Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFE: trace TLS in container #65

Closed
vincentmli opened this issue May 30, 2022 · 28 comments
Closed

RFE: trace TLS in container #65

vincentmli opened this issue May 30, 2022 · 28 comments
Labels
good first issue Good for newcomers

Comments

@vincentmli
Copy link
Contributor

vincentmli commented May 30, 2022

Hi,

for example I have a netshoot pod running in kubernetes, when I run curl

kubectl exec -it netshoot-hostnetwork  -- curl -k -v https://10.1.34.88/

I want to trace the TLS connection from the curl from the netshoot-hostnetwork pod, the curl in netshoot-hostnetwork pod has libssl below in the pod namespace

kubectl exec -it netshoot-hostnetwork  -- ldd /usr/bin/curl
	/lib/ld-musl-x86_64.so.1 (0x7f932eda0000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0x7f932ece5000)
	libz.so.1 => /lib/libz.so.1 (0x7f932eccb000)
	libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f932eda0000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0x7f932eca6000)
	libssl.so.1.1 => /lib/libssl.so.1.1 (0x7f932ec25000)
	libcrypto.so.1.1 => /lib/libcrypto.so.1.1 (0x7f932e9a4000)
	libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0x7f932e998000)
	libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0x7f932e975000)

is this possible? or in general, could we improve ecapture to capture container TLS traffic?

@vincentmli
Copy link
Contributor Author

I could do this:

find the netshoot docker ID

docker ps | grep -w netshoot
b37ffd7a8341   nicolaka/netshoot           "/bin/sleep 3600"        50 minutes ago      Up 50 minutes                k8s_netshoot_netshoot-hostnetwork_default_751666e9-5a23-4da1-952f-379288b47f97_0

docker inspect the ID

docker inspect b37ffd7a8341  | grep '"MergedDir"'
                "MergedDir": "/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged",

find libssl

find /var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged -name "libssl*"

/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/lib/libssl.so.1.1
/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/usr/lib/libssl.so.1.1

use ecapture with the correct libssl path

ecapture tls --libssl="/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/lib/libssl.so.1.1" --hex
2022/05/30 14:02:27 pid info :2825032
2022/05/30 14:02:27 start to run EBPFProbeOPENSSL module
2022/05/30 14:02:27 start to run EBPFProbeGNUTLS module
2022/05/30 14:02:27 HOOK type:2, binrayPath:/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/lib/libssl.so.1.1
2022/05/30 14:02:27 libPthread so Path:/lib64/libpthread.so.0
2022/05/30 14:02:27 target all process. 
2022/05/30 14:02:27 start to run EBPFProbeNSPR module
2022/05/30 14:02:27 stat /usr/lib/libnspr4.so: no such file or directory
2022/05/30 14:02:27 HOOK type:2, binrayPath:/lib64/libgnutls.so.30
2022/05/30 14:02:27 target all process. 

execute the curl from netshoot pod

kubectl exec -it netshoot-hostnetwork  -- curl -k -v https://10.1.34.88/
*   Trying 10.1.34.88:443...
* Connected to 10.1.34.88 (10.1.34.88) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=US; ST=WA; L=Seattle; O=MyCompany; OU=IT; CN=localhost.localdomain; emailAddress=root@localhost.localdomain
*  start date: Oct  1 19:29:04 2020 GMT
*  expire date: Sep 29 19:29:04 2030 GMT
*  issuer: C=US; ST=WA; L=Seattle; O=MyCompany; OU=IT; CN=localhost.localdomain; emailAddress=root@localhost.localdomain
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET / HTTP/1.1
> Host: 10.1.34.88
> User-Agent: curl/7.83.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: BigIP
* HTTP/1.0 connection set to keep alive
< Connection: Keep-Alive
< Content-Length: 10
< 
* Connection #0 to host 10.1.34.88 left intact
 IT WORKS

the ecapture output:

2022/05/30 14:03:34 PID:2825502, Comm:curl, TID:2825502, Version:TLS1_2_VERSION, Send 74 bytes to [ADDR_NOT_FOUND], Payload:
GET / HTTP/1.1
Host: 10.1.34.88
User-Agent: curl/7.83.1
Accept: */*


2022/05/30 14:03:34 PID:2825502, Comm:curl, TID:2825502, Version:TLS1_2_VERSION, Recived 88 bytes from [ADDR_NOT_FOUND], Payload:
HTTP/1.0 200 OK
Server: BigIP
Connection: Keep-Alive
Content-Length: 10

 IT WORKS 

@cfc4n
Copy link
Member

cfc4n commented May 31, 2022

yes, you are right. eh....Is your problem solved?

@vincentmli
Copy link
Contributor Author

yes it is resolved, thanks for this great project :)

@cfc4n cfc4n added the good first issue Good for newcomers label May 31, 2022
@cfc4n cfc4n pinned this issue May 31, 2022
@cfc4n cfc4n changed the title RFE: trace TLS in container? RFE: trace TLS in container May 31, 2022
@BurlyLuo
Copy link

BurlyLuo commented May 31, 2022

yes it is resolved, thanks for this great project :)

Thanks the test.
Can we have a sample to use this tool into the container?

@cfc4n
Copy link
Member

cfc4n commented May 31, 2022

@BurlyLuo That is a good idea. can you send a PR for it? or an article?

@vincentmli
Copy link
Contributor Author

hm, this brings me thoughts about multi tenant cloud environment, that someone could run ecapture as container and gain privilege on the node and sniff other tenant containers TLS connection, is that possible ?

@cfc4n
Copy link
Member

cfc4n commented May 31, 2022

Of course, if the container is authorized with the SYS_ADMIN permission, then it can obtain the communication plaintext of all networks on this host.

So, That is an other topic about eBPF security on runtime.

ref: https://github.com/ehids/ebpf-slide/blob/master/security/us-21-With-Friends-Like-EBPF-Who-Needs-Enemies.pdf
demo: https://www.youtube.com/watch?v=zgsHc8apFGs
posts on chinese: https://www.cnxct.com/evil-use-ebpf-and-how-to-detect-ebpf-rootkit-in-linux/?f=ecapture-github

@vincentmli
Copy link
Contributor Author

I wonder if https://github.com/cilium/tetragon is able to detect this scenario, I assume your ehids could detect this too

@cfc4n
Copy link
Member

cfc4n commented Jun 1, 2022

tetragon is the learning objective of ehids.

@vincentmli
Copy link
Contributor Author

I am not familiar with docker, can we create a docker file for ecapture to run ecapture in container/pod ?

@cfc4n
Copy link
Member

cfc4n commented Jun 1, 2022

I think It is not necessary. via: #23

@vincentmli
Copy link
Contributor Author

ok, good, just one more question, is it possible to give --libssl argument multiple libssl path so ecapture could capture multiple TLS connections for multiple processes/commands ? I guess we can run multiple ecapture commands to achieve that, just wonder :)

@cfc4n
Copy link
Member

cfc4n commented Jun 1, 2022

1, eCapture can capture all process TLS paintext who use the same libssl.so default. and can use pid arg to filter.
2, Can run multiple eCapture with every libssl path to achieve that.
3, Of course, an eCapture process can hook multiple ssl/tls SO file, but I don't think this is a necessary requirement.

@vincentmli
Copy link
Contributor Author

ah, right I got 1), I am thinking in k8s pod/container scenario that each pod/container might have their own libssl copy, yes 2) can achieve that.

@vincentmli
Copy link
Contributor Author

by the way, I recorded a short video playing with ecapture https://youtu.be/Au1YeB0nz3g

@cfc4n
Copy link
Member

cfc4n commented Jun 1, 2022

Good job. I'll create User Manual WIKI tomorrow with your video, thanks. 😊

@vincentmli
Copy link
Contributor Author

vincentmli commented Jun 3, 2022

Hi @cfc4n sorry to bother you again, I added the ecapture binary in netshoot pod like vincentmli/netshoot@72633ba so I can run ecapture from netshoot pod in k8s, the netshoot pod yaml file has privilege permission like below

apiVersion: v1
kind: Pod
metadata:
 name: netshoot-ecap
spec:
 hostPID: true
 hostNetwork: true
 nodeSelector:
    dedicated: master
 containers:
   - name: netshoot-ecap
     image: vli39/netshoot:ecap
     command:
       - /bin/sleep
       - "3600"
     volumeMounts:
       - mountPath: /mnt
         name: host-slash
     securityContext:
       privileged: true
 volumes:
   - name: host-slash
     hostPath:
       path: /
       type: ''

you can see I mounted the node root / in netshoot pod /mnt to access the host libssl, but it errors out with "lstat /etc/ld.so.conf: no such file or directory", I wonder why ecapture is trying to check host /etc/ld.so.conf, ecapture is statically build, I suspect the host libssl requiring host /etc/ld.so.conf, but I can't mount host /etc to netshoot pod /etc, wonder if you have better idea :)

kubectl exec -it netshoot-ecap -- /bin/bash
bash-5.1# ls -l /mnt/lib64/libssl*
lrwxrwxrwx    1 root     root            16 Mar 30  2021 /mnt/lib64/libssl.so -> libssl.so.1.1.1g
lrwxrwxrwx    1 root     root            16 Mar 30  2021 /mnt/lib64/libssl.so.1.1 -> libssl.so.1.1.1g
-rwxr-xr-x    1 root     root        615576 Mar 30  2021 /mnt/lib64/libssl.so.1.1.1g
-rwxr-xr-x    1 root     root        394632 Dec 18  2020 /mnt/lib64/libssl3.so

bash-5.1# ecapture tls --libssl="/mnt/lib64/libssl.so.1.1.1g" --hex
2022/06/03 14:51:29 pid info :1391071
2022/06/03 14:51:29 start to run EBPFProbeOPENSSL module
2022/06/03 14:51:29 lstat /etc/ld.so.conf: no such file or directory
2022/06/03 14:51:29 invalid argument

@cfc4n
Copy link
Member

cfc4n commented Jun 4, 2022

eh, eCapture will find pthread.so to capture IP ADDRESS default , and you can set pthread.so argument with pthread falg.

@vincentmli
Copy link
Contributor Author

ecapture is statically built

ldd /usr/local/bin/ecapture
	not a dynamic executable

but are you saying eCapture dynamically linked with pthread.so? how do I specify ecapture with pthread flag? sorry I am not following you :)

@BurlyLuo
Copy link

BurlyLuo commented Jun 5, 2022

eh, eCapture will find pthread.so to capture IP ADDRESS default , and you can set pthread.so argument with pthread falg.

bash-5.1# ecapture tls --libssl="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libssl.so.1.1" --pthread="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libpthread.so.0" 
2022/06/05 04:36:54 pid info :10668
2022/06/05 04:36:54 start to run EBPFProbeOPENSSL module
2022/06/05 04:36:54 start to run EBPFProbeGNUTLS module
2022/06/05 04:36:54 invalid argument
bash-5.1# 
bash-5.1# ecapture tls --libssl="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libssl.so.1.1" --pthread="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libpthread.so.0"  -h
NAME:
        tls - alias name:openssl , use to capture tls/ssl text content without CA cert.

USAGE:
        ecapture tls [flags]

DESCRIPTION:
        use eBPF uprobe to capture process event data, not used libpcap.
        Can used to trace, debug, database audit, security event aduit etc.

OPTIONS:
      --curl=""         curl or wget file path, use to dectet openssl.so path, default:/usr/bin/curl
      --firefox=""      firefox file path, default: /usr/lib/firefox/firefox.
      --gnutls=""       libgnutls.so file path, will automatically find it from curl default.
  -h, --help[=false]    help for tls
      --libssl=""       libssl.so file path, will automatically find it from curl default.
      --nspr=""         libnspr44.so file path, will automatically find it from curl default.
      --pthread=""      libpthread.so file path, use to hook connect to capture socket FD.will automatically find it from curl.
      --wget=""         wget file path, default: /usr/bin/wget.

GLOBAL OPTIONS:
  -d, --debug[=false]   enable debug logging
      --hex[=false]     print byte strings as hex encoded strings
  -p, --pid=0           if pid is 0 then we target all pids

bash-5.1# 

@cfc4n
Copy link
Member

cfc4n commented Jun 5, 2022

ecapture is statically built

ldd /usr/local/bin/ecapture
	not a dynamic executable

but are you saying eCapture dynamically linked with pthread.so? how do I specify ecapture with pthread flag? sorry I am not following you :)

sure, ecapture is statically built . but eCapture hook connect()function to capture IP ADDRESS ,sometimes connect function was define inpthread.so. so need to search the path.

https://github.com/ehids/ecapture/blob/ed5ebf13bb12873292625d550ffa902c96124388/kern/openssl_kern.c#L285-L290

https://github.com/ehids/ecapture/blob/ed5ebf13bb12873292625d550ffa902c96124388/user/probe_openssl.go#L157-L161

@cfc4n
Copy link
Member

cfc4n commented Jun 5, 2022

sometimes , defined in /lib64/libc.so.6 , you can use ldd which curl|grep libc.so|awk '{print $3}'|xargs objdump -T |grep connect to confirm it.

[root@VM-16-13-centos bin]# ldd `which curl`|grep libc.so|awk '{print $3}'|xargs objdump -T |grep connect
0000000000112f90  w   DF .text	000000000000009b  GLIBC_2.2.5 __connect
0000000000112f90  w   DF .text	000000000000009b  GLIBC_2.2.5 connect

@BurlyLuo @vincentmli
It will skip directory search if lib path args was set.
image

@vincentmli
Copy link
Contributor Author

ah, sorry I missed the output libPthread so Path:/lib64/libpthread.so.0 when run ecapture normally

 ecapture tls  --libssl=/lib64/libssl.so.1.1.1g
2022/06/05 10:53:59 pid info :3232826
2022/06/05 10:53:59 start to run EBPFProbeOPENSSL module
2022/06/05 10:53:59 start to run EBPFProbeGNUTLS module
2022/06/05 10:53:59 stat /usr/lib/libgnutls.so.30: no such file or directory
2022/06/05 10:53:59 start to run EBPFProbeNSPR module
2022/06/05 10:53:59 stat /usr/lib/libnspr4.so: no such file or directory
2022/06/05 10:53:59 HOOK type:2, binrayPath:/lib64/libssl.so.1.1.1g
2022/06/05 10:53:59 libPthread so Path:/lib64/libpthread.so.0 <====MISSED THIS 
2022/06/05 10:53:59 target all process. 

but still error when I run ecapture in container with --pthread=/mnt/lib64/libpthread.so.0, it looks to me when putting ecapture in container, for some reason, ecapture could not find the correct pthread lib even with --pthread argument, and still try to locate /etc/ld.so.conf

kubectl exec -it netshoot-ecap -- /bin/bash
bash-5.1# 

bash-5.1# ls -l /mnt/lib64/libpth*
-rwxr-xr-x    1 root     root        320504 Jul 20  2020 /mnt/lib64/libpthread-2.28.so
lrwxrwxrwx    1 root     root            27 Jul 20  2020 /mnt/lib64/libpthread.so -> ../../lib64/libpthread.so.0
lrwxrwxrwx    1 root     root            18 Jul 20  2020 /mnt/lib64/libpthread.so.0 -> libpthread-2.28.so

bash-5.1# ecapture tls --libssl=/mnt/lib64/libssl.so.1.1.1g --pthread=/mnt/lib64/libpthread.so.0
2022/06/05 15:01:40 pid info :3237812
2022/06/05 15:01:40 start to run EBPFProbeOPENSSL module
2022/06/05 15:01:40 start to run EBPFProbeGNUTLS module
2022/06/05 15:01:40 lstat /etc/ld.so.conf: no such file or directory
2022/06/05 15:01:40 invalid argument

@vincentmli
Copy link
Contributor Author

tried create /etc/ld.so.conf with mounted host / in container, still can't find connect I guess

bash-5.1# echo "/mnt/lib64" > /etc/ld.so.conf
bash-5.1# cat /etc/ld.so.conf
/mnt/lib64

bash-5.1# ecapture tls --libssl="/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libssl.so.1.0.2k"  --hex
2022/06/05 15:32:04 pid info :3257576
2022/06/05 15:32:04 start to run EBPFProbeOPENSSL module
2022/06/05 15:32:04 start to run EBPFProbeGNUTLS module
2022/06/05 15:32:04 invalid argument

bash-5.1# ecapture tls --pthread="/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libpthread.so.0" --libssl="/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libssl.so.1.0.2k"  --hex
2022/06/05 15:29:01 pid info :3255607
2022/06/05 15:29:01 start to run EBPFProbeOPENSSL module
2022/06/05 15:29:01 start to run EBPFProbeGNUTLS module
2022/06/05 15:29:01 invalid argument

@cfc4n
Copy link
Member

cfc4n commented Jun 5, 2022

may be it's a bug. can you send a new issue for it?

@vincentmli
Copy link
Contributor Author

sure, will do, appreciate the help!

@vincentmli
Copy link
Contributor Author

opened #69

@vincentmli
Copy link
Contributor Author

close this since question is answered by @cfc4n

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

3 participants