Tracing Encrypted Traffic
Using eBPF to sniff encrypted traffic
Kubeshark can sniff the encrypted traffic (TLS) in your cluster using eBPF without actually doing decryption. It hooks into entry and exit points in certain functions inside the OpenSSL library and Go’s crypto/tls package.
Kubeshark offers tracing kernel-space and user-space functions using eBPF (Extended Berkeley Packet Filter). eBPF is an in-kernel virtual machine running programs passed from user space. It’s first introduced into Linux kernel with version 4.4 and quite matured since then.
Capturing Unencrypted TLS Traffic
To deploy Kubeshark with TLS sniffing capability, simply add the --tls
option:
kubeshark tap --tls -n sock-shop
Kubeshark supports the most commonly used encryption/decryption library OpenSSL library and Go’s custom implementation crypto/tls package.
Note: By default the capturing unencrypted TLS traffic. Only
--tls
option enables it.
OpenSSL
It attaches uprobe(s)
to SSL_read
and SSL_write
functions to simply reads the incoming unencrypted response and outgoing encrypted request in any TLS/SSL connection.
Languages like Python, Java, PHP, Ruby and Node.js use OpenSSL library for their encryption/decryption work. So, pretty much any program or service that’s doing encrypted communication (using TLS) falls into this category.
Go
Go is a little bit more complicated than OpenSSL but the basic principle is the same.
Go language has two ABIs; ABI0 and ABIInternal. We support both amd64 and arm64 so that translates into a good number of offsets to handle.
We basically probe the crypto/tls.(*Conn).Read
and
crypto/tls.(*Conn).Write
just like OpenSLL’s SSL_read
/ SSL_write
.
In addition to that we dismantle the targeted Go binaries using Capstone to get the offsets of ret
mnemonics.
Because uretprobe
does not function properly in Go thanks to its unique ABI.
Lastly, we keep track of the Goroutine ID by using some offsets that we learn by looking at the DWARF table.
Kernel
We kprobe
certain tracepoints in the kernel for reasons
like doing address resolution (by learning IP and port number for both source and destination) or for matching
request-response pairs.
While the methods explained in here sounds a little bit complicated the TLS sniffer has little to no performance impact thanks to the efficient eBPF in-kernel virtual machine of Linux and our carefully written C code. In any case, Linux kernel does not allow injecting a huge number of instructions for probing purposes. So, the kernel itself guarantees no slowdown or crash.