Tracing Encrypted Traffic

Using eBPF to sniff encrypted traffic

Kubeshark can sniff the encrypted traffic (TLS) in your cluster using eBPF without actually doing decryption. It hooks into entry and exit points in certain functions inside the OpenSSL library and Go’s crypto/tls package.

Kubeshark offers tracing kernel-space and user-space functions using eBPF (Extended Berkeley Packet Filter). eBPF is an in-kernel virtual machine running programs passed from user space. It’s first introduced into Linux kernel with version 4.4 and quite matured since then.

Capturing Unencrypted TLS Traffic

To deploy Kubeshark with TLS sniffing capability, simply add the --tls option:

kubeshark tap --tls -n sock-shop

Kubeshark supports the most commonly used encryption/decryption library OpenSSL library and Go’s custom implementation crypto/tls package.

TLS Traffic Example

Note: By default the capturing unencrypted TLS traffic. Only --tls option enables it.

OpenSSL

It attaches uprobe(s) to SSL_read and SSL_write functions to simply reads the incoming unencrypted response and outgoing encrypted request in any TLS/SSL connection.

Languages like Python, Java, PHP, Ruby and Node.js use OpenSSL library for their encryption/decryption work. So, pretty much any program or service that’s doing encrypted communication (using TLS) falls into this category.

Go

Go is a little bit more complicated than OpenSSL but the basic principle is the same.

Go language has two ABIs; ABI0 and ABIInternal. We support both amd64 and arm64 so that translates into a good number of offsets to handle.

We basically probe the crypto/tls.(*Conn).Read and crypto/tls.(*Conn).Write just like OpenSLL’s SSL_read / SSL_write . In addition to that we dismantle the targeted Go binaries using Capstone to get the offsets of ret mnemonics. Because uretprobe does not function properly in Go thanks to its unique ABI.

Lastly, we keep track of the Goroutine ID by using some offsets that we learn by looking at the DWARF table.

Kernel

We kprobe certain tracepoints in the kernel for reasons like doing address resolution (by learning IP and port number for both source and destination) or for matching request-response pairs.

While the methods explained in here sounds a little bit complicated the TLS sniffer has little to no performance impact thanks to the efficient eBPF in-kernel virtual machine of Linux and our carefully written C code. In any case, Linux kernel does not allow injecting a huge number of instructions for probing purposes. So, the kernel itself guarantees no slowdown or crash.