Kernel Tracing

Kubeshark offers tracing kernel-space and user-space functions using 🐝 eBPF (Extended Berkeley Packet Filter). eBPF is an in-kernel virtual machine running programs passed from user space. It's first introduced into Linux kernel with version 4.4 and quite matured since then.

Capturing Unencrypted TLS Traffic

To deploy Kubeshark with TLS sniffing capability, simply add the --tls option:

kubeshark deploy --tls -n sock-shop

Kubeshark supports the most commonly used encryption/decryption library OpenSSL library and Go's custom implementation crypto/tls package.

TLS Traffic Example

Note: By default the capturing unencrypted TLS traffic. Only --tls option enables it.

OpenSSL

It attaches uprobe(s) to SSL_read and SSL_write functions to simply reads the incoming unencrypted response and outgoing encrypted request in any TLS/SSL connection.

Languages like Python, Java, PHP, Ruby and Node.js use OpenSSL library for their encryption/decryption work. So, pretty much any program or service that's doing encrypted communication (using TLS) falls into this category.

Go

Go is a little bit complicated than OpenSSL but the basic principle is the same.

Go language has two ABIs; ABI0 and ABIInternal. We support both amd64 and arm64 so that translates into a good number of offsets to handle.

We basically probe the crypto/tls.(*Conn).Read and crypto/tls.(*Conn).Write just like OpenSLL's SSL_read / SSL_write . In addition to that we dissamble the targetted Go binaries using Capstone to get the offsets of ret mnemonics. Because uretprobe does not function properly in Go thanks to its unique ABI.

Lastly, we keep track of the Goroutine ID by using some offsets that we learn by looking at the DWARF table.

Kernel

We kprobe certain tracepoints in the kernel for reasons like doing address resolution (by learning IP and port number for both source and destination) or for matching request-response pairs.

While the methods explained in here sounds a little bit complicated the TLS sniffer has little to no performance impact thanks to efficient eBPF in-kernel virtual machine of Linux and our carefully written C code. In any case, Linux kernel does not allow injecting a huge number of instructions for probing purposes. So, the kernel's itself guarantee no slowdown or crash.

Microservice Tracing

Work in Progress...