Rootkit Turns Kubernetes from Orchestration to Subversion

As software development focuses on continuous integration and deployment, orchestration platforms like Kubernetes have taken off, but that popularity has put them in attackers' crosshairs.

Most successful attacks — at least those publicly reported — have led to the deployment of cryptomining-focused containers, basically stealing cloud compute resources from businesses to power cryptocurrency mining. Yet the attacks could be much worse — infecting Kubernetes clusters with rootkits would result in collections of containers controlled by attackers, says Nicholas Lang, a security researcher with cloud-infrastructure security firm Sysdig, who will present a prototype rootkit at Black Hat Europe next month.

The successful compromise of a Kubernetes cluster by a rootkit could allow an attacker to hide malicious containers on the system, for example. The rootkit can hide other containerized payloads and take more sophisticated actions escaping notice in the system, because they are hidden from the operating system, he says.

"Even though it interacts with the kernel to get these containers up and in place, after that ... the rootkit is able to hide these containerized payloads," Lang says. "The rootkit is part of the initial payload ... and then, you know, future stages will do more sophisticated things in secret because they're hidden from the operating system by the rootkit."

Kubernetes is a popular way of automating the configuration, deployment, and management — that is, "orchestration" — of containers, virtualized software environments that can run a wide variety of workloads, from servers to applications to software-defined networks. As such, the technology is critical for cloud applications in today's fast-moving world of software development and deployment.

Vulnerabilities and misconfigurations are top concerns for Kubernetes. Source: Red Hat

For the same reason, however, attackers have targeted the infrastructure. In February, an attacker compromised a misconfigured Kubernetes cluster, first installing cryptojacking containers and then stealing intellectual property and sensitive data. A month earlier, Microsoft researchers discovered that the Kinsing malware had started targeting poorly configured database containers on Kubernetes platforms.

Kubernetes Under Attack

The spate of attacks have software firms worried. Two-thirds of companies (67%) have delayed or slowed down an application deployment due to a security concern with Kubernetes, according to Red Hat's "2023 State of Kubernetes Security Report."

While attackers have exploited vulnerabilities in Kubernetes infrastructure, misconfigured applications running in containers are, by far, the most common way that the orchestration platform is compromised, says Sysdig's Lang.

"A misconfigured Web server or Web application gives the attacker shell access to that virtual machine or inside that container," he says. "Depending on the attacker's sophistication level, they'll either realize that they're inside of a container or a virtual machine, or whatever, and try to escape to the host, or they realize that they're in a Kubernetes environment by doing a little bit of poking and prodding."

While a specific Linux kernel rootkit, known as Diamorphine, has occasionally been used to compromise Kubernetes clusters, Kubernetes-focused rootkits have not yet become popular.

Lang argues that will change, and as a view into the future, he and another security researcher, Andrew Hughes of Narf Industries, plan to demonstrate their own Kubernetes rootkit at the Black Hat Europe Conference in December. "The real change is the attackers learning that Kubernetes is increasingly common in the cloud, and how to deal with it and how to get around it, and how to make use of it even," he says.

Kubernetes Admins Need Visibility

Typically, a victim would have to see a rootkit getting loaded or a vulnerability being exploited by the attacker to catch an attack on a Kubernetes cluster, Lang says. However, admins can also look out for kernel modules that get loaded during runtime, which really should not happen in a production setting, he says.

"These systems don't really do a whole lot of crazy stuff, so if you see [kernel modules loading] inside a container or on host that runs containers, you can be pretty confident something bad is going on," he says. "Otherwise, catching it is very, very difficult, because a lot of it happens in user space or in an application layer where you don't have a lot of deep insight."

Admins should also ask their red team to conduct a group exercise, working against defenders to attack, and then with defenders to analyze the attack — a process referred to as purple teaming.

Sysdig runs its own honeypot, exposing Kubernetes ports to potential attacks, and typically, the first probes come quickly, Lang says.

"Within minutes, sometimes seconds, it's already getting attacked," he says. "So purple teaming is how you will find and close your gaps, and then not putting Kubernetes on the Internet in the first place is a great way to not get attacked."