Exposed Kubernetes Clusters, Kubelet Ports Can Be Abused in Cyberattacks

Kubernetes clusters provide a scalable and resilient backbone to many modern Internet-facing applications. However, if adversaries can access the nodes in those clusters, they essentially take over your infrastructure. They can compromise the integrity of your systems and hijack the infrastructure and use it for their own purposes.

Recent data from Shodan shows 243,469 Kubernetes clusters that are publicly exposed. These clusters also exposed port 10250, used by the kubelet (the agent that runs on each node and ensures that all containers are running in a pod) as a default setting. Attackers could potentially use the kubelet API as an entry point in targeting Kubernetes clusters to mine for cryptocurrency.

Trend Micro researcher Magno Logan looked at how cybercriminals could abuse these clusters and exposed kubelet ports.

First, there is the problem of sensitive information leakage by returning data on the running pods on the node.

In addition, since the kubelet API is exposed, there is another endpoint /run that would allow an attacker to execute commands inside the running pods of the cluster just by sending a POST request to the specific pods and using the parameter cmd to execute the desired shell commands. Trend Micro says threat actor TeamTNT performed multiple /run commands in just this manner to compromise multiple clusters last year. This technique can make things easier for attackers to take over clusters, Logan says in the report.

Logan called it "very concerning" that hackers could use the kubelet API as an entry point when targeting Kubernetes clusters.

"Those 600 kubelets we've found to be completely exposed and without authentication or authorization could easily be compromised via simple API requests," he said. "That would allow an attacker to execute commands on the pods running inside that node, most of the time to mine cryptocurrencies."

Exposed Kubelets Leave Door Open to Malicious Actors

According to Michael Isbitski, director of cybersecurity strategy for Sysdig, when Kubernetes clusters or kubelets are improperly exposed or don't enforce proper access control, it leaves the door open for a wide range of malicious activity.

"Attackers can potentially harvest sensitive data being transmitted within the cluster, spin-up new workloads, reconfigure elements of a node, disable access controls, erase audit trails, add vulnerable dependencies, bootstrap malicious cryptominers, and more," he says.

Isbitski notes that many Kubernetes configurations are secure by default with current platform offerings, but some organizations may be sitting on old or misconfigured deployments.

He points out organizations also sometimes inadvertently override secure defaults to get a cluster to an operational state without understanding the potential security risks.

"We've seen issues with vulnerabilities in runtime components, which can result in container escapes and lateral movement within networks if attackers are successful in their exploitation attempts," he says.

Practice Defense In-Depth, Zero Trust

Matt Dupre, director of software engineering at Tigera, a provider of security and observability for containers, Kubernetes, and cloud, points out that sufficiently privileged access to the kubelet amounts to a complete compromise of that host and potentially any other workloads running on it.

Access to the Kubernetes API has the same potential impact: Admin access essentially provides complete control of the cluster and everything in it.

He notes that while the security risk is significant, an overwhelming majority of the clusters that accepted connections from the Internet rejected the requests due to lack of authentication or authorization.

"Given that, there are two concerns: firstly, that you fall in that misconfigured 613 clusters, or that a new critical vulnerability that bypasses authn or authz is found, and this would be a very significant vulnerability," Dupre says. "Organizations' internal APIs are probably a bigger worry in practice."

He advises practicing defense in depth by following zero-trust principles and not allowing connections to your kubelets from unknown sources, such as the Internet.

"Additionally, you could port-scan your infrastructure and investigate any responses," he adds. "Keeping careful control of access tokens is always important — they should never be published, and you should have processes in place to ensure that they and other secrets are stored properly."

Avoid Exposing the Kubelet Default Port

As a basic kubelet security practice, Logan says organizations shouldn’t expose their kubelet port (10250 by default) to the Internet.

"If you need to do that, at least enable kubelet authentication and authorization on the kubelet API to avoid attackers being able to perform requests to the API and receive the 401 – Unauthorized response," he adds.

Mark Lambert, vice president of products at ArmorCode, an application security provider, says when deploying these types of systems, take a "zero-trust mindset" and remember that the default configurations are usually set up for ease of use, not security.

"This means you need to pay close attention to configuration files, disable features you are not using, change default ports, and minimize information leakage so that hackers cannot gain insight that could provide them another point of attack," he says.

Finally, all this needs to be operationalized as part of your application security program, and development teams must be engaged early, as they play a key role in building security into the design of the application from the start.

Besides enabling the kubelet authentication and authorization on the kubelet API, Logan advises restricting the kubelet permissions via the least privilege principle and periodically rotating the kubelet certificates to reduce the attack surface.

"Organizations should also investigate tools for runtime protection such as Falco to prevent and alert when there are suspicious execution happening inside their containers," he says.

Constantly Analyze IaaC, Monitor Clusters in Runtime

Isbitski says native capabilities and tooling from cloud providers and Kubernetes platform providers can provide a starting point for keeping kubelets protected.

He adds that security teams must continuously analyze the infrastructure-as-code used to configure and operate clusters, scan dependencies used by workloads, and monitor clusters in runtime to detect malicious activity, such as when an attacker attempts unauthorized access to the Kubernetes APIs.

"Appropriate access control should also be implemented at multiple points of a cluster," he says. "Native capabilities like Kubernetes network policy also help with restricting communication within a cluster and enforce zero trust principles."

Isbitski points out the Kubernetes control plane is also multilayered when working with managed Kubernetes.

In those scenarios, security teams should also continuously validate the cloud tenant configurations, along with IAM policies, for misconfigurations and excessive permissions.