Researcher Finds etcd Servers Leak Passwords Like a Sieve

Kubernetes is an open source system for automating deployment, scaling and management of containerized applications in the cloud. First developed by Google, the orchestration software has had some serious security problems show up lately.

Now, there's another problem coming to light, but it's due to one of Kubernetes components: etcd. (See Cybercriminals Using Kubernetes, Docker to Bitcoin Mine.)

etcd is an open source distributed key value store written in the Google Go language that provides a "reliable" way to store data across a cluster of machines. One can read and write values with curl and other HTTP libraries. It's sort of like an expandable database that changes with how the clusters that are in use change.

(Source: Geralt via Pixabay)

Security researcher Giovanni Collazo got curious when he saw in some etcd documentation that:

"etcd before 2.1 was a completely open system; anyone with access to the API could change keys. In order to preserve backward compatibility and upgradability, this feature is off by default."

This made him wonder if the kind of problems that had been affecting MongoDB --misconfiguration that could stop authentication -- was in play here. It was.

As he wrote in his blog on the issue, Collazo did a simple Shodan search and found 2,284 etcd servers on the open Internet. From these servers he was able to download "passwords for databases of all kinds, AWS secret keys, and API keys and secrets for a bunch of services. Also came across a few certificates, you name it."

The data included the passwords and keys required for cms_admin, mysql_root and postgres server infrastructure.

So, in conclusion, etcd can be easily misconfigured and then leak secrets like a sieve.

But -- and this ties the whole thing together -- etcd is used as Kubernetes' backing store for all cluster data. Usually that data in the backing store will include secrets encoded as base64 strings. An attacker can get into one of these poorly configured servers and come back with stuff that will compromised the entire Kubernetes cluster.

Using the etcd API, an attacker could also write to the database. This might invalidate the cluster data stored in it, or serve as a way to hide data that was taken in another attack.

In an email to Security Now, Zohar Alon, CEO of security company Dome9, wrote:

"There were two levels of security failure here -- one, no password protection for the service by default, and two, servers with ports exposed to the whole world when they should have been locked down."

Kubernetes dependency upon etcd is a non-obvious vulnerability, but a serious one. If this kind of system is deployed by an organization, they would do well to ascertain that it has been locked down by them before someone gets the idea to attack it.

Related posts:

— Larry Loeb has written for many of the last century's major "dead tree" computer magazines, having been, among other things, a consulting editor for BYTE magazine and senior editor for the launch of WebWeek.