Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

News & Commentary

12:05 PM
Curtis Franklin
Curtis Franklin
Curt Franklin

Cloudflare Bleeds Bad News – & Good

A slip of the fingers brings privacy concerns for millions and lessons in how to handle a security incident.

"For the want of a nail the shoe was lost, For the want of a shoe the horse was lost, For the want of a horse the rider was lost, For the want of a rider the battle was lost, For the want of a battle the kingdom was lost, And all for the want of a horseshoe-nail."
— Benjamin Franklin

Substitute the ">" symbol for an old-fashioned nail, and you have a synopsis of the latest privacy catastrophe, dubbed "Cloudbleed" by security researchers. How bad is Cloudbleed? The advice to users echoed around the Internet gives a hint of the breach's scope:

Change your passwords. All of them.

What happened to cause the problem and how it has been addressed each carry instructions for enterprise security and IT professionals. What happened is simple, and the sort of coding error that can be frustratingly difficult to catch absent the most rigorous testing protocols. In a single line of code, the intended ">=" operator was typed "==". As a result, certain operations were able to fill a buffer and keep right on writing, planting data across the system.

The flaw
According to Tavis Ormandy, a researcher with Google Project Zero, that planted data can include cookies, private messages, IP addresses, and passwords. The worst part? In a blog post about the event (made public only after the vulnerability had been repaired and remediated), Ormandy wrote, "PII [personally identifiable information] was actively being downloaded by crawlers and users during normal usage, they just didn't understand what they were seeing."

Cloudflare is a service provider with a client list that includes Uber, OKCupid, Fitbit, and various financial institutions. In addition to basic network performance services that include DNS and load balancing, Cloudflare provides security capabilities such as SSL and DDoS mitigation. Because Cloudflare provides the service to businesses rather than consumers, it's nearly impossible for an individual to know whether their personal information was exposed. And it's absolutely impossible to know whether any exposed information was actually grabbed by criminals -- until, of course, some form of identity theft takes place. Hence, the advice to change all your passwords. Now.

The problem was compounded by the way that Cloudflare operates. Cloudflare's data can be crawled and cached by services such as Google. This means that personal data was not just exposed, it was exposed in a persistent manner. That's a lot of bad news wrapped up in a single wayward character. Fortunately, there is some good news in the story to balance a bit of the bad.

Rapid response
When Ormandy discovered the issue, he immediately contacted Cloudflare using Twitter (in, as one commenter noted, the sort of message that should be absolutely horrifying to anyone on a corporate security team). In very short order, Cloudflare security personnel had reproduced the issue and shut down the services making use of the affected code. According to the company, they then put teams in the US and UK on 12-hour shifts, handing code off between the teams to keep efforts to find and remediate the problem going 24 hours a day.

Cloudflare reports that all services that included the problem were disabled globally within roughly three hours of notification. Figuring out precisely where in the code the problem lay, and how it could be repaired, took a bit longer. The company's blog post on the issue (referenced above) goes into great detail about where the problem was, what its effects were, and why it had not previously appeared. The level of detail in the disclosure is part of the good news surrounding the vulnerability; in speaking with several security researchers and IT professionals, all said that this disclosure is a model of what they would like to see in future industry events. All felt that this level of disclosure is likely to assuage customer fears and reassure regulators that all proper steps have been taken.

The final step in remediation was Cloudflare working with search engines to clear caches of released information. As the company wrote in its disclosure blog post, "With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains. The leaked memory has been purged with the help of the search engines."

Six days after the initial contact from Ormandy, Cloudflare disclosed the incident to the public. The vulnerability had existed for approximately four months and the number of customer records revealed to criminals is unknown. What is known is that this will likely be used as an example of the white-hat research system working as intended; independent discovery of an issue, verification by the affected company, rapid repair and remediation, with speedy, transparent disclosure to the public.

— Curtis Franklin, Security Editor, Light Reading

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
COVID-19: Latest Security News & Commentary
Dark Reading Staff 7/2/2020
Ripple20 Threatens Increasingly Connected Medical Devices
Kelly Sheridan, Staff Editor, Dark Reading,  6/30/2020
DDoS Attacks Jump 542% from Q4 2019 to Q1 2020
Dark Reading Staff 6/30/2020
Register for Dark Reading Newsletters
White Papers
Current Issue
How Cybersecurity Incident Response Programs Work (and Why Some Don't)
This Tech Digest takes a look at the vital role cybersecurity incident response (IR) plays in managing cyber-risk within organizations. Download the Tech Digest today to find out how well-planned IR programs can detect intrusions, contain breaches, and help an organization restore normal operations.
Flash Poll
The Threat from the Internetand What Your Organization Can Do About It
The Threat from the Internetand What Your Organization Can Do About It
This report describes some of the latest attacks and threats emanating from the Internet, as well as advice and tips on how your organization can mitigate those threats before they affect your business. Download it today!
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2020-07-02
Apache Guacamole 1.1.0 and older may mishandle pointers involved inprocessing data received via RDP static virtual channels. If a userconnects to a malicious or compromised RDP server, a series ofspecially-crafted PDUs could result in memory corruption, possiblyallowing arbitrary code to be executed...
PUBLISHED: 2020-07-02
A vulnerability in the web-based management interface of Cisco Unified Communications Manager, Cisco Unified Communications Manager Session Management Edition, Cisco Unified Communications Manager IM & Presence Service, and Cisco Unity Connection could allow an unauthenticated, remote attack...
PUBLISHED: 2020-07-02
In versions 3.0.0-3.5.0, 2.0.0-2.9.0, and 1.0.1, when users run the command displayed in NGINX Controller user interface (UI) to fetch the agent installer, the server TLS certificate is not verified.
PUBLISHED: 2020-07-02
In versions 3.0.0-3.5.0, 2.0.0-2.9.0, and 1.0.1, the Neural Autonomic Transport System (NATS) messaging services in use by the NGINX Controller do not require any form of authentication, so any successful connection would be authorized.
PUBLISHED: 2020-07-02
In versions 3.0.0-3.5.0, 2.0.0-2.9.0, and 1.0.1, the NGINX Controller installer starts the download of Kubernetes packages from an HTTP URL On Debian/Ubuntu system.