Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

News & Commentary

2/27/2017
12:05 PM
Curtis Franklin
Curtis Franklin
Curt Franklin
50%
50%

Cloudflare Bleeds Bad News – & Good

A slip of the fingers brings privacy concerns for millions and lessons in how to handle a security incident.

"For the want of a nail the shoe was lost, For the want of a shoe the horse was lost, For the want of a horse the rider was lost, For the want of a rider the battle was lost, For the want of a battle the kingdom was lost, And all for the want of a horseshoe-nail."
— Benjamin Franklin

Substitute the ">" symbol for an old-fashioned nail, and you have a synopsis of the latest privacy catastrophe, dubbed "Cloudbleed" by security researchers. How bad is Cloudbleed? The advice to users echoed around the Internet gives a hint of the breach's scope:

Change your passwords. All of them.

What happened to cause the problem and how it has been addressed each carry instructions for enterprise security and IT professionals. What happened is simple, and the sort of coding error that can be frustratingly difficult to catch absent the most rigorous testing protocols. In a single line of code, the intended ">=" operator was typed "==". As a result, certain operations were able to fill a buffer and keep right on writing, planting data across the system.

The flaw
According to Tavis Ormandy, a researcher with Google Project Zero, that planted data can include cookies, private messages, IP addresses, and passwords. The worst part? In a blog post about the event (made public only after the vulnerability had been repaired and remediated), Ormandy wrote, "PII [personally identifiable information] was actively being downloaded by crawlers and users during normal usage, they just didn't understand what they were seeing."

Cloudflare is a service provider with a client list that includes Uber, OKCupid, Fitbit, and various financial institutions. In addition to basic network performance services that include DNS and load balancing, Cloudflare provides security capabilities such as SSL and DDoS mitigation. Because Cloudflare provides the service to businesses rather than consumers, it's nearly impossible for an individual to know whether their personal information was exposed. And it's absolutely impossible to know whether any exposed information was actually grabbed by criminals -- until, of course, some form of identity theft takes place. Hence, the advice to change all your passwords. Now.

The problem was compounded by the way that Cloudflare operates. Cloudflare's data can be crawled and cached by services such as Google. This means that personal data was not just exposed, it was exposed in a persistent manner. That's a lot of bad news wrapped up in a single wayward character. Fortunately, there is some good news in the story to balance a bit of the bad.

Rapid response
When Ormandy discovered the issue, he immediately contacted Cloudflare using Twitter (in, as one commenter noted, the sort of message that should be absolutely horrifying to anyone on a corporate security team). In very short order, Cloudflare security personnel had reproduced the issue and shut down the services making use of the affected code. According to the company, they then put teams in the US and UK on 12-hour shifts, handing code off between the teams to keep efforts to find and remediate the problem going 24 hours a day.

Cloudflare reports that all services that included the problem were disabled globally within roughly three hours of notification. Figuring out precisely where in the code the problem lay, and how it could be repaired, took a bit longer. The company's blog post on the issue (referenced above) goes into great detail about where the problem was, what its effects were, and why it had not previously appeared. The level of detail in the disclosure is part of the good news surrounding the vulnerability; in speaking with several security researchers and IT professionals, all said that this disclosure is a model of what they would like to see in future industry events. All felt that this level of disclosure is likely to assuage customer fears and reassure regulators that all proper steps have been taken.

The final step in remediation was Cloudflare working with search engines to clear caches of released information. As the company wrote in its disclosure blog post, "With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains. The leaked memory has been purged with the help of the search engines."

Six days after the initial contact from Ormandy, Cloudflare disclosed the incident to the public. The vulnerability had existed for approximately four months and the number of customer records revealed to criminals is unknown. What is known is that this will likely be used as an example of the white-hat research system working as intended; independent discovery of an issue, verification by the affected company, rapid repair and remediation, with speedy, transparent disclosure to the public.

— Curtis Franklin, Security Editor, Light Reading

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
COVID-19: Latest Security News & Commentary
Dark Reading Staff 9/21/2020
Hacking Yourself: Marie Moe and Pacemaker Security
Gary McGraw Ph.D., Co-founder Berryville Institute of Machine Learning,  9/21/2020
Startup Aims to Map and Track All the IT and Security Things
Kelly Jackson Higgins, Executive Editor at Dark Reading,  9/22/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
Special Report: Computing's New Normal
This special report examines how IT security organizations have adapted to the "new normal" of computing and what the long-term effects will be. Read it and get a unique set of perspectives on issues ranging from new threats & vulnerabilities as a result of remote working to how enterprise security strategy will be affected long term.
Flash Poll
How IT Security Organizations are Attacking the Cybersecurity Problem
How IT Security Organizations are Attacking the Cybersecurity Problem
The COVID-19 pandemic turned the world -- and enterprise computing -- on end. Here's a look at how cybersecurity teams are retrenching their defense strategies, rebuilding their teams, and selecting new technologies to stop the oncoming rise of online attacks.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2015-4719
PUBLISHED: 2020-09-24
The client API authentication mechanism in Pexip Infinity before 10 allows remote attackers to gain privileges via a crafted request.
CVE-2020-15604
PUBLISHED: 2020-09-24
An incomplete SSL server certification validation vulnerability in the Trend Micro Security 2019 (v15) consumer family of products could allow an attacker to combine this vulnerability with another attack to trick an affected client into downloading a malicious update instead of the expected one. CW...
CVE-2020-24560
PUBLISHED: 2020-09-24
An incomplete SSL server certification validation vulnerability in the Trend Micro Security 2019 (v15) consumer family of products could allow an attacker to combine this vulnerability with another attack to trick an affected client into downloading a malicious update instead of the expected one. CW...
CVE-2020-25596
PUBLISHED: 2020-09-23
An issue was discovered in Xen through 4.14.x. x86 PV guest kernels can experience denial of service via SYSENTER. The SYSENTER instruction leaves various state sanitization activities to software. One of Xen's sanitization paths injects a #GP fault, and incorrectly delivers it twice to the guest. T...
CVE-2020-25597
PUBLISHED: 2020-09-23
An issue was discovered in Xen through 4.14.x. There is mishandling of the constraint that once-valid event channels may not turn invalid. Logic in the handling of event channel operations in Xen assumes that an event channel, once valid, will not become invalid over the life time of a guest. Howeve...