Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Perimeter

9/26/2014
10:20 AM
Paul Vixie
Paul Vixie
Commentary
Connect Directly
Facebook
Twitter
LinkedIn
Google+
RSS
E-Mail vvv
50%
50%

Shellshocked: A Future Of ‘Hair On Fire’ Bugs

Most computers affected by Bash will be updated within 10 years. The rest will be vulnerable for the lifespans of all humans now living. This should concern us. But then, global warming should also concern us.

It seems that the more we integrate our lives and our national economies into the Internet, the less safe we, and our privacy, and our money, become. The human population will eventually grow weary of the endless parade of worldwide hair-on-fire technology problems like Y2K, Conficker, Heartbleed, and, as of this week, Shellshock.

By “weary” I mean thoughts like “Another day, another bug, I have no time for this, I’ll upgrade everything when I get the time.” We may already be there, given that six full years after the Conficker worm was announced, we still see about a million unique IP addresses per day in the Conficker sinkhole. Those can’t all be student researchers checking to see if the botnet still exists. Sadly, every hour of delay dramatically increases the likelihood that the device will become more deeply infected before it is patched -- after which time, patches won’t actually make anything better.

Aggregate attack surface is a progressive concept. When the total number of infected or vulnerable computers, and the total number of infections and vulnerabilities themselves, were small, the criminal miscreants of the world did not have as much free infrastructure to draw from when attacking new targets. Make no mistake, the bad guys don’t pay for hardware or connectivity -- they’re using ours instead, since ours is free, and this kind of indirection helps them hide their tracks and misdirect our investigations. Today the computational resources available to the bad guys have grown to the point where no victim is out of reach, and no attacker has to be all that frugal or even quiet about its attacks.

Shellshock is the name of a bug in “Bash,” an acronym for Bourne Again Shell, a command line interpreter present on most computers in the world except for Windows. Bash is in Linux, and Linux is in just about every embedded computer including smart TVs, smartphones and tablets, home gateways, wireless access points, Internet servers, and many industrial control systems.

There is not a quality problem with Linux per se. In fact, Linux may be the most reviewed, most tested, highest quality (per line of code) system in history. However, there’s an awful lot of code and an awful lot of interfaces -- connections between different parts of code -- and with complexity comes error just as surely as night follows day. More importantly, many devices containing Bash are not field upgradeable, either for cost reasons or because their makers died out. Even among devices that are still upgradeable, most are silent, unknown trolls in dark closets with no monitoring or auditing or management at all.

Fix they did, but…
Within a few hours of the announcement of Shellshock and the first software update to fix the bug (which fix did, by the way, turn out to be incomplete), those of us who watch the Internet for malicious activity saw hundreds of researchers scanning billions of Internet addresses, all trying to measure and catalogue attack surface.

It’s safe to assume that some of those researchers have evil intent, such as adding the vulnerable computers to a botnet, to be used later for launching DDoS attacks or similar malfeasance. And as always during times of emergency, misinformation and misunderstanding was everywhere, even in some official announcements from people and companies and agencies whose purpose in life is to keep us informed and safe against online threats.

It’s conservative to estimate that for every modern computing device we actually know is a computing device, and whose manufacturer still has the ability to update its software, there are at least 10 other computing devices that we don’t know much about, which will not have their software updated ever, and which have years or even decades of life left.

Experience tells us to expect that the computers that will be updated will be mostly updated within about 10 years. Most of the rest will be vulnerable until they die, which will take some decades, and a few million of them will still be running and still be vulnerable to Shellshock for the lifespans of all humans now living. This should concern us, but then, global warming should also concern us.

As the saying goes, “When you’re stuck in a hole, the first thing to do is stop digging.” Given the inevitability of software bugs and the growing dependence on technology for banking, communications, infrastructure, agriculture, and food supply, can we afford to continue driving innovation guided only by near-term profit, where technology’s winners are always those who pursue time-to-market over quality?

By all means, let us patch Bash wherever we can find it. But that’s busy work. The vulnerable Bash instances that we won’t find vastly outnumber those we will, and our future is going to be dominated by leftovers from an endless parade of hair-on-fire bugs that we eventually learn to live with when the next one comes along and steals our attention.

Dr. Paul Vixie is an Internet pioneer and thought leader who designed, implemented, and deployed several Domain Name System (DNS) protocol extensions and applications that are used throughout the Internet today. He is CEO of Farsight Security Inc. Previously, he served as ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Threaded  |  Newest First  |  Oldest First
Kelly Jackson Higgins
50%
50%
Kelly Jackson Higgins,
User Rank: Strategist
9/26/2014 | 4:09:35 PM
hair-on-fire bug fatigue
Your point about this being another in a series of major flaws we'll face is so true, @Paul. What is worrisome is how this could spawn bug fatigue, almost like we're seeing in the wave of retail data breaches. Another day, another major Internet flaw is exposed. How does the industry avoid that?
rjones2818
50%
50%
rjones2818,
User Rank: Strategist
9/30/2014 | 11:03:40 AM
Re: hair-on-fire bug fatigue
Two things:


1) Simplify as much as possible, as has been mentioned in the comments. This is particularly true in the entrance to any programs.  The fewer doors, the fewer ways for the rats to get in.  I know it's a broad brush, but complexity for its own sake is unsafe.  The likelyhood is that every system is probably unsafe due to designers not thinking of every way their code is going to be attacked.  This isn't because they're bad designers, it's because not every way code is going to be attacked has been thought of by anybody yet.


2) The people who aren't patching aren't fatigued.  Regular patchers shouldn't be fatigued, it's just part of what they do. People who patch absolutly everything the moment a patch comes out probably are fatigued.
Marilyn Cohodas
50%
50%
Marilyn Cohodas,
User Rank: Strategist
9/26/2014 | 4:14:13 PM
Great analysis but is it really so hopeless?
It patching is simply busy work, what real work needs to be done? I for one don't look forward to a future with my hair on fire watching the bug parade go on and on.... 
paulvixie
50%
50%
paulvixie,
User Rank: Author
9/26/2014 | 4:57:21 PM
Re: Great analysis but is it really so hopeless?
<< It patching is simply busy work, what real work needs to be done? I for one don't look forward to a future with my hair on fire watching the bug parade go on and on....  >>

the situation at hand is the result of mass numbers of people acting in their short-term self interest, to the detriment of the whole, sort of in a "tragedy of the commons" remake. my comparison of this to global warming is not an accident. historically, the way we incentivize mass numbers of people to act in their longer-term self interest as opposed to their short-term self interest, has involved some combination of three things:

1. give them better choices (so, innovation)

2. increase the cost of some of their current choices (so, regulation)

3. make them better aware of their long-term self interest, and their choices (so, education)

one innovation i've been pondering for the last few years in the face of a similarly difficult problem which is the wide spread lack of source address validation at the edge of the internet, is a certification programme for devices which have been well tested under difficult conditions, along the lines of the "underwriters laboratory" sticker i looked for before buying a toaster oven for my kitchen.

but to your question ("what real work needs to be done?"), i think the first thing we've got to do is stop panicking every time one of these bugs comes along. we still have millions of devices vulnerable to attacks that had our hair on fire one, two, three, four, and five years ago -- clearly setting our hair on fire didn't have a big impact. so, do patch, but, also consider what i said on a private security forum yesterday:

correct reaction would be strategic: get an inventory of the contents of every smart device your agency or your company owns or operates or depends upon, and enact a phase-out plan that replaces every non-upgradeable or un-auditable device with something you can actually control. let normal apple/redhat/$vendor upgrade/patch take care of their products on your network in due course. note for example that the first patch released by redhat yesterday did not go far enough to fix this bug.

thank you for your question, it's an important one. --vix

aws0513
50%
50%
aws0513,
User Rank: Ninja
9/29/2014 | 8:34:19 AM
Re: Great analysis but is it really so hopeless?
Great article Paul.

Since the announcement of the bug finding, I have not considered this a full on "sky is falling" situation.  My reaction was more in line with my military background. It wasn't that we were not aware of the gravity of the situation, we just reacted as we always react to high risk vulnerability announcements.  Business as usual for this IT security program.

First we went into full assessment mode to identify as many systems that could be affected as possible.  This meant taking our inventory and removing the obvious non-players, actioning the known affected systems, and validation of the unknowns.  We also ramped up monitoring practices for the time being until we feel most of the immediate risk has been mitigated.
We were fortunate in that we just finished implementing an inventory management practice that gives us a very comprehensive data set to start with.  BTW...  Inventory data on enterprise systems is probably the most powerful tool an IT security team can have.  The classic "know thyself" factor.  Having such a data set allows for efficient triage and planning.  It is some overhead to keep an inventory current, but worth every penny when the sky does seem to be falling.

As it stands today, we are looking good on the patching front thanks to some dedicated work over the weekend.  Now we are still sorting through and validating each unknown system/device to determine impact and remediation options.  The fire fight is ongoing, but we have reinforced the weak points and are assessing the rest of the infrastructure to see if there are more.
paulvixie
50%
50%
paulvixie,
User Rank: Author
9/29/2014 | 9:06:35 AM
if you patched over the weekend, you're out of date, and vulnerable, again.
<< As it stands today, we are looking good on the patching front thanks to some dedicated work over the weekend. >>

if you patched over the weekend, you're out of date, and vulnerable, again. here's the latest as of this moment:

Shellshock (CVE-2014-6271CVE-2014-7169CVE-2014-7186CVE-2014-7187CVE-2014-6277) is a vulnerability in GNU's bash shell that gives attackers access to run remote commands on a vulnerable system. If your system has not updated bash in since Sun Sep 28 2014: 1:11AM EST (See patch history), you're most definitely vulnerable and have been since first boot. This security vulnerability affects versions 1.14 (released in 1994) to the most recent version 4.3 according to NVD.

(that text is from https://shellshocker.net/)

Kelly Jackson Higgins
50%
50%
Kelly Jackson Higgins,
User Rank: Strategist
9/29/2014 | 9:21:44 AM
Re: if you patched over the weekend, you're out of date, and vulnerable, again.
So @Paul, how would orgs who patched over the weekend actually know that their patch has to be redone? Is US-CERT or ICS-CERT planning to issue an alert to let them know? There's so much information flying around, it must be a nightmare to stay on top of this. 
paulvixie
50%
50%
paulvixie,
User Rank: Author
9/29/2014 | 9:32:04 AM
Re: if you patched over the weekend, you're out of date, and vulnerable, again.
<< So @Paul, how would orgs who patched over the weekend actually know that their patch has to be redone? Is US-CERT or ICS-CERT planning to issue an alert to let them know? There's so much information flying around, it must be a nightmare to stay on top of this.  >>

i recommend looking at the shellshocked.net web site, writing down the list of CVE's, and then looking at your vendor's patch literature, and making sure that every CVE listed at the shellshocked.net web site, is listed as fixed by some recent patch from your vendor.

if not, call your vendor.
Kelly Jackson Higgins
50%
50%
Kelly Jackson Higgins,
User Rank: Strategist
9/29/2014 | 9:37:59 AM
Re: if you patched over the weekend, you're out of date, and vulnerable, again.
Thank you, @Paul. This is good to know. 
Kelly Jackson Higgins
50%
50%
Kelly Jackson Higgins,
User Rank: Strategist
9/29/2014 | 12:59:58 PM
Re: if you patched over the weekend, you're out of date, and vulnerable, again.
@Paul--A bit of confusion here on the vulns Shellshocker is talking about and what SANS ISC has posted: https://isc.sans.edu/diary/Shellshock%3A+We+are+not+done+yet+CVE-2014-6277%2C+CVE-2014-6278/18723. These appear to be different bugs, unless I'm missing something? 
paulvixie
50%
50%
paulvixie,
User Rank: Author
9/29/2014 | 1:51:04 PM
Re: if you patched over the weekend, you're out of date, and vulnerable, again.
<< These appear to be different bugs, unless I'm missing something? >>

To me the bug is that GNU Bash ever evals the contents of an environment variable. In other words, all of this week's drama comes from a misfeature. Based on the fix now present on FreeBSD systems, I am not the only one thinking this way. However, the maintainers of GNU Bash are doing their darndest to make this feature safe by making terribly fine distinctions about the exact form, syntax, and content of these environment variables. The reason you see five different CVE's (as of this moment) at http://shellshocked.net/ is that people keep finding new ways to fool the latest patch and access the underlying remote execution vulnerability.

I prefer FreeBSD's fix. Don't evaluate the contents of environment variables by default. To those who warn that this will break some existing GNU Bash scripts, I answer: yes, and that's a bitter pill, but since this is actually misfeature, my feeling is that adding logic to make finer and finer distinctions about the content of environment variables is increasing complexity (and therefore danger), and decreasing auditability and provability (and therefore safety).

I also prefer the Debian approach (/bin/sh is "dash" not "bash") over RedHat and Apple's. GNU Bash is a great interactive shell, but it's way too large and too complex to be allowed to be in the execution path for libc's popen() and system() calls, which are used by Apache and QMail to run commands. /bin/sh should be as simple as possible, which is to say, like "dash" on Debian (which comes from "ash" which is used as /bin/sh on FreeBSD, NetBSD, OpenBSD, and DragonflyBSD).

It's very strange after the last couple of years of hair-on-fire bug-of-the-week theater, to have to argue that complexity ought to be avoided wherever possible in control systems.
Kelly Jackson Higgins
50%
50%
Kelly Jackson Higgins,
User Rank: Strategist
9/29/2014 | 2:01:41 PM
Re: if you patched over the weekend, you're out of date, and vulnerable, again.
This is great insight and perspective, Paul. Many thanks.
RyanSepe
50%
50%
RyanSepe,
User Rank: Ninja
9/29/2014 | 9:42:09 AM
Re: if you patched over the weekend, you're out of date, and vulnerable, again.
I agree @Kelly. We need to define a better means of communciation for not only this vulnerability but all vulnerabilities. I would imagine that corporate security teams are in many conversations with their MSSP's if they have them available but for ones that don't they are relying much on this information outlet.

What have people felt are the best avenues for consistent and validated data regarding this vulnerability?
aws0513
50%
50%
aws0513,
User Rank: Ninja
9/29/2014 | 9:46:16 AM
Re: Great analysis but is it really so hopeless?
Yep, we caught that info yesterday. We got caught up on our key systems accordingly.
Monitoring for further info as we go.

Thanks for the follow up info @Paul.
Robert McDougal
50%
50%
Robert McDougal,
User Rank: Ninja
9/29/2014 | 10:05:05 AM
Re: Great analysis but is it really so hopeless?
So far our org has patched everything that can be patched.  However, we are also not seeing very many attack attempts either.  Since we deployed our Shellshock IDS alerts we have only seen around 40 attempted attacks.
Marilyn Cohodas
50%
50%
Marilyn Cohodas,
User Rank: Strategist
9/29/2014 | 10:28:40 AM
Re: Great analysis but is it really so hopeless?
That sounds like a good thing @Robert McDougal (that you've patched and also have seen only 40 attempted attacks). Are you confident that your patched all the holes and that your information is correct?
Robert McDougal
50%
50%
Robert McDougal,
User Rank: Ninja
9/29/2014 | 10:39:35 AM
Re: Great analysis but is it really so hopeless?
As with any security problem I am only as confident with regards to the intel I have available.  Based on the exploits that are currently known to be in the wild I am very confident that I am able to detect them all.  However, if there is an attack that is drastically different than those I am tracking then they could slip under my radar.

With that said, based off the nature of this vulnerability I am fairly confident that we are seeing everything that is heading our way and the attacks that are directed at us are not getting through.
kstaron
50%
50%
kstaron,
User Rank: Apprentice
9/26/2014 | 6:36:25 PM
Ok, so now what?
Okay ,so it's out there, other than patching what can we do about it? It's not feasible to throw all the devices out and start fresh, so where does that leave us? As consumers what can we do to protect ourselves?
paulvixie
50%
50%
paulvixie,
User Rank: Author
9/26/2014 | 6:44:22 PM
Re: Ok, so now what?
<< Okay ,so it's out there, other than patching what can we do about it? It's not feasible to throw all the devices out and start fresh, so where does that leave us? As consumers what can we do to protect ourselves? >>

my recommendation is as follows:

correct reaction would be strategic: get an inventory of the contents of every smart device your agency or your company owns or operates or depends upon, and enact a phase-out plan that replaces every non-upgradeable or un-auditable device with something you can actually control. let normal apple/redhat/$vendor upgrade/patch take care of their products on your network in due course. note for example that the first patch released by redhat yesterday did not go far enough to fix this bug.

i hope this helps. --vix

Marilyn Cohodas
50%
50%
Marilyn Cohodas,
User Rank: Strategist
9/29/2014 | 8:08:13 AM
Re: Ok, so now what?
Thanks for these links, Paul, not to mention your spot-on advice for dealing with Shellshock, both long- and short-term. Hope you'll keep us apprised of new insights/developoments as this story continues to unfold. 
SgS125
50%
50%
SgS125,
User Rank: Ninja
9/29/2014 | 11:22:35 AM
You sound tired Paul
Yup another week, another lip gets bit, we wonder, we wring our hands and guess what, the dang thing was there last week.  We did not know it was there last week, at least most of us anyway.

Time to double down on your game and get to it.

Just think of all the great exploits that we still don;t know about, always work to be done here.

It's like we are the gravediggers and everyone is in their 90's..... only a matter of time till we need to dig another hole.

 
44% of Security Threats Start in the Cloud
Kelly Sheridan, Staff Editor, Dark Reading,  2/19/2020
Zero-Factor Authentication: Owning Our Data
Nick Selby, Chief Security Officer at Paxos Trust Company,  2/19/2020
Firms Improve Threat Detection but Face Increasingly Disruptive Attacks
Robert Lemos, Contributing Writer,  2/20/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
6 Emerging Cyber Threats That Enterprises Face in 2020
This Tech Digest gives an in-depth look at six emerging cyber threats that enterprises could face in 2020. Download your copy today!
Flash Poll
How Enterprises Are Developing and Maintaining Secure Applications
How Enterprises Are Developing and Maintaining Secure Applications
The concept of application security is well known, but application security testing and remediation processes remain unbalanced. Most organizations are confident in their approach to AppSec, although others seem to have no approach at all. Read this report to find out more.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2020-9351
PUBLISHED: 2020-02-23
An issue was discovered in SmartClient 12.0. If an unauthenticated attacker makes a POST request to /tools/developerConsoleOperations.jsp or /isomorphic/IDACall with malformed XML data in the _transaction parameter, the server replies with a verbose error showing where the application resides (the a...
CVE-2020-9352
PUBLISHED: 2020-02-23
An issue was discovered in SmartClient 12.0. Unauthenticated exploitation of blind XXE can occur in the downloadWSDL feature by sending a POST request to /tools/developerConsoleOperations.jsp with a valid payload in the _transaction parameter.
CVE-2020-9353
PUBLISHED: 2020-02-23
An issue was discovered in SmartClient 12.0. The Remote Procedure Call (RPC) loadFile provided by the console functionality on the /tools/developerConsoleOperations.jsp (or /isomorphic/IDACall) URL is affected by unauthenticated Local File Inclusion via directory-traversal sequences in the elem XML ...
CVE-2020-9354
PUBLISHED: 2020-02-23
An issue was discovered in SmartClient 12.0. The Remote Procedure Call (RPC) saveFile provided by the console functionality on the /tools/developerConsoleOperations.jsp (or /isomorphic/IDACall) URL allows an unauthenticated attacker to overwrite files via vectors involving an XML comment and /.. pat...
CVE-2020-9355
PUBLISHED: 2020-02-23
danfruehauf NetworkManager-ssh before 1.2.11 allows privilege escalation because extra options are mishandled.