Risk
4/2/2015
10:30 AM
Andrew Ginter
Andrew Ginter
Commentary
Connect Directly
LinkedIn
RSS
E-Mail vvv
100%
0%

Stuxnet Five Years Later: Did We Learn The Right Lesson?

No! That's despite an abundance of best practices and standards that are shining light into the dark corners of industrial control system security.

The 5th anniversary of the day the Stuxnet worm came into public view is upon us. The worm triggered changes in industrial control systems products, networks, and methodologies. Looking back, what did we learn? Did we look for lessons where the light was brightest? Or did we look in the dark corners where our need was greatest?

The bright light shining was "best practice violations." What did the light show us?

  • Stuxnet spread between sites on USB sticks. Poor USB device control is a best practice violation, and so we took action. Some of us glued USB ports shut. Many of us changed procedures to send of all of our industrial control system (ICS) information through firewalls to control the use of removable media. 
  • Stuxnet spread across networks for months using zero-day vulnerabilities. Hmm – zero-days happen in all software; there is no avoiding them. Some of us pushed the vendors for speedier security updates, and maybe invested a bit more in patch management, or maybe not. The bright light of best practice violations has little to say about zero-days.
  • Stuxnet spread through IT/OT firewalls on SQLServer connections using a Siemens S7 hard-coded password. Hard-coded passwords are a serious best-practice violation, so we all criticized Siemens. In the best-practice theme of “passwords matter,” many of us accelerated our IT/OT integration plans and deployed Active Directory servers to centralize all password policies and password management.

In fact, many of us accelerated our IT/OT integration plans even more generally, moving responsibility for all ICS security functions into central, expert, IT security teams. Now we do ICS security “by the book,” applying all IT security best practices.

Breaching a zero-vulnerabilities network
Is this progress? I was talking to an ICS security architect for an industrial firm at a conference a few months ago. They were doing everything "by the book." They had just passed an internal security audit by their IT security experts and, after years of work, their control system had come up squeaky clean. Zero vulnerabilities and compliant with all relevant cyber security standards and regulations.

So -- icing on the cake -- they brought in a penetration tester. They set her up on the corporate network. They told her:

“Pretend the receptionist’s PC had been compromised in a standard targeted attack. Let’s say the malware evaded anti-virus because there are less than one hundred copies circulating world-wide, so there are no anti-virus signatures. Let’s say the malware escalated privilege by telling the receptionist he needed to install a video CODEC. Let’s say the malware stegographically tunnels a command-line interface through one of the thousands of web application protocols that the next-gen corporate firewalls understand.”

This is a classic, targeted attack. The pen-tester’s mission is to use her control over this one machine to break into the zero-vulnerabilities ICS network. How far does she get? Five minutes later, she was in, controlling equipment on the ICS.

Five minutes.

Really? Five minutes? How is this possible? She simply continued with the standard targeted attack pattern. She downloads a password hash scraper and finds the domain administrator's password hash on the compromised machine. The domain administrator, after all, was on the machine the day before adding the machine into the domain for the test.

She uses the hash to log into the domain controller and creates an account for herself with administrative privileges. She logs into one of the plant systems she sees exposed through the firewall. (I was not told which system. Remote Desktop? A SQL Server? The file server deployed to reduce the use of USB sticks?) There are many targets to choose from on a typical, zero-vulnerabilities, best-practice, integrated IT/OT firewall.

After five minutes, she says “I’m in. Do you want me to stir this pot?”

“Step away from the keyboard,” she is told. This system controls a large, complex, dangerous physical process. Any unqualified individual “stirring the pot,” however briefly, is an unacceptable risk.

Darker lessons
Five minutes. Clearly, we did not take the right lessons from Stuxnet. What should we have learned?

  • Control networks are different: Industrial control systems, not surprisingly, control things. Industrial sites are full of powerful physical systems, and the subtlest sabotage can cause lasting damage. Stuxnet is credited with destroying 1,000-2,000 uranium gas centrifuges. Once a centrifuge physically disintegrates at 75,000 rpm, there is no way to “restore it from backup.” 
  • Every site can be hacked: The first law of cyber security is that no site is ever completely secure. Given enough time, money, and talent, any site can be hacked – even a uranium enrichment site with mil-spec protections. Every vulnerability assessment must finish with a description of how the site can attacked and compromised, and that description must use words that our senior management understands.
  • Attack training is essential to defense: Our defenses must reflect the capabilities and methods of our attackers. In Stuxnet’s case, the target was militarily strategic; the attackers were nation-state militaries prepared to spend billions on the attack if necessary, because even at that cost, an effective cyber assault is far cheaper than a conventional conflict. The lesson for “normal” sites is that we need to understand, at the very least, those attack tools and techniques that are widely available to any adversary. Today, remote control attacks exploiting trust relationships through firewalls are neither “unusual,” nor “unexpected.” This is the standard, modern, targeted attack pattern. Every control network design must anticipate this class of attack.

Evolving best practices
The French ANSSI ICS security standards published just last year got all of this right. They outright forbid the use of firewalls to connect certain classes of control networks to untrustworthy corporate networks, and they specifically forbid remote control of critical networks from less-trusted networks. The American NERC CIP V5 standards are moving in this direction as well. Look at their definition of External Routable Connectivity.

Note that these new ICS security best practices are not advocating air gaps; the trend towards increased network connectivity is inexorable. These best practices say two things: use unidirectional security gateways, and use secure tunnels. Unidirectional gateways replicate industrial servers through one-way hardware modules to less-trusted networks, so that corporate users and servers can query the replicas without ever sending any message at all back into protected control networks. The unidirectional hardware is able to send information out of a trusted network into a less-trusted network, but is fundamentally, physically, incapable of sending any message, any bit, or any signal at all back into the trusted network from the less-trusted network.

Secure tunnels connect equally-trusted networks in different geographies. To manage critical networks remotely, the ANSSI standards allow the use of strong VPN tunnels to extend critical networks to the head office. Keep those critical networks separate from less-trusted corporate and other networks at the head office. Connect critical networks and less-trusted networks only via unidirectional gateways, and never through firewalls. Physically and otherwise secure the head-office network extensions of critical networks the same way that those networks are themselves secured at their respective plants. With secure extensions in place, our central experts simply need two computers on their desks: one physically connected to the extended, critical control network, and one to the Internet-exposed corporate network.

Ask the right question
The bright light of "best practice" advice and standards is finally starting to shine into the dark corners of our ICS security programs. Control systems control large, costly, complex physical processes. Any unauthorized operation of these systems, however briefly, risks irreversible damage. Since no site is ever “secure,” it is vital that we understand what approaches are still open to our enemies to attack these sensitive systems, including modern remote-control attack techniques. Specifically, to defend the intrinsically “soft” interior of our critical networks, we must be deeply suspicious of every piece of information that enters our network – every USB stick, every laptop, every anti-virus update, and especially every network message.

Best practices are evolving to demand that we simply stop sending messages from probably-compromised corporate networks into our control networks. A decade ago, all we had were firewalls, but today we know better, and best-practice authorities are capturing that knowledge.

Today, we all need to start asking the question, which is becoming the central question for ICS security for the rest of this decade, namely: "which of our employees, our industrial equipment and our industrial plants are expendable enough to protect with only firewalls?"

 

Andrew Ginter is vice president of industrial security at Waterfall Security Solutions. He has managed the development of commercial products for computer networking, industrial control systems and industrial cyber security for vendors such as Develcon, Agilent Technologies ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Marilyn Cohodas
50%
50%
Marilyn Cohodas,
User Rank: Strategist
4/9/2015 | 11:11:17 AM
Standards take time
So true. But time is a commodity that is in short supply (as are the standards being developed. ) Let's hope the industry will the awareness to jump on them once they are available. 
Andrew_Ginter
50%
50%
Andrew_Ginter,
User Rank: Author
4/9/2015 | 9:17:13 AM
Re: Still in the dark?
Many control system and utility sites are already applying these lessons and making their sites orders of magnitude more difficult to attack, but many more are not yet doing so. At Waterfall we are working with control system security standards groups to update their advice to reflect the latest threats, but standards take time. 

I hope that what it takes to motivate action at the majority industrial sites is the on-going shift in expert consensus and advice/standards - similar to the shift that took place in many industries 25 years ago for process safety.
Marilyn Cohodas
50%
50%
Marilyn Cohodas,
User Rank: Strategist
4/6/2015 | 1:48:18 PM
Still in the dark?
Great blog,Andrew. But you don't paint a very optimistic picture. What will it take take for critical industries to effectively apply  the lessons that we should be learning from Stuxnet? Is that even realistic?
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
Security Operations and IT Operations: Finding the Path to Collaboration
A wide gulf has emerged between SOC and NOC teams that's keeping both of them from assuring the confidentiality, integrity, and availability of IT systems. Here's how experts think it should be bridged.
Flash Poll
New Best Practices for Secure App Development
New Best Practices for Secure App Development
The transition from DevOps to SecDevOps is combining with the move toward cloud computing to create new challenges - and new opportunities - for the information security team. Download this report, to learn about the new best practices for secure application development.
Slideshows
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2013-7445
Published: 2015-10-15
The Direct Rendering Manager (DRM) subsystem in the Linux kernel through 4.x mishandles requests for Graphics Execution Manager (GEM) objects, which allows context-dependent attackers to cause a denial of service (memory consumption) via an application that processes graphics data, as demonstrated b...

CVE-2015-4948
Published: 2015-10-15
netstat in IBM AIX 5.3, 6.1, and 7.1 and VIOS 2.2.x, when a fibre channel adapter is used, allows local users to gain privileges via unspecified vectors.

CVE-2015-5660
Published: 2015-10-15
Cross-site request forgery (CSRF) vulnerability in eXtplorer before 2.1.8 allows remote attackers to hijack the authentication of arbitrary users for requests that execute PHP code.

CVE-2015-6003
Published: 2015-10-15
Directory traversal vulnerability in QNAP QTS before 4.1.4 build 0910 and 4.2.x before 4.2.0 RC2 build 0910, when AFP is enabled, allows remote attackers to read or write to arbitrary files by leveraging access to an OS X (1) user or (2) guest account.

CVE-2015-6333
Published: 2015-10-15
Cisco Application Policy Infrastructure Controller (APIC) 1.1j allows local users to gain privileges via vectors involving addition of an SSH key, aka Bug ID CSCuw46076.

Dark Reading Radio
Archived Dark Reading Radio
In past years, security researchers have discovered ways to hack cars, medical devices, automated teller machines, and many other targets. Dark Reading Executive Editor Kelly Jackson Higgins hosts researcher Samy Kamkar and Levi Gundert, vice president of threat intelligence at Recorded Future, to discuss some of 2016's most unusual and creative hacks by white hats, and what these new vulnerabilities might mean for the coming year.