The 5th anniversary of the day the Stuxnet worm came into public view is upon us. The worm triggered changes in industrial control systems products, networks, and methodologies. Looking back, what did we learn? Did we look for lessons where the light was brightest? Or did we look in the dark corners where our need was greatest?
The bright light shining was "best practice violations." What did the light show us?
- Stuxnet spread between sites on USB sticks. Poor USB device control is a best practice violation, and so we took action. Some of us glued USB ports shut. Many of us changed procedures to send of all of our industrial control system (ICS) information through firewalls to control the use of removable media.
- Stuxnet spread across networks for months using zero-day vulnerabilities. Hmm – zero-days happen in all software; there is no avoiding them. Some of us pushed the vendors for speedier security updates, and maybe invested a bit more in patch management, or maybe not. The bright light of best practice violations has little to say about zero-days.
- Stuxnet spread through IT/OT firewalls on SQLServer connections using a Siemens S7 hard-coded password. Hard-coded passwords are a serious best-practice violation, so we all criticized Siemens. In the best-practice theme of “passwords matter,” many of us accelerated our IT/OT integration plans and deployed Active Directory servers to centralize all password policies and password management.
In fact, many of us accelerated our IT/OT integration plans even more generally, moving responsibility for all ICS security functions into central, expert, IT security teams. Now we do ICS security “by the book,” applying all IT security best practices.
Breaching a zero-vulnerabilities network
Is this progress? I was talking to an ICS security architect for an industrial firm at a conference a few months ago. They were doing everything "by the book." They had just passed an internal security audit by their IT security experts and, after years of work, their control system had come up squeaky clean. Zero vulnerabilities and compliant with all relevant cyber security standards and regulations.
So -- icing on the cake -- they brought in a penetration tester. They set her up on the corporate network. They told her:
“Pretend the receptionist’s PC had been compromised in a standard targeted attack. Let’s say the malware evaded anti-virus because there are less than one hundred copies circulating world-wide, so there are no anti-virus signatures. Let’s say the malware escalated privilege by telling the receptionist he needed to install a video CODEC. Let’s say the malware stegographically tunnels a command-line interface through one of the thousands of web application protocols that the next-gen corporate firewalls understand.”
This is a classic, targeted attack. The pen-tester’s mission is to use her control over this one machine to break into the zero-vulnerabilities ICS network. How far does she get? Five minutes later, she was in, controlling equipment on the ICS.
Really? Five minutes? How is this possible? She simply continued with the standard targeted attack pattern. She downloads a password hash scraper and finds the domain administrator's password hash on the compromised machine. The domain administrator, after all, was on the machine the day before adding the machine into the domain for the test.
She uses the hash to log into the domain controller and creates an account for herself with administrative privileges. She logs into one of the plant systems she sees exposed through the firewall. (I was not told which system. Remote Desktop? A SQL Server? The file server deployed to reduce the use of USB sticks?) There are many targets to choose from on a typical, zero-vulnerabilities, best-practice, integrated IT/OT firewall.
After five minutes, she says “I’m in. Do you want me to stir this pot?”
“Step away from the keyboard,” she is told. This system controls a large, complex, dangerous physical process. Any unqualified individual “stirring the pot,” however briefly, is an unacceptable risk.
Five minutes. Clearly, we did not take the right lessons from Stuxnet. What should we have learned?
- Control networks are different: Industrial control systems, not surprisingly, control things. Industrial sites are full of powerful physical systems, and the subtlest sabotage can cause lasting damage. Stuxnet is credited with destroying 1,000-2,000 uranium gas centrifuges. Once a centrifuge physically disintegrates at 75,000 rpm, there is no way to “restore it from backup.”
- Every site can be hacked: The first law of cyber security is that no site is ever completely secure. Given enough time, money, and talent, any site can be hacked – even a uranium enrichment site with mil-spec protections. Every vulnerability assessment must finish with a description of how the site can attacked and compromised, and that description must use words that our senior management understands.
- Attack training is essential to defense: Our defenses must reflect the capabilities and methods of our attackers. In Stuxnet’s case, the target was militarily strategic; the attackers were nation-state militaries prepared to spend billions on the attack if necessary, because even at that cost, an effective cyber assault is far cheaper than a conventional conflict. The lesson for “normal” sites is that we need to understand, at the very least, those attack tools and techniques that are widely available to any adversary. Today, remote control attacks exploiting trust relationships through firewalls are neither “unusual,” nor “unexpected.” This is the standard, modern, targeted attack pattern. Every control network design must anticipate this class of attack.
Evolving best practices
The French ANSSI ICS security standards published just last year got all of this right. They outright forbid the use of firewalls to connect certain classes of control networks to untrustworthy corporate networks, and they specifically forbid remote control of critical networks from less-trusted networks. The American NERC CIP V5 standards are moving in this direction as well. Look at their definition of External Routable Connectivity.
Note that these new ICS security best practices are not advocating air gaps; the trend towards increased network connectivity is inexorable. These best practices say two things: use unidirectional security gateways, and use secure tunnels. Unidirectional gateways replicate industrial servers through one-way hardware modules to less-trusted networks, so that corporate users and servers can query the replicas without ever sending any message at all back into protected control networks. The unidirectional hardware is able to send information out of a trusted network into a less-trusted network, but is fundamentally, physically, incapable of sending any message, any bit, or any signal at all back into the trusted network from the less-trusted network.
Secure tunnels connect equally-trusted networks in different geographies. To manage critical networks remotely, the ANSSI standards allow the use of strong VPN tunnels to extend critical networks to the head office. Keep those critical networks separate from less-trusted corporate and other networks at the head office. Connect critical networks and less-trusted networks only via unidirectional gateways, and never through firewalls. Physically and otherwise secure the head-office network extensions of critical networks the same way that those networks are themselves secured at their respective plants. With secure extensions in place, our central experts simply need two computers on their desks: one physically connected to the extended, critical control network, and one to the Internet-exposed corporate network.
Ask the right question
The bright light of "best practice" advice and standards is finally starting to shine into the dark corners of our ICS security programs. Control systems control large, costly, complex physical processes. Any unauthorized operation of these systems, however briefly, risks irreversible damage. Since no site is ever “secure,” it is vital that we understand what approaches are still open to our enemies to attack these sensitive systems, including modern remote-control attack techniques. Specifically, to defend the intrinsically “soft” interior of our critical networks, we must be deeply suspicious of every piece of information that enters our network – every USB stick, every laptop, every anti-virus update, and especially every network message.
Best practices are evolving to demand that we simply stop sending messages from probably-compromised corporate networks into our control networks. A decade ago, all we had were firewalls, but today we know better, and best-practice authorities are capturing that knowledge.
Today, we all need to start asking the question, which is becoming the central question for ICS security for the rest of this decade, namely: "which of our employees, our industrial equipment and our industrial plants are expendable enough to protect with only firewalls?"