Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Vulnerabilities / Threats

05:12 PM
Connect Directly

Lessons From The Ukraine Electric Grid Hack

New SANS analysis on how the attackers broke in and took control of the industrial control systems at three regional power firms in the Ukraine and shut off the lights.

New analysis and details about the devastating and unprecedented cyberattack that resulted in a power blackout in a region of the Ukraine last December illuminate glaring holes in security and operations that could have thwarted the attackers from shutting off the lights.

Security experts from SANS today in conjunction with the North American Reliability Corporation (NERC)'s E-ISAC published an in-depth postmortem analysis by SANS ICS experts of the attack, based on details revealed by ICS-CERT late last month as well as other public information. Aside from the glaring question of whether the attack indeed was sponsored by the Russian government, most of the nagging questions of how the attackers were able to black out a portion of Ukraine’s power grid have now been answered. The smoking gun has been confirmed: the attackers used stolen user credentials to remotely access and manipulate the industrial control systems and shut down power for some 225,000 Ukrainian power customers on Dec. 23 of last year.

“I think that the puzzle pieces are together now,” says Robert M. Lee, a SANS instructor and ICS/SCADA expert, as well as co-author of the report. “We’re missing the definitive attribution ... but the technical details” are mostly fleshed out, he says.

Ukraine officials have accused Russia, an obvious suspect given the military and geopolitical conflict between the two nations over Crimea. But the US reports steered clear of confirming that the attacks were the handiwork of a Russian state-sponsored initiative.

One thing US officials have confirmed is that the attackers staged a well-coordinated attack that relied on deep reconnaissance over a six-month period after they first embedded themselves into the network of three regional energy distribution companies. The attacks went live within 30 minutes of one another, and there also were three other organizations hit by the attacks that didn’t suffer any disruption to operations.

Like most targeted attacks, the Ukraine power grid attack began with a phishing email containing a malware-rigged attachment. In this case, Word Documents and Excel spreadsheets that when opened by users in the companies’ business network, dropped BlackEnergy3 malware that lurked around and stole legitimate user credentials. The attackers then used stolen VPN credentials to reach the industrial control systems network, and remote access tools to control the HMIs and pull the breakers.  

The attackers covered their tracks and bought themselves time, too, by installing their own custom firmware on serial-to-Ethernet devices at substations in order to knock them offline, and using KillDisk to wipe master boot records of the systems they hacked as well as to delete some logs. They waged a denial-of-service attack on the power companies’ telephone systems as well, thwarting their ability to communicate. In one case, KillDesk overwrote Windows-based HMIs in remote terminal units. The attackers also remotely disconnected Uninterruptable Power Supplies (UPS) systems to cripple power-restoration operations.

“It was extremely well-done -- how you would expect a well-funded team to operate,” Lee says.

In an interview with Dark Reading last month, Phyllis Schneck, the deputy under secretary for cybersecurity and communications with the Department of Homeland Security, said members of ICS-CERT’s team had been invited to Kiev to study and learn more about the attacks. “They spent four days working with our Ukraine counterparts to understand what happened,” she said. They learned that BlackEnergy malware was widespread in the victim networks, and the attackers “had their way with the systems” using stolen credentials, she said.

ICS-CERT’s findings showed how such an attack could “happen to anyone,” she said, and the agency wanted to provide recommendations for preventing such an attack on US critical infrastructure.

DHS undersecretary for the National Protection and Programs Directorate (NPPD) Suzanne Spaulding says she hopes the report will be a reality-check for US critical infrastructure owners. “I want ... [executives to say], ‘what are we doing about this?’” to prevent similar attacks, she said during an interview last month with Dark Reading.

There are plenty of lessons to be gleaned for power grid and other critical infrastructure operators in the US and around the globe.

For one thing, a cyberattack that results in a power outage takes some heavy lifting, and a bit of time, to pull off. “It took them six months or more to figure out these environments ... And it was only a partial outage,” says Lee, who notes that their methods weren’t necessarily sophisticated but were definitely coordinated. “We consistently see [the] theme for attackers who do the things we care about most in ICS networks ... it’s much more difficult” for them to do damage and it takes time, he says.

And that’s lesson number one: if attackers need a sufficient period of time for reconnaissance and learning the environment in order to control industrial equipment, the good news is that there’s actually a window for detecting their activity -- and stopping them from doing damage.


Network security monitoring could have helped spot the attackers before they shut off the power.

The Ukraine power grid attackers hid in plain sight for six months, gradually gathering enough intelligence and and knowledge to figure out how to access and manipulate the HMI and turn out the lights. Had the power companies been running network security monitoring tools, they could have spotted that activity.

There are many free and open-source network security monitoring (NSM) tools out there that can spot all kinds of bad activity in an ICS/SCADA environment, including unusual file traffic, a PLC code update, or command and control communications. Rob Caldwell and Chris Sistrunk ICS/SCADA experts from FireEye Mandiant recommend NSM for plants, and say NSM would have caught Stuxnet, for instance, and could be set to catch BlackEnergy. Some of the more popular tools come via the Security Onion Linux suite including Wireshark, NetworkMiner, Bro, and Snorby.

“If they had used network security monitoring practices, they could identify any reconnaissance ... and multiple VPN connections at times that were not normal,” SANS’ Lee notes.

Monitoring tools would have detected unusual data flows, something that’s relatively easy to spot in ICS networks because data flows are mostly static and predictable, he says. “When attackers are trying to learn [the environment], they disrupt those pathways.”

The attack punctuates the danger of remote access to ICS/SCADA networks.

VPN connections between the Ukraine power companies’ ICS and enterprise networks did not appear to use two-factor authentication, according to the report. “Additionally, the firewall allowed the adversary to remote admin out of the environment utilizing a remote access capability native to the systems,” the NERC SANS report says.

Ralph Langner, founder of the Langner Group, says critical infrastructure operators shouldn’t allow remote access to these systems.

“Limit remote access only to the people who need it,” SANS’ Lee says.

The report recommends using multi-factor authentication for any remote access communications.

Uninterruptible power supplies need protection, too.

The attackers commandeered a remote management interface to the UPS systems to schedule an outage for power at the energy company’s own buildings or datacenters.

“The online command interface to UPSes is another stupid flaw. These UPSes are located within the same building, so by controlling them via the network you just save five minutes for a maintenance job,” says Langner, who notes the CLI most likely would have been an embedded Web browser. He recommends disabling remote command interfaces to UPS systems.

The attackers also generated a DoS of thousands of phone calls to the energy company’s call center to derail restoration and communications.

“The reconfiguration of the UPS and the telephone DDoS: those two things added to the confusion, and to make the Ukrainians look incompetent. Those are two things I wouldn’t have predicted would have happened” in at attack like this, he says.

While the disabled UPS system and the telephone system DDoS were separate from the blackout portion of the attack, the goal appeared to be to embarrass the Ukrainians as well as to thwart restoration, he says. “During this attack, there seemed to be elements that highlight incompetence ... I think that’s interesting.”

Lee points out that the Russian media for the past year and a half has been reporting on the “incompetence” of the Ukraine infrastructure, and how they need Russia’s help. “The consistent theme [in the cyberattack] was not only being highly sophisticated in logistics and planning, but also in this showing” perceived weaknesses in Ukraine’s management of the power grid, he says.

Attackers can install malicious firmware on industrial equipment.

DHS in 2008 issued an alert to ICS/SCADA operators about a vulnerability in ICS/SCADA firmware update processes dubbed “Boreas.” It basically leaves an industrial systems’ firmware updates open to abuse, where an attacker installs his own malicious firmware to sabotage the system.

That’s basically what happened to the serial-to-Ethernet gateways in the Ukraine attacks, according to Langner, rendering them inoperable such that the operators were unable to communicate with the substations.

SANS’ Lee says the the custom firmware installed on the Ukraine networks’ serial-to-Ethernet gateways to “brick” them and disrupt the restoration of power was most surprising element of the attack. “That was extremely clever and it hurt the restoration effort of the Ukrainians,” he says. “I didn’t think we’d see an adversary clicking the breakers open and with what happened with the firmware.”

The gateways, or converters, basically translate communications between the serial protocols at physical substations and the overall Ethernet network that connects them. “By opening the breakers and modifying the firmware on those devices, it makes them unusable. In essence, they blew the bridges” up, Lee explains.

“They were cut off from the remote sites and had to physically drive out to them.”

Without a ‘cyber’ element to incident response and disaster recovery, a cyberattack is a disaster.

The Ukrainian power companies had no way to maintain control of their ICS/SCADA environment after the attack. That was an “eye-opener,” Lee says, and shows the crucial need for a “cyber” element in incident response and disaster recovery plans.

“You know they are opening breakers, so how do you quickly disable those features ... No one has that capability,” he says of ICS/SCADA operators.

That type of contingency planning is a big piece of the security picture, and until now, there’s been no experience in fighting back and regaining control when the bad guys have taken over, he says.

“There has never been a public case where the power grid was [affected] due to a cyberattack. This is the first time it’s happened, and it’s our only case study of what it looks like.”

Meanwhile, the lights may be back on in the Ukraine, but the nation remains vulnerable to another attack, Lee says. “It takes a long time to change processes, systems, and [get] trained personnel,” he says.


Related Content:


Interop 2016 Las VegasFind out more about security threats at Interop 2016, May 2-6, at the Mandalay Bay Convention Center, Las Vegas. Register today and receive an early bird discount of $200.

Kelly Jackson Higgins is the Executive Editor of Dark Reading. She is an award-winning veteran technology and business journalist with more than two decades of experience in reporting and editing for various publications, including Network Computing, Secure Enterprise ... View Full Bio

Recommended Reading:

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
Joe Stanganelli
Joe Stanganelli,
User Rank: Ninja
3/27/2016 | 11:35:52 AM
Re: Phishing
So yes... Lesson 1: Have better training for phishing defense.  Lesson 2: Check the logs and use analytics to analyze and compare past and present network activity to detect intrusions.
Kelly Jackson Higgins
Kelly Jackson Higgins,
User Rank: Strategist
3/19/2016 | 9:16:07 AM
Re: Phishing
Indeed. The themes are so familiar, right? It's just much more dramatic and unnerving when in the end a phish leads to a power outage. What happened after that phish for 6 months was really where the power cos. had a shot at stopping this attack. Now we have a case study, as Rob Lee said.  
Joe Stanganelli
Joe Stanganelli,
User Rank: Ninja
3/19/2016 | 6:47:34 AM
The really disheartening thing about this news is that it all started with a phishing attack -- something that is so easily prevented via proper training.

You can have the best locks in the world, but if your people unlock them and open the doors wide for the bad guys, there's not a lot else you can do.
COVID-19: Latest Security News & Commentary
Dark Reading Staff 9/21/2020
Hacking Yourself: Marie Moe and Pacemaker Security
Gary McGraw Ph.D., Co-founder Berryville Institute of Machine Learning,  9/21/2020
Startup Aims to Map and Track All the IT and Security Things
Kelly Jackson Higgins, Executive Editor at Dark Reading,  9/22/2020
Register for Dark Reading Newsletters
White Papers
Current Issue
Special Report: Computing's New Normal
This special report examines how IT security organizations have adapted to the "new normal" of computing and what the long-term effects will be. Read it and get a unique set of perspectives on issues ranging from new threats & vulnerabilities as a result of remote working to how enterprise security strategy will be affected long term.
Flash Poll
How IT Security Organizations are Attacking the Cybersecurity Problem
How IT Security Organizations are Attacking the Cybersecurity Problem
The COVID-19 pandemic turned the world -- and enterprise computing -- on end. Here's a look at how cybersecurity teams are retrenching their defense strategies, rebuilding their teams, and selecting new technologies to stop the oncoming rise of online attacks.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2020-09-24
The client API authentication mechanism in Pexip Infinity before 10 allows remote attackers to gain privileges via a crafted request.
PUBLISHED: 2020-09-24
An incomplete SSL server certification validation vulnerability in the Trend Micro Security 2019 (v15) consumer family of products could allow an attacker to combine this vulnerability with another attack to trick an affected client into downloading a malicious update instead of the expected one. CW...
PUBLISHED: 2020-09-24
An incomplete SSL server certification validation vulnerability in the Trend Micro Security 2019 (v15) consumer family of products could allow an attacker to combine this vulnerability with another attack to trick an affected client into downloading a malicious update instead of the expected one. CW...
PUBLISHED: 2020-09-23
An issue was discovered in Xen through 4.14.x. x86 PV guest kernels can experience denial of service via SYSENTER. The SYSENTER instruction leaves various state sanitization activities to software. One of Xen's sanitization paths injects a #GP fault, and incorrectly delivers it twice to the guest. T...
PUBLISHED: 2020-09-23
An issue was discovered in Xen through 4.14.x. There is mishandling of the constraint that once-valid event channels may not turn invalid. Logic in the handling of event channel operations in Xen assumes that an event channel, once valid, will not become invalid over the life time of a guest. Howeve...