Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Attacks/Breaches

5/20/2019
09:30 AM
Connect Directly
Google+
Twitter
RSS
E-Mail
50%
50%

How a Manufacturing Firm Recovered from a Devastating Ransomware Attack

The infamous Ryuk ransomware slammed a small company that makes heavy-duty vehicle alternators for government and emergency fleet. Here's what happened.

The tiny IT team at C.E. Niehoff & Co. had been working for two weeks to run down and clean up a malware infection that had infiltrated its network after an employee clicked on a URL in a phishing email. Unbeknownst to the company as it scrambled to quell the attack, the malware, which was later identified as Trickbot, was quietly spreading among its endpoints and servers, gathering intel about the manufacturing firm and stealing credentials from the compromised machines.

It wasn't until the morning of Sunday, Oct. 14, when C.E. Niehoff IT manager Kelvin Larrue logged into the company's network from home, that it became clear to the company that the attack was something much more serious than a bot infection. A stunned Larrue could see that an intruder was running a PowerShell session on one of the company's servers, moving from server to server with stolen credentials and disabling security tools.

"I could see what he was actually doing. I knew we were in real trouble and someone was in our system," Larrue recalls. "They literally had the keys to the kingdom."

Larrue jumped into his car and drove the 20-minute route to the data center on the company's campus in Evanston, Ill., which houses its corporate headquarters and the manufacturing plant where it builds heavy-duty alternators for government and emergency vehicles. Racing to shut down the network in order to shut out the attacker, Larrue and his team pulled the plug in hopes of preventing the attacker from getting any deeper into the network, but it was too late.

"By that time, the perpetrator had done extensive damage to our network," he says. The attacker had begun dropping ransomware: "He had started routines to encrypt files on all of the servers and any workstations he happened to be on at that point," Larrue says.

What Larrue was witnessing firsthand, he later learned, was a Ryuk ransomware attack on his company. Ryuk is part of the recent generation of ransomware variants that is typically used for custom and targeted attacks on bigger and potentially more financially lucrative targets. According to Check Point Security, which has studied Ryuk and its attack methods, Ryuk's authors built it with an encryption scheme that targets critical resources and assets in a victim's network; for maximum impact, its payload is released manually by the attackers once they have the intel and stolen credentials they need.

"When [Ryuk attackers] infect a new victim, they can stay for a while to observe the network ... and see if the infected machine or network is interesting," explains Itay Cohen, a security researcher with Check Point who tracks Ryuk. "They do not automatically drop Ryuk; they drop it manually" if they decide it's a useful target. That's a departure from earlier ransomware attack campaigns that were more random and automated, he says.

Ryuk has claimed several high-profile victims since the fall of 2018, including newspapers such as the Chicago Tribune and the Los Angeles Times; the city of Stuart, Fla.; and Onslow Water and Sewer Authority, which was hit with a ransomware attack in October 2018, around the same time frame as C.E. Niehoff.

Ryuk and other ransomware, such as GandCrab and LockerGoga, which crippled Norwegian aluminum manufacturer Norsk Hydro, are all about targeting what CrowdStrike calls "big game," or large organizations theoretically able to pay a higher ransom than randomly infected consumers or small organizations.  

Larrue says C.E. Niehoff believes the malicious URL in the phishing email that dropped Trickbot was the first phase of its attack and where the intel-gathering and credential-stealing occurred. In some Ryuk attacks on other victims, the gang has used Emotet as the bot and Trickbot as the intel- and credential-stealer in advance of dropping Ryuk and locking down the victims' machines.

"What was happening behind the scenes was that Trickbot got in and set up the whole command-and-control thing, and we later found out what was actually going on. They siphoned off credentials, set up the C2, and then we got hit with the big one," the Ryuk ransomware, Larrue explains.

While he and his team were "chasing our tails" trying to quell the infection's spread, the attackers had set up a reverse-shell attack, he says, possibly exploiting an unpatched vulnerability in Java. The company's Vipre anti-malware tools didn't recognize or catch the variant.

With the stolen credentials, the Ryuk attackers then set up Remote Desktop Protocol (RDP) connections to the network and, via the PowerShell commands, set off the Ryuk ransomware payload, server by server, he says.

But what Larrue and his team didn't realize at the time was that unplugging machines from the network actually exacerbated the attack: The Ryuk attackers apparently had set the attack to corrupt the firmware of the infected machines if the ransomware's encryption process was disrupted. Larrue and his team of three IT staffers had not seen the ransom note warning them not to shut down or risk their systems getting corrupted when they frantically did the shutdown; they finally got a look at the message in the wake of the response.

"They were expecting us to come in Monday morning [to the ransom message]," he says. "They didn't expect us on Sunday."

Unplugging the machines "was a mistake on my part," Larrue adds. "Part of the encryption scheme ... was if we did pull the plug, something would corrupt the firmware on all the servers," including the manufacturing firm's email and ERP servers.

"At that point it was totally lost. Even if we wanted to pay ransom, we couldn't," he recalls.

It turned out the ransom note had warned that only the attackers could help decrypt the files, and that resetting or shutting down systems could damage the files. It didn't include a ransom fee, but instead instructions on how to proceed in working with the attackers to get the files decrypted.

"I've had bad days in my life, but I've never had one like that," Larrue says. "I had the weight of the world bearing down on me." 

Paper and Pen
C.E. Niehoff is a relatively small, privately held manufacturing firm, with 400 employees and a three-person IT department that also works on security issues. Its customers include the US military, which uses its industrial alternators for vehicles, for instance. One of the first worries in the wake of the attack was the loss of its ERP manufacturing server to the Ryuk attack.

The good news was the attackers hadn't stolen any customer or sensitive information, but the bad news was the manufacturing process had to rely on paper and existing orders to keep the shop floor open. "We had enough paperwork to keep the manufacturing floor running on jobs already issued," Larrue says. "The ERP system provides information to execute on the shop floor, but we can still produce without it. Production didn't come to a grinding halt."

But "we couldn't see too far into the future" until the ERP system was back online, he recalls.

By some stroke of luck, the company's human resources and payroll server wasn't infected with ransomware. Neither was its two backup appliances, although there were signs the attackers had tried to encrypt the Arcserve 8200 Series devices but had failed for some unknown reason. One appliance sat in Building A, and the other in Building B, on the campus, and were set to run a data backup rotation and handle file compression for terabytes of the firm's data.

"So this was more or less all we had," as well as some older backup tapes that only contained data for the past four years, Larrue says.

And C.E. Niehoff had not actually set up the appliances for full system recovery yet — the devices were relatively new — so Larrue had to get help from an Arcserve engineer/technician to restore the backups to the new computers, which the manufacturing firm had to quickly purchase to replace the compromised systems. A couple of the systems that had been configured for bare-metal restoration were back online quickly, he recalls, but there were challenges with several other systems that had not been configured for full restoration.

"We had to more or less rebuild the machine," which took longer to restore, he says.

One way to keep backup systems safe from ransomware attacks is to keep them on a separate domain, advises Gary Sussman, the Arcserve engineer who helped Larrue restore the manufacturing company's systems. He also recommends setting them with strong credentials and ensuring that hardware encryption "is turned on."  

In all, it took C.E. Niehoff two-and-half weeks to get all of its systems fully back up and running, starting with its email server. 

Larrue says the company since has added additional layers of security and is working on beefing up redundancy in its systems and storage, including some cloud-based storage. Ransomware threats are the new normal.

"The lessons learned here is this is an ongoing campaign and it's not going to stop," he says of the threat of ransomware attacks.

Related Content:

Kelly Jackson Higgins is the Executive Editor of Dark Reading. She is an award-winning veteran technology and business journalist with more than two decades of experience in reporting and editing for various publications, including Network Computing, Secure Enterprise ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
packetfusion11
50%
50%
packetfusion11,
User Rank: Apprentice
11/1/2019 | 3:02:42 AM
virtual pbx
You're constantly bombarded with marketing materials and promises that cloud communications will help transform your business. But there are over 400 Unified Communications vendors and dozens of different services. Figuring out where to start can be a serious challenge.

 
REISEN1955
100%
0%
REISEN1955,
User Rank: Ninja
5/21/2019 | 10:21:50 AM
Re: When all is said and done
Good point - always amazing to me how many users of any category still fall for an invoice, an infected word file and such.  If you don't need it, don't read it, delete it.

In January of 2014, before moving to Georgia, a 501C3 museum I supported and administered was wiped out by Cryptolocker - executive workstation to server and everything was gone.  I am damn proud of my response.  I had a basement full of computers, some dedicated to backups of clients.  Infection broke 3:45 am.  At 8am I picked up the dedicated system, put it into car and drove to museum.  It had the same defacto NAME as the server, so hooking it in enabled all data access, backed up earlier, just without active directory.  Then I was able to analyze and restore the server.  This took 3 hours, 98% recovery with only desktop of executive station lost.  Not too damn bad. 
Kelly Jackson Higgins
100%
0%
Kelly Jackson Higgins,
User Rank: Strategist
5/20/2019 | 6:35:09 PM
Re: When all is said and done
This wasn't included in my piece, but the user who fell for the phish was actually a security-savvy one, according to Larrue. So it just goes to show anyone can fall for a good phish.
REISEN1955
0%
100%
REISEN1955,
User Rank: Ninja
5/20/2019 | 3:07:57 PM
When all is said and done
Line 3 - an employee clicked on an email link.  Education!!!   If you don't need it, don't read it, delete it.  All this nightmare could have been prevented by a bit of knowledge and a few hours of training.  Should be mandatory for any user in any corporation these days.  How much did that employee directly COST the company?  Perhaps put it out of business (which did happen in a way).   Start at step 1 and all else follows.  Oh, and tested backup and restore protocol?  Or is just keeping servers running away the job of an IT staff.  
I 'Hacked' My Accounts Using My Mobile Number: Here's What I Learned
Nicole Sette, Director in the Cyber Risk practice of Kroll, a division of Duff & Phelps,  11/19/2019
DevSecOps: The Answer to the Cloud Security Skills Gap
Lamont Orange, Chief Information Security Officer at Netskope,  11/15/2019
Attackers' Costs Increasing as Businesses Focus on Security
Robert Lemos, Contributing Writer,  11/15/2019
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Current Issue
Navigating the Deluge of Security Data
In this Tech Digest, Dark Reading shares the experiences of some top security practitioners as they navigate volumes of security data. We examine some examples of how enterprises can cull this data to find the clues they need.
Flash Poll
Rethinking Enterprise Data Defense
Rethinking Enterprise Data Defense
Frustrated with recurring intrusions and breaches, cybersecurity professionals are questioning some of the industrys conventional wisdom. Heres a look at what theyre thinking about.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2019-19033
PUBLISHED: 2019-11-21
Jalios JCMS 10 allows attackers to access any part of the website and the WebDAV server with administrative privileges via a backdoor account, by using any username and the hardcoded dev password.
CVE-2019-19191
PUBLISHED: 2019-11-21
Shibboleth Service Provider (SP) 3.x before 3.1.0 shipped a spec file that calls chown on files in a directory controlled by the service user (the shibd account) after installation. This allows the user to escalate to root by pointing symlinks to files such as /etc/shadow.
CVE-2019-15511
PUBLISHED: 2019-11-21
An exploitable local privilege escalation vulnerability exists in the GalaxyClientService installed by GOG Galaxy. Due to Improper Access Control, an attacker can send unauthenticated local TCP packets to the service to gain SYSTEM privileges in Windows system where GOG Galaxy software is installed....
CVE-2019-16405
PUBLISHED: 2019-11-21
Centreon Web 19.04.4 allows Remote Code Execution by an administrator who can modify Macro Expression location settings.
CVE-2019-16406
PUBLISHED: 2019-11-21
Centreon Web 19.04.4 has weak permissions within the OVA (aka VMware virtual machine) and OVF (aka VirtualBox virtual machine) files, allowing attackers to gain privileges via a Trojan horse Centreon-autodisco executable file that is launched by cron.