Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


09:30 AM
Connect Directly

How a Manufacturing Firm Recovered from a Devastating Ransomware Attack

The infamous Ryuk ransomware slammed a small company that makes heavy-duty vehicle alternators for government and emergency fleet. Here's what happened.

The tiny IT team at C.E. Niehoff & Co. had been working for two weeks to run down and clean up a malware infection that had infiltrated its network after an employee clicked on a URL in a phishing email. Unbeknownst to the company as it scrambled to quell the attack, the malware, which was later identified as Trickbot, was quietly spreading among its endpoints and servers, gathering intel about the manufacturing firm and stealing credentials from the compromised machines.

It wasn't until the morning of Sunday, Oct. 14, when C.E. Niehoff IT manager Kelvin Larrue logged into the company's network from home, that it became clear to the company that the attack was something much more serious than a bot infection. A stunned Larrue could see that an intruder was running a PowerShell session on one of the company's servers, moving from server to server with stolen credentials and disabling security tools.

"I could see what he was actually doing. I knew we were in real trouble and someone was in our system," Larrue recalls. "They literally had the keys to the kingdom."

Larrue jumped into his car and drove the 20-minute route to the data center on the company's campus in Evanston, Ill., which houses its corporate headquarters and the manufacturing plant where it builds heavy-duty alternators for government and emergency vehicles. Racing to shut down the network in order to shut out the attacker, Larrue and his team pulled the plug in hopes of preventing the attacker from getting any deeper into the network, but it was too late.

"By that time, the perpetrator had done extensive damage to our network," he says. The attacker had begun dropping ransomware: "He had started routines to encrypt files on all of the servers and any workstations he happened to be on at that point," Larrue says.

What Larrue was witnessing firsthand, he later learned, was a Ryuk ransomware attack on his company. Ryuk is part of the recent generation of ransomware variants that is typically used for custom and targeted attacks on bigger and potentially more financially lucrative targets. According to Check Point Security, which has studied Ryuk and its attack methods, Ryuk's authors built it with an encryption scheme that targets critical resources and assets in a victim's network; for maximum impact, its payload is released manually by the attackers once they have the intel and stolen credentials they need.

"When [Ryuk attackers] infect a new victim, they can stay for a while to observe the network ... and see if the infected machine or network is interesting," explains Itay Cohen, a security researcher with Check Point who tracks Ryuk. "They do not automatically drop Ryuk; they drop it manually" if they decide it's a useful target. That's a departure from earlier ransomware attack campaigns that were more random and automated, he says.

Ryuk has claimed several high-profile victims since the fall of 2018, including newspapers such as the Chicago Tribune and the Los Angeles Times; the city of Stuart, Fla.; and Onslow Water and Sewer Authority, which was hit with a ransomware attack in October 2018, around the same time frame as C.E. Niehoff.

Ryuk and other ransomware, such as GandCrab and LockerGoga, which crippled Norwegian aluminum manufacturer Norsk Hydro, are all about targeting what CrowdStrike calls "big game," or large organizations theoretically able to pay a higher ransom than randomly infected consumers or small organizations.  

Larrue says C.E. Niehoff believes the malicious URL in the phishing email that dropped Trickbot was the first phase of its attack and where the intel-gathering and credential-stealing occurred. In some Ryuk attacks on other victims, the gang has used Emotet as the bot and Trickbot as the intel- and credential-stealer in advance of dropping Ryuk and locking down the victims' machines.

"What was happening behind the scenes was that Trickbot got in and set up the whole command-and-control thing, and we later found out what was actually going on. They siphoned off credentials, set up the C2, and then we got hit with the big one," the Ryuk ransomware, Larrue explains.

While he and his team were "chasing our tails" trying to quell the infection's spread, the attackers had set up a reverse-shell attack, he says, possibly exploiting an unpatched vulnerability in Java. The company's Vipre anti-malware tools didn't recognize or catch the variant.

With the stolen credentials, the Ryuk attackers then set up Remote Desktop Protocol (RDP) connections to the network and, via the PowerShell commands, set off the Ryuk ransomware payload, server by server, he says.

But what Larrue and his team didn't realize at the time was that unplugging machines from the network actually exacerbated the attack: The Ryuk attackers apparently had set the attack to corrupt the firmware of the infected machines if the ransomware's encryption process was disrupted. Larrue and his team of three IT staffers had not seen the ransom note warning them not to shut down or risk their systems getting corrupted when they frantically did the shutdown; they finally got a look at the message in the wake of the response.

"They were expecting us to come in Monday morning [to the ransom message]," he says. "They didn't expect us on Sunday."

Unplugging the machines "was a mistake on my part," Larrue adds. "Part of the encryption scheme ... was if we did pull the plug, something would corrupt the firmware on all the servers," including the manufacturing firm's email and ERP servers.

"At that point it was totally lost. Even if we wanted to pay ransom, we couldn't," he recalls.

It turned out the ransom note had warned that only the attackers could help decrypt the files, and that resetting or shutting down systems could damage the files. It didn't include a ransom fee, but instead instructions on how to proceed in working with the attackers to get the files decrypted.

"I've had bad days in my life, but I've never had one like that," Larrue says. "I had the weight of the world bearing down on me." 

Paper and Pen
C.E. Niehoff is a relatively small, privately held manufacturing firm, with 400 employees and a three-person IT department that also works on security issues. Its customers include the US military, which uses its industrial alternators for vehicles, for instance. One of the first worries in the wake of the attack was the loss of its ERP manufacturing server to the Ryuk attack.

The good news was the attackers hadn't stolen any customer or sensitive information, but the bad news was the manufacturing process had to rely on paper and existing orders to keep the shop floor open. "We had enough paperwork to keep the manufacturing floor running on jobs already issued," Larrue says. "The ERP system provides information to execute on the shop floor, but we can still produce without it. Production didn't come to a grinding halt."

But "we couldn't see too far into the future" until the ERP system was back online, he recalls.

By some stroke of luck, the company's human resources and payroll server wasn't infected with ransomware. Neither was its two backup appliances, although there were signs the attackers had tried to encrypt the Arcserve 8200 Series devices but had failed for some unknown reason. One appliance sat in Building A, and the other in Building B, on the campus, and were set to run a data backup rotation and handle file compression for terabytes of the firm's data.

"So this was more or less all we had," as well as some older backup tapes that only contained data for the past four years, Larrue says.

And C.E. Niehoff had not actually set up the appliances for full system recovery yet — the devices were relatively new — so Larrue had to get help from an Arcserve engineer/technician to restore the backups to the new computers, which the manufacturing firm had to quickly purchase to replace the compromised systems. A couple of the systems that had been configured for bare-metal restoration were back online quickly, he recalls, but there were challenges with several other systems that had not been configured for full restoration.

"We had to more or less rebuild the machine," which took longer to restore, he says.

One way to keep backup systems safe from ransomware attacks is to keep them on a separate domain, advises Gary Sussman, the Arcserve engineer who helped Larrue restore the manufacturing company's systems. He also recommends setting them with strong credentials and ensuring that hardware encryption "is turned on."  

In all, it took C.E. Niehoff two-and-half weeks to get all of its systems fully back up and running, starting with its email server. 

Larrue says the company since has added additional layers of security and is working on beefing up redundancy in its systems and storage, including some cloud-based storage. Ransomware threats are the new normal.

"The lessons learned here is this is an ongoing campaign and it's not going to stop," he says of the threat of ransomware attacks.

Related Content:

Kelly Jackson Higgins is the Executive Editor of Dark Reading. She is an award-winning veteran technology and business journalist with more than two decades of experience in reporting and editing for various publications, including Network Computing, Secure Enterprise ... View Full Bio

Recommended Reading:

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
11/1/2019 | 3:02:42 AM
virtual pbx
You're constantly bombarded with marketing materials and promises that cloud communications will help transform your business. But there are over 400 Unified Communications vendors and dozens of different services. Figuring out where to start can be a serious challenge.

User Rank: Ninja
5/21/2019 | 10:21:50 AM
Re: When all is said and done
Good point - always amazing to me how many users of any category still fall for an invoice, an infected word file and such.  If you don't need it, don't read it, delete it.

In January of 2014, before moving to Georgia, a 501C3 museum I supported and administered was wiped out by Cryptolocker - executive workstation to server and everything was gone.  I am damn proud of my response.  I had a basement full of computers, some dedicated to backups of clients.  Infection broke 3:45 am.  At 8am I picked up the dedicated system, put it into car and drove to museum.  It had the same defacto NAME as the server, so hooking it in enabled all data access, backed up earlier, just without active directory.  Then I was able to analyze and restore the server.  This took 3 hours, 98% recovery with only desktop of executive station lost.  Not too damn bad. 
Kelly Jackson Higgins
Kelly Jackson Higgins,
User Rank: Strategist
5/20/2019 | 6:35:09 PM
Re: When all is said and done
This wasn't included in my piece, but the user who fell for the phish was actually a security-savvy one, according to Larrue. So it just goes to show anyone can fall for a good phish.
User Rank: Ninja
5/20/2019 | 3:07:57 PM
When all is said and done
Line 3 - an employee clicked on an email link.  Education!!!   If you don't need it, don't read it, delete it.  All this nightmare could have been prevented by a bit of knowledge and a few hours of training.  Should be mandatory for any user in any corporation these days.  How much did that employee directly COST the company?  Perhaps put it out of business (which did happen in a way).   Start at step 1 and all else follows.  Oh, and tested backup and restore protocol?  Or is just keeping servers running away the job of an IT staff.  
Inside the Ransomware Campaigns Targeting Exchange Servers
Kelly Sheridan, Staff Editor, Dark Reading,  4/2/2021
Beyond MITRE ATT&CK: The Case for a New Cyber Kill Chain
Rik Turner, Principal Analyst, Infrastructure Solutions, Omdia,  3/30/2021
Register for Dark Reading Newsletters
White Papers
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you today!
Flash Poll
How Enterprises are Developing Secure Applications
How Enterprises are Developing Secure Applications
Recent breaches of third-party apps are driving many organizations to think harder about the security of their off-the-shelf software as they continue to move left in secure software development practices.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2021-04-15
An issue was discovered in Zulip Server before 3.4. A bug in the implementation of replies to messages sent by outgoing webhooks to private streams meant that an outgoing webhook bot could be used to send messages to private streams that the user was not intended to be able to send messages to.
PUBLISHED: 2021-04-15
An issue was discovered in Zulip Server before 3.4. A bug in the implementation of the can_forge_sender permission (previously is_api_super_user) resulted in users with this permission being able to send messages appearing as if sent by a system bot, including to other organizations hosted by the sa...
PUBLISHED: 2021-04-15
An issue was discovered in Zulip Server before 3.4. A bug in the implementation of the all_public_streams API feature resulted in guest users being able to receive message traffic to public streams that should have been only accessible to members of the organization.
PUBLISHED: 2021-04-15
In the topic moving API in Zulip Server 3.x before 3.4, organization administrators were able to move messages to streams in other organizations hosted by the same Zulip installation.
PUBLISHED: 2021-04-15
The issue navigation and search view in Jira Server and Data Center before version 8.5.12, from version 8.6.0 before version 8.13.4, and from version 8.14.0 before version 8.15.1 allows remote attackers to inject arbitrary HTML or JavaScript via a DOM Cross-Site Scripting (XSS) vulnerability caused ...