11. Get an Incident Command System Going
In the sad event that your city is overtaken by ransomware, an incident command system (ICS) will help you organize — providing your response team an action plan and a place to work.
The ICS will also give idle government staff new direction. According to Hinkle, when all the computers went down in Valdez, a lot of people who'd never lived in an analog world were rather at loose ends.
"There's a certain generation of people that don't know how to write. So without their computers, there was a lot of idle time and sitting on hands and 'what do we do now.' ... There was a whole workforce out there that was lost without a computer." he says.
"The ICS model really helped give those people tasks that they could do," says Osburn.
"More than anything it helped keep frivolous requests off Matt's desk," says Hinkle.
12. Isolate Systems and Contain the Impact
Because Valdez didn't know the entry point of infection, they also collected all employees' scattered thumb drives. In an abundance of caution — perhaps the attack was a distraction from a related data exfiltration attack — they froze all the city's credit cards.
"First thing I knew ... You do not want to power down systems," says Valdez IT director Matthew Osburn. "You want to isolate them. And that really saved us in the long run."
For example, even though financial systems were locked down by ransomware, Osburn's team was still able to access the SQL Server, which expedited their ability to get back to issuing payroll and tax receipts.
13. Check for Decryption Keys Online
NoMoreRansom.org maintains a list of published decryption keys. The one you need might very well be there already. Nevertheless, be careful. This might be better done by your professional cyber incident response team.
14. Watch What You Say to Ransomware Operators
Osburn made it very clear that nobody in the city of Valdez communicate with the attackers directly, just as other experts advise that victims let the experts lead communications with criminals.
The city wanted to ensure that the decryption key would work before sending payment. But, the attackers, who often automate their attacks, did not necessarily know what victim they were talking to. So, when communicating about decryption keys, the Valdez response team protected the city's identity.
"[The attackers] said give us a couple filenames, we'll decrypt them for you to show we have that capability and that it works," says Hinkle. "And so we put our group to work finding three files of which I knew the content and which could not be traced back to the City of Valdez.
"Obviously you don't want it on letterhead. Anything that could be traced back. Once they learn that it's the city of Valdez they have, a simple internet search would reveal that we're able to afford more than 4 Bitcoin. Because up to this point they don't know if we're just a small little mom-and-pop grocery stand. ... They deal in bulk."
IP Architects' Pironti also encourages negotiation.
"You never pay full price," he says. Many of the systems are automated, he says, so if you drop one Bitcoin in, even if they asked for five, they might automatically issue the decryption key. "In some cases, they're happy to get anything," he says.
15. Be Careful With Decrypted Files
Osburn didn't just welcome the files back with open arms, though.
"We got our files back," says Osburn, "but I don't trust 'em."
He's carefully reintroducing files, making sure not to invite weaponized documents or new malware into the system.
16. Learn From the Experience
Since the initial attack, Valdez has begun incident response plans and a wholesale overhaul of its IT infrastructure so it never happens again.
"Communities prepare for those natural disasters," says Hinkle. A ransomware outbreak "is one that should be tabletopped or practiced or put in place. There was not one for us. We did a lot of things very well, but this was literally learning as we went."
They have since sped up all their other planned upgrades. Osburn says that what would have been rolled out over three years has now been rolled out over three months instead.
They've upgraded outdated systems to Windows 10 and Server 16. They've virtualized legacy bare metal servers. They're boosting their firewall and endpoint protections.
"We have a different backup system that's not integrated with Active Directory just for this reason," says Osburn. ""There were certain security procedures I was wanting to implement that were a little unpopular. One of them was more difficult passwords. Well, after not having your documents for a month, having your documents and a more difficult password seems like an easy compromise. [The attack] really paved the way to get us where we need to be."
It's such a beautiful and completely necessary overhaul that Osburn seems secretly grateful to the ransomware operators.
"Usually these systems are very hard to upgrade because you can't have downtime ... Well, that decision was made for us."
"While I'd never want to go through this again, I'd say it's probably one of the best things that could have happened to the city. It gave us the opportunity."
(OK, maybe not-so-secretly grateful.)