Sometime after 14:30 UTC on Friday, July 2, network traffic combining three vulnerabilities started compromising scores of Internet-connected Kaseya Virtual System Administrator (VSA) servers hosted by managed service providers. The attackers' code synchronized to a specific time and then hibernated.

At 4:30 p.m. UTC, all within the same second, the compromised servers woke up and ran a command script that disabled a variety of security controls and sent malicious payloads to every system managed by those servers, according to an analysis conducted by Huntress Labs. While security firms are still sifting through the data, reverse engineering has revealed that the attack — from the first packets exploiting dozens of VSA servers, to the deployment of ransomware on the endpoints of hundreds to thousands of MSP customers — took less than two hours.

The speed of automation gave managed service providers and their customers only a very narrow window in which to detect attacks and block them, says John Hammond, a senior threat researcher for Huntress Labs. Companies would have to run frequent monitoring and alerts to have caught the changes, he says.

"Unfortunately, this form of hyperactive logging and detection is rare — managed service providers often don't have the resources, let alone the personnel to frequently monitor massive components of their software and stack," Hammond says. "With that said, the efficacy and potential for human-powered threat hunters is never something to be left out of the equation."

The quick turnaround of the attack underscores the compressed timeline for defenders to respond to automated attacks. The REvil group and its affiliates, who are thought responsible for the attack, scanned for Internet-connected VSA servers and, when found, sent the initial exploit, which chained three vulnerabilities. 

At 14:48 UTC on Friday, July 2, the first packets started hitting on-premise Kaseya VSA servers, according to logs collected from affected MSPs by Huntress Labs. The exploited flaws included an authentication bypass, an arbitrary file upload, and a command injection. The activity continued, until the hibernating processes reactivated at 16:30 UTC, and antivirus firms suddenly started seeing spikes in detections of the ransomware payload. 

In the hour after the attack's activation, between 16:30 and 17:30 UTC, antivirus firm Sophos detected a massive spike in blocked ransomware activity on its endpoints. 

"We started seeing telemetry immediately as the client systems started getting hit," says Sean Gallagher, senior threat researcher at Sophos. "The telemetry spiked all at one time, in a very small time window." After that, the attack most went quiet, he says.

Because Kaseya VSA manages other systems, the software not only has higher privileges — usually administrator privileges — on other systems but also often has exclusions in place so that antivirus software does not flag its activity as malicious. The command-line script that executed at 16:30 UTC on Friday ran a PowerShell script, disabling many security measures, loading in certificates, and running a malicious executable disguised as a certificate, agent.crt.

The final insult: The attackers installed in an obsolete version of Microsoft's antivirus program, Defender, to load in the final ransomware payload.

"It uses an antivirus product to load a virus," Sophos' Gallagher says. "It dropped an obsolete version of Windows Defender that is susceptible to side-load attacks ... and installs a malicious DLL [dynamic linked library] that is named the same as a DLL that Windows Defender would load." Because that was a piece of code signed by Microsoft, it would evade some malware protection as it looks like a legitimate piece of code, even though it is over 6 years old.

The Dutch Institute for Vulnerability Disclosure (DIVD), which had found at least one of the vulnerabilities used in the attack, published more information on the issues on July 7. The vulnerabilities were discovered in April, disclosed to Kaseya, and DIVD had worked with the company while it shored up its security and started producing patches. 

Overall, Kaseya has not "been slacking" and did everything that DIVD expected of them, says Victor Gevers, chairman of DIVD. The company did not have the security processes in place in April to handle the requirements of patching and incident response, but quickly ramped up, he says.

"If you go back through the timeline, a few days after notification, they knew they needed to hire more security people, and they did," he says. "It showed that their security posture was not up to par yet."

On Monday, Kaseya estimated that fewer than 60 customers, each using the on-premises version of the VSA server, had been affected, with fewer than 1,500 total downstream businesses affected. In interviews, security experts expected that number to rise. 

As of 8 a.m. ET on July 7, Kaseya continued to have problems patching the issue and has delayed rolling out a fix to on-premises customers.

'Overblown' Numbers?

While the attackers claim that more than a million endpoints were encrypted by the ransomware, the number is likely overblown, say security experts. However, many companies have suffered disruption due to the attack. The Swedish grocery chain Coop had to close several hundreds of stores on Saturday because of the ransomware attack, and several schools in New Zealand were affected, according to a Reuters report. 

The Biden administration maintained on Tuesday that the attack did minimal damage to US businesses, but intends to put increasing pressure on Russia to curb attackers that act from within its borders. "If the Russian government cannot or will not take action against criminal actors residing in Russia, we will take action or reserve the right to take action on our own," White House press secretary Jen Psaki said Tuesday, according to the Washington Post.

The attack could have been worse. At the onset, about 2,200 vulnerable VSA servers were connected to the Internet, according to data from DIVD. Without a patch from Kaseya, or even a notification or workaround, the MSPs hosting those servers would have been hard pressed to defend against the attack, says Corey Nachreiner, chief security officer at WatchGuard Technologies.

"In many cases, there are no real protections against zero-day network exploits, which may leave people blind to the indicators of that attack until after that fact," he says. "That said, there are security solutions that did detect the ransomware involved and could prevent it from an individual endpoint perspective."

While the use of a previously unknown vulnerability and the fast, automated attack may lead to many calling the attack a zero-day exploit, Huntress Labs' Hammond takes issue with that description.

"In my mind, a zero-day is defined as the defenders having zero days to prepare. But Kaseya had already been working with DIVD, so I have to put an asterisk around the notion they had zero days to prepare."

The small and midsize business clients of the managed service providers subscribed to their services because they did not have the expertise to manage their own technology. The vendors and MSPs need to take responsibility for their security, Hammond says.

"We have been a bit vocal about these services, by design, giving administrative access and godlike superpowers on all the potential clients," he says. "Vendors and companies, including us, have to review the source code, having that internal red teaming, and being absolute certain to make sure that the technology is hardened to the world and secure."