A few days ago, FireEye made a stunning announcement: Its network had been breached by attackers backed by the Russian government. Now we know that there are even more victims. SolarWinds, a popular provider of tools to manage networks and IT systems, appears to be the vector the state-backed attackers used to get in. Its customer list is a who's who of major corporations, claiming more than 400 of the Fortune 500, not to mention an extensive list of US government agencies.

It's now reported that the US government has been breached by the same attackers, and FireEye has stated that it is seeing attacks against "government, consulting, technology, telecom and extractive entities in North America, Europe, Asia and the Middle East." The scope of the attack is still far from clear, and the list of those known victims is growing steadily. For the US government, the list has expanded to a number of departments, including State, Defense, Homeland Security, Treasury, and Commerce, with possible impact elsewhere, including some vital to national security, such as the National Security Administration and Las Alamos National Laboratory. 

In a rare move, the Cybersecurity & Infrastructure Security Agency released an emergency directive to shut down SolarWinds Orion systems immediately. This is a move to protect these important networks, though it comes with its own problems, losing the monitoring and services provided. During an incident like this, a clear line of sight into network traffic, logs, and system activity is vital to determining impact; by leveraging these systems in the attack, the malicious actors have compounded the problem for those potentially affected. 

SolarWinds boasts a customer list of more than 300,000 entities — though based on an Securities and Exchange Commission filing, about 33,000 of those customers are using the affected Orion suite of products and 18,000 of those may have been affected. While 18,000 possibly impacted is far better than 300,000, it's still hard to overstate the possible effects of this attack. The value of information that may have been stolen is incalculable, and the results of this attack could be felt for years.

While there is much to be learned, some things are already becoming clear, including the mechanism that the attackers used to launch their attacks — inserting malicious code into an application update. Software supply chain attacks have been a major concern, and have been used in the past to great effect. Crafting a malicious update, especially one that includes proper code signing, is complicated, and often requires deep access to the vendor's development systems. 

Developer machines, source control management systems, build servers, or even sites that developers download tools from may be compromised, giving an attacker an entry point to inject malicious code. Too often, these are the weakest links in the chain, and attackers will always focus on the weak links. There's no need to spend the time and effort to attack the hard targets when there are easier options available; attackers — especially those that work for state-backed operations — have deadlines too.

Due to the nature of software development, these systems too often don't have the level of monitoring, access controls, and security hardening that other systems in a network do. While this makes troubleshooting easier for developers, it also makes it easier for attackers to get in and to remain undetected while they explore and insert their backdoors.

While details of how the attackers were able to backdoor the software updates delivered to customers are scarce, a few things have come out: The attack seems to have started with SolarWinds' email and the attackers collecting sensitive information; the attackers then compromised the build systems used for the Orion products — this would allow them to inject the malicious code into the product without it being added to the source code management system. 

Compromising the SolarWinds' build system in turn allowed attacks against their customers, such as FireEye; the data stolen from FireEye and other victims could potentially be leveraged to attack others, or even create new backdoors and malicious updates. One breach leads to another, and may lead to yet more. Attacks like this can have ripple effects that endure and spread surprisingly far.

For instance, as recently as earlier this year, a malicious change made to a single file, SolarWinds.Orion.Core.BusinessLayer.dll, distributed as a hotfix to the application affected untold business and agencies around the world.

There are always many lessons that can be learned from events like this, and as the story continues to become clearer in the days and weeks ahead, more opportunities to learn will likely be exposed. One clear lesson that the community has been discussing for years: Supply chain attacks can have a truly massive impact and should be carefully considered.

For a company that releases software that is used on its customers' networks, ensuring that it doesn't cause those customers harm is a sacred duty. While Hippocrates may or may not have used the exact phrase "first, do no harm," it's still an important consideration for all those that have trust placed in them, medical or otherwise. There is a moral duty to protect customers, and it should be taken seriously. This is not to lay blame at the feet of anyone involved (other than the attackers) but is meant as a reminder of the importance of finding the weak links and fixing them.

Any device, system, application, or service that can result in malicious code being added to a software release should be carefully monitored, secured, and treated with the utmost care to minimize the risk to customers. Trusting, for example, a build server to work and be secure without the proper caution can lead to disaster. Trusting, for example, a developer's laptop to be free of malicious influence can lead to disaster. Trusting development systems without applying the proper security controls places everyone at risk.

While perfect security is an impossible goal, risks should be understood, controls should be in place, and care should be taken at every opportunity to provide the safest possible software.