Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Vulnerabilities / Threats //

Advanced Threats

10:30 AM

Why Patching Software Is Hard: Technical Challenges

Huge companies like Equifax can stumble over basic technical issues. Here's why.

First of a two-part post.

Like everyone else, I wish that Equifax had patched the software flaw that caused the breach right away. However, I also understand why this is difficult. I was an employee of one of the largest banks in the United States, with over 45,000 employees. For the first few years, I worked as a development lead for a team that created and updated applications that processed transactions for billions of dollars in investment assets. I later worked in public cloud, network, and security engineering roles helping teams across the entire company move applications to production.

Patching one software vulnerability on a few servers sounds easy. However, patching that one vulnerability in the context of thousands of devices, software applications, and software libraries across multiple locations and lines of business is another story.

Tracking Devices, Applications, and Software Libraries
Companies need to have an inventory of every single device that runs software. Equifax has close to 10,000 employees worldwide. Each person may have computers, phones, tablets, and other devices. The company must track every piece of hardware connected to its network and every virtual machine in a public cloud environment. Additionally, the company needs to know all software it runs, including operating systems, applications, and software libraries running on each of those devices. Some of the software doesn't have an automated update or notification process. Companies must vet the software to make sure that update process is not delivering malicious code, as happened in recent cases involving NotPetya, CCleaner, and malicious libraries in public Python repositories.

According to Crunchbase, Equifax acquired 16 companies between 1995 and 2017. Each time a company buys another company, myriad new technologies and software libraries are part of that acquisition. The acquiring company needs to make sure all software is up to date on the company systems it has acquired. Acquisitions involve many complex issues, and patching may not be a top priority. Merging different networks and IT systems is complicated and can take up to a year or longer. Acquisitions and restructuring may mean companies have different lines of business. Different people may manage software in various parts of the organization.

Updating Critical, Complex, and Legacy Applications
Many applications may share a single software library. Updating that library can break processes handling millions or billions of dollars in transactions. The company must test each application that uses the upgraded library before deploying a new version to production. In one case, it took a development team months to update a custom-built library to a new version of Java. The team had to test over 50 different financial processing applications that depended on that library before deploying them into production and removing the old version of the library.

Testing complex legacy applications can be challenging. Imagine all the rules related to US tax laws for a company that handles investment transactions. There are hundreds of variations that can occur that change the tax implications of a trade and what must appear on tax forms. The type of change made to the system will dictate how many of those variations a development team will need to test to ensure any tax or financial processing by the system works correctly. Hopefully, documentation exists for the application, or someone still works at the company who knows how to test infrequently updated legacy applications.

In some cases, installing and testing a patch is extremely risky. A software patch can break devices that cost millions of dollars, such as SCADA systems, medical devices, and research lab systems. No spare machine exists that system administrators can use to test the software update in advance. Patching the software may cause operations at an organization to cease. In the case of a medical facility, it could be a literal matter of life and death.

Patching Solutions and Alternatives
Just because patching is hard doesn't mean companies can ignore the problem. Organizations need to invest time and money into solutions that automate software deployments and track software inventory. If companies are not aware what software exists in the organization, they won't be able to make sure it is all up to date. When patching is very risky, companies can limit network access to the port that exposes the vulnerability or turn off the vulnerable features of the software.

In addition, companies should move legacy software to new software architectures with security designed in from the start. Companies can measure the return on this investment based on the cost other companies are facing due to massive data breaches. Additionally, if this keeps happening, companies should consider the cost of increased legislation designed to prevent data breaches — some of which may add overhead without solving the problem, like regulations related to PCI-DSS compliance

In the second part of this two-part post, I examine the organizational challenges involved. 

Related Content:

Join Dark Reading LIVE for two days of practical cyber defense discussions. Learn from the industry’s most knowledgeable IT security experts. Check out the INsecurity agenda here.

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
10/27/2017 | 2:33:41 PM
Fixing the patching problem...
Thank you for the comments. I posted a second article related to organizational challenges faced by employees in different roles in the company who may be responsible for patching. Patching likely needs to be addressed at a level above the individuals who might actually do the work. I hope you will read the second article and share your thoughts on that as well.
User Rank: Ninja
10/27/2017 | 8:24:17 AM
Re: Patching Legacy
Outsourcing per se is not an intelligence problem though it is often easy, and sometimes unfair, to bash a help desk in Bangalore.  I have found some highly intelligent folks there (as anywhere really) and the are to be treasured.  But an outsource firm has it's own VESTED interest at heart first - getting paid and looking after their own firm first.  This is NOT how it shold be but - real world - this is true.  Client interests come in second, sometimes third (if it is IBM doing the work). 
User Rank: Ninja
10/26/2017 | 12:06:30 PM
16 companies?
Article mentioned Equifax acquired 16 companies between 1995 and 2017, what exactly they are doing nobody knows then, you would think credit status check company would not need that many acquisition.

User Rank: Ninja
10/26/2017 | 12:03:49 PM
Re: Outsource
"It is my understanding that Equifax outsourced IT support "

Most companies do that, that should not mean that there is less of chance being secure. I would think Equifax did not want to spend enough money to keep themselves secure.
User Rank: Ninja
10/26/2017 | 12:01:44 PM
Patching Legacy
As article suggested, we should start getting rid of legacy applications , they do not belong to today's world, patching will not take us anywhere.
User Rank: Ninja
10/25/2017 | 10:15:10 AM
It is my understanding that Equifax outsourced IT support to that once fine American company - IBM.  Now, that means mostly India and that means train wreck in situations that require real heft.   If it is HARD to do, as it seems to be at old Equi, then that is what IT is PAID FOR if staff is internal and if you have outsourced staff then you deserve the level of support you pay for - generally BAD in my experience.  But hey, reduces that nasty salary expense and benefits, RIGHT????
Windows 10 Migration: Getting It Right
Kevin Alexandra, Principal Solutions Engineer at BeyondTrust,  5/15/2019
Artist Uses Malware in Installation
Dark Reading Staff 5/17/2019
Baltimore Ransomware Attack Takes Strange Twist
Kelly Jackson Higgins, Executive Editor at Dark Reading,  5/14/2019
Register for Dark Reading Newsletters
White Papers
Cartoon Contest
Write a Caption, Win a Starbucks Card! Click Here
Latest Comment: This comment is waiting for review by our moderators.
Current Issue
Building and Managing an IT Security Operations Program
As cyber threats grow, many organizations are building security operations centers (SOCs) to improve their defenses. In this Tech Digest you will learn tips on how to get the most out of a SOC in your organization - and what to do if you can't afford to build one.
Flash Poll
The State of Ransomware
The State of Ransomware
Ransomware has become one of the most prevalent new cybersecurity threats faced by today's enterprises. This new report from Dark Reading includes feedback from IT and IT security professionals about their organization's ransomware experiences, defense plans, and malware challenges. Find out what they had to say!
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2019-05-20
In GoHttp through 2017-07-25, there is a stack-based buffer over-read via a long User-Agent header.
PUBLISHED: 2019-05-20
eLabFTW 1.8.5 is vulnerable to arbitrary file uploads via the /app/controllers/EntityController.php component. This may result in remote command execution. An attacker can use a user account to fully compromise the system using a POST request. This will allow for PHP files to be written to the web r...
PUBLISHED: 2019-05-19
There is XSS in browser/components/MarkdownPreview.js in BoostIO Boostnote 0.11.15 via a label named flowchart, sequence, gallery, or chart, as demonstrated by a crafted SRC attribute of an IFRAME element, a different vulnerability than CVE-2019-12136.
PUBLISHED: 2019-05-18
MacDown 0.7.1 (870) allows remote code execution via a file:\\\ URI, with a .app pathname, in the HREF attribute of an A element. This is different from CVE-2019-12138.
PUBLISHED: 2019-05-17
Typora (1913) allows arbitrary code execution via a modified file: URL syntax in the HREF attribute of an AREA element, as demonstrated by file:\\\ on macOS or Linux, or file://C| on Windows. This is different from CVE-2019-12137.