Vulnerabilities / Threats //

Advanced Threats

10/24/2017
10:30 AM
Teri Radichel
Teri Radichel
Commentary
Connect Directly
Twitter
LinkedIn
RSS
E-Mail vvv
50%
50%

Why Patching Software Is Hard: Technical Challenges

Huge companies like Equifax can stumble over basic technical issues. Here's why.

First of a two-part post.

Like everyone else, I wish that Equifax had patched the software flaw that caused the breach right away. However, I also understand why this is difficult. I was an employee of one of the largest banks in the United States, with over 45,000 employees. For the first few years, I worked as a development lead for a team that created and updated applications that processed transactions for billions of dollars in investment assets. I later worked in public cloud, network, and security engineering roles helping teams across the entire company move applications to production.

Patching one software vulnerability on a few servers sounds easy. However, patching that one vulnerability in the context of thousands of devices, software applications, and software libraries across multiple locations and lines of business is another story.

Tracking Devices, Applications, and Software Libraries
Companies need to have an inventory of every single device that runs software. Equifax has close to 10,000 employees worldwide. Each person may have computers, phones, tablets, and other devices. The company must track every piece of hardware connected to its network and every virtual machine in a public cloud environment. Additionally, the company needs to know all software it runs, including operating systems, applications, and software libraries running on each of those devices. Some of the software doesn't have an automated update or notification process. Companies must vet the software to make sure that update process is not delivering malicious code, as happened in recent cases involving NotPetya, CCleaner, and malicious libraries in public Python repositories.

According to Crunchbase, Equifax acquired 16 companies between 1995 and 2017. Each time a company buys another company, myriad new technologies and software libraries are part of that acquisition. The acquiring company needs to make sure all software is up to date on the company systems it has acquired. Acquisitions involve many complex issues, and patching may not be a top priority. Merging different networks and IT systems is complicated and can take up to a year or longer. Acquisitions and restructuring may mean companies have different lines of business. Different people may manage software in various parts of the organization.

Updating Critical, Complex, and Legacy Applications
Many applications may share a single software library. Updating that library can break processes handling millions or billions of dollars in transactions. The company must test each application that uses the upgraded library before deploying a new version to production. In one case, it took a development team months to update a custom-built library to a new version of Java. The team had to test over 50 different financial processing applications that depended on that library before deploying them into production and removing the old version of the library.

Testing complex legacy applications can be challenging. Imagine all the rules related to US tax laws for a company that handles investment transactions. There are hundreds of variations that can occur that change the tax implications of a trade and what must appear on tax forms. The type of change made to the system will dictate how many of those variations a development team will need to test to ensure any tax or financial processing by the system works correctly. Hopefully, documentation exists for the application, or someone still works at the company who knows how to test infrequently updated legacy applications.

In some cases, installing and testing a patch is extremely risky. A software patch can break devices that cost millions of dollars, such as SCADA systems, medical devices, and research lab systems. No spare machine exists that system administrators can use to test the software update in advance. Patching the software may cause operations at an organization to cease. In the case of a medical facility, it could be a literal matter of life and death.

Patching Solutions and Alternatives
Just because patching is hard doesn't mean companies can ignore the problem. Organizations need to invest time and money into solutions that automate software deployments and track software inventory. If companies are not aware what software exists in the organization, they won't be able to make sure it is all up to date. When patching is very risky, companies can limit network access to the port that exposes the vulnerability or turn off the vulnerable features of the software.

In addition, companies should move legacy software to new software architectures with security designed in from the start. Companies can measure the return on this investment based on the cost other companies are facing due to massive data breaches. Additionally, if this keeps happening, companies should consider the cost of increased legislation designed to prevent data breaches — some of which may add overhead without solving the problem, like regulations related to PCI-DSS compliance

In the second part of this two-part post, I examine the organizational challenges involved. 

Related Content:

Join Dark Reading LIVE for two days of practical cyber defense discussions. Learn from the industry’s most knowledgeable IT security experts. Check out the INsecurity agenda here.

Teri Radichel is the Director of Security Strategy and Research at WatchGuard Technologies. She was on the initial team that helped Capital One move to the cloud, implementing security controls and networking for multiple lines of business. She joined WatchGuard Technologies ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
tradichel
50%
50%
tradichel,
User Rank: Apprentice
10/27/2017 | 2:33:41 PM
Fixing the patching problem...
Thank you for the comments. I posted a second article related to organizational challenges faced by employees in different roles in the company who may be responsible for patching. Patching likely needs to be addressed at a level above the individuals who might actually do the work. I hope you will read the second article and share your thoughts on that as well.
REISEN1955
50%
50%
REISEN1955,
User Rank: Ninja
10/27/2017 | 8:24:17 AM
Re: Patching Legacy
Outsourcing per se is not an intelligence problem though it is often easy, and sometimes unfair, to bash a help desk in Bangalore.  I have found some highly intelligent folks there (as anywhere really) and the are to be treasured.  But an outsource firm has it's own VESTED interest at heart first - getting paid and looking after their own firm first.  This is NOT how it shold be but - real world - this is true.  Client interests come in second, sometimes third (if it is IBM doing the work). 
Dr.T
50%
50%
Dr.T,
User Rank: Ninja
10/26/2017 | 12:06:30 PM
16 companies?
Article mentioned Equifax acquired 16 companies between 1995 and 2017, what exactly they are doing nobody knows then, you would think credit status check company would not need that many acquisition.

 
Dr.T
50%
50%
Dr.T,
User Rank: Ninja
10/26/2017 | 12:03:49 PM
Re: Outsource
"It is my understanding that Equifax outsourced IT support "

Most companies do that, that should not mean that there is less of chance being secure. I would think Equifax did not want to spend enough money to keep themselves secure.
Dr.T
50%
50%
Dr.T,
User Rank: Ninja
10/26/2017 | 12:01:44 PM
Patching Legacy
As article suggested, we should start getting rid of legacy applications , they do not belong to today's world, patching will not take us anywhere.
REISEN1955
50%
50%
REISEN1955,
User Rank: Ninja
10/25/2017 | 10:15:10 AM
Outsource
It is my understanding that Equifax outsourced IT support to that once fine American company - IBM.  Now, that means mostly India and that means train wreck in situations that require real heft.   If it is HARD to do, as it seems to be at old Equi, then that is what IT is PAID FOR if staff is internal and if you have outsourced staff then you deserve the level of support you pay for - generally BAD in my experience.  But hey, reduces that nasty salary expense and benefits, RIGHT????
5 Reasons the Cybersecurity Labor Shortfall Won't End Soon
Steve Morgan, Founder & CEO, Cybersecurity Ventures,  12/11/2017
BlueBorne Attack Highlights Flaws in Linux, IoT Security
Kelly Sheridan, Associate Editor, Dark Reading,  12/14/2017
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Current Issue
The Year in Security: 2017
A look at the biggest news stories (so far) of 2017 that shaped the cybersecurity landscape -- from Russian hacking, ransomware's coming-out party, and voting machine vulnerabilities to the massive data breach of credit-monitoring firm Equifax.
Flash Poll
The State of Ransomware
The State of Ransomware
Ransomware has become one of the most prevalent new cybersecurity threats faced by today's enterprises. This new report from Dark Reading includes feedback from IT and IT security professionals about their organization's ransomware experiences, defense plans, and malware challenges. Find out what they had to say!
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2017-0290
Published: 2017-05-09
NScript in mpengine in Microsoft Malware Protection Engine with Engine Version before 1.1.13704.0, as used in Windows Defender and other products, allows remote attackers to execute arbitrary code or cause a denial of service (type confusion and application crash) via crafted JavaScript code within ...

CVE-2016-10369
Published: 2017-05-08
unixsocket.c in lxterminal through 0.3.0 insecurely uses /tmp for a socket file, allowing a local user to cause a denial of service (preventing terminal launch), or possibly have other impact (bypassing terminal access control).

CVE-2016-8202
Published: 2017-05-08
A privilege escalation vulnerability in Brocade Fibre Channel SAN products running Brocade Fabric OS (FOS) releases earlier than v7.4.1d and v8.0.1b could allow an authenticated attacker to elevate the privileges of user accounts accessing the system via command line interface. With affected version...

CVE-2016-8209
Published: 2017-05-08
Improper checks for unusual or exceptional conditions in Brocade NetIron 05.8.00 and later releases up to and including 06.1.00, when the Management Module is continuously scanned on port 22, may allow attackers to cause a denial of service (crash and reload) of the management module.

CVE-2017-0890
Published: 2017-05-08
Nextcloud Server before 11.0.3 is vulnerable to an inadequate escaping leading to a XSS vulnerability in the search module. To be exploitable a user has to write or paste malicious content into the search dialogue.