{Image 1}

Two men are walking through a forest. Suddenly, they see a bear off in the distance, running toward them. Adrenaline pumping, they start running away. But then one of them stops, takes some running shoes from his bag and starts putting them on.

“Frank, what are you doing?” says the other man. “Do you think you will run faster than the bear with those?”

“I don’t need to run faster than the bear,” Frank replies. “I just have to run faster than you.”

This scenario repeats itself every time a new security vulnerability is discovered in a widely used open source component. Imagine the bear as your adversary. Rushing to attack when easy prey is present. Your response time is critical.

Sneakers on. Go!

Racing to Answer Four Questions

Late last year, security researchers from FoxGlove Security announced a critical vulnerability in applications using Apache Commons Collection. The component is so widely used in Java development, countless organizations were impacted. The FoxGlove report revealed vulnerable instances of the component in open source applications like WebLogic, WebSphere, JBoss and Jenkins. Simultaneously, Java development and IT operations teams around the globe raced to discover four things:

As soon as the vulnerabilities were announced, the bears were charging to their doorstep. Response times were vital.

Lessons learned at PayPal

Earlier this year, PayPal shared its own experience with the deserialization vulnerability. Their engineering team blogged, “we began forking a few work-streams to assess the impact to our application infrastructure … As it turned out, this bug was found in one of our apps that was not on our core Java frameworks. Once we validated the bug, the product development and security teams quickly moved to patch the application in a couple of days.”

Knowing they were not alone in the development community racing to fix a vulnerable application, PayPal shared several lessons learned, like:

When it comes to eliminating known vulnerabilities in critical applications, we can see from PayPal that speed, focus and community engagement all play a critical role in staying ahead of the bear.

Houston, we have a problem

When I first read about the vulnerability deserialization issue, I reached out to our Sonatype security team. They were already on the case to assess any impact to our own products (luckily, we were not impacted). My next step was reaching out to my friends at CloudBees (creators of Jenkins). Thankfully, their development team was already on top of it and released a newer version within just a few days of the alert.

But, the story does not end there. Researchers at Sonatype have identified deserialization issues in over 30,000 unique components stored in our company’s Central Repository.

The issue of vulnerabilities is far reaching. When we started looking at the download patterns of open source components from the Central Repository, we noticed the download consumption patterns of the known vulnerable versions of popular components were undistinguished from the good ones. People are using known vulnerable components for years and years after safer versions of those same components have been announced and released. In many cases, the open source projects themselves are not aware of vulnerabilities they are harboring.

 Why is this? It is not because people intend to build flawed software. It’s most likely, because they were not aware of the problem.  This is because:

Why components age like milk

We also realize that components sometimes age like milk. Components that were once good can sour instantly when new vulnerabilities are discovered within them. When that happens, the race is on. The bears are coming, and you have to respond.

[Read more about vulnerability management from Derek Weeks in 5 Steps to Improve Your Software Supply Chain Security.]

For some, the pace of their response resembles walking from the attacking bear. They have no idea what components have been consumed over the years across their software development practices. No records are kept. When a new vulnerability is announced (assuming they hear about it), the hunt reveals a manual, step-by-step analysis.

In one specific instance, a global banking entity told me that it took four people two months to detect if they were using vulnerable versions of Commons Collection across their application portfolio. (They had no running shoes.)

For others, it is a sprint at world-record pace. Organizations like Capital One, Intuit and FedEx have full visibility to the software supply chain that feeds components into their development organization. They know what components have been used and where. When new vulnerabilities are announced, they are automatically alerted to the use and location of that component.  

But responding at record pace is not an anomaly. You have to work at it. Global manufacturing giants like Toyota, Gilead Sciences, Apple and Lockheed have figured out how to run production lines at a ludicrous velocity while maintaining extremely low defect rates by continuously managing their supply chains. Knowing the bears won’t relent, it is time and it is possible for those managing software supply chains to do the same.

Related Content:

Gain insight into the latest threats and emerging best practices for managing them. Attend the Security Track at Interop Las Vegas, May 2-6. Register now!