Vulnerability Management: The Most Important Security Issue the CISO Doesn't Own

This piece was co-written with Amber Record, a security engineer at F5 Networks. In her role, Amber is responsible for leading the company's vulnerability management program.

The number of attacks like the recent one against Equifax have risen dramatically in the last few years, resulting in the exposure of hundreds of millions of private records. Almost without exception there has been some fundamental flaw related to configuration or patching of systems. This trend will continue without systems designed to automatically identify, patch, and close vulnerabilities in core IT systems that can reduce the chance of human error.  We can accomplish this with automation typically found in large operational cloud deployments and the Constant Delivery (CD)/Constant Integration (CI) principles of DevOps.  These principles are already being used to automatically stop active attacks within the information security community and should now extend to IT operations to improve protections and stop the bad guys from getting in at all.

Where do organizations start when they realize that a standardized, managed vulnerability management program does not exist? There are almost too many options. For one, you could create your own vulnerability scanning solution. If you walk into a security team that has more than one solution, you could review them both and select the best option for your needs.  Another way is to establish a program by hiring a consultant to assist in reviewing the existing program or build one for you. No matter which option you choose, experienced contractors can easily provide a very generalized plan and you can then tune it to your company's specific needs. In the end the goal is to achieve a safer environment for your data and applications. Evaluation criteria must include manageability, accuracy, interpretability, and the ability to identify specific actions that server owners can perform.  

Once you've selected an approach, then what? The next step in the process is getting a peer review of your solution and what it might be lacking. Coordinating the plan outside of the organization is necessary to getting the program into a fully functioning state complete with ticketing to remediate vulnerabilities. The primary preference to reduce vulnerabilities is to implement as much automated patching as possible. This has two net effects: it provides protection earlier and reduces the load on application owners to manually patch systems.

Once you implement the technical scanning solution and ticketing support, the process becomes much more simplified. The primary drive to decide what to scan first should be based on risk.  One of the biggest problems with vulnerability management programs is the application owner's fear that scanning will negatively affect their applications. The information security team can throttle scans, target certain times of the day for lower application traffic, and scan applications prior to implementation in production to catch vulnerabilities sooner and reduce application loads.

The sad truth is that vulnerability management programs have either no or extremely limited ability to actively correct the flaws that they find. Most generate tickets that task system maintainers to patch systems or correct improperly configured systems. There are a few systems that either inject code into connection streams or provide generic intrusion prevention signatures that cover up underlying flaws in Internet-facing services, but to limited effect. Looking at the problem from the standard ITIL lens of people, process, and technology, we offer a new approach:

Today, most enterprises still rely on people in IT to manually patch operational IT systems, especially e-commerce and other customer-facing systems. But that's the problem — even when completely accurate vulnerability scans are delivered, there aren't enough people to patch or correct the systems in a timeframe that is relevant to prevent attack. It's a black hole for head count to continue along this sort of path, driving total cost of ownership up with limited return because, to be truly effective, nearly all patches and fixes need to be made without exception.  Instead, IT departments should hire developers that can automate the correction of infrastructure flaws as needed. Changing the makeup of the IT Ops department may create upheaval among workers who are worried about their jobs, but this isn't insurmountable if retraining is offered and embraced.

Switching to automation to address the vulnerability management problem doesn't just require different skills, it requires entirely new processes. Processes designed strictly for manual work in user interfaces won't do the trick. New processes that leverage automation tools are needed to cut out waste and "wait states" that consume time without yielding benefit. Proper testing of the operational impacts of patching and system configuration changes should be integrated with existing DevOps processes in the enterprise to prevent outages. Engineering integration skills to interconnect products like vulnerability scanners and infrastructure management systems are needed for success. The automation created should also provide metrics on the state of the systems under management to ensure that a secure state is truly being achieved.

Finally, new automation technologies will be required to complete a full cycle of vulnerability detection that automatically corrects and verifies that the fixes have been made. Orchestration tools like Puppet and Chef should be integrated with the APIs built into vulnerability scanners and infrastructure management and patching systems to make this vision a reality. It will be challenging for enterprises to accept this level of automation and "trust the machine," but it's a far better option than today's automated intrusion prevention technologies, which can result in false positives that block legitimate traffic and possibly interfere with revenue-generating business systems. 

Related Content: