How To Handle Patch Overload

Special to Dark Reading

There are Patch Tuesdays, and then there are mega-Patch Tuesdays like this month's, when Microsoft released a record-tying number of 13 security bulletins fixing 26 vulnerabilities. Handling this heavy load of patches -- many of them requiring system shutdowns and reboots -- with minimal disruption to business and the rare risk of the patches themselves causing problems is no easy feat.

Much of it depends on how well-prepared your business is for handling the lighter, more typical patch releases. Those releases can help prepare you for the "big one," like February's cycle.

Prep work for any Patch Tuesday cycle should include careful review of the Microsoft material that precedes patch releases, says Jason Miller, security and data team manager at Shavlik Technologies.

"Researching each bulletin is key. Microsoft issues advanced notification every Thursday before Patch Tuesday," he says. "The information can give administrators an idea of what is coming, and how it will affect their business. Reviewing this information can save time and allow for planning. In addition, Microsoft has been supplying more relevant information on these notifications that aid in this effort."

And when the release is a whopper like this month's?

"Large patch days require additional resources, but the same techniques used for light days are applicable," Wolfgang Kandek, CTO of Qualys.

These resources -- available IT personnel, test platforms that match company configurations for ensuring the patch won't cause more problems than it solves, and firm understanding and coordination of the effect of reboots on business and operations -- can themselves be configured in ways that make mammoth patch releases more manageable.

This largely depends on a company's existing patch strategy. Automated patching and reporting can ease some of the burden, Miller says. "Patch maintenance windows may expand, though, as companies deal with troubleshooting any patch that may have failed to deploy or caused issues with systems and business applications," he says. "Some patch strategies require each patch to be tested against machines that have business applications. This ensures the patches will not cause widespread issues that can cripple a business. With large releases, there are delays in patching because the testing takes more time."

As with trimming patch deployment time and other strategies for increasing the efficiency and safety of patch deployments, the key is to have a solid, thorough understanding of all of affected systems and operations, deploying and installing the patches in stages that match the specific needs and peculiarities of those configurations. Qualys' Kandek suggests a two-tier approach to large patch deployments, such as the February Microsoft Advisory. "By partitioning the organization's computers into fast and slow patch groups, companies can attack the problem in pieces," he says. "A fast patch group contains machines that essentially run standard software and do not have any special needs. In the fast group, the patch testing that vendors have performed before releasing the patch to the public should be sufficient, and longer internal testing of patches can be avoided."

More complex or custom configurations, which Kandek calls the "slow patch group," are where companies will take things more slowly and apply more of their resources preinstallation. "The slow group contains systems that have complex setups, run legacy applications, or are systems that have had problems in the past," he says.

And it's best to hold back on patching complex, difficult, or legacy systems too quickly. Gather information by monitoring public news sources for coverage of the patches, as well as following interest groups (i.e., patchmanagement.org) for notification of potential problems, and invest in internal testing in QA and staging setups for specific configurations, Kandek recommends.

Shavlik's Miller points to a trend in Microsoft's monthly security bulletins during the past three years, where a large patch release one month is followed by a smaller one the next month. "It is important to note what each bulletin addresses. The February patch day had 13 bulletins released; this was the same number of bulletins that were released last October, although October's patch day was much more difficult due to the number of products that were affected," he says. "The number of products affected will increase the amount of patching that actually needs to incur."

While the mountain of Microsoft patches this month is the exception rather than the rule, Miller advises that companies get used to increasing numbers of patches -- and not just from Microsoft.

"A bigger challenge for businesses will be addressing non-Microsoft patches. There has been an increase in active exploits for products such as Adobe Reader and Apple QuickTime. These products also need to be addressed, which can make patching systems very difficult," he says.

Kandek says the number of patches will continue to climb. "We are likely to see increases in patches in the near future," he says, the increase being one result of having more products installed on even "slow group" computers. "Attackers that in the past focused on the operating system have started to attack applications such as Adobe Reader."

Adobe, in fact, has been standardizing its own patch release schedule for the same reasons of efficiency and consolidation that led Microsoft to implement Patch Tuesdays. But like Microsoft, Adobe occasionally encounters surprises that must be dealt with immediately: Just this week, for example, the company released two emergency Reader patches out-of-cycle.

Attackers will continue to extend their reach as current applications become more robust and new applications become more common. "Think Twitter clients, music players, etc.," Kandek says. "IT administrators need to have an accurate picture of their installed software and monitor all applicable sources for their patching intelligence."

Doing so might not lessen the burden of large patch releases, but it can make managing their deployment less daunting and risky, as well as providing for emergency off-cycle patch releases and increasing the efficiency and effectiveness of dealing with normal patch release schedules that arrive on time -- and in small numbers.

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Dark Reading's editors directly, send us a message.