Incident Response Prep Extends Beyond Tools, Training

Whenever you read information on how to perform forensics and incident response, there is a preparation phase that comes before anything else. Preparation steps cover how to prepare for dealing with an incident in your environment -- but what about making sure your environment is ready for an incident?I've seen many incidents in which the team responsible for its handling had the standard forensic and IR tools in place. Their jump bag was ready to go with hard drives, forensic imaging equipment, laptops, etc., but when they approached the system in question, it was a proprietary or legacy system that wasn't supported by the typical cadre of forensic software available.

While legacy systems and proprietary systems may be absolutely necessary in some situations, more often I've seen them left hanging around because it's easier (and cheaper) to let them do what they do than to upgrade them to a more modern OS that's less esoteric and easier for IR staff to handle. This is a situation where "if it ain't broke, don't fix it" can seriously throw a monkey wrench into an investigation.

Legacy systems, however, are not the source of so many other problems I've come across. The number one core issue is changes made by system administrators and help desk staff during the course of managing systems and during troubleshooting that don't take into consideration what impact it will have if one of those systems is compromised and must be analyzed.

The best example is an issue that resulted from cached pages in an ERP Web portal. The users were experiencing odd behavior with stale pages and values in forms. Troubleshooting the issue led to a "fix," by having the cache and history cleared upon exit, or not record at all. Shortly after the "fix" was put in place, two incidents occurred that were complicated by it. One dealt with inappropriate Web surfing and the other was malware contracted through Web browsing.

I've seen several others, including turning off firewalls to troubleshoot a problem and never turning it back on, logging disabled not to be re-enabled, and other oversights. Some of the problems are from simple forgetfulness after solving a problem and not having any sort of auditing to confirm that settings were what they should be. The Web caching and history problem, though, was a workaround that no one ever considered the ramifications on other areas, such as IR and forensics.

There's no simple solution to preparing your environment to be ready for an incident. Knowing what's there and making sure you have the necessary tools to handle them is key, but so is having clearly defined policies that get regularly reviewed and updated to address changes in the environment. Security needs to be plugged into all the IT processes in an organization and not just be something that one group is responsible for.

This is one of those problems that I'm sure permeates organizations no matter what industry you're in. Leave a comment or send me an e-mail if you've run into the same problem and how you solved it (if you were able to).

John H. Sawyer is a senior security engineer on the IT Security Team at the University of Florida. The views and opinions expressed in this blog are his own and do not represent the views and opinions of the UF IT Security Team or the University of Florida. When John's not fighting flaming, malware-infested machines or performing autopsies on blitzed boxes, he can usually be found hanging with his family, bouncing a baby on one knee and balancing a laptop on the other. Special to Dark Reading.