Digital Detectives: Making The Most Of Your Incident Response And Forensics Efforts

Security professionals are overworked, stressed, and don't have enough hours in the day to do their jobs. Why? Too often it's because they're reacting, fighting security fires as they happen rather than providing proactive protection.

Reactive security is a common pitfall caused by staff shortages, lack of skills, and an overall low level of preparedness. Security pros need to make sure they're executing the full security incident response cycle: Prepare for problems, identify incidents, fix the problem and restore systems to service, and then loop back to preparation to make sure the lessons learned are applied to head off future problems.

It's essential today because there are now so many targets inside a company. "Before, with your firewall up and servers secured, you were relatively safe," says Chris Cuevas, a senior security analyst at the consulting firm Secure Ideas, who spent 13 years as a system administrator. "Now, everything is a target, from your cellphone to your printer and your webcam."

Prepare For Success

Companies need to implement routine activities to identify security incidents. Your staff will need to do daily log reviews to look for anomalous activities, do IDS event reviews, and follow up on user reports of abnormal system behavior. Events that surface from these reviews may prove unimportant, but they could provide the first hint of a serious incident in progress.

Preparation also includes staff knowledge. Regularly reading security news sites and blogs--such as DarkReading.com, SANS Internet Storm Center, and Krebs On Security; Twitter; and mailing lists, such as the ones at SecLists.Org--is needed to keep security pros current on threats and the tools to counter them. Experience isn't a substitute for knowing the latest security trends.

For formal training, some great programs are available from companies such as Mandiant and the SANS Institute that can be used to supplement the knowledge of experienced staff and get new employees quickly up to speed. The courses start with the basics of incident response and delve into advanced forensic topics, including memory analysis and malware reverse engineering.

But training alone isn't enough. Staff must have a clear process for how to effectively respond to incidents and work together constructively. "Teamwork and experience are critical to the success of an incident-response program," says Don Weber, a senior consultant at security consulting firm InGuardians.

To do that, security teams also need the right tools in place. The basic premise behind incident response is to have as little impact on the system at issue as possible while still getting the data needed to identify and contain the problem.

The tools you choose to use for a specific incident will depend on where it occurs and the particular responder's experience and training with the tools. Just because one tool works great on Windows 2000 doesn't mean it will work on Windows 7, and you certainly don't want to throw an advanced forensics tool like EnCase into the hands of someone who hasn't been trained in forensics and how the tool should be used.

Tool selection efforts must be synchronized with training strategies. Most training courses include a basic methodology that's based on a set of tools that have been vetted in a range of incident-response situations and tested to determine their accuracy and impact on a system. Examples include tools from AccessData, Guidance Software, Mandiant, and Microsoft Sysinternals. Others can be found in the books and cheat sheets listed on the next page.

Volatile Data, Respond With Care

Just the act of collecting information from a live system can lose volatile data--such as memory contents, network connections, open files, and the list of running processes--so you must collect that data first. The order in which you run your tools typically follows the "order of volatility" described by Dan Farmer and Wietse Venema in their book Forensic Discovery.

The recently released Tr3Secure Volatile Data Collection Script is a great example of a tool that considers the order of volatility as it collects data from Windows systems. The script first creates a forensic copy of the contents of the physical memory, then collects volatile data such as processes, network information, users logged on, open files, clipboard contents, and system information--anything that would be lost if power to the system were turned off. It then collects nonvolatile data like software installed, users and groups, and computer devices.

One common, and problematic, response to a security incident is to go right to the system in question and start running commands--kind of like what we see on TV. That approach goes against the goal of treading lightly during the incident-response process.

In an enterprise environment, there are countless places that incident-related information can be found and analyzed without ever touching or even running a tool on the target system. That's an important point, because if an attack is still going on, attackers will likely notice response efforts and try to cover their tracks by clearing system logs, or worse.

Centralized log management and a security information and event management (SIEM) systems are ideal sources of data that can be analyzed without interacting with the system in question. Intrusion-detection systems, antivirus software, network firewalls, backup servers, and network flow data are a few others that provide insight into an incident by detecting downloaded malware, port scans by a suspect system, and attempts to exploit a network-based service.

Social Media Exposure

To get a better idea of how these types of systems figure into a response effort, let's look at a fictitious, but all too real, scenario where an employee's system is compromised after the person browsed a social networking site. It's a big risk: 75% of companies in InformationWeek's 2011 Strategic Security Survey (see chart below) cite malware from malicious links as a risk from social networks.

Suppose an employee browsing a social networking site is tricked into clicking a link from someone who appeared to be a friend. The link turns out to be to a malicious website that exploits a flaw in a popular document viewer. Malware is then installed on the person's computer, letting an attacker pilfer sensitive company data and probe the internal corporate network for additional information.

Using an SIEM, an incident responder can identify the social network the person visited because a company's Web proxy, which monitors employee Internet usage, logged the activity. The subsequent download of executable content and repeated requests to a website overseas indicate the user's workstation is infected and is phoning home for further instructions. System logs and network traffic show installation of several back doors that are accepting incoming network connections from other compromised systems.

In addition to SIEM, there are other centralized logging tools and anti-malware software on the user's workstation. Antivirus software might detect a few pieces of the malware that were installed but not catch all of them. Subsequent components of the malware may turn off the antivirus and install new services. Those are all useful clues that should show up in the central logs and can be analyzed--even a lack of logs can indicate a problem because the malware may have cleared them in order to cover its tracks.

The network layer also is rich with log data from an IDS, firewall, and the routers themselves that can export network flow data. Backup servers also hold clues in situations where attackers wipe a server hard disk or have access for an extended period and add or delete files.

You must look beyond the compromised system to see the big picture and find all the information you need. Always ask where else in your network is there information that could help in determining what's going on. Then pull that data together, and analyze and cross-reference it with resources on the Internet to further identify and contain the incident.