Security information and event monitoring (SIEM) systems are typically used to create a forensics trail so that evidence of digital intrusions can be investigated after an attack is discovered. Increasingly, however, SIEM and log management systems have the ability to detect attacks in real time, security experts say.

While many security professionals have wished for such capabilities, only recently has the technology caught up with the promise of catching attackers through real-time correlation of disparate networking events, says David Pack, manager of LogRhythm Labs, a maker of SIEM systems. A combination of factors is leading to the evolution of the systems: The technology can better handle the volume of log data generated by corporate systems, the capabilities of the correlation engines have grown, and the researchers developing such systems better understand attack patterns, he says.

"What is going on in the SIEM industry is that it is becoming a real-time monitoring solution -- it's becoming the SIEM 2.0 that it should be," Pack says. "All the framework and pieces are there to turn this into real-time analysis and real-time alarming."

In a presentation at the United Security Summit in San Francisco, Pack spelled out ways that log management systems can use advanced correlation to detect attackers and showed the signs that a typical attacker will leave in the logs. While many security firms focus on detecting attackers' attempts to exploit vulnerabilities and compromise systems, event correlation can put together evidence from other systems to detect an attack.

In a spear-phishing incident, for example, a log trail can show the user's Outlook e-mail client connecting to an Exchange server and downloading a message with a malicious attachment, the user's Adobe Acrobat opening up a PDF file, and, following exploitation, see the client communicating with an unknown Internet address.

"Pretty much every piece of that activity is logged somewhere, often away from where the exploit happened," Pack says. "What is interesting about that is you don't have to worry about whether the exploited vulnerability is known or a zero-day. You only have to worry about how [the attack] is delivered and what happens afterward."

Information security managers are increasingly looking for a greater awareness of what is happening on their networks. In addition, they are being asked to look at a greater variety of threats than in the past, says Mark Seward, senior director of security and compliance for network-intelligence firm Splunk.

"It used to be that I -- as the information security manager -- was only responsible for the information security infrastructure -- listening to it and reacting to it," Seward says. "Now I'm being asked to look at malicious insiders and being asked to look at real-time threats to the business."

[ Any security monitoring system comes with a certain amount of good old-fashioned alerting, but interpreting those alerts in the context of a whole running system is something else entirely. See The Most Expensive Part Of The Monitoring System. ]

Turning networking and physical events that have been logged into intelligence about what is happening on the network is necessary in today's world, Seward says.

In the lion's share of breaches, for example, the attacker leaves traces in the log files. In its latest Data Breach Investigations Report, telecommunications and security provider Verizon found that 86 percent of incidents could have been detected from the events collected in the logs -- if the victim's staff had only been looking.

Part of the problem is that the evidence is lost in the massive volume of log data, which has increased dramatically, says Chris Novak, managing principal with Verizon. In the past, when his investigations team visited a potential client and asked for log data, the company's executives would often give them a blank look, he says. Now, because of compliance regulations, companies are collecting the data, but not generally mining it for useful data on security events.

"Now many of your larger organizations are storing their logs indefinitely," Novak says. "But ... organizations have so much log data that it has created a new problem that they don't know how to mine and search and correlate that data to make it useful, so it just becomes something we store."

With better correlation engines, companies could not only better use the data, but detect attacks in real time. The first step is to create a baseline for the company so that strange behavior can be detected. Then technology providers and their customers can describe complex patterns that may indicate an attack. LogRhythm uses self-contained rules, called rule blocks, to build more complex structures. Some half of the company's technical workers are developing rules and patterns to translate device logs into patterns that can be detected, LogRhythm's Pack says.

"The real-time alerting has been there for simple issues," he says. "But now we are able to do it for complex behavioral patterns."

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.