Whether to improve performance, gather business intelligence, or detect security threats, log management boils down to three steps: Collect the logs, store the data, and analyze the data to identify patterns.

Yet, while the collection and analysis of log data is one of 20 critical security controls identified by the SANS Institute, most companies do not regularly collect and analyze their logs unless required by regulations. With so much data, information technology professionals can be confused as to where to start, says Nicole Pauls, product manager for SolarWinds, a maker of IT management and monitoring software.

"When people come to log management, they are flooded with a lot of data," she says. "What people are trying to find are the anomalies, the patterns that hint at something going on, but it's difficult."

Good security log analysis revolves around four principals, says Ben Feinstein, director of operations and development for Dell SecureWorks' Counter Threat Unit. First, companies need to monitor the right logs, including data from firewalls, virtual private networking (VPN) appliances, Web proxies, and DNS servers. Next, the security team must collect data on what "normal" looks like inside the company's network. Third, analysts must identify the indicators of attacks in their log files. Finally, the security group must have a procedure for responding to incidents identified by log analysis.

"Just pulling all these logs into your SIEM systems is not going to get you anywhere if your security team does not know what bad or suspicious looks like to your monitoring system," Feinstein says.

Here are five types of events that companies should be checking, according to security experts.

1. User access anomalies
The Windows security log and the records of Active Directory domain controllers are a good first stop to finding malicious activity on the network. Changes in permissions, users logging in remotely from unknown locations, and users accessing one system and using that system to access another are all possible signs of malicious activity, says Kathy Lam, product marketing manager for HP ArcSight.

"When we look at the types of attacks and how hackers have been getting into the environment, they have typically been inside a network posing as a user for months to longer than a year," she says. "By really looking at the baseline and seeing how current activity deviates from that can really pinpoint attacks."

Especially important are privileged accounts -- those users who have administrator permissions on various systems in the network. Because those accounts have more power in the network, they should be monitored more closely.

[Enterprises have been leveraging big data tools and technologies to analyze everything from consumer buying patterns to competitors' product strategies. See How Enterprises Can Use Big Data To Improve Security.]

2. Patterns that match threat indicators
Companies should also run comparisons between the data in their logs and whatever indicators of compromise they are able to obtain, whether through established blacklists or a more complete threat-intelligence service, SecureWorks' Feinstein says.

Threat indicators can help companies identify suspicious IP addresses, host names, domain names, and malware signatures in firewall, DNS server, or Web proxy logs.

"Web proxy logs are a powerful point of visibility into the Web traffic that is traversing your network -- how your endpoint systems are reaching out to the Web," he says.

3. Configuration changes outside the "window"
Attackers who have gained access to a system will typically try to change configurations to further compromise and gain a more certain foothold in the network.

Because most companies limit configuration changes to a limited time each week, month, or quarter, those malicious configuration changes -- whether to open the system up to attack or just turn off logging -- can be a certain sign that an attack is in progress, says Sanjay Castelino, vice president with SolarWinds.

"Those changes typically happen inside a very narrow window, and so if there are changes happening to the configuration outside of that window, you are going to want to know," he says.

Such analysis can help in certain cases. The rules created to manage security products are typically very complex, and it can be difficult to detect whether the rule is malicious by simple analysis, Castelino says. Instead, security teams will find it easier to flag any changes made outside of a specific maintenance window, he says.

4. Strange database transactions
Because databases are such an important part of a company's infrastructure, the business should monitor database transactions to detect malicious activity. A query that attempts to select and copy a large range of data, for example, should be more closely scrutinized.

In addition, monitoring database communications is not enough. While logging transactions can hamper database performance, a journal of what transactions actually occurred becomes invaluable during investigations of whether any compromise resulted in a successful data breach, says Rob Kraus, director of research for security-management firm Solutionary's Engineering Research Team (SERT).

"When clients ask us what records were accessed and what records can we prove were not accessed, the trail leads up to the database," he says. "If they were not logging, it makes it a real challenge. In the end, unless you are logging database transactions, you cannot say which records were touched."

5. New device-user combinations
Before mobile devices and the bring-your-own-device trend, companies could treat any new devices connecting to the network as suspicious. Now that's no longer a good indicator, SolarWinds' Castelino says.

Instead, companies should link devices to their users and treat changes as incidents, he says.

"You probably still want to flag a device, but you may want to flag devices and users together," he says. "Because if I bring my tablet to work, no one else should be logging in with it."

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.