Back to Basics with Log Management, SIEMs & MSSPs

For security professionals and forensics investigators, it doesn't get any more basic than the unrelenting flow of log data generated by countless machines attached to the enterprise backbone.

The activity, health, status, and anomalies of these endpoints and systems are time-stamped and delivered to some repository, usually a syslog, a security information and event management (SIEM) tool, or some managed security service provider's MSSP) archive in the cloud. The vast majority of these log records will never be reviewed or needed; for many organizations, they're simply checking a box by archiving log data to comply with state, federal, and international laws.

This fundamental collection of health and performance data has been going on for decades. Why, then, do so many organizations still manage to fumble the details of log data and actually undermine the very security they're charged with protecting?

The short answer: Log data can be really challenging to manage, and it requires some heavy-lifting on the part of the entire organization – not just the IT department and security professionals – to consider why its collecting log data and where. More on that in a minute.

The discomfiting example of Target isn't so flattened on the highway of infosec forensics that we can't run it over one more time. Overinstrumented and drowning in alert messages, Target's IT and security teams completely missed multiple intrusions, malware payloads, and massive theft of credit card data in 2013. It's noteworthy that in the bitterly competitive retail sector, none of Target's competitors rushed to proclaim, "That could never happen here." Their forbearance masks an acknowledgment that everyone can do a better job, not just with alerts but all their log data.

With that context in mind, here's some advice from experts and a CISO about how to get the basics right with log data, your SIEM vendor, or MSSP.

Pick Your Log Sources Carefully
Every source we talked to for this article agreed vehemently on two essential points: Collecting every single bit of log data is a mistake. And the log data you do collect must be mapped back to some business requirement or strategic issue. Otherwise, you end up boiling the ocean. 

"The most common mistake CISOs and others who want enterprise visibility make is they onboard every log source they can get their hands on," confirms James Carder, CISO at SIEM vendor LogRhythm and formerly the director of security informatics at the Mayo Clinic.

But more is not more in this instance; in log management, more leads to overload for those who must manage and analyze the data. It's also a dubious choice from a cost perspective: Though storage prices have dropped dramatically thanks to the cloud, it simply makes no financial sense to store all log data from every attached machine.

More logging data also means a greater volume of false-positives, the bane of any security pro's existence. Chasing down false-positives isn't an activity where any organization wants its security staff spending precious time. False-positives waste money, they're bad for morale, and, worst of all, they don't really improve the security profile of the organization.

Carder also encourages security pros to watch their assumptions, since they often have preconceptions or don't understand what's important to the business. In a previous job with the Mayo Clinic, Carder says he assumed clinical research data was critical to protect, only to find out that the organization gave away that data for free. "I made an incorrect assumption," he notes. And it changed how he collected log data.

Therefore, dig in and do the due diligence; otherwise your log data collection efforts are likely to fall short, if not fail. Carder's advice? Take a look at your business and threat sectors, and then map out end-to-end use cases of log data to get a clearer picture of the organization's needs. "What you get is a business case for what you're trying to solve," he adds. And you can leverage that information as you move from one division of the organization to another and get buy in to your log data initiative.

Another useful resource is NIST document 800-92, "Guide to Computer Security Log Management." It's a thorough, high-level discussion of logging concepts and technologies, not a step-by-step guide to implementing them. Though written in 2006, the document is time-tested and still useful, according to Mike Lloyd, CTO of security analytics vendor RedSeal. "It's not a holy bible, but 800-92 is still a really good document that precisely addresses security log management," he says.

Got the Time?
Once the business use cases are settled, there's the basic issue – frequently overlooked – of how to set the timestamping for each log message that gets sent and stored. This is particularly challenging for organizations that span multiple time zones, whether daylight saving time gets observed, and the inevitable internal politics and pushback ("Corporate wants us to do what?").

"Time is a big issue," notes RedSeal's Lloyd. But it's essential the organization specifies a time zone to use for syncing up its machinery, apps, and log alerts. He's a big advocate of standardizing on Coordinated Universal Time, or UTC, as it's referred to internationally with its French acronym. This is what gets used in most aeronautical applications and systems; most IT networking gear also has UTC defaults.

"UTC is what pilots use, otherwise the airways would be chaos. But it may be impossible to get everybody's laptop to use it," Lloyd says. He recommends an approach that's a variation of begging forgiveness rather than asking permission. "Go as far as you can [with UTC] till the organization stops you," he says.

Bad actors rarely break into just one machine, and deconstructing the sequence of their actions requires logs that can be easily correlated against a time line. "If you don't have a good measure of time, you can't reconstruct the right order of what they did," Lloyd adds.

Organizing Your Logging Sources
Another potential hurdle to effective log management is how you organize your sources of log data – hierarchical, geographical, by business unit? There are also details about logging data that can be customized, like which time zone you use for timestamping, an essential piece for correlating events as part of incident response and any forensics investigation.

LogRhythm's Carder is a big advocate of dividing up sources (and their software) by function:

--Endpoint sources: Servers, computers, laptops, and mobile.

--Network data: Routers, switches, Wi-Fi access points, and others.

--End-user behavior data: Networks and applications accessed, changes in authorizations, volumes of data uploaded or downloaded.

By monitoring log alerts in these categories, security pros can more quickly pinpoint where the trouble is in real time – and where it's not. "Even if end-point and network behaviors look normal, there are behavioral shifts that could still bubble up that indicate a person isn’t who they say they are," Carder explains.

But tracking these three areas can help isolate, mitigate, or prevent incidents if the security pros associated with each side of the triangle are monitoring alerts and communicating with each other.

Implementing SIEM Functions
Once the strategic aspects of logging data have been resolved, end-user organizations must then turn to another piece of heavy lifting: How will they manage log data? With the emergence of security information and event management (SIEM) software in the past 10 years, customers get some combination of intelligence and automation to analyze security alerts.

Typically, SIEMs collect and store log files from a variety of sources and systems, and perform some combination of real-time monitoring, event correlation, and analysis. SIEMs help move organizations closer to a meta view of their security IT infrastructure – that "single pane of glass" management vendors have promised for years.

And in the age of cloud computing and applications on demand, managed security service providers increasingly offer some version of SIEM as a service. While organizations in highly regulated, compliance-bound industries (financial services, healthcare) may be restricted from using third parties for security management, the MSSP model has nonetheless attracted plenty of customers who want to contain ballooning security costs.

Another model has SIEM vendors partnering with MSSPs to help customers get the most from their log data and keep their security posture strong. But some caveats apply here as well, according to Fred Kwong, CISO for Delta Dental Plans Association. To serve his member organizations and their customized requirements, he's working with both SIEM vendors and an MSSP to preserve members' options.

"Some SIEMs work better in the cloud than others, and some SIEM vendors are better partnered with cloud vendors," Kwong explains. Delta needed support for portability of SIEM as a Service, without a lot of operational overhead for its members to preserve transition options for the future.

One size for SIEM definitely doesn't fit all for Delta's members, and it underscores the need for understanding different business units' requirements and objectives. "Know the use case and what you need from the tool to make that happen," Kwong says. "It makes a big difference."

And while new or emerging technology can simplify and aggregate vast amounts of log data, it still costs money, notes LogRhythm's Carder. And that reinforces the imperative to understand what data you're collecting and why. "AI or machine learning may help you on the analytics side, but knowing which logs you need to be effective is the key," he adds.

Related Content: