Searching For Security's Yardstick

Despite rising threats, most security organizations still don't have clear metrics for measuring their performance -- or their enterprises' security posture
But no matter how customized your measurement system, every company needs some basic metrics to start off with, notes Steven Piliero, who heads up the Benchmarks Division of the Center for Internet Security (CIS). The CIS Consensus Information Security Metrics benchmark is perhaps the closest thing the industry has to a set of standards for security metrics today.

"There are three kinds of metrics: those that are broad enough and well-enough understood that they can be used across industries; those that are industry-sector specific; and those that are organization-specific," Piliero says. "We're helping to define that first category, the metrics that many industries can use."

The CIS Consensus defines some basic metrics that organizations can measure frequently, as companies do with certain financial numbers, or as hospitals measure post-surgical infection rates, Piliero says. "They're a starting point for building out your metrics -- some unambiguous standards for measuring specific security functions."

"It is possible to get some level of agreement on high-level metrics," Rothman agrees. "CIS Consensus is a great resource to kick-start the metrics effort."

The CIS Consensus offers standardized methods for tracking measurable activities, such as the frequency of incidents and the time/cost to mitigate them; scanning of vulnerabilities and the time/cost to repair them; and the frequency/time required to do patch management.

"You can measure things like the number of times you investigated potential indicators of anomalous activity," says EMA's Crawford. "You can track the number of cases of investigation and the number of cases that have had to be escalated to mitigation. You can measure the percent of unplanned IT work related to that escalation, and the resulting security spend."

However, experts warn against measuring aspects of security that may not be meaningful to the business -- or worse, may cause the security department to focus its efforts on the wrong priorities.

"Tracking vulnerabilities, days to patch, [antivirus] performance -- these might be useful at an operational level, but measuring these in order to show security effectiveness is a load of crap," Rothman says.

And while tracking incident response or mitigation time might be useful in benchmarking the performance of the security department for upper management, these metrics still don't provide enough meat to serve as a gauge for the organization's security posture, experts advise.

"To measure security posture means arriving at compound and composite metrics -- something like the cost of sales numbers that many companies track," says CIS' Piliero. "I'm not aware of any standard metrics that can measure that today."

There is an emerging class of tools for security posture management (SPOM), such as those made by RedSeal, currently on the market. Such products harvest firewall configuration data and other information to show the potential for access to critical business data -- a measure of both vulnerabilities and risk.

"Companies use us to do a risk analysis on a specific vulnerability -- what's the potential impact if it's exploited?" Dauber explains. "They can use this data to help with prioritization of security actions -- to help figure out what issues they should handle first."

Rothman says the SPOM concept has merit for measuring security posture, but the market hasn't taken off. "The big problem is the cost," he says. "Executives have to see that it's worth that much to be able to judge security posture, and that's only going to happen in industries where that sort of data is critical to the business."

Still, there is clearly a need for tools that can not only provide simple metrics for reporting to upper management, but can provide real insight into the company's state of security, Rothman observes. Core Security's new Core Insight tool -- essentially a penetration-testing appliance -- is one such emerging product, and nCircle's Suite 360 Intelligence Hub offers a way to benchmark one company's security against other, similar companies, he notes.

"One promising way to get some security metrics is to benchmark one organization's state and processes relative to others," Rothman says. "The problem with that is how do you attribute the data back without giving away too much about its source? Sharing between security companies is still the main constraint on this."

Organizations such as the CIS and the new Open Security Intelligence forum are attempting to provide a basis for the definition and sharing of security data and metrics, but there is still a lot of work to be done, experts say. Part of the problem is that there are so many different functions and players in the security metrics game.

"There is still a big gap between operations people, compliance people, and security people," says Joe Gottlieb, CEO of SenSage and founder of the Open Security Intelligence initiative. SenSage earlier this week published a study in which a majority of security people said they thought their security processes are less effective because data is not effectively shared among the various functional areas, such as compliance, incident response, and real-time monitoring.

"Most security metrics [initiatives] start because some enlightened executive up the chain asks for the numbers," says CIS' Piliero. "Once that happens, you see companies trying to get their own house in order, working together to pull together operational metrics before they start reporting up the chain."

Have a comment on this story? Please click "Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.