The Four Big Problems With Security Metrics

There’s a sort of can’t-live-with-'em-can’t-live-without-'em quality to a lot of the metrics that are used by security organizations to report on the effectiveness of enterprise security programs.

Analysts consider metrics vital not just to measuring how well a security program might be doing, but also in communicating that to executive management and the C-suite. Metrics, when used effectively, can help identify strengths and weaknesses in controls and processes in an organization’s cybersecurity program and provide a sense of the value being derived from it.

The problem, say practitioners and security experts, is finding and gathering the right metrics. Often, the metrics that security organizations track and present to management are not aligned with business objectives. They tend to be too focused on compliance and do little to convey how effective a security program is in reducing overall risk.

More than 8 out of 10 respondents in an April 2014 survey of nearly 600 IT and security professional conducted by the Ponemon Institute on behalf of FireMon said that it is important to have metrics that are aligned with business goals. But 43 percent said the metrics that are actually used today do little to convey the true state of security in an organization while 11 percent said they were unsure how effective their metrics were.

Here, in no particular order, are some of the most common problems with the metrics that are used today, according to security practitioners and experts.

Metrics report activity, not outcomes

Security professionals themselves consider threat detection and risk metrics to be the top indicators of the effectiveness of their security program. In a recent survey (registration required) conducted by Dimensional Research on behalf of privileged account security software vendor CyberArk, respondents ranked metrics like the time to detect attempted attacks and the potential costs from security attacks as the most effective metrics. Yet, the same respondents also said that the metrics they most often actually provided to executive management were compliance-related or had to do with systems availability.

The fact is that it often is easier to report on activities, like the progress in implementing the security controls needed to meet a compliance objective, than talking about how effective those controls actually are in reducing risk, says John Bruce, CEO of Resilient Systems. “Yes, ‘we are compliant, check’ doesn’t mean ‘yes we are secure, check’,” Bruce says.

Sacrificing Detail For Simplicity

Dashboards that boil down the security status of an organization into a simple-to-understand Green, Yellow, and Red color code can be useful. They can help quickly convey important information about the security preparedness of an organization in an easy-to-digest manner. But the key is in the details that lie underneath.

“Dashboards provide the ultimate way to provide security information,” says Pete Lindstrom, an analyst with IDC. “The question is, when you click your way down, are you getting real information,” on security preparedness, he says.

In order to really understand risk, an organization has to, among other things, have a sense of the value that business derives from IT, the control framework in place to protect the systems that deliver that value, a sense of the threats that are being blocked and the potential losses that could result from a security incident.

There often is a huge disconnect between what executives should be told and how that information is presented to them, Lindstrom says. In trying to keep things simple, there is a tendency for instance to report on simple "pass" or "fail" metrics associated with a compliance audit, instead of the more relevant data.

Metrics That Are Useful To Security Pros Are Too Complicated For Management

As the CyberArk survey showed, information security professionals consider threat detection and risk-related metrics to be the top security indicators, though the metrics they end up reporting are something else entirely. The problem has a lot to do with the communication gap that exists between the security function and the executives to whom they report.

“I have found that most metrics that we collect are relatively meaningless to them,” says Matt Kesner, CIO at Mountain View, Calif.-based lawfirm Fenwick & West. “Modern security systems do not report metrics in a way that seems meaningful to most business people.”

It is nearly impossible for security professions to use the very large numbers reported by most systems in a way that is easily digested by executives. “Whether the systems report them as attacks, or attempts, or even advanced persistent attacks, the numbers are so large as to seem meaningless,” Kesner says. “Worse yet if you report those numbers, the perception can be that those large numbers did not result in any real harm -- so we must be invincible.”

Because of this, Kesner says, he cites outside surveys and industry trends when speaking with the law firm’s executive committee. “I only talk about specific incidents, when I speak about our firm’s experience,” he says.

Viewing Metrics As An Exact Science

Metrics are vital to any risk-based enterprise information security program. The right metrics can help an organization get a pretty good idea of how effective their security program is and how well aligned it is with business objectives. But metrics are not an exact science. They might tell you how many attacks your security controls stopped, but not how many attacks will be stopped or how many attack they might have missed.

Management executives want security organizations to tell them precisely what is going on in language they can understand, Bruce from Resilient says. “The most competent way to converse with them is to describe the nature of the problem and to make clear that it not an exact science.”

It is important to convey the nature of the risks that all organizations face including the potential for cyberattacks and to explain that there are ways to control and mitigate such attacks he says.

“If you go ask for more technology and more money, then you are not going to get the audience you are looking for,” he says. “It is well understood that you are going to to be subject to a lot of attacks. It is what it is. But it is not the end of the world.”