Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


05:20 PM
Connect Directly

The Four Big Problems With Security Metrics

Metrics can be very useful, but only if they track the things that matter.

There’s a sort of can’t-live-with-'em-can’t-live-without-'em quality to a lot of the metrics that are used by security organizations to report on the effectiveness of enterprise security programs.

Analysts consider metrics vital not just to measuring how well a security program might be doing, but also in communicating that to executive management and the C-suite. Metrics, when used effectively, can help identify strengths and weaknesses in controls and processes in an organization’s cybersecurity program and provide a sense of the value being derived from it.

The problem, say practitioners and security experts, is finding and gathering the right metrics. Often, the metrics that security organizations track and present to management are not aligned with business objectives. They tend to be too focused on compliance and do little to convey how effective a security program is in reducing overall risk.

More than 8 out of 10 respondents in an April 2014 survey of nearly 600 IT and security professional conducted by the Ponemon Institute on behalf of FireMon said that it is important to have metrics that are aligned with business goals. But 43 percent said the metrics that are actually used today do little to convey the true state of security in an organization while 11 percent said they were unsure how effective their metrics were.

Here, in no particular order, are some of the most common problems with the metrics that are used today, according to security practitioners and experts.

Metrics report activity, not outcomes

Security professionals themselves consider threat detection and risk metrics to be the top indicators of the effectiveness of their security program. In a recent survey (registration required) conducted by Dimensional Research on behalf of privileged account security software vendor CyberArk, respondents ranked metrics like the time to detect attempted attacks and the potential costs from security attacks as the most effective metrics. Yet, the same respondents also said that the metrics they most often actually provided to executive management were compliance-related or had to do with systems availability.

The fact is that it often is easier to report on activities, like the progress in implementing the security controls needed to meet a compliance objective, than talking about how effective those controls actually are in reducing risk, says John Bruce, CEO of Resilient Systems. “Yes, ‘we are compliant, check’ doesn’t mean ‘yes we are secure, check’,” Bruce says.

Sacrificing Detail For Simplicity

Dashboards that boil down the security status of an organization into a simple-to-understand Green, Yellow, and Red color code can be useful. They can help quickly convey important information about the security preparedness of an organization in an easy-to-digest manner. But the key is in the details that lie underneath.

“Dashboards provide the ultimate way to provide security information,” says Pete Lindstrom, an analyst with IDC. “The question is, when you click your way down, are you getting real information,” on security preparedness, he says.

In order to really understand risk, an organization has to, among other things, have a sense of the value that business derives from IT, the control framework in place to protect the systems that deliver that value, a sense of the threats that are being blocked and the potential losses that could result from a security incident.

There often is a huge disconnect between what executives should be told and how that information is presented to them, Lindstrom says. In trying to keep things simple, there is a tendency for instance to report on simple "pass" or "fail" metrics associated with a compliance audit, instead of the more relevant data.

Metrics That Are Useful To Security Pros Are Too Complicated For Management

As the CyberArk survey showed, information security professionals consider threat detection and risk-related metrics to be the top security indicators, though the metrics they end up reporting are something else entirely. The problem has a lot to do with the communication gap that exists between the security function and the executives to whom they report.

“I have found that most metrics that we collect are relatively meaningless to them,” says Matt Kesner, CIO at Mountain View, Calif.-based lawfirm Fenwick & West. “Modern security systems do not report metrics in a way that seems meaningful to most business people.”

It is nearly impossible for security professions to use the very large numbers reported by most systems in a way that is easily digested by executives. “Whether the systems report them as attacks, or attempts, or even advanced persistent attacks, the numbers are so large as to seem meaningless,” Kesner says. “Worse yet if you report those numbers, the perception can be that those large numbers did not result in any real harm -- so we must be invincible.”

Because of this, Kesner says, he cites outside surveys and industry trends when speaking with the law firm’s executive committee. “I only talk about specific incidents, when I speak about our firm’s experience,” he says.

Viewing Metrics As An Exact Science

Metrics are vital to any risk-based enterprise information security program. The right metrics can help an organization get a pretty good idea of how effective their security program is and how well aligned it is with business objectives. But metrics are not an exact science. They might tell you how many attacks your security controls stopped, but not how many attacks will be stopped or how many attack they might have missed.

Management executives want security organizations to tell them precisely what is going on in language they can understand, Bruce from Resilient says. “The most competent way to converse with them is to describe the nature of the problem and to make clear that it not an exact science.”

It is important to convey the nature of the risks that all organizations face including the potential for cyberattacks and to explain that there are ways to control and mitigate such attacks he says.

“If you go ask for more technology and more money, then you are not going to get the audience you are looking for,” he says. “It is well understood that you are going to to be subject to a lot of attacks. It is what it is. But it is not the end of the world.”

Jai Vijayan is a seasoned technology reporter with over 20 years of experience in IT trade journalism. He was most recently a Senior Editor at Computerworld, where he covered information security and data privacy issues for the publication. Over the course of his 20-year ... View Full Bio

Recommended Reading:

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
COVID-19: Latest Security News & Commentary
Dark Reading Staff 5/28/2020
The Problem with Artificial Intelligence in Security
Dr. Leila Powell, Lead Security Data Scientist, Panaseer,  5/26/2020
GDPR Enforcement Loosens Amid Pandemic
Seth Rosenblatt, Contributing Writer,  5/27/2020
Register for Dark Reading Newsletters
White Papers
Cartoon Contest
Current Issue
How Cybersecurity Incident Response Programs Work (and Why Some Don't)
This Tech Digest takes a look at the vital role cybersecurity incident response (IR) plays in managing cyber-risk within organizations. Download the Tech Digest today to find out how well-planned IR programs can detect intrusions, contain breaches, and help an organization restore normal operations.
Flash Poll
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2020-05-28
IBM Security Identity Governance and Intelligence 5.2.6 could allow a remote attacker to obtain sensitive information when a detailed technical error message is returned in the browser. This information could be used in further attacks against the system. IBM X-Force ID: 175484.
PUBLISHED: 2020-05-28
A denial of service vulnerability was reported in the firmware prior to version 1.01 used in Lenovo Printer LJ4010DN that could be triggered by a remote user sending a crafted packet to the device, causing an error to be displayed and preventing printer from functioning until the printer is rebooted...
PUBLISHED: 2020-05-28
A denial of service vulnerability was reported in the firmware prior to version 1.01 used in Lenovo Printer LJ4010DN that could be triggered by a remote user sending a crafted packet to the device, preventing subsequent print jobs until the printer is rebooted.
PUBLISHED: 2020-05-28
IBM Security Identity Governance and Intelligence 5.2.6 could allow an authenticated user to perform unauthorized commands due to hazardous input validation. IBM X-Force ID: 175335.
PUBLISHED: 2020-05-28
IBM Security Identity Governance and Intelligence 5.2.6 could allow an attacker to enumerate usernames to find valid login credentials which could be used to attempt further attacks against the system. IBM X-Force ID: 175336.