Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


05:20 PM
Connect Directly

The Four Big Problems With Security Metrics

Metrics can be very useful, but only if they track the things that matter.

There’s a sort of can’t-live-with-'em-can’t-live-without-'em quality to a lot of the metrics that are used by security organizations to report on the effectiveness of enterprise security programs.

Analysts consider metrics vital not just to measuring how well a security program might be doing, but also in communicating that to executive management and the C-suite. Metrics, when used effectively, can help identify strengths and weaknesses in controls and processes in an organization’s cybersecurity program and provide a sense of the value being derived from it.

The problem, say practitioners and security experts, is finding and gathering the right metrics. Often, the metrics that security organizations track and present to management are not aligned with business objectives. They tend to be too focused on compliance and do little to convey how effective a security program is in reducing overall risk.

More than 8 out of 10 respondents in an April 2014 survey of nearly 600 IT and security professional conducted by the Ponemon Institute on behalf of FireMon said that it is important to have metrics that are aligned with business goals. But 43 percent said the metrics that are actually used today do little to convey the true state of security in an organization while 11 percent said they were unsure how effective their metrics were.

Here, in no particular order, are some of the most common problems with the metrics that are used today, according to security practitioners and experts.

Metrics report activity, not outcomes

Security professionals themselves consider threat detection and risk metrics to be the top indicators of the effectiveness of their security program. In a recent survey (registration required) conducted by Dimensional Research on behalf of privileged account security software vendor CyberArk, respondents ranked metrics like the time to detect attempted attacks and the potential costs from security attacks as the most effective metrics. Yet, the same respondents also said that the metrics they most often actually provided to executive management were compliance-related or had to do with systems availability.

The fact is that it often is easier to report on activities, like the progress in implementing the security controls needed to meet a compliance objective, than talking about how effective those controls actually are in reducing risk, says John Bruce, CEO of Resilient Systems. “Yes, ‘we are compliant, check’ doesn’t mean ‘yes we are secure, check’,” Bruce says.

Sacrificing Detail For Simplicity

Dashboards that boil down the security status of an organization into a simple-to-understand Green, Yellow, and Red color code can be useful. They can help quickly convey important information about the security preparedness of an organization in an easy-to-digest manner. But the key is in the details that lie underneath.

“Dashboards provide the ultimate way to provide security information,” says Pete Lindstrom, an analyst with IDC. “The question is, when you click your way down, are you getting real information,” on security preparedness, he says.

In order to really understand risk, an organization has to, among other things, have a sense of the value that business derives from IT, the control framework in place to protect the systems that deliver that value, a sense of the threats that are being blocked and the potential losses that could result from a security incident.

There often is a huge disconnect between what executives should be told and how that information is presented to them, Lindstrom says. In trying to keep things simple, there is a tendency for instance to report on simple "pass" or "fail" metrics associated with a compliance audit, instead of the more relevant data.

Metrics That Are Useful To Security Pros Are Too Complicated For Management

As the CyberArk survey showed, information security professionals consider threat detection and risk-related metrics to be the top security indicators, though the metrics they end up reporting are something else entirely. The problem has a lot to do with the communication gap that exists between the security function and the executives to whom they report.

“I have found that most metrics that we collect are relatively meaningless to them,” says Matt Kesner, CIO at Mountain View, Calif.-based lawfirm Fenwick & West. “Modern security systems do not report metrics in a way that seems meaningful to most business people.”

It is nearly impossible for security professions to use the very large numbers reported by most systems in a way that is easily digested by executives. “Whether the systems report them as attacks, or attempts, or even advanced persistent attacks, the numbers are so large as to seem meaningless,” Kesner says. “Worse yet if you report those numbers, the perception can be that those large numbers did not result in any real harm -- so we must be invincible.”

Because of this, Kesner says, he cites outside surveys and industry trends when speaking with the law firm’s executive committee. “I only talk about specific incidents, when I speak about our firm’s experience,” he says.

Viewing Metrics As An Exact Science

Metrics are vital to any risk-based enterprise information security program. The right metrics can help an organization get a pretty good idea of how effective their security program is and how well aligned it is with business objectives. But metrics are not an exact science. They might tell you how many attacks your security controls stopped, but not how many attacks will be stopped or how many attack they might have missed.

Management executives want security organizations to tell them precisely what is going on in language they can understand, Bruce from Resilient says. “The most competent way to converse with them is to describe the nature of the problem and to make clear that it not an exact science.”

It is important to convey the nature of the risks that all organizations face including the potential for cyberattacks and to explain that there are ways to control and mitigate such attacks he says.

“If you go ask for more technology and more money, then you are not going to get the audience you are looking for,” he says. “It is well understood that you are going to to be subject to a lot of attacks. It is what it is. But it is not the end of the world.”

Jai Vijayan is a seasoned technology reporter with over 20 years of experience in IT trade journalism. He was most recently a Senior Editor at Computerworld, where he covered information security and data privacy issues for the publication. Over the course of his 20-year ... View Full Bio

Recommended Reading:

Comment  | 
Print  | 
More Insights
Oldest First  |  Newest First  |  Threaded View
COVID-19: Latest Security News & Commentary
Dark Reading Staff 8/10/2020
Pen Testers Who Got Arrested Doing Their Jobs Tell All
Kelly Jackson Higgins, Executive Editor at Dark Reading,  8/5/2020
Researcher Finds New Office Macro Attacks for MacOS
Curtis Franklin Jr., Senior Editor at Dark Reading,  8/7/2020
Register for Dark Reading Newsletters
White Papers
Cartoon Contest
Current Issue
Special Report: Computing's New Normal, a Dark Reading Perspective
This special report examines how IT security organizations have adapted to the "new normal" of computing and what the long-term effects will be. Read it and get a unique set of perspectives on issues ranging from new threats & vulnerabilities as a result of remote working to how enterprise security strategy will be affected long term.
Flash Poll
The Changing Face of Threat Intelligence
The Changing Face of Threat Intelligence
This special report takes a look at how enterprises are using threat intelligence, as well as emerging best practices for integrating threat intel into security operations and incident response. Download it today!
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2020-08-11
Telegram Desktop through 2.1.13 allows a spoofed file type to bypass the Dangerous File Type Execution protection mechanism, as demonstrated by use of the chat window with a filename that lacks an extension.
PUBLISHED: 2020-08-11
Turcom TRCwifiZone through 2020-08-10 allows authentication bypass by visiting manage/control.php and ignoring 302 Redirect responses.
PUBLISHED: 2020-08-11
An elevation of privilege vulnerability exists in ManageEngine ADSelfService Plus before build 6003 because it does not properly enforce user privileges associated with a Certificate dialog. This vulnerability could allow an unauthenticated attacker to escalate privileges on a Windows host. An attac...
PUBLISHED: 2020-08-11
SABnzbd 2.3.9 and 3.0.0Alpha2 has a command injection vulnerability in the web configuration interface that permits an authenticated user to execute arbitrary Python commands on the underlying operating system.
PUBLISHED: 2020-08-11
SOPlanning 1.46.01 allows persistent XSS via the Project Name, Statutes Comment, Places Comment, or Resources Comment field.