Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


05:20 PM
Connect Directly

The Four Big Problems With Security Metrics

Metrics can be very useful, but only if they track the things that matter.

There’s a sort of can’t-live-with-'em-can’t-live-without-'em quality to a lot of the metrics that are used by security organizations to report on the effectiveness of enterprise security programs.

Analysts consider metrics vital not just to measuring how well a security program might be doing, but also in communicating that to executive management and the C-suite. Metrics, when used effectively, can help identify strengths and weaknesses in controls and processes in an organization’s cybersecurity program and provide a sense of the value being derived from it.

The problem, say practitioners and security experts, is finding and gathering the right metrics. Often, the metrics that security organizations track and present to management are not aligned with business objectives. They tend to be too focused on compliance and do little to convey how effective a security program is in reducing overall risk.

More than 8 out of 10 respondents in an April 2014 survey of nearly 600 IT and security professional conducted by the Ponemon Institute on behalf of FireMon said that it is important to have metrics that are aligned with business goals. But 43 percent said the metrics that are actually used today do little to convey the true state of security in an organization while 11 percent said they were unsure how effective their metrics were.

Here, in no particular order, are some of the most common problems with the metrics that are used today, according to security practitioners and experts.

Metrics report activity, not outcomes

Security professionals themselves consider threat detection and risk metrics to be the top indicators of the effectiveness of their security program. In a recent survey (registration required) conducted by Dimensional Research on behalf of privileged account security software vendor CyberArk, respondents ranked metrics like the time to detect attempted attacks and the potential costs from security attacks as the most effective metrics. Yet, the same respondents also said that the metrics they most often actually provided to executive management were compliance-related or had to do with systems availability.

The fact is that it often is easier to report on activities, like the progress in implementing the security controls needed to meet a compliance objective, than talking about how effective those controls actually are in reducing risk, says John Bruce, CEO of Resilient Systems. “Yes, ‘we are compliant, check’ doesn’t mean ‘yes we are secure, check’,” Bruce says.

Sacrificing Detail For Simplicity

Dashboards that boil down the security status of an organization into a simple-to-understand Green, Yellow, and Red color code can be useful. They can help quickly convey important information about the security preparedness of an organization in an easy-to-digest manner. But the key is in the details that lie underneath.

“Dashboards provide the ultimate way to provide security information,” says Pete Lindstrom, an analyst with IDC. “The question is, when you click your way down, are you getting real information,” on security preparedness, he says.

In order to really understand risk, an organization has to, among other things, have a sense of the value that business derives from IT, the control framework in place to protect the systems that deliver that value, a sense of the threats that are being blocked and the potential losses that could result from a security incident.

There often is a huge disconnect between what executives should be told and how that information is presented to them, Lindstrom says. In trying to keep things simple, there is a tendency for instance to report on simple "pass" or "fail" metrics associated with a compliance audit, instead of the more relevant data.

Metrics That Are Useful To Security Pros Are Too Complicated For Management

As the CyberArk survey showed, information security professionals consider threat detection and risk-related metrics to be the top security indicators, though the metrics they end up reporting are something else entirely. The problem has a lot to do with the communication gap that exists between the security function and the executives to whom they report.

“I have found that most metrics that we collect are relatively meaningless to them,” says Matt Kesner, CIO at Mountain View, Calif.-based lawfirm Fenwick & West. “Modern security systems do not report metrics in a way that seems meaningful to most business people.”

It is nearly impossible for security professions to use the very large numbers reported by most systems in a way that is easily digested by executives. “Whether the systems report them as attacks, or attempts, or even advanced persistent attacks, the numbers are so large as to seem meaningless,” Kesner says. “Worse yet if you report those numbers, the perception can be that those large numbers did not result in any real harm -- so we must be invincible.”

Because of this, Kesner says, he cites outside surveys and industry trends when speaking with the law firm’s executive committee. “I only talk about specific incidents, when I speak about our firm’s experience,” he says.

Viewing Metrics As An Exact Science

Metrics are vital to any risk-based enterprise information security program. The right metrics can help an organization get a pretty good idea of how effective their security program is and how well aligned it is with business objectives. But metrics are not an exact science. They might tell you how many attacks your security controls stopped, but not how many attacks will be stopped or how many attack they might have missed.

Management executives want security organizations to tell them precisely what is going on in language they can understand, Bruce from Resilient says. “The most competent way to converse with them is to describe the nature of the problem and to make clear that it not an exact science.”

It is important to convey the nature of the risks that all organizations face including the potential for cyberattacks and to explain that there are ways to control and mitigate such attacks he says.

“If you go ask for more technology and more money, then you are not going to get the audience you are looking for,” he says. “It is well understood that you are going to to be subject to a lot of attacks. It is what it is. But it is not the end of the world.”

Jai Vijayan is a seasoned technology reporter with over 20 years of experience in IT trade journalism. He was most recently a Senior Editor at Computerworld, where he covered information security and data privacy issues for the publication. Over the course of his 20-year ... View Full Bio

Recommended Reading:

Comment  | 
Print  | 
More Insights
Threaded  |  Newest First  |  Oldest First
COVID-19: Latest Security News & Commentary
Dark Reading Staff 8/14/2020
Lock-Pickers Face an Uncertain Future Online
Seth Rosenblatt, Contributing Writer,  8/10/2020
Hacking It as a CISO: Advice for Security Leadership
Kelly Sheridan, Staff Editor, Dark Reading,  8/10/2020
Register for Dark Reading Newsletters
White Papers
Current Issue
7 New Cybersecurity Vulnerabilities That Could Put Your Enterprise at Risk
In this Dark Reading Tech Digest, we look at the ways security researchers and ethical hackers find critical vulnerabilities and offer insights into how you can fix them before attackers can exploit them.
Flash Poll
The Changing Face of Threat Intelligence
The Changing Face of Threat Intelligence
This special report takes a look at how enterprises are using threat intelligence, as well as emerging best practices for integrating threat intel into security operations and incident response. Download it today!
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2020-08-14
Lack of authentication in the network relays used in MEGVII Koala 2.9.1-c3s allows attackers to grant physical access to anyone by sending packet data to UDP port 5000.
PUBLISHED: 2020-08-14
** REJECT ** DO NOT USE THIS CANDIDATE NUMBER. ConsultIDs: CVE-2020-10751. Reason: This candidate is a duplicate of CVE-2020-10751. Notes: All CVE users should reference CVE-2020-10751 instead of this candidate. All references and descriptions in this candidate have been removed to prevent accidenta...
PUBLISHED: 2020-08-14
** REJECT ** DO NOT USE THIS CANDIDATE NUMBER. ConsultIDs: CVE-2017-18270. Reason: This candidate is a duplicate of CVE-2017-18270. Notes: All CVE users should reference CVE-2017-18270 instead of this candidate. All references and descriptions in this candidate have been removed to prevent accidenta...
PUBLISHED: 2020-08-14
** REJECT ** DO NOT USE THIS CANDIDATE NUMBER. ConsultIDs: none. Reason: This candidate was withdrawn by its CNA. Further investigation showed that it was not a security issue. Notes: none.
PUBLISHED: 2020-08-14
Lack of mutual authentication in ZKTeco FaceDepot 7B 1.0.213 and ZKBiosecurity Server 1.0.0_20190723 allows an attacker to obtain a long-lasting token by impersonating the server.