05:20 PM
Connect Directly

The Four Big Problems With Security Metrics

Metrics can be very useful, but only if they track the things that matter.

There’s a sort of can’t-live-with-'em-can’t-live-without-'em quality to a lot of the metrics that are used by security organizations to report on the effectiveness of enterprise security programs.

Analysts consider metrics vital not just to measuring how well a security program might be doing, but also in communicating that to executive management and the C-suite. Metrics, when used effectively, can help identify strengths and weaknesses in controls and processes in an organization’s cybersecurity program and provide a sense of the value being derived from it.

The problem, say practitioners and security experts, is finding and gathering the right metrics. Often, the metrics that security organizations track and present to management are not aligned with business objectives. They tend to be too focused on compliance and do little to convey how effective a security program is in reducing overall risk.

More than 8 out of 10 respondents in an April 2014 survey of nearly 600 IT and security professional conducted by the Ponemon Institute on behalf of FireMon said that it is important to have metrics that are aligned with business goals. But 43 percent said the metrics that are actually used today do little to convey the true state of security in an organization while 11 percent said they were unsure how effective their metrics were.

Here, in no particular order, are some of the most common problems with the metrics that are used today, according to security practitioners and experts.

Metrics report activity, not outcomes

Security professionals themselves consider threat detection and risk metrics to be the top indicators of the effectiveness of their security program. In a recent survey (registration required) conducted by Dimensional Research on behalf of privileged account security software vendor CyberArk, respondents ranked metrics like the time to detect attempted attacks and the potential costs from security attacks as the most effective metrics. Yet, the same respondents also said that the metrics they most often actually provided to executive management were compliance-related or had to do with systems availability.

The fact is that it often is easier to report on activities, like the progress in implementing the security controls needed to meet a compliance objective, than talking about how effective those controls actually are in reducing risk, says John Bruce, CEO of Resilient Systems. “Yes, ‘we are compliant, check’ doesn’t mean ‘yes we are secure, check’,” Bruce says.

Sacrificing Detail For Simplicity

Dashboards that boil down the security status of an organization into a simple-to-understand Green, Yellow, and Red color code can be useful. They can help quickly convey important information about the security preparedness of an organization in an easy-to-digest manner. But the key is in the details that lie underneath.

“Dashboards provide the ultimate way to provide security information,” says Pete Lindstrom, an analyst with IDC. “The question is, when you click your way down, are you getting real information,” on security preparedness, he says.

In order to really understand risk, an organization has to, among other things, have a sense of the value that business derives from IT, the control framework in place to protect the systems that deliver that value, a sense of the threats that are being blocked and the potential losses that could result from a security incident.

There often is a huge disconnect between what executives should be told and how that information is presented to them, Lindstrom says. In trying to keep things simple, there is a tendency for instance to report on simple "pass" or "fail" metrics associated with a compliance audit, instead of the more relevant data.

Metrics That Are Useful To Security Pros Are Too Complicated For Management

As the CyberArk survey showed, information security professionals consider threat detection and risk-related metrics to be the top security indicators, though the metrics they end up reporting are something else entirely. The problem has a lot to do with the communication gap that exists between the security function and the executives to whom they report.

“I have found that most metrics that we collect are relatively meaningless to them,” says Matt Kesner, CIO at Mountain View, Calif.-based lawfirm Fenwick & West. “Modern security systems do not report metrics in a way that seems meaningful to most business people.”

It is nearly impossible for security professions to use the very large numbers reported by most systems in a way that is easily digested by executives. “Whether the systems report them as attacks, or attempts, or even advanced persistent attacks, the numbers are so large as to seem meaningless,” Kesner says. “Worse yet if you report those numbers, the perception can be that those large numbers did not result in any real harm -- so we must be invincible.”

Because of this, Kesner says, he cites outside surveys and industry trends when speaking with the law firm’s executive committee. “I only talk about specific incidents, when I speak about our firm’s experience,” he says.

Viewing Metrics As An Exact Science

Metrics are vital to any risk-based enterprise information security program. The right metrics can help an organization get a pretty good idea of how effective their security program is and how well aligned it is with business objectives. But metrics are not an exact science. They might tell you how many attacks your security controls stopped, but not how many attacks will be stopped or how many attack they might have missed.

Management executives want security organizations to tell them precisely what is going on in language they can understand, Bruce from Resilient says. “The most competent way to converse with them is to describe the nature of the problem and to make clear that it not an exact science.”

It is important to convey the nature of the risks that all organizations face including the potential for cyberattacks and to explain that there are ways to control and mitigate such attacks he says.

“If you go ask for more technology and more money, then you are not going to get the audience you are looking for,” he says. “It is well understood that you are going to to be subject to a lot of attacks. It is what it is. But it is not the end of the world.”

Jai Vijayan is a seasoned technology reporter with over 20 years of experience in IT trade journalism. He was most recently a Senior Editor at Computerworld, where he covered information security and data privacy issues for the publication. Over the course of his 20-year ... View Full Bio

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
8 Ways Hackers Monetize Stolen Data
Steve Zurier, Freelance Writer,  4/17/2018
Securing Social Media: National Safety, Privacy Concerns
Kelly Sheridan, Staff Editor, Dark Reading,  4/19/2018
Firms More Likely to Tempt Security Pros With Big Salaries than Invest in Training
Sara Peters, Senior Editor at Dark Reading,  4/19/2018
Register for Dark Reading Newsletters
White Papers
Current Issue
How to Cope with the IT Security Skills Shortage
Most enterprises don't have all the in-house skills they need to meet the rising threat from online attackers. Here are some tips on ways to beat the shortage.
Flash Poll
[Strategic Security Report] Navigating the Threat Intelligence Maze
[Strategic Security Report] Navigating the Threat Intelligence Maze
Most enterprises are using threat intel services, but many are still figuring out how to use the data they're collecting. In this Dark Reading survey we give you a look at what they're doing today - and where they hope to go.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
Published: 2017-05-09
NScript in mpengine in Microsoft Malware Protection Engine with Engine Version before 1.1.13704.0, as used in Windows Defender and other products, allows remote attackers to execute arbitrary code or cause a denial of service (type confusion and application crash) via crafted JavaScript code within ...

Published: 2017-05-08
unixsocket.c in lxterminal through 0.3.0 insecurely uses /tmp for a socket file, allowing a local user to cause a denial of service (preventing terminal launch), or possibly have other impact (bypassing terminal access control).

Published: 2017-05-08
A privilege escalation vulnerability in Brocade Fibre Channel SAN products running Brocade Fabric OS (FOS) releases earlier than v7.4.1d and v8.0.1b could allow an authenticated attacker to elevate the privileges of user accounts accessing the system via command line interface. With affected version...

Published: 2017-05-08
Improper checks for unusual or exceptional conditions in Brocade NetIron 05.8.00 and later releases up to and including 06.1.00, when the Management Module is continuously scanned on port 22, may allow attackers to cause a denial of service (crash and reload) of the management module.

Published: 2017-05-08
Nextcloud Server before 11.0.3 is vulnerable to an inadequate escaping leading to a XSS vulnerability in the search module. To be exploitable a user has to write or paste malicious content into the search dialogue.