Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


10:12 PM

The Trouble With Security Metrics

A Q&A with the author of The Security Risk Assessment Handbook

Security practitioner Doug Landoll is passionate about risk assessments and security measurements. Author of The Security Risk Assessment Handbook and CEO of Assero Security, a risk consultancy for SMBs, Landoll believes the industry engages in far too many theatrical risk assessments for the sake of audits. These assessments never return solid measurements of risk because the collection methods are faulty, he says. As organizations seek to meet risks head on, they need better visibility into which security initiatives work, which don't, and which need improvement. Done right, security metrics can help provide the estimates to plan out effective strategies. Dark Reading recently caught up with Landoll to talk about his thoughts on how organizations can improve their collection methods to create security metrics that mean something.

Dark Reading: I know you're a big believer that if you depend on a faulty method of gathering data, security metrics won't really matter for much. Can you explain a little about your philosophy when it comes to measuring risk? Landoll: You want to measure what's important to the organization, instead of thinking what you measure is what's important. There's a difference.

It is what I call the SEIM dog wagging the governance tail. When you've got some great technology that's pumping out tons of data, the bottom-up approach is to say, 'Wow, look at all this data. Let's build a dashboard.' So you get this dashboard with all these really cool metrics and trends and everything, and now that becomes what you measure and that becomes what's important to you -- which is way backward. What's important to you should be based on your company mission. Based on your company strategy, your security strategy should have some goals you're trying to get to -- those are the things to measure.

Just because that metric is not being produced by a piece of technology, it's a mistake if we don't grab it, we don't report it, we don't trend it, we don't correct our strategy. Top down is the only way to do metrics.

Dark Reading: Can you give an example of a risk measurement that people wouldn't think of because maybe they don't have an automated dashboard to spit it out at them? Landoll: Let's say you're going to spend money to develop a computer incident response. I think you have some expected improvements from that. Before the project even starts, you can collect metrics on it. I think you're expecting a decreased time from incident discovery to recovery. I think you're expecting to be able to demonstrate compliance with breach notification requirements, and I think you want to minimize damage from incidents. If I can prove those things, that has been a successful project.

I'm not sure that people are trying to take the time from incident discovery to recovery. I don't think it would be that hard, but I think that's not pumped out by a tool, so, therefore, you're not usually going to measure it. So how would you do that?

One example would be to grab three to five common incidents that happen and measure the various phases. I think you should be able to have a date and time for the detection phase, the analysis phase, containment, eradication, and recovery. Here's how we get the call, how long it takes them to figure out what else it is, etc., etc. And then you check to see if that is getting any better six months down the line.

That would be interesting. But as you know, there is no tool that does that. But you have a ticketing system and maybe it's in the incident report. But it's there somewhere.

[How are CISOs preparing for 2013? See 7 Risk Management Priorities For 2013.]

Dark Reading: What about metrics around security awareness? It's seems it is easy to game that system by measuring how many people have been trained versus actual results. Landoll: Some other things you can do are testing for susceptibility to phishing attacks. Use a company like PhishMe. It's a pretty affordable service. Do it on a regular basis and get reports on the results. I would hope that in six months, when the culture gets used to these phishing emails going around and people get embarrassed by clicking on them, nobody's going to fall for that stuff anymore. Or there will be a considerable improvement.

Dark Reading: When it comes to thinking up new metrics or effective ways to measure security, what kind of mindset do you need to take into that process? Landoll: I find it's not cookie-cutter, but it tends to be easier than you'd think. The methodology I use is, I start at the high level and say, 'What would I like to know?' Then I think about whether there is any information out there. And if it's not there, I figure out how to generate it. So let's say, for example, I want to know how many laptops might be stolen this year from my traveling consultants. I think that data's there, or at least last year's data is there, so who would have that at a company? I think, the guy that ordered those new laptops.

He'll just tell you, it was six last year and there were eight the year before. So you have a pretty good guess it's between five and 10 this year -- unless you've done some kind of change control.

I think if you start at the high level and think, here's what I'd like to know, where would that information be, or how would I create it. Very few times you'll have to create it, but then when you create it, it's not as hard as you think. It could be a survey, it could be, 'Let's talk to the guy in charge and see what value he's put on that.' People shy away from that because they feel inaccurate. But measurement is reduction of uncertainty. It is estimation. We don't have to say we'll lose exactly 7.1 laptops per year. It's PK to say it's between five and 10.

I would also recommend Doug Hubbard's book. It's called How To Measure Anything: Finding the Value of "Intangibles" in Business. There's a chapter on security. He certainly understands our industry. The mistake everyone makes is that they think their industry is unique and they've never had this problem before, and he says nonsense to that. People had all of these problems before and they thought of clever ways to grab metrics.

Dark Reading: In your opinion, what would you say are some of the biggest mistakes people make when measuring security? Landoll: I would say letting the available data drive the metrics program. That's a huge mistake. Another one is not collecting enough data points and relying on single data points, or assuming that the data you want isn't available and settling for something else. I think that's a poor assumption.

Dark Reading: You've certainly crusaded against relying on single sources of data for risk assessments and measurable, advocating for what you call the RIIOT (Review, Interview, Inspect, Observe and Test) method of risk assessment. Why is it so important to pull metrics and assessment data from different collection methods? Landoll: There are a lot of approaches to risk management that are not accurate. When you boil down your security program to a 50-question questionnaire, and you divide it among the people in the organization, send it out, and compile it, you know it's not accurate and they know it's not accurate. We're just checking a box and when the auditor comes, we say, 'Here, we did a risk assessment. Good luck.'

This is your plan. This is your strategy. This is where you determine which security activities you're going to be doing in the next few years. And yet you're not collecting the data, and you're making wrong conclusions, and you're getting budget, making a plan based on faulty data. That's alarming.

Let's say you do a traditional risk assessment where you're reviewing documents and you're interviewing people about security awareness training. The first interview is going to come out really good -- people tend to want to make themselves look good. If that's your only input, you have to conclude that security awareness training doesn't need to be improved. However, if you were to observe that post-it notes were on those screens, and you just tried one social engineering trick and it worked, now you've got to conclude that it's not effective. That's a complete 180. And just by doing a few more tests.

Another good example is on system hardening. It's not about finding the error; it's about finding the root cause. You can do a scan, find a vulnerability, and say you patched it. But what if I reviewed the documents and then saw that the hardening documents weren't detailed enough? And I interviewed people and they said, 'There's change management procedures, but we don't use those, and we didn't write that hardening document and don't use it.' So the recommendation from that shouldn't just be to patch the system. It's really a little training, some governance, some hardening documents.

I'm just really concerned that the one security activity that we do as professionals to help plan out our strategy suffers from way too many shortcuts.

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Strategist
12/17/2012 | 4:18:18 PM
re: The Trouble With Security Metrics
It's so tough to measure security because all the measurements are negative -- the best you can say is "Today we didn't get hacked."- Do any of you readers have tips on positive security metrics or methods for measuring risk?
--Tim Wilson, editor, Dark Reading
Ransomware Is Not the Problem
Adam Shostack, Consultant, Entrepreneur, Technologist, Game Designer,  6/9/2021
How Can I Test the Security of My Home-Office Employees' Routers?
John Bock, Senior Research Scientist,  6/7/2021
New Ransomware Group Claiming Connection to REvil Gang Surfaces
Jai Vijayan, Contributing Writer,  6/10/2021
Register for Dark Reading Newsletters
White Papers
Cartoon Contest
Write a Caption, Win an Amazon Gift Card! Click Here
Latest Comment: Google's new See No Evil policy......
Current Issue
The State of Cybersecurity Incident Response
In this report learn how enterprises are building their incident response teams and processes, how they research potential compromises, how they respond to new breaches, and what tools and processes they use to remediate problems and improve their cyber defenses for the future.
Flash Poll
How Enterprises are Developing Secure Applications
How Enterprises are Developing Secure Applications
Recent breaches of third-party apps are driving many organizations to think harder about the security of their off-the-shelf software as they continue to move left in secure software development practices.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2021-06-18
Contiki-NG is an open-source, cross-platform operating system for internet of things devices. The RPL-Classic and RPL-Lite implementations in the Contiki-NG operating system versions prior to 4.6 do not validate the address pointer in the RPL source routing header This makes it possible for an attac...
PUBLISHED: 2021-06-18
Contiki-NG is an open-source, cross-platform operating system for internet of things devices. In verions prior to 4.6, an attacker can perform a denial-of-service attack by triggering an infinite loop in the processing of IPv6 neighbor solicitation (NS) messages. This type of attack can effectively ...
PUBLISHED: 2021-06-18
Contiki-NG is an open-source, cross-platform operating system for internet of things devices. It is possible to cause an out-of-bounds write in versions of Contiki-NG prior to 4.6 when transmitting a 6LoWPAN packet with a chain of extension headers. Unfortunately, the written header is not checked t...
PUBLISHED: 2021-06-18
Contiki-NG is an open-source, cross-platform operating system for internet of things devices. A buffer overflow vulnerability exists in Contiki-NG versions prior to 4.6. After establishing a TCP socket using the tcp-socket library, it is possible for the remote end to send a packet with a data offse...
PUBLISHED: 2021-06-18
Contiki-NG is an open-source, cross-platform operating system for Next-Generation IoT devices. An out-of-bounds read can be triggered by 6LoWPAN packets sent to devices running Contiki-NG 4.6 and prior. The IPv6 header decompression function (<code>uncompress_hdr_iphc</code>) does not pe...