Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


10:30 AM
Jay Jacobs
Jay Jacobs
Connect Directly
E-Mail vvv

How To Put Data At The Heart Of Your Security Practice

First step: A good set of questions that seek out objective, measurable answers.

“When you can measure what you are speaking about, and express it in numbers, you know something about it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts advanced to the stage of science.” -- Lord Kelvin, 1883

Lord Kelvin wrote those words over 130 years ago. He was addressing a group of civil engineers on the topic of practical applications of electricity. While electricity as a physical science may (or may not) seem like a far jump from information security, how we improve our knowledge and understanding has remained relatively constant: we learn by observing and measuring our environment. Security programs are just now beginning to realize this lesson and there are a lot of questions being asked around data-driven security programs and how to build a security practice that has data analysis at the heart of the decision-making process. 

However, before we talk about how to approach the integration of data and measurement into the security decision-making process, let’s talk about how you should not to begin. This is important because many data-driven security programs are doomed in their first step because people don’t ask the right questions. Instead, organizations will look at the data they have and try to pull out things that are “interesting” or that they think will help drive their program. This only leads to metrics that are convenient and not necessarily useful and will end up wasting a lot of time and energy from everyone involved.

Anything worth doing is worth asking questions about
To build a data-driven security practice, start by defining a list of questions that, if answered, would help not only drive decisions, but also help how you evaluate how good those decisions were down the road. Defining such questions are tricky, because they can’t be just any old questions, they must seek out objective answers. As Bill James, who spent his life studying and reporting on the statistics of baseball, once said, “My job was to find questions about baseball that have objective answers, that’s all that I do, that’s all that I’ve done.” So rather than ask, “How secure am I?” perhaps a better question is “How many security events did we have last quarter?” Or maybe even dig deeper with, “What types of security events do we spend the most time on?” Through this approach, you can objectively answer, “How secure am I?” with multiple points that are grounded in data.

Another approach is to pause before you sign off on the next security purchase and ask what observable actions in the environment the team would expect to influence with this purchase. This is not an easy challenge; sometimes decisions are made to stop something that hasn’t happened yet. In that case, the questions may look externally, “How many breaches were disclosed by other organizations like us?” Taking an outside-in approach will broaden the sources of data and help answer some those tough questions. 

Once you have a list of questions that you’d like answered, look for data sources to answer them.  Chances are extremely high that you aren’t collecting all the data you’ll need. The good news is that many organizations are asking the same questions and vendors are beginning to respond with data-driven solutions. For example, perhaps you don’t have to measure all of your industry peers: there are vendors and industry reports offering up answers that you can draw from.

Simple answers are still answers
Start trying to answer your questions with simple counts. Counting things is the first big step in being data-driven and the number of questions that can be answered with a simple count may surprise you. Simple counts paint with large brush strokes and may answer many of your initial questions. But sooner or later, you will want to compare two different counts. Perhaps the comparison will be as simple as comparing one month to the next, but eventually the need to compare two different counts is inevitable. Be forewarned: this comparison is the second big step towards being data-driven. Someone will ask, “Is the difference significant?” and that new question will set you traveling down a path towards statistical thinking. Don’t panic, statistics have already helped shape the evolution of many other fields and resistance is futile.

There is good news and bad news at this point. The good news is that we can get a lot of answers (and therefore benefit) with some relatively entry-level calculations.

The bad news is that the list of questions will grow exponentially as previous ones get answered. “How do I compare to my peers?” and “How can I measure my vendors and third parties?” Congratulations, you are well on your way with a security practice that has data analysis at the heart of the decision-making process!


Jay Jacobs has over 15 years of experience within IT and information security with a focus on cryptography, risk, and data analysis. Most recently, he has joined BitSight Technologies, the Standard in Security Ratings, as their Senior Data Scientist. Previously, he was a Data ... View Full Bio

Recommended Reading:

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
8/6/2015 | 12:12:56 PM
The lego block analogy

Good article Jay. Always a useful reminder to break down macro objectives into building block level questions and answers first 

User Rank: Ninja
7/28/2015 | 1:19:07 PM
Occams Razor
To understand something you need to break it down to its most basic components. This is an idea prevalent throughout physics. Relating to "Simple Answers are still answers", using Occam's Razor can definitely help by simplifying procedures such as ones that deal with the filtering of data. Using the fewest assumptions to develop security principles will provide a stronger framework than the oppossite idealogy which could over-complicate.
Inside the Ransomware Campaigns Targeting Exchange Servers
Kelly Sheridan, Staff Editor, Dark Reading,  4/2/2021
Beyond MITRE ATT&CK: The Case for a New Cyber Kill Chain
Rik Turner, Principal Analyst, Infrastructure Solutions, Omdia,  3/30/2021
Register for Dark Reading Newsletters
White Papers
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you today!
Flash Poll
How Enterprises are Developing Secure Applications
How Enterprises are Developing Secure Applications
Recent breaches of third-party apps are driving many organizations to think harder about the security of their off-the-shelf software as they continue to move left in secure software development practices.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2021-04-16
The unofficial vscode-rpm-spec extension before 0.3.2 for Visual Studio Code allows remote code execution via a crafted workspace configuration.
PUBLISHED: 2021-04-16
Broken Authentication in Atlassian Connect Express (ACE) from version 3.0.2 before version 6.6.0: Atlassian Connect Express is a Node.js package for building Atlassian Connect apps. Authentication between Atlassian products and the Atlassian Connect Express app occurs with a server-to-server JWT or ...
PUBLISHED: 2021-04-16
Broken Authentication in Atlassian Connect Spring Boot (ACSB) from version 1.1.0 before version 2.1.3: Atlassian Connect Spring Boot is a Java Spring Boot package for building Atlassian Connect apps. Authentication between Atlassian products and the Atlassian Connect Spring Boot app occurs with a se...
PUBLISHED: 2021-04-16
A cross-site scripting (XSS) vulnerability has been reported to affect earlier versions of File Station. If exploited, this vulnerability allows remote attackers to inject malicious code. We have already fixed this vulnerability in the following versions: QTS build 20210202 (and later) QT...
PUBLISHED: 2021-04-16
Command Injection in Tenda G0 routers with firmware versions v15.11.0.6(9039)_CN and v15.11.0.5(5876)_CN , and Tenda G1 and G3 routers with firmware versions v15.11.0.17(9502)_CN or v15.11.0.16(9024)_CN allows remote attackers to execute arbitrary OS commands via a crafted action/setDebugCfg request...