Operations

3/17/2016
04:00 PM
Adam Shostack
Adam Shostack
Commentary
Connect Directly
Twitter
LinkedIn
RSS
E-Mail vvv
50%
50%

Security Lessons From My Stock Broker

Or, how to lie with metrics.

A few years back, I called my stock broker and asked for help selecting a growth fund to diversify my holdings a bit. He said he had this great fund that was totally a fit for what I needed.  (Have you ever called a salesperson and not heard that they had what you needed? When you do, pay attention. Those are the ones doing real strategic sales.)

This fund had great performance relative to the Russell 2000, had a low beta (a measure of volatility), and blah blah. Frankly, I don’t remember the points he made when selling me. They were his points, not my points. Some of them were real metrics, that were relevant to what I wanted to know, and some were what the lean startup movement calls “vanity metrics.”

But they were his metrics, not mine. I had not done the hard work of figuring out what mattered to me, and ensuring that the things I wanted were being measured. So I was an easy mark.  There are two lessons here: one for people buying products and services, and one for those producing metrics for “the business.” 

Source: Pixabay
Source: Pixabay

Walking around at RSA, it seems that every product today has its own “single pain of glass.”  (No, not pane, trust me, they’re misspelling it.) These pains of glass take metrics that a product manager selected, just like my stock broker selected his metrics. And you’re going to have a lot of them, and they’ll be pains. They’ll be numbers that you can, with work, influence, but that work doesn’t mean your business is more secure. But now that you’re measuring them, you better start influencing them. You’re going to be held accountable for the numbers that you bought.

Let’s take an example of vulnerability counts. Vulnerability counts have, at best, a complex relationship to consequential events. As someone who helped get the CVE off the ground, I know that there are plenty of real issues (word macros, dll injection) which real attackers exploit and which don’t get fixed. Others, like Autorun, do get fixed, without a CVE, because they’re not bugs, but features. There are also plenty of real vulnerabilities, such as SQL injection in your custom database, that don’t get a CVE. (I hope that those are bugs, not features.)

The question you’d like to ask, the thing that you’d like to measure, is not vulnerabilities. You probably want to influence vulnerabilities because you think they correlate with the consequential events that your business cares about, and they might. But as we’ve just discussed, they are not a complete metric of what matters to the business, and we don’t have a good way to estimate their incompleteness. So, not measuring what you care about or being tightly correlated with what you care about means they’re a bad executive metric.

And here’s the lesson my stock broker can teach those producing metrics for the business. Don’t be like my stock broker. It’s a short-term business model. Business has a way of looking at issues. Profit and loss. Return on capital. Now, it’s cliché to complain about how hard it is to link security to those issues, and so we invent stuff to report on, like “maturity,” thinking it sounds strategic. It doesn’t. 

Look, executives become executives because they’re good at making decisions about complex questions with big impacts. Is it harder in security? Well, yes, we blindfold ourselves, we rail against talking about our mistakes, and then wonder why no one ever gets better. But that’s a problem we have to face within security, and in the meanwhile, we need to find metrics or frameworks that matter to our executives, that the business understands, and that we can speak. In that order.

So the lesson is: figure out the metrics that matter to you, and figure out the metrics that matter to the business. Some of it will be hard to gather, some of it will be impossible. But you don’t want to be like the drunk looking for their keys under the streetlight, even if the light is better there.

Next week, we’ll get down and dirty and talk about what those metrics are not. Here’s a hint: they’re not about things you can’t control. 

Oh -- and incidentally, that fund? Down 20% when I sold it.

Related content:

 

Interop 2016 Las VegasFind out more about security threats at Interop 2016, May 2-6, at the Mandalay Bay Convention Center, Las Vegas. Register today and receive an early bird discount of $200.

Adam is an entrepreneur, technologist, author and game designer. He's a member of the BlackHat Review Board, and helped found the CVE and many other things. He's currently building his fifth startup, focused on improving security effectiveness, and mentors startups as a ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Joe Stanganelli
50%
50%
Joe Stanganelli,
User Rank: Ninja
3/18/2016 | 9:28:01 AM
Measuring in the real world
I strongly suspect that the security industry (or, at least, security industry marketing) focuses on these vulnerability-related metrics because other security metrics are so difficult to -- well -- measure.

It's also interesting to note how lab tests can different from "real-world" environments and results.  NSS Labs (which was at RSA) released a NGFW study that -- in addition to its basic tests -- purported to offer results from tests emulating various "real-world" environments.  What was interesting here is that where one of the NGFWs (made by Palo Alto Networks) smoked the competition on performance in all the other tests, others performed better in NSS's "real-world" datacenter test.
Cybersecurity's 'Broken' Hiring Process
Kelly Jackson Higgins, Executive Editor at Dark Reading,  10/11/2017
Ransomware Grabs Headlines but BEC May Be a Bigger Threat
Marc Wilczek, Digital Strategist & CIO Advisor,  10/12/2017
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Write a Caption, Win a Starbucks Card! Click Here
Latest Comment: What did you expect from this SOC? A unicorn....
Current Issue
Security Vulnerabilities: The Next Wave
Just when you thought it was safe, researchers have unveiled a new round of IT security flaws. Is your enterprise ready?
Flash Poll
The State of Ransomware
The State of Ransomware
Ransomware has become one of the most prevalent new cybersecurity threats faced by today's enterprises. This new report from Dark Reading includes feedback from IT and IT security professionals about their organization's ransomware experiences, defense plans, and malware challenges. Find out what they had to say!
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2017-0290
Published: 2017-05-09
NScript in mpengine in Microsoft Malware Protection Engine with Engine Version before 1.1.13704.0, as used in Windows Defender and other products, allows remote attackers to execute arbitrary code or cause a denial of service (type confusion and application crash) via crafted JavaScript code within ...

CVE-2016-10369
Published: 2017-05-08
unixsocket.c in lxterminal through 0.3.0 insecurely uses /tmp for a socket file, allowing a local user to cause a denial of service (preventing terminal launch), or possibly have other impact (bypassing terminal access control).

CVE-2016-8202
Published: 2017-05-08
A privilege escalation vulnerability in Brocade Fibre Channel SAN products running Brocade Fabric OS (FOS) releases earlier than v7.4.1d and v8.0.1b could allow an authenticated attacker to elevate the privileges of user accounts accessing the system via command line interface. With affected version...

CVE-2016-8209
Published: 2017-05-08
Improper checks for unusual or exceptional conditions in Brocade NetIron 05.8.00 and later releases up to and including 06.1.00, when the Management Module is continuously scanned on port 22, may allow attackers to cause a denial of service (crash and reload) of the management module.

CVE-2017-0890
Published: 2017-05-08
Nextcloud Server before 11.0.3 is vulnerable to an inadequate escaping leading to a XSS vulnerability in the search module. To be exploitable a user has to write or paste malicious content into the search dialogue.