Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Operations

3/17/2016
04:00 PM
Adam Shostack
Adam Shostack
Commentary
Connect Directly
Twitter
LinkedIn
RSS
E-Mail vvv
50%
50%

Security Lessons From My Stock Broker

Or, how to lie with metrics.

A few years back, I called my stock broker and asked for help selecting a growth fund to diversify my holdings a bit. He said he had this great fund that was totally a fit for what I needed.  (Have you ever called a salesperson and not heard that they had what you needed? When you do, pay attention. Those are the ones doing real strategic sales.)

This fund had great performance relative to the Russell 2000, had a low beta (a measure of volatility), and blah blah. Frankly, I don’t remember the points he made when selling me. They were his points, not my points. Some of them were real metrics, that were relevant to what I wanted to know, and some were what the lean startup movement calls “vanity metrics.”

But they were his metrics, not mine. I had not done the hard work of figuring out what mattered to me, and ensuring that the things I wanted were being measured. So I was an easy mark.  There are two lessons here: one for people buying products and services, and one for those producing metrics for “the business.” 

Source: Pixabay
Source: Pixabay

Walking around at RSA, it seems that every product today has its own “single pain of glass.”  (No, not pane, trust me, they’re misspelling it.) These pains of glass take metrics that a product manager selected, just like my stock broker selected his metrics. And you’re going to have a lot of them, and they’ll be pains. They’ll be numbers that you can, with work, influence, but that work doesn’t mean your business is more secure. But now that you’re measuring them, you better start influencing them. You’re going to be held accountable for the numbers that you bought.

Let’s take an example of vulnerability counts. Vulnerability counts have, at best, a complex relationship to consequential events. As someone who helped get the CVE off the ground, I know that there are plenty of real issues (word macros, dll injection) which real attackers exploit and which don’t get fixed. Others, like Autorun, do get fixed, without a CVE, because they’re not bugs, but features. There are also plenty of real vulnerabilities, such as SQL injection in your custom database, that don’t get a CVE. (I hope that those are bugs, not features.)

The question you’d like to ask, the thing that you’d like to measure, is not vulnerabilities. You probably want to influence vulnerabilities because you think they correlate with the consequential events that your business cares about, and they might. But as we’ve just discussed, they are not a complete metric of what matters to the business, and we don’t have a good way to estimate their incompleteness. So, not measuring what you care about or being tightly correlated with what you care about means they’re a bad executive metric.

And here’s the lesson my stock broker can teach those producing metrics for the business. Don’t be like my stock broker. It’s a short-term business model. Business has a way of looking at issues. Profit and loss. Return on capital. Now, it’s cliché to complain about how hard it is to link security to those issues, and so we invent stuff to report on, like “maturity,” thinking it sounds strategic. It doesn’t. 

Look, executives become executives because they’re good at making decisions about complex questions with big impacts. Is it harder in security? Well, yes, we blindfold ourselves, we rail against talking about our mistakes, and then wonder why no one ever gets better. But that’s a problem we have to face within security, and in the meanwhile, we need to find metrics or frameworks that matter to our executives, that the business understands, and that we can speak. In that order.

So the lesson is: figure out the metrics that matter to you, and figure out the metrics that matter to the business. Some of it will be hard to gather, some of it will be impossible. But you don’t want to be like the drunk looking for their keys under the streetlight, even if the light is better there.

Next week, we’ll get down and dirty and talk about what those metrics are not. Here’s a hint: they’re not about things you can’t control. 

Oh -- and incidentally, that fund? Down 20% when I sold it.

Related content:

 

Interop 2016 Las VegasFind out more about security threats at Interop 2016, May 2-6, at the Mandalay Bay Convention Center, Las Vegas. Register today and receive an early bird discount of $200.

Adam is a consultant, entrepreneur, technologist, author and game designer. He's a member of the BlackHat Review Board and helped create the CVE and many other things. He currently helps organizations improve their security via Shostack & Associates, and advises startups ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Joe Stanganelli
50%
50%
Joe Stanganelli,
User Rank: Ninja
3/18/2016 | 9:28:01 AM
Measuring in the real world
I strongly suspect that the security industry (or, at least, security industry marketing) focuses on these vulnerability-related metrics because other security metrics are so difficult to -- well -- measure.

It's also interesting to note how lab tests can different from "real-world" environments and results.  NSS Labs (which was at RSA) released a NGFW study that -- in addition to its basic tests -- purported to offer results from tests emulating various "real-world" environments.  What was interesting here is that where one of the NGFWs (made by Palo Alto Networks) smoked the competition on performance in all the other tests, others performed better in NSS's "real-world" datacenter test.
97% of Americans Can't Ace a Basic Security Test
Steve Zurier, Contributing Writer,  5/20/2019
How a Manufacturing Firm Recovered from a Devastating Ransomware Attack
Kelly Jackson Higgins, Executive Editor at Dark Reading,  5/20/2019
Why AI Will Create Far More Jobs Than It Replaces
John DiLullo, CEO, Lastline,  5/14/2019
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Write a Caption, Win a Starbucks Card! Click Here
Latest Comment: Talk about vendor lock in...
Current Issue
Building and Managing an IT Security Operations Program
As cyber threats grow, many organizations are building security operations centers (SOCs) to improve their defenses. In this Tech Digest you will learn tips on how to get the most out of a SOC in your organization - and what to do if you can't afford to build one.
Flash Poll
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2019-11816
PUBLISHED: 2019-05-20
Incorrect access control in the WebUI in OPNsense before version 19.1.8, and pfsense before 2.4.4-p3 allows remote authenticated users to escalate privileges to administrator via a specially crafted request.
CVE-2019-10076
PUBLISHED: 2019-05-20
A carefully crafted malicious attachment could trigger an XSS vulnerability on Apache JSPWiki 2.9.0 to 2.11.0.M3, which could lead to session hijacking.
CVE-2019-10077
PUBLISHED: 2019-05-20
A carefully crafted InterWiki link could trigger an XSS vulnerability on Apache JSPWiki 2.9.0 to 2.11.0.M3, which could lead to session hijacking.
CVE-2019-10078
PUBLISHED: 2019-05-20
A carefully crafted plugin link invocation could trigger an XSS vulnerability on Apache JSPWiki 2.9.0 to 2.11.0.M3, which could lead to session hijacking. Initial reporting indicated ReferredPagesPlugin, but further analysis showed that multiple plugins were vulnerable.
CVE-2019-12239
PUBLISHED: 2019-05-20
The WP Booking System plugin 1.5.1 for WordPress has no CSRF protection, which allows attackers to reach certain SQL injection issues that require administrative access.