Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Operations

3/17/2016
04:00 PM
Adam Shostack
Adam Shostack
Commentary
Connect Directly
Twitter
LinkedIn
RSS
E-Mail vvv
50%
50%

Security Lessons From My Stock Broker

Or, how to lie with metrics.

A few years back, I called my stock broker and asked for help selecting a growth fund to diversify my holdings a bit. He said he had this great fund that was totally a fit for what I needed.  (Have you ever called a salesperson and not heard that they had what you needed? When you do, pay attention. Those are the ones doing real strategic sales.)

This fund had great performance relative to the Russell 2000, had a low beta (a measure of volatility), and blah blah. Frankly, I don’t remember the points he made when selling me. They were his points, not my points. Some of them were real metrics, that were relevant to what I wanted to know, and some were what the lean startup movement calls “vanity metrics.”

But they were his metrics, not mine. I had not done the hard work of figuring out what mattered to me, and ensuring that the things I wanted were being measured. So I was an easy mark.  There are two lessons here: one for people buying products and services, and one for those producing metrics for “the business.” 

Source: Pixabay
Source: Pixabay

Walking around at RSA, it seems that every product today has its own “single pain of glass.”  (No, not pane, trust me, they’re misspelling it.) These pains of glass take metrics that a product manager selected, just like my stock broker selected his metrics. And you’re going to have a lot of them, and they’ll be pains. They’ll be numbers that you can, with work, influence, but that work doesn’t mean your business is more secure. But now that you’re measuring them, you better start influencing them. You’re going to be held accountable for the numbers that you bought.

Let’s take an example of vulnerability counts. Vulnerability counts have, at best, a complex relationship to consequential events. As someone who helped get the CVE off the ground, I know that there are plenty of real issues (word macros, dll injection) which real attackers exploit and which don’t get fixed. Others, like Autorun, do get fixed, without a CVE, because they’re not bugs, but features. There are also plenty of real vulnerabilities, such as SQL injection in your custom database, that don’t get a CVE. (I hope that those are bugs, not features.)

The question you’d like to ask, the thing that you’d like to measure, is not vulnerabilities. You probably want to influence vulnerabilities because you think they correlate with the consequential events that your business cares about, and they might. But as we’ve just discussed, they are not a complete metric of what matters to the business, and we don’t have a good way to estimate their incompleteness. So, not measuring what you care about or being tightly correlated with what you care about means they’re a bad executive metric.

And here’s the lesson my stock broker can teach those producing metrics for the business. Don’t be like my stock broker. It’s a short-term business model. Business has a way of looking at issues. Profit and loss. Return on capital. Now, it’s cliché to complain about how hard it is to link security to those issues, and so we invent stuff to report on, like “maturity,” thinking it sounds strategic. It doesn’t. 

Look, executives become executives because they’re good at making decisions about complex questions with big impacts. Is it harder in security? Well, yes, we blindfold ourselves, we rail against talking about our mistakes, and then wonder why no one ever gets better. But that’s a problem we have to face within security, and in the meanwhile, we need to find metrics or frameworks that matter to our executives, that the business understands, and that we can speak. In that order.

So the lesson is: figure out the metrics that matter to you, and figure out the metrics that matter to the business. Some of it will be hard to gather, some of it will be impossible. But you don’t want to be like the drunk looking for their keys under the streetlight, even if the light is better there.

Next week, we’ll get down and dirty and talk about what those metrics are not. Here’s a hint: they’re not about things you can’t control. 

Oh -- and incidentally, that fund? Down 20% when I sold it.

Related content:

 

Interop 2016 Las VegasFind out more about security threats at Interop 2016, May 2-6, at the Mandalay Bay Convention Center, Las Vegas. Register today and receive an early bird discount of $200.

Adam is a consultant, entrepreneur, technologist, author and game designer. He's a member of the BlackHat Review Board and helped create the CVE and many other things. He currently helps organizations improve their security via Shostack & Associates, and advises startups ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Joe Stanganelli
50%
50%
Joe Stanganelli,
User Rank: Ninja
3/18/2016 | 9:28:01 AM
Measuring in the real world
I strongly suspect that the security industry (or, at least, security industry marketing) focuses on these vulnerability-related metrics because other security metrics are so difficult to -- well -- measure.

It's also interesting to note how lab tests can different from "real-world" environments and results.  NSS Labs (which was at RSA) released a NGFW study that -- in addition to its basic tests -- purported to offer results from tests emulating various "real-world" environments.  What was interesting here is that where one of the NGFWs (made by Palo Alto Networks) smoked the competition on performance in all the other tests, others performed better in NSS's "real-world" datacenter test.
Cybersecurity Team Holiday Guide: 2019 Gag Gift Edition
Ericka Chickowski, Contributing Writer,  12/2/2019
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Current Issue
Navigating the Deluge of Security Data
In this Tech Digest, Dark Reading shares the experiences of some top security practitioners as they navigate volumes of security data. We examine some examples of how enterprises can cull this data to find the clues they need.
Flash Poll
Rethinking Enterprise Data Defense
Rethinking Enterprise Data Defense
Frustrated with recurring intrusions and breaches, cybersecurity professionals are questioning some of the industrys conventional wisdom. Heres a look at what theyre thinking about.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2019-19647
PUBLISHED: 2019-12-09
radare2 through 4.0.0 lacks validation of the content variable in the function r_asm_pseudo_incbin at libr/asm/asm.c, ultimately leading to an arbitrary write. This allows remote attackers to cause a denial of service (application crash) or possibly have unspecified other impact via crafted input.
CVE-2019-19648
PUBLISHED: 2019-12-09
In the macho_parse_file functionality in macho/macho.c of YARA 3.11.0, command_size may be inconsistent with the real size. A specially crafted MachO file can cause an out-of-bounds memory access, resulting in Denial of Service (application crash) or potential code execution.
CVE-2019-19642
PUBLISHED: 2019-12-08
On SuperMicro X8STi-F motherboards with IPMI firmware 2.06 and BIOS 02.68, the Virtual Media feature allows OS Command Injection by authenticated attackers who can send HTTP requests to the IPMI IP address. This requires a POST to /rpc/setvmdrive.asp with shell metacharacters in ShareHost or ShareNa...
CVE-2019-19637
PUBLISHED: 2019-12-08
An issue was discovered in libsixel 1.8.2. There is an integer overflow in the function sixel_decode_raw_impl at fromsixel.c.
CVE-2019-19638
PUBLISHED: 2019-12-08
An issue was discovered in libsixel 1.8.2. There is a heap-based buffer overflow in the function load_pnm at frompnm.c, due to an integer overflow.