Operations

3/17/2016
04:00 PM
Adam Shostack
Adam Shostack
Commentary
Connect Directly
Twitter
LinkedIn
RSS
E-Mail vvv
50%
50%

Security Lessons From My Stock Broker

Or, how to lie with metrics.

A few years back, I called my stock broker and asked for help selecting a growth fund to diversify my holdings a bit. He said he had this great fund that was totally a fit for what I needed.  (Have you ever called a salesperson and not heard that they had what you needed? When you do, pay attention. Those are the ones doing real strategic sales.)

This fund had great performance relative to the Russell 2000, had a low beta (a measure of volatility), and blah blah. Frankly, I don’t remember the points he made when selling me. They were his points, not my points. Some of them were real metrics, that were relevant to what I wanted to know, and some were what the lean startup movement calls “vanity metrics.”

But they were his metrics, not mine. I had not done the hard work of figuring out what mattered to me, and ensuring that the things I wanted were being measured. So I was an easy mark.  There are two lessons here: one for people buying products and services, and one for those producing metrics for “the business.” 

Source: Pixabay
Source: Pixabay

Walking around at RSA, it seems that every product today has its own “single pain of glass.”  (No, not pane, trust me, they’re misspelling it.) These pains of glass take metrics that a product manager selected, just like my stock broker selected his metrics. And you’re going to have a lot of them, and they’ll be pains. They’ll be numbers that you can, with work, influence, but that work doesn’t mean your business is more secure. But now that you’re measuring them, you better start influencing them. You’re going to be held accountable for the numbers that you bought.

Let’s take an example of vulnerability counts. Vulnerability counts have, at best, a complex relationship to consequential events. As someone who helped get the CVE off the ground, I know that there are plenty of real issues (word macros, dll injection) which real attackers exploit and which don’t get fixed. Others, like Autorun, do get fixed, without a CVE, because they’re not bugs, but features. There are also plenty of real vulnerabilities, such as SQL injection in your custom database, that don’t get a CVE. (I hope that those are bugs, not features.)

The question you’d like to ask, the thing that you’d like to measure, is not vulnerabilities. You probably want to influence vulnerabilities because you think they correlate with the consequential events that your business cares about, and they might. But as we’ve just discussed, they are not a complete metric of what matters to the business, and we don’t have a good way to estimate their incompleteness. So, not measuring what you care about or being tightly correlated with what you care about means they’re a bad executive metric.

And here’s the lesson my stock broker can teach those producing metrics for the business. Don’t be like my stock broker. It’s a short-term business model. Business has a way of looking at issues. Profit and loss. Return on capital. Now, it’s cliché to complain about how hard it is to link security to those issues, and so we invent stuff to report on, like “maturity,” thinking it sounds strategic. It doesn’t. 

Look, executives become executives because they’re good at making decisions about complex questions with big impacts. Is it harder in security? Well, yes, we blindfold ourselves, we rail against talking about our mistakes, and then wonder why no one ever gets better. But that’s a problem we have to face within security, and in the meanwhile, we need to find metrics or frameworks that matter to our executives, that the business understands, and that we can speak. In that order.

So the lesson is: figure out the metrics that matter to you, and figure out the metrics that matter to the business. Some of it will be hard to gather, some of it will be impossible. But you don’t want to be like the drunk looking for their keys under the streetlight, even if the light is better there.

Next week, we’ll get down and dirty and talk about what those metrics are not. Here’s a hint: they’re not about things you can’t control. 

Oh -- and incidentally, that fund? Down 20% when I sold it.

Related content:

 

Interop 2016 Las VegasFind out more about security threats at Interop 2016, May 2-6, at the Mandalay Bay Convention Center, Las Vegas. Register today and receive an early bird discount of $200.

Adam is an entrepreneur, technologist, author and game designer. He's a member of the BlackHat Review Board, and helped found the CVE and many other things. He's currently building his fifth startup, focused on improving security effectiveness, and mentors startups as a ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Joe Stanganelli
50%
50%
Joe Stanganelli,
User Rank: Ninja
3/18/2016 | 9:28:01 AM
Measuring in the real world
I strongly suspect that the security industry (or, at least, security industry marketing) focuses on these vulnerability-related metrics because other security metrics are so difficult to -- well -- measure.

It's also interesting to note how lab tests can different from "real-world" environments and results.  NSS Labs (which was at RSA) released a NGFW study that -- in addition to its basic tests -- purported to offer results from tests emulating various "real-world" environments.  What was interesting here is that where one of the NGFWs (made by Palo Alto Networks) smoked the competition on performance in all the other tests, others performed better in NSS's "real-world" datacenter test.
Microsoft President: Governments Must Cooperate on Cybersecurity
Kelly Sheridan, Staff Editor, Dark Reading,  11/8/2018
5 Reasons Why Threat Intelligence Doesn't Work
Jonathan Zhang, CEO/Founder of WhoisXML API and TIP,  11/7/2018
Why the CISSP Remains Relevant to Cybersecurity After 28 Years
Steven Paul Romero, SANS Instructor and Sr. SCADA Network Engineer, Chevron,  11/6/2018
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
Flash Poll
Online Malware and Threats: A Profile of Today's Security Posture
Online Malware and Threats: A Profile of Today's Security Posture
This report offers insight on how security professionals plan to invest in cybersecurity, and how they are prioritizing their resources. Find out what your peers have planned today!
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2018-19205
PUBLISHED: 2018-11-12
Roundcube before 1.3.7 mishandles GnuPG MDC integrity-protection warnings, which makes it easier for attackers to obtain sensitive information, a related issue to CVE-2017-17688. This is associated with plugins/enigma/lib/enigma_driver_gnupg.php.
CVE-2018-19206
PUBLISHED: 2018-11-12
steps/mail/func.inc in Roundcube before 1.3.8 has XSS via crafted use of <svg><style>, as demonstrated by an onload attribute in a BODY element, within an HTML attachment.
CVE-2018-19207
PUBLISHED: 2018-11-12
The Van Ons WP GDPR Compliance (aka wp-gdpr-compliance) plugin before 1.4.3 for WordPress allows remote attackers to execute arbitrary code because $wpdb->prepare() input is mishandled, as exploited in the wild in November 2018.
CVE-2018-1786
PUBLISHED: 2018-11-12
IBM Spectrum Protect 7.1 and 8.1 dsmc and dsmcad processes incorrectly accumulate TCP/IP sockets in a CLOSE_WAIT state. This can cause TCP/IP resource leakage and may result in a denial of service. IBM X-Force ID: 148871.
CVE-2018-1798
PUBLISHED: 2018-11-12
IBM WebSphere Application Server 7.0, 8.0, 8.5, and 9.0 is vulnerable to cross-site scripting. This vulnerability allows users to embed arbitrary JavaScript code in the Web UI thus altering the intended functionality potentially leading to credentials disclosure within a trusted session. IBM X-Force...