Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


04:46 PM
George V. Hulme
George V. Hulme

Microsoft's Security Development Life Cycle (SDL) Metrics: Microsoft Can Do Better

Microsoft can, and should, provide more insight into how well its security development life cycle is working.

Microsoft can, and should, provide more insight into how well its security development life cycle is working.Burton Group security analyst Pete Lindstrom set the security blogoshere ablaze with his post Microsoft's SDL has Saved the World!. David Maynor at Errata Security calls Lindstrom an old man. While Michael Howard, a senior security program manager at Microsoft responded here.

At issue isn't the value of improving software development processes to incorporate security throughout the development life cycle. What's at issue is whether or not Microsoft's SDL has improved the security of its software when it's shipped at GA. I certainly believe it has come a long way since this day (link), and that Microsoft is using public vulnerability discovery counts as its measurement of success.

Lindstrom makes a number of points, some are dubious, others are worth consideration as to why the number of publicly discovered vulnerabilities is on the wane. And not all of these have anything to do with an improvement in development:

1. Microsoft has hired/contracted with the most talented bugfinders, and they're no longer going public with the bugs they find. 2. Bugfinders are selling (or keeping) the vulnerabilities they find underground. (There's been more and more of this going on in the past few years). 3. Bugfinders are growing bored with Microsoft operating systems, and focusing more on the client applications. 4.Microsoft has improved its software development processes.

While I find the first point dubious at best, even Microsoft can't hire every skilled bugfinder. But, points two, three, and four are all contributing factors as to why the number of publically disclosed vulnerability counts is down.

In Howard's retort to Lindstrom's post, Howard quoted Lindstrom as saying the following:

"Microsoft has systematically hired and/or contracted with every one of their most vocal critics (and most seasoned bugfinders) to do the work behind the scenes and they don't count those vulns!"

This is part of Howard's response:

But in making this assertion, he's saying the vulnerabilities we remove (and do not add to the code in the first place) as part of the SDL process should be counted as though they were part of the product after we shipped it. We don't count vulnerabilities that don't affect customers, regardless of the vendor.

My question is: Why doesn't Microsoft count the vulnerabilities it finds and eliminates during development? How else can success be measured, as well as improve internal processes and the focus on developer training and education?

Instead, the market just fed the rhetoric that since there are fewer publically discovered vulnerabilities, the secure development life cycle is a success. This just isn't intrinsically so. Correlation isn't always causation. There may be many, many other external factors that could lead to a reduction in publicly discovered vulnerabilities.

From Michael Howard's blog:

Some of the many SDL principles that reduce or mitigate security bugs include:

• Mandatory education (Net effect: fewer security bugs up front)

• Design decisions based on threat models (Net effect: fewer security design bugs up front)

• Cannot use known insecure APIs (Net effect: fewer security bugs up front)

• Use of static analysis tools (Net effect: fewer security bugs enter the central source code repository)

• Cannot use known weak crypto primitives and key lengths (Net effect: fewer security bugs up front)

• Compiler and linker requirements (Net effect: extra defenses, in case you miss a bug)

• Fuzz testing (Net effect: implementation bugs found before shipping)

There are a lot of claims here, so why not share some numbers that substantiates them. Surely we can't expect Microsoft to share all of its internal numbers, but why can't the company reveal the percentage reduction of security bugs discovered earlier in development, say per thousand lines of code? Or what's the percentage of vulnerabilities reduced during static code analysis in products developed in the SDL?

How well is the mandatory developer education is working? How well is threat-modeling and mis-use cases working at reducing the attack surface -- it must be very difficult improvement the processes. There's much the industry could learn here.

I'm not aware of any other company on the planet that relies on the random process of public defect finding as their primary measurement of quality control. Surely, Microsoft can do better than just pointing to publicly discovered vulnerability counts as the only measure of success.


Recommended Reading:

Comment  | 
Print  | 
More Insights
Threaded  |  Newest First  |  Oldest First
COVID-19: Latest Security News & Commentary
Dark Reading Staff 8/3/2020
'BootHole' Vulnerability Exposes Secure Boot Devices to Attack
Kelly Sheridan, Staff Editor, Dark Reading,  7/29/2020
Average Cost of a Data Breach: $3.86 Million
Jai Vijayan, Contributing Writer,  7/29/2020
Register for Dark Reading Newsletters
White Papers
Cartoon Contest
Current Issue
Special Report: Computing's New Normal, a Dark Reading Perspective
This special report examines how IT security organizations have adapted to the "new normal" of computing and what the long-term effects will be. Read it and get a unique set of perspectives on issues ranging from new threats & vulnerabilities as a result of remote working to how enterprise security strategy will be affected long term.
Flash Poll
The Threat from the Internetand What Your Organization Can Do About It
The Threat from the Internetand What Your Organization Can Do About It
This report describes some of the latest attacks and threats emanating from the Internet, as well as advice and tips on how your organization can mitigate those threats before they affect your business. Download it today!
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2020-08-04
In solidus before versions 2.8.6, 2.9.6, and 2.10.2, there is an bility to change order address without triggering address validations. This vulnerability allows a malicious customer to craft request data with parameters that allow changing the address of the current order without changing the ship...
PUBLISHED: 2020-08-04
Extreme Analytics in Extreme Management Center before allows unauthenticated reflected XSS via a parameter in a GET request, aka CFD-4887.
PUBLISHED: 2020-08-04
save-server (npm package) before version 1.05 is affected by a CSRF vulnerability, as there is no CSRF mitigation (Tokens etc.). The fix introduced in version version 1.05 unintentionally breaks uploading so version v1.0.7 is the fixed version. This is patched by implementing Double submit. The CSRF...
PUBLISHED: 2020-08-04
An exploitable arbitrary file delete vulnerability exists in SoftPerfect RAM Disk 4.1 spvve.sys driver. A specially crafted I/O request packet (IRP) can allow an unprivileged user to delete any file on the filesystem. An attacker can send a malicious IRP to trigger this vulnerability.
PUBLISHED: 2020-08-04
An issue was discovered in the Gantt-Chart module before 5.5.4 for Jira. Due to a missing privilege check, it is possible to read and write to the module configuration of other users. This can also be used to deliver an XSS payload to other users' dashboards. To exploit this vulnerability, an attack...