Metrics and Audience

There has been some chatter recently about a post Pete Lindstrom made about Microsoft's SDL and their publicly disclosed metrics. I chimed in on Pete's blog as well as on the Microsoft SDL blog, here is a little more.

The fundamental confusion here is about the audience for the vulnerability numbers, and metrics in general.

There are several audiences here:

1. Microsoft's customers, competitors, and the public at large.
2. Security folks, especially software security folks that want to improve the quality of their software.
3. People who want more metrics about all things generally, the costs of security, etc.

Microsoft's vulnerabilities in shipped software metric is really only targeted to audience #1. Like it or not, what customers care about, as Michael Howard rightly points out, is how likely they are to get attacked/hacked, and how many patches they have to deploy. Microsoft for its part also cares about #1 for the reasons above, and the fact that releasing patches is extraordinarily expensive.

Security folks, especially those working on their own software security initiatives find the vulnerabilities metric mostly useless. It gives us no insight into *how* Microsoft has achieved the reduction in vulnerabilities. What we'd all love to know is how much each element of the SDL contributes to reducing vulnerabilities. A percentage break out on how effective each element is, Training, Threat Modeling, Testing, at reducing vulnerability counts, especially as broken out by design/architecture defects and implementation defects.

At the same time, I'm willing to acknowledge that developing these metrics is a full time job for multiple people. And, tracking the metrics over time is difficult, since its hard to normalize the defects between products and across time. New attacks are always surfacing, so how do you track the impact of new attack types across time. How do you track the impact of better code scanning/static-analysis tools over time. As the tool improves you'll find more defects when you run it, but that will skew your metrics somewhat.

The fundamentally unanswered question though is how do we actually measure the security of software. From a general customer standpoint what you care about is how likely you are to get attacked and compromised for one piece of software vs. another, what that software is going to cost to buy and run, etc.

For the security community what we're looking for is a metric that more closely tracks the "real" security properties of a piece of software. How hard it is for the expert to attack, how it does in real world deployments, etc.

Unfortunately no one metric is going to capture this. As I've previously mentioned the QoP workshop is a great place to go if you're looking for answers to the "how do we measure software security" question. But if what you want to know is how much is it generally going to cost me to run/implement a piece of software, looking at things like number of required patches and their frequency/severity, then perhaps Microsoft's vulnerability metric is for you.