Is Vulnerability Fix Time Really the Best Software Security Metric?

Johnathan Nightingale has a new post up over at the Mozilla Security Blog. His post is in response to a report by Bit9 about vulnerabilities in software and a comparison between packages and their published vulnerabilities.

Jonathan makes the claim (Apologies for quoting so much here):

To suggest that this openness is a weakness because it means that we have “reported vulnerabilities” is to miss the reality: that software has bugs. A product’s responsiveness to those bugs and its ability to contain them quickly and effectively is a much more meaningful metric than counting them.

The Firefox vulnerabilities Bit9 discusses are long-since fixed, with the majority of these fixes coming within days of it being announced. That is the real measure of application security: are known vulnerabilities fixed promptly, tested carefully, and deployed thoroughly? When people have asked that question, Firefox and Mozilla have consistently come out ahead.

Bug counting is unfortunately common because it’s easy, but it should not be a substitute for real security measurement.

I unfortunately have to take issue with Jonathan's definition. The vulnerability exposure isn't the only measure of software security, especially if the vulnerability exposure window ois only measured from the time a defect is disclosed, not the time it existed in the wild. Other important metrics are:

• How many defects and of what severity are in the shipped product
• What type of defects are they and why did they show up (and weren't found in testing
• Defects discovered in the wild over time (is the software getting better or worse)
  

Software does get attacked with 0day exploits, witness last weeks IE exploits. For people exploited due to this vulnerability knowing Microsoft patched it quickly will be poor consolation for the damage they have suffered.

I don't doubt that Mozilla takes security seriously but a statement that "Software has bugs" isn't really responsive to the idea that over time software ought to have fewer of them.