Software Security Metrics and Commentary - Part 2

Part 1 here

In Part-1 of this entry I talked about the first 5 metrics from the paper "A Metrics Framework to Drive Application Security Improvement".

In part-2 of this piece I'll try to cover the remaining 5 metrics as well as discuss a few thoughts on translating survivability/Quality-of-Protection into upstream SDL metrics.

First, onto the other five metrics from the paper:

• Injection Flaws
• Again, I think the metric posited in the paper is too tilted towards incident discovery rather than prevention. Just like the XSS metric I added - OutputValidation , this is really the key to prevention here. Most static analysis tools can detect tainted input and have a set of untrusted input functions (things that read from sockets, stdin, etc). It should be relatively straightforward to model our own application-specific output functions to detect where we're handing unchecked/unfiltered input to an output routine, potentially those across a trust boundary. If we can model these, we can at least make sure we have good sanitization coverage for each output type. We'll want to have this type of output filtering anyway, we might as well combine metrics from our XSS example.
• Improper Error Handling
• I think the metric posed in the paper - counting unchecked returns is a pretty good idea. This isn't going to catch web-server layer errors unfortunately, and won't necessarily detect errors in things like app servers, db-layers, etc. We can test for these, but the best metrics might be those related to following secure configuration guidance such as the CIS guide for individual web servers and/or app servers. The CIS benchmark for example requires a compliant configuration to handle standard web errors (4xx and 5xx) through rewrites and/or custom handlers. There are cases (SOAP comes to mind) where we need to throw a 5xx error back to a client, but this is the exception rather than the norm. Configuring application and web servers to minimize this sort of data disclosure is certainly a good thing, and in this sense we can check for compliance at this layer as almost a binary config - you pass the CIS guidance or you don't.
  
• Insecure Storage
• I don't think the metric of percent encrypted hard drives is really a meaningful metric in this context. If we look at typical web attacks that fall into this category we'd be looking at exploits that leak things like passwords, CC-data, etc. that is stored in an improper manner on the webserver. Some of this is going to be related to the implementation in the code, and so our best bet is probably a detailed audit of each piece of information that falls into this criticality range to confirm that it is being handled in an appropriate manner. I struggle to find a concrete metric that helps to measure this however. PercentCriticalDataCovered for proper encryption/hashing technique? Still not a very convincing metric unfortunately.
• Application Denial of Service
• Two metrics spring to mind here:
• Memory/Resource Allocations Before Authentication
• Memory Leaks
• Both of these are a lot more likely to lead to application denial of service than any other errors I can think of. Both of these should be minimized. Tracking them and having the absolute fewest of them is probably a good bet. That doesn't mean we're not going to have a DoS issue, but these are at least 2 places to look.
  
• Insecure Configuration Management
• This item probably goes back to the same metrics I posited for Improper Error Handling. Things like the CIS benchmarks for OS, webserver, and appserver are our first pass candidates for measuring this.

On the question of survivability I was struck by a presentation and paper Steve Bellovin did last year about this topic at the first Metricon - "On the Brittleness of Software and the Infeasibility of Security Metrics." He published a paper and a presentation about it.

Steve makes what I believe are two major points in this paper:

• Software is brittle, it fails catastrophically
• Unlike other engineering disciplines, we don't know how to get to certainty about the strength of a piece of software.

I won't disagree with either of these points, but to an extent you can say this about all new technologies. We've had catastrophic failures in physical engineering before as well. Old materials fail in sometimes new ways, new materials fail in unpredictable ways, and we still rely on sample and testing for analysis of a batch of materials.

The Quality of Protection workshop at the CCS conference is probably the best place to look for research in this area. Previous papers from the workshop can be found here. This years conference and workshop is starting next week, if you're in the DC area and interested in software security metrics it looks like its going to be a good event. The accepted papers list contains a number of papers that I think might shed some light on my speculation above.

I plan to put together a few more thoughts on brittle failure modes of software in a followup to this, I haven't had time to pull all of my thoughts together yet.