Measuring the Wrong Things?

I'm not sure why I'm always finding interesting articles in NPR about medicine that seem to resonate so much in relation to software security. Nonetheless that seems to be how things go, so here comes another one.

NPR ran a story the other day titled "Doctors' 'Treat the Numbers' Approach Challenged". The main idea in the story is that doctors have been treating patients and using the results of certain tests as the metrics by which they judge health. They treat a patient with drugs, therapies, etc. to get to the diagnostic numbers they want, but now we're finding out that perhaps the numbers are not necessarily representing what we'd like them to.

The example from the article was:

Doctors call it "treating the numbers" — trying to get a patient's test results to a certain target, which they assume will treat — or prevent — disease. But earlier this year, a study on a widely used cholesterol drug challenged that assumption.

Vytorin, a combination of two cholesterol-lowering agents, certainly lowers cholesterol. But patients taking it didn't have any less plaque in a major artery than those taking a less-potent drug.

I'm assuming that less plaque generally does translate to fewer adverse events, but the article doesn't cover this. Helpfully, in medicine we generally have a pretty clear definition of an adverse event, and we're not dealing with intelligent active threats. Active threats (virus, bacteria, fungus, parasite), but not intelligent... We don't try to design cholesterol treatments to fend off a malicious food company that has designed a new more dangerous form of cholesterol that our drug can't fight :)

Knowing what to measure in security is hard though. We've covered a little of this before here.

If you're looking for more formal treatments of security metrics - check out the Quality of Protection (QoP) workshop held as part of the ACM CCS Conference.

"The goal of the QoP Workshop is to help security research progress towards a notion of Quality of Protection in Security comparable to the notion of Quality of Service in Networking, Software Reliability, or measures in Empirical Software Engineering."

Over the next few posts I'll take a few of the papers from the workshop and discuss a bit of their results. If you're interested in the TOC for the workshop, you can find it here.