Building Effective Metrics

The topic of metrics for Information Security comes up quite often. I've been in quite a number of situations where a relatively sparse infosec program exists and no metrics exist. The question often comes up of what types of metrics to gather first to measure program status, effectiveness, etc. And, when rolling out a new element of an infosec program what metrics to focus on first.

I've come to the conclusion that process maturity based metrics are the best thing to worry about when you're building an infosec program or a new feature of an existing program.

Let's take several areas of Infosec and examine my premise.

• Vulnerability Management (discovery and remediation)
• Anti-virus software
  
• Software Security

Vulnerability Management

When you're first starting to build a vulnerability management program you're worried about a few things:

1. Measuring existing vulnerabilities
2. Remediating vulnerabilities
3. Eventually, reducing the number of vulnerabilities that get "deployed"

Most people try to tackle these items in numerical order. They buy a vulnerability scanner, they start scanning their network, they come up with a giant list of security vulnerabilities, and then they try to tackle #2, remediation. They generally set the bar pretty high in terms of what they expect the organization to fix. For example, all level 5,4,3 vulnerabilities in a Qualys scan. They push the vulnerability report to the systems administration staff, tell them to go fix the vulnerabilities, and wait an eternity to hear back about the vulnerabilities, what has been done, etc. Usually they get upset that things are being fixed faster, that new vulnerabilities surface faster than they can close the old ones, and they either give up and start ignoring their vulnerability scans, or they get extremely frustrated with the admins and a constant battle ensues.

Instead, I like to tackle these items in reverse order of the above list:

1. Reduce the number of new vulnerabilities that get deployed
2. Implement a remediation process
3. Search for vulnerabilities and feed them into #2.

In my experience most people want to go a good job at what they do. They don't want to release systems with holes in them, have their systems get hacked, etc. Unfortunately they aren't security experts and don't know what to focus on. They need assistance and prescriptive guidance on exactly what to do and when to do it.

Step 1: Reduce the Number of New Vulnerabilities

Start with something like a system hardening guide and approved software list. You pick things like the CIS hardening standards and ensure that all new systems getting built go out the door with your hardening applied. In this way you cut down on the number of new vulnerabilities you're introducing into your environment.

Step 2: Implement a Remediation Process with Metrics

Work on a remediation process. Focus on elements such as:

• Who is responsible for requesting something be remediated
• Who is responsible for determining the scope of the request and its priority/severity
  
• What testing has to be done, and who must approve it in order to push something to the environment
• How do you track status through each of these items including time taken, roadblocks, etc.
• How much did it cost to fix each vulnerability
  

Building your remediation process before you start up the firehose gives you several advantages:

1. You can start slow at feeding vulnerabilities into the remediation process and get useful metrics about the costs of remediation.
2. You don't cause undue friction with the operations staff by asking them to take on too much too soon.
3. You have a well-established process for fixing any/all vulnerabilities you discover.
   

Once you've got this process created you can measure how effectively you're remediating any given vulnerability you discover. You have process metrics for your remediation process, rather than an ad-hoc best effort situation.

Step 3: Search for Vulnerabilities and Feed Them to Your Remediation Process

One you have a repeatable remediation process, you're ready to start feeding the process new vulnerabilities. In an organization that isn't used to routine patching, turning off services, remediating vulnerabilities you can't start out with a firehose of vulnerabilities to unprepared staff. The best approach is to use the metrics you've created in step-2 and be selective with what vulnerabilities you ask to be fixed. Once you have the process in place you can choose to stat with a subset of your vulnerabilities - your example your Qualys level-5 vulnerabilities. Ramp up slowly to the organization so that you can adequately measure the impact of your changes, the value they are providing, and the costs of remediating.

Get people used to being accountable for fixing vulnerabilities, for testing the fixes, and for measuring the results. Once you have that in place you're free to ramp up the security level you want to achieve in a measured fashion.

Eventually, once you finally have a handle on these three steps you can move on to more advanced metrics such as:

• Average time to remediation
• Overall vulnerability score

Until you have the first three pieces in place though focusing on your overall risk/vulnerability isn't that interesting. Even if you don't like the score, you're never going to get it lower without a repeatable process in place to remediate.

More on process related metrics for Anti-virus and Software Security in a later post.