I've come to the conclusion that process maturity based metrics are the best thing to worry about when you're building an infosec program or a new feature of an existing program.
Let's take several areas of Infosec and examine my premise.
- Vulnerability Management (discovery and remediation)
- Anti-virus software
- Software Security
When you're first starting to build a vulnerability management program you're worried about a few things:
- Measuring existing vulnerabilities
- Remediating vulnerabilities
- Eventually, reducing the number of vulnerabilities that get "deployed"
Instead, I like to tackle these items in reverse order of the above list:
- Reduce the number of new vulnerabilities that get deployed
- Implement a remediation process
- Search for vulnerabilities and feed them into #2.
Step 1: Reduce the Number of New Vulnerabilities
Start with something like a system hardening guide and approved software list. You pick things like the CIS hardening standards and ensure that all new systems getting built go out the door with your hardening applied. In this way you cut down on the number of new vulnerabilities you're introducing into your environment.
Step 2: Implement a Remediation Process with Metrics
Work on a remediation process. Focus on elements such as:
- Who is responsible for requesting something be remediated
- Who is responsible for determining the scope of the request and its priority/severity
- What testing has to be done, and who must approve it in order to push something to the environment
- How do you track status through each of these items including time taken, roadblocks, etc.
- How much did it cost to fix each vulnerability
- You can start slow at feeding vulnerabilities into the remediation process and get useful metrics about the costs of remediation.
- You don't cause undue friction with the operations staff by asking them to take on too much too soon.
- You have a well-established process for fixing any/all vulnerabilities you discover.
Step 3: Search for Vulnerabilities and Feed Them to Your Remediation Process
One you have a repeatable remediation process, you're ready to start feeding the process new vulnerabilities. In an organization that isn't used to routine patching, turning off services, remediating vulnerabilities you can't start out with a firehose of vulnerabilities to unprepared staff. The best approach is to use the metrics you've created in step-2 and be selective with what vulnerabilities you ask to be fixed. Once you have the process in place you can choose to stat with a subset of your vulnerabilities - your example your Qualys level-5 vulnerabilities. Ramp up slowly to the organization so that you can adequately measure the impact of your changes, the value they are providing, and the costs of remediating.
Get people used to being accountable for fixing vulnerabilities, for testing the fixes, and for measuring the results. Once you have that in place you're free to ramp up the security level you want to achieve in a measured fashion.
Eventually, once you finally have a handle on these three steps you can move on to more advanced metrics such as:
- Average time to remediation
- Overall vulnerability score
More on process related metrics for Anti-virus and Software Security in a later post.