Tuesday, November 27, 2007

Poll: How Important is a POC When Getting Someone to Fix a Security Issue

Working on security inside a company that takes security seriously sometimes blinds me to how other people work and the challenges they face with getting security issues taken seriously.

I've noticed that lots of people that work as consultants and/or inside companies have to jump through lots of hoops to get a security vulnerability taken seriously.

In many cases I see people spending hours and hours crafting a working proof-of-concept exploit for a vulnerability and needing to actually demonstrate that exploit to get the issue taken seriously.

To understand this better, I set up a small poll to get some data about why people are needing to craft a working POC when demonstrating a vulnerability exists.

I've only ever had to do this once, and yet it seems that every time I read about a penetration test I see people spending lots of time crafting sample exploits rather than spending more time on finding more vulnerabilities, or fixing classes of vulnerabilities that are similar and offering solutions to those.

In my experience the only time a POC has been really useful is when I need to make sure that the person fixing the issue has the necessary information/tests to make sure they've closed the issue.

For those who do penetration tests (network or application) - how often do you feel that you need to create working POCs for exploits in order for the company's management to take it seriously?


Anonymous said...

POC's are important because you can't always be sure a hole has been closed without testing it. I always try to have a POC for every find I report. It also gives the organization the ability to test it at a later time once you are off-site.

Anonymous said...

POCs are helpful in demonstrating the reproducibility of a problem. I happen to subscribe to DREAD as a threat modeling methodology (I subscribe to others as well). By understanding what it takes to reproduce a vulnerability is helpful in determining the overall impact of a given vulnerability. This also helps in prioritizing software maintenance activities. Just my $.02.

Andy Steingruebl said...

Good points both. I can definitely see using a POC to demonstrate work factor for breaking something, especially if we need to prioritize fixes for multiple vulnerabilities.

That said, I'm pretty disturbed at times by how much of a testing engagement goes towards fully working exploits rather than on finding more defects.

When I engage folks for this sort of work its a fine line between them giving you tests that let you know when you've fixed something, and spending lots of time demonstrating that something can be fully exploited even when there is an obvious hole.

Take a look at the work teh Core Impact guys had to do to prove to the OpenBSD folks that the kernel error earlier this year was exploitable. I'm thinking they shouldn't have needed to spend as much time on it to get the point across, and if I'd been paying for their engagement I wouldn't have insisted on the exploit, I'd have filed a priority defect right away on it.