Automated Web Application Security Scanners

Jeremiah Grossman recently made this blog post, in regards to a comparison on automated web application security scanners.

I personally found the comparison a really interesting read. This is a topic that tends to spark a lot of discussion within organisations - certainly did amoungst my team. :)

I'd like to say that I am all in favor of these tools but there has to be some caveats -

1. They are no replacement for a human being.
2. They are no replacement for a skilled human being.
3. They are useful when conducting a normal web app pentest across a large number of sites is infeasible.
4. Do not rely on results which haven't been validated and put into context.
5. Be prepared to scrap them when they fail.

I'm aware of some enterprises using these within their security teams, often as a method for addressing web application security concerns. The problem is they aren't staffed with appropriately skilled individuals with penetration testing, software development and coding experience.

As this report demonstrates, these tools in point-and-shoot mode pose numerous issues:

E.g.

• They do not return all the vulnerabilities that exist.
• Many of the vulnerabilities found are actually false.
• The speed at which they operate is often woeful.
• The tools often require extensive time tuning to a given website.

The list goes on. And to some extent, this can't be helped. I mean even a human penetration tester isn't going to necessarily find every vulnerability. That's just the way it goes. They might come damn close but it just ain't the same.

But these issues can only be addressed by an intelligent penetration tester. One who can chuck the tool when needed, perform a SQL injection attack or modify a HTTP response on the fly via Burp proxy, etc. I mean nobody complains about having a skilled firewall administrator behind Checkpoint, or a DBA behind Oracle RAC, or sysadmin rebuilding a SAN array. So why the hell is there so much resistance behind hiring a skilled pentester often ignored?

If you're a security manager or application support or development manager thinking of purchasing one of these tools (or already own one), have a good long hard think about your reasons for purchasing one and how are you maximising its use. If you don't have a suitably experience penetration tester steering it, I would suggest that the money would be better spent scrapping that product and using the money saved on consulting to evaluate your software assurance processes and how you can better integrate application security testing into it. And if you cannot afford that, maybe you're better off outsourcing that next penetration test on a critical application rather than rely on automated tools to try and cover the lot and then make sense of the trainwreck.

- J.