Counterpoint: Commoditising Penetration Testing

Recently Drazen Drazic put out an article on his blog at Beast Or Buddha about penetration testing and how it has become heavily commoditised. Or more specifically how businesses and individuals are attempting to do so, when infact they can't.

Drazen points out that there are some things you can commoditise, good penetration testing you can't. While I agree with this 100%, I do disagree with his view on commoditisation. I hope I can present a different viewpoint.

Commoditisation in this sense means that the skills required are becoming something that can be trained. Look at penetration testing back in say 2000. It wasn't that many years earlier the first book 'Maximum Security', the first book to really look at this in any depth was written. Hacking texts were limited to old BBSes, early underground forums, newsgroups, etc. You had to know who to talk to and where to look. More often than not, you were told to go figure out stuff yourself. If you had to ask, you fail. So in its earliest forms, hacking was really a form of exploration, self-discovery and learning (and understanding) technology. It was like this for many years prior to 2000 as well but the point being is that this is the mold that many of the "old school" penetration testers come from.

Fast forward to 2010. We now have course after course after course, certification after certification, book after book. For those of us who have been at this long enough many of these options are regarded as a mixed bag. Some people see these as "dumbing down" the content and presenting the technical tools and methods without any understanding/appreciation for the underlying technology. Others see them as not providing true value to prospective students. Some see them as an entry path into the field, lowering the barrier to entry that has often existed in this field. And there are many compelling arguments for each position.


Where I sit, the commoditisation is good.  I'll try and explain why but I'll also explain why commoditisation doesn't actually hurt penetration testing.


1) Unit testing

Many of the tests to perform a specific function are very simple. Lets say you want to test that input validation is working on a single form field. It is relatively simple for anyone to peform this test. Training your application testers on basic web application penetration testing (lets assume developers are doing some form of development training) means then they are able to validate that no cross site scripting is present. While this certainly doesn't obviate the need for penetration testing at the final phase or even multiple phases, what it does do is reduce the likelihood of critical security errors common to applications. It reduces the load on the penetration testers. But most importantly, it means these defects are detected earlier. Now, if you're a security manager overseeing large volumes of application development and testing, wouldn't you want to see your testing teams attempt to validate your security requirements as much as possible prior to obtaining independent penetration testing? I know I do.

2) Trained vs. Skilled

Secondly, good penetration testers is actually hard to come by. It is a very specific skillset that requires someone with a good grasp on just about all technical areas and in some cases, a specialisation or two. Networks, operating systems, applications, databases, programming. All this and more is part of their skillset. They understand how people think, how programmers write code, how administrators handle their environments. They can think like an attacker and more frighteningly, it comes naturally. They can pull apart just about any piece of technology you give them and figure out what makes it tick and love every minute of it. Now do you really think any training course can commoditise that skillset and that mindset? Hell no! That comes with instinct and experience - something no course alone can provide.

3) Nature vs. Nuture

Thirdly, apart from the fact this is a skill you cannot train, it is one that must be nutured. Most in-house security teams won't pay their staff to continually upskill themselves, stay abreast of the latest technology and trends. They really get to go on courses, let alone conferences. Don't even get me started on building test labs or pay for software licensing. Most of these guys might do it on their own time but they won't remain inhouse security when just about any consultancy team will grab these guys, pay them a lot more money to do what they love to do, pay for all their training, toys, etc AND do it all on company time.  And given that in-house security teams cannot retain good penetration testers for long, surely it makes sense to go to a consultancy that you rely upon as a trusted partner? That makes perfect sense to me.

4) Methodology Matters

Fourthly, the methodology that a lot of these consultancies use is to make penetration tests a measurable, repeatable process. That's not to say all penetration tests, or the testers, are equal, but the intention is to come up with a proven approach to ensuring that each test will yield a high quality result as possible. In that respect, I actually think this is a good approach. Not just from ensuring the quality of the testing but even from a viewpoint of training up your new staff (who may be very technically proficient btw). Now the methodology is not the "be all and end all" but the fact that it can provide a common framework designed specifically to yielding results based on a proven approach cannot be understated.

5) Knowing The Path vs. Walking The Path

I'm paraphrasing Morpheus here, but this is the final distinction between 'good' and 'great'. A decent penetration tester may be able to follow their methodology and generate consistent results and findings for their client. However, a great pentester is one that is able to look at their methodology, their current findings and say "I know the methodology says this, but I am going to go off the beaten path and explore this...". They are operating on either a hunch (which I covered above), evidence, whatever -  there are somethings that simply only experience can yield. And this is the distinction I am trying to make. No course or certification alone is ever going to provide this edge. Ever.

6) Increased Awareness

Raising the profile of penetration testing is good for the profession. It is good because IT shops begin to realise the value of a good pentester. Consultancies develop a clearer picture of what different roles exist - and that a penetration tester becomes a clearly defined role, rather than an implicit skillset that every security professional is assumed to have.

Now I am not saying the issues raised by Drazen aren't that bad - they are. But he did neglect to touch on these points I've raised and I think its remiss to not mention these benefits.

There are always going to be consultancies that try to sell vulnerability assessments as 'cheap penetration tests', or tools like nmap and use open ports as 'critical security defects'. There are going to be people who rely on certification as evidence of a great penetration tester. There are going to be consultancies or individuals that just deliver terrible quality penetration tests and useless reports. C'est la vie.

However, that's why it is up to security professionals to cast a discerning eye over these potential partners. It's up to us to provide that insight and find the value.

In my last role I worked with a few great security professionals and we had to evaluate many consultancies and their penetration testing offering (so it was great, I got to see a stack of our competition in the field now! LOL). Here's some of  the tips my teammates and I used to help flush out the good guys:

FIVE WAYS TO FIND A GREAT PENETRATION TESTER:

1. Ask for a copy of a sanitised pentest report. 
What are the findings? Do they discuss serious application faults? Are they able to articulate how this vulnerability can lead to a business impact (and I mean a business impact, not a technical impact)? You'd be amazed how rare this ability is. Poor ones often just do a bad job alround of explaining what a technical risk actually means.


2. Ask what conferences they go to or what professional associations they're apart of.
Do they share their information? Do they present? Do they mingle with other security professionals and swap war stories from the trenches? Some pentesters keep this information very close to their chest and don't like sharing. While this doesn't make them any less technically capable, I question the wisdom behind this approach. I believe security professionals should be able to educate the wider community as well as our own.


3. Ask what research are they doing.
What projects are they working on. Are their tools they are developing? Are they doing vulnerability research? Perhaps its a presentation they are working on? Research represents an interest in their field over and beyond what a cert monkey is capable of. Additionally, some security consultants that perform pentests are often people who are heavily utilised on other projects, in which case penetration testing might be just one of many service offerings. This means at best, they might be good but they definitely won't fall in the "great" category.


4. Ask about their methodology
What does it align with? What doesn't it? If it doesn't align - ask why? Sometimes a company will borrow heavily from a standard but choose to modify it or disregard it based on their own experience. Some of them will actually have their own and it will borrow heavily from several standards. Any group that is consistently refining their metholody and can cite their influences gets big brownie points in my book.

5. Ask what their speciality is?
This is a trick question really. Don't trust anyone who says EVERYTHING. A consultancy may have capability in all areas but either their collective skills will fall into a particular arena, or their individual testers will have unique specialisations. So ask the testers themselves and listen to what they say.


- J.