Monday, December 20, 2010

My $0.02 worth on outsourcing security

I have had some discussions with people surrounding this topic lately and I want to highlight a few points on this very tender topic.

Firstly, I am not in favor of offshoring security testing. It needs to be said. I acknowledge it is happening, I acknowledge there is an economic benefit in doing so, I understand why this is happening - but it isn't something I'm happy with. Then again I'm not happy with a lot of things but we have to make do with what we've got sometimes. I also think a lot of people rage about it without a clear grasp of what is "really" the problem. So I want to put out a number of ideas out there what I think people are sensitive about and with that in mind, try and put it into perspective. Moreover, I want to stress in economic terms why this happens and is "normal" (in an economic sense of the word).

I've included some common economic principles below that I have personally found really interesting in studying this for my MBA. Contrary to popular belief, economics is not just about business. At its core, it is about how people make decisions and evaluate choices. Everyone should study a bit of economics IMHO. Even if you have no interest in business, it can help you with your social engineering efforts. :) Anyway, this segues into my next point.

Economists discuss that when people make a decision (such as purchasing another unit of a given good) they evaluate what is the benefit in doing so. Some companies have already made the decision to move a good number of critical functions offshore gives them a much greater benefit. They can put the cash saved towards purchasing more security consulting, buying more widgets, giving execs larger bonuses - whatever. The point is the decision has been made that they perceive there to be a greater "gain" in this move.

Thirdly, the whole "outsourcing" security is precisely why security consultancies are built. Businesses outsource their function to specialists every day. It makes sense. Anyone that is even slightly familiar with economic theory will understand that it makes sense to employ specialists where needed when a specific skill competency is outside of your core business. So the whole idea that "outsourcing = bad" must be utterly dispelled. It isn't "bad". To say it is bad is a strawman argument. When I was hiring pentesters in my last job, its because I am snowed under, I wasn't getting time to develop my own skills and yet expected to attend 5+ hrs of meetings/day. It made perfect sense for me to hire guys who stayed ontop of their game and were paid to keep their skills sharp, to say nothing of obtaining objective, independent testing. So lets get that out of the way. It happens every day, what we're talking about are these services either moving offshore or to more unskilled labour - that is what gets most security consultant's goats up. If you get a plumber in to fix your pipes, that is "outsourcing". Service your car? Outsourced. We do this every day, only we don't perceive it that way. Ever buy a car? Probably imported. I can go on but the point is that "outsourcing" is another word for "trade" (see Principle #5 below).

Fourthly. the outsourcing argument assumes that Australia has a monopoly on technical acumen - and that's simply not true. We are a very small player on a global stage. To be fair, I also think a good number of our homegrown talent is largely uncredited (but that's another issue). But I'll pick on India since its a hotspot for outsourcing and there's a lot of good discussions and research on this topic in relation to India (so it suits for reference purposes).

For starters, India's IT schools are regarded as an asset of national critical importance. They also have a much larger population, an incredibly larger proportion of people going through university (1.1 billion people estimated, 7% post secondary education vs our 22 million population and 34% post secondary education). Right away, they have three times our national population with a tertiary education. Granted, there are cultural issues which suggest that many graduates are not well suited for many jobs (cultural attitude to rote learning apparently) but if you consider the pool of potential graduates coming through, statistically they have enough people they can (almost literally) throw at the wall and see what sticks! Even if there is evidence to suggest they aren't all appropriate, the fact remains that even if you look at the guys with talent, there is almost a statistical certainty that they will have more technically capable guys than we do in Australia. It's arrogant presumption to presume we're so awesome that they can't compete in the same space as us. One of the beautiful things about technical skill is that it is not constrained by country, economic wealth or privilege and does not care about culture barriers. This is what is great about tech and what brings techies together around the world. Likewise, its the Achilles heel which ensures that the industry will always trend to being outsourced.

Bringing the point home, its not difficult to assume that if I can keep a close eye on how I outsourced my pentest and manage assurance with virtual team then certainly other companies can't with outsourcing.  That's not to say crap work can be delivered by unskilled labour but lets be frank, we've seen that with local assets too, right? :D I see that as a management problem, not a delivery problem. Don't believe me? Ask any tradie who has to oversee first year apprentices fresh out of high school.

Finally, I.T. security has been largely a growth industry and despite hearing it is almost a recession proof industry for years, its interesting that only now we're being hit with the outsourcing. Having grown up in I.T, I've seen this so many outsourcing gigs hit jobs of either friends or family that I've almost become inured to it. I know people are thinking "OMFG pentesting gone to India wtfbbq" and its like yes, welcome to the 21st century. We are all expendible. Now as mentioned, I am opposed to outsourcing this function because I think sensitive information gathered from pentesting needs to be kept close to home with strict governance around it along with stringent quality assurance. However, we all know about the insider threat so again - its a strawman argument to assume it can't happen here. But the reality is we are not unique and beautiful snowflakes. People will outsource to companies if they can do it cheaper - even if there is a drop in quality. If the quality drop is still within an acceptable level, then meh - they'll wear it. I'm not saying it is right, but this is what Mankiw calls "the margin" and its here to stay (always has been really). I was actually going to graph this point but I'm tired and can't be arsed. If you're actually interested, go study 'price equilibrium' or 'equilibrium theory' (the two aren't the same but they are related) and you'll see what I'm talking about.

Ultimately, the problems that arise from outsourcing are largely the same problems our clients today face when decide to engage us as consultants. I share the same trepidation as everyone else but I don't think the problem is as concerning as everyone makes out. We just need to be very clear on what those problems are and ensure that we impose suitable checks to offset those risks and have some sound advice for clients looking to move down this path.

If you're a company competing against outsourcing, my recommendation is to study Mankiw (see below) and have a think about how you could apply these principles in effect to your work. I have many ideas on how I'd compete against anyone simply offering a lower daily rate against my business. Some of them aren't entirely fair either. ;-) In any case, there are somethings I won't post because we're competing against this crap too. :) One strategy I will offer is that I am a firm believer in uplifting our capability and focusing on delivering business consulting and security strategy (see #4 below and my previous rant).


- J.

N. Gregory Mankiw - 10 Principles of Economics.

#1 People face tradeoffs
#2 The cost of something is what you give up to get it
#3 Rational people think at the margin
#4 People respond to incentives
#5 Trade can make everyone better off
#6 Markets are usually a good way to organize economic activity
#7 Governments can sometimes improve market outcomes
#8 A country’s standard of living depends on its ability to produce goods and services
#9 Prices rise when the government prints too much money
#10 Society faces a short-run tradeoff between inflation and unemployment

PS: I recommend "Economists Do It With Models"  as a great resource for understanding economic concepts in bite sized chunks. Jodi Beggs is actually a former student of Mankiw and better able to conveying information than the textbook I have to use for my unit. :-(   Yes, the link is worksafe btw, don't be put off by the name.

PPS: I had a whole rant where I was actually going to discuss price elasticity of demand with regards to security services but I realised that its not relevant to this discussion and decided to drop it, but I'd be keen to hear from other security folk who are actually interested in this stuff on what they perceive as the elasticity of security services to be. I'm of the very its actually quite elastic based on my own experiences but I realise that flies in the face of a lot of evidence to the contrary. Then again, I suspect I might be an anomaly in this area...

Sunday, December 19, 2010

Pentesting isn't enough - "Part 3"

My friend Serg over @ recently made these two posts about pentesting and I must admit it gladdens me immensely to see posts like this. I won't divulge too much about Serg in the interests of maintaining his privacy but I will say he is an experienced penetration tester and security consultant and in short, knows what he's talking about. Before I begin this post, you really need to read what he's written first.

I think I'd learned this lesson several years ago in my last job - and I know he has too but I think for Serg had the points hammered home to him over the past two years in particular on the basis of some of the clients and types of engagements he was working with. It's one thing to know something but another entirely to have the point really hammered home to you or see it demonstrated time and time again.

One of the things I am convinced you become aware of the longer you work in technical security space, the more you realise that the governance and management aspects play a critical factor in ensuring the success of a security program. Having said that, there's a world of difference between wanting to be a security manager and recognising the need for security management.

I have no desire to be a manager myself but I've seen first hand why non-technical skills are absolutely critical to ensuring the success and I'll give you an example:

I was a security lead on a very large project (large iconic brand shall we say?) and the one project was going to bulldoze ahead to go live without a critical security feature which was absolutely necessary to ensuring success. Without it, I am talking front page news headlines, loss of consumer trust, irrepairable brand damage, basically a post-apocalyptic security nightmare.  At this point in my career I had come mostly from a tech background and I had flipped out that this was going on. I realised that the Project Managers, Business Analysts, Programmers, etc. -- NO-ONE cared about security.
I couldn't understand why.

I had to learn (this job was a real baptism by fire let me tell you) how to convince people. I had to relate these risks back into business terms that they could relate to and why they needed to.

Pretty soon, nobody saw the problem I pointed out as a "security problem" it became "a problem" that each project member saw in their own way. The Marketing head saw it as being "detrimental to the end user experience". The Governance board envisioned "irrepairable brand damage". The Head of IT imagined his entire online presence having to be rebuilt and his Application Support team working overtime for the next six months. I eventually got funding to do the work that was needed and divert resources to develop the necessary controls to prevent this from happening.

Now if you are just a pentester and all you know is how to break stuff, then I suspect you would find it highly challenging having the above conversations. Infact, I know plenty that have no interest in having those conversations either - and that's fine. It takes all sorts of people to make this world go around and it would be horribly boring if we were all the same.

But to be a consultant you do need to be able to have these discussions with a variety of people. You have to understand what motivates people and speak to that motivation. Tony Robbins is often quoted as saying that people will do more to avoid pain than to gain pleasure - and this is certainly true in a business context.

Serg's point is that too many people lack these skills and it is these skills are most important. With the increase of technical jobs going overseas, the critical differentiator becomes the understanding of the client's business and how security fits into that. This is the lifecycle he's referring to. These are the sorts of questions such a consultant might find himself asking:

  • How does penetrationt testing fit into your application development process?
  • How much of that testing is handled by other test teams before a security specialists does?
  • Are you doing code analysis, threat modelling, architecture reviews prior to development?
  • Post launch, who is monitoring your applications, infrastructure and your data and how?
By no means is this list comprehensive but demonstrates the point.Technical skills are important. But to work in today's world as a consultant, you need more than that. I've worked with some brilliant technical minds (and I mean brilliant, not just skilled) but could also understand clients and their businesses. They can pentest anything you put infront of them (and I mean anything). It is certainly possible to have a good mix of both. The question is whether you want that mix of skills? If you want to remain a consultant in today's market, you have to have a service differentiator.

Now if you're relying on technical excellence as the only factor, you may be in for a very rough time. The basic law of economics dictates that people seek alternatives. If they can get the same service from someone overseas for a fraction of the price for equal skill (or even lesser skill) then a lot of people will make the trade-off if they perceive it to be of value. I'm not saying this is right or wrong, but that's the way it is. There are ways of dealing with this but we - as an industry - cannot continue to do what we always used to, never change and expect things to function as they used to. Times are changing and so too must we.

I guess the crux of the post is this -

It's one set of skills to be a hacker. It's another set of skills entirely to be a security consultant. Being one doesn't automatically make you the other. Serious hackers laugh at guys like me with their CISSP. That's fine. I laugh at guys like them who wouldn't last a week in a corporate enterprise environment dealing with the baggage I've had to. Like I said before, it takes all sorts to make this world go 'round.

I read Serg's post is a wakeup call to the pentesting community. If all you know is how to pentest, then times are looking grim. If you cannot properly engage the business, understand their language and communicate in their terms, think in abstract terms of principles and concepts and not just lines of code, then you better start learning or otherwise re-evaluate your day job. Your job is going to go to India or the nearest CompSci grad with a CEH who will undercut your daily rate.

If you don't believe me, then you better open your eyes. It's already happening.

- J.

Friday, December 3, 2010

Wikileaks getting shafted

I remember seeing Julian Assange speaking at one of the Ruxmon meetings earlier this year and not long after, I got into a discussion with another infosec consultant who was in attendance. We were discussing whether we thought Wikileaks should have posted the Collateral Murder video. His view was that, while disclosures are necessary, he wondered if this was really a case of "fog of war" and whether the video was more an indication of an unfortunate, grizzly accident rather than a grotesque abuse of power (force of arms).

(Now, to be fair, Assange went to great lengths to explain the rules of engagement as officially passed down to military personnel to highlight the fact they had clearly violated said rules to carry out the attack.)

I was polarised at the time. I used to be a far left hippy in my youth but I think overtime I'd become more bipartisan in my thinking. I could see both sides and I have a tendency to play devil's advocate. I think I said something about "yeah but we need groups like Wikileaks, to keep governments accountable." Say what you will of the Collateral Murder video but you have to admit, if the purpose of government is to serve its people then the idea that it should also be answerable to the people, isn't a stretch (unless you support dictatorships or other forms of non-democratic government, in which case I think we're on different views and you should stop reading).

The point I'm making was I really didn't feel strongly enough one way or another, but notionally, I supported the organisation.

Then someone else this week suggested creating a new root DNS server, in opposition to ICANN's management. My initial thought was "Ok great, someone wants to setup a new root but how do you secure it and prevent flagrant abuse?" This is a governance problem, not a technical problem - hence my tweet on the subject.

Not long thereafter, Wikileaks got DDOSed. Then the site was dropped from Amazon. Then their DNS provider dropped them.

At first I was shocked. Then I was scared (yes, scared that this could happen as naive as that may sound).

And then I got angry.

One of the roles I used to work in was Network Abuse, where we used to deal with investigations ranging from professional spamhaus gangs, to child pornographers, to kiddies dealing trojans to steal Diablo II accounts, you name it. We used to deal with other ISPs to collaboratively take down sites or offenders of clearly malicious intent. It was like a code amoungst ISPs - even if there is no direct law for some of the things we did, we did it because it made sense to work together as a global community (of course emailing non-English speaking countries was always a challenge but I digress).

You could argue we took the law in our hands, but we acted when the evidence was overwhelming that these people were malicious. E.g. evidence of spam traceable to certain IP blocks. Abused credit card numbers. URLs of sites allegedly hosting kiddie porn, etc.

Today I just saw a bunch of companies give strawman excuses to drop Wikileaks like a hot potato, for reasons I can only attribute to political pressure or unsavory conditions. I get DDOS as a weapon of hacktivism - I understand the motive. But these companies wiggled out of the arrangement for very dubious reasons. The US Government which claims to protect individual freedoms and rights is using every means at its disposal to capture Assange and arrest him. Swedish authorities have acted against their own law, with largely uncredible testimony.

The bottom line is this - even if you didn't support Wikileaks before, the actions of all these various groups is actively working against them, as it will polarise various groups that would otherwise have remained enemies. I'm only one random dude with an Internet connection and a pedestal, but I can find myself in agreement with so many people I wouldn't have yesterday, how would some other folks out there feel, particularly those with more spare time, motivation and technical savvy? I can only imagine.

In Australia we've never had to fight for our independence or freedoms. We have no Bill of Rights. Subsequently most Australians really are largely apathetic to the notion of free speech. However, if you've lived abroad or travelled to communist or non-democratic countries, you begin to realise just how valuable it is. We may have no such war but I cherish it.

Today made me think that I am far, far more left wing than I ever thought I was and it sent a shock through me. I wound up making a donation to Wikileaks. Nothing large but enough to at least send a token of support.

I support Wikileaks because there may come a time when I need a voice for something I cannot say myself. I support Wikileaks for my family, friends and other people on this planet who may find themselves one day in a god awful place where they need a voice and Wikileaks is the only one who can provide it. There are some real scumbags on this planet who should be punished but they hold the balance of power. Wikileaks has real power to make those people answerable to a higher power. Without groups like them, the bad guys can and will win.

Even if you haven't always agreed with their actions to date, ask yourself:
  • Do you believe that businesses and governments alike should be answerable to the people?
  • Do you believe that no-one is above justice?

If you answer yes to either of the above, I also encourage you to make a donation to Wikileaks and support the cause.

- J.

Thursday, December 2, 2010

Problem Solving

I recently read a presentation by Ivan Ristic (ModSecurity fame). You can find it here. It struck a chord with me and I wanted to share this gem.

In infosec we suffer either one of two conditions (generally) - either we suffer from tunnel vision, focusing on minutae that are rarely of any real relevance to the task at hand, or we attempt to "boil the ocean" in an attempt to solve unsolvable problems.

Ivan's post basically made a very practical recommendation - don't boil the ocean, focus on one problem at a time.

We are all confronted with a variety of problems, either at work or at home and it becomes very, very easy to start lumping isolated problems together, creating a snowball effect. Sometimes they are related but sometimes when you have an emotional investment in something you can't see that the forest is made of individual trees. :)

I am meandering a bit (I tend to if you haven't figured it out by now - sorry) but my point is that while its important to look at the big picture, make no mistake we can only solve one problem at a time.

Extrapolating this to infosec, each of us see something that we as individuals can "fix". I want to put it out there that there are problems we can fix, we just need to pick them off. I've got my own bugbears I'm thinking about and if I have to pick one I want to focus on, it would probably be patch management on an enterprise scale. I'm not sure I can "fix" this but I am pretty sure I have some pretty practical suggestions that would go a long way towards doing so. I'm still contemplating how/where/when I'll go abouts working on a presentation on it.

Anyway, I'm sure you each have one thing you want to fix or more importantly, can fix. I want to emplore anyone reading this to do it.

We may never solve the "security problem" but we can all certainly play a part in trying to fix it. I'd rather be part of the solution than part of the problem.

- J.