Saturday, January 22, 2011

What did we learn from the floods and fires?

Years ago, I was lucky enough to be invited to closed session at Deloitte where Adele Melek (Global Leader of Information & Technology Risk Service) talked about one of their annual global security reports.

One of the points in the report was about Business Contuinity Planning and Disaster Recovery. What was interesting was he made the very poignant observation that there was not a significant uptake of these consulting services when compared to other regions globally. I can't remember the exact numbers but he made a point that in late 2000/early 2001 a marginal percentage, lets say around 10% (I am throwing up arbitrary numbers which sound right from memory). Post 9/11, 12 months after, that figure skyrocketed to somewhere well above 90%. Watching those two towers go down and the number of businesses and lives affected, made a significant impact for businesses. He made a point - as have many others - it will take a 9/11 (or equivalent there of) for US before we start thinking of BCP/DR a little more.

Well three years since I attended that session we've seen not one but TWO MAJOR tragedies I would argue would be our equivalent - the Victorian Bushfires of 2009 (aka. Black Saturday) and the Queensland Floods.

Before anyone dismisses the impact of these in relation to 9/11, let me through up some numbers/stats:

Victorian Bushfires
  • Destroyed 2,030 houses, 3,500+ structures in total and damaged thousands more.
  • The following townships utterly destroyed: Kinglake, Marysville, Narbethong, Strathewen and Flowerdale.
  • The fires affected 78 individual townships in total and displaced an
  • Displaced an estimated 7,562 people.
  • Total death toll: 178
  • The Black Saturday bushfires were the 8th deadliest singular bushfire/wildfire event in recorded history.

Queensland Floods
  • At least 70 towns affected to date.
  • Over 200,000 people were affected.
  • Damage initially was estimated at around A$1 billion.
  • The estimate of lost revenue from Australia's GDP is about A$30 billion.
  • Many state that the factual losses cannot be calculated but can be readily counted in the billions of dollars.
  • (These numbers could increase as of this time of posting btw).
Even if your business or livelihood was not directly impacted by these events, chances are you had a business interruption as a result if you did business with anyone in these areas - or had friends and family who did.

These incidents teach us that the Availability in information security goes beyond having data recovery and backup. It's about an understanding of your true assets - your buildings, your equipment, phones, desks, software, hardware and of course, your people. Sound business continuity planning relies on thinking in terms of business processes and functions (not just IT infrastructure) and absolutely defining what is critical and what isn't. What can you afford to run at a reduced capacity? Who are the key staff in your business and what happens if you lose them? How is knowledge transferred to prevent loss of critical corporate intellectual property? This isn't just useful in a crisis either. A lot of it just good corporate governance.

Also, sound strategy here goes beyond thinking in terms of fires and floods. What is a real disaster for YOUR business? What are the likely risks you could face? Have you done a threat and risk assessment for your business? Few people could have anticipated the planes hitting the towers in 9/11. But some did predict the possibility of losing access to a building and developed strategies to ensure that the loss of their sites (and in some csaes their staff) to ensured they had a measure of redundancy continue operation - even if it was at a reduced capacity - in the case of such an event.

Real contigency plans can be constructed around a number of scenarios. For example:
  • Loss of buildings,
  • Loss of people,
  • Loss of critical services (gas, water, power, telecommunications).
These could be trigged by anything (fire, flood, even acts of violence such as shootings, etc).

While it is impossible to have a scenario for everything, developing even a single strategy is simply the beginning to a wider strategy. Having a process (even an imperfect one) means that you have something which can tested annually and improved it over time. This already puts you ahead of the pack.

I've personally worked with businesses that have fully accepted the risk of not having a DR or BCP strategy based on their estimation of likelihood. I think these incidents have well highlighted that such short sightedness can be damning.It doesn't have to be a fully redundant cutover. You just need to be realistic. Your BCP/DR strategy may only allow for limited or even reduced capacity based on cost constraints. But hey, something is better than nothing.

I don't want people to think that I am trying to milk these tragedies for chalking up posts ont his blog. My intention is the opposite infact. My point is that there is a greater tragedy here - that these incidents have occured and yet, we still don't seem to be learning.

- J.


K9Pup said...

Wow. If you brainstorm like this at 5.30 in the morning only the birds will know.

The reason so many fail to see the lighthouse is the fog masks the jagged rocks and it is not until we founder on the rocks that we realise the peril.

Most of the apathy originates from the perception of "It won't happen to me" attitude. The rest is based on cost vs benefit. The same reason there are those driving unregistered vehicles, uninsured vehicles, lack private medical insurance, don't invest in pre-paid funeral plans, have no health insurance and most definitely lack anything to do with BCP or DR.

On the fiscal side - funds are allocated on a needs by needs basis determined by the budget and or available funds.

BCP and DR need to be promoted in a similar light to the way College Education is promoted in the USA. Societal conditioning over the years now has parents squireling away funds to cover the costs of their children's education the day they are born. But for the most part, today's society is all about the 'here and now' rather than about 'tomorrow and the day after'.

In Australia we think ourselves lucky if our company has a 3 year plan of operation. In the USA it is closer to 5 or 10 years. In Japan they plan for 30.

I would have to agree that with the advent of low cost drives and USB, it is affordable for any business to at least have some kind of DR plan in place (even a weekly backup taken offsite).

Then we also have the entry of low cost cloud computing. The jury verdict on cloud computing though is still out.

PT is also a difficult task. A nightmare for any IT Manager. With the globalisation of business, the drive by the employee, CEO and business partners to offer real-time communication, facebook, twitter, android, rim and iphone, access not to mention wifi is enough to drive anyone insane.

In today's world, you have to apply to 80/20 rule set. Apply what safety measures one can and hope for the best. But what are the preventative measures? They are the known gaps in firewalls or bugs in software. And how far do you go? This is no different to protecting a home. You can lock the doors and windows in addition to installing a home alarm and putting a big nasty dog in your yard. But does all that prevent the smash and grab? Does it stop the professional? Does it prevent someone coming in through the roof? And what about your visitors - have you had them security checked?

The same applies to businesses. You can apply some rudimentary preventative measures, but never completely plug all the gaps.

For a business though it is a hard sell to make them see the benefit.

The best suggestion is not to term thing in the future tense. Rather, term them in the past tense. e.g. Yesterday your business burnt down, all your computer information is gone. What are you going to do today? When the CEO looks blankly at you, wait 3 seconds, then hand them a 1 Terabyte Hard Drive and say 'Here you go - this drive has all your information backed up and ready to re-install'.

Now you have their attention.

Jarrod said...

It's much easier to come up with a sound DR/BCP strategy for your own home than it is for an entire enterprise because there are orders of magnitude in difference.

With enterprises, if you lack the executive support, you're screwed. I've seen environments where people chose the USB route for highly sensitive data and it was not encrypted and not pretty. This was chosen because the business unit found the official enterprise-grade solution "too expensive." Often this is said because they don't understand the cost breakdown and the value of the service.

Holding together a business with USB storage and low cost disk can be a risky proposition. It really depends on the size of the business. If you don't have executive support for it on some level, you're building your strategy in a house of straw.

But you are dead on the money - it is a cultural problem and ultimately, it is possible to have some plan, even if it isn't perfect.

- J.

K9Pup said...

J - Totally agree with you.

The problem is more widespread though due to the very nature of technology and its acceptance all the way downstream to SMB's.

As more and more SMB's adopt softweare like MS Small Business Server 2008 and upcomming 2011, VAR's, Consultants and Support Services are finding it harder to lay even a small foundation towards BCP or RD, let alone educate SMB's on privacy concerns.

But even large Enterprises are even struggling to cope with the myriad of areas they now need to be concerned about in IT.

Early adoption of Technology had easily definied cost benefits that were clear and precise. Something the average CEO or CIO could fathom. e.g. That MS Word Program or (Showing age Wordperfect 5.1) can do a mail merge that would take my Typist pool xx days to do.

Today however, we seem to have crested a wave in Technology automation with an abundance of savings yet those savings are absorbed by new training, education, integration and additional staff. The cost savings vanish like a needle in a haystack.

Is it all information overload?

I wonder how many IT managers have any kind of IT policy to ensure total erasure of hard drives in laser printers, photocopiers and flushing memory from SSD's.

It can be quite fascinating to buy a second hand colour laser printer from VicRoads I can tell you from experience. Even more hairaising if it is from Centrelink or Dept of Health. And yes.. most of these now have some kind of policy but there are still instances of these items falling through the cracks.

You and I both jumped into IT back in the days of CCA, and I'm certain we've seen our fair share of good and bad BCP & DR. All those nice plans that went into the Southern Cross Pipe with its special redundancy went up in smoke after just a month. Granted, it could have been worse without planning, but needless to say, planning for every contigency is difficult and costly.

And if the big boys in the Enterprise can't manage to get a handle on it, what hope do the SMB's have. :(

As a consultant, one has a better sphear of influence but the size of the cake is always the same. You need to dice and slice the cake in proportions that merrit the best balance. All the while watching the jockeying of Hardware, Software and Staffing.

Then within each of those categories you have the subsets petitioning for their share. Severs, Desktops, PC's, Mobile Phones, Laptops, or with Software, its Enterprises Apps like Antivirus, Email, Spam control, DB's etc.

Oh, and lets not forget... BCP and DR.

Sadly, I do not see increased takeup or adoption happening any time soon.

To do so and do so effectively would require legislation and enforcement with penalties. :(

To my mind, the best sales blurb for BCP and DR would have to be NASA and the Apollo 13 Incident. That was 40 odd years ago and how far have we progressed?