Book Review: Security Metrics: Replacing Fear, Uncertainty and Doubt

As promised here is my review of 'Security Metrics: Replacing Fear, Uncertainty and Doubt' by Andrew Jacquith.
--

Firstly, to put this book into perspective -
This book was written by someone who is an information security veteran with, by my estimation, over ten years experience. This individual is a student of multiple fields and disciplines has been involved in everything from penetration testing to programming, business analysis, project management, communications and consulting.

My impression was that at the author's core, he is an analyst - first and foremost (and this is reflected in the first sentence of Chapter 1). He takes great pride at being able to understand not just security but business as a whole and being able to convey relevant information to all people at any level and across all levels of business and verticals.

With that statement in mind, I will proceed with the review.

The book is split into eight chapters, with a summary at the end of each, covering concisely what the chapter covered.

To paraphrase what he wrote in fewer words (and here's my hoping I do not do him any injustice) he believes that risk management as it stands today is far too immature for the average security practictioner because it fails to adequately measure "true" risk. He talks about risk management as it stands today being a dead art. I think what he means to say was that "as it stands" it is, well, dead. We cannot improve what we cannot measure - let alone hope to understand. And since current risk management practises are well and truly inadequate in this regard, he says we cannot accurate assesss risk. This book is intended to come up ways and methods of devising metrics of your own.

These metrics can be used to better measure risk and this could be for something as simple as reporting to your inline manager or as a business driver to improve security across the organisation (which is the goal I would assume of any security practitioner). But the end game all about measurement.

• Foreword
• Preface
• Acknowledgments
• About the Author
• Chapter 1 Introduction: Escaping the Hamster Wheel of Pain
• Chapter 2 Defining Security Metrics
• Chapter 3 Diagnosing Problems and Measuring Technical Security
• Chapter 4 Measuring Program Effectiveness
• Chapter 5 Analysis Techniques
• Chapter 6 Visualization
• Chapter 7 Automating Metrics Calculations
• Chapter 8 Designing Security Scorecards
• Index

'Chapter 1: Introduction: Escaping the Hamster Wheel of Pain'
This chapter focuses on where organisations typically sit during this process. The book spells out how information security is typically managed (read: poorly) with minimal measurable, meaningul results. Vendor promises a product to solve everything. Client buys product. Client panicks at the information cleaned. Client proceeds to fix those problems. Hope problems go away (until new lot of problems or type of problems appear). He refers to this as the "Hamster Wheel of Pain" - aka "detect-fix-patch-pray". He explains that having meaningful metrics means that you never rely on vague notions of perceived risk or a vendor promises for enhanced security. In other words, enables you to escape the "Hamster Wheel of Pain".

'Chapter 2: Defining Security Metrics'
This was arguably one of my favorite chapters. Maybe it is the analyst style hats I've worn over the years but what I really took away from this chapter was the following:

• What makes a 'good' metric.
  
• What makes a 'bad' metric.
  
• How do you measure improvement over the years.

I silently got a chuckle over was the examples of 'bad' metrics and where I've seen them used, especially given I pointed out what terrible measures they were and this was long before I'd even read this book. It was this chapter that really struck me how much the author enjoys analytics as not only a science but an artform (the art component becomes more relevant in Chapter 6) and just how much our profession has yet to come. He pointed out that while ISO17799 (or 27002 today) provides a good framework for building a security program, it actually provides minimal guidance and zero metrics in which one can use to measure the success of a security program and is often relied upon as a poor man's crutch. He argues that more specific measures are required if you truly want to measure improvement of security. He also rails against the use of ALE as a woefully inadequate measurement - and one of my key gripes of its frequent application to information security. While I found myself at times challenged by Andrew's comments as I read on I found many of his opinions resonating very strongly with me. As a result, I slowly came around to his way of thinking. Particularly as his examples from real life scenarios became more frequent, truly driving the points home.

Chapter 3: Diagnosing Problems and Measuring Technical Security
This chapter focused heavily on how building a series of metrics can be used to diagnose problems within your environment. Andrew is complete in his coverage of the different categories of problems: perimeter and security threats, coverage and control, availability and reliability and finally, application security. What I enjoyed about this chapter was how he walked through the process of how you begin to analyse an environment, devise a hypothesis and then build your metrics around testing that hypothesis. In my mind, Chapter 2 and 3 read as if they were one chapter to me, it was so seamless.

(Admittedly the whole book did, but these two chapters in particular).

Chapter 4: Measuring Program Effectiveness
This chapter examines the COBIT, ITIL and ISO17799 frameworks and how they can be leveraged to help define metrics that can shed light as to how you can create meaningful measures of the degree of success in your application of these frameworks. Again, Andrew's examples show his application of how metrics can be created and applied to simple but powerful, well thought out questions.

Again, Andrew takes another swipe at ALE and asset valuation as a process. But what I liked was that while Andrew made clear his position he was able to present an opposing view - and at the same time make it no less valid than his own. It is rare in this day and age to find reference books that are capable of doing this. Most seem to present then own ideas or framework as gospel. Andrew is very conscious in his writing not to make this error.

Chapter 5: Analysis Techniques, focuses largely on the methods of how one can use the numbers gathered to gain insight. This chapter covers basic statistics and mathematics (but nothing too complicated) and how you can use means, averages, standard deviations, group and aggregation, as well as time series and cross sectional analysis, etc, across your data sets. He walks the reader through each of these methods, explaining how and where to apply them.

Chapter 6: Visualisation walks the reader through the different types of graphs and charting, explaining the common mistakes (which I grimaced as I read, knowing I was all too guilty of some of them) and providing guidance on how to construct models that are not only more meaningful but asthetically pleasing and easier to absorb.

(This actually reminded me to earmark 'Applied Security Visualisations' for future reading btw - which has also come to me as highly recommended).

In 'Chapter 7: Automating Metrics Calculations', Jacquith recovers old territory. Earlier in the book he makes specific reference to how metrics must be automated (not just for the person doing the reporting but to ensure accuracy and consistency in the results). This chapter focuses more on the benefits of automation as well as providing guidance on how and where some of the data we are after can be gathered to build and measure our newfound metrics. The previous chapters explained a lot of the theory behind building metrics. This chapter is how you put it together and provides ideas on how you can automate the process, providing suggestions of various technologies than can facilitate.


'Chapter 8: Designing Security Scorecards' focuses on building a scoring system that combines security metrics but relates them back to core business objectives. The key purpose here is to ensure the metrics are relevant to the business and can be absorbed by them. He explains that this is a framework he has used (remember: framework like ISO17799, ITIL, COBIT) and is very clear that there are limitations. I have to admit, despite the nice correlary back to the business, I did not like this format but that is a personal issue. I'm sure others will rave about it.

Earlier on, he made a good point about how metrics should be evenly applied across multiple businesses and verticals. I think that is what he was trying to was create - a framework that would enable us to do precisely that. Unfortunately the metrics you need to create to populate the card are so unique to a given environment, that the framework provided seems like a moot point to me.

At the end of the day however, I think the true purpose was to demonstrate how one would evaluate security in an organisation, define a series of metrics and use those metrics to report on the state of security back to the organisation. In other words, how to apply the lessons in this book from start to finish. In that respect, I think he succeeded.

If I have to pick one gripe (and I admit I am nitpicking here), it would be that the Summary at the end of each chapter is largely a condensed copy of the entire chapter, often using entire slabs of text. When he covers a large chunk of material in a given chapter (e.g. Chapter 6) you feel grateful, but for others (e.g. Chapter 8) it feels redundant. I would have preferred to see less redundant language in the summaries, perhaps keeping them to bullet points if he wanted to use the same text. But again, that is just a personal preference. I expect many reader's will truly appreciate the detailed summaries.

Overall, I think this book is mandatory reading for security managers, consultants (especially those who assess and audit Technical Security Controls), security architects, and even operations folk who are looking to explain/justify their existence (or find themselves in the position where they have to continually do so) or even just gain management support to their efforts.

Too many security people say they are "too busy" to deal with report writing and measuring. I think this book creates a compelling case why we need to start thinking about security metrics more - and start today.

Rating: 9.5/10

Mandatory Reading

Hi all

Occasionally you read a book that flips you on your head and gives you a unique spin on things. I've just found two books that fit the bill:

'Security Metrics' by Andrew Jaquith
'Beautiful Security' by Andy Oram, John Viega

I've almost finished Security Metrics and just started reading spot chapters from Beautiful Security. I fully intend on writing reviews here for both shortly but please don't wait for my review.

I strongly recommend you read these books. I am finding both fascinating reads and I can assure you that these will give you new insights and add real value to your work.

- J.

Sources on "Best Practise" security

I had a question on 'best practise information security' today by a family member and it then occured to me that unless you're in the know, the sources for "best practise" security aren't actually listed anywhere. It's all stored on a series of disparate sources with nothing to unify them.

If information is intended to be free, why isn't this stuff more widely communicated to the non-security folk out there? Anyone would think we're trying to keep this stuff to ourselves in an effort to prop up our industry.

For example, there is no wikipedia entry on "best practise information security".

So my question is - what sources do YOU consider to be definitive when it comes to information security "best practise"?

I'll get the ball rolling with a few:

• NIST,
• CIS,
• NSA.gov,
• Standards (ISO/AS/NZ),
• OWASP.
  

Anyone?

- J.

Good to Great

I read a post on 'Infamous Agenda' recently and it really got me thinking... security practitioners often so caught up on putting out spotfires that we never seem to get a chance to refine our internal processes and practises.

If you're engaged on multiple projects, your requirements should be well understood, documented and easily repeatable. If there is common functionality that keeps appearing on your applications then you should have design patterns available for guidance. Classification of information assets should be defined prior the asset being built, and so on. Common sense security stuff that's a no brainer in theory but often harder to get in place in practise.

How many places can _honestly_ say that their ship runs like this - regular as clockwork?

As I am about to leave my current role and move onto greener pastures, I have spent a lot of time reflecting. I've reflected on my career highlights and contributions - my big wins. I've also spent a lot of time thinking about what I could have done differently. What mistakes did I make? How would I approach the same problem again?

They say a wise man learns from not just his mistakes but the mistakes of others. I like to think that I am my own worst critic (at least I hope so!).

Lately I've been thinking about idealism in security (as opposed to the pragmatic posts of yesterday). If we are too pragmatic we risk becoming cynical. We start to slip and lose our edge. You run the risk of not keeping focus on the items that really matter and you're no longer the gatekeeper but a rubber stamp for the business. Our idealism is what inspires us to lead and to create, it ensures we are passionate about our jobs as protectors and advisors. Lose that and what value to we add?

The difference between a good security practitioner and a great one is one that contributes some of their time each week to enhancement.
Some ideas:

• Creating design patterns and security architecture.
• Defining meaningful security metrics for reporting.
• Clarifying how business units formally engage Security.
• Embedding security into application development and testing processes.
• Security training for architects and developers as well as general staff.

I'm not talking about the stuff that's compliance driven either. Just the stuff that we should do a little each day to make the business run a little better. The Japanese have a word "kaizen" which embodies this principle.

It's not all about chalking up quick wins on the board like pleasing business owners, but the consistency. When you build value into the business, one day at a time, and you get to see the fruits of your labour grow over time - that's where security really demonstrates its value and that is the sense of accomplishment I have always found in my work in security.

Good security is more than just plugging holes. It's about building in the right processes into an organisation than enable them to respond to evolving security threats so that even if there are gaps that the business can respond with a degree of speed and agility. Likewise, developing a better security program means devoting time to building those processes even when you think you can't. More importantly, being a better security practitioner means devoting time to ourselves to review past failures and triumphs and learning from them (ontop of the usual insane amount of additional study we do).

In hindsight, I see a lot of things I wish I handled differently. But when I see that a single communications piece not being able to go through Marketing without a review by the Fraud teams to ensure the company message maintains a consistent approach, when Usability teams working alongside Security to ensure key functionality works as intended but doesn't pose a security risk, or even if it is something as intangible as an increased level of security awareness by multiple business units where senior management are asking security-related questions even when I'm not there, I feel like I've left my mark on an organisation and left it a little better than when I started.

- J.