Book Review: Security Metrics: Replacing Fear, Uncertainty and Doubt

As promised here is my review of 'Security Metrics: Replacing Fear, Uncertainty and Doubt' by Andrew Jacquith.
--

Firstly, to put this book into perspective -
This book was written by someone who is an information security veteran with, by my estimation, over ten years experience. This individual is a student of multiple fields and disciplines has been involved in everything from penetration testing to programming, business analysis, project management, communications and consulting.

My impression was that at the author's core, he is an analyst - first and foremost (and this is reflected in the first sentence of Chapter 1). He takes great pride at being able to understand not just security but business as a whole and being able to convey relevant information to all people at any level and across all levels of business and verticals.

With that statement in mind, I will proceed with the review.

The book is split into eight chapters, with a summary at the end of each, covering concisely what the chapter covered.

To paraphrase what he wrote in fewer words (and here's my hoping I do not do him any injustice) he believes that risk management as it stands today is far too immature for the average security practictioner because it fails to adequately measure "true" risk. He talks about risk management as it stands today being a dead art. I think what he means to say was that "as it stands" it is, well, dead. We cannot improve what we cannot measure - let alone hope to understand. And since current risk management practises are well and truly inadequate in this regard, he says we cannot accurate assesss risk. This book is intended to come up ways and methods of devising metrics of your own.

These metrics can be used to better measure risk and this could be for something as simple as reporting to your inline manager or as a business driver to improve security across the organisation (which is the goal I would assume of any security practitioner). But the end game all about measurement.

• Foreword
• Preface
• Acknowledgments
• About the Author
• Chapter 1 Introduction: Escaping the Hamster Wheel of Pain
• Chapter 2 Defining Security Metrics
• Chapter 3 Diagnosing Problems and Measuring Technical Security
• Chapter 4 Measuring Program Effectiveness
• Chapter 5 Analysis Techniques
• Chapter 6 Visualization
• Chapter 7 Automating Metrics Calculations
• Chapter 8 Designing Security Scorecards
• Index

'Chapter 1: Introduction: Escaping the Hamster Wheel of Pain'
This chapter focuses on where organisations typically sit during this process. The book spells out how information security is typically managed (read: poorly) with minimal measurable, meaningul results. Vendor promises a product to solve everything. Client buys product. Client panicks at the information cleaned. Client proceeds to fix those problems. Hope problems go away (until new lot of problems or type of problems appear). He refers to this as the "Hamster Wheel of Pain" - aka "detect-fix-patch-pray". He explains that having meaningful metrics means that you never rely on vague notions of perceived risk or a vendor promises for enhanced security. In other words, enables you to escape the "Hamster Wheel of Pain".

'Chapter 2: Defining Security Metrics'
This was arguably one of my favorite chapters. Maybe it is the analyst style hats I've worn over the years but what I really took away from this chapter was the following:

• What makes a 'good' metric.
  
• What makes a 'bad' metric.
  
• How do you measure improvement over the years.

I silently got a chuckle over was the examples of 'bad' metrics and where I've seen them used, especially given I pointed out what terrible measures they were and this was long before I'd even read this book. It was this chapter that really struck me how much the author enjoys analytics as not only a science but an artform (the art component becomes more relevant in Chapter 6) and just how much our profession has yet to come. He pointed out that while ISO17799 (or 27002 today) provides a good framework for building a security program, it actually provides minimal guidance and zero metrics in which one can use to measure the success of a security program and is often relied upon as a poor man's crutch. He argues that more specific measures are required if you truly want to measure improvement of security. He also rails against the use of ALE as a woefully inadequate measurement - and one of my key gripes of its frequent application to information security. While I found myself at times challenged by Andrew's comments as I read on I found many of his opinions resonating very strongly with me. As a result, I slowly came around to his way of thinking. Particularly as his examples from real life scenarios became more frequent, truly driving the points home.

Chapter 3: Diagnosing Problems and Measuring Technical Security
This chapter focused heavily on how building a series of metrics can be used to diagnose problems within your environment. Andrew is complete in his coverage of the different categories of problems: perimeter and security threats, coverage and control, availability and reliability and finally, application security. What I enjoyed about this chapter was how he walked through the process of how you begin to analyse an environment, devise a hypothesis and then build your metrics around testing that hypothesis. In my mind, Chapter 2 and 3 read as if they were one chapter to me, it was so seamless.

(Admittedly the whole book did, but these two chapters in particular).

Chapter 4: Measuring Program Effectiveness
This chapter examines the COBIT, ITIL and ISO17799 frameworks and how they can be leveraged to help define metrics that can shed light as to how you can create meaningful measures of the degree of success in your application of these frameworks. Again, Andrew's examples show his application of how metrics can be created and applied to simple but powerful, well thought out questions.

Again, Andrew takes another swipe at ALE and asset valuation as a process. But what I liked was that while Andrew made clear his position he was able to present an opposing view - and at the same time make it no less valid than his own. It is rare in this day and age to find reference books that are capable of doing this. Most seem to present then own ideas or framework as gospel. Andrew is very conscious in his writing not to make this error.

Chapter 5: Analysis Techniques, focuses largely on the methods of how one can use the numbers gathered to gain insight. This chapter covers basic statistics and mathematics (but nothing too complicated) and how you can use means, averages, standard deviations, group and aggregation, as well as time series and cross sectional analysis, etc, across your data sets. He walks the reader through each of these methods, explaining how and where to apply them.

Chapter 6: Visualisation walks the reader through the different types of graphs and charting, explaining the common mistakes (which I grimaced as I read, knowing I was all too guilty of some of them) and providing guidance on how to construct models that are not only more meaningful but asthetically pleasing and easier to absorb.

(This actually reminded me to earmark 'Applied Security Visualisations' for future reading btw - which has also come to me as highly recommended).

In 'Chapter 7: Automating Metrics Calculations', Jacquith recovers old territory. Earlier in the book he makes specific reference to how metrics must be automated (not just for the person doing the reporting but to ensure accuracy and consistency in the results). This chapter focuses more on the benefits of automation as well as providing guidance on how and where some of the data we are after can be gathered to build and measure our newfound metrics. The previous chapters explained a lot of the theory behind building metrics. This chapter is how you put it together and provides ideas on how you can automate the process, providing suggestions of various technologies than can facilitate.


'Chapter 8: Designing Security Scorecards' focuses on building a scoring system that combines security metrics but relates them back to core business objectives. The key purpose here is to ensure the metrics are relevant to the business and can be absorbed by them. He explains that this is a framework he has used (remember: framework like ISO17799, ITIL, COBIT) and is very clear that there are limitations. I have to admit, despite the nice correlary back to the business, I did not like this format but that is a personal issue. I'm sure others will rave about it.

Earlier on, he made a good point about how metrics should be evenly applied across multiple businesses and verticals. I think that is what he was trying to was create - a framework that would enable us to do precisely that. Unfortunately the metrics you need to create to populate the card are so unique to a given environment, that the framework provided seems like a moot point to me.

At the end of the day however, I think the true purpose was to demonstrate how one would evaluate security in an organisation, define a series of metrics and use those metrics to report on the state of security back to the organisation. In other words, how to apply the lessons in this book from start to finish. In that respect, I think he succeeded.

If I have to pick one gripe (and I admit I am nitpicking here), it would be that the Summary at the end of each chapter is largely a condensed copy of the entire chapter, often using entire slabs of text. When he covers a large chunk of material in a given chapter (e.g. Chapter 6) you feel grateful, but for others (e.g. Chapter 8) it feels redundant. I would have preferred to see less redundant language in the summaries, perhaps keeping them to bullet points if he wanted to use the same text. But again, that is just a personal preference. I expect many reader's will truly appreciate the detailed summaries.

Overall, I think this book is mandatory reading for security managers, consultants (especially those who assess and audit Technical Security Controls), security architects, and even operations folk who are looking to explain/justify their existence (or find themselves in the position where they have to continually do so) or even just gain management support to their efforts.

Too many security people say they are "too busy" to deal with report writing and measuring. I think this book creates a compelling case why we need to start thinking about security metrics more - and start today.

Rating: 9.5/10