A primer on the objects of our measurement
By Richard Seiersen: Risk Management Author, Serial CISO, CEO Soluble
If someone asks how to measure “damage to reputation” or “threat” or “business disruption,” we simply ask, “What do you mean, exactly?”
- How To Measure Anything In Cybersecurity Risk
As a self proclaimed “security measurement provocateur” I’m on the lookout for the next thing to explore. And risk surface has been that thing for the past several months. It’s a new distinction meant to draw our attention to an emerging risk. And if you don’t already know what risk surface is then consider perusing the The Internet Risk Surface Report and or The Cloud Risk Surface Report. They are full of data, graphs and commentary based on RiskRecon’s third party data lake.
In this post I will explore the intersection of risk surface and metrics. Metrics feature strongly in my soon to be released book and in our first tome. This first in a series of articles will introduce you to approaches coming from those books applied to the concept of risk surface. But first, a bit of metrics perspective. This will be relevant to all of the forthcoming metrics articles.
Metrics Measure Capabilities - Not Technology
Metrics measure capabilities. Metrics ask, “are our security capabilities meeting the goals we set for them?” I expect some people don’t think about metrics this way, i.e from a capability and goal centric perspective. That is ok. Just see this as one of several points of view you can adopt as needed. Our first step then is defining capability.
A capability is a set of people, process and technologies. And capabilities are grouped to form a system. Security professionals, particularly leaders, are capability designers. For example, they design and then deploy vulnerability management, threat intelligence, incident response and other capabilities. It’s more than just technology - it’s teams and their processes too.
And the CISO is the system designer. They bring the aforementioned capabilities together to reduce the likelihood and impact of security risk. Meaning, the CISO is ultimately accountable to the outcomes produced by the overall security system and its capabilities. We know the CISO is accountable because of what happens to the CISO when the system they designed fails.
But how do you know the capabilities you have deployed are working? More importantly, how do you know they are improving, scaling and optimizing? Could they be “descaling” i.e. getting less effective? Metrics answer these questions. And the first steps in designing a good metric is getting clear about the capabilities outcome. Formally, we call this the object of measurement.
The Object Of Measurement
If you haven’t heard of the concept, object and methods of measurement we dedicate a whole chapter to it in our book “How To Measure Anything In Cybersecurity Risk.” My co-author, Doug Hubbard, simply calls it “The COM Model.” And if you were to retain anything from our book, or its predecessors, the COM model is it! For the remainder of this post we are going to focus on the “O” in the COM Model - the object of measurement.
Having the wrong object of measurement can lead to devastating outcomes. A whole industry can actually be misled by having the wrong objects. A good case and point was illustrated in Moneyball.
For those of you who didn’t see the movie or read the book, pro baseball had the RBI as its chief metric. In Moneyball they realized that the RBI actually didn’t correlated with “winning.” It did correlate with expensive baseball players. Conversely, the metric that had the highest correlation to winning (winning is the true object of measurement) was getting on first base. And players that were good at getting on first base were cheap. The Oakland Athletics had both “cheap” and “winning” as outcomes. Cheap was in scope because they had the smallest budget in Major League Baseball (MLB) and winning because who doesn’t want to win!
Hundreds of millions of dollars were likely blown over decades in MLB using the wrong metric in pursuit of the wrong object of measurement! But now that we know the moral of the story, let’s reverse engineer our way into a general outcome statement for the Oakland A's:
Minimizing the cost of players while maximizing the likelihood of getting to (and winning) the world series.
The prior “wrong” outcome statement might have been something like:
Spend as much as needed to score more without breaking the bank.
The Oakland A’s metrics had to make sense to winning and extreme cost containment. It ended up being an optimization problem for them that led to profound results. Ask yourself, are you clear on your objects of measurement for your capabilities and your system?
Finding Risk Surface’s Object Of Measurement
What is winning for risk surface? Here is as pithy as I can get at the moment. And I am curious to hear more about what you come up with:
Keeping the likelihood and impact of third party driven breach within risk tolerance given possible third party gains.
You may say, “that is impossible to measure!” Remember, I am bias. The title of our book is “How To Measure Anything...” And yes, if you conflate measurement with exacting precision then it is impossible to measure this and anything else of practical interest in security.
So, just stick with me on this. Imagine a third party risk chart that says you have a 20% chance of losing $10M or more a year and a 5% chance of losing $30M or more and a 1% chance of losing $40M or more on average. The chart shows a lot of variance (uncertainty) this is just an average.
Now, what if you simultaneously forecast a 25% chance of gaining $50M or more and a 5% chance of gaining $150M and a 1% chance of $250 or more in revenue tied to third party use? Given those two potential, yet uncertain outcomes, would you accept the risk of loss for those potential gains?
This is of course hypothetical, but it is how one might go about making risk based decisions. And it would be just one of a variety of things you might use as cognitive prosthesis while making forecasts under uncertainty. And if your are uncomfortable with uncertainty there really is only one practical solution. More Data! And in this case, one of the main factors that reduces our uncertainty is information regarding the state of control of our third party managed assets. This is of course RiskRecon’s speciality.
In the next paper I am going to help you to start building a basic model for measuring your third party risk management capabilities. It should be fun...at least for you risk and security metrics nerds out there!
