You Scored 46: when SOC metrics stop meaning anything
Security reporting is very good at producing numbers. Whether those numbers tell you the service is working is a different question, and it is the one Dave's BSides London 2025 talk sets out to ask.
A talk examining SOC and MSSP measurement and the problem with security reporting that produces numbers without necessarily showing whether the service is effective.
WATCH THE ORIGINAL (opens on an external site)You Scored 46 is a talk about SOC and MSSP measurement. It looks at the reporting most buyers receive, and at the gap between a report that is full of numbers and a service that is demonstrably effective.
The argument did not stop at the conference. A later conversation on Cyber All Day with Michael Virgone picks the same thread up and develops it further.
What it gets into
- How SOC and MSSP performance is commonly measured, and what those measurements are actually describing.
- Why a report can be full of numbers without showing whether the service is effective.
- The difference between events, alerts, and incidents, and why treating them as interchangeable distorts reporting.
- Goodhart's Law: what happens to a measure once it becomes the target.
- MSSP economics, and how commercial pressure shapes what gets reported.
If you buy, run, or report on a managed detection service, the useful question the talk leaves you with is a simple one: what would this reporting have to show for you to conclude the service is not working?
MSSP SOC reporting has a metrics problem #03
A later conversation that develops the ideas explored in You Scored 46, covering SOC and MSSP metrics, Goodhart's Law, MSSP economics, service reporting, and the distinction between events, alerts, and incidents.
GO TO SOURCE (opens on an external site)
Want to talk it through?
If any of this is live in your business right now, a short conversation is usually the fastest way to work out what to do next.