There’s a fantastic series of essays by Cedric Chin about becoming a data driven business. It starts off on a really inspiring essay dissecting Goodhart’s law. Most of the essays are focused around Amazon and often reference the book that Cedric had helped with: “Working Backwards” by Colin Bryar and Bill Carr.

For those that don’t know it, Goodhart’s law is:

When a measure becomes a target, it ceases to be a good measure.

Every measure is a proxy for what you actually want — and is always an imperfect proxy for it. Even when you’re using a tape measure you’re not quite getting what you want - an exact measurement of a distance. Instead, you’re getting something that’s “close enough” in precision to the real deal that it works fine.

In another essay, Cedric summarized a paper by David Manheim and Scott Garrabrant that breaks down Goodhart’s law into four “flavors”: regressive, extremal, causal, and adversarial. This is reflected in Donald Wheeler’s take on Goodhart’s law from Understanding Variation:

When people are pressured to meet a target value there are three ways they can proceed:

1. They can work to improve the system
2. They can distort the system
3. Or they can distort the data

To me, this is a fantastic way to frame Goodhart’s law. Instead of it being a problem declared out into the ether it’s something that can be solutioned against. The solutions would be to make it more difficult to distort the data or the system - or you can make it easier to improve the system.

Wheeler notes that this can be seen between the Voice of the Customer - the expectations - and the Voice of the Process - what can be done in the real world. Focusing entirely on the Voice of the Customer without understanding the Voice of the Process leads to gaming the measurements, inadvertently or not.

The essay series goes into depth on how Amazon has approached this problem - via their Weekly Business Review (WBR). The goal of the WBR is to provide flexibility to make the various systems work for them and also to drop measurements that aren’t useful. It promotes iteration and learning, synchronizing leadership rather than promoting blindly following numbers.

Proxy measures will always allow for some form of gaming the system. At the same time, you can’t run a business without these kinds of proxy measures. Goodhart’s law isn’t wrong - but instead of approaching it by choosing to avoid measuring, we instead need to find solutions to prevent it.

The problem isn’t having targets - it’s failing to reconcile those targets with the real world.