Blog

Notes on decision-making, uncertainty, and AI evaluation.

Posts related to the technical reports behind my projects, written to make the methods, assumptions, and measurement questions easier to follow.

Featured series

Evaluating AI Decision-Makers Under Uncertainty

A three-part introduction to why decision quality needs measurement, what such a measurement has to provide, and how to tell whether the measurement is meaningful for the agent in front of you.

Part 1 Why measure how an AI agent decides? Why intervention, comparison, and monitoring all require a real measurement of decision quality. Part 2 What a measurement of decision quality has to look like The four requirements a useful measurement has to satisfy. Part 3 Is the measurement telling you anything? Why adequacy checks are needed before a score becomes evidence.

All Posts

Part 1 - Why measure how an AI agent decides?

If you've put an AI agent in a decision-making seat, you need a way to tell whether it's actually deciding well.

Part 2 - What a measurement of decision quality has to look like

Four requirements for measuring how well an AI agent decides under uncertainty.

Part 3 - Is the measurement telling you anything?

Why adequacy checks determine whether a decision-quality measurement is evidence or noise.