essay
Closing the Loop on ML Evals
A practical framework for turning evaluation insights into deployable improvements.
Most eval work stops at measurement — a dashboard nobody acts on. The loop only closes when a regression has a named owner, a reproducible case, and a gate that blocks the deploy. That takes slices instead of averages, a suite small enough to run on every change, and treating eval failures as backlog items rather than trivia.
Reliable ML systems rarely fail in the obvious ways. They degrade silently, drift, or encounter input regimes nobody planned for. The model still returns something. The endpoint still returns 200. The graph still looks fine, because the graph is an average.
Most evaluation work I have seen gets the measurement half right and stops there. What comes out is a number that goes up and down, and no mechanism for turning a bad number into a code change. That is not a loop. That is a dashboard.
The problem with a single number
Aggregate accuracy is the metric everyone starts with and the one that hides the most. A system at 94% overall can be at 99% on the two thirds of traffic that is easy and 71% on the slice that actually generates revenue, and the aggregate will barely twitch when that slice degrades further.
Worse, aggregates are anti-diagnostic. When the number drops two points you learn that something got worse — which you already suspected — and nothing about where to look. You cannot assign a two-point drop to an engineer.
The fix is not a better metric. It is a different unit of measurement.
Slices, not averages
A slice is a subset of traffic you can name, care about, and defend to someone who does not work on the model. “Queries from enterprise accounts.” “Requests that hit the retrieval path.” “Conversations longer than ten turns.” “Anything mentioning a regulated product.”
Slices are useful in ways aggregates are not:
They localise failure. When one slice moves and eight do not, you know which code path to open. The diff between a healthy slice and a broken one is usually the bug.
They survive redesign. Your model will change, your prompts will change, your retrieval strategy will change. “Enterprise queries should work” outlives all of it. A slice is a statement about the product, not the implementation.
They force a conversation about priorities. Deciding which slices matter is a product decision that eval work makes explicit. Teams that skip it are implicitly deciding all traffic is equally important, which is never true.
The practical move is to define five to ten slices, weight them by how much you care, and report each separately. Never let them collapse back into one headline figure.
The loop
Measurement is the first third. The loop is:
- Define a real-world slice you care about, in language a non-engineer would recognise.
- Measure it on every change, with a probe set small enough that nobody is tempted to skip it.
- Convert regressions into work — a training fix, a routing rule, a retrieval change, a prompt change — with an owner and a ticket.
Step three is where almost everyone stalls, and the reason is structural rather than technical. A regression that arrives as a Slack message competes with everything else in Slack. A regression that arrives as a failing check on a pull request gets fixed.
Make the suite cheap enough to always run
The highest-leverage property of an eval suite is not coverage. It is that it runs on every change without anyone deciding to run it.
That constraint drives the design. A suite that takes forty minutes and a meaningful amount of inference spend gets run before releases, then before major releases, then never. A suite of twenty to fifty cases that finishes in a couple of minutes gets run constantly — and constant beats thorough, because you catch the regression in the change that caused it, while the author still has the context to fix it.
Start with twenty cases drawn from real traffic, weighted toward the slices you named. Then add a case every time something breaks in production. That habit matters more than the initial design: a suite grown from real failures converges on the shape of your actual problem far faster than one designed up front.
You do need judgement about what “passing” means. For anything generative, exact-match scoring is useless and a human reviewing every run does not scale. The middle ground is a scorer per property you care about — did it choose the right tool, is the citation real, is the reasoning coherent, did it refuse when it should have. That is the model I ended up building into Neon: named scorers that each answer one question, run as a gate before deploy rather than as a report afterwards.
Gate on it, or it does not count
An eval that reports is advisory. An eval that blocks is a specification.
This is where resistance shows up, and the resistance deserves to be taken seriously — a flaky gate that blocks good changes gets disabled within a week, and then you have neither a gate nor a report. So build it for trust:
Gate on slices you have already stabilised. A slice that moves three points run to run for no reason is not ready to gate on. Fix the variance first, or leave it out.
Distinguish a regression from noise. Run enough cases per slice that one unlucky sample cannot fail the build, and set an explicit threshold rather than gating on any decrease at all.
Make the failure legible. A failed gate should say which slice moved, by how much, and link to the specific cases that changed verdict. If the author has to do archaeology to find out what broke, they will route around it.
Allow a documented override. Sometimes shipping a known regression is correct. A gate with no escape hatch gets deleted; a gate with a logged override stays.
A few things worth doing
- Write tests for your data, not just your code. Schema drift, a silently changed upstream field, a shift in what users actually send — none of these break a unit test.
- Trend metrics, do not just aggregate them. Direction and variance matter more than the absolute number. A slice that has quietly moved two points a month for four months is the interesting story, and no single measurement contains it.
- Treat eval failures as backlog items. They need an owner, a priority and a place to live. Failures that exist only in a dashboard are not work; they are trivia.
- Version the suite alongside the code. An eval result is meaningless without knowing which cases and which scorers produced it. When both move in the same commits, comparing two runs is trivial.
The honest summary
The measurement half of evaluation is largely a solved problem and mostly a matter of doing the work. What separates systems that improve from systems that merely get observed is the plumbing between a bad number and a code change: named slices, a suite fast enough to be automatic, and a gate people trust enough to leave switched on.
Everything else is a dashboard.