Teaching an Agent to Hold Still

A pipeline of language-model agents had been built to turn a conversation with a client and their submitted documents into a recommended plan. Five stages, hand-offs between them, a good seventeen minutes end to end. I was amazed by the output: a polished report with sound reasoning, an action plan, and a detailed analysis to support the main conclusions. But it had a quiet problem: run it twice on the same input and it could hand back two different recommendations.

Nothing had crashed and it turned out that the agents had not lied either. The pipeline was simply improvising in the places the specification left open, and doing it differently each time. This is the story of how I made that visible, then made it stop. The domain doesn't matter; the shape of the problem could be familiar in every agent system.

This is the last note in the series for a reason: the eval leans on everything the earlier ones built: the inspectable, deterministic harness, the single-sourced facts, and the graded test profiles it runs against. With those in place, measuring stability became possible; without them, it would have been guesswork.

Measure whether it holds, not whether it's right

The instinct is to grade the output against a "correct" answer. But for an open-ended recommendation there often isn't one, and for a PoC I did not have access to a ground truth provided by a human professional. So I set accuracy aside and asked a different, cheaper question:

Does the workflow give the same answer when nothing about the question changed?

That reframing did most of the work. Decision stability, i.e., whether the recommended strategy holds across runs. The underlying numbers were tracked too, but demoted to secondary. A recommendation that flip-flops is a failure whether or not any single run is defensible, and instability is something you can measure today. In the future, when a human expert sees a questionable recommendation, they can be sure that the root cause is a coding error, a misinterpretation of the knowledge base, or missing information, all of which can be corrected. They should not be chasing random fluctuations of the pipeline. This is essential for building trustworthy AI systems.

Two facts made it tractable

First, the stochastic core was seeded. The simulation underneath the agents ran off a fixed random seed, so identical inputs produced bit-identical numbers. That single property is worth more than it sounds: it means any numeric movement between runs is signal (the agent fed a different parameter, or the code changed), and never simulation noise.

Second, the pipeline had a headless mode: feed it a pre-built input and it skips the human conversation. So the whole thing could run unattended, on a schedule, as many times as I liked. A stability eval you can't run cheaply and often is a stability eval you won't run.

Build the measuring instrument out of ordinary code

The measurement layer, the thing that reads two runs and decides whether they agree, is plain, deterministic code with no model in it. If you grade an LLM's output with another LLM, your ruler drifts with the thing it's measuring, and you can never tell a real regression from a bad mood in the judge. This is more relevant in this particular use case as opposed to other LLM applications because the recommendations are based on strong logical, and numerical foundations.

On top of that sat three artifacts: an append-only ledger, one row per run per metric, the data of record that lets you attribute a drift to the change that caused it; a set of dashboards regenerated from the ledger; and a small library of frozen baselines, the golden runs everything is measured against.

Categorical up top, numbers underneath

The scorecard put the decisions first, as categoricals: the recommended strategy and the sub-choices feeding it. Any change at all is a drift. There is no "close enough" on a decision. The numbers were graded on tolerance bands: a movement was fine within a small absolute-and-relative window, flagged outside it.

One root cause, one failure

Early on, when a single decision flips, every number downstream of it moves too. Report each of those as its own failure due to one root cause becomes fifteen red cells. So the ruler attributes: when a decision flips, the numbers that mechanically follow it are annotated and not counted again. The high-signal event, i.e., the one you actually chase, is a deterministic number that moved while its governing decision held still. Given the seeded core, that can only mean the math changed.

The other half of not crying wolf was measuring the margin. A recommendation chosen from a field of near-identical options is a coin toss; one chosen over a clear runner-up is a real verdict. So I recorded how close the race was. A flip within a dead heat is expected and shrugged off. A flip away from a clear winner is an alarm.

The first run came back red

Three of five test profiles drifted on the recommendation itself. A green first run mostly means the eval isn't looking hard enough. The point of building the instrument was to see something, and it saw plenty. The question became: why.

From "it drifts" to a coordinate

"It's flaky" is not a bug report. To turn it into one I ran each profile K times in a fixed state and aggregated the values instead of the pass/fail: for each decision, its mode-share (how often the most common answer won); for each number, its coefficient of variation. Suddenly the vague complaint had coordinates. A decision at 71% mode-share is a specific fork taken two runs out of seven, with a specific pair of numbers on either side of it. Calibration is the microscope you point at whatever the eval flags.

Almost all of it was unpinned degrees of freedom

Every root cause I chased turned out to be the same species: a spot where the agent had latitude the specification never nailed down, so independent runs resolved it independently.

The agent wasn't unreliable. I'd underspecified it, and the agent was improvising in the gaps I left, faithfully, and differently every time.

Pin one, re-calibrate, watch the mode-share snap to 100% and the coefficient of variation collapse toward zero. Then the next factor, previously hidden under the first, steps into view. Peel, measure, pin, repeat.

One of them wasn't a bug at all

The last unstable profile resisted every pin, because it was a different animal. Its input was genuinely missing a value the task needed: a brief mentioned a cost but named no figure. Some runs invented a reasonable number; others declined to invent and modeled nothing. That fork swung the whole result, and no amount of specification can pin it, because the information simply isn't there.

This is the distinction worth carrying away: an unpinned mapping is an engineering fix (write the rule). A missing input is a product decision (refuse to fabricate and surface it, or go capture the value at intake). Conflating the two sends you writing rules that can't help. I flagged it, and deliberately left it open.

Re-baseline, then prove it

Some fixes correctly changed the recommendation. That's not drift, it's progress. But it means the old golden runs are wrong now. So the fixes earned a new baseline epoch, versioned, so the trend line doesn't falsely lurch when you improve the thing on purpose. Old history stays attributed to the old baseline; new runs measure against the new one.

And then the step it's tempting to skip: an independent confirmation run against the fresh baselines.

A baseline you can't reproduce isn't a baseline. It's a screenshot.

The confirmation cycle came back reproducing every golden to the last digit. Decision stability went from two-of-five to five-of-five; the suite flipped from red to green; the unexplained-drift count sat at zero. The workflow finally held still.