One Axis at a Time

You can't test a pipeline you can't feed. Real client situations were scarce, sensitive, and inconveniently shaped, so before I could measure whether the agents were stable I needed a way to manufacture clients: synthetic profiles, plausible enough to exercise the workflow and controlled enough to reason about. The naive version, "generate a random client," fails twice over: it hands you either a toy nobody resembles, or a tangle so knotted that when the agent stumbles you can't say what tripped it.

What I wanted was the opposite of random: inputs I could dial deliberately, so that every failure pointed back at a cause. That meant giving "complicated" a structure.

Complexity isn't a scalar. It's a set of axes.

The move that unlocked everything was refusing to treat difficulty as a single number. A client's situation isn't just "hard", it's hard in particular ways, and those ways are largely independent of one another. So I broke the situation into a set of complexity vectors: dimensions, each its own spectrum running from easy-to-model at one end to hard-to-model at the other. A synthetic profile is then just a point in that space, one level chosen on every axis.

"Hard" isn't a number. It's a set of coordinates.

Localized difficulty means localized blame

A one-to-five difficulty dial would have been easier to build and useless to debug. When a case rated "hard" fails, the rating tells you nothing about why. A vector does: it says this profile is level-three on assets and near the floor on everything else. So if it breaks, assets is your first and only suspect. The structure I put into the input is exactly the structure I get back in the diagnosis.

This is the same instinct as the harness note, pushed one step upstream into the test data itself: localize the blame by construction. The harness localizes a failure to a single stage; graded inputs localize it to a single dimension, i.e. the one hard axis kept from hiding inside a vague "complex case." Design the inputs so the outputs can only point one way.

The common case, not the easy case

The temptation is to begin at the origin where every axis is at its minimum. I didn't. An all-easy profile is a strawman: no real client is that tidy, and stabilizing on it would teach me nothing about the distribution I'd actually serve. So I anchored at the most common vector instead: the typical client, which is emphatically not the easiest on every axis. Most people carry real complexity somewhere; the common case already sits a notch or two up several dimensions.

Start where your clients actually are, not where the model is most comfortable.

That choice matters more than it looks. The first profile you stabilize becomes the trunk everything else grows from. Root it in a fiction and every branch inherits the fiction; root it in the modal client and every expansion is a step into territory you'll genuinely have to serve.

Advance one axis at a time, behind a regression gate

From that anchor the plan was a curriculum, not a data dump. First, get the agents stable on the common profile. Stable in the sense the eval that closes this series measures: the recommendation reproduces run to run. Only then grow the complexity, and only in one dimension, two at the very most per step. Re-run the whole suite, check for regression, and move outward only once it's green again.

The size of the step is the entire discipline. Advance three axes at once and something breaks, and you're back to guessing which of the three did it. You've re-created the very ambiguity the vectors were built to kill. Move one, and a failure names its own cause.

Move one axis, and a failure names its own cause.

The regression gate is what makes the ground permanent. Every profile already passing stays in the suite forever, so each expansion has to earn its place without costing the ones behind it. Capability only ratchets forward. What you get is a difficulty curriculum where the agents are made incrementally more capable. Each step a little harder, none of them a leap.

A map, not a pile

Run this for a while and the test set stops being a heap of cases and becomes a map: a lattice of covered profiles with a visible frontier, the edge of what the agents can reliably handle. Growth is deliberate and legible. You can point at the boundary and say, precisely, "we're solid two levels up on assets and timeline, one level up on tax, and we haven't touched multi-party households yet." Coverage stops being a box you check and becomes a frontier you push.

And the hardest end of one axis: a profile that names a cost but never a figure, is exactly the kind of genuinely missing input that pinning can't fix: the open problem the eval closing this series runs headlong into. Putting it on an axis was how I turned "we don't handle that yet" from a surprise into a coordinate: a known point beyond the current frontier, waiting its turn, not a landmine.