Skip to content
VectorStep
Blog

Readiness/calibration tier won't turn green

Symptom: a pipeline has real run history, you’ve marked plenty of outcomes, and a readiness tier (or a step’s own calibration: gate) still reads insufficient_data — or a tier that was passing suddenly isn’t anymore. Three genuinely different causes produce this same symptom; check them in this order.

1. n_min is per confidence band, not a total — the single most misread number in the product

Section titled “1. n_min is per confidence band, not a total — the single most misread number in the product”

This is worth reading twice, because the config reads like it means something more forgiving than it does.

readiness:
  calibration:
    n_min: 20       # looks like "20 marked results total"

It isn’t a total. A step with 100 marked results spread evenly across 10 confidence bands has only 10 results in each band — and it will not validate at n_min: 20, no matter how healthy the 100 looks. Calibration measures “how accurate is this step when it says ~70% confident,” and that question can only be answered from results that actually landed in the 70–80% band; the other 90 results, however many there are, don’t count toward it.

Check the fullest band’s own count, never the total. The Insights — Steps page shows the count per band directly; a tier reading insufficient_data almost always means the fullest band is below n_min, even though total_n looks comfortably past it. Raising n_min because the total looks fine is the wrong fix — if anything, it makes the bar harder to clear. The actual fix is either accumulating more marked runs at that confidence level, or lowering n_min deliberately if 20-per-band is more than this pipeline’s volume can realistically produce.

2. Editing a prompt or agent config resets the evidence — on purpose

Section titled “2. Editing a prompt or agent config resets the evidence — on purpose”

If a tier was passing and suddenly reads insufficient_data right after a config change, this is almost certainly it, not a bug. require_current_config (default true on accuracy/calibration) filters evidence to runs matching the step’s current prompt_template text and the agent’s most recently observed version. Editing either one — even a one-word wording change — starts that tier’s evidence over at zero, deliberately: a bucket blending results from two different prompts isn’t a bigger sample, it’s a wrong one.

The run detail page’s Trust panel names this explicitly when it’s the cause — “Calibration was reset when this step’s prompt changed on <date>. The previous version had N marked results; this one has M of the K needed.” If you see that note, this is expected behaviour working as designed, not something to debug further. If a reverted prompt is involved, reverting it back to its exact previous text automatically rejoins the original bucket and every label it earned — nothing is permanently lost by trying an edit and changing your mind.

3. You’re looking at the wrong stage’s evidence

Section titled “3. You’re looking at the wrong stage’s evidence”

Calibration and readiness evidence for a pipeline is scoped to one stage at a time, and the two populations are never blended on the same view. A step’s calibration: {enforce: true} gate reads production history by default — if the pipeline hasn’t been promoted yet, or has only just been promoted, that population can genuinely be thin even if stage: testing has plenty of history. The Insights — Steps page’s Stage selector switches which population you’re looking at; make sure it’s set to match what you’re actually trying to validate against. A step can opt into blending both stages for its own gate with calibration.include_testing: true — but that’s an explicit, per-step choice, not something any view does for you by default.

The other classic misreading, while you’re here

Section titled “The other classic misreading, while you’re here”

Not an insufficient_data cause, but the other trap the Promotion readiness card explicitly warns about: adding a status to acceptable_statuses makes the operational bar laxer, not stricter. [completed, escalated] reads like a longer, more thorough list than [completed] alone, but it accepts runs where a human had to step in — a weaker claim about the step’s own performance. If a tier turns green right after “improving” this list, that’s the bar getting weaker, not the pipeline getting better.

Where next

Section titled “Where next”