Choosing readiness criteria

Promotion readiness documents what every tier and knob does, exhaustively. This guide is meant to be its Choosing confidence thresholds companion — the opinionated “what do I actually set” complement, aimed at someone standing in front of the promotion tutorial’s criteria builder with sixteen knobs and no instinct yet for which four matter for this pipeline.

Start with operational alone, always

It’s the cheapest possible bar — pure PipelineStep.status counting, no labelling required — and the only tier a non-LLM step (webhook/notify/ human/pipeline executor) can ever satisfy, since those never write effective_confidence. It also can’t lie to you the way the other three can: it never reports fail, only pass or insufficient_data, so there’s no way to misconfigure it into a false negative. For a low-stakes step, operational with a real min_runs is a perfectly reasonable bar on its own — you don’t owe every step a full four-tier gauntlet just because the option exists.

Add accuracy before calibration, not instead of it

Calibration is the strongest signal readiness has, but it’s also the slowest to earn, because n_min counts per confidence band, not in total — a step needs enough marked history clustered at the same score to say anything. Accuracy with a real min_human_marked floor is a faster, still-meaningful bar while that calibration history accumulates: it asks “were the outputs actually good,” on whatever labels exist today, rather than waiting for a full distribution across bands. Don’t treat accuracy as calibration’s poor cousin and skip straight to the stronger tier — configure both, and let accuracy carry the bar honestly until calibration has enough history to carry its share too. min_human_marked matters here specifically: without it, an accuracy tier can be satisfied entirely by automated or deterministic marks, which answer a narrower question than “is this step good” and can make a population that’s failed every deterministic check read as accurate by omission.

confidence alone is close to worthless as a gate

confidence only checks “did the step claim confidence,” and self-reported confidence is, by design, the least trustworthy signal VectorStep has — the agent grading its own work with no outside check at all. A step can clear a min_confidence: 0.90 bar every single time while being wrong just as often as before; the tier never asks whether it should have been confident. Use it as a cheap early smoke test — did the agent even get far enough to produce a real answer — never as the bar a step gets promoted on by itself, and always pair it with a real min_runs so a lucky handful of runs can’t stand in for a trustworthy mean.

Two misreadings that make a “stricter-looking” config weaker

Both of readiness’s worst traps look like they’re raising the bar and actually lower it.

calibration.n_min is per band, not a total. A step with 100 marked results spread evenly across 10 confidence bands has only 10 results in each band, and won’t validate at n_min: 20 no matter how healthy total_n looks. Before raising n_min because a tier reads insufficient_data, check the fullest band’s own count — it’s usually the real number, and it’s usually smaller than the total suggests.

Adding a status to acceptable_statuses makes the bar laxer, not stricter. [completed, escalated] reads like a longer, more thorough list than [completed] alone, but it accepts runs where a human had to step in and fix things — a weaker claim about the step’s own performance, not a stronger one. If a pipeline’s not_ready badge turns green right after someone “improves” this list, that’s the config getting weaker, not the pipeline getting better; treat it as a signal to look at why runs are escalating, not a config change worth making on its own.

Match the tier set to the step’s actual risk

This is Choosing confidence thresholds’s core argument one level up: a step that only informs the next one doesn’t need the same bar as a step that authorises a real, hard-to-reverse action on its own. readiness: null is a legitimate choice, not a gap to fill in later — a notify step has nothing meaningful for any of the four tiers to measure, and forcing one on anyway just adds a chip that never says anything useful. Reserve the full stack — accuracy with a real min_human_marked, plus calibration with a tightened max_divergence — for the steps whose mistakes are actually expensive, the same way you’d reserve a 0.85 confidence threshold for a step that remediates rather than one that only reports.

Use require_own_evidence/require_current_config deliberately

Both default to something reasonable, but “reasonable default” isn’t the same as “right for this step,” and both are easy to leave unexamined.

calibration.require_own_evidence (default false) lets a shared library step’s production track record from a different pipeline count toward this one’s calibration tier, as long as agent/model/prompt/agent version all match exactly — the response names which pipelines contributed, so a green tick is never mysteriously someone else’s traffic. That’s usually the right default for a genuinely shared step: it means calibration doesn’t have to start from zero in every pipeline that reuses it. Set it true when a step’s context inside this pipeline is different enough — a different downstream consumer of its output, different stakes — that another pipeline’s history shouldn’t vouch for it here.

require_current_config defaults to true on accuracy/calibration and false on operational/confidence, and that asymmetry is deliberate, not an oversight to normalize away. Editing a step’s prompt drops its accuracy/calibration evidence to insufficient_data immediately — with an explicit note naming how many earlier results were excluded, never a silent zero — because a calibration bucket is keyed by (prompt_hash, agent_version) by definition; there’s no way to “ignore the version” there without merging buckets that measured different things. operational‘s false default means a one-line typo fix doesn’t wipe out 30 clean runs’ worth of operational evidence, since “did it run cleanly” usually doesn’t depend on exact prompt wording the way accuracy and calibration do. One trap worth knowing before you rely on either knob: overriding prompt_template locally on a use: step changes its prompt hash and silently forfeits inherited evidence regardless of how require_own_evidence is set — the override itself, not the knob, is what breaks the link.

Where next

• Promotion readiness — the full mechanical reference this guide assumes throughout, including the complete knob table and the YAML examples the sections above draw their numbers from.
• Promote your pipeline to production — the hands-on companion this guide is meant to prepare you for, walking the criteria builder end to end on a real pipeline.
• Choosing confidence thresholds — the same “what number do I actually pick” question, one level down, for gating a single step’s output rather than promoting a whole pipeline.
• How confidence and calibration work — why self-report is the weakest signal and calibration the strongest, which is most of the reasoning behind the tier ordering above.