Tutorial: promote your pipeline to production

Every tutorial in this series has run ~/.vectorstep/pipelines/alert-triage.yaml in stage: testing — the default — without saying so explicitly. This capstone tutorial makes that visible, then walks through actually earning and executing the promotion to stage: production, using readiness evidence rather than a gut call.

Prerequisites

The pipeline as it stands after however much of the series you’ve done — this tutorial works against any shape of alert-triage.yaml. The more tutorials behind it (confidence threshold, verifier, fan-out, notifications), the more interesting its accumulated run history is for this one; readiness reads real evidence, and a pipeline with only a couple of runs will mostly show insufficient_data, which is itself a fine thing to see once.

1. See what `stage: testing` has actually been hiding

Re-trigger the pipeline once, the same way you have all series. Every route needs a bearer token — /webhook specifically needs the webhook token, /reload the admin token; both used repeatedly through this tutorial, so set them once:

WEBHOOK_TOKEN=$(grep VECTORSTEP_WEBHOOK_TOKEN ~/.vectorstep/.env | cut -d= -f2)
ADMIN_TOKEN=$(grep VECTORSTEP_ADMIN_TOKEN ~/.vectorstep/.env | cut -d= -f2)

curl -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
  -H "Authorization: Bearer $WEBHOOK_TOKEN" \
  -H "Content-Type: application/json" \
  -d @webhooks/alertmanager_critical.json

Open the run in the UI — the amber TESTING badge next to the run status is the first thing stage: testing changes: it’s shown on every browse surface (runs list, dashboard, this run) precisely so testing activity never looks like real traffic by accident.

If you did Route escalations to a real channel and left check-upstreams’s confidence_threshold at its real 0.70 (you reverted it from the artificial 0.99 at the end of that tutorial), a healthy re-trigger here has nothing to escalate — the Run log will show a clean sequence of Step started/Step completed events and no notification event at all, because none fired. That’s expected, not a sign something’s wrong: stage: testing only mutes a notification that would otherwise be sent, it doesn’t manufacture one. If you want to see the muting itself in action, temporarily push confidence_threshold back up past what a real check will clear (the same trick that tutorial used) and re-trigger — now the Run log shows notification_suppressed_testing instead of notification_sent — “[testing] Notification routed to log: escalate → would have been log” — even though a real channel is configured, testing forces every notification through log instead. Put the threshold back to 0.70 afterwards either way.

What’s genuinely unaffected: the trust-gating mechanics from every earlier tutorial — confidence thresholds, verifiers, grounding, fan-out joins, budget aborts — behave identically in either stage. Only four specific outbound paths are muted by stage: testing (notifications, executor: notify, on_failure.webhook, and the external side of executor: human) — see what testing mutes for the complete list. Nothing about whether the pipeline reasons well is different; only whether its side effects reach the outside world.

2. Build up real run history

Readiness reads accumulated evidence — it can only ever report what this pipeline has already earned, not what it’s capable of in principle. Trigger it several more times:

for i in 1 2 3 4 5; do
  run_id=$(curl -s -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
    -H "Authorization: Bearer $WEBHOOK_TOKEN" \
    -H "Content-Type: application/json" \
    -d @webhooks/alertmanager_critical.json | python3 -c "import json,sys; print(json.load(sys.stdin).get('run_id',''))")
  echo "Triggered $run_id — waiting for it to finish before the next one..."
  while :; do
    status=$(curl -s "http://localhost:8000/runs/$run_id" -H "Authorization: Bearer $ADMIN_TOKEN" \
      | python3 -c "import json,sys; print(json.load(sys.stdin).get('status','?'))" 2>/dev/null)
    [ "$status" != "running" ] && [ -n "$status" ] && break
    sleep 5
  done
done

Every trigger here reuses the same static alertmanager_critical.json payload, so it’s the same alert fingerprint every time — and VectorStep only ever runs one pipeline per fingerprint at once. Firing the next trigger while the previous run for that fingerprint is still running doesn’t start a second run; it’s treated the same way a real Alertmanager resend of a still-firing alert would be. That’s why this waits for each run to actually finish (usually under a minute, but fan-out + verifier + grounding steps can take longer depending on model latency) rather than using a fixed short sleep between triggers.

Then mark a few outcomes — this is what feeds the accuracy and calibration tiers below; without it they’ll sit at insufficient_data no matter how many runs pile up.

There are two places to do this, and which one you use actually matters:

Run-level — the Accuracy feedback widget at the bottom of the run detail page (Correct / Partial / Incorrect). One click, one verdict for the whole run.
Step-level — a separate Correct / Partial / Incorrect control inside each individual step’s own expanded detail panel, next to its parsed output. One verdict for that one step’s execution only.

How they interact: a run-level mark is a fallback — it’s applied to every step in that run for accuracy/calibration purposes, but only for steps that don’t already have something more specific. A step’s own mark always wins over the run-level fallback for that one step; so does an automated deterministic-check failure on that step, if it has one. Nothing below the winning label is even looked at. This is exactly why run detail describes step-level feedback as not affecting “gating or flow control” — that’s about the live run itself behaving no differently, not about readiness. It’s fully counted, and counted first.

In practice: for this series’ pipeline, marking the run is enough most of the time — one meaningful chain of reasoning, one verdict. Reach for a step-level mark instead when a run’s overall outcome doesn’t reflect one particular step fairly — e.g. the run as a whole was fine but one check-upstreams branch reasoned badly and you don’t want that dragging down that branch’s own calibration history, or the reverse (a run escalated because of one bad step, but a different step in the same run behaved correctly and deserves its own accurate record rather than inheriting the whole run’s “incorrect”).

Open a couple of the resulting runs and mark a few — run-level is fine to start with. For finding every step still missing a mark across a whole pipeline (rather than clicking into runs one at a time), see the Marking queue.

3. Look at readiness before you set a bar

Open the pipeline’s detail page (/ui/pipelines/alert-triage) and find the Promotion readiness card — it only appears for stage: testing pipelines. This pipeline has no readiness: block yet, so the card reads “No criteria configured — showing observed evidence at service defaults”, and each step shows an observed: ... chip instead of tier chips — real evidence, just not measured against a bar you chose. Hover one: “Observed at service defaults (bin_width 0.1, n_min 20) — not a bar you chose.”

This is what evidence looks like with nothing configured: not nothing, but not a verdict either.

4. Use the criteria builder

Click the Build criteria button in the Promotion readiness card’s own header — not a per-step tier chip; clicking one of those opens the builder scoped to just that one step instead, which isn’t what we want here. This opens a preview-only builder — turning a knob sends the candidate config to POST .../preview and updates the result against this pipeline’s real accumulated evidence within about 300ms. Nothing is written until you paste something yourself; the banner says so directly: “Preview only — nothing is saved. Copy the YAML and ship it through git.”

Check the Scope dropdown at the top of the builder reads “Whole pipeline (default for every step)” — that’s what it opens to when you use the header button, but confirm it before continuing, since the resulting YAML lands in a completely different place (pipeline-level vs. one step’s own readiness: block) depending on what’s selected here.

Start with two knobs:

Operational → min_runs: 20, acceptable_statuses: [completed]
Confidence → min_confidence: 0.80

Watch the live result update per step as you type — a verdict badge (ready / not_ready / insufficient_data) and a narrative explaining why. Two knobs on this same screen are worth reading carefully before you trust them elsewhere, because both are counter-intuitive in the same direction — laxer than they look:

Once the preview looks right, click Copy and paste the generated snippet into ~/.vectorstep/pipelines/alert-triage.yaml, at the pipeline level (a sibling of budget:/notifications:):

readiness:
  operational:
    min_runs: 20
    acceptable_statuses: [completed]
  confidence:
    min_confidence: 0.80

5. Reload and confirm the bar applies for real

curl -X POST http://localhost:8000/reload -H "Authorization: Bearer $ADMIN_TOKEN"

Refresh the pipeline detail page. The per-step chips should now read operational / confidence verdicts instead of a plain observed: ... badge — pass (green), not_ready (red), or insufficient_data (amber) depending on what the last few triggers actually produced. Click How is this judged? on a step for the same plain-language narrative the builder showed, now reading the real configured bar instead of a preview.

If a tier reads not_ready and you don’t understand why, this is the moment to actually trust the number rather than raise the bar until it turns green — that’s what stage: testing and this whole card exist to surface before real traffic depends on it.

6. Promote

Every tutorial in this series has relied on stage: testing being the implicit default — nothing in alert-triage.yaml has said so explicitly until now. Add one line at the pipeline level (a sibling of budget:/ notifications:/readiness:):

stage: production

Reload, then trigger it without allow_testing=true this time — this is the concrete difference promotion just made:

curl -X POST http://localhost:8000/reload -H "Authorization: Bearer $ADMIN_TOKEN"
curl -X POST "http://localhost:8000/webhook?source=alertmanager" \
  -H "Authorization: Bearer $WEBHOOK_TOKEN" \
  -H "Content-Type: application/json" \
  -d @webhooks/alertmanager_critical.json
# → {"status": "accepted", "run_id": "..."}   — no allow_testing needed any more

What you should see

The new run has no TESTING badge, and — assuming check-upstreams’s confidence_threshold is still the real 0.70 from Route escalations to a real channel rather than the 0.99 used there to force a demo — it will most likely just complete normally, same as every healthy run earlier in this series. That’s not an anticlimax: a promoted pipeline behaving exactly like it did in testing, on the same real evidence, is the entire point.

It can also genuinely escalate here — grounding’s own judge call varies run to run, same as it did in Turn on the trust knobs, and a real check on a real GitHub/npm status response can occasionally come back weaker evidence than last time. That’s not a failure of this tutorial either: it’s the same trust vector, now gating a real alert on real evidence instead of a rehearsal, and this is what it looks like when the evidence genuinely doesn’t clear the bar on a given run. If it happens, skip ahead — you’ve already seen the “real notification” case this next section manufactures on purpose, since your escalation just did it for real.

One consequence worth being explicit about if your run completed cleanly: no notification fires on a healthy run either, even if Telegram is configured — notifications: only reacts to an escalate/abort/notify transition, and a healthy completed run never triggers one. If you were expecting to see something in your phone right now, that’s why nothing arrived — not a bug, and not a testing-stage artifact this time; a production run with nothing to escalate genuinely sends nothing.

The Promotion readiness card is gone entirely from the pipeline detail page — it’s a stage: testing-only tool, not a permanent fixture, since a production pipeline is judged by its live metrics and dashboards from here on, not a pre-promotion checklist.

Seeing a real notification actually fire

To confirm end-to-end that a promoted pipeline’s notifications genuinely reach a real channel — not suppressed, not simulated — force the same escalation used earlier in the series, now for real. In ~/.vectorstep/pipelines/alert-triage.yaml:

  - fan_out:
      name: check-upstreams
      ...
      confidence_threshold: 0.99   # deliberately unreachable — see below

curl -X POST http://localhost:8000/reload -H "Authorization: Bearer $ADMIN_TOKEN"
curl -X POST "http://localhost:8000/webhook?source=alertmanager" \
  -H "Authorization: Bearer $WEBHOOK_TOKEN" \
  -H "Content-Type: application/json" \
  -d @webhooks/alertmanager_critical.json

This time the Run log should show “Notification sent: escalate → log” and “Notification sent: escalate → telegram”, not the [testing] Notification routed to log: ... phrasing from earlier in this series — and if Telegram is configured, check your phone; this is the first message in the whole series that actually reaches it for real, rather than being routed to log because the pipeline was still stage: testing.

Put confidence_threshold back to 0.70 once you’ve confirmed it — same as every other time this series has used an artificially unreachable bar to force a demo, it only exists to manufacture the escalation, not to describe how this pipeline should actually run.

7. Close the loop

Nothing about this promotion was automated. Readiness is strictly advisory — the tooling never intercepted the stage: edit, never added a UI toggle, never blocked /reload. It informed a decision you made in a one-line YAML diff, reviewed in git exactly like any other config change. A team that wants an automated gate on top of this can script one against GET /pipelines/{name}/promotion-readiness — the same JSON the builder itself calls — but building that automation is a decision for that team to make deliberately, not something this feature does for you by default.

That’s the end of the tutorial series. Every mechanism from the quick start through this page is now sitting in one real, promoted pipeline — trust knobs, grounding, fan-out, artifacts, notifications, budget, and now a readiness bar it actually earned.

Where next

Promotion readiness — the full tier reference (operational/confidence/accuracy/calibration), every readiness knob, and the merge rules for a step that adds to or overrides the pipeline’s house standard.
Testing vs production stages — exactly what stage gates and what it doesn’t, and the full promotion workflow.
Marking queue — finding every step still missing a human accuracy mark before its tiers can resolve.
Replay / shadow evaluation — once a step has real calibration history, this re-runs a bucket’s own recorded, labelled inputs against a candidate prompt/model change, so you can compare the two on identical evidence without waiting for fresh production traffic.