Tutorial: turn on grounding

Build your first agent had you watch first-responder’s trace show two real tool calls — fetch against GitHub’s status API, filesystem reading known-issues.md — and told you that’s exactly what grounding checks automatically. This tutorial turns that on: a judge agent cross-references upstream_incident and known_issue against that same trace instead of you eyeballing it, then goes one rung further and lets that judgement actually gate the step.

Prerequisites

Turn on the trust knobs completed — ~/.vectorstep/pipelines/alert-triage.yaml’s triage step already has a confidence_threshold and a verifier. Grounding is executor: gateway-only, which triage already is.

1. The grounding-judge agent is already installed

The installer seeds every sample agent — grounding-judge included — into ~/.vectorstep/agents/ on first install, alongside generic-pipeline-step from the quick start. There’s nothing to add here.

Worth reading ~/.vectorstep/agents/grounding-judge/soul.md before moving on: it has tools: [] and is explicitly told not to use outside knowledge — its only job is to check whether the trace it’s handed backs up a claim, not whether the claim is actually true. A claim that just repeats something the primary agent was already told (the alert’s severity, service, environment) doesn’t need evidence; only claims that go beyond that — a root cause, a lookup result, a ticket ID — do. It’s also explicit that seeing a tool call is not the same as seeing its result: a truncated or missing TOOL RESULT means the claim is unsupported, not “probably fine because the right tool ran” — worth remembering for step 3.

Like first-responder in the previous tutorial, grounding-judge’s agent.yaml defaults to a direct Anthropic model — swap model: there the same way if you’re on a different provider (see Providers and the quick start’s OpenRouter setup steps). Without a working Anthropic key configured, grounding fails silently into G: null rather than blocking the run — the step still completes, just without a grounding score, which is easy to miss if you’re not watching for it.

2. Turn on grounding (shadow mode)

In ~/.vectorstep/pipelines/alert-triage.yaml, add a grounding: block to the triage step — everything else in the file is unchanged from the previous tutorial:

name: alert-triage
description: First-responder agent gathers evidence before anyone escalates
trigger:
  match: { source: alertmanager, severity: critical }
  dedup:
    enabled: false

context_template:
  include:
    - severity
    - summary

steps:
  - name: triage
    executor: gateway
    executor_config:
      agent: first-responder
      session_key: "agent:first-responder:{{pipeline_run_id}}:triage"
    confidence_threshold: 0.70
    on_low_confidence: escalate
    prompt_template: |
      ...                          # unchanged from the previous tutorial
    verifier:
      executor: gateway
      executor_config:
        agent: first-responder
        session_key: "agent:first-responder:{{pipeline_run_id}}:triage-verify"
      combination_strategy: minimum
      trigger:
        always: true
    grounding:
      agent: grounding-judge

agent: grounding-judge is actually the default — it’s shown explicitly here for clarity. Nothing else is required to turn grounding on: no threshold, no cap, just a score that gets recorded.

Reload and re-trigger:

curl -X POST http://localhost:8000/reload \
  -H "Authorization: Bearer $(grep VECTORSTEP_ADMIN_TOKEN ~/.vectorstep/.env | cut -d= -f2)"
curl -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
  -H "Authorization: Bearer $(grep VECTORSTEP_WEBHOOK_TOKEN ~/.vectorstep/.env | cut -d= -f2)" \
  -H "Content-Type: application/json" \
  -d @webhooks/alertmanager_critical.json

What you should see

The step should still show completed, at the same confidence as before — shadow-mode grounding never changes the outcome. What’s new is a G figure alongside S and V in the Trust panel, plus a per-claim breakdown under it, each row marked supported (✓) with a short excerpt of the evidence the judge found in the trace, or unsupported (✗) if it couldn’t find any. Don’t expect exactly two rows for upstream_incident and known_issue specifically — the judge decomposes the response into whatever load-bearing claims it actually identifies, which is often more granular than the two named JSON fields (four is typical: the two structured claims plus supporting sub-claims it pulls out of the summary text). With both tools working, expect G at or near 1.0 — all of them should be well-supported. “How was this calculated?” now includes a grounding line in its narrative alongside self-report and verifier.

3. Make it actually catch something

Shadow mode is only interesting once you’ve seen it flag a real gap, and the reliable way to force one doesn’t touch first-responder at all — no prompt edits, no soul.md edits, none of the original two-task pipeline changes. In ~/.vectorstep/pipelines/alert-triage.yaml, just cap grounding.max_trace_chars down to something absurdly small:

    grounding:
      agent: grounding-judge
      max_trace_chars: 10   # deliberately tiny — see below

Reload and re-trigger — no Gateway reload needed, nothing about the agent or its tools changed:

curl -X POST http://localhost:8000/reload \
  -H "Authorization: Bearer $(grep VECTORSTEP_ADMIN_TOKEN ~/.vectorstep/.env | cut -d= -f2)"
curl -X POST "http://localhost:8000/webhook?source=alertmanager&allow_testing=true" \
  -H "Authorization: Bearer $(grep VECTORSTEP_WEBHOOK_TOKEN ~/.vectorstep/.env | cut -d= -f2)" \
  -H "Content-Type: application/json" \
  -d @webhooks/alertmanager_critical.json

What you should see

G should drop sharply, though not necessarily to exactly 0 — expect somewhere in the 0.0–0.5 range. The bundled judge correctly distinguishes two kinds of claims: ones that merely restate something first-responder was already told (the alert’s severity, service, summary — see step 1), which need no evidence and stay marked supported regardless of truncation, and ones that depend on what the tools actually returned (upstream_incident, known_issue, and any sub-claims pulled from them), which now flip to unsupported, each citing the truncated TOOL_RESULT as the reason. A run with no restated-context claims at all will bottom out at 0; one with some will land partway there — either way, the load-bearing, evidence-dependent claims are what you’re checking, and those should all read unsupported. Nothing about what actually happened changed: the same two tool calls ran and returned the same real data, and both of first-responder’s original claims (upstream_incident, known_issue) are exactly as true as they were in step 2. What changed is that max_trace_chars: 10 truncates every tool-result event in the transcript handed to the judge down to about ten characters plus an ellipsis — and the bundled grounding-judge is explicit that a TOOL CALL line alone (which survives untouched — only result content gets truncated) is not evidence, and that a truncated result should be marked unsupported rather than charitably assumed fine. If your evidence text calls out the truncation explicitly rather than just saying “unsupported,” that’s the judge doing exactly what it’s told.

This is the truncation gotcha the reference doc warns about, deliberately provoked rather than stumbled into: a claim that looks exactly like a hallucination from the grounding score alone can actually be a real, true claim whose supporting evidence just didn’t make it into what the judge was shown. In production this usually shows up by accident on a step with unusually long tool output, not because someone set the cutoff to 10 — see Grounding keeps flagging real evidence as unsupported for the full troubleshooting path, including the second, independent truncation point on the Gateway side that raising this setting alone won’t fix.

Getting a truncated result reliably marked unsupported took real iteration to land on the bundled judge’s exact wording — earlier attempts that didn’t explicitly rule out “a tool was called, so it’s probably fine” let plenty of genuinely-unsupported claims through, and results varied noticeably between models used as the judge. Don’t assume a grounding judge behaves a given way just because the sample does, especially if you swap in a different model or write your own from scratch — verify it against a case where you already know the right answer, the same way this tutorial just did. See Writing your grounding judge, Writing good agents, and Writing good prompts for what actually shapes this behaviour.

This isn’t a flaw in the exercise — it’s the actual reason the trust vector exists at all. You just watched grounding scores swing across identical setups, and a verifier call fail outright, from nothing more than ordinary model variance. An agent’s own self-reported confidence would have sailed through every one of those runs sounding equally sure of itself. VectorStep doesn’t try to make that variance go away; it catches it — a verifier, a grounding check, a deterministic check, each able to lower trust but never inflate it, so a fluky run gets escalated to a human instead of shipped silently. See Why I built VectorStep for the fuller version of this.

4. Turn on enforced grounding

So far G is purely informational. Add enforce: true to make it participate in the gate:

    grounding:
      agent: grounding-judge
      max_trace_chars: 10
      enforce: true

The gate formula becomes combined_trust = min(effective_confidence, G) — see the full formula rather than re-deriving it here. There’s no separate grounding threshold; it reuses the step’s existing confidence_threshold (0.70).

Reload and re-trigger again, with max_trace_chars: 10 from step 3 still in place.

What you should see

With G still well below confidence_threshold: 0.70 (per step 3’s range), combined_trust = min(effective_confidence, G) comes out at G’s value regardless of how high the primary’s own confidence is — so expect the run to escalate. The Trust panel header should now read “Trust (enforced)” instead of “(shadow)”, with a Combined trust figure shown alongside S/V/G. This is the actual payoff: a confident, correct, well-evidenced response still gets treated as untrusted the moment its evidence trail is cut off — grounding enforcement doesn’t know why G is low, only that it is.

5. Clean up

Remove max_trace_chars: 10 from the grounding: block (or set it back to the default, 1500) — it only existed to manufacture this demo, and leaving evidence permanently invisible to the judge defeats the point of having grounding at all. Reload VectorStep again after reverting. Leave grounding: and enforce: true in place; that’s the actual end state this tutorial was building toward. Nothing about first-responder itself was ever touched, so there’s nothing to revert on that side.

A note on how narrow this agent still is

first-responder’s soul.md hardcodes an exact numbered task list — two tools, then summarise. That’s a reasonable shape for a single-purpose tutorial agent, but it doesn’t scale to reusing the same agent across pipelines with different needs. A more general design keeps soul.md to identity and standing behaviour (how carefully to reason, when to say “I don’t know,” what “supported by evidence” means to this agent) and lets each pipeline’s own prompt_template supply what this run specifically needs. See Writing good agents and Writing good prompts for where that line usually belongs, including why a prompt-level ask that conflicts with soul.md’s stated scope tends to lose.

The grounding-judge sample, in contrast, is already written the general way: nothing in its own soul.md is task-specific, which is exactly why the same one agent could judge a completely different pipeline’s claims tomorrow without changes. Most real deployments reuse one judge across many steps rather than writing a bespoke one per pipeline; you’d reach for a specialised judge only when a domain has its own notion of what counts as evidence (legal, financial, medical claims, for instance) that a generic cross-referencer wouldn’t reliably apply. See Writing your grounding judge for when that’s actually worth doing, and what to change.

Where next

Go to Fan out over multiple services next — the next tutorial in the series.

Once you’re comfortable with the mechanics:

Grounding — the full reference this tutorial walks through hands-on, including deterministic checks (D), the third trust-vector signal this tutorial doesn’t cover.
Writing your grounding judge — going beyond the bundled sample: model choice, and what to change for a step with many load-bearing claims.
Grounding keeps flagging real evidence as unsupported — if a claim you’re confident is real gets flagged unsupported, this is almost always a truncation cutoff, not a judge mistake — start here.