Writing your grounding judge

~/.vectorstep/agents/grounding-judge/, seeded by the installer, is a working starting point, used in Turn on grounding as written. This guide is about what to change and why, once the bundled default isn’t quite right for a specific step.

Start from the sample, don’t write one from scratch

Its soul.md earns its length: read closely, it encodes three disciplines that are easy to state but easy to lose in a rewrite. First, a claim that merely restates something already present in the original task — the alert’s severity, service name, environment — needs no evidence, because it was given to the primary agent, not discovered by it; without this distinction a judge marks plain input facts as false “unsupported” verdicts. Second, no outside knowledge and no tools — the judge decides whether a claim is anchored to the trace it was shown, never whether the claim happens to be true. Third, and the easiest of the three to skip when writing from a blank page: seeing a TOOL CALL line in the trace is not evidence by itself — the judge has to actually check the corresponding TOOL RESULT content backs the specific claim, not just that some tool was invoked. Rewriting a judge’s soul.md from zero is the most likely way to accidentally drop one of these three, and each failure mode looks identical from the outside: a grounding score that’s technically computed but not actually checking what it claims to check.

Model choice: usually the cheapest model that follows instructions reliably

The bundled sample’s own agent.yaml picks anthropic/claude-haiku-4-5-20251001 with the reasoning right there in a comment: cross-referencing a claim against a transcript is a constrained, mechanical task, not open-ended reasoning — it doesn’t need the same model as the primary agent it’s judging. The real requirement is reliably returning the exact JSON shape every time, including the full reasoning.claims list, not just a plausible-sounding summary. If a cheap model starts drifting off-format under real traces — malformed JSON, claims silently dropped from the list — that’s the signal to move up to a stronger model, not a reason to reach for one pre-emptively before you’ve actually seen it struggle.

Keep tools: [], deliberately

The sample ships with an empty tool list and says why directly in the comment: a judge that can browse or query isn’t cross-referencing anymore — it’s a second investigator, and its score stops meaning “was this backed by the primary’s own evidence” and starts meaning something closer to “did a second agent independently agree.” Those are different signals, and conflating them loses exactly the thing grounding is for: checking the primary’s claims against its own trace, not against a fresh investigation of the underlying facts. Resist adding tools even when it would make the judge “smarter” — a judge that goes and checks Grafana itself instead of reading what the primary already gathered is closer to an independent-mode verifier than a grounding judge, and the two aren’t interchangeable.

Size max_tokens to the step’s claim count, not a guess

The grounding-accuracy troubleshooting guide covers this as a symptom: a step with many load-bearing claims can produce a reasoning.claims list long enough that the judge’s own response gets cut off mid-generation, and the failure looks like a parse error, not a low score — genuinely different from a claim being scored unsupported, and worth telling apart before assuming the judge is being harsh. This section is the design-time version of the same fix: a step whose output routinely makes many claims (a triage step that lists several findings, not a step with one clean verdict) needs a judge max_tokens sized for a proportionally long claims list from the start, set on the grounding-judge agent’s own agent.yaml on the Gateway, not discovered after the first parse failure shows up in production. The sample’s default (4096) is tuned for a modest claim count; raise it for any step whose primary agent’s prompt structurally invites a long list of findings.

One shared judge, or several domain-specific ones?

grounding.agent defaults to grounding-judge but is settable per step, so a single shared judge isn’t the only option. A generic judge is the right default for most steps — the discipline in its soul.md (given input vs. discovered claim, tool call vs. tool result, no outside knowledge) is domain-agnostic by design. Reach for a step-specific judge only when a step’s evidence is specialised enough that a generic judge would need to be told, every single time, what “supported” actually looks like in that domain — a step whose claims hinge on a particular ticket-ID format or dashboard-naming convention a generic judge has no way to recognise as evidence versus noise. A step-specific judge here just means the same soul.md discipline plus a short, concrete addendum naming what counts as evidence in that one domain — not a rewrite from scratch, which is exactly what the first section above warns against.

Trust the judge before you enforce it

Grounding’s shadow mode exists exactly for this: it computes and records G on every run a step’s grounding: block declares, whether or not grounding.enforce: true is set, so you can watch its verdicts against steps you can manually verify for a while before flipping the switch that lets it actually gate. This is the same discipline Choosing confidence thresholds recommends for a primary agent’s own threshold — don’t guess, watch the real numbers first — applied to the judge itself rather than the step it’s judging. A judge that’s confidently wrong in shadow mode is a config problem to fix (model, max_trace_chars, the soul.md itself); a judge that’s confidently wrong under enforce: true is a step failing runs for the wrong reason.

Where next

• Grounding — the full judge contract and config reference.
• Turn on grounding — the hands-on tutorial this guide is the deeper companion to.
• Grounding keeps flagging real evidence as unsupported — the max_tokens sizing problem from the symptom side.
• Verifiers — the other second-opinion mechanism, and the line between “grounding judge with tools” and “independent-mode verifier” the tools section above draws.