Production Prompt Orchestration
VectorStep turns any webhook into a multi-step AI pipeline — and only lets a step act autonomously when its confidence survives a second opinion, evidence checks, hard deterministic facts, and its own measured track record.
Every pipeline’s steps, agents and run history in one page — the structure, not just the outcome
An agent finishes a task and says “I’m 95% confident.”
Should you believe it?
LLMs can be completely wrong and still sound calm, structured, and sure of themselves. Confidence, as an agent reports it, is a style of writing as much as a measurement. VectorStep never trusts one number — it builds a trust vector from independent signals and gates every autonomous action on the weakest one. Not an average. The floor.
What you get on day one
Every gating signal on this page — verifier, grounding, calibration, readiness — is additive. None of it is required to run a production pipeline.
Triggers, steps and flow control are declarative config, versioned like everything else you own. No UI required to author or ship one.
Fan out over discovered work at runtime, run branches concurrently, and merge the results back into the next step.
Conditional steps, a per-step failure policy, and automatic retries — because real pipelines don't get to assume every step succeeds.
Trigger on a schedule as easily as on a webhook, and chain pipelines together without an external scheduler in the loop.
Route an approval to Telegram, Slack or Teams and the pipeline waits — no separate approvals system to bolt on.
Any LLM provider and MCP tools through the Gateway; webhooks in from any source, notifications out to any destination.
The reusable step library is why the config in the next section is a dozen lines instead of a hundred. Write the prompt template and the output contract once, then reference it with use: sre-investigationfrom any pipeline. An inline pipeline — the kind with the prompt template and the LLM output contract written out in full — gets long fast; a step-library pipeline stays readable in review.
Developer experience
Every pipeline and every step is YAML in version control. Changes go through the same code review as the rest of your codebase.
POST /reloador SIGHUPpicks up config changes without dropping a run in flight.
A dedicated validate endpoint checks a pipeline against the schema and never mutates state — wire it into CI before anything merges.
SQLite for zero-infra local development, Postgres for production — no branching logic, no surprises when you promote.
Prometheus metrics and OpenTelemetry tracing ship with the service — plug in your existing stack, nothing to bolt on.
Two MCP servers expose pipeline and agent authoring directly to an AI coding assistant — write, inspect and iterate on pipelines without leaving your editor.
The trust vector
What the agent says about its own work. The starting point — and the least trustworthy number in the whole system, treated accordingly.
A second agent reviews the reasoning — or independently redoes the task, blind. A verifier can lower confidence, never raise it. Consensus is not comfort.
Is every load-bearing claim backed by an actual tool call in the agent’s own trace? Plausible-sounding prose doesn’t count. Evidence does.
Real commands, real queries, real approvals — no model in the loop. One failed hard check forces trust to zero. No averaging. No partial credit.
Threshold 75% — the agent was “95% confident” the whole time.
Every step of this chain — with the exact numbers for that specific run — is visible in the run’s Trust panel under “How was this calculated?”. Nothing is a black box.
Calibration
VectorStep bins every marked outcome per agent, model, provider, prompt version and agent version, then measures what each confidence band was actually worth. If an agent says 90% but its history at that level is 67% — you’ll see it, in plain English, before it ever bites you.
sre-agent · claude-sonnet · confidence 90–100%
31 marked runs in this band
Config-driven
No SDK, no DSL, no redeploy. Pipelines, reusable steps, verifiers, grounding, calibration and hard checks are declarative config — versioned in git, hot-reloaded on change, validated before a byte is written.
name: alert-triage
description: Triage critical alerts and notify on completion
trigger:
source: alertmanager
match: { severity: critical }
steps:
- name: investigate
use: sre-investigation # reusable step library
executor: gateway
executor_config:
agent: sre-agent
confidence_threshold: 0.75
on_low_confidence: escalate # a human sees it insteadNo verifier, no grounding, no calibration, no readiness — a legitimate way to run this pipeline in production.
name: alert-triage-critical
description: Triage critical alerts, act only when trust survives every check
trigger:
source: alertmanager
match: { severity: critical }
steps:
- name: investigate
use: sre-investigation # reusable step library
executor: gateway
agent: sre-agent
confidence_threshold: 0.75
on_low_confidence: escalate # a human sees it instead
verifier:
mode: independent # blind second agent
combination_strategy: minimum
grounding:
enforce: true # claims need evidence
calibration:
enforce: true # trust the track record
on_uncalibrated: escalate
deterministic_checks:
- type: shell # still breaching, live?
command: ./checks/alert-still-firing.shThe whole safety story is YAML you can read in review, too.
Gated is Simple plus four opt-in blocks — verifier, grounding, calibration, deterministic checks. Nothing in Simple is removed to get there.
Operate with evidence
Per-step breakdowns by agent, model and provider — success rate, tokens, latency and judged accuracy, un-blended. “Did that prompt edit help?” has a diff and a number.
Owner-defined readiness criteria across four tiers — operational, confidence, accuracy, calibration — judged against real evidence, with a guided builder that previews any criteria change in ~300ms.
A marking queue of every step lacking human accuracy feedback — the exact labels calibration and readiness are built from. Live-tail any run while it happens.
From first webhook to a gated, calibrated production pipeline — the quick start takes about ten minutes.