VectorStep | Blog

How I think about agent design

Mon, 07 Sep 2026 00:00:00 GMT

When I first started tinkering with agents my first instinct was to keep making the agent more capable. I’d give it more tools, let it handle more of the job, trust it to figure out the rest. That instinct is probably right for an agent you interact with all the time but the more I thought about it, the more I felt that was the wrong way to look at it for deliberate, repeatable production workflows.

One prompt, then five

My first pass at the alert-triage workflow that eventually became the samples in this repo was, predictably, one prompt. Look up the service doc, raise the ticket, query the metrics, work out a probable cause, propose a remediation, mark your own confidence - all of it, one agent, one soul.md trying to be a first-responder, an SRE and a reviewer at the same time.

It sort of worked. For a production workflow it also fell over in exactly the way you’d expect. The confidence number meant something different every time depending on which part of the job the model happened to be doing when it wrote it, so gating on it was close to useless. Splitting the same job into five smaller steps - triage, investigate, verify, decide, act - wasn’t a refactor I did after finding problems. It’s the thing that made the whole approach work at all, almost immediately. I didn’t have to fix the giant prompt. I just had to stop asking one agent to be five people.

That’s not really an accident, and it’s not really a discovery either. Once I stopped trying to design the “right” agent architecture from scratch and started asking how this job already gets done by people, most of the decisions made themselves.

Everyone’s already doing this, just not with LLMs

So how does an incident actually get triaged today, on a team that is already doing this without an LLM in the process?

A senior engineer doesn’t open a raw alert and start hand-typing service context into a Jira ticket. That’s not what they’re there for, and it would be a waste of what they are there for. There’s a first-line function - call it L1, call it whoever’s on the on-call rota this week - whose entire job is to look at the alert, pull up the service documentation, and raise a ticket with enough context that a senior engineer isn’t starting from a blank page.

And a senior SRE doesn’t sit refreshing a Jira board waiting to write problem reports off the back of raw pages either. They get handed a ticket that already has a shape to it - a service, a summary, a link to the doc - and their job starts from there. They query the metrics, work out what’s actually going on, decide whether this needs someone even more senior.

Once you start thinking like that, splitting a pipeline into first-line-triage and sre-investigation isn’t a design decision I had to invent. It’s just how this already works, everywhere I’ve ever been on-call. The process already exists. I didn’t design it - I just started copying it.

The same is true one level up. Nobody senior double-checks every single ticket L1 raises - that would defeat the point of having an L1. But for the things where being wrong actually costs something, someone more senior generally reviews before it goes further ad this is true of any business process, not just in software engineering or the IT world. It’s not a special AI safety mechanism, that’s just an org chart. In VectorStep, and our sameple use-case it shows up as a principal-sre agent sitting in the verifier slot on a step - sometimes triggered conditionally, sometimes wired to fire on every run when the stakes justify it - but the shape of the idea predates the software by decades.

This is the bit I’d want a reader to take away more than any specific YAML field. You don’t need to invent an organisational structure for your agents. Look at how the humans doing this job today are already organised - who’s allowed to decide what, who checks whose work, who escalates to whom and build the pipeline to match. The org chart is the spec. It’s a lot easier to draw a process that already exists than to guess at a new one.

Grant only the tools the job needs

That same division of labour is exactly why tools: in agent.yaml matters as much as it does. L1 has access to Confluence and Jira. They don’t have a Grafana login, and they definitely don’t have a way to push a change to production - not because anyone doubts their competence, but because it’s not their job, and giving them that access wouldn’t make them faster at the job they do have. The senior SRE can query Grafana and comment on the ticket, but they don’t get to skip the ticket and go straight to remediation off their own initiative in most places I’ve worked in.

An agent’s tool list is the same thing, just enforced by the gateway instead of by an access request form (and a long drawnt out manual human process getting in the way). first-line-triage can reach Confluence and Jira and nothing else. sre-investigation can reach Grafana and the ticket, and nothing else. Neither can reach anything genuinely dangerous, so whatever the model’s reasoning does under pressure - or under a prompt injection buried in a tool result - there’s a hard ceiling on how much damage it can do. That’s not a courtesy. It’s the actual security boundary, and it falls out for free once you’ve split the job the way the humans already split it.

Confidence has to mean one thing

A triage agent’s sense of “how bad does this look” isn’t the same signal as an SRE’s sense of “how confident am I in this probable cause,” and neither of those is “how completely did I do my job” which is the only one of the three that’s actually safe to gate a pipeline on. Bundle triage, investigation and remediation into one agent and its confidence score ends up quietly averaging across three different questions. Split them, and each agent’s confidence has to answer exactly one question, because it’s the only question that agent ever gets asked. Calibration only works once that’s true - see how confidence and calibration work for the mechanics.

Worth establishing now, while it’s still cheap

There’s a cost argument for all of this too, and I think it’s really important to think about this now before (hopefully!) it becomes an obviously good idea rather than a forward-looking one.

Token costs right now are, relatively speaking, a rounding error for most of the workloads I’m describing. That’s a honeymoon, not a permanent state of affairs. Anyone who was around for the early days of cloud compute has seen this before. “Just call the biggest model, it’s cheap enough not to matter” is actually a fairly reasonable thing to say today and a completely unreasonable habit to have baked into your architecture in two years’ time, once someone in finance starts asking how much we’re spending on AI (I am already seeing these conversations coming up).

Splitting a workflow into steps is what makes it possible to do anything about that at all. A first-line-triage step that runs two hundred times a day looking up a doc and raising a ticket doesn’t need the same model as a principal-sre verifier whose entire job is catching the primary’s mistakes on the handful of runs where it’s worth paying for a second, genuinely independent opinion. That’s not a distinction you can make once everything is one prompt and one model. It’s a direct consequence of having five smaller jobs instead of one big one, each cheap enough to reason about in isolation, including reasoning about what it should cost. See cost accounting for what that actually looks like once you want real numbers instead of intuition.

I’d rather build the habit of matching the model to the job while it barely matters than wait until it very much does.

The practical version

The engineering checklist version of this post lives at Writing good agents — this post is the “why,” that page is the “how.”

Why I built VectorStep

Mon, 24 Aug 2026 00:00:00 GMT

I arrived unfashionably late to the OpenClaw party around March this year. When I started reading about implementing a personal AI assistant I couldn’t necessarily see what I might need a fleet of agents for in my personal life. I have no interest in having AI check-in for flights (I don’t go on many flights these days) or produce a morning briefing (what would AI brief me on?) but I have for many years tried to find ingenious ways to try and make my weekend hobby of sports betting profitable by delving into stats using tools such as Splunk and streaming odds into grafana cloud and analysing from there.

What interested me wasn’t really the betting itself, but the idea of building a system that could continuously watch a bunch of data, spot something interesting and then help me decide what to do with it.

So it was this use-case that got me tinkering with Openclaw. I liked the idea of agents just continually keeping context and essentially getting to know you and keeping a memory. But in practice, in my opinion, memory management and the idea that your personal agents just “remember” and “know” stuff about you is still some way off.

But what tinkering with these tools and features did do is cement my thinking on how AI could be useful in my working life.

I have worked in observability for an awfully long time and I have heard the term AIOps used so often and for so long i’ve almost become immune to anything an article says after it. Quite frankly, AIOps used to mean some sort of automation and at best some machine learning. At a stretch you could tag that “Artificial Intelligence”, but not as we know it today with LLMs.

Things have moved on though, and genuine alert triage and remediation is possible today. There are loads of tools already out there that do just that.

Most of my career has been spent in highly regulated industries like banking, insurance and medicine. Using AI is becoming more prevalent for engineers in their every day workflow (copilot etc.) but is still fairly scarce when moving into production and asking AI to work completely autonomously.

That got me thinking about something slightly broader than observability. If AI agents were going to become genuinely useful at work, I didn’t think they should necessarily be limited to one particular use case. An incident response workflow is one example, but there are potentially thousands of workflows across a company where you might want an agent to gather information, make a decision, take an action and then move on to the next step.

At the time, I was also a bit naive about what tooling already existed. I knew there were plenty of workflow engines and orchestrators, and obviously there were already tools for building AI agents, but I wasn’t really seeing the combination I was thinking about. Something that treated an AI workflow as a series of explicit steps, where each step could have its own inputs, outputs, checks and rules around whether the workflow should be allowed to continue.

I also wanted it to feel like something an engineer would actually want to use. I’m a big fan of configuration and infrastructure as code, so I didn’t want to build something where the important parts of the workflow lived inside a visual editor that was difficult to version, review, reproduce or understand.

I wanted to be able to look at a pipeline in a repository, understand what it was going to do, see the decisions it could make and check it into source control like any other piece of software.

And that got me thinking about (yes, I do a lot of thinking about this sort of thing!) a more fundamental question.

If we are going to break an AI-driven workflow down into explicit steps, what should determine whether it is allowed to move from one step to the next?

So, how do we gate those steps?

This is essentially the question that led me to VectorStep.

Disclaimer: the original tool was called “PORK”, for Prompt Orchestrator. I still love that name, but sensible me thought VectorStep sounded more professional!

I started thinking about each step in an AI workflow as something that shouldn’t automatically be trusted just because an LLM had produced an answer. If a step is going to trigger another step, call an API, make a recommendation or ultimately take some action, there should be some way of deciding whether the output is good enough to proceed.

I originally just asked the LLM to report back its own confidence on a task and, in my initial testing, that worked surprisingly well, but I should add here, I think that’s because I was very deliberate about the scope of the step and the very specific instructions for the agent (prompt) and its very limited soul (essentially what you tell your agent it can/can’t do).

For now, i’ll leave that subject there but I do have some further articles and guides on this subject in the docs site so please have a look if you want to know more about writing good agents and prompts.

But asking an LLM to mark its own homework is fraught with risk, as you can imagine, so this mechanism evolved. In some cases you might want independent verification. In others you might need evidence from a known source, a deterministic check, or some understanding of how well the agent has performed historically.

I layered all these checks and balances into what I call the “Truth Vector” (is that woefully corny?).

So VectorStep is my attempt to build all of that into the workflow itself.

What actually makes up the Truth Vector?

The first version was very simple. The agent did a piece of work and told me how confident it was in the result. That was useful, but I didn’t want to rely on the agent marking its own homework, so I started looking at what other signals I could use.

The Truth Vector is currently made up of five signals. They each answer a slightly different question about the output of a step.

Confidence - how confident is the agent in the result it has produced?

Verification - can another process independently check the result?

Grounding - can the important claims in the result be backed up by actual evidence?

Deterministic checks - does the output pass rules that don’t require another LLM to make a judgement?

Calibration - does this particular agent’s confidence actually bear any relationship to how often it is right?

In my opinion, I don’t think any one of these is necessarily enough on its own.

An agent can be highly confident and wrong. A verifier can be wrong. Evidence can be incomplete or misleading. A deterministic check can tell you that something meets a particular rule without telling you whether the overall answer makes sense.

The idea is that these signals give me different ways of looking at the same decision.

And importantly, I’m not trying to turn all of this into one magic number that says “the AI is 87% trustworthy”. That would just move the problem somewhere else.

You are the weakest link. Goodbye!

One of the decisions I made fairly early on was that I didn’t want to simply average these signals.

Imagine an agent reports 95% confidence. The verifier is happy. The grounding looks good. But a deterministic check that is critical to the particular action fails.

I don’t think it makes sense to say that the first three signals were good enough to outweigh the one that failed.

If that check is important enough to be a requirement for the next step, the workflow should stop.

This is why I tend to think of the Truth Vector less as a score and more as a collection of evidence about whether a step has earned the right to continue.

And that last bit is important because the answer isn’t always the same.

Not every action deserves the same level of trust

I don’t think there is a single threshold at which an AI becomes “trusted”.

If an agent is writing a draft for me to review, I’m happy to accept a very different level of uncertainty than I would if it was about to make a change to a production system.

The same is true across pretty much any business.

An agent might be allowed to categorise an incoming support ticket with relatively little scrutiny. Asking it to send a customer an email without human approval is a different proposition. Asking it to change something in production is different again.

So rather than asking:

“Is this AI trustworthy?”

it’s probably better to ask:

“Has this particular step produced enough evidence for the action we are about to take?”

It belongs to the step, the evidence available to that step and the consequences of allowing it to continue.

And this is where I think treating an AI workflow as a series of explicit steps becomes particularly useful.

An example

Take a fairly simple example from the sort of work I’ve spent most of my career around: an agent investigating a production alert.

The first step might gather information from several sources and produce a triage report.

Without any gating, that’s all we’re doing. The agent investigates and gives us an answer.

Now we can start adding some gates.

Perhaps the agent needs to have a confidence above a certain level.

Then we might independently verify some of its conclusions.

We might require the important claims in the report to be backed by evidence from the systems it has queried.

We might have a deterministic check that says the proposed remediation is only valid for a particular type of alert.

And, once we’ve accumulated enough historical runs, we can start asking whether the confidence the agent reports is actually useful.

At each point, the workflow has more information available when deciding whether to continue.

And if one of the things we’ve decided is essential fails, the workflow doesn’t need to pretend everything is fine. It can stop, retry, or escalate to a human.

That, to me, is much more interesting than simply asking an LLM to give me an answer and then deciding whether I happen to like the answer.

So what actually is VectorStep?

This is where VectorStep started to take shape.

I wanted a way to describe an AI workflow as a series of explicit, version-controlled steps, while also making the conditions for progressing between those steps explicit.

The AI can do the work. It can use tools, gather information, reason over it and produce an output. But the workflow gets to decide what happens next.

Sometimes that will be another automated step.

Sometimes it will be a retry.

Sometimes it will be a human.

And sometimes the correct decision is simply to stop.

VectorStep is my attempt to make those decisions a first-class part of the workflow rather than something that gets bolted on afterwards.

It’s free to use, it’s designed to be engineer-friendly and the workflows are defined as code. I’m deliberately trying to make the whole thing inspectable and reproducible rather than hiding the important decisions inside a visual workflow builder.

I don’t think this solves “AI trust”, i’m not sure that’s possible right now.

I don’t think VectorStep can tell you whether an AI agent is telling the truth with absolute certainty, that’s a much harder problem because a verifier can be wrong, a source can be wrong. An agent can find convincing evidence for the wrong conclusion. And no amount of configuration can remove uncertainty from a system that is ultimately making decisions in a changing environment.

What I think we can try and do though, is make that uncertainty much more explicit.

Instead of an agent simply producing an answer and the system assuming it is good enough, we can ask what evidence we have for that answer, what checks it has passed, how the agent has performed historically and whether that is sufficient for the action we are about to allow.

That’s the idea behind VectorStep.

Don’t blindly trust the agent. Give it a way to earn the right to take the next step.

I’m going to be exploring that idea through the project, and I’m sure some of my assumptions will turn out to be wrong along the way.

I’m particularly interested in hearing from people who are trying to put agents into real production workflows, especially where the consequences of getting the decision wrong actually matter.

If that’s the kind of project you’d want to poke at and try to break, it’s free to install and the docs are at vectorstep.io.