Agent observability: how do you know the agent was right?

Every review of an agent-written number ends up in the same place. Fine, it ran. How do you know it's right?

Not how do you know it ran. You can see that it ran. There's a trace, a token count, a latency number and a green check next to all of it.

We'll cover what that tracing actually buys you, the three ways the answer goes wrong anyway, and four things you can do to get to a number somebody else can check.

What a trace actually tells you

Agent observability took its vocabulary from application monitoring and most of it moved over cleanly. OpenTelemetry keeps a set of GenAI semantic conventions for exactly this, covering "spans, metrics, and events for GenAI clients, MCP (Model Context Protocol), and provider-specific conventions". You get gen_ai.operation.name to tell a chat apart from a tool call or an agent invocation, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens for what the run cost you, and error.type when something throws. PostHog, Datadog, Splunk, and Langfuse all sell a good version of this.

It answers what happened, how long it took, what it cost, and where it broke.

It doesn't answer whether the number was right, and that isn't a gap in the tooling. It's a different question. A run that reads the wrong table, joins on the wrong key and returns a confident number produces a completely clean trace. Every span succeeds, latency is fine, cost is fine, and the number is wrong.

What actually goes wrong

The definition changes between runs

Ask an agent how many active users you had last month, on Monday and then again on Thursday, and it writes the SQL fresh both times. Active can mean signed in, performed an action, or held a session longer than thirty seconds. The agent picks one. It doesn't tell you which, and there's nothing making it pick the same one on Thursday.

You get two answers that both look right. They're answers to two different questions.

There's nothing behind the number

The number pops up in somebody's chat window. There's no row set anyone can open, no query anyone can read, and nothing the person you forward it to can check for themselves. Six weeks later you can't reconstruct it, because the session is gone and the data has moved on.

That's fine while you're exploring. It stops being fine the moment the number goes in front of somebody who wasn't in the session.

Nobody is reviewing the scheduled runs

This is the one that actually costs you. The whole reason to put an agent on a schedule is so that you stop looking at it. A run at 6am on a Tuesday has no reviewer. If it starts returning a wrong number in week three, the problem isn't the wrong number. It's the three weeks.

How to know the answer was right

Store the query instead of regenerating it

Two of those three problems have the same fix. Write the query once, check it, store it, and have the agent read the stored result instead of composing new SQL every time somebody asks.

That's what gives you provenance. The definition is the record of what was asked and the rows are the record of what came back, so "where did this number come from" has an answer that isn't "the agent wrote something on Monday".

With Dreambase: A dataset is a stored, named query plus its materialized result. Build it as a scratch dataset first and check the numbers against something you already trust, because scratch is the cheap place to be wrong. Then promote it, which pins the definition and makes it durable. refresh_dataset replays that same definition and replaces the rows, so the thing that changes between two runs is the data and never the question.

Ask for a value, not a paragraph

Most agent answers arrive as prose, and prose is the part you can't check. "Signups look healthy this week" isn't a fact. A number, a category or a boolean is a fact, and a fact can be put next to the last one.

The model layer is moving this way too. TypeSafe AI shipped Jev in September 2026, out of Diogo Almeida, who helped build ChatGPT at OpenAI. It's a System One model and it doesn't write text at all. Their docs describe the output as "typed values and probability distributions that your code can branch on, sort by, and route with", built around "No text generation, no parsing". Because the possible outputs are ones you defined up front, there's nothing for it to invent. Vercel measured it 5 to 18 times faster than the OpenAI model they compared it against, with better accuracy.

You don't have to use Jev to take the point. Ask for a typed value with a confidence attached, not a sentence with the confidence baked into the tone.

With Dreambase: A metric snapshot comes back as a value and a change, not a paragraph about the value. That's what makes it comparable next week without anyone rereading it.

Keep every run so you can compare them

Most wrong answers aren't obviously wrong. They're plausible, which is exactly why they survive the meeting. You catch a plausible wrong answer by putting it next to the one before it. If active users drop 40% on a Tuesday, either something happened or the definition changed, and one number on its own can't tell you which.

With Dreambase: Every durable dataset gets aggregates, and that's where history over time lives. So the check is mechanical rather than a judgement call: pull this period, pull the last four, look at the size of the move.

Label what the query means

Storing the query tells you what ran. It doesn't tell you what it meant. You still end up with three datasets that all count users in slightly different ways and no record of which one is the one finance uses.

That's the job a semantic layer does for analytics. A label sitting over a stored query, so a metric has a name and that name has exactly one definition underneath it.

With Dreambase: Top level metrics sit on top of datasets. You label a query over a dataset, so the metric carries its definition and the place its data is actually stored in the same object. The dataset already gives you the provenance. The metric gives it a name you can use anywhere you work and it always means the same in the underlying query.