The question every agent operation gets sooner or later: "why did it do that?". A customer got a wrong answer, an order was canceled, a charge came out at a strange amount. If the answer is a reconstruction by approximation, the project is at risk, not because of the error itself, but because of the inability to explain it.
Agent observability is the ability to answer that question with evidence. This text covers what has to be recorded for that to be possible, and what you measure with what was recorded.
Why application logging does not work
Traditional software is deterministic: the same input produces the same output, and the log records what happened because the "why" is in the code. An agent is not like that. The same input can follow different paths, and the path taken depends on the context assembled, the model used, the version of the instructions and the result of each tool along the way.
In an agent, the decision is the data. Recording only the input and the output is like keeping the question and the answer from an exam without keeping the working: you can see that it got it wrong, you cannot see where. And without knowing where, the fix becomes trial and error against a system that changes behavior every run.
The eleven items of the trail
An agent execution has to leave at least this on record. The order is the order in which things happen.
- The original input, as it arrived, before any processing.
- The context assembled: what was retrieved from memory and from the knowledge base, and what was sent to the model. It is where you discover the agent answered with an out-of-date document.
- The model and version used on that call. Providers change versions without warning, and the behavior changes with them.
- The version of the instructions (the prompt). Without it, there is no way to know whether the error came from a wording change made the week before.
- Every routing decision: which specialist received the request, and on what basis.
- Every tool call, with parameters and return. It is the most important item and the most frequently absent. "Looked up the order" is not enough; it has to be "looked up order 4471 and received status: canceled".
- Failures and retries: what failed, how many times it tried, what it did next.
- Tokens and cost per call. It is what enables cost per resolved case, rather than per call.
- Latency per step, not only total. The slow step is the one you optimize.
- The final output and who it went to.
- The handoff to a human, with the reason: out of scope, low confidence, customer request, risk.
The eleven together enable the property that decides everything: reproducibility. Given the record, it is possible to reconstruct the execution step by step and point to where the path diverged from what was expected. Without it, observability is a summary dashboard, and a summary dashboard does not explain a case.
What to measure with what was recorded
| Metric | What it reveals |
|---|---|
| Resolution without handoff, by type of case | What the agent actually takes out of the queue. It is the value metric. |
| Handoffs by reason | Out of scope calls for a change of scope; low confidence calls for a change of instructions; risk calls for a review of the rules. Each reason is a different fix. |
| Cost per resolved case | A cheap call that resolves nothing is the most expensive. Cost per call misleads. |
| Routing accuracy on the first try | If it falls, the problem is the supervisor, not the specialists. |
| Hops per execution | Reveals loops between agents before they show up on the invoice. |
| Tool failure rate, per tool | An unstable integrated system shows up here first. |
| Weekly variation in resolution | A slow decline is the sign that the catalog or the policy changed and the agent did not. It is the metric nobody looks at until it is late. |
The three uses of the trail
Debugging
A case went wrong. With the trail, you open the execution, see the context that was assembled, the tool that returned out-of-date data, the decision the model took from it. The fix is targeted: update the knowledge base, adjust the tool, change the instructions. Without the trail, the fix is rewriting the whole instruction set and hoping.
Auditing
The risk team asks why the agent approved those terms in August. The trail answers with the whole execution: who asked, what was looked up, which rule was applied, who approved. It is what turns "the model decided" into an acceptable answer, and it is a condition for operating an agent in a regulated sector. The trail, however, contains personal data and is itself processing; the post on governance and the LGPD covers what that requires.
Improving
A weekly review of a sample of executions, with the handoffs by reason up front. It is the cycle that keeps quality up when the process changes, and it is the work that needs an owner after go-live. An agent without that cycle degrades quietly, and the trail is what makes the review possible instead of it being an aimless read of conversations.
Three common mistakes
- Recording the output and not the path. It is the pattern of whoever adapts application logging. It shows that it got things wrong, and hides where.
- A dashboard with no executions. A chart of volume and average rate, with no ability to open a specific case. It serves a report, not an operation.
- A trail with no defined retention. Keeping everything forever creates a store of personal data with no inventory; deleting too early removes the evidence when the audit arrives. The period is a decision, and it has to be taken before go-live.
How Runflow records it
On the platform, every execution, decision and tool call is recorded and reproducible: you audit the past and debug the present from the same trail. Defining tools with types in the SDK is what allows parameters and returns to be recorded with structure, rather than as loose text. Cost is visible by team and by user, and the Metrics dashboards are built in code, on the same foundation. When you ask for a demo, ask to see a real execution opened up, with its tool calls: it is the test that separates a trail from a dashboard.
