Observability is the product: why agents without traces don't survive a compliance review

A compliance reviewer opens a prior-authorization denial from last quarter and asks one question: why was this request denied?

The agent that handled it has had two prompt updates and a model upgrade since then. The clinician who signed off has moved to another team. The answer has to come from what the system wrote down at the time. If what it wrote down was application logs, the answer is a list of timestamps and API calls. If it wrote down a decision record, the answer is the policy version the agent read, the criteria it checked, which of them it could confirm, and who approved the outcome.

The title of this piece claims that observability is the product. In regulated work I think that is close to literally true. An agent that gets most cases right but can't account for a particular one is hard to approve, and hard to keep in production once a regulator, an auditor or a complainant starts asking. But the claim need to be defined precisely, because "observability" usually means a tracing dashboard, and a dashboard is not what the reviewer is asking for.

What a reviewer asks

The questions are much the same in insurance, life sciences and banking. What did the agent see, and which versions of those records? Which rule did it apply, and which version of the rule? Which tools did it call, with what inputs, and what came back? Why did it reach this outcome: which criteria were met, which weren't, and on what evidence? Who reviewed, edited or overrode the result, and when? And what produced it: which model, which prompt, which versions of the tools? Behind all of these sits a question about the system rather than the case: how do you know it works, and has that changed since it was approved?

Current compliance rules ask for versions of the same things, though few of them prescribe a format. Let's take couple of examples. Since January 2026, CMS-0057-F has required Medicare Advantage, Medicaid, CHIP and certain marketplace plans to give a specific reason for every prior-authorization denial, within 72 hours for urgent requests and seven calendar days for standard ones. In lending, CFPB Circular 2022-03 states that creditors must give specific and accurate reasons for adverse action even when a decision relies on complex algorithms, and that being unable to explain the model doesn't excuse the requirement.

The EU AI Act covers the system side. Article 12 requires high-risk systems to record events automatically over their lifetime, with deployers keeping the logs for at least six months. Article 14 requires that the people overseeing a system can interpret its output, stay alert to the tendency to over-rely on it, and decide to disregard or reverse it. In US insurance, the NAIC Model Bulletin, latest adopted jurisdiction map here, expects insurers to validate and retest their AI systems and to produce documentation during a market conduct examination.

Read side by side, these rules ask for two things: a specific, accurate reason for each decision that matters, and evidence that the system is tested and monitored over time. Neither can be reconstructed afterwards from a running system.

A log vs. a decision record

Most agentic AI teams already collect something they call observability. LangChain's 2025 survey of agent builders found that almost nine in ten had some form of it. What they usually have is operational telemetry: every model call and tool call, with timings, token counts and errors. That is valuable for debugging and for cost. Our work for a global IT-services and consulting firm was largely in this layer, distributed tracing tied to cloud-cost governance, and it is where questions about latency and spend get answered.

Telemetry is high volume, so teams sample it, and there lies the problem. If the evidence a reviewer needs lives in the same pipeline, sampling skips the audit evidence along with debugging data. A decision record is a different object. It is small and structured, so keeping one for every case costs little, and it answers a different question: what the agent decided and on what basis, as opposed to how long each step took. It belongs in its own append-only store, linked to the telemetry by a shared trace ID, with retention set by regulation and legal hold instead of by the defaults of a logging tool.

The same split is reflected in the tooling. OpenTelemetry's conventions for generative AI describe an agent run as a tree of spans (the agent invocation, each model call, each tool call), and they are a sound basis for telemetry. As of mid-2026, every attribute in them was still at development status, and they define no fields for the rule applied, the criteria outcomes or the human sign-off. Those belong in a schema the organization defines and keeps.

For example, this is what one looks like, for a specialty-medication prior authorization:

• Saw: policy v3.2, four clinical criteria, member history
• Applied: formulary rule 12.3, coverage threshold
• Why: three of four criteria met, one unclear, below threshold
• Signed: clinician reviewed and approved

The example is from what we show on our trust page.

Note that each line above answers one of the reviewer's questions. The "why" is a list of criteria with an outcome for each, which is also what CMS-0057-F asks a health plan to state when it denies a request. And the unclear criterion is the reason the case reached a clinician at all: the agent's verify step couldn't confirm it, so the agent routed the case for review instead of deciding it. The provenance record in the prior-authorization system we built works this way, with every decision carrying what was read, which rule applied, and why.

The same record is visible when the agent is running. An operator looking at that case on the day sees the request, the agent's stated intent (assess eligibility against formulary policy), its plan (retrieve the policy, check the criteria, verify), the tools it called (the formulary API and the clinical guidelines), the verification result (three criteria met, one flagged), and the review status (clinical reviewer notified). Two of the AI Experience patterns we design for, planning visibility and tool-use disclosure, are this record shown at the moment it is written. The operator on the day and the reviewer in last quarter, are reading the same thing at different times, which is the point: there is no second, reconstructed account to reconcile with the first.

Where the "why" comes from

The obvious way to fill the "why" field is to ask the model to explain its reasoning and store whatever it says. This is another form of post-hoc rationalization bias, where the model generates plausible-sounding excuses for its decision after the fact. That answer is not reliable evidence of how the decision was made.

Anthropic tested this directly in 2025. Researchers gave reasoning models questions together with a hint pointing to an answer. They then took the cases where the hint changed the model's answer and checked whether the model's written reasoning mentioned using it. Claude 3.7 Sonnet mentioned the hint about 25 percent of the time on average, and DeepSeek R1 about 39 percent. The researchers concluded that monitoring a model's chain of thought cannot yet reliably rule out behavior you would want to catch.

For a regulated decision, that makes it clear where the reason has to come from. The reason has to come from the structure the agent works within, and not from the account the model gives of itself. Each criterion is checked separately, with a recorded outcome (met, not met, unclear) and a pointer to the evidence. Anything that can be computed deterministically, such as a threshold, a date rule or a formulary lookup, is computed by a tool, and the result is recorded. The model's job is extraction, mapping, and judgment on the parts that genuinely need it. A separate verify step checks the outcomes against the evidence, and whatever it can't confirm goes to a person. A model-written summary can still sit in the record to help a reviewer read the case, as long as it is labelled as generated text and not as the basis of the decision.

A reason built that way, specific and traceable, is what both the CMS rule and the CFPB circular ask for.

What produced the decision

Out of all the reviewer's questions, "what produced this decision?" is the one I see answered worst.

An agent's behavior depends on much more than the model: the system prompt, the tool definitions, the retrieval index, the policy documents, the rule set. A one-line prompt edit can change outcomes as much as a model upgrade does. So each of those needs a version identifier, and each decision record needs to carry the versions in force for that case. Every change should go live only after the regression suite passes, with the results and the approver stored against the release. That chain answers the system question, "has it changed since approval?", for every change made since.

There is a further wrinkle. Providers retire model versions on published schedules, and a decision made on a retired model can't be replayed to show what happened. The record therefore has to hold enough to explain the outcome without a replay: references to the inputs, the passages retrieved, the tool results and the outcome of each criterion.

What a trace cannot show

So far this has been an argument for writing more down. It is worth highlighting what writing things down doesn't do.

A complete record of a wrong decision is still a wrong decision. Traces show what happened. Only evaluation shows whether it was right, and that needs a labelled set of real cases, run on every change, with the results kept. The LangChain survey that found nine in ten teams with observability also found only about half running offline evaluations. If observability only means dashboards, the claim in the title of this post doesn't hold. What a regulated buyer is paying for is a correct decision that can be defended, and the records are how the two parts get shown.

Automated grading needs monitoring too. Many teams use a model to grade agent outputs (LLM-as-a-judge). The study that popularized the approach found that GPT-4's judgments agreed with human preferences more than 80 percent of the time, about as often as humans agree with each other. The same study documented biases toward whichever answer was shown first, toward longer answers, and toward answers resembling the judge's own. A model grader needs periodic checks against a human-labelled sample, and the grader's own version should be in the configuration record.

A logged approval is not proof of oversight. The EU AI Act names over-reliance on system output as something human overseers must be able to counter, and with reason. If clinicians approve nearly every case the agent sends them, a complete log of approvals shows that people clicked. This is what we are calling reviewer fatigue now. Capturing the override rate, the time spent on each review, and agreement on seeded cases with known answers help us alleviate that. An override rate near zero for several months is a reason to look harder.

Traces also hold sensitive data. Prompts and tool results routinely contain health, financial and personal information, which is why OpenTelemetry has made capturing prompt and completion content opt-in. Retention duties pull one way and data minimization pulls the other. So the decision record should reference source records held in systems that are already governed, redact what it can, sit behind its own access controls, and have its retention agreed with legal and compliance before go-live.

The standards have some catchup to do. The OpenTelemetry conventions aren't stable yet, and the European technical standards for logging are in draft. Building to an internal schema mapped onto OpenTelemetry, with a migration expected later, is reasonable path today. Waiting for a final standard isn't.

A test worth running

There is a simple way to find out where a program stands. Pick a case at random from three months ago. Ask someone outside the build team to reconstruct the decision from the records alone: what the agent saw, which rule it applied, why it reached the outcome, who signed off, and which configuration was running. If that takes days, or needs an engineer to query three systems, the records won't hold up when a reviewer asks. Adding them after go-live means changing how the agent calls tools, reports criteria and gets released, at the moment when everyone wants it finished. That is a large part of why the piece on stalled pilots argues for building the decision record and the eval suite during the pilot.

So pick the case from last quarter. Who in your organization could tell you why it was decided, using only what the system wrote down at the time?

If you are preparing agents for a compliance review, talk to us and bring a case. We'll walk through what its record would need to show.