Observability for AI

The situation

At 3 a.m., the incident agent got a latency alert on the payments service. It read logs, checked metrics, and restarted the catalog service. The alert kept firing. The on-call engineer woke up to two problems instead of one.

At the next day’s review there was a single question: why did the agent do that? The application log said “restart requested by agent” and nothing else. There was no way to know what the model read, which tool returned what, or which prompt version was running.

In regular software, a stack trace tells the story of an error. In an AI system, the decision comes from what the model saw in its context. If you did not store what it saw, the decision has no explanation. And without an explanation, the only possible fix is a guess: change the prompt, change the model, hope.

The concept

Observability for AI means recording each run so you can answer, afterwards: what went in, what the model decided, what the tools did, how much it cost, and how long it took. The basic unit is the trace (the record of a whole run), divided into spans (the pieces of that run).

flowchart TD
  T(["1. Trace: one task"]):::io --> S1(["2. Span: model call"]):::llm
  S1 --> S2["3. Span: tool"]:::tool
  T --> S3(["2. Span: next model call"]):::llm
  S1 --> A["4. Attributes: prompt, version, tokens, cost"]:::tool
  S2 --> B["4. Attributes: arguments, result, error"]:::tool
  T --> D[("5. Trace storage")]:::store
  D --> Q{"6. Search and analysis"}

1. A trace covers one task from start to finish: a message answered, a document processed, an investigation.
2. Each model call becomes a span, with a start, an end, and a parent.
3. Each tool execution becomes another span, a child of the call that requested the tool.
4. Attributes keep what matters for explaining the decision.
5. Traces go to storage that lets you search by user, by error, by cost.
6. With that, you find the bad run, open it, and read what happened, step by step.

The same incident, with a trace

Go back to 3 a.m. Suppose the agent had tracing. Opening the run, you would see something like this (illustrative example):

trace incident-7f3a                          total 48s   cost US$0.21
├─ llm   decide_next_step   prompt=sre@12   in=3,104  out=88
│  └─ tool fetch_logs   args={"service": "payments", "limit": 50}
│        result: 50 lines, all "timeout calling catalog:8080"
├─ llm   decide_next_step   prompt=sre@12   in=6,877  out=41
│  └─ tool restart_service   args={"service": "catalog"}   result: ok
└─ llm   final_answer   "Restarted catalog, likely cause of the timeouts."

Now the decision has an explanation. The log tool returned only the last 50 lines, and all of them mentioned the catalog. The model made a reasonable inference from what it saw. The defect is not in the model: it is in the 50-line limit and in restart_service running without approval. You leave the review with two concrete fixes instead of “improve the prompt”.

Check +20 XP on first tryIn an agent's trace, the span for the fetch_logs tool sits loose, directly under the trace, instead of under the model call that requested the search. What information do you lose?

The tool's token cost

Which model decision led to that call

The tool's latency

What to store on each span

Span	Minimum attributes
Model call	model, prompt version, input messages, output, input and output tokens, cache tokens, latency, stop reason
Tool	name, arguments, result (or a summary), error, latency
Decision in code	chosen route, confidence, limit reached
Human	who approved or rejected, when, what was shown

The stop reason looks like a detail and is not. An answer cut off by the token limit (max_tokens) and a complete answer can look the same in the application log. In the span, the difference is written down.

Logs, metrics, and traces

• Metrics answer “is everything fine?”: cost per hour, error rate, p95 latency.
• Traces answer “what happened in this run?”.
• Logs hold loose events. In AI systems, almost everything that matters fits better as a span attribute.

The three complement each other. The cost-per-hour metric fires the alert. The filter “traces costing more than US$1” shows which runs pulled the average up. Opening one shows why.

The industry standard for this is OpenTelemetry, which has its own semantic conventions for generative AI (gen_ai.* attributes). Those conventions are still marked as in development, so attribute names may change between versions. Several LLM observability tools read this format.

Check +20 XP on first tryYour LLM provider bill went up 40% this week. Which investigation order uses each tool for what it does best?

Read the application logs line by line until something looks odd

Open random traces from the day and look for the problem

Check the metric for when and where cost rose, filter the most expensive traces, and open a few

Common mistakes

• Storing only the final answer. It is like keeping only the last line of a stack trace. The error is almost always in a middle step.
• Sampling 1% of traces. That works for web traffic, where requests look alike. In agents, the rare expensive run is exactly the one you need. Keep every run that errored, hit a limit, or cost the most.
• Forgetting the prompt version. Without it, you cannot tell whether yesterday’s complaint came from the old prompt or the new one.
• Truncating tool results without saying so. If the span shows a summary, mark it as a summary. Otherwise you will investigate data the model never saw in that form.

Privacy

Traces store the text the user sent. That includes personal data: national ID numbers, addresses, purchase history, sometimes health data. Decide who can open traces, how long they are kept, and what gets masked before storage.

Mask before writing, not after. A trace with a raw ID number (in Brazil, the CPF), kept for a year and readable by anyone on the team, is a privacy-law problem waiting to happen, whether the law is GDPR or Brazil’s LGPD. Replace the value with a placeholder ([CPF]) or with a hash when you need to correlate runs from the same user.

Try it

This example creates a minimal trace with OpenTelemetry and prints it to the terminal. Install opentelemetry-sdk and anthropic.

import anthropic
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("ai-guru.demo")

client = anthropic.Anthropic()
MODEL = "claude-haiku-4-5"
PROMPT_VERSION = "classifier@3"

with tracer.start_as_current_span("handle_message") as root:
    root.set_attribute("user.id", "u-123")
    with tracer.start_as_current_span(f"chat {MODEL}") as span:
        reply = client.messages.create(
            model=MODEL, max_tokens=50,
            system="Classify as: billing, order, or other.",
            messages=[{"role": "user", "content": "I was charged twice"}],
        )
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.request.model", MODEL)
        span.set_attribute("prompt.version", PROMPT_VERSION)
        span.set_attribute("gen_ai.usage.input_tokens", reply.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", reply.usage.output_tokens)
        span.set_attribute("gen_ai.response.finish_reasons", [reply.stop_reason])
        span.set_attribute("app.output", reply.content[0].text)

Run it and read the JSON it prints. Notice three things:

1. The chat claude-haiku-4-5 span has a parent_id equal to the span_id of handle_message.
2. Both spans share the same trace_id. That is what ties all the steps of one task together.
3. prompt.version is there. Change it to classifier@4, run again, and see how easy it is to split runs by version.

Now picture a task with 15 spans like this, and a trace opened on a screen that shows the whole tree. That is what was missing at the review at the top of this page. In production, replace ConsoleSpanExporter with an OTLP exporter that ships spans to your observability tool.

What this changes

• Without traces, no smell in this catalog can be diagnosed with certainty. The smell of having no traces is No Tracing, and the refactoring is Add Tracing.
• Every pattern page has an “In production” section, with the spans that should exist and the suggested alerts.
• The prompt version must be on the span. Without it you cannot reproduce the answer behind a complaint. See Unversioned Prompt.
• Production traces are the best source of eval cases. A bad run becomes a case in the Golden Dataset.
• When the system decides about money, data, or people, the trace becomes evidence. The pattern that handles that is the Audit Log.

Check your understanding

Check +20 XP on first tryA customer complains that the support bot gave a wrong delivery estimate yesterday at 2 p.m. Your system keeps a trace of every run, but without the prompt version. The team changed the prompt at 1 p.m. What can you not establish?

Which tools the bot called in that conversation

Whether the wrong answer came from the old prompt or the new one

How much that conversation cost

Check +20 XP on first tryTo save storage, your team proposes keeping only 1% of traces, picked at random. What is the problem with that in a system with agents?

None, because 1% is already a statistically valid sample

Rare runs with errors or high cost almost never make it into the sample

OpenTelemetry does not allow sampling

Check +20 XP on first tryA finance assistant stores full customer messages in its traces, with ID numbers and balances, forever, readable by the whole engineering team. What do you change first?

Mask personal data before writing and set retention and access rules

Stop recording traces, since they hold sensitive data

Encrypt the trace database and leave everything else as is