No Tracing

Signs and symptoms

At 3:12 a.m., the incident agent restarted the catalog service instead of the payments service. The next morning, this was the entire available record (anonymized):

03:11:58 INFO  incident-agent  alert received: payments-latency-p99
03:12:41 INFO  incident-agent  action: restart service=catalog
03:12:43 INFO  incident-agent  done

Forty-three seconds of decision-making and three log lines. Nobody knows what the agent read, which tools it called, what they returned, which prompt version ran, or what it cost.

At a bank, this is more than an annoying bug. If the agent moved money, blocked Pix (Brazil’s instant payments), or restarted a settlement service, privacy law and audit will ask why. “I think the model mixed up the name” is not an answer. Brazil’s LGPD, like the GDPR, gives people a right to understand automated decisions that affect them.

Signs of No Tracing:

• When someone asks “why did the agent do that?”, the answer is a guess.
• Logs record start and end, but not model calls or tool results.
• There is no way to know which prompt version produced a specific answer.
• Cost is only known from the monthly invoice, with no breakdown by task or user.
• To reproduce a bug, the team has to ask the user to try again.
• Everyone debugs with local print statements, and nothing is kept.

flowchart LR
  A(["1. Alert"]):::io --> M(["2. Model decides"]):::llm
  M --> T["3. Tools"]:::tool
  T --> M
  M --> X["4. Action"]:::tool
  X --> L[("5. Log: 'done'")]:::store

1. Something triggers the task.
2. The model reads a context nobody stored.
3. Tools return results nobody stored.
4. The action happens.
5. The only record says it finished.

Check +20 XP on first tryThe incident agent restarted the wrong service. The logs show only 'alert received' and 'done'. What can you actually claim?

The model hallucinated the service name

You need to raise the log level to DEBUG

Without spans for the calls and the tools, you cannot rebuild the decision

Reasons

• Application logs seem enough. For regular code, one line per event works. For a model, the decision lives in the context, and the context is not in the log.
• Fear of storing data. Prompts contain user data, and storing it feels risky. Without a masking plan, the choice becomes storing nothing.
• Observability can wait. In the prototype, the developer sees everything in the terminal. In production, the terminal is gone.
• New tooling. The team does not know the conventions and tracing tools for AI.

Common trap: saving prompts and calling it a trace

A file with thousands of loose prompts does not rebuild a run. You still need the task id, the tool order, each result, the prompt version, and the cost. It is a black box with more paper inside.

The other extreme fails too. In 2023, Samsung employees pasted confidential code into ChatGPT. With no retention and masking policy, the team either stores too much (and leaks) or stores nothing (and flies blind). Both mistakes come from the same gap: a plan for what to keep, for how long, and which fields to mask.

Cost of ignoring

• Unexplained incidents. Without knowing why the agent went wrong, you cannot fix it, only hope.
• Cost with no owner. A user or task costing ten times the average goes unnoticed.
• Evals without a source. The best test cases come from bad production runs. Without traces, they are lost.
• Compliance. For decisions about money or people, having no record can be a violation.

Compare the two mornings after the incident:

Question	No trace	With spans
What did the model read?	A guess	The alert, the runbook, and the service list it got back
Which tool failed?	Nobody knows	list_services returned catalog at the top
Which prompt?	Friday’s, maybe	incident-agent@v12
Can it become an eval?	No	Yes, one click on the case

Treatment

Refactoring	Choose it when
Add Tracing	Always. Every model call, every tool, and every code decision becomes a span, with prompt, version, tokens, cost, and result. For decisions that need proof, the target is the Audit Log.

The concept is covered in Observability for AI. A useful span carries at least: run id, prompt version, tool arguments and result (the payload, never only “ok”), tokens, cost, and latency. Mask tax ids, cards, and secrets at write time, not at incident time.

Debug traces and an audit trail are not the same thing. The first can expire in days. The second, for payments, credit, or account blocks, follows the retention legal asked for.

Check +20 XP on first tryThe team wants to record traces, but legal blocks it because the prompt carries a national ID number. What is the right next step?

Mask personal data in the spans and record the rest

Drop tracing and keep only start and end logs

Record everything and restrict access to engineering

Payoff

• Any run can be opened and read step by step.
• Cost and latency per task, per user, and per prompt version.
• Production bugs become eval cases with one click.
• Answers to “why did the system do that?” backed by evidence.

When to ignore

• A local script for personal use. If only you use it and you see the output in the terminal, full tracing is overkill.
• The first hours of a prototype. Even there, instrumenting with an off-the-shelf library takes minutes and usually pays for itself at the first bug.

No agent that takes action in production, with nobody watching, should fly without spans.

Is it this smell?

Check +20 XP on first tryThe team writes every prompt and response to a log file, but without identifying which task or user each line belongs to, and without tool results. Is this No Tracing?

No, the prompts are stored

Only if the prompts are longer than a thousand tokens

Yes

Check +20 XP on first tryTraces are complete but kept for only 3 days. A customer complaint arrives 10 days after the problem. What is the situation?

No Tracing

Retention too short for the case

Unversioned Prompt

Check +20 XP on first tryTraces record calls and tools, but not the prompt version. A bad answer from two weeks ago cannot be reproduced because the prompt has changed three times since. Which smell is closest?

Unversioned Prompt

No Tracing

No Cost Budget

Check +20 XP on first tryTraces record the model call, but each tool result is just the string 'ok', with no payload. Can you explain why the agent restarted catalog?

Yes, the action name is enough

Yes, if you ask the model to explain afterwards

No, the tool result and the model's context are still missing