Why agents are hard to debug
A single agent turn may involve 10+ LLM calls, 5+ tool executions, and dozens of intermediate messages. When the output is wrong, tracing which step caused it is the difference between a 5-minute fix and a two-day investigation.
The naive approach — 'log everything to stdout' — falls over at any real scale. Structured tracing designed for LLM workflows is now the baseline expectation for production agents.
OpenTelemetry for LLMs
The OpenLLMetry and Traceloop projects extend OpenTelemetry with LLM-specific semantic conventions: prompt, completion, model, tokens, latency, tool calls. Every major framework (LangChain, LlamaIndex, Vercel AI SDK) now emits these traces.
Send traces to Langfuse, Braintrust, LangSmith, or your existing OTel backend (Honeycomb, Grafana Tempo). All of them show the tree structure of an agent run with drill-in on every LLM call.
Pick one tool and standardize. Multi-tool tracing fragments your investigation — the win comes from having every call in one queryable place.
Replay is the killer feature
When production produces a bad output, you want to re-run that exact input through a new prompt without waiting for a real user to hit it. This is replay.
Every serious LLM observability tool now supports it: click a trace, edit the prompt, hit rerun, see the new output side-by-side. It compresses a debugging cycle from hours to seconds.
This changes how prompt changes ship. Instead of 'edit prompt → deploy → wait for a user to hit the bug', you replay the top-10 recent failures and know before deploying whether the fix worked.
Sampling and cost
Full-trace logging on high-volume agents gets expensive fast. Sample: 100% of failures, 100% of user-flagged calls, 5-10% of everything else.
Truncate prompt bodies over ~10K tokens in the trace payload; keep the full call available for replay via a signed URL to blob storage. Storage cost drops 90%.
What to alert on
Tool call failure rate > 5% over 10 min. Median latency doubling week over week. Cost per session drifting up more than 20%. LLM grader score dropping more than 5% on nightly sample.
Do not alert on individual failed calls — LLMs fail. Alert on trends. Every alert should point at a Grafana or Langfuse view where you can drill in within one click.