AI Agent Observability:
Tracing, Monitoring, Debugging & Incident Response

AGENT OBSERVABILITY · THE CONTROL PLANE

What should you observe in an AI agent?

An agent may return a reassuring answer while leaving a database half-updated, calling a prohibited API or retrying the same expensive request. To debug that gap, you need more than a chatbot transcript. You need three complementary signals: traces to explain a particular run, metrics to detect patterns across many runs, and structured logs/events to capture controlled lifecycle facts. The fourth requirement is independent verification of the actual outcome.

A trace ID is a correlation key, not proof of correctness. Record a run ID tied to the product's state and use server-side outcome checks for high-impact actions. For conceptual metrics, see AI Agent Reliability; for pre-production tests, see AI Agent Evaluation Checklist. SXF's synthetic marketing agent study shows why run-level outcomes and false alerts matter, while explicitly noting that its operator-pasted responses are not full execution traces. This page is about obtaining evidence continuously in live environments.

LLM TRACING · SEMANTIC CONVENTIONS

Execution traces: spans, context propagation and OpenTelemetry

A distributed trace is a tree of timed operations. Put an agent-run span around the whole attempt, then create child spans for model calls, retrieval, tool execution, authorization and recovery. Preserve parent/child relationships across async tasks, services and message queues using trace context. Use stable, opaque identifiers rather than customer messages as identifiers.

OpenTelemetry's Generative AI semantic conventions provide common vocabulary across SDKs. Their repository and documents evolve, and some agent semantic conventions remain in Development. Pin the instrumentation version and test the actual exported field names before building dashboards or promises around them. [1–4]

PYTHON IMPLEMENTATION · REAL TRACES

Instrument an AI agent with OpenTelemetry in Python

Start small: create a root span for the run and nested spans around your own tool calls. The following stdlib-compatible example apart from the OpenTelemetry packages uses synthetic, local data. It does not call a live model or payment API. The GenAI operation fields are illustrative where they overlap with spec conventions; application-specific fields are explicitly prefixed app..

# pip install opentelemetry-api opentelemetry-sdk
# Example uses ConsoleSpanExporter; replace it with an approved OTLP
# exporter + collector config once privacy and transport are reviewed.
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
from opentelemetry.trace import Status, StatusCode

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("sxf.agent-observability", "1.0")

def authorized_lookup(run_id: str, allowed: bool) -> dict:
    with tracer.start_as_current_span("execute_tool lookup_inventory") as span:
        span.set_attribute("app.run_id", run_id)  # opaque, non-PII ID
        span.set_attribute("app.tool.name", "lookup_inventory")
        span.set_attribute("app.authorization.allowed", allowed)
        if not allowed:
            span.set_attribute("app.tool.outcome", "denied")
            return {"status": "denied"}
        span.set_attribute("app.tool.outcome", "ok")
        return {"status": "ok", "count": 3}  # synthetic response

def execute_agent(run_id: str) -> dict:
    with tracer.start_as_current_span("invoke_agent support_agent") as root:
        root.set_attribute("app.run_id", run_id)
        root.set_attribute("app.agent.name", "support_agent")
        with tracer.start_as_current_span("chat plan") as model_span:
            # Only set actual provider-reported usage, not made-up tokens.
            model_span.set_attribute("gen_ai.operation.name", "chat")
            model_span.set_attribute("gen_ai.provider.name", "example_provider")
        result = authorized_lookup(run_id, allowed=True)
        root.set_attribute("app.outcome.verified", result["status"] == "ok")
        return result

if __name__ == "__main__":
    execute_agent("demo-run-001")
# Do not export prompts, customer text, secret tokens or raw tool payloads.

The console exporter is for local demonstrations; production systems should use an approved SDK processor and a secure OpenTelemetry Collector with suitable authentication, sampling, storage and retention. Automatic instrumentation is convenient, but inspect precisely which attributes it exports before enabling it in a customer environment. [4–8]

INTERACTIVE DEMO · SYNTHETIC TELEMETRY

AI agent monitoring dashboard: traces, errors, latency and costs

Explore a synthetic production-style dataset. Change the scenario to see how success rate, p95 latency, token use, tool errors and cost can tell different stories. This is a browser-only demonstration using authored sample records; it does not connect to your site's telemetry or any provider.

CASE STUDY · END-TO-END INCIDENT RESPONSE

Production incident: an agent retried a failing tool until cost and latency spiked

Scenario (fictional): at 14:03 UTC, a customer-support agent receives an inventory question. Its inventory tool times out repeatedly. The agent retries three times without a strict retry budget, prompts the LLM to re-plan, and approaches an unauthorized write path. A server-side approval gate blocks the write. The answer arrives late; although the final text appears calm, operationally the workflow did not complete normally.

The key diagnosis is not simply “the model hallucinated.” The agent architecture allowed a failing dependency to create repeated orchestration work. Reliability comes from explicit timeout budgets, independent side-effect verification and safe human escalation. Read AI Agent Security for permission boundaries and AI Agent Authorization for policy gates.

OPERATIONAL METRICS · ACTIONABLE ALERTS

Which SLIs, SLOs and alerts matter for AI agents?

Define service-level indicators from verifiable events, not prose sentiment. Set objective thresholds from your own baseline, traffic, criticality and error budget. Illustrative targets below are planning examples, not universal industry SLOs. The most dangerous failure can be rare: authorization bypass should page immediately, while modest cost variance may warrant investigation rather than a 3 AM alarm.

Sampling matters: sampled traces help investigate, but sampled runs cannot serve as the denominator for complete service rates without correction. Use unsampled or appropriately aggregated counters for requests, errors, costs and verified outcomes; keep representative traces for root-cause analysis and a controlled retention policy for error traces. Be careful that tail sampling may require collector resources and may see incomplete traces in distributed deployments. [5, 7]

TOKEN USAGE · BUDGET ENFORCEMENT

Monitor token usage and enforce AI agent cost limits

Token accounting must be model- and provider-aware: input tokens, output tokens, cached-input discounts, tool costs, retries, premium modes, requests rejected upstream, and billing-period differences affect the real total. A trace can show where tokens were spent, while aggregate metrics show whether an agent is consuming its error budget or financial budget.

Enforce a hard application-side maximum on model calls, total tokens, tool invocations and elapsed time per run. An LLM instruction such as “spend less than $1” is not a budget enforcement mechanism. See AI Agent Cost & TCO and Model Pricing for cost modeling.

PRIVACY · SECURITY · SAFE TRACING

How to avoid leaking sensitive data into AI agent logs

Full prompts, tool responses, stack traces and retrieved documents can contain personal data, secrets, access tokens and privileged business records. The most durable principle is data minimization at source: emit safe IDs, numeric measurements, bounded enum labels and separately authorized summaries instead of raw content. Masking at a backend is a useful second line, but it cannot guarantee removal from copies that left your process earlier.

Self-hosting does not automatically make traces private if operators and integrations can read them. Review SDK defaults, the collector configuration, your hosted vendor's retention and access policies, and any tracing enabled by frameworks. OpenAI's Agents SDK, for example, has built-in tracing features with configurable sensitivity behavior: confirm them for the exact SDK version and your data controls. [8, 9]

AGENT OBSERVABILITY TOOLS · COMPARISON

OpenTelemetry vs LangSmith vs Langfuse vs Phoenix: which tool fits?

These products are not identical substitutes. OpenTelemetry defines instrumentation and interoperability plumbing rather than a complete agent debugging UI. Other tools add trace inspection, prompt evaluation, datasets, sessions or workflow views. This is a capability and deployment comparison, not an invented benchmark or price ranking.

Shortlist based on your framework, whether you must self-host, on-call workflow, dashboards, alert integrations, token/cost attribution, RBAC and retention rules. Link telemetry to your existing incident system; a good trace viewer without paging and remediation practice is insufficient for production.

PRODUCTION RELEASE CHECKLIST

AI agent observability checklist: deploy, detect, diagnose and learn

The checklist is interactive only in your browser; checking a box does not certify your deployment. Use the 25-test evaluation scorecard alongside live monitoring, rather than treating either as a replacement for the other.

SEARCH QUESTIONS · DIRECT ANSWERS

Frequently asked questions about AI agent observability

OFFICIAL SOURCES · EDITORIAL METHOD

Primary research and source ledger

Reviewed October 17, 2026. This reference uses OpenTelemetry documentation for the conceptual telemetry model and current GenAI semantic-convention names, plus official vendor documentation for tool capabilities. Schema stability and version compatibility must be rechecked before instrumenting production. The incident scenario, diagram, dashboard numbers and demo cost values are authored synthetic examples; SXF has not claimed access to any customer's private telemetry or completed a live production benchmark. Vendor pricing and feature tiers are intentionally not guessed.

SXF / AI · AGENT ENGINEERING LIBRARY

Continue from observability to reliability, security and architecture