agent-observabilityComprehensive skill for tracing reasoning paths, debugging non-deterministic agent loops, and monitoring agent behavior in production systems. Covers reasoning trace visualization, OpenTelemetry integration for agent systems, distributed tracing across multi-agent chains, decision audit logging, performance profiling, anomaly detection, cost tracking and op…
Independent GitHub snapshot · Automated classification · Original documentation governs
name: agent-observability description: > Comprehensive skill for tracing reasoning paths, debugging non-deterministic agent loops, and monitoring agent behavior in production systems. Covers reasoning trace visualization, OpenTelemetry integration for agent systems, distributed tracing across multi-agent chains, decision audit logging, performance profiling, anomaly detection, cost tracking and optimization, and latency analysis for AI agent deployments. version: "2.0.0" author: "j4flmao" license: "MIT" type: skill compatibility: claude-code: true cursor: true codex: true windsurf: true tags:
This skill provides the complete knowledge required to build production-grade observability systems for AI agent deployments. Unlike traditional software where execution paths are deterministic and debuggable with standard tools, agent systems exhibit non-deterministic reasoning loops, branching decision trees, variable-length tool call chains, and stochastic output variations that demand specialized observability infrastructure.
The skill covers every dimension of agent observability: capturing and visualizing reasoning traces that show why an agent made each decision, integrating OpenTelemetry to instrument every LLM call and tool invocation, propagating distributed trace context across multi-agent chains, maintaining tamper-evident decision audit logs, profiling performance bottlenecks in agent pipelines, detecting anomalous behavior patterns that indicate drift or failure, tracking costs across model providers and optimizing token usage, and analyzing latency to identify and eliminate bottlenecks in the agent execution critical path.
Trace Everything, Sample Intelligently: Every LLM call, tool invocation, and decision point must be instrumentable. In production, use head-based and tail-based sampling to control volume while guaranteeing capture of errors and anomalies.
Reasoning is the Primary Signal: Traditional metrics (latency, throughput, errors) are necessary but insufficient. The agent's reasoning trace—the chain of observations, thoughts, and actions—is the primary debugging and auditing artifact.
Correlation Across Boundaries: A single user request may traverse multiple agents, tools, sandboxes, and external APIs. Trace context must propagate across all boundaries to enable end-to-end visibility.
Cost is a First-Class Metric: Token consumption, API call counts, and dollar costs must be tracked with the same rigor as latency and error rates. Cost anomalies often indicate reasoning loops or prompt inefficiencies.
Detect Drift Before Failure: Anomaly detection on agent behavior distributions (response length, tool call frequency, reasoning depth) catches degradation before it manifests as user-visible failures.
{
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"span_id": "00f067aa0ba902b7",
"agent_id": "agent-planner-01",
"request_id": "req-7f8a9b0c",
"reasoning_trace": {
"steps": [
{
"step_id": 1,
"type": "observation",
"content": "User requests weather forecast for Tokyo",
"timestamp": "2026-06-04T09:00:00.123Z",
"tokens_in": 47,
"tokens_out": 0
},
{
"step_id": 2,
"type": "thought",
"content": "Need to call weather API tool with location=Tokyo",
"timestamp": "2026-06-04T09:00:00.456Z",
"tokens_in": 0,
"tokens_out": 32,
"model": "claude-sonnet-4-20250514",
"cost_usd": 0.00048
},
{
"step_id": 3,
"type": "action",
"tool": "weather_api",
"input": {"location": "Tokyo", "units": "metric"},
"output": {"temp_c": 22, "condition": "partly_cloudy"},
"latency_ms": 340,
"timestamp": "2026-06-04T09:00:00.796Z"
}
],
"total_steps": 5,
"total_tokens": 847,
"total_cost_usd": 0.00127,
"total_latency_ms": 2340
},
"metrics": {
"llm_calls": 2,
"tool_calls": 1,
"retries": 0,
"cache_hits": 1,
"reasoning_depth": 5
}
}
START: Observability requirement identified
│
├─ What is the primary goal?
│ ├─ DEBUGGING → Focus on reasoning trace capture
│ │ ├─ Single agent? → Instrument with local span collection
│ │ └─ Multi-agent? → Deploy distributed tracing with context propagation
│ │
│ ├─ COMPLIANCE → Focus on decision audit logging
│ │ ├─ Tamper-evident required? → Use append-only log with Merkle tree
│ │ └─ Standard audit? → Structured JSON logs with retention policy
│ │
│ ├─ COST CONTROL → Focus on cost tracking & optimization
│ │ ├─ Per-request attribution? → Tag spans with cost metadata
│ │ └─ Aggregate trends? → Build cost dashboards with model breakdown
│ │
│ └─ PERFORMANCE → Focus on latency analysis
│ ├─ Identify bottleneck first → Capture critical path analysis
│ ├─ LLM latency dominant? → Optimize prompts, enable caching
│ └─ Tool latency dominant? → Parallelize tool calls, add timeouts
│
├─ What sampling strategy?
│ ├─ Development → 100% sampling (capture everything)
│ ├─ Staging → Head-based 10% + tail-based on errors
│ └─ Production → Head-based 1% + tail-based on errors/anomalies
│
└─ What alerting is needed?
├─ Latency P99 > threshold → PagerDuty critical alert
├─ Error rate > 5% over 5min → Slack warning + auto-investigation
├─ Cost per request > 2x baseline → Cost alert + reasoning review
└─ Anomaly score > 3σ → Anomaly alert + full trace capture
┌───────────────────────────────────────────────────────────────────┐
│ AGENT RUNTIME │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ INSTRUMENTATION LAYER │ │
│ │ ┌────────────┐ ┌────────────┐ ┌────────────────────────┐ │ │
│ │ │ LLM Call │ │ Tool Call │ │ Reasoning Step │ │ │
│ │ │ Interceptor│ │ Interceptor│ │ Recorder │ │ │
│ │ └─────┬──────┘ └─────┬──────┘ └──────────┬─────────────┘ │ │
│ │ │ │ │ │ │
│ │ ▼ ▼ ▼ │ │
│ │ ┌──────────────────────────────────────────────────────────┐ │ │
│ │ │ OPENTELEMETRY SDK (Traces + Metrics) │ │ │
│ │ │ ┌──────────┐ ┌───────────┐ ┌───────────────────────┐ │ │ │
│ │ │ │ Tracer │ │ Meter │ │ Context Propagator │ │ │ │
│ │ │ │ Provider │ │ Provider │ │ (W3C TraceContext) │ │ │ │
│ │ │ └──────────┘ └───────────┘ └───────────────────────┘ │ │ │
│ │ └──────────────────────┬───────────────────────────────────┘ │ │
│ └─────────────────────────┼────────────────────────────────────┘ │
│ │ │
└────────────────────────────┼──────────────────────────────────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
┌──────────────────┐ ┌─────────────┐ ┌──────────────────┐
│ TRACE BACKEND │ │ METRICS │ │ LOG AGGREGATOR │
│ ┌────────────┐ │ │ BACKEND │ │ ┌────────────┐ │
│ │ Jaeger / │ │ │ ┌─────────┐│ │ │ Loki / │ │
│ │ Tempo / │ │ │ │Prometheus││ │ │ Elastic / │ │
│ │ Honeycomb │ │ │ │/ Mimir ││ │ │ CloudWatch │ │
│ └────────────┘ │ │ └─────────┘│ │ └────────────┘ │
└──────────────────┘ └─────────────┘ └──────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────────┐
│ ANALYSIS LAYER │
│ ┌─────────────┐ ┌──────────────┐ ┌────────────────┐ │
│ │ Reasoning │ │ Anomaly │ │ Cost │ │
│ │ Trace │ │ Detection │ │ Attribution │ │
│ │ Visualizer │ │ Engine │ │ Engine │ │
│ └─────────────┘ └──────────────┘ └────────────────┘ │
│ ┌─────────────┐ ┌──────────────┐ ┌────────────────┐ │
│ │ Latency │ │ Decision │ │ Alert │ │
│ │ Critical │ │ Audit │ │ Manager │ │
│ │ Path │ │ Explorer │ │ (PagerDuty) │ │
│ └─────────────┘ └──────────────┘ └────────────────┘ │
└─────────────────────────────────────────────────────────┘
LLM Call ──► Span Created ──► Attributes Added ──► Span Exported
│ │ │ │
▼ ▼ ▼ ▼
Token Count Trace Context Model, Tokens, OTLP Collector
Recorded Propagated Cost Metadata │
├──► Traces → Jaeger
├──► Metrics → Prometheus
└──► Logs → Loki
| Symptom | Primary Cause | Mitigation Action |
|---|---|---|
| Traces missing for some agent steps | Async context lost in callback chains | Use OpenTelemetry context propagation utilities; ensure async hooks are instrumented |
| High cardinality metric explosion | Unique trace/request IDs used as metric labels | Use bounded label sets; move unique IDs to trace attributes, not metric labels |
| Reasoning trace shows infinite loop | Agent re-evaluating same observation without progress | Add loop detection middleware; set max_reasoning_steps per request |
| Cost tracking shows $0 for all requests | Model pricing metadata not configured | Configure per-model token pricing in the cost attribution engine |
| Latency P99 10x higher than P50 | Tail latency from cold LLM inference or rate limiting | Implement request hedging; add LLM response caching; monitor rate limit headers |
| Audit log gaps during high traffic | Log buffer overflow dropping entries | Use durable queue (Kafka) between agent and audit log; increase buffer capacity |
| Anomaly detector fires too many false positives | Baselines computed during non-representative period | Retrain baselines on 7+ days of production data; implement adaptive thresholds |
| Distributed trace context not propagating | Missing W3C TraceContext headers in HTTP/gRPC calls | Verify auto-instrumentation covers all HTTP clients; add manual propagation for custom protocols |
User Request: "Summarize the Q3 financial report and compare to Q2"
│
▼
┌──────────────────────────────────────────────────────────────┐
│ 1. TRACE INITIATED │
│ trace_id: 4bf92f3577b34da6 │
│ root_span: "user_request" (agent-planner-01) │
│ Sampling decision: RECORD (matches 1% head-based sample) │
└─────────────────────┬────────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────────┐
│ 2. REASONING TRACE CAPTURED │
│ Step 1: [observation] Parse user intent → "compare Q3/Q2" │
│ Step 2: [thought] Need to retrieve both reports first │
│ Step 3: [action] tool=doc_retriever input={q: "Q3 report"} │
│ Step 4: [action] tool=doc_retriever input={q: "Q2 report"} │
│ Step 5: [thought] Both retrieved, now analyze differences │
│ Step 6: [action] tool=llm_analyze input={docs: [q3, q2]} │
│ Step 7: [result] Summary generated with comparison table │
└─────────────────────┬────────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────────┐
│ 3. METRICS EMITTED │
│ llm_call_count: 3 | tool_call_count: 2 │
│ total_tokens: 12,847 | total_cost: $0.0193 │
│ total_latency: 4,230ms | reasoning_depth: 7 │
│ Critical path: LLM analyze (2,100ms) → doc_retriever (890ms)│
└─────────────────────┬────────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────────┐
│ 4. AUDIT LOG ENTRY │
│ request_id: req-7f8a9b0c | agent: agent-planner-01 │
│ decision: "Retrieved Q3+Q2 reports, performed comparative │
│ analysis using claude-sonnet-4-20250514" │
│ outcome: SUCCESS | confidence: 0.92 │
│ hash_chain: sha256(prev_entry + this_entry) │
└─────────────────────┬────────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────────┐
│ 5. ANOMALY CHECK │
│ Reasoning depth 7: within 1σ of baseline (μ=6.2, σ=2.1) │
│ Cost $0.0193: within normal range │
│ Latency 4,230ms: slightly above P90 (3,800ms) → MONITOR │
│ No anomaly alert triggered │
└──────────────────────────────────────────────────────────────┘
Never log raw prompts or completions in metrics: Prompts and completions contain sensitive data and have unbounded cardinality. Store them in traces with appropriate access controls, never as metric labels.
Propagate trace context through every boundary: Every HTTP call, gRPC call, queue message, and sandbox invocation must carry W3C TraceContext headers. Broken trace context creates observability blind spots.
Cost attribution must be real-time: Cost data older than 5 minutes is too stale for anomaly detection. Compute cost at span-completion time using pre-configured pricing tables.
Audit logs are append-only and immutable: Once written, audit entries must never be modified or deleted within the retention period. Use cryptographic hash chains to detect tampering.
Alert on absence, not just presence: Missing traces, gaps in metric streams, and silent agents are often more dangerous than explicit errors. Monitor heartbeats and expected event rates.
from typing import Dict, List, Optional, Any
from datetime import datetime
import json
import uuid
class AgentTrace:
def __init__(self, agent_id: str, session_id: str):
self.trace_id = str(uuid.uuid4())
self.agent_id = agent_id
self.session_id = session_id
self.spans: List[Dict] = []
self.start_time = datetime.utcnow()
def start_span(self, name: str, parent_span_id: Optional[str] = None) -> str:
span_id = str(uuid.uuid4())[:16]
span = {
"span_id": span_id,
"trace_id": self.trace_id,
"parent_span_id": parent_span_id,
"name": name,
"start_time": datetime.utcnow().isoformat() + "Z",
"attributes": {},
"events": [],
"status": "ok",
}
self.spans.append(span)
return span_id
def end_span(self, span_id: str, status: str = "ok", attributes: Optional[Dict] = None):
for span in self.spans:
if span["span_id"] == span_id:
span["end_time"] = datetime.utcnow().isoformat() + "Z"
span["status"] = status
if attributes:
span["attributes"].update(attributes)
def add_event(self, span_id: str, name: str, attributes: Optional[Dict] = None):
for span in self.spans:
if span["span_id"] == span_id:
span["events"].append({
"name": name,
"timestamp": datetime.utcnow().isoformat() + "Z",
"attributes": attributes or {},
})
def to_dict(self) -> Dict:
return {
"trace_id": self.trace_id,
"agent_id": self.agent_id,
"session_id": self.session_id,
"duration_ms": (datetime.utcnow() - self.start_time).total_seconds() * 1000,
"spans": self.spans,
}
class CostTracker:
def __init__(self):
self.entries: List[Dict] = []
self.pricing = {
"gpt-4": {"input": 0.03, "output": 0.06},
"gpt-3.5-turbo": {"input": 0.0015, "output": 0.002},
"claude-3": {"input": 0.015, "output": 0.075},
}
def track_call(self, model: str, input_tokens: int, output_tokens: int, agent_id: str):
pricing = self.pricing.get(model, {"input": 0.01, "output": 0.03})
cost = (input_tokens / 1000 * pricing["input"]) + (output_tokens / 1000 * pricing["output"])
self.entries.append({
"timestamp": datetime.utcnow().isoformat() + "Z",
"agent_id": agent_id,
"model": model,
"input_tokens": input_tokens,
"output_tokens": output_tokens,
"cost": round(cost, 6),
})
def get_total_cost(self, agent_id: Optional[str] = None) -> float:
entries = self.entries
if agent_id:
entries = [e for e in entries if e["agent_id"] == agent_id]
return round(sum(e["cost"] for e in entries), 4)
def get_report(self) -> str:
total = sum(e["cost"] for e in self.entries)
by_model = {}
for e in self.entries:
by_model[e["model"]] = by_model.get(e["model"], 0) + e["cost"]
lines = ["## Cost Report"]
lines.append(f"Total Cost: ${total:.4f}")
for model, cost in sorted(by_model.items(), key=lambda x: -x[1]):
lines.append(f" {model}: ${cost:.4f}")
lines.append(f"Total Calls: {len(self.entries)}")
return "\n".join(lines)
What aspect of agent behavior to observe?
├── Reasoning path (what the agent thought)
│ └── Reasoning traces: span per reasoning step with input/output
│ ├── Debug incorrect decisions
│ ├── Validate safety constraints
│ └── Optimize prompt chains
│
├── Tool usage (what tools were called)
│ └── Tool call spans: arguments, results, duration
│ ├── Track tool success/failure rates
│ ├── Detect tool misuse patterns
│ └── Measure tool latency impact
│
├── Cost (how much each interaction costs)
│ └── Token tracking per model call
│ ├── Attribution to user/agent/session
│ ├── Budget enforcement
│ └── Cost optimization
│
└── Safety (did the agent violate policies)
└── Audit events: policy checks, guardrail triggers, violations
├── Immutable audit log
├── Violation rate trending
└── Root cause investigation
| Anti-Pattern | Why It Fails | Correct Approach |
|---|---|---|
| Not tracing agent reasoning | Can't debug why agent made wrong decision | Trace every LLM call with input/output |
| Cost tracking with stale pricing | Budget reports are inaccurate | Update pricing table on model/version change |
| No audit log immutability | Tampered logs can't be used for compliance | Cryptographic hash chain on audit entries |
| Alerting on absence without baseline | Too many false positives | Establish baseline metrics, alert on deviation |