AI Agent Observability Needs More Than a Successful Trace

An agent run can return HTTP 200 after choosing the wrong tool, using stale context, repeating work, or crossing a policy boundary. A successful request is therefore not a successful trace.

The comparison below separates full tracing and evaluation platforms from governance guidance and local security evidence. AgentGuard is evaluated as the latter, not counted as an observability platform.

Decide which evidence you need

Pick one reproducible failure and work backward from the user-visible outcome. The trial must connect the run, span, evaluator, tool action, policy decision, and final result without manual payload copying.

*Join evidence with stable run, agent, tool, target, and timestamp fields. Claims remain tied to the listed evidence sources.*

Replay the same failure synchronously and across an asynchronous or sub-agent boundary. Compare lineage, retry side effects, evaluator state, policy evidence, sampling, retention, and incident reconstruction time.

Separate trace capture, evaluation, production diagnosis, and security decision evidence. A trace records the sequence and timing of model, retrieval, tool, and agent steps. An evaluation judges selected behavior against a dataset or rule. Diagnosis connects a production symptom to the responsible step. Security evidence records identity, policy, requested action, approval, and result.

Coverage is more important than a polished trace view. List every framework, model route, retrieval system, tool host, sub-agent, queue, and asynchronous job in the representative workflow, then mark where context propagation can break. A root trace without the tool side effect or policy decision may explain model behavior while leaving the incident boundary invisible.

Map trace, evaluation, and incident owners with enterprise AI agent security practices. The observability purchase still needs end-to-end correlation through the tested run.

Four observability jobs for agents

An agent run crosses model, retrieval, tool, sub-agent, queue, and policy boundaries that need stable correlation rather than one undifferentiated log. Preserve parent and child IDs, agent identity, model and retrieval spans, tool target, policy version, evaluation, timestamps, and final user outcome.

Use the OpenTelemetry specification to define relevant risk language, then translate it into an expected product behavior. The specification cannot prove that a candidate preserves lineage through this worker, tool host, evaluator, or security decision.

For observability, separate trace capture, evaluation, diagnosis, alerting, and security decision evidence. Require the incident owner to start from the user outcome and navigate backward without manual payload copying.

Define a minimum event contract before vendor evaluation: trace and span IDs, timestamps, environment, agent and user identity, model, safe content references under the data policy, tool name, arguments or hashes, target, result, token and latency data, evaluation outcome, policy decision, and error. Not every field belongs in every system, but correlation must be deterministic.

Test sampling, redaction, retention, export, and access controls. A rare security failure can disappear under head-based sampling, while full prompt capture can create a data exposure. Ask whether teams can retain decision metadata without sensitive content and whether an incident export preserves ordering across asynchronous agents.

Sampling should be tested with a rare tail event, not an average successful run. Generate a small series of normal traces and one delayed tool failure or policy denial near the end of the workflow. Confirm whether tail-based rules retain the complete parent and child path, including spans created before the failure was known. If the platform keeps only the terminal error, diagnosis still lacks the retrieval, model, and delegation context that led to it.

Finally, break correlation deliberately. Remove or alter one trace header at an asynchronous boundary and verify how the platform exposes the orphan instead of silently presenting two complete-looking runs. The POC should show the exact instrumentation change needed to restore lineage and whether historical events can be reconciled after that fix.

Test one bypass explicitly: an asynchronous job, sub-agent, custom tool host, sampled trace, or redacted event that loses correlation. Attribute the orphan to instrumentation, propagation, sampling, or retention and define the replay that proves it is repaired.

Three observability platforms, governance guidance, and one security control

Braintrust, LangSmith, and Arize Phoenix are the three observability platforms counted here. IBM provides governance and monitoring context, while AgentGuard supplies adjacent local security-decision evidence rather than full tracing or evaluation. AgentGuard publishes this article; all entries use the same first-party evidence rule, and the cross-trace replay tests their operational value.

Braintrust

Braintrust documents active observability and evaluation for agents. Test online evaluation and debugging against your production traces.

For Braintrust, run one known agent failure through production tracing and an online evaluation. Confirm how the evaluation result attaches to the responsible span, how a reviewer reaches the input and output evidence, and whether the same test can become a regression case without copying sensitive production content.

LangSmith

LangSmith documents tracing, evaluation, and observability for LLM applications and agents. Verify framework coverage and data controls.

For LangSmith, instrument the frameworks and tools used by the representative agent, including one sub-agent or asynchronous step. Verify parent-child correlation, error propagation, dataset creation, evaluation feedback, and export. Record any path that requires custom instrumentation rather than assuming framework support is universal.

Arize Phoenix

Arize Phoenix documents open-source tracing and evaluation for AI applications. Test deployment ownership and production scale.

For Phoenix, test the open-source deployment model the team would actually maintain. Capture a trace, run an evaluation, apply redaction, and export the incident evidence. Include storage growth, retention operations, upgrades, and access control in the POC because ownership is part of the product decision for a solo or small team.

IBM watsonx.governance / observability guidance

IBM watsonx.governance / observability guidance documents enterprise monitoring and governance context. Confirm product boundaries for agent traces.

The reviewed IBM source provides enterprise monitoring and governance context. Ask which purchasable component captures agent traces, which records evaluations or governance decisions, and how the proposed architecture correlates them. Keep guidance evidence separate from behavior demonstrated in the team's environment.

AgentGuard

AgentGuard documents security decision evidence near high-risk actions. Use it alongside quality observability rather than as a full tracing platform.

For AgentGuard, attempt one harmless high-risk developer-agent action and retain the identity context, tool, target, policy, decision, approval state, and result. Then correlate that record to the surrounding trace by stable identifiers or timestamp. This proves security decision evidence without claiming full model, retrieval, and evaluation observability.

The LLM agent exploit vectors review offers candidate failure scenarios for trace replay. It does not establish that every observability platform captures the related identity, tool, or policy fields.

Replay one failure across the trace

Select one deterministic workflow failure that the team can reproduce safely. A retrieval miss, tool timeout, duplicate tool call, denied harmless action, or broken handoff works if the expected trace and final outcome are known before the run.

Define the correlation keys first: run ID, trace ID, agent and sub-agent identity, model request, tool call, policy decision, evaluation result, and final user outcome. Run the workflow through the normal synchronous path and retain the raw trace plus any derived evaluation.

Replay the same input through an asynchronous worker or sub-agent branch. Confirm whether parent and child spans remain connected, whether retries create duplicate business actions, and whether an evaluator can distinguish a recovered technical error from a failed user task.

Introduce the security decision next. Deny one harmless action and verify that the trace records the requested tool and target while the security record retains its policy reason. Correlate the two without copying sensitive prompts or tool arguments into every store.

Use the NIST AI Risk Management Framework to assign owners for measurement, incident review, and policy. It does not define the telemetry fields or prove vendor interoperability.

Now sample the same failure under load. Check ingestion delay, missing spans, evaluation lag, alert deduplication, retention, and cost. The useful result is a reconstructed decision path, not a dashboard screenshot.

Build the correlation record before comparing dashboards:

Instrument one asynchronous boundary before the trial. Record which service creates the child run, how trace context is propagated, and what happens when the worker starts after the parent request has ended. Then break that propagation deliberately in the test environment. The platform should make the orphan visible instead of presenting an apparently complete parent trace.

Treat retries as a business-state problem, not only a span count. Use a harmless tool action with an idempotency key and force one timeout after dispatch. The trace should show whether the action completed, whether a retry was issued, and whether the final user outcome contains one or two side effects. A product that displays both calls but cannot connect them to the intended outcome leaves the operator to infer the incident.

Version evaluation and policy evidence. Change one evaluator threshold or local policy rule between two identical runs, then confirm that the incident view preserves which version applied. Current configuration is not sufficient for explaining an earlier decision. The record should also distinguish an evaluation score from an enforcement outcome; a low score does not prove that an action was blocked.

Inspect sampling and retention by event type. A system may retain error traces while sampling successful runs, or keep tool metadata longer than prompt content. Verify that fields required for security investigation survive the relevant period without retaining unnecessary sensitive payloads. Record redaction and access controls for each stream.

The final replay should start from the user outcome and move backward through evaluation, tool action, policy decision, and parent-child execution. Time how long an operator takes and list every manual join. That readout tests whether the evidence can answer a real operational question.

Replay one agent failure after the expected trace, evaluation, and policy outcome are written.

Where AgentGuard adds security evidence

AgentGuard can contribute a security-specific record near a risky developer-agent action. That record answers whether a bounded action was allowed, denied, or required approval. An observability platform answers how the broader run reached that request and what happened around it.

Join the records using stable run, agent, tool, target, and timestamp fields. Confirm that an operator can move from the trace to the policy decision without exposing more prompt or data content than each system needs.

Inspect the AgentGuard decision inside the replayed failure. The result does not establish model-quality evaluation, full distributed tracing, or observability retention.

Keep the roles separate during incident review. A trace can show that the agent asked for a command; the policy record should show why that command was governed. Together they can explain the failure without relabeling one product as the other.

The MCP security tools guide can help identify adjacent MCP gateway or scanning evidence for the same trace.