An AI agent completes its request with a 200 response and clears all faithfulness validation checks. Yet the customer receives an incorrect answer. The explanation for this failure exists within the execution trace, not in the code diff.

A code diff represents intent, not evidence. When a system goes live and problems emerge—whether slowness, wrong answers, or both—developers naturally examine what changed. The diff points to a modified function, and a plausible explanation follows. But plausibility is not proof.

This observability gap becomes critical with AI features. According to Dynatrace's 2026 State of SRE and Platform Engineering report, which surveyed 919 enterprise leaders globally, 77% of platform engineering teams have embedded observability in at least some services. However, only 40% have achieved full integration across all deployments. This gap was tolerable when systems behaved deterministically, but AI agents expose it as a serious liability.

A conventional service fails loudly… An AI agent fails quietly. It returns a 200. It passes faithfulness checks. And the customer still gets the wrong answer.

Traditional services announce their failures through 500 errors, latency spikes, or unresponsive dependencies. AI agents fail silently. They send a 200 status, pass groundedness metrics, and deliver incorrect information. Alerting on "wrong" is impossible without evidence from the running system—something deeper than request traces and error rates.

One release, two symptoms

Consider a support agent that answers questions using product documentation. A customer asks how to configure export in version 2026.3. The team recently rewrote the documentation lookup with help from a coding assistant. CI passed. Existing evaluations passed. After deployment, two problems appear: answers take longer to generate, and some describe older product versions.

Begin with a single affected request. Capture its release version, retrieval configuration, and feature-flag state as attributes on the root span at span creation time, not reconstructed later from deployment logs. Place this run alongside a similar question from before the change.

For an agent, this means the complete trajectory: every model call and tool invocation, in sequence, with their arguments and results. A distributed trace captures these as spans and connects them across service boundaries through context propagation.

# Illustrative pseudotelemetry, not a captured incident.
# Names, IDs, timings, and labels are invented, not a standard schema.
# Selected spans shown in execution order; other work is omitted.

trace: example-run-a | session: example-session-7 | release: 2026.9.2
requested.product_version: "2026.3"

agent.run                                  12.4s
  model.choose_tool                         1.0s
  tool.search_docs                          0.9s
    args: {query: "configure export", product_version: null}
  tool.search_docs                          0.8s
    args: {query: "configure export", product_version: null}
  tool.search_docs                          0.9s
    args: {query: "configure export", product_version: null}
    returned.doc_versions: ["2024.1", "2024.1", "2023.9"]
  model.generate_answer                     8.1s

linked_evaluation:
  trace: example-run-a
  faithfulness: pass
  requested_version_answered: fail

Two issues emerge: repeated searches and a null version filter. Three identical searches consumed 2.6 seconds. The trace reveals the symptom but not its cause. However, examining what lies between the searches provides a clue. A single model.choose_tool span appears at the top, with no model call between the second and third search. The model did not request those retries; something in the execution harness did—the code managing tool execution, retry logic, and context. A model.choose_tool span between each search would indicate the opposite: a model repeatedly requesting the same tool, pointing to a prompt or tool-description issue. Same symptom, different source file.

This doesn't automatically make the retries incorrect. Review the retry policy and examine the tool results. A 200 response from a search backend can contain an empty result set or results below the relevance threshold, making retries legitimate. The generation call dominates the timeline at 8.1 seconds anyway. Compare its input tokens and duration against similar runs. If the harness concatenated all three result sets into the context, the retries inflated the prompt, creating costs in both latency and tokens. Examine downstream services and traffic patterns before attributing the slowdown to the release.

To bring this investigation back into the development environment, narrow the question. Provide the assistant with the service name, release version, time window, and trace IDs. Have it align the modified code path against the dependency calls visible in the affected trace. Then distinguish what the evidence actually supports from what it merely assumes. This same workflow applies to debugging a checkout service making three identical database calls—no agent required.

A grounded answer can still fail

Now examine the answer itself. In this case, it accurately reproduces the retrieved documentation. Faithfulness passes, or groundedness, depending on terminology. The customer still receives instructions for the wrong version.

Whether labeled faithfulness or groundedness, this metric only confirms whether the answer is supported by the supplied sources. It reveals nothing about whether those sources were correct. The natural next step is a retrieval evaluator. It still won't catch this problem. Retrieval evaluators score whether retrieved context is relevant to the query. The 2024.1 export instructions are relevant to configuring export. They are simply invalid for the requested version.

Those documents are relevant to the query. They are not valid for the version the customer requested. Relevance is not validity.

This is not a generation failure. It is a retrieval precondition that was never enforced, and the null filter identifies it: the requested version never reached the lookup. Reproduce this condition before modifying the prompt or model.

Most of this is testable with standard code. Provide fixtures with version metadata in the documents, then assert directly on the lookup without involving a model:

def test_lookup_filters_to_requested_version(docs_fixture):
    hits = search_docs(query="configure export", product_version="2026.3")
    assert hits, "no hits for a version that has docs"
    assert {h.product_version for h in hits} == {"2026.3"}

This is deterministic, fast, and belongs in CI. Evaluate the answer separately, which is the part that cannot be asserted: does it provide usable 2026.3 instructions, or does it acknowledge that available documentation cannot support one? Two tests, because they fail for different reasons and distinguishing which one broke matters. The assertion won't catch every wrong answer, but it will catch this missing constraint every time—more reliable than a human judge scoring helpfulness on a scale.

This is why evaluation requires retained context. Record the prompt version, model ID, retrieval configuration, and document IDs and versions alongside the release. Preserve enough permitted evidence to reconstruct the answer later, with sensitive content redacted before export. Link results by trace and span ID. If scoring occurs after the span closes, store a separate linked result. Do not attempt to write attributes to a finished span; the OpenTelemetry tracing API specifies that implementations should disregard updates after End. GenAI semantic conventions remain in flux, and different instrumentation projects use different attribute names for similar concepts.

Make the failure part of the next release check

Once causes are confirmed, validate each fix against the behavior it should change. For repeated searches, create a regression test that reproduces the repetition without breaking legitimate retries. Do not demand a single exact tool sequence when multiple orderings complete the task correctly; a trajectory test requiring one specific path will fail on every valid refactor.

For the version mismatch, restore the filter. Add test cases covering the current version, an older supported version the customer explicitly names, irrelevant documentation, and scenarios where no supportable answer exists. Run answer evaluations multiple times where output varies, since a single pass is not a result. Use code for anything that can be directly asserted. Use a model-based judge for answer quality, and validate that judge against examples reviewed by people. An unvalidated judge is simply another model accepted on faith.

To score production traffic rather than test fixtures, establish a method to sample spans already in the environment, score them with a judge model, and link each result back to the source trace—the linked-result pattern mentioned earlier, not a write to a closed span. Whatever tooling is chosen, version the evaluator. A scoring change that appears to be a product improvement may not be one.

After deployment, compare latency and task success on similar requests, and display tool-call and token counts on the same view. Read them together, or they will mislead. Tool calls dropping from three to one can suggest the fix is working, but it can also indicate an accidentally removed lookup. Fewer output tokens appear as a cost reduction, but they also suggest an answer that quietly omitted step four.

Bring one debugging question

If the starting point for an investigation is unclear, instrumentation is incomplete. For a conventional service, this means the request path and dependency timing. For an AI feature, add what was retrieved, what was produced, and how to determine whether that was correct.

Dynatrace is sponsoring WeAreDevelopers World Congress Americas, September 23-25, 2026, in San José. Attendees are invited to bring a debugging question from an AI-assisted release or from an AI feature under development, and work through it together.

Source: The New Stack