Moving past informal demonstrations requires building evaluation into the delivery pipeline. When a team tests an agent with a handful of representative prompts and sees satisfactory responses, it proves the system worked under those specific conditions—nothing more. A model upgrade or retrieval configuration change weeks later can introduce regressions that remain hidden until users encounter them or monitoring alerts fire. Effective evaluation must become part of how products ship.
"If it can't reproduce a run or a material regression in a high-risk workflow, the product isn't ready to pass the release gate." A repeatable evaluation system executes fixed scenarios along the agent's execution path and gathers evidence to inform release decisions. The system should test the code assembling context, the tools available to the agent, and the permissions enforced at runtime. Without this capability, teams cannot reliably determine whether a new version maintains required behavior.
Define correct behavior before writing tests
Vague criteria like "the answer was good" cannot be tested consistently. Before selecting evaluation tools, teams should document the jobs their agent performs, the constraints around each job, and the outcomes that fall outside acceptable operating boundaries.
Consider a support agent tasked with answering billing questions using records from the correct account while citing current policy. Its constraints might prohibit plan changes or exposure of other customers' data. When policy information is unavailable, the agent should acknowledge the gap rather than present unsupported information as fact. The agent may need to request an account number before proceeding or escalate exceptions to authorized personnel.
Distinguishing the outcome from the process matters significantly. An agent might deliver the correct answer after retrieving the wrong document, complete a task after calling an unnecessary tool, or escalate a routine request it should have handled independently. These runs may appear successful in transcripts while concealing weaknesses that emerge under different requests. Observable requirements for each job should specify that required facts must be supported by named sources, writes must await confirmation, and missing data should prompt the agent to ask rather than guess. High-risk rules warrant exact assertions, though supporting language can vary.
Build scenarios from real user work
The initial test set should remain small enough for a team to maintain. "Ten real tasks are more valuable than a large benchmark filled with prompts your users never send." Support tickets, workflow logs, incident reports, and user conversations provide valuable raw material. Include the ordinary requests comprising most workload volume, then add cases with ambiguous instructions or missing account data. Test scenarios where documents are outdated or tools time out. Some scenarios should require approval before the agent acts, and a few should cover unusual but valid requests.
Since agents operate across multiple turns, test scenarios should reflect this. Ask for an account change, supply the missing identifier in the next message, and confirm the proposed change in a third turn. The test should verify the agent carries the account identifier and proposed change forward without introducing unrelated details into the final action.
Each scenario requires fixtures. Freeze the documents and tool responses used during execution, and pin account state to a known snapshot. Policy version and agent permissions matter equally. A failure that cannot be reproduced becomes a debate about what the agent may have encountered. Fixed fixtures transform it into an engineering problem. Production failures should become permanent regression cases, allowing the suite to record mistakes the team has already learned from.
Test the entire execution path
Scoring only the final answer misses what distinguishes an agent from a chatbot. "Final-answer scoring misses much of what distinguishes an agent from a chatbot. Any step in that loop can diverge from the intended path even when the response appears convincing." Agents retrieve data, select tools to call, supply arguments, interpret results, and decide whether to continue. Any step can deviate from the intended path despite a convincing response.
Capture the request and system instructions. Record the exact model and application build. Version the prompt and retrieval configuration, including tool schemas. Document every retrieved source with its version, every tool call with its arguments and result, permission checks, and the final response. Include latency, token usage, and cost. The trace should answer practical questions without requiring reconstruction from disparate logs.
Combined with server-side enforcement and audit records, the trace should demonstrate that searches remained within the correct tenant and customer account, identify the approved policy source and cited records, and show user confirmation and server-side permission checks for writes. These are deterministic checks—they either occurred or did not. Clarity and usefulness are less deterministic. Human reviewers or model-based evaluators can score whether the response answered the request, explained limitations, or posed sensible follow-up questions. Keep these judgments attached to the trace so teams can identify which step changed when scores drop.
Evaluator failures become easier to spot this way. Model-based evaluators may produce different judgments after an upgrade or respond differently to a revised rubric. Save the evaluator's version and instructions with its result. Regularly compare a sample of those scores against human reviews.
Keep fixed rules separate from variable scores
Agent quality cannot be reduced to a single unexplained metric. Track task completion and factual support separately from retrieval quality. Distinguish policy compliance from user experience, latency, and cost.
Some signals permit tolerances—a response taking 200 milliseconds longer may remain acceptable, and a slightly longer answer might even improve clarity. Others allow no failures. Unapproved updates, cross-tenant retrieval, or missing approval should remain release-blocking conditions regardless of other scores.
Compare a candidate against a known baseline using the same scenarios and fixtures. Show reviewers the changed answers and supporting records, letting tool paths and individual scores explain the differences. If a new version completes more tasks but doubles latency, that may be a reasonable product decision. If it improves average score while bypassing a permission gate, it is not.
Repeat scenarios when behavior is variable. A task succeeding inconsistently—for example, once in ten attempts—has not yet met a reliable release threshold. Set thresholds based on risk, and reserve absolute gates for rules the system must obey every time.
Building evaluation workflows does not require custom tooling. Tools such as Promptfoo, DeepEval, LangSmith, and Braintrust provide capabilities for running scenarios and capturing traces. Some also use models to grade results. Understanding metrics vocabulary helps too. Pass@k asks whether at least one of k attempts succeeds, while pass^k asks whether all k attempts do. Pass^k is useful when consistent behavior matters, but it does not replace exact gates for rules an agent must obey.
Evaluation carries costs. Every live, end-to-end run calling a model spends tokens. Judge models cost more than string checks, and a large suite on every commit accumulates quickly. Reserve expensive judgments for scenarios carrying real risk.
Make evaluation a release gate
Run the suite whenever the team changes a model or prompt. New retrieval configurations, memory policy changes, and tool interface modifications all trigger evaluation. Use a fast set for ordinary changes and a broader set before major releases or model migrations. When a behavior change is intentional, require a reviewer to approve the new expectation rather than rewrite the test.
Records supporting this process need the same controls as the agent itself because evaluation inputs may contain customer data. Traces can include retrieved text and internal instructions, and may capture tool arguments containing credentials or personal data. Version the records and scope access carefully. Consider redacting sensitive values before storage and applying appropriate encryption, access controls, and retention policies based on data involved.
Keeping this work near operational data shortens the path. Oracle AI Vector Search stores vector embeddings alongside business data, and SQL queries can combine similarity search with relational filters and lexical search. Teams using Oracle AI Database can maintain operational records and their vectors in a data platform already under their control. The same platform can hold evaluation traces and enforce access rules. Database-enforced access controls can apply row- and column-level policies, providing another layer for enforcing data-access boundaries. The specific platform matters less than the invariant: teams need durable evaluation cases, the inputs used for each run, and a record of why each build passed.
A release gate should prevent releases with known high-risk conditions while recognizing that some judgments require context. Exact checks block permission and policy regressions. Thresholds catch measurable quality drops, and human review handles ambiguous changes scores cannot settle.
Start with ten cases and keep every important failure
Select ten real tasks this week. Record the expected result and the evidence the agent should use. Note the actions it must not take. Freeze the fixtures, capture the trace, and run the cases before the next release.
"Trust in an agent grows when the team can replay what happened and show that the next release still respects the boundaries users depend on." When an incident occurs, add it to the suite. When a user discovers a failure nobody predicted, keep it. The suite will expand with the product.
The evaluation system is part of the product that ships. Working examples of these patterns in Oracle AI Database, including agentic RAG patterns with hybrid search, are available in Oracle's AI Developer Hub.
Source: The New Stack