Anthropic reports that Claude generates and evaluates approximately 90% of the code written by engineers developing Claude Code, supported by a verification system designed to handle dozens or hundreds of concurrent agents without requiring manual inspection of every change.

In a Monday engineering post, Anthropic technical staff member Addy Osmani outlined a verification architecture built to scale across multiple agents, where automated checks provide the evidence engineers need to assess agent-generated work. The system has drawn attention from the developer community, particularly regarding the inference costs of model-driven verification and whether deterministic tools could replace some of that functionality.

Anthropic addressed questions from The New Stack about controlling inference expenses, preventing agents from circumventing tests, and measuring the effectiveness of its automated reviewers.

Testing AI code live

Anthropic deploys Claude's changes within a running application, using browser automation for web interfaces, MCP-connected simulators for mobile applications, and local services for backend components. Verification instructions are stored in repository-specific skills under .claude/skills/, allowing agents to execute and validate their own changes before committing.

Osmani also recommends property-based testing to identify edge cases and differential testing to detect unintended behavior shifts between old and new implementations.

Keeping verification costs down

Skepticism persists about the economics of this approach. Some developers have questioned whether additional inference expenses justify the benefits, with Hacker News commenters suggesting that reusable Playwright tests could handle much of the verification without repeated model calls.

According to Anthropic, inference costs vary significantly depending on the change, and the company minimizes them by executing inexpensive deterministic checks before involving any model. Anthropic stated to The New Stack: "Deterministic checks like the build, linters, unit tests and property tests go first, because once Claude writes them they run as ordinary software and cost very little to rerun."

Model review is for what a test can't answer, like whether a change takes the right approach or misses a case nobody wrote a test for.

Anthropic

Anthropic declined to disclose specific figures on the added inference cost attributable to model-based verification.

Stopping agents gaming tests

A separate concern emerged from Hacker News user xnorswap: agents might silently alter code to make failing tests pass rather than addressing root causes. "It's not the per-run costs I worry about, it's the costs of churn," xnorswap wrote, noting that an agent could modify the test itself or introduce a temporary fix instead of solving the underlying issue.

Anthropic has implemented rules governing how Claude responds to failing tests and which repository sections its reviewer can access.

Our rule is that if a change can modify the check that judges it, the check is broken.

Anthropic

In practice, Code Review retrieves REVIEW.md from the base branch rather than from pull request changes, and review results apply only to the exact commit examined. Claude cannot skip flaky tests, insert delays, or repeatedly run a test until it passes; when the agent cannot identify the root cause, it removes the test and documents the reason.

The team also records how each commit underwent verification, including instances where no verification occurred.

Turning old bugs into guardrails

The Claude Code team has assembled 39 reliability checks, though Claude typically runs only three to ten per change. For modifications to session storage, for instance, Claude verifies that older transcripts still load with –resume.

Commit gates require developers to verify changes through /code-review or /verify. Teams can track compliance before enforcing the checks, and developers can override a gate if they provide justification.

Between June and September, Anthropic reports that merged pull requests to Claude Code's codebase more than doubled, while the rate of reverted changes fell by more than 40%. The company did not specify how much of this improvement stemmed from its verification loops or provide separate metrics for production defects versus regressions.

Grading the code reviewers

To evaluate its own reviewers, Anthropic tests them against historical incidents and reverts, reconstructing the repository state at the moment of review and creating a one-sentence answer key for each case. A reviewer receives credit only when its finding accurately describes the bug.

Previously merged changes that remained stable serve as controls for detecting false positives, and the company maintains a separate test set isolated from the examples used to refine the reviewers.

Anthropic measures recall at multiple points in the review pipeline, and even a reviewer that identifies a genuine bug is marked as a miss if the finding does not reach an engineer. The company currently does not publish false-positive or false-negative rates from Code Review evaluations.

Where engineers still earn their keep

Anthropic's framework provides guidance for determining which checks require model inference and which deterministic tools can handle more economically. Without published reviewer error rates or detailed inference costs, developers lack benchmarking data for their own implementations.

A person is accountable for every change that ships, whoever typed it

Anthropic

Engineers review the verification evidence, including test results and screenshots. When a check still fails after three attempts, Claude escalates the problem to an engineer.