This week, Amazon Web Services unveiled AWS Emulator, a locally-runnable replica of AWS infrastructure designed to let coding agents validate their work. The tool encompasses support for over 200 AWS services and was developed entirely through agent-driven coding. The most compelling aspect of this release centers on an anecdote shared by Swami Sivasubramanian in the announcement.

Two decades ago, Sivasubramanian participated in creating the replication engine for Alf, a Paxos-based lock service that continues to underpin AWS infrastructure including S3 and DynamoDB. The original implementation required approximately four months of work. When he later tasked Kiro, AWS's agentic coding system, with reconstructing it under his supervision, a critical limitation emerged after four nights of effort: the agent lacked the ability to validate its own output. Once provided with a specification and test environment, the same agent completed the rebuild in roughly two hours.

The transformation from four nights to two hours reveals a fundamental principle about agent effectiveness. The difference lay entirely in the feedback mechanism available to the agent.

An agent is only as good as its verification loop, and a verification loop is only as good as what it runs against.

Swami Sivasubramanian

Sivasubramanian emphasizes that testing distributed systems presents inherent challenges. This observation aligns with patterns observed across organizations deploying coding agents on production infrastructure. While AWS Emulator addresses verification for AWS API interactions, it does not solve the problem for proprietary services developed internally—which is precisely where agent-generated modifications most frequently encounter failures.

AWS puts infrastructure inside the agent's loop

Coding agents operate through iterative cycles: generating code, executing it, analyzing results, and determining necessary modifications. Sivasubramanian's insight highlights that within distributed systems, this cycle depends on infrastructure behavior. When invoked services do not behave identically to production versions, agents draw incorrect conclusions and continue down problematic paths.

AWS Emulator brings AWS services into this loop on developer machines. By pointing existing AWS tools and code at the emulator using a single environment variable, test execution occurs against simulated services without requiring code modifications.

Agents authored every component of the emulator, drawing from AWS's published API specifications. Each implemented behavior undergoes fidelity validation—first passing against actual AWS, then against the emulator—and all modifications pass through a merge gate where security-focused agents perform review. Sivasubramanian's perspective is that when agents generate code, engineers shift focus toward establishing specifications, tests, and merge gates that determine correctness.

Initial commentary on Hacker News immediately raised concerns about trustworthiness. One user noted approximately 30,000 commits accumulated over three weeks and questioned the resulting quality. This represents a legitimate concern directed at the wrong metric. The identity of the code author provides limited insight. The systems against which code undergoes verification provide substantially more information—in this case, actual AWS infrastructure.

Slow or unrealistic feedback was a tax on people. For agents, it decides whether the loop works at all.

Swami Sivasubramanian

The fundamental shift involves who executes the verification loop. A human developer typically validates changes several times daily, while an agent performs validation on each iteration. For people, sluggish or inaccurate feedback represented an efficiency cost. For agents, it determines whether the verification mechanism functions at all.

Every loop runs against mocks, an emulator, or the running system

Agents can verify their output through three distinct approaches, each presenting particular limitations.

Mocks represent the weakest verification method, though artificial intelligence has reduced their creation costs substantially. A Hacker News contributor suggested that AI-generated mocks now accelerate test coverage development, noting that "AI has been a boon for test coverage." However, when agents generate their own mocks, they construct them according to their own internal assumptions. Tests succeed because they were designed to succeed, resulting in increased coverage metrics without corresponding increases in actual confidence.

Emulators provide meaningful advancement. AWS Emulator undergoes comparison against actual AWS services, meaning its behavior derives from the genuine service rather than from agent assumptions.

Emulators remain models nonetheless, and AWS acknowledges this reality openly. The project repository maintains a directory documenting behaviors agents discovered that diverge from actual AWS, and unimplemented operations return errors rather than simulated success. An IDC analyst observed that the emulator currently lacks simulation of throttling, service outages, or per-Region availability differences—the exact scenarios where retry and failover mechanisms receive testing.

An agent that writes its own mocks writes them to match its own assumptions. The tests pass because they were built to pass, so coverage goes up while confidence should not.

Swami Sivasubramanian

Sivasubramanian draws parallels to his own experience with Alf. Simulation resolved most concurrency-related defects before deployment to actual servers. Subsequent testing on real infrastructure revealed different categories of problems, because "the real world exposes things a simulation cannot."

Infographic detailing the three things a loop runs against.

You can't emulate your own system the way AWS emulated its own

In the same Hacker News discussion, someone identifying as an AWS employee described the strategy most organizations currently employ with agents: isolate as much of the system as feasible, mock or simulate unavoidable external dependencies, then permit the agent to operate. This strategy succeeds when the isolated component is a cloud service featuring a documented specification. It fails when that component represents proprietary internal infrastructure.

AWS Emulator functions because AWS possessed two resources most organizations lack for their own applications: a comprehensive, machine-parseable specification of every operation and a stable reference implementation in actual AWS against which to validate.

Consider a payments infrastructure supporting 60 microservices. Service contracts exist scattered across implementation code, several OpenAPI documents that have diverged over months, and the collective knowledge of original developers. No unified specification exists, and the authoritative reference is the operational system itself, which evolves with each merged change. When agents generate pull requests multiple times daily, the system transforms faster than any model could track.

This scenario also represents where agent-generated application code typically fails. Failures seldom involve how a managed queue manages message visibility. They involve a field renamed in one service that a downstream consumer still expects, event structures that receiving services cannot process, queries performing acceptably on test data but slowly on production volumes, or configuration, feature flags, and credentials existing only within the running cluster. No cloud provider emulator will ever encounter these issues.

The environment belongs on the engineer's list

Sivasubramanian identifies specifications, tests, and merge gates as core engineer responsibilities in this new paradigm. The environment merits addition to this list, as it determines what agents can detect.

A production-grade verification loop exhibits six essential characteristics. Changes execute within an environment mirroring production, alongside the actual services, datasets, and configuration they will encounter after deployment. Each modification receives its own isolated environment, preventing interference when multiple agents operate simultaneously. Environment initialization completes in seconds, because agents that wait 20 minutes abandon the mechanism and revert to unit testing. Agents independently create environments and retrieve results without requiring manual intervention or ticket submission.

The loop additionally executes integration and end-to-end test suites the team already relies upon, rather than tests the agent created to validate itself. It generates artifacts that reviewers or automated merge gates can evaluate before integration.

The central difficulty involves achieving production-equivalent fidelity while maintaining rapid startup times. Provisioning a complete stack replica for each change provides fidelity but sacrifices speed and cost efficiency. A ten-person engineering team running multiple concurrent agent sessions might generate 100+ pull requests daily requiring verification. No organization can absorb the expense of maintaining 100+ full system copies.

Emulate what you don't own, and run what you own for real

Emulators and production-equivalent environments represent complementary layers rather than competing approaches.

For managed cloud services offering published interfaces, emulators provide agents with rapid, cost-free feedback on developer machines. When code interacts with S3 and queue services, AWS Emulator identifies API misuse within seconds without consuming AWS account resources.

Emulators and production-like environments are not competing ideas. They are layers in the same loop.

For proprietary infrastructure, deploy exclusively the components a change modifies rather than replicating the entire stack. Route requests and messages designated for that change's environment to new versions, while serving all other traffic from the shared stable environment already operational. Implement lightweight per-environment schemas or topics in databases and queues only where the change requires data isolation. This approach enables environment startup in seconds while maintaining production-like behavior. Uber and Lyft constructed internal systems following this identical pattern for identical reasons.

Infographic detailing a framework where you emulate what you don't own, and run what you own for real.

Configure the merge gate to require both validations. A pull request passing both emulator verification and validation against the target system represents code a reviewer can confidently approve.

Every agent will have a loop, and realism is what separates them

Sivasubramanian's agent accelerated from four nights to two hours once it gained the ability to validate its work. For application teams, the subsequent acceleration depends on what the agent validates against. Cloud API coverage comes from emulators, while only the operational system covers proprietary services.

Implementing a verification loop has become standard practice. The fidelity of that loop determines how much confidence the resulting code merits. Organizations seeking agents to validate every change within production-equivalent environments before merging must establish this as their default operating model.