A group of AI coding agents completed a 3D browser reconstruction of San Francisco's Union Square in just two hours, consuming approximately 8 million tokens at a cost of roughly $33 in API charges. PhiloLabs conducted this experiment to explore the capabilities of AI agents when tasked with building applications where functional correctness alone is insufficient.

The agents worked from real-world geographic data and reference photographs to construct a Three.js implementation featuring 453 building footprints, 75 custom façades, and 129 named storefronts. The scene also included dynamic elements: 220 pedestrians and 109 vehicles navigating the environment, with Powell Street's cable cars among the moving objects.

Beyond simply achieving a working application, PhiloLabs wanted to demonstrate how agents could identify visual problems that conventional testing frameworks would overlook. The team integrated Playwright into the development workflow to capture and analyze visual output.

Playwright as agent vision

PhiloLabs organized the project into specialized subagents responsible for geographic research, building geometry, textures, storefronts, and other scene components. Once the browser-based version was operational, Playwright captured screenshots from 34 predetermined camera angles, which could then be compared against actual photographs of the real Union Square.

The agents generated 147 comparison sheets that made it straightforward to identify discrepancies between what was technically accurate and what appeared visually incorrect. A building might occupy the correct coordinates but display wrong proportions, or a storefront could end up positioned on the wrong side of a street. Consistent camera positions across iterations also enabled tracking of changes between successive development cycles.

In all, the agents produced 147 comparison sheets, making it easier to spot things that were technically correct but still looked wrong.

Agents reviewing agents

PhiloLabs assigned specialist agents to evaluate the output, with different agents focusing on architecture and geography versus technical art and interactions. These reviewers collectively produced nine reports documenting issues with the Union Square reconstruction.

The reports served as a task list for subsequent development iterations. Development agents could then address the identified problems and regenerate the scene.

This multi-agent review structure addresses a fundamental challenge: not all errors translate into testable conditions. Verifying that a building sits at the correct coordinates is straightforward. Determining whether a street genuinely resembles Union Square is substantially more difficult to encode as an automated test.

Filling gaps in source data

The agents did not begin with a finished 3D model ready for replication. Instead, they synthesized open geographic data—primarily OpenStreetMap and USGS elevation information—along with reference material about the actual location, transforming this data into geometry, façades, and browser-compatible objects.

Significant data gaps remained throughout the process. Geographic databases could indicate where a building belonged without revealing its façade appearance, while photographs only showed building faces from specific viewing angles. When neither source provided complete information, the agents had to make independent decisions about details.

Having another agent review the work does not guarantee correctness, particularly when source material is inherently incomplete. A reviewer can overlook the same issues that the original agent missed.

What $33 buys

Eight million tokens represents substantial model activity for a single project. The reported API expenditure for the Union Square experiment was approximately $33.

PhiloLabs achieved this cost efficiency by distributing work across subagents, executing tasks concurrently, and leveraging cached context to avoid having a single agent reprocess the entire project repeatedly.

However, Union Square represents a relatively straightforward experimental scope. Screenshot-based validation becomes less practical as agents tackle more complex applications. Spline recently rebuilt its 3D editor using Claude Code agents, but the resulting interface displays only a portion of what those agents constructed. Issues embedded within code or triggered through actual user interaction patterns may never surface in a screenshot.

Eight million tokens is a lot of model activity for a single application. Yet the reported API cost for the Union Square run was about $33.

Source: The New Stack