OpenAI's Astra Shows Why the Software Wrapper Around AI Models Now Matters More Than the Model Itself

When ARC Prize tested GPT-6 Astra through its own standard harness, the model scored 62.7%. Running the identical model through OpenAI's Provider Adapter produced a 98.6% result. The model itself remained unchanged; only the software framework shifted. That 36-point gap reveals a fundamental truth about modern AI systems: the harness engineering now rivals model selection in importance.

The performance difference carried financial implications as well. OpenAI's adapter completed the task for $17,332, while ARC Prize's harness cost $26,098. Across 167 game-reasoning pairs that both systems solved, the Provider Adapter consumed 49% fewer tokens and ran roughly 3.66 times faster. Investor Matt Turck, whose podcast has featured the architects behind ARC-AGI, captured the significance with a comment: "This is wild," he said, adding in parentheses: "w/ its native harness."

The benchmark measures the system, not the model

ARC-AGI was designed to resist simple scaling approaches that compromise other benchmarks. This year's ARC-AGI-3 iteration elevated the challenge further by placing models into interactive environments without instructions, explicit goals, or stated rules, then measuring how quickly they learn to function. When ARC Prize released the benchmark, humans achieved 100%, while Frontier AI managed just 0.51%.

The two harnesses operate differently in ways that matter. ARC Prize's standard system allows models to retain notes of their choosing. OpenAI's adapter preserves the opaque reasoning state between requests and compresses extended conversations, enabling the model to resume its own thinking rather than starting from scratch. These architectural choices explain much of the performance variance.

The scoring spread across reasoning levels tells the story. Astra running through OpenAI's harness with no reasoning effort scored 96.7% for $23,457. The same model at maximum reasoning within ARC Prize's standard harness achieved 62.7% and cost $26,098. The harness itself outperformed the reasoning setting by a substantial margin.

OpenAI has not concealed how the adapter functions—it relies on documented Responses API capabilities available to anyone. However, the complete assembled system that delivered the 98.6% score remains unavailable for purchase. This distinction matters when evaluating claims about progress. OpenAI President Greg Brockman stated during a press briefing that "I think it's not unreasonable to feel that we are now in the AGI era," but this assertion describes a benchmark outcome produced by a specific system configuration, not proof that the underlying model itself constitutes AGI.

The harness is becoming the product

Frontier models now cluster within a few percentage points on coding tasks. Artificial Analysis measures its Coding Agent Index by evaluating each model within a harness rather than in isolation: Astra scores 67 in Codex, Opus 5 and Fable 5 reach roughly the same level in Claude Code, Muse Spark 1.3 performs similarly in Muse Code, and Fable 5.1 leads at 70 in Claude Code. The unit being assessed is already the pairing of model and harness.

The major AI labs recognized this shift months ago. Anthropic, OpenAI, Google, and Microsoft each treat the harness as a commercial product, disagreeing only on pricing models. Anthropic charges $0.08 per session hour for Managed Agents, in addition to token costs. OpenAI released its Agents SDK without any runtime fee. Google and Microsoft bill sessions, memory, code execution, and observability as separate line items. None treat the harness as a complimentary add-on to the model.

This pattern extends beyond the major labs. Stripe acquired OpenRouter in August for a reported $8 billion, gaining control of a routing layer that processes 10 trillion tokens daily across more than 400 models for 10 million developers. The company framed tokens as the central currency for AI-powered businesses, and the acquisition targeted the infrastructure layer positioned in front of the models themselves. Nvidia has constructed its own harness as well.

Nvidia's approach illustrates the principle. Claude Opus 5 scores 30.2% on ARC-AGI-3's public benchmark when operating independently. When wrapped in Nvidia's AVO system, which provides persistent memory and programmatic supervision that activates when progress stalls, it cleared all 183 levels across 25 environments. Nvidia articulated the concept succinctly: "Model capability matters enormously, but the surrounding system determines how effectively that capability can be converted into sustained autonomous progress."

Harness engineering is the job

Token consumption varies dramatically depending on the harness. Testing an identical model across Aider, Claude Code, and OpenClaw revealed a 70-fold variation in token usage. Cache hit rates ranged from approximately 70% down to 1.5% based on the serving path. No model selection alone accounts for such disparity.

The engineering work itself involves conventional decisions: determining what an agent retains in memory and what it discards, defining what it can access and when it must pause for human approval. These choices compound into measurable differences in efficiency and capability.

The skill hierarchy is shifting. Model selection was once the critical hiring criterion, but that advantage erodes as frontier models converge. Within a year, teams operating agents will likely allocate less time to model selection and more effort to engineering the systems surrounding them. The model remains important, but the architecture that deploys it increasingly determines real-world outcomes.

Source: The New Stack