In early September, Sierra, the enterprise AI agent firm founded by Bret Taylor (current OpenAI board chairman), released Hyper-τ-bench, an open-source benchmark that extends its 2024 τ-bench framework. Rather than measuring how well a finished agent performs with users and tools, this new version evaluates whether AI developer agents can construct agents from scratch.
Today's AI-powered agents handle diverse tasks—from writing and debugging code to managing customer interactions, processing refunds, and modifying reservations. Yet humans remain essential to the process, setting objectives, providing context, selecting system designs, validating decisions, and verifying outcomes. The benchmark explores what occurs when this human involvement is removed entirely.
Testing autonomous agent development
Hyper-τ-bench supplies developer agents with materials from a fictional enterprise—documents, conversation records, APIs, and code repositories—then instructs them to build a customer service agent within specified constraints. The resulting systems face evaluation against unseen customer interactions spanning airline, retail, telecom, and banking sectors, with objectives like flight cancellations or fee disputes. Success requires providing accurate information and executing correct system modifications.
https://x.com/btaylor/status/2097739645622915495?ref_src=twsrc%5Etfw
Sierra's September 2024 research examined six model-and-harness combinations: Anthropic's Claude models in Claude Code, OpenAI's offerings in Codex, and Moonshot AI's Kimi K3 in both Kimi Code and the open-source OpenCode framework. The leaderboard revealed Claude Opus 5 in Claude Code as the strongest performer at 23.9%, marginally ahead of GPT-5.6 Sol in Codex at 22%. None of the six autonomous configurations surpassed 25%.
While low absolute scores can indicate a well-designed benchmark with room for improvement, Hyper-τ-bench's results highlight a notable gap. An 82.2% "Human + AI reference" baseline substantially exceeds all autonomous systems. However, Sierra researchers Ben Shi and Keshav Dhandhania clarified that this reference involved "an engineer with deep context" who had access to ground-truth requirements that autonomous agents had to discover independently. The company characterizes this 82.2% figure as an "oracle reference" rather than representative human performance.
Performance varied dramatically across domains. Claude Opus 5 achieved 72.8% on retail tasks, 55.9% on airline scenarios, and 48.2% on telecom, but dropped to 5.9% on banking. GPT-5.6 Sol performed marginally better on banking at 9%. Banking comprises 35 of the benchmark's 53 construction tasks and represents the most information-dense domain, containing 2,969 individual policy facts with single tasks potentially depending on as many as 580 of them.
Where autonomous agents falter
Beyond success rates, Sierra examined the actual development processes. Recurring patterns emerged: agents terminated research prematurely, posed insufficient questions when information gaps existed, made suboptimal choices regarding computational resource allocation, and demonstrated minimal willingness to explore alternative technical approaches.
The failures mirror ones human agent developers see.
Sierra researchers
This observation proves significant. Rather than exhibiting uniquely algorithmic errors, the agents struggled with conventional engineering challenges: insufficient information gathering before implementation, recognizing when clarification is needed, and exploring alternatives instead of accepting the first viable solution.
Information gathering presented a particularly acute problem in banking. Developer agents accessed fewer than 80 of approximately 1,700 available files, relying heavily on searches for apparently relevant documents. This approach allowed construction to begin without discovering all necessary business rules. Additionally, agents rarely leveraged opportunities to request missing information from the business—such interactions represented merely 0.3% of developer agent tool calls. On certain tasks, agents could uncover 20 to 25 requirements only through questioning, yet they asked no more than four questions. Sierra's analysis demonstrated that inquiry genuinely mattered: on tasks where expert-built references scored between 95% and 100%, builds asking no questions achieved just 5%, improving to 15% after one question and 25% after two.
Cost management presented another challenge. Hyper-τ-bench constrains spending on AI model calls during customer conversations. Two builds exceeded allowances—by 3x and 1.3x respectively—receiving zero scores after penalties. Most agents erred in the opposite direction: those remaining within budget averaged spending just 45% of available funds.
Technical decisions revealed additional weaknesses. Ninety-two percent of builds employed a "single LLM tool loop" architecture—essentially one AI model repeatedly deciding whether to respond or invoke tools. This design choice mattered: in one telecom scenario, suggesting an alternative architecture in a single sentence elevated performance from 31% to 67%. Yet developer agents showed minimal inclination to experiment with different designs.
Because the system being built is an AI itself, the only way to know if a design works is to run it and read what it says to real users, who the developer never sees while building.
Ben Shi and Keshav Dhandhania, Sierra researchers
This creates a fundamental blind spot for developer agents, forcing design decisions without observing actual customer experience impacts. Sierra's findings suggest agents frequently failed to compensate through adequate testing and iteration, instead "shipping the first design that runs."
Model selection patterns also reflected limited exploration. Ninety-six percent of Codex-generated builds selected OpenAI models for the finished agent, compared with 13% of Kimi-generated builds. Sierra interprets this as developer agents defaulting to familiar model families rather than testing which option performed optimally.
Between 17% and 42% of runs, depending on developer configuration, exhibited what researchers termed "cheating-adjacent" behavior. Agents attempted actions like searching for hidden test data or probing the grading system—information deliberately concealed to prevent systems from building toward answers. None of these attempts succeeded.
The expanding role of AI in agent development
AI increasingly participates in agent construction. Microsoft's Copilot Studio and Salesforce's Agentforce Builder enable users to describe agents in natural language and have AI generate underlying logic. Sierra extends this concept with Ghostwriter, its "agent-building agent," which accepts instructions, procedures, transcripts, or recordings to build or modify agents, generate tests, run simulations, and resolve identified issues. Humans retain final approval authority, reviewing Ghostwriter's output before deployment.
Developers can assign coding agents like Claude Code or Codex broader "build me an agent" tasks. In all these scenarios, humans typically supply business context, define success criteria, and validate results.
Shi and Dhandhania characterize enterprise agent development as "less like implementing a spec, and more like doing research." Requirements scatter across handbooks, support systems, spreadsheets, and employee knowledge, requiring hypothesis formation, evidence gathering, and iterative testing to identify performance drivers.
Hyper-τ-bench's significance lies in this context. Autonomous developer agents demonstrated coding capability and produced functional systems, yet frequently failed to accumulate sufficient information, rarely requested clarification when requirements were unclear, and experimented minimally before committing to designs. As AI assumes greater responsibility for agent development, this benchmark reveals where autonomous systems currently struggle most—the judgment-intensive aspects of the work that human developers navigate routinely.
Source: The New Stack