Claude Fable 5.1 topped a newly released coding benchmark while simultaneously demonstrating substantial limitations. The model achieved a 38.8% success rate on Real-SWE, a test developed by Y Combinator-backed Specific Labs that departs from conventional evaluation methods by using proprietary codebases from operating companies rather than publicly available repositories.
The benchmark's distinguishing feature lies in its reliance on private code and solutions that remain inaccessible to the public. This approach reduces the likelihood that models encountered the test material during training. Specific Labs estimates that 99% of tokens present in real-world enterprise environments remain hidden from frontier models. When agents were placed in these unfamiliar contexts, performance metrics declined sharply.
Benchmark results across leading models
Fable 5.1 running via Claude Code led the rankings at 38.8%. GPT-6 Astra on Codex CLI placed second with 33.8%, followed by Gemini 3.8 Flash on Gemini CLI at 31.2%.
Performance dropped considerably beyond the top three. GLM 5.3 scored 28.8%, Grok 4.6 and Muse Spark 1.3 tied at 23.8%, Kimi K3 achieved 18.8%, and GPT-5.6 Sol finished at 16.2%. Each model received eight attempts per task, and scores reflected the complete system including each model's paired coding tool rather than the model alone.
The testing framework demonstrates that scaffolding choices surrounding a model influence its results. Fable 5.1 might produce different outcomes if deployed through Cursor instead of Claude Code, similar to how GPT-6 Astra's ARC-AGI performance shifted based on its implementation environment.
Task-level performance reveals deeper struggles
Real-SWE draws proprietary code licensed from functioning businesses, including a consumer application serving over 200,000 users and a fintech system that has processed more than 100,000 bank statements. Solutions within the benchmark span across codebases, with Real-SWE fixes touching a median of 11 files—nearly double the six-file median found in comparable benchmarks like FrontierCode and DeepSWE.
Individual task results proved even more challenging than overall scores suggested. Six of the 10 tasks showed success rates below 15%. A billing schedule migration achieved a 14.1% fix rate, API token metering reached 12.5%, S3 storage tracking hit 10.9%, and a linearizable scan managed only 4.7%, while a tax jurisdiction bug was resolved just 3.1% of the time.
Not a single model solved the analytics stream reducer across 64 attempts.
Astra and Gemini achieved perfect scores on a multi-region sweep task, with Fable solving seven of eight attempts, yet all three models failed every attempt at the linearizable scan. No agent demonstrated consistent reliability across the full benchmark.
Failure patterns across models
Fable 5.1's failures clustered around two primary categories: missing requirements accounted for 36.7% of failures, while integration errors represented 34.7%. Astra's failures split evenly between integration errors and unverified assumptions, each at 34%.
Integration errors dominated nearly half of Gemini 3.8 Flash's failed attempts. GPT-5.6 Sol made unverified assumptions in 43.3% of its failures, indicating different failure modes across the tested agents.
Implications for real-world deployment
Real-SWE does not definitively establish that public coding benchmarks suffer from data contamination inflation, and 10 tasks represent a limited sample size. However, the leading agent's failure rate exceeding 60% on private code it almost certainly had not encountered during training suggests a substantial distinction between solving isolated coding problems and navigating unfamiliar production environments.