Anthropic unveiled Claude Fable 5.1 on September 1, touting a doubling of performance capabilities for agentic research work. When tested directly against its predecessor across four real-world tasks with careful token tracking, both models achieved flawless accuracy. However, on the most demanding test, the newer version consumed more than twice the tokens and generated substantially higher costs.

The company positioned Claude Fable 5.1 as "our most advanced model for coding and knowledge work." The release generated considerable enthusiasm in developer circles. CEO Dan Shipper declared after a week of evaluation that it represented "the strongest coding model we've used." AI analyst Min Choi documented instances of users "one-shotting games, building 3D worlds + creating insane simulations" within 24 hours of availability.

Anthropic highlighted the Terminal-Bench-Science benchmark as evidence of improvement, where Fable 5.1 achieved 52.6% versus Fable 5's 24.7%. This benchmark presents models with a terminal interface and multi-step scientific research assignments, measuring the percentage of tasks completed successfully. The pricing remained identical: $10 per million input tokens and $50 per million output tokens.

Yet benchmark performance often diverges from practical application. Standardized tests measure constrained problem sets under conditions vendors help shape, and some organizations have optimized models specifically for evaluation criteria. While no suggestion is made that Anthropic engaged in such practices, the structural incentives underlying benchmark marketing remain inherent to the process.

A 52.6% research benchmark score provides limited insight into whether the model's actual output improves for specific user needs. To understand what these metrics mean in practice, testing was conducted on four representative tasks reflecting genuine use cases.

The tests

  • Agentic research: Processing experimental data containing five intentional errors, with lab documentation explaining detection methods. The model must filter corrupted entries, calculate batch averages, and document findings.
  • Agentic coding: A Python project with two deliberate bugs and a failing test suite. The model must identify and resolve both issues until all tests pass.
  • Reasoning: Two mathematical problems with verified correct answers, requiring only analytical thinking without terminal interaction.
  • Sensor data audit: Problematic readings from five sensors with all issues documented in equipment logs, including timing drift, mid-operation hardware replacement, data corruption, and unit inconsistencies. Included as a final differentiator.

Prompts are typically shared for reproducibility, but these tests rely on data file folders with embedded errors, rendering prompts meaningless without the accompanying files.

Agentic research

Both models completed the task flawlessly in three iterations. Each correctly interpreted the lab notes and removed precisely the five problematic rows, including the subtle scenario where a duplicated trial's initial entry is corrupted while its repeat is sound. Both calculated batch means matching verified answers exactly.

Fable 5.1 demonstrated marginally superior speed (19.2 seconds versus 20.6 seconds) and cost efficiency ($0.086 versus $0.100). According to the benchmark this task mirrors, Fable 5 supposedly fails in roughly 75% of instances. In this evaluation, it produced no errors.

Agentic coding

The coding evaluation produced comparable outcomes. Both models executed the test suite and identified the obvious bug—a remove function performing addition instead of subtraction. Both also detected the subtle error: an off-by-one mistake in a threshold check. Each corrected both issues, finishing with all 8 tests passing in three iterations.

Fable 5.1 completed in 13.6 seconds while Fable 5 required 17.2 seconds, with both runs costing $0.07. The newer model showed speed advantages, but performance remained equivalent otherwise.

Reasoning

Anthropic's published data predicted minimal separation here, and results confirmed this expectation. Both models solved the two problems correctly with valid step-by-step reasoning. Fable 5.1 was marginally quicker on both: 12.0 seconds against 12.5 on the first problem and 9.7 seconds against 10.7 on the second. It also proved more economical with output tokens, consuming 771 versus Fable 5's 1,045 on the first problem and 647 versus 798 on the second.

Sensor audit data

This served as the deciding test. After three rounds of perfect accuracy parity, a fourth challenge was introduced specifically designed to be more demanding, since a model claiming to double its predecessor's capabilities should demonstrate that advantage somewhere.

Both models navigated every obstacle successfully. Each adjusted the fast clock before applying time-window filtering, which also eliminated two anomalous readings. Each properly calibrated the swapped sensor at the transition point, removed corrupted entries, and converted temperature units after calibration rather than before. The resulting files remained identical and completely accurate.

Divergence emerged in execution metrics. Fable 5 completed in four iterations, 23.9 seconds, and $0.134. Fable 5.1 required five iterations, consumed 28.4 seconds, and cost $0.304—more than double the price. The additional iteration drove the cost increase, as each turn resubmits the full conversation history. Fable 5.1 transmitted 23,602 input tokens through the API compared to Fable 5's 7,940.

Fable 5 was as accurate, cheaper, and faster on the hardest task of the set. I wasn't expecting that.

Results

Both models achieved perfect scores of 24 out of 24 across all four tests. Fable 5.1 completed the full evaluation slightly faster at 82.9 seconds against 84.9 seconds, but consumed 70% more tokens and incurred 34% higher costs (results were influenced by the sensor test, though the overall comparison still stands).

These findings don't invalidate the Terminal-Bench-Science results. That benchmark was constructed using lengthy, complex research tasks where Fable 5 purportedly fails most of the time. However, this scenario may not reflect how most users actually deploy Fable 5. Some certainly do, but many do not.

Two important limitations warrant mention. The claimed cost reductions depend on cache read pricing, which has declined 75%. The short tasks in this evaluation didn't utilize caching at all, so cost comparisons don't validate that claim. Additionally, Anthropic notes that Fable 5.1 was benchmarked with production safety measures enabled, which occasionally reduced its own scores.

The search for a 2x improvement yielded a model indistinguishable from its predecessor in practical testing. This doesn't mean differences don't exist, but for routine work already handled by Fable 5—bug fixes, data processing, analysis with known complexities—the upgrade may produce no visible difference.

I went looking for a 2x improvement and found a model I couldn't differentiate from its predecessor.

For users currently running Fable 5 on work resembling these tests—code repairs, data sanitization, analysis involving documented edge cases—upgrading will not alter outcomes. On extended agentic research tasks, costs may actually increase per execution. If your workload matches the benchmark profile, involving multi-hour research agents with high failure rates, Anthropic's published data indicates Fable 5.1 delivers the claimed gains. Constructing such a test in limited time proved infeasible.

Source: The New Stack