Anthropic's announcement of Claude Fable 5.1 highlighted a striking benchmark achievement: the model achieved a Terminal-Bench-Science score of 52.6%, compared to Fable 5's 24.7%. However, those results came from testing conditions unavailable to most users—each model received up to eight hours per task, and Anthropic has not disclosed the specific harness or budget parameters used. When the benchmark's independent leaderboard evaluated Fable 5, it consumed $14,180 across 210 attempts, averaging $67 per task.

To understand how these models perform under typical usage constraints, independent testing was conducted using five representative tasks from the benchmark's five science categories, each with a $12 budget and 60-turn limit per test. The results painted a different picture than Anthropic's published figures.

The tests

Five tasks were selected, one from each Terminal-Bench-Science category:

  • Symbolic regression (mathematics) – Identify a hidden formula from a dataset with 100 variables to predict yes-or-no labels on unseen data
  • Lorenz-96 assimilation (Earth sciences) – Reconstruct a chaotic atmospheric model using data from a few uncalibrated sensors with unknown clock offsets, graded on five all-or-nothing criteria
  • Reactor safety control (engineering) – Develop a controller for a chemical reactor that completes batches rapidly while maintaining temperature limits across known and hidden fault scenarios
  • Foraging cognitive model (life sciences) – Predict which lever each of 20 mice will press on a trial-by-trial basis, evaluated on unseen sessions
  • Nanoindentation (physical sciences) – Extract material properties from raw indentation curves containing drift, adhesion, defects, and an unknown tip shape

Each model received a plain terminal interface with a $12 spending limit and 60 turns per test. The complete testing cycle required approximately 12 hours.

Symbolic regression

This was the sole test where a model successfully passed the benchmark's hidden evaluation. Fable 5.1 completed the task in 27 turns over 11.8 minutes, generating 27,088 output tokens and spending $1.96. The model identified the underlying structure, created a working predictor, and halted independently.

Fable 5 exhausted all 60 turns across 53.5 minutes, producing 39,461 output tokens at a cost of $4.20, without solving the problem. A second attempt with Fable 5 consumed all 60 turns again, required 60 minutes, generated 60,608 output tokens, cost $6.38, and also failed.

Lorenz-96 assimilation

This pair of runs proved the most expensive. Fable 5 reached the $12 budget ceiling at 45 turns after 97.7 minutes and 92,091 output tokens, totaling $12.63. Fable 5.1 used the full 60 turns across 126 minutes, created 89,789 output tokens, and spent $10.70. Both models failed to solve the task. The independent leaderboard shows Fable 5 performing near zero in Earth sciences, consistent with these results.

Reactor safety control

Neither model produced a controller that satisfied the grader's test scenarios. Fable 5.1 generated the highest token count of any test, 157,710 tokens, before hitting the 60-turn limit after 40.9 minutes and costing $11.53. Fable 5 reached its $12 budget limit at 49 turns following 63.8 minutes, with 121,978 output tokens and $12.04 in charges.

Foraging cognitive model

This represented the longest test in the series. Fable 5.1 was the only model to signal completion, constructing a model, validating it against its own scoring mechanism, and declaring success at 43 turns after 53.5 minutes. It produced 65,518 output tokens and cost $5.65. The official grader rejected the submission.

Fable 5 never signaled completion. It hit the $12 limit at 60 turns following 139.3 minutes and 62,587 output tokens, ending at $12.13.

Nanoindentation

Both models failed. Each spent most of the run examining raw curves and writing segmentation code. Neither generated a results file that the grader would accept. Fable 5.1 exhausted its turns at 29.6 minutes, 115,687 output tokens, and $10.91. Fable 5 ran out of budget at 48 turns after 34.1 minutes and 114,239 output tokens, totaling $12.59.

Results

The benchmark evaluates models across all 70 tasks with three trials each. Anthropic's published scores of 24.7% and 52.6% derive from this comprehensive suite. The independent leaderboard places Fable 5 at 21.4%, closely matching Anthropic's figure. Fable 5.1 has not yet appeared on the independent leaderboard, so its 52.6% comes solely from Anthropic. The independent testing yielded 0% and 20%, falling below both published figures. Given that only five tasks were tested, these results represent a small sample.

The probability of obtaining these results by chance remains plausible even if the published scores are accurate, meaning this testing neither validates nor refutes Anthropic's doubling claim. The direction of results aligned with the published data, as Fable 5.1 performed better. Notably, the single task Fable 5.1 solved fell within mathematics, the field where the leaderboard shows Fable 5 achieving its strongest performance.

What I think

A typical user is unlikely to perceive a substantial performance difference between Fable 5 and Fable 5.1. While this testing covered only a limited sample and cannot definitively confirm or refute Anthropic's benchmark results, the evidence suggests the performance gap will not materialize for average users. The clearest distinction that emerged involved cost efficiency. Fable 5.1 failed faster and more cheaply, and never approached the budget ceiling, whereas Fable 5 hit the cost limit four times.

If your use cases resemble the benchmark tasks, both the testing harness and the budget become as critical as the model itself. Using a purpose-built harness, extended time per task, and substantially larger budgets could bring results closer to Anthropic's published numbers.

Source: The New Stack