The selection of a coding agent harness can dramatically affect operational expenses, with the same underlying AI model potentially costing five times more depending on which harness framework wraps it, according to findings from a new benchmark study. Runta and Baseten evaluated 22 different harness configurations paired with five open-weight models, executing 27,280 test runs that consumed approximately 83 billion tokens.

While the model itself drove 92.8% of cost variation, the harness selection still produced substantial differences. A harness that offered the lowest cost with one model could rank among the most expensive options with another, complicating optimization decisions for teams attempting to reduce expenses through harness switching or model substitution.

Nearly 65% of estimated inference spending went to runs that failed verification or timed out.

Harness Rankings Shift Dramatically Between Models

The FrontierHarness Eval v1.1 benchmark tested 22 configurations across five models—GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Muse Spark Open, and Kimi-K3—all served through Baseten infrastructure. Each setup ran 62 terminal-based tasks four times, with researchers tracking pass rates, inference costs and runtime performance.

Performance rankings proved unstable across model changes. Codex CLI achieved the lowest average cost per task with GLM-5.3 but fell to 16th position out of 22 when paired with Kimi-K3. DeepSeek Harness Creator demonstrated even more volatility, ranking first with Muse before dropping to 21st with GLM-5.3 Flash.

Across the five models, harnesses shifted a median of 11 ranking positions when the model changed, while costs varied by as much as fivefold even when the model stayed the same.

The most extreme cost disparity appeared with DeepSeek, where Cline CLI consumed five times the tokens of OpenCode v1, despite delivering only a two percentage point higher pass rate. To rule out random variation as an explanation, researchers shuffled trial sequences within each model 300 times. The actual cost gaps exceeded those from randomized trials with statistical significance (p-value of 0.003).

Failed Runs Dominate Token Consumption

Failed runs consumed 52.7% of estimated inference spending, with timeout failures accounting for an additional 11.8%. Only 35.5% of token expenditure went toward successful task completion. These figures, calculated using Baseten's published pricing, quantify a persistent challenge in coding agent reliability.

Among the five best-value harnesses, 68% to 82% of unsuccessful runs ended with the agent submitting a final answer that failed verification rather than timing out.

The researchers identified 216 instances of reward hacking in execution logs, with 163 involving online lookups. All such attempts received zero scores. Among the top-performing harnesses by cost efficiency, the majority of failures resulted from incorrect final answers rather than timeout conditions.

Step Count Emerges as Primary Cost Driver

Runta analyzed whether prompt size, tool availability, cache performance, or agent steps most influenced per-task expenses. Step count proved the only factor with statistically significant correlation to cost per successful completion, showing a 0.51 correlation coefficient.

Mini-SWE-agent illustrated how multi-step execution escalates token usage. Despite starting with the benchmark's smallest prompt at 882 tokens, it averaged 38 steps per successful task and ranked 17th out of 22 configurations for cost efficiency.

The five most cost-effective harnesses relied heavily on shell commands, which represented 61% to 93% of their tool invocations, and minimized subagent usage and context summarization. However, Runta did not test these design patterns in isolation, so the analysis cannot definitively attribute lower costs to architectural simplicity rather than other factors.