Beyond Benchmark Numbers: Why the Inference Stack Matters More Than Model Size

Benchmark scores paint an incomplete picture of how open models perform in the real world. While these metrics showcase optimal conditions, production systems must deliver reliable results repeatedly under unpredictable circumstances. Nebius, an AI cloud provider, contends that the solution lies not in scaling up model parameters, but in engineering a more sophisticated inference architecture.

Engineers often assume that strong benchmark performance translates directly to production success. In reality, factors like input quality, task-specific accuracy requirements, and long-horizon workflows create gaps between lab results and deployed systems. Benchmarks cannot predict whether an AI system will complete its assigned task accurately or at all, particularly for complex, multi-step operations.

Ryan Hanrui Wang, SVP of Applied AI at Nebius, emphasizes the importance of combining public benchmarks with custom internal testing. "Customers often have their own benchmarks, so we can use their agent workflow and its traces to see the level of success we can work with. We always work closely with them to define the benchmarks," Wang explains. He joined Nebius following its acquisition of Eigen AI, an inference-optimization startup he co-founded and led as CEO, in a deal valued at approximately $643 million announced in May. Nebius delivers inference capabilities through its Token Factory platform.

Wang notes that standardized benchmarks frequently fail to capture the diversity of tasks required in actual deployments. "The public benchmark may not cover the multi-domain tasks that an engineer wants inference to do in the actual production environment," he states. This mismatch can obscure the true cost of inference operations and misalign agent workflows with what benchmarks actually measure. "The workflow is not tested by the benchmark design. There needs to be in-context learning."

Rather than investing excessive effort in model selection alone, organizations should evaluate the complete inference stack. The choice between a cost-effective, flexible open-model approach and a frontier model should depend on specific business requirements and accuracy thresholds.

For cost-conscious deployments, Nebius assesses open models including Kimi and DeepSeek, alongside lighter "flash" variants, to determine whether they meet accuracy targets and whether fine-tuning is necessary. Organizations prioritizing superior accuracy typically narrow their focus to two or three leading open models, then apply fine-tuning to boost performance on their specific tasks.

Inference and post-training

Post-training leverages organizational data to enhance accuracy and task alignment. The serving layer—encompassing quantization, batching, caching, and kernel optimization—drives inference speed. Engineers must also design cache systems for production traffic, maximizing hit rates so repeated queries retrieve cached results rather than triggering recomputation.

Throughput, measured as tokens generated per unit time, represents a critical trade-off against responsiveness. Wang explains: "The other important thing is throughput, the measure of how many tokens are generated in a certain time span, as they are a major trade-off against interactivity." Teams must balance efficient resource utilization and lower per-token costs against acceptable per-user response speeds.

Concurrency management requires careful calibration. Underutilized compute raises token costs, while aggressive concurrency can degrade individual user latency. Monitoring tail latencies—not merely average response times—proves essential. Wang emphasizes: "P95 and P99 are important metrics for the per-user token speed and time to first token (TTFT); the time they need to wait needs to be below five seconds for some of our customers." A system with excellent average performance can become unusable if occasional requests experience significant delays.

Modeling for accuracy

Benchmark selection matters as much as benchmark scores. A test measuring mathematical reasoning may prove irrelevant for a customer support agent where emotional intelligence carries greater weight. Wang observes: "So there is a misalignment between the different benchmarks and actual production." Cost-per-successful-task calculations must account for model accuracy; inexpensive tokens provide no value if the model generates incorrect outputs.

Evaluating the complete end-to-end workflow enables engineers to select appropriately-sized models for their specific tasks. Nebius refined a small model using 6,000 traces from a specialized assistant agent, achieving an 11% accuracy improvement while reducing tool calls by 22%. This example challenges the assumption that larger always means better; accuracy gains from scaling model size eventually plateau.

"There is typically a tipping point for this kind of task," Wang says. For customer support applications, 10 billion parameters may suffice, while 100 billion or even trillion-parameter models deliver only marginal improvements. "What we want to do is find the sweet spot."

Engineers can use large models as reference points, then refine reasoning with proprietary data to identify the optimal model size and accuracy balance. "We typically do a sweep of several models, using the largest model as a reference to know where the sweet spot for model size is," Wang explains. "For a complicated task, we can dissect the complex task into smaller subtasks, working with the customer, which connects workflows and can then use smaller models."

Wang frames this methodology as a division of labor, invoking economist Adam Smith's principles. This progression reflects AI engineering maturation: early enthusiasm favors scale, but as the field matures, focus shifts toward outcomes and engineering rigor, often requiring partnerships with specialists who understand how to achieve them. Sometimes a larger model proves optimal; other times a smaller, specialized approach delivers superior results.