Chip Huyen is set to return to P99 CONF on October 21–22, and her October 2025 presentation on reducing inference expenses deserves renewed attention in an era dominated by agentic systems. The online conference draws developers working on high-performance, low-latency systems, and Huyen—who authored the widely-read AI Engineering—delivered a keynote that laid out the fundamental challenge: training a frontier model represents a single expenditure, while inference costs accumulate repeatedly.
Huyen framed the problem with straightforward arithmetic. Across a model's lifetime, the compute allocation between training and inference typically ranges from 1:10 to 1:100, with reasoning models pushing that ratio even further toward inference. This imbalance creates a stark economic reality: if inference remains prohibitively expensive, the initial training investment never gets recouped. Huyen distilled months of research from her book into a 30-minute conference talk, focusing on practical optimization strategies.
Key metrics to track
https://www.youtube.com/embed/-MIv3mWAlBc?feature=oembed
Huyen identified several critical latency measurements that matter for inference performance:
- Time to first token (TTFT): elapsed time before the user sees output
- Time per output token (TPOT): average duration between successive tokens, also called inter-token latency
- End-to-end latency: TTFT plus TPOT multiplied by output token count minus one
Reasoning models introduce a complication: some generated tokens never display to users. Huyen explained: "The first generated token might not be the same as the first visible token," noting that models may perform internal reasoning before revealing the first token of the final answer. Some practitioners measure Time to Publish instead, tracking when users first see output. The appropriate metric depends on user priorities.
Beyond latency, Huyen advocated measuring "goodput" alongside throughput. Throughput counts requests processed within a time window; goodput counts only those meeting performance targets. Her illustration: an application targeting 200 milliseconds for TTFT and 100 milliseconds for TPOT processes 10 requests per minute, but only three satisfy both constraints.
Three optimization angles
Inference optimization can be approached through hardware, the model itself, or the service layer managing requests and responses. Huyen, drawing on prior experience at Nvidia, sidestepped hardware discussion as impractical for most practitioners. She also dismissed replica parallelism—simply adding more machines—as expensive and operationally complex, particularly when managing heterogeneous hardware configurations.
This left model and service optimizations as the focus. Huyen offered guidance on choosing between them: "If you want to host the models yourself, or if you have access to the model weights, or if you train a model yourself, or you want to fine-tune or distill a model, then model optimizations might be for you. However, if you want to take a model as-is and make it more efficient on your own inference service, you might want to look into service optimizations."
Model optimization
These techniques modify model weights and thus affect outputs. Quantization reduces the bit precision of weights and activations—for instance, from 32-bit (four bytes per parameter) to 8-bit (one byte). Huyen noted the efficiency gain: "Reducing the precision not only reduces the memory requirement to run the model, making it cheaper. It can also make the model a lot faster. If you do additions bit by bit and each weight is 32 bits, you have to do it 32 times. If it's 8 bits, you only have to do it eight times." The tradeoff involves modest quality loss, yet Huyen observed: "I rarely see any companies running a model at full precision anymore."
Distillation uses a large model to generate training examples for a smaller one. If you possess a very large model (Huyen cited o1 as an example) and want a smaller version with similar performance, you collect a prompt set, run it through the larger model, and train the smaller model on those outputs. Caution is warranted, however. Huyen warned: "A lot of model providers have the condition that they do not allow their models to be used to train competitive models. So even though it's a very common technique, you need to check licensing."
Service optimization
These techniques govern how requests are scheduled, routed, and reused without altering model weights. Batching combines multiple requests into a single forward pass, far more efficient than sequential processing. Huyen outlined three batching strategies:
- Static batching waits for the batch to fill, maximizing compute use but potentially raising latency for early requests
- Dynamic batching operates on a timer (e.g., every 15 milliseconds), reducing compute efficiency but improving latency
- Continuous batching accommodates requests finishing at different times—common with LLMs—returning each request as it completes and filling the slot with another, improving both resource utilization and latency
Decoupling prefill and decode separates input processing from output generation onto different machines. Huyen explained the distinction: "Input tokens can be processed in parallel, whereas output tokens need to be generated sequentially. With parallel processing, it's bounded by compute, the processing power of the chip. With decoding, it's bounded by memory, because you have to move model weights." Since each phase stresses different resources, most services now separate them. To improve TTFT, allocate more capacity to prefill; to improve TPOT, shift resources to decode.
Parallelism distributes work across machines in different ways. Replica parallelism copies the entire model to additional machines. Tensor parallelism divides large matrices so different machines compute different sections. Pipeline parallelism splits the model by layer, allowing requests to flow through as a pipeline.
Prompt caching processes repeated text segments once, then reuses them, cutting both cost and latency. Many requests to the same application share common elements: system prompts, examples, code bases, or documents referenced in different queries. Processing that shared prefix once and caching it makes economic sense. When Huyen wrote AI Engineering, prompt caching was obscure—one research paper, little awareness. She included it anyway, and now it is ubiquitous. Savings scale with the cached portion size. Huyen's open-source tool Sniffly discovered cache hit rates of 90% to 97% in Claude Code logs. Some providers internally rewrite prompts to boost hit rates, but you can structure them yourself. Huyen's recommendation: "It's pretty easy to do, and it can improve your application performance significantly." Place stable prompt elements first and variable parts later, since caching works on shared prefixes.
Selecting an inference provider
Huyen cautioned those evaluating inference providers to look beyond advertised cost and latency figures. "There are many inference companies that provide inference optimizations for models you want to use, and a lot of them advertise just cost and latency. But pay attention to how many inference optimization techniques also change the model behavior or reduce the model quality. So when evaluating an inference service, it's important to look not just at cost and latency, but also at model quality. Does this model, provided on this service, also perform similarly on standard benchmarks?"
How the advice has aged
A year after the keynote, most of Huyen's core insights remain sound. Her economic framing—that inference dominates training costs—holds. For many practitioners, however, not all optimization levers are accessible. Understanding what is happening remains valuable. Local LLM users can relate to her points on parallelism (often unavailable locally) and prompt caching and quantization (within reach).
Prompt caching, which Huyen described as novel when writing her book, is now bundled into API token pricing. Her observation about Claude Code achieving 90% cache hit rates likely explains why agentic coding assistants remain affordable for ordinary users. Some developments proved harder to predict. Huyen flagged reasoning models as making inference even more significant; one year later, agents executing multi-step loops with tool calls have transformed that observation from a minor point into a major factor reshaping Claude's rate limits and availability constraints.
The metrics Huyen outlined—time to first token, time to publish, goodput under latency SLOs—are now standard terminology, though they may require different reasoning in the agent era. Huyen's return to P99 CONF 2026 on October 21 and 22 offers an opportunity to hear how these ideas have evolved.
Source: The New Stack