Optimize before you buy: A framework for right-sizing GPU capacity in Kubernetes

When GPU pools reach full allocation and batch-job queues grow, the impulse to provision additional capacity can feel inevitable. Yet a lengthening queue may signal scheduling inefficiencies, insufficient serving capacity, or mismatches between model choice and hardware rather than a genuine shortage of resources. Teams should establish clear success criteria, deadline requirements, and cost targets before committing to expansion.

The foundation is recognizing a critical distinction: an allocated GPU, a busy GPU, and a workload producing valuable results are not the same thing. Understanding how these states relate enables teams to extract more value from existing infrastructure before purchasing additional capacity they may not truly need.

Start with the outcome and deadline

Workload urgency depends on business requirements, not generic classifications. A customer service chatbot requires continuous inference availability, while a model evaluation job might only need results before the next internal release meeting. Batch inference can tolerate delays, but a training job may have a firm completion date.

For each workload, document the deadline, expected runtime, resource needs, and whether it can tolerate interruption. Define what constitutes valuable output: an inference service should have task-specific quality thresholds and latency targets. A completed request or valid JSON response alone does not confirm the workload met its objective.

Distinguish between flexibility in start time and tolerance for interruption. A job that can wait to begin may still require uninterrupted execution once started.

Use scheduling flexibility before expanding the pool

Consider a scenario with four compatible GPUs available for batch work from noon to 5 p.m. and four independent evaluation jobs ready at noon, each requiring two GPUs for one hour with all results due by 5 p.m.

Running all four jobs immediately would require eight GPUs. Alternatively, starting two at noon and two at 1 p.m. completes all four by 2 p.m. using only the four GPUs already available. Both approaches consume eight GPU-hours (excluding startup overhead), but staggering reduces peak demand.

Staggering reduces peak demand. The jobs still finish on time without expanding the pool.

Tools like Kueue provide admission control using configured quotas and policies to determine when work enters the system. Teams must decide how to represent and enforce deadlines and workload requirements. Flexible queueing can support capacity planning and reduce the need to start jobs early only to preempt them later.

Measure how much useful work the deployment can sustain

Flexible scheduling eases capacity constraints, but inference teams must also evaluate how well the model and serving configuration match the workload.

High GPU activity does not reveal how much additional throughput remains available. Nvidia's profiling documentation notes that streaming multiprocessor activity can include warps waiting on memory. Serving metrics such as running requests, waiting requests, and cache occupancy provide context, while preemption counts explain behavior. In vLLM, KV-cache usage measures the fraction of occupied cache blocks but does not directly measure total serving capacity.

Establish sustainable capacity through representative load testing. Hold the model, hardware, serving settings, and request mix constant. Increase the arrival rate and observe throughput, quality, and latency. The highest sustained rate meeting the workload's requirements—including error limits and selected latency percentiles—yields a useful capacity estimate. For interactive generation, measure time to first token and subsequent token delivery, not just average completion time. DistServe demonstrates why serving capacity should be evaluated against application-level objectives like latency.

Compare production traffic against that tested capacity. Low cache occupancy may warrant investigation, but claiming ten times more throughput requires a throughput test. Brief queues represent temporary bursts, while persistent queues might indicate a diagnosis opportunity. Neither alone identifies the limiting resource.

Connect OpenCost data to verified outcomes

The FinOps community frames AI efficiency around cost and useful outcomes. For AI workloads, cost becomes more meaningful when connected to the outcomes the infrastructure delivers. Kubernetes teams can make this relationship concrete by combining workload cost, serving behavior, and application results.

Use OpenCost's allocation API to retrieve workload costs at defined intervals, then join that data with serving telemetry and outcome records using workload identities and timestamps. OpenCost also provides opt-in inference cost tracking, including cost per million tokens. When calculating cost per outcome, include applicable GPU, CPU, memory, and other infrastructure costs, and document how shared and idle costs are assigned. Add relevant costs outside OpenCost separately without double-counting.

A useful metric is cost per verified outcome: total attributable cost divided by the number of results meeting the defined quality requirements and delivered within the required time. Account for retries, failures, startup, and idle periods where they contribute to total cost. Count the final business result once. When correctness is estimated from sampled reviews, document the sampling method and uncertainty.

OpenCost supplies the cost allocation details; the application supplies the outcome definition and evaluation evidence. Capacity still requires separate measurement. OpenCost proposal 3915 explores bringing serving-engine telemetry into the cost model, though it remains a draft rather than a released saturation or correctness score.

Test whether the spare capacity earns its cost

Consider a service receiving 1,000 tasks per hour with a 98% correctness requirement. A small model achieving 90% correctness is unsuitable, regardless of low GPU cost and full utilization.

A stronger model might produce 990 correct and timely results for $2 per hour while using only a fraction of its tested serving capacity. Its cost is approximately $2.02 per 1,000 qualifying outcomes, and that spare capacity may be an acceptable price for meeting quality requirements and absorbing bursty activity.

The next test is whether a smaller GPU, fewer replicas, or different batching settings can preserve those outcomes and required reliability. If a tested configuration produces the same 990 qualifying results for $1.50, the cost falls to approximately $1.52 per 1,000 qualifying outcomes. Savings come from using fewer resources to deliver the same outcome. Increasing utilization on the original GPU without improving the outcome provides no comparable value.

When two models both meet minimum requirements, an additional quality improvement may justify higher cost if it produces fewer consequential errors or requires less human review. Compare that value with actual demand. Cost per correct outcome is most useful when the outcomes being compared have similar value and error consequences.

Change one major variable at a time, then compare cost, verified outcomes, latency, and deadline attainment under representative demand. Preserve capacity for bursts and failures, and confirm that low activity corresponds to usable capacity. A GPU allocated exclusively to one Pod cannot be reassigned to another until it is released.

The target is a deployment that delivers the required outcomes at a defensible cost, with enough headroom to meet its requirements.

Teams can then choose the change the evidence supports: delay flexible jobs, improve batching, resize a deployment, select a different model, or provision additional GPUs.