Kubernetes platforms typically ship with native visibility into CPU and memory consumption: per-pod metrics, straightforward rightsizing, and cost allocation by namespace. GPU workloads arrive with none of these conveniences, forcing engineering teams to engineer their own observability layer for the single most expensive line item in the infrastructure budget.
Most Kubernetes operators maintain Grafana dashboards that compare container requests against actual usage, making it simple to spot a deployment requesting four cores while consuming half of one. Namespace-level breakdowns show which team consumes what, a practice that has worked reliably for CPU and memory for years. The moment workloads move to GPUs, nearly every assumption breaks down. Visibility, accuracy, guardrails, and billing rules all become the platform team's responsibility to build and maintain.
Bring your own metrics
The kernel automatically tracks CPU and memory usage per container, making per-pod visibility nearly free. GPUs operate differently. NVIDIA's DCGM exporter, the standard starting point, reports metrics at the physical device level by default. This works fine when a single pod owns an entire GPU, but breaks immediately under GPU sharing via time-slicing or MPS. When three pods share one GPU, DCGM reports the same utilization value for all three, producing a nonsensical 300% utilization of a single GPU. The driver exposes one counter set for the whole GPU, leaving DCGM unable to attribute usage to individual workloads.
The resulting graphs look clean and professional, yet teams build dashboards and reports on demonstrably incorrect values that remain difficult to spot as wrong.
Bring your own rightsizing math
CPU resource management relies on millicores, overcommitment, burst capacity, throttling on limit breach, and OOM kills on memory limits. GPUs discard most of this model. A GPU appears as a whole-number resource: nvidia.com/gpu: 1. Requests and limits must match exactly, the scheduler enforces no overcommitment, and fractional sharing for inference workloads happens below the scheduler through time-slicing or MIG. Kubernetes resource requests do not map cleanly onto these mechanisms.
The fundamental rightsizing exercise—comparing requested capacity against actual consumption to reclaim waste—barely exists for GPUs initially. Teams must first determine what fraction of GPU capacity a workload actually received before they can even begin a waste conversation.
Bring your own guardrails
The NVIDIA container runtime reads an environment variable called NVIDIA_VISIBLE_DEVICES, and base CUDA container images set it to all by default. Normally the device plugin overrides this and injects only the GPUs Kubernetes allocated to the pod. If nothing overrides the variable or the container runtime honors the env var from the image, a pod can see and claim every GPU on the node regardless of what the scheduler allocated. The discrepancy between what the scheduler recorded and what actually ran on hardware can grow substantial. Determining the real allocation requires examining process-level activity and reconciling it against the scheduler's records.
Bring your own definition of utilization
A CPU utilization reading of 80% conveys clear meaning. GPU utilization is far murkier. The standard metric indicates whether a kernel ran on the GPU during some portion of the sample window, but reveals nothing about actual compute or memory bandwidth consumed. GPUs frequently sit at 100% utilization while compute barely progresses. Memory often becomes the bottleneck first, since frameworks like vLLM reserve most GPU memory at startup regardless of actual need. Teams must identify early which signal matters for their workloads—compute occupancy, memory footprint, bandwidth, or power consumption. Relying on the headline utilization number leads to wrong conclusions.
Bring your own attribution
NVIDIA's NVML library reports per-process GPU memory consumption and allows tracing each process back to its owning pod and team. The numbers prove reliable with one caveat: they do not sum to the card's total because the driver reserves memory for itself that belongs to no workload.
Compute attribution falls apart, and understanding why matters. Consider a shared kitchen: memory is counter space where you can see what ingredients sit where. Compute is the stove where everyone takes millisecond-long turns. NVIDIA provides not "how much stove time did each cook get" but rather a flag checked many times per second asking "is this cook doing something right now?" When two people work simultaneously, both flags read yes almost constantly, producing two jobs that each appear 90% busy—true but meaningless for determining who accomplished more.
This measurement works in one scenario: when the GPU strictly time-shares between jobs one at a time, where tools like DCGM's accounting mode or nvidia-smi PID-level sampling can approximate each job's share. It fails when jobs genuinely run side by side—precisely the setup teams choose for better utilization. If two teams share a GPU, no available tool today reveals which one consumed 70% and which consumed 30%.
The practical solution is billing on memory, where the data holds up, and treating compute as a signal for chargeback purposes. Fairness through accuracy between teams matters when moving beyond showback, and GPU memory remains the most accurate attribution method with existing tooling.
GPU sharing encompasses several different mechanisms, each changing the underlying math. Time-slicing round-robins the GPU with no isolation and permits oversubscription. MPS allows kernels from different processes to run simultaneously. MIG splits the GPU into fixed hardware partitions and enjoys popularity, though most cloud providers still charge for the entire physical GPU even when using only a slice, and A100/H100-class GPUs support only up to 7 instances. DRA, Dynamic Resource Allocation, has matured over recent Kubernetes releases and represents the clear direction forward, but the surrounding ecosystem, including autoscalers like Karpenter, continues catching up. Each mechanism redefines what "allocated" means, so visibility depends on which mode a node uses.
Bring your own splitting rules (and the bill is brutal)
GPU costs per hour far exceed CPU costs and resist per-workload splitting. Cloud providers bill a GPU node as a single line item combining CPU, memory, and GPU. Reaching a real dollar figure per workload requires breaking that line apart and weighting the GPU piece by model, since a T4 and an H200 carry vastly different prices. Then split that across whatever workloads shared the GPU. On AWS, the cheapest MIG-compatible A100 GPU costs $22/hr, roughly $14/hr on spot, while a comparable CPU-only instance of equivalent size runs about $5/hr. Even on spot, the difference stings.
So what does this actually cost you?
The true cost extends beyond the GPU bill itself to the engineering time and toil spent rebuilding visibility that CPU and memory provided for free. Metrics, accuracy expectations, guardrails, attribution, splitting rules—every layer now belongs to the platform team to own and maintain.
The encouraging news is that the space moves quickly. DRA matures and adoption grows alongside it, hardware partitioning becomes easier, and the tooling gap narrows with each release. Make deliberate choices about which layers to build now versus which to defer while the ecosystem catches up, and do not assume your CPU and memory playbook transfers directly to GPUs.