Training AI agents presents a paradox: organizations invest heavily in powerful GPUs only to watch them sit idle. Before models can learn, agents must execute tasks and generate results—a process called rollout generation. When this inference stage moves slowly, it becomes the constraint that holds back the entire training loop, yet many teams treat it as an afterthought.
Di Jin, VP of AI Training at Amsterdam-headquartered Nebius, observes that teams transitioning from pretraining or supervised fine-tuning often underestimate the infrastructure demands of this phase. "If you staff and design it as a training project, inference becomes the bottleneck nobody owns," Jin explains. Rollout generation typically consumes the majority of each training step's elapsed time, making batching, cache management, and scheduling critical to learning speed and overall cost.
Measuring what matters: time to trainable batch
Teams should track time to a trainable batch alongside conventional throughput metrics. Average tokens per second can obscure the long tail of slower operations: in a synchronous batch, training cannot proceed until the final trajectory completes. Agent trajectories vary in length—some finish quickly while others take longer, leaving training hardware idle waiting for stragglers. This reality means treating the inference engine as an integral component of the post-training reinforcement learning system, not a separate concern.
Policy staleness: measured risk, not absolute evil
Using data generated by older model versions introduces tradeoffs. Some policy lag can boost GPU efficiency by allowing generation and training to overlap, but safety depends on how much the policy has shifted and how the training algorithm handles that divergence. Jin tells us: "In fully synchronous RL, inference GPUs wait while training runs, and training GPUs wait while generation runs. Letting generation use weights that are one or a few updates old allows both to run in parallel. That overlap is how most large-scale systems reach reasonable utilization." When success on difficult tasks is uncommon, reusing an already successful trajectory can provide value even if slightly outdated.
The real danger emerges when policies drift too far from their generating version. Rather than age, the meaningful measure is divergence between the current policy and the one that produced the data. "Staleness isn't the enemy. Unmeasured staleness is," Jin states. Teams should measure and manage staleness through importance-sampling corrections (typically truncated to limit variance), dropping or down-weighting samples beyond staleness thresholds, and tracking divergence between generating and training policies.
Environments as the new frontier
In Jin's experience, data and environments consume most of the effort in successful RL post-training, while algorithm design represents the smallest component. Infrastructure enables rapid iteration, but what models actually learn depends on the tasks, environments, and reward structures provided. For agentic systems, this means focusing on the environments that generate training experience. Unlike supervised fine-tuning, which uses a pre-collected dataset, agentic RL generates trajectories through model interaction with environments.
Reproducible containers, codebases, websites, and APIs with realistic states help learning transfer to real-world use. Reliable verifiers and appropriately difficult tasks are equally essential. As models improve, task sets require refreshing to prevent saturation and maintain learning signals. "Building environments is the new data curation," Jin says.
Failures frequently occur at system boundaries—where the trainer, inference engine, sandbox, and verifier hand off to one another. Models can exploit verifier loopholes to receive rewards for unintended behaviors, and sandboxes can fail or misbehave, corrupting training data. Teams should record whether episodes ended in model success, model failure, infrastructure failure, or unknown outcome. Infrastructure-caused failures should be excluded from policy-gradient updates, while model-caused failures (including timeouts or errors triggered by the model) must still register as failures to prevent models from learning to avoid penalties through infrastructure disruption.
Scheduling and batch assembly
Long-horizon agentic RL combines trajectories, tool calls, and external latencies, creating bursts of GPU demand and idle periods. For GRPO-style training, where sibling attempts compete against each other, scheduling should prioritize group completion rather than launch order. A group becomes ready once all sibling attempts finish; the trainer then processes eligible complete groups regardless of launch timing. Time to trainable batch governs performance, while tokens per second remains useful for diagnosing inference bottlenecks. Routing matters because keeping conversations close to cached history avoids repeated prefill work.
In a Nebius experiment using a Terminal-Bench-based workload, modifications to routing, sandbox execution, and batch assembly reduced fixed-weight collection time for a 256-trajectory batch by 66.9%. This measurement covered batch collection only, excluding the trainer's forward pass, backward pass, and optimizer work—it did not measure complete training speed.
For agentic AI rollouts, a core infrastructure goal is reducing the time expensive GPUs spend waiting on sandboxes, browsers, code execution, and other non-GPU dependencies. "Your most expensive GPU is often waiting on your cheapest container," Jin recommends. Scaling CPU and sandbox capacity independently of GPUs and provisioning generously makes sense because idle containers cost relatively little. Keeping sandboxes ready through snapshot and fork methods removes environment startup from the critical path, though tool execution time remains. Placing sandboxes on the same network fabric as GPUs minimizes cumulative latency from the hundreds of tool calls agents may execute. Per-step timeouts and budgets limit disruption from hung browsers or processes. Teams should track GPU-hours spent waiting on environments alongside utilization and time to trainable batch to identify where additional investment would help.
Numerical alignment between systems
During RL post-training, the rollout engine and trainer can assign different probabilities to identical tokens even when starting from the same weights. Quantization, kernels, batching, and in mixture-of-experts models, expert routing can all contribute to these differences, introducing systematic bias into training updates. Jin recommends aligning numerics first, then measuring and correcting remaining mismatches. This might involve matching the quantization scheme and computation in the trainer's forward pass to the rollout engine while retaining higher-precision weights for optimization. Importance-sampling corrections can help manage residual differences, but clipping alone should not be assumed to resolve every form of mismatch.
Evaluation should use the precision, inference engine, and version that will serve the model in production. Any change to quantization or kernels requires re-evaluation.
Weight distribution and policy synchronization
As training updates the policy, rollout workers need dependable mechanisms to receive new weights while maintaining policy lag within acceptable bounds. Jin describes three broad patterns: resharding weights in memory when training and inference share GPUs; transferring weights directly to separate inference workers; and publishing checkpoints that inference engines hot-load through storage. The optimal choice depends on model size, hardware topology, and freshness requirements.
Full service restarts can introduce unnecessary downtime, but storage-based publication does not require one. Asynchronous transfer and compact updates can move much of the work off the critical path. Teams should measure end-to-end publication latency separately from the pause needed to apply weights. Weight publication also requires an explicit plan for active requests and cached state. Depending on the engine, teams can finish, abort, or pause requests before applying an update, and clear or recompute cached context accordingly. If a trajectory spans policy versions, record which version generated its tokens so the training system can handle that mixture intentionally. Coordinate transfers with rollout traffic where they share resources, and monitor publication delays alongside policy age.
Designing these components together helps teams improve iteration speed while keeping training data and policy updates reliable. Jin advises smaller teams to begin with a simple setup and add asynchronous complexity when measured bottlenecks justify it.
Key metrics to track
- Batch readiness: How long does it take to assemble a complete, usable batch, including the straggler tail?
- Policy freshness: Can every rollout be traced to the policy version that generated it, and what drift makes it too stale to use?
- Numerical alignment: How different are the trainer's and rollout engine's probabilities, and what correction handles the remaining gap?
- Environment reliability: Can you distinguish infrastructure failures from model failures and reproduce an affected episode?
- Weight publication: What happens to active requests and KV caches during an update, and how long does publication actually take?
- Evaluation: Are you testing the model in the precision and serving stack users will encounter?