Infrastructure health and GPU saturation offer little insight into whether an AI service actually serves users well. A model might emit its opening token quickly yet stall during generation. It could respond fast while delivering incorrect answers. Meanwhile, operations teams watching GPU metrics see only utilization rates, not the user experience underneath. The real challenge lies in connecting AI service performance with the Kubernetes, GPU, compute, storage, and network layers that support it—rather than monitoring each component separately.
The solution requires measuring inference behavior before production goes live, then comparing live performance against that baseline. In a conversation with Marius Bogoevici, Senior Principal Product Manager for HPE Morpheus Software; Mahesh Ramachandran, Vice President of Product Management for HPE OpsRamp Software; and Karthik Subramanian, Principal Product Manager for HPE Morpheus Software, The New Stack explores how to connect these layers and catch performance gaps early.
Start with the experience, then establish a baseline
For conversational AI services, Bogoevici recommends tracking time to first token, time per output token, reliability, capacity, resource efficiency, cost, and answer quality. The specific metrics that matter depend on what experience the team wants to deliver. A fast first token can hide slow token generation that follows; rapid completion does not guarantee accuracy or usefulness.
Time to first token and time to full response answer distinct questions. Inter-token latency and per-token generation speed illuminate what happens after the first token appears. Teams must document where each measurement occurs and validate results under sustained load and traffic spikes. Define performance targets for the specific workload and user experience intended.
Ramachandran emphasizes profiling the serving system before customers rely on it. Test with representative prompt and context lengths, output sizes, and traffic volumes. Capture token throughput, latency, and service-level indicators under those conditions. This baseline becomes the reference point for measuring how production behavior changes over time.
profile it before you launch
Mahesh Ramachandran, Vice President of Product Management for HPE OpsRamp Software
Correlating workload measurements with infrastructure performance and capacity helps teams separate rising inference demand from resource contention and plan expansion more accurately. Answer quality demands separate evaluation. Bogoevici notes that teams must determine whether responses are correct or hallucinated. Platform telemetry confirms a request was processed, but data science and domain experts need representative test cases and acceptance standards for output validation.
The ebook Kubernetes in 2027: Building platforms for AI, autonomy, and scale recommends evaluating output quality alongside service performance metrics and reassessing both after changes to the model, prompt, retrieval system, or serving infrastructure.
When the GPU is busy but the service is slow
GPU utilization provides a signal, but it does not pinpoint where performance breaks down. Bogoevici describes a common misinterpretation: "Our GPUs are maxed at 100%. So we're doing well." This metric tells operations teams almost nothing about actual service performance. Ramachandran suggests investigating whether inference pods scale based on workload-appropriate metrics, whether requests reach the intended model, and whether dependent components can be scheduled when required. Different models impose different latency and capacity requirements, so a shift in request distribution can degrade user experience even when GPU activity remains normal.
Subramanian identifies dependencies surrounding the model itself: network setup and response times from databases or vector stores. He raises another consideration: "Is the GPU partition too small to serve that model?" These represent investigation paths, not universal diagnoses. Teams should correlate them with affected requests and timing windows before implementing fixes.
Kubernetes' Horizontal Pod Autoscaler accepts custom or external metrics, so teams need not abandon it simply because CPU utilization fails as an inference trigger. They must supply and validate the metrics pipeline, scaling limits, and available accelerator capacity for their serving stack. A scaling policy cannot instantly create additional GPU resources.
What HPE Morpheus and OpsRamp contribute
HPE describes Morpheus Software as a platform for controlled provisioning and automation across hybrid infrastructure. Its Advanced and Enterprise editions include HPE Kubernetes Service (HKS); the Enterprise version extends operations across hybrid and public-cloud settings. Platform teams leverage catalog items, blueprints, access controls, and approval workflows to define how they request and manage supported environments. This approach delivers consistent, governed self-service and manages VMs and Kubernetes clusters through a single interface, reducing tool sprawl and manual handoffs. These capabilities alone do not benchmark models or assess answer quality.
OpsRamp Software delivers full-stack observability and monitoring across multi-vendor hybrid infrastructure and applications, featuring service maps, AI-driven event correlation, and operational automation to help teams understand dependencies and move from detection to resolution. Ramachandran characterizes Morpheus as the provisioning engine and OpsRamp as the observability engine within HPE's broader cloud operations strategy.
Ramachandran describes HPE's combined offering as Morpheus, OpsRamp, and runtimes working together. The solution can measure token service rates in model-serving environments and token request rates in agentic workloads. HPE is expanding OpsRamp's AI workload observability to link model and agent activity, token volumes, token serving rates, inference latency, and time to first token with underlying infrastructure telemetry.
The operational benefit emerges from correlation: teams can investigate whether slow responses originate in the application layer, model-serving layer, or infrastructure, while using workload demand to guide capacity planning and cost attribution. For each deployment, teams must verify which component produces each metric, how it is gathered, and what additional application instrumentation is required.
For agentic applications, HPE's planned observability upgrades will extend visibility to AI traces, tool calls, errors, and quality signals alongside Kubernetes, CPU, GPU, memory, storage, and network telemetry. Lightweight eBPF and OpenTelemetry-based instrumentation are also planned to expand visibility while keeping collection overhead minimal. These signals support investigation but do not replace domain-specific answer evaluation or security controls.
Collecting telemetry across a stack depends on deployed agents, integrations, permissions, and application instrumentation. For an inference service, connect user-facing measurements with model-server metrics and traces, Kubernetes events, and relevant network, database, and accelerator signals. Identify where visibility gaps exist, particularly at external model or service boundaries.
Plan for AI-specific routing and scheduling
Looking ahead to 2027, Ramachandran urges teams to treat autoscaling, model routing, and scheduling as deliberate design decisions for AI workloads. He discusses content-aware request routing, co-scheduling of related components, and disaggregated serving. These represent areas to evaluate for specific architectures, not features that every Kubernetes deployment or HPE product provides by default.
An experimental Gateway API proposal describes pre-routing filters and references the Inference Extension's Body-Based Router as an example. That router can extract a model name from a request body for routing purposes, but teams need a compatible implementation and configuration. They should measure any latency or efficiency gains from routing and scheduling decisions using their own models, traffic patterns, and hardware.
Inference observability introduces a data-handling challenge. Bogoevici cautions that prompts, responses, and retrieved documents may end up in logs and traces. He asks, "What information is actually captured and where does it go?" Teams must decide what to collect, where to store it, retention duration, and access permissions. He emphasizes a fundamental principle: prompts are data. Sensitive information requires protection even during troubleshooting collection.
Ramachandran observes that operations and security observability are converging as AI agents introduce new events and access records. He notes that OpsRamp is preparing to interpret existing telemetry from a security lens, while agent-specific activity may demand additional instrumentation. Teams must still design and validate their security controls and incident response procedures.
Before launch, Ramachandran recommends that business and IT stakeholders align on the value the service should deliver, including a realistic inference volume tied to a business outcome. "There has to be measurable business value," he says.
Subramanian emphasizes a critical point: "It can't be a single person." Platform administrators, machine learning and data science teams, and security teams must collaborate and prepare for failures or attacks. Include the relevant business stakeholder, test representative load and failure scenarios, and document the capacity, quality, cost, and recovery evidence supporting the decision.
A launch decision should rest on evidence that the service meets agreed user and business targets and that responsible teams understand what to measure and how to respond when performance falls short. After launch, continue comparing AI workload behavior with infrastructure conditions to detect regressions, troubleshoot across layers, and adjust capacity as demand evolves. "The goal is not simply busy GPUs, but reliable AI services with performance and resource use aligned to business needs."