At 3 a.m., an AI agent begins calling the same tool repeatedly. The operations team scrambles. This scenario captures a fundamental problem plaguing enterprise AI: a joint IDC and Lenovo study shows that 88% of AI proof-of-concept projects fail to transition into production environments. The breakdown stems from three core issues—vague project goals, inadequate data preparation, and insufficient in-house expertise with agentic AI systems. The result is a dangerous split: the team that built and deployed the agents often hands off to an operations group with minimal training, creating a gap where failures cascade.

Serhii Semenchenko, CTO of Dedicatted, a Toronto-based AWS Premier Tier Partner for Agentic AI Specialization, frames the core tension between controlled testing and unpredictable production. "The demo is a clean environment where we have clean, prepared data. No edge cases, so everything is defined — input and output." Production systems face the opposite reality: messy data, unexpected edge cases, and users who behave in ways the designers never anticipated. Moving from proof-of-concept to production requires fundamentally different infrastructure.

Semenchenko outlines what production readiness demands: "Every single step should include the logging, tracing, monitoring, breakers, all the checks, and validation." His basic checklist includes timeouts on every agent, comprehensive logging of all outputs for traceability, and self-assessment mechanisms that let agents decide whether to continue, retry, or escalate after unexpected events. These safeguards exist to catch failures before they cascade into business outages.

Treating agents as services

Semenchenko draws a parallel to microservices architecture: each agent should operate with its own narrowly scoped permissions rather than broad, long-lived credentials. Agents receive short-lived keys that grant access only to the specific tools they need—whether those are APIs, databases, or other services. This approach prevents unauthorized tool calls, data leaks, and prompt injection attacks. The principle is straightforward: "Every single agent should be treated as a separate service."

Multi-agent orchestration requires a router that gathers inputs from the user, system, and platform, then decides how to route the work. Specialist agents handle documentation, analysis, video, audio, and livestream processing, each with its own toolset. A final composer agent assembles outputs from all upstream agents and tools. Critically, if the composer fails, it does not invent missing data—it either retries or marks the task as failed, ensuring clear state boundaries and accountability.

Memory poisoning—corruption of an agent's stored context—poses a distinct threat. Semenchenko describes "guardrails" as the defense: deterministic validation of all documents and data to catch inconsistencies, duplicates, and invalid records before they propagate through the system.

From POC to production: two case studies

Dedicatted built a multi-agent system for CASSEN Testing Laboratories, a Canadian chemical and environmental testing firm facing multi-day report turnaround times due to manual regulatory testing. The project had clear input and output specifications, but production revealed human-centered edge cases: chemists had personal preferences about how chemical compounds and error investigations should be presented in reports. The solution required adjusting the composition agent so final reports matched the style chemists would produce manually. As Semenchenko explains, "We adjusted our flow, adjusted the agent that composed this report. So, everything in the final, after these adjustments, reports look the same as if chemists had manually prepared them."

A second case involved Zivian Health, a U.S. compliance platform for organizations managing nurse practitioner and physician assistant workforces across multiple states. The company migrated from AWS to Azure under regulatory pressure—a re-platforming with high stakes. AWS and Azure use completely different access control systems, making least-privilege permissions critical. Semenchenko notes that "everything should be mapped, adjusted, and validated first." The team deployed cloud-agnostic monitoring tools to trace all agentic AI activity across both platforms, then ran a comprehensive validation suite covering all use cases and user flows. Only after confirming matching success rates, latency, and token costs across the migration did the system move to production.

The 3 a.m. misbehavior problem

Semenchenko describes a common failure mode: an agent decides that a tool's output is insufficient and calls it repeatedly—2, 3, 5 times—hoping new data will appear. Advanced monitoring systems detect this anomalous behavior and flag it. The response depends on the operational model. If an agentic AI partner operates the system, they monitor dashboards and alerts continuously, reacting immediately to anomalies without waiting for morning. If the client's internal team operates the system, they receive error messages and anomaly alerts, then use runbooks provided by the partner developers to respond.

The advantage of outsourced operation is clear: continuous monitoring, immediate response, and investigation of edge cases to prevent recurrence. Partners can stop agents or revoke access instantly, removing the burden from in-house teams and reducing the window of exposure.

The commercial cost of failure

Denys Yermakov, CEO of Dedicatted, emphasizes that agent failures often go unnoticed on the business side until they cause real damage. Production agentic AI systems require extensive monitoring, regression tests, varied prompts, and behavior testing. How companies ship fixes matters as much as how they detect problems. "The price of failure is going to be much higher compared to other platforms," Yermakov warns, pointing to outages from infrastructure issues or bad deployments.

For change management, solutions must be flexible and adaptable to future requests. Yermakov advises enterprises to question potential partners about project scope, code ownership, and platform ownership—details that matter because the in-house team will eventually handle incident response and maintenance.

Evaluating a partner's true expertise requires scrutiny. Yermakov suggests starting with a small piece of the POC or demo and observing how the partner delivers it. The critical distinction lies in whether the partner works with real data, real workflows, and real business challenges, or relies on mock data and predefined scenarios. A partner who surfaces challenges during demos—asking the right questions and taking the right approach—signals genuine expertise. One who glosses over complexity signals a reseller rather than a true service provider.

Data security and the perimeter problem

A Modern Data Company survey found that 75.9% of enterprise data leaders cite data quality and trust as barriers to AI systems reaching production. When sensitive data moves between partners, Yermakov stresses the business imperative: "It's so important that your data stay within your perimeter and not leave your secure perimeter, which you already have in your platform." Traditional enterprises with large on-premises data repositories face a particular challenge: many AI providers train public models on customer data, creating unacceptable risk.

Semenchenko describes the technical solution deployed in healthcare projects where personally identifiable information around health is critical. The constraint was absolute: PII could not be stored anywhere, yet agents needed access to it for processing. The answer was an additional layer that strips all PII from agent outputs before they reach observability platforms. As Semenchenko explains, "The traceable data goes to the observability platform. So no PII data in logs and monitoring."

Build in-house or hire partners

Yermakov emphasizes how new agentic AI remains to enterprise operations. Many organizations are still grappling with foundational questions: how to set up platforms, what architecture makes sense, what benefits accrue, and what the cost per user looks like. Partners bring hard-won experience—they have shipped systems, overcome obstacles, and built platforms efficiently with predictable outcomes. "It's quite important to build and architect all the platforms from the very beginning in the very right way, to not redo what you've implemented into the business processes."

The operational reality emerges during platform maintenance, which must remain adaptable and adjustable. Yermakov stresses the importance of avoiding early failures and building in-house knowledge and capability for long-term maturity. The decision to build or outsource should consider two perspectives: how the business operates and what technical challenges the organization faces.

The key to success, Yermakov argues, is staffing: "Always the team effort of the in-house resources and the outsourcing resources." Outsourcing everything fails because partners cannot understand business processes in sufficient detail. Subject matter experts and partners must engage from the start to ensure knowledge transfer. The in-house team should own and protect the business process, while the partner architects and operates the core runtime.