Scaling AI Agent APIs: Microservices Principles Meet Stateful Workloads

Most interactions with frontier AI models flow through structured APIs provided by model vendors like OpenAI and Anthropic. A practical approach involves deploying an OpenAI-compatible API that can manage multi-turn agent conversations, allowing developers to point the official OpenAI SDK at a compatible inference endpoint and maintain the same interface across conversation turns.

These systems require deliberate architectural choices to scale gracefully. When traffic increases or subagents spawn, engineers must scale the infrastructure—for instance, expanding from a single replica to three behind a load balancer—without users experiencing lost context or broken conversation state. The challenge lies in retaining proper conversation context and previous turns through the scaling process.

The microservices principles everyone stopped talking about five years ago turn out to be exactly what agent traffic needs.

Why agent traffic is different from web traffic

The load characteristics of AI agents diverge fundamentally from browser-based traffic. The difference lies in how stateful systems absorb load spikes and request patterns.

Four properties distinguish agent clients from traditional web clients:

1. Conversations span many turns and maintain state, yet individual requests remain stateless at the HTTP layer. A 40-turn agent loop generates 40 independent HTTP requests with no protocol-level binding to a specific server. The application must track which conversation each request belongs to and where its state resides.
2. Tool calls branch unpredictably. A single chat request may trigger zero downstream calls or six, with one potentially hitting a vector index and consuming 400 milliseconds. Because model outputs and tool-selection paths vary, identical inputs can produce different sequences and counts of tool calls.
3. Agents implement aggressive retry logic. AI agents typically run within a harness containing its own control loop, error handling, and retry policy. When timeouts or errors occur, the system may receive rapid retry sequences from the same workload, amplifying pressure on already-stressed dependencies.
4. Load arrives in bursts with machine pacing. Little or no think time separates turns. While an AI agent operates, it generates continuous work bursts constrained mainly by hardware and concurrency limits.

The first property is most intuitive: browser sessions survive on a single server through sticky routing and human think time. An agent can fire many requests in quick succession, and without durable shared state, requests distributed across replicas lose the context they require, creating friction for the user.

The first version everyone builds

A minimal, working OpenAI-compatible endpoint might look like this:

SESSIONS: dict[str, list] = {}
SLOTS = asyncio.Semaphore(int(os.environ.get("CONCURRENCY", "32")))  # one pool

@app.post("/v1/chat/completions")
async def chat(req: Request):
    body = await req.json()
    sid = req.headers.get("x-session-id") or body.get("user")
    async with SLOTS:                                 # everything shares it
        history = SESSIONS.setdefault(sid, [])
        history.append({"role": "user", "content": body["messages"][-1]["content"]})
        if body.get("tools"):
            await TOOLS[body["tools"][0]["function"]["name"]]()   # inline, in-path
        ...

This endpoint is correct and minimal, passing all tests. Yet it harbors a fundamental flaw. In a proof-of-concept benchmark driving 200 conversations of four turns each, with each turn routed to the next available replica in rotation—the distribution pattern that emerges when stickiness configuration is absent—the monolithic deployment on a single machine versus three replicas revealed the problem.

The design is not broken; it is conditionally correct. The condition is that every turn reaches the same machine—precisely the condition abandoned when scaling horizontally.

The failure is silent. The model still answers, but it may answer without the right context because the request landed on another machine that does not have the previous chat history.

This silent failure represents the core problem. The model produces a response, but without the right context because the request arrived at a different replica lacking the previous chat history. Observability may still report HTTP 200 responses, yet the system misbehaves from the user's perspective. This distinction matters when designing APIs for AI applications.

Returning to the principles

The solution draws from established patterns. Lewis and Fowler's microservices characteristics (2014) and Michael Nygard's bulkhead pattern from Release It! apply directly to request shapes that did not exist when those works were published.

Three principles carry the weight:

Statelessness: Conversation state migrates from process memory into a shared store. Any replica can serve any turn of any conversation. This single change enables the previous endpoint to scale across replicas without relying on process-local conversation state. The agent can retrieve what it needs from a centralized location rather than requiring the full chat history to traverse every request.

Bulkheads (isolation): Different system parts isolate to prevent cascading failures. In this case, isolation occurs by dependency rather than globally.

Smart endpoints, dumb pipes: The OpenAI chat-completions protocol serves as an unusually effective transport layer: relatively compact, stable, already supported by many agent frameworks, and familiar to AI developers. Intelligence—routing, budgeting, memory engineering, context engineering, tool-calling policies—sits atop it rather than inside it. The pipe remains simple.

The POC: four services, one protocol

The decomposed stack splits along failure-domain lines rather than nouns:

• gateway — speaks the OpenAI protocol, holds nothing, scales horizontally
• memory — owns conversation state and the tool audit trail; no other component may hold a message list
• tools — executes tools in its own process with its own concurrency budget and timeout
• Oracle AI Database Free — the shared substrate beneath all components

The architecture isolates as much as possible while keeping each component focused on a single job. This makes components easier to evolve independently and provides room to set separate concurrency budgets, timeouts, and other configuration parameters for each dependency.

Semaphores implement this control, allowing management of access to shared resources:

CHAT_SLOTS = asyncio.Semaphore(int(os.environ.get("CHAT_CONCURRENCY", "24")))
TOOL_SLOTS = asyncio.Semaphore(int(os.environ.get("GW_TOOL_CONCURRENCY", "8")))

Because the protocol remains unchanged, the official SDK communicates with this OpenAI-compatible endpoint without SDK modifications:

from openai import OpenAI
c = OpenAI(base_url="http://127.0.0.1:8110/v1", api_key="")

c.models.list()          # ['agent-gateway/router-v1']
r1 = c.chat.completions.create(model="agent-gateway/router-v1",
        messages=[{"role": "user", "content": "hello"}], user="demo-1")
r2 = c.chat.completions.create(model="agent-gateway/router-v1",
        messages=[{"role": "user", "content": "and again"}], user="demo-1")
# r1 -> "turn 1"   r2 -> "turn 3"   (state survived, on a different replica)

As the gateway writes to the memory component independently with semaphore-controlled concurrency, the design avoids relying on local file system state across replicas.

Are bulkheads worth it?

Bulkheads isolate different application components to prevent failures from spreading. For example, the tool-calling path can separate from the standard chat-completion path.

Consider a system that correctly partitions components and separates tool-calling requests from standard chat completions. Both stacks receive 32 admission slots per replica. The monolithic approach draws from one request pool. The gateway splits them: 24 for plain chat, eight for tool-bearing requests. A benchmark driving 120 chat requests per second while 300 concurrent callers hammer a 400-millisecond tool demonstrates the difference.

Without bulkheads, a user making a standard chat request can end up waiting behind slow tool calls because every request is competing for the same pool.

With isolation, tool pressure remains contained within its own pool, helping preserve capacity for standard chat traffic. Without bulkheads, standard chat requests queue behind slow tool calls, both competing for identical resources.

Where Oracle AI Database Free fits in

The microservices principle of "database per service" may not apply uniformly in AI applications. The 2014 consensus held that giving every service its own datastore reduced coupling costs compared to eventual consistency coordination across shared schemas.

For agents, this calculus inverts. A single agent turn writes four things that must agree:

• the conversation state
• the tool invocation record and order
• the semantic, episodic and workflow memory facts extracted from the turn
• the idempotency ledger entry that prevents a retry from re-executing a paid tool

Splitting these across data stores—one in Postgres, one in a vector store, another in Redis—creates a consistency problem the application must manage. Databases have solved transactional consistency for decades, so this architecture can leverage database transaction capabilities to manage related writes together.

Keeping them in one database makes it a single transaction commit:

CREATE TABLE agent_sessions (
  session_id   VARCHAR2(64) PRIMARY KEY,
  messages     JSON NOT NULL,                 -- native JSON, not a CLOB you parse
  version      NUMBER DEFAULT 1 NOT NULL,     -- optimistic concurrency
  updated_at   TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL
);

CREATE TABLE agent_memory (
  memory_id    VARCHAR2(64) PRIMARY KEY,
  session_id   VARCHAR2(64),
  content      CLOB NOT NULL,
  embedding    VECTOR(1024, FLOAT32)          -- same row, same transaction
);

CREATE VECTOR INDEX ix_memory_vec ON agent_memory (embedding)
  ORGANIZATION NEIGHBOR PARTITIONS DISTANCE COSINE WITH TARGET ACCURACY 95;

Relational, JSON and vector data coexist in one engine, one transaction, one backup, one set of credentials. Oracle AI Database Hybrid Vector Search enables a retrieval query to fetch what it needs, join the tool history and rank by cosine distance in a single statement—rather than joining in Python across three systems, which otherwise requires application-level coordination.

The database also helps coordinate concurrent turns across horizontally scaled APIs. Two replicas serving turns of the same conversation is normal, not exceptional. SELECT … FOR UPDATE plus a version column can serialize conflicting updates and implement retry handling, preventing competing updates from overwriting conversation state.

The counterintuitive part: decompose compute, converge data

Many teams reach for microservices to scale AI applications, but applying the principle uniformly creates consistency challenges across the agent's memory, audit log, and vector index.

The useful split is asymmetric: decompose the compute, converge the data.

The 2014 principles endure. They simply require adaptation to the new era of AI agents. Download Oracle AI Database Free, install the Oracle AI Agent Memory Python package, and follow the local quickstart to add persistent state and durable memory to an agent API. Additional examples are available at the Oracle AI Developer Hub.