Large language models can field the same question repeatedly, each time incurring a charge. The path to lower costs begins with a straightforward question: has anything changed that would alter the correct answer? By tracking the request itself, its surrounding context, model configuration, and source data, teams can build a fingerprint of inputs and dependencies. When that fingerprint matches a previous query, returning the cached response skips the model call entirely and preserves budget.

This pattern is not new. Production data pipelines have long struggled with nightly jobs that recalculate aggregations unchanged since the previous run. The waste becomes visible only during cost reviews, when someone realizes that significant compute is re-answering questions whose inputs never changed. The solution—hashing upstream inputs, fingerprinting dependencies, and skipping recomputation when fingerprints match—cuts waste substantially. The same principle applies to LLM workloads, where billing by token makes duplicate requests costly.

Duplicate requests are nearly inevitable. Upstream users converge on similar questions. Batch jobs repeat boilerplate with each run. Prompt-engineering experiments in development and CI invoke the same prompt repeatedly. Tool-calling agents may hit the same knowledge-base tool many times in a single day. Meanwhile, many APIs treat duplicate requests as new ones anyway, compounding the cost.

Response caching differs from native prompt caching offered by providers. In prompt caching, providers reuse cached prompt computation and charge reduced rates for eligible cache reads; output generation remains billable. Response caching attempts to skip the call entirely when an answer already exists in the team's own infrastructure.

Tier 1: exact match

The simplest approach normalizes the model request body, runs it through a cryptographic hash like SHA-256, and looks up the hash in an in-memory store such as Redis. A match returns the answer without waiting for model inference. Exact-match caching works best when model requests are bounded and predictable—a common scenario for batch pipelines, CI runs, and boilerplate summarization tasks.

Tier 2: semantic match

Many workloads require more flexibility than exact matching provides. The approach embeds the user's query through an embedding model and stores the resulting vector in a vector database. When a new query arrives, it undergoes the same embedding, and the system searches for close matches by cosine similarity.

How close is close enough? A common starting point is a cosine-similarity threshold between 0.90 and 0.95, but this should be treated as a tuning parameter, not a default. The right value depends on the embedding model and the data; teams should validate it against real queries. Vector stores vary in their output: some report cosine similarity rising toward 1 for closer matches, while others report distance falling toward 0. Confirm which metric the threshold applies to. A looser threshold increases the risk of wrong matches, where the system answers one query while the user asked about another—for example, conflating weather in one town with weather in another.

Tier 3: hybrid

A common approach runs both tiers in sequence: check the exact-match store first, and run semantic search only on a miss. When semantic search returns a close-enough match, the result is promoted back into the exact-match store under the hash of the new query, so the paraphrase and its answer become an exact hit next time. This strategy favors cheap exact matches on repeat traffic.

Both tiers key on more than the query text alone: the context and documents in the prompt, the model and its settings, the version of any retrieved source, and the caller's access scope. Two identical questions asked against different documents, or by users with different permissions, must not share a cache entry.

def cached_completion(query, ctx):

    # ctx bundles everything that changes what the correct answer is:

    # the context/documents in the prompt, the model and its settings,

    # the source-version of any retrieved content, and the caller's access scope.

    key = sha256(normalize(query, ctx))

    # Tier 1: exact-key lookup on Redis (O(1)).

    # Correctness still depends on cache contents, request scope, and freshness.

    if (hit := redis.get(key)):

        return hit

    # Tier 2: semantic search, restricted to the same scope as the request.

    emb = embed(query)

    match = vector_db.search(emb, top_k=1, filter=scope_of(ctx))

    if match and same_scope(match, ctx) \

            and match.score >= threshold_for(category(query)):

        # Promote, but preserve the original freshness deadline.

        remaining = match.expires_at - now()

        if remaining > 0:

            redis.set(key, match.response, ttl=remaining)

            return match.response

    # Miss on both tiers: call the model, validate before writing back.

    resp = llm(query, ctx)

    if is_valid(resp):  # no errors, no empty payloads, no malformed JSON

        ttl = ttl_for(category(query))

        redis.set(key, resp, ttl=ttl)

        vector_db.insert(emb, resp, ttl=ttl, scope=scope_of(ctx))

    return resp

One threshold does not fit all. Code-like queries often need stricter thresholds, around 0.95 or higher, because small wording changes can produce entirely different results. Conversational queries can tolerate looser thresholds, in the 0.85 to 0.90 range. These numbers are starting points, not settled values—teams should validate them for their own workload and embedding model before relying on them. Cache freshness follows the same principle: the right TTL depends on how much staleness the use case can tolerate, not on the data type alone.

A cached market-data answer might be acceptable for only a minute or two, because a stale price can be actively misleading. An internal HR policy answer can often be reused for weeks, because the underlying document rarely changes and a slightly old answer is usually still correct. The interval is a judgment about acceptable staleness, not a fixed property of the content.

The math

Consider a workload of 1,000,000 calls per month at $0.006 per call, roughly $6,000 with no caching. A hybrid cache delivering about a 60% hit rate avoids 600,000 calls to the model. If embedding and vector-store costs total about $150, monthly spend drops closer to $2,550, a 57.5% reduction, plus the latency win of answering many questions without waiting on the model. One essential caveat: measure the hit rate before projecting any savings.

The decisions

Beyond the tiered framework, teams must tune TTLs to the freshness each data type actually needs and invalidate entries when content updates. A fine-grained approach assigns a per-category TTL based on how quickly each answer goes stale: a news summary might hold up for an hour, while a live sports score is worthless within seconds and shouldn't be cached at all during a game.

Live scores require a freshness policy matched to the application. Verified final scores can support much longer caching, with invalidation for corrections. The distinction is whether the underlying value is still moving. A simpler approach skips per-category tuning and purges the whole cache whenever source content changes. Either way, run the cache in shadow mode first, logging what would have been returned without changing behavior. Evaluate cached answers against verified reference answers or expert review. A fresh model response can help identify differences, but it is not ground truth.

Warm the cache from a historical set of common queries before relying on it, and validate answers before writing them back, so the cache doesn't become poisoned with errors, empty responses, or malformed content.

When should caching be skipped? Avoid it for requests with personal or account-specific data, to prevent leaking one user's cached output into another's request. Skip it for creative tasks, where a different answer each run is desired. And skip it for genuinely real-time data like stock prices and live inventory, where an answer even a minute old may be too stale for the application.

The takeaway

The principle predates the web: Donald Michie described memo functions in 1968. When possible, fingerprint the question and store the hashed exact form alongside the semantic-variant form, so repeated model calls are avoided while a valid cached answer remains available.