Stop Paying for the Same AI Work Twice

AI applications have a strange cost problem. We spend a lot of time asking: Which model is cheaper? Which provider has the lowest token price? Can we route this request to a smaller model?

Those are important questions.

But there is another question that is often overlooked:

Why are we asking the model to do the same work again?

An AI agent can send the same system instructions, tool definitions, documents, conversation history, and context through a model over and over again. The model may already have processed most of that information. Yet the application still sends it. The provider still processes it. And you may still pay for it. This is where KV caching and prompt caching become much more than performance optimizations. They become an important part of the economics of running AI agents at scale.

Before We Talk About Caching, What Actually Happens When an LLM Gets a Prompt?

To understand caching, we first need to understand what an LLM actually does with your input.

Suppose an agent receives:

"Analyze this customer account and tell me whether there are any risks."

The model doesn't directly understand that sentence as text. The input is first converted into tokens. Those tokens are processed through the model's transformer layers. For a decoder-only transformer, this initial processing is often called prefill. During prefill, the model processes the input sequence and produces internal representations used by the attention mechanism. This is where the Key-Value cache, commonly called the KV cache, comes in.

The KV Cache: Remembering What the Model Already Processed

Inside transformer attention, every token produces three important representations:

• Query (Q) — what the current token is looking for
• Key (K) — what information a token can be matched against
• Value (V) — the information retrieved when that token receives attention

During autoregressive generation, the model generates one token at a time. Without a KV cache, the model would repeatedly recompute the keys and values for all previous tokens every time it generated a new token. That would be extremely wasteful. Instead, the model stores the previously computed keys and values. When the next token is generated, the model reuses them. That's the KV cache.

The KV cache therefore turns previously computed context into reusable inference state.

As the Yandex research team recently argued, it is useful to think of the KV cache as more than just a decoding optimization: it can be viewed as part of the model's active execution state, potentially shared and scheduled across different streams of an agent runtime.

That distinction becomes especially interesting when we move from chatbots to agents.

Why the KV Cache Gets Expensive

The cache is not free. For every active sequence, the model needs to retain key and value representations for the tokens it needs to attend to. As context grows, the cache grows. More tokens. More layers. More KV heads. More memory. More GPU bandwidth.

For example: for Llama 3.1 70B at 128K tokens, the BF16 KV cache for one full-length sequence can be roughly 40 GB. Multiple concurrent sequences can multiply that requirement dramatically.

This creates two related problems:

1. Memory — the KV cache occupies GPU memory.

2. Memory bandwidth — during decoding, the model repeatedly needs to read the cached keys and values.

So even when the model weights are already loaded, long contexts can become a major serving bottleneck.

This is why modern inference systems use techniques such as GQA, MQA, sliding-window attention, quantization, paging, eviction, prefix reuse, and offloading to reduce memory consumption or unnecessary cache reads.

But there is another layer of caching that matters enormously for AI applications.

Prompt caching.

KV Cache vs. Prompt Cache: They Are Related, But Not the Same Thing

These two concepts are often mixed together. They shouldn't be. KV cache is primarily an inference-runtime mechanism. It stores intermediate attention state so the serving engine doesn't have to recompute previous tokens during generation. Prompt caching exposes the idea of reusing already-computed prompt state across separate requests.

Imagine an agent has this large context:

20,000-token system instructions + Tool definitions + Company policies + Retrieved documents + Conversation history + Current user question

The current user question changes every time. But much of the context doesn't.

Why should the model repeatedly process the entire stable prefix from scratch?

Only the changing portion needs fresh processing.

And This Is Where Agents Change Everything

Consider a customer-support agent. Its system context might contain agent instructions, company policies, product information, tool definitions, safety rules, formatting requirements, and customer-service examples. That context may be thousands or tens of thousands of tokens. Now the agent handles 10,000 conversations. If every request contains the same large prefix, the system may repeatedly send and process essentially the same context. The user question changes. The stable foundation doesn't. This is exactly the type of workload where prompt caching becomes valuable.

Modern provider APIs explicitly support this idea. Anthropic describes prompt caching as reusing a cached prompt prefix to reduce processing time and cost, while OpenAI has also implemented automatic prompt caching for repeated prefixes.

The Economics Are Bigger Than They Look

Suppose an agent sends a 10,000-token stable prefix and then a 200-token user request. If the agent makes 1,000 requests, that's potentially 10 million stable input tokens being sent as part of those requests.

If the stable prefix can be reused through prompt caching, a large portion of that repeated computation can be avoided or priced differently depending on the provider, and that's before considering what happens inside an agent.

One user request may produce a planning call, a retrieval call, a tool-selection call, a reasoning call, a summarization call, and a final response call. The same system instructions and tool definitions can appear across many of those requests.

So the question becomes:

How much of an agent's context is actually new on every model call?

Often, much less than we think.

But There Is a Problem: Every Provider Caches Differently

This is where things get interesting. Prompt caching sounds like a universal feature. It isn't. Different providers expose and implement it differently.

For example, Anthropic supports explicit cache breakpoints using cache_control. OpenAI uses automatic prefix caching. AWS Bedrock's Converse API uses cachePoint markers.

The APIs also differ in how caching is triggered, what can be cached, TTL behavior, cache-write pricing, cache-read pricing, usage fields, cache invalidation, minimum token requirements, and provider-specific limitations. The provider-specific differences are not just API syntax. They affect routing, cost calculation, observability, and reliability.

That creates a problem for anyone building a multi-provider AI system.

The Same Prompt Does Not Mean the Same Cache

Imagine your application sends a 10,000-token system prompt to Claude. The provider warms its cache. The next 100 requests reuse that prefix. Great. Now the provider experiences an outage. Your AI gateway automatically fails over to another provider. The application still sends the same 10,000-token prompt. But the second provider does not magically inherit the first provider's KV cache. The new provider sees a cold request. The context needs to be processed again.

This is one of the most important concepts in multi-provider AI infrastructure:

A cache is generally local to the serving path that created it.

Caches don't simply travel with your prompt when you switch providers.

The provider-specific gateway research shared with this article illustrates exactly this problem: traffic that was warm on one provider can become cold after fallback to another provider, creating a temporary cost spike while the new cache warms.

So Even "The Same Model" Can Have Different Cache Economics

This is an important distinction.

You might think: "I'm using the same model, so the cache should be the same." Not necessarily.

Suppose the same model family is available through Provider A and Provider B. The underlying model may be the same or closely related, but the serving infrastructure is different. The cache is associated with the infrastructure actually processing the request.

So Request 1 → Model X via Provider A may warm Provider A's cache. Then Request 2 → Model X via Provider B may still be a cache miss.

This means model routing and caching cannot always be treated as two completely independent systems.

Routing decisions can affect cache behavior. And cache behavior can affect the real economics of routing.

The Routing Problem Becomes More Interesting

Traditional model routing asks: Which model should handle this request? A more advanced routing layer should eventually be asking:

Which model and provider should handle this request, given capability, cost, latency, availability — and cache state?

Imagine two providers can both serve the same model.

Provider A: Cache hit rate: 88% Provider B: Cache hit rate: 46%

Their listed model prices might look similar. But their effective cost can be very different.

This is why a pricing table like the one below becomes interesting: the listed input price isn't necessarily the number that determines what you actually pay. Cache hit rate changes the effective input economics.

A gateway can therefore look beyond listed input price and evaluate effective input cost based on actual cache behavior.

That is a much more useful metric for production AI.

Prompt Caching Is Not Just "Turn It On"

There is another important catch. Caching typically depends on what exactly is in the prefix. A tiny change can create a different cache key.

Consider:

System Prompt
+
Tool Definitions
+
Company Policies
+
User Request

If you put the changing information in the wrong place:

System Prompt
+
Today's Date
+
Request ID
+
Random Tool Ordering
+
Tool Definitions

you can accidentally destroy the stability of the prefix.

The result? Lower cache hit rate. More recomputation. Higher cost.

The provider documentation makes this explicit: prompt-cache hits depend on matching the relevant prefix, and changing supposedly stable content can invalidate reuse.

Common Things That Quietly Kill Cache Hit Rates

Timestamps in system prompts — "Today is September 11, 2026." Tomorrow, it's different. Different prefix. Cold cache.

Request IDs — putting request_id = 7f82... inside the system prompt makes every request unique.

Random tool ordering — if the same tools appear in a different order, the prefix changes.

Per-user instructions — putting "You are assisting John." inside a shared system prefix creates a different cache for every user.

Model switching — caches are typically model-specific. Switching the model can mean recomputing the context even when the prompt itself hasn't changed. Anthropic's own current guidance explicitly notes that model changes can invalidate the cached state.

This Is Why Agent Architecture Matters

A well-designed agent isn't just a sequence of prompts. It is a sequence of state transitions.

Consider: System Instructions → Tools → Knowledge → Conversation → Reasoning → Tool Result → Next Reasoning Step

Some of those blocks are highly stable. Some change occasionally. Some change every turn. Some are unique to a single request.

A good caching strategy understands the difference.

Instead of treating the entire prompt as one giant string, we can think of it as layers of state.

The more carefully we separate those layers, the more opportunity we create for reuse.

The Cache Can Become Part of the Agent Runtime

This is where the recent thinking around KV cache becomes particularly interesting.

The traditional view is: KV cache = optimization for faster generation.

A more ambitious view is: KV cache = reusable execution state for an agent.

The Yandex Research work explores this idea directly, arguing that KV-cache blocks can potentially represent different streams of an interactive system — such as user input, reasoning, tool results, and other workers — and that inference-time manipulation of those blocks can support concurrent agent behavior without retraining the model.

That changes how we think about caching. It is no longer only "How do I save money on tokens?" It becomes "How do I manage the computational state of my AI system?"

From Prompt Cache to Cache-Aware AI Infrastructure

Now bring all of this together.

Your AI gateway sits between Applications & Agents and Models & Providers.

The gateway already knows which model was requested, which provider is serving it, how much the request costs, how many tokens were used, which routing policy was applied, whether a fallback occurred, whether the request hit a cache, and how much context was reused.

That creates an opportunity.

Instead of treating caching as a provider-specific feature hidden inside individual integrations, the gateway can expose a common caching abstraction.

The application says, conceptually: "This portion of the request is reusable." The gateway then translates that intent into whatever the target provider supports.

One Cache Policy. Multiple Provider Implementations.

This is the problem with provider-specific caching APIs.

Your application shouldn't need to know:

Anthropic -> cache_control
OpenAI -> automatic prefix caching
Bedrock -> cachePoint

The application should ideally deal with something more consistent:

cache:
  enabled: true
  key: customer-support-agent
  ttl: appropriate

Then the gateway determines how to express that intent to the selected provider.

If the target provider supports the requested behavior: Translate → Forward → Observe. If it supports a different mechanism: Translate → Forward → Observe. If it doesn't support the requested behavior: Downgrade / Strip / Route differently → Log.

The provider-specific gateway material highlights exactly this normalization challenge: a multi-provider gateway may need to translate, strip, or reject cache directives depending on the target provider's capabilities.

The Gateway Also Needs to Know When Caching Is Actually Working

Saying "caching is enabled" is not enough. You need to know: are requests actually hitting the cache?

Consider this:

The interesting metric isn't just "Provider D charges $X per million tokens." It's:

"What did this workload actually cost after cache reuse?"

That is why a gateway needs normalized cache metrics: cache hits, cache misses, cache writes, cache-read tokens, cache-write tokens, cache hit rate, provider, model, TTL, and effective input cost.

The TrueFoundry material makes this point strongly: provider-specific usage fields need to be normalized if you want meaningful cross-provider cost attribution and cache analysis.

Cache-Aware Failover

Now consider the failure scenario again.

Your primary provider has a 95% cache hit rate. Your fallback provider has 20% cache hit rate.

A traditional failover system sees: "Provider A unavailable → use Provider B."

A cache-aware system sees: "Provider A unavailable → Provider B is available, but its cache is cold."

That matters. Because the fallback may technically keep your application running while temporarily increasing input processing cost, time to first token, GPU utilization, latency, and cache write volume.

This doesn't mean you should refuse to fail over. Availability still matters. But it means the gateway should understand the economic and performance consequences of the fallback path.

Reliability and caching are connected.

Cache Warming Can Also Matter

Suppose you know that an agent uses a large system prompt and tool registry.

Instead of waiting for the first real user request to populate the cache, some providers support mechanisms for pre-warming reusable prompt state.

Anthropic, for example, documents cache pre-warming for suitable workloads to eliminate the first-request cache-miss latency penalty.

This creates another infrastructure question: Which provider should be warmed, and when?

If your gateway knows that 80% of traffic normally goes to Provider A, it can reason about cache state as part of traffic management.

Again, caching starts looking less like an application feature and more like infrastructure.

But There Is No Single "Cache"

This is perhaps the most important lesson. There are multiple layers of caching in modern AI systems.

These layers solve different problems. They shouldn't be confused.

The Most Expensive AI Request Might Be the One You Never Needed to Make

Think about the optimization hierarchy.

First: Can we avoid the model call entirely? If not: Can we reuse the previous result? If not: Can we reuse the prompt prefix? If not: Can we route the request to a cheaper model that can still complete the task? If not: Can we use the most appropriate model and provider?

The industry often starts at the last step. But the biggest savings can happen earlier in the chain.

What This Means for AI Agents

As agents become more sophisticated, they generate more repeated context: system instructions, tool definitions, memory, retrieved information, previous reasoning, workflow state, policies, environment information.

Many of those things remain stable across multiple model calls. That means caching isn't a small optimization. It becomes a fundamental part of agent infrastructure.

The future agent runtime may not simply ask "Which model should I call next?" It may ask:

"What state already exists, what can I reuse, what needs to change, and where should the next computation happen?"

Where SPRO Fits

This is where an AI gateway becomes much more than an API proxy.

SPRO can sit between your applications and AI providers as the control layer responsible for:

• Connect — One interface across providers.
• Route — Select models and providers based on requirements.
• Cache — Reuse eligible AI context and reduce unnecessary repeated processing.
• Protect — Apply security and privacy policies.
• Control — Manage usage and cost.
• Observe — Track cache behavior, tokens, latency, cost, models, providers, and requests.
• Fail Over — Move traffic when a provider becomes unavailable.

The important part is that these capabilities work together. Because caching cannot be treated independently from routing. Routing cannot be treated independently from cost. Cost cannot be treated independently from observability. And provider failover cannot be treated independently from cache state.

The AI Gateway Is Becoming a Cache-Aware Control Plane

The old architecture looked like: Application → LLM API → Model.

The new architecture is becoming a gateway-mediated one.

The gateway becomes the place where the organization understands not only "Which model answered this request?" but also "How much of this request actually needed to be computed?"

That is a much more powerful question.

Stop Paying for Work You've Already Done

AI optimization used to be mostly about model selection. Then it became about model routing. Now it is increasingly about computation reuse.

KV caching prevents the model from repeatedly recomputing previous context during generation. Prompt caching extends that idea across requests by reusing stable prefixes. Provider-aware gateways take the idea one step further by making caching part of a multi-model infrastructure strategy.

Because in a multi-provider world:

The same model doesn't necessarily mean the same cache. The same prompt doesn't necessarily mean the same cache behavior. The cheapest listed token price doesn't necessarily mean the cheapest effective cost.

And: a successful request isn't necessarily an efficient request.

The most efficient AI system is one that knows when to generate, when to reuse, when to cache, and where to route.

Because the next frontier of AI optimization isn't just "Use a cheaper model." It is:

"Don't pay to compute what you already know."