Prompt Caching - Cutting LLM Costs for Recurring Context

Prompt caching (also context caching) is a technique in which the unchanging parts of a prompt are cached and reused across multiple requests - instead of being processed and paid for anew each time. This mainly concerns long, stable building blocks: an extensive system prompt, fixed context documents, or recurring examples.

Without caching, a language model reprocesses the entire input text on every request, including the identical parts. This is exactly where prompt caching comes in: the stable prefix is processed once, its internal state is stored, and it is reused directly on the next request. This saves compute and lowers token costs.

With the providers, cached context tokens are billed noticeably cheaper than freshly processed input tokens. The effect grows with the length of the repeated prefix and the frequency of its reuse.

How Prompt Caching Works

• Stable prefix: The recurring start of the prompt (system prompt, knowledge documents, few-shot examples, agent context) is marked as a cacheable block.
• Cache write: On the first pass, the block is processed and cached. A cache write can be somewhat more expensive than a regular pass.
• Cache read: On subsequent requests with the same prefix, the stored state is read - cheaper and faster.
• Amortization: The one-time surcharge for the cache write pays off as soon as the prefix is reused several times.

When Prompt Caching Applies

The lever works when a long, stable prefix repeats across many requests. Typical situations:

• Extensive system prompt: Roles, rules, and framing instructions that stay identical on every request.
• Fixed context documents: Manuals, guidelines, or knowledge bases that serve as the basis for a conversation.
• Few-shot examples: A fixed collection of sample cases that steer behavior.
• Agent context: Tool descriptions and history that an agent carries across many steps.

Limits and Conditions

A cache has a limited lifetime (TTL): if the prefix goes unused for a while, the entry expires and has to be rewritten. The cached block must also stay exactly identical - even small changes to the prefix invalidate the cache. That is why the stable parts belong at the front, the variable parts at the back. Providers such as Anthropic, OpenAI, and Google support prompt caching; the specifics (pricing, minimum lengths, TTL) differ by provider and must be weighed against the available context window.

As a building block of well-considered LLM cost optimization, prompt caching complements other levers and is especially effective for recurring context. How to plan and implement such measures in combination is shown on our service page AI Cost Optimization - the spend for repeated context then often amounts to only a fraction of the uncached cost.

FAQ

What is prompt caching?

A technique that caches the unchanging parts of a prompt and reuses them across multiple requests, instead of processing and paying for them each time.

Why does prompt caching lower costs?

Cached context tokens are billed far cheaper by the providers than freshly processed input tokens. The longer and more frequent the repeated prefix, the greater the savings.

Which cases is it suited for?

Long, stable prefixes that repeat across many requests - such as extensive system prompts, fixed knowledge documents, few-shot examples, or agent context.

Further Resources

• Glossary: LLM Cost Optimization, Token Costs, Context Window
• Service: AI Cost Optimization