KV Cache
The mechanism that lets a LLM avoid recomputing the whole start of a conversation for every new token it generates. It keeps the work already done on the context in memory (the 'keys' and 'values') and only processes what follows. Direct consequence for you: if the start of your prompt stays identical across calls, the model reuses it and it costs far less. It is the engine behind the 'prompt caching' that Anthropic and others bill.
Strengths
- Stable prompt prefix reused = input token bill heavily reduced
- Also reduces latency: less compute to redo each turn
Limitations
- Breaks the moment you change the start of the prompt: a single upstream character invalidates the cache
- In a Claude Code session, switching models or adding an MCP midway invalidates the whole cache: instructions, CLAUDE.md and history are billed again at full price. Configure before you start
Best for
- Reducing the cost of an agent replaying the same system context in a loop