The article breaks down prompt caching, a technique used in large language models (LLMs) to enhance efficiency by reusing previously computed key-value tensors for identical prompt prefixes. The author shares their personal experience, highlighting mistakes made under pressure while developing a feature that involved chat and tool calling components. Initially, they misunderstood how prompt caching worked, thinking it was limited to a single user session. In reality, prompt caching allows different users to share the same system prompt across sessions, leading to faster responses and reduced costs. Key to understanding prompt caching is the concept of KV cache reuse, which is enabled through various techniques like paged attention. The author emphasizes the importance of maintaining a stable prefix in prompts, suggesting that removing user-specific data helps increase cache hits. For example, they moved towards an append-only context model, avoiding truncation of tool outputs, which improved performance by keeping the prefix intact. The article also discusses how different LLM providers, like OpenAI and Anthropic, approach caching and pricing, revealing how cache retention policies impact performance and costs. Practical tips for improving cache hits include ensuring that prompts are stable and deterministic. The article references guidance from both OpenAI and other resources, stressing that following these practices can significantly reduce costs associated with token usage. The author’s experiences and insights aim to help readers gain a better understanding of how prompt caching works and how to leverage it effectively in their own projects.