Why prompt caching matters
The dominant cost in most production LLM workflows isn't reasoning — it's the same system prompt, tool schemas, and retrieved context getting billed on every request. A well-designed RAG stack might send 15,000 tokens of unchanging preamble to answer a 200-token question, and pay for the preamble at full rate every time.
Prompt caching flips that. Anthropic charges 10% of the input rate for cached tokens; OpenAI charges 50%. On a workload that repeats a 15K-token system message 100 times an hour, that's the difference between a $2,000/month bill and a $250/month bill for identical output.
How the two APIs differ
Anthropic's caching is explicit: you mark blocks with cache_control: {type: 'ephemeral'}. TTL is ~5 minutes; the first request pays a 25% write premium, subsequent hits pay 10% of the input rate. You control what to cache.
OpenAI's caching is automatic on GPT-4o family and above for prompts ≥1,024 tokens. Hits pay 50% of the input rate. You don't opt in — but you also don't control it, which means small edits to the top of your prompt invalidate the whole cache.
Both cache from the beginning of the prompt forward. Static content must live at the top, dynamic content at the bottom. This is the single most important design rule.
Three patterns that win
First: park your tool schemas and system prompt at the very top, mark them cached (Anthropic) or accept the automatic cache (OpenAI). This alone recovers 30-50% of cost on any tool-using agent.
Second: for RAG, cache the retrieved chunks in a separate block below the system prompt. If your retrieval is stable across a conversation (e.g. one document per session), a 100K-token doc gets billed at 10% of input on every follow-up question.
Third: batch similar requests into a shared session where possible. Anthropic's 5-minute TTL means bursts win; if you can group 10 related queries into a single window, cache hit rate approaches 100%.
Pitfalls
Timestamps in your system prompt destroy caching. Move them to the user block or omit entirely.
Randomly ordered tool definitions look different to the cache key. Sort tools alphabetically at build time.
OpenAI's cache doesn't survive across regions or accounts — use one project consistently for cacheable traffic.
Do not cache PII-heavy contexts across users. Cache keys are per-organization on both providers, but scoping remains your responsibility.
Measuring impact
Both APIs return cache_read_input_tokens in the response. Log it. Dashboard the ratio of cached to total input tokens by workflow. A healthy production workflow with static context lives above 70%.
If your ratio is below 30%, you have prompt drift — something is changing across requests that shouldn't be. Common culprits: timestamps, user IDs at the top of the prompt, dynamic retrieval order.