Introduce Cache

Problem

The same large prefix and the same questions are processed from scratch every time, and cost grows with traffic. This is the Missing Cache smell.

Solution

Reuse whatever repeats, at three levels: prompt caching for the fixed prefix (instructions, tools, documents), an embedding cache for text that does not change, and an answer cache (exact or semantic) for equivalent questions, with an expiry rule.

Before and after

-def answer(question):
-    q_vec = embed(question)                                      # paid every time
-    docs = index.search(q_vec, k=6)
-    return llm(SYSTEM_9K_TOKENS, docs, question)                 # prefix paid every time
+def answer(question):
+    key = normalize(question)                                    # lowercase, no punctuation
+    if (hit := answer_cache.get(key)) and hit.sources_unchanged():
+        return hit.answer                                        # 1. answer cache
+
+    q_vec = embedding_cache.get_or_compute(key, embed)           # 2. embedding cache
+    docs = index.search(q_vec, k=6)
+    reply = client.messages.create(
+        model=MODEL, max_tokens=1500,
+        system=[{"type": "text", "text": SYSTEM_9K_TOKENS,
+                 "cache_control": {"type": "ephemeral"}}],      # 3. prompt cache
+        messages=[{"role": "user", "content": render(docs, question)}],
+    )
+    answer_cache.set(key, reply, ttl="24h", sources=[d.id for d in docs])
+    return reply

Before: everything processed from scratch.

flowchart LR
  Q(["1. Question"]):::io --> E["2. Embedding"]:::tool
  E --> S["3. Search"]:::tool
  S --> M(["4. Model reads 9,000 fixed tokens"]):::llm

After: three layers of reuse.

flowchart LR
  Q(["1. Question"]):::io --> C{"2. Already answered?"}
  C -->|yes, same sources| OUT(["Answer from cache"]):::io
  C -->|no| E["3. Embedding from cache or new"]:::tool
  E --> S["4. Search"]:::tool
  S --> M(["5. Model with a cached prefix"]):::llm
  M --> OUT2(["Answer, stored in the cache"]):::io

1. The question arrives and is normalized.
2. If the same question was answered and its sources did not change, the answer returns at once.
3. Embeddings for repeated questions are not recomputed.
4. Search runs as usual.
5. The fixed prefix is read from the prompt cache, at a discount and with lower latency.

Why it works

Each layer targets one kind of repetition. Prompt caching exploits the fact that instructions and tools are identical on every call: the provider keeps the prefix’s processing and charges a fraction to reuse it. The embedding cache avoids recomputing vectors for identical text. The answer cache skips the whole call when the question has already been answered and nothing has changed since.

The expiry rule is what makes the answer cache safe: a cached answer records which sources it came from, and stops being valid if any of them changes. The target patterns are Prompt Caching and Semantic Cache.

Prompt caching reuses a byte-identical prefix. On Anthropic, the order is tools, then system, then messages, and cache_control marks how far the prefix should be stored. The default lifetime is a few minutes, renewed on each hit, with a one-hour option that costs more to write. One extra space, or a timestamp at the start, breaks the hit. If calls are 20 minutes apart, the cache expires between them and almost every call pays the write.

Safe mechanics

0. Measure. Gather the question distribution (how many repeat), the fixed prefix size, and the cost per question. Check the cache fields in the API’s usage.
1. Start with prompt caching: order the prompt with fixed content first and variable content last, and mark the cache point.
2. Confirm hits: from the second call on, cache_read_input_tokens should be positive.
3. Add an embedding cache for repeated text.
4. If repetition is high, add an answer cache with a normalized key, an expiry, and invalidation on source changes.
5. Consider a semantic cache only with a threshold tested in the eval, so you do not return the answer to a similar but different question.
6. Measure again: hit rate per layer, cost and latency per question, and eval quality.

If cache_read_input_tokens stays at zero, look for something variable at the start of the prompt, or for traffic so sparse that the cache expires between calls.

Risks

• Stale answers. An answer cache with no expiry serves outdated information. Tie expiry to the sources.
• Someone else’s answer. Questions with personal data (“what is my balance?”) cannot share answers. Include the user in the key or do not cache.
• A loose semantic cache. “Cancel subscription” and “subscribe to a plan” are close in vector space. Test the threshold with hard pairs.
• An unstable prefix. One variable detail at the start of the prompt resets the prompt cache without anyone noticing.
• A cache that hides a regression. After you change the prompt, old answers keep coming from the answer cache. Include the prompt and model version in the key.

Questions that differ by a city, amount, or date (shipping to Manaus versus Belém, two cities in Brazil) are close in vector space and far in meaning. Those need an exact cache with the parameter in the key, or no cache.

Challenge

Why does this prompt never hit the cache, and how would you reorganize it?

system = f"""Date: {datetime.now().isoformat()}
User: {user.name} (plan {user.plan})
You are the store's assistant. [8,000 tokens of rules and tool catalog]"""

system = [
    {"type": "text", "text": RULES_AND_TOOLS_8K, "cache_control": {"type": "ephemeral"}},
    {"type": "text", "text": f"User: {user.name} (plan {user.plan}). Date: {date.today()}"},
]

Check your understanding