Move Retrieval Out of the Prompt

Problem

The system prompt carries whole documents (policies, catalogs, FAQs) on every call, keeps growing, and goes stale. This is the Prompt Bloat smell, in its pasted-knowledge form.

The company’s HR assistant looked good in the demo: ten policies in the system prompt, vacation and remote-work questions answered on the spot. In six months legal sent 27 new policies. Someone pasted them. The prompt went from about 12,000 tokens to about 85,000. Every “how many vacation days do I get?” paid for reading all 37.

In March, HR changed remote work from 3 days to 2. The new policy hit the internal wiki the same day. The bot’s prompt still had the 2024 paragraph, plus a later note “new rule (Mar/26): 2 days”, and the model picked one of the two depending on the question. In Moffatt v. Air Canada (2024), an airline chatbot invented a refund policy and the company was ordered to honor what the bot promised. A wrong policy in the prompt becomes a commitment to the customer.

Solution

Take the knowledge out of the prompt and into a search index. On each question, retrieve only the relevant passages and put them in the context as data, with their source. The prompt keeps the behavior instructions.

Before and after

-SYSTEM = f"""You are the HR assistant. Answer based on the policies below.
-{open('policies/vacation.md').read()}
-{open('policies/expenses.md').read()}
-{open('policies/remote_work.md').read()}
-... [37 policies, 85,000 tokens]"""
-reply = llm(SYSTEM, question)
+SYSTEM = """You are the HR assistant. Answer only from the passages inside
+<passage>. Cite the policy and section for each claim. If the passages do not
+answer the question, say you did not find it and point to the HR channel."""
+
+chunks = index.search(question, k=6, filters={"status": "current"})
+context = "\n".join(f'<passage source="{c.doc}" section="{c.section}">{c.text}</passage>' for c in chunks)
+reply = llm(SYSTEM, f"{context}\n\nQuestion: {question}")

Before: every policy on every call.

flowchart LR
  P[("1. 37 policies in the system prompt")]:::store --> M(["2. Model"]):::llm
  Q(["Question"]):::io --> M
  M --> OUT(["3. Answer, sometimes with an old rule"]):::io

After: the index holds the knowledge; the prompt receives only what matters.

flowchart LR
  Q(["1. Question"]):::io --> S["2. Retrieve current passages"]:::tool
  I[("Policy index")]:::store --> S
  S --> M(["3. Model with 6 passages and sources"]):::llm
  M --> OUT(["4. Cited answer"]):::io

1. The question arrives.
2. Search brings back a few relevant passages, filtered to the current version.
3. The model answers only from them.
4. The answer cites its source; with no relevant passage, it says it did not find one.

One question, two contexts

Question: “how many remote days is it now?”

Before, the model saw 85,000 tokens, including the 2024 paragraph (“3 days”) and the March note (“2 days”), far apart. After, the status: current filter returns one passage:

<passage source="remote-work-policy-v4.md" section="3. Remote days">
From 1 March 2026, the arrangement is 2 remote days per week.
The 2024 version (3 days) is revoked.
</passage>

The answer cites remote-work-policy-v4.md, section 3. If search finds no current passage, the bot says it did not find one and points to the HR channel, instead of improvising.

Why it works

The prompt holds behavior, which changes rarely. The index holds knowledge, which changes all the time. Separating them solves three problems at once.

• Cost. Each call carries a few passages instead of dozens of documents. The 85,000 tokens become, in this example, on the order of 2,000.
• Freshness. A new policy enters the index the same day, with no prompt PR. The current-version filter hides the old one.
• Attention. The model reads only what matters for that question. There is no 2024 paragraph competing with the 2026 one in the same context.

With a source on every passage, the answer can be checked. “Not found” becomes a useful signal of a documentation gap, instead of a polite hallucination. The destination pattern is RAG.

Check +20 XP on first tryThe HR bot still answers '3 remote days' a week after policy v4 (2 days) was published on the internal wiki. The system prompt has all 37 policies pasted in. What does the refactoring change first in this case?

The model, because a larger one can prefer the newest paragraph in a long prompt

The prompt gets a line at the top saying 'always use the newest rule'

The policy leaves the prompt; search returns only passages with current status, and v4 enters the index the same day HR publishes

Safe mechanics

0. Measure. Build an eval with real questions and the policy that answers each. Include questions about rules that changed. Record accuracy, cost, and latency, and note whether the answer cites the right version.
1. Separate instructions from knowledge in the current prompt. Tone, limits, and what to do when you do not know stay. Vacation, expenses, and remote work leave.
2. Split the documents into passages that respect their structure (section, clause), with a title and source document on each. See Contextual Chunking.
3. Index them with metadata: document, section, date, status (current / revoked). Without status, search returns v3 and v4 together.
4. Evaluate retrieval alone: for each eval question, is the right passage among the top results? If it is not, the model never sees the rule. Improve retrieval before judging the new prompt.
5. Replace the prompt with the version that has delimited passages and an instruction to cite. Tags make it clear what is data and what is instruction.
6. Measure again. Equal or better accuracy, lower cost and latency, and citations matching the current version.

Risks

• Search misses the right passage. The answer ends up wrong or empty. Measure retrieval separately and improve it with Hybrid Search and Reranking.
• Questions that span documents. “Can I combine vacation with the year-end shutdown?” needs two passages. A k that is too small cuts the second policy. Raise k or use multi-step retrieval.
• Small material. If everything fits comfortably in the context, sending it whole with caching may be better. See RAG for Small Data and Replace RAG with Direct Context.
• Pasting it back when retrieval fails. The reflex to “then send all 37 so the model does not miss” recreates prompt bloat. Fix retrieval and the eval; do not reopen the door.

Common mistake: an index with no version

If the document in the index is a vacation.md that someone overwrites, you cannot filter to current and you cannot audit what the bot saw on Tuesday. Store each publication as a version, with a date and a status. Default search reads only current. The trace stores source and version for every passage sent.

Check +20 XP on first tryAfter the swap, the answer eval dropped. The retrieval eval shows that for 30% of questions the right passage is not in the top 6. The team wants to 'improve the assistant prompt'. What do you do?

Improve the prompt, asking the model to admit uncertainty more often

Treat retrieval as the bug: measure recall of the right passage, tune chunking, filters, and reranking, and only then re-evaluate the answer

Raise k to 40, so almost the whole index fits in every question

Challenge

A sales assistant’s prompt holds the price table for 300 products and the discount rules. The discount rules change once a year; the table, every week. What goes into the index and what stays?

Show solution

Neither needs to stay in the prompt in full. The price table changes weekly and each question uses a few rows: the best fit is a per-product lookup tool (exact data, not semantic search). The discount rules are short and stable: they can stay in the prompt, or in a final-price calculation tool so the model does not do arithmetic. The prompt keeps the sales behavior.

Check your understanding

Check +20 XP on first tryAn expense-reimbursement question needs the expenses policy and the corporate-card policy. With k=3, search returns three sections of the first and none of the second. What is the most direct fix?

Paste those two policies back into the system prompt, only those two

Tell the prompt to recall the card policy, since it has barely changed

Raise k enough to cover cross-document questions, or retrieve in two steps (rewrite the question and search again)

Check +20 XP on first tryLegal publishes vacation policy v5 at 9:00. At 9:12 an employee asks the bot and gets the v4 rule. The index has a single vacation.md file, overwritten. Which gap in the refactoring shows up here?

Missing version and status metadata: publishing should create a current v5 and mark v4 revoked, and search should filter to current

Missing a larger model, which would notice that the text 'looks old'

None: a 12-minute delay is acceptable for an internal policy

Check +20 XP on first tryYour policy corpus is 8 pages in total, fits easily in the window, and changes twice a year. Search sometimes misses the exception that sits in another section. Is this refactoring still the first choice?

Yes, RAG is always better than sending the whole document

No: small material that fits in the window usually goes in whole, with prompt caching; RAG here is the RAG for Small Data smell

No, and the knowledge should go back into the system prompt for good

Check +20 XP on first tryThe bot starts saying 'not found' on questions the wiki covers. The retrieval eval looks fine. Which hypothesis do you test before touching the index again?

The new prompt is too strict about answering only from the passages, and the passages arrive without the neighboring section that completes the answer

The model got worse and needs to be swapped

Citing sources is getting in the way; drop the duty to cite