What is Replace RAG with Direct Context?
Replace RAG with Direct Context is an AI refactoring for LLM systems, in the Organizing data group. When the material fits in the window, send it whole and use prompt caching.
Trail content
Chapters
Transcript
Opening
Back to the legal assistant from the last lesson. One services contract, eighteen pages, about twelve thousand tokens.
The pipeline splits it into five hundred token chunks, and retrieves the four nearest to each question. Four chunks of five hundred is two thousand tokens. Out of twelve thousand.
So when someone asked about terminating for late delivery, the model answered looking at one sixth of the document. And it didn't know that. It said no penalty. Clause fourteen point three said otherwise.
In this lesson, you'll take the retrieval out, safely, put the whole contract in front of the model, and keep the bill under control.
The idea
The refactoring fits in two sentences. Send the whole material in the context, in a fixed block at the start of the call. And turn on prompt caching, so you don't pay for the full read on every question.
Vector search exists to pick the little that fits, when the material is large. With small material, that choice only removes information. The split separates clauses that depend on each other, and the search misses when the question's words don't match the text.
Before and after
Before, the contract becomes chunks and embeddings. The question picks four of them. And the model answers with whatever the search chose.
After, the whole document goes into a fixed block, right after the instructions.
On the first question, that prefix is written to the cache.
On the next ones, the model reads the contract from the cache, at a fraction of the cost.
And the answer can connect distant clauses, because nothing was cut.
The same question
So let's ask the same question again. With four chunks, the context had termination, and not the penalty. With the whole contract, both clauses sit in the same prefix.
And the instruction asks the model to cite article and paragraph. So the answer becomes this. No. Clause eight allows termination for delay, but fourteen point three applies a penalty on default, and late delivery falls under that definition. That's an answer a lawyer can check in ten seconds.
One warning still stands. A huge context doesn't guarantee the model finds the clause in the middle. The research calls it lost in the middle. That's why the condition is fits with room to spare. Twelve thousand tokens in a large window is room. A hundred thousand tokens of stacked annexes is a different design.
A challenge
Here's one for you. After the switch, the user's second question shows zero tokens read from cache. Looking at the prompt, the system block starts with today's date, so the model knows the day, and then the contract. What's the fix?
Take the date out of the prefix, and send it in the user message, after the cached document. The cache reuses an identical start of the prompt. A date that changes every day means there's never anything to reuse.
It's not the provider's fault, and caching the user message doesn't help either, because that changes on every question. Instructions and contract stay stable. Date, user and question come after.
Safe mechanics
Now the mechanics.
Step zero, measure. Run the question eval, and include questions that span sections, like a rule and its exception. Record accuracy, cost and latency per question. Those cross-section questions are the reason for the swap.
Check the size. Count the tokens with the provider's token counter, not a guess. Leave room for the conversation and the answer. If there's no room, this refactoring doesn't apply.
Build the fixed prefix. Instructions first, then the document. And nothing that varies before the end of the document. No date, no user id, no time.
Turn on prompt caching on the document block.
Confirm it works. From the second call on, the usage report should show tokens read from cache. If it shows zero, the prefix is changing somewhere.
Measure again, and compare accuracy and cost per question. The cross-section questions are the test that matters. In the example, retrieval scored sixty one percent, and the whole document scored eighty eight.
And only after the new version is stable, remove the retrieval pipeline. Don't leave it behind a flag, just in case. While both exist, someone will just tune the number of chunks, during an incident.
Risks
A few risks. The material grows. Today it's one contract. Tomorrow it's a hundred and twenty vendor contracts, and the prefix no longer fits with room. Size changed, so the tool changes. Go back to an index, filtered by the user's contract, and keep direct context for the single document that still fits.
Widely spaced questions. The cache expires after a few minutes without use. With low traffic, the discount shows up less. The accuracy gain on cross-clause questions still holds.
Parts the user may not see. If some users may only see the pricing annex, a prompt asking the model to hide the rest is not access control. A leading question leaks it. Permission applies before the context is built, with filtered retrieval, or a prefix per role.
And thinking that it fits is enough. If the cross-section eval stays poor with the whole document, the problem wasn't retrieval. It may be the prompt, the model, or a contract that's genuinely ambiguous.
And a quick boundary. An online store thinks about pasting its catalog of eight thousand products into the prefix, with caching. That's not this refactoring. It doesn't fit with room, each question needs a few rows, and a product's price is a lookup by its code. That's a query tool, or retrieval, not an eight thousand item prefix.
The code, by intent
The code gets shorter, which is a good sign.
The contract is loaded once, as text.
The system part has two blocks. The instructions, and then the contract, marked for caching.
And the question goes alone in the user message, after everything that's stable. The three lines that used to embed the question, search the database and join the chunks are simply gone.
On the page, you'll find the same change across the code tabs. What changes between them is only how each one marks the block for caching, and where it reports the tokens read from cache.
Wrap-up
So here's the idea to take with you. When the material fits with room to spare, retrieval is a filter that only removes. Put the document first, the question after, let the cache carry the cost, and measure on the questions that cross sections.
But sometimes the material really doesn't fit, and you do need chunks. In the next lesson, we'll make each chunk carry the context it lost when it was cut: contextual chunking. See you there.
Replace RAG with Direct Context
Trocar RAG por Contexto Direto
Replace RAG with Direct Context is an AI refactoring for LLM systems, in the Organizing data group. When the material fits in the window, send it whole and use prompt caching.
- Element
- 18/35
- Level
- Beginner
- Read
- 5 min
- Updated
Problem
A vector search pipeline serves small material that would fit whole in the context, and loses important passages along the way. This is the RAG for Small Data smell.
The legal assistant answers questions about an 18-page services contract (about 12,000 tokens). The architecture that left the demo: split into 500-token chunks, embed, retrieve the 4 nearest. The question “can I terminate without a penalty if the vendor delivers late?” brought back the termination clause and not the penalties clause, which talked about “default” instead of “delay”. The model answered “yes, without a penalty”. Clause 14.3 said the opposite.
The whole contract fit in the window with room to spare. Four 500-token chunks are 2,000 of the 12,000: the model answered looking at one sixth of the document, and did not know that. Retrieval did the job it exists for when the corpus does not fit. Here the corpus fit.
Solution
Send the whole material in the context, in a fixed block at the start of the call, and use prompt caching so you do not pay for the full read on every question.
Before and after
-chunks = vector_db.search(embed(question), k=4)
-context = "\n".join(c.text for c in chunks)
-reply = llm(SYSTEM, f"{context}\n\nQuestion: {question}")
+CONTRACT = load_text("contract.pdf") # 18 pages, about 12,000 tokens
+
+reply = client.messages.create(
+ model=MODEL, max_tokens=2000,
+ system=[
+ {"type": "text", "text": SYSTEM},
+ {"type": "text", "text": f"<contract>\n{CONTRACT}\n</contract>",
+ "cache_control": {"type": "ephemeral"}}, # cached prefix
+ ],
+ messages=[{"role": "user", "content": question}],
+)Before: the model sees the 4 chunks the search picked.
flowchart LR
D(["1. Contract"]):::io --> C["2. Chunks and embeddings"]:::tool
Q(["Question"]):::io --> S["3. Retrieve 4 chunks"]:::tool
C --> S
S --> M(["4. Model with part of the contract"]):::llm
After: the model sees the whole contract, and the cache absorbs the cost.
flowchart LR
D(["1. Whole contract in the prefix"]):::io --> K[("2. Prompt cache")]:::store
Q(["Question"]):::io --> M(["3. Model with the full contract"]):::llm
K --> M
M --> OUT(["4. Answer that considers every clause"]):::io
- The whole document goes into a fixed block, after the instructions.
- On the first question, the prefix is written to the cache.
- On later ones, the model reads the contract from the cache, at a fraction of the cost.
- The answer can connect distant clauses, because nothing was cut.
The question RAG got wrong
With the 4 chunks, the context had termination and not the penalty. With the whole contract, the same question sees both clauses in the same prefix. The instruction asks for article and paragraph. The answer becomes: “no: clause 8 allows termination for delay, but 14.3 applies a penalty on default, and late delivery falls under that definition”.
The figures on this page follow the 18-page contract already used in the smell. They are not a new product measurement.
Why it works
Vector search exists to pick the little that fits in the context when the material is large. With small material, that choice only removes information: the split separates clauses that depend on each other, and retrieval misses when the question’s words do not match the text’s.
With the whole document, the model does what it does well: read, connect, and cite. Prompt caching answers the cost objection, because the identical prefix (instructions and document) is billed at a discount from the second call on. And the system gets simpler: no index, no embeddings, no retrieval tuning.
This refactoring is the other side of Move Retrieval Out of the Prompt. There the corpus does not fit and changes all the time. Here the material fits and retrieval is what is getting in the way.
The paper “Lost in the Middle” (Liu et al., 2023) still stands as a warning: a huge context does not guarantee the model finds the clause in the middle. That is why “fits with room to spare” matters. Twelve thousand tokens in a large-window model is spare room. A hundred thousand tokens of stacked annexes is not the same design.
Safe mechanics
- Measure. Run the question eval, including questions that span sections. Record accuracy, cost, and latency per question. The cross-section questions are the reason for the swap.
- Check the size. Count the material’s tokens with the API’s token counter. Leave room for the conversation and the answer. If there is no room, this refactoring does not apply.
- Build the fixed prefix: instructions first, then the document, with nothing variable (date, user id, time) before the end of the document.
- Turn on prompt caching on the document block.
- Confirm the cache works: from the second call on,
usageshould show tokens read from cache. If it shows zero, the prefix is changing. - Measure again and compare accuracy and cost per question. Accuracy on the cross-section questions is the test that matters.
- Remove the retrieval pipeline only after the new version is stable. While both exist, someone will “just tune k”.
Risks
- The material grows. Today it is one contract; tomorrow it is a hundred. Monitor the size and go back to retrieval once it passes the limit where spare room remains.
- A cache that never hits. A date or an id before the document changes the prefix and resets the cache. Keep what varies afterwards, in the user message.
- Widely spaced questions. The cache expires after a few minutes without use (longer durations are available). With low traffic, the discount shows up less. The accuracy gain on cross-clause questions still holds.
- A document with parts the user may not see. Sending everything ignores per-passage permissions. In that case, filtered retrieval is required, even if the file fits.
- Thinking that “it fits” is enough. If the cross-question eval stays poor with the whole document, the problem was not RAG. It may be the prompt, the model, or a genuinely ambiguous contract.
Common mistake: a prefix that looks fixed and is not
The team puts the contract in the system prompt and, in the same block, today’s date “so the model knows the day”. Each day the prefix changes, the cache never hits, and the bill returns to a full-read price. Date, user name, and session id go in the message, after the document.
Challenge
An assistant answers questions about a condominium’s bylaws (25 pages). Today it uses RAG and gets questions like “can I do renovation work on Saturday afternoon?” wrong, because the general rule is in one section and the Saturday exception in another. What does the direct-context version look like, and what should you check?
Show solution
The whole set of bylaws goes into a fixed system prompt block, with caching, and the instructions ask to cite article and paragraph. Check: the token size, leaving room for the conversation; a positive cache_read_input_tokens from the second question on; and the eval with questions that depend on a rule and an exception in different sections, which were exactly the ones RAG got wrong.
Check your understanding
Lesson check
Answer the 6 questions to complete this lesson. Watching alone does not complete it: the check is what counts toward the trail and the certificate.
Quick answers
What problem does Replace RAG with Direct Context solve?
It treats RAG for Small Data.
Which pattern does Replace RAG with Direct Context lead to?
The Prompt Caching design pattern.
Go further
Relations
- Smells it treats 1
- Target patterns 1
- Shows up in scenarios 1
- Part of the tracks 1