Opening

A legal assistant answers questions about one services contract. Eighteen pages. About twelve thousand tokens.

The team built it the way the tutorials show. Split the contract into small chunks, turn each chunk into an embedding, store them in a vector database. For each question, retrieve the four most similar chunks, and let the model answer.

Then a user asks. Can I terminate without a penalty if the supplier delivers late? The search brings back the termination clause. It doesn't bring back the penalties clause, which sits on another page and says contractual default, not late delivery.

The model answers yes, no penalty. Clause fourteen point three says the opposite. And here's the detail that makes this a smell. The whole contract fit comfortably in the model's context. In this lesson, you'll learn to spot when retrieval is a bet you didn't need to make.

The idea

Here's the core of it. The search finds chunks that look like the question. It doesn't find the chunks you need to answer it.

Delivers late sounds close to deadlines and termination. The penalties clause talks about default, and in vector space it sits far away. So the model answers from a fifth of the contract, and it doesn't know what was left out.

When the whole document fits, let the model that read everything decide what's relevant. Not a search that only read the question.

How it breaks

Let's walk the pipeline.

It starts with material that's small and complete. One contract, every clause in it.

Then it gets split. And the split separates clauses that depend on each other. The rule is on page four, the exception is on page eleven.

The chunks become vectors, and go into the database.

For each question, the search picks a few. And it misses when the words don't match, even when the meaning does.

And the model answers from a fraction of the material, with full confidence. Nothing in the prompt tells it that clause fourteen exists.

So, the signs. The total content is small, tens of pages, and would fit whole in the context window.

There's chunking, embeddings and a vector database, for a handful of documents.

The wrong answers trace back to the search missing the right passage. Especially questions that combine distant parts of the document.

And the team spends more time tuning chunk sizes than solving the user's problem.

Why it happens

Nobody builds this out of carelessness. There are good reasons.

RAG became the default answer for AI over documents. It's the first architecture every tutorial shows.

Context windows grew. What needed retrieval a few years ago now fits whole, with room to spare.

It was designed for future scale. What if there are a thousand contracts? Maybe there will be. Today there's one.

And the cost of context was misjudged. Sending the whole document on every question sounds expensive, until you count prompt caching.

A challenge

Here's one for you. The assistant misses a question because the exception clause didn't come back from the search. A colleague proposes going from four chunks to eight per question. The whole contract is twenty four chunks. What do you say?

Eight chunks fixes this question. The next one may depend on the ninth. You're still betting that the search is right, for material that already fit whole.

And swapping the embedding model doesn't solve it either. A better embedding brings similar words closer. It doesn't connect a clause to an exception that looks nothing like the question. With twelve thousand tokens, send everything, and measure it on the eval.

The treatment

The treatment is the next lesson's refactoring. Replace RAG with direct context. Send the whole document, and use prompt caching so you don't pay full price for reading it on every question.

One detail decides whether the cache works. The document has to sit at the start of the prompt, byte for byte the same on every call.

The question comes after. If the question comes first, the start of the prompt changes every time, and the cache never hits.

A team learned that the hard way. They switched to the whole contract, and the bill went up far more than expected. The trace showed the question sitting in front of the contract. The fix wasn't going back to RAG. It was moving one line.

What you get back is worth it. The model sees the links between clauses. There's no chunking, no index, no retrieval to tune. And the citations get more precise, because the model sees each clause with its number and its page.

When to keep RAG

Now the other side. This isn't a rule against RAG.

If the material doesn't fit, or grows fast, you need retrieval. A knowledge base of four thousand articles that changes every week, with hybrid search, is not this smell. There, retrieval pays for itself.

Access control is the subtle one. An HR handbook with thirty pages fits easily. But if each employee may only see the section for their own role, sending everything shows the model sections that person must not see. Filtered retrieval, or a context built per role in code, is how you enforce the permission.

And if caching doesn't apply to your traffic, and the question volume is very high, work out the cost per question before you decide.

And the common mistake runs the other way too. A forty page manual, questions about isolated sections, retrieval almost always right, low cost. Small material alone isn't a reason to rebuild it. Run both versions on the eval. If direct context is more accurate at the same cost, switch. If they tie, keep the one that's simpler to maintain.

In production

What to watch. In the eval, tag every wrong answer by cause. Was the right passage in the prompt or not? If most errors are passages that never arrived, that's this smell, in numbers.

Keep a set of questions that need two distant sections, like the rule and its exception. Those are the ones chunking breaks first.

And after the switch, watch the cache hit rate in the trace. A rate near zero means something in the start of the prompt is changing. A date, a user name, or the question in the wrong place.

Wrap-up

So here's the idea to take with you. Retrieval is a bet that the search will find what the answer needs. When the document fits in the window, you don't have to place that bet. Give the model the whole thing, and let it decide what matters.

In the next lesson, we'll do the refactoring step by step, safely, with the cache in the right place: replace RAG with direct context. See you there.