Opening

The company's HR bot started with ten lines. A friendly tone, a couple of rules, and that was it.

Today, its system prompt looks like this. Forty paragraphs, about four thousand tokens. And there are three things worth noticing.

First, the remote work rule appears twice. Once it says three days a week.

And further down, it says two. With different values, and nobody deleted the old one.

Second, the prompt asks for brevity, at the very end of a four thousand token text.

And third, half the content isn't instruction at all. It's knowledge. The vacation policy, the reimbursement policy. In this lesson, you'll learn to recognize this smell, why it grows, what it costs, and how to shrink it without breaking anything.

The diagnosis

That last point is the key to the diagnosis.

An instruction tells the model how to act, in any conversation. Be brief. Never discuss salaries.

Knowledge is the subject of some conversations. The vacation policy only matters when someone asks about vacation. But when knowledge lives in the system prompt, it rides along on every single call. Including the ones that don't need it.

So how do you recognize it? The system prompt is past a few thousand tokens, and it only grows.

Every fixed bug became a new paragraph. Never the removal of an old one.

Some rules contradict each other, and nobody knows which one wins.

Whole blocks of knowledge are pasted in. Policies, catalogs, frequently asked questions.

And nobody on the team can explain what each paragraph does. Changing one line changes answers about topics that have nothing to do with it.

Why it happens

Nobody writes a bloated prompt on purpose. It grows out of reasonable decisions, one at a time. And it grows in a cycle.

A case shows up that the bot handles badly.

The quickest fix, at six in the evening, is adding one more instruction. Do not do this.

The prompt grows, and every new instruction competes with the old ones.

The model follows the existing rules a little less reliably. New bugs appear. And the cycle starts over.

Adding is faster than understanding.

There are no evals. Without a test set, removing a paragraph is scary, because nobody knows what will break.

The prompt is the only place. With no search index and no tools, every piece of information ends up there.

And several people edit it. Each one adds their rule, without reading everyone else's.

What it costs

Now the cost of ignoring it. Let's start with money, using some illustrative numbers. A four thousand token prompt, times twenty thousand messages a day, is eighty million input tokens a day.

At a hypothetical one dollar per million, that's about two thousand four hundred dollars a month, just for the prompt. Before the user has typed a single word.

Then time. Bigger input means slower answers, especially before the first word shows up.

Then quality. Instructions compete for the model's attention. Research has shown that models make worse use of what sits in the middle of a long context. An important rule in paragraph twenty-three of forty tends to be followed less consistently. And rules that contradict each other produce answers nobody can predict.

And team speed. Nobody wants to touch it. Every change becomes a session of manual testing, and hoping.

A challenge

Here's one for you. With that prompt, an employee asks how many remote days they get. Sometimes the bot says three. Sometimes it says two. What's the most likely cause?

The prompt has two contradicting rules, and the model picks one each time. It's not temperature. Even at zero, the contradiction would still be there. And it's not that the model misunderstood. The question is simple. The problem is in what the model receives.

The fix is keeping each rule in exactly one place. Not asking the model to pay more attention.

The treatment

There are three refactorings, depending on what the bloat is made of.

If the prompt handles several topics or tasks, extract subprompt. Each one gets its own call, often behind a router.

If the bloat is style and format rules, replace rules with examples. A few good examples teach more than twenty rules in capital letters.

And if the bloat is content, like policies, catalogs and FAQs, move it out of the prompt. That content goes to an index, and comes in through retrieval only when it's relevant.

Apply all three to the HR bot, and the system prompt shrinks to four lines. Answer from the policy excerpts. If it's not there, say you don't know and point to the HR channel. Never discuss salaries. And answer in at most three sentences.

Notice that be brief became a limit you can check. At most three sentences.

The policies become documents in an index, each with an effective date. The three day rule simply disappears, because only the current version is indexed. And a reimbursement question brings in only the reimbursement excerpt.

But before any of it, build an eval with real questions, and measure the current prompt. That's what gives you the courage to delete paragraphs.

The common mistake here is cutting by gut feeling, on a Saturday, with no eval. A paragraph that looked useless might be the one holding a rare case, like the rule for interns. Cut one block at a time, run the eval, and keep going only if the numbers hold.

When to ignore

And when can you live with it? A prototype. To validate an idea in a week, a big messy prompt can be the fastest path. Just don't take it to production that way.

Low volume, stable content. An internal tool used ten times a day, with a base that barely changes, can live with a big prompt. Turn on caching, and move on.

And a prompt that's long because it holds eight well chosen examples isn't bloated. If accuracy is high and nobody's touched it in months, it's healthy. The smell is growth without criteria, not size.

One last trap. Someone turns on prompt caching, the cost of the eight thousand token prompt drops a lot, and they say the problem is solved. Caching lowers the price of repeated tokens. But the model still reads the whole prompt, on every call, with the same contradictions, and the same competition for attention.

Wrap-up

So here's the idea to take with you. A system prompt should hold how to behave, not everything the bot might ever need to know. Each rule in one place, knowledge in an index, and an eval so you can delete with confidence. A prompt someone can read in two minutes is a prompt your team can change.

In the next lesson, we'll move part of what the prompt asks for out of the text, and into a contract your code can check: structured output. See you there.