Opening

Julia built a chat over her company's internal manuals. There are forty of them.

To be safe, her code pastes all forty manuals into every single question. It works. But every answer is slow, and the monthly bill came in ten times over the forecast.

And there's something worse. When the answer is in manual twenty-three, the bot sometimes says it found nothing.

Two concepts explain both problems: tokens, and the context window.

By the end of this lesson, you'll be able to do the math on any design before it reaches production. And you'll know why fitting in the window isn't the same as working.

The idea

The model doesn't read letters or words. It reads tokens, chunks of text cut by a tokenizer. A common English word is often one token. Long, accented or rare words become several.

Tokens matter for three reasons. Price, because you pay for every token going in and every token coming out. And the output usually costs more.

Time, because the model writes one token at a time. A long answer is a slow answer.

And the limit, because everything going in and out has to fit in one space, the context window.

One detail that surprises people. Not all text costs the same. A question in Portuguese often becomes more tokens than the same question in English. Code full of symbols, invoice files, and data with repeated field names swell even more. If your product speaks Portuguese and pastes those files, the estimate you did in your head is low.

How it works

The context window is the total number of tokens the model sees in one call. Let's look at what's inside it.

The system prompt goes into every call. A thousand extra tokens here are a thousand extra tokens on every single message.

Each tool you offer takes space too, with its name, its description and its schema.

The history grows every turn. The model has no memory, so you resend everything.

Documents and tool results are usually the biggest part.

The user's question is usually the smallest.

And the answer takes room in the window too, with its own separate limit.

Those two limits get mixed up all the time. The window is the cap for the whole call. The output limit, called max tokens, is only the cap for the answer.

Say the window is two hundred thousand tokens, and you already sent one hundred ninety-eight thousand as input. The answer gets about two thousand, at most, even if your output limit asks for eight thousand. Most APIs reject the call when the total doesn't fit. And some frameworks, to make it fit, quietly drop your oldest messages.

A real case

Back to Julia, with illustrative numbers. Say each manual is about four thousand tokens.

Pasting all forty sends about one hundred sixty thousand input tokens on every question. Her question is a tiny fraction of that. And she pays for all of it on every message, even when the answer lives in three paragraphs.

Now retrieve only three relevant passages, five hundred tokens each. That's about two thousand tokens, plus the system prompt. The question competes with far less noise, and the price drops with the size.

But that's only the bill. Remember manual twenty-three? Context windows got huge. That doesn't mean the model uses everything in them well.

There's a research paper from twenty twenty-three called Lost in the Middle. It showed that models tend to use what sits at the start and the end of the input better than what sits in the middle. Newer models improved on this. But the principle holds: every irrelevant token competes with the relevant ones for the model's attention.

That's why Julia's chat failed on manual twenty-three. The information was in the window. It was just drowned among thirty-nine other manuals, right in the middle.

A challenge

Here's one for you. Julia's team upgrades to a model with a much bigger window, two hundred thousand tokens instead of thirty-two thousand. They paste the forty manuals again. The bill stays high, and manual twenty-three still vanishes sometimes. What did the bigger window fix?

Neither problem. You still pay for every token you send. And the middle of the input is still at a disadvantage. A bigger window is slack, not a strategy. The fix is to choose passages, not to ask for more room.

The common mistake

The beginner trap is treating a big window as a reason not to choose. It fits becomes the only criterion.

The right question is different. What needs to be here for this question, and what's only taking up space?

Here's a version you'll meet in real life. On turn one, someone pastes a twenty page document into the chat. On turn fifteen, they ask for the store's address. And your code is still resending that document, on every turn, twelve turns after anyone needed it. History isn't a mandatory archive. Send what the question needs, and summarize or drop the rest.

Do the math

Now the habit that saves you from surprises. Count tokens before you send. The provider's API has an endpoint just for that, which generates no answer and uses the same tokenizer as the model.

Then do the monthly math. Input tokens, times messages per day, times thirty. Add the average output. Multiply by the model's price per million tokens. That number tends to be scary. And it's much better to be scared before production.

Let's do one together. You measured eight thousand input tokens per question, ten thousand questions a day. The team wants to just add the vacation policy, another six thousand tokens, on every call.

Before, that's eighty million input tokens a day.

After, one hundred forty million. Your daily input nearly doubles, and the policy rides along on questions that have nothing to do with vacation.

And don't estimate by dividing characters by four. That rule sometimes works for plain English. For Portuguese, code and structured data, it undercounts. Measure a few real examples with the provider's counter.

In production

So what changes in your designs? Sending less and better beats sending everything. That's what retrieval does, instead of pasting whole manuals.

The system prompt needs a diet. When it grows unchecked, it becomes a smell called prompt bloat.

Long conversations rot. Old history confuses the model and raises the cost. That one's called context rot.

And the fixed parts at the start of the input can be reused across calls with prompt caching, which cuts both cost and latency. Just don't use caching as an excuse to paste a whole file the question doesn't need.

Wrap-up

So here's the idea to keep. The window is a budget, not a storage room. Every token you put there costs money, costs time, and competes for attention. Token math is the first thing to do in any design.

In the next lesson, we'll open up that dice roll inside the model: sampling and temperature, and what they really control. See you there.