Opening

Same month, same company, two complaints. The marketing team asked the model for ten campaign headlines, and got ten headlines that were almost identical.

Meanwhile, the data team was extracting the company tax ID from invoices. And on every run, the field came back in a different format.

Both teams asked the engineer for the same thing: tweak the temperature. One wanted more variation. The other wanted less.

To know whether temperature is even the right knob, you need to understand where variation comes from. That's this lesson. And you'll see why, for one of these two teams, the answer is a different tool altogether.

The idea

At every step, the model computes a probability for every possible next token. Sampling is the rule that picks one token from those probabilities.

Picture a weighted die. The most likely word has the biggest face. But the other faces are still there, and sometimes they come up.

How it works

Let's follow one step. The model looks at everything already in the sequence.

It produces a distribution. Say, Electronics at eighty percent, Audio at fifteen, and dozens of other words sharing the rest.

Then the sampling settings reshape that distribution, before the draw.

One token is drawn from the adjusted distribution.

And the chosen token joins the sequence. Every later choice depends on it.

That last step matters more than it looks. Since each token depends on the ones before it, a different early choice changes everything after it. Two answers can drift far apart, even when each step varies only a little.

That's what the data team saw. The tax ID came back with dots and a slash, then digits only, then with a period at the end. Three paths, a few tokens apart, and all three plausible.

The knobs

There are three classic knobs. Temperature sharpens or flattens the distribution. Low temperature raises the peak, so the leading token leads even more. High temperature flattens the chart, and second and third place get a real chance.

Top p doesn't stretch the chart. It cuts the tail, and draws only among the most likely tokens that add up to a given probability.

And top k keeps only a fixed number of the most likely tokens.

One piece of advice. Tuning more than one at once makes the effect impossible to read. If you use these knobs at all, pick one. Temperature alone is enough for an experiment.

Back to the category example. Near zero temperature, Electronics comes out almost every time. At high temperature, Audio shows up, and even Computing. But notice what none of those draws does. None of them turns a punctuated tax ID into fourteen clean digits. Format doesn't live on that knob.

Zero is not a contract

Low temperature reduces variation. It doesn't turn the model into a deterministic function.

Details of how the provider runs the model can still change the output.

A tiny change in the prompt changes everything after it.

And some providers offer a seed setting, to try to repeat a draw. Even when it exists, the docs usually call it best effort. So don't build a test that depends on two identical calls.

A challenge

Here's one for you. The tax ID extractor returns dots and a slash on one run, and digits only on the next. The data team wants temperature zero. What actually fixes the format?

A schema that asks for a string of fourteen digits, and validation in code before you store the value. At temperature zero, all three spellings are still plausible, so nothing in the draw forces fourteen digits.

And writing always use the same format, please, in capitals, helps a little and guarantees nothing. The model can still be wrong. Your code is what refuses to let the wrong value in. That's the job of structured output, not temperature.

Where models are heading

Here's something telling. The newest models from several providers are removing these knobs. On current Anthropic models, like Claude Opus and Sonnet five point five, the API rejects any temperature other than the default. Older models, like Haiku four point five, still accept it.

Control moved elsewhere: the output format, the reasoning effort, and the prompt itself. And that tells you something useful. Tuning sampling was never the right way to guarantee format or correctness. Providers are just making that explicit.

So if you migrate the extractor to one of those models, and the data team panics, the answer is calm. Nothing changes in the contract. The schema, the validation and the eval were always the control.

Try it

Now the marketing side. On the page there's an experiment, with a model that still accepts temperature. You generate five headlines at low temperature, and five at high temperature.

Compare the variety between the two blocks. Then try a different route.

Ask, in a single call, for ten headlines that differ from each other, in different tones, for different audiences. Variety you ask for in the prompt usually beats variety you draw at random.

And here's the trap on that side. High temperature spreads the headlines, and it also spreads the quality. One is sharp, the next is an empty slogan. Push it to the extreme, and you trade similar for useless. Ten options in one call let you pick with your own criterion, instead of hoping the draw hits the right tone.

In production

So what changes? For extraction, classification, and any output your code reads, use structured output plus validation. Don't lean on temperature.

For creativity, ask for variety explicitly, and generate several options in one call.

For important decisions, think about where variation hurts. A credit opinion at high temperature means two similar customers can get different answers, with no business reason. There, you want less variation, not more. And running the same task several times and comparing is a pattern of its own, called voting.

And variation is always there. If you run the same test case three times and get three passes, then two, then three, that's normal. The number that matters is the rate over many runs. That's why every system with a model needs evals, and not a test that runs once.

Wrap-up

So, back to the two teams. Marketing didn't need a hotter die. It needed to ask for variety. And the data team didn't need a colder one. It needed a schema and a check in code. Temperature changes how wide the draw is. It never guarantees what comes out.

In the next lesson, we'll look at the prompt itself, and treat it like what it really is: an interface, with a contract, versions and tests. See you there.