Opening

The back office team has an invoice extractor that works well. One day, someone notices the prompt says: extract the total amount. And they change it to: extract the total amount, with two decimal places.

It looks harmless. The deploy goes out on a Friday.

On Monday, finance complains. Invoices with a discount are coming in at the gross amount. A one sentence change altered how the model read the field. And nobody could have known beforehand, because the only test was opening three files and looking at them.

With regular code, you'd have a unit test. With a model, the same input can produce different answers, and correct isn't always an exact match. The test built for that world is called an eval. In this first lesson of the evals and reliability trail, you'll learn what one is made of, how to grade it, and how to read its numbers without fooling yourself.

The idea

An eval is a test for model output. It has three parts.

A set of cases. Inputs, each with the expected result.

A way to run the system on all of them.

And a grading rule, that turns each output into a score.

How it works

The cases come from real data, anonymized, with the expected answer labeled.

The whole system runs, with the same prompt, model and tools as production.

Each case produces an output, stored together with the prompt version.

Grading compares the output with the expected result.

Each case gets a score. Pass or fail, or a value from zero to one.

And the report shows the overall rate, the rate per category, and the difference from the previous version.

Friday's three files failed at step one, and at step six. Three examples by eye aren't a set. And looking isn't a grading rule. On Monday, the team discovered a kind of invoice, one with a discount, that wasn't among the three.

Three ways to grade

There are three ways to grade. Exact match. Code compares field by field, or checks that the output fits a schema. Use it when there's a right answer, like a classification, an extraction, or a database query. On the invoice extractor, the total is an exact match against the labeled number.

A rule in code. Small functions in your code check properties of the output. Has a citation, stays under a hundred words, never names a competitor. Use it when the text varies, but there are objective criteria.

And a model as judge. Another model scores the output, with a written rubric. Use it for open-ended quality. Tone, helpfulness.

The rule is to start with the simplest one that does the job. Exact match is cheap, fast, and never wrong. A judge costs money, and it needs evaluating too. It inherits the model's variation, and it can agree with the mistake.

Here's a case where that matters. A tone eval for emails used only a model as judge. And the judge approved emails that promised a discount the store doesn't offer. A bigger judge wouldn't fix it. It can still find the promise helpful. Add rules in code first. Don't invent a price, don't name a competitor. Then use the judge on what remains.

A challenge

Here's one for you. The extractor gets forty-eight of fifty cases right. Both misses are discounted invoices, the only two of that kind in the set. The team celebrates ninety-six percent. What is that number hiding?

It's hiding the rate per category. On the kind of invoice finance fears most, the rate is zero for two. The overall number diluted exactly the case that breaks the accounting system.

And calling it noise doesn't help. It might be noise overall, and it's still a hundred percent failure on discounts. You need more of those cases, not a pat on the back. Fill that slice, fix it, and only then look at the overall number.

Where cases come from

There are three sizes of eval. A unit case. A bug seen in production becomes a case. It's the equivalent of a regression test.

An eval dataset. From fifty to a few hundred cases covering your product's categories. It runs on every change.

And a public benchmark. Well known sets that measure models in general. They help you pick a candidate model. They don't tell you whether your system works. If product wants to swap models because a public benchmark rose two points, and your internal eval dropped one point, your eval wins. You don't ship a benchmark. You ship your classifier.

And here's the beginner trap. Treating the dataset as a dead list. The best cases are born in production. The invoice finance sent back. The ticket the classifier routed to the wrong team. Each incident becomes one more line in the file. Six months later, that file is worth more than the prompt. You can rewrite a prompt in an afternoon. A good dataset takes months to gather.

Reading the numbers

Because output varies, a single run can mislead. For important cases, run three to five times, and look at the pass rate, not one round's result.

A drop from ninety-two to ninety percent on fifty cases is a single case. It may be noise.

And fifty typical cases won't cover a rare category. If discounted invoices are five percent of volume, and you sampled twenty files from the top of the folder, that kind may never get in. Stratify. Include some of every category the business fears, instead of only what shows up first.

Try it

On the page there's an eval that runs in a few seconds. Five customer messages, each labeled billing, order or other.

The model classifies each one, and code counts how many match the label exactly.

Now change the prompt to: answer with the category and a short justification. Run it again, and all five fail, even though the category at the front of the sentence is right.

That's not the model getting worse. Your grading is coupled to the old format. The eval and the prompt are the same contract. If the output gains a reason, grading reads the category field, not the sentence. That's the point. An eval turns I think it got better into a number, and shows you right away what a change broke.

In production

So what changes? No prompt, model or tool change ships without running the eval first.

Trying five questions by hand before a deploy isn't an eval. That's a smell called vibe check eval.

And the eval has to run on its own, in your pipeline, or with a single command. That pattern is called an eval harness.

One last case. Your pipeline runs eighty cases on every pull request. Someone also tried four more questions in a chat on Friday, and shipped. The pipeline passed. And finance still came back with a complaint. Where did it leak? In those four questions. If they were different from the eighty, they should have become cases before the merge. If they weren't, the chat added nothing but false confidence.

Wrap-up

So here's the idea to take with you. An eval is how you stop guessing. Real cases with the right answer, the real system running on them, and a grade that doesn't depend on anyone's mood. Read it per category, run it on every change, and feed it with every incident.

In the next lesson, we'll look at what happens when a team doesn't have one: the untested prompt. See you there.