Opening

You've met this story in the last two lessons. One line changed in the invoice extractor. Three test invoices looked fine. And in production, invoices with discounts got worse.

Today we do the refactoring that prevents it. And it has a strange first step. Before you change anything, you change nothing.

This refactoring treats three smells. The untested prompt, where changes ship with no test set.

The vibe check eval, where someone asks a few questions and eyeballs the answers.

And the model pinned prompt. A classifier nobody has dared to move to another model in eight months, because the prompt was tuned on this one. In this lesson, you'll build the safety net that makes all three go away.

The idea

The solution fits in one sentence. Before any tweak, build a set of real cases with expected answers, run the current version on them, and store the result as a baseline. From then on, every change is compared against that baseline.

The baseline is the piece that's usually missing. Without it, a lone number, like ninety-one percent accuracy, doesn't tell you whether the change helped or hurt. With it, every discussion becomes a comparison.

Before and after

Before, the change ships with an opinion. Someone changes the prompt, looks at three cases, and deploys.

After, the change ships with a number. Someone proposes a change to the prompt, the model, or a parameter.

The eval runs the new version on the versioned dataset, a few times.

The result is compared with the previous version, field by field, category by category.

With no regression, the change goes ahead.

With a regression, the change stops, and the cases that got worse come up for analysis.

Here's what that looks like on the pull request, as an illustrative report. Version six against version seven. The issuer field holds. The date holds. And the total drops six points, on the discounted invoices.

The run fails, across three runs. Friday's bug never reaches production.

Why it works

The problem with testing by hand isn't laziness. It's statistics. Three examples won't catch a six point drop in a category that shows up in fifteen percent of cases. A set of dozens of real cases, run a few times, will.

The baseline turns a number into a comparison. That's what ends the it got better, it got worse debate.

And the eval changes how the team behaves. When deleting a prompt paragraph costs a few minutes of measurement, people start cleaning the prompt, instead of only adding rules.

Safe mechanics

Step zero. Don't change anything yet. Freeze the current prompt. The baseline belongs to the version that's live, not the one you want to ship.

Gather real cases. Start with thirty to a hundred production inputs, anonymized, covering the main categories and the bugs you already know. Invoices with a discount, with service tax, with more than one line item.

Label the expected result. For extraction and classification, the right value. For open text, criteria you can verify, or a rubric.

Pick the simplest grading that works. Exact match, a rule in code, or a model as judge when there's no alternative. And if the judge is a model, measure how often it agrees with a person.

Run the current version three to five times, and store the mean and the spread, per category. That's your baseline.

Version the dataset and the script in the same repository as the prompt.

Now, finally, make the change, and run the same eval.

Compare per category, not only the average. Agree with the team on which drop blocks a deploy. And wire the eval into your pipeline, so every prompt change runs it automatically.

A challenge

Here's one for you. You ran the eval once on the new version and got ninety-four percent. The baseline is ninety-two, plus or minus three points, over five runs. Can you ship?

Not yet. A single run inside the baseline's noise band isn't an improvement. You may just be looking at variation the baseline already had. Run the new version three to five times, and compare the mean and the range per category with the baseline. Then decide.

And write the ship or block rule before you see the numbers. Afterwards, everyone finds a reason to like the result they wanted.

The traps

Now the traps. The first is the average that hides a rare queue. Overall accuracy rises one point. The chargeback queue drops twenty. If your pipeline looks only at the mean, the pull request passes. The fix is to fail the pull request if any agreed category drops past its threshold, and to print the cases that got worse. That turns the number into a fix.

A dataset that doesn't look like production. A team had forty cases, invented on a whiteboard, all easy. The pipeline always passed, and production failed on service invoices. Replace the whiteboard with real anonymized invoices, including the errors you've already seen.

Overfitting to the eval. Tweaking the prompt until it passes the fifty cases can make the rest of production worse. Hold out part of the dataset, and use it only for the final decision.

A judge nobody evaluated. If grading uses a model, check its agreement with a person from time to time.

And noise mistaken for improvement. We just saw that one.

Switching models

Now the payoff. The team wants to move the ticket classifier to a cheaper model. What needs to exist before you run the test?

A dataset of real tickets with the right queue, covering every queue, including the rare ones.

The current model's baseline. Accuracy per queue, averaged over a few runs.

A decision rule, written before the test. For example, no queue may drop more than two points, and the average may not drop.

And the cost per thousand tickets for both models, to weigh against the quality difference.

With that, the switch becomes an afternoon of work, and a decision backed by a number. Not a bet. And don't wait for the provider to retire your current model. That's the worst moment to start, switching under a deadline, with no baseline. Build the eval now, while the current model still runs.

Wrap-up

So here's the idea to take with you. The eval comes before the change, never after. Freeze what's live, measure it on real cases, a few times, and write down the rule. Then change, and compare per category. That's what turns fear of touching the prompt into an afternoon of measured work.

In the next lesson, we'll look at the machine that runs all of this for you, on every change: the eval harness. See you there.