Move Few-Shots to a Dataset

Problem

The prompt’s examples live in code, grew without curation, and only change with a deploy. These are the Few-Shot Hoarding and Hardcoded Few-Shots smells.

The bank’s triage classifies tickets about Pix (Brazil’s instant-payment rail), boleto (a Brazilian payment slip), access, and cards. Someone pasted three examples into classifier.py in the first month. Every misclassified ticket earned one more. Today there are 31. The prompt sends all 31 on every call. Ops cannot add the case “the boleto expired and access was blocked” without opening an engineering pull request. And 12 of the 60 eval cases are copies of the examples: the 94% accuracy is measuring memorization.

The destination of this refactoring is the Few-Shot Selector, fed by a Golden Dataset that does not mix with the eval.

Solution

Take the examples out of the code and into a versioned set, with an owner, a reason, and a date for each. At request time, pick the examples most similar to the input instead of sending them all.

Before and after

-# classifier.py
-FEW_SHOTS = [
-    ("I can't access the dashboard", "access"),
-    ... [31 examples]
-]
-prompt = SYSTEM + render(FEW_SHOTS)
+# examples/triage.jsonl  (versioned, edited through pull requests, reviewed by ops)
+# {"id": "ex-041", "input": "invoice overdue and access got blocked", "label": "billing",
+#  "added_by": "ops/marina", "reason": "a block for non-payment is billing", "since": "2026-05"}
+
+EXAMPLES = ExampleSet.load("examples/triage.jsonl")      # embeddings computed at build time
+chosen = EXAMPLES.select(ticket_text, k=4, diversity=0.3)
+prompt = SYSTEM + render(chosen)
+trace.set("few_shot.ids", [e.id for e in chosen])

Before: every example, fixed in code.

flowchart LR
  C[("1. 31 examples in code")]:::store --> P(["2. Prompt with all of them"]):::llm
  IN(["Ticket"]):::io --> P

After: a curated set outside the code, and only the relevant ones on each call.

flowchart LR
  D[("1. Versioned, curated set")]:::store --> S["2. Pick the 4 most similar"]:::tool
  IN(["Ticket"]):::io --> S
  S --> P(["3. Prompt with the chosen ones"]):::llm
  P --> T[("4. Trace with the ids used")]:::store

1. Examples live in a file or table with history, editable by domain experts.
2. A similarity search picks a few examples for each input.
3. The prompt has a fixed size.
4. The trace records which examples were used, to track down bad ones.

Worked example

Suppose each example is about 150 tokens (ticket plus label). The numbers below are illustrative.

Design	Examples sent	Example tokens per call
31 fixed	31	about 4,650
4 chosen	4	about 600

At 10 thousand tickets a day, the gap is on the order of 40 million tokens. The set can grow to 200 Pix, boleto, and card cases without the prompt growing: the selector still sends 4.

The ticket “the boleto expired and access was blocked” in the before competed with 30 neighbours, including 8 pure-access ones. In “Lost in the Middle” (Liu et al., 2023), models pay less attention to what sits in the middle of a long context. The 31 examples spread the signal. With k=4 and a diversity criterion, the selector returns two billing examples and one access example, and the model sees the contrast that matters: a block for non-payment is billing, not access.

The span records few_shot.ids=["ex-041","ex-018","ex-007","ex-033"]. If triage is wrong, you know which examples were on the table.

Check +20 XP on first tryTriage sends 31 examples on every call. The team wants to reach 80 to cover Pix, boleto, and cards. With no selector, what happens to the prompt?

Nothing much: examples at the end are cheap and the model uses them all

The prompt grows with every new example, cost rises, and attention to the right example thins out

The eval will improve in proportion, so the growth pays for itself

Why it works

Separating examples from code separates two change cycles: code (engineering, deploys) and domain knowledge (operations, curation). Each moves at its own pace, with its own review. Ops adds ex-041 with a reason and an author; engineering only changes the selector when the picking logic changes.

Similarity selection solves the hoarding: the set can grow to hundreds of examples without the prompt growing. And the metadata (reason, author, date) makes curation possible: you can find contradictory examples, remove stale ones, and understand why each one exists.

The id on the trace closes the diagnosis. A bad example stops being “the classifier got it wrong” and becomes “ex-018 is pulling boleto into access”. You remove the example, not the model.

Safe mechanics

0. Measure. Run the eval with the current examples. Keep the eval dataset separate from the examples: they must never overlap. If 12 cases are identical, take them out of one of the two sides and rerun the baseline. The accuracy that drops is the real one.
1. Export the examples from code into a file or table, adding the reason and author where known. Stable ids from day one.
2. Do an initial curation: remove duplicates, resolve contradictions, balance the categories. Three Pix examples and twenty card examples bias the selector.
3. Implement selection by similarity, with a small k (3 to 5) and some diversity criterion, so you do not get four clones of the same class.
4. Define the editing flow: who can propose, who reviews, and the eval running before a change takes effect. A jsonl in the repo with a pull request is already a flow.
5. Record on the trace the ids of the examples used.
6. Measure again and compare with the fixed-examples version, on the eval with no overlap.

Prompt cache and position

The fixed instructions (role, categories, output format) sit in the prefix, where the provider’s cache hits. The chosen examples change per ticket: they go after the prefix, before the input. If you mix an example into the middle of the rules, the cache breaks and the rules also move around on every call.

Check +20 XP on first tryThe triage eval has 60 cases, and 12 are identical to examples in the prompt. The team wants to move the 40 examples into a dataset. What should happen before you compare?

Migrate first and compare with the current 94%, so you have a continuous line

Remove the overlap, rerun the baseline, and only then migrate and compare

Copy the 12 into both the dataset and the eval, so the selector has anchors

Risks

• Leakage into the eval. An example identical to an eval case inflates accuracy. Check for overlap automatically on every change.
• Poor selection. Embedding similarity can bring in examples that look alike but mean something different (“I can’t log in” vs. “access blocked over a boleto”). Test the selector on its own: given a ticket, which ids it returns, and is the class right?
• Uncontrolled edits. An editable set with no review and no eval becomes a direct path to production. One wrong jsonl line classifies thousands of tickets.
• Prompt caching. Examples that change per request fall out of the cached prefix. Put them after the fixed instructions.
• k too large. k=15 “to be safe” recreates the hoarding. If the eval does not rise from k=4 to k=8, stay at 4.

Challenge

The team wants to move 40 fixed examples into a dataset. The current eval has 60 cases, and 12 of them are identical to examples in the prompt. What should happen before the migration?

Show solution

Separate the sets. The 12 identical cases must leave the eval (or the examples), or the eval measures memorization. Then run the baseline again: accuracy will probably drop a little, and that is the real number. Only then migrate and compare.

Keep the 12 as candidate examples if they are good, or as eval cases if they are what you want to measure. Not both.

Check your understanding

Check +20 XP on first tryOps wants to add an example from a Pix ticket that failed triage yesterday. Engineering only deploys on Thursday. With the versioned dataset, who should edit, and when does the change take effect?

Ops proposes the example with a reason and an author; the change only takes effect after review and the eval

Ops edits the jsonl in production at once, because the point is not to wait for engineering

Engineering remains the one who edits, so the prompt stays under control

Check +20 XP on first tryThe selector returned four ‘access’ examples for the ticket ‘the boleto expired and access was blocked’. The classification came out ‘access’. What is the most useful reading?

The model is weak on this kind of sentence; swap the classifier

You need to raise k to 12, so some billing example gets in

The selector had no diversity and the set may lack the billing vs. access contrast; fix both

Check +20 XP on first tryTriage rules live in the system prompt and are cached. The 4 chosen examples are being concatenated at the start, before the rules, so ‘the model sees the cases first’. What breaks?

Nothing. Examples at the start improve few-shot

The prompt cache, because the prefix stops being stable

The selector, because it assumes examples come at the end

Check +20 XP on first tryAfter the migration, someone suggests k=31 so you ‘do not lose information’, since the dataset has 31 items. What is the right answer?

Accept it, because the versioned dataset already solved the smell

Refuse. k stays small and fixed; it is the dataset that grows

Accept it only on the eval, to measure the ceiling, and use k=4 in production