Few-Shot Hoarding

Signs and symptoms

A ticket classifier started with 4 examples. Every “it classified this wrong” complaint became a new example in the prompt. Today it has 31 (anonymized excerpt):

Example 1: "can't log in" -> access
Example 2: "forgot my password" -> access
Example 3: "password doesn't work" -> access
...
Example 17: "overdue boleto, can I still pay?" -> billing   # added Mar/25, customer X
Example 18: "boleto overdue and access got blocked" -> access   # added May/25, ?
...
Example 31: "app closes on its own on Android 9" -> bug

Examples 17 and 18 are nearly identical and teach different answers. Nobody remembers why 18 was added. Nine examples are variations of “I can’t get in”. The billing queue, the hardest one, has two. (A boleto is a Brazilian bank slip, a common support topic.)

Signs of Few-Shot Hoarding:

• The prompt carries dozens of fixed examples.
• Every fixed bug became an example, and none were ever removed.
• There are near-identical examples, sometimes with different answers.
• The distribution of examples does not follow difficulty: easy cases have many, hard cases have few.
• Nobody can explain why each example is there.
• Removing an example is scary, because there is no eval.

flowchart LR
  B(["1. Reported error"]):::io --> E["2. One more example in the prompt"]:::tool
  E --> P[("3. 31 fixed examples")]:::store
  P --> M(["4. Model follows the closest ones, not always the right ones"]):::llm
  M --> B

1. A ticket is misclassified.
2. The quick fix is pasting that case in as an example.
3. The set grows without curation, with redundancies and contradictions.
4. The model mimics the closest examples, and contradictory examples produce new errors.

Check +20 XP on first tryA ticket arrives: 'boleto expired and I can't get into the app'. The prompt has example 17 (billing) and 18 (access), nearly identical. What does the model tend to do?

Ask for clarification, because the examples disagree

Follow the business rule in the system prompt and ignore the examples

Copy the nearest example, which may be 17 or 18, and send the ticket to the wrong queue

Reasons

• An example is the easiest fix. Pasting the failed case fixes that case on the spot.
• It works in the short term. The fixed case passes, and the effect on the others goes unmeasured.
• The examples live in code. There is no place to review, version, and remove them.
• No owner. Whoever added example 18 has left the team.

The trap: treating the prompt as an incident notebook. Every complaint gets a line, and the notebook never loses a page. Without an eval, nobody notices that nine login examples do not help the boleto queue.

A concrete case: the ticket “boleto expired, app blocked” should go to billing if that is the new product rule. With example 18 still in the prompt, the nearest output remains “access”. You do not have a model bug. You have an old example beating the new rule because it sits closer to the input text.

Cost of ignoring

• Cost and latency. Dozens of examples on every call weigh on input.
• Quality. Contradictory examples confuse; the model may copy the wrong one.
• Bias. The example distribution pulls answers toward the categories with more examples.
• Maintenance. Changing a business rule means finding and fixing every example that contradicts it.

A back-of-the-envelope bias

The numbers below are illustrative. Suppose 80 tokens per example. Thirty-one examples are about 2,500 tokens on every classification.

Category	Examples in the prompt	Real difficulty (the team’s view)
access	9	low
bug	6	medium
billing / boleto	2	high
other	14	mixed

The classifier sees “can’t log in” all day. “Boleto overdue and access blocked” drifts toward access because that queue has more neighbors. You pay 2,500 tokens to over-teach the easy case and under-teach the hard one.

If the business rule changes (overdue boleto with a blocked app becomes billing, not access), the 31 examples become 31 lines to hunt through in the diff.

Treatment

Refactoring	Choose it when
Move Few-Shots to a Dataset	Whenever the examples go beyond a handful. They leave the prompt for a versioned, curated set and are picked by similarity for each request. Targets: Few-Shot Selector and Golden Dataset.

Practical order:

1. Build an eval from the incidents that produced the examples. Without it, you cannot tell what is safe to delete.
2. Deduplicate. Two near-identical examples with different labels are a data bug, not a feature.
3. Balance by difficulty, not by complaint volume. The noisy queue does not need nine copies.
4. Leave in the prompt only the format (one or two examples), or select 3 to 5 neighbors per request.

A new error does not go back into the prompt the same day. It goes into the eval. If the eval drops, then you discuss whether you need an example, a rule, or a new category.

Check +20 XP on first tryThe eval shows that removing 30 of the 40 examples in the prompt does not change accuracy. What is the next step?

Remove the 30 and keep the 10, the ones you measured

Keep all 40 to be safe, because one of them might save a case someday

Paste this week's errors into the prompt to make up for the ones you will drop

Payoff

• A fixed-size prompt with the 3 to 5 most relevant examples for each input.
• Examples with an owner, a reason, and a date, reviewable in a pull request.
• Contradictions found and resolved during curation.
• New errors become eval cases, not examples pasted in a hurry.

When to ignore

• A few stable examples. Five well-chosen examples that cover the cases and have not changed in months make a good prompt.
• A task with a rigid format. When the example teaches format and all examples are equivalent, one or two are enough.

Is it this smell?

Check +20 XP on first tryThe prompt has 6 examples, one per category, picked by the team a year ago. Accuracy is stable and every category is in the eval. Is this Few-Shot Hoarding?

Yes, fixed examples are always bad

No

Yes, because nobody reviewed the examples this year

Check +20 XP on first tryThe examples sit in a constant in the code. To change one, the operations team files a ticket with engineering and waits for the next deploy. Which smell fits best?

Few-Shot Hoarding

Hardcoded Few-Shots

Prompt Bloat

Check +20 XP on first tryThe classifier has 31 examples. Nine are login. Billing has two. The eval is strong on access and weak on boleto. What is the diagnosis?

The model does not understand finance and needs to be replaced

Add an IF boleto block to the system prompt

Few-Shot Hoarding with category bias: the prompt over-teaches the easy case

Check +20 XP on first tryA production ticket failed. The person on call wants to paste the text into the prompt today. What do you ask for first?

That they paste it, and tomorrow the team reviews whether the example should stay

That they boost the example with the word IMPORTANT in front

That the case enter the eval today, and only become an example after someone decides the label and the reason