Golden Dataset

Intent

Keep a curated set of inputs with expected answers that grows with every real bug and serves as the reference for every change. It is for any AI system that needs a trustworthy eval over time.

Problem

Marcos built the ticket classifier’s first eval in an afternoon: he asked the model itself to generate 100 sample tickets with the right queue. The eval scored 96%. In production, the support team complained about misrouted tickets every day.

When he looked at real tickets, he understood. Customers write with typos, mix two topics, paste screenshots, reply to old emails. The 100 generated tickets were clean, short, and always about one thing. The eval measured a world that did not exist.

There was a second, subtler problem. The same model that generated the tickets also classified them. It wrote cases the way it understood them, so it got almost all of them right. The eval was a mirror.

Worse: when a real ticket landed in the wrong queue, someone fixed the prompt, but the case never made it into the eval. Three months later, the same error came back with another prompt change, and nobody noticed until support complained again.

The question this pattern answers: where do the cases for an eval that truly represents production come from, and how does it keep representing it over time?

Solution

Treat the dataset as a product asset, with an owner, versions, and rules for what gets in:

• Real origin. Cases come from production, anonymized. Synthetic data only enters to cover known gaps, and it stays labeled as synthetic.
• Reviewed labels. The expected answer is labeled by someone who knows the domain, and hard cases get a second reviewer.
• Coverage per category. The dataset has cases for every category, including rare ones, and hard cases on purpose.
• Growth by bug. Every production error becomes a case before it is fixed.
• Separation. Part of the dataset is for development; another part is held out for the final decision, to avoid overfitting.

Of the five rules, growth by bug pays off the most over time. A dataset that only gets cases on the day it was created ages along with the product. One that receives every production error becomes the team’s memory: everything that broke once stays watched forever.

The order “case before fix” matters too. You record the case, run the eval, and watch the case fail. Only then do you fix it. If the case does not fail before the fix, it does not reproduce the bug, and the test protects nothing.

Structure

flowchart LR
  P[("1. Production: traces and feedback")]:::store --> S["2. Sample and anonymize"]:::tool
  B(["1. Reported bug"]):::io --> S
  S --> L((("3. Label and review"))):::human
  L --> D[("4. Versioned dataset")]:::store
  D --> DEV["5a. Development split"]:::tool
  D --> HO["5b. Holdout split"]:::tool
  DEV --> E(["6. Eval"]):::llm
  HO --> E

1. New cases come from production samples and reported bugs.
2. Personal data is removed or masked before anything else.
3. A domain expert labels the expected answer; ambiguous cases get a second review.
4. The dataset is versioned like code, with a history of who added what and why.
5. One split is for iterating on the prompt; the other is used only for the deploy decision.
6. The Eval Harness runs the system against the dataset.

Pseudocode

schema GoldenCase {
    id: string, input: any, expected: any, category: string,
    source: "production" | "bug" | "synthetic", difficulty: "easy" | "hard",
    split: "dev" | "holdout", added_by: string, reason: string
}

function add_from_bug(report):
    raw = traces.find(report.trace_id)
    case = GoldenCase(input=redact_pii(raw.input), source="bug", difficulty="hard",
                      split=assign_split(), reason=report.summary)
    case.expected = human.label(case)                     # 3
    if disagreement(case): case.expected = human.review(case)
    dataset.append(case)                                  # 4
    dataset.commit("bug " + report.id)

function sample_production(n):
    for raw in traces.sample(n, stratify_by="category"):  # 1 and 2
        queue_for_labeling(redact_pii(raw.input))

function coverage_report():
    return dataset.count(by=["category", "difficulty", "source"])

Real-world scenarios

• Invoice and contract extraction. Every invoice layout that failed becomes a case, with the right value for each field. The dataset is organized by issuer and document type, because a new issuer often brings a new layout.
• Natural language to SQL. Real analyst questions with the correct query and the expected result on a fixed test database. The eval compares results, not query text, because two different queries can both be right.
• Automatic ticket triage. Real tickets with the correct queue, stratified so that rare queues have enough cases. This is Marcos’s case, and the worked example below.

Applicability

• You will run evals more than once.
• The system has categories with very different frequencies.
• Production bugs happen and get fixed in the prompt.
• More than one person needs to trust the eval’s result.
• You will switch models one day and want to know what changes before you switch.

When not to use

• A prototype with no real users. Start with 20 hand-written cases and replace them with real ones as soon as they exist.
• Tasks with no expected answer (creative text). Use written criteria and a rubric scored by people or by a judge model; the dataset becomes a set of inputs without an answer key.
• Examples for the prompt. The eval dataset must not be the same thing as the Few-Shot Selector examples; if it is, the eval measures memorization.

How to implement

1. Pull 100 to 200 production inputs, stratified by category.
2. Remove personal data with an automated routine and review a sample.
3. Label with domain experts, using written labeling instructions.
4. Mark difficulty and source on every case.
5. Hold out 20% to 30%, drawn at random within each category.
6. Version it in text files (JSONL) in the repository or in storage with history.
7. Create the rule: no bug is closed without becoming a case.
8. Review the dataset every quarter: wrong labels, new categories, cases that no longer make sense.

The labeling guide

Two labelers only agree if the rule is written down. The guide is a short document that answers the questions that will come up. An excerpt from the triage guide:

Queue for a ticket with two topics:
  use the queue of the topic that blocks the customer from using the product.
  "I paid and the order is still pending, and I want to change the size"
  -> billing (the unconfirmed payment blocks the order)

Ticket with only a screenshot, no text:
  label by the screenshot's content; if unreadable, mark "unclear" and keep it out of the eval.

Unsure between two queues:
  list both under "alternatives" and send it to second review.

Every time a review finds a new tie, the rule goes into the guide. The guide is versioned with the dataset, because labels and rules travel together.

How many cases are enough

The math is simple and scary. In a category with 20 cases, each error moves the score by 5 points. In one with 4 cases, by 25.

Suppose the “legal” queue is 2% of tickets. A random sample of 200 cases brings about 4 of them. One error and the queue shows 75%; no errors and it shows 100%. Neither number says much.

That is why the sample is stratified: you deliberately pull more cases from rare queues, until each has a few dozen. The side effect is that the dataset’s plain average no longer reflects production. Handle it like this:

• Report each category’s score separately.
• Compute the overall score weighted by each category’s real frequency in production.
• Before celebrating a two or three point gain, look at which cases changed. In 100 cases, two points are two tickets.

Worked example: from bug to case

On a Monday, support flags it: a customer wrote “hi, I paid the bill but order 4521 still shows pending and I want to change the size”. The ticket went to the orders queue. Billing only saw it two days later.

1. Find the trace. The on-call engineer clicks “send to dataset” on the ticket’s trace. The input lands in the labeling queue, already masked.
2. Label. The labeler applies the guide: two topics, the payment blocks the order, so billing. Since it is a hard case, a second person confirms.
3. Record and watch it fail. The case goes in as tk-0912, with source: "bug" and the reason. The eval runs on the current prompt:

golden/tickets v41  (212 cases: 158 dev, 54 holdout)   illustrative example
prompt triage@v17
  dev      91.1%   holdout  88.9%
  FAIL tk-0912  expected=billing  got=orders  reason: two topics

4. Fix. The prompt gets the blocking-topic rule, with an example that is not tk-0912.
5. Watch it pass and compare. The eval runs again. tk-0912 passes, and you check whether any case that used to pass now fails. Only then is the bug closed.

Step 5 is what protects the future. Six months from now, when someone rewrites the prompt for something else, tk-0912 is still there and speaks up if the error returns.

Common traps

• The prompt author labels alone. They tend to label the way the prompt already answers. Bring in someone from support or billing.
• Deleting the case that always fails. If a case fails on every version, review the label. If the label is right, the case is exactly what the dataset needs.
• Peeking at the holdout. Checking the holdout on every iteration and keeping the version that did best on it turns the holdout into development data.
• Copying cases into the prompt. The case becomes a few-shot example, the eval passes from memory, and the score rises without production improving.

Variations

• Directed synthetic data. A model generates variations of hard real cases (with typos, two topics, another language). Useful for covering gaps, always marked as synthetic.
• Live dataset from feedback. Answers with a thumbs-down enter a labeling queue.
• Dataset per policy version. When the business rule changes, the labels change with it; keep the policy version on each case.

Cost, latency, and failure modes

How to evaluate

• Inter-labeler agreement on a subset: if two people disagree a lot, the task or the instructions are poorly defined. The most common measure is Cohen’s kappa, which discounts the agreement you would get by chance.
• Correlation with production: does the eval score track complaints and corrections in production? If the eval goes up and complaints do not go down, the dataset does not represent production.
• Coverage: cases per category and difficulty, and the share of the quarter’s bugs that became cases.

{ "id": "tk-0912", "input": "hi, I paid the bill but order 4521 still shows pending and I want to change the size", "expected": "billing", "category": "billing", "source": "bug", "difficulty": "hard", "split": "holdout", "reason": "two topics; routed to orders in 2026-08" }

In production

• A button or command that sends a production trace straight to the labeling queue.
• A dashboard with dataset size per category and the date of each category’s latest case.
• An alert when a new category appears in production with no cases in the dataset.
• A labeling queue with an owner and a deadline. A queue with no owner becomes a graveyard of traces.

Security

The dataset holds real data. Mask personal data before storing it, restrict access, and record the legal basis for using it, in line with privacy laws such as GDPR or Brazil’s LGPD.

Masking only ID numbers and emails is not enough. Tickets carry names, addresses, phone numbers, card numbers and, sometimes, health data. Run a fuller detection pass and review a sample by hand on every batch.

Pros and cons

Code

Both tabs implement adding a case from a bug and the coverage report.

import hashlib
import json
import re
from collections import Counter
from pathlib import Path

DATASET = Path("evals/golden/tickets.jsonl")
SSN = re.compile(r"\b\d{3}-?\d{2}-?\d{4}\b")
EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")


def redact(text: str) -> str:  # 2. mask before storing
    return EMAIL.sub("<email>", SSN.sub("<ssn>", text))


def assign_split(case_id: str, holdout_share: float = 0.25) -> str:
    bucket = int(hashlib.sha256(case_id.encode()).hexdigest(), 16) % 100
    return "holdout" if bucket < holdout_share * 100 else "dev"  # stable per id


def add_from_bug(bug_id: str, text: str, expected: str, category: str, reason: str, author: str) -> dict:
    case_id = f"bug-{bug_id}"
    case = {
        "id": case_id, "input": redact(text), "expected": expected, "category": category,
        "source": "bug", "difficulty": "hard", "split": assign_split(case_id),
        "added_by": author, "reason": reason,
    }
    with DATASET.open("a", encoding="utf-8") as f:  # 4. versioned with git
        f.write(json.dumps(case, ensure_ascii=False) + "\n")
    return case


def coverage() -> Counter:
    with DATASET.open(encoding="utf-8") as f:
        cases = [json.loads(line) for line in f if line.strip()]
    return Counter((c["category"], c["difficulty"], c["split"]) for c in cases)

import { createHash } from "node:crypto";
import { appendFileSync, readFileSync } from "node:fs";

const DATASET = "evals/golden/tickets.jsonl";

type GoldenCase = {
  id: string; input: string; expected: string; category: string;
  source: "production" | "bug" | "synthetic"; difficulty: "easy" | "hard";
  split: "dev" | "holdout"; added_by: string; reason: string;
};

const redact = (t: string) =>
  t.replace(/[\w.+-]+@[\w-]+\.[\w.]+/g, "<email>").replace(/\b\d{3}-?\d{2}-?\d{4}\b/g, "<ssn>");

const split = (id: string): GoldenCase["split"] =>
  parseInt(createHash("sha256").update(id).digest("hex").slice(0, 8), 16) % 100 < 25 ? "holdout" : "dev";

export function addFromBug(bug: { id: string; text: string; expected: string; category: string; reason: string; author: string }) {
  const id = `bug-${bug.id}`;
  const c: GoldenCase = {
    id, input: redact(bug.text), expected: bug.expected, category: bug.category,
    source: "bug", difficulty: "hard", split: split(id), added_by: bug.author, reason: bug.reason,
  };
  appendFileSync(DATASET, JSON.stringify(c) + "\n");
  return c;
}

export function coverage(): Record<string, number> {
  const cases: GoldenCase[] = readFileSync(DATASET, "utf8").split("\n").filter((l) => l.trim()).map((l) => JSON.parse(l));
  return cases.reduce<Record<string, number>>((acc, c) => {
    const key = `${c.category}/${c.difficulty}/${c.split}`;
    acc[key] = (acc[key] ?? 0) + 1;
    return acc;
  }, {});
}

Check your understanding