Evaluator-Optimizer

Intent

One model generates, another grades against clear criteria, and the loop repeats until it passes or hits a limit. It is for tasks with a well-defined quality criterion where a first version is rarely enough.

Problem

Gustavo maintains a unit test generator. The model receives a function and writes tests. Some tests do not compile, some compile but test nothing useful, and some test the wrong behavior.

Asking for “good tests” in the prompt changed nothing. Adding a list of rules helped a little. What helped most was running the tests and showing the model the error: with the compiler message in front of it, it fixed things almost every time on the second try.

The same holds for other tasks: a SQL query that does not return the expected rows, a text that breaks the character limit, a post that does not follow the brand guide. The first version is reasonable; the second, with specific criticism, is good.

The question this pattern answers: how do you turn “try again” into a controlled loop, with a clear criterion and a guaranteed end?

Solution

Separate two roles:

• Generator: produces a version of the output. In later rounds, it also receives the previous evaluation.
• Evaluator: compares the output with written criteria and returns a score and a list of specific problems. It can be code (compile, run tests, validate a schema), a model with a rubric, or both.

The loop ends when the evaluation passes, when the maximum number of rounds runs out, or when the score stops improving. The evaluator must be more reliable than the generator on the criterion it judges; otherwise, the loop just spins.

Anthropic’s “Building effective agents” post (December 2024) describes this pattern as a generator and an evaluator in a cycle, and warns: it pays off when there are clear criteria. Without that, the loop becomes the Redundant Self-Critique smell.

Structure

flowchart TD
  IN(["1. Task and criteria"]):::io --> G(["2. Generator"]):::llm
  G --> E{"3. Evaluator"}
  E -->|"passed"| OUT(["5. Approved output"]):::io
  E -->|"failed, with problems"| B{"4. Rounds left?"}
  B -->|yes| G
  B -->|no| BEST(["6. Best version with the report"]):::io

1. The task arrives with written criteria, ideally verifiable ones.
2. The generator produces a version; from the second round on, with the previous evaluation.
3. The evaluator checks the criteria: code first, a model with a rubric when needed.
4. Code decides whether to try again, with a round cap and stagnation detection.
5. If it passed, the output goes out.
6. If rounds ran out, the best version so far goes out, marked with what still fails.

Worked example

Gustavo asks for tests for apply_discount(price, pct). The generator’s first version imports a module that does not exist. The evaluator in code is pytest:

ImportError: cannot import name 'moneyfmt' from 'utils'

That message goes back to the generator. The second version compiles, the tests pass, and every assert is assert True. pytest is green. The criterion “compiles and passes” is loose.

Then the second criterion, still in code: mutation. Replacing pct with 0 in the original code should break at least one test. None break. The evaluator returns “kills 0% of mutants”. The third version tests the actual discount.

Notice the order. Compilation first (cheap, infallible on its own criterion). Tests next. Mutation last. A model with a rubric, if you have one, only sees what already passed those checks. Asking the same model “are the tests good?” on the first version would have approved assert True with a glowing paragraph.

Pseudocode

schema Evaluation { passed: bool, score: float, problems: list[string] }

function generate_with_feedback(task, criteria, max_rounds=3):
    best, history = null, []
    for round in range(max_rounds):
        draft = llm(GENERATOR_PROMPT, task, history)                 # 2
        # 3. checks in code before the model
        code_problems = run_checks(draft)                            # compile, tests, schema
        if code_problems:
            ev = Evaluation(passed=false, score=0, problems=code_problems)
        else:
            ev = llm(EVALUATOR_PROMPT, criteria, draft) -> Evaluation
        trace.span("round", round, ev)

        if best is null or ev.score > best.score:
            best = { draft, score: ev.score, problems: ev.problems }
        if ev.passed:
            return draft                                              # 5
        if stagnated(history, ev):
            break                                                     # 4
        history.append({ draft, problems: ev.problems })
    return best                                                       # 6

Real-world scenarios

• Test generator. The evaluator is the compiler and the test runner. The loop ends when the tests compile, pass, and kill mutants of the original code.
• Natural language to SQL. The evaluator runs the query on a test database and checks for syntax errors, missing columns, and empty results.
• Content creation pipeline. An evaluator with the brand guide’s rubric returns specific problems; the writer fixes them until it passes.

Applicability

• There is a written quality criterion, ideally verifiable in code.
• The first version usually fails in ways specific criticism can fix.
• The cost of extra rounds is lower than the cost of a bad output.
• The extra latency is acceptable.

When not to use

• The first version already passes in most cases. The evaluator becomes cost with no gain; use only a check in code.
• There is no clear criterion. A vague evaluator (“is it good?”) tends to approve everything or request random changes. See Redundant Self-Critique.
• The evaluator is the same model, with the same context, and no new information. That usually just repeats the first opinion. See Reflection for when criticism comes from an external signal.
• The task needs an immediate answer.

Common mistake: keeping the last version

Round 2 clears the ImportError. Round 3, trying to kill mutants, breaks compilation again. If you return the last one, the user gets code that does not even import. The best version was round 2, with the report of what still fails.

The cap also needs a visible stop reason: passed, stagnated (same error twice), or rounds exhausted. Without that, the trace does not say why the task exited.

How to implement

1. Write the criteria as a verifiable list. Separate the ones code can check.
2. Implement the code checks first: compilation, tests, schema, rules.
3. For whatever remains, write a rubric for the evaluator, with examples of high and low scores.
4. Have the evaluator return specific, actionable problems along with the score.
5. Pass the generator the previous version and its problems, not the whole round history.
6. Add a round cap and stagnation detection.
7. Keep the best version, not the last one.
8. Evaluate the evaluator: agreement with a person on the cases where it uses a model.

On the second round, the generator does not need draft 1 and draft 2 and the full rubric again. It needs the current draft and the short problem list. A long history pushes the model into defending itself (“I already tested that”) instead of fixing.

If the evaluator is code (pytest, compiler, schema), agreement is trivial: it does not “opine”. If it is a model, build a dataset where a person already marked pass/fail and measure how often the judge is right. An evaluator that disagrees with the person on 30% of cases will train the generator in the wrong direction.

Variations

• Code-only evaluator. Compiler, tests, validators. The cheapest and most reliable.
• Rubric evaluator. A model with written criteria. See LLM-as-Judge.
• Hybrid evaluator. Code first; the model only judges what passed the objective checks.
• Several candidates per round. The generator produces N versions and the evaluator picks the best. More expensive, fewer rounds.

The hybrid is the variant people regret least. Code first: if it did not compile, the model judge does not even run. That cuts cost and stops an eloquent paragraph from approving a test that does not import.

Several candidates per round make sense when the first version varies a lot (two tones of a post, two shapes of SQL) and a code evaluator can rank them cheaply. Generating 5 and grading all 5 with an expensive rubric usually loses to 2 hybrid rounds.

A code-only evaluator does not cover brand tone. That is where the hybrid wins again: pytest (or the character cap) first, rubric second, on the ones that passed. Reversing the order is the classic waste: the style judge approves a test that does not even import, and only then the compiler says no.

Also store why the loop stopped. “Passed”, “stagnated”, and “cap” are three different outcomes on the dashboard. Without the reason, a cost spike looks like “the generator got worse”, when the evaluator actually got nitpicky.

Cost, latency, and failure modes

How to evaluate

• First-round pass rate and pass rate after the loop. The difference is the pattern’s gain.
• Average rounds and cost per task.
• Evaluator quality: agreement with a person, and false positive and false negative rates.

A false positive here is the evaluator rejecting text a person would accept. A false negative is approving what the person would reject. In content, the false negative is the danger: the loop ships a post that breaks the brand and nobody sees it. In tests, a false positive costs rounds; a false negative ships assert True.

Measure both rates on a small hand-labeled set before you turn the loop on in production. If the model judge is wrong on 1 in 4, it is not more reliable than the generator.

{ "function": "fixtures/apply_discount.py", "criteria": ["compiles", "all tests pass", "kills at least 80% of mutants"], "max_rounds": 3 }

In production

• One span per round with the score and problems.
• A dashboard of the round distribution per task. Many tasks at the cap point to a weak generator or a strict evaluator.
• An alert when average cost per task rises, a sign the loop is running longer.
• A sample of outputs approved by the model evaluator, for human review.

Security

The loop sends the previous output back to the generator. If the task includes outside text (a PDF, an email), that text can try to instruct the generator more strongly on the second round. Treat the draft as data, not as orders, and do not let the generator call side-effect tools mid-loop. Effects (merge, email, payment) sit after approval, with Validate Before Execute.

A lenient evaluator that approves SQL generated from a user question is the usual risk: model output going to an engine. The evaluator needs to run the query on a test database, not only check that it “looks like SQL”.

Pros and cons

Code

The tabs generate, evaluate with code, and repeat with the errors until it passes or hits the cap. Python and TypeScript write the rounds as a loop. LangGraph splits generator and evaluator into nodes: looping or stopping is an edge, not a request from the model.

import subprocess
from pathlib import Path

import anthropic

client = anthropic.Anthropic()
MAX_ROUNDS = 3


def evaluate(test_code: str, workdir: Path) -> tuple[bool, str]:  # 3. evaluator in code
    path = workdir / "test_generated.py"
    path.write_text(test_code, encoding="utf-8")
    out = subprocess.run(["pytest", str(path), "-q", "-x"], capture_output=True, text=True, timeout=120)
    return out.returncode == 0, (out.stdout + out.stderr)[-4000:]


def generate_tests(source: str, workdir: Path) -> dict:
    feedback, best = "", None
    for round_ in range(MAX_ROUNDS):
        prompt = f"Write pytest tests for this function.\n\n{source}"
        if feedback:
            prompt += f"\n\nYour previous version failed. pytest output:\n{feedback}\nFix it."
        reply = client.messages.create(  # 2. generator
            model="claude-sonnet-5-5", max_tokens=8000,
            system="Answer with Python code only, no markdown.",
            messages=[{"role": "user", "content": prompt}],
        )
        code = reply.content[-1].text
        passed, output = evaluate(code, workdir)
        best = {"code": code, "passed": passed, "report": output, "rounds": round_ + 1}
        if passed:
            return best  # 5
        if feedback and output == feedback:
            break  # 4. stagnated: same error twice
        feedback = output
    return best  # 6. best version with the report of what fails

import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import { z } from "zod";

const client = new Anthropic();

const Evaluation = z.object({
  passed: z.boolean(),
  score: z.number().min(0).max(10),
  problems: z.array(z.string()),
});

const text = (m: Anthropic.Message) => m.content.find((b) => b.type === "text")?.text ?? "";

export async function writePost(brief: string, rubric: string, maxRounds = 3) {
  let best = { draft: "", score: -1, problems: [] as string[] };
  let feedback: string[] = [];
  for (let round = 0; round < maxRounds; round++) {
    const draft = text(await client.messages.create({ // 2
      model: "claude-sonnet-5-5", max_tokens: 4000,
      messages: [{ role: "user", content: `Write the post.\n${brief}\n${feedback.length ? "Fix:\n- " + feedback.join("\n- ") : ""}` }],
    }));

    const codeProblems = draft.length > 2200 ? ["Over 2,200 characters."] : []; // 3. code first
    const { parsed_output: ev } = codeProblems.length
      ? { parsed_output: { passed: false, score: 0, problems: codeProblems } }
      : await client.messages.parse({
          model: "claude-opus-5-5", max_tokens: 2000,
          system: `Grade the text against this rubric and list specific problems.\n${rubric}`,
          messages: [{ role: "user", content: draft }],
          output_config: { format: zodOutputFormat(Evaluation) },
        });
    if (!ev) break;
    if (ev.score > best.score) best = { draft, score: ev.score, problems: ev.problems };
    if (ev.passed) return { ...best, passed: true }; // 5
    feedback = ev.problems;
  }
  return { ...best, passed: false }; // 6
}

from typing import Literal

from langchain.chat_models import init_chat_model
from langgraph.graph import END, START, StateGraph
from typing_extensions import TypedDict

class State(TypedDict):
    task: str
    code: str
    errors: str
    best: str
    rounds: int

writer = init_chat_model("anthropic:claude-opus-5-5", temperature=0.2)

def generate(state: State) -> dict:
    # 2. generator, with the previous review from the second round on
    reply = writer.invoke([("human", f"{state['task']}\n\nErrors:\n{state['errors']}\n\nVersion:\n{state['code']}")])
    return {"code": reply.text, "rounds": state["rounds"] + 1}

def evaluate(state: State) -> dict:
    ok, log = pytest_run(state["code"])  # 3. code first
    return {"errors": "" if ok else log, "best": state["code"] if ok or not state["best"] else state["best"]}

def route(state: State) -> Literal["generate", "done"]:
    # 4. cap and stagnation in code
    if not state["errors"] or state["rounds"] >= 4:
        return "done"
    return "generate"

graph = StateGraph(State)
graph.add_node("generate", generate)
graph.add_node("evaluate", evaluate)
graph.add_edge(START, "generate")
graph.add_edge("generate", "evaluate")
graph.add_conditional_edges("evaluate", route, {"generate": "generate", "done": END})
app = graph.compile()

Framework support

Check your understanding