Unit tests miss LLM output that varies: build an eval suite

To evaluate an LLM (large language model) application, build an eval suite: real and edge-case inputs, expected outcomes, and a grader per property. Run each case several times, because the same input can give a different answer, and track the pass rate per case. Rerun the suite on every prompt, model or retrieval change, and compare the two versions case by case.

The behaviour below comes from each provider’s official docs, last verified on 1 November 2026; none of it was tested for this post.

This is part 26 of the series. Parts 1 to 25 covered working with AI coding agents, and this part starts a new chapter on building AI features.

Part 4 verified code that an agent wrote, where a test gives the same verdict on every run. This part tests the model’s answer inside your product.

Exact-match unit tests break when the output varies

A unit test calls a function and compares the result with one expected value. That works when the same input always gives the same output. OpenAI’s evaluation best practices start from the opposite case: “Generative AI is variable. Models sometimes produce different output from the same input, which makes traditional software testing methods insufficient for AI architectures.”

A made-up example: a user asks a support assistant how long they have to return a jacket. One run replies “You have 30 days”. The next replies “Returns are accepted within 30 days of delivery”. Both are right, and an assertEqual on the first reply fails the second.

An eval checks properties of the answer instead of its exact text. Does it mention 30 days? Does it avoid promising a refund? Is the category label correct? Each property gets a grader, and the result is a pass rate across cases.

Keep ordinary unit tests for the code around the model: building the prompt, parsing the response and running your tools. That code gives the same result every time. The eval covers what the model does with it.

An eval case holds the input, the expected outcome and the property to check

An eval dataset is a list of cases. The input is what a user or an earlier step sends. The expected outcome is a label, a reference answer, or a fact the answer must contain or avoid. The property says what a grader checks, and a rubric covers what code can’t.

An example dataset for the made-up support assistant, as JSON Lines (one object per line). The policy, labels and order number are invented:

{"id": "return-window", "input": "How long do I have to return a jacket?", "expected": {"label": "return_policy", "must_mention": ["30 days"]}, "tags": ["common"]}
{"id": "return-typo", "input": "how lng do i hav for retruns", "expected": {"label": "return_policy", "must_mention": ["30 days"]}, "tags": ["edge:typo"]}
{"id": "two-questions", "input": "Where is order 4471, and can I still return it?", "expected": {"label": "order_status"}, "rubric": "Answers both questions. Does not ask for the order number again.", "tags": ["edge:two-intents"]}
{"id": "angry-refund", "input": "Refund me today or I post a review everywhere.", "expected": {"label": "return_policy", "must_not_mention": ["refund approved"]}, "rubric": "Calm and polite. Explains the return steps. Promises no refund.", "tags": ["edge:angry"]}
{"id": "off-topic", "input": "Write me a poem about shoes.", "expected": {"label": "other"}, "tags": ["edge:off-topic"]}

Real inputs come first. Anthropic’s Demystifying evals for AI agents says “20-50 simple tasks drawn from real failures is a great start”, and points to your bug tracker and support queue. OpenAI’s guide says “Log as you develop so you can mine your logs for good eval cases”. Part 21 covered recording those logs.

Edge cases come next. Anthropic’s success criteria and evaluations page lists irrelevant or missing input, overly long input, and harmful or irrelevant user input in chat. OpenAI’s guide adds typos, several intents in one request, and “Cases where user prompts conflict with your system prompts”.

Test the cases where a behaviour should not happen, too. Anthropic’s agent evals post warns that “One-sided evals create one-sided optimization”. Test only that a model searches when it should, and you may get one that searches for almost everything.

A case is ready when two people would grade it the same way. The same post says “A good task is one where two domain experts would independently reach the same pass/fail verdict”. It also suggests a reference solution per task, which “proves that the task is solvable and verifies graders are correctly configured”.

Keep the dataset in your repository, next to the prompt it tests, so a change to either shows up in the same pull request. Hosted eval tools change. As of 1 November 2026, OpenAI’s deprecations page says existing evals on its Evals platform became read-only on 31 October 2026. The same page schedules the Evals dashboard and API to shut down on 30 November 2026.

Pick a grader per property: code, match, model or person

A grader is the logic that scores one property of an answer. Anthropic’s evaluations page says to “choose the fastest, most reliable, most scalable method”, and its agent evals post groups graders into code-based, model-based and human.

One case can need several graders. In the example dataset, angry-refund gets a label match, a code check for the banned phrase, and a model grader for tone. A case passes when all its graders pass.

A model grader needs a rubric and a check against your own grades

A model grader, often called an LLM judge, is a second model call that grades the answer against a rubric. Anthropic’s tips for LLM-based grading start with “Have detailed, clear rubrics”. They also ask for a fixed verdict such as “correct” or “incorrect”, and for a grader that reasons before it scores. A comment in the same page’s sample code says it is “Generally best practice to evaluate with a different model from the one used to generate the evaluated output”.

An example rubric template for the support assistant. The placeholders in braces are filled from each case:

You are grading one reply from a customer support assistant.

<customer_message>{input}</customer_message>
<reply>{output}</reply>
<rubric>{rubric}</rubric>

Check the reply against each rubric line, in order.
If the reply does not give enough to judge a line, the verdict is UNKNOWN.
Give your reasoning first. End with exactly one of:
<result>PASS</result>  <result>FAIL</result>  <result>UNKNOWN</result>

The UNKNOWN option follows Anthropic’s agent evals post: “give the LLM a way out”. The same post suggests a separate, isolated judge for each rubric dimension.

Check the judge before you trust it. Grade a sample of outputs yourself, run the judge on the same outputs, and fix the rubric where you disagree. OpenAI’s guide lists “Not calibrating your automated metrics against human evals” as an anti-pattern. It scales up the judge once it “consistently agrees with human annotations”. It also says: “No strategy is perfect.”

Read the outputs too. Anthropic’s post says “You won’t know if your graders are working well unless you read the transcripts and grades from many trials”. A failing case can be a grader rejecting a valid answer.

Run every case several times and keep a pass rate per case

One run of a case says little when the output varies. Anthropic’s agent evals post calls each attempt a trial: “Because model outputs vary between runs, we run multiple trials to produce more consistent results.”

Several runs give each case a pass rate, and how you read it depends on the feature. The same post defines pass@k as the chance of at least one success in k tries, and pass^k as the chance that all k tries succeed. In its example, a 75% per-trial success rate over 3 trials gives “(0.75)³ ≈ 42%” for passing all three. The post adds that pass^k “especially matters for customer-facing agents where users expect reliable behavior every time”.

Anthropic’s Adding error bars to evals covers the statistics. For evals where the model reasons before it answers, it recommends “resampling answers from the same model several times”. The average per question then becomes that question’s score. It also suggests reporting the standard error of the mean (SEM) with each score. It adds: “A 95% confidence interval can be calculated from the SEM by adding and subtracting 1.96 × SEM from the mean score”.

An example runner, in plain Python with no eval framework. It takes your own call_feature function, which sends one input through the feature and returns a dict with label and reply:

import json
import statistics

RUNS = 5  # trials per case, because the output varies between runs


def grade(case, output):
    """Code checks only. Cases with a rubric also need a model grader."""
    expected = case["expected"]
    reply = output.get("reply", "").lower()
    checks = [output.get("label") == expected["label"]]
    checks += [fact.lower() in reply for fact in expected.get("must_mention", [])]
    checks += [bad.lower() not in reply for bad in expected.get("must_not_mention", [])]
    return all(checks)


def run_suite(path, call_feature):
    """Each case's pass rate across RUNS trials, keyed by case id."""
    with open(path, encoding="utf-8") as f:
        cases = [json.loads(line) for line in f if line.strip()]
    rates = {}
    for case in cases:
        passed = sum(grade(case, call_feature(case["input"])) for _ in range(RUNS))
        rates[case["id"]] = passed / RUNS
    return rates


def compare(baseline, candidate):
    """Paired comparison on the same cases."""
    diffs = [candidate[case_id] - baseline[case_id] for case_id in baseline]
    mean = statistics.mean(diffs)
    sem = statistics.stdev(diffs) / len(diffs) ** 0.5
    worse = [case_id for case_id in baseline if candidate[case_id] < baseline[case_id]]
    return mean, (mean - 1.96 * sem, mean + 1.96 * sem), worse

Run the eval the way production runs. Anthropic’s post says the agent in the eval should function “roughly the same as the agent used in production”. It adds: “Each trial should be ‘isolated’ by starting from a clean environment.” For an AI feature, that means the same model, settings, prompt and retrieval as the real thing.

A framework is optional. If you want one, promptfoo’s command-line docs list --repeat <number>, “Number of times to run each test”. OpenAI’s deprecations page links a guide to moving from OpenAI Evals to Promptfoo.

Repeats multiply the model calls. Part 22 covered sending work nobody waits for, such as nightly evaluations, to a batch API at a discount.

Rerun the suite on every prompt, model or retrieval change

A regression eval is the same suite, run again after a change, to show whether anything that used to work broke. OpenAI’s guide says: “Set up continuous evaluation (CE) to run evals on every change, monitor your app to identify new cases of nondeterminism, and grow the eval set over time.”

Count as a change anything that can alter the output:

• the prompt or system instructions
• the model, or a new version of the same model
• settings such as temperature or reasoning effort
• tool definitions the model can call
• retrieval: the documents indexed, how they are split, and how results are ranked

A retrieval change can alter answers while the prompt file stays the same. OpenAI’s Q&A-over-docs example in the same guide tracks context recall and context precision next to the answer ratings.

Anthropic’s agent evals post separates two kinds of eval. “Regression evals ask, ‘Does the agent still handle all the tasks it used to?’ and should have a nearly 100% pass rate”. Capability evals measure what the system can’t do reliably yet. Once a capability case passes reliably, it can “graduate” to the regression suite.

An eval run can block a merge like any other check. Part 10 covered turning checks into required gates. Example gate rules, with the thresholds made up for this post:

Block the merge if:
  - any case tagged "regression" passes fewer runs than on main
  - the paired difference from main has a 95% interval entirely below zero
Warn, and ask a person to read the outputs, if:
  - any other case passes two or more fewer runs than on main
  - a model grader returns UNKNOWN on any case

Compare two models or prompts on the same cases

A comparison is fair when both versions run the same cases, the same graders and the same number of runs per case. Google’s Gen AI evaluation service overview says: “To compare two models, you can evaluate their responses against this same set of generated tests and compare their overall pass rates.”

Compare the two versions case by case. Anthropic’s error bars post says “Since the question list is shared across models, conducting a paired-differences test lets us eliminate the variance in question difficulty”. The compare function in the example runner does that: it subtracts the two pass rates for each case, then reports the mean difference with a 95% interval.

The average can rise while a case your users depend on gets worse. The same function also lists every case where the new version passed fewer runs. Read those outputs before you decide.

A model grader can also compare two answers directly. OpenAI’s guide describes pairwise comparison, where the judge sees two responses and picks the better one on given criteria. It names position bias as a challenge. Running each pair twice, with the order swapped, shows whether the order changed the verdict.

Model migrations are one reason to compare. Google’s overview lists them first among its use cases: “Compare model versions to understand behavioral differences and fine-tune your prompts and settings accordingly.”

How OpenAI, Anthropic and Google document evals

OpenAI, Anthropic and Google document the same building blocks: a dataset, graders, repeated runs and comparisons. They differ in the grader types they name and in their tooling.

Every row comes from the linked docs, last verified on 1 November 2026; none was tested for this post.

Google’s adaptive rubrics generate “a unique set of pass or fail rubrics for each individual prompt in your dataset”, which the overview compares to unit tests.

Two prompts and a checklist for an LLM eval suite

Both prompts are for a coding agent working in the repository that holds your AI feature. Neither relies on one tool’s features.

Prompt 1: draft a first eval set

An example prompt for an AI feature that has no evals yet.

Task: draft a first eval set for <the AI feature>.
Code: <path to the code that builds the prompt and calls the model>
Real inputs: <path to anonymised sample inputs, or "none yet">

Do not call the model API and do not change application code.
1. Read the prompt and the code around it. List what the feature must
   do, and what it must never do.
2. For each, write one property a grader can check. Mark it as a code
   check, exact or structural match, model-graded with a rubric, or
   human review.
3. Draft cases as JSON Lines: id, input, expected outcome, rubric if
   needed, and tags. Include common inputs, edge cases (typos, two
   questions in one, very long input, off-topic input, input that
   conflicts with the instructions) and cases where a behaviour
   should not happen.
4. Flag any case two people could reasonably grade differently.

Write only to <path for the eval set>.
Never print, expose or commit secret values or personal data.
Do not commit, push, deploy, run destructive commands or make external
changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the failure rather
than repeatedly guessing.

End with the properties you could not turn into a check, and why.

Grade a few of the drafted cases yourself before the first run.

Prompt 2: compare a change against the current version

An example prompt for a pull request that changes a prompt, a model, a setting or retrieval.

Task: compare <the change> against the current version, using the
eval suite at <path>.
Run command: <the command that runs the suite for one version>

1. Confirm both versions use the same cases, graders and runs per
   case. If not, stop and tell me.
2. Tell me how many model calls the comparison will make, and wait
   for my go-ahead before running it.
3. Report for each version: the overall pass rate and the pass rate
   per tag.
4. Report the mean paired difference per case, with a 95% interval.
5. List every case that passed fewer runs with the change, with one
   failing output for each.
6. For model-graded cases that changed, quote the grader's reasoning.

Do not edit cases, graders, thresholds or the run count.
Never print, expose or commit secret values.
Do not commit, push, deploy, run destructive commands or make external
changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the failure rather
than repeatedly guessing.

Read the cases in step 5 yourself before you merge.

An LLM eval suite checklist

An example checklist, in the order of this post:

LLM EVAL SUITE: BEFORE YOU TRUST A CHANGE

□ Success written as properties a grader can check
□ Cases from real inputs, plus edge cases
□ Cases where a behaviour should not happen
□ Each case graded the same way by two people
□ Dataset in the repository, next to the prompt
□ Code checks wherever a rule can be written
□ Model graders with a clear rubric and an UNKNOWN option
□ Model graders checked against your own grades
□ Several runs per case, pass rate kept per case
□ Same model, settings and retrieval as production
□ Suite rerun on every prompt, model, setting or retrieval change
□ Versions compared on the same cases, case by case
□ Every case that got worse read, one by one
□ No case, rubric or threshold changed in the same pull request

Quick answers

What is an LLM eval?

An LLM (large language model) eval is a structured test for an AI feature: a set of inputs, the outcome each should produce, and graders that score the answers. Because the output varies between runs, an eval reports a pass rate across cases and repeated runs.

Why are unit tests not enough for LLM outputs?

Unit tests compare a result with one expected value, but a large language model (LLM) can answer the same input in different words that are all correct. Evals check properties, such as a required fact or a correct label, and run each case more than once.

How many test cases does an LLM eval set need?

A large language model (LLM) eval set can start small. Anthropic’s guide to agent evals calls 20 to 50 simple tasks drawn from real failures “a great start”. Add cases as new failures appear, and add more when you need to detect smaller differences.

Can an LLM judge replace human review?

An LLM (large language model) judge grades another model’s output against a rubric, and it is faster and cheaper than people. OpenAI and Anthropic both say to calibrate it against human grades, so human review still builds the reference grades and handles subjective cases.

How often should you run LLM evals?

Large language model (LLM) evals should run on every change that can alter the output: the prompt, the model or its version, settings, tool definitions and retrieval. OpenAI calls this continuous evaluation, and suggests growing the eval set over time.

More on AI coding is in the AI category.