LLM feedback loop: turn production failures into eval cases

To improve an LLM (large language model) feature after release, run a feedback loop on a schedule. Collect feedback, read the failing traces, name the failure types, and fix the layer behind the most common one. Each fixed failure becomes an eval case, measured on the eval set before release and on production signals after.

The methods below come from OpenAI’s, Anthropic’s and Google’s evaluation docs, Google’s People + AI Guidebook, and two guides on Hamel Husain’s personal site. All were last verified on 1 November 2026, and none of it was tested for this post.

This is part 31 of the series. Parts 26 to 30 built the pieces of an AI feature: an eval suite, model routing, structured outputs, retrieval and agent state. This part connects them into one loop that runs after release.

Six steps in an LLM feedback loop, and who runs each

An LLM feedback loop has six steps, and a person owns every one of them. A thumbs-down stored in a log changes no prompt. Someone has to read the trace and decide which layer to change.

OpenAI’s evaluation best practices describe continuous evaluation, which covers part of this loop. They advise you to “run evals on every change, monitor your app to identify new cases of nondeterminism, and grow the eval set over time.”

Google’s People + AI Guidebook names two uses for explicit feedback. “You or your team can review user feedback for themes and make changes to the product accordingly. It can be fed directly back into the AI model as a signal.” This post covers the first use. Training a model on feedback is a separate job, with its own data and review.

Collect explicit and implicit feedback against each trace

User feedback on an AI feature comes in two kinds. The People + AI Guidebook defines explicit feedback as “when users deliberately provide commentary on output from your AI”. Its examples are surveys, ratings, thumbs up or down, and open text fields. Implicit feedback is “data about user behavior and interactions from your product logs”.

For an LLM feature, implicit signals include a user rephrasing the same question, regenerating an answer, asking for a human or leaving mid-task. Anthropic’s success criteria docs list “User feedback: Implicit measures such as task completion rates” among the ways to measure an LLM application.

Neither kind is a verdict. The guidebook warns that “there isn’t always a clear link between what a user does and what a user wants from the AI model.” A rephrased question can mean a bad answer, or a user who changed their mind. Treat feedback as a pointer to traces worth reading.

A pointer helps when it leads somewhere, so store each signal with the trace it belongs to. Part 21 shows how to join late feedback to a trace. An example feedback record, with made-up field names:

trace_id         the trace of the request being rated
response_id      the model response the signal is about
kind             explicit | implicit
signal           thumbs_down | rephrased | asked_for_human | abandoned
comment          free text, if the user wrote any
prompt_version   the prompt version that produced the answer
model            the model that answered

Feedback comes from the users who chose to give it, so it is a skewed sample. Hamel Husain and Shreya Shankar’s evals FAQ calls user feedback “a good way to narrow in on problematic traces”. It also suggests reviewing a random sample of traces.

Read the failing traces, then name the failure types

Error analysis is the work of reading real traces, writing down what went wrong, and grouping those notes into failure types. The evals FAQ calls it “the most important activity in evals”, and describes it in steps borrowed from qualitative research:

1. A domain expert reads each trace and writes a free-text note on what looks wrong. The FAQ calls this open coding, and suggests noting the first failure in a trace, “as upstream errors can cause downstream issues”.
2. Similar notes get grouped into a “failure taxonomy”, which the FAQ calls axial coding. Then you “count the number of failures in each category.”
3. The review continues until nothing new turns up. The FAQ suggests annotating at least 30 traces yourself and reviewing at least 100. You stop at “theoretical saturation”, when new reviews stop changing the taxonomy.

The failure types come from the traces you read. Hamel Husain’s A Field Guide to Rapidly Improving AI Products says a top-down list of generic metrics, such as hallucination, “often misses domain-specific issues”.

The FAQ also says to record problems that are not the model’s fault, such as a missing feature. It suggests leaving the cause for later: “deferring root-cause analysis and focusing on problems allows you to write higher-quality annotations while looking at more data.”

A made-up failure log for a support assistant shows the shape:

Sorted by count, the log tells you which failure type to fix first. A model can help group the notes, but the FAQ warns against letting it suggest failures too early: “its guesses can bias your judgment.”

Fix the layer behind the failure type

A failure type points at one layer of your AI feature, and the fix belongs in that layer. The table maps common failure types to a layer and the first thing to check. Most layers have a part of this series behind them.

For the prompt row, the evals FAQ notes that “Many teams discover their LLM doesn’t meet preferences they never actually specified”, such as short answers or a set format. Its advice is to “Fix these obvious gaps first before building complex evaluation infrastructure.”

Before you pick a layer, shrink the failure. The FAQ says: “When you find a failure, reproduce it with the simplest possible test case.” Its example is a wrong answer on turn four that still fails as a one-line question. The FAQ says that error is “likely a basic retrieval or knowledge issue”.

Retrieval gets checked before the answer. The FAQ’s advice for RAG (retrieval-augmented generation) is to “debug retrieval first using IR metrics, then tackle generation quality”. If the right document is missing, no prompt change will make the answer use it.

A model switch changes behaviour across every case, so it needs the whole eval set, plus the checks in Part 27.

Turn every fixed failure into an eval case

An eval case built from a real failure catches that failure if it ever comes back. The evals FAQ builds the repeatable set that way: “Start with examples from error discovery that capture important failure modes. Add confirmed failures as you find them.” OpenAI’s best practices say the same from the logging side: “Log as you develop so you can mine your logs for good eval cases.”

The case needs a check that would fail on the old behaviour. The FAQ says to “Use a code-based eval when an objective rule can identify the failure”, and an LLM judge when the failure needs human judgement. Part 26 covers graders and how to build the suite around them.

An example eval case for trace t-0440 from the made-up log, with made-up names and values:

id: returns-window-t0440
source: production trace t-0440, redacted
failure_type: wrong_document_retrieved
input: "Can I still return a jacket I bought five weeks ago?"
fixture: help-articles snapshot of 2026-10-28
checks:
  - type: retrieval
    expect_in_top_k: returns-policy-current
    k: 5
  - type: code
    must_contain: "60 days"
  - type: llm_judge
    rubric: "Pass if the answer states the current return window and how to start a return."
fixed_by: filter retrieval to the current policy version

The source, failure_type and fixed_by fields cost nothing to add. They tell the next person why the case exists, and whether it can be retired.

Not every failure needs an automated check. The FAQ advises: “Focus automated evaluators on failures that persist after fixing your prompts.” A one-line prompt fix can get a cheap code check. Save LLM judges for failures you will keep working on.

Measure the change before release and after it

A change to an AI feature gets measured twice. Before release, run the whole eval set on the old version and the new one, with the same cases and graders. The new cases should pass, and no old case should start failing.

Anthropic’s success criteria docs list A/B testing as one method: “Compare performance against a baseline model or earlier version.” Model outputs vary between runs, so run each case more than once before you trust a difference. Part 26 covers repeated runs.

After release, watch the signals that started the loop. Compare thumbs-down rate, rephrasing and handoffs by prompt version or model. Part 21’s traces carry gen_ai.prompt.version and gen_ai.response.model for that split.

The evals FAQ separates checks in CI (continuous integration) from monitoring. “CI evals protect against known regressions before deployment. Online monitoring find failures in production traffic and estimate how often they occur.” New failures from monitoring go back to step 2.

Each row comes from that provider’s docs, last verified on 1 November 2026; none was tested for this post.

Four ways a feedback loop stops measuring anything

Changing several layers in one release

A release that changes the prompt and the retriever together tells you little. If the eval score moves, you can’t say which change moved it. One change may have helped while the other hurt. Change one layer per release where you can, and write down which one.

Tuning against the eval set until it stops measuring

An eval set you tune against for many rounds starts to measure your tuning. The evals FAQ describes this for LLM judges: “Each time you use dev results to change the prompt or choose a model, information from those examples influences the evaluator. After many rounds, it may do well on the dev set but poorly on new examples.”

The FAQ says this about judges. A prompt tuned against the same cases is likely to drift the same way, though the FAQ does not say so. Keep a set of cases you never look at while tuning, and check against it before release. Refresh it from new production failures.

Treating feedback scores as the goal

A thumbs-down rate makes a poor target. The field guide warns that generic metrics “create a false sense of measurement and progress”. A feature can collect fewer thumbs-down because users stopped rating, and nothing got better.

Letting the eval set go stale

An eval set ages as the product and its users change. The FAQ says: “If everything keeps passing, this is a sign that the eval is less useful and should be run less often or retired.” On timing, the FAQ suggests re-running error analysis after “new features, prompt updates, model switches, or major bug fixes”.

Two prompts for running the loop

Both prompts are for a coding agent working in the repository of your LLM feature, with exported traces it may read. Neither relies on a tool-specific feature.

Prompt 1: find the failure types in a batch of traces

An example prompt for steps 2 and 3, after you have annotated your first traces by hand.

Task: help me with error analysis on <the exported traces, path>.
My notes on the first traces are in <path to my notes>.

Do not change code, prompts or data. Read only the traces and notes.

1. Read my notes and list the failure types they describe, in my words.
2. Search the remaining traces for more examples of each type.
   Cite the trace id for every example.
3. List traces that look wrong but fit no type, with a one-line note.
4. For each trace, note the first failure only.
5. Do not guess root causes yet.

Never print, expose or commit secret values or personal data. Refer to
traces by id.
Do not commit, push, deploy, run destructive commands or make external
changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the failure rather
than repeatedly guessing.

End with a table: failure type, count, example trace ids, and the
traces you were unsure about.

Check every suggested trace yourself before it counts toward a type.

Prompt 2: turn a fixed failure into an eval case

An example prompt for steps 5 and 6, once you know which layer you changed.

Task: add an eval case for <failure type> from trace <trace id>.
The fix changed this layer only: <prompt | retrieval | model | schema | tool>.
Eval set: <path>. Eval command: <command>.

1. Write a redacted case from the trace: input, fixture and checks.
   Use a code check if a rule can detect the failure; otherwise
   propose an LLM judge rubric for me to approve.
2. Record source, failure_type and fixed_by on the case.
3. Show that the new case fails on <the old version>.
4. Run the full eval set on the old and new versions, more than once.
5. Report new passes, new failures and anything that changed.

Do not edit existing cases or graders to make them pass.
Never print, expose or commit secret values or personal data.
Do not commit, push, deploy, run destructive commands or make external
changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the failure rather
than repeatedly guessing.

Step 3 matters most: a case that passes on the old version tests nothing.

Quick answers

What is an LLM feedback loop?

An LLM (large language model) feedback loop is an engineering routine for an AI feature in production. People collect user feedback, read failing traces, group failures into types, fix the responsible layer, add each fix to the eval set, and measure the change before and after release.

What is error analysis for LLM applications?

Error analysis for large language model (LLM) applications means reading real traces, writing a free-text note on each failure, and grouping the notes into a failure taxonomy with counts. The most common failure types decide which prompt, retrieval, model, schema or tool change comes next.

What is the difference between explicit and implicit feedback?

Explicit feedback is a deliberate rating or comment from a user, such as a thumbs-down or a written complaint. Implicit feedback comes from behaviour in product logs, such as rephrasing a question, regenerating an answer, asking for a human or abandoning the task.

Does an LLM application learn from user feedback automatically?

A large language model (LLM) application stays the same when users click thumbs-down. Feedback sits in logs until people read the traces and change a prompt, retriever, model, schema or tool. Training a model on feedback is a separate process with its own review.

How do you check an LLM change before release?

An LLM (large language model) change needs the full eval set run on the old and new versions, with the same cases and graders, several times each. The new cases should pass, no old case should start failing, and production signals are compared by version after release.

More on AI coding is in the AI category.