LLM model routing: test on evals, pin it, fall back safely

To choose a model for each task in an LLM (large language model) application, test the candidates on your own eval set, smallest first. Route each request by task type, a classifier or a failed check, and keep a tested fallback for errors and rate limits. Pin every model version, and run the evals again when a model changes or is deprecated.

The provider behaviour below comes from OpenAI’s, Anthropic’s and Google’s official docs, last verified on 1 November 2026. None of it was tested for this post.

This is part 27 of the series. Part 26 built the eval set, and this part uses it to compare models. Cost per successful task is in Part 22, so here it gets one row.

Write down what each task needs before you compare models

Model choice starts from the task, and one LLM application usually has several. A made-up support assistant labels tickets, pulls order details into JSON, answers customer questions and drafts replies. Those jobs need different things, so each gets its own answers to these questions.

Agree on the pass bar before any model runs. OpenAI’s model selection guide says to “Choose acceptance criteria before comparing models”.

Context window and feature support are documented facts, so read them from the provider. Anthropic’s Models API returns max_input_tokens, max_tokens and a capabilities object for every available model. The context window has to hold the prompt, any retrieved text and the answer, so test with your biggest real input.

Test the smaller model first on your eval set

Start with the smallest model that might do the task, and move up when the eval set shows a gap. Anthropic’s choosing a model guide calls this starting “efficiency-first”, one of the two starting points it describes. Its last step is “Upgrade only if necessary for specific capability gaps.” The same page says “Having a good evaluation set is the most important step in the process.”

OpenAI’s model selection guide gives similar advice: “If you can, experiment using the same inputs to compare results and keep the lightest setting that meets your quality bar.”

An example comparison run, one task at a time:

1. Freeze the inputs: the eval set version, the prompt, the output schema and the settings.
2. Run each candidate on that task’s cases more than once, because outputs vary between runs.
3. Compare pass rates for the task, then read the failing cases one by one.
4. Add cost per successful task and latency from the same runs.
5. Keep the lightest model that clears the bar, and write the choice down.

Try a lower effort setting before you try a bigger model. Anthropic’s guide says “Tuning effort is often a better lever than switching models.” OpenAI’s guide ends its comparison steps with “vary reasoning effort while keeping the model and inputs fixed.”

A written record of each choice helps later, when the model is retired and nobody remembers why it was picked. An example record, with names made up for this post:

ticket_label:
  model: small-model-2026-08-01    # a pinned snapshot ID, never an alias
  effort: low
  eval_set: support-evals v14
  pass_bar: <agreed before the run>
  tried: smaller model and lower effort first
  fallback: other-provider-small-2026-07-15
  decided: <date>
draft_reply:
  model: large-model-2026-09-10
  effort: medium
  eval_set: support-evals v14
  pass_bar: <agreed before the run>
  fallback: none                   # hand the ticket to a person
  decided: <date>

Route each request by task type, a classifier or a failed check

A router decides which model handles a request. Anthropic’s Building effective agents defines it in one sentence. “Routing classifies an input and directs it to a specialized followup task.” One of its examples sends “easy/common questions to smaller, cost-efficient models” and “hard/unusual questions to more capable models”.

Route by task type when your code already knows the task

Routing by task type is a lookup in your code. The ticket labeller always calls one model and the reply writer calls another. Nothing has to guess the route, so there is nothing extra to evaluate. Start here, and add a classifier later if one task needs it.

Use a simple classifier when one task mixes easy and hard inputs

A classifier routes requests inside one task, for example short, common questions to a small model and long, unusual ones to a larger model. The same article says routing works “where classification can be handled accurately, either by an LLM or a more traditional classification model/algorithm.”

That makes the classifier a model choice of its own. Give it eval cases: labelled requests, each with the route it should take. Send any request the classifier is unsure about to the larger model.

Escalate to a larger model when a check fails

Escalation runs the small model first and calls a larger one when the answer fails a check. The check has to be cheap and trustworthy: schema validation, a test run, a rule in code, or a grader you have already evaluated. Anthropic’s cost and intelligence guide describes the same pattern for effort: “run every task at a low setting and re-run only the failures at a higher one.”

Cap escalation at one step. The failed first attempt still counts in cost per successful task.

An example router config, with names made up for this post:

routes:
  ticket_label:
    model: small-model-2026-08-01
  order_details:
    model: small-model-2026-08-01
    check: validate against order_schema v3
    on_check_fail: escalate to large-model-2026-09-10, once
  customer_question:
    classifier: question-router v2   # has its own eval cases
    easy: small-model-2026-08-01
    hard_or_unsure: large-model-2026-09-10
log: route, model ID, escalated (yes or no), for every request

Record the route and the model ID on every request. They belong on the same trace as the rest of the request, as in Part 21.

Fall back on errors and rate limits without changing the contract

A fallback sends a request to another model or provider when the call fails: an error, a timeout or a rate limit. Escalation handles an answer that failed a check. A fallback handles a call that never returned a usable answer.

Retry the errors a retry can fix, then fall back

Each provider documents which errors are worth retrying. The table also shows how each one scopes rate limits, which decides whether another model on the same account can help.

Every row comes from the linked docs, last verified on 1 November 2026; none was tested for this post.

On OpenAI, models in the same shared-limit list draw from one pool, per its rate limits guide. Falling back to a sibling in that list may hit the same 429. A spend-cap or billing error belongs to the account. Switching models on that account is unlikely to help, so send those to a person.

Count the retries you already have. Anthropic’s SDK retries transient failures “twice by default”, per its errors page. Gemini’s Python SDK retries “up to four times”, per its troubleshooting guide. OpenAI’s rate limits guide warns: “If you manage retries in your application, disable SDK retries or account for them in those limits so nested retry loops don’t multiply requests.”

Timeouts need a deadline for the whole request, fallback included. The same OpenAI guide notes that “A timeout for each attempt isn’t necessarily a deadline for the entire operation.”

Keep the prompt, schema and safety settings the same across a fallback

A fallback that changes the contract ships a different feature during an outage. These parts stay the same across a fallback.

Structured output support differs between models and providers. Anthropic’s structured outputs page lists the supported models by ID. OpenAI’s guide says older models “may use JSON mode instead”. Both OpenAI and Gemini support a subset of JSON Schema, so test the fallback with your real schema.

Region can change with the platform too. Anthropic’s models overview notes that Amazon Bedrock offers “global endpoints (dynamic routing) and regional endpoints (guaranteed data routing)” for Claude Sonnet 4.5 and later.

Pin model versions and run the evals again when a model changes

Pinning means calling one exact model snapshot, so the model under your application can’t change without you. OpenAI’s text generation guide recommends “Pinning your production applications to specific model snapshots” to “ensure consistent behavior”, and building evaluation suites for “when you change and upgrade model versions”.

Every row comes from the linked docs, last verified on 1 November 2026; none was tested for this post.

A pinned ID can still shift a little. Anthropic’s model IDs page says “Occasionally, infrastructure updates produce minor differences in observable behavior even when the model ID and weights have not changed.” Run the eval set on a schedule, as well as on every change.

Run it again when any of these happen:

• a new model or snapshot you might move to
• a deprecation notice for any model you call, fallbacks included
• an alias change notice, if you still call an alias
• a change in behaviour that nobody can explain
• any change to the prompt, the schema or the router

Put the retirement date in the team calendar on the day the notice arrives. Anthropic’s deprecations page advises: “Test your applications with newer models well before the retirement date of your current model.”

Two prompts and a checklist for model routing

Neither prompt relies on one coding agent’s features.

Prompt 1: map every model call and its routing

An example prompt for a coding agent, to review how an application picks and calls models.

Task: map how <the application or service> chooses and calls models.
Start from: <path or entry point>

Do not change code and do not call any model API.
For every model call, report:
1. Where it is called, and which task it does.
2. The model ID: a pinned snapshot, an alias such as "latest",
   or a value from config?
3. How the model is chosen: fixed per task, a classifier, or
   escalation after a failed check.
4. What happens on a 429, a 5xx error or a timeout: retries,
   backoff, overall deadline, fallback model.
5. Do SDK retries and application retries stack?
6. Does a fallback keep the same prompt, output schema, tools,
   safety settings and region?
7. Which eval cases cover this call, if any?

Cite the file and line for each answer. Mark anything you could not
confirm as unverified.
Never print, expose or commit secret values.
Do not commit, push, deploy, run destructive commands or make external
changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the failure rather
than repeatedly guessing.

End with the calls that have no eval cases, no pinned version or an
untested fallback.

Prompt 2: plan a move off a deprecated model

An example prompt for when a deprecation notice names a model you call.

Task: plan the move from <current model ID> to <candidate model ID>
for <the task or feature>. Retirement date: <date from the notice>.
Eval set: <name and version, or path>

Do not change production config and do not deploy.
1. List every place the current model ID is used, fallbacks and
   config files included.
2. From the provider's docs, check the candidate's context window,
   output limit and support for the features these calls use.
   Link each source.
3. Write the command that runs the eval set on both models, with the
   same prompts, schema and settings, more than once.
4. Before running it, tell me how many model calls it will make, and
   wait for my go-ahead.
5. After the run, compare pass rates per task and list the cases
   that newly fail.

Never print, expose or commit secret values.
Do not commit, push, deploy, run destructive commands or make external
changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the failure rather
than repeatedly guessing.

End with a recommendation per task: move, adjust the prompt and test
again, or try another candidate.

A model routing checklist

An example checklist, in the order of this post:

MODEL SELECTION AND ROUTING: REVIEW

□ Criteria and pass bar written per task before comparing models
□ Candidates run on the eval set, smallest first
□ Lower effort tried before a larger model
□ Each choice recorded: model ID, eval set version, date
□ Routing by task type, unless one task mixes easy and hard inputs
□ Any classifier has its own eval cases
□ Escalation on a failed check, capped at one step and logged
□ Retries only on errors a retry can fix, inside one deadline
□ SDK retries and application retries counted together
□ Fallback keeps prompt, schema, tools, safety settings and region
□ Fallback model passed the same eval set
□ Every model ID pinned; no "latest" alias in production
□ Retirement dates in the calendar, fallbacks included
□ Eval set run on every model change, and on a schedule

Quick answers

What is LLM model routing?

LLM (large language model) model routing sends each request to the model that suits it. The router can use a fixed map from task type to model, or a classifier that sorts easy from hard requests. It can also escalate to a larger model after a failed check.

How do you choose an LLM for a task?

Choosing a large language model (LLM) for a task starts with written criteria: quality on your own eval cases, latency, cost, context window, output format, tools and data rules. Run the candidates on the same eval set, smallest first, and keep the lightest model that passes.

What is the difference between LLM fallback and escalation?

An LLM (large language model) fallback calls another model or provider when the first call fails with an error, a timeout or a rate limit. Escalation calls a larger model when the first call returned an answer that failed a check, such as schema validation.

Should you pin LLM model versions in production?

Pinning a model version means calling one exact snapshot ID instead of an alias that moves to new releases. OpenAI recommends pinning production applications to model snapshots, and Google says most production apps should use a specific stable Gemini model.

What should you do when an LLM model is deprecated?

A deprecated large language model (LLM) keeps working until its retirement or shutdown date, and then requests to it fail. Note the date, test replacement candidates on your eval set with the same prompts and schema, and move every task before then, fallbacks included.

More on AI coding is in the AI category.