LLM fallback models

Day 25: Your fallback model is a second production model nobody onboarded

Your gateway config has a fallback model. That means you're running two production models, and you've tested one of them. Most teams treat the fallback as an uptime feature: the primary fails, traffic moves elsewhere, the system stays up. In LLM systems the fallback also changes how the system behaves, and that change won't show up on any availability dashboard.

The silent 2am failover

It usually goes like this. At 2am the primary provider starts returning 529 overloaded errors. The gateway does exactly what you told it to and sends every request to the fallback. Every call comes back 200 OK. Latency looks normal. Error rate is zero. Nobody gets paged.

At 9am a customer writes in. The support agent has been closing tickets without issuing the refunds it promised. Or its summaries skip the one field their workflow depends on. The customer found the problem before your team did. Nothing went down. The system just got quietly worse. Your monitoring tells you whether the system is available. It doesn't tell you whether it's still doing the job correctly.

Where this sits in the course

In earlier days you built eval suites, tool calling and cost-aware routing, and you tuned all of it against one specific model. Today is about what happens to that work when the model underneath changes without warning.

The mental model shift: uptime vs. behavioural contract

The uptime view comes from classic infrastructure. A server is a server: if one dies, you send traffic to another. That works when the replacement is identical. LLMs aren't interchangeable. Your system prompt, tool schemas and few-shot examples are effectively a contract with one specific model. When you switch models, you're asking a different party to honour a contract it has never read.

| Question                 | Uptime view                     | Behavioural-contract view                          |
|--------------------------|---------------------------------|----------------------------------------------------|
| What is a fallback?      | A spare that takes traffic      | A second model with its own prompt and evals       |
| Success signal           | 200 OK, latency, error rate     | Eval score, task success, tool-call validity       |
| When do you switch?      | Per call, on 5xx/429/529        | Per task, at a checkpoint                          |
| What gets tested?        | That the route works            | That the behaviour holds on that route             |
| Who owns it?             | Whoever owns the gateway config | A named owner of the fallback path                 |
| Failure mode             | Visible outage                  | Invisible degradation                              |

You do need the fallback line in your LiteLLM, OpenRouter or Portkey config. But it only covers the uptime half of the problem. The rest of this lesson is about the behavioural half.

What actually breaks, concretely

Four kinds of failure come up again and again in agent pipelines. None of them returns an HTTP error.

• ▹Tool-call formatting drift. The primary reliably fills in a nested `filters` object. The fallback flattens it into a string, wraps the arguments in markdown, or adds an optional field your schema doesn't have. Then your parser either rejects the call and starts a retry loop, or coerces it leniently and runs the wrong query. You get `search_orders(filters="status:refunded")` instead of `{"status": "refunded"}`, and that returns every order you have.
• ▹Instruction-following on long system prompts. Your system prompt is 3,000 tokens, and line 140 says 'never close a ticket with an open refund'. The primary keeps hold of that rule. A smaller or differently trained fallback tends to attend to the start and end of the prompt and lose rules buried in the middle. You find out when a ticket gets closed that shouldn't have been.
• ▹Refusal and verbosity differences. The fallback refuses a legitimate request the primary handled fine, like summarising a medical-billing dispute. Or it puts three paragraphs of caveats around a JSON payload that the downstream agent expected to be bare. In a multi-agent chain, the next agent reads that preamble as content.
• ▹Few-shots that over-steer a weaker model. All three of your few-shot examples involve refunds. The primary generalises from them. The weaker fallback copies them and pushes every ticket toward a refund, password resets included. Few-shots written to correct a strong model's small quirks often become templates that a weaker model fills in literally.

The splice problem: why agents make this worse

In a single-shot feature, a fallback gives you one worse answer. In an agent, it can give you a trajectory that doesn't make sense. Here's how that happens. The primary writes a five-step plan and runs steps 1 and 2. It times out on step 3, and the gateway retries on the fallback. The fallback gets the full context: a plan it didn't write, reasoning it wouldn't have produced, and tool results it has to interpret through someone else's framing. It carries on confidently, because the context says the plan is already underway. It might skip a precondition the primary meant to check in step 4. It might re-run step 2 because it misread the state.

Every step returns 200 OK, and the overall result doesn't hold together. You've spliced two models into one trajectory, and no single model owns the outcome.

The rule is to switch models per task, not per call. If the primary fails mid-task, don't hand its trajectory to another model. Restore the last checkpoint, meaning the state before the plan or the last verified step. Then either re-plan from scratch on the fallback with the fallback's own prompt, or pause and resume on the primary later. Per-call retries on the same model are fine. Per-call retries across models are where splicing happens.

Step 1: Onboard the fallback

Onboard the fallback like a new engineer joining the production rotation. It gets its own docs, someone checks it can use the tools, and it passes a test before it carries the pager.

• ▹Give it its own prompt variant. Start from the primary's prompt and cut it down. Move critical rules to the top and bottom, spell out output formats, and check every few-shot example for over-steering. Version it next to the primary's prompt, for example `support_agent.v14.primary` and `support_agent.v14.fallback`.
• ▹Check its tool schemas. Run every tool through the fallback with representative inputs and validate the arguments strictly. Simplify the schemas it struggles with: flatten nesting, swap free text for enums. If it still can't call a tool reliably, take that tool out of the fallback tier.
• ▹Run your Day-N eval suite against it. Same golden set, same graders, applied to the fallback using its own prompt. Keep the result as a baseline you'll compare against later, not a check you run once.

Set your pass/fail thresholds before you look at the numbers. Otherwise you'll find a way to explain away a bad result. Here's a minimal table to start from. Tune the values to your own risk tolerance:

| Check                          | Measured how                         | Fallback must                      | If it fails                          |
|--------------------------------|--------------------------------------|------------------------------------|--------------------------------------|
| Tool-call validity             | Strict schema validation on eval set | Match primary on allowed tools     | Remove failing tools from tier       |
| Task success (core eval)       | Your Day-N graders                   | Within an agreed gap of primary    | Restrict to read-only tier           |
| Critical-rule adherence        | Targeted adversarial cases           | Zero violations                    | Block fallback for that workflow     |
| Refusal on legitimate requests | Labelled benign-but-sensitive set    | No higher than primary + margin    | Adjust prompt variant, re-test       |
| Output format compliance       | Parser success rate                  | Parses cleanly every time          | Add structured output / constrain    |

Critical-rule adherence has zero tolerance on purpose. You can live with a fallback that writes slightly worse summaries. You can't live with one that sometimes issues refunds it shouldn't.

Step 2: Drill it like disaster recovery

A backup you've never restored from isn't really a backup, and the same goes for a fallback model. Providers repoint model aliases to new versions, deprecate and retire old snapshots, and change platform defaults. A fallback that passed in March can quietly fail in October.

• ▹Schedule forced-failover runs. Use a weekly CI job, or a small canary slice in staging, that pins the system to the fallback tier and runs the full eval suite. Simulate the failure at the gateway, for example by making the primary return 529, so you test the real routing and checkpoint logic and not only the fallback model.
• ▹Alert on drift as well as failure. Track the fallback's eval score over time. Alert when it drops below threshold or moves noticeably away from its baseline. In production, also alert whenever real traffic is running on the fallback tier. That one alert would have caught the 2am incident.
• ▹Name an owner. The fallback path needs one person, not 'the platform team'. That person owns the prompt variant, reviews drill results and decides when a tier gets demoted. Fallbacks nobody owns rot.
• ▹Log the model per task. Every trace should record which tier and model handled the task. Then incident review can tell 'the agent is bad' apart from 'the agent was on the fallback'.

Step 3: Degrade, don't substitute

A fallback shouldn't pretend nothing changed. Its job is to keep doing the part of the work it can still do safely. A good fallback tier narrows what the agent is allowed to do. It doesn't copy the primary onto a weaker model.

• ▹Fewer tools. Remove write tools, irreversible actions and anything that failed the schema checks. A read-only agent that answers questions correctly beats a full agent that occasionally deletes the wrong record.
• ▹Lower autonomy. Switch from auto-execute to human approval. The agent drafts the refund, and a person clicks approve.
• ▹Honest capability messaging. Have the agent say something like 'I can look up your order status right now, but I can't make changes. I've saved your request and it will be completed shortly.' In my experience, users cope with honest limits much better than with confident mistakes.
• ▹In multi-agent setups, keep the strong model where the judgment calls are. Planner and reviewer roles stay on the primary, or pause if it's down. Executors that run narrow, well-specified, verifiable steps can drop a tier. A weak executor with a strong reviewer degrades gracefully. A weak planner drags down everything after it.
• ▹Sometimes the best fallback is a safe stop. Write a checkpoint, queue the task and tell the user when to expect it. For high-stakes workflows like payments, account changes or production deploys, pausing is the right fallback.

A whiteboard-able fallback policy

Sketch this on a whiteboard before you touch the gateway config. Each tier maps to a model, a prompt variant, a tool allowlist, an autonomy level and a condition for pausing. The model names are placeholders, so swap in your actual primary and fallbacks.

fallback_policy:
  switch_granularity: per_task        # never per_call across models
  on_mid_task_failure: restore_checkpoint_then_replan
  owner: "@fallback-path-owner"
  drill: { schedule: weekly, suite: evals/core_v14, alert_on_drift: true }

  tiers:
    - name: T0_primary
      model: primary-model
      prompt: support_agent.v14.primary
      tools: [search_orders, read_ticket, issue_refund, close_ticket, send_email]
      autonomy: auto_execute

    - name: T1_fallback
      trigger: primary unavailable at task start
      model: fallback-model
      prompt: support_agent.v14.fallback
      tools: [search_orders, read_ticket, draft_refund, draft_email]
      autonomy: human_approval          # drafts only, a human commits
      user_message: "I can check status and prepare changes; a teammate will confirm them."
      roles: { planner: pause_if_primary_down, executor: fallback-model, reviewer: primary_or_human }

    - name: T2_safe_stop
      trigger:
        - fallback eval score below threshold in last drill
        - workflow tagged high_stakes (payments, account_changes)
        - mid-task failure with no valid checkpoint
      action: checkpoint_and_queue
      user_message: "Your request is saved and will be completed when full service resumes."

  observability:
    log_per_task: [tier, model, prompt_version, checkpoint_id]
    alert: production traffic on T1 or T2 for > 5 minutes

Notice what isn't there: no line that says 'on 529, retry on another model and keep going'. Each tier is a behavioural state, and moving to a new tier changes the job as well as the model.

Common pitfalls

• ▹Reusing the primary's prompt on the fallback because 'it's mostly the same'.
• ▹Testing the fallback once at setup and never again, while the provider repoints the alias or retires the version underneath you.
• ▹Setting up per-call cross-model retries inside an agent loop. That's a splice waiting to happen.
• ▹Monitoring only availability and error rate, with nothing that tells you 'we're running on the fallback right now'.
• ▹Putting the weakest model on the planner because planner calls cost the most.

Exercise for today

In staging, force your system onto its fallback for one hour. Simulate the failure at the gateway instead of editing app code, so you're testing the real path. Run your eval suite, then drive a few real tasks by hand. Write down three behaviours that changed: a tool argument that drifted, a rule that got skipped, a new refusal or a shift in tone. For each one, decide whether to fix the prompt variant, remove that capability from the fallback tier, or route it to a safe stop. That list of decisions is the first draft of your fallback policy.

Tomorrow, Day 26: you know your system degrades, so how do you spot it in production before your customers do? We'll cover observability for agents: traces, per-task model attribution, and the signals that actually tell you behaviour has shifted.

Extend your knowledge

• ▹Read your gateway's fallback and routing docs (LiteLLM, OpenRouter or Portkey) with one question in mind: does it retry per call or per task, and can I attach a different prompt to each fallback?
• ▹Look at how agent frameworks handle checkpointing, for example LangGraph's persistence and checkpointer concepts, to see how to restore state before re-planning instead of splicing.
• ▹Read up on chaos engineering (Netflix's Chaos Monkey and the Principles of Chaos Engineering). Take its 'steady-state hypothesis' and apply it to eval scores instead of uptime.
• ▹Read the model migration or deprecation guides your providers publish. Every forced migration is really an unplanned fallback, so run today's onboarding checklist against it.