Add Fallback Path

Problem

At 2pm, the primary model provider starts returning 503. A bank’s voice bot has no second path. Every call falls into silence after four seconds. The call center fills up. When the model, a tool, or the provider fails or slows down, the whole product stops with it, and nobody defined what should happen. This is the Single Provider Dependency smell.

Solution

Define, for each dependency, what happens when it fails: a short timeout, an already evaluated alternative path (another model, another provider, a fixed answer, a queue, a person), and a circuit breaker that stops hammering a service that is down.

Before and after

-def answer(call):
-    reply = client.messages.create(model=PRIMARY, messages=call.messages)
-    return speak(reply)
+def answer(call):
+    if breaker.is_open("primary"):                         # breaker open: do not even try
+        return fallback(call)
+    try:
+        reply = client.with_options(timeout=2.5, max_retries=0).messages.create(
+            model=PRIMARY, messages=call.messages, max_tokens=400)
+        breaker.record_success("primary")
+        return speak(reply)
+    except (APITimeoutError, APIConnectionError, InternalServerError, RateLimitError):
+        breaker.record_failure("primary")
+        return fallback(call)
+
+def fallback(call):
+    reply = backup_client.create(model=BACKUP, prompt=BACKUP_PROMPT, ...)  # evaluated in the eval
+    if reply is None:
+        return speak_fixed("I will transfer you to an agent.") or transfer(call)
+    return speak(reply)

Before: a single path.

flowchart LR
  C(["1. Call"]):::io --> P(["2. Primary model"]):::llm
  P -->|"failure or slowness"| X(["3. Silence and a dropped call"]):::io

After: a timeout, a circuit breaker, and fallback tiers.

flowchart LR
  C(["1. Call"]):::io --> B{"2. Breaker open?"}
  B -->|no| P(["3. Primary model, short timeout"]):::llm
  B -->|yes| F(["4. Backup model"]):::llm
  P -->|"error or timeout"| F
  F -->|"also failed"| H((("5. Human agent"))):::human
  P --> OUT(["Answer"]):::io
  F --> OUT

1. The request arrives.
2. If the primary service failed several times recently, the breaker is open and no attempt is made.
3. The primary model has a timeout that fits the product (seconds, not minutes).
4. The backup has already been evaluated with its own prompt.
5. If everything fails, there is a defined path that does not depend on a model.

Why it works

Provider failures are rare, but certain. Major providers’ status pages record public incidents with some regularity. Without a plan, every failure becomes a manual decision, made late and under pressure. With a plan in code, the switch happens in seconds and the same way every time.

The three parts complement each other. A short timeout stops hanging requests from consuming every resource. The circuit breaker avoids hammering a service that is down, which worsens the outage and delays recovery. The evaluated fallback guarantees the alternative path works before you need it.

Worked example: voice with a 3-second ceiling

Product: a bank IVR on the phone. The pause a customer will tolerate is about 3 seconds. The SDK default waits 10 minutes and retries twice. Leave that alone, and a provider slowdown holds 200 concurrent calls until timeout.

Configuration: timeout=2.5, max_retries=0. Breaker opens after 5 failures in 30 seconds. Fallback: a model at another provider, with its own prompt. Suppose the eval shows the backup about 4 points below the primary on intent classification. Third tier: a fixed sentence and transfer to a human agent.

At 2pm the primary returns 503. The first 5 calls fail in 2.5s and fall to the backup. The breaker opens. Later calls do not even try the primary. The call center only gets what the backup also could not cover. Without the plan, every call went to silence.

Check +20 XP on first tryThe provider SDK has a 600s timeout and 2 retries. Your voice bot needs to answer in 3s. You only implemented a backup model, without touching the HTTP client. What happens on a slowdown?

The backup kicks in immediately, because the failure is obvious

The call waits for the SDK timeout and retries; the backup arrives minutes later, if at all

The breaker opens on its own after the first slow call

Safe mechanics

0. Measure. Gather the model’s p95 and p99 latency, the error rate, and what happens today on a failure (simulate it in staging).
1. Set timeouts from the experience the product can accept, not from the library default.
2. Choose each dependency’s fallback: another model, another region, another provider, a fixed answer, a queue, or a person. A nightly job does not need the same tier as live voice.
3. Evaluate the fallback with the eval, with its own prompt if needed. Know how much quality drops. Without that, the backup is a surprise on the worst day.
4. Implement the circuit breaker with a failure threshold, an open period, and a recovery probe (half-open: one probe, not a flood).
5. Record on the trace which path served each request.
6. Rehearse the failure: take the primary down in staging and confirm the fallback takes over.
7. Measure again availability and quality on the alternative path.

Trap: two models at the same provider

gpt-4o and gpt-4o-mini fail together when that provider’s API, in that region, is down. Diversify provider or region when the product cannot stop. If you cannot, the last tier cannot be a model: a fixed sentence, a queue, a person.

Another trap: a fallback nobody ran through the eval. Different format, different tone, a tool with an old name. The outage day is the first time that prompt sees production.

Check +20 XP on first tryThe backup never went through the eval. In the staging rehearsal it answers in free text, and the product expects JSON with intent. What do you do before production?

Nothing. On outage day, free text is still better than silence

Run the same eval as the primary on the backup, with its own prompt, and only promote it if the format matches and quality is acceptable

Tell the backup prompt to ‘answer like the primary’, with no dataset

Risks

• A never-tested fallback. A backup model nobody evaluated may answer worse or in another format. The backup’s eval is mandatory.
• A fallback that also falls. Two models at the same provider and region fail together. Diversify when the product requires it.
• Flapping. A badly configured breaker opens and closes constantly. Tune the threshold and window. One probe in half-open, not 100% of traffic back at once.
• Unexpected cost. The backup may be more expensive. Monitor cost while it is in use.

Challenge

A nightly report generator uses a model to summarize sales. If the provider goes down, the report does not go out. Which fallback makes sense here?

Show solution

For an asynchronous task, the simplest fallback is time: retry with growing backoff for a few hours, only on transient errors, with an alert if it has not finished by a deadline. If the report must go out no matter what, a second tier can produce a version without the summary (only the tables computed in code), marked as incomplete. A backup model only if the eval shows it summarizes well.

Related

• Treats the Single Provider Dependency smell.
• Errors that deserve a retry, not a path change: Classify Errors Before Retry.
• No baseline on the backup: Add Eval Before Changing the Prompt.

Check your understanding

Check +20 XP on first tryThe breaker opens. You dump 100% of traffic back to the primary after 30s, with no probe. The provider is still unstable. What happens?

Flapping: the breaker closes, failures return, it opens again, and the product shakes to the recovery's rhythm

Nothing serious. 30s is long enough for any incident to end

The backup keeps serving in parallel, so the flood does not matter

Check +20 XP on first tryA nightly NF-e job (Brazilian electronic invoices) summarizes invoices. The provider goes down at 2am. Which fallback tier comes first?

Hand every invoice to a human agent at 2am

Wait and retry transient errors until a deadline; if that fails, ship the report with totals computed in code only, marked incomplete

Switch immediately to a more expensive model at another provider, with no eval, because the report has to go out

Check +20 XP on first tryPrimary and backup sit in the same region of the same provider. The status page goes red in that region. What should your design have planned for?

Nothing. A single region keeps latency simple, and the breaker is enough

A third tier that is not a model: a fixed sentence, a queue, or a person, because both models go down together

Longer retries on the primary, to ride out the incident