Single Provider Dependency

Signs and symptoms

A call center’s voice assistant depends on a single model provider. On a peak day, the provider had an outage. The incident log (anonymized):

14:02  model p95 latency rises from 1.1 s to 9.8 s
14:05  503 error rate at 34%
14:06  calls start dropping on IVR timeouts
14:30  team decides to switch off the assistant; all calls go to the human queue
14:31  human queue: 412 people waiting, average wait 38 min
16:10  provider recovers; assistant switched back on

The system had no alternative path. No other model was configured, there was no degraded mode, and no automatic cutoff. Switching it off was a manual decision, made after half an hour of lost calls.

On the same bank’s WhatsApp, the Pix bot (Brazil’s instant payments) would share that fate: silence on the channel, a human queue, customers thinking the payment failed. Model providers have already had public incidents long enough to take down anyone without a backup plan. In November 2023, an OpenAI API outage knocked offline a visible slice of products that depended only on it.

Signs of a Single Provider Dependency:

• The whole product depends on one model, at one provider, in one region.
• There is no plan for when the API slows down, returns errors, or hits the rate limit.
• Timeouts are high or missing, and requests hang.
• There is no degraded mode (simplified answer, queue, fixed message).
• Nobody has tested the system with the provider down.

flowchart LR
  U(["1. Calls"]):::io --> A(["2. Assistant"]):::llm
  A --> P["3. Single provider"]:::tool
  P -->|"slow or down"| X(["4. The whole product stops"]):::io

1. All traffic reaches the assistant.
2. The assistant depends on a single API.
3. When the API degrades, there is no detour.
4. The product stops along with it.

Check +20 XP on first tryp95 jumps from 1.1 s to 9.8 s and 34% of calls return 503. There is no fallback. Which automatic action should already exist?

Switch the assistant off by hand after half an hour

A short timeout and a circuit breaker that stops sending traffic to the provider

Raise the timeout to 10 minutes and wait for the provider to recover

Reasons

• One provider is enough at first. Availability is high, and failures seem rare.
• The provider’s format is everywhere. Switching models means rewriting code, so nobody prepares a second one. See Provider Format Leak.
• Prompts tuned to one model. A backup model would answer differently, with no prior evaluation.
• Failure was never rehearsed. Nobody knows what the system does when the API does not answer.

Common trap: thinking another region fixes it

Moving the same model to another region covers one data center. It does not cover a suspended account, a global rate limit, an API bug, or a whole-provider incident. Another region is a good first step. It is not independence.

Cost of ignoring

• Total unavailability during provider incidents.
• Cascading failure. Hanging requests tie up connections and take down other services.
• Lost customers on real-time channels, such as voice and chat.
• Panic decisions. Switching everything off by hand, with no criteria.

The worst minute in the incident above was not the 503. It was 14:06 to 14:30, when the product still sent traffic to a provider that already could not answer in time, while the IVR dropped call after call.

Piece	What it prevents	What it does not prevent
Short timeout	Stuck workers	A provider that answers wrong, fast
Circuit breaker	Hammering the sick one	Poor quality on the backup
Another region	A local outage	An account or global API incident
Evaluated backup	Going mute when the primary dies	A different prompt and schema, if you did not measure

Treatment

Refactoring	Choose it when
Add Fallback Path	Whenever the product cannot stop. Define what happens when the model fails: another evaluated model or provider, a degraded mode, a queue, or a handoff to a human, with timeouts and a circuit breaker that cuts over automatically.

The order that actually holds the channel:

1. A short timeout on the HTTP client, matching the IVR or WhatsApp SLA. Without it, nothing below fires.
2. A circuit breaker. After N failures or latency above a threshold, stop sending traffic to the sick provider for a while.
3. An immediate degraded path. A fixed message, a queue, or a human. The channel does not vanish.
4. An evaluated backup model. Another provider or model, with its own prompt and its own eval, already run. With no eval, the switch is a second incident.

Rehearse the failure: take the API down on purpose in a test environment and watch whether the breaker opens, the backup answers in the right format, and the IVR stays up. The drill is the only way to know the plan exists for real.

Check +20 XP on first tryYou added a backup model from another provider. What do you do before relying on it in production?

Run the same eval on the backup, with its own prompt, and check the output format

Change the URL and watch the monitors

Send 100% of traffic to the backup for one afternoon

Payoff

• The product keeps working, perhaps at lower quality, during incidents.
• Automatic cutovers in seconds, not manual decisions in half an hour.
• A backup model already evaluated, with no surprises when switching.
• A failure drill that shows the real behavior.

When to ignore

• Asynchronous tasks that can wait. A nightly job can simply try again later.
• Non-critical internal tools. If nobody is hurt by an hour of downtime, a timeout and a clear message are enough.

Is it this smell?

Check +20 XP on first tryThe system uses a primary model and, if it fails, tries the same model in another region of the same provider. Is this a Single Provider Dependency?

No, there is a fallback

Yes, and the only fix is another provider

Partly

Check +20 XP on first tryThe system has a backup model from another provider configured, but nobody has ever run the eval on it. What can go wrong at switch time?

Nothing, you just change the URL

Only the price changes

Worse answers, or a different format

Check +20 XP on first tryWhen the API slows down, requests wait up to 10 minutes, the HTTP client's default timeout. All workers get stuck and the service stops. Which fix comes first?

A short timeout and a circuit breaker

More workers

Infinite retries on the same provider

Check +20 XP on first tryThe bank's WhatsApp bot, with the provider down, shows the fixed message 'we are having issues, use the app for Pix'. Is the smell treated?

Yes, a fixed message removes the smell

Partly: the product does not vanish, but you still need an evaluated backup if the channel must keep serving

No, only another provider counts as treatment