Introduce Model Cascade

Problem

Every request uses the most expensive model, easy ones included, and there is no spending cap. This is the No Cost Budget smell, in its high-volume form.

Solution

Try a cheaper model first and move up to a larger one only when a verifiable criterion shows the cheap answer will not do. Before building the cascade, also measure the simplest alternative: the same capable model with lower reasoning effort.

Before and after

-def triage(ticket):
-    return llm_parse(TRIAGE_PROMPT, ticket, schema=Triage, model="large")
+TIERS = [("small", 0.85), ("large", 0.60)]          # thresholds calibrated in the eval
+
+def triage(ticket):
+    for model, min_conf in TIERS:
+        t = llm_parse(TRIAGE_PROMPT, ticket, schema=Triage, model=model)
+        trace.set("cascade.tier", model)
+        if valid(t) and t.confidence >= min_conf:   # validation in code + confidence
+            return t
+    return human_queue(ticket)

Before: the large model on everything.

flowchart LR
  IN(["1. Ticket"]):::io --> L(["2. Large model"]):::llm
  L --> OUT(["3. Classification"]):::io

After: the cheap one first, the large one when needed.

flowchart LR
  IN(["1. Ticket"]):::io --> S(["2. Small model"]):::llm
  S --> C{"3. Valid and confident?"}
  C -->|yes| OUT(["5. Classification"]):::io
  C -->|no| L(["4. Large model"]):::llm
  L --> OUT

1. The same ticket comes in.
2. The small model tries first.
3. A criterion in code decides whether the answer will do.
4. Only the cases that fail the criterion move up to the large model.
5. The tier used is recorded on the trace.

Worked example

Imagine support for a Brazilian marketplace. Thousands of tickets arrive each day: “where is my order”, returns, invoices, Pix payment disputes. Pix is Brazil’s instant payment system. Triage needs a queue, a priority, and a product.

In the eval, the large model scores 94% overall. The small one scores 89%. The gap looks small until you split by type:

Type (example)	Volume	Small is right	Large is right
Tracking and delivery time	high	97%	98%
Returns and exchanges	medium	88%	93%
Pix and chargeback	low	71%	92%

Almost all volume sits in the first two rows. Paying the large model on tracking buys 1 percentage point. On Pix and chargeback, the same money buys 21 points.

The cascade in this example looks like this. The small model tries. Code validates the schema (known queue, priority in the list, existing product). If validation passes and calibrated confidence clears the threshold, triage is done. The rest moves up. A long chargeback ticket can have a cheap rule that sends it straight to the large model, skipping the first call.

Suppose 10,000 tickets a day and that the large model costs about 10 times the small one per call. If 80% stay on the small model, daily cost falls to about 28% of “large on everything”, and you still pay the large model on the hard 20%. These numbers are illustrative: run the arithmetic with your provider’s price and your volume.

The FrugalGPT paper (Chen, Zaharia, and Zou, 2023) documents the same idea by chaining models from different APIs: try the cheap one and move up only when the answer misses a quality threshold. The lesson that matters here is the same. The gain comes from the share that does not move up, not from the extra model.

Check +20 XP on first tryIn the eval, the small model writes confidence 0.92 on almost every ticket and is right on 78% of those cases. The team uses 0.85 as the escalation threshold. What is wrong?

The threshold should be 0.99, because 0.85 is too low for any model

Stated confidence does not separate hits from misses, so it cannot be the criterion on its own

The small model has to go, because 78% is unacceptable in triage

Why it works

In high-volume workloads, most inputs tend to be easy, and any competent model gets them right. Paying for the larger model on those inputs is cost with no gain. The cascade reserves the expensive model for the hard fraction.

The criterion is what makes the cascade work or fail. Validation in code (schema, business rule) is the most reliable. The model’s stated confidence is only useful once calibrated in the eval: if the small model says 90% and is right 70% of the time, the threshold must rise. The target pattern is Model Cascade.

The alternative most teams skip

Before the cascade, run the same capable model at lower reasoning effort. Several providers expose this as “effort”, “thinking”, or a reasoning flag. If that version ties the cascade on quality and cost, keep one model.

One model is easier to maintain. The prompt cache does not split. The trace has one tier, not two. The cascade only pays off when the eval shows that two models, with an escalation criterion, beat that alternative.

Safe mechanics

0. Measure. Run the eval with the current model and record accuracy per category and cost per item.
1. Measure the simple alternative: the same model at lower effort. If it ties on quality and cost, keep it; one model is easier to maintain and does not split the cache.
2. Run the eval on the cheap model and see where it fails.
3. Choose the escalation criterion: validation in code first, calibrated confidence second.
4. Calibrate thresholds with the eval: accuracy per confidence band on the cheap tier.
5. Simulate the cascade on the dataset: share that moves up, final accuracy, total cost.
6. Implement it with the tier on the trace and a final destination for whatever no tier resolves.
7. Measure again in production, sampling answers from the cheap tier.

How to calibrate the threshold

Split the cheap-tier dataset into bands: confidence below 0.6, 0.6 to 0.8, above 0.8. In each band, measure real accuracy.

The useful threshold is the point where the band starts being accurate enough for the product. If the “above 0.8” band scores 96% and “0.6 to 0.8” scores 74%, a cut near 0.8 makes sense. If every band scores about the same, confidence carries no information. Then the criterion has to be something else: text length, presence of a tax ID, a business rule that failed.

Never pick 0.7 because it “feels reasonable”. That number only exists after the eval.

Risks

• Confident errors. The cheap model is wrong with high confidence, and the error does not move up. Review samples of answers approved on the first tier.
• Everything moves up. A criterion that is too strict sends almost everything to the large model, and the cascade only adds latency.
• Latency for hard cases. Whatever moves up pays for two calls. Check that the product can accept it.
• Split cache. Different models do not share the prompt cache.

Common mistake: cascade as decoration

A team puts the small model in front, escalates on any doubt, and 95% of requests go to the large one. On the trace, “cascade” shows up on every span. On the bill, cost got worse: an extra call on almost everything, plus the large model.

That is not a cascade. It is the large model with a toll booth in front. The metric that gives it away: share that moves up. If it clears, say, two thirds of your volume, go back to the eval and loosen the criterion, or drop the cascade.

Check +20 XP on first tryA week after turning the cascade on, 94% of tickets move up to the large model. What is the most useful next step?

Open the eval and the escalation criterion: the cascade is only adding latency

Add a third, even larger tier for the remaining 6%

Swap the small model for a medium one, keeping the same 0.85 threshold

Challenge

The eval shows: the small model gets 96% of short tickets and 71% of tickets over 300 words right. The large one gets 97% and 93%. How would you design the cascade?

Show solution

Put a cheap routing rule before the cascade: tickets over 300 words go straight to the large model, because the small one fails too often on them. Short tickets go to the small model, with validation in code and calibrated confidence to move doubtful cases up. Check in the eval the final accuracy per length band and the total cost against the “large on everything” version.

Check your understanding

Check +20 XP on first tryThe eval shows that the same capable model, at low effort, ties the cascade on accuracy and costs about the same. What do you do?

Ship the cascade anyway, so the trace has visible tiers

Raise effort on everything, because the capable model is already in the system

Keep one model. The cascade would not pay for the extra complexity

Check +20 XP on first tryThe cheap tier approves a Pix dispute ticket with the payee tax ID digits reversed, confidence 0.97. Pix is Brazil's instant payment system. How do you catch this kind of error?

Sending every ticket that mentions Pix straight to the large model

Sampling answers approved on the first tier and validating rules in code, such as tax ID check digits

Asking the small model to 'think more' before answering

Check +20 XP on first tryTickets over 300 words: the small model scores 71%, the large one 93%. Short tickets: 96% and 97%. Which design best matches these numbers?

The same cascade for everyone, with the same confidence threshold

Only the large model, because 71% on the small one is unacceptable

A length rule sends long tickets straight to the large model; short ones enter the cascade

Check +20 XP on first tryIn production, overall cascade accuracy is stable, but the human queue grew. The trace shows cascade.tier=large on most of those cases. What does that suggest?

The small model got worse and needs retraining

The large model is also failing a new slice, and the final destination (a human) is overloaded

You need a middle tier between small and large