Serial Latency Stack

Signs and symptoms

A call center voice assistant picks up, understands the request, and answers. Before speaking, it does four things (anonymized excerpt):

intent    = classify_intent(transcript)       # 1.1 s
customer  = summarize_customer(crm_history)   # 2.4 s
policy    = find_policy(transcript)           # 1.8 s
sentiment = detect_sentiment(transcript)      # 0.9 s
reply     = generate_reply(intent, customer, policy, sentiment)   # 2.2 s

The first four steps do not depend on each other, but they run in a queue. Add them up: 1.1 + 2.4 + 1.8 + 0.9 + 2.2 = 8.4 s. The caller sits in silence for more than eight seconds. Many hang up before hearing the answer.

On a bank’s WhatsApp, the same design delays confirmation of a Pix payment (Brazil’s instant transfer). The person already paid in the app and is waiting for the bot to say it landed. Every extra second feeds “I already paid, why isn’t it showing?”.

Signs of a Serial Latency Stack:

• The code makes several model or tool calls, one after another, with sequential awaits.
• Some of those calls do not use the previous calls’ results.
• The trace shows spans in a staircase, none of them overlapping.
• Total latency is the sum of the latencies, and it grows with each new step.
• Users complain about slowness, but each call alone is fast.

flowchart LR
  IN(["1. Input"]):::io --> A(["2. Intent"]):::llm
  A --> B(["3. Customer summary"]):::llm
  B --> C(["4. Policy"]):::llm
  C --> D(["5. Sentiment"]):::llm
  D --> R(["6. Reply"]):::llm

1. The input arrives.
2. to 5. Four independent analyses run in a queue, each waiting for the previous one to finish.
3. The reply only starts after all of them.

Check +20 XP on first tryThe trace shows four staircase spans: intent 1.1 s, customer 2.4 s, policy 1.8 s, sentiment 0.9 s. None uses the previous result. What is the latency of those four if you parallelize?

6.2 s, the sum does not change

2.4 s, that of the slowest step

0.9 s, that of the fastest step

Reasons

• Code written top to bottom. The most natural way to write is one line after another.
• Each step arrived at a different time. The pipeline started with one step; the others were appended to the end of the list.
• Parallelism looks complicated. asyncio.gather or Promise.all require thinking about dependencies and errors.
• Rate limits. Fear of hitting the provider’s limit makes the team avoid simultaneous calls.

Common trap: parallelizing what depends

After you see the staircase, the temptation is to throw everything into a gather. If field extraction needs the document type, the two in parallel return garbage or a race. The test is simple: does this call use the other call’s value? If yes, the queue is honest. If not, it is charging you for idle waiting.

Anthropic’s “Building effective agents” (Dec. 2024) recommends parallelizing independent tool calls. The point is not “always parallel”. It is “do not serialize out of habit”.

Cost of ignoring

• Experience. In voice and chat, every second of waiting raises abandonment.
• Timeouts. Integrations with time limits (webhooks, gateways) fail when the sum crosses the limit.
• Scale. Each new step makes latency linearly worse, so nobody wants to add anything.

The money cost does not change: the calls are the same. What changes is time.

Design	Time for the four analyses	Total until speech
Queue, as today	6.2 s	8.4 s
Four in parallel	2.4 s	4.6 s
Sentiment in code, the rest in parallel	2.4 s	4.6 s, one fewer call

The third row is no faster than the second in this example, because the customer summary still sets the pace. It is cheaper and simpler to keep.

Treatment

Refactoring	Choose it when
Parallelize Independent Steps	There are steps that do not depend on each other. They run at the same time, and latency becomes that of the slowest one. Targets: Sectioning and Map-Reduce.

If some steps do not even need a model (rule-based sentiment, lookup by id), replace them with code. See LLM Doing Deterministic Work. And consider a Model Cascade for steps that can use a smaller, faster model.

Before you parallelize, draw the graph: an arrow exists only when a step reads another step’s result. Whatever has no arrow between them goes in the same gather. Handle rate limits with a queue and a concurrency cap, not with a serial stack on purpose.

Check +20 XP on first trySentiment in your pipeline is a three-label classifier the team already has in code. What do you do with that step?

Replace the call with code and parallelize only what still needs a model

Parallelize sentiment with the other models and keep the call

Fold sentiment into the reply prompt

Payoff

• Latency close to that of the slowest step, not the sum of all.
• Room to add analyses without making things slower.
• An explicit dependency graph that documents the pipeline.

When to ignore

• Truly dependent steps. If each step uses the previous result, the queue is necessary. Reduce the number of steps or the size of each.
• Batch processing. If nobody waits for the answer (a nightly job), latency matters little. Parallelize across documents, not within each one.
• Tight rate limits. If parallelism will cause limit errors, deal with the limit first.

Is it this smell?

Check +20 XP on first tryA pipeline extracts text from a PDF, then classifies the document type, then extracts that type's fields. Each step uses the previous result. Is this a Serial Latency Stack?

Yes, the steps are in sequence

Yes, if each step takes more than a second

No

Check +20 XP on first tryA PR reviewer checks security, style, and tests, one aspect after another, each with its own call over the same diff. Is this a Serial Latency Stack?

Yes

No, each aspect needs its own call

Only if the diff is small

Check +20 XP on first tryAn agent takes 12 loop turns to solve a task, each waiting for the previous one. The task takes 40 seconds. Is this a Serial Latency Stack?

Yes

Not necessarily

Yes, and the fix is to parallelize the turns

Check +20 XP on first tryA provider webhook dies at 10 s. Your flow makes five queued calls of about 3 s each and blows the limit. After you parallelize, the slowest takes 3 s. What was the diagnosis?

The smell was a Serial Latency Stack, and parallelizing fits the timeout

The problem was the short timeout, so raise it to 20 s

Each 3 s call is already too slow, so replace the model first