Cost and latency

The situation

The internal assistant passed its pilot with 20 people. The monthly bill was small and nobody looked closely. The week it opened to the whole company, the provider invoice tripled in four days, and “it is slow” complaints arrived along with it.

When the team opened the traces, they found three things: a 6,000-token system prompt sent with every message, a conversation history that never shrank, and four model calls made one after another, two of which depended on nothing.

None of these decisions was wrong during the pilot. They had simply never gone through a basic calculation that fits in a spreadsheet and should happen before the first deploy.

The concept

Every model call has two bills: how much it costs and how long it takes. Both depend on how many tokens go in, how many come out, and how many calls run in a row.

flowchart LR
  P["1. Input tokens"]:::tool --> M(["3. Model call"]):::llm
  H[("2. History and documents")]:::store --> P
  M --> O["4. Output tokens"]:::tool
  O --> C{"5. Cost and latency"}
  C --> T(["6. Total per task"]):::io

1. Input is everything in the call: system prompt, tools, history, documents, and the question.
2. History and documents are usually the largest share of input, and they grow without anyone noticing.
3. The chosen model sets the price per token and the speed.
4. Output is what the model writes, including internal reasoning when it is on.
5. Output costs several times more per token than input and is generated one token at a time.
6. What matters is the total per completed task, not per call.

The pilot hides item 6. Twenty people, short chats, one call per question: the bill looks like a rounding error. Across the company, history grows, the four calls in a row show up, and “cheap per call” becomes expensive per task.

The cost formula

call_cost  = input_tokens × input_price + output_tokens × output_price
task_cost  = sum of the task's calls (including retries and agent steps)
month_cost = task_cost × tasks_per_day × 30

Prices are on the provider’s page, in dollars per million tokens. In October 2026, Anthropic’s list had, for example, Claude Haiku 4.5 at US$1 input and US$5 output, and Claude Opus 5.5 at US$4 and US$20. Always check the official page before doing your own math: prices change.

An illustrative bill, using the Haiku row above. A 6,000-token system prompt, a 200-token question, 400 tokens of output. Input 6,200 × US$1 / million, output 400 × US$5 / million. That is fractions of a cent per call. Multiply by 5,000 questions a day and the month stops being a fraction. Now add history that, by turn 15, already extra 4,000 tokens: the same question got much more expensive with nobody changing the model.

Two factors move this math more than anything else:

• Prompt caching. When the start of the call repeats (system prompt, tools, a fixed document), the provider charges a fraction of the price for tokens read from cache. See Prompt Caching.
• Number of steps. An agent that makes ten calls resends the history ten times. Cost grows faster than the step count, because every turn pays again for what was already sent.

Blind retry lands in the same sum. Three attempts are not three times the price of the first if history already grew, and they still add three latencies. Classify the error before you repeat.

Check +20 XP on first tryThe pilot was cheap with 20 people. At launch, the same 6,000-token prompt went out on thousands of questions a day. Which reading of the bill is right?

The provider changed the price list over the weekend, because the prompt did not change

Cost per call was predictable; what was missing was multiplying by real volume and by history that grows

Output got more expensive, so the team should cut max_tokens and ignore input

The latency formula

The latency your user feels has two parts:

Part	What drives it	How to reduce it
Time to first token	Input size, provider queue, model	Smaller input, prompt caching, smaller model
Generation time	Number of output tokens and model speed	Shorter outputs, lean schema, streaming

Sequential calls add their latencies. Independent calls can run at the same time, and then total latency is that of the slowest one. That is the difference between the Serial Latency Stack smell and the Sectioning pattern.

Streaming does not shorten total generation time. It shortens the time until the user sees the first word. For JSON that code consumes at the end, streaming does not help the backend; a short schema does.

The internal assistant’s four calls are the textbook case. If two of them are “ticket summary” and “reply tone” and share no result, they can run in parallel. The other two, if one needs the previous output, stay in a line. The trace shows the sum. The spreadsheet should have shown it earlier.

Try it

This script measures the cost and latency of a real call and projects the month. It uses the API’s token counter, which does not charge for generation.

import time

import anthropic

client = anthropic.Anthropic()

MODEL = "claude-haiku-4-5"
PRICE_IN, PRICE_OUT = 1.00, 5.00   # US$ per million tokens; check the current list
SYSTEM = "You are the company's HR assistant. " * 200   # simulates a large prompt
QUESTION = "How many vacation days do I get per year?"

counted = client.messages.count_tokens(
    model=MODEL, system=SYSTEM, messages=[{"role": "user", "content": QUESTION}],
)
print("input tokens (counted up front):", counted.input_tokens)

start = time.perf_counter()
reply = client.messages.create(
    model=MODEL, max_tokens=300, system=SYSTEM,
    messages=[{"role": "user", "content": QUESTION}],
)
elapsed = time.perf_counter() - start

u = reply.usage
cost = u.input_tokens / 1e6 * PRICE_IN + u.output_tokens / 1e6 * PRICE_OUT
print(f"input {u.input_tokens}, output {u.output_tokens}, {elapsed:.2f}s")
print(f"cost per call: US$ {cost:.5f}")
print(f"5,000 questions a day, 30 days: US$ {cost * 5000 * 30:.2f}")

Then shrink SYSTEM to one line and run it again. Compare the latency and the monthly projection. The difference is the price of a bloated prompt.

Run a third time with internal reasoning on, if the model offers it. Billed output grows even when the user only sees the final sentence. The spreadsheet needs the column the trace already has: real output tokens, not visible tokens.

Check +20 XP on first tryThe script shows 0.8s with the large prompt and 0.4s with the one-line prompt, on the same question. The team wants streaming to 'fix the slowness'. What did the number already say?

That the slowness is the provider queue, so streaming is the right fix

That a large share of the wait is bloated input; streaming hides it, a prompt diet cuts it

That 0.8s is fine and not worth touching

What this changes

• Do the math before the deploy, per task, with the expected volume. A spreadsheet with tokens per call and calls per task prevents most surprises.
• Every pattern page in this catalog has a “Cost, latency, and failure modes” section. It says how many calls the pattern adds and where the bill shows up in the trace.
• With no spending limit, a loop or an abusive user drains the budget. See the No Cost Budget smell and the Cost Budget pattern.
• Not every task needs the largest model. A Model Cascade tries the cheap one first.
• A repeated question does not need processing again. See Missing Cache.
• You only control what you measure. Record tokens and cost on every trace span, as shown in Observability for AI.

Check your understanding

Check +20 XP on first tryFour calls of 600 ms each. Two are independent, the other two must run in sequence. What task latency do you put on the spreadsheet if you parallelize what you can?

600 ms, because in parallel everything becomes the slowest one

About 1.8 s: max of the independent pair plus the sum of the dependent pair

2.4 s still, because the provider serializes anyway

Check +20 XP on first tryAn agent solves a ticket in 8 steps at the median and 40 in the worst case, with no cost cap. What risk does a 'mean cost per ticket' spreadsheet hide?

None, if the median sits inside the monthly budget

The tail: the worst case resends history 40 times, and the mean dilutes that until the card complains

Only latency, because 40 steps are slow but cache zeros the cost

Check +20 XP on first tryThe team wants Opus on every HR assistant question 'so vacation answers never fail'. The eval shows Haiku is enough on 95% of cases. Which design matches cost to quality?

Haiku on everything, because a 5% error in HR is fine if the bill drops

Opus on everything, because the 95% eval may be optimistic

A cascade: Haiku first, Opus only when confidence or validation fails

Check +20 XP on first tryTwo users ask the same vacation policy on the same day. There is no cache. What is the second call paying?

Almost nothing, because the provider remembers the first answer

The whole input again, including the bloated prompt, and a new generation

Only the output, because input about a public policy should be free