Replace History with Summary

Problem

The session carries its entire history, which only grows, confuses the model, and makes every turn more expensive. This is the Context Rot smell.

The school’s WhatsApp math tutor agreed, in lesson 1: “hints only, never the finished answer”. In lesson 6 the student asked for a review sheet with an answer key, and the tutor allowed it as an exception for that sheet. In lesson 12 she asks for the equation of a new problem. The history holds both rules, several pages of wrong working, three schedule changes, and a note from her mother. The model returns the finished answer. Or, the next day, it refuses even the review sheet, because “the rule is hints only”.

The paper “Lost in the Middle” (Liu et al., 2023) documented the same kind of failure in long-context reading: models find information in the middle of a large context worse than information at the start or the end. A session that lasts weeks is that context, renewed every turn.

The numbers below are illustrative. Suppose a 1,500-token system prompt and about 200 tokens per turn (question plus answer). By lesson 12, with 40 turns, each call already sends about 9,500 tokens of history. By lesson 40, that passes 17,000, and the lesson-1 rule competes with everything that came after.

Solution

Replace old turns with a summary and keep the facts that must last in explicit state, outside the raw history. Send the model the state, the summary, and only the recent turns.

Before and after

 def answer(session, user_msg):
-    session.messages.append(user(user_msg))
-    reply = call_model(SYSTEM, session.messages)          # everything, always
+    session.messages.append(user(user_msg))
+    if tokens(session.messages) > COMPACT_AT:
+        old, recent = split(session.messages, keep_last=8)
+        session.summary = summarize(session.summary, old)  # rolling summary
+        session.facts.update(extract_facts(old))           # durable facts
+        session.messages = recent
+    context = [state_block(session.facts, session.summary), *session.messages]
+    reply = call_model(SYSTEM, context)
     session.messages.append(assistant(reply))
     return reply

Before: the entire history goes into every call.

flowchart LR
  U(["1. Message"]):::io --> H[("2. Entire history")]:::store
  H --> M(["3. Model"]):::llm
  M --> R(["4. Answer"]):::io

After: state, summary, and recent turns, with compaction off the main path.

flowchart LR
  U(["1. Message"]):::io --> H[("2. Recent turns")]:::store
  F[("3. Facts and summary")]:::store --> M(["4. Model"]):::llm
  H --> M
  M --> R(["5. Answer"]):::io
  H -->|"over the limit"| C(["6. Compact"]):::llm
  C --> F

1. The new message arrives.
2. Only the latest turns stay in raw form.
3. Durable facts (preferences, decisions, ids) and a summary of the rest live in a separate block.
4. The model receives the state block, the summary, and the recent turns.
5. The answer goes out and joins the recent turns.
6. When recent turns pass the limit, a compaction step updates the summary and facts and drops the old turns.

What the model now sees

After the lesson-12 compaction, the state block can be:

{
  "student": "Ana",
  "level": {"equations": "intermediate", "geometry": "beginner"},
  "format": "step-by-step hints",
  "solution_mode": "hints_only",
  "solution_mode_exceptions": [
    {"scope": "lesson 6 review sheet", "mode": "full answer key"}
  ]
}

The summary, in a few lines: “Lessons 1 to 11: quadratic equations, stuck on completing the square. Lesson 6: review sheet with answer key, agreed exception. Lesson 11: started geometry, Pythagorean theorem.” The eight recent turns stay whole. The current rule sits on a single line. The mother’s note and the wrong working from lesson 3 do not go in.

Why it works

Raw history mixes three things: facts that still hold, context that explains the moment, and noise that no longer matters. The model gets all three with equal weight. Separating them gives each a different weight: facts sit in a short block that is always present, the summary tells the story in a few lines, and the noise goes away.

Explicit state solves the case of the rule that changes. Instead of the model finding two contradictory rules at different points in the history, there is one line: solution_mode: hints_only, with the lesson-6 sheet exception beside it. Compaction updates that line, and you can read and test what it wrote.

Cost stays flat: context size stops depending on the session’s age. Recent turns have a bounded size; the facts block grows slowly; the summary has a word cap. The destination pattern is Context Compaction.

Safe mechanics

0. Measure. Build an eval from real long sessions (anonymized) with questions that depend on something said early: the hints rule, the account id, the delivery ZIP code. Record accuracy, cost per turn, and time to first token at the end of the sessions.
1. List your domain’s durable facts: ids, preferences, agreed rules, decisions, exceptions with scope. They become a schema, in the same spirit as Replace Free Text with Structured Output.
2. Write the compaction prompt asking for two outputs: updated facts (structured) and a short summary of what happened. Ask for the source (turn or lesson) of each fact change.
3. Pick the trigger: a token or turn count. Always keep the latest turns whole. A cap of 8 to 12 recent turns is a common starting point; tune it against the eval.
4. Run compaction off the critical path when you can, between turns or in the background. The user should not wait for the lesson-11 summary to get the lesson-12 hint.
5. Keep the full history outside the context, for auditing and for rebuilding the summary if it goes wrong.
6. Record every compaction on the trace, with facts before and after. If solution_mode changed without the student asking, you see it immediately.
7. Measure again. Accuracy on questions that depend on early turns should hold or rise; cost per turn at the end of sessions should drop.

Risks

• A summary that loses the right fact. An important detail disappears in compaction. Detect it with eval cases that ask about old facts, and by comparing facts before and after on the trace.
• A summary that makes things up. The compaction model may join things that were not said together (“the student asked for answer keys on every sheet”). Ask for facts with a reference to their source turn, and reject a fact with no source.
• Compounding error. Each summary builds on the previous one. A small drift in lesson 8 becomes a false rule in lesson 20. Keep the raw history and rebuild now and then from it, not only from the last summary.
• Invalidated cache. Changing the start of the context on every compaction can reset the prompt cache. Compact in blocks, not every turn, so the prefix (system prompt plus facts block) stays stable between compactions.

Common mistake: compacting only when the window fills

If you wait for the API to reject the context, the session is already expensive and the model is already lost in the middle. The trigger is a limit you choose, below the window, at the point where the eval still gets the old questions right. Compacting too early also hurts: you drop detail from the current conversation. That is why the latest turns stay whole.

Challenge

This compaction prompt produces nice summaries that lose information. What is it missing?

Summarize the conversation below in one paragraph, clearly and concisely.

You update the memory of a tutoring session.

Input: the current facts (JSON), the previous summary, and the new turns.

1. Update the facts: the student's level per topic, format preferences,
   and agreed rules. If a new rule has a limited scope (one sheet, one
   lesson), record the scope. Cite the source turn for each change.
2. Rewrite the summary in at most 120 words, folding in the previous one.
   Only what helps the next lesson: what was covered, where she got stuck.
3. Do not invent anything that is not in the turns.

Check your understanding