Replace Prompt Rule with Graph Edge

Problem

The prompt describes the flow (“first ask for X, then Y; if Z, do W”) and the model skips, repeats, or reorders steps. These are the Rule Stacking and Prompt as State Machine smells.

The marketplace returns bot asks for the order number, checks the 30-day window, requires a photo if the item is electronic, and only then generates the label. All of that lives in one system-prompt paragraph. On Tuesday a customer wrote “order 4521, headphones crackling, photo attached”. The model skipped the delivery date, generated the label, and the product was already 47 days old. The rule was in the text. The text is not an if.

The question this refactoring answers: how do you make the mandatory step happen every time, without turning the prompt into an unreadable state machine? The nearby destination is a chain or routing in code.

Solution

Turn each “IF this, do that” in the prompt into an explicit transition in code: a state, a verifiable condition, and the next step. The model keeps doing what only it can do (understand and write), but the graph decides the path.

Before and after

-SYSTEM = """You handle exchange requests.
-First ask for the order number.
-Then confirm the order was delivered less than 30 days ago.
-If it has been more than 30 days, explain that an exchange is not possible.
-If the product is electronic, ask for a photo of the defect before continuing.
-Only generate the return label after everything is confirmed."""
+def next_step(state):
+    if not state.order_id:
+        return ask("order_id")                       # 1
+    order = orders.get(state.order_id)
+    if (today() - order.delivered_at).days > 30:
+        return reply("outside_window")               # 2
+    if order.category == "electronics" and not state.defect_photo:
+        return ask("defect_photo")                   # 3
+    return create_return_label(order)                # 4
+
+# the model only extracts data from the message and phrases the question naturally
+state.update(llm_parse(EXTRACT_PROMPT, message, schema=ExchangeFields))
+step = next_step(state)
+answer = llm(PHRASE_PROMPT, step)

Before: the flow is a paragraph the model has to follow.

flowchart LR
  M(["1. Message"]):::io --> P(["2. Prompt with the whole script"]):::llm
  P --> R(["3. Reply, sometimes with skipped steps"]):::io

After: the flow is a graph in code; the model works at the edges.

flowchart TD
  M(["Message"]):::io --> X(["Extract fields"]):::llm
  X --> S1{"1. Has order number?"}
  S1 -->|no| A1(["Ask for the number"]):::llm
  S1 -->|yes| S2{"2. Within 30 days?"}
  S2 -->|no| A2(["Explain it is not possible"]):::llm
  S2 -->|yes| S3{"3. Electronic without photo?"}
  S3 -->|yes| A3(["Ask for photo"]):::llm
  S3 -->|no| L["4. Generate label"]:::tool

1. Without the order number, the only possible action is asking for it.
2. The 30-day window is calculated by code, from the real delivery date.
3. The photo requirement for electronics is an edge, not a sentence.
4. The label is only generated when code reaches this node.

Worked example

Tuesday’s message: “order 4521, headphones crackling, photo attached”.

In the before, the model receives the script and the message together. Sometimes it extracts 4521, accepts the photo, and generates the label without looking at delivered_at. There is no test that proves the window was checked: the check lives inside the model.

In the after, the extractor returns:

{ "order_id": "4521", "defect_photo": "cid:foto1", "symptom": "crackling" }

next_step loads the order, computes 47 days, and returns reply("outside_window"). The phrasing model only explains, in a courteous tone, that the window has closed. The label is not a tool at its disposal. There is no way to skip it, because the path to it does not exist when the date overflows.

A unit test covers the edge: order 31 days old, photo present, expect outside_window, never create_return_label. That test did not exist while the rule was a paragraph.

Why it works

A model follows flow instructions most of the time, and most of the time is not enough for mandatory steps. An edge in code has no “sometimes”: if the condition is not true, the next step does not happen.

The division of labor also gets clearer. The model is good at understanding messy messages (“the headphones I bought last week, order 4521, arrived crackling”) and at writing natural questions. Code is good at comparing dates, checking categories, and enforcing order. Each does what it does best.

And the flow becomes something you can read, test, and draw. Every edge gets a unit test. The leftover prompt talks about tone and format, not about “first” and “only when”.

Anthropic’s “Building effective agents” post (December 2024) describes workflows with steps defined in code, instead of letting the model orchestrate everything. This refactoring is that move, applied to a paragraph of rules.

Safe mechanics

0. Measure. Build an eval from real conversations, including out-of-order answers and attempts to skip steps. Record the rate of correctly completed flows, and especially the rate of skipped mandatory steps.
1. List the prompt’s flow rules: every sentence with “first”, “then”, “if”, “only when”.
2. Draw the graph: states, conditions, and transitions. Check it with the process owner. The 30-day window is a business rule, not a prompt guess.
3. Define explicit state (fields collected, validated, pending).
4. Write the extractor: a call that reads the message and fills state fields as structured output. It does not pick the next step.
5. Implement transitions in code, with a test for each edge, including the ones that refuse (window expired, category without a photo).
6. Shrink the reply prompt to tone and format. What to ask comes from the graph.
7. Measure again. Correctly completed flows should rise; skipped steps should drop to zero.

What stays in the prompt

Tone, vocabulary, language, how to treat an angry customer. The graph does not do that well. “Be courteous and use the first name” stays. “Only generate the label after everything is confirmed” leaves.

Risks

• Robotic conversation. A rigid graph may ignore it when the customer answers two things at once. Extract every field from every message, not only what was asked.
• Cases outside the graph. The customer asks something no state anticipates (“do you accept Pix on the refund?”). Pix is Brazil’s instant-payment rail. Have a path for general questions and for a person.
• A graph that grows out of control. Every exception becomes an edge. If the graph gets huge, maybe the process needs simplifying, not encoding.
• Lost state. Long conversations, or ones resumed days later, need persisted state. See Move to Durable Execution.

Challenge

Which rules in this scheduling prompt should become edges, and which can stay in the prompt?

You book appointments.
Only offer slots within opening hours (8 a.m. to 6 p.m.).
Always confirm the insurance plan before showing slots.
If the plan is not accepted, offer a private appointment.
Be courteous and use the patient's first name.

state: { patient, insurance?, insurance_accepted?, slot? }
if not insurance: ask(insurance)
elif not insurance_accepted: offer(private)
elif not slot: show(calendar.slots(open_hours_only=True))
else: confirm(slot)
prompt: "Be courteous and use the patient's first name."

Check your understanding