VOICE AI ENGINEERING · PART 2

The Anatomy of a Correct Playbook (and the Agent Script That Runs It)

Previously in this series: How We Built an IDE for Voice AI Agents

In our last post, we talked about why a voice agent's conversation logic shouldn't live buried inside its audio pipeline — why "a conversation is text in, text out," and why pulling that logic into a readable Playbook, separate from the audio seam, finally made building a voice agent feel like building software instead of performing a soundcheck.

That post was about the shape of the solution. This one is about what happens after you've adopted that shape and start actually writing Playbooks for real calls — dozens of them, across hospitality bookings, real-estate qualification, tee-time reservations, lead follow-ups. It turns out a Playbook being readable doesn't automatically make it correct. We've shipped stuck loops, off-script wandering, silent hangups, and echoed speech — not because the architecture was wrong, but because a flowchart in YAML has its own ways of lying to you, and the script running that flowchart has its own ways of quietly breaking it.

This is the postmortem-turned-guide we wish we'd had before writing our first Playbook.

Everything below runs itself, in Claude Code and Codex

The deterministic-exit rule, the uses_kb scoping trap, the interrupt-wording pattern, the Director-model tradeoff — all of it is now a packaged playbook-builder skill.

15 failure patterns drawn from real production incidents, a schema reference checked against the installed library (not remembered from an example), and the host-side code guard for the one bug class YAML alone can't close. Point it at a client spec or an existing Playbook and it drafts or reviews against this exact list.

/plugin marketplace add ankitai-s/superdialog-playbook-skill

Repo · Install for Codex

A Playbook is a flowchart, not a prompt

The mental model is simple: a Playbook is a flowchart for a phone call. Each checkpoint is one step — "get the name," "explain pricing," "confirm the booking." The agent is always sitting at exactly one checkpoint. Every turn, it speaks (from that checkpoint's guidance or a fixed say_verbatim line), the caller replies, and an advance_when rule decides whether to stay or move on. Separately, interrupts are checked on every single turn regardless of checkpoint — "caller says wrong number" jumps straight to a wrong-number checkpoint from anywhere in the flow.

Writing a Playbook is drawing this flowchart. The trouble is that a flowchart with one broken edge doesn't fail loudly — it fails as a call that quietly loops forever, and nothing in the logs says "error." It says the flowchart is working exactly as written; the flowchart was just wrong.

Rule #1: every slot-gated checkpoint needs a deterministic exit

Here's the bug that taught us this the hard way. A checkpoint's only job was to capture one slot — "can the caller talk right now?" — and its only advance_when rule was judged by the LLM: "Caller agrees to continue." On a clean "yes," it worked. On real replies — "uh-huh," "haan bolo," "yes it's fine" — the model had to do two things in one verdict: extract the slot and decide to advance. It kept choosing not to. The agent re-greeted the caller, forever. Nothing errored. The logs read "verdict: STAY" turn after turn, which looks exactly like a model being appropriately cautious — until you notice it's the same verdict on turn 40.

An identical flow, written for a different client, never hit this bug — because every checkpoint there had one extra rule, evaluated before the LLM was ever asked anything:

advance_when:
- when: "slots.can_talk_now is not None"   # deterministic — fires first, every turn
  judge: expr
  to: main.check_name
- when: Caller agrees to continue.          # llm fallback / extra branches
  to: main.check_name
  requires: [can_talk_now]

The engine evaluates judge: expr rules before asking the model anything at all. The moment the slot is filled, it advances — no matter what the LLM would have decided. Same engine, same speech-to-text, same underlying model; the only difference was this one line. If a slot-capture checkpoint's only exit depends on an LLM choosing to leave, you have shipped a coin flip disguised as a rule, and eventually it will land on "stay" forever.

The one real exception

If a checkpoint needs to speak something once the slot fills — a wrap-up line, an explanation that depends on what was just captured — don't add the expr rule. It will fire the same turn the slot is set, before that line is ever spoken, and the caller never hears it. Split it: one checkpoint that captures (with the expr rule), one that speaks and then advances on an LLM rule of its own. And always add a turn_budget loop-guard regardless — the backstop for the turn the LLM genuinely never fires on.

Knowledge bases wander if you let them

The second bug was less obvious to spot and more obvious in hindsight. A checkpoint's entire job was "get the family size." It had a knowledge base attached — uses_kb: true — so it could answer a stray pricing question if one came up. Then someone added an unrelated FAQ to that same knowledge base, for a different checkpoint. Because uses_kb feeds the entire blob to any checkpoint that has it turned on — not just the section relevant to that step — the family-size checkpoint suddenly had unrelated facts sitting in its context. The next time a caller gave an ambiguous answer, the model reached for the more interesting content it could see and wandered off into that topic instead of just re-asking for family size. Fifteen turns of a checkpoint talking about anything except its own job.

The fix isn't "don't use knowledge bases" — open-ended lookup genuinely needs them. It's: turn uses_kb on only where a checkpoint's job is actually open-ended (pricing, objection-handling), and on every checkpoint that does use it, say explicitly in its guidance: stay only on X, do not bring up Y here. A knowledge base is a floor plan with every door unlocked; you still have to tell the agent which room it's standing in.

Two brains, one script: the Talker and the Director

This is the part that doesn't show up anywhere in the YAML, and it's the part most likely to quietly wreck a call that reads perfectly on paper.

Every turn runs through two different model calls, not one:

• Talker — Generates what the caller actually hears. Reads guidance / say_verbatim and speaks.
• Director — The judge. Evaluates every advance_when rule, extracts slots, decides where the flowchart goes next. The caller never hears it.

On checkpoints marked gate: hard, the Talker waits on the Director before it's allowed to speak at all. That wait has a budget — a barrier timeout, then a hold timeout, stacked — and once that budget runs out the caller hears a generic filler line ("one moment, let me confirm that…") instead of the actual answer.

Gate: hard — the Talker's wait budget: barrier_timeout · 3s, then hold_timeout · 4s (7s total).

Past 7 seconds combined, the caller hears the filler line instead of the real answer — regardless of whether the Director was one second or one minute from finishing.

We had this budget set to a combined seven seconds. It was fine for months, until we pointed the Director at a slower model, and every hard-gated checkpoint started stalling into filler lines and hold music, on every call, because the Director simply couldn't finish its silent judgment inside the window the Talker was willing to wait.

Production bug — the twist

The instructive part of that story wasn't the timeout — it was what happened next. We swapped the Director to a faster provider and immediately got a worse symptom: every single turn returned DEGRADED, detail=llm_error, the checkpoint frozen at greeting, the Talker hallucinating a fake booking on top of a Director that was silently failing every call.

It looked like a model-capability problem. It was an expired API key — the Director's exception handler, by design, swallows the underlying error and returns a generic degraded verdict rather than crashing the call, which is the right call for a live phone conversation and the wrong thing to debug against without stepping outside the framework and reproducing the raw provider call directly.

The lesson: a Director failure and a Director misconfiguration look identical from the transcript. Reproduce against the raw model client before you trust either as the root cause.

Why is the Director's model choice invisible until it isn't? The caller only ever experiences Talker latency and Talker text. Director latency is pure overhead stacked in front of every hard-gated response, and nobody notices it's the bottleneck until they go looking for why the agent keeps saying "bear with me."

One playbook, two models, and it depends who's running it

We recently added a way to declare both models directly in the Playbook itself:

llm:
  provider: custom
  model: lk-inference/google/gemma-4-31b-it   # Talker
  director:
    provider: groq
    model: llama-3.3-70b-versatile             # Director (optional — falls
                                                 # back to the Talker's model
                                                 # if omitted)

Here's the part that surprised even us the first time we wired it up: whether this field does anything depends entirely on who's running the file. If you're using the open-source superdialog package directly — a playground, a bot you're wiring up yourself, anyone who isn't running our specific voice stack — calling Playbook.resolve_llm_providers() reads this block, live-validates each model against its provider, and hands you back two ready-to-use model clients. One field, both roles, done.

Our own production stack doesn't call that method at all. Our Talker model comes from a per-agent database configuration, not from this YAML — so on our calls, the top-level provider/model pair is inert. It has to be present and valid for the file to be a complete, portable Playbook (anyone else running the same file needs it), but it changes nothing about what our Talker actually runs on. The director field, though, is fully live on our stack too — it's the actual, current source of truth for which model judges every turn of every call, and it's the one field in this block worth spending real attention on if you're chasing hard-gate latency.

The takeaway generalizes past our own stack: a Playbook is meant to be portable, and portability means the same file can be interpreted differently by different hosts. Know which parts of your declared config your specific runtime actually reads back before you assume changing a YAML value changed a running call.

The host matters as much as the file

Every bug above lives in the YAML. The rest of what we learned lives in the code that executes the YAML — and it turns out a perfectly-written Playbook can still produce a broken call if the runtime underneath it isn't equally careful. A short list of what that looked like in practice:

• Echoed, duplicated speech. Speculative generation and committed generation both reached text-to-speech independently for the same turn, and our dedup guard only covered one code path. The caller heard every line twice.
• Callers left dangling after the call "ended." Removing a participant from the room doesn't reliably send a SIP hangup signal down the trunk — the caller's phone stayed connected after the agent had already torn everything else down.
• Silence dressed up as success. A "never leave the caller in dead air" fallback was wired to fire whenever no audio chunks were generated — except a legitimate line could be generated and then get deduplicated away, leaving zero spoken audio while the chunk list was technically non-empty. The fallback checked the wrong flag and never fired.
• A JWT with a ten-minute shelf life. A gateway token was minted once at call setup and cached as a static credential. Any call that ran past ten minutes started failing every subsequent Talker turn with an authentication error it had no way to recover from mid-call.
• A second question, asked while the first was still being judged. The Director takes real, measurable time to return a verdict. A caller who asked a new, different question in that window had it silently dropped — the turn-lock meant to prevent a duplicate re-trigger was dropping every concurrent utterance, not just literal repeats.
• "Are you there?" ending the call. An interrupt meant to catch genuine dead air fired on a bare "Hello?" check-in right after the agent repeated itself — the caller was actively talking, and the call still ended on "no response detected."

None of these are Playbook-authoring mistakes. They're the kind of bug that only shows up once a Playbook is correct enough that the runtime becomes the remaining source of error — which is exactly the argument for treating the engine executing your flowchart with the same rigor as the flowchart itself.

A checklist, for the next Playbook you write

• Every slot-gated checkpoint has a judge: expr rule as its first advance_when entry, checking slots.<name> is not None — not just an LLM rule hoping the model both extracts and advances in one shot.
• Every checkpoint has an explicit turn_budget, and non-linear steps have an explicit on_failure — the backstop for the turn nothing else fires on.
• uses_kb: true only where a checkpoint's job is genuinely open-ended, paired with an explicit "stay only on X" line in its own guidance.
• Every advance_when.to and interrupts[].to is journey-qualified (main.checkpoint_id, not a bare id) — a bad reference fails to load; the mistakes that don't fail to load are the ones worth double-checking.
• interrupts are worded defensively — spell out what should not count, not just what should. A silence interrupt that doesn't exclude a genuine reply will eventually fire on one.
• If you're declaring llm.director for latency reasons, confirm which model is actually judging your hard-gated checkpoints in the logs — don't assume the top-level llm.model field changed anything for the Talker until you've checked what your specific host reads back.
• Load the file through the real engine before trusting it. A clean load only proves the schema is valid — it proves nothing about a stray bare Jinja name silently evaluating to false, or a judge: expr string referencing a name that was never a real slot.

Appendix: every field in the schema, in one place

Everything above was the story of what breaks. This is the reference we keep next to it — every field the Playbook format defines, grouped by the object it lives on. If you're writing a Playbook right now, this is what to have open in the other tab. Fields tagged (real bug) are the ones implicated in a story above.

Playbook — top level

llm — LLMConfig / LLMRoleConfig

guidelines — GuidelineConfig

Checkpoint

Slot (inside a checkpoint's slots:)

advance_when — one exit rule

Interrupt (checked every turn, any checkpoint)

dispatch entry (call-start routing only)

tools / pipelines / handlers / middleware — engineering-owned

A typo here can break the call entirely or leak data to the wrong endpoint — don't touch without engineering.

policies — Policies / SilencePolicy

Separating the conversation from the audio pipeline made a Playbook something you could read like software. It didn't make that software correct by default — flowcharts still have unreachable states, race conditions between two model calls per turn, and runtimes with their own failure modes underneath. Writing a Playbook that survives contact with a real caller means treating it exactly like the software it now genuinely is: with rules for control flow, a threat model for the parts that fail silently, and a habit of testing the actual load path instead of eyeballing the YAML.

That's the discipline. The IDE gave us the readable artifact. This is what it takes to make the artifact right.