Voice shouldn’t have its own idea of the task

Suppose three coding threads are running, one of them is waiting for permission to run a database migration, another has started rewriting tests you wanted left alone, and you are nowhere near a keyboard. “Tell the auth one to leave the tests alone” and “yes, go ahead” would be perfectly clear to a colleague who had been following along. They are much harder for software, and the difficulty has little to do with audio. Each sentence refers to work that already exists, whose state is changing while you speak, and the obvious way to build a voice assistant gives it a private understanding of that work, which starts going out of date as soon as it forms.

Our voice work began in the spring of 2026 as a dedicated voice-agent pipeline. It then moved toward sharing the coding agent’s state, with speech input and output attached, and by late July it had become a realtime model that controlled work through tools. I think that direction was right, for one reason: voice should be one more client of the thread operations that already exist, and it should hold nothing about the work that the thread itself does not hold.

Every verb is an existing operation

On the browser’s direct path, a person talked over WebRTC to a model on OpenAI’s Realtime API, using a short-lived client secret minted by our gateway. The audio did not pass through the gateway on that path. What reached it were tool calls, and a dispatcher turned each one into an operation the rest of the system already had.

Starting work created a worker session, set its goal and started a turn, with the turn marked as coming from voice. It also gave the thread a name you could actually say back later, and created the project if the one you named did not exist. Steering went to the same steer handler the conversation layer exposed to everything else. Stopping went to the same cancellation, with a reason recording that voice had asked, and stop was the only control verb the tool offered. Listing and describing threads read the same state, and the results came back as short summaries fit to be spoken.

Steering is a race

A spoken instruction takes time to say, and the work does not wait. By the time a sentence has been heard, understood and turned into a tool call, the turn it was aimed at may have finished, which is why steering needed more care than anything else on that list.

Two guards dealt with it. Steering was opt-in per agent, and an agent that did not declare support was treated as not steerable. An unguarded agent could treat the instruction as a concurrent turn, so for those agents the dispatcher stopped the active turn before delivering the redirect as a fresh one. For an agent that supported steering, the request carried the ID of the turn it expected to find. If that turn was no longer active, the handler answered 409, and voice delivered the instruction as a fresh turn. Only that conflict response permitted this fallback after a steering attempt; every other error was rethrown. The tempting version falls through to a new turn on any failure, and that version can deliver an instruction twice when a steer succeeded but its response was lost.

“Yes” doesn’t name anything

Speech also removes the identifiers other clients take for granted. A thread picker sends a thread ID. A person says “the auth one”, a phrase that may fit several thread titles. The resolver worked through a fixed order of interpretations and, when more than one thread fit, returned the ambiguity to the model so it could ask you, rather than picking the likeliest.

The same refusal applied to approvals, where a wrong guess costs more. If exactly one request was pending, “yes” answered it. If there were several, the tool listed up to four and asked which you meant. A question from the agent took a different branch from an approval: “yes” selected the first option, “no” the last, and the answer was keyed to that question’s field.

An approval, however it was phrased, ended up in the same server-side function as one made in the interface. That function reported a request that had already been resolved rather than applying it twice, and it re-evaluated the permission at that moment against layered policies in which a child scope cannot broaden a restrictive parent, downgrading an allow to a one-time deny if the policy now forbade the call. An approval then became a signed, one-use receipt bound to the tool, a digest of its input, the policy revision, the caller’s context, the decision and an expiry, and a shared insert-if-absent store consumed it across replicas, so it could not casually be replayed for a changed command, another thread or a newer policy. A coding agent wrote the main permission implementation with me as co-author; voice reused that shared system.

None of this makes speech recognition right. A misheard “yes” is still a yes. What the design does is confine the mistake to the single request that word could unambiguously refer to, and to what policy still allows.

Limits that stop at the gateway

The voice tool endpoints sat outside the JSON-RPC rate limiter that protected the rest of the gateway, yet they started real agent turns, so they needed a budget of their own: a sliding 10-second window allowing 30 calls in all and 6 of the costly ones. Only allowed calls counted against it. The budget was keyed on tenant and user rather than on the session, because a takeover could mint a new session up to six times a minute, and a new session ID must not arrive with a fresh budget.

Sessions lived in a durable registry backed by SQLite: one per user, with a new session taking over from the old, a per-tenant cap and a lifetime of about 15 minutes. But the browser held the call directly with the model provider, so those controls stopped at the gateway. An expired session lost access to our tools while an existing WebRTC call could continue, and the caps bounded how many sessions we brokered rather than what they spent. It is easy to assume a session registry is a kill switch, and this one wasn’t.

Deciding when to speak

Someone on a voice call cannot glance at a sidebar, so something has to decide what is worth interrupting them for. That job sat in a hub fed by the same stream and status broadcasts that drove the other clients. The default was narrow: threads that were blocked and threads that had finished. What came through could be pushed silently into the model’s context, spoken aloud or turned into an action in the interface, and the digest reached the model as data, with state and event markers. Slow questions got similar treatment. A deep question returned a handle immediately so the conversation did not stall, the answer was pushed when it was ready, and a watchdog bounded how long the conversation waited for an answer that never came.

Where channels meet

Telegram, which came earlier, in April and May, applied the same idea in a channel with nothing in common with realtime audio. Account, chat and topic together identified each conversation, so the plugin could find or create the right thread and send the reply back where the message came from, and two topics in one group stayed two threads. The turn then started through the same conversation handler as any other, under a service identity whose claims recorded the channel and account. The two hard problems from voice got simpler answers here, because someone typing a message can afford to be explicit and to try again. References were spelled out, with the verbs prompt, ask and tell taking a selector and a way to list threads and projects. The race was refused outright: if the selected thread already had an active turn, the bot said it was still working and asked you to try again when that finished, rather than steering or queueing.

Voice and Telegram shared an account context, which decided whose threads were on offer. Each channel bound to threads independently; the thread held the work they could both act on. That left one place to ask what was happening.

A realtime model will still remember what it was told a minute ago, and that memory can be stale, which no conversational model can avoid. What keeps it from becoming a second model of the task is that the memory has no authority. A steering request has to name a thread the resolver accepts and pass the current-turn guard, or take the explicit redirect path. An approval has to name a request that is still pending and pass policy evaluated at that moment. If its picture is out of date, it finds out from the answer.

Chris Chabot, November 2026