Counting interventions instead of adding watchdogs

Counting why people step in to an agent platform, letting threads own goals judged by observed progress, and running recovery code in shadow before it may act.

October 20264 min readAgent systemsRuntimeEvaluation

Small chalk marks paired with dull gold marks descend the left side of deep olive, one pair broken by grey scrapes.
On this page
  1. Counting interventions and root causes
  2. Why the external watchdog failed
  3. Moving goal ownership into the platform
  4. Shipping recovery code in shadow mode

Five long-running agent threads were working on a hosted coding-agent platform I built to carry software projects across days of autonomous execution, including its own development. Even after those agents had tools to inspect logs, fix bugs, and deploy changes, each thread still needed a person to poke, restart, or repair it roughly every half hour.

When an autonomous system still demands that much babysitting, the usual engineering reflexes work against you. Patching whatever annoyed you last optimizes for memorable incidents rather than frequent ones, while collapsing reliability into an autonomy score hides what kind of human help was actually needed. Adding another external watchdog is worse, because it bolts unverified recovery code onto a runtime whose lifecycle model is incomplete. Needing an outside monitor at all is the critical bug. Reducing the remaining human dependencies took two practices instead: counting specific interventions deterministically with root causes under predicates that separate agent actions from human ones, and keeping any recovery path that steers, stops, or restarts work in shadow mode until it proves clean against live traffic.

Counting interventions and root causes

Instead of guessing where operator time went, I ran a coordinated research project on the platform to audit its own operation. Deterministic predicates over project histories and logs from the gateway, the service that dispatches turns and manages thread lifecycles, counted each day’s owner and operator interventions. Each intervention mapped to a root cause, a concrete fix, and a verification check running in minutes.

Instrumenting that count surfaced an immediate attribution bug. Administrative API calls did not preserve agent identity, so an agent recovering by restarting its own project coordinator was billed to the owner’s intervention tally. Preserving caller identity on those calls became one of the autonomy fixes itself.

The audit produced a ranked list of 204 causes. About half were ordinary platform defects; prompting accounted for only a dozen. Worker agents pulled from the top of that list, and as 77 fixes went live, daily operator interventions fell from around a hundred to one or two while nudge alerts to the owner stopped. No day reached zero interventions, and because a live platform does not hold still for a controlled experiment, how much of that drop the fixes themselves caused remains unknown.

Why the external watchdog failed

Before fixing the thread lifecycle, I had tried the instinctive safeguard: a stall supervisor meant to detect wedged threads and restart them. In principle, an independent watchdog is a reasonable backstop when long-running agents block on hung tool calls or timer faults. This one, however, had been verified only in unit tests. Once deployed, it never received activity events from the live gateway, judged every busy thread silent after ten minutes, and ended all five active turns at once. Recovery code that relies on telemetry never exercised against live traffic turns isolated faults into correlated outages while leaving the underlying lifecycle gap intact.

Moving goal ownership into the platform

Most agent runtimes scope a standalone thread to a single turn, so any turn that ends early or hits a transient fault leaves work idle until someone intervenes. Here, a thread owns its goal instead. When a turn ends unfinished, the gateway schedules the next turn automatically, continuing until the agent explicitly declares the goal done or blocked, or until five consecutive turns produce no progress.

Judging progress from observed state is what keeps automatic continuation from burning tokens in a loop. Output tokens do not count, because a model can always say it is still working. Instead, the gateway looks for a new workspace digest or a successful tool output the thread has not produced before. Each fruitless turn increases the retry backoff until the fifth halts the goal with a single alert, and a person’s Stop always means stop, overriding any scheduled continuation.

Long-lived threads also create a deployment bottleneck when a thread stays pinned to the release its turn started on while fixes ship underneath it. To let fixes reach busy threads without discarding in-flight work, the gateway migrates a lagging thread onto the new release at a model-call boundary while preserving its active turn and execution sandbox. Adopting the kept sandbox takes about two seconds where it had taken up to ten minutes, so platform patches take effect mid-goal.

Shipping recovery code in shadow mode

The stall supervisor failure became a standing rule in the repository: any code path that steers, stops, restarts, or re-sends work ships in shadow mode first, logging what it would do against live traffic while acting on nothing until an overnight run records zero false actions. Enabling it also requires an end-to-end test against a real local gateway that kills the gateway process mid-goal to verify recovery across restarts.

Goal continuation was the first mechanism through that gate. On its initial night in shadow mode, it logged three false turns against live threads, all of which were fixed before it was switched on.

Chris Chabot, October 2026