Monitoring should keep working when one of its own checks breaks

This used to be the loop at the centre of alerting in Ferrite’s observability engine:

pub fn evaluate_all(&self, now_ms: u64) -> ObserveResult<()> {
    let policies: Vec<AlertPolicy> = self.policies();
    for p in policies.iter().filter(|p| p.enabled) {
        self.evaluate_policy(p, now_ms)?;
    }
    Ok(())
}

Most Rust programmers have written this loop, and most reviewers would pass it, because ? is the idiomatic way of refusing to ignore an error. A background task calls it on an interval, logs a warning when it fails and tries again on the next tick.

The policies come back sorted by name. When one fails, the ? leaves the loop and every policy sorted after it goes unevaluated, and since a failure of this kind is rarely transient, the same thing happens on the next tick and the one after that. A policy called a.broken would switch off every alert that sorts after it for as long as it stayed installed, and the only sign would be a repeating warning in a log, while the incident list stayed empty. Which of your alerts still worked would be decided by the alphabet. This is the particular danger of a bug in monitoring, because a monitoring system that has stopped looks exactly like a system with nothing to report.

The engine came into Ferrite, the application platform I own, as an imported prototype, and I’ve been hardening it with agents. It’s proven locally and through sample dashboards rather than running as a deployed service, and what holds the behaviour below in place is a set of regression tests.

Why validation doesn’t settle it

The obvious objection is that a broken policy should never have been installed, and structural validation does refuse plenty at install time. It can’t refuse everything, because some failures depend on things it doesn’t consult. The regression test uses a percentile aligner on a gauge: the policy is well-formed, and the metrics engine only refuses it when a query runs and finds that percentiles need a distribution metric. A policy can legitimately be installed before its metric has a descriptor at all, in which case it reads as no data. The equivalent test for SLOs, which share this machinery and use multi-window burn-rate alerts, has a burn window spanning more one-second windows than a single query is allowed to read, a refusal you very much want the engine to make and one it makes on every tick. And a durable state key can always turn out to be unwritable.

So evaluation has to assume a check may fail, and the fix puts the failure boundary around each check rather than around the pass:

pub fn evaluate_all(&self, now_ms: u64) -> ObserveResult<()> {
    let policies: Vec<AlertPolicy> = self.policies();
    let mut first_failure = None;
    for p in policies.iter().filter(|p| p.enabled) {
        if let Err(e) = self.evaluate_policy(p, now_ms) {
            tracing::warn!(policy = %p.name, "alert policy evaluation failed: {e}");
            first_failure.get_or_insert(e);
        }
    }
    first_failure.map_or(Ok(()), Err)
}

This is different from swallowing the error. The log names the policy, the remaining policies still run, and the pass still returns the first failure when it’s finished, so the caller hears that something went wrong. The test installs a.broken next to a healthy policy, feeds the healthy one a breach over three ticks and asserts two things at once: every pass returns the engine’s refusal, and the healthy policy still reaches firing and sends exactly one notification. The SLO test does the same with an error burst and expects one incident, for the healthy SLO.

A failing policy still doesn’t page anyone. Its evaluation cursor stops advancing and it produces a warning on every tick, but no incident opens to say that a check has stopped running, so the fix stops one broken check from hiding the others and leaves the broken check itself depending on someone reading the log.

What one failure is allowed to cost

The evaluation loop is the plainest case of a question the rest of the engine has to answer too, which is how much one failure is allowed to cost.

A failed notification is allowed to cost the notification and nothing else. Policy state, the incident and the incident index commit atomically, and delivery comes afterwards on a best-effort basis. A failed delivery leaves the incident open with a failure event on it, a later evaluation tries again, and a marker absorbs the duplicates, so a webhook that’s down costs you a page and never the record that there was something to page about.

Overloaded ingestion is allowed to cost data, and that’s a deliberate trade. The engine is never on the data plane’s critical path: admission is a push onto a bounded buffer, a background flusher commits batches, and a full buffer sheds new points and counts them. The alternative is a monitoring system that can stall the application it’s watching, so the loss is accepted and the count is what keeps it visible, as a series you can query and a field in the diagnostic bundle. The overload test pushes 20,000 points at a 256-point buffer while a slow scrape runs alongside, and asserts that admitted plus shed equals 20,000, that the folded total equals exactly the admitted count, and that no call took as long as 250 milliseconds.

An expensive diagnostic request is allowed to cost completeness, as long as it says so. An agent or an operator asking what happened during an incident gets a bundle with the incident, the policy, the triggering datapoints, related series and logs with their source coordinates, and commands to reproduce the queries, all under a budget on bytes, log rows and metric windows. Queries underneath refuse with an estimate before doing the work if they’re too broad. The byte budget is enforced last: it drops the related logs, then series, then traces, then the repro commands, measuring again after each, records every drop in a truncated list, and refuses outright if what’s left is still over. A log query too broad for one bucket becomes a truncation entry rather than a failed bundle, and a related-series query that fails puts its error in that section. A caller also only sees what it could query directly. The log leg is confined to buckets the caller’s capability can read, so a budgeted evidence tool isn’t also a way round scoping.

When the answer is empty

The last version of the problem is the least obvious, because nothing fails at all. Log buckets index declared fields, and the query planner treats a missing posting as proof that no retained entry carries that value. That proof only holds if the index covers every retained entry under the declarations it’s read with. Declaring a field on a bucket that already had data used to leave the older entries out, so a filter on that field answered with an empty page while three matching entries sat on disk. A change to the indexed fields now rebuilds the index from the retained log before the new declaration becomes visible, under the index lock, so that every entry is indexed either by the rebuild or by the writer under the new declaration. Anyone reading that empty page would have concluded there was nothing there, which is the same failure as the alert that never fired.

The next function along

A failure boundary is easy to add to the loop in front of you and easy to miss in the one beside it. The SLO evaluator’s loop got its fix, but the SLO status read still stops at its first error, and the context API, which composes anomalies, error logs, error traces, open incidents and firing SLOs into one answer to “what happened here?”, calls it with a ?. An SLO the engine refuses can no longer silence its neighbours during evaluation, and it can still cost a caller that whole bundle, including all the evidence that had nothing to do with it. That failure is at least loud, since the caller gets a clear error, but it’s the same ?, one function along.

Chris Chabot, November 2026