The report that refused to pick a winner

I wrote two fake benchmark reports about a search indexer that does not exist. Benchmark Alpha report says Indexer A was the fastest implementation for the measured query mix; Benchmark Beta report says the opposite about the same indexer and the same query mix. I fed both to my own research pipeline. If it had crowned a winner, it would have lied for me, fluently, about software I made up.

The pipeline is Arcwell, my local assistant layer for research and long-running agent work. I forged the reports to test a specific failure mode. A simple pipeline gathers links, asks a model to summarise them, and stores the summary. Disagreement between sources can disappear inside fluent prose while the report becomes more confident. I have not observed that failure in a real report, so I built this fixture to test for it.

In June I put the two reports into my evidence store as source cards, Arcwell’s unit of stored evidence. Both were marked primary and high trust. I added them to a disposable research run and asked the pipeline to compile a report.

Two rows, yes and no

Extraction boiled the two cards down to this:

[
  {
    "subject": "Indexer A",
    "predicate": "fastest implementation",
    "object": "yes",
    "confidence": 0.8,
    "source_card_id": "src-b4c8ffa00eaff80f"
  },
  {
    "subject": "Indexer A",
    "predicate": "fastest implementation",
    "object": "no",
    "confidence": 0.8,
    "source_card_id": "src-26019cd58d71d268"
  }
]

Each row carries the ID of the card it came from, so the disagreement has an address a later pass can point at.

What the skeptic pass returned

Arcwell grouped the claims into a cluster for indexer a, then ran a skeptic pass over the linked sources, source roles, claims, and clusters. This was the step the whole forgery existed to reach.

I watched it at the terminal. What came back was a refusal:

{
  "ok": false,
  "contradictions": [
    {
      "severity": "error",
      "notes": "`Indexer A is the fastest implementation for the measured query mix.` conflicts with `Indexer A is not the fastest implementation for the measured query mix.`"
    }
  ],
  "findings": [
    {
      "severity": "error",
      "code": "structured_claim_contradiction",
      "message": "Structured claims appear to conflict and require resolution or caveat."
    }
  ]
}

The refusal comes from research_claims_conflict, a normalised comparison over structured rows. A conflict has the same subject and predicate but a different object. No model decides whether the extracted claims disagree at this stage. That judgement happened during extraction, and the ledger retains it.

Both callers live in backup_wiki_sources_store.rs: run_research_skeptic_pass checks source roles and contradictions, and compile_research_report uses the skeptic and audit results to decide whether the run counts as complete.

Contradiction is one error condition. A run also fails without a linked primary source or when a generated or model-answer card is linked as evidence (generated_source_card_linked). Models participate throughout the pipeline, so this check prevents their output from being stored as evidence.

I staged this contradiction; in the field it’s routine. Siyi Liu and colleagues, building the QACC dataset, found conflicting contexts in about a quarter of open-domain queries, and ConflictScore, a metric that checks whether an answer acknowledges the conflict, detects overconfident claims across domains.

What the report wrote

The compiled report came out marked incomplete:

status: incomplete
source cards: 2
primary source cards: 2
extracted claims: 2
clusters: 1
contradictions: 1

Neither claim won; both sit in the evidence section, with the audit finding attached:

structured_claim_contradiction:
Structured claims appear to conflict and require resolution or caveat.

Three corpus warnings made it through too:

thin_source_corpus: linked_sources=2
thin_primary_source_coverage: primary_cards=2
measurement_claim_without_document_anchor

Those warnings stop a two-card fixture from being presented as broad research. In a real run they also provide the next tasks, beginning with more benchmark sources. Different workloads, dates, versions, or measurement practices could explain the disagreement, and two cards cannot distinguish them. Disagreeing sources remain in the corpus as new ones arrive. The design document forbids triage from discarding a source “merely because it disagrees with the emerging narrative.”

At size, the same path has gone the other way: STATUS.md preserves two live-test runs that compiled completed and audit-clean, one on code-execution safety with 129 linked sources and 31 source cards, one on the London AI ecosystem with 133 and 32.

This run establishes one result: when two primary, high-trust sources contradict each other on the same claim, the skeptic pass returns ok: false and the report remains incomplete. It does not test broad research quality, scheduling, or live provider behaviour. I still want a live run where I expected agreement and the skeptic pass caught a real conflict. If your pipeline has done that, I’d like to hear what it found.