Every draft passed the judges

“The small worker looked, at first, almost too plain to be interesting.” That was the opening of a technical post on my site in May 2026, and I asked for it to be rewritten, because nobody starts a practical write-up as if it were the lead of a cheap five-cent story. Agents kept producing sentences like it, so I did what engineers do with a recurring defect and wrote rules to prevent it.

By July the rules had become a system. There was a canonical voice guide with addons for blog posts, essays, documentation and social media, and its kill list reached about fifty entries before we consolidated it down to thirty-two: corrective definitions of the “X is not Y. It is Z.” kind, recap endings, engineered quotability, simulated witness and the rest. A pipeline of around 150 agents rewrote nine posts and eleven essays against it, judged each one, fixed what the judges found and judged it again, and a piece passed with a score of at least eight and no hard tells. By the end every piece passed. That sounds like a result, and it mostly describes the loop: a pipeline that fixes whatever its judges object to and then asks the same judges again is going to arrive at a pass, and the pass says that the prose now agrees with the list, which is a long way short of saying that anyone would want to read it.

Along the way, three of the things I objected to turned out to have come from the list itself. I had asked for technical posts to stay grounded in things I had built, run or observed, and seven of nine drafts opened with a date followed by “I”, which every reviewer approved because each was looking at one article. Another rule asked for evidence, so drafts started narrating their compliance with it; one informed the reader that tests had been written, that they had passed and that this came as no surprise, which spends a reader’s time on nothing at all. The third changed how I thought about the whole exercise. An old paragraph full of constants had been moved to the top of a post without its context, and the system responded by minting a permanent rule against opening a post with jargon. That was a clumsy edit, not a law of prose. I asked for every line of the guide to be reviewed for whether it was a general rule or a reaction to a single occurrence, because a style guide fitted too closely to its last few mistakes does as much damage as having none. We also ended up adding a pass to put texture back into prose the prohibitions had flattened, which is a sign you are maintaining a system rather than writing.

A banned phrase describes the surface of a sentence, and a score built from banned phrases inherits the limit. What I cared about was what the sentence was doing there, and neither the list nor the number had any way of saying that.

Which one I would rather read

A passing score only answers the question its rubric asks. I had been asking whether the prose obeyed the guide, when what mattered was whether I wanted to carry on reading. Those questions can produce different judgments of the same paragraph.

A better way of judging started to take shape in October, when I was designing a writing partner with agents: a collaborative application where an article develops through conversation, with the text taking shape on a page beside it. An application like that is only as good as the prose that comes out of it, so in the end it has to be judged by someone who will actually read it. In November I ran a proper experiment on prompts and effort levels, and part of the work was reading variants of the same passages and saying which I preferred and why.

Reading variants side by side is a different act from giving one passage a mark. A mark asks how a passage measures up against something written down beforehand, whereas a comparison asks only which of these you would rather read, and having to say why leaves a reason attached to a particular piece of text. One of those comparisons was the opening of a technical explanation, and the version I preferred had nothing showy in it. A judge hunting for good lines would walk straight past it, because none of its lines is worth quoting. What I liked was the way each sentence prepared the next, so that by the time the passage reached its point I had nearly got there myself, and the sentence that made the point would have been forgettable anywhere else.

That is what I mean by what a sentence is doing, and prohibition can’t get you there. Neither can a rubric, which has to name in advance what it is looking for, when what I was responding to was how this passage, on this subject, arranged what it had to say.

A preference has a scope

A preference belongs to the passage that produced it, which is the thing a scorecard is least equipped to record. Qualifiers are the smallest case of it. Any list of model habits would flag a hedge, yet in a technical sentence the hedge can be the only accurate thing to say, and a judge counting hedges can’t tell which kind it is looking at.

Length behaves the same way. In that technical explanation I liked the fuller versions, because the extra clauses were carrying reasoning, while in a narrative passage the extra words made a variant worse. Those judgments don’t contradict each other, but a rubric with a line for concision would have to pick a side, and an average of the two would describe neither.

Some of those verdicts were also provisional, the best of the versions in front of me, and at least one comparison never produced a clear winner. A ranking has nowhere to put either fact. Write a preference down without its passage and its reason and you have the next entry for the kill list, which is how one misplaced paragraph became a permanent rule in July.

What the judges are for

I still use model judges, though I’ve stopped asking them for the verdict. The November experiment kept its source corpus, its outputs, blind judgments of those outputs and, separately, an adjudication of the facts, and I think that separation is the most transferable part of it. Whether a passage is pleasant to read and whether it is true are different questions, and they need different evidence. A preference can’t settle a fact, and the pressure runs in one direction: a variant that improves a number or tidies a messy cause into a clean one may well read better for it, and a judge reading for pleasure will reward exactly the change that someone checking the facts has to refuse. Blinding does a similar job for taste, since whoever is judging, person or model, should be responding to the passage, and the knowledge that one variant came from the expensive setting is enough to change what you see.

A reader’s preference and a model’s score can sit side by side in the same record and still answer different questions, the first about what a particular person would rather read and the second about what a model has been told, or has come, to reward. Judging by model is also where the model’s taste becomes visible, because it isn’t mine. Put next to what I preferred, the judges sometimes liked the kind of flourish I would cut and grew suspicious of plain statements that any working engineer would let pass, and a review left to run drifted from real problems to smaller and smaller ones. A pass has to find something, so once the real defects have gone it carries on swapping adequate sentences for other adequate sentences, and a scorecard will record each swap as progress. None of that makes the judges useless, but somebody has to decide in advance whose judgment settles a disagreement, and for writing that goes out under my name that person is me.

A scorecard is useful for finding a passage worth looking at. It can’t tell me which passage I prefer, and a clean scan for known tells says nothing about who wrote something, whatever a detector’s score appears to imply.

What I kept

The decision rests on what I would rather read, with the facts adjudicated separately and the scores treated as leads, and it rests on outputs I can still open and read again.

The last of those matters more than it sounds, because one preference from an earlier round survives only as a note: I liked a variant enough to write down why, and by the time of the experiment its text had gone. There’s nothing to set beside the note and nothing anyone can score or reread, and the honest thing is to leave the hole where it is rather than reconstruct a passage from what I once said about it. I trust that hole more than most of those thirty-two rules, and it’s the best argument I know for keeping the outputs along with the verdicts, since a verdict with no text attached can only be believed.

Choosing a method settles how a draft gets made and leaves open whether any given sentence deserves to stay. That part is still my job.

Chris Chabot, November 2026