Ask a coding agent to add tests and it usually proposes a happy path, a few edge cases, and a build command. I kept approving that plan. The tests passed and the summary sounded finished, but important claims remained unchallenged. Could the wrong user read the object? Could an interrupted save mark broken state as complete? For ordinary code this may be enough. For permissions, files, external input, generated code, identity, money, or durable state, defects can ship behind a green build. I have not yet recorded this failure harming one of my projects. I built the stricter process first, and its own scoring table near the end of this post records the limited evidence.
In May I wrote severe-testing to make the agent do the part the polite plan skips: write down what observation would prove the claim false, then seriously try to produce it. The whole thing is one SKILL.md that both Codex and Claude Code read. The habit it enforces is much older than the repo: Karl Popper argued in the 1930s that a claim earns confidence by surviving serious attempts to refute it, and an agent won’t hold a test plan to that standard unless the instructions demand it.
Name the claim, then attack it
The first thing SKILL.md demands is the claim, written down before a single test is designed: what the code says is true about its valid input space, postconditions, invariants, performance budgets, security boundaries, and failure semantics. Then the agent lists the observations that would prove the claim false. Only after that do tests get written.
Take an agent tool: written out, its claim covers tool scope, prompt-injection resistance, data exfiltration, and whether an indirect instruction can influence a privileged action. Once those are on paper, “add some tests” has a target.
What counts as a finding
When the surface is broad, SKILL.md separates the work into authorisation, injection, concurrency, resource exhaustion, supply chain, and secrets or privacy. Each lens runs independently so one blind spot does not hide another. A finding must be an observable failure:
- wrong user can read object by direct ID
- symlink escape writes outside the workspace
- replayed request creates a duplicate charge
- markdown payload becomes script execution
- tool output carries a secret into the model prompt
- interrupted save leaves corrupt state marked complete
Each one also needs an oracle that can disagree with the implementation: a permission matrix, a schema validator, a state-machine invariant, a normalized diff, a trusted library, an accessibility tree, a mathematical identity. Generated code needs this the most, because a model can write a test that agrees with the bug, and then both pass. An independent oracle breaks that circularity.
The attacks are real, so SKILL.md is explicit about the blast radius: destructive, malicious, and failure-injection scenarios stay inside disposable fixtures, sandboxes, temporary directories, test tenants, or authorized staging environments, never against third-party systems, production systems, or real user data.
The score that gates the report
Before a finding is allowed into the report, I make the agent score it:
| Score | Meaning |
|---|---|
| 0 | The failure mode does not apply to this code. |
| 25 | Plausible risk, but no concrete path or reproducer. |
| 50 | Reproduced under contrived conditions. |
| 75 | Reliably reproduced under realistic conditions. |
| 100 | Demonstrated in the real runtime with captured evidence. |
Findings at 50 and above go in the report, with their evidence and the conditions that produced them. The 25s get listed separately as untested risk, so they stay on the record without being claimed as bugs. The 0s get dropped. For a fix to a previously demonstrated bug the bar is 75 or 100; the parenthetical in SKILL.md reads, “a regression test that does not actually fail on the broken code is not evidence.”
About 30 of SKILL.md’s 122 lines are instructions for what not to report: SSRF on a pure function with no network egress, prompt injection on code that never calls an LLM, “could race” on code with no shared mutable state, a 10 GB string into a function only ever called with a CLI flag bounded by argv limits. An agent told to attack will come back with scary-sounding findings either way, and if the imagined ones get presented as bugs a few times, people learn to skim the whole review.
The table gives severe-testing a score of 25 in my hands: plausible, with no published reproducer showing that it caught something a normal test plan missed. That is the same grade of evidence the skill forbids agents from reporting as a confirmed bug. I am recording it as an untested risk. The hypothesis is that a stricter harness produces fewer follow-up sessions after a green build. When I have evidence worth 75 points, I’ll write that post.
If the bet sounds worth taking, installing it is one symlink: the repo goes into ~/.codex/skills/ or ~/.claude/skills/, and the optional Codex list metadata lives in agents/openai.yaml.
One thing I can vouch for at 100, though:
$ grep -o 'adveserial\|vunerability' SKILL.md | sort | uniq -c
2 adveserial
1 vunerabilityBoth misspellings sit in the trigger list on line 3 of SKILL.md, so the skill still fires when the prompt is typed quickly. Those are mine.