The cheapest models are the best reviewers
We assumed a stronger model should audit a weaker model's output. 144 runs across 6 models later, recall is roughly saturated across the cohort — but models discriminate sharply on precision, and the cheapest ones win.
The assumption
agntro's safety net is a reviewer-observer-adversary triad: after a cheap model (the "flash" pass) does the work, a stronger model audits it in three stances — as a reviewer (does it meet spec?), an observer (is the reasoning sound?), and an adversary (assume it's wrong, prove it).
The load-bearing assumption: a stronger model reliably catches under-scoped work. If that's false, the safety net is theater — optimistic triage ships silently.
The probe
Six models, five planted bugs + three clean implementations, three review
roles each. The bugs range from overt (a wrong HTTP method) to subtle (a
len(x)//4 tokenizer heuristic that's a flawed mental model). The
clean items test for false alarms — does the reviewer invent
defects in correct code?
144 runs total (6 models × 8 items × 3 roles). Per-model temperature, truncation-retry, and a fallback parse for robustness.
The results
| Model | Recall | Precision | False alarms | Avg cost | Read |
|---|---|---|---|---|---|
| GLM-5.2 | 80% | 100% | 0 | $0.0006–0.002 | cleanest; 0 FA anywhere |
| MiniMax-M3 | 80% | 100% | 0 | $0.0005–0.0007 | tied-cleanest, cheapest |
| deepseek-v4-pro | 80% | 100% (rev/obs), 80% (adv) | 1/3 (adv) | $0.0004–0.0011 | solid; adversary over-fires |
| kimi-k2.7-code | 80% | 100% (rev) | 1/1 (adv) | $0.0009 | 3 retry-error cases (undercount) |
| qwen3.7-plus | 60–80% | 67–100% | 1–2 | $0.0017–0.0033 | weakest reviewer (missed subtle) |
| grok-build-0.1 | 80% | 57% (adv) | 3/3 (adv) | $0.008–0.026 | 5–13× cost, adv hallucinates |
The four findings (honest)
1. The cheapest models are the best reviewers
GLM-5.2 and MiniMax-M3 — 80% recall, zero false alarms in every role, among the cheapest in the cohort. More disciplined than DeepSeek, Qwen, and grok at equal recall.
This is the counterintuitive one. We expected the heavy reasoners to dominate review. Instead, the models that cost fractions of a cent per run were the most disciplined — they flagged real bugs and stayed silent on clean code. The expensive models either matched them on recall (grok) or underperformed (qwen).
2. One bug was missed by every model, in every role
SUBTLE-3 — a conceptual len(text)//4 tokenizer
heuristic — was missed by all 6 models across all 3 roles. This is a
universal gap: conceptual flaws that look locally reasonable need a
different mechanism. Not a stronger reviewer — a test as the oracle.
This finding shaped the architecture: a reviewer catches overt defects, but conceptual soundness requires executable contracts. agntro now has a contract-verifier role that writes and runs pytest snippets from the spec — the reviewer's blind spot is exactly what tests catch.
3. Heavy reasoning doesn't pay off in review
grok-build-0.1 thinks hard — and its adversary role false-alarmed on all three clean items, at 5–13× the cost. More reasoning didn't mean more accuracy; it meant more skepticism, which over-fired. "Thinks hard, not more accurate" is the read.
4. The adversary role is where models diverge most
GLM and MiniMax stayed disciplined in the adversary stance (0 false alarms); DeepSeek, Qwen, and grok all false-alarmed. This is strong evidence that the adversary role should be armed selectively — only disciplined models can run it safely. The precision cost is model-dependent, not role-dependent.
What this means for the harness
Two architectural decisions follow directly from this probe:
- Role → model assignment is measured, not assumed. Reviewer/observer = GLM-5.2 or MiniMax-M3 (cheap, disciplined). Adversary = GLM/MiniMax only. Avoid grok for the safety net — it costs more and is less precise.
- Conceptual flaws need tests, not reviewers. The contract-verifier exists because SUBTLE-3 proved no reviewer catches a heuristic that looks reasonable but is structurally wrong. An executable contract doesn't care about plausibility — it runs.
Honest limitations
n=5 bugs is small. These results are directional — good enough to set initial role assignments, but they need a wider bug set to firm up. The recall ceiling (~80% overall, ~50% on subtle bugs) may rise with a more capable reviewer, but the precision inversion (cheap models more disciplined than expensive ones) is a pattern we haven't seen falsified yet.
The SUBTLE-3 miss is the most important negative result. It's not "we need a better reviewer" — it's "reviewing is the wrong mechanism for conceptual soundness." That's a stronger claim than a recall number, and it's held up across every subsequent run.