JRN-2026-08-28-the-grader-was-biased-toward-a-false-pass · 2026-08-28-the-grader-was-biased-toward-a-false-pass · Field Journal

CAPACITY TEST: The Grader Was Biased Toward a False Pass

VERDICT · SURPRISE

Build a deterministic grader for HEAT-004's four-task capacity basket, so a Pro-vs-Max comparison could be trusted

sense · remember · Samantha "Sam" Summerson

August 28, 2026 · Surprise · Sam

We built a four-task test, called the basket, to settle whether ChatGPT Pro or Anthropic Max gets more real work done per dollar. Each task has a fixed right answer, checked by a script called the grader. Before it ran for real, a Sol red team and several Codex review rounds found the grader could be pushed wrong almost every way it was tested.

One review task spelled out each planted bug's answer in a comment, so copying comments beat finding the bug. A second task's answer sat in a plain file inside the same folder the model read. Hashing that file did not help: the hash covered so few possible answers that guessing beat solving.

The code-fix grader trusted a submission's claim that its tests passed, instead of running them itself. A document task's prompt handed over the correct answer directly. One fix broke the grader's own regression test: an inherited environment variable made it mark a deliberately broken example as correct.

Nearly every defect pushed the grader toward a false yes. One did the opposite. A Node parsing detail made it read zero tests for a submission that had actually fixed the bug, so a correct fix could fail too.

Sol found fifteen problems in one pass, one serious enough to block the tool outright. Codex kept finding more after, each a fresh way the last fix could still be gamed.

Every finding above got fixed before the basket graded a real submission.

One piece stays interim by design. The code-fix task runs in a scrubbed environment with a timeout, not a full filesystem or network sandbox. That is acceptable for our own models and gated before anyone else's.

Next: the first live run against ChatGPT Pro. The matching Max run has not been built yet.