Four ways my evals lied to me

My eval suite lied to me four times this month. Not the model. The harness.

1. The eval scored its own preprocessing. The text-cleanup eval reported 53% content loss on the cheapest config. Real loss: zero. Some fixtures only assert on body text, so the harness blanked the model's generated title before scoring, and every fact the model had placed in the title got counted as deleted. That number nearly blocked a 5x latency win.

2. The judge that never ran. I wrote an LLM-judge scorer for the quality that substring checks can't see. It sat in judge.ts, imported by nothing. Every fixture stayed green because only the trivial scorers ran. A scorer you forget to register doesn't error. It just quietly agrees with everything.

3. Billing faults scored as behavior. One run reported 40 of 64 fixtures failing. All of them were insufficient_quota. The trap inside the trap: as far as I can tell, OpenAI reserves a request's max cost up front, so real 23k-token turns got rejected while a tiny "say ok" health check passed fine.

4. Fixtures that could never fail. A typo'd key (expected instead of expect) was silently dropped, so the fixture asserted nothing and passed forever.

What I changed: strict Zod schemas plus a fixture linter (pnpm eval:lint) that rejects unknown keys, empty assertions, and no-op fixtures before any model call. It runs in CI, so nobody has to remember it. The runner now preflights quota with a call sized like a real turn. Three all-error fixtures in a row and it aborts, because a full run costs about $1.50 and a half-dead one manufactures plausible failures.

None of that catches an unregistered scorer. I still don't have a guard for number 2 beyond reading the score names in the output.

And one discipline no linter covers: a new fixture has to fail on purpose once. Revert the rule it guards, watch it go red, put the rule back. Then trust its green.