The Evaluation Metric Is Reading the Code

Level 1: Check the Homework

Imagine using three clever robots: one writes homework instructions, one completes the homework, and one checks it. Someone asks how you know the system worked, and you reply, “I read the homework.” That is funny because the complicated robot team still ends with the simplest sensible action: a person looks carefully at what they produced before trusting it.

Level 2: Review Beats Vibes

An LLM agent is a language model given instructions and tools to perform a task. Orchestration assigns specialized parts of one job to different agents. Here, one model plans, another writes, and the first model reviews. The arrangement resembles a small software team: architect, implementer, reviewer.

A metric is a measurable signal used to compare outcomes. For generated code, useful signals could include whether tests pass, how many defects reviewers find, how long the task takes, and how much the model calls cost. None of these alone proves quality. Tests only check the cases someone encoded; a cheap patch can be unmaintainable; a readable patch can still contain a boundary bug. That is why actual code review remains important.

For a newer developer, “I read the code” should not mean glancing at the diff until it feels respectable. It means asking concrete questions:

• Does the implementation match the requirement and the plan?
• Are errors, empty inputs, and unusual states handled?
• Do names and structure make the behavior understandable?
• Do tests cover the important paths rather than merely turn green?
• Is every changed line necessary?

The screenshot’s humor comes from the contrast in scale. The setup uses premium models, multiple roles, and explicit cost accounting; the verification technique is the same basic skill developers used before AI coding tools existed. New machinery has made reading more important, not obsolete.

Level 3: The Biological Judge

The screenshot builds an elaborate planner–coder–judge pipeline, asks for its evaluation methodology, and then collapses the entire stack into four words:

I read the code

That answer is funny because it is simultaneously primitive and rigorous. The first post describes Fable at xhigh acting as planner and architect, GPT 5.5 xhigh (subscription) implementing the work, and Fable returning as judge. It even compares “~few dollar” planning-and-judging calls with “typical $50+ full round trips.” After all that model routing, cost optimization, and orchestration, the final trust boundary is still a developer inspecting what changed.

Each role in this pipeline addresses a different failure mode. The planner converts a goal into constraints and an implementation strategy. The coder turns that plan into a patch. The judge searches for mismatches between intention and result. Using a separate pass—and possibly a different model family—can expose errors that self-review misses because the reviewer begins with a fresh framing. It does not guarantee independence, however: models can share blind spots, accept the same mistaken premise, or be persuaded by plausible-looking code. Three confident agents can still form a very efficient committee for approving the wrong abstraction.

Human code review supplies information that a generic model score may not capture. A maintainer can ask whether the patch fits local conventions, preserves an undocumented invariant, makes future debugging tolerable, and solves the actual problem rather than the easiest adjacent one. “Reading the code” means tracing data flow, checking boundaries, comparing the diff with the stated intent, and noticing suspicious complexity. It is not a replacement for tests, static analysis, type checking, security scans, or production telemetry; those tools observe different kinds of evidence. The strongest evaluation combines executable checks with informed inspection.

The question in the middle—“What metrics are you using to measure the success of this approach, and how do you handle potential errors or inconsistencies introduced by the AI tools?”—sounds like it expects a benchmark suite, rubric, or dashboard. Hashimoto’s deadpan reply refuses the premise that useful engineering judgment must arrive as a neat scalar. In day-to-day repository work, success may be qualitative: the patch is understandable, behavior is correct, edge cases are handled, and the maintainer is willing to own it. A metric can count passing tests or review findings, but it cannot automatically certify that a design belongs in this codebase.

There is a healthy warning inside the joke. AI-assisted development can increase output faster than it increases review capacity. If generation becomes cheap while careful reading remains scarce, the bottleneck moves from typing code to establishing trust. The screenshot’s workflow recognizes that by paying for an AI judge, then quietly reveals that accountability has not been delegated. The last evaluator is a human who can say, “No, this is clever nonsense,” without opening another API round trip.