Interlaced eucalypt canopies with birds threading between the crowns, in a three-ink risograph print

Evaluating agent output

The agent says it's done. Verifying that claim means running the thing, and keeping enough understanding of it to answer for what shipped

Simon Willison puts the job in one line: the skill is confidently instructing agents, then confidently verifying the results. Weeks 2 and 3 were the instructing half — the brief, the spec, the plan, the harness that pushes back. This page is the other half.

The names on this page are borrowed from his Agentic Engineering Patterns, a catalogue of practice that is still being added to. This page uses his terms so that you can find the same techniques elsewhere.

On this page: running it · agentic manual testing · cognitive debt · what checks settle · what no sensor reaches

Never assume it works until it has run

That rule is Willison’s, and it is short enough to keep. A passing suite makes a narrower claim than it appears to: the paths somebody thought to write down still behave. The same commit can crash on startup, render nothing, or work only on the one route the test walked.

Three patterns fit together as a loop:

The third one matters because manual testing finds a bug once, and nothing stops it returning on the next change. Promoting the finding into a test is what turns a one-off observation into permanent backpressure.

Agentic manual testing

Somebody has to open the thing and use it, and the pattern is that the somebody can be the agent. The tactic changes with the shape of what you built:

The prompt that gets the widest coverage is the vaguest one: asking the agent to explore makes it try many angles at once, which is what you want from a pass whose job is finding what nobody predicted.

Driving the browser

For the course prototypes, week 3 already introduced the tool: agent-browser arrived as the sensor that shows the agent what it actually rendered. It appears in Willison’s chapter alongside Playwright; both are established options for browser automation.

Week 3 used it defensively, to close one loop when the agent claimed a layout was fixed. The wider pass uses the same instrument deliberately: drive the artefact the way a marker will, at both marking viewports, with the keyboard rather than the mouse and the console open. Ask for the screenshots and look at them yourself.

Browser tests have an old reputation for flakiness, and it is deserved — a trivial HTML edit cascades into breakage. What changed is who pays the maintenance: an agent can handle the repetitive updates.

Ask for the artefact

The failure worth designing against is an agent writing up what did not happen. “I tested the form and it submits correctly” is prose, and prose is cheap.

Willison’s answer is a tool that records a testing session as a document, where each command and its real output are captured together rather than narrated; he is explicit that this exists to prevent a test report from substituting for evidence. That tool is his own practice, still spreading. The principle behind it is the one to take, and the course already marks you on it: a claim the captured output corroborates counts; a claim it does not reach is just a sentence.

In practice, ask for the screenshot, the console output, the response body or the exit code, and paste those into PROCESS.md when they document a decision.

Cognitive debt

Here is the failure green checks cannot see: when we lose track of how code written by our agents works, we take on cognitive debt. The term in this sense is Margaret-Anne Storey’s, and Willison’s write-up is how most of the field met it. Storey traces the idea to Peter Naur, who argued in 1985 that the people who built a program hold its theory, and that rebuilding that theory from the documentation alone is “strictly impossible”.

It doesn’t always matter. Nobody needs a deep model of the part that reads rows and emits JSON. It starts mattering when the core of the application is a black box, because then you can’t plan against it, can’t judge whether a change is safe, and can’t tell a plausible fix from a real one. It behaves like the other kind of debt: velocity looks fine for a while, then stalls. So the practical rule is to pay it down where you’ll go again — the core, the part you’ll change next week, the part you’d have to debug an hour before a crit — and to leave it standing on the leaf that reads rows and emits JSON.

An agent can hand you a passing suite over code you could not describe to your pod. The suite has verified the artefact; your understanding still needs its own check.

The course has been asking about this since week 1 without naming it. The crit asks whether you can account for how your agent produced the work, and the assessment weights process above artefact.

Noticing it

The expensive kind is the debt you haven’t noticed. Two quick checks expose it. First, describe the change you just accepted with the file shut — to your pod, to a rubber duck, out loud on the walk home. Note the point where you need to reopen the file. Second, ask the agent to quiz you on the code it just wrote. Both take about a minute, and the crit uses the first whether you’ve rehearsed it or not.

Paying it down

You pay down cognitive debt by improving your understanding, and two named techniques cost one prompt each:

Ask for a walkthrough in reading order rather than by feature name:

walk me through src/tides.js in the order I’d need to read it to change the interpolation — entry point first, and what each part is there for

You have already built the second one. Assignment 1 asked for an interactive explainer, which is this pattern pointed at an idea. Pointing it at code — your agent’s, or a library you are about to depend on — is the same skill.

What checks settle

Types, linting and focused tests settle the machine-checkable fraction, and CI runs them away from your machine so the settling happens without you. That is what makes them worth wiring in: every question the machine answers for free is attention you get back for the ones it cannot.

The specialised sensors narrow the gap a little further, and each one is bounded evidence — a clean accessibility scan leaves the keyboard experience unexamined.

What stays yours is whether the result was worth building, whether the code is one you’d want to own, and whether you still understand it.

What no sensor reaches

Some qualities never become a red check. Whether the architecture is right, whether the naming holds, whether twelve weeks of a course add up to one idea: you steer these before the agent generates rather than measuring them after. In the vocabulary of backpressure they are guide work, and the guides are your CLAUDE.md, your spec and your plan.

Birgitta Böckeler splits the checking itself in two. Computational controls (tests, linters, type checkers) are deterministic and fast; inferential controls, an LLM asked to judge, are slower and non-deterministic. Neither reaches the tacit part: an agent does not evaluate whether a 300-line function is aesthetically objectionable, and cannot tell which of your conventions is essential and which is habit.

Storey calls undocumented purpose intent debt. It makes the system’s purpose hard to recover from either the artefacts or the code. In Assignment 2, record the course’s promises so that both you and the agent can check the implementation against them.

At the crit

Your pod provides fast independent review. Four people who have never seen your prototype can find in ten seconds what you stopped seeing three days ago. The crit mark also requires you to provide that review for them.

Bring the artefact running and demonstrate it on screen rather than only saying that it works.