# Week 5: Verification > The agent says it's done — but has anyone run it, and do you still understand it? Verification, and the cognitive debt that green checks don't measure # Welcome --- # Week 5 ## Verification --- ## what changed this week? [SemiAnalysis](https://newsletter.semianalysis.com/p/openai-jalapeno-better-than-nvidia) on OpenAI's **Jalapeño** inference chip, and tokens per _joule_ is the key metric ICML keynote by Arvind Narayanan [What will be left for us to work on?](https://icml.cc/virtual/2026/invited-talk/67274) --- ## today's beats never assume it works until it has run cognitive debt, agentic coding and you assignment 2 released --- ## demo: cold music-making then we leave it running and talk about what verifying actually means --- # never assume it works --- ## instruct, then verify > the skill is confidently **instructing** agents, then confidently > **verifying** the results > never assume that code generated by an LLM works until that code has been > executed [Willison](https://simonwillison.net/guides/agentic-engineering-patterns/); weeks 2–3 were the instructing half --- ## three patterns **[first run the tests](https://simonwillison.net/guides/agentic-engineering-patterns/first-run-the-tests/)**: a green baseline before you ask for anything **[agentic manual testing](https://simonwillison.net/guides/agentic-engineering-patterns/agentic-manual-testing/)**: the agent drives the running artefact and reports what it saw **[red/green TDD](https://simonwillison.net/guides/agentic-engineering-patterns/red-green-tdd/)**: whatever the drive-through found becomes a permanent test --- # **words**; what do they even mean? --- ## first run the tests ```text $ pnpm test ✓ spec/invariants.test.ts (4 tests) 812ms ✓ spec/tides.test.ts (3 tests) 141ms Test Files 2 passed (2) Tests 7 passed (7) ``` run them first, and a red result later comes from _your change_ --- ## agentic manual testing ```text $ agent-browser open http://localhost:4398/ $ agent-browser snapshot -i - heading "Hobart tides" [level=1, ref=e1] - combobox "Port" [expanded=false, ref=e2]: Hobart … - button "Show tides" [ref=e3] $ agent-browser click @e3 ✓ Done ``` the snapshot (trimmed) is the accessibility tree: the agent reads roles and names, then acts on a ref; `agent-browser errors` lists what the page threw ([agent-browser](https://github.com/vercel-labs/agent-browser)) --- ## a passing suite is not a working feature --- ## make it show its work > I tested the form and it submits correctly ```text $ curl -s -X POST localhost:4321/api/tide -d 'port=hobart' {"error":"unknown port"} ``` the agent's report can say it worked when nothing ran ask for the artefact: the screenshot, the console, the response body, the exit code --- ## manual testing finds it once so close it with a test: write it, watch it fail for the right reason, then fix ```ts // spec/page.test.ts, against the built page it("tides.js finds its elements", () => { for (const id of ["port", "show", "refresh"]) expect(doc.getElementById(id)).not.toBeNull(); }); ``` red now, green once markup and script agree, and it guards every later change --- the suite verified the artefact nothing verified **you** --- ## cognitive debt > when we lose track of how code written by our agents works we take on > **cognitive debt** from Margaret-Anne Storey, [_From Technical Debt to Cognitive and Intent Debt_](https://arxiv.org/abs/2603.22106) (2026) the test: could you change this file without asking the agent what it does? pay it down where it's worth it: the thing you'd have to debug an hour before a crit --- ## Ben uses… "simplify" requests maybe I should make it a slash command... --- ## find out before your pod does **say it out loud**: describe the change you just accepted without looking **make it examine you**: ask the agent to quiz you on the code it just wrote your pod runs the first one on you this week anyway --- ## the first remedy is one prompt ```text > walk me through src/tides.js in the order I'd need to read it to change the interpolation — entry point first, and what each part is there for ``` that's a **linear walkthrough**: a structured tour, in the order a reader needs rather than the order the files sit in point it at your agent's code, or at the library you're about to depend on --- ## you have already built the second one **interactive explanations**: an explorable, for when the tour still leaves you without intuition Willison's example is a word-cloud placement algorithm he couldn't follow from its description until he had an animation with a speed slider --- ## Ben uses… pristine output > pristine output means zero failures, errors, warnings, and backtraces another standing line in my `~/.claude/CLAUDE.md` because if there's junk in the log I miss the stuff I should care about --- ## yes, and… how does it get pushed further? weird ideas welcome --- ## drive it the way a marker will **both viewports**: week 2's device toolbar, still the phone that breaks **the keyboard**: tab through it without touching the mouse **the console open**: a page can look completely fine and be throwing on every interaction --- ## looks fine, throws anyway ```text GET /assets/tide-data.json 404 (Not Found) Uncaught (in promise) SyntaxError: Unexpected token '<', " design the university course you wish you could take, and build the website > that runs it worth 20%, due noon Monday 21 September — the first day back after the break no (separate) reflection _doc_ this time assignment 1 was one page with one interaction; this is twenty-odd pages that have to be consistent with each other --- ## Slop University > research-led, metrics-forward, and continuously improving 1st in the world for research throughput (_The World University Index_, 2026) --- ## what's fixed, what's yours **fixed**: the Slop identity, the content collections, and the generated API that the cohort-wide programs-and-courses page is built from **yours**: the pages, the decks, the components, the navigation, the visual treatment and every word your course code's last three digits arrive with the repo and belong to nobody else in the cohort; you pick the level digit --- ## make it niche narrow enough that no real university would run it, with enough depth to be taught for a full semester a course that would pass a curriculum committee unnoticed is too broad, and COMP4020 with the nouns find-and-replaced is too low-effort sincere, deadpan and satirical all work — it still has to be a real course website --- ## one idea, held for a semester [Calling Bullshit](https://callingbullshit.org/) — a point of view stated in the title and held all semester [How to Make (Almost) Anything](https://fab.cba.mit.edu/classes/863.25/) — a syllabus that is a list of machines, and a course that is the sum of them [CS 007](https://cs007.blog/) — personal finance for engineers: narrow audience, plain register, no filler --- ## your course site makes claims it claims assessment adding up to 100%, every teaching week inside the semester, no session citing a lecture you never wrote which of those deserve a check is this morning's lesson at site scale — `spec/` is submitted work, read the way `PROCESS.md` is an agent will write you twelve weeks in a minute, and no check tests whether they add up to a course anyone would take --- ## the repo lands next week provisioned early next week, then `/comp4020:start` clones it like every other deliverable a working Slop course site: the content model, a real deck, placeholder content and one data-integrity check — and placeholder artwork that `pnpm check:evidence` will not pass the [brief](https://comp.anu.edu.au/courses/comp4020-agentic-coding-studio/assessments/assignment-2/) is live now — read it before the repo shows up --- ## ass1 marks out tomorrow on canvas see [feedback and re-marks](https://comp.anu.edu.au/courses/comp4020-agentic-coding-studio/topics/assessment/#appeals) process --- too **vague**? --- ## the world is vague marking this course is the same problem you have every rubric line and every spec line now has to hold up against an agent that will satisfy it to the letter dealing with vagueness is our raison d'être --- ## the existential bit don't make it a load-bearing part of your personality --- > These best practices are still being figured out across our industry. I'm > still figuring them out myself. Simon Willison, Agentic Engineering Patterns --- ## three diagnoses, one symptom each change passes its checks, and the codebase still drifts **[Böckeler](https://martinfowler.com/articles/harness-engineering.html)**: a check can't tell a critical convention from a habit **[Ronacher](https://lucumr.pocoo.org/2026/7/13/the-tower-keeps-rising/)**: each change is reasonable in isolation; the coherence goes anyway **[Osmani](https://addyosmani.com/blog/agentic-engineering/)**: agents can implement, but can't judge whether it's the right design --- ## the hard case **[Gas Town](https://steve-yegge.medium.com/welcome-to-gas-town-4f25ee16dd04)**: Yegge's agent orchestrator — [75k lines in 17 days](https://maggieappleton.com/gastown), dozens of agents at once, thousands of dollars a month in API spend he describes it as complicated "not because I wanted it to be, but because I had to keep adding components until it was a self-sustaining machine" [Klabnik](https://steveklabnik.com/writing/how-to-think-about-gas-town/) calls it "_aggressively_ not rigorous" — and is glad someone is running the experiment (week 9 is where claims like these go under the microscope) --- ## a spec a machine can satisfy > when a measure becomes a target, it ceases to be a good measure [Goodhart's law](https://en.wikipedia.org/wiki/Goodhart%27s_law), in Marilyn Strathern's phrasing e.g. for the spec line "every page links home", a hidden `` on each page turns the test green your spec is the measure and your agent is the machine --- ## mark weighting **process 45%**: can a reader see how you got from the brief to the thing? **artefact 20%**: does the deployed thing work, and is it good? **response 35%**: is the response to the brief any good? --- humans making things for other **humans** --- ## you will be asked this > tell me what value humans still bring to the software development process — > examples if you like "I use Claude Code" is not an answer to it --- ## yes, and… how does it get pushed further? weird ideas welcome what will a stranger try first? --- ## C5: a game > build a tiny browser game — one mechanic is usually enough — obvious in ten > seconds, still interesting at five minutes, no tutorial in sight last week: no way to be played wrong; this week: rules, stakes and an ending one rule under a focused test, and one change that came from _playing_ the finished game --- ## deadline radar **Monday**/**Wednesday**: C5 live at its URL two hours before your session, with the reflection in the repo **Monday 31 August**: census date **noon, Monday 21 September**: assignment 2 — the Monday you come back from the break, so it's closer than it looks --- ## tl;dr never assume it works until it has run — and get the agent to show you, not tell you cognitive debt is the failure a green suite doesn't test for — explain it with the file shut, then walk it back one prompt at a time both viewports, the keyboard, the console — then hand it to a stranger --- ## ask your agent ask until you can explain these back: - **agentic manual testing**: how to drive the running artefact like a user, and what evidence to bring back - **red/green TDD**: turning one drive-through finding into a permanent test - **cognitive debt**: which part of your own prototype you could no longer explain to your pod - **linear walkthroughs**: get one for a file you inherited and didn't write --- # See you in the studio Source: \ Licence: [CC-BY-NC-SA-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)\ Cite as: Swift, B. (2026). _COMP4020/COMP8020 Agentic Coding Studio_ [Course materials]. ANU School of Computing. https://doi.org/10.5281/zenodo.21395760