What Happened When I Had an AI Agent Audit My Own Site

A Next.js site. Nine published posts. One new pillar waiting to go live.

I handed the whole thing to an agent and told it to audit.

Seven passes later I had a list of real defects and a working site. That part went how you would expect.

The bugs were not the interesting part.

The interesting part was which ones were findable at all. Every single real defect was caught by rendering a page, blocking a request, or inspecting a live output. None of them came from reading code and thinking hard about it.

And twice, the agent came back on its own and told me an earlier report it had given me was wrong.

That is the post. Not the audit. The measuring, and the correcting.

What does an AI agent actually find that a scan doesn't?

Things that only exist at render time. A crawler checks what the page returned. An agent that actually runs the page checks whether what it returned matches what it was supposed to return, and that gap is where the real defects live.

Three of them, in order of how much they surprised me.

The first was a layout shift on every article page. The avatar image squeezed to 37 pixels wide while the file was still loading, then snapped to 64 the moment the bytes landed, shoving everything beside it sideways. Width and height attributes are the standard fix for this. They were already there. On a flex item they do nothing.

The agent found it by blocking the image request and measuring the box before and after. Not by reading the component. The fix was one utility class, shrink-0.

The second was a component that had never rendered in production. A copy-to-clipboard block, sitting in the codebase, about to carry 14 prompts in the pillar post I was publishing that week.

The agent smoke tested it before using it. Two defects came back immediately. Blank lines disappeared from the copied text, so any multi-paragraph prompt would paste as a wall. And prose styles leaked into the block, which meant all 14 prompts would have rendered in 19px body text instead of 15px mono.

Reading the component showed neither problem. The code looked correct. It was correct, in isolation. It was wrong in the page.

The third was the one nobody would ever report. Every displayed date on the site rendered one day early. UTC midnight got parsed, then rendered in local time, and rolled backwards. The machine-readable output was correct the whole time, so no crawler, no validator, and no linter had anything to complain about. The only way to see it was to look at a rendered page and compare it to the source file.

Why does "measure, don't reason" matter more than the prompt?

Because reading code tells you what should happen and rendering tells you what does. Every real find in this project came from the second one. The prompt engineering mattered far less than whether the instruction had a measurable check attached to it.

Here is the strongest evidence. Standard, correct, widely documented advice was wrong twice in this project, and only measurement caught either one.

Width and height attributes prevent layout shift. True, except on a flex item, where I had one.

And Next.js metadataBase, which is documented to propagate URLs site-wide. It was configured. It was correct. It was completely inert, because all 27 URLs in the site were hardcoded absolute strings that ignored it. Every canonical tag on the site pointed at a hostname that redirected.

An agent reasoning from documentation would confirm both settings were right. Both settings were right. The site was still broken.

The same pattern showed up in the cheapest fix of the whole project. A 3.4 MB image, displayed at 64 by 64 pixels, on every single article page. Served payload after the fix was about 1 KB. Nothing in the source code says "this file is 3.4 MB." You have to look at what actually went over the wire.

So the instruction shape that worked was not clever. It just refused to accept a conclusion without the measurement that produced it.

The verification clause

For each issue you report, give me three things in this order:

1. The exact command or action you ran to observe it
2. The raw output that command produced
3. Your conclusion, stated separately from the evidence

If you did not run something that produced observable output, label the item UNVERIFIED and put it in a separate list. Do not mix inferred issues into the confirmed list.

That last line does more work than the rest of it combined. Agents are not usually lying to you. They are collapsing "this is probably true" and "I watched this happen" into the same confident sentence. Force them apart and the confident sentence becomes trustworthy.

What happens when the agent is wrong?

It tells you, if you built the job so that it finds out. Both times this happened were more useful to me than any single bug in the audit, because an unreported error is one you go on believing.

First one. The agent reported 12 raw links that needed converting to the proper component. I approved the work. On execution, the real number was 4.

The grep filter it had used to count them never actually applied, because the attribute it was matching sat on its own line in most of the files. The agent found this itself, mid-task, and said so plainly instead of quietly converting 4 and reporting 12 done.

Second one, and this is the better of the two. It reported one pre-existing lint error and set the expectation that everything else was clean. Later it came back and corrected itself. It had only linted the files it had touched. The real count was 11 errors inside the scope of the work and 23 across the site.

I did not catch that. I had no way to catch it. I was not going to independently lint a site I had just been told was clean.

It volunteered it.

The practical version of this: whatever you are running, add a step at the end that asks the agent to review its own earlier claims against what it learned while executing them. Errors surface during the work, not during the analysis. That is exactly when nobody is asking.

Why did the same context have to be re-established seven times?

Because the agent started every pass from zero and I was the only thing carrying memory between them. That is the whole cost. Not the tokens, not the time. The fact that the operator becomes the persistence layer.

It showed up three ways in this project.

An instruction I gave to add a page to the sitemap, which got reversed one pass later once the page's actual purpose surfaced. Correct instruction, wrong context. The agent executed it perfectly and I had to undo it.

An assumption about whether a page was indexed, stated with full confidence, carried forward across two passes and used as the basis for other decisions before an audit finally caught it. Nothing in the loop was checking assumptions against reality, so the wrong one just kept compounding.

And a factual error about years of experience, which propagated into a second live post. Not a reasoning failure. There was simply no stored fact to check the claim against, so nothing ever flagged it.

None of these are model failures. A smarter model does not fix any of them, because the information required was not in front of the model. It was in my head, or in a previous session, or in a file nobody thought to open. This is the cold-start problem, and it gets worse the more capable the agent gets, because a capable agent will act confidently on incomplete context instead of stalling.

This is the problem I am building Focus Pilot to close. A memory layer that holds the decisions, the corrections, and the facts about the business, so the next session starts where the last one ended instead of starting at zero. I am building it in public and it is not finished. But I did not pick the problem from a list. I picked it because I spent seven passes being the memory layer myself.

How should you actually run this?

Recon first, build second, always as two separate jobs. The single biggest improvement in this project was refusing to let the agent investigate and change things in the same breath.

Recon has one output, which is a report. No edits, no fixes, no helpful cleanup along the way. You read the report and you decide. Then you hand back a scoped instruction and it executes only that.

Recon pass, no changes

This is a read-only investigation. Do not edit, create, or delete any file.

Render the pages listed below in a real browser and inspect them. For each one, report:

- what you observed, with the command or action that produced it
- what you expected instead, and why
- what you could not check, and what would be required to check it

Rank findings by severity. At the end, list every assumption you made that you did not verify.

Then the second half, which is the part almost nobody adds.

Self-correction pass

Before you finish, go back through every claim you made earlier in this session.

For each one, state whether what you learned while executing the work confirms it, contradicts it, or leaves it unresolved. Where you reported a count or a scope, say exactly what you counted and what you excluded.

Report contradictions first, in plain language, even if it means correcting something I already approved.

Run those two around the actual work and you get a very different kind of report. Not a longer one. A more honest one. There is more on structuring these in AI agent workflows, and if this is your first time driving one of these, start with Claude Code for non-developers.

The part that actually matters

Agents are exceptional at execution and completely neutral about judgment.

The agent that found a layout shift by blocking an image request would have just as happily added the wrong page to my sitemap, because I told it to. It did, in fact. It carried a wrong assumption about index status forward for two passes without pausing, because nothing in the loop asked it to doubt itself.

It never got tired. It never skipped a page. It also never once asked whether the thing it was doing was worth doing.

So my job moved. I used to do the work. Now I design the verification.

Deciding what counts as proof, what gets measured instead of reasoned about, and what the agent has to report back before I trust a single line of it. That is the actual skill now, and it is not a technical one.

If you want to see how I run this day to day, I build in public inside The Sprint. You can get in here.