A user typed “a simple todo app.”

What came back had categories. Due dates. Priorities. Badges. A Calendar tab and a Settings tab. A profile screen. An attachments block.

Nothing crashed. No test failed. No agent reported an error. Every stage of the pipeline declared success, and the app it produced runs - it’s just not the app anybody asked for.

That is the hardest class of bug in AI software. There’s no stack trace, no failing assertion, and the only oracle for correctness is a human remembering what they typed twenty minutes ago. You cannot grep for it. You cannot write a unit test for it, because you don’t know in advance what the model will invent.

I spent a long time treating this as a prompt problem. It isn’t.

English is a terrible wire format

Naapy turns a description into a working mobile app - React Native frontend, Python backend - via a pipeline of agents. A product manager agent reads the request. An architect plans the files. A design agent draws the screens. Two developer agents write the code. A repair agent fixes what fails.

Every one of those handoffs was a paragraph of prose. Agent A wrote a description, Agent B read it and re-interpreted it, Agent C read that and re-interpreted again.

If you’ve ever shipped microservices, you already know how this ends. It’s a distributed system whose serialization format is English. Nobody would accept that between two HTTP services. We accepted it between five agents because the components happened to be language models, and language is what they take.

Every drift bug I chased looked like a prompt bug, so I fixed prompts, and the drift moved somewhere else. The prompts were never the problem. The channel was.

The standard I didn’t write

The fix I landed on isn’t mine. It’s an open standard called ProductSpec - a “product harness for AI-native software work: what to build, what not to build, how to prove completion, and when intent changes.”

It defines a structured format for product intent: Problem, Hypothesis, Product Summary, Scope, Acceptance Criteria, Success Metrics. Durable identifiers - REQ-001, AC-001, SM-001. And two companion artifacts, an Agent Run and a Decision Trace, for recording what an agent actually did and how a decision changed.

I want to be precise about which part is load-bearing, because it isn’t the six sections.

It’s the durable ID.

AC-001 is a join key. That’s all. But once every acceptance criterion has a stable identity, four things become possible that were flatly impossible before:

• you can lint whether every criterion was accounted for
• you can slice the spec and hand each agent only its own rows
• you can join criteria against evidence after the build
• you can fence the ones the user explicitly excluded

None of that works on a paragraph. All of it is trivial on an ID.

One implementation detail that matters more than it looks: the IDs are assigned by a deterministic compiler, never by the model. The PM agent returns a draft with no identifiers at all; a pure function assigns REQ-001, AC-001 by position and computes a content hash. Same intent in, same IDs and same hash out, every time. Let the model name its own keys and they drift between runs - and every join you built on top of them silently stops joining.

Making a human standard hold with no human in the loop

The standard is written for teams: people read the markdown, people notice when reality diverges. I needed it to hold at 3am with nobody watching. That meant four additions.

Agents read a compiled handoff, not the spec. Each planning agent gets a slice: its in-scope requirements with their stable IDs, pinned to a spec hash - and an explicit exclusion fence. The compiler deliberately does not paste the whole spec; excluded items appear only as terse “do not build” lines. Telling an agent what not to build turned out to matter roughly as much as telling it what to build.

A lint that blocks, before any code-generation spend. Every in-scope criterion must be accounted for: covered by a real file, or explicitly declared foundation-only or deferred. A covered claim with no implementing file is rejected. A criterion marked MUST cannot be deferred. A file that references a fenced-off requirement is rejected - it’s building scope the user ruled out. A reference to a criterion ID that doesn’t exist in the spec is rejected as a hallucination. All of it runs before a single token of code gets generated.

The approval gate pins the hash. When a user approves a design, the gate records the spec hash. On resume it re-reads the gate and the spec straight from the database and fails closed if it can’t verify the approved spec, or if the spec has moved underneath it.

And we stopped asking the AI whether it was done.

That last one is the one I’d put in front of a product manager. Previously, the summary shown to the user was written by a summarizer agent that was told what the other agents claimed. Agents claimed success. The summary said success.

Now there’s a deterministic reconciliation step that joins three independent sources - what the architect said it covered, what the platform is actually capable of, and what the final validation gate observed - into a status for each criterion, walking a ladder where the least-complete status the evidence supports wins:

blocked → deferred → foundation → generated → verified

A criterion is only verified if a file implements it and the validation gate passed. If code was written but a gate failed, it’s generated - not done. If the platform fundamentally can’t do it, it’s blocked, and that gets its own Decision Trace explaining why. The status block is prepended to the user-facing summary deterministically, so honesty doesn’t depend on the model choosing to be honest.

Completion became a measurement instead of a claim.

What I actually got: not fewer bugs, but visible ones

Here’s the part I didn’t expect, and it’s the real lesson.

The spec did not stop the agents from drifting. What it did was turn drift into something with a name and a location. Within two days of it landing, three specific bugs fell out - all of which had been in production the entire time, indistinguishable from “the AI is just flaky.”

The design agent was never constrained. All that lint and coverage machinery targeted the architect and the generated code. Nothing touched the design layer - which was still cheerfully inventing the entire todo-app surface. Worse, when I read its prompt, I found I had been instructing it to invent: a nudge suggesting “4-8 screens” and a settings screen. It was doing what I told it. The fix was a compiled per-screen scope brief injected into every mockup prompt (each screen prompt had previously seen a single line of purpose), plus a smallest-screen-set default - many real apps are one to three screens.

An impossible contract, generated and then re-generated. A todo run kept the frozen template’s useItems.ts hook while declaring a todos collection in its API contract. The hook has no backend to type against. No amount of repair can satisfy that - it’s not a bug, it’s a contradiction. The root cause was a trap in my own prompt: it said “rename or repurpose,” so the architect repurposed the file in place, and the self-check only demanded “a matching collection,” never the same resource word.

An error that never reached the repair loop. A listing app half-renamed the scaffold - hook renamed, orphaned file left behind, form screen still carrying its old filename while wired to a new route, two stale navigation calls. A scaffold-cleanup check consumed both repair iterations, and the actual type error was gated behind it: it only surfaced after the budget was gone. Fix: run the type-checker alongside the failing check rather than after it, filtering out errors from files that are about to be deleted anyway. A repair loop that shows you one error at a time can spend its entire budget without ever showing you the one that matters.

Three real bugs, all pre-existing, all invisible until there was a written-down definition of what the run was supposed to produce.

You cannot debug drift without a ruler. The spec is the ruler.

One tradeoff I took deliberately

When a user sends a follow-up - “make it more colorful” vs “add a due date” - the pipeline classifies it as a style tweak or a scope change. Style is cheap. Scope reruns planning.

Ambiguous requests now default to scope change, on purpose, because the two errors don’t cost the same. A wrong “style” silently mutates intent: the design regenerates against a stale plan and the thing the user asked for is quietly dropped, with no error anywhere. A wrong “scope change” costs one planning rerun and some money.

I’ll pay for reruns all day to never silently drop something a user asked for. Not every asymmetric tradeoff is this legible, but when one is, take it.

What it still can’t see

In the spirit of the thing: the reconciliation ladder is one claim deep. A criterion counts as covered when some file backs it - and part of “some file” is the architect’s own declaration of which files it wrote. A bare covered with no file at all is correctly rejected, but a declared path is still a claim, not proof. Verifying the file exists and contains something relevant to that criterion is the next honest step, and until it lands, verified means “the evidence chain is complete” rather than “I checked.”

Writing that limitation down is the cheapest thing in this entire post and probably the highest return. A known limit is a backlog item. An unknown one is an incident.

The part worth generalizing

For engineers: schemas at service boundaries, not prose. You already believe this. Agents are services. The fact that they speak English is not a reason to send them paragraphs.

For product managers: acceptance criteria with durable IDs are the single highest-leverage artifact you can produce right now - not because they’re rigorous, but because they’re the only form of intent that can be mechanically joined against what got built.

And for anyone deciding where the defensibility lives in an AI product: it is not the model. Everyone calls the same API. What compounds is the harness around it - knowing what was asked for, knowing what was built, and reporting the gap honestly even when the gap is embarrassing.

I didn’t invent the format for that. I found it, adopted it, and had to build the enforcement underneath it. That was still by far the cheapest part of the whole project.

The spec turned out to have a second use, too: once you have a written-down definition of what was asked for, you can finally measure whether a change to your pipeline made things better or worse. That’s the next post - and it goes badly before it goes well.