Quality is hard to find when it’s subjective, and most of what we build is subjective. You can argue all day about whether a feature feels right. You can’t argue about whether the date is in the wrong format.
So the question I’ve been sitting with is how we optimize our way to determinism: how do we get better at spotting the parts of a problem we can determine, so we can verify them quickly, automate the check, and spend our attention on the parts we genuinely can’t?
That feels like a project of its own, and I think it belongs before almost everything else we do with AI agents.
Where the criteria come from
The starting point is a conversation. Let people talk freely, with prompts that guide them so they feel fully heard. Don’t ask them to fill out a checklist yet. Let them describe what they want and what would bother them.
Then use AI to read through what they said and sort it. Which statements can be checked deterministically, by a script? Which ones need a judgment call, where an LLM acts as judge? The first list becomes automated checks. The second list becomes the thing humans and judges focus on.
My Write Stories skill is close to the starting point here, but I suspect I need something before it: a step that captures the intent behind the stories, so the stories are targeted enough to produce evaluation criteria in the first place. A story that says “users can search” doesn’t give you much to check. A story that says what searching is for gives you a lot.
What to do with the rest
Once you have the deterministic checks, the rest of the work gets clearer. Generate the evals from the criteria. Build the skill that satisfies them. Then define the adversarial agents that attack the output.
Those agents need their own treatment: what criteria they apply, what inputs they get, what they produce, and where their findings go. Does a finding send the work back for rework, or to a specialist? That routing decision is where much of this falls apart today.
The home inspection analogy helps here. The inspector doesn’t need the full history of how the house was built. They walk through, look for what the builder’s own checklist missed, and write a report. Someone else decides what to do with it.
Why this is worth the effort
If a check is deterministic, people can verify it in seconds and we can automate it. That frees attention for the subjective parts, where there’s no shortcut and human judgment is the whole point.
I’d rather have an argument about what makes a feature good than an argument about whether the form accepts a date. Getting the second kind out of the way leaves room for the first.
That’s the project I want to work on: strategies for surfacing which criteria are deterministic and which aren’t, and a repeatable way to turn the first group into checks and the second group into something we can evaluate deliberately.
Leave a Reply