Long before I ever typed a prompt into an AI tool, I was standing at a whiteboard writing stories with people who don’t write code.
The format was always the same. The reason something matters, a persona, and a thing they want to do (why, who, what). Then the scenarios underneath, in Given-When-Then. I’d write them with business people, product people, testers, whoever was in the room. The point was to get the intent out of everyone’s heads and onto the wall where we could all see it and argue about it.
I’ve been doing that for about twenty years. If you’ve followed me for more than five minutes, you’ve probably read me say that stories are not requirements. Stories describe desired behavior change, in a language people can read. Requirements describe a checklist, usually in a language only the author can read.
When evaluation-driven development showed up in the class that prepares us for Stage 3, the third step in Improving’s AI Maturity model, my brain relaxed almost immediately. It sounded like the same idea pointed at a new target. If you can describe the behavior you want in a story, you can describe the behavior you want from a prompt the same way.
The rest of the class did not go as smoothly, and I’ll get to that. But the stories part, that part I already owned.
BDD on one side, TDD on the other
There’s a distinction that matters more now than it did before AI. I’ve written about it before.
When I write the story and its scenarios with the business, that’s Behavior-Driven Development. When I take those scenarios and turn them into a design to achieve that behavior, that’s Test-Driven Design. Same behavior, two different conversations. One is with the people who care about the outcome. The other is with the machine that has to verify it.
Here’s the story side, written the way I write it with a team:
The reason the distinction matters is where the thinking happens. The critical thinking about what “correct” means belongs in the story, with humans, at the board. The implementation of the check is a separate job, and it can be mechanical.
For years I’ve been turning stories into tests that don’t look like the tests most people write. Show a typical developer one of my Playwright tests and they’ll say, that doesn’t look like Playwright. It doesn’t. It looks like the conversation we had with the business. That’s on purpose.
I didn’t invent that approach for AI. It’s how I’ve worked for a long time, whenever I had any influence over how a team worked.
My first skill with evaluations
The other piece of ground I was standing on came from a tool I was just trying out.
I’d had this intent for a while to turn stories into graphic-novel style visualizations, so people could see the story instead of just reading it. I installed Claude Cowork and started explaining what I wanted. It asked questions first. What did I mean by a storyboard? What did a good page look like? What should it avoid?
Those questions turned out to be the whole point. By answering them, I was writing down my criteria without calling it that.
Cowork built the storyboard skill, then built a little visualizer that showed me sample outputs one at a time and asked me for my feedback. I went through a handful, saying what I liked and what I didn’t. When I said I was done, it used all of that feedback to refine the prompt and ran it again. Then it packaged the approved behavior as a skill.
I did not write the evaluation myself. What I did was judge the outputs, and that was enough for a skill that worked. I took notice of that experience: explain the intent, verify samples, let the tool package the result. Later I ported that skill to Devin. It was the first skill I had that came with evaluations attached, and it showed me the shape I wanted for everything else.
Grounded expectations
Here’s the principle underneath all of it: make the intent clear through clear stories, and then hold the output to that intent. Persona, acceptance criteria, background, scenarios. That’s the whole trick.
It worked for software because the stories described what success looked like in a way everyone could check. It works for prompts for the same reason. If I can write down what a good result looks like before I run anything, then I have something to measure against. Without that, “this looks better” is the only test I’ve got, and “better” moves around depending on my mood and how tired I am.
The stories are the grounded expectation. Everything else is plumbing.
The rubric experiment
After the self-assessment told me evaluation was my weak spot, I started small. I picked one of the skills I use the most, wrote a rubric for what a good output looked like, and used an LLM to grade outputs against that rubric.
I’d run the skill, hand the result to the judge, and look at the score. Then I’d change the skill and run it again. The numbers moved. Not always in the direction I wanted, but they moved, and that told me the change did something.
This was before I’d heard of Promptfoo or any evaluation framework. I just figured this was how you’d do it. Write down what good looks like, get something to score against it, change one thing, and see if the score moves.
It had real problems. The judge is non-deterministic, so the same output can score differently on two runs. I was still doing a lot of re-prompting by hand to keep the loop going. And it wasn’t a repeatable harness. If I wanted to re-run the whole thing a month later, I’d have to rebuild the steps from memory.
But it worked well enough to make progress. And progress was what I needed, because the thing I was about to point it at was a suite of more than 900 end-to-end tests. I wrote about that migration when it happened.
What the old habit bought me
The reason the rubric experiment got off the ground at all is that I already knew how to write down what good looks like. That’s the twenty years of stories doing the work. The tool was new. The habit underneath it wasn’t.
When people tell me evaluation-driven development feels like a foreign discipline bolted onto their workflow, I understand the reaction. It does look foreign, especially when the first thing you see is a YAML file. But the core of it, writing down what you expect and then checking whether you got it, is the same thing most good teams have been doing at the whiteboard for years.
The gap is that they’ve never pointed it at their own prompts.
Leave a Reply