The pushback I get most often on automation goes something like this: writing the evaluation criteria by hand is necessary. You can’t skip that thinking.
I agree with every word of it. The criteria have to be thought through by a human. That’s not negotiable, and I’d argue against anyone who says otherwise.
The part I keep pushing on is where that thinking happens.
At the board, not in the file
When I write stories with a team, I’m standing at a whiteboard whenever possible. I write every sentence by hand. I argue about the persona. I ask what “done” means for this scenario and watch people disagree until they agree. That’s the critical thinking, and it’s some of the most valuable time I spend on a project.
When I hear people say they don’t do BDD because it doesn’t fit their pipeline, I think they’ve misread what it’s for. The value isn’t in the file format. It’s in the conversation. I don’t want to move that conversation into a YAML file. I want to have it at the board, with the people who care about the outcome, and then let something else handle the transcription.
That’s what I mean when I say I moved where the critical thinking happens. I didn’t remove it. I put it where it has always belonged, with humans, in a story, and I automated the part that never needed me: turning a clear story into structured checks.
The criteria are still written by hand. They’re just written in English, by a person, at a whiteboard, instead of typed into a config file by someone who’s tired.
What deserves evaluation
Not everything does.
The self-assessment question that stuck with me was about which part of my AI-assisted work is hardest to evaluate. When I went through my skills looking for an answer, I noticed the ones I care about most aren’t the ones that are easy to measure.
The skills that help me improve a conversation with stakeholders take priority. Anything that helps me understand what someone is actually asking for, turn it into something we can all see, and move a decision forward. Those are the ones I want evaluated first, because those are the ones where a bad output costs real time and trust. I need to build the right thing first, and then I think about building it right.
The skill that checks whether there’s a semicolon at the end of a line? I don’t care. The braces? I don’t care. Let the linters figure that out. Those are deterministic checks, and I can write them in a second and forget about them.
The judgment checks are where the value is. And the judgment checks are also where the evaluation usually stops short of the experience I actually care about. My checks can confirm that a story has the right structure. They can’t confirm that it would land with the person I’m writing it for. That gap is worth knowing about, because knowing where my evaluation stops is more useful than pretending it covers everything.
The journals are the reason I could tell this story
I want to close on the practice underneath all of it, because it’s the thing that made this series possible.
For a while now, I’ve gotten into the habit of putting an activity log on anything I’m working on. When I finish with an agent, I say, log that. So I have a record of the decisions I made, the challenges I ran into, the things I tried that didn’t work. It’s not a formal process. It’s a habit, and it takes almost no effort.
Everything I write, every talk I give, every journal entry I dictate on the drive to the office, gets pushed into a personal knowledge system I call my digital brain. It’s a retrieval layer over my own writing, with an MCP server on top so I can query it.
This talk came out of that. Two days before I gave it, I asked an agent to help me build a one-hour talk about my road to Stage 3. It pulled from my journals, my blog posts, my certification submission. It reminded me of things I’d forgotten. When I half-remembered something and got it wrong, it corrected me and cited the entry where I’d written the real version.
The stories in this series came from that material. The Cypress migration, the treadmill submission, the PRD frustration, the wrong unit, all of it was sitting in my notes because I’d bothered to write it down (or voice it out) when it happened. I didn’t reconstruct any of this solely from memory. I read it back.
Whatever form it takes for you, a journal, a log, a folder of notes, do it. It compounds in ways you can’t see while you’re doing it. I gave a talk that didn’t exist until the night before, and it only worked because the raw material had been accumulating for years.
What’s next
Stage 3 was one evaluated agent. Stage 4 takes stories and pushes them through planning, implementation, tests, review, and a human approval step. It’s the build side. Stage 5 is the side in front of it: a discovery workflow that produces the stories in the first place. Talk to stakeholders, look at the business model, figure out the customer segments and the value proposition, sketch the experience, build a prototype, put it in front of people, take the transcript, refine it, and ask whether this is what they meant.
Once that’s cleared, it hands off to the harness I already built. Discovery in, working software out, with a human deciding at every gate that matters.
The road to Stage 3 certification in Improving’s AI Maturity model was messier than it looked from the outside. I suspect the next one will be too, and I’m planning to write it down as I go.
Leave a Reply