I was having a good month. Features were coming out of my AI-assisted workflow faster than I could review them, client work was moving, and I was finally feeling like the investment in this stuff was paying off.
Then I hit my monthly usage limit and my access got cut off.
It landed right around the time our AI usage dashboard showed up, which meant I got to see the number that did it. I remember looking at it and thinking, I’m spending that much, and I’m not even using the expensive model. I was on the same setup most people were on: the mid-tier model for planning, the small one for implementation, and a lot of care about how much context I was dragging around. I was being careful. It still wasn’t enough.
When your access stops mid-flight, it feels like going 200 miles an hour and having someone yank the brake. Your neck snaps forward. Everything you were in the middle of is just stopped.
Then I heard the part that changed the plan. Stage 3 certification, the third step in Improving’s AI Maturity model, came with an automatic increase to the limit. So the thing I’d been putting off, the evaluation work I found so unappealing, was now the thing standing between me and my normal pace.
Time to go revisit it.
The PRD wall
I went back to the self-directed learning material, the module that walks you through creating a PRD for something small, a Pomodoro timer kind of thing. A few of us went through it together over a few weeks, step by step, the way it was laid out.
I got stuck almost immediately, and it wasn’t the tooling.
PRDs, product requirements documents. I’ve never used them. Not once in my career. I first heard the term maybe a year earlier, when everyone started saying you need to give AI a PRD. My honest reaction was, what on earth is a PRD?
And the word requirements did the rest. When I hear requirements, I think waterfall. I think a thick document written once and argued over for months. It doesn’t matter whether that’s fair to the concept. That’s the association my brain makes, and it’s strong enough that it stops me cold.
On top of that, the module wanted me writing YAML and evaluation criteria by hand. So I had two walls in front of me: a document format I like, but not for that purpose, and a pile of configuration I’d already decided I wasn’t going to touch.
I sat in one of our meetings, scratching my head, and a colleague noticed. What’s up? So I said it out loud. This PRD thing isn’t my workflow. I don’t think in requirements. I think in stories.
He said, then drop the PRD thing. Focus on your thing.
That was all I needed. It sounds small, but I’d been treating the sequence in the course as the only path, and it isn’t. The course teaches a way to get to a result. If my natural way gets to the same result, and gets there faster, I should use it.
My natural way goes like this: discovery with the people who have the problem, then some sense of the experience we’re trying to create, then the interface, and then stories. Not requirements. Stories. The ones people will tell each other if we actually pull this off.
I’ve had a skill for writing those stories for a few years now. It started as a custom GPT, back when that was the only shape this stuff came in. I fed it my blog posts about how I write stories and how I like to have the conversation, and it became the first agent I ever built. I used it constantly, dictating in stakeholder rooms and asking it to turn the conversation into stories.
So I had the habit. I had the skill. I just needed to point them at evaluation.
An hour on the treadmill
Once I stopped fighting the PRD, the certification package came together fast.
I do an hour-long walk most evenings, and that night I was on the treadmill with the problem in my head. Somewhere in that hour, the package was submitted.
What made that possible wasn’t speed of typing. It was that I already had most of the raw material sitting around.
The certification requirements were spelled out in a rubric, the last slide of the class. The course repository had every module laid out: write the evaluation, commit it, create the iteration log, create the load-bearing log, create the model ladder log. All of it was specified.
And I had the skill I’d been refining for years, plus something I’d forgotten I had. In many of my blog posts, I’d already written out the evaluation criteria for that skill, in my usual verbose English. I’d documented what good output looked like, in prose, long before I needed it for a certification.
So I didn’t have to write the criteria from scratch. I handed all of it to an agent: the rubric, the course repo, my skill, my blog posts. It worked on its own for a while. Then it told me the package was ready.
The order it worked (which was part of my plan) in is the part I’d been avoiding doing by hand:
- Write the manual test cases first.
- Turn the skill into a structured prompt.
- Build the evaluation harness around it.
- Iterate and produce the audits.
And the iteration log told the story of the four passes it took:
| Iteration | What changed | Result |
|---|---|---|
| 1 | First prompt and eval harness | 1 passed, 4 failed, 1 error (17%) |
| 2 | Swapped LLM-rubric graders for deterministic assertions | 1 passed, 5 failed (17%) |
| 3 | Added a complete example story | 6 passed, 0 failed (100%) |
| 4 | Removed guidance no test depended on | 6 passed, 0 failed (100%) |
I looked at what it built. It had pulled the evaluation criteria out of my blog post, set up the harness, and produced the YAML and Python I never wanted to see. It produced the audits. I read through the artifacts, because I’m accountable for whatever the AI creates on my behalf, and then I submitted.
The examiner’s report came back PASS. It noted that 22 of the 23 prompt instructions were load-bearing and that every evaluated case passed on both models. That’s a statement about this prompt, these inputs, and those runs. It is not a promise that the skill works everywhere.
What the friction produced
I tried to follow the instructions exactly, and I couldn’t. I tried two or three different ways to make the PRD sequence work for me, and I asked other people for help, but I still couldn’t connect the material to how I actually work.
That’s not a knock on the material. It’s a knock on the idea that there’s one path. The sequence in the course is one way to reach the result. My way reaches the same result, and it’s built on something I’ve been doing for twenty years.
The certification was the catalyst. It forced me to finally do the evaluation work I’d been avoiding. But the way I did it came straight out of the friction, not despite it.
The certification solved one problem and left another one wide open. I had the evaluation habit now, and I had a submitted package behind me. I still had a budget that ran out, and a stack of skills that were only as good as the model carrying them. That was the next thing to attack.
Leave a Reply