The thing I did to get certified was a one-off. I had a deadline, I had a pile of raw material, and I muscled a package together. Then I did the same kind of work again to make my skills survive a cheaper model. When it was done, I had a choice: treat all of that as a recurring chore, or turn it into something I could run again.
I turned it into something I could run again, because I already knew I’d need it.
Here’s the problem with getting certified. It’s easy to finish Stage 3 certification in Improving’s AI Maturity model and think, great, that’s handled. It isn’t. Stage 4 is a collection of Stage 3 agents, and every one of them needs its own evaluated receipts. Not one package. Every agent in the harness. And then the people building on top of that are looking at tens or hundreds of agents eventually.
I am not hand-writing evaluations for hundreds of agents. I’ll do it once to learn the skill. I won’t do it as a career.
The ideal I started with
My ideal is to feed stories into AI, describe what I need, and get a prompt back. A prompt that’s already been evaluated, already has its audits, and is ready to install as a skill.
I don’t want to write the evaluation criteria by hand. I don’t want to touch the harness config. I don’t want to read through the reports. I want to talk like a human on one end and get a trustworthy, installable skill on the other end.
That doesn’t mean the criteria don’t get written. They absolutely do. It means I’m not the one transcribing them into YAML at midnight. The critical thinking still happens, and it happens where I want it to happen, which I’ll get to in the last installment.
Story to prompt
So I built the skill that does it. I call it story to prompt.
It takes a story, or an existing prompt I already have, and runs it through the same engine I used for the certification. It generates the evaluation criteria, sets up the harness, produces the iteration log, runs the audits, and hands me a prompt that’s ready to install.
The part I care most about is where the criteria come from. I don’t write them as criteria. I write them as stories, with my Given-When-Then scenarios, in the same human language I’d use with a stakeholder. The skill knows how to turn that into the structured checks the harness needs.
So the input is still a conversation about intent. The output is a machine-ready evaluation. Everything in between is the skill’s busy work. Here’s the shape of one package, so you can see what lands on the other side:
product-management/create-impact-map/
├── human-stories.md # what I wrote
├── prompt.md # what the skill produced
├── tests.md
├── eval.js
├── promptfoo.yaml
├── skill-config.json
├── audits/
│ ├── iteration-log.md
│ ├── load-bearing-audit.md
│ └── model-ladder-audit.md
└── results/
I write the top line. The skill produces everything else.
The AI assistance repository
I’ve been putting the skills I use most into one place. I call it my AI assistance repository.
It’s organized by area, so there’s a section for product management, for instance, where skills like turning a stakeholder conversation into an impact map live. Each one has been through the story-to-prompt engine, so each one has evaluations behind it.
A lot of these skills started as prompts from before we had the concept of skills at all. Some of them had been sitting in my notes for years, doing useful work, with zero evaluation around them. Now they go through the same pipeline and come out with receipts.
The repository isn’t complete. I migrate skills as I start leaning on them heavily. When I notice I’m depending on something, I check whether it has evaluations. If it doesn’t, I push it through and now it does. That’s the whole maintenance model: use it a lot, notice, migrate.
Let the AI read the report
One of the decisions I’m most confident about is this: I don’t read the harness output anymore. The AI does.
The evaluation tooling spits out an HTML report with everything in it, and the guidance is to open it and read through it. I did that once. I scrolled through pages of results, trying to hold the whole thing in my head, and thought, why is a human doing this?
Language models are built for exactly this. Take all that output, look at my stories, and tell me what I actually need to know. Which checks failed, what changed, and what decisions I have to make as the human in the loop. Bring me the two or three things that matter and leave the rest in the log.
Now that’s what happens. The skill reads the report, synthesizes it, and surfaces the decisions. I get a short list instead of a wall of HTML.
Deterministic checks and judgment checks
There’s a distinction I’m building into this now, and it’s the piece I’m most excited about.
Some checks are deterministic. They’re structural. Did the output have the shape I asked for? Does the story have a persona, an action, and a reason? Does it have the scenarios? Those are pass or fail, and a machine can verify them without judgment.
Other checks need judgment. They’re qualitative. You can have a story with all the right structure that says something nonsensical, and a deterministic check will wave it through. A judge needs to read it and decide whether it actually makes sense in this company’s context.
I want the skill to know which checks are which, split them automatically, and track them separately. That way I can tell the difference between “the structure broke” and “the structure is fine but the content is wrong,” which are two completely different problems with two completely different fixes.
I’m also working to make the harness itself swappable. Right now the evaluations run through Promptfoo. But another client uses a different tool for the same job, and I want the same skill to be able to target either one. The stories stay the same. The engine underneath is replaceable.
Swapping models on demand
The last piece is the one that makes all of this feel like an actual system instead of a pile of scripts.
When a new cheapest model shows up, I can say, replace the model everywhere and re-run everything, then tell me what the scores say and what needs to change. The skill goes through every evaluated prompt, re-runs it against the new model, compares the results, and surfaces the ones that broke. I can do that on demand now.
It logs as it goes. The iteration log records the hypothesis, the change it made, and the result after the change. The load-bearing log shows the checks that actually matter. The model ladder log shows how each model performed across the criteria. It’s all structured, and it all gets recorded the way the certification asked for, except now nobody has to sit there and do it by hand.
The point of all of it
I wrote a skill that builds skills. That sounds like a cute phrase until you do the Stage 4 math and realize it’s the only way the math works.
If every agent in the harness needs evaluated receipts, then the receipts have to come from somewhere automated, or the whole approach collapses under its own paperwork. The critical thinking stays with me. The transcription doesn’t.
Where that thinking actually happens is the part I want to end this series on.
Leave a Reply