I recently got my stage four certification in Improving’s AI maturity program. Then I thought about the timeline I’d been keeping. From the day I put my head down to prepare to the day I submitted, it really was quick.
Then I took a step back and asked what made it quick. That question is where Episode 28 of Reflective Practice Radio started.
Matthew and I recorded this one after three weeks apart. He’d been on production support, which ate up his time and my energy. But the thread that kept pulling us forward was the gap between how fast this certification looked and how long it actually took to earn. xxx
It Wasn’t Quick
The stage four criteria were only released a few weeks ago. So from release to my submission, it looks fast. Two weeks, maybe. That framing misses everything that came before.
Trace it back, and it’s about two and a half years of work. I started by writing my team’s process up on a whiteboard: the distinct tasks, how work flowed from one to the next, who owned each step. About twelve months later, I looked at the same workflow and asked where AI could help. For the next year and a half, I built what we now call stage three agents, one for each of those tasks. Stage four was connecting the arrows, letting AI drive through the workflows instead of a human passing each thing forward.
So the certification wasn’t quick. The groundwork had been done and exercised long before the criteria existed. I’d actually been working toward stage four before stage four was a thing for me.
Matthew put his finger on it: the date the certification was released was not the starting point. We just don’t hear the underlying story of what it took to get there.
You Can Only Automate What You Understand
It is easier to automate a process you already understand manually. If you start from automation, from a vague “I want to start here and end there” with no idea what happens in between, it is much harder.
You do need to visualize the whole thing. Not at fine granularity. Just enough to see the boxes and how they flow. Once you can describe that to a person, they can fill in the details. Now AI can too.
That’s how I think about my own craft, producing user stories from conversations with stakeholders. I’ve done it manually for years. I understand the input, I know what good output looks like, and I know the process in between. So when I gave AI my blog posts documenting that process and asked it to make that reusable, it boiled down to one prompt. All those years of understanding came before that prompt.
The Human Checkpoint
Early on, I built a checkpoint into my harness. Once the agent has implemented everything, written the tests, and run them, it stops. Then it’s my turn.
I want to actually try it. I pop it open in the browser and click through, wearing the shoes of the person who’s going to use it. I think about what they have in front of them, what’s up in their mind, whether the software helps them through their flow. I don’t know a way AI can do that for me. It’s a human quality control process, and it’s hard to delegate.
Because a technically perfect app that doesn’t suit the person using it is useless to them.
Pacing Yourself
The go-kart analogy keeps coming up. The first time I went to a go-kart track, I paid for ten seven-minute sessions. I couldn’t finish three. I was going fast, but I was tense the entire time. I didn’t know when to grip the wheel and when to relax.
Working with AI has felt the same. Do it, do it, do it. One more prompt, one more prompt. Keep going. If you’re tense the whole way, you come out exhausted.
The key is understanding where to put the exertion and what the release looks like. I put my effort into framing the problem well, building a prototype when it helps, so I have a higher chance of hitting the result. Then I push it through the system and, for that hour, it’s working; I can relax and do something less taxing.
The Conductor, Not the Musician
Matthew pointed out that we still tend to fill my relaxing time with more taxing work. We launch three agents, and while they run, we launch into a fourth thing. No time to recover. That’s a source of burnout we keep falling into.
The harder part is context. If three features finish around the same time, three different contexts come back to me at once. I have to reorient, play each persona, then switch to the next. Jumping between them is where the exhaustion lives. You can even start hallucinating in your own context, dropping a prompt meant for one session into another.
The orchestra analogy helps me. Each musician thinks about the note they’re playing and the one coming up. The conductor isn’t thinking about every note. They step back and guide the whole performance. With AI, we have to stop trying to check every semicolon. We have to sit where the conductor sits.
It’s like teaching a kid to ride a bike. Training wheels first, then guard rails and knee pads, then you let go. You build trust incrementally. You don’t hand the AI a huge task and go read every line it generated.
More Than a Fancy Google Search
We talked about how easy it is to treat AI as a fancier search box. We’ve both found better uses for it.
Matthew gets a DDD lesson delivered to him every morning, scheduled and waiting when he wakes up. On Saturdays, a digest of the week’s tech news. Proactive, not reactive. That’s a different relationship with the tool.
I used AI to bridge the gap in our training content for stage four. The material doesn’t fit how I learn. So I told the AI what I already know, what I need to know, and asked it to help me connect the boxes and arrows. We worked through it daily, self-guided. I got to the point where I could see the whole picture, then double-click into the parts I didn’t fully understand. That’s something I could not do in a traditional class years ago.
We both had small, personal examples. Matthew is growing cucumbers and checking with AI at every stage, from spotting the first sprouts to knowing which male flower to remove. I’ve been feeding my lyrics and guitar parts into AI to make songs, then practicing to reach the notes I couldn’t hit before. I have recordings from months ago where I sound terrible, and recent ones where I can hear the progress. Measuring against the old recording is what makes that visible.
The education point follows naturally. If we can individualize a lesson to a kid who grew up playing soccer in Brazil, using examples they already know instead of forcing everything into an American frame, that changes how they learn. We have the tools for that now.
The Right Tool for the Problem
Matthew’s cucumber story reminded me of a conversation I overheard between technologists, bragging about throwing a powerful agent at their problems. Throw Anthropic’s Fable at it, and it’s solved. That’s the bazooka approach.
Sure, you can kill ants with a bazooka. But it’s expensive, and you burn through your ammo, and you still have ants. Killing ants isn’t a solution to an ant problem. You figure out where they’re coming in and address that instead.
It’s the same with helping real people. My uncle doesn’t need a custom app built with a dozen containers. He needs help remembering to take his medication, setting an alarm, getting his calendar sorted. Those are real problems. The impressive approach isn’t the bazooka. It’s using what you know to help someone like him solve what’s actually in front of them.
This ties back to a point from Dario Amodei at Anthropic. People are souring on AI because we’re not keeping our promises. If AI can actually solve something hard, we should pursue it in earnest. But there’s a fair question that came up from a colleague: once a cure is found, with all the data that went into training the models, how do we make sure people can actually benefit from it? Show me the plan.
Matthew keeps landing on optimism. There’s potential for good and potential for bad, and that was true before AI too. The difference is whether we collectively decide to look for the good and do something about it.
This episode wandered from certifications to go-karts to cucumbers to a grandparent’s medication reminders. It landed, as these conversations tend to, on optimism and on the responsibility for the people who can see what’s possible to help those who can’t.
If any of this resonated, the full conversation is worth your time.
Leave a Reply