I recently earned Improving’s Stage 4 certification in AI Governance, which is part of our AI Maturity model. From the day I started preparing to the day I submitted, the whole thing looked like a couple of weeks of work. It wasn’t.
Thirteen days if you only count from the day the criteria arrived. Two and a half years if you count the work that made it possible, as far as leveraging AI. Much longer than that if you count a few decades of solving problems through building software. The certification was never the start of the story.
It Started With a Whiteboard
I tend to tell this story starting from the day the criteria were released, because that’s when the clock that everyone sees starts. But tracing it back, the work of really leveraging AI assistance began about two and a half years ago.
I started by writing my team’s process up on a whiteboard. The distinct tasks. How work flowed from one to the next. Who owned each step. About twelve months later, I looked at the same workflow and asked where AI could help. For the next year and a half, I built what we now call Stage 3 agents, one for many of those tasks.
Stage 3 was about making a single prompt trustworthy: getting it to hit a quality bar at least 95% of the time, and proving it with evaluations. Stage 4 was the part where I connected the arrows. Instead of a human passing each artifact forward, the agents started passing things to each other, with guardrails between them to keep the whole thing from sliding into chaos.
The certification wasn’t quick. The groundwork had been exercised long before the criteria existed.
What Stage 4 Actually Asks For
Stage 4 certification comes down to five artifacts. Each one forced me to treat my AI-assisted workflow like a real piece of production software instead of a collection of prompts:
-
Workflow definition. Multiple evaluated Stage 3 prompts wired into one end-to-end flow, with an automated harness running it.
-
Guardrails. Automated checks and adversarial review agents between steps, not a human eyeballing each one.
-
Punch-out. Explicit points where a human has to make a decision an AI shouldn’t, and proof that the workflow actually stops there.
-
End-to-end success rate. Evidence the whole workflow succeeds 95%+ of the time across real inputs.
-
Audit trail. Every step traceable, with model, tokens, and cost per step.
The standard also had automatic-fail conditions. No automated harness, fail. Guardrails that rely on human review, fail. A success rate with no composite end-to-end number, fail. Window-dressing hooks that just check a file exists, fail. That bar is what made this certification worth earning.
Turning a Workflow Into a Program
The centerpiece is a harness I built in the Engage (an internal platform) codebase. It takes the spec of a real feature and runs it through five steps: plan, implement, tests, verify, log. One agent session per step. Each step is driven by the exact evaluated Stage 3 prompt, so I wasn’t trusting a different, less-tested version of the logic than the one I’d already proven.
The harness is feature-agnostic. A new feature is just a new story and a run id. It built eighteen distinct real features, shipped through the same governed pipeline.
Guardrails That Catch Real Bugs
Between steps sit deterministic hooks written in plain Python. No LLM in the gate. They reject a plan with an empty scope, a manifest that lists files that don’t exist, test results with a single failing suite, a verification that doesn’t pass its light and dark mode check. These are real content checks, and they caught actual problems.
On top of those, two adversarial review agents challenge the highest-risk steps from a lens the producing step doesn’t share. The implement step gets reviewed for security and permission gaps. The verify step gets reviewed for dark mode and contrast, computing actual color ratios instead of trusting the claim.
These weren’t theater. During the runs they flagged a missing authorization attribute on endpoints, and a dark mode contrast bug where two backgrounds were effectively identical. Each one became a fix, then a re-review that passed.
Knowing When a Human Has to Decide
The harness has two places where it stops and waits for me, and they’re deliberate.
The first is right after planning. Before any implementation code is written, a human has to approve the scope and acceptance criteria. An AI can draft a plan, but committing a team to build the wrong thing shouldn’t happen without a person signing off.
The second is the check I’ve never found a way to delegate. After the tests pass and the automated verification is done, I open the app and try it. I click through wearing the shoes of the person who’ll use it. I think about what they have in front of them and whether the software helps them through their flow. A technically perfect app that doesn’t suit the person using it is useless to them, and I don’t know a way to hand that judgment to an agent.
I had to prove these gates actually stop the workflow. The self-tests run the harness with no approvals, with one approval, and with both, and show it blocks exactly where it should and never sneaks past an unapproved gate.
The Number That Mattered
The success-rate artifact forced me to be honest in a way I found uncomfortable at first.
My first real measurement batch ran thirteen workflows and came back at 30.8%. The bottleneck was the implement step. Those weren’t flaky prompts, they were genuine defects: missing authorization, missing end-to-end coverage, dark mode contrast, and an adversarial review that kept failing to converge. I root-caused each one and fixed it.
After the fixes, a fresh batch of ten distinct features completed end-to-end at 100%. Every step passed. The composite across all eighteen features held at 100%, with no failure trace to chase.
The methodology mattered too. If you aim for 95% success on each of five steps and just multiply, you predict about 77% end-to-end. Measuring the whole pipeline as a single outcome is the only honest way to know whether the workflow actually works, and it’s what caught the compounding problem the first batch exposed.
Every Step, Accounted For
The audit trail is one line of JSON per step execution. It records the model, the input and output tokens, the real cost in ACUs and dollars, the status, the guardrail results, and a trace pointing any failure back to the step that caused it. The cost is harvested from the live agent session, never estimated or fabricated.
To build all the features through the harness, I’ve used the cheapest model I have available. This is a direct benefit from having stage 3 agents that have been evaluated, including load-bearing and model ladder audits.
What I’m Taking Away
The certification that looks like 13 days was really years of building trust in a process, one prompt evaluated at a time. Stage 4 was about making it safe to let run, knowing exactly where the guardrails are and where a human still has to weigh in. Making AI more capable was never the hard part.
Governance is what lets you trust the system enough to let it drive, and to know precisely when to take the wheel back.
Leave a Reply