At one point, I had a suite of more than 900 Cypress tests, and for a while I knew I wanted them in Playwright. I also knew I was never going to migrate them by hand.
Nine hundred tests meant weeks of tedious, apparently low-value work that would rot the moment I looked away. Before AI, I had no chance of doing it. I’d think about it, decide the math didn’t work, and move on to something more useful.
Then I did it. It took five days on the calendar and about two hours of my actual attention.
How the two hours were spent
I want to be precise about that number, because “it took two hours” sounds like a magic trick, and it wasn’t.
The five days were mostly me doing sprint work. At some point I said, all right, I’m ready to make this change, and I handed an agent a plan. Then I went back to my normal work while it churned. I spent two hours creating the initial plan, then reviewing each phase to make sure the migration stayed on track.
The important word there is plan. I didn’t say “migrate all 900 tests.” That’s the instruction that turns a hard job into a mess. Instead, I broke it into phases: this group first, verify it, then the next group. Each phase had a defined outcome and a way to check it. The agent worked through a phase, I looked at the result, and we moved on.
That decomposition is the same thing I’d do with a person. It’s also the same thing I’d do with myself if I were doing the work by hand. The agent didn’t change the shape of the job, only how fast each piece moved.
Why I trusted it enough to start
The honest answer is that I wouldn’t have trusted it a year earlier.
The rubric experiment I wrote about in the last installment had given me a loop that, while rough, closed. I could write down what a good converted test looked like, run the skill, score the output, change the skill, and see the score move. It wasn’t a perfect harness. The judge was non-deterministic. But it was enough for me to see that the skill was actually getting better over iterations, not just that it felt better that afternoon.
That’s what made the migration possible. I trusted the loop around the model more than the model itself.
I could see the skill improving across the 900 tests. When a phase came back wrong, I could trace it to a specific weakness in the skill, fix the skill, and watch the next phase come back cleaner. That’s the difference between trusting a tool and hoping a tool works.
What the migration bought me
Two things, and both were worth the effort.
The first was speed. The suite used to take 35 minutes to run. After the migration, it ran in 25. Ten minutes off every full run, on a suite I ran constantly. That compounds.
The second was a better skill for writing end-to-end tests, which is the part I care about more. Migrating 900 tests is a lot of practice. Every phase gave me another look at how the skill handled a real case, and I refined it as I went.
There was also an observation buried in the work that turned out to matter. Watching the models convert tests, I kept noticing the same pattern. They struggled with Cypress. They did fine with Playwright. Same tests, same intent, different framework, and the results were noticeably different in quality.
I don’t have a deep explanation for that. What I have is a lot of observations from a lot of conversions, and the observations pointed in one direction. When I’m choosing a framework for end-to-end tests, I now factor in how well AI can write it, because that’s part of how the work gets done.
What this proved
This was the moment I stopped treating evaluation as an optional add-on.
The rubric loop was non-deterministic and held together with tape. It still took a task I had written off as impossible and made it routine. Nine hundred tests, five days on the calendar, two hours of my attention, a faster suite at the end, and the ability to write better and faster end-to-end tests moving forward.
That’s when I understood what the self-assessment had been trying to tell me. I had been treating evaluation as a compliance step to check a box before Stage 3 certification. It turned out to be the thing that decided whether I could trust a skill with real work. The gap it flagged was the gap between hoping and knowing.
Improving’s AI Maturity model frames Stage 3 as an agent that does one thing well, wrapped in evaluations that keep it honest. I was starting to understand why the wrapping is the hard part.
I had closed the loop enough to trust it with 900 tests. Now I needed to understand the tools people told me to use, and that part fought me the whole way.
Leave a Reply