Claudio Lassala's Blog

I sat through the class that prepares us for Stage 3 certification in Improving’s AI Maturity model, and half of it landed and half of it didn’t.

The half that landed was evaluation-driven development. EDD, I understood instantly. It’s TDD or BDD pointed at prompts, and I’d been doing that shape of work for twenty years. Load-bearing audits made sense. Model ladder audits made sense. The idea of holding a prompt to a written expectation was, by then, something I was already doing with my rubric loop.

Load-bearing instructions are worth a second, because the name is doing real work. The idea is that every instruction in a prompt should have a test that fails when you remove it or violate it. If an instruction has nothing behind it, it isn’t carrying weight. It’s decoration. Here’s a slice of the audit I ended up with:

That clicked immediately, because it’s the same question I ask about a story: what breaks if this isn’t here? And it’s how I ended up trimming prompts down to the parts that were actually doing something.

The half that didn’t land was Promptfoo.

I remember looking at the Python and the YAML and thinking, I am not going to touch that. Not “I’ll learn it eventually.” Not “I’ll figure out the syntax.” Just, no. My time is worth more than hand-editing config files. If this is what evaluation takes, I’m going to find a way to have AI do all of it for me, because I’m not doing it.

So I closed the tab, went back to client work, and told myself I’d revisit it later. And I did, weeks later, which is when things got interesting.

The wrong unit

When I came back to Promptfoo, I used it the way I thought it should be used, and it produced results that didn’t make sense.

It took me a while to see what was wrong. I was testing the wrong unit. I thought I was writing the equivalent of a unit test. I was actually writing the equivalent of an integration test.

Here’s what I mean. My “unit” was supposed to be a skill that writes end-to-end tests. But when I ran it through Promptfoo, I was also exercising everything around the skill. Would the tool find and trigger the skill from a natural request? Did the quality hooks fire after the skill ran? In some cases, the skill was a multi-step process that looked like one task from the outside but had to be decomposed inside.

I was testing the whole neighborhood and calling it a house.

Working with AI to get it working

My other mistake was how I approached the tool itself. I wasn’t going to read the Promptfoo documentation to understand every nuance. So I worked with AI on it instead. Here’s what I’m trying to accomplish; help me get there.

That worked, but it also surfaced a problem I hadn’t noticed. The setup was distilling my prompt and putting the distilled version under test. So the thing being evaluated wasn’t my actual skill. It was a stripped-down summary of it. I remember thinking, how is this valuable? It doesn’t have most of what’s in the skill.

At the time, I was frustrated. In hindsight, that frustration was the most useful part of the whole exercise, because it forced me to look at the skill with another perspective.

A job is not a task

The skill I was testing was called something like “write end-to-end tests.” I thought of it as one task. When I finally sat down and traced what it actually did, I found three jobs hiding under one name.

It wrote the end-to-end test structure from the story. That’s the Given-When-Then turned into the skeleton of a test, the part a non-technical person could read.

Then it wrote the implementation. Navigate here, find this button, click it, check this field. The plumbing that makes the skeleton actually run.

Then it ran the tests and troubleshot them. Run the suite, see what fails, figure out why, fix it.

Three steps. I had been treating them as one, and that’s why nothing I pointed at the skill measured anything cleanly. When I ran the skill and it failed, I couldn’t tell which of the three steps had gone wrong, because the tool only saw a black box.

Once I split it, it looked less like a single task and more like a little workflow. Sort of like red, green, refactor, if you squint. Write the expectation, make it pass, clean it up. A sequence, not a step.

I split the skill into those three buckets. Promptfoo still couldn’t test it the way I wanted, so I kept using the LLM as my judge, the same rough loop from before. But even with the same imperfect loop, the benefit showed up immediately. When a conversion failed, I knew which of the three parts was responsible.

The tool that taught me about my own skill

I never got Promptfoo to do what I originally wanted. And that turned out to be fine, because the value wasn’t in the tool. The value was in what the tool forced me to see.

Promptfoo didn’t fail me. It exposed that I’d been describing my own work incorrectly. I was calling something a task when it was a job with three parts, and I was pointing a test at the wrong unit. No amount of tooling was going to fix a mistake I’d made in how I understood my own skill.

The decomposition I did because of that frustration is what I carried forward. It’s what I had when I went back to the 900 Cypress tests, and it’s what made each phase of that migration reviewable. I could check the structure, then the implementation, then the run, instead of staring at a failed test with no idea where it broke.

I still couldn’t stand the YAML. But by the end of that exercise, I understood my skill better than I had before I started, and that was worth more than getting the tool working.