Claudio Lassala's Blog

I picked the topic for this talk two days before I gave it. The slide deck didn’t exist until the night before. Then I stood up in front of the room at TLC, the Learning Circle we run at Improving, and told the story of how I got to Stage 3 certification.

I could have shown everyone the finished thing. Here are my agents, here are the evaluations, here are the audit logs, here is the iteration log. Look how clean it all is.

I didn’t want to do that.

It’s easy to make this work look like magic. It’s much harder, and much more useful, to show the route I actually took: the things I tried that failed, why they failed, what I learned from the failure, and how I iterated from there. That’s the part people can take with them. The finished diagram is just the receipt.

So this is a series about the painful route. It starts with a questionnaire.

The questionnaire that named my gap

In May, I went through Improving’s AI Maturity self-assessment, the one with 24 questions. I answered it with AI’s help, because of course I did. I took it twice that month, and the same dimension came up short both times.

The result was interesting. It put me close to Stage 4 on prompt engineering, which I believed. It gave me credit for thinking in terms of agents that perform a task. But output evaluation and quality judgment came back at 2.7.

I read that and thought, yeah, that’s right. I don’t have evaluations. Not really.

My evaluation at the time was me. I would prompt the thing, look at the output, decide whether it was good, change the skill, and run it again. If the new result looked better, I kept the change. If it didn’t, I tried something else.

That’s not nothing. It’s why the dimension didn’t score zero. I was iterating on the skill and evolving it instead of fighting the agent. But it wasn’t measurement. It was one person’s judgment, in the moment, with no record and no way to tell whether a change helped or just felt like it helped.

Why the score didn’t bother me as much as it should have

The part that got me thinking was the shape of the gap. Everything else in that assessment pointed forward. Prompting was strong. Task agents were decent. Evaluation was the one thing holding the whole picture back, and it was the one thing I had been treating as optional (due to my time constraints).

Here’s the thing about evaluation: when you don’t measure, you don’t actually know. You know how a handful of runs felt. You know that the output looked good that one time. You don’t know whether the last change you made to a skill broke something three cases over, because you never ran those cases again.

I had a strong instinct for what good output looked like. What I didn’t have was a way to prove that my instinct survived a model change, a prompt edit, or a month of not touching the skill.

What this series covers

I’m going to walk through the whole road, in order, including the dead ends. Roughly:

• The story-writing habit I’d been leaning on for twenty years, and how it turned out to be the ground I was already standing on when evaluation-driven development showed up.
• The rubric experiments that came before I’d heard of any evaluation tooling, and the 900-test migration they made possible.
• The tool I resisted, the wrong unit I was testing, and the moment I realized a job is not a task.
• The month the budget ran out, the PRD that never resonated, and the certification package I put together in about an hour.
• The model switch that improved every skill I had, and the tool that came out of all of it.
• Where the critical thinking actually happens when you automate the rest.

If you’ve been told you need evaluations and you’ve been avoiding it because it looks like a pile of YAML and Python, this is for you. It looked that way to me too. I found another way in, and the way in was something I already knew how to do.

Making it visible on purpose

I titled the talk the way I did because of a choice I kept making on purpose. Every time I was tempted to show the polished result, I showed the mess instead. The first Promptfoo setup that tested the wrong thing. The skill I thought was one task that was really three. The class where I sat there thinking, I am not going to touch that stuff.

The mess is the useful part. Anyone can show you the finished harness. Not everyone will show you how many tries it took, and why.

I’m going to do that here, installment by installment.