Claudio Lassala's Blog

I reran my AI maturity assessment today. The first run was noisy. It kept picking up references to the skill-creator skill that Anthropic built, not me. Counting someone else’s skill as my own work made the numbers less useful.

I improved my digital brain app to let me exclude files and folders from indexing. I reran the assessment on cleaner data and started analyzing the middle dimensions.

This is the first half of that analysis. The score itself matters less than the patterns it surfaced.

Main Findings

1. I fix AI failures at the system level

When the assessment asked about recent AI output I could not ship as-is, the evidence showed I respond by updating the underlying skill or workflow, not by rewriting the prompt one more time.

Three failure modes keep showing up:

• Outdated knowledge. LLM training data has a cutoff. I feed the model current ground truth with Context7 and the latest changelogs.

• Unreliable test generation. AI kept struggling with Cypress tests. I migrated to Playwright and built skills for writing, implementing, and troubleshooting end-to-end tests in that framework.

• Incorrect file structure. Generated files sometimes land in the wrong folders. The fix is to tighten the skill until the pattern is enforced, not to move files one by one.

A repeated failure becomes a skill update.

2. AI instructions are living documents

My persistent instructions live in the .devin/skills folder as version-controlled artifacts. They get updated when real output falls short.

Recent examples:

• writing-style keeps rules like linking to Improving and avoiding overused em-dashes.

• The draft-blog-post skill produced AI-isms, so I added an edit-blog-post skill that revises the output to sound like me.

• prototype-spec now contains constraints, target tech stack, and persona guidance so the prototype can be used as a conversation tool with stakeholders.

These are not static prompts. They evolve.

3. Workflows are mapped into discrete steps

The team has mapped the AI-assisted development process from stakeholder conversations to feature wrap-up. I turned that into a document with Mermaid diagrams. The diagrams matter less than the clarity about where we already have AI help and where the next improvement should live.

That is how designing endpoints became a focus. It happens every sprint, and it is worth automating well.

4. Sub-agents work best with limited context

I now hand work to multiple small agents with tight context windows instead of loading everything into one long conversation.

The prototype-spec skill is a good example that can be improved to leverage that approach by constraining agents to a specific target stack, a set of personas, and the data shapes needed to facilitate stakeholder conversations. That limited scope gives more useful output than asking one big agent to do everything.

5. Handoffs are the weakest area

The lowest score in this batch was on automated transitions between steps. Most of my workflows still require me to manually invoke the next skill.

The design-endpoint workflow is one of the exceptions. It runs a designer agent, then a reviewer agent, then loops back to the designer if needed. After a few iterations, it escalates to a human. I want the rest of the workflows to work the same way.

6. Quality gates are already in place

The evidence showed several quality checks:

• A reviewer agent checks the output of the designer agent.

• I use a separate LLM to critique a plan generated by a first LLM.

• The prototype build skill runs npm run build as part of its execution.

The harness is there. It needs more coverage.

7. Evaluation is moving from “vibe” to harness

I am using PromptFoo to run repeatable tests, compare models, and track scores over time. The most useful thing I learned is that I was treating PromptFoo like an integration test for entire workflows, but it is better suited as a unit test for individual prompts.

That distinction matters. If a prompt is the unit under test, you can refine it, bundle it into a skill, and register that skill. The design-endpoint skill is the prime candidate for this because we use it every sprint.

What I’m Taking Away

The cleaner dataset already made the assessment more useful. Three things are worth acting on:

1. Treat skills as the product, not the prompts. A working prompt is a start. A reusable, versioned skill that encodes the fix is the asset.

2. Test prompts like unit tests, not workflows like integration tests. Workflows are hard to evaluate as one black box. Individual prompts, and the small agents that wrap them, are the right unit of measurement.

3. Automate the handoffs. Manual invocation between steps is the friction. The design-endpoint reviewer loop is the model for the next set of workflows.

What’s Next

I will finish the second half of the assessment, covering the remaining dimensions and the stage mapping, and share that as Part 2. In the meantime, I am improving the design-endpoint evaluation, adding more exclusions to the digital brain, and looking at which handoffs can be automated next.

The assessment is not a destination. It is a way to see which recurring problems are worth solving systematically, and whether my AI instructions are actually living documents.