A few months ago a colleague at Improving asked me to run the business proposal for a client through some AI to extract the personas we would need to staff a project. The client wanted AI-enabled consultants. So I did some prompting and pulled out maybe five of those AI-enabled roles. When I looked at the personas, I felt a familiar knot in my stomach. There were words there that I would never fit into. Holy cow, there is a lot of stuff I do not know.
Then I remembered I have a whole pile of journals I digitized, twenty years of blog posts, and transcripts from talks I have given. I bundled all of that up, dropped it into an LLM with the job openings, and asked a simple question: here is a candidate, tell me if there is a fit.
The answer came back with numbers and reasoning. Fifty percent on this one, ninety-five percent on that one. And when I looked at the reasoning, I saw that the big words I panicked over had equivalents in what I already did. One role wanted something I had literally done for years; I just never described it in those terms. Another gave me a plan: learn these specific things, week one this, week two that, and the fit jumped to eighty percent.
That single exercise took me from “I cannot do any of this” to “oh, actually, I can.” It was the spark behind the whole thing I have been building since.
Journaling became the raw material
When I told that story to a colleague, he pushed back. “Easy for you to say”, he told me, “you have twenty years of blog posts. What about the rest of us?” Just start journaling today. Right now. Write down the things you work on, the challenges you hit, the way you got unstuck.
So I helped design and run a journaling deliberate practice with a group of people who were interested. Every day we wrote a journal entry. We used a simple structure: the need, the problem, the solution. Not “I wrote C# code.” That does not matter. The need was this, the problem was that, and here is the solution I found, including my opinions about it and the level of skill I had with the tools involved.
That journaling is the asset. Every entry is data about how I actually work, what I actually know, and how I actually solve problems.
The problem: reprocessing the same files
The journaling practice worked, but I kept running the same markdown files through AI workflows over and over. When I wanted an assessment, I loaded my journals, my blog posts, my meeting transcripts, and my sprint docs into my AI IDE and chatted with all of it. It worked. It also burned tokens every single time, reprocessing content that had not changed.
Why am I paying for this repeatedly? If I have already processed a file, what is the most optimal way to reuse that content? That is exactly what RAG is for.
RAG stands for retrieval-augmented generation. You pre-process your documents into chunks, embed them into a vector database, and when you ask a question, you retrieve just the relevant pieces and feed those to the model. You are not dumping your whole brain into every prompt.
From markdown chat to a multi-agent workflow
My first attempt was basically a “mere mortal’s RAG.” I took the AI maturity questionnaire from Improving, threw my folders into my AI IDE, and prompted my way through it. It gave me results and reasoning. Some answers were off, which got me thinking about how to improve the input. But it proved the concept.
The second attempt was a multi-agent workflow. I wanted to run the assessment monthly, and I knew my data would only keep growing. So I recorded myself describing how I thought the work should split, got a transcript, and handed it to the AI to build a workflow I could run with a slash command.
The result was a single markdown file. That was the whole lesson. I had built up multi-agent orchestration as something mysterious, but it came down to one markdown file that said: spin up one agent to process journals, one for blog posts, one for sprint docs, one for transcripts. They can run in parallel because they are independent. Each one comes back with evidence. Then one agent consolidates that evidence, answers the questionnaire, scores the assessment, and writes the summary.
The important part is I did not type that file by hand. I described the need and the sequence, and I let the AI scaffold it. My job was to articulate the workflow: what the pieces were, what could run in parallel, what had to wait.
From workflow to a real app
The multi-agent workflow worked, but I ran into the same wall again. Reprocessing the same files. So I decided to build a small tool that pre-processes everything once and lets me ask questions against it.
It is a React and Vite app in the browser, a Python FastAPI backend running on my machine, and a Postgres database using pgvector as the vector store. It runs in Docker on my computer. All the embedding happens locally in LM Studio using a local model, so my journals never leave my hard drive until I decide what parts should. Only once I have retrieved the relevant chunks does it send that small context to my Gemini API for the analysis.
I am not a Python developer. I do not know a lot of Python. That was the point. I did not start from the technology. I started from a problem I needed to solve for myself. I was my own customer. Only after it worked did I go back and learn how the API was structured, how the ingestion worked, how the retrieval produced the responses. I let AI scaffold it, then I used the running system to teach me the parts I did not understand.
What I can do with it now
The app answers the AI maturity questionnaire based on my actual records, not my memory of the last two weeks. It gives me reasoning and citations. Here is the question, here is the evidence I found, here is where in your content it came from. That matters because I am terrible at self-assessment. Ask me to rate myself in C#, and I will say two. That is not a thoughtful answer; that is reflex. Worse, my memory suffers from recency bias. Whatever happened most recently becomes the truth in my head. The records correct that. I have done things I completely forgot, and the system brings them back with receipts.
I have added other assessments on top of the same data. StrengthsFinder, which gave me an executive dashboard of my signature strengths and the evidence behind each one. A skills review, where I exported a skills catalog from our Engage platform and asked what actually fits me, and it surfaced things like agile coaching and test automation that I would have undersold. Even Java showed up once, not because I know Java, but because I coached a teammate on refactoring Java tests by applying principles from C#. I do not know Java. I do know how to solve that kind of problem, and the language turned out not to matter.
I also added a chat feature. I can ask it to find my blog posts about BDD or TDD, and it pulls them up with summaries and links to the source files, all sitting on my disk. And I built a small MCP server so I can query my digital brain from my AI IDE. In the middle of a sales call, when someone asks whether we have anyone with a certain experience, I can ask my own system and cut a profile straight out of the evidence.
One of my favorite additions was the visualizers. The assessment output comes back as markdown, which is a lot of reading. I am a visual thinker, so I took that markdown into Gemini, used the canvas feature to design a dashboard I liked, downloaded the HTML and JavaScript it produced, dropped it into my project, and prompted “implement a visualizer for my assessment like this.” A few minutes later, the feature existed. Now every assessment gets a visual layer on top, and I can drill from a maturity radar down to the per-question evidence. From markdown and JSON to insight, with a much shorter path between the data and my understanding of it.
Lessons learned
The whole build has taught me more than I expected. The main lessons are these.
- Ingestion and context matter more than the model. When I got a bad answer, my first instinct was that the model was stupid and I needed a better one. It was not. The data going in was bad. Fix the filters, the chunking, the retrieval before you blame the LLM.
- Trust evidence, not scores. A score is only as good as the citations behind it. I want to see what it found and where, not just a number.
- Use the cheapest model that fits the task. Local embeddings in LM Studio, a modest model for retrieval, and my Gemini key only for the deep analysis on a small context. Cost stays low because the heavy lifting is local.
- Your own data is an asset. Journals, sprint docs, transcripts all become reasonable evidence the moment you make them queryable.
- Start from a real workflow, then productize. I did not build the app first and hope for adoption. I ran the manual workflow until I understood how the pieces flowed, then automated parts of it. That ordering is essential.
What this comes down to
None of this started with knowing how to build a RAG system. It started with a need and a problem, and I let AI help me fill the gaps. The tool is only as useful as the records behind it. Journal every day, keep the raw material, and suddenly the AI is not guessing about you. It is reading the receipts.
Here is the point: people do not come to us asking whether we know C# or JavaScript. They ask whether we have solved a certain problem. That is exactly what my digital brain lets me answer, with evidence, instead of from memory.
Leave a Reply