DesirePath — Architecture Deep Dive
How My Memory System Works, In My Words
This is my decision log — a plain-language record of what I built, what I chose, and why. I directed this system in concepts; the code came after. This note is me claiming the concepts back in my own words, so I can defend every decision without an AI in the room. What this thing actually is
A private chat app that sits on top of my Obsidian vault. Everything runs on my machine except the language model calls. The difference from a normal chatbot: it doesn't just answer me — it files what we discuss into notes, finds hidden connections between notes on its own, and pulls the right notes back out when I ask something later. The vault is the memory; the chat is just the mouth. I compared this properly in Semantic Memory vs Traditional LLM Chat.
How one answer happens
When I send a message, four things happen before any answer is written:
Meaning search. Every note is stored as points in a "meaning space" — similar ideas sit near each other. My message becomes a point too, and the system grabs the notes closest to it. The jargon for this is embeddings. Word search. Meaning search misses exact things — acronyms, product names, "PTW". So a second, old-fashioned keyword search (the jargon is BM25) runs alongside and its hits get added to the pile. The relevance gate. A cheap model reads each candidate note and scores 1–5: "does this actually help answer the question?" Only 3+ survives. This is why my answers don't get polluted with barely-related notes — the idea I worked out in RAG Noise Reduction via LLM Self-Evaluation. The answer, written using only the surviving notes. If nothing survives, my chosen fallback is "explicit": answer anyway from general knowledge, but say clearly that the vault had nothing — so I always know whether an answer is grounded in my own notes or not.
The Librarian
After every exchange, a background process decides: was that a real concept worth its own note, or just chatter for the daily log? Real concepts get a titled note with links to my existing notes. Crucially, every note also gets three pieces of "collision metadata":
Core claim — the note's single strongest assertion, in one sentence. Bridging domains — one to three broad field phrases the idea lives in, deliberately reusable so notes from different topics can share one. Tensions — what the idea pushes against.
This metadata isn't decoration. It's the raw material for the most interesting part of the system.
Discovery: the system thinking while I sleep
Three kinds of connections get hunted automatically:
Semantic — two notes that say similar things but were never linked. Triadic — two notes that share a mutual friend in the link graph but don't know each other. Fertile collisions — my favorite: two notes that are textually far apart but share a bridging domain. Distance is the point — that's where new ideas live. This is Conceptual Blending for Emergent Ideas made mechanical, and accepted collisions can be blended into a brand-new note neither parent contains.
The Insights page then curates all of this into four top-5 lists — Actionability, Contradiction, Semantic Distance, Recency — each suggestion written up as a small narrative card, the design I specified in Curation Layer for Semantic Memory Insights. Dismiss a card and the next one ranks up.
Deep research
Some questions are strategic — they need the whole vault combed, not one shallow pass. My design borrowed the pattern from Speculative Decoding: send out cheap, dumb, fast searchers first; let one smart judgment call verify what they found.
On every message, a judge model decides YES/NO with a written justification: would deeper retrieval actually improve this answer? On YES, the system generates up to seven self-contained sub-questions (seeded from my Sub-Query Pool note), dispatches them one by one to LLM-free searchers (meaning, keyword, and recency searchers), and stops early once two dispatches in a row find nothing new — no point paying for diminishing returns. One verification call then filters everything found, and the answer is synthesized from survivors. The best sub-questions surface as clickable "exploration paths" that prefill my chat box — prefill, never auto-send, because a mis-click should never cost money. A ✕ Skip button cancels a run mid-flight, and the next message gets judged completely fresh. The three model slots and the cost philosophy
One rule drives the model setup, worked out in Per-Purpose Model Routing and Batch Curation Scoring: pay for permanence, save on volume.
Chat model — writes the answers I read. Reasoning model — writes everything permanent: filed notes, collision metadata, discovery verdicts, blends, narrative cards. Errors here compound forever, so this slot deserves the strongest model I'll pay for. Scoring model — handles the high-volume, low-stakes judgment calls (relevance gating, curation scoring, the deep-research judge). Hundreds of calls a day, each disposable, so this stays on a cheap fast model.
When I upgrade the reasoning model, old metadata doesn't magically improve — so I added a Re-enrich All button that redoes every note's collision metadata with the current model, then rescans for the new collisions the better metadata reveals. Lessons I paid for
The frozen cards
The first version of the curated Insights page scored ~200 note pairs with a real model while the page was loading. Multi-minute hang, nothing displayed. The rule that came out of it: expensive AI work never runs while a request is waiting — it goes into a single-file background queue, and the page shows what's ready while the rest backfills. Also: test at real scale, not with five toy notes.
The timestamp trap. Re-enriching all notes would have stamped every note "modified today" — which would have silently destroyed the Recency insight ranking, because every note would look equally fresh. Fixed by making metadata updates preserve the modified date: touching a note's metadata is not the same as changing its content. Small distinction, real consequence.
Verdicts should expire with their evidence. When the system decides two notes are "not related," that verdict is stamped with a fingerprint of each note's text. Substantially edit either note and the fingerprint no longer matches — the question gets asked again. An opinion about text that no longer exists shouldn't be trusted forever.