A Retrieval Gate for Large Knowledge Systems

Overview

Retrieval-augmented systems are usually described as a search problem. In practice, past a few hundred documents, they become a filtering problem: the retriever returns ten plausible passages, seven of them are loosely on-topic, and the model dutifully writes an answer grounded in all ten. The failure is not that nothing relevant was found. It is that nothing irrelevant was excluded.

DesirePath is a working prototype of an architecture built around that observation. It sits on a vault of plain Markdown notes — 308 of them — and everything runs locally except the language-model calls.

The three ideas worth stealing

Score every candidate before it reaches the answering model. Hybrid retrieval — vector search for meaning, BM25 for the exact names and acronyms that embeddings smooth over — produces a candidate pool. A cheap model then scores each candidate 1–5 for whether it actually helps answer this question, and only 3 and above survive. The gate sits on the critical path of every message, which is precisely why it runs on the cheap model rather than the good one.

Pay for permanence, save on volume. Three model slots, chosen by consequence rather than by task. The scoring slot handles hundreds of disposable judgements a day. The reasoning slot writes anything that persists — filed notes, metadata, concept formation — because errors there compound forever. The chat slot writes the prose a human reads. Getting this split right was worth more than any prompt change.

Make the expensive step rare, not merely cheaper. This is the part I would defend hardest, and it came out of a failure documented in the Desire paths tab.

What it is honest about

A system that writes its own output back into its own knowledge base will eventually confuse its proposals with its records. That is the central architectural lesson here, it is not hypothetical, and the Limitations tab explains how it manifested and what it cost. Anyone building an agent with persistent memory will meet this, and it is easier to design against than to repair.

The other honest constraint: the quality ceiling of the whole system is set by the model in the Librarian slot, because that model writes the metadata every later retrieval and discovery step depends on. A weak model there does not degrade gracefully — it produces confident, plausible, permanent metadata that quietly misdirects everything downstream.

Status

A prototype that ran daily for months against a real vault, not a product. It has no multi-tenancy, no access control, and no evaluation beyond my own usage. The Where it fits tab is explicit about which of these patterns transfer to an organisational deployment and which are single-user conveniences that would have to be rebuilt.

Architecture

One answer, four steps

Meaning search. Every note is embedded as a position in a vector space, and the incoming message is embedded alongside it. Long notes are chunked at roughly 1,500 characters and embedded per chunk, so a match against one section is not diluted by the rest of a document about five other things. Similarity between two notes is max-cosine — compare every chunk against every chunk and take the closest single pair, because one strongly matching paragraph is a real connection even when the rest of both notes disagree.

Keyword search. Meaning search reliably misses exact strings: acronyms, product names, tag codes. A BM25 pass runs alongside and its hits join the candidate pool. This is unglamorous and it is the difference between finding "PTW" and not.

The relevance gate. A cheap model scores each candidate 1–5 on whether it helps answer this specific question. Only 3 and above reach the answering model. This is the component that stops answers being polluted by the loosely-related, and it is the single highest-leverage piece of the design.

The answer, written using only the survivors — with an explicit statement when nothing survived. Three fallback behaviours were available and the choice matters more than it looks:

Explicit was chosen because the alternative is a system whose confident answers are indistinguishable from its grounded ones.

The Librarian

After each exchange a background process decides whether what was discussed is a real concept worth its own note or merely conversation. Concepts get a titled note with links into the existing graph, plus three pieces of metadata that exist to make later connection-finding possible:

• Core claim — the note's single strongest assertion, in one sentence.
• Bridging domains — one to three broad field phrases the idea lives in, deliberately reusable so that notes from different topics can share one.
• Tensions — what the idea pushes against.

Bridging domains are the load-bearing one. Two notes that are textually distant but share a bridging domain are exactly the pair worth surfacing, and no similarity metric will ever find them, because by construction they do not look alike.

Operational lessons that changed the design

Expensive work never runs while a request is waiting. The first version of the insights page scored around 200 note pairs with a real model during page load. Multi-minute hang, nothing rendered. Everything expensive now goes into a single-file background queue and the page shows what is ready. The corollary: test at real scale, not with five toy notes.

Verdicts expire with their evidence. When the system judges two notes unrelated, that verdict is stamped with a content hash of both. Edit either note substantially and the hash no longer matches, so the question is asked again. An opinion about text that no longer exists should not be trusted forever — and without this, a permanent judgement layer silently accumulates wrong answers as the corpus evolves.

Touching metadata is not editing content. Re-running enrichment across every note would have stamped them all as modified today, which would have destroyed recency ranking by making everything equally fresh. Metadata updates preserve the modified date. A small distinction with a real consequence, and the kind of thing only found by looking.

Off means off. Discovery and deep retrieval have switches, and off means zero background model calls rather than reduced ones. A system that spends money in the background needs a brake that is believed.

Desire paths

The connection-finding engine was the most expensive component in the system and the least useful. This is how it was measured, and what replaced it.

The problem, in numbers

The original engine compared notes pairwise and asked an expensive model whether each pair was related. The usage log was damning:

• 1,401 calls — more than half of every token the system had ever spent
• 213 pairs judged, of which it rejected 10
• 3 suggestions accepted by me

A 1.4% hit rate, on the most expensive call type in the system. The model was enthusiastically producing suggestions and I was paying for every miss.

The deeper flaw was structural rather than economic: it compared everything against everything, with no notion of what mattered. And it fed on its own output — every accepted link created new open triangles in the graph, which generated more candidate pairs. A system that generates its own future workload needs a brake designed in, not bolted on.

The replacement

In parks, planners lay footpaths and then people wear dirt trails through the grass where they actually wanted to walk. Those are desire paths.

Every note is a point on a field. Every time a note is pulled into an answer, or a new link is made to it, that is a footstep. Notes visited repeatedly, by many different other notes, become intersections where paths converge — hubs. A validated hub becomes a concept: a category that emerged from behaviour rather than one anybody predefined.

A note becomes a candidate when its novelty score stands out from the whole field. The score combines how concentrated its traffic is recently (a spike), how fresh the last visit is (freshness halves weekly), and how many distinct notes visit it (variety). The bar is the field mean plus two standard deviations — so as the vault grows, the bar raises itself rather than needing to be retuned.

Then three tiers, each rarer and more expensive than the last:

1. Detection — pure arithmetic over the visit log. Free. Runs daily.
2. Screening — one cheap batched call labels every candidate at once; duplicates of existing concepts and generic labels are discarded.
3. Formation — only survivors reach the expensive model, which writes a real description and its relations to existing concepts.

The philosophy is the same one behind the model routing, applied structurally: the expensive step is not merely routed to a cheaper model, it is made rare.

What was deliberately kept

Pairwise semantic and link-of-a-link scanning were deleted, along with a 200-suggestion backlog, in one sweep.

Fertile collisions were kept — pairs that are textually far apart but share a bridging domain. That is a different mechanism from similarity, it produced none of the noise, and distance is the entire point: it is the only path in the system that can produce an idea neither parent note contained.

Lessons

Measure before fixing. I came in with a feeling — "too much noise, too many tokens". The usage log converted it into a diagnosis: one call type, one cause, an unfiltered candidate source feeding a judge that never said no. The fix was obvious once the number existed and invisible before it.

Structure beats instruction. The old prompt said "be selective" and the model ignored it roughly 95% of the time. The new design never asks a model to be selective. The arithmetic is selective, and the model only ever sees what already survived.

Watch for self-feeding loops. Any component whose output becomes its own input will find a way to grow without bound, and it will look like success while doing it.

Where it fits

This was built for one person's notes. The interesting question is which parts survive contact with an organisation, and I would rather be specific than enthusiastic.

The patterns that transfer

The scored relevance gate is the one to take. Any organisation running retrieval over more than a few hundred documents has the same failure: plausible-but-unhelpful passages reaching the model and being faithfully incorporated. A cheap scoring pass in front of an expensive answering model improves grounding and reduces token spend at the same time, which is a rare combination. It is also cheap to retrofit — it sits between two components that already exist.

Consequence-based model routing. Splitting model slots by what happens if this is wrong rather than by task type generalises directly. Volume judgements that are re-made constantly can run cheap; anything written into a persistent store should not, because errors there are not retried, they accumulate. In an orchestrator with many agents, this is the difference between a sane bill and a surprising one.

Rarity over cheapness. The 1.4% hit rate was not fixed by finding a cheaper model. It was fixed by making the expensive call rare through arithmetic that ran first. Applied to an agent fleet: the win is usually in a deterministic filter that decides whether to invoke, not in negotiating a better rate on invocations you should not have made.

Expiring verdicts. Any system caching model judgements about content that changes needs content-hash invalidation, or it becomes confidently stale. This is a genuine correctness hole in a lot of production RAG, and it is a few lines of work.

Bridging metadata. If the goal is surfacing non-obvious connections — across teams, incidents, product lines — similarity search structurally cannot do it, because the interesting pairs do not look alike. Something like bridging domains is required, and it must be written at ingestion.

Where an orchestrator deployment would benefit most

The shape that fits is an organisation with a large corpus of unstructured internal text that people already fail to find things in: incident write-ups, maintenance history, engineering decision records, customer correspondence. Somewhere the cost of a wrong-but-confident answer is real, and somewhere the corpus keeps changing.

In a multi-agent orchestrator specifically, the gate becomes more valuable rather than less. Ten agents each retrieving unfiltered context multiply the noise problem by ten, and they do it inside a loop where nobody reads the intermediate results.

What would have to be rebuilt

Honestly, quite a lot:

• Access control. There is none. Retrieval must respect permissions at query time, not filter afterwards, and that is an architectural change rather than a feature.
• Multi-tenancy. One vault, one user, one process. Everything about the background queue assumes a single writer.
• Ingestion at scale. The corpus is Markdown files on disk, re-scanned. Real connectors, incremental indexing and failure handling are all absent.
• Evaluation. Judged entirely by whether it was useful to me. No benchmark, no regression suite, no measure of whether the gate's threshold of 3 is right for anyone else's corpus.
• Prompt-injection handling. A retrieval system that ingests documents written by other people has an attack surface this prototype does not address at all.

The honest pitch

The value here is a set of design decisions that were tested against real usage and real cost, including one that was measured and found badly wrong. It is not a product, and the sensible way to use it is as an argument about architecture rather than a codebase to deploy.

The Limitations tab covers the one that constrains everything else.

Limitations

The Librarian model sets the ceiling for everything

This is the constraint that matters most, and it is not a tuning problem.

Every piece of metadata the system depends on — the core claim, the bridging domains, the tensions, the concept descriptions — is written by whichever model occupies the Librarian slot. Retrieval quality, connection discovery and every insight downstream are all consuming that model's output. The rest of the architecture is plumbing around its judgement.

A weak model in that slot does not fail loudly. It writes metadata that is fluent, plausible, and subtly wrong: a core claim that captures the second-most-important assertion, a bridging domain that is too generic to collide usefully, a tension that was not actually the tension. Nothing errors. The notes look fine. Discovery quietly stops working, and the reason is invisible because the metadata reads well.

Worse, it is permanent by default. Upgrading the model does not improve metadata already written, which is why a Re-enrich All pass exists — it rewrites every note's metadata with the current model and then rescans for the connections the better metadata reveals. That pass is itself expensive, and it is an admission: the system's history is only as good as the model that was cheapest at the time.

Anyone adopting this pattern should budget for the Librarian slot first and the rest afterwards. It is the opposite of the usual instinct, which is to spend on the model the user talks to.

A system that writes into its own knowledge base confuses proposals with records

The design files what it discusses. Session summaries are written back into the vault so that a later, stateless query can retrieve them like any other note. That is the mechanism that gives the system apparent memory, and it works.

It also means that when a conversation explores an option, the record of that exploration becomes indistinguishable from a record of a decision. Ask it months later what you did, and things you considered come back as things you chose. The vault slowly fills with the system's own suggestions wearing the same clothes as your facts.

The desire-path rebuild fixed one half of this deliberately — concepts stay internal to the graph and the insights view, and never become notes. The other half, session summaries, is still there.

The general lesson is the one I would put in front of anyone building agent memory: provenance is not optional metadata. A record needs to know whether it describes something that happened or something that was proposed, and no amount of retrieval quality repairs the difference once it is lost. This is the reason I keep the knowledge I actually rely on in a corpus I curate by hand.

Smaller, but real

No evaluation worth the name. Every quality claim here is "it was useful to me". The relevance gate's threshold of 3 was chosen because it looked right, not because it was tuned against a labelled set.

Cost scales awkwardly. Detection is free arithmetic, but the discovery layer's expense grows with corpus size, and the constants that keep it quiet — candidate pool sizes, scan limits, ore caps — were set by watching a bill rather than by analysis.

Single writer, single user. The background queue is a single file. Nothing about concurrency has been considered, let alone tested.

Local-first is a real constraint, not just a virtue. Everything runs on one machine except model calls, which is good for privacy and bad for everything else: no shared state, no horizontal scaling, and an embedding model chosen because it runs on a laptop rather than because it was the best available.

No security model. The prototype ingests only my own files. A deployment ingesting documents written by other people inherits a prompt-injection surface that nothing here addresses.

What it is

A prototype that ran daily against a real corpus for months, produced a measurable failure, and was rebuilt around what the measurement said. That is the whole claim. The parts worth reusing are described in Where it fits; the parts worth avoiding are on this page, and the second list is the more useful of the two.