Lorekeeper doesn't retrieve notes. It retrieves spans.

A while ago I wrote about field-testing my second brain against a real ticket. The honest grade was a B: useful, not decisive.

The most interesting thing that test produced wasn't the 92% context reduction. It was a complaint.

The agent kept retrieving files when what it needed was paragraphs.

Lorekeeper is the tool I'm building to fix that, and over the last few weeks it stopped being a folder convention and became an actual retrieval contract.

Most of the work wasn't ranking. It was deciding what a retrievable thing is.

Folders are a viewer's problem

The first thing I threw out was numbered PARA folders.

Numeric prefixes exist to force sort order in a file browser. Encoding a viewer's concern as architecture is how a knowledge base becomes unmovable.

So the folders are thin and carry only an epistemic role: where a thing sits in the chain from raw artifact to synthesized knowledge:

external artifact → sources/ → notes/ → entities/

And frontmatter is authoritative for everything that matters: type, status, provenance, links.

That single split is what lets me reorganize my vault on a Sunday afternoon without breaking retrieval.

One contract is frozen: provenance edges always point from the derived artifact to its origin. Inverse edges are never stored. Backlinks are computed, never written. A note that says where it came from stays true no matter what gets added later.

A file is the wrong unit

The index doesn't address files. It addresses spans: a path, a heading anchor, and a 1-based line range.

A three-thousand-word note about deployment isn't one fact. It's forty. Retrieving the whole thing to answer one question is the same mistake as pasting a repository into a prompt. You get the answer, and thirty-nine irrelevant things competing with it for attention.

Ranking is BM25 over the span text plus a weighted metadata document. A prototype settled that early, and I deliberately stopped treating ranking as a Feature that needed inventing.

A question rarely arrives in the wording the note was written in. So a search takes several alternate phrasings in one call and fuses the ranked lists with Reciprocal Rank Fusion:

lore search ~/brain "retry policy" "backoff strategy" "what happens when a request fails"

Near-duplicate spans get suppressed, so a paragraph copy-pasted across nine files doesn't take nine slots in a list of five.

Every result carries path, anchor, line range, score and the span text itself. Five fields, because an agent weighing evidence needs the address, not just the claim.

Two things it refuses to do

There is no relevance cutoff. Anywhere.

A low score never withholds a result, and no "nothing found" verdict is ever derived from a number. This matters more than it looks: a retrieval score cannot prove that knowledge is absent. It's a documented limitation, not a defect waiting for a fix, and the agent-facing instructions say so explicitly. The agent treats results as evidence, may search again, and may conclude the evidence is insufficient, which is a very different sentence from "there is nothing."

And correctness never depends on the metadata being good. A file with no frontmatter, or with frontmatter nobody recognizes, indexes as ordinary knowledge. Ranking reads neither type nor about as structure.

That's deliberate. The moment retrieval quality depends on perfect tagging, the system only works for the version of me that has time to tag perfectly.

Reading and indexing never write to your notes either. Indexing a vault leaves every file byte-identical.

What it costs to ask

I'd rather measure than claim.

Over a deterministic synthetic corpus of 3,000 notes (10,537 spans, about 3.2 MB, with 907 paragraphs deliberately duplicated across files):

| Measure | Value | | --- | --- | | Walk, read and split | 81 ms | | Build the index | 64 ms | | Query, one wording | 2.3 ms | | Query, three wordings fused and suppressed | 6.9 ms | | lore search end to end, three wordings, --json | 129 ms |

Measured on an M5 with Node 26, query rows as medians of 20 runs.

The index is built in memory on every single command and never persisted. That looks wasteful until you read the table: persisting it would save roughly 150 ms and cost a managed file that can drift, go stale, and need invalidating.

At this size that trade is obviously bad. When the walk and index phases, the two that scale with the vault, start approaching the interactive reading, I'll revisit it. Not before.

There are no embeddings, no vectors, and no LLM call anywhere in the CLI. It runs offline with no credentials and no vendor. For a few thousand Markdown files, lexical retrieval isn't a compromise. It's the correct amount of machinery.

The agent has to be told, and told correctly

None of this helps if a coding agent doesn't know the brain exists.

So initializing a brain generates an AGENTS.md at its root: a manifest-owned file stating what the brain is, the exact command including how to pass several wordings, the five result fields, and the absence limitation.

Regenerating it over a copy you've edited reports drift instead of overwriting. Your edits win.

The instruction to issue multiple wordings lives in that file deliberately. It isn't an optimization tip. It's part of the retrieval contract, because fusing three phrasings is what the ranking was designed around.

And the number I didn't get to choose

The whole premise is that span retrieval is cheaper than reading notes. I measured that too, using the exact call the artifact teaches.

Five results, from five distinct files.

The whole notes those spans live in: 5,604 bytes. The JSON payload the agent actually receives: 2,847 bytes.

Roughly 2.0x. Modest.

I'm reporting it as measured rather than as improved, and three things bound it. All three I could have tuned away and didn't.

The baseline is deliberately generous. It charges only for the files the returned spans came from, as if the agent had already known which five notes to open. Without span retrieval it wouldn't have known, so 2.0x is a floor, not a best case.

The synthetic notes average about a kilobyte, so a span can only be around 3.4 times smaller than the note holding it. A vault of longer notes would show a bigger saving. I didn't lengthen the corpus to produce one.

And the payload is pretty-printed, which is 12% of its bytes. Compact JSON would read 2.2x, but the output format is a contract other things depend on, and changing a contract to improve a benchmark is exactly the kind of thing this project exists to avoid.

Bytes are a labelled proxy for token cost, not a token count. Counting real tokens means shipping a tokenizer and picking a vendor, and the CLI makes zero LLM calls by design.

The thing that made me laugh

AGENTS.md, the file that teaches an agent how to search, is itself indexed by the search.

On its own illustrative example query it outranks the note it was trying to demonstrate. 0.0492 against 0.0484.

I could exclude it by filename. That would mean retrieval deriving meaning from a path, which contradicts the entire folder decision above. I could exclude it by frontmatter, which needs vocabulary I've deliberately left unfrozen.

There is no small correct fix, so there isn't one. It's written down as a known behavior instead of quietly patched.

Across four realistic questions the intended note ranked first every time, with the artifact appearing only lower down at about a third of the winning score.

Where this is going

Pathfinder finds the way. Lorekeeper remembers the journey.

The retrieval spine is the part I wanted to get right first, because everything later (capability sync, an MCP adapter, mobile capture) is a transport question over the same core.

I still don't need an AI that remembers everything.

I need the right seven paragraphs.

Lorekeeper on GitHub: https://github.com/rikilamadrid/lorekeeper

📜 Plain Markdown. Your files. No vendor in the middle.