bert

What I built

bert is a local MCP server with hybrid retrieval. sqlite-vec stores bge-base-en-v1.5 dense vectors, BM25 covers exact keywords, reciprocal-rank fusion merges the two lists, and a cross-encoder reranks the top hits. It runs as a single-user stdio process over JSON-RPC 2.0. Retrieval needs no API keys and makes no LLM calls. The host calls memory_search, gets chunks back, and its own model answers.

The repo also carries an orchestration harness. The server registers 12 tools, and the memory and code runs use memory_search.

What I measured

The memory test uses one reader, Claude Sonnet with a 200K window, and changes only how it reaches the notes. 50 questions go to 3 non-Claude judges, graded by majority. All three are Llama models, so they are independent of Claude, not of each other. The corpus is synthetic, and I measured Claude Code only, no other MCP host.

At the 1.26M-token size, about 6× the window, bert and grep both scored 0.90 (45/50 and 44/49), with no significant difference. The paired difference is +0.020, interval -0.08 to +0.12, on 7 discordant questions. Grep used a median 1.46× bert's tokens per question, counting cache reads in full.

Full context fell to 0.08. That reader was capped at a 120K budget, which keeps only about the newest 10%. I read this as truncation losing history, not as evidence for bert. A 1M-context reader would hold about 80% of this corpus and would not collapse the same way. I did not run one.

Where grep scored higher

On code, grep scored higher, 0.97 to 0.86, and used fewer tokens, a mean of 70,537 per question against 149,648. Of the 249 questions, 228 are graded by exact value and 21 by the same judges.

On 30 conceptual questions, grep also scored higher, 0.967 to 0.900, on 4 discordant questions, too few to separate them. There, grep used more tokens.

Orchestration

I tested the harness with Opus on both sides. It changed judged quality by a median +0.007 on a 0 to 1 scale, n=11, at 1.3-5.7× the tokens and 1.4-4.5× the wall clock. I read that as no measured gain.

The bug the eval caught

A dict-key mismatch zeroed the dense signal in the shipped retriever, and a 240-character cap cut answer spans, so hybrid was really keyword-only. The fix moved the same 20-question eval from 0.10 to 0.85. That is directional: no significance test, a llama-3.3-70b reader rather than Claude Code, and the same questions exposed the bug and scored the fix.

Public retrieval data

I read 0.745 against 0.740 as the embedding model earning most of it. Rerank moved hybrid 0.356 to 0.434 on fiqa and 0.355 to 0.354 on nfcorpus, so its effect depends on the dataset.

What limited it

• On code, the gap sits in one corpus. On the 41M-token set, bert got 45/80 and grep 78/80. On the other three, 168/169 and 163/169, p = 0.125. I measured where the gap is, not why.
• One-shot retrieval. The same retriever scored 0.86 over several turns, against 0.66 in one shot, a significant gap. These rows can't say why.
• Rerank latency. On scifact the full stack averaged 3376 ms per query against 17 ms for dense, on an 18 GB laptop.

What it doesn't show

• That bert scores higher than grep. No comparison here shows that at a significant level.
• That they are equal on memory. This sample leaves the tie unresolved.
• Anything about real projects. The memory corpus is one fictional project, with filler from 8 templates.
• What hybrid plus rerank adds in multi-turn use. Naive vector is one-shot and bert multi-turn, so two things changed. I did not run multi-turn dense alone.
• Facts that change. The one such question scored 0.00 on every arm.
• Dollars. Token counts lump input, cache reads and cache writes together.

Links

• Repository
• Memory benchmark report
• Code benchmark report
• Orchestration results
• Bug eval result
• BEIR results