Loading post…
Every time an AI searches our notes, the search box sends the question to Google, gets back 768 numbers, and uses them to find notes with a similar meaning. That is an embedding, and ours come from Gemini.
It works well. It is also a network round trip on every single search, a dependency on someone else's API, and every note we write passes through it once on the way in. So we asked a simple question: can a small open model running on our own hardware do the same job?
Short answer, measured on 269 real searches and graded blind:
| Gemini (today) | Small open model, our hardware | |
|---|---|---|
| Right note ranked first | 68.9% | 67.7% |
| Share of answer notes in the top 5 | 77.0% | 76.7% |
| Mean reciprocal rank | 0.793 | 0.781 |
| Related notes ranked well (nDCG@10) | 0.810 | 0.778 |
On finding the answer, it is a tie. The difference is well inside the noise. On ranking the useful-but-secondary notes below the answer, Gemini is still a little better, and that gap is real.
The model is EmbeddingGemma, 300 million parameters, running on the CPU of a 2019 Intel MacBook. Out of the box it lost. Here is what it took, and what it means for you.
When your AI looks something up in a docs pack, or in team packs searched alongside one, Warmhand runs two searches side by side: one on keywords, one on meaning. The meaning search is the one that finds "who are our customers?" in a note that never uses the word "customer". Packs that hold only memories are searched by keywords alone today.
That meaning search runs on Gemini. Each search that uses it goes to Google's API, and the text of your pages, documents and memories goes there once, when we index it. For docs packs, a Gemini model also writes a few questions each section answers, so plain questions find technical text. Our privacy policy lists all of this.
A small model on our own hardware would change three things:
It could also bring meaning search to packs that only hold memories, without sending them anywhere.
What it does not change yet: anything you use today. This test ran on the search behind our own team notes, not on Warmhand's pack search. Before we switch Warmhand, we run the same test on packs. The bar is answers at least as good as Gemini's, and no more "we have nothing on this" when there is something.
It is not about money. Embeddings cost us between one and three dollars a month.
The notes. A copy of our own team's working memory, about 6,000 notes of decisions, rules, project updates and research, kept in an internal tool we use every day. Same search engine, same code, before and after. Search there is hybrid too: a keyword search and a meaning search run side by side and their results are merged.
The questions. Not ones we wrote for the test. We pulled the actual searches our AI assistants ran over those notes during normal work: 269 of them, things like a project name plus a few keywords, or a half-remembered decision.
The grading. For each search we took the top 10 results from both systems, shuffled them together and had an AI judge (Claude Sonnet) grade every note 0, 1 or 2 without knowing which system returned it. 3,842 judgments in total. 2 means "this is the answer", 1 means "useful context", 0 means "not relevant".
We started on our older, smaller test set of 34 hand-written questions. Gemini answered 23.
| Model | Size | Answered (of 34) |
|---|---|---|
| Gemini embedding | hosted | 23 |
| EmbeddingGemma | 300M | 20 |
| Qwen3-Embedding | 600M | 19 |
| EmbeddingGemma 2 | 300M | 18 |
Newer and bigger did not help. The gap was always on the same kind of question: a casual question whose answer is written in jargon. Ask "who are we selling this to?" and the right note says "the ICP is…". Gemini knows those mean the same thing. A 300M model ranked that note 250th.
Two things we tried that made it worse, so you do not have to:
The fix that worked is old and simple. It is usually called doc2query, and it is the same idea our docs packs already use.
For every note, we generated three questions someone might ask when that note is the answer: one plain, one casual, one short keyword search. Then we folded those questions into the note's embedding, half note and half questions. Now when someone types a casual question, it lands near notes that were already "asked" something similar.
No new model. No training. The search code did not change, only its distance cutoffs and the note vectors.
On pure meaning search, with keyword search switched off, it nearly caught up. On our hand-written questions, the average rank score went from 0.61 to 0.67, against Gemini's 0.69. The right note landed in the top 20 for 27 of 28 questions, against Gemini's 24. Two questions Gemini ranks at 73 and 208 now land in the top 20.
That is the version in the table at the top.
We also fine-tuned the model on the same generated questions. It helped a little on ranking and not at all on questions answered. The expensive option lost to the cheap one.
On our test machine, search took a median of 51 ms with the local model and 352 ms with Gemini. That is not a fair comparison: the test machine sits far from Google's servers, behind a VPN, so Gemini pays extra for distance.
The fairer number comes from a small server in Frankfurt, with one CPU core, on an earlier day. Embedding a single question took a median of 135 ms with EmbeddingGemma and 264 ms through the Gemini API. Faster, from a model that costs nothing and runs on the cheapest hardware you can rent.
The goal is not "almost as good". It is as good or better, on hardware we own. We are moving search and the question-writing step onto one small box, closing the gap on the related notes, and running the same test on Warmhand packs. When that is done, we will publish the result, whichever way it goes.