We benchmarked our packs search against Context7 and ended up rebuilding it

On 60 fresh Next.js questions, Context7 put the right docs page first 80 percent of the time. We managed 60 percent. Seven measured changes later, on held-out questions across six libraries, Warmhand ranks the right page first at least as often as either Context7 index on every library, and the median answer costs an agent 420 to 881 tokens instead of 6,050 to 20,702. This post covers how we measured that, including the part where our first result was wrong.

Why a docs benchmark, for a product built on a team's own memories

Warmhand's core is a team's own knowledge: decisions, conventions, how things are done here, packaged as packs and delivered to whatever AI tools the team uses. That material is private by nature. Nobody else holds a copy, so there is no competitor we can run against it.

Docs packs are the exception. A pack built from the Next.js documentation holds the same pages Context7 indexes, so both can be asked identical questions and scored with identical code. And the search that answers from a docs pack is the same delivery engine that answers from every other pack. If it hands an agent the wrong page, or a 20,000-token wall of text, it does the same to your team's memories. Docs just make that measurable.

"Good delivery" for an agent is concrete:

• It gets the right page.
• The answer is actually inside what it receives, not one more call away.
• It costs few tokens, since retrieval tokens are tokens not spent on the work.
• It is fast.
• When the pack has nothing on the question, it says so instead of returning the closest-sounding noise.

What we count, per question

Right page first, and right page in the top 3. Every result cites the page it came from, and we check that page against the one the question's author marked as answering it (a few questions accept more than one page).

Answer delivered. Each question carries a short key string from its answer, and the question counts only if that string reaches the agent. For the squished-thumbnails question quoted below, the string is object-cover. A search can rank the right page first and still fail here, if what it sends stops short of the part that answers.

Tokens are bytes received divided by four, reported as the median per question. Latency is the median per call. Abstention is how often search says "nothing here" when a pack is asked another library's questions.

We also compute mean reciprocal rank, which averages 1 divided by the position of the right page (1.0 means always first).

The questions we tuned on said we had won

On 2026-10-01 we ran our first comparison on a 56-question Next.js set. Warmhand looked ahead: the right page was in our top 3 for 53 of 56 questions, against 45 of 56 for Context7. We had a draft post claiming the win.

The catch: our search had been tuned on those 56 questions. So we wrote 60 fresh Next.js questions, blind, and ran both systems once. Context7 won clearly. It ranked the right page first 0.80 of the time against our 0.60, delivered the answer 0.90 of the time against our 0.73, and sent about 1,000 tokens per question where we sent about 5,900. The draft was wrong, and the held-out set is the only reason we caught it before publishing.

The full set of rules:

• Same questions, same scoring code, both sides. Context7 was queried with an API key, on both its repo-built and its website-built index for each library.

• Questions written blind. Agents that read only the docs wrote them, never either system, so nobody could write questions a particular search happens to be good at. They span identifier lookups, how-tos, error messages, conceptual questions and version changes. Two how-tos from the Tailwind held-out set, verbatim:

  Thumbnail images are getting squished, I want them to fill the box and crop instead

  My row of tag chips overflows the container, I want them to continue on the next line

  Neither names the utility it needs (object-cover, flex-wrap), which is the point.

• A tuning set to iterate on, held-out sets run once. Per library there is a tuning set, held-out set 1 (the head-to-head) and held-out set 2 (60 fresh questions, 40 for Knex).

• Every failed call is a miss. Errors and timeouts stay in the denominator. Context7's "no relevant snippets" response counts as finding nothing.

• Raw answers are cached. Context7's responses are stored with their source lines, so a scoring fix re-scores without re-querying. That matters when the other side's quota is finite (see the caveats).

We also found four bugs in our own harness, two each way. Two were unfair to Context7: file names containing spaces, and how its website URLs mapped back to pages. Both were fixed before its numbers were frozen. Two were unfair to us: results with no kind field, and answer strings containing quotes, which were checked against escaped JSON (that one touched 5 of 968 questions). The "before" rows below were scored before our two fixes, so if anything they understate the old search slightly.

Before: one vector per page, then read the whole page

Each docs page was one embedding, built from up to 24,000 characters, and once search picked a page, the agent read all of it. One vector had to stand for everything on a long page, so a specific answer in the middle had little pull on it. And when the page did rank, the agent paid for every word.

Held-out set 1. Each cell reads Warmhand first, then Context7's repo-built / website-built index.

Library (questions)	Right page first	Right page in top 3	Answer delivered	Tokens (median)
Next.js (60)	0.65 · 0.80 / 0.70	0.88 · 0.93 / 0.90	0.90 · 0.90 / 0.92	6,050 · 1,024 / 907
React (40)	0.65 · 0.65 / 0.65	0.80 · 0.93 / 0.80	0.80 · 0.82 / 0.78	8,650 · 1,094 / 1,038
NestJS (40)	0.58 · 0.85 / 0.42	0.92 · 0.93 / 0.42	0.66 · 0.88 / 0.60	9,395 · 1,068 / 653
Tailwind CSS (40)	0.57 · 0.70 / 0.88	0.85 · 0.85 / 0.97	0.75 · 0.88 / 0.82	6,792 · 695 / 462
TypeScript (40)	0.40 · 0.62 / 0.23	0.62 · 0.75 / 0.25	0.57 · 0.85 / 0.50	9,435 · 902 / 613
Knex (30)	0.77 · 0.87 / 1.00	0.97 · 0.93 / 1.00	0.80 · 0.87 / 0.80	20,702 · 623 / 473

(These rows come from a later run on our local eval stack, and its Next.js figures differ a little from that first day's.)

Sections, not pages, in the order we measured them

Each change below was kept only if it won on the tuning sets.

1. Rank heading sections instead of pages. A page ranks by its best section, and search returns the best sections as passages, so the agent no longer reads whole pages.
2. At sync time, a model writes the questions each section answers, and those are indexed alongside it (never shown to the agent), plus a short description per page. This is what lets plain-language how-tos find utility pages they never name. It beat rewriting the agent's question at search time, which added about 1.4 seconds per search.
3. A lean response: passages plus bare page titles, with no per-result snippet or metadata.
4. A similarity floor that decides one thing only, whether to say "nothing here". It no longer filters the ranking.
5. Adaptive passage counts and a budget scaled by confidence: a confident match gets up to 4 passages and 3,000 characters, a less confident one up to 6 and 4,800.
6. BM25 for the keyword side of the search, because Postgres's built-in ranking has no notion of how rare a word is or how long a section is.
7. Sections of 250 to 1,000 characters, and MDX demo markup and comments stripped before indexing.

Same held-out set, same layout, Warmhand now running section search:

Library (questions)	Right page first	Right page in top 3	Answer delivered	Tokens (median)
Next.js (60)	0.83 · 0.80 / 0.70	0.98 · 0.93 / 0.90	0.95 · 0.90 / 0.92	824 · 1,024 / 907
React (40)	0.88 · 0.65 / 0.65	0.95 · 0.93 / 0.80	0.80 · 0.82 / 0.78	788 · 1,094 / 1,038
NestJS (40)	0.95 · 0.85 / 0.42	1.00 · 0.93 / 0.42	0.85 · 0.88 / 0.60	680 · 1,068 / 653
Tailwind CSS (40)	0.88 · 0.70 / 0.88	0.95 · 0.85 / 0.97	0.95 · 0.88 / 0.82	593 · 695 / 462
TypeScript (40)	0.70 · 0.62 / 0.23	0.88 · 0.75 / 0.25	0.97 · 0.85 / 0.50	881 · 902 / 613
Knex (30)	1.00 · 0.87 / 1.00	1.00 · 0.93 / 1.00	0.97 · 0.87 / 0.80	420 · 623 / 473

Where we now lead: on Next.js we are ahead of both Context7 indexes on every column. On TypeScript we lead both on ranking and answer delivery, and use fewer tokens than its repo index (the website index is leaner at 613 tokens, and ranks the right page first 0.23 of the time). On Knex we tie its website index at a perfect 1.00 on ranking and lead on delivery and tokens. Right page first is a lead on four libraries and a tie on the other two (Tailwind at 0.88, Knex at 1.00).

Where we do not:

• Answer delivery on React (0.80 against 0.82) and NestJS (0.85 against 0.88). Each gap is one question of 40.
• Tailwind. Context7's website index uses fewer tokens (462 against our 593), gets the right page into the top 3 one question more often (0.97 against 0.95), and edges us on mean reciprocal rank (0.93 against 0.91).
• On NestJS, Context7's website index is slightly leaner than us (653 tokens against 680), though it ranks the right page first 0.42 of the time.

Abstention is a column we only measured for ourselves. Asked another library's questions, the new search said "nothing here" for 93 to 100 percent of them (held-out set 2 runs). We did not run Context7 on those (an agent using it would pick a different library first), so we cannot say how it compares. For an agent, "nothing here" is a useful answer: it can go look elsewhere instead of building on a near miss.

On held-out set 2, which we never tuned on (Warmhand only), section search ranks the right page first 0.72 to 0.92 of the time, puts it in the top 3 for 0.95 to 1.00, and delivers the answer for 0.92 to 1.00, at 446 to 894 tokens.

What these numbers do not tell you

• Latency is not comparable here. Warmhand ran on localhost (464 to 543 ms median), Context7 over the internet (1,839 to 2,297 ms). Before this work, production Warmhand answered in about 1.2 seconds against Context7's 1.8 to 2.3, and production still needs re-measuring with the new search.
• The head-to-head is held-out set 1 only. Context7's monthly API quota ran out before we could run held-out set 2 on its side. We do not know how it would score there.
• Held-out set 1 stopped being clean. Once set 2 existed, we used set 1 as a development set, which is exactly the habit that fooled us on 2026-10-01. Set 2 is the clean verdict for Warmhand; it just has no Context7 column.
• These sets are small. On 40 questions, one question moves a rate by 2.5 points, and most of Context7's remaining leads are one question each.
• The tables show the measured branch. The version we shipped, with code-review fixes, was re-measured on five of the six libraries and moved by a few points either way (Next.js answer delivery 0.97, TypeScript 0.93).
• Docs packs are not open to everyone yet.

What should we test next?

This time the post came after the held-out set, not before.

We would rather find our next 0.60 in a held-out set than in a draft, and six libraries is a small sample of what agents get asked. If there is a library your agent keeps getting wrong, or a kind of question you suspect our search would fail (vague how-tos, error messages pasted whole, version migrations), reply and tell us which.