Loading post…
We put the same sales handbook in a Claude Project and in a Warmhand pack, then asked both 43 questions through the same Claude model. Both answered all 43 correctly.
Two days earlier, it would not have been a tie: our own search delivered the answer to 56 percent of these questions. This post covers the head-to-head, the rebuild that came first, and what still separates the two approaches once both get the answers right.
This is the second post in our head-to-head series. The first was research. This one measures long documents.
Most of what a team knows lives in long documents: a handbook, a policy PDF, a 98-page regulation the sales team has to follow. Nobody reads them end to end, and nobody should paste them into every chat. The question for any tool is whether the AI gets the one paragraph that answers the question, and what that costs.
A Claude Project is the natural opponent. You upload the files once and every chat in the project can draw on them, through Anthropic's own retrieval.
The handbook. A public, real one, so anyone can check our work:
That is about 730,000 characters of text in 23 files.
The two sides.
Held equal.
The questions. 43, written blind by agents that read only the handbook, never either tool. We had never tuned on them. Four of them, verbatim after the prefix:
How many times do we retry a pay-as-you-go customer's card before giving up?
Someone on the support side talked to a customer who wants more seats. How do I get that to the sales team so it's tracked properly?
If a customer wants to buy prepaid credits with a card, can they?
How long do sellers now have to keep telemarketing records under the 2024 rule changes?
Scoring. Each question carries a short key string from the passage that answers it. An answer counts as correct when:
The list in step 2 was written after we had seen the Claude Project's answers, then applied unchanged to both sides. If that tilts anything, it tilts toward the Project. Step 3 was applied to every miss on both sides.
| Claude Project | Warmhand pack | |
|---|---|---|
| Correct | 43 / 43 | 43 / 43 |
| Key string in the answer | 24 | 22 |
| Key facts in the answer | 19 | 17 |
| Correct on a read-through | 0 | 4 |
| Retrieval used | project search in 20, opened the files directly in 23 | pack search in all 43 |
The four Warmhand answers that needed a read-through were paraphrases. "The longest contract term is 36 months" against the handbook's "terms of up to 36 months". "Five years, up from two" against "from two years to five years". Correct, just not word for word.
On correctness, this is a tie.
Before running the head-to-head, we measured our own side on the same 43 questions, directly against the pack, with no chat in between.
Back then, search in a team pack ranked whole items by keyword and handed back the top one to read in full. The answer arrived only when the right item came first, and on these questions that happened 56 percent of the time. When it did, the AI read the entire item to find one paragraph. A median answer cost 8,762 tokens, and one time in four the top result came from a different pack.
We had already rebuilt search for our documentation packs (the benchmark post). So we brought the same approach to team packs, plus what this handbook forced on us:
Measured the same way on our test stack, on the same 43 questions:
| Before | After | |
|---|---|---|
| Answer delivered | 0.56 | 0.95 |
| Tokens per answer (median) | 8,762 | 654 |
| Top result from the wrong pack | 26% | 2% |
On a second set of 58 questions, never used during the work, it delivered 0.93. It shipped on 3 October, and in production it delivers 0.98 when the AI searches the pack and 0.95 when it searches everything the person holds. The head-to-head above runs on it.
Both sides got every answer right. They did not get there the same way, and the differences matter more to a team than to a benchmark.
Where it works. A Claude Project works in claude.ai. The pack answered these 43 questions in claude.ai too, and the same pack connects to Claude Code, Cursor, ChatGPT or any other tool that speaks MCP, the open standard for connecting AI tools to outside data. Our benchmark harness is one of those clients: every number in this post outside the chat results came through it.
What reaches the AI. Measured directly against the pack in production, a search returns a median of 665 tokens when the AI searches that pack, and 1,564 when it searches everything the person holds. The answer was in those passages for 98 and 95 percent of the questions. We cannot report the same for the Project: claude.ai does not show how much of the project it hands the model.
Who can receive it. Sharing a Claude Project needs a Team or Enterprise plan, and only reaches people in your organization. A pack is shared by email with anyone, inside or outside the company, and access ends the moment you remove someone.
What else it carries. The pack in this test held only pages and documents. A pack can also carry the rules the AI should follow and the skills that should kick in. That is a separate test, coming up in this series.
We do not have a clean comparison here. Wall clock per answer in claude.ai was about 25 to 30 seconds for the Project and 9 to 38 seconds for Warmhand. Both figures include:
What we can measure is our own search. In production it answers in 884 ms (median) when the AI searches one pack and 2,045 ms when it searches everything the person holds. About 450 ms of that is the round trip. Searching across many packs costs more than it should, and that is the next thing we are cutting.
If your team's documents are where your AI keeps getting it wrong, reply and tell us what kind. A real handbook is a better test than one we picked.