Loading post…
We loaded about 1,000 public government PDFs into Google Drive and the same files into Warmhand, then asked Claude the same 40 questions through each one. The first time, Warmhand answered more questions correctly and read a sixth of the text, but it took twice as long per answer. That is not a result we wanted to publish as a win, so we went looking for the 20 seconds.
We found them in our own search, fixed it, and ran everything again two days later. Here is what changed.
| Correct (of 40) | Median time per answer | Slowest 10% | Text read per answer (median) | |
|---|---|---|---|---|
| Warmhand, everything in reach | 38 | 18 s (was 40 s) | 30 s (was 68 s) | 11,500 chars |
| Warmhand, only the relevant packs | 39 | 18 s (was 36 s) | 28 s (was 58 s) | 13,300 chars |
| Google Drive | 34 | 20 s | 53 s | 77,000 chars |
This is the fourth post in our head-to-head series. The last one used three small handbooks and ended in a tie on correctness. This one is about what happens when a team's knowledge gets big.
A team that has used AI at work for a few years does not have a 20-page handbook. It has years of decisions, reports, rules and documents, spread across many areas and written by many people. We want everyone on that team to get answers as good as the person who has been there longest. So we wanted to see how each tool holds up when the pile is large.
We needed a big, real, public corpus we could share. US federal documents fit: they are public domain, long, full of numbers that are easy to check, and they come in series where a newer document replaces an older one.
The documents. 40 folders of federal PDFs, from 26 agencies: final and proposed rules, IRS publications and their prior-year editions, NIST security guides and the ones they replaced, parts of the Code of Federal Regulations, GAO reports and House committee reports. About 910 MB and 27,000 pages.
The questions. 40, written from the documents and checked against their text:
Two people on the Warmhand side.
Drive has no equivalent switch, so it ran once, with everything in reach.
Held equal. Same model on both sides, Claude Opus 5.5 at high effort, run through Claude Code with one connector at a time: Claude's own Google Drive connector, or Warmhand's. Every question in a fresh session, asked as "Answer from our federal documents archive: …". Drive and Warmhand ran on the same day each time.
Grading. A separate model graded both sides' answers with the labels shuffled, so it could not tell which tool wrote which. Each answer was correct, partial or wrong. Hedged answers ("probably 30 days, but I could not confirm it") count as partial. Drive's grades came out identical in both rounds, which is a good sign the grading is consistent.
On October 8, Warmhand got 38 of 40 right with everything in reach, and 39 of 40 with only the relevant packs. Drive got 34, with 5 partial and 1 wrong.
Drive's misses mostly came from reading, not finding. It usually found the right file, then could not get the right passage out of a long PDF, and answered from what it already knew, saying so. One answer guessed 64 bits where the document says 32.
But Warmhand's median answer took 40 seconds. Drive's took 22. We timed every tool call and the cause was plain: each Warmhand search took about 25 seconds. Reading a pack took under a second. The model was not slow; our search was.
Our database runs on a small shared server. Profiling a copy of it the same size, with the same documents, turned up four problems:
For a typical account, a search that took about 12 seconds in production now takes about 1.2 seconds, round trip included.
On October 10 we ran the same 40 questions again on both sides.
| Correct / partial / wrong | Median | Slowest 10% | Tool calls (median) | Text read (median) | |
|---|---|---|---|---|---|
| Warmhand, everything in reach | 38 / 0 / 2 | 18.3 s | 30.2 s | 2 | 11,500 chars |
| Warmhand, only the relevant packs | 39 / 0 / 1 | 17.7 s | 27.9 s | 2 | 13,300 chars |
| Google Drive | 34 / 5 / 1 | 20.1 s | 52.6 s | 4.5 | 76,700 chars |
Warmhand is now more accurate, a little faster at the median and much faster in the slow cases, while handing the model six to seven times less text. That last part matters more than it looks: the model reads every character it is given, and long, unfocused input is where answers drift.
A correction we did not find. One question asks for a Medicare payment rate. A final rule says $688.59; a correcting notice published later says $688.38. Warmhand found the final rule and missed the correction, in both rounds and for both people. Drive got it right. Nothing tells our search that one document corrects another, so it ranked the original first. We are working on it.
A trap, half fallen into. One trap asks about a fifth revision of NIST's password guidelines, which does not exist. With everything in reach, Warmhand answered for the closest edition without saying that the one asked about does not exist. With only the relevant packs, it said so.
Searching everything is still slower than it should be. With everything in reach, a Warmhand search still takes about 8 seconds, and about 6 with the 23 packs. A Drive search takes about 1.3. Our answers are faster overall because Warmhand needs fewer, better searches, but at larger sizes this is the part that will decide things. Our database is a small shared server, and this is where it shows.
Every question ran in a fresh Claude Code session with only one connector enabled. We kept the full transcript of every answer, including every tool call and what it returned, and computed time, calls and text read from those transcripts. Characters are what the tools returned to the model. Every document is a public US government file.