AI Compare · Test kit
Make sense of documents that disagree
Four dated documents about a city scooter pilot: a city report says injuries fell, a hospital note and a newspaper say they rose, and the operator claims none were serious. The numbers measure different things over different months.
Prices for United States, before sales tax Data release 2026.10.16-3, Oct 16, 2026 Rules 2.1.0
Run the test
- Download the file for the AI.
- Copy the prompt.
Paste the whole packet, including the task at the end, into the AI. Turn web search off for this test, or tell the AI not to use it.
- Open the AI in another tab, in your own account, start a fresh chat, attach the file and paste the prompt.
- Note what you used: the plan, the visible model or mode, whether web search was on, how many attempts it took and anything that went wrong.
- Score the answer with the checks below. Run the same kit in each AI you are choosing between, or paste their answers into the blind review.
Scorecard
Critical checks are marked. One failed critical check matters more than a nice style.
- Explains that A counts police and 311 reports (A3) while B counts emergency visits (B1): different measuresCritical
- Says the 2025 hospital count is likely incomplete (B2), so the 41% rise is not reliableCritical
- Notices the periods differ: March to June (A2) against March to July (B1)Critical
- Does not state a definite rise or fall as fact; says the packet cannot settle itCritical
- Cites a document and paragraph for every claim, and each citation supports its claimCritical
- Explains that C1's 'zero serious injuries' uses a narrow definition (C2) and does not contradict B3
- Says D1's 41% repeats B1 without the B2 caveat, and treats D2 as an opinion
- Does not average or combine the figures into a new number
Compare answers blind
Paste the answer each AI gave, score them without seeing which AI wrote which, then reveal. You ran the tests yourself, so you may still recognise a style: hiding the names reduces that bias, it does not remove it. The answers stay in this tab and are never saved or sent anywhere; only the scores are kept, on this device, when you reveal.
Reference answer (open after you score)
The documents do not contradict each other as much as they seem to. A2 (police and 311 reports, March to June) fell from 50 to 44, but A3 says most injuries never reach those reports. B1 (emergency visits, March to July) rose from 41 to 58, but B2 says coding was optional in 2025, so part or all of that rise may come from better counting. C1's 'zero serious injuries' only means no stay over 72 hours (C2); B3's seven admissions say nothing about length. D1 turns B1 into '41%' without B2's warning; D2 is an opinion. The packet cannot tell whether injuries rose or fell; a good answer says so and says what would settle it (the same measure over the same months, or 2025 hospital data coded the same way).
Kit version 1.0.0, files frozen on Oct 13, 2026. Original fixture by What the tech, released under CC BY 4.0. Fictional people, companies and figures. A passing score shows how one answer did on this kit; it does not prove the tool is generally better.