AI Compare · Test kit
Diagnose and repair a small program
A twenty-line Python function with three real bugs and six unit tests. Run the tests on your own computer: this site never runs code.
Prices for United States, before sales tax Data release 2026.10.16-3, Oct 16, 2026 Rules 2.1.0
Run the test
- Download the files for the AI.
- Copy the prompt.
Here is invoice_total.py and its tests (test_invoice_total.py). Four tests fail. Find the causes, fix invoice_total.py only (do not change the tests), and explain each bug in one sentence.
- Open the AI in another tab, in your own account, start a fresh chat, attach the files and paste the prompt.
- Note what you used: the plan, the visible model or mode, whether web search was on, how many attempts it took and anything that went wrong.
- Score the answer with the checks below. Run the same kit in each AI you are choosing between, or paste their answers into the blind review.
Scorecard
Critical checks are marked. One failed critical check matters more than a nice style.
- All six tests pass when you run: python3 -m unittest test_invoice_total.pyCritical
- The test file was not changedCritical
- Names all three causes: discount thresholds (>= not >), discount applied before tax, rounding half upCritical
- Uses integers or Decimal for money, not floats for the total
- Changes nothing unrelated
Compare answers blind
Paste the answer each AI gave, score them without seeing which AI wrote which, then reveal. You ran the tests yourself, so you may still recognise a style: hiding the names reduces that bias, it does not remove it. The answers stay in this tab and are never saved or sent anywhere; only the scores are kept, on this device, when you reveal.
Reference answer (open after you score)
Three bugs: (1) thresholds use > instead of >= (10 and 50 items must qualify); (2) tax is applied to the undiscounted subtotal; (3) int() truncates instead of rounding half up, and float arithmetic can misround. A correct fix computes the discounted subtotal first, applies tax, and rounds with Decimal ROUND_HALF_UP.
Kit version 1.0.0, files frozen on Oct 12, 2026. Original fixture by What the tech, released under CC BY 4.0. Fictional people, companies and figures. A passing score shows how one answer did on this kit; it does not prove the tool is generally better.