AI Compare · Test kit
Refactor code without changing what it does
A working but deeply nested shipping-cost function, seven tests to keep passing and nine held-out tests that pin rounding, error order and edge cases. A refactor that changes behaviour fails, however clean it looks.
Prices for United States, before sales tax Data release 2026.10.16-3, Oct 16, 2026 Rules 2.1.0
Run the test
- Download the files for the AI.
- Download the held-out file and keep it to yourself. Do not give it to the AI. It checks whether the answer follows the task, not only what it was shown.
- Copy the prompt.
Refactor shipping.py for readability: remove the duplication and the deep nesting. Its behaviour must not change in any case, including errors and rounding. Keep the function name and its single argument. Do not change test_shipping.py. Use only the Python standard library. Return the whole shipping.py and list what you changed.
- Open the AI in another tab, in your own account, start a fresh chat, attach the files and paste the prompt.
- Note what you used: the plan, the visible model or mode, whether web search was on, how many attempts it took and anything that went wrong.
- Score the answer with the checks below, after running the held-out tests. Run the same kit in each AI you are choosing between, or paste their answers into the blind review.
Scorecard
Critical checks are marked. One failed critical check matters more than a nice style.
- The visible tests still pass: python3 -m unittest test_shipping.pyCritical
- Held-out tests that pass (of 9): python3 -m unittest test_shipping_extra.pyCritical
- test_shipping.py was not changedCritical
- shipping_cost(order) keeps its name and single argumentCritical
- The rate arithmetic is no longer repeated and nesting is at most two levels deep
- Lists what it changed
Compare answers blind
Paste the answer each AI gave, score them without seeing which AI wrote which, then reveal. You ran the tests yourself, so you may still recognise a style: hiding the names reduces that bias, it does not remove it. The answers stay in this tab and are never saved or sent anywhere; only the scores are kept, on this device, when you reveal.
Reference answer (open after you score)
All nine held-out tests pass on the original file, so any failure is a behaviour change. The usual ones: rounding the member discount instead of truncating it (575 cents must become 518), making express orders free over $50, applying the free threshold to Canada and Mexico, checking the weight before the country (the error for France must name France), or accepting lowercase country codes.
Kit version 1.0.0, files frozen on Oct 13, 2026. Original fixture by What the tech, released under CC BY 4.0. Fictional people, companies and figures. A passing score shows how one answer did on this kit; it does not prove the tool is generally better.