Why such simple tasks?

Exact arithmetic and character-level tasks are sensitive to precision loss and have no grading ambiguity, which makes differences between hosts easy to see.

Which quantization does each host use?

Where the provider reports it, it is shown next to the host.

Gecko Tests · Same Model, Different HostFirst run in progress

Same Model, Different Host · Who Serves It Right?

Open models are sold by many providers, some quantized or misconfigured. We send the same 30 exact-answer tasks to every host of a model and compare. A host far below the best one is flagged.

First run in progress.

How it works

30 tasks with one exact answer each: multiplication, letter counting, string reversal and sorting. They are generated from a fixed seed and the answers are computed by code, so the key cannot be wrong.
Each request is pinned to one provider with fallbacks off, temperature 0, lowest reasoning setting.
Score = share of exact answers. A host is flagged when it scores more than 15 points below the best host of the same model.
Models: open-weight models from tracked labs with at least four hosts. Re-run monthly.

Open dataset · v1

30 items, every model answer and every score are public under CC BY 4.0. Cite as “BenchGecko Gecko Tests, Same Model, Different Host v1” with a link.

Download JSON

Frequently Asked Questions

No. Lower precision, a different inference engine or a wrong chat template can all cost accuracy. The flag says the same model behaves measurably worse there.