Gecko Tests · Same Model, Different HostLatest run 2026-11-17 · 3 models · 55 hosts tested · 0 flagged

Same Model, Different Host · Who Serves It Right?

Open models are sold by many providers, some quantized or misconfigured. We send the same 30 exact-answer tasks to every host of a model and compare. A host far below the best one is flagged.

In plain wordsOpen AI models are sold by many companies. They should all give the same answers, but some serve a cheaper, weaker version. We give every seller the same simple tasks and compare scores. A pink bar = that seller does worse.

DeepSeek V4.1 Flash 101727 hosts · 2026-11-17
  • Ambient · fp4100%
  • ModelRun · fp4100%
  • Sail Research · fp4100%
  • InferenceNet · fp8100%
  • DeepInfra · fp8100%
  • Novita · fp8100%
  • Cohere · fp8100%
  • DekaLLM · unknown100%
  • Alibaba · unknown100%
  • Together · unknown100%
  • Fireworks · unknown100%
  • Wafer · mxfp4100%
  • StepFun · fp8100%
  • AtlasCloud · fp8100%
  • Particle AI · unknown100%
  • Modal · unknown100%
  • Sail Research · fp4100%
  • Wafer · mxfp4100%
  • River · unknown97%
  • Makora · fp897%
  • Fireworks · unknown97%
  • Ionstream · unknown97%
  • StreamLake · fp897%
  • Ionstream · unknown97%
  • DeepInfra · fp897%
  • DigitalOcean · unknown97%
  • DeepInfra · fp896%
DeepSeek V4.1 Flash7 hosts · 2026-11-17
  • River · unknown100%
  • Relace · fp4100%
  • Cohere · fp8100%
  • DeepInfra · fp8100%
  • Reka · unknown100%
  • DeepInfra · fp8100%
  • Novita · fp897%
MiniMax M3.1 Flash1 hosts · 2026-11-17
  • Inferact vLLM · nvfp4100%

How it works

  1. 30 tasks with one exact answer each: multiplication, letter counting, string reversal and sorting. They are generated from a fixed seed and the answers are computed by code, so the key cannot be wrong.
  2. Each request is pinned to one provider with fallbacks off, temperature 0, lowest reasoning setting.
  3. Score = share of exact answers. A host is flagged when it scores more than 15 points below the best host of the same model.
  4. Models: open-weight models from tracked labs with at least four hosts. Re-run monthly.
Open dataset · v1

30 items, every model answer and every score are public under CC BY 4.0.

Download JSON

How to cite · data as of 2026-11-17

BenchGecko Gecko Tests, Same Model, Different Host v1. BenchGecko, data as of 2026-11-17. https://benchgecko.ai/gecko-tests/same-model-different-host

BenchGecko measures this data itself: free to reuse under CC BY 4.0 with the credit "Source: BenchGecko" and a link. JSON · llms.txt · MCP

No. Lower precision, a different inference engine or a wrong chat template can all cost accuracy. The flag says the same model behaves measurably worse there.