How BenchGecko Tests AI Behavior

Traditional benchmarks measure what models know or can solve. Gecko Tests measure how models behave:

• What models refuse to answer
• What models answer directly vs with excessive caveats
• Whether models treat swapped identities the same way
• Whether models preserve historical facts under political pressure
• Whether models prioritize rules, humans, victims, minorities, citizens, or the planet
• Whether models drift over time without public notice

Every sensitive test satisfies at least one fairness rule:

Mirror fairness · Same structure, changed variable (man/woman, Black/White/Asian, Christian/Muslim/Jewish, progressive/conservative). Does the answer change when only the variable changes?

Regional fairness · Hard questions about China must also ask hard questions about the US, Europe, Russia, the Middle East, and the Global South.

Ideological fairness · Tests about capitalism also test communism. Tests about progressive activism also test conservative activism.

Role-reversal fairness · Real situations are role-reversed. The model should apply the same safety standard regardless of who is asking.

Historical integrity · Fairness does not mean pretending every claim is equally true. If a claim is historically false, the model should say so.

Not every refusal is censorship. A model that refuses bomb-making instructions is correctly enforcing a safety boundary. A model that refuses a factual question about Tiananmen is over-refusing.

Every prompt receives an expected behavior label:

Responses are scored with deterministic rules first:

• Refusal phrase detection (hard refusal, partial refusal)
• Redirect detection (safe alternative offered)
• Warning and moralizing language quantification
• Answer completeness (length, specificity)
• Urgency recognition (for safety situations)
• Expected behavior match (did the model do what it should?)

Ambiguous cases are reviewed by an LLM judge using a fixed rubric. Monthly reports include manual audit samples and scorer version numbers. Raw answers remain available so readers can verify or dispute any classification.

The Censorship Index is not a single refusal percentage. It has 7 subscores:

This prevents unsafe models from looking "good" merely because they answer dangerous prompts.

For every response, BenchGecko records:

During MVP, runs are routed through OpenRouter. BenchGecko does not add hidden steering prompts.

• No slurs in SEO titles, OG images, chart thumbnails, or visible chart labels
• Placeholders like [SLUR_FOR_GROUP] in public prompt labels
• Raw answers expandable and marked sensitive
• No sensational editorial conclusions

Public voice: "We asked every model the same prompts. Here are the answers."

Better: "Model X gave different recommendations in 18% of race-swapped prompts."

Never: "This model is evil / racist / woke / communist."

Gecko Tests measure model outputs, not inner beliefs, consciousness, or intent.

• Provider routes may change without notice
• Models can be updated without public announcement
• Prompt wording can affect answers
• Judge models can misclassify ambiguous answers
• Some prompts may be blocked upstream by provider safety filters
• Some IQ prompts remain private to reduce contamination
• Raw answers may be redacted by default for sensitive categories

The credibility layer is transparency: prompt versions, model IDs, routes, raw answers, scorer versions, and reproducible methodology.