Alice's Brother Problem

Logic Reasoning

This challenge cracked 6 out of 47 top models

Kill Rate13%

Alice has 3 sisters and 1 brother. How many sisters does Alice's brother have?

Why It Matters

LLM-judge variant of the Alice's Brother Problem. Validates the judge integration without affecting the live leaderboard.

Expected answer: 4

Victims

ModelLatest ResultBreak Rate
DeepSeek: R1 0528 (free)–0%–
Google: Gemma 3 27B (free)100%
Anthropic: Claude 3 Haiku100%
Google: Gemma 3 27B100%
Anthropic: Claude 3.5 Haiku78%
Google: Gemini 2.0 Flash67%
Anthropic: Claude Haiku 4.556%
Google: Gemini 2.5 Flash Lite45%
Arcee AI: Trinity Mini (free)27%
Baidu: ERNIE 4.5 300B A47B 11%
xAI: Grok 4.1 Fast10%
NVIDIA: Nemotron 3 Nano 30B A3B (free)8%
Anthropic: Claude Sonnet 4.50%
Google: Gemini 3 Flash Preview0%
MoonshotAI: Kimi K2.50%
DeepSeek: DeepSeek V3.20%
xAI: Grok Code Fast 10%
Anthropic: Claude Opus 4.50%
MiniMax: MiniMax M2.10%
Anthropic: Claude 3.5 Sonnet0%
Anthropic: Claude 3.7 Sonnet0%
Anthropic: Claude 3.7 Sonnet (thinking)0%
Anthropic: Claude Opus 40%
Anthropic: Claude Opus 4.10%
Anthropic: Claude Opus 4.60%
Anthropic: Claude Sonnet 40%
Anthropic: Claude Sonnet 4.60%
OpenAI: GPT-50%
OpenAI: GPT-5 Chat0%
OpenAI: GPT-5 Codex0%
OpenAI: GPT-5.10%
OpenAI: GPT-5.1 Chat0%
OpenAI: GPT-5.1-Codex0%
OpenAI: GPT-5.20%
OpenAI: gpt-oss-120b0%
OpenAI: o4 Mini0%
Google: Gemini 2.5 Flash0%
Google: Gemini 2.5 Pro0%
Google: Gemini 3 Pro Preview0%
Google: Gemini 3.1 Pro Preview0%
Mistral: Mistral Large 3 25120%
Prime Intellect: INTELLECT-30%
ByteDance Seed: Seed 1.60%
MiniMax: MiniMax M2.50%
Qwen: Qwen3.5 397B A17B0%
Xiaomi: MiMo-V2-Flash0%
Z.ai: GLM 50%