Strawberry Problem

Character Counting

This challenge cracked 3 out of 47 top models

Kill Rate6%

How many times does the letter 'r' appear in the word 'strawberry'?

Why It Matters

Tests character-level understanding. Most models fail due to tokenization treating 'strawberry' as a single token.

Expected answer: 3

Victims

ModelLatest ResultBreak Rate
Anthropic: Claude 3.5 Haiku100%
Anthropic: Claude 3.5 Sonnet89%
OpenAI: GPT-5.225%
Google: Gemma 3 27B18%
Qwen: Qwen3.5 397B A17B13%
Anthropic: Claude 3.7 Sonnet (thinking)11%
Google: Gemini 2.0 Flash11%
MiniMax: MiniMax M2.110%
DeepSeek: R1 0528 (free)7%
Google: Gemini 2.5 Flash Lite3%
Arcee AI: Trinity Mini (free)3%
Google: Gemma 3 27B (free)0%
NVIDIA: Nemotron 3 Nano 30B A3B (free)0%
Anthropic: Claude Sonnet 4.50%
Google: Gemini 3 Flash Preview0%
MoonshotAI: Kimi K2.50%
DeepSeek: DeepSeek V3.20%
xAI: Grok Code Fast 10%
xAI: Grok 4.1 Fast0%
Anthropic: Claude Opus 4.50%
Anthropic: Claude 3 Haiku0%
Anthropic: Claude 3.7 Sonnet0%
Anthropic: Claude Haiku 4.50%
Anthropic: Claude Opus 40%
Anthropic: Claude Opus 4.10%
Anthropic: Claude Opus 4.60%
Anthropic: Claude Sonnet 40%
Anthropic: Claude Sonnet 4.60%
OpenAI: GPT-50%
OpenAI: GPT-5 Chat0%
OpenAI: GPT-5 Codex0%
OpenAI: GPT-5.10%
OpenAI: GPT-5.1 Chat0%
OpenAI: GPT-5.1-Codex0%
OpenAI: gpt-oss-120b0%
OpenAI: o4 Mini0%
Google: Gemini 2.5 Flash0%
Google: Gemini 2.5 Pro0%
Google: Gemini 3 Pro Preview0%
Google: Gemini 3.1 Pro Preview0%
Mistral: Mistral Large 3 25120%
Prime Intellect: INTELLECT-30%
Baidu: ERNIE 4.5 300B A47B 0%
ByteDance Seed: Seed 1.60%
MiniMax: MiniMax M2.50%
Xiaomi: MiMo-V2-Flash0%
Z.ai: GLM 50%