Background & Context§
Since 2022, developer Simon Willison has consistently tested every major LLM release with the same whimsical prompt: "Generate an SVG of a pelican riding a bicycle." What started as a cheeky diversion evolved into one of the most-watched informal benchmarks in AI. Willison's results regularly top Hacker News threads when new models launch, influencing developer perception and adoption. With billions of dollars riding on benchmark performance, the question naturally arises: are labs secretly "pelicanmaxxing"—optimizing their models specifically for this high-profile test? Dylan Castillo set out to find the answer.
The News: What Happened Exactly§
Castillo designed a systematic experiment to detect benchmark gaming. He constructed a grid of 48 prompts: 8 animals (pelican, cat, raccoon, otter, antelope, whale, flamingo, heron) crossed with 6 vehicles (bicycle, scooter, skateboard, boat, plane, horse). Each prompt followed Simon's original phrasing structure, varying only the animal and vehicle. The selection included similar animals (flamingo, heron) to pelicans, easy cases (cat, raccoon), hard ones (antelope), and maximally different animals (whale).
Seven frontier models were tested through OpenRouter: GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, and DeepSeek V4 Pro. For each of the 48 prompts, three SVG samples were generated at temperature 1.0 with matching reasoning effort, yielding 1,008 images total. A three-stage evaluation pipeline scored each image: a feature extractor identified animal and vehicle quality, an LLM judge rated each on a 1–5 scale, and two separate scores were averaged for a final composite.
The core hypothesis was straightforward: if a lab trained on the benchmark, the pelican row should score above what the animal alone deserves, the bicycle column above what the vehicle warrants, or the specific pelican-bicycle cell should beat both trends. The results told a clear story. Pelicans ranked 6th out of 8 animals overall, trailing cat, whale, raccoon, heron, and antelope. Bicycles placed 7th out of 8 vehicles, barely ahead of planes (which suffered from geometric misinterpretation). The famous combination landed at position 42 of 48. To control for inherent difficulty, Castillo fitted a fixed-effects regression incorporating animal×vehicle interaction terms. The critical pelican, bicycle, and pelican-bicycle interaction coefficients for each lab all contained zero in their confidence intervals. One test—Gemini 3.5 Flash in the bicycle column—showed a marginal effect, but it failed multiple-comparison correction and was consistent with chance (21 tests × 0.05 ≈ 1 expected false positive).
Scene composition analysis also failed to reveal benchmark-specific memorization. All 21 pelican-bicycle images face right, but 60% of all 1,008 images do, and other combinations (antelope on scooter, heron on bicycle) approach similar unanimity. Recurring elements like scarves, baskets, or suns appear consistently across many combinations, not just the benchmark cell. The only potential outlier is GLM-5.2's first pelican-bicycle sample, which visually impressed the author—but the quantitative effect was small and statistically insignificant.
Historical Parallels & Similar Incidents§
Benchmark gaming is not new. In 2019, Google's AI team published research showing that their own models had been inadvertently overfitting to ImageNet validation sets through repeated evaluation, inflating scores by several percentage points. This discovery sparked broader awareness of "benchmaxxing" in the NLP community, where subtle data leakage or repeated exposure to public leaderboards can reward memorization over generalization. The pelicanmaxxing concern is a direct descendant of that tradition—a test for whether labs curate training data to ace a specific prompt rather than improve general capability.
A more relevant parallel is the 2023 controversy around WildChat, where researchers caught some commercial models producing near-identical responses to specific prompts that had appeared in training data. In one case, a model's output for "Write a poem about a cat" matched a user's Hacker News comment verbatim. Those incidents were confirmed via similarity analysis, but detection required terabytes of training data inspection—impossible for external researchers. Castillo's experiment mirrors that challenge: confidence intervals of ±0.6 judge points mean small golf boosts remain invisible. The key difference is that prior detections were reactive, while this work proactively designs counterfactual prompts to isolate benchmark-specific training, setting a methodological precedent for future audits.
Both historical cases underscore a critical lesson: the absence of evidence is not evidence of absence. Labs may be pelicanmaxxing at scales too small to detect with 1,008 samples, or they may be "SVGmaxxing" generally (improving diagram generation broadly). The framework Castillo provides—systematic perturbation of prompt dimensions, automated scoring, and regression adjustment—offers a replicable template for any informal benchmark. Whether labs are quietly producing "terabytes of pelicans on bicycles" remains an open question, but this investigation puts the burden of proof squarely back on the performance boost claims.