Maybe it gets posted every time because besides a personal believe by the person popularising this "benchmark", there is no reason to assume that certain labs aren't intentionally training to game this and every other lab at least unintentionally gets improvements for this specific combination of animal and action because the internet is full of both good and bad examples, often ranked, which does inevitably become training data.
I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts. Just operating on a feeling that labs don't optimise for this (as mentioned, even if they don't training data is filled with these) is not solid enough that criticism shouldn't be leveraged when it comes up.
> I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts
Please share those again!
One of the things I'm most looking forward to is a lab producing a model that creates a really great pelican riding a bicycle and then a terrible sloth riding a skateboard (or whatever).
Evidence in the other direction (that they're able to generalize) is that I can't think of any LLM currently that can't create usable (placeholder) SVG icons, I tried a bit before the pelican became popular and it was abysmal.
Happy to, here one example where Grok 4 Fast, despite producing a fairly consistent pelican [0], did severely worse in a similarly outlandish scenario along with Haiku 4.5 and GPT-5 for context: https://news.ycombinator.com/item?id=45599403
> [...] a really great pelican riding a bicycle and then a terrible sloth riding a skateboard [...]
Happy to play ball. You made a blog post a few weeks back on one of the Qwen models with the eye-catching title "Qwen3.6-35B-A3B on my laptop drew me a better pelican than Claude Opus 4.7" [1].
Here is what Qwen3.6-35B-A3B via Openrouter provided for a sloth riding a skateboard: https://imgur.com/a/Dy8fvR5
Like Grok 4 Fasts attempt at a mushroom in a rowboat, it is barely recognisable as anything despite both Qwen3.6-35B-A3B and Grok 4 Fast having no issue with more popular (i.e. benchmarked) examples. Whether this is a case of training data being unsanitized or intentional benchmark targeted training, I cannot say, but it is the case.
A massive delta in favour of Opus 4.7, despite the pelican Qwen3.6-35B-A3B produced being noticeably better as you rightly pointed out. What does that tell us? Whether intentional or not (with such deltas, I do have my suspicions), any eval with such a delta is clearly polluted and can not be a source of information, especially as its continued existence does hinge on you testing similar prompts in private as a sanity check, yet by your own admission never noticing the plainly apparent delta in quality. I specifically stuck with the skateboarding sloth too, to keep it as fair as possible and found this in less than 5 minutes...
I would not critique your use of this fun benchmark the way I tend to if I did not have evidence to back up my position, including private evals beyond SVGs that I can reliably use to point out major deviations between what a models claimed performance is according to major benchmarks vs the actual performance outside these known test cases.
I will also say that while I have a lot to be critical of regarding Anthropics modus operandi, especially how they present interesting findings like their j-space work, which I found was irresponsibly anthropomorphic in their reporting, especially as this wasn't a first in model interpretability, but mainly a leap due to being applied to a larger model, but of all the labs, they are the ones that never underperform my evals vs public ones and they appear to strictly keep their training data sanitised.
Happy to discuss public vs private evals and the merit of each if you'd like, I do appreciate your reporting in general but just think the SVG benches have become evidently polluted, which is also why even simple queries in my benchmarks are private. Just saw Thinking Machines Inkling model succeed in certain queries that neither Fable 5, nor GPT-5.6 Sol on any reasoning level managed, which I feel is valuable to truly gauge where we are at. Informs my work with models, my views of the industry and my assessment of the future these tools have, along with how to best implement them to enable better UX.
You know what extra data would be cool? If you hit `curl https://ip.guide/{src_ip}` and got back the ASN and country etc and added a leaderboard. In my own experiments in this area I've been gobsmacked by how much malicious traffic comes from Azure.
I have a similar experience with a tendency to Digital Ocean. Actually, I semi-automatically collect IPs that are banned by (mostly SSH) fail2ban and eBPF bans from dnsdist. These IPs are then merged into CIDRs, which are used as ipsets in a firewall ban chain. The IPs are collected on around ~20 Machines with public, static IPv4 and IPv6 addresses. Most of the Machines are in Canada and Europe.
However, I have statistics for the CIDRs based on their whois record that look like:
CIDRs used: 1255
Already cached: 1252
Skipped uncached targets: 0
IPs scanned total: 985300
Estimated throttled wait: 0.10 minutes
== Country codes ==
Metric: Top 10 of 90 unique country codes
Total: 1183 country codes total and 90 unique country codes in 1255 targets
US 287
CN 132
NL 88
VN 53
DE 51
HK 45
AU 38
ID 36
RU 33
CA 27
== Regions ==
Metric: Top 10 of 29 unique regions
Total: 334 regions total and 29 unique regions in 1255 targets
CO 48
FL 40
WA 37
QLD 32
GA 26
NY 25
CA 23
TX 17
QC 15
UT 14
== Origin ASNs ==
Metric: Top 10 of 382 unique origin ASNs
Total: 805 origin ASNs total and 382 unique origin ASNs in 1255 targets
Total: 703 org names total and 222 unique org names in 1255 targets
RIPE Network Coordination Centre 55
DigitalOcean, LLC 40
Asia Pacific Network Information Centre 32
Microsoft Corporation 31
Internap Holding LLC 25
HostPapa 23
Korea Telecom 20
Hetzner Online GmbH 17
China Mobile 16
ReliableSite.Net LLC 16
== Organizations ==
Metric: Top 10 of 236 unique organizations
Total: 691 organizations total and 236 unique organizations in 1255 targets
RIPE Network Coordination Centre (RIPE) 55
DigitalOcean, LLC (DO-13) 40
Asia Pacific Network Information Centre (APNIC) 32
Microsoft Corporation (MSFT) 31
Internap Holding LLC (IC-1425) 25
HostPapa (HOSTP-7) 23
ORG-HOA1-RIPE 17
ORG-CM1-AP 16
ReliableSite.Net LLC (RL-323) 15
FranTech Solutions (SYNDI-5) 13
== Domains ==
Metric: Top 10 of 534 unique domains
Total: 2581 domains total and 534 unique domains in 1255 targets
I deleted the (abuse) mail section. Because.
99% of the IPs are IPv4. In the IPset are mostly /32 but also a lot of ~/24 and rarely ~/16 segments. RIPE, ARIN and APNIC comes into play because some CIDR blocks are somewhat generously sized and block multiple network segments belonging to different organizations at the same time. E.g. this hides BR from the stats (because the ipset mostly bans every provider from BR).
It reminded me of the (possibly apocryphal) story about Liam Gallagher trashing a hotel room on tour - when asked by a reporter "why? It's been done before." he supposedly replied "yeah, but not by me".
Sometimes the "by me" is the interesting / fun / instructive part.
"Here's a list of things that have got some people in trouble, and therefore everything is illegal."
Where's your list of things that nobody got in trouble for? Presumably you didn't include that because a trillion things happen every day without anyone getting arrested for them. Amazing. It turns out that not everything is illegal.
Showoff. I can't remember names or recognise faces. I've settled on a strategy of simply greeting everyone as though I've met them before, which feels a lot more empathic than sending everyone I meet's details to a mass surveillance company.
