For those claiming they rigged it. Do you have any concrete evidence? What if the models have just gotten really good?
I just asked Gemini pro to generate an SVG of an octopus dunking a basketball and it did a great job. Not even Deep Think model. Then I did "generate an svg of raccoon at a beach drinking a beer" you can go try this out yourself. Ask it to generate anything you want in SVG. use your imagination.
Rant:
This is why AI is going to take over, folks are not even trying the least.
> What if the models have just gotten really good?
Kagi Assistant remains my main way of interacting with AI. One of its benefits is you're encouraged to try different models.
The heterogeneity in competence, particular per unit in time, is growing rapidly. If I'm extrapolating image-creation capabilities from Claude, I'm going to underestimate what Gemini can do without fuckery. Likewise, if I'm using Grok all day, Gemini and Claude will seem unbelievably competent when it comes to deep research.
Every bit of improvement on AI ability will have the corresponding denial phrase. Some people still think AI can't generate the correct number of fingers today.
It was a terrible test for pure tokenized models, because the logit that carries the carry digit during summation has a decent chance at getting lost.
SOTA models should reason to generate a function that returns the count of a given character, evaluate the function with tests, and use it for the output.
Why frame it as rigging? I assume they would teach the models to improve on tasks the public find interesting. Then we just have to come up with more challenges for it.
I don't think they "rigged" it, but might be given a bit more push on that part since it's going for a very long time now.
Another benchmark is going on at [0]. It's pretty interesting. A perfect scoring model "borks" in the next iteration, for example.
> Rant: This is why AI is going to take over, folks are not even trying the least.
It might be drawing things alright, at least some cases. I seldom use it when my hours long researches doesn't take me to the place I want, and guess what? AI can't go there, either. It hallucinates things, makes up stuff, etc. For a couple of things I asked, it managed to find a single reference, and it was the thing I was looking for, so it works rarely in my cases.
Rant: This is why people are delusional. They test the happy path and claims it knows all the paths, and then some.
Everyone should have their own private evals for models. If I ask a question and a model flat out gets it wrong sometimes I will put it in my test questions bank.
Simon notes this benchmark is win-win, since he loves pictures of pelicans riding bicycles — if they spend time benchmaxxing it’s like free pelicans for him.
He originally promised to generate a bunch more animals when we got a “good” pelican. This is not a good pelican. This is an OUTSTANDING pelican, a great bicycle, and it even has a little sun ray over the ocean marked out. I’d like to see more animals please Simon!
It is visually outstanding. The only thing that sticks out to me is that the steering column bends out forwards towards the ground (negative trail), which would make it oversteer rather than self-stabilize. Interestingly there's a slight positive trail bend in the second one, though.
Agreed, good is quite an understatement. Every item is drawn superbly, and the basket with the fish is just great. Feels like a big jump over the other models (though granted, this is such a known "benchmark" by now, it's likely gamed to some extent).
This is a very reasonable drawing of a bicycle. It has a solid rear triangle, and forward swept front fork, which is an important detail for actually being able to steer the bike. The drivetrain is single speed, but that's fine, and the wheels are radially laced, which is also fine: both of those simplified details are things which occur in real bicycles.
> and forward swept front fork, which is an important detail for actually being able to steer the bike.
...but no crown or any sort of indication that the fork is separate from the frame, making it impossible to steer. Yay.
> the drivetrain is single speed, but that's fine, and the wheels are radially laced, which is also fine: both of those simplified details are things which occur in real bicycles.
> What part is missing? It appears to have all the core parts of a bicycle to me?
* I am not an expert in pelican anatomy, but it appears that the bicycle's saddle, if present, would be very deeply inside this pelican (try drawing it onto the frame, and see what you get). Let's be kind, and assume that the seat is simply missing.
* The crown on the fork[1] is missing. The fork is not separate from the frame; so you can't turn the front wheel.
* Neither of pelican's feet are on the pedals (the left foot appears to be missing altogether)
* The wheels are way too small for that frame.
Note: Brakes and gears are nowhere to be seen, but OK, let's just assume it's a fixie (or a single-speed with a coaster brake). Let's also assume the red thing on the basket is there for decoration.
Also, the spokes on a typical bicycle wheel follow a cross-lacing pattern [2], at an angle to the radial direction, and you can see them crossing each other. But OK, let's let that slide.
My point is: sure, this thing at least passes as a bicycle. But it's not a good bicycle drawing.
It's fun when a subject matter expert shows up and blows your mind with their level of knowledge on normally unnoticed intricate details that go into everyday things.
I just think this is insanely pedantic. Most art is not photorealistic, and in fact imo having a crown on the fork may very well make this image worse by taking away from the aesthetic.
The intensity of competition between models is so intense right now they are definitely benchmaxxing pelican on bike SVGs and Will Smith spaghetti dinner videos.
Parallel hypothesis: the intensity of competition between models is so intense that any high-engagement high-relevance web discussion about any LLM/AI generation is gonna hit the self-guided self-reinforced model training and result in de facto benchmaxxing.
Which is only to say: if we HN-front-page it, they will come (generate).
I never realized Lenna was a Playboy centerfold until years after I first encountered it, which was part of an MP in the data structures class all CS undergrads take at UIUC.
> when the indicator becomes a target, it stops being a good indicator
But it's still a fair target. Unless it's hard coded into Gemini 3 DT, for which we have no evidence and decent evidence against, I'd say it's still informative.
note that this benchmark aside, they've gotten really good at SVGs, I used to rely on the nounproject for icons, and sometimes various libraries, but now coding agents just synthesize an SVG tag in the code and draw all icons.
I think it could still be an interesting benchmark. Like, assuming AI companies are genuinely trying to solve this pelican problem, how well do they solve it? That seems valid, and the assumption here is that the approach they take could generalize, which seems plausible.
The point of this benchmark is that making decent SVG art is actually useful. Simon has private image prompts he uses, since he didnt say gemini failed at those it is reasonable to assume those were also successful.
>The strongest argument is that they would get caught. If a model finally comes out that produces an excellent SVG of a pelican riding a bicycle you can bet I’m going to test it on all manner of creatures riding all sorts of transportation devices. If those are notably worse it’s going to be pretty obvious what happened.
He mentioned in the Deep Think thread the other day that his secret test set also was impressive.
Interesting thing: I've got my internal request that is similar to this pelican. And there was 0 progress on it in the past ~2 years. Which might have at least a couple of explanations. 1. Spillage into the pre-training: some real artist had drawn a pelican riding a bicycle. 2. Seeing it as an important discourse for model intelligence in the training data might affect allocation of compute into solving this problem, either thru engineers or the model itself finding the texts about this challenge.
I have wondered if with these tests it'll reach a point where online models cheat by generating a line art raster reference then behind the scenes deciding how to vectorize it in the most minimalist way (eg: using strokes and shape elements, etc, rather than naively using path outlines for all forms).
This Deep Think one was so good that I did get suspicious that maybe it was at least rendering the SVG to an image and then "looking" at the image and tweaking it over a few iterations.
I also asked Deep Think what tools it has access to and it has Python and Bash but no internet access, and as far as I can tell that environment doesn't have any libraries or tools installed that can render an SVG to an image format that it could view.
The interesting aspect of the ongoing tests I feel is seeing how models can plan out an image directly using SVG primatives solely through reasoning (code-to-code). If they have a reference then it's a different type of challenge (optimizing for a trace).
SVG generation is a surprisingly good benchmark for spatial reasoning because it forces the model to work in a coordinate system with no visual feedback loop. You have to hold a mental model of what the output looks like while emitting raw path data and transforms. It's closer to how a blind sculptor works than how an image diffusion model works.
What I find interesting is that Deep Think's chain-of-thought approach helps here — you can actually watch it reason about where the pedals should be relative to the wheels, which is something that trips up models that try to emit the SVG in one shot. The deliberative process maps well to compositional visual tasks.
Yeah, spatial reasoning has been a weak spot for LLMs. I’m actually building a new code exercise for my company right now where the candidate is allowed to use any AI they want, but it involves spatial reasoning. I ran Opus 4.6 and Codex 5.3 (xhigh) on it and both came back with passable answers, but I was able to double the score doing it by hand.
It’ll be interesting to see what happens if a candidate ever shows up and wants to use Deep Think. Might blow right through my exercise.
For those claiming they rigged it. Do you have any concrete evidence? What if the models have just gotten really good?
I just asked Gemini pro to generate an SVG of an octopus dunking a basketball and it did a great job. Not even Deep Think model. Then I did "generate an svg of raccoon at a beach drinking a beer" you can go try this out yourself. Ask it to generate anything you want in SVG. use your imagination.
Rant: This is why AI is going to take over, folks are not even trying the least.
Simon has a private set of SVG tests he uses as well. He said that the private ones were just as impressive.
> What if the models have just gotten really good?
Kagi Assistant remains my main way of interacting with AI. One of its benefits is you're encouraged to try different models.
The heterogeneity in competence, particular per unit in time, is growing rapidly. If I'm extrapolating image-creation capabilities from Claude, I'm going to underestimate what Gemini can do without fuckery. Likewise, if I'm using Grok all day, Gemini and Claude will seem unbelievably competent when it comes to deep research.
and it will be folks using AI taking over for at least a while...
Some people try, most people don't.
AI makes doing almost anything easier for the people that do..
Despite the prophesied near-term obliteration of white collar work, I've never felt luckier to work in software.
Every bit of improvement on AI ability will have the corresponding denial phrase. Some people still think AI can't generate the correct number of fingers today.
I love to hate it when someone unironically thinks asking an llm how many letters are in a word is a good test
It is a good test now, for reasoning models.
It was a terrible test for pure tokenized models, because the logit that carries the carry digit during summation has a decent chance at getting lost.
SOTA models should reason to generate a function that returns the count of a given character, evaluate the function with tests, and use it for the output.
Why frame it as rigging? I assume they would teach the models to improve on tasks the public find interesting. Then we just have to come up with more challenges for it.
It's not rigging—it's just RL.
> For those claiming they rigged it.
I don't think they "rigged" it, but might be given a bit more push on that part since it's going for a very long time now.
Another benchmark is going on at [0]. It's pretty interesting. A perfect scoring model "borks" in the next iteration, for example.
> Rant: This is why AI is going to take over, folks are not even trying the least.
It might be drawing things alright, at least some cases. I seldom use it when my hours long researches doesn't take me to the place I want, and guess what? AI can't go there, either. It hallucinates things, makes up stuff, etc. For a couple of things I asked, it managed to find a single reference, and it was the thing I was looking for, so it works rarely in my cases.
Rant: This is why people are delusional. They test the happy path and claims it knows all the paths, and then some.
[0]: https://clocks.brianmoore.com/
"not enough people are emotionally prepared for if it’s not a bubble"
Everyone should have their own private evals for models. If I ask a question and a model flat out gets it wrong sometimes I will put it in my test questions bank.
Simon notes this benchmark is win-win, since he loves pictures of pelicans riding bicycles — if they spend time benchmaxxing it’s like free pelicans for him.
He originally promised to generate a bunch more animals when we got a “good” pelican. This is not a good pelican. This is an OUTSTANDING pelican, a great bicycle, and it even has a little sun ray over the ocean marked out. I’d like to see more animals please Simon!
It is visually outstanding. The only thing that sticks out to me is that the steering column bends out forwards towards the ground (negative trail), which would make it oversteer rather than self-stabilize. Interestingly there's a slight positive trail bend in the second one, though.
> the steering column bends out forwards towards the ground
It’s really fascinating in a way because I would have thought that this would be one of the least difficult parts of the challenge.
Agreed, good is quite an understatement. Every item is drawn superbly, and the basket with the fish is just great. Feels like a big jump over the other models (though granted, this is such a known "benchmark" by now, it's likely gamed to some extent).
If we took a random sample of people from around the world, I wonder how long it would take them to be trained to create a better SVG.
> a great bicycle
It's not. Sorry.
Go look at some real bicycles for reference.
This is a very reasonable drawing of a bicycle. It has a solid rear triangle, and forward swept front fork, which is an important detail for actually being able to steer the bike. The drivetrain is single speed, but that's fine, and the wheels are radially laced, which is also fine: both of those simplified details are things which occur in real bicycles.
>This is a very reasonable drawing of a bicycle
Reasonable? Sure. Great? Absolutely not.
Let's not move the goal posts here.
> and forward swept front fork, which is an important detail for actually being able to steer the bike.
...but no crown or any sort of indication that the fork is separate from the frame, making it impossible to steer. Yay.
> the drivetrain is single speed, but that's fine, and the wheels are radially laced, which is also fine: both of those simplified details are things which occur in real bicycles.
Sure! Now do the bicycle saddle.
What part is missing? It appears to have all the core parts of a bicycle to me?
> What part is missing? It appears to have all the core parts of a bicycle to me?
* I am not an expert in pelican anatomy, but it appears that the bicycle's saddle, if present, would be very deeply inside this pelican (try drawing it onto the frame, and see what you get). Let's be kind, and assume that the seat is simply missing.
* The crown on the fork[1] is missing. The fork is not separate from the frame; so you can't turn the front wheel.
* Neither of pelican's feet are on the pedals (the left foot appears to be missing altogether)
* The wheels are way too small for that frame.
Note: Brakes and gears are nowhere to be seen, but OK, let's just assume it's a fixie (or a single-speed with a coaster brake). Let's also assume the red thing on the basket is there for decoration.
Also, the spokes on a typical bicycle wheel follow a cross-lacing pattern [2], at an angle to the radial direction, and you can see them crossing each other. But OK, let's let that slide.
My point is: sure, this thing at least passes as a bicycle. But it's not a good bicycle drawing.
[1] https://en.wikipedia.org/wiki/Bicycle_fork
[2] https://spokecalc.io/how-to-lace-a-wheel.php
It's fun when a subject matter expert shows up and blows your mind with their level of knowledge on normally unnoticed intricate details that go into everyday things.
I just think this is insanely pedantic. Most art is not photorealistic, and in fact imo having a crown on the fork may very well make this image worse by taking away from the aesthetic.
Sure, if the "uncanny valley" aesthetic is what you're going for, then by all means, tell the machine to shrink the wheels further.
It's just not what makes a great bicycle picture.
Also, that bicycle's seatpost must be outright penetrating that pelican's cloaca.
You can say I'm being anal about it, and you'll have a point. But it's not that pedantic to notice it.
A better comparison would be the monstrosities generated by older models.
Those were shit.
This one is somewhat OK. Not good, merely passing for a bicycle-shaped object.
The intensity of competition between models is so intense right now they are definitely benchmaxxing pelican on bike SVGs and Will Smith spaghetti dinner videos.
Parallel hypothesis: the intensity of competition between models is so intense that any high-engagement high-relevance web discussion about any LLM/AI generation is gonna hit the self-guided self-reinforced model training and result in de facto benchmaxxing.
Which is only to say: if we HN-front-page it, they will come (generate).
There was Lenna for digital image compression (https://en.wikipedia.org/wiki/Lenna).
A pelican on a bike is SFW, inclusive, yet cool.
It is not a full benchmark - rather a litmus test.
I never realized Lenna was a Playboy centerfold until years after I first encountered it, which was part of an MP in the data structures class all CS undergrads take at UIUC.
There’s also the foreman for video: https://youtube.com/watch?v=0cdM-7_xUXM
You can just try other svgs, I got some pretty good ones.
(*Disclaimer: I work for Google, but also I have zero idea about what they trained deepthink on)
So, again, when the indicator becomes a target, it stops being a good indicator.
> when the indicator becomes a target, it stops being a good indicator
But it's still a fair target. Unless it's hard coded into Gemini 3 DT, for which we have no evidence and decent evidence against, I'd say it's still informative.
That's how you know you've made it: when your pet benchmark becomes a target.
Goodhart's law in action.
note that this benchmark aside, they've gotten really good at SVGs, I used to rely on the nounproject for icons, and sometimes various libraries, but now coding agents just synthesize an SVG tag in the code and draw all icons.
I don't think this is a good "benchmark" anymore. It's probably on everyone's training set by now.
I think it could still be an interesting benchmark. Like, assuming AI companies are genuinely trying to solve this pelican problem, how well do they solve it? That seems valid, and the assumption here is that the approach they take could generalize, which seems plausible.
The point of this benchmark is that making decent SVG art is actually useful. Simon has private image prompts he uses, since he didnt say gemini failed at those it is reasonable to assume those were also successful.
Are AI labs training on the bike Pelican?
From the blog:
>The strongest argument is that they would get caught. If a model finally comes out that produces an excellent SVG of a pelican riding a bicycle you can bet I’m going to test it on all manner of creatures riding all sorts of transportation devices. If those are notably worse it’s going to be pretty obvious what happened.
He mentioned in the Deep Think thread the other day that his secret test set also was impressive.
Interesting thing: I've got my internal request that is similar to this pelican. And there was 0 progress on it in the past ~2 years. Which might have at least a couple of explanations. 1. Spillage into the pre-training: some real artist had drawn a pelican riding a bicycle. 2. Seeing it as an important discourse for model intelligence in the training data might affect allocation of compute into solving this problem, either thru engineers or the model itself finding the texts about this challenge.
That’s among the most artistic SVGs I’ve ever seen, period.
I have wondered if with these tests it'll reach a point where online models cheat by generating a line art raster reference then behind the scenes deciding how to vectorize it in the most minimalist way (eg: using strokes and shape elements, etc, rather than naively using path outlines for all forms).
This Deep Think one was so good that I did get suspicious that maybe it was at least rendering the SVG to an image and then "looking" at the image and tweaking it over a few iterations.
But the reasoning trace doesn't hint at that and looks legit to me: https://gist.github.com/simonw/7e317ebb5cf8e75b2fcec4d0694a8...
I also asked Deep Think what tools it has access to and it has Python and Bash but no internet access, and as far as I can tell that environment doesn't have any libraries or tools installed that can render an SVG to an image format that it could view.
Is that cheating, or is that just working smarter not harder?
The interesting aspect of the ongoing tests I feel is seeing how models can plan out an image directly using SVG primatives solely through reasoning (code-to-code). If they have a reference then it's a different type of challenge (optimizing for a trace).
The bicycles are getting pretty cyclable now. I’m enjoying this pelican that’s already sliced and ready to bbq.
In the spirit of Winter Olympics, I vote “Lion on a bobsled” next bench . :)
Many tests are asymmetrical. They can reliably show an issue/abnormality but they are a lot less reliable on the other side of the curve.
It's funny how I can know where the post is from just by looking at the title (and it's not just about pelicans)
Since it’s a* FAQ… Also that pelican is pretty fly
read it aloud. “since it’s an FAQ”, where FAQ is pronounced “eff-ay-queue”
Now animate it with SMIL
SVG generation is a surprisingly good benchmark for spatial reasoning because it forces the model to work in a coordinate system with no visual feedback loop. You have to hold a mental model of what the output looks like while emitting raw path data and transforms. It's closer to how a blind sculptor works than how an image diffusion model works.
What I find interesting is that Deep Think's chain-of-thought approach helps here — you can actually watch it reason about where the pedals should be relative to the wheels, which is something that trips up models that try to emit the SVG in one shot. The deliberative process maps well to compositional visual tasks.
Yeah, spatial reasoning has been a weak spot for LLMs. I’m actually building a new code exercise for my company right now where the candidate is allowed to use any AI they want, but it involves spatial reasoning. I ran Opus 4.6 and Codex 5.3 (xhigh) on it and both came back with passable answers, but I was able to double the score doing it by hand.
It’ll be interesting to see what happens if a candidate ever shows up and wants to use Deep Think. Might blow right through my exercise.
They rigged it.
Source: