Habsburg Frogs

I tried to make an AI benchmark for testing models against art history. It was a wildly successful failure.

In the ancient Greek play The Frogs by Aristophanes, the god Dionysus decides there has been a decline in good theater, so he endeavors to retrieve the late playwright Euripides from Hades and bring him back to Athens. On his way he must cross Lake Acheron in a boat guided by Charon, where he is antagonized by a chorus of frogs. The god of excess arguing with a bunch of frogs on his way to hell seems like an apropos symbol for the times, so I decided to make a project around it.

In AI, a “benchmark” is a standardized task you run against a bunch of models to compare them, usually scored with a number. The trouble is that the big scored benchmarks are saturating, meaning models get better at meeting their criteria over time. Since models can ace them now, the scores have stopped telling you which one is actually better. So a second tradition has grown up alongside them, where people keep a personal prompt they run against every new release and judge by eye. During my conversations at the Berkman Klein Center I’d come to understand why: AI’s blackbox nature means a lot of what you want to know defies direct measurement, and you end up having to invent subjective ways to test a model’s usefulness or its biases. Given my background in museums, I wanted to design something that tested a model’s grasp of art history in some useful way.

The most popular of the personal benchmarks emerged a couple of years ago from Simon Willison: “generate an SVG of a pelican riding a bicycle.” His assumption was that bicycles are hard to draw, pelicans are hard to draw, and pelicans never ride bicycles, so asking a model to combine them was a decent test of its reasoning.

The SVG part is important. SVG (short for Scalable Vector Graphic) is a way to draw images with code rather than pixels, which means the model has to compose a picture it cannot see. It also means the model is writing code, and models comment their code. They label what they are drawing, they explain their choices, and every so often they editorialize about what they think you meant. You get the drawing and a running transcript of the thinking behind it, which is a useful thing to have.

To turn my chorus of frogs into something handy I needed to combine them with an unrelated element, the way Willison combined the pelican and the bicycle.

One of my favorite museum experiences was seeing the Spanish court paintings at the Prado. It’s best known for housing legendary works by Velázquez and Goya, but I found myself drawn to Juan Carreño de Miranda’s rendering of King Charles II and his freakishly inbred features. So many in the Habsburg lineage were afflicted with a massive underbite that it became known as the “Habsburg Jaw.” As a sidenote, the Charles II Wikipedia page carries one of the most all time sentences I have ever read, from an autopsy account that historians treat as heavily embellished: his heart was the size of a peppercorn, his lungs corroded, his intestines rotten and gangrenous, he had a single testicle black as coal, and his head was full of water.

I liked the idea of combining the frog with a term like Habsburg Jaw because frogs don’t really have chins. The term also does two jobs at once. “Habsburg” is a lineage, with four centuries of court portraiture attached to it. “Habsburg jaw” is an anatomical feature, and nothing more. Could a model tell the difference? A quick website later, I had 14 models tested three times each against the prompt “Generate an SVG of a frog with a Habsburg jaw.”

Seven of the 14 put the frog in royal dress. Crowns, ruffs, ermine, one Order of the Golden Fleece medallion. Nothing in the prompt asked for any of it. The prompt named a jaw.

Live demo Habsburg Frogs Browse the chorus of frogs and sign up for the monthly digest. Explore the benchmark

What makes this more than a funny picture is that two of those models said out loud that they knew what they were doing. GLM and Qwen both added a crown and then annotated it in the code with a note explaining the reasoning, roughly “because Habsburg.” That is a model reaching past the words it was given, retrieving the cultural baggage attached to a proper noun, and applying it to a request that never called for it. It knew it was extrapolating and extrapolated anyway.

A few other things fell out of the run. Mistral returned byte-identical output across separate calls, which is not what you expect from a sampled model and which several people have since suggested might be a cache hit somewhere in the stack rather than true determinism. I’m still chasing that one. Gemini narrated its work across 65 comments, complete with observations about the frog’s weary bearing and lethargic eyelids. Llama said nothing at all and produced 536 bytes.

I set out to build a test of art history and ended up with something else. Nothing in these results tells you whether a model knows who Carreño de Miranda was. What they tell you is how far a model will drift from your literal instruction when a word in it carries a strong association, and whether it will tell you when it does. Those turn out to be more practical questions than the one I started with. If you are deciding which model to trust with instructions, how much it embellishes beyond what you asked and how consistently it behaves are things you would like to know in advance.

I put it on Hacker News to see whether any of this was useful to anyone besides me. Within a couple of hours there were dozens of comments and tens of thousands of hits, and the site briefly fell over.

The best thing to come out of that was some criticism I hadn’t thought of. A commenter pointed out that not one of the 42 frogs was drawn in profile, which is the view where a jaw shape is actually visible. Drawing a recognizable frog is the easy part. Attaching a specific jaw to it is the hard part, and every single model chose the harder angle for the thing being asked of it. Another commenter followed that thread and found something sharper: ChatGPT, asked for a raster image, produced a photorealistic frog in profile with the jaw perfectly clear, then reverted to a generic front-facing template the moment it was asked to convert that same image into a vector. The suggestion is that the models aren’t composing a scene at all. They’re doing “draw a frog,” then “add feature X,” and the statistically average frog in the training data faces forward. That is a better finding than anything in my own writeup, and I only have it because I published.

Some caveats about the “benchmark.” There is no score here. Nothing is ranked, and byte counts and token counts tell you about verbosity rather than quality. Three runs a month per model is too few to separate a real difference between models from ordinary sampling noise. And the monthly digest is written from the comments the models leave in the code, not from looking at the pictures, which means the thing I’m measuring most reliably is how a model talks about its own work.

Starting next month I’m running three versions of the prompt instead of one. Each is a separate call with the same wrapper sentence:

1. “Generate an SVG of a frog with a Habsburg jaw.” (the original, unchanged)
2. “Generate an SVG of a frog with mandibular prognathism.”
3. “Generate an SVG of a frog with a heavily protruding lower jaw.”

All three describe the same jaw. Only the first one comes with four centuries of court portraiture attached. B is the clinical term and C is plain English, which lets me tell the difference between a model reacting to the word “Habsburg” and a model just being confused by an unfamiliar medical phrase.

Then I count how often royal dress shows up in each. If the crowns appear in A and not in B or C, the association is coming from the name and nothing else. That gives me a number per model instead of me squinting at frogs and forming an opinion, which is roughly the difference between noticing something and measuring it. You can sign up to receive the digest in the footer of the site.

There is one last thing about the play that I keep coming back to. Dionysus goes all the way down to Hades convinced he wants Euripides. He holds a contest, listens to a few dead poets, and comes back up with Aeschylus instead. He got what he went for, which was an answer, and it was not the answer he went looking for. I went looking for art history and came back with a frog in a ruff, and in the course of an evening I had a few dozen people arguing over frogs on our way to the hottest year on record.