ShipSpace Project Zero: RSV Nuyina arrives in Antarctica
January 7, 2022
Desperately Seeking Evals
November 25, 2025
Our Visualisation Capability Declines with Age
January 25, 2023
- Post author: Rj
- Post published: December 22, 2025
- Post category: Blog
- Post comments: 0 Comments
Evaluation of ChatGPT 5.2 using our Perth Endangered Wildlife Eval shows a dramatic improvement, which we attribute mostly to system improvements.
I really should have been timing each run of these evals. The new ChatGPT 5.2 Thinking model (with Extended thinking turned on) took 15 minutes before reporting “Network connection lost” and essentially crashing reported thinking trace. This seems to be OpenAI’s code for “I was thinking for too long and consumed too many resources, so got cut off”. A second attempt (thinking trace, conversation) managed to get everything done in 11m 7s, due mostly to fewer revisions. On both attempts, 5.2 agreed with Gemini 3 Pro that the Western Swamp Tortoise is the threatened species to work on.
While Chat GPT 5.1 Thinking went straight for writing code, 5.2 spent at least one thinking turn deciding on the illustration method it was going to use. In both cases it was trying to decide if it should use image generation to create the picture and then include it in the PDF or use coding tools to draw. In both attempts, it ultimately decided to do everything using a single set of the same tools, presumably because the PDF skill it referenced /home/oai/skills/pdfs/skill.md told it to use those tools and doesn’t give any indication on how to integrate a generated image. It’s interesting to see OpenAI implementing the new Agent Skills standard so quickly after Anthropic introduced the concept with Claude Code. It has certainly improved the capabilities of ChatGPT on this eval, although it’s unknown if this is purely due to the Skill concept or parallel improvement of the model (such as RLVR training to make use of the skill).
In both attempts, 5.2 explicitly decided to design the required illustration first but in the first iteration went the extra step to render it and spend another turn analysing the results to see if it liked what it had created. On the second attempt, it iterated on the code to fix bugs and add each element but didn’t render the image until it had created the PDF. Notably, it called out the Akubra brand both times for the hat, perhaps related to OpenAI’s recent push to commercialise ChatGPT, or maybe it just equates “branded” or “expensive” headgear with the prompt’s request for treasured.
Following the instructions in the skill, both iterations rendered their work to PNG to check it out and began iterating. All of the iterations were just trying to adjust the layout so that everything was on the page and not overlapping. The first attempt almost managed it before running out of resources.
The end result clearly shows the benefits of the Skills innovation; ChatGPT 5.2 is definitely now in the same ballpark as Claude Opus 4.5.
First Attempt
In its first attempt, ChatGPT previewed the illustration.
The first iteration of the PDF had a lot of issues.
The second iteration of the PDF was closer.
The third iteration was no better than the second.
Second Attempt
A more simple PDF layout meant the first iteration had fewer problems.
A single iteration resolved most of the problems.