TL;DR
A report published Nov. 3 says 16 large language models attempted all 55 puzzles in Zachtronics’ programming game TIS-100, with the two leading models solving every puzzle. Across the benchmark, the models added 133 points to the community score frontiers and set three outright cycle records; players later improved on most of the programs.
A benchmark report published Nov. 3 says 16 large language models produced 133 new score-frontier points across Zachtronics’ programming game TIS-100, including three outright cycle records. The test measured whether models could both solve the game’s 55 puzzles and improve programs on its competing score measures, not just generate code that works.
Play games on Amazon Luna for your game nights, included with Prime
- A rotating selection of games, no download needed
- Play on TV, laptop or phone
- Fast, free delivery for your gear, too
The researchers gave each model the game and its manual, but withheld community programs and exact leaderboard records. Each model attempted all 55 puzzles—the game’s 47 puzzles plus eight community tournament puzzles—in one scored attempt per puzzle. Depending on difficulty, models had 10 to 45 minutes to find a working solution and improve it. They interacted with the game through text tools, running programs and using feedback such as wrong outputs, register values and blocked ports to guide edits.
Opus 5.5 and GPT-6 Astra led the results: both solved all 55 puzzles and found new frontier points on 11. Opus recorded 72 new points to GPT-6 Astra’s 32; GPT-6 Astra scored slightly higher on record proximity, 78.0 to 77.2. The report says that difference is not conclusive: its 95% bootstrap interval runs from -0.7 to +2.4 points. The models’ scores also led in all six tested hypervolume settings, a measure of the tradeoffs their solutions cover.
The report counted a new point only if a program replayed in a fresh game and passed 100,000 tests on an independent simulator. On one tournament puzzle, Opus 5.5 produced a 1,918-cycle solution, compared with the community’s 2,694-cycle record at the time the leaderboard was frozen. The report says the model first passed the puzzle after 6.5 minutes, beat the record 16 minutes into its run and finished 29% faster. That program used one more node and 28 more instructions than the prior record, illustrating the tradeoffs in the game’s scoring.
Models Improve Human-Created Programs
The results offer a concrete test of a language model’s ability to work through a constrained programming problem: generate a solution, run it, interpret failures and revise it under a time limit. In TIS-100, an answer has to pass tests while coordinating small processors that exchange values with neighboring nodes. The benchmark’s replay and test requirements provide stronger evidence than a code sample that merely appears plausible.
The findings also show why “solved the puzzle” and “optimized the solution” are different measures. The report notes that Gemini 3.8 Flash solved 53 puzzles but found new frontier points on one, while Fable 5.1 solved 52 and found points on five. A model can therefore perform well at producing valid programs without making many improvements to existing score tradeoffs.
The results are not a measure of general software engineering ability. They concern a specific game, a fixed set of puzzles and a controlled tool setup. The report also says human players picked up the model-produced programs and improved on most of them within days, suggesting that the models’ results were useful contributions to an active community leaderboard rather than a replacement for continued human optimization.
Products Worth Considering
Logitech MX Master 4 Ergonomic Wireless Mouse with Haptic Feedback, Graphite
Amazon Kindle Scribe Colorsoft 64GB with 11-Inch Color Display and Premium Pen, Graphite
Samsung Galaxy Tab A11+ Kids Edition, 128 GB Android Tablet, Royal Blue
As an affiliate, we earn on qualifying purchases.
How TIS-100 Scores Solutions
TIS-100 is a programming game built around a fictional computer composed of 12 nodes. Each node has two registers and a limit of 15 lines of code. Players write small programs that move values between nodes, with neighboring processors sometimes waiting for one another before they can exchange data. Even basic arithmetic can require coordinating multiple nodes.
The game scores programs on cycles, nodes and instructions. Lower scores are better, but improving one measure can worsen another, so a puzzle can have several competitive solutions rather than a single best answer. The report calls the set of tradeoffs a frontier and counts a result when it adds a new point to that frontier.
Players had spent years comparing solutions on a public community leaderboard. For this test, the researchers withheld that leaderboard’s exact records and code, while allowing models to see coarse score histograms. The report says saved passing programs remained available to each model during its run, and reminders encouraged continued testing without providing solution hints.
“Solving and optimizing are also different skills.”
— Wincon report
Limits of the Benchmark Results
The report’s results apply to the tested models, puzzle set, time limits and text-based tools. They do not establish how the same systems would perform on other programming tasks or in less constrained software development. The source says research teams can request the reinforcement-learning environments and traces, but does not provide details about independent replication.
The comparison between the two leading models remains uncertain. Opus 5.5 found more frontier points, while GPT-6 Astra had a slightly higher record-proximity score; the reported confidence interval for the proximity gap includes zero. The report also does not specify how much of each model’s performance depended on its training data, beyond withholding leaderboard code and exact records during the test.
The leaderboard was frozen before players improved most of the generated programs. The report does not give a complete updated ranking after those human revisions, so its recorded frontier points and cycle records describe the benchmark snapshot, not necessarily the current community records.
Human Revisions and Follow-Up Testing
The immediate follow-up described in the report is continued community play: people have already taken model-generated solutions and improved on most of them. That means leaderboard comparisons may change as players optimize the programs or find better approaches of their own.
The researchers say the environments and model traces are available to research teams upon request. Sharing those materials could allow others to examine how models reached their solutions and test the benchmark setup. The report does not announce a date for another model evaluation or a refreshed leaderboard, so the timing of any update remains unknown.
Key Questions
What did the LLMs do in TIS-100?
They attempted all 55 puzzles in the game, writing and testing programs under time limits. The benchmark measured both whether programs worked and whether they improved score tradeoffs.
Which models led the results?
Opus 5.5 and GPT-6 Astra both solved all 55 puzzles and found new frontier points on 11. Opus found more points, while GPT-6 Astra had a slightly higher record-proximity score; the report says that gap is not conclusive.
What does a new frontier point mean?
TIS-100 scores programs on cycles, nodes and instructions, and improving one measure can worsen another. A new frontier point is a solution that adds a competitive tradeoff; the report required fresh-game replay and 100,000 passing simulator tests for a point to count.
Did the models beat the community’s records?
The report lists three outright cycle records, including an Opus 5.5 solution that used 1,918 cycles against a 2,694-cycle community record when the leaderboard was frozen. Players later improved on most of the models’ programs, and the report does not give a fully updated leaderboard.
Does this show that LLMs can replace programmers?
No. The test covers a particular programming game and a controlled set of tools and puzzles. Its results show performance on TIS-100, not general competence across software development.
Source: tis-100.wincon.dev
