Claude model versions benchmarked on Pokémon Red milestones in staircase chart

Level 1: Smarter AI, Better Gamer

Imagine three friends of different skill levels playing the same Pokémon game. The first friend is a newbie and barely manages to leave the starting house before getting confused. The second friend does better, making it through a couple of towns but eventually getting stuck at a tough boss. The third friend is super experienced and zooms ahead, winning three boss battles with ease. This meme is like a chart of those progress stories, but the “friends” are actually versions of an AI. Each step up in the chart is like completing a level or earning a badge. The newest AI (the orange line) is the smartest, so it climbs the highest – it completes more of the game and does so faster, with fewer wrong turns. The chart looks like stairs because the AI goes step by step, finishing one goal at a time. It’s funny because we usually think of measuring a computer’s intelligence with math tests or fancy benchmarks, but here we’re measuring it by how far it gets in a Pokémon adventure! It’s a playful way to show that the latest AI is much better – almost like saying, “Look, our AI can really play – it’s the Pokémon champion compared to its previous versions.” Even if you don’t know the technical details, you can giggle at the idea of a super-smart computer being proud that it beat a video game level that its younger self couldn’t. It’s a simple story of each new model being a better game-player, told in a fun, colorful chart form.

Level 2: AI vs Pokémon Red

Let’s break down what’s going on in this meme in simpler terms. Claude is the name of a series of AI models created by the company Anthropic – think of it like Anthropic’s version of GPT. These are Large Language Models (LLMs), which usually excel at understanding and generating text. Here, however, they’re being used in a pretty novel way: to play a video game (specifically Pokémon Red from the late ’90s). Now, Pokémon Red is an adventure role-playing game where you, as the player, travel from town to town (in a region called Kanto) and collect Gym Badges by defeating powerful trainers (Gym Leaders). The progression in the game is fairly linear and milestone-based – you start in your home (Pallet Town), you get your first Pokémon (your “starter”), you go to the next city, run errands like getting a parcel for Professor Oak, navigate through a forest, and eventually challenge Gym Leaders like Brock (Pewter City’s leader with Rock-type Pokémon), Misty (Cerulean City’s Water-type leader), and Lt. Surge (Vermilion City’s Electric-type leader). Each of those achievements (leaving the house, getting the starter Pokémon, reaching a city, obtaining a badge) is a milestone in the game’s story. They’re kind of like level checkpoints or quests completed.

The meme shows a chart titled “Claude models playing Pokémon* – Milestone progress over time.” On the left (y-axis) are listed those game milestones from Start (at the bottom) all the way up to “Get Surge’s Badge” (toward the top). On the bottom (x-axis) we see “Number of Actions” going from 0 to 35.0k (meaning 0 to 35,000). Essentially, as the AI plays the game, we count how many actions it takes – an action might be moving one step, choosing an attack in battle, talking to an NPC, etc. The graph has four lines, each representing a different version of the Claude AI model: 3.0 Sonnet (green), 3.5 Sonnet (blue), 3.5 Sonnet (new) (purple), and 3.7 Sonnet (orange). “Sonnet” seems to be the codename or family name of these model versions. Possibly, Claude 3.5 (new) was an improved update of Claude 3.5 that Anthropic released, and then Claude 3.7 is an even more advanced successor. Think of it like software version updates – 3.0 was the old one, 3.5 was better, and 3.7 is newer still.

Now, the lines are “step-shaped” because of how milestones work: the line stays flat while the model hasn’t reached a new milestone, then it jumps up when the model hits that next milestone. So it literally looks like stairs climbing up. For example, consider the blue line for Claude 3.5: it starts at “Start” on y-axis when 0 actions have been taken (the beginning of the game). It might remain flat until, say, a few hundred actions in when it manages to “Leave House” – then the line steps up to that milestone level. It stays flat again as the model wanders around until it “Get Starter” Pokémon – up goes the line. This continues. If a model stops making progress (say it never got past Viridian Forest), its line will plateau there and never reach higher milestones. By looking at the chart, we can see the green 3.0 Sonnet line barely rises at all – that implies Claude 3.0 only ticked off the very first objectives (maybe it left the house, but perhaps it didn’t even successfully obtain a starter or reach the next city). In contrast, the orange 3.7 Sonnet line climbs all the way to the top milestone listed (Surge’s Badge). That tells us Claude 3.7 actually made it through all those early game hurdles: it left Pallet Town, got its Pokémon, went to Viridian City, delivered Oak’s Parcel, traversed Viridian Forest, beat Brock to get the Boulder Badge, went through Mt. Moon, reached Cerulean City, beat Misty for the Cascade Badge, continued on to Vermilion City, and finally beat Lt. Surge for the Thunder Badge. Phew! That’s a lot of game accomplished – roughly the first third of Pokémon Red – all by an AI controlling the game. Meanwhile, the blue and purple lines (both versions of 3.5) reach somewhere in the middle. The purple 3.5 Sonnet (new) line goes a bit higher than the blue 3.5 line, indicating the “new” update of 3.5 was able to progress further in the game than the original 3.5 could. For instance, if 3.5 (blue) got stuck at Brock’s Gym and never earned Brock’s Badge, but 3.5 (new) managed to beat Brock, then purple would step one level higher than blue. The chart suggests exactly that kind of improvement: each newer model overcomes one or two more challenges that stumped the previous one.

The x-axis (number of actions) also tells a story about efficiency. Notice that the orange 3.7 line not only reaches higher milestones, it often reaches them with fewer actions (the steps are shifted to the left relative to the other lines). For example, suppose both blue and orange lines reach “Get Oak’s Parcel” – if orange did it in, say, 1,000 actions while blue took 2,000 actions, the orange step up will appear earlier on the x-axis. This means Claude 3.7 didn’t waste as much time and moves to complete that quest. In practical terms, a more efficient AI might have navigated directly to the destination without getting lost, or fought fewer wild Pokémon because it chose to run away more often to save time, etc. In contrast, if Claude 3.0 is floundering, it might be walking in circles or repeating unnecessary actions, which racks up the action count without gaining new milestones – hence its line would be far to the right (lots of actions) even for low on the y-axis (few milestones). The faintness of the green line and how little it moves tells us 3.0 basically failed to progress — it might have just meandered until hitting some limit.

So, in plainer terms: Claude 3.7 is the champ here. It played further into the game and did so more smoothly than the older Claudes. The caption under the image says as much in a formal tone: “Claude 3.7 Sonnet demonstrates that it is the very best of all the Sonnet models so far at playing Pokémon Red.” That line is both informative and a cheeky nod to Pokémon’s catchphrase about being “the very best.” For a viewer, the humor comes from the idea that we’re treating “playing Pokémon” as a serious benchmark test for AI. It’s like if you came into an AI lab and saw researchers solemnly watching an AI try to beat a Game Boy game; it’s a bit unexpected and fun. But actually, it’s rooted in real trends: AI researchers do use games to test AI abilities! (Games provide clear goals and metrics – perfect for benchmarks). In fact, classic games like those on the Atari or even complex ones like StarCraft have been popular for AI testing. Pokémon Red, though, is a bit unique because it involves exploration, memory, and planning, not just reflex or high-score – it’s almost like a role-playing puzzle for the AI.

To make sure everything’s clear: an LLM playing Pokémon means the AI is effectively reading the game’s state as text and deciding on the next move to type in. The model presumably isn’t seeing the actual Game Boy screen pixels – instead, the game’s state (like “You are in Pewter City. There is a Gym ahead.” or battle text like “Wild Pidgey appeared!”) is fed to the AI as a prompt, and the AI outputs an action (like “walk north” or “use Thunderbolt”). This loop continues, action after action. So “Number of Actions” on the chart is literally how many command inputs the AI made. If the AI took 30,000 actions to get to Surge, that includes every little move, menu selection, and battle command along the way. That magnitude also explains why an older model might fail – controlling a game for tens of thousands of steps without messing up or getting stuck is really hard! Claude 3.7 managing it suggests it had the smarts to keep track of objectives and perhaps correct itself when things went wrong. Meanwhile, Claude 3.0 maybe lost track or did something that the game didn’t allow (like trying to walk through a wall) and got stuck early.

Finally, notice the professional look of the graph – white background, neat labels, legend, and even the trademark note "Pokémon is a registered trademark of Nintendo... No affiliation, sponsorship, or endorsement is implied." This is intentionally mimicking a real benchmark report or a slide from a tech presentation. It gives the meme a dry, deadpan delivery: treating something inherently silly in a serious, analytical way. That’s a common style in developer humor: present a joke as if it’s a formal report. Here the joke is that Claude 3.7 beating more of Pokémon Red is like a “performance win” to boast about. And truth be told, if you’re into AI, it is kind of cool! It means the AI had to handle navigation, inventory (getting Oak’s Parcel is basically a fetch quest), and battling efficiently. So for a junior developer or someone new to AI, this meme is a lighthearted introduction to how AI progress gets measured. Instead of showing you a confusing table of numbers or abstract tasks, it uses a familiar game to illustrate: “Our new AI is smarter – see, it can play the game further!” It’s both funny and actually informative. If you know Pokémon, you immediately get a sense of each model’s capability: one couldn’t even get out of the starting area, another made it to Pewter City, the best one got as far as Vermilion City. It’s like saying one AI is a novice, another is intermediate, and the newest is approaching expert gamer level, all shown in one quick graphic.

Level 3: Gotta Benchmark ’Em All

For seasoned developers and ML enthusiasts, this meme nails a blend of nostalgia and cutting-edge AI benchmarking. It presents what looks like a serious performance chart – the kind you’d see in an AI research paper or product release – but the task being benchmarked is delightfully off-beat: playing Pokémon Red. The title “Claude models playing Pokémon” with the subtitle “Milestone progress over time” instantly tells us that multiple versions of the Claude AI (Anthropic’s language model series) have been tested on how far they can progress in a 1990s Game Boy game. This juxtaposition is hilarious and telling. On one hand, it’s a legitimate model comparison of successive AI versions (3.0, 3.5, 3.5 (new), and 3.7 nicknamed "Sonnet"), but on the other hand, the metric for “best model” isn’t a usual NLP score or throughput – it’s who can get the furthest Gym Badge in Pokémon! It’s a crossover of AI research culture with gaming nostalgia, and that’s catnip for dev humor. Many of us grew up with Pokémon’s milestones (defeating Brock, Misty, Surge, etc.), so seeing them on a formal chart triggers a knowing grin.

The chart itself is a staircase-style line graph – an appropriate visualization since progression in a game is discrete (you either beat a boss or you haven’t, it’s not a smooth continuum). Each colored step-line corresponds to a Claude model version, showing how far it got (y-axis milestone) by the time it took a certain number of actions (x-axis). The orange line for Claude 3.7 Sonnet goes the highest – all the way to “Get Surge’s Badge” at the top. That immediately tells a tech-savvy audience: the newest model outperforms its predecessors by a wide margin. In fact, the caption explicitly boasts that Claude 3.7 “is the very best of all the Sonnet models so far at playing Pokémon Red,” cheekily echoing the Pokémon theme song lyrics (“I wanna be the very best, like no one ever was”). It’s a playful brag about AI performance using Poké-terminology. Meanwhile, the faint green line for 3.0 Sonnet fizzles out near the bottom (“barely leaves the house”), implying the earliest model got stuck almost immediately (maybe it kept walking into a wall or didn’t even figure out how to exit the starting house in Pallet Town). The blue 3.5 Sonnet line and the purple 3.5 Sonnet (new) line climb a bit higher – they achieve some milestones but plateau well before the third Gym. The “(new)” tag hints at an interim improvement (perhaps an optimized version of 3.5) – an inside joke about how sometimes minor version bumps hide major improvements. Any developer who’s seen version numbers used creatively (looking at you, Semantic Versioning abuse) will smirk at 3.5 having a “new” flavor that outperforms the old 3.5. It’s like an admission: “Yeah, 3.5-new is basically 3.6, but we didn’t want to change the number.”

The humor resonates on multiple levels. Firstly, it satirizes the AI industry trend of showing flashy benchmarks whenever a new model comes out. We’re used to charts where models are compared on tasks like translation accuracy or code generation. Here that trope is applied to a whimsical task – beating parts of a Pokémon game – but treated with full seriousness (complete with a proper labeled chart, legend, and even a legal Pokémon trademark footnote at the bottom!). That footnote – “Pokémon is a registered trademark... No affiliation is implied” – adds to the comedic effect by being overly formal on such a nerdy scenario, as if Anthropic had to lawyer-proof their Pokémon experiment. It mimics the tone of a corporate slide deck or a research poster, which seasoned devs find funny because we’ve all seen over-the-top professionalism applied to something inherently playful. It’s a wink: we know playing Pokémon isn’t an official benchmark… but let’s pretend it is!

Secondly, it highlights how far AI tools have come. Ten years ago, if you wanted a bot to play Pokémon, you’d probably write a specialized program or use reinforcement learning algorithms in an emulator. But here in the mid-2020s, you just throw a general LLM at the problem and see how well its general intelligence handles it. That’s wild! The meme taps into that collective amazement. Claude 3.7 presumably wasn’t built just to play games, yet it navigates a significant chunk of a JRPG’s quest line through sheer general prowess. This evokes the recent trend of using LLMs in novel ways (like AutoGPT agents that can loop through tasks, or people hooking ChatGPT up to Minecraft or web browsing). The AI research community has indeed been testing LLMs in more agent-like roles, so the idea of an LLM gameplay benchmark feels both absurd and plausible. It’s absurd in a funny way (“our chatbot is now a Pokémon speedrunner”), but also plausible because we’ve seen papers where GPT-4 plays chess via text or controls character in a text adventure. For veterans, it’s humor with a side of “huh, this might actually be a thing now.”

The shared experience being satirized is also the iterative improvement of models. Many of us remember when GPT-3’s first versions would mess up basic logic or get confused by multi-step instructions. Then came GPT-4 and suddenly it could handle much more complex sequences reliably. That leap is mirrored here by Claude 3.7 vis-à-vis 3.0/3.5. We nod knowingly because we’ve seen those dramatic jumps in capability – the meme just uses a fun yardstick (game progress) instead of a serious one. It’s machine learning humor at its finest: to those in the know, beating Misty (the second Gym Leader) becomes as triumphant a metric as optimizing BLEU score in translation. And because it’s Pokémon, there’s an extra layer of geek camaraderie: everyone remembers the personal “milestones” of that game (“Have you gotten to Cerulean City yet?”), so using them to measure an AI gives us a cozy, nerdy feeling. It’s like we’re sharing an inside joke that spans our childhood gaming and our present-day AI work.

There’s also an implicit commentary on performance vs. efficiency. The x-axis in thousands of actions suggests how efficiently each model reached its best milestone. Claude 3.7’s orange line not only goes highest, it also reaches each checkpoint with fewer actions (notice how its steps are further left compared to the blue/purple lines at the same milestones). For a senior tech crowd, this screams “improved algorithm!” It’s reminiscent of how a new software version might complete a task in 3 seconds that used to take 30 – here the task is navigating Viridian Forest or defeating Brock, and 3.7 does it in fewer in-game moves. Maybe Claude 3.0 had our poor virtual trainer running around in circles or wasting turns in battle, whereas Claude 3.7 beelines to the objectives and employs effective tactics (no more spending 10 turns using “Splash” – the AI now knows to use super-effective moves!). This ties into real-world AI optimization: the joke that “it’s always the newest model that finally doesn’t bump into walls.” Seasoned devs have painful memories of v1 of some system being clunky, and by v3 it’s finally smooth – that’s essentially what’s humorously portrayed here with game milestones. The Sonnet series naming even subtly pokes at how companies name models in fancy ways (why Sonnet? maybe because it’s poetic how much better it is?), which we chuckle at but also accept as part of AI industry trends of branding models.

In practice, we can imagine the scenarios behind each line on the graph, which makes it fun. A room of engineers might’ve watched Claude 3.0 try to play Pokémon: it likely got out of bed, bungled around (“Where’s the door again?”), maybe it didn’t even manage to Get Starter Pokémon because it failed to type “yes” to Oak’s question or got stuck in Professor Oak’s lab. That run ends early – hence the tiny green blip. Then Claude 3.5 comes along, now a bit smarter and with more training. It probably grabs Squirtle or Pikachu successfully, manages to fight some wild Pidgeys, and make its way to Viridian City. But perhaps it can’t figure out how to get Oak’s Parcel delivered (a minor fetch quest) or gets lost in Viridian Forest getting constantly attacked by Caterpies until it runs out of HP. Its line plateaus – game over or progress stalls – somewhere around those first challenges. Enter Claude 3.5 (new) (the upgraded half-step): this one actually completes the parcel quest, navigates Viridian Forest, and even beats Brock’s Gym to earn the Boulder Badge. Victory! The purple line jumps higher than the blue, showing it cleared that first big boss. But maybe it struggles with the next segment – perhaps it doesn’t efficiently get through Mt. Moon or loses to Misty’s Starmie repeatedly. So 3.5-new’s journey ends in the mid-game. Now, Claude 3.7 struts in like a seasoned Pokémon Master. It remembers to stock up on Potions, it knows Pikachu won’t help versus Brock so it trains a Butterfree or catches a Mankey for type advantage, breezes through Mt. Moon (maybe even picks the right fossil just for style points), trounces Misty by having a Bellsprout ready, and goes on to get Lt. Surge’s Thunder Badge. All this with far fewer random detours. Its orange step-line shows steady, controlled progress — the kind of line a proud dev would circle in a presentation saying “look, fewer actions, higher reward.” In a way, Claude 3.7 is depicted as having achieved what every gamer claims on their best play-through: a near-optimal run.

That scenario is both impressive and tongue-in-cheek. Impressive because it suggests the AI can handle complex sequence tasks (mixing exploration, combat strategy, and remembering to backtrack for story events). Tongue-in-cheek because it’s Pokémon! An AI benchmarked on a Game Boy RPG is not something you’d find in a formal benchmark suite like GLUE or SuperGLUE – it’s clearly done for fun and engagement (and it succeeded, since we’re all engaged and amused!). This also resonates with how AI companies sometimes use fun demos to communicate progress — it’s more exciting to say “Our AI beat Pokémon Red” than “Our AI achieved F1=92.1 on XYZ dataset,” right? So the meme is likely riffing on an Anthropic blog or talk that showcased this Pokémon experiment as a visually intuitive proof of Claude 3.7’s improved performance. It’s both a genuine display of technological advancement and an inside joke for the community. After all, “Gotta benchmark ’em all!” could easily be a battle cry in AI research now – if there’s a task out there, someone’s going to try the newest model on it, be it protein folding or a 8-bit video game. And as seasoned devs, we find that equal parts admirable and absurd, which is exactly why this meme elicits a knowing laugh.

Level 4: Long-Horizon Planning

At this deepest level, the meme highlights a cutting-edge intersection of reinforcement learning and large language model capabilities. Pokémon Red is treated as a long-horizon sequential decision problem, essentially a classic Markov Decision Process (MDP) with a vast state space (towns, battles, items) and delayed rewards (Gym Badges). Each Claude model here is an LLM acting as a game-playing agent, deciding one action at a time (like “move up” or “use Water Gun”) based on the game’s state feedback. The chart’s x-axis (Number of Actions) measures how many action-decisions the model needed, up to ~35,000, to reach certain milestones. This is huge in terms of token sequences – it means the AI must handle extremely long contexts and plans. Claude 3.7 Sonnet’s ability to reach Surge’s Badge (the third Gym Leader’s badge) with comparatively fewer actions implies superior planning efficiency and memory management across thousands of game steps. In other words, the newest model exhibits far better sample efficiency: it doesn’t waste as many moves wandering or backtracking; it strategizes more directly toward goals. This kind of efficient long-horizon planning is something we usually expect from specialized game-playing algorithms or deep RL agents, so seeing it emerge from a general LLM is both surprising and thrilling to AI researchers.

Under the hood, this likely leverages the LLM’s massive context window and world knowledge. Newer Claude models (like 3.7) can retain and reason over very long sequences of text (Claude is known for 100k-token contexts), which here translates to remembering the game’s history and objectives over many actions. Older Claude 3.0 might “forget” crucial instructions (like “bring Professor Oak his Parcel”) after a few thousand tokens, causing it to get stuck or repeat mistakes. In contrast, Claude 3.7 can maintain an internal chain-of-thought about long-term goals (“get the parcel, return to Oak, then head north to Pewter City for Brock’s Gym”). Its improved architectural and training refinements yield more coherent strategies over a prolonged sequence. Essentially, Claude 3.7 is better at the temporal credit assignment problem – it can connect an action taken now (e.g. buying a Poké Doll) to a payoff much later (using it to skip a battle in Rock Tunnel, for instance), whereas a less advanced model might not anticipate such long-range consequences.

We’re also witnessing how an LLM’s world knowledge can substitute for direct experience. A model like Claude has likely “read” countless guides, Wiki pages, or forum discussions about Pokémon Red during training. This means Claude 3.7 might already know the map layout of Kanto or the fact that Electric attacks won’t affect Brock’s Rock/Ground Pokémon. It can use this latent knowledge to plan effective moves (like training a Butterfree to use Confusion against Brock’s Geodude). Earlier versions (Claude 3.0/3.5) may have had the knowledge but lacked the ability to apply it consistently over many steps or they might not parse the game state accurately. The stepwise shape of each model’s line reflects the model encountering sub-goals and challenges: the horizontal stretches are plateaus where the AI is presumably attempting actions without achieving a new milestone (for example, wandering in Viridian Forest or grinding levels before Brock). A shorter plateau means the model solved that segment faster. Claude 3.7’s orange line has briefer plateaus– it likely navigated areas and won battles with fewer redundant actions – indicating more optimal decision-making. Those plateaus vanish upward in a jump when a milestone is finally achieved (like a sudden “Aha!” moment when the AI figures out how to get past a roadblock or win a Gym battle). This resembles how a well-tuned RL agent’s reward graph might look: flat during learning phases, then a spike when it “gets it.” Here, each step up is literally the AI clearing a level of the game.

Another advanced aspect is how the game state is represented to the language model. Pokémon Red wasn’t designed as a text-based environment, but to use an LLM, the game’s visuals and events must be described in text form (like “You are in Pewter City. There is a Gym here. An old man blocks the road east.”). The AI then responds in text (e.g. “Go to Gym”). The system likely converts that into game actions. This pipeline turns a classic video game into an interactive text adventure for the AI. It’s a clever fusion of modalities, essentially bridging embodied game agents with pure text reasoning. The Claude Sonnet series improvements might also include better tool use or API call abilities, meaning Claude 3.7 could be more adept at utilizing a provided game-interface API or remembering to use game commands correctly (like knowing the exact input needed to save the game or use an item, which might have been finicky for a smaller model). Ensuring the AI doesn’t cheat or go out-of-bounds is also important: with all its knowledge, a model might try a known glitch or exploit (imagine it attempting the old MissingNo glitch or a sequence break trick that it read about online!). In a rigorous benchmark, they’d sandbox the AI to use only legitimate actions. The fact that 3.7 goes the furthest without such tricks hints that it legitimately solved the game’s challenges through reasoning, not by “breaking” the game. That demonstrates emergent problem-solving capabilities: beyond just parroting solutions, the model can adapt its knowledge to the specific run of the game – for example, handling random wild Pokémon encounters or managing limited Pokémon health and inventory.

In summary, this meme’s chart isn’t just tech humor – it’s showcasing a complex concept: using a Large Language Model as a general game-playing agent. It touches on research ideas like language-model planning and model-based reinforcement learning (since the model might internally simulate outcomes via its next-token predictions). For those of us steeped in AI theory, it’s fascinating (and a bit absurd) that a single pre-trained model, without explicit game-specific training, can coordinate 30k+ actions to reach mid-game in a classic RPG. It’s like witnessing a weird hybrid of GPT meets AlphaGo: instead of calculating moves by pure trial-and-error or search, the AI is reasoning through Pokémon as if it were a massive interactive story. The humor hides a deep truth: as AI models scale in knowledge and context, they begin to exhibit agent-like abilities in long-horizon tasks that were once firmly the domain of specialized algorithms. Claude 3.7’s performance suggests that large generative models might eventually become general problem solvers, capable of handling not just Q&A or code, but sequential decision challenges – from navigating a game world to potentially navigating real-world tasks – all through the lens of language.