GPT-5 Performance Benchmark on Expert-Level Questions

Level 1: Scratch Paper Allowed

Imagine you’re taking a really hard test that covers everything – history, math, science, you name it – and it’s full of super tricky questions. Now, picture two scenarios:

1. In the first scenario, you have to answer each question immediately without any extra help – no calculator, no textbook, and you’re not even allowed to jot down your thoughts. It’s just whatever pops into your head as your final answer.
2. In the second scenario, you’re allowed to think out loud or on paper, work through the problem step by step, and even check a calculator or an encyclopedia if you need to.

It’s pretty obvious which scenario would make you more likely to get the questions right, right? Of course the second one! If you can take your time to reason through a tough question and use tools to help (like looking up a fact or calculating a big number), you’re going to do way better than if you just guess with no working.

That’s exactly the point this meme is making about AI. The bar chart joke is basically saying: “Look how much better the AI did on the test when it was allowed to actually think and use tools versus when it had to answer without thinking!” The tall magenta bars are the AI “with its thinking cap on,” and the short pink bars are the poor AI “with its brain switched off.” It’s funny because we normally assume computers are always methodical, but here we literally have to tell the AI, “Hey, slow down and show your work,” otherwise it might just blurt out an answer and get it wrong. In human terms, it’s like an exam where one student rushes and writes down an answer without scratch paper, while another student carefully works through the problem and checks a calculator. No surprise – the careful student does much better! The humor and relief come from seeing that even for a super-smart AI, thinking things through makes a night-and-day difference. Essentially, the meme is winking at us and saying: turns out giving the AI a moment to think (and maybe a calculator and the internet) is the secret sauce to making it “smart” – who would’ve guessed?

Level 2: Tools and Scratchpad

Let’s break down what’s going on in simpler terms. This meme shows a bar chart comparing how well different AI models do on a super hard test called “Humanity’s Last Exam.” The test contains expert-level questions across many subjects – think of it like a giant trivia and problem-solving gauntlet where each question is really challenging. The y-axis is labeled Accuracy (pass@1), which basically means “the percentage of questions the AI got correct on the first try.” If an AI has 42% accuracy, it answered 42 out of 100 tough questions right on the first go. Not great by school exam standards, but for this kind of all-subject expert quiz, it’s actually impressive.

Now, the bars are colored in two shades: dark magenta and light pink. According to the legend, dark magenta means the AI was used “with thinking,” and light pink means “without thinking.” Of course, an AI isn’t literally thinking or not thinking – this refers to whether we prompted the AI to reason things out step-by-step (with thinking) or made it answer directly (no step-by-step, just give the answer – that’s without thinking). In AI lingo, “with thinking” mode is achieved via chain-of-thought prompting. That means we instruct the AI to show its work or break the problem into smaller steps in its answer. “Without thinking” means the AI just gives a final answer without explaining or working it out explicitly. It’s similar to you solving a math problem by doing calculations on paper versus just trying to intuit the answer in one go. Generally, writing out the steps (even for an AI) leads to a better result on hard problems – and the chart’s differences confirm that. For example, one pink bar shows GPT-5 (no tools) getting only 6.3% correct with no thinking steps – basically random guessing territory. But the magenta bar right next to it for GPT-5 (no tools, with thinking) shoots up to 24.8%. That’s the same GPT-5 model, same questions, but quadrupling its score just by tackling questions step-by-step in a logical way. Big difference! 🎉

Another factor here is the use of tools. You’ll notice some labels like “(python + search with blocklist)” next to model names. This means that model was allowed to use extra helpers: a Python interpreter (to run code or calculations) and a web search engine (to look up information). Blocklist likely means they restricted certain websites or answers so the AI couldn’t just copy a known solution directly – it had to do some work on its own. When a model can use such tools, we call it a tool-augmented LLM or an agentic LLM workflow. It’s like giving the AI a little toolbox with a calculator and an encyclopedia. For instance, GPT-5 pro (python + search) is using tools, whereas GPT-5 pro (no tools) isn’t – it’s just the AI model by itself with its built-in knowledge. The chart shows huge gains when the AI has those tools and is allowed to think step-by-step. GPT-5 pro with tools + thinking got 42.0% correct, whereas GPT-5 pro with no tools (and presumably no step-by-step) got 30.7%. Even ChatGPT agent – which is basically ChatGPT given the ability to act like an agent (it can open a browser, run code, use a terminal, etc.) – shot up to 41.6% with tools, compared to 23.0% without tools. So having a “browser + computer + terminal” at its disposal almost doubled ChatGPT’s score on this exam. In simpler terms: give the AI access to the internet and a calculator and let it work through problems methodically, and it becomes way more accurate.

Let’s clarify who these players are: GPT-5 is the hypothetical next big version of OpenAI’s flagship AI (successor to GPT-4). The chart even has a GPT-5 pro, which sounds like an even more souped-up version – perhaps a model with extra fine-tuning or more capabilities (maybe it’s like the “pro edition” with larger capacity or special training). ChatGPT agent is like an advanced mode of ChatGPT where it can do more than just chat – it can actually take actions, such as browsing the web, running code, and using a terminal to complete tasks. Think of it as ChatGPT with a virtual computer and internet connection to help it out. OpenAI Q3 might refer to some OpenAI model (possibly a research prototype from a certain quarter) that is not as powerful as GPT-5, included for comparison. Deep research likely refers to another research model (could be from another company or a codename project) that also had tool-use (Python + browser). And GPT-4.0 (no tools) at the far right is basically the older GPT-4 model with no extra help; it only got 5.3% correct, which shows how much harder these expert questions are without the new techniques.

The term pass@1 is borrowed from coding benchmarks – it usually means the chance that the first attempt the model gives is a correct solution. Here it’s used similarly: if the AI’s first answer to a question is right, that counts. So a higher pass@1% means the AI is more often correct on the first try. This is a strict metric, especially for open-ended expert questions. The chart is a classic benchmark visualization – it’s comparing the accuracy of different approaches side by side so you can visually see which is better. Every dark bar towering over a light bar is basically saying, “Hey, thinking things through (and using tools) is beating the straight no-thinking approach.” It’s a comparative accuracy snapshot of various LLM strategies.

In summary, the meme graphic is highlighting how chain-of-thought prompting (“with thinking”) and tool use make a huge difference in an AI’s performance. It’s poking a bit of fun at how we present this fact. The title “Humanity’s Last Exam” is grandiose – suggesting this test is like the final boss battle between AI and human knowledge. Of course, in reality, a 42% score isn’t beating humanity yet, but it’s a big jump for AI. And the phrase “With thinking” almost sounds like an upgrade toggle you switch on an AI: Brain Mode: ON. The humor is that for an AI, “thinking” is literally a feature you have to prompt or enable! Developers find that both funny and fascinating, because it’s true: if you don’t prompt the AI to think things through, it often won’t. But if you do, it suddenly gets a lot smarter – just like a person taking the time to reason instead of guessing.

To illustrate, here’s a tiny example of an AI Q&A without chain-of-thought vs with chain-of-thought:

Question: What is the product of the first five prime numbers?
Assistant (no thinking): 46
Assistant (with thinking): Let’s break this down. The first five prime numbers are 2, 3, 5, 7, and 11. First, 2×3 = 6. Then 6×5 = 30. 30×7 = 210. Finally, 210×11 = 2310. So the product is 2310.

See the difference? The second answer took the time to work through each step (and got it correct: 2×3×5×7×11 = 2310), whereas the first answer was just a quick guess (46 was way off). In the chart, with thinking is akin to that step-by-step answer process, often combined with actually using a tool (like if the assistant had a calculator or wrote a short script to multiply those numbers). Without thinking is like the AI blurting out “46” with no working shown. The lesson is clear: even for AIs, thinking carefully and using the right tools leads to much better results.

Level 3: Thinking Cap On

For seasoned engineers familiar with machine learning A/B tests, this bar chart instantly evokes a knowing grin. It’s basically the “before and after” of telling a large language model to put its thinking cap on. The dark magenta bars show performance with the model thinking things through (and often using some tools), while the lighter pink bars are the same setups without that step-by-step reasoning. The meme’s humor lies in how obvious yet crucial this difference is: of course an AI does better when it’s allowed to reason out loud and use resources – but here it’s packaged like a shiny new feature called “With thinking”. Senior developers have seen this pattern since the GPT-3 days: add a prompt like “Let’s think step by step” or give the model a calculator and boom – your accuracy on hard problems jumps dramatically. This plot is essentially that familiar CoT bump (Chain-of-Thought bump) presented with dramatic flair.

Look at GPT-5 pro on the left: with the full toolset (Python + web search) and chain-of-thought, it hits 42.0% accuracy, whereas GPT-5 pro (no tools) lags at 30.7%. That ~11% absolute lift is huge in benchmark terms – roughly a 36% relative improvement just by letting the model use its “brain” (and by brain we mean a reasoning prompt and some handy APIs). The funniest part is the phrasing: “With thinking” vs “Without thinking.” It implies we’ve been running our poor AI on brain-turned-off mode this whole time. 😅 It’s a tongue-in-cheek way to describe what’s formally called chain_of_thought_prompting and tool augmentation. An experienced ML engineer might chuckle, “So, we gave the model a careful reasoning process and internet access – and you’re telling me it did better? Shocker!” The AIHypeVsReality tag comes into play because this very ordinary A/B test (enable the advanced reasoning agent vs use raw model) is being presented as “Humanity’s Last Exam” triumph. It’s a bit of classic OpenAI dramatic marketing: framing a routine benchmark upgrade as an existential milestone. The OpenAI swirl logo in the corner and the title suggest this came from an official reveal, likely hyping GPT-5 as a major leap toward AGI by showing it acing “the hardest test we could throw at it.” Yet, the senior crowd can’t help but note that a 42% score is far from acing – it’s more like a gentleman’s F if this were a real exam. The memester’s caption captures it: GPT-5 vaults past the rest, but only after flipping on thinking mode.

There’s also a shared war-story here about tool_augmented_llm experiments. Anyone who integrated an LLM with a Python REPL or a web browser (remember those early GitHub agents and the ReAct paper?) has seen surprisingly strong results. Letting ChatGPT run code or do web searches was all the rage in 2023 with projects like LangChain, AutoGPT, and OpenAI’s own plugins. This chart shows that trend going mainstream: ChatGPT agent (with browser + computer + terminal access) shoots up to 41.6%, nearly matching GPT-5 pro’s 42.0%. In other words, an older model given the right tools and the ability to think in multiple steps can rival a more advanced model that’s unassisted. That’s a key industry lesson: an architecture upgrade (agentic workflow) can be just as impactful as a model upgrade. It also validates the community’s intuition that chain-of-thought and tool use are basically “free lunches” for boosting AI performance – you don’t need a new model, just use the existing one more cleverly. The meme facetiously calls it Humanity’s Last Exam, as if this is the final boss test for AI supremacy. The senior folks know it’s likely a mashup benchmark (maybe an extreme version of MMLU or a collection of expert-level questions across subjects that even top humans would struggle to get perfect). We’ve heard grandiose claims before, so the title reads as playful hyperbole. And indeed, despite the ominous “last exam” wording, the best AI score here is 42%. (One can’t help but notice 42 – the Hitchhiker’s Guide answer to life, the universe, and everything – as the top score, adding an extra nerdy chuckle.) Humanity’s reign isn’t over quite yet, folks – our AI overlords still get more than half of the hardest questions wrong on the first try.

For a senior engineer, a chart like this also screams LLM evaluation gotchas and the realities of deploying these systems. Sure, you can crank up accuracy by letting the model think and use tools, but that often means higher latency, more cost, and complex orchestration in production. Each “thinking” step or code execution is extra tokens and compute. It’s reminiscent of the early days of search engines, where more complex queries yielded better results but at a processing cost. So while the meme highlights the almost comical gap between a model with brain vs no brain, it also reminds us why we don’t always run in full agentic_llm_workflows mode for every single request in practice. It’s a trade-off: AIHypeVsReality indeed – the reality is that those flashy 42% results might come with timeouts, higher bills, or tricky sandbox issues (imagine your GPT agent accidentally rm -rf / in that terminal tool 😜). Still, from a pure results standpoint, any veteran will acknowledge the impressive progress: GPT-4 (labeled here as GPT-4.0) was down at a pathetic 5.3% with no tools, essentially flunking out. In just a couple of generations, and with the clever use of agentic reasoning, we’re at 42%. The meme nails the sentiment: it’s “before you enable CoT” versus “after you enabled CoT”, dressed up as a dramatic bar chart. It’s equal parts technobabble and truth – something only AI humor can get away with in a single slide.

Level 4: Deliberation Dividend

At the cutting edge of AI reasoning, this chart highlights an advanced technique known as chain-of-thought prompting and its transformative effect. In essence, “with thinking” means the model is explicitly guided to deliberate step-by-step and even use external tools (like a Python interpreter or web search) before finalizing an answer. This taps into a more algorithmic problem-solving mode of the AI. Instead of one giant forward-pass guess, the model simulates an iterative reasoning process – akin to how a theorem prover or classical AI algorithm might break a problem into parts. The technical payoff is enormous: for example, GPT-5 (no tools) leapt from a mere ~6.3% accuracy to 24.8% on this expert-level exam just by enabling this internal reasoning chain. That’s a 4x increase in pass@1 accuracy, purely from prompting the model to "show its work" internally. We’re essentially seeing the model leverage a scratchpad in its token space, which aligns with research findings (e.g., Wei et al.’s Chain-of-Thought paper in 2022) that large transformers can solve much more complex tasks when they generate intermediate steps. Mathematically, it’s like unrolling a big computation: a single forward pass without chain-of-thought is a shallow mapping, but a prompted multi-step solution lets the network iteratively apply its 175B+ parameters to subproblems, compounding its reasoning depth.

Moreover, incorporating external tools transforms the model into an agentic system rather than a static Q&A machine. When GPT-5 pro is allowed to run Python code, perform web searches, and use a blocklist-filtered knowledge base, it effectively has an extended computational graph that can query external state and perform deterministic calculations. Theoretically, this pushes the system closer to Turing-completeness in practice – the language model can delegate tasks to a Python interpreter (which is Turing-complete), thereby overcoming the inherent limits of a fixed-depth transformer. From a research perspective, this is the fusion of learning (the neural network’s stored knowledge and patterns) with search (active information retrieval) and symbolic reasoning (through code execution). It’s a modern revisit of classic AI hybrids: a neural net brainstorm assisted by a logic engine sidekick. The results validate a core principle in AI-complete tasks: iterative reasoning + tool use yields qualitatively higher performance than a closed-box model. We see comparative accuracy skyrockets (42.0% vs 30.7%, 41.6% vs 23.0%, etc.) when the “thinking” switch is flipped on. This highlights an important point in LLM evaluation: many tasks that stump an LLM in direct response mode become solvable when the model can decompose the problem and consult external knowledge systematically. In other words, the limitations of a gigantic parametric model can be overcome by giving it a way to actively think and work – a result both theoretically intriguing and, as the meme humor implies, unsurprising in hindsight. After all, even in AI, deliberation pays dividends.