Anthropic Releases Claude 4 Benchmark Chart, Fueling the AI Model Wars
Description
A clean, professional-looking chart titled 'Claude 4 benchmarks' presents a performance comparison of various large language models. The table lists several models in columns: Claude Opus 4, Claude Sonnet 4, Claude Sonnet 3.7, OpenAI o3, OpenAI GPT-4.1, and Gemini 2.5 Pro Preview. The rows represent different benchmark tests, including 'Agentic coding (SWE-bench Verified)', 'Graduate-level reasoning (GPQA Diamond)', and 'Visual reasoning (MMMU validation)'. The cells are populated with percentage scores, some of which show two numbers separated by a slash. At the bottom, a 'Methodology' section provides five footnotes detailing the testing conditions, such as sampling techniques. This image is a typical example of a competitive benchmark report released by an AI company (Anthropic, indicated by the 'AI' logo) to showcase its models' capabilities against major rivals from OpenAI and Google. For experienced engineers, these charts are viewed with interest but also healthy skepticism, as the results are highly dependent on the specific tests and methodologies, which may not always reflect real-world performance on complex, production-level tasks
Comments
30Comment deleted
Ah, the quarterly LLM benchmark release. A time-honored tradition where every company presents their model as a valedictorian, conveniently omitting the subjects it failed and the open-book policy detailed in the methodology footnote
Turns out even models with eight-digit parameter counts panic at the Terminal-bench - proof that "saving the demo with sudo !!" remains unsolved AI alignment
Watching Claude 4 beat GPT-4.1 at coding benchmarks is like watching your junior engineer fix that legacy codebase you've been avoiding for months - impressive, slightly concerning, and makes you wonder if you should update your LinkedIn to 'AI Prompt Architect' before the models start reviewing your PRs
Claude Opus 4 scoring 90% on high school math while only hitting 50% on terminal tasks perfectly captures the AI industry's current state: we've built models that can ace calculus but still struggle to `cd` into the right directory. It's like hiring a PhD who needs help with basic shell navigation - technically brilliant, practically humbling. The real benchmark isn't whether AI can solve AIME problems; it's whether it can survive a production incident at 3 AM without accidentally `rm -rf /` the entire deployment
Claude 4 aces agentic coding benchmarks - now if only it could untangle our 20-year monolith's cyclic deps
Nothing says rigorous science like a leaderboard where the big numbers are above tiny footnotes about bash tools, nucleus sampling, and parallel test-time sequence selection - the peer-reviewed version of “works on my GPU.”
Amazing how we’re 80% on SWE-bench once you allow pass@k, sampling, and toolchains - then Terminal-bench drops to ~30%, i.e., the difference between your slide deck and ssh-ing into prod
You? Comment deleted
FFS! Comment deleted
No, for real, wtf is going on 😂 Comment deleted
Well, i know want to see their naming for releases. Comment deleted
You might have guessed it V0-1.0 🤣 Comment deleted
Ohhh no, I’m sorry, it’s even more funny v0-1.0-md Comment deleted
bias(math.random()) - "ohhhh this model is so smart in reasoning and science" Comment deleted
What problem does it solve? No, those numbers don't count, I meant problems, not hand picked tasks that are very likely in the training data Comment deleted
But can it count letters 'o' in 'strawberry'? Comment deleted
Just did it for fun Comment deleted
Time to buy a land plot and become farmer, I know Comment deleted
Finally! Comment deleted
I didn’t get what the point of the question tho Like, if you really thought it will fail then you really need to spend some time to finally experiment with frontier models (and spend some money on it, yes) Comment deleted
My friend once tried to get right answer from gpt, he asked to count letters 'o' in word 'молоко'. He spent at least 30 min teaching him how to count, lol Comment deleted
Tell LLaMa 3.1 7B (local) to carefully evaluate and it oneshots the strawberry question interestingly enough. Just ask and you get a correct answer once every blue moon Comment deleted
What is special in counting? Comment deleted
Everyone is still obsessed with tokenizer issues I suppose 🥲 Comment deleted
Yes because thats in the training data now.. but ask a different word and the "sota" models fuck up again 😂 Comment deleted
This is much more complicated now Companies preparing training data is actually worth much more money than they should Comment deleted
SWE-bench methodology For the Claude 4 family of models, we continue to use the same simple scaffold that equips the model with solely the two tools described in our prior releases here—a bash tool, and a file editing tool that operates via string replacements. We no longer include the third ‘planning tool’ used by Claude 3.7 Sonnet. On all Claude 4 models, we report scores out of the full 500 problems. Scores for OpenAI models are reported out of a 477 problem subset. For our “high compute” numbers we adopt additional complexity and parallel test-time compute as follows: We sample multiple parallel attempts.We discard patches that break the visible regression tests in the repository, similar to the rejection sampling approach adopted by Agentless (Xia et al. 2024); note no hidden test information is used.We then use an internal scoring model to select the best candidate from the remaining attempts.This results in a score of 79.4% and 80.2% for Opus 4 and Sonnet 4 respectively. Comment deleted
Neither is a slightly bigger number on some bs "benchmark" Comment deleted
I meant if you straight up ask. I added "carefully evaluate your answer" and it splits it into letters and answers correctly. Weird. Comment deleted
Well, but where's vibe coding benchmark?🤔 Comment deleted