Gemini's Clean Sweep on the AI Leaderboard
Description
The image displays a leaderboard-style comparison of various large language models (LLMs) on a light-themed table. The top row, for the model 'gemini-exp-1206', is highlighted with a bright yellow background and shows a rank of '1' in every single category across the board. The categories include 'Overall', 'Coding', 'Math', 'Creative Writing', 'Instruction Following', and others. Competing models listed below it, such as 'chatgpt-4o-latest', 'gemini-exp-1121', 'o1-preview', and 'claude-3-5-sonnet', all have varying, higher numerical ranks, indicating lower performance. Overlaid on the image in a large red font is the text 'Gemini top-1 in all domains!'. In the bottom right corner, the watermark 'lmarena.ai' is visible. This image is a benchmark result, likely from a platform like LMSys Arena (lmarena.ai), which pits LLMs against each other. The visual unequivocally declares that a new experimental Gemini model has achieved the top rank in all measured capabilities, a significant event in the highly competitive field of AI development. For developers, this signifies a potential new state-of-the-art model to experiment with or integrate into their workflows
Comments
27Comment deleted
The 'gemini-exp-1206' model's performance is so dominant, its developers are probably running benchmarks now to see if it can achieve a rank of 0
Pretty sure the query behind this slide was just `SELECT 1 AS rank FROM benchmarks;` - marketing-driven AGI achieved!
Ah yes, the quarterly 'our model is #1' leaderboard shuffle - where every vendor cherry-picks their benchmarks until everyone's simultaneously winning. Next week's update: 'Actually, we meant #1 on Tuesdays, evaluated on palindromes, with style control set to 'baroque.'
When your model achieves perfect 1s across all benchmarks, you know it's either genuinely revolutionary or the evaluation criteria were suspiciously well-aligned with your training data. Either way, the marketing team is already drafting the 'AGI achieved' press release while the engineers nervously check if anyone tested adversarial prompts
Great - Gemini is #1 everywhere; wake me when there’s a column for “writes idempotent migrations, respects rate limits, and doesn’t hallucinate new APIs during a Sev‑1.”
Gemini crushes arena benchmarks in coding and math - too bad it can't outlast your monolith in a production outage
Great - rank‑1 across every column; Goodhart’s Law just filed a Sev‑2: “optimized for the leaderboard, regressed in production.”
All my experience with Gemini is worse than even GPT-3.5. It's bullshitting everything, effectively transforming every request into some kind of trendy social media agenda. Comment deleted
try api version of gemini, it should solve some of them with prompt, however totally agree about those weird formatting Comment deleted
I even tried ScholarAI Comment deleted
Yeah, same here, that’s why I posted it since it finally something Comment deleted
Aider Code editing benchmark Comment deleted
To me it's kinda the best benchmark out there right now in regard of coding capabilities Totally aligned with my own experience of models usage Comment deleted
Oh, here's an example from today: I asked Gemini how to enable caching for requests with authorization using OutputCache in C#, .NET 8. It took me 5 attempts. Three of the answers were just hallucinations and syntax errors (lol), one was complete BS with a lecture about caching, and on the fifth attempt, it gave me an answer about custom policies - but Gemini didn't override the default behavior (excludeDefaultPolicy), so that was a fail too. I'm glad Gemini can handle childish coding tasks with a 70% success rate, but beyond that, its coding capabilities are a joke. Comment deleted
Why did you asked Gemini about something? Comment deleted
Or you’re talking about 1206? Comment deleted
Nope, it's gemini-pro-experimental-0827. My bad. It seems I don't have access to the latest one Comment deleted
sonnet still the real goat tho fr 🙏🙏 Comment deleted
my own flesh-bag brain beats all of these, checkmate Comment deleted
do you know what version is on the google assistant on pixel phones? Comment deleted
Disabled. Really, who uses it? Comment deleted
I try it Comment deleted
Tested. Nope. Same BS and hallucinations. I usually ask LLMs for how-to instructions to avoid reading tons of documentation. They provide code snippets I can use or modify, but for anything more complex than basics, they mix up everything and include unnecessary code. While o1 might try to debug the issues, Gemini just goes off the rails and produces even more hallucinations. Claude, however, is still the best. Comment deleted
Gemini also has a very low response size limit Comment deleted
Also, with ChatGPT o1 or even 4o, if you role play a bit and are willing to go back and forth with revisions, it can actually be useful. It works great if you don't have somebody that can help you debug for instance, but you shouldn't expect it to just spit out a fully functional app first try Comment deleted
In my experience if you actually want it to write some code, you have to be very specific with your design requirements, as in, you have to offer it implementation options to your best understanding. You actually need to know what roughly it has to write for it to write anything remotely useful. Comment deleted
After all, an LLM is not an engineer, you are the engineer. Garbage in, predicted garbage out Comment deleted