Skip to content
DevMeme
6434 of 7435
LLM Task Duration Chart Shows Exponential Growth in AI Agent Capabilities
AI ML Post #7055, on Aug 18, 2025 in TG

LLM Task Duration Chart Shows Exponential Growth in AI Agent Capabilities

Why is this AI ML meme funny?

Level 1: The Super-Speed Helper

Imagine you have a helper friend who keeps getting smarter and faster every year. 😊 At first, this friend could only help with tiny chores, like quickly fetching a fact from a book for you. Next year, they got better – they could help you do a small homework problem or write a short email. As time went on, your helper friend started doing bigger jobs: one day they could even help build a whole little toy or solve a tough puzzle that would normally take you an hour! Now, the funny picture here is like a chart showing how much work this helper can do as it gets smarter. By the far right side of the chart (which is like a couple of years in the future), it suggests your helper might handle a 2-hour task all by itself. That’s a long task – something like cleaning your whole room or doing a big school project – and the helper could maybe do it just as well as you about half the time.

Why is this cartoon chart amusing to people who make software? Because it feels like saying: “Pretty soon, this super-speed AI helper will be able to do a job that takes a person two hours, and it’ll do it in a snap!” It’s both cool and a little spooky. It’s cool because who wouldn’t want a magical helper that can save you time? Imagine your chores or homework being done in minutes rather than hours. But it’s a bit spooky because the people who normally do that work (like you, or grown-up developers writing code) might start wondering, “Uh, so what do I do now? And is someone going to expect me to get twice as much done since I have this helper?” It’s like if a teacher suddenly gives you twice the homework because they know you have a calculator or the internet – instead of the tool making life easier, it could make the teacher raise their expectations.

In the meme’s caption, a person jokingly asks, “Is this 2-hour task that GPT-5 can do by itself in the room with us right now?” They’re treating the AI a bit like an invisible ghost or an imaginary friend that might be hiding nearby, ready to jump in and do the work. It’s a playful way to express that weird feeling: we can’t see this AI helper, but everyone’s talking about it as if it’s here and capable of almost anything. The emotional core of the joke is a mix of wonder and worry. We’re amazed that technology is improving so fast – tasks that used to take a lot of human effort might soon be done by a computer talking like a person. But we’re also a tiny bit nervous, like when a super smart robot shows up in a story: will it be a great helper, or will it make things confusing for the humans? In simple terms, it’s funny because it’s like telling a kid, “Next year, your robot buddy will be able to do all your math homework in no time!” – you’d grin at how awesome that sounds, but you might also wonder if the teacher will just make the homework harder.

Level 2: AI Taking on Tasks

Let’s break down what’s going on in this chart and why developers find it funny (and a bit scary). The chart is showing how Large Language Models (LLMs) – that is, huge AI systems like GPT-3 and GPT-4 – have been getting better at handling more complex tasks over time. The x-axis at the bottom is labeled “LLM release date” (years 2020 through 2026), so as you move right, you’re looking at newer and more advanced AI models. The y-axis on the left shows “Task duration (for humans)”, going from 0 minutes up to 2 hours. A point on this graph basically says: “By this date, we think the AI has a 50% chance of succeeding at a task that would take a human this long to do.” In other words, as we get to newer AI releases (toward 2025-2026 on the right), the green and grey dots climb higher on the y-axis – meaning the tasks that AI can tackle are longer/harder. A dashed green curve is drawn through the green dots, showing an overall trend that looks exponential (starting almost flat near 0 minutes, then shooting up towards 2 hours by 2025+). Exponential is just a math-y way of saying “starts off slow, then skyrockets fast.” It’s like how each generation of smartphones gets not just a bit better, but a lot better in some capabilities – here it’s AI getting a lot better at doing work for us.

The green dots are labeled with model names: GPT-3, Claude 3.5 Sonnet (Old), o1, Claude 4 Sonnet, o4-mini, Grok 4, and GPT-5. These represent various AI models over the years:

  • GPT-3 and GPT-5 are from OpenAI (GPT stands for Generative Pre-trained Transformer). GPT-3 came out in 2020 and was one of the first very powerful language models that could do things like write essays or code snippets. GPT-4 (released 2023) is even more advanced, and GPT-5 is the expected next version in this sequence (not actually released as of 2025, but here it’s imagined or “projected” to exist soon).
  • Claude models (Claude 3.5, 3.7, 4 Sonnet, etc.) are from another AI company called Anthropic. Claude is similar to GPT in that it’s also a big language model you can chat with or ask to do tasks. The “Sonnet” might be an internal name or version (the meme uses it presumably to differentiate versions, like Claude 4 “Sonnet” being a particular variant). Realistically, Anthropic had models like Claude 1, Claude 2, etc., but the exact names here are a bit tongue-in-cheek.
  • o1, o3, o4-mini are a bit mysterious in name – possibly they stand for other models or open-source efforts. They’re shown as green too, so likely notable AI releases. For example, they could hint at something like an open-source model series (just using “o” as a generic label). The specifics aren’t as important; the key idea is there have been multiple players releasing models.
  • Grok 4 sounds like a fictional model name (there isn’t a well-known AI called Grok 4 as of now). The term “grok” means to understand deeply (it’s a slang from a sci-fi novel). So it’s probably a playful way to name a hypothetical AI that really understands things. It suggests another competitor’s model or maybe a future version of something like Google’s AI.

All those grey dots on the chart represent other AI models (unnamed here) in the same time frame – the grey color likely means “not the main ones we’re focusing on.” They form a cluster around the green trend line, implying lots of models had various capabilities, but the best ones (the green labeled ones) are defining the frontier. Think of grey dots as the dozens of AI models or versions that came out (some from universities, some from companies like Meta, Google, etc.) over the years, each with their own level of capability.

Now, the labels next to the dots like “Find fact on web” and “Train classifier” are examples of tasks and roughly how long those tasks take a human:

  • “Find fact on web” – maybe a quick Google search that a human does in a minute or two. That label is placed near the very bottom (near 0 on the y-axis, around year 2020). It suggests that an early model (like Claude 3.5 Sonnet (Old) or GPT-3) could handle a very short task like finding a fact online with ~50% success. Indeed, GPT-3 was often used for quick Q&A or retrieving facts it had ingested during training.
  • “Train classifier” – this is a more involved task, placed around the 1 hour mark. Training a classifier means taking some dataset and teaching a simple AI (a classifier) to distinguish things (like recognizing spam vs non-spam emails). For a human, setting up and running a training job, then evaluating it might take roughly an hour or more (especially if coding from scratch or using a new dataset). The chart positions “Train classifier” around early 2024 or so with a model labeled Claude 3.7 Sonnet – implying that by that time, an AI could half the time successfully do the steps to train a classifier on its own. In practical terms, that could mean the AI writing Python code to train a model, running it (maybe in a cloud environment), and getting results without human help.
  • There might be other tasks implicitly on the chart even if not explicitly labeled – presumably something at 1.5 hours (maybe “Code review” or “Write a complex script”). The context tags mention code_review_autopilot, which hints that one of the tasks around the 1.5h region could be an AI doing an entire code review automatically. Another could be writing a significant piece of software or a “build_vs_buy_decision” analysis (like researching whether to build something or use a library, which can take hours of reading docs – an AI might do that faster soon).

So, what does “50% chance of succeeding” mean here? It means if you give that task to the AI, there’s a half-and-half chance it will do it correctly (or well enough) without human intervention. A 50% success rate is far from perfect (it’s basically a coin flip whether you’ll get a good result or not), but it’s also not negligible – it means AI is sometimes capable of it. For context, earlier models had 0% chance on an hour-long coding task – they’d almost certainly fail or produce junk. By the time it’s 50%, we treat that as noteworthy progress. It’s like saying: “We’re at the point where an AI is as likely as not to actually finish this fairly big task correctly.”

The meme is poking fun at sprint planning and developer life. Backlog grooming is the process where a dev team looks at all the upcoming tasks (the backlog), estimates them, and decides which ones to do in the next sprint (usually a 1-2 week cycle of work). In backlog grooming, tasks that are multi-hour are common – like a ticket that might take a dev a day or two is broken into sub-tasks of few hours each. The joke here is that if AI can now do, say, a 2-hour task on its own with decent probability, how do we plan around that? Do we assign such tasks to the AI? Do we count them in our team’s velocity (velocity = how much work a team completes per sprint, typically measured in story points or hours)? The line about “negotiating with your model about velocity” imagines a funny scenario: you treat the AI as if it’s a team member. Picture asking ChatGPT, “Hey, can you commit to implementing the login feature this sprint? Are you 50% confident you’ll finish it?” It’s absurd, because AIs don’t quite work like a person in a team – they don’t promise things or give reliable timelines – but teams are kind of already using AI tools to speed up work.

Now, let’s connect that to AI_hype and AI_limitations. There’s a lot of hype that AI will massively boost developer productivity – some say by 2x or more. If one sprint you delivered 5 features, maybe with AI you’ll deliver 10 in the same time. That’s the hype, reflected by this steep chart suggesting ever more capability. The reality is more nuanced: yes, AI can handle certain tasks quickly (like writing boilerplate code, or suggesting how to fix a bug, which might save a dev 30 minutes here, an hour there), but it often still requires a human to oversee and correct it. 50% success rate means the AI might do half the job, and the other half a human needs to either redo or guide. Developers who try these tools learn their quirks – e.g., an AI might write code that looks confident but actually has a security flaw or doesn’t handle an edge case. That’s an AI limitation: it lacks true understanding of what it doesn’t know or when it’s wrong (we call this tendency to “hallucinate” in AI – producing convincing but incorrect or made-up answers). So we can’t just trust it blindly, especially for tasks that really matter if done correctly.

Nevertheless, the trend is undeniable that each new model is tackling bigger challenges. GPT-4, for instance, can often write pretty complex code or analyze a big chunk of text way better than GPT-3 could. By the time GPT-5 rolls around (if it follows the pattern), maybe it can manage multi-hour tasks like writing a whole module of software or doing an in-depth technical research summary. The meme teases that this might happen “within the next sprint cycle,” which is an exaggerated way to say “very soon.” (Usually a sprint cycle is just a couple of weeks, so saying an AI will swallow multi-hour tasks by the next sprint is hyperbole for humorous effect – it really means “imminently, practically tomorrow!”). Developers often use that kind of phrasing jokingly, like “Careful, by next sprint the new framework will replace all our code,” knowing that things don’t change that instantly.

Let’s also clarify sprint_capacity_planning: teams have to plan how much work they can get done in a sprint. If an AI can do some tasks, theoretically that increases the team’s capacity. For example, if an AI assistant can handle writing all the unit tests (something that might have taken a junior dev a full day), the team can plan to do more feature work in that sprint. But planning that is tricky – you have to trust the AI’s output quality and factor in the time for a human to verify the AI’s work. Right now, many teams treat AI as a helper tool, not as an autonomous team member. So it’s more like each developer might complete tasks 10-30% faster using the AI for help on parts of it. We’re not yet at the point where you’d hand a task entirely to an AI and say “see you in 2 hours with the finished work” – though the graph humorously suggests that is exactly what some might hope for with GPT-5.

Finally, the meme draws its “quiet dread” vibe from developer_productivity pressure. If the boss sees this chart, they might think, “Great, we can squeeze in more tasks per sprint because AI has got our back.” Developers are a bit anxious about that because it might lead to unrealistic expectations. Instead of AI giving us a breather or time to focus on really hard problems, it could just mean we’re expected to deliver the same work faster. It’s like when we got faster computers and better tools – sometimes it just meant the deadline moved up correspondingly. The joke, in simple terms, is that as soon as AI can do something in 2 hours that used to take you 2 hours, your manager might assign you two extra hours of work elsewhere. So the promise of making life easier comes with a catch: the bar for “normal output” might get raised. That feeling is summed up perfectly by the joking question, “Is this 2-hour task that GPT-5 can do by itself in the room with us right now?” It’s as if someone is looking over their shoulder for the AI that’s coming to steal their task (or conversely, to help with it). It’s funny and a bit unnerving at the same time – which is exactly why this meme resonates with developers figuring out how they and AI will collaborate (or compete) on the next big project.

Level 3: Exponential Expectations

From a senior engineer’s perch, this meme hits a nerve: it’s a graph of AI progress that looks equal parts exciting and ominous. The humor comes from seeing our sprint planning realities distilled into a sleek exponential plot. Those green dots (GPT-3, GPT-4, Claude, “o” models, Grok 4, etc.) creeping up into multi-hour territory mean one thing in manager-speak: “If AI can handle bigger tasks, let’s raise the targets!” Seasoned devs know that tone all too well. The chart might as well be titled “Recalibrate Your KPIs, Human”. We chuckle (and maybe cringe) because we’ve sat through meetings where some exec brandishes a trend line of productivity or technology improvement and asks for impossible leaps. AI_hype_vs_reality is a familiar rollercoaster and this meme visualizes the hype: it compresses years of LLM improvements into that steep green dashed line, shooting almost vertically by 2025. It’s Moore’s Law vibes, but instead of transistors it’s task complexity. And just like Moore’s Law, folks in the trenches feel the exponential expectations – every year, deliver twice as much with the new tools, or so we’re told.

The specific combination of elements here is brilliant satire. We have an academic-looking chart (complete with axis labels and confidence jargon “50% chance of succeeding”) used to convey something every dev has joked about: “Will GPT-5 take my job… or at least this annoying two-hour task?” By plotting tasks like “Find fact on web” near 2020 and “Train classifier” around 2023, it reminds experienced devs of how our workflow has already changed. Searching the web for a fact? That’s practically instantaneous with modern AI assistants – what used to take us 10 minutes of Googling, an LLM can now summarize in seconds by regurgitating its trained knowledge (with varying accuracy, of course). Writing boilerplate code or a unit test that might take 30 minutes? GitHub Copilot and friends are on it. Heck, we’ve gone from Stack Overflow-driven development to AI-assisted development in such a short time. So, seeing “1h 30m” and “2 hours” up there for future models is both a logical next step and a tongue-in-cheek “brace yourself” for devs.

Why is this funny to us? It’s that quiet dread mixed with amusement. On one hand, who wouldn’t want an annoying two-hour grunt-work task automated? Senior devs certainly aren’t sentimental about, say, trawling through logs for an hour or converting mockup to HTML again. But on the other hand, we know the trade-off: if AI handles the easy half of a task, management might assume the other half magically vanishes too. The meme caption nails this feeling with a joke:

“is this 2 hour task that gpt5 can do by itself in the room with us right now?”

It riffs on the classic therapy quip (“Is the ghost that haunts you in the room with us now?”), implying we’re almost paranoid about an invisible GPT-5 lurking around ready to snatch up work. It’s funny because some of us have caught ourselves thinking, “Could I just let ChatGPT handle this while I grab coffee?” – a mix of hope and nervousness that maybe it could, and what that would mean.

IndustryTrends_Hype element: We’ve all seen charts like this at tech conferences and in glossy reports, predicting AI will do X by year Y. They’re usually overly optimistic. A senior dev reading “50% chance of succeeding” also recalls that 50% success for an automated system is basically a coin flip – not something you’d bet the production server on. It’s a sly nod: yes, GPT-5 might attempt a two-hour coding task, but there’s a decent chance it fails hilariously (or introduces a subtle bug that takes four hours to debug later). So there’s an AI_limitations undercurrent here. We wink at each other knowing that “50% success” means you’ll still be babysitting that AI. It’s like working with a super-speedy but moody junior developer: they might crank out a feature by lunchtime or produce gibberish that you must untangle. Real-world early adopters of AI coding tools have seen both outcomes.

The meme also touches on developer_productivity and how it might be measured in the AI era. The line “backlog grooming might soon include negotiating with your model about velocity” is a fantastic image. Imagine sprint planning with an AI: Product Manager: “Can we commit to these 5 user stories?” Lead Dev: “Our AI helper promises it can handle two of them – it swears it has a 50% chance on the complex one!” It’s absurd, yet some teams are already experimenting with assigning certain tasks to AI assistants. Code review, for instance – what the meme tags call code_review_autopilot – is becoming a thing (AI suggests fixes or spots bugs in pull requests). But seniors know the caveats: the AI might catch the low-hanging fruit and miss the deeper issues that a human would notice from experience. So you can’t (yet) just let it fully autopilot. Still, the sprint_capacity_planning conversations are changing. Instead of a flat “we have 5 engineers, 2 weeks, that’s X story points of capacity,” it’s now “we have 5 engineers + maybe GPT-4 pair-programming, so maybe X+? story points.” That build_vs_buy_decision now sometimes includes a third option: “build, buy, or prompt”. Do we code a feature from scratch, purchase a tool, or ask the AI to generate a solution on the fly? A senior dev knows the trap here – that third option can be quick to start but costly to finish if the AI output is misleading.

Historically, every big hype (from cloud, to DevOps, to low-code platforms) came with promises of “freeing us for more important work.” In practice, it often means the goalposts move. This meme’s vibe of “productivity KPIs being recalibrated yet again” is so on point. Veterans will recall how adoption of cloud meant the team was expected to maintain more systems since “infrastructure is easy now,” or how adding automation test suites meant testers were expected to handle more projects because “the tests do the work.” The same is happening with AI: if GPT-5 can handle a two-hour task, maybe your manager decides that task is now worth zero points and doesn’t allocate human time—until it fails, of course, then it’s all hands on deck to fix it Friday at 5 PM. It’s the classic hype vs reality cycle: initial productivity boost, followed by a reality check (and sometimes a late-night bug hunt).

In essence, the meme is “too real” because it jokingly predicts a near future many of us are already negotiating in small ways. It captures both the marvel of these LLMs’ rapid progress and the side-eye skepticism of those who will actually integrate this progress into a team workflow. We laugh, perhaps a bit nervously, because the graph suggests our role is changing fast. And as any senior dev knows, when something seems to be moving exponentially, you’d better buckle up – because somewhere on that curve, things get messy before (if ever) the promise is fully realized. For now, we’ll keep joking about that ghost in the backlog (GPT-5, we’re looking at you) while carefully reviewing the pull requests it claims it can handle. After all, it might only have a 50/50 shot… but our name is on the commit history if it merges broken code into main!

Level 4: Emergent Complexity Frontier

Behind this tongue-in-cheek chart lies a serious AI trend: each new generation of Large Language Model pushes the frontier of tasks they can handle, almost like a computational Moore’s Law for cognitive capability. The dashed green curve hugging an exponential trajectory isn’t arbitrary hype – it echoes real observations from research on scaling laws. In formal terms, as we double or triple model parameters and training data, we often see non-linear performance jumps on complex tasks. This means an LLM like GPT-5 (presumably larger and more finely tuned than GPT-4) might display emergent abilities that GPT-3 or earlier simply couldn’t tackle. For example, GPT-3 could manage trivial queries or generate paragraphs of code with guidance, but GPT-4 surprised many by handling nuanced multi-step reasoning and even passing professional exams. These are emergent properties – skills that weren’t explicitly programmed but “surface” once the model’s scale crosses a threshold. The chart’s green dots for GPT-3, Claude, o1, Grok 4, etc., are essentially markers on this frontier of emergent complexity. They suggest that by 2025-2026, LLMs might reliably attempt tasks taking up to ~2 hours of human effort.

Consider the task “Train classifier,” annotated around the one-hour mark. That’s a multi-step project for a human: writing a training script, running it on data, tweaking parameters, evaluating results. For an AI, this involves not just regurgitating facts but performing coordinated reasoning and tool use. How could an LLM succeed here? One theoretical avenue is integrating the LLM with external tools or having it generate code which, when executed, trains the model. In AI research, this falls under AI planning and tool orchestration. Early signs are already here: there are frameworks where an LLM calls APIs or writes and runs code in a loop (think of experimental projects like AutoGPT and others). To have a 50% success rate on a one-hour coding task, GPT-5 might internally simulate those hours of work in minutes: it could output a complete Python script for training a classifier, self-check the code, maybe even debug errors by reasoning through stack traces – all within its extended context window. Speaking of context, note that one limiter for LLM “thinking” time is context length (how much the model can read & remember at once). Each new generation often has a larger context window (GPT-4 can handle ~32K tokens ≈ 50 pages of text). If GPT-5 extends this further or employs a form of long-term memory, it could ingest more information and plan longer tasks without forgetting earlier steps, essentially stretching its effective attention span closer to hours.

Yet, even an exponential trend faces physics and scaling limits. There’s an implicit half-joke in “50% chance of succeeding” – it hints at the stochastic nature of these models. Unlike a deterministic program, an LLM generates outputs probabilistically, so complex tasks aren’t guaranteed successes each run. In theoretical computer science terms, you might say we’re pushing up against tasks that approach the AI’s current “competency ceiling.” Achieving >50% reliability on 2-hour tasks might need qualitative model improvements, not just more FLOPs: think better problem decomposition (breaking big tasks into smaller subtasks, a known hard problem called AI planning), or model architectures beyond today’s Transformers to overcome issues like long-term dependency and context fragmentation. There’s active research into making AI reasoning more robust – from incorporating symbolic logic, to neural executive functions that plan and verify steps (almost like how a developer writes pseudocode to outline a solution). In short, the meme’s chart exaggerates with humor, but it reflects a real technical frontier: the race to expand an AI’s effective working duration on a problem, approaching human-scale continuous work (multi-hour cognitive tasks). It’s a dizzying exponential model scaling curve: if progress continues unabated, tasks that once took an afternoon of human concentration might soon be solvable by letting a supercharged LLM churn away through lunch. The real question is, will these models just get faster at predicting text, or will they evolve new internal mechanisms to truly “think” for two hours straight? The answer determines if the trendline keeps shooting up or finally plateaus when raw scale alone no longer cuts it.

Description

A scatter plot titled 'Task duration (for humans) where we predict the AI has a 50% chance of succeeding' with the Y-axis showing time from 0 to 2+ hours and X-axis showing LLM release dates from 2020 to 2026. Green dots represent specific models: GPT-3 (near 0), Claude 3.5 Sonnet Old (~15min), o1 (~35min), Claude 3.7 Sonnet (~50min), Claude 4 Sonnet (~1hr), o4-mini (~1h15m), o3 (~1h30m), Grok 4 (~1h45m), and GPT-5 (~2h15m). Gray dots with error bars show additional data points. A dashed green trendline shows exponential growth. Reference tasks on the Y-axis include 'Find fact on web' and 'Train classifier'

Comments

26
Anonymous ★ Top Pick At this rate, by 2028 AI agents will be able to handle tasks that take a human 8 hours - which means they'll finally be ready to attend a full day of meetings and still get nothing done
  1. Anonymous ★ Top Pick

    At this rate, by 2028 AI agents will be able to handle tasks that take a human 8 hours - which means they'll finally be ready to attend a full day of meetings and still get nothing done

  2. Anonymous

    The scary part isn't that GPT-5 can do a 2-hour task; it's that it will probably spend the first hour arguing about the requirements in a Slack channel, just like a human

  3. Anonymous

    If this curve holds, GPT-6 will finish your code review, draft the retro notes, and file a Jira ticket reminding you that humans are now the blocking dependency

  4. Anonymous

    Ah yes, the classic exponential curve where we're always just 2 years away from AGI - reminds me of how we've been 5 years away from fully autonomous vehicles since 2015. At least this chart is honest about the 50% success rate, unlike my junior devs claiming their code is 'production ready'

  5. Anonymous

    Ah yes, the classic 'AI will replace us by 2025' chart - where every new model release pushes the goalposts from 'find fact on web' to 'replicate two hours of human cognitive labor.' Notice how we've gone from GPT-3 barely managing web searches to GPT-5 allegedly matching a junior developer's afternoon sprint. The real joke? By the time GPT-5 actually ships, we'll have redefined 'human-level performance' to mean 'can survive a 4-hour architecture review meeting without suggesting we rewrite everything in Rust.' The dashed green line of exponential progress conveniently ignores that humans also get coffee breaks, context switching overhead, and existential dread - none of which scale logarithmically

  6. Anonymous

    By 2026 GPT-5 handles two-hour tasks at 50% success; the production pattern is idempotent retries with exponential backoff - until Finance pages us

  7. Anonymous

    Great, we finally automated two hours of work with a 50% SLO - marketing calls it AGI, SRE calls it a postmortem

  8. Anonymous

    Inference latency's true scaling law: O(parameters × existential_doubt)

  9. @apBUS_amp_K 10mo

    Houston, the error bars have entered the space!!!

  10. @fkarg10 10mo

    the task in question

    1. @Vlasoov 10mo

      Generic answer more or less complicated Linux question

  11. @sysoevyarik 10mo

    "50% chance of succeeding" OpenAI calculations is: it's always 50% chance. It succeeds or it doesn't.

    1. @ZgGPuo8dZef58K6hxxGVj3Z2 10mo

      Bro they cant even scale graphs consistently

      1. @sysoevyarik 10mo

        Oh I forgot 😁

  12. @sysoevyarik 10mo

    Also OpenAI dudes too dumb to use logarithmic scale

    1. @Iizvullok 10mo

      I think they chose to make it linear on purpose so it looks better.

    2. @deadgnom32 10mo

      where? I see its quite linear here

    3. @deadgnom32 10mo

      ah I get it. they could have used it, but they didn't. I think it's because most people can't properly comprehend log scale

  13. @Algoinde 10mo

    And if you read the error bars, you will find the average LLM error rate

  14. @deerspangle 10mo

    "Find fact on web" , 10 minute task. Ah yes, the contractor billing by time approach

    1. @SamsonovAnton 10mo

      That's what my bosses think, especially on facts that nobody would publish on the web, at least for free. 🥲

  15. @rliskovenko 10mo

    Procrastinating for two hours -- achievement unlocked

  16. @mrYakov 10mo

    Its totally wrong comparing old models, that try to give answer instantly and modern agentic workflow, that consuming a lot of time rechecking everything. apply modern workflow to older models and their results will be much higher

  17. @jackietanyen 10mo

    Error bars used to mean something

  18. @qtsmolcat 10mo

    That 50% success rate is doing a lot of heavy lifting here

  19. @cacojo15 10mo

    An example of task that takes 2h for a human: "Exploit a buffer-overflow in libiec61850" Source: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/

Use J and K for navigation