LLM Task Duration Chart Shows Exponential Growth in AI Agent Capabilities
Why is this AI ML meme funny?
Level 1: The Super-Speed Helper
Imagine you have a helper friend who keeps getting smarter and faster every year. đ At first, this friend could only help with tiny chores, like quickly fetching a fact from a book for you. Next year, they got better â they could help you do a small homework problem or write a short email. As time went on, your helper friend started doing bigger jobs: one day they could even help build a whole little toy or solve a tough puzzle that would normally take you an hour! Now, the funny picture here is like a chart showing how much work this helper can do as it gets smarter. By the far right side of the chart (which is like a couple of years in the future), it suggests your helper might handle a 2-hour task all by itself. Thatâs a long task â something like cleaning your whole room or doing a big school project â and the helper could maybe do it just as well as you about half the time.
Why is this cartoon chart amusing to people who make software? Because it feels like saying: âPretty soon, this super-speed AI helper will be able to do a job that takes a person two hours, and itâll do it in a snap!â Itâs both cool and a little spooky. Itâs cool because who wouldnât want a magical helper that can save you time? Imagine your chores or homework being done in minutes rather than hours. But itâs a bit spooky because the people who normally do that work (like you, or grown-up developers writing code) might start wondering, âUh, so what do I do now? And is someone going to expect me to get twice as much done since I have this helper?â Itâs like if a teacher suddenly gives you twice the homework because they know you have a calculator or the internet â instead of the tool making life easier, it could make the teacher raise their expectations.
In the memeâs caption, a person jokingly asks, âIs this 2-hour task that GPT-5 can do by itself in the room with us right now?â Theyâre treating the AI a bit like an invisible ghost or an imaginary friend that might be hiding nearby, ready to jump in and do the work. Itâs a playful way to express that weird feeling: we canât see this AI helper, but everyoneâs talking about it as if itâs here and capable of almost anything. The emotional core of the joke is a mix of wonder and worry. Weâre amazed that technology is improving so fast â tasks that used to take a lot of human effort might soon be done by a computer talking like a person. But weâre also a tiny bit nervous, like when a super smart robot shows up in a story: will it be a great helper, or will it make things confusing for the humans? In simple terms, itâs funny because itâs like telling a kid, âNext year, your robot buddy will be able to do all your math homework in no time!â â youâd grin at how awesome that sounds, but you might also wonder if the teacher will just make the homework harder.
Level 2: AI Taking on Tasks
Letâs break down whatâs going on in this chart and why developers find it funny (and a bit scary). The chart is showing how Large Language Models (LLMs) â that is, huge AI systems like GPT-3 and GPT-4 â have been getting better at handling more complex tasks over time. The x-axis at the bottom is labeled âLLM release dateâ (years 2020 through 2026), so as you move right, youâre looking at newer and more advanced AI models. The y-axis on the left shows âTask duration (for humans)â, going from 0 minutes up to 2 hours. A point on this graph basically says: âBy this date, we think the AI has a 50% chance of succeeding at a task that would take a human this long to do.â In other words, as we get to newer AI releases (toward 2025-2026 on the right), the green and grey dots climb higher on the y-axis â meaning the tasks that AI can tackle are longer/harder. A dashed green curve is drawn through the green dots, showing an overall trend that looks exponential (starting almost flat near 0 minutes, then shooting up towards 2 hours by 2025+). Exponential is just a math-y way of saying âstarts off slow, then skyrockets fast.â Itâs like how each generation of smartphones gets not just a bit better, but a lot better in some capabilities â here itâs AI getting a lot better at doing work for us.
The green dots are labeled with model names: GPT-3, Claude 3.5 Sonnet (Old), o1, Claude 4 Sonnet, o4-mini, Grok 4, and GPT-5. These represent various AI models over the years:
- GPT-3 and GPT-5 are from OpenAI (GPT stands for Generative Pre-trained Transformer). GPT-3 came out in 2020 and was one of the first very powerful language models that could do things like write essays or code snippets. GPT-4 (released 2023) is even more advanced, and GPT-5 is the expected next version in this sequence (not actually released as of 2025, but here itâs imagined or âprojectedâ to exist soon).
- Claude models (Claude 3.5, 3.7, 4 Sonnet, etc.) are from another AI company called Anthropic. Claude is similar to GPT in that itâs also a big language model you can chat with or ask to do tasks. The âSonnetâ might be an internal name or version (the meme uses it presumably to differentiate versions, like Claude 4 âSonnetâ being a particular variant). Realistically, Anthropic had models like Claude 1, Claude 2, etc., but the exact names here are a bit tongue-in-cheek.
- o1, o3, o4-mini are a bit mysterious in name â possibly they stand for other models or open-source efforts. Theyâre shown as green too, so likely notable AI releases. For example, they could hint at something like an open-source model series (just using âoâ as a generic label). The specifics arenât as important; the key idea is there have been multiple players releasing models.
- Grok 4 sounds like a fictional model name (there isnât a well-known AI called Grok 4 as of now). The term âgrokâ means to understand deeply (itâs a slang from a sci-fi novel). So itâs probably a playful way to name a hypothetical AI that really understands things. It suggests another competitorâs model or maybe a future version of something like Googleâs AI.
All those grey dots on the chart represent other AI models (unnamed here) in the same time frame â the grey color likely means ânot the main ones weâre focusing on.â They form a cluster around the green trend line, implying lots of models had various capabilities, but the best ones (the green labeled ones) are defining the frontier. Think of grey dots as the dozens of AI models or versions that came out (some from universities, some from companies like Meta, Google, etc.) over the years, each with their own level of capability.
Now, the labels next to the dots like âFind fact on webâ and âTrain classifierâ are examples of tasks and roughly how long those tasks take a human:
- âFind fact on webâ â maybe a quick Google search that a human does in a minute or two. That label is placed near the very bottom (near 0 on the y-axis, around year 2020). It suggests that an early model (like Claude 3.5 Sonnet (Old) or GPT-3) could handle a very short task like finding a fact online with ~50% success. Indeed, GPT-3 was often used for quick Q&A or retrieving facts it had ingested during training.
- âTrain classifierâ â this is a more involved task, placed around the 1 hour mark. Training a classifier means taking some dataset and teaching a simple AI (a classifier) to distinguish things (like recognizing spam vs non-spam emails). For a human, setting up and running a training job, then evaluating it might take roughly an hour or more (especially if coding from scratch or using a new dataset). The chart positions âTrain classifierâ around early 2024 or so with a model labeled Claude 3.7 Sonnet â implying that by that time, an AI could half the time successfully do the steps to train a classifier on its own. In practical terms, that could mean the AI writing Python code to train a model, running it (maybe in a cloud environment), and getting results without human help.
- There might be other tasks implicitly on the chart even if not explicitly labeled â presumably something at 1.5 hours (maybe âCode reviewâ or âWrite a complex scriptâ). The context tags mention code_review_autopilot, which hints that one of the tasks around the 1.5h region could be an AI doing an entire code review automatically. Another could be writing a significant piece of software or a âbuild_vs_buy_decisionâ analysis (like researching whether to build something or use a library, which can take hours of reading docs â an AI might do that faster soon).
So, what does â50% chance of succeedingâ mean here? It means if you give that task to the AI, thereâs a half-and-half chance it will do it correctly (or well enough) without human intervention. A 50% success rate is far from perfect (itâs basically a coin flip whether youâll get a good result or not), but itâs also not negligible â it means AI is sometimes capable of it. For context, earlier models had 0% chance on an hour-long coding task â theyâd almost certainly fail or produce junk. By the time itâs 50%, we treat that as noteworthy progress. Itâs like saying: âWeâre at the point where an AI is as likely as not to actually finish this fairly big task correctly.â
The meme is poking fun at sprint planning and developer life. Backlog grooming is the process where a dev team looks at all the upcoming tasks (the backlog), estimates them, and decides which ones to do in the next sprint (usually a 1-2 week cycle of work). In backlog grooming, tasks that are multi-hour are common â like a ticket that might take a dev a day or two is broken into sub-tasks of few hours each. The joke here is that if AI can now do, say, a 2-hour task on its own with decent probability, how do we plan around that? Do we assign such tasks to the AI? Do we count them in our teamâs velocity (velocity = how much work a team completes per sprint, typically measured in story points or hours)? The line about ânegotiating with your model about velocityâ imagines a funny scenario: you treat the AI as if itâs a team member. Picture asking ChatGPT, âHey, can you commit to implementing the login feature this sprint? Are you 50% confident youâll finish it?â Itâs absurd, because AIs donât quite work like a person in a team â they donât promise things or give reliable timelines â but teams are kind of already using AI tools to speed up work.
Now, letâs connect that to AI_hype and AI_limitations. Thereâs a lot of hype that AI will massively boost developer productivity â some say by 2x or more. If one sprint you delivered 5 features, maybe with AI youâll deliver 10 in the same time. Thatâs the hype, reflected by this steep chart suggesting ever more capability. The reality is more nuanced: yes, AI can handle certain tasks quickly (like writing boilerplate code, or suggesting how to fix a bug, which might save a dev 30 minutes here, an hour there), but it often still requires a human to oversee and correct it. 50% success rate means the AI might do half the job, and the other half a human needs to either redo or guide. Developers who try these tools learn their quirks â e.g., an AI might write code that looks confident but actually has a security flaw or doesnât handle an edge case. Thatâs an AI limitation: it lacks true understanding of what it doesnât know or when itâs wrong (we call this tendency to âhallucinateâ in AI â producing convincing but incorrect or made-up answers). So we canât just trust it blindly, especially for tasks that really matter if done correctly.
Nevertheless, the trend is undeniable that each new model is tackling bigger challenges. GPT-4, for instance, can often write pretty complex code or analyze a big chunk of text way better than GPT-3 could. By the time GPT-5 rolls around (if it follows the pattern), maybe it can manage multi-hour tasks like writing a whole module of software or doing an in-depth technical research summary. The meme teases that this might happen âwithin the next sprint cycle,â which is an exaggerated way to say âvery soon.â (Usually a sprint cycle is just a couple of weeks, so saying an AI will swallow multi-hour tasks by the next sprint is hyperbole for humorous effect â it really means âimminently, practically tomorrow!â). Developers often use that kind of phrasing jokingly, like âCareful, by next sprint the new framework will replace all our code,â knowing that things donât change that instantly.
Letâs also clarify sprint_capacity_planning: teams have to plan how much work they can get done in a sprint. If an AI can do some tasks, theoretically that increases the teamâs capacity. For example, if an AI assistant can handle writing all the unit tests (something that might have taken a junior dev a full day), the team can plan to do more feature work in that sprint. But planning that is tricky â you have to trust the AIâs output quality and factor in the time for a human to verify the AIâs work. Right now, many teams treat AI as a helper tool, not as an autonomous team member. So itâs more like each developer might complete tasks 10-30% faster using the AI for help on parts of it. Weâre not yet at the point where youâd hand a task entirely to an AI and say âsee you in 2 hours with the finished workâ â though the graph humorously suggests that is exactly what some might hope for with GPT-5.
Finally, the meme draws its âquiet dreadâ vibe from developer_productivity pressure. If the boss sees this chart, they might think, âGreat, we can squeeze in more tasks per sprint because AI has got our back.â Developers are a bit anxious about that because it might lead to unrealistic expectations. Instead of AI giving us a breather or time to focus on really hard problems, it could just mean weâre expected to deliver the same work faster. Itâs like when we got faster computers and better tools â sometimes it just meant the deadline moved up correspondingly. The joke, in simple terms, is that as soon as AI can do something in 2 hours that used to take you 2 hours, your manager might assign you two extra hours of work elsewhere. So the promise of making life easier comes with a catch: the bar for ânormal outputâ might get raised. That feeling is summed up perfectly by the joking question, âIs this 2-hour task that GPT-5 can do by itself in the room with us right now?â Itâs as if someone is looking over their shoulder for the AI thatâs coming to steal their task (or conversely, to help with it). Itâs funny and a bit unnerving at the same time â which is exactly why this meme resonates with developers figuring out how they and AI will collaborate (or compete) on the next big project.
Level 3: Exponential Expectations
From a senior engineerâs perch, this meme hits a nerve: itâs a graph of AI progress that looks equal parts exciting and ominous. The humor comes from seeing our sprint planning realities distilled into a sleek exponential plot. Those green dots (GPT-3, GPT-4, Claude, âoâ models, Grok 4, etc.) creeping up into multi-hour territory mean one thing in manager-speak: âIf AI can handle bigger tasks, letâs raise the targets!â Seasoned devs know that tone all too well. The chart might as well be titled âRecalibrate Your KPIs, Humanâ. We chuckle (and maybe cringe) because weâve sat through meetings where some exec brandishes a trend line of productivity or technology improvement and asks for impossible leaps. AI_hype_vs_reality is a familiar rollercoaster and this meme visualizes the hype: it compresses years of LLM improvements into that steep green dashed line, shooting almost vertically by 2025. Itâs Mooreâs Law vibes, but instead of transistors itâs task complexity. And just like Mooreâs Law, folks in the trenches feel the exponential expectations â every year, deliver twice as much with the new tools, or so weâre told.
The specific combination of elements here is brilliant satire. We have an academic-looking chart (complete with axis labels and confidence jargon â50% chance of succeedingâ) used to convey something every dev has joked about: âWill GPT-5 take my job⌠or at least this annoying two-hour task?â By plotting tasks like âFind fact on webâ near 2020 and âTrain classifierâ around 2023, it reminds experienced devs of how our workflow has already changed. Searching the web for a fact? Thatâs practically instantaneous with modern AI assistants â what used to take us 10 minutes of Googling, an LLM can now summarize in seconds by regurgitating its trained knowledge (with varying accuracy, of course). Writing boilerplate code or a unit test that might take 30 minutes? GitHub Copilot and friends are on it. Heck, weâve gone from Stack Overflow-driven development to AI-assisted development in such a short time. So, seeing â1h 30mâ and â2 hoursâ up there for future models is both a logical next step and a tongue-in-cheek âbrace yourselfâ for devs.
Why is this funny to us? Itâs that quiet dread mixed with amusement. On one hand, who wouldnât want an annoying two-hour grunt-work task automated? Senior devs certainly arenât sentimental about, say, trawling through logs for an hour or converting mockup to HTML again. But on the other hand, we know the trade-off: if AI handles the easy half of a task, management might assume the other half magically vanishes too. The meme caption nails this feeling with a joke:
âis this 2 hour task that gpt5 can do by itself in the room with us right now?â
It riffs on the classic therapy quip (âIs the ghost that haunts you in the room with us now?â), implying weâre almost paranoid about an invisible GPT-5 lurking around ready to snatch up work. Itâs funny because some of us have caught ourselves thinking, âCould I just let ChatGPT handle this while I grab coffee?â â a mix of hope and nervousness that maybe it could, and what that would mean.
IndustryTrends_Hype element: Weâve all seen charts like this at tech conferences and in glossy reports, predicting AI will do X by year Y. Theyâre usually overly optimistic. A senior dev reading â50% chance of succeedingâ also recalls that 50% success for an automated system is basically a coin flip â not something youâd bet the production server on. Itâs a sly nod: yes, GPT-5 might attempt a two-hour coding task, but thereâs a decent chance it fails hilariously (or introduces a subtle bug that takes four hours to debug later). So thereâs an AI_limitations undercurrent here. We wink at each other knowing that â50% successâ means youâll still be babysitting that AI. Itâs like working with a super-speedy but moody junior developer: they might crank out a feature by lunchtime or produce gibberish that you must untangle. Real-world early adopters of AI coding tools have seen both outcomes.
The meme also touches on developer_productivity and how it might be measured in the AI era. The line âbacklog grooming might soon include negotiating with your model about velocityâ is a fantastic image. Imagine sprint planning with an AI: Product Manager: âCan we commit to these 5 user stories?â Lead Dev: âOur AI helper promises it can handle two of them â it swears it has a 50% chance on the complex one!â Itâs absurd, yet some teams are already experimenting with assigning certain tasks to AI assistants. Code review, for instance â what the meme tags call code_review_autopilot â is becoming a thing (AI suggests fixes or spots bugs in pull requests). But seniors know the caveats: the AI might catch the low-hanging fruit and miss the deeper issues that a human would notice from experience. So you canât (yet) just let it fully autopilot. Still, the sprint_capacity_planning conversations are changing. Instead of a flat âwe have 5 engineers, 2 weeks, thatâs X story points of capacity,â itâs now âwe have 5 engineers + maybe GPT-4 pair-programming, so maybe X+? story points.â That build_vs_buy_decision now sometimes includes a third option: âbuild, buy, or promptâ. Do we code a feature from scratch, purchase a tool, or ask the AI to generate a solution on the fly? A senior dev knows the trap here â that third option can be quick to start but costly to finish if the AI output is misleading.
Historically, every big hype (from cloud, to DevOps, to low-code platforms) came with promises of âfreeing us for more important work.â In practice, it often means the goalposts move. This memeâs vibe of âproductivity KPIs being recalibrated yet againâ is so on point. Veterans will recall how adoption of cloud meant the team was expected to maintain more systems since âinfrastructure is easy now,â or how adding automation test suites meant testers were expected to handle more projects because âthe tests do the work.â The same is happening with AI: if GPT-5 can handle a two-hour task, maybe your manager decides that task is now worth zero points and doesnât allocate human timeâuntil it fails, of course, then itâs all hands on deck to fix it Friday at 5 PM. Itâs the classic hype vs reality cycle: initial productivity boost, followed by a reality check (and sometimes a late-night bug hunt).
In essence, the meme is âtoo realâ because it jokingly predicts a near future many of us are already negotiating in small ways. It captures both the marvel of these LLMsâ rapid progress and the side-eye skepticism of those who will actually integrate this progress into a team workflow. We laugh, perhaps a bit nervously, because the graph suggests our role is changing fast. And as any senior dev knows, when something seems to be moving exponentially, youâd better buckle up â because somewhere on that curve, things get messy before (if ever) the promise is fully realized. For now, weâll keep joking about that ghost in the backlog (GPT-5, weâre looking at you) while carefully reviewing the pull requests it claims it can handle. After all, it might only have a 50/50 shot⌠but our name is on the commit history if it merges broken code into main!
Level 4: Emergent Complexity Frontier
Behind this tongue-in-cheek chart lies a serious AI trend: each new generation of Large Language Model pushes the frontier of tasks they can handle, almost like a computational Mooreâs Law for cognitive capability. The dashed green curve hugging an exponential trajectory isnât arbitrary hype â it echoes real observations from research on scaling laws. In formal terms, as we double or triple model parameters and training data, we often see non-linear performance jumps on complex tasks. This means an LLM like GPT-5 (presumably larger and more finely tuned than GPT-4) might display emergent abilities that GPT-3 or earlier simply couldnât tackle. For example, GPT-3 could manage trivial queries or generate paragraphs of code with guidance, but GPT-4 surprised many by handling nuanced multi-step reasoning and even passing professional exams. These are emergent properties â skills that werenât explicitly programmed but âsurfaceâ once the modelâs scale crosses a threshold. The chartâs green dots for GPT-3, Claude, o1, Grok 4, etc., are essentially markers on this frontier of emergent complexity. They suggest that by 2025-2026, LLMs might reliably attempt tasks taking up to ~2 hours of human effort.
Consider the task âTrain classifier,â annotated around the one-hour mark. Thatâs a multi-step project for a human: writing a training script, running it on data, tweaking parameters, evaluating results. For an AI, this involves not just regurgitating facts but performing coordinated reasoning and tool use. How could an LLM succeed here? One theoretical avenue is integrating the LLM with external tools or having it generate code which, when executed, trains the model. In AI research, this falls under AI planning and tool orchestration. Early signs are already here: there are frameworks where an LLM calls APIs or writes and runs code in a loop (think of experimental projects like AutoGPT and others). To have a 50% success rate on a one-hour coding task, GPT-5 might internally simulate those hours of work in minutes: it could output a complete Python script for training a classifier, self-check the code, maybe even debug errors by reasoning through stack traces â all within its extended context window. Speaking of context, note that one limiter for LLM âthinkingâ time is context length (how much the model can read & remember at once). Each new generation often has a larger context window (GPT-4 can handle ~32K tokens â 50 pages of text). If GPT-5 extends this further or employs a form of long-term memory, it could ingest more information and plan longer tasks without forgetting earlier steps, essentially stretching its effective attention span closer to hours.
Yet, even an exponential trend faces physics and scaling limits. Thereâs an implicit half-joke in â50% chance of succeedingâ â it hints at the stochastic nature of these models. Unlike a deterministic program, an LLM generates outputs probabilistically, so complex tasks arenât guaranteed successes each run. In theoretical computer science terms, you might say weâre pushing up against tasks that approach the AIâs current âcompetency ceiling.â Achieving >50% reliability on 2-hour tasks might need qualitative model improvements, not just more FLOPs: think better problem decomposition (breaking big tasks into smaller subtasks, a known hard problem called AI planning), or model architectures beyond todayâs Transformers to overcome issues like long-term dependency and context fragmentation. Thereâs active research into making AI reasoning more robust â from incorporating symbolic logic, to neural executive functions that plan and verify steps (almost like how a developer writes pseudocode to outline a solution). In short, the memeâs chart exaggerates with humor, but it reflects a real technical frontier: the race to expand an AIâs effective working duration on a problem, approaching human-scale continuous work (multi-hour cognitive tasks). Itâs a dizzying exponential model scaling curve: if progress continues unabated, tasks that once took an afternoon of human concentration might soon be solvable by letting a supercharged LLM churn away through lunch. The real question is, will these models just get faster at predicting text, or will they evolve new internal mechanisms to truly âthinkâ for two hours straight? The answer determines if the trendline keeps shooting up or finally plateaus when raw scale alone no longer cuts it.
Description
A scatter plot titled 'Task duration (for humans) where we predict the AI has a 50% chance of succeeding' with the Y-axis showing time from 0 to 2+ hours and X-axis showing LLM release dates from 2020 to 2026. Green dots represent specific models: GPT-3 (near 0), Claude 3.5 Sonnet Old (~15min), o1 (~35min), Claude 3.7 Sonnet (~50min), Claude 4 Sonnet (~1hr), o4-mini (~1h15m), o3 (~1h30m), Grok 4 (~1h45m), and GPT-5 (~2h15m). Gray dots with error bars show additional data points. A dashed green trendline shows exponential growth. Reference tasks on the Y-axis include 'Find fact on web' and 'Train classifier'
Comments
26Comment deleted
At this rate, by 2028 AI agents will be able to handle tasks that take a human 8 hours - which means they'll finally be ready to attend a full day of meetings and still get nothing done
The scary part isn't that GPT-5 can do a 2-hour task; it's that it will probably spend the first hour arguing about the requirements in a Slack channel, just like a human
If this curve holds, GPT-6 will finish your code review, draft the retro notes, and file a Jira ticket reminding you that humans are now the blocking dependency
Ah yes, the classic exponential curve where we're always just 2 years away from AGI - reminds me of how we've been 5 years away from fully autonomous vehicles since 2015. At least this chart is honest about the 50% success rate, unlike my junior devs claiming their code is 'production ready'
Ah yes, the classic 'AI will replace us by 2025' chart - where every new model release pushes the goalposts from 'find fact on web' to 'replicate two hours of human cognitive labor.' Notice how we've gone from GPT-3 barely managing web searches to GPT-5 allegedly matching a junior developer's afternoon sprint. The real joke? By the time GPT-5 actually ships, we'll have redefined 'human-level performance' to mean 'can survive a 4-hour architecture review meeting without suggesting we rewrite everything in Rust.' The dashed green line of exponential progress conveniently ignores that humans also get coffee breaks, context switching overhead, and existential dread - none of which scale logarithmically
By 2026 GPT-5 handles two-hour tasks at 50% success; the production pattern is idempotent retries with exponential backoff - until Finance pages us
Great, we finally automated two hours of work with a 50% SLO - marketing calls it AGI, SRE calls it a postmortem
Inference latency's true scaling law: O(parameters Ă existential_doubt)
Houston, the error bars have entered the space!!! Comment deleted
the task in question Comment deleted
Generic answer more or less complicated Linux question Comment deleted
"50% chance of succeeding" OpenAI calculations is: it's always 50% chance. It succeeds or it doesn't. Comment deleted
Bro they cant even scale graphs consistently Comment deleted
Oh I forgot đ Comment deleted
Also OpenAI dudes too dumb to use logarithmic scale Comment deleted
I think they chose to make it linear on purpose so it looks better. Comment deleted
where? I see its quite linear here Comment deleted
ah I get it. they could have used it, but they didn't. I think it's because most people can't properly comprehend log scale Comment deleted
And if you read the error bars, you will find the average LLM error rate Comment deleted
"Find fact on web" , 10 minute task. Ah yes, the contractor billing by time approach Comment deleted
That's what my bosses think, especially on facts that nobody would publish on the web, at least for free. 𼲠Comment deleted
Procrastinating for two hours -- achievement unlocked Comment deleted
Its totally wrong comparing old models, that try to give answer instantly and modern agentic workflow, that consuming a lot of time rechecking everything. apply modern workflow to older models and their results will be much higher Comment deleted
Error bars used to mean something Comment deleted
That 50% success rate is doing a lot of heavy lifting here Comment deleted
An example of task that takes 2h for a human: "Exploit a buffer-overflow in libiec61850" Source: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ Comment deleted