Key takeaways
- In a 2025 METR study, 16 experienced developers on 246 real tasks were 19% slower with AI while believing they were about 20% faster, and had expected a 24% speedup going in.
- The durable lesson is the perception gap, not the number: smart people are confidently wrong about their own speed, so your gut is a bad instrument for judging whether AI saves you time.
- In a February 2026 update, METR said their newer data was an unreliable signal and that developers are likely more sped up by AI now than in early 2025, so the original result is a snapshot, not a permanent verdict.
- Only measure speed on tasks you can already do well by hand. Where AI lets you do something you otherwise couldn't (coding, design, video), it wins by default and speed is beside the point.
- Run a one-week test: pick one repeatable task, do it some times with AI and some times by hand, time the whole job to truly finished, and rate the quality 1 to 5.
- Hand the raw logs to Claude and ask it to act as a skeptical analyst using the median, not the average, and to refuse to flatter you.
- Eight to twelve rows is enough to overrule a feeling, not enough to prove a number, and losing that false certainty is the whole prize.
The Solo Creators AI Studio
Get this skill, and every one I build.
The whole vaultA new drop monthlyBuild live with meYours to keep, forever
See inside the Studio
30-day money-back guarantee. Cancel anytime.
Questions people ask
Did AI really make experienced developers slower?
In METR's 2025 study of 16 experienced developers across 246 real tasks, yes: they took about 19% longer with AI, even though they had expected a 24% speedup and still believed afterward that AI had made them roughly 20% faster. The striking part is that gap between what they felt and what actually happened.
Doesn't that prove AI is a waste of time?
No, and it's important to cite the whole story. In a February 2026 update, METR said their newer data gave an unreliable signal and that developers are likely more sped up by AI now than in early 2025. Treat the original result as a snapshot of one moment. The point that survives is that people are poor judges of their own speed. I build everything I sell with AI and couldn't do most of it otherwise.
Does this study apply to coaches, consultants, and other non-developers?
Not directly. It was a study of developers writing open-source code, so don't apply the speed number to your work. What does carry over is the human failure mode underneath it: experienced people were confidently wrong about their own pace on familiar work. That can happen to anyone in any field, which is why measuring beats guessing.
How do I actually measure whether AI saves me time?
Pick one task you already do well by hand, then run it in two lanes for a week: some times with AI, some times the old way, timing each one to truly finished and rating the result 1 to 5. The full step-by-step protocol, the blank log template, and the analysis prompt are in the one-week AI time test section above.
What if AI is slower but the result is better?
Then it can still be worth keeping, and that's exactly why the log has a quality column sitting next to the minutes. Decide what the task is actually for. On a client proposal, a few extra minutes for a noticeably better draft is a good trade. On routine follow-up emails that were already fine, slower and prettier is a loss. Logging both numbers lets you make that call on purpose instead of by feel.
Why should I use the median instead of the average when reviewing my times?
Because with only a handful of tasks, one disastrous session (a bad re-prompt spiral, an interruption) can drag an average way off and mislead you. The median, the middle value, is far more stable on small samples, which is why the copy-paste prompt in this article tells Claude to use it.
What if the results are different for different tasks?
That's the most likely and most useful outcome. You'll probably find AI clearly wins on some tasks, loses on others, and ties on the rest. Keep it where it earns its keep, stop forcing it where it doesn't, and re-run the test every few months since tools and your own skill both change over time.
Written by Ronnie Nijmeh
Ronnie spent 18 years building a SaaS with a team of 20 that served over 650,000 customers, generated over $14M in sales, and sent over 550M emails. Now he's solo, solving real business bottlenecks and turning them into working AI skills, workflows, and automations. He teaches all of it, with direct access to him, inside the Solo Creators AI Studio Skool community. See what he's built →
“I completely rebuilt my site using the Website Launch Builder. If you've ever built a website, you'll really appreciate this skill. I'm blessed to be a member.”
Michael HattawayFounder, Iron Strengthens IronKeep reading
YOUR EDGE
Keep the Thinking That Makes You Worth Hiring
The honest version of the MIT brain study, plus the exact line between the work you should automate and the judgment you should never hand off.
PRICING
Get Paid for the Result, Not the Hours AI Just Saved You
The honest arithmetic of pricing your work now that AI does the slow part in minutes, and how to charge for the result instead.
CLIENT-READY AI
Send Work That Lands: Why AI Output Becomes Someone Else's Homework
Polished AI output that reads done and isn't quietly hands your client the real work. Learn the tell, and the fix, so your deliverables land the first time.