AI felt faster. The stopwatch disagreed.

It is possible to enjoy a tool, want to keep it and still be wrong about how much time it saves.

That distinction makes the debate about AI productivity more interesting than a contest between enthusiastic users and skeptical observers. They may be describing different things: the experience of working, the effort required, or the amount of finished work delivered.

A research result can help separate those questions. It cannot answer all of them at once.

The result that made the argument uncomfortable

In its July 10, 2025 randomized study, METR followed 16 experienced open-source developers across 246 tasks in repositories they knew. Allowing early-2025 AI tools increased completion time by 19%. Afterward, participants estimated that AI had made them 20% faster. The result concerns that setting, not all developers or subsequent tools.

The striking part is the disagreement between experience and measurement. A favorable impression is valuable evidence about preference. It is not automatically a measurement of duration.

That does not make the people involved foolish. An interface can make one part of work feel easier while a different part consumes the time. If the question is whether a process became faster, the attractive moment—the appearance of a draft, for example—is not enough to settle it.

The follow-up matters as much as the headline

In its February 24, 2026 update, METR said the newer experiment could not reliably estimate the current productivity effect. Participation and task-selection biases, changed compensation and difficulty measuring concurrent agent work complicated the comparison. The researchers believed gains had likely improved, but did not consider the data strong evidence of their magnitude.

The defensible reading is therefore not “AI slows developers down forever.” Nor is it “the old result has been canceled, so every productivity claim is now credible.”

A bounded finding can remain valid for its original conditions while becoming a poor answer to a broader, later question. That is a reason to identify the conditions of a claim—not to discard measurement whenever it becomes inconvenient.

Three questions can hide inside “Does AI save time?”

Here is how I would separate them when evaluating a project.

The first question is whether comparable work reaches the same acceptable result with less human effort. That requires a definition of completion. A drafted answer and an answer checked against the customer’s actual situation are not interchangeable endpoints.

The second is whether the work reaches the recipient sooner. Human effort and elapsed time can move differently. A process may require fewer working minutes yet still wait overnight for approval. This is a conceptual distinction, not a finding attributed to METR.

The third is whether the team can now do something it otherwise would not have attempted. A prototype that becomes feasible is not well described as “three minutes saved on the old task,” because there may be no completed old task to compare it with.

Those questions can all matter. Combining them into one percentage makes the percentage difficult to interpret.

Count the work after the draft appears

Consider a hypothetical service request. Without AI, preparing, checking and sending an acceptable response takes 12 minutes. With AI, the draft takes 1 minute, verification takes 9 and correction and sending take 3. The complete task takes 13 minutes.

This arithmetic is an illustration, not a METR finding or a client result. I would investigate the nine minutes of verification before changing the writing model. The person may be searching several applications for information the draft needs. Measure comparable completed requests after addressing that obstacle, keeping unsuccessful attempts in the record. Track the customer’s waiting time separately.

Freed hours still need a useful destination

In a different hypothetical case, 200 requests each require 3 fewer minutes of human effort at the same acceptable quality. That frees 10 hours a month. Subtract 2 hours of recurring maintenance and 8 hours remain. Setup time and subscription or usage costs still belong in the overall comparison; this is not a full financial return calculation.

Those eight hours represent capacity. They become a reduction in spending only if an expense actually falls. If the same team uses them to clear a backlog or improve follow-up, examine those outcomes instead. Fragmented minutes may also be harder to reuse than a continuous block of time.

I would choose the intended use before presenting the project as a saving. For a backlog, record completed cases and their age. For better service, define the deadline and quality expected. A useful benefit should remain visible in the work, even when payroll is unchanged.

The useful experiment starts with a decision

For a proposed internal assistant, I would choose the decision the trial needs to support before choosing the headline metric.

If the decision concerns staffing a queue, I would examine completed, acceptable cases and the human effort they require. If it concerns reducing waiting, I would follow the work through to the recipient. If it concerns opening a new capability, I would specify what must become possible and what resources the attempt can consume.

I would also record unsuccessful attempts rather than retain only the outputs worth showing. Otherwise, a polished demonstration and an operational estimate become easy to confuse.

This is not an argument for turning every small experiment into a research project. It is a way to make a small experiment answer a real question.

The conclusion I take from this comparison is not a verdict for or against AI. A productivity claim needs a unit of finished work, a comparison and a date. Without those, a persuasive number can describe almost anything.

That leaves plenty of room for ambition. It also makes it easier to recognize a valuable result when it appears, whether that result is less effort, less waiting or work that has only just become possible.

For a different angle on the same problem, consider whether the work itself should be redesigned before it is automated.