The AI coding productivity data keeps contradicting itself. Here's why.

Every few weeks, a new study lands on AI coding assistants. The AI coding productivity numbers swing between a 26% boost and a 19% slowdown. If you're an engineering manager deciding whether to expand your team's AI tooling budget, add new tools, or question whether the current ones are paying off, neither number tells you what to do. That's because they're measuring different things entirely.

They're not both wrong. The contradiction isn't noise in the data. It's a clue about where AI tools actually work, and the difference between knowing which is which is the difference between a tooling decision and a guess.

The numbers that keep going in opposite directions

The pro-AI productivity evidence is easy to find and largely consistent. GitHub's own research on Copilot reported developers completing isolated tasks 55% faster. McKinsey and Boston Consulting Group studies put productivity gains in the 20-45% range for various software tasks. Index.dev surveys find developers saving an average of 3.6 hours per week. Every major vendor has their own version of this number, and they broadly agree on direction if not magnitude.

Then there's the METR study, published in July 2025. METR recruited experienced open-source contributors (median 5+ years of experience, regular AI tool users in their own work), and ran them through 246 real GitHub issues from actual open-source repositories. Half got access to AI tools (Cursor with Claude and GPT-4 available), half didn't. When they measured wall-clock time, the AI group took 19% longer to complete the same tasks.

Crucially, the developers in the AI group reported feeling more productive than the control group. Perceived AI coding productivity and measured productivity moved in opposite directions. That perception mismatch is its own finding. It has direct implications for survey-based research, which is what most vendor studies rely on.

Why the unit of measurement is the problem

Most productivity studies that find large gains from AI tools use some variant of: here's a clearly specified function to write, here's a clean file, here's an explicit acceptance criterion. This is the best-case scenario for AI code generation. The problem is well-bounded, the context is minimal, and the output is easy to evaluate. Of course AI tools help here.

Real software work looks different. It involves reading an ambiguously written ticket, tracing through a codebase you only partially understand, proposing a change that fits an existing architecture someone else designed, writing code that will get reviewed by someone with different context than you, iterating on that review across two or three rounds, and eventually shipping something that doesn't break the twelve things that currently work. None of that looks like 'write a function to spec in a clean file.'

The studies that find the biggest AI coding productivity gains tend to measure the first type of work. The METR study measured the second. That's not a flaw in either study. It's the explanation for why their results point in opposite directions. The question isn't which study is right. The question is which type of work fills most of your team's week.

Where AI coding productivity gains are real

Strip away the contradictions and the evidence is actually consistent about the task types where AI coding tools deliver reproducible gains:

• Autocomplete on well-understood patterns: boilerplate, repetitive structure, familiar idioms in a language the developer already knows.
• Writing unit tests for code that's already been written and understood by the developer.
• Format conversions, data transformations, and one-off SQL queries against a schema the developer has described.
• Documentation for well-typed, clearly structured functions where the types are explicit.
• Working in unfamiliar syntax: a Python developer writing a small Bash script, a backend engineer touching CSS for the first time in years, a frontend developer writing a database migration.

And the task types where the evidence is neutral to negative:

• Complex multi-file refactors where the change must respect dozens of implicit architectural decisions spread across the codebase.
• Diagnosing performance regressions in production systems where the symptom and cause are separated by several layers.
• Reasoning about concurrency, distributed state, or subtle ordering constraints where correctness depends on understanding the full system.
• Writing code that depends on your team's specific conventions, historical context, and undocumented architectural decisions.
• Reviewing AI-generated code: junior developers specifically report this as more effortful than writing from scratch, because evaluating plausible-but-wrong output is harder than writing it yourself.

The METR study in detail

The METR result deserves more attention than it's received, because it's methodologically the most rigorous study done to date on this question. 246 tasks drawn from real GitHub issue trackers. Real open-source codebases, not toy repositories constructed for research purposes. Developers with genuine experience who already use AI tools in their own work. Random assignment to AI-available and AI-unavailable conditions, with controlled access to Cursor and the underlying models.

The 19% slowdown isn't a result of developers being bad at using AI tools. It's a result of what the tasks actually required: understanding the issue in enough depth to reproduce it, navigating a large unfamiliar codebase to find the relevant code, reasoning about the right fix given constraints the codebase imposes, writing code that integrates cleanly, and making sure tests pass in a test suite they didn't write. At each of these steps, AI suggestions require evaluation. Some are useful. Many are not. Evaluating a wrong suggestion, recognising it's wrong, and re-prompting takes time. That time stacks on top of the ordinary cognitive work of software development.

What the study doesn't prove is that AI tools are net-negative across all work. The tasks it selected are representative of one end of the complexity spectrum. For the other end: boilerplate, format conversions, and unit test scaffolding. The gains from other studies hold. What METR adds is a constraint on how far you can generalise from those studies.

The senior/junior asymmetry that hiring decisions are missing

Buried in most AI coding productivity research is a split that matters more for hiring decisions than for tool purchasing decisions. Senior engineers and junior engineers don't get the same thing from AI tools, and the direction of that asymmetry has real career consequences.

The reason is straightforward. A senior engineer using an AI coding assistant gets a first draft of code they already know how to evaluate. They can spot what's structurally wrong in ten seconds, modify it precisely, and move on. Their pattern recognition is the limiting factor in their work — AI gives them more patterns to work with, faster. A junior engineer using the same tool gets a first draft they may not be able to fully evaluate. They can check whether the tests pass. They can't always tell whether the code handles the edge case three callers downstream, whether it will hold up under concurrent access, or whether it fits the architecture their team has been building toward for two years.

The data on career-level outcomes is pointed. A 2025 LeadDev survey found 54% of engineering leaders planned to hire fewer junior engineers, citing AI tools as part of their rationale. Stack Overflow research from late 2025 found 61% of junior developers described the current market as challenging, versus 34% of seniors. The junior roles most at risk are those that consisted of well-scoped implementation tasks, exactly the category where AI tools are most effective and most substitutable.

This isn't an argument against AI tools. It's an argument for clarity about what junior engineers are actually getting from AI-assisted work. Writing more code is not the same as building the understanding needed to debug it later, review it under pressure, or make architectural decisions about it in two years. Teams that measure only output volume while ignoring whether junior engineers are developing genuine understanding are making a slow-moving mistake.

What to actually measure in your own team

If you want to know whether AI tools are helping your specific team on your specific work, not GitHub's internal benchmark or a controlled study with open-source repositories, these are the metrics that tell you something real:

1. Time from ticket creation to production deploy, on features that touch three or more files. This is the closest proxy for real product work in most teams, because it captures the navigation, review, and integration work that AI tools don't consistently help with.
2. PR review cycle time. AI-generated code is often syntactically correct and structurally plausible but wrong in subtle ways. If review is getting slower as AI usage increases: more rounds, more comments, more back-and-forth. That's a meaningful signal.
3. Bug rate in AI-assisted PRs versus non-AI-assisted PRs, measured at 30 days post-deploy. This requires tagging PRs by AI involvement, which adds friction, but a single quarter of data is enough to be directional.
4. A weekly developer question: 'I understand the code I shipped today.' One item, five-point scale, ten seconds to answer. The aggregate trend over a few months is more informative than any velocity metric.

What not to measure: lines of code per developer per day, AI acceptance rate in your IDE, percentage of committed code that was AI-generated. These metrics flatter the tools without telling you whether your team is shipping better software, faster, with fewer defects.

The AI coding productivity question isn't going to resolve cleanly across the industry. The tools keep improving, the tasks teams use them for keep shifting, and the research will always lag both. What changes when you run your own numbers on your own codebase is that the benefits land exactly where you'd expect — and so do the gaps.