Productivity claims in AI-assisted software development

What has been measured, and in whom

Product Research. Independent desk research. Not peer reviewed.

Abstract

Two figures circulate in discussions of AI-assisted development: a controlled trial reporting that developers using an AI assistant completed a task around 56 percent faster, and a 2025 study reporting that experienced developers working in their own repositories were around 19 percent slower. Both are real, both are competently conducted, and they are not in conflict.

This note reviews the available studies and argues that the apparent contradiction dissolves once the outcome variables are separated. Five distinct constructs appear in the corpus, quoted interchangeably: task completion time, task throughput, perceived productivity, output quality including security, and maintainability of the resulting code. Three moderators explain most of the variance between studies: task novelty, codebase familiarity, and whether the task falls inside the model's competence.

The most consistent finding across the corpus is not an effect size. It is that developers systematically misjudge whether the tools made them faster, in both directions.

Method: independent desk research, meaning a structured review of publicly available studies and reports. No measurements were taken.

Research question

Across the available empirical studies of AI-assisted software development, what outcome variable was measured, on what population and task type, and what do the results support once those differences are made explicit?

A second question follows: which decision-relevant quantity is currently unmeasured?

Background

Adoption decisions about these tools are being made at organisational scale, and the numbers used to justify them are drawn from studies with sharply different designs. The problem is not that the studies disagree. It is that a percentage from a randomised trial on an isolated greenfield task is quoted as though it estimated the same quantity as a field experiment counting merged pull requests, or as a survey asking developers how productive they felt.

Software productivity measurement has a long history of exactly this failure. Lines of code, function points and story points have each been used as proxies and each has been shown to be gameable and weakly related to delivered value. The current literature inherits the problem rather than solving it, and it inherits it at a moment when the answer is being used to make hiring and tooling decisions.

Methodology

Independent desk research: a structured review of publicly available studies, working papers and industry reports.

Studies were included if they reported a quantitative comparison between developers using and not using an AI coding assistant, on a defined task, with the design documented. Industry reports without peer review are included and are labelled as such, because they are the only source addressing one of the constructs.

Three limits on the method are stated here rather than deferred. No systematic search protocol was used, so the corpus is not exhaustive. Several of the largest studies are authored by researchers affiliated with the vendor of the tool studied, which is declared in the evidence section rather than treated as disqualifying. And no original measurement was conducted. Where the author's build experience appears, it is declared as practitioner observation from a single-operator sample.

Evidence

Peng, Kalliamvakou, Cihon and Demirer, 2023, "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot". A randomised controlled trial with 95 recruited developers. Task: implement an HTTP server in JavaScript. The treatment group completed it approximately 55.8 percent faster.

Design characteristics that bound the result: a single, self-contained, greenfield task with a well-known shape; no existing codebase to understand; recruited participants; and time-to-completion as the outcome. Several authors were affiliated with GitHub or Microsoft, which is disclosed in the paper.

METR, 2025, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity". Sixteen experienced open-source developers, 246 real tasks drawn from their own large repositories, randomised at the task level to allow or disallow AI tooling. Result: task completion took approximately 19 percent longer with AI allowed.

The finding that matters more than the effect size: participants forecast a 24 percent speedup beforehand, and after completing the work still believed they had been roughly 20 percent faster. Their estimate was wrong in sign.

Bounds on the result: small sample of developers, though a substantial number of tasks; highly experienced participants working in codebases they knew deeply; and a period during which the tooling was changing.

Cui, Demirer, Jaffe, Musolff, Peng and Salz, 2024, "The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers". Field experiments across several thousand developers at Microsoft, Accenture and an anonymised firm. Reported approximately a 26 percent increase in completed tasks, measured largely by pull requests, with larger effects for less experienced developers.

This is the largest study in the corpus and its outcome variable is throughput rather than time. It also reports the clearest moderator: the gain is concentrated among juniors.

Ziegler et al., "Measuring GitHub Copilot's Impact on Productivity" (Communications of the ACM, 2024). Survey of roughly two thousand developers combined with usage telemetry. The finding is that suggestion acceptance rate was the telemetry measure most strongly associated with self-reported productivity.

This study is frequently cited as evidence that the tool improves productivity. It measures the correlation between usage and perceived productivity, which is a different claim, and the authors are affiliated with GitHub.

Vaithilingam, Zhang and Glassman, 2022, "Expectation vs. Experience" (CHI Late-Breaking Work). Twenty-four participants. No statistically significant improvement in task completion time, with some tasks taking longer, and yet most participants preferred working with the tool.

Dell'Acqua et al., 2023, "Navigating the Jagged Technological Frontier" (Harvard Business School working paper). 758 consultants at Boston Consulting Group, not software developers, so the domain transfer is an assumption. On tasks inside the model's competence, AI users performed substantially better on quality measures. On a task designed to fall outside it, AI users were markedly less likely to reach the correct answer than the control group.

The contribution is the concept rather than the numbers: capability is jagged rather than graded, the boundary is not visible from inside the task, and crossing it inverts the effect.

Perry, Srivastava, Kumar and Boneh, 2023, "Do Users Write More Insecure Code with AI Assistants?" (ACM CCS). Forty-seven participants across security-relevant programming tasks. Participants with access to an AI assistant produced less secure solutions on most tasks, and were more likely to believe their solutions were secure.

The perception gap appears again, now attached to a different construct.

Sandoval et al., 2023, "Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants" (USENIX Security). Included deliberately as contradicting evidence. In a low-level C context, the study did not find that assistance increased severe security bugs at the rate that had been anticipated.

Taken with Perry and colleagues, the honest reading is that the security effect is task and language dependent and not settled.

GitClear, 2024 and 2025, code quality reports. Analysis of a large corpus of changed lines, reporting rising code duplication and a decline in refactored or moved code over the period of assistant adoption.

This is industry research, not peer reviewed, with a proprietary methodology that has not been independently replicated, and the observed trend is confounded by everything else that changed in the same years. It is included because it is the only source addressing maintainability at scale, and it should be read as a hypothesis rather than a finding.

Imai, 2022, "Is GitHub Copilot a Substitute for Human Pair-Programming?" A small study reporting that assistance produced more code faster, with a higher proportion subsequently removed, while human pair programming produced higher-quality output.

Analysis

Five constructs, one number

The corpus measures five different things, and public discussion collapses them.

Time to complete an isolated task. Peng et al., Vaithilingam et al.
Task throughput over weeks. Cui et al.
Perceived productivity. Ziegler et al., and the self-estimates inside METR and Vaithilingam.
Output quality, including security. Perry et al., Sandoval et al., Imai.
Maintainability of the resulting code. GitClear only, weakly.

A team quoting a figure should be able to say which of the five it refers to. Most cannot, and the two most-quoted figures refer to constructs one and one respectively, measured under conditions that share almost nothing.

Three moderators account for the apparent contradiction

Task novelty. Peng's task was greenfield and canonical. METR's tasks were incremental changes inside mature systems. The first is close to a pure generation problem. The second is dominated by comprehension, constraint-finding and integration, and generation is a small share of the work.

Codebase familiarity. METR's participants were expert in their own repositories. Their tacit knowledge of the code was an asset that the assistant could not access and, on the evidence, partly displaced. Peng's participants had no codebase to be familiar with.

Position relative to the competence frontier. Dell'Acqua and colleagues give the mechanism. Inside the frontier, assistance helps substantially. Outside it, assistance actively harms, because it supplies plausible output that the user is not positioned to reject. The frontier is invisible from inside the task, which is what makes the effect dangerous rather than merely variable.

Add the experience moderator from Cui and colleagues, where juniors gain most, and a coherent picture emerges. Assistance substitutes for knowledge the developer lacks and competes with knowledge the developer has.

The most robust finding is about perception, not speed

Across four separate studies with different designs, developers' beliefs about their own performance diverged from measurement. METR's participants were wrong about the sign. Vaithilingam's preferred the slower condition. Perry's believed insecure code was secure. Ziegler's association is between usage and felt productivity.

This is more consistent than any effect size in the corpus, and it has a direct organisational consequence: internal developer surveys and satisfaction metrics are not evidence about productivity, and a team that adopts on the strength of them has measured enthusiasm.

The reason is not obscure. The tools remove the subjectively effortful part of the work, which is producing the first draft, and add effort in review and integration, which is less salient. Perceived effort falls while total time may not.

The decision-relevant quantity is unmeasured

Every study measures a cost incurred during authoring. None measures the cost incurred afterwards.

What a team actually needs to know is the total cost of ownership of the code produced: review time, defect rate over the following two quarters, incident rate, time to modify it a year later, and the onboarding cost for a developer who did not write it. GitClear gestures at part of this with a method that cannot support the inference.

This is an argument from the absence of studies, which is weaker than a positive finding, and it is stated as such. But the absence is real, and it is the reason the literature cannot currently answer the question organisations are asking it.

Vendor affiliation should be declared and not weaponised

Two of the most cited studies have authors affiliated with the vendor of the tool being evaluated. This is normal in industrial research, the affiliations are disclosed, and the designs are not obviously compromised.

It is still a fact a reader needs, particularly given that the independent study with real tasks in real codebases found the opposite sign. Weighting the corpus without noting the pattern would be a misreading of it.

Findings

Five different outcome constructs are measured in this literature and quoted as one. Any productivity claim should name which one.
The apparent contradiction between the largest positive and the most negative results is explained by task novelty, codebase familiarity and position relative to the model's competence, not by methodological error in either.
Gains concentrate where the developer lacks knowledge: greenfield tasks, unfamiliar territory, junior developers. Losses concentrate where the developer holds deep tacit knowledge of a mature system.
Capability is jagged rather than graded, and the boundary is not visible from inside the task. Crossing it reverses the effect rather than diminishing it.
The single most consistent result across the corpus is that developers misjudge their own performance with these tools, sometimes in sign. Self-report and satisfaction metrics are therefore not evidence of productivity.
Security effects are task and language dependent, and the published evidence is genuinely mixed. The one consistent element is that users overestimate the security of what they produced.
Downstream cost of ownership is essentially unmeasured, and it is the quantity most relevant to an adoption decision.

Limitations

This is not a systematic review. No pre-registered search, no inclusion protocol, no assessment of study quality against a standard instrument. Papers were selected because they are widely cited or because they contradict the popular reading, and the second criterion introduces a bias toward negative results that a systematic review would correct.

The corpus is small and heterogeneous. Seven or eight studies with different populations, tasks, tools, model generations and outcome measures cannot be pooled. Nothing here is a meta-analysis and no combined effect size is offered, because one would be meaningless.

Effect sizes have short shelf lives. Every study fixes a model generation. The 2022 and 2023 results concern tools substantially weaker than current ones, and the 2025 result concerns tooling that has since changed. Any specific percentage in this note should be read as a historical measurement rather than a current estimate. The moderators are the part more likely to survive.

Task-level experiments cannot see team-level effects. Review load, coordination, code ownership and the distribution of understanding across a team are all plausibly affected and none are measured. A per-developer speedup that increases reviewer load could be net negative at team level, and no study in this corpus could detect that.

The domain transfer on the frontier concept is assumed. Dell'Acqua and colleagues studied management consultants. The jagged frontier is a compelling framing for software and it has not been demonstrated there.

One key construct rests on a non-peer-reviewed industry source. The maintainability discussion depends on GitClear, whose methodology has not been independently replicated and whose observed trend is heavily confounded. The honest position is that construct five is unmeasured rather than measured badly.

The author has no data. The practitioner observation available is that of one person who used these tools to produce seventeen repositories over five months. No throughput was recorded, no defect rate was tracked, no review time was measured, and no comparison condition exists. This is worth stating plainly because the temptation in a note like this is to add a personal figure, and there is none to add. The record shows what was built, and it is silent on how fast or how well.

Implications

For teams making an adoption decision. Decide which of the five constructs you care about before reading any number. If the answer is throughput of merged work, the field experiments are the relevant evidence. If it is speed on unfamiliar greenfield work, the controlled trial is. If it is maintenance cost, no evidence exists and you should measure your own.

For measurement. Instrument what the studies do not: review time per change, defect rate by authorship condition over two quarters, and time to modify a module a year after it was written. This is the gap, and an organisation with production systems can close it for itself in a way the public literature currently cannot.

Do not use developer surveys as productivity evidence. They measure perception, and the corpus establishes that perception diverges from measurement in this specific setting.

Expect the effect to depend on who and what. Juniors on unfamiliar work is the strong case. Experts on mature systems they know is the case where the evidence points the other way, and the appropriate policy is different for the two rather than uniform.

Treat the competence frontier as a review policy. Where a task is likely to sit outside the model's reliable range, and security-sensitive and unusual work often does, increase review rather than trusting output that will look exactly as confident as the reliable kind.

For further work. The study the field needs is longitudinal and organisational: authorship condition recorded per change, tracked forward against review time, defects and modification cost. It is unglamorous, it requires access to production repositories over quarters, and it would answer the question that seven task-level experiments cannot.

Conclusion

The two headline numbers do not conflict. One measures a novice-equivalent situation on a canonical greenfield task, the other measures experts making incremental changes in systems they know well, and the difference between those situations is the actual finding.

What the corpus supports is narrower than either camp claims. Assistance substitutes for absent knowledge and competes with present knowledge. Its benefit is largest for the inexperienced on unfamiliar work. Its effect reverses outside the model's competence, and that boundary is invisible from inside the task.

The most reliable result is uncomfortable and easy to act on: developers cannot tell whether these tools are making them faster. Which means the decision cannot be delegated to how the work feels, and the measurement that would settle it, of what the code costs after it is written, has not been done by anyone.

References

S. Peng, E. Kalliamvakou, P. Cihon and M. Demirer, "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot", arXiv:2302.06590, 2023. Author affiliations include GitHub and Microsoft.
METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", 2025.
Z. Cui, M. Demirer, S. Jaffe, L. Musolff, S. Peng and T. Salz, "The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers", 2024.
A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam and E. Aftandilian, "Measuring GitHub Copilot's Impact on Productivity", Communications of the ACM 67(3), 2024. Measures perceived productivity. Author affiliations include GitHub.
P. Vaithilingam, T. Zhang and E. L. Glassman, "Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models", CHI 2022 Extended Abstracts.
F. Dell'Acqua, E. McFowland III, E. Mollick, H. Lifshitz-Assaf, K. Kellogg, S. Rajendran, L. Krayer, F. Candelon and K. R. Lakhani, "Navigating the Jagged Technological Frontier", Harvard Business School Working Paper 24-013, 2023. Management consulting, not software development.
N. Perry, M. Srivastava, D. Kumar and D. Boneh, "Do Users Write More Insecure Code with AI Assistants?", ACM CCS 2023.
G. Sandoval, H. Pearce, T. Nys, R. Karri, S. Garg and B. Dolan-Gavitt, "Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants", USENIX Security 2023. Included as contradicting evidence.
GitClear, AI code quality reports, 2024 and 2025. Industry research, not peer reviewed, proprietary methodology, not independently replicated.
S. Imai, "Is GitHub Copilot a Substitute for Human Pair-Programming? An Empirical Study", ICSE 2022 poster track.
github.com/TheDevChopra, seventeen repositories built with the tools listed at B10 and B11. Recorded as A2. https://github.com/TheDevChopra

Productivity claims in AI-assisted software development: what has been measured, and in whom