AI Coding Agents: Code Output, Software Delivery, and Organizational ROI

Review period: January 1, 2021–October 13, 2026 Scope: Effects of AI coding tools on code production, review, issue resolution, software quality, delivered business value, and organizational ROI. Source policy: Prioritize original studies, controlled experiments, official institutional research, and first-party deployments; distinguish reported results, commercial claims, observational associations, and analytical inferences.

Executive summary

AI coding tools can increase productivity, but faster code generation is not equivalent to faster software delivery or positive organizational ROI. The 718-firm study finds a pronounced divergence between increased coding activity and completed software work, with review emerging as a downstream constraint. Controlled experiments nevertheless demonstrate genuine benefits in some tasks and settings—and slowdowns or learning costs in others. These findings are compatible, not mutually exclusive. (fion.ac)

The defensible organizational conclusion is conditional: AI creates value when the organization converts cheaper implementation into independently validated, deployed, useful software—or verified operating-cost reductions. Neither code volume nor merged pull requests establish that conversion. Enterprise deployments provide positive throughput results and substantial claimed cost savings, but generally lack independently verified, all-in financial ROI calculations. (arxiv.org)

Report synthesis: The most promising strategy is not simply to generate more code. It is to improve specification, context, verification, review capacity, deployment, and product prioritization together, while measuring total human effort and production outcomes.

1. What should count as actual productivity?

For this report, the analytical distinction is:

This is a report framework, not a claim that one published metric captures all five levels. Microsoft’s agent-rollout researchers explicitly acknowledge that merged PRs are not the same as their delivered value; DORA’s delivery metrics instead focus on delivering software safely, quickly, and efficiently. (arxiv.org)

The central evaluation question should therefore be: Does AI increase valuable, quality-adjusted production output per unit of total organizational input? Authoring time, reviewer time, QA, integration, operations, compute, and subsequent maintenance all belong in that accounting.

2. The 718-firm study: what it establishes

Reference, sample, and design

The primary reference is Fiona Chen and James Stratton’s Artificial Intelligence in the Firm: Bottlenecks in Software Production, using the author-hosted August 4, 2026 version. Observations span January 2021–March 2026 and include 725,938 workers, not exclusively developers. (fion.ac)

The study covers approximately 300 million work events across 718 firms, combining engineering activity with issue-management and calendar information. It uses staggered difference-in-differences based on firm adoption timing, separately examining assistants and agents. (fion.ac) Its specification includes firm/month effects, size controls, and firm-clustered errors; activity rates are normalized by active worker-month, not labor-hour. (fion.ac)

Reported results following agent adoption

The following are source-reported pooled estimates, not independently recalculated statistics.

Interpretation

Report synthesis: Within this panel, agents increased intermediate production more clearly than final delivery. The combination of more submissions, longer PR lifecycles, and more revision signals supports a downstream review constraint. It does not establish that the entire additional calendar interval was spent actively reviewing code, or that every extra comment represented additional human labor.

Several distinctions are essential:

• A statistically insignificant output estimate is not proof of zero benefit. The reported interval still accommodates a modest issue-completion gain, though not a pass-through matching the coding-volume estimate.
• PR creation is not PR acceptance. More submitted work can expand work-in-progress without increasing production delivery.
• Reviewer participation is not review hours. A person performing one review and a person spending most of a week reviewing both count as participants.
• Review friction is not a production-defect rate. The study does not audit code content or calculate revenue/profit ROI. (fion.ac)

AI review adoption also should not be confused with comprehensive automated verification: the paper’s 10.8% figure concerns PRs receiving AI-generated review comments, not PRs authored by agents. (fion.ac)

3. Methodological assessment

Why this study is particularly valuable

The study’s major contribution is organizational scope. Rather than asking whether one developer finishes one task faster, it follows multiple stages of work across many firms. Its distinction between intermediate productivity and downstream output is precisely what is missing from demonstrations of rapid code generation. (fion.ac)

Its robustness work includes pre-trend assessment, text/LLM-based task-size controls, and alternative staggered-adoption estimators illustrated for code-line output. These are useful checks, although they do not independently validate every downstream result. (fion.ac)

Limitations and their implications

The following judgments are report analysis, rather than additional findings published by the authors.

The paper’s adoption measurement can miss unsigned or individually licensed use. That creates uncertainty around treatment timing and intensity, rather than automatically proving bias in one direction. (fion.ac)

Overall assessment: This is strong evidence against treating code-volume gains as automatic delivery gains. It is weaker evidence for the universal claim that agents reduce code quality, and insufficient evidence for the financial claim that agents have negative ROI. Its observational organizational design and task-level randomized experiments answer different questions. (fion.ac)

4. Controlled experiments: genuine gains, genuine costs

Comparison of major experiments

Why the results are compatible

Report synthesis: AI can be particularly useful when implementation is bounded and requirements are explicit. It can be less useful when a developer already knows the relevant code deeply, while the tool needs prompting, context reconstruction, and extensive verification. A coding assistant’s gain on a specified task also need not pass through a team’s review queue or deployment process.

Randomization strengthens causal identification for the tested task and population; it does not make that population representative of all engineering work. Conversely, the broad firm panel improves organizational relevance without achieving the same protection against confounding as an RCT. The experiments should therefore be triangulated, not averaged into a universal “AI productivity percentage.” (arxiv.org)

Updating the negative METR result responsibly

The early-2025 slowdown should not be presented as an October 2026 effect estimate. METR’s February 24, 2026 update covered 10 returning and 47 newly recruited developers, with more than 800 tasks overall. Estimated task-time changes were −18% for returnees, interval −38% to +9%, and −4% for newcomers, interval −15% to +9%. Both intervals include no effect. (metr.org)

METR judged these newer estimates unreliable because developers and tasks increasingly selected out of AI-disallowed conditions; lower participant pay and concurrent-agent time measurement added difficulties. This is evidence that the original estimate should not be mechanically carried forward—not a clean demonstration of a universal newer-tool speedup. (metr.org)

A separate May 2026 METR survey of 349 technical workers reported median self-estimated changes in work value of 1.4–2×. Those are perceived counterfactuals, not independently measured delivery productivity or financial ROI. (metr.org)

5. Enterprise deployments: throughput gains are possible

These deployments rule out the categorical statement that AI only generates unusable code. They do not establish an industry-wide financial return: the endpoints, generations of tooling, task types, and selection processes differ materially.

There is also a distinction between causal research and vendor benchmarking. Jellyfish’s March 17, 2026 announcement reported 2× PR throughput among top-quartile AI adopters relative to low adopters. That comparison is not equivalent to Chen and Stratton’s adoption-timing design and does not measure issue completion or financial ROI. (jellyfish.co)

Business value remains the least-established link

Demirer, Musolff, and Yang’s NBER working-paper abstract reports that coding gains attenuate at the release level, while more applications across four marketplaces do not produce increased total usage. The original abstract is reproduced in publisher-supplied RePEc metadata; the full paper was inaccessible during this review. Accordingly, this report uses it only as qualitative corroboration, not as a verified numerical effect estimate or independently replicated ROI result. (ideas.repec.org)

Interpretation: Releasing more software is still not equivalent to creating more demand. Aggregate marketplace usage also cannot determine whether an individual enterprise’s internal automation or particular product investment was valuable.

6. Software quality: concern is justified, a universal verdict is not

The relevant quality endpoints must be separated:

• Draft quality: How much correction is needed before acceptance?
• Production correctness: What defects escape review and testing?
• Operational reliability: What fails after deployment?
• Maintainability: How much effort does subsequent evolution require?
• Human understanding: Can the responsible engineers explain and safely modify the system?

The focal study provides review-process evidence, not an audited production-defect comparison. The maintainability experiment supplies counterevidence to inevitable short-term harm, while the skill-formation experiment identifies a potential weakness in human understanding. These are different quality dimensions. (fion.ac)

Additional evidence reinforces the need for workflow discipline:

• An empirical study of 33,596 agent-authored public PRs reported 24,014 merged, or 71.48%. Its qualitative rejection analysis had 562 accessible PRs from 600 sampled cases and identified failures involving CI, duplication, unwanted features, and absent reviewer engagement. This is not a matched human-versus-agent defect-rate experiment; repository and agent selection limit generalization. (arxiv.org)
• DORA’s 2024 research associated greater AI adoption with lower delivery throughput and stability. Its 2025 findings, discussed in March 2026, instead associated adoption with higher throughput and greater instability. These are observational survey relationships, not randomized proof that AI causes either outcome; they nevertheless caution against treating faster delivery as automatically safer delivery. (dora.dev)

Report synthesis: The evidence supports monitoring quality closely, not claiming that every AI-assisted change is inferior. More change requests could reflect poorer drafts, higher standards, weaker author familiarity, or changed review processes. Longer elapsed review can also arise from queuing. Those mechanisms require separate measurement.

7. Organizational ROI: what can be concluded?

Verdict

The correct conclusion is therefore conditional ROI, not zero ROI and not automatic ROI. The focal study does not calculate financial returns. An absence of employment reductions would not itself disprove value: avoided future hiring, useful redeployment, better reliability, or operating-cost savings could matter. Equally, unchanged payroll means saved authoring hours cannot simply be booked as cash savings.

A defensible measurement framework

The following is a report recommendation, consistent with treating AI adoption as a broader organizational investment rather than a code-count exercise. DORA provides an explicit framework for evaluating the ROI of AI-assisted development. (dora.dev)

ROI = (realized incremental benefit − incremental adoption cost) / incremental adoption cost.

Report two ledgers separately:

1. Financial return: Additional realized contribution profit and verified avoidable cash costs, less actual incremental adoption costs.
2. Capacity and opportunity cost: Total human effort freed or consumed, including additional review, QA, integration, operations, and maintenance—even when payroll does not change.

Include licenses, inference, CI and sandbox compute, integrations, training, administration, security controls, and remediation. Do not count the same saved hours once as labor savings and again as incremental output value. Do not convert calendar review delays directly into paid reviewer-hours.

How an organization should test the proposition

A credible local evaluation should:

• Compare teams or comparable workstreams over multiple full release cycles, preferably with randomized phased access or randomized workflow interventions.
• Record actual use and noncompliance, not just purchased seats.
• Predefine authorized tasks and include failed, abandoned, unmerged, and reopened attempts.
• Separate authoring time, active human review, waiting, QA, integration, deployment, and post-release correction.
• Track deployed outcomes against customer or operational acceptance criteria—not merely PR totals or story points.
• Report the numbers of teams, developers, tasks, releases, and incidents, with uncertainty and subgroup limitations.
• Follow quality and maintenance after release, rather than stopping at merge.

METR’s newer measurement difficulties illustrate why task selection and concurrent-agent time accounting need explicit treatment. (metr.org)

8. Bottlenecks and engineering practices that can overcome them

The practices below are recommendations grounded in the reviewed evidence, not a claim that an entire practice bundle has been causally proven to generate ROI.

Task selection and rollout matter

Report synthesis: Start with well-specified, bounded work for which verification is relatively inexpensive: repeatable migrations, localized changes with strong tests, and maintenance tasks with clear acceptance criteria. Treat poorly understood legacy behavior, broad architectural changes, and high-consequence business logic more conservatively.

This recommendation is consistent with the contrasting migration, expert-maintenance, and longitudinal enterprise evidence; it is not a universal ranking of task types or tools. The B2B deployment’s gains accumulated with use and were stronger in newer code, while Microsoft’s rollout found social networks important for first adoption. Enablement and peer learning are therefore plausible complements—but not proven substitutes for measuring outcomes. (press.aboutamazon.com)

Why coding speed alone has a ceiling

Revoqra illustrative calculation—not an empirical estimate: Suppose coding represents 40% of total labor and becomes twice as fast, with no additional verification work. Total labor falls to 60% + 40%/2 = 80% of baseline: a 20% labor reduction, corresponding to 25% greater throughput capacity under those simplifying assumptions—not a doubling. If review or deployment already constrains throughput, even that gain may not materialize.

The practical priority is consequently to shorten the whole validated delivery path, rather than maximizing the number of parallel agents or generated changes.

9. Bottom line

The 718-firm study is best read as a warning about incomplete pass-through, not a verdict that AI coding is useless. It shows why code production and review activity are inadequate organizational success measures. Controlled trials and deployments demonstrate real gains in some settings, but the evidence for typical all-in financial ROI remains incomplete. (fion.ac)

The strongest operating principle is: expand generation only as fast as the organization can specify, independently verify, deploy, maintain, and obtain value from the resulting software.

Evidence and calculation note

Empirical percentages above are source-reported estimates or descriptive results, not Revoqra reanalyses. Proprietary enterprise microdata were unavailable for independent replication. No original significance tests or pooled productivity estimate were performed. The one toy calculation is explicitly hypothetical. Different tools, periods, populations, and endpoints are not treated as directly interchangeable.

Sources

• Focal organizational study: Chen and Stratton’s working paper and author research page. (fion.ac)
• Randomized productivity evidence: Peng et al.; Google’s enterprise-task trial; Cui et al., Management Science. (arxiv.org)
• Independent measurement and updates: METR’s early-2025 study, February 2026 update, and May 2026 survey. (metr.org)
• Learning and maintainability: Anthropic’s skill-formation research; Borg et al., Empirical Software Engineering. (arxiv.org)
• Enterprise agent deployments: Microsoft CLI rollout; He et al.’s longitudinal mandate study. (arxiv.org)
• Assistant deployment and financial-benefit claim: CodeFuse quasi-experiment; Amazon’s modernization announcement. (sciencedirect.com)
• Delivery, stability, and evaluation guidance: DORA’s generative-AI findings, 2026 interpretation, AI capabilities model, metrics, and ROI framework. (dora.dev)
• Agent rejection and verification: Ehsani et al.’s public-PR study; SIG’s vendor-controlled guardrail experiment. (arxiv.org)
• Limited-access corroboration: Demirer, Musolff, and Yang’s original abstract, reproduced in RePEc; full-text verification was unavailable. (ideas.repec.org)