Adopting AI: The Evidence So Far
What the current evidence shows about bringing AI into an engineering organization: where the gains land, what they depend on, and how the work changes.
Updated July 2026
This is research on what organizations have actually measured while adopting AI in software engineering, and how it went. It is not a recommendation. Five themes, seventeen evidence cards graded by confidence. Each theme closes with working questions it deliberately leaves for you to answer, and the resources at the end are what to reach for when you go to answer them.
- 01AI amplifies an organization; it doesn't fix one
- 02The engineer's role moves from author to director
- 03Verification becomes the bottleneck
- 04Rollout is phased and governance-first
- 05The failure modes are real, and they don't announce themselves
Also inside: how it was made, its sources, and how claims were verified (the panels below) · working questions after each theme · resources to run.
Provenance
MethodSix research angles, worked independently, then adversarially reviewed.
An AI deep-research run, then edited and fact-checked by hand. A scoping pass split the question into six angles:
- Anthropic's own primary sources
- Vendor and analyst rollout playbooks
- Productivity measurement research (DORA, DX)
- The shift in engineering roles
- Practitioner writing (Fowler, Thoughtworks, Osmani)
- Risks, governance and contrarian evidence
Independent search agents worked each angle. A separate pass then took the opposite side, going back over every finding to look for reasons it was wrong.
SourcesControlled trials, measurement research, vendor guidance, and practitioner writing.
Four kinds of source, and they are not equally independent:
- Controlled trials. Anthropic's randomized trial on what AI assistance does to coding skill, and METR's randomized trial on experienced open-source developers.
- Survey and measurement research. Google's DORA 2025 report, drawn from nearly 5,000 developers, and Stanford's software engineering productivity research, which measures output directly across 100,000+ engineers at 600+ companies.
- Vendor guidance. Anthropic, GitHub and DX, describing their own tools and how to roll them out.
- Practitioner writing. Birgitta Böckeler on Martin Fowler's site, Thoughtworks' Technology Radar, and Addy Osmani.
Roughly twice as many sources were read as are cited. Nothing was dropped for disagreeing: what fell away either yielded no citable claims at all (blog indexes, a customer-story page, articles that restated other sources) or dissolved during synthesis. METR, whose measured result cuts hardest against the optimistic ones, kept its own card and a place in the overview's belief-vs-measurement tile. Half of the cited sources are vendors writing about their own products, every card that leans on one says so, and every cited source was read at the page, including the DORA report, which is a 142 page PDF.
VerifiedClaims were argued against before they were allowed to stand. Not all of them survived.
The research passes behind this document put 258 claims through the same gauntlet: three reviewers per claim, instructed to refute, not to confirm, covering whether AI helps or hurts delivery, how engineering roles shift, what a rollout actually needs, where the risks land, and more. Thirty-seven were refuted, among them two fabricated quotations and a false attribution; none of them appear in this document.
The refuted claim worth naming is a figure still widely quoted: McKinsey's 2023 claim that developers can complete coding tasks "up to twice as fast" with generative AI. All three reviewers rejected it, not because the savings are unreal but because a best case was being quoted as the norm: the same study's task-by-task numbers reach roughly 50 percent on routine work like documentation and fall below 10 percent on high-complexity tasks, and the study is a small 2023 lab exercise run before modern agentic tools. It appears nowhere in this document; card E15 records the task-by-task figures that did hold up and names the refuted claim so nobody repeats it.
QuotationsQuotation marks mean verbatim. Nothing here is a paraphrase in disguise.
Anything between quotation marks in this document is character for character from the source page. Where a source makes a point but has no single sentence worth quoting, you will find plain narration instead, never a paraphrase dressed up as a quote.
Every quotation was checked against its source before publication, and the ones that failed were replaced with the source's real words. No statistic changed in the process; where a quote was wrong, it was the wording that was wrong and not the finding.
At a glance
Across this research, the measured gains are real, and the spread is enormous: the same tools deliver up to 40% on one kind of work and as low as 0% on another. What separates those outcomes is rarely the tool. It is the conditions the tool lands in: the type of task it is given, the health of the codebase, the strength of the verification infrastructure, the shape of the engineer's role. Those are things an organization can shape. The evidence describes work moving from writing code toward directing and verifying it, and the organizations that build for that shift are the ones that capture the gains.
Measured
Up to 40%
productivity gain on simple new-code tasks, as low as 0% on complex work in an existing codebase
Stanford SWEPR 2025 · 136 teams
Measured
Code health
explains roughly 40% of the variation in measured AI gains: tests, types, documentation, modularity
Belief vs measurement
Measured
1 in 3 got it right
In a Stanford study of 43 engineers, two-thirds misjudged their own productivity by roughly 30 percentile points.
That cuts both ways: confidence that AI is helping and confidence that it is not are the same kind of estimate. A 2025 study made the gap vivid: on complex work in codebases they knew well, developers felt about 20% faster while measurement showed the work taking longer. Its authors have since revisited that result: as of early 2026, they believe developers are more sped up than their original estimates showed. Belief and measurement diverged in both directions here. See E16.
Stanford SWEPR 2025 · METR 2025-26
The analogy the evidence suggests
When test automation arrived, testers read it as a threat. It was going to take the job. What it took was the monotonous, repetitive part of the job. What it left was the part that needed a person: mindset, perspective, experience, patterns and practices. Those skills did not go obsolete. They became the work.
The evidence here does not settle whether software engineering is making the same passage. But it does point the same way: the skills that survive are judgment, verification, and knowing when to distrust the output. See E4 and E6.
Themes
AI amplifies an organization; it doesn't fix one
AI raises the ceiling for teams with their fundamentals in order and lowers the floor for teams without them. Where the control systems arrived after the acceleration rather than before it, the measured result was instability rather than throughput (E2).
E1High confidence
AI is an amplifier, not a remedy
"AI's primary role in software development is that of an amplifier. It magnifies the strengths of high-performing organizations and the dysfunctions of struggling ones."
Google DORA 2025source, context & caveats
SourceGoogle DORA · State of AI-assisted Software Development 2025 (~5,000 professionals). dora.dev/dora-report-2025. Stanford figure: Yegor Denisov-Blanch, SWEPR, conference talk, December 2025 (a talk, not a published paper). youtube.com/watch?v=JvosMkuNxF8
ContextStanford's measurement work points the same way, from outside Google: plotting an environment cleanliness index (tests, types, documentation, modularity, code quality) against measured AI productivity lift, Stanford SWEPR reports an R-squared near 0.40, so roughly 40% of the variation. The cleaner the fundamentals, the more the same tools return.
CaveatCorrelational, not a controlled trial. DORA's own contrast is between "high-performing organizations" and "struggling ones"; it defines team capability through its seven-practice AI Capabilities Model, not through a term called "strong team." The Stanford figure is weaker still: it comes from a conference talk rather than a published paper, its author calls the cleanliness index "quite experimental", and no sample size is given for that chart. Read it as directional corroboration, not as a second measurement.
E2High confidence
Speed without control systems produces instability
"AI-assisted coding can increase the volume and velocity of changes, which can also lead to more instability. Your version control system is a critical safety net."
DORA 2025 + Stanfordsource, context & caveats
SourceGoogle DORA 2025. dora.dev/dora-report-2025. Stanford case study: Denisov-Blanch, SWEPR, presented deck and talk, 2025 (not a published paper). youtube.com/watch?v=JvosMkuNxF8
ContextA Stanford case study measures the outcome side of this, under the title "Higher PR Counts, Lower Quality". At one large enterprise, a 350-person organization adopted AI; comparing the four months before with the four months after, pull requests rose 14% while code quality fell 9%, rework rose 2.6x, and effective output stayed roughly flat. Stanford does not diagnose why. Its point is that the PR count alone would have read as a win.
CaveatThe quoted line sits in DORA's advice chapter, though DORA measured what it rests on: AI adoption is associated with higher delivery instability, and more frequent use of rollback amplifies AI's positive influence on team performance. What DORA did not measure is the causal chain the line implies; it says plainly that rollback reliance does not directly reduce instability. Stanford measured outcomes, not control systems, and names no cause: reading its degradation as controls arriving late is this dossier's inference, not Stanford's finding. One organization, before and after: it shows the pattern, not how often it occurs.
E3High confidence
Near-universal use, conditional trust
"The majority of survey respondents (90%) use AI as part of their work and believe (more than 80%) it has increased their productivity. Yet a notable portion (30%) currently report little to no trust in the code generated by AI, indicating a need for critical validation skills."
Google DORA 2025source, context & caveats
SourceGoogle DORA 2025. dora.dev/dora-report-2025
ContextThe 90% and 80%+ are the same DORA figures behind the overview's two self-reported tiles; this quote is the sentence they come from. The number that is new here is trust, and DORA reports it as a distribution rather than a verdict: 70% of respondents express some degree of confidence in the quality of AI output (24% "a great deal" or "a lot"), while 30% are more reserved, trusting it "a little" (23%) or "not at all" (7%).
Caveat"Productivity increased" is self-reported perception, and E16 shows perceived speedup can diverge from measured reality. DORA does not read the gap as a paradox: its executive summary heads the same finding "Broad AI adoption with healthy skepticism" and calls the "trust but verify" stance "a sign of mature adoption", and its Trust chapter holds that "absolute trust is not a prerequisite for AI-generated outputs to be useful."
The engineer's role moves from author to director
This is where the tester analogy earns its keep: the mechanical work gets absorbed and the role relocates "up" to judgment, direction, and quality ownership. Skill doesn't become obsolete: it moves. What the organization has to build so that direction is possible is Theme 03's subject; this theme is about the person.
E4High confidence
Skill remains essential even when the AI succeeds
"Even in those successful sessions, I intervened, corrected and steered all the time." "AI goes down rabbit holes quite frequently when it misdiagnoses a problem. Many of those times I can pull the tool back from the edge of those rabbit holes based on my previous experience with those problems."
Böckeler, Thoughtworkssource, context & caveats
SourceBirgitta Böckeler (Thoughtworks), The role of developer skills in agentic coding, March 2025. martinfowler.com
CaveatThe judgment on display is not only about spotting wrong turns: Böckeler's worked sessions apply it to code quality, root-cause analysis, test quality, reuse, and over-complexity. Illustrated through worked examples, not a controlled study: directionally strong, not quantified.
E5Medium · against vendor interest
Directing is a loop, not a handoff
Anthropic's own guidance opens by warning against the thing people try first: "Letting Claude jump straight to coding can produce code that solves the wrong problem." The prescribed sequence is explore, then plan, then code, and for anything substantial the advice is to be interrogated before a line is written: "For larger features, have Claude interview you first." The stated payoff: "Time spent making the spec precise pays off more than time spent watching the implementation."
Anthropicsource, context & caveats
SourceAnthropic · Claude Code best practices. code.claude.com/docs/en/best-practices · How Anthropic teams use Claude Code (internal-teams report, PDF). anthropic.com (PDF)
ContextWhat the unplanned prompt actually returns, from the vendor's own engineers: Anthropic's RL engineering team reports that on small-to-medium PRs, it "only works on first attempt about one-third of the time, requiring either additional guidance or manual intervention." They can afford that bet because they commit constantly and roll back cheaply. Two times in three, the human is back in the loop.
CaveatVendor sources, and the internal report is ten teams who do not agree on tactics: one treats the model "like a slot machine", discarding and rerunning rather than correcting, while others "brainstorm and prompt plan before coding". No methodology sits behind the one-third, and it is scoped to one team's small-to-medium PRs. It is carried because a vendor publishing a two-thirds first-attempt miss rate for its flagship agent is publishing against its own interest.
E6Medium confidence
The value and the risk both move to review
"Maker becomes checker." "The best software engineers won’t be the fastest coders, but those who know when to distrust AI."
Osmani + Codacysource, context & caveats
SourceAddy Osmani, The Next Two Years of Software Engineering, January 2026. addyosmani.com/blog/next-two-years. The post is built as paired scenarios the author declines to predict between ("These aren't really predictions, but lenses for preparation"); "Maker becomes checker." is his diminished-role pole, and the second line is an unnamed senior engineer he quotes. Codacy, a code-quality and review-tooling vendor, names the same shift in the title of a marketing post, "AI Agents Are Turning Developers Into Engineering Orchestrators and Moving the Risk to Review". blog.codacy.com
CaveatOsmani is practitioner commentary, widely echoed but not empirically measured. Codacy is not a second independent voice: it is a vendor marketing post, and Codacy sells the review-and-enforcement layer this framing implies you need, so its commercial interest runs with the claim. Its one survey figure (38% find AI code harder to review) comes from Sonar, another code-review vendor, and is a plurality: 27% said the opposite.
Verification becomes the bottleneck
AI generates faster than a team can check, so checking becomes the constraint, and the rest of the software development lifecycle (SDLC) adapts around it. The vendor guidance in this theme converges on making the loop close automatically; the measured wins in it landed where the loop already closed.
E7High · measured
Writing code speeds up far more than shipping code
"we find that autocomplete, interactive coding agents, and autonomous coding agents each significantly increase coding activity (“commits”), with respective cumulative effects of 40%, 140%, and 180%. These gains, however, attenuate sharply across the production hierarchy: the 180% cumulative effect falls to 50% for the number of projects, and to 30% for actual releases."
NBER working papersource, context & caveats
SourceDemirer, Musolff & Yang · Writing Code vs. Shipping Code, NBER Working Paper 35275, May 2026 (100,000+ GitHub developers, matched event study). nber.org/papers/w35275
ContextThe authors' own summary: "the strong productivity gains from AI are attenuated by human bottlenecks in the production chain". Those human bottlenecks, the checks between written code and shipped code, are what E8 and E9 aim to automate.
CaveatWorking paper, not yet peer reviewed, and two of the three authors are paid Microsoft research consultants (Microsoft owns GitHub, the data source). Treat that conflict as live rather than cancelled by the direction of the result: the same paper headlines a 17.3x cumulative lines-of-code effect and argues the attenuation can compress as AI automates more of the chain. The headline 180-to-30 contrast is also not like-for-like, because the paper reports no release effect for autonomous agents (an async agent cannot cut a release itself), so the 30% covers autocomplete and sync agents only. The attenuation holds inside each generation regardless: autocomplete lifts commits 40% and releases 10%; sync agents lift commits 140% and releases 20%. Scope: public repositories only, and the authors say their data "does not cover some other important parts of the market", "most notably enterprise and internal-only software". Take the shape of the attenuation into your own release process, not the magnitudes.
E8High · prescriptive
Anthropic's central prescription is a pass/fail check the agent can run
"Claude stops when the work looks done. Without a check it can run, 'looks done' is the only signal available, and you become the verification loop… Give Claude something that produces a pass or fail, and the loop closes on its own."
Anthropicsource, context & caveats
SourceAnthropic · Claude Code best practices. code.claude.com/docs/en/best-practices
CaveatVendor guidance, but the underlying logic (automated checks reduce human verification burden) is broadly accepted practice independent of any tool. The same logic runs the other way in E10: the largest reported wins landed on work that arrived with a pass/fail check already attached.
E9High · split vote
Anthropic attributes performance to the "harness" more than the model
The harness (the context files, the scoping, the loop the agent runs inside) "determines how Claude Code performs more than the model alone."
Anthropicsource, context & caveats
SourceAnthropic · How Claude Code works in large codebases. claude.com/blog
CaveatVendor source, and prescriptive guidance rather than a measured result: no measurement sits behind it, and one of three hostile reviewers voted to refute it. The lever it points at is the repository rather than the release notes, the claim being that what the agent runs inside moves performance more than a model upgrade does.
E10High · first-party
The largest reported wins landed on bounded, mechanically checkable work
Migrations, framework conversions, version upgrades, repetitive changes across many files. Airbnb: "We’d originally estimated this would take 1.5 years of engineering time to do by hand", and instead migrated "nearly 3.5K Enzyme test files to React Testing Library in just 6 weeks using automation and LLMs". Google, on an int32-to-int64 ID migration inside its 500M+ line Ads codebase (tens of thousands of code locations across thousands of files): "The total time spent on the migration was reduced by an estimated 50% as reported by the engineers doing the migration", with 80% of the code modifications in the landed changelists fully AI-authored.
Airbnb + Googlesource, context & caveats
SourceAirbnb Engineering · Accelerating Large-Scale Test Migration with LLMs, March 2025. airbnb.tech · Google · How is Google using AI for internal code migrations?, January 2025. arxiv.org/2501.06972
ContextE8 is about building a pass/fail signal where none exists. This is the other move: some work arrives with one, and that is where the wins concentrate. The check is what makes the work delegable, not what makes it free: Airbnb ran each file through a state machine of automated validations with retry loops and prompts pulling in as many as 50 related files, which reached 75% in four hours; four days of hand-tuning took that to 97%, and the last 3% was fixed by hand over another week. For contrast, a 2023 lab study of that era's assistants measured under 10% time saving on work developers rated high in complexity (E15): a gradient across task shapes and tool generations, not one controlled comparison.
CaveatFirst-party reports against estimated baselines; nobody ran the manual arm, and Google says itself: "It is not a research study". Slack ran the same Enzyme-to-RTL migration a year earlier and reported a 22% developer-time saving, drawn from a 338-file sample counting only test cases that converted, ran and passed, which Slack itself flags as an undercount (slack.engineering). Same task, different pipeline and model generation, very different published numbers, and no writeup separates the two causes. And volume is not value: bulk changes shift the remaining cost to review (E7).
Rollout is phased and governance-first
Theme 01 found that the control systems have to be in place before the acceleration. This is what that order looked like when real organizations ran it. Every vendor and analyst playbook read for this dossier leads with organizational and governance work, then a small pilot, then expansion, rather than a technical big-bang, and the adopters who published their rollouts did the same. The shape matches the one organizations used for CI/CD, Git, and test automation, though no source here draws that comparison: it is the dossier's.
E11High · measured
Meta rolled out language by language, 25% of developers at a time
"Our rollout strategy for CodeCompose consists of gradual deployment in waves of languages: (i) only Python, (ii) Hack, Flow (Javascript), and C++, (iii) others. Within each wave, we rolled it out to increments of 25% of the developer population until we enable it for 100% of developers. The rollout was completed after four weeks in Spring of 2023." The phasing was the measurement plan: "This method of doing the rollout steadily in phases helped us measure the effects of CodeCompose in practice at every step (using the quantitative and qualitative feedback) and iterate on the product experience to improve the means to achieve intended outcomes before rolling out further."
Meta, FSE 2024source, context & caveats
SourceMeta · AI-Assisted Code Authoring at Scale (CodeCompose), Proceedings of the ACM on Software Engineering (FSE), 2024, peer reviewed. arxiv.org/2305.12050
Context"In this paper, we make over 4.5M suggestions to 16K engineers. We see an acceptance rate of 22% which is comparable to those at Google and GitHub." What the phased rollout surfaced, through developer feedback rather than telemetry: CodeCompose competing with the traditional auto-complete already installed. "Sometimes, when these two systems compete to show suggestions, it creates a disruptive experience for developers," with the Tab key "overloaded to accept CodeCompose suggestions and traditional auto-complete suggestions." Meta reports this as still open, not solved: "We are actively exploring this area to make CodeCompose a productive experience for all the developers." A new tool lands beside the tools people already run, and reconciling them is its own workstream, not something phasing does for you.
CaveatMeta reporting on Meta's own internal tool, in a peer-reviewed venue. A 2023 completions-era assistant: evidence for the rollout method, not for today's gain sizes. The paper's "overwhelmingly positive" satisfaction figure came from an opt-in group of 70 engineers and is not repeated here.
E12High · prescriptive
Vendor rollout guidance is staged and governance-led
GitHub's enterprise rollout guide is an ordered set of tracks: Getting started, then Governance basics, then Adopting agents. The first getting-started step is approval, not deployment: "Get ready to adopt Copilot by sending resources to legal and security teams in your company." Governance sits ahead of agents, aiming to "Set a governance posture that balances compliance requirements with developer productivity, so your rollout succeeds from day one."
GitHubsource, context & caveats
SourceGitHub · Adopting GitHub Copilot in your enterprise. docs.github.com
CaveatVendor guidance for GitHub's own product: a recommended sequence, not an industry consensus. GitHub's separate rollout guide, Rolling out GitHub Copilot at scale, opens with license assignment and places governance second, so even this vendor does not always lead with approval. Read the track order as one vendor's opinion about sequence.
E13High · prescriptive
DX's framework measures impact, not usage, across counterbalanced dimensions
"Effectively measuring AI code assistants and agents requires focusing on three key dimensions: utilization, impact, and cost." DX frames these three dimensions as tracking the natural lifecycle of AI adoption: teams first prioritize adoption and usage, then measuring impact, then the governance of standardization and spend. Impact is time saved, satisfaction, and delivery, not raw usage.
DX + GitHubsource, context & caveats
SourceDX · Measuring AI code assistants and agents. getdx.com/research · Measuring developer productivity with the DX Core 4. getdx.com/research
ContextIn DX's framing, impact is not one number. The DX Core 4 (speed, effectiveness, quality, business impact) is built so its dimensions check each other: "Multiple dimensions are needed to capture software development comprehensively because changes to one dimension, such as speed, may negatively affect others (e.g., quality or effectiveness)." DX calls these "four counterbalanced dimensions." GitHub ships one implementation of the utilization dimension: its usage-metrics API sorts engaged users by the deepest capability in use (code-first: completions and/or IDE agent mode; agent-first: one GitHub-hosted agent surface such as the cloud agent, code review or CLI; multi-agent: two or more) over "a rolling 28-day window," aiming to "Move beyond simple active-user counts." Note what that API reports per user: lines added and deleted, and code-generation acceptance activity. github.blog/changelog
CaveatVendor source: DX sells the platform that measures this, and a framework prescribes how to look, not what will be found. It does constrain its own instrument, though not by excluding output metrics: diffs per engineer is a Core 4 key metric that DX says "requires caution" and permits only when counterbalanced, never tied to a target, and rolled out without abuse. What it warns against is throughput in isolation: "Speed and throughput metrics, when used in isolation, often incite fear and counterproductive behaviors from developers." DX also attaches preconditions this card cannot drop: it "strongly caution[s] against top-down mandates or using metrics for individual performance evaluation", singles out code-generation volume as "particularly susceptible to gaming", and tells adopters to say plainly that "These metrics will not be used in individual performance evaluations." That constraint bites here, because the GitHub API above reports exactly that kind of per-user data.
The failure modes are real, and they don't announce themselves
New skill quietly fails to form, the gains are smaller than they feel, duplication accumulates without anyone noticing, and in every case the people involved reported that things were going well. None of these show up from the inside, which is the whole argument for measuring rather than assuming. (Where the gains do land is Theme 03.)
E14High · against vendor interest
Over-reliance stunts skill formation: how AI is used matters more than whether
"There was, however, a significant difference in test scores: the AI group averaged 50% on the quiz, compared to 67% in the hand-coding group, or the equivalent of nearly two letter grades." "The largest gap in scores between the two groups was on debugging questions."
Anthropic Researchsource, context & caveats
SourceAnthropic Research, How AI assistance impacts the formation of coding skills (RCT, 52 mostly-junior engineers, none familiar with the library under test). anthropic.com/research
ContextParticipants who stayed cognitively engaged scored highest, 65% to 86%, and engagement took more than one shape: the largest high-scoring cluster asked the AI conceptual questions only and wrote the code itself. The heavy-reliance patterns averaged under 40% as a group, and they are not all what you would expect: alongside wholesale delegation sits a cluster that wrote its own code but leaned on AI to debug and verify it, asked more questions, and still landed in that group. Across the two arms the AI group finished about two minutes faster, not a statistically significant difference. The trade is real for the people who take it, though: the paper's abstract says participants who fully delegated "showed some productivity improvements, but at the cost of learning the library." One calibration from practitioner review: the randomized design supports the group-level gap, while the cluster splits rest on small subgroups; read them as directional, not individual-level.
CaveatThis measures failure to acquire a skill nobody had, not erosion of one someone already has: every participant was screened as unfamiliar with the library under test (async Trio). Anthropic leaves the decay case open, saying it is possible AI both accelerates work on well-developed skills and hinders the acquisition of new ones, and it scopes the speed null too, expecting AI to help more on repetitive or familiar tasks. n=52, single study, mostly-junior participants; not yet replicated at scale. Read the gap as points, not as a relative drop: 50% versus 67% is a 17 percentage-point gap, not a 17% relative decline. Notably against the vendor's commercial interest, which strengthens it.
E15Medium confidence
Gains are real but task- and experience-dependent, not universal
~45-50% time saved on documentation, 35-45% on generation, 20-30% on refactoring, but time savings "shrank to less than 10 percent on tasks that developers deemed high in complexity."
McKinseysource, context & caveats
SourceMcKinsey · Unleashing developer productivity with generative AI (internal lab, 40+ of its own developers, June 2023). mckinsey.com
ContextThe gradient tracks speed, not value: McKinsey also reports that developers using the tools on complex tasks were "25 to 30 percent more likely than those without the tools to complete those tasks within the time frame given". Read the low end as where time savings thin out, not as where AI stops helping. McKinsey attributes the variance to two things, and the second is the sharper result: "in some cases, tasks took junior developers 7 to 10 percent longer with the tools than without them." What the tools are pointed at is one lever; who is holding them is the other. The other end of the task axis is E10, where later tooling on bounded, mechanically checkable work posted the largest gains anyone has reported: a gradient across task shapes and tool generations, not a controlled comparison.
Caveat⚠ The broader "up to twice as fast" generalization failed verification: it appears nowhere in this document. What survived are the task-by-task figures above, not a blanket multiplier. 2023 lab study on that era's tools.
E16Contrarian · perception gap
Perceived speedup and measured speedup can diverge
"…developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%."
METRsource, context & caveats
SourceMETR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (RCT, 16 developers, 246 tasks). metr.org/blog
ContextIn that study developers took 19% longer with AI, but that is a point estimate inside a wide 95% interval, +2% to +39%, drawn as the error bar on the original post's headline chart. A point estimate inside a large range, not a precise number. METR now banners the original study: "These results are out of date." "We have released results that are current as of early 2026, in a continuation of this study. We believe these historical results no longer reflect the current impact of AI models on open-source developer productivity." Its current read: "we believe it is likely that developers are more sped up from AI tools now — in early 2026 — compared to our estimates from early 2025." And on magnitude: "our data is only very weak evidence for the size of this increase." metr.org/blog/2026-02-24-uplift-update
CaveatThis is a finding about human self-assessment, not about tool capability, so it does not expire as models improve. The durable lesson is that perceived speedup is not evidence of speedup: in this study the two pointed in opposite directions.
E17High confidence
Complacency with AI-generated code is a named anti-pattern
"Complacency with AI-generated code" placed in the Hold ring (the "proceed with caution" signal), citing rising code churn/duplication. Thoughtworks' own countermeasure: "reinforcing established practices such as TDD and static analysis, and embedding them directly into coding workflows."
Thoughtworks + GitClearsource, context & caveats
SourceThoughtworks Technology Radar, Vol. 33 (November 2025). On Hold in every appearance since Vol. 31 (October 2024). thoughtworks.com/radar
ContextGitClear puts large-N telemetry behind the duplication half: across 211 million changed lines, 2024 was the first year copy/pasted lines outnumbered moved lines (its proxy for refactoring). In a separate commit scan, 6.66% of the 56,495 commits authored in 2024 contained a duplicated block of five or more lines, against 1.80% of the 41,561 scanned for 2023; the earlier years were backfilled from each repo's largest commits, a sampling bias GitClear says should if anything understate the rise. It rhymes with E2's Stanford case rather than replicating it: Stanford measured rework inside one organization, GitClear measures duplication across its own corpus, which is roughly two-thirds opted-in private repositories and one-third open source, not an inspectable public record. Different instruments, compatible direction. gitclear.com (PDF)
CaveatA qualitative advisory ring, not a metric. Hold in all three appearances (October 2024, April 2025, November 2025), but not carried onto the current Radar (Vol. 34, April 2026); Thoughtworks banners lapsed entries as likely still relevant and says it lacks the bandwidth to re-review previous editions, so the absence is a missing re-review rather than a downgrade.
Resources
Each card above links its own source. These are the things a team reaches for after the working questions: artifacts it can actually run, and a short list worth reading in full. Everything here was fetched and checked. Where something is gated, or assumes a particular vendor's tools, it says so.
Norms: what AI is for, and what it must never touch
- atlassian.com/team-playbook/plays/ai-working-agreementsFree, and no Atlassian account is needed to run it. The Confluence whiteboard template is optional; the play tells you to recreate it by hand if you would rather. Atlassian's Rovo is listed as an optional tool, not a requirement. This is the direct answer to the norms question in Themes 01 and 04.
Rollout and enablement
- github.com/github/ai-adoption-playbookFree, CC BY 4.0, cloneable, pull requests open. Copilot appears only as an example tool. It assumes an organization large enough to have a named owner and an executive sponsor. Its two-tier tool policy (vetted tools, safe for confidential data; everything else, public and non-sensitive only) is a drop-in answer to Theme 04's approved-tool question. This is the only case found of a large adopter packaging its internal program as a free kit rather than publishing a story about it.
- code.claude.com/docs/en/champion-kitFree and public. The examples are Claude Code specific; the structure is not. This is the closest thing found to an answer for Theme 02's question about what would spread the skill fastest.
- atlassian.com/team-playbook/plays/ai-team-microlearning · /ai-demosFree, no vendor assumption. Atlassian's thirteen AI plays include an AI Training Workshop whose name promises more than it delivers here: it is built on Rovo and Confluence AI and aimed at beginner business users in marketing, HR and legal, not at an engineering division.
Measurement
- services.google.com · 2025 DORA AI Capabilities Model (PDF)Free and ungated at the link above. Vendor-neutral, though note the report is licensed CC BY-NC, so non-commercial use only. The dora.dev "Get the report" button routes through a lead-capture form; this link does not.
- dora.dev/ai/capabilities-model/questionsFree, CC BY 4.0, no gate. The fastest way to find out whether anyone in the organization can actually state the policy.
- getdx.com/blog/ai-roi-calculatorFree, no form. Read it against E16 before you quote it: its central input is self-reported hours saved per developer per week, which is precisely the number this document shows people get wrong. Use it to make the cost side legible, not to prove the benefit side. DX's two flagship AI measurement whitepapers are gated behind name, work email and a demo request; the free DX material is in its research and blog sections.
Going deeper
- digitaleconomy.stanford.edu/publication/enterprise-ai-playbookFree PDF. General enterprise rather than software-specific, which is why it is reading and not a play.
- getdx.com/research/measuring-developer-productivity-with-the-dx-core-4 · /blog/revisiting-the-dx-core-4-in-the-age-of-aiFree, no gate.
- sei.cmu.edu · The AI Adoption Maturity Model (PDF)Free and ungated, but 124 pages and pitched at whole-enterprise altitude. The assessment tool that accompanies it is a commercial Accenture engagement, not part of the download: this is a framework, not an instrument. Start with DORA's.
Where this leaves us
The evidence gathered here is uneven and often disagrees, and that spread is itself the finding rather than something to average away. Where the gains landed, they tracked the conditions the tool met more than the tool itself: the shape of the task, the health of the codebase, the strength of verification, the shape of the engineer's role. The role moved from writing toward directing and verifying; verification became the constraint the rest of the work adapts around; the rollouts that worked led with governance and measured as they went; and the failure modes stayed invisible unless someone measured for them.
The studies here run from 2023 to 2026, a stretch over which the tools themselves changed out from under the research, from autocomplete suggestions to agents that plan work and run their own checks. Half the sources are vendors writing about their own products, and every card that leans on one says so. The two randomized trials are small, 52 and 16 participants. And nothing here is longitudinal on organizational adoption, because that research does not exist yet: what exists is measurement studies, vendor guidance, and a handful of published rollouts.