I really wish this study had chosen to focus on more useful endpoints like revenue, profitability, customer retention, conversion rate, etc....at a minimum, don't look at PRs merged without also looking at bugs in production code, number of incident reports, features shipped, debt log canceled, etc. Even if you only want to consider merged PRs, you should weigh the productivity of the reviewer and whether they are also 24% boosted or else it's just an accounting trick.
The problem that I'm finding in my own work on figuring out the impact of AI is that there's just no reliable way to connect things directly to AI usage. Most of the tooling does things like "The user used AI on the same day that they opened this PR, therefore we'll assume the AI was used to write the code in the PR." In a mature AI-driven org that might be true, but in the rollout phase of an AI experiment it absolutely isn't.
Until there's a good way to fix that gap in the data any measure of AI impact is going to be horribly flawed.
The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.
24% of increased productivity (yes, this is assuming of course that the “proxy” of merged PRs reflects productivity) is actually a pretty big deal. Given the salary of developers, this translates to tens of thousands of dollars per year, per developer.
My guess is they used # of PRs as a measure as it’s easy to obtain, while other measures are hard, may be due to other factors, etc.
FWIW I saw a similar number for myself, around 30% more PRs in the last 6 months, compared to the 6 months before that (I picked up agentic coding around at the start of the year). And a similar increase for closed issues.
In my case this clearly doesn’t translate to as much value for the organization, or rather, it’s hard to say, as many of those PRs were things I wouldn’t even have done without AI support. This means they were low priority. However, many were of the cleanup/refactor type, so they might result in speedups later.
I've also seen an increase in merged PRs, but it coincided with developers opening smaller PRs. In other words, AI made devs break work down more so they opened more PRs for the same work. That's still good, because smaller increments are better, but there' no actual increase in coding productivity, and it means the context-switching burden from review work went up e.g developers slowed down in a different area.
24% PRs isn't 24% more productivity. Lines of code isn't productivity, and neither is CLs landed. What's the feature velocity of the team? How much time is being spent on rollbacks, outage responses, etc.?
Here's a quick hack to triple your PRs landed: Land a PR, then land ANOTHER PR undoing that one when you realize it was full of bugs, then land the PR again once you realize management doesn't care about quality, they just care about the number of PRs landed.
The 24% more PRs was merged, not «landed». Presumably after human code review (but I didn’t check the paper…) Not sure about this particular study, but I «think» my own productivity (coding wise) has improved by more than 30%. In many ways I wouldn’t even have started on many of the things I’ve completed these last six months because I would have viewed their effort to be insurmountable within my time budget as a professor (with all my other duties).
Given that Microsoft's overhead to actually landing features has, from what I've heard, long been more about dodging the cross-org and inter-org "guns" famously depicted in [0] and [1] than coding time... a 24% increase in merged PRs is massive!
> In my case this clearly doesn’t translate to as much value for the organization, or rather, it’s hard to say, as many of those PRs were things I wouldn’t even have done without AI support.
That's not necessarily bad. It could be a sign of effective prioritization. If you're good at working on the most important thing, and suddenly find yourself 24% more productive, what extra are you working on? The things that wouldn't have quite made the cut before.
To me the biggest gain I see is that you take the programmers out of the loop. Instead of formulating your ideas to start a project and then acquiring the resources to do a single iteration on it which may take months if not years, now many specialists "just do it" and do several such iterations in a single day. Afterwards they may still go the ordinary route via programmers but on a completely different level and a lot of fruitless work is frontloaded and 100x cheaper. This doesn't show up in any such statistic.
So, the question is not "Does it make our programmers more productive?" but "Does it make our organization faster?".
I think number of features released to customers (not behind a feature flag or still being rolled out, but fully rolled out). And number of bug fixes (only those reported by customers).
Also just in general, customer satisfaction, acquisition, conversion, retention, etc.
Number of completed org-level roadmap items, org-level goals achievement rate, and so on.
I also think a good one would be seeing an increase in meeting estimation, like if project was estimated to take X days with Y devs, does the use of AI increased how often you met or beat those estimates in actual time/dev effort?
And you'd want to compare that against prior years, where no AI was used, within the same org, or try going 1 quarter without AI and another with and compare quarter to quarter.
I also think something along these lines is the correct answer. It can be hard to pin down an exact metric because once you start optimizing for a metric it tends to not be a good measure of the original thing anymore. But in general I think it comes down to some measure of feature velocity combined with a counter metric on support/maintenance burden.
"Number of PRs merged" seems like "number of lines of code" wearing a trenchcoat, and I thought we all agreed back in the 90s that number of lines of code was a terrible measure of software productivity...
Feature velocity is another that's extremely easy to game. My company is trying this right now: instead of measuring PRs or lines of code, we are measuring number of customer facing features shipped. Well guess what? Everything is now a customer-facing feature. You did a big internal code refactor and data migration? Well guess what, that's a customer feature now because it unlocks future such and such. Deploy a new piece of infra? Customer feature. Dev tool improvement? Customer feature.
IMO trying to measure productivity gains is a fools' errand. The only thing that matters is CSAT, escaped defects, retention, cost per contact, and other metrics that measure actual business outcomes.
Incredibly hard problem, but METR had a good method. They had people estimate how long a task would take (before knowing whether AI would be used), and then randomly assigned each task to “with AI” or “without AI.” When the data was in, they compared actual/estimate ratios of the two populations.
(Presumably, they used a t-test that only compared people against themselves.)
Interestingly, for that study (released in 2025), participants self-rated themselves as 20% more productive, but were measured as being 19% less productive.
It'll probably take a really good product built by a profitable company evangelizing an AI workflow with reproducible examples dating back a few years.
We use a tool called Weave (I believe YC 25?) that analyzes PRs for "expert units of work" and shows lift from AI tools. My understanding is they have their own proprietary model that assesses the difficulty of each PR. I find the organization level view and pivots useful and aligned with intuitive expectations.
Yeah welcome to the state of the art in measuring AI impact. I have contacts at a few different larger tech companies that are fully AI pilled (the one I work at included) and every single one has forgotten the last 50 years of lessons in measuring dev productivity and hyperfocused on PR throughput and token usage.
Fun fact: all the data I've seen suggests at most a 50% uplift in those metrics. And that's at the top percentiles. Its very clear that the already high performers see the greatest uplift but anyone in that meaty middle will only see incremental gains.
more features doesn't mean anything if they don't translate into economic value, refactoring prs and dependency updates are easily automated now and it saves alot of productivity, but if you compare the costs and economic value gained it might not actually justify enterprise token spending.
Highly doubt. Pace for merging requests have not improved and teams at MSFT are terrible at reviewing said PRs. Longer PRs and more frequent requests were clearly creating more friction.
My personal experience was that I saw AI adopters opening and merging more PRs but I also spent more of my day reviewing PRs than I did in the past, in order to keep up with the flood of new stuff. Reconcile that with MSFT's repeated layoffs however you like.
The number of merged PRs increased. But what does that actually tell us?
There is no comparison of the complexity, or depth of those PRs with work previously done without agents. The authors themselves acknowledge that PR count rewards small, frequent PRs and that whether the additional output justifies its cost remains unanswered. We do not know whether end-to-end feature delivery became faster, whether the additional output justified the cost, or what happened to defects, maintainability, review effort, and rework.
The conclusion seems to be little more than: Microsoft engineers are using AI tools, Microsoft is spending more on AI usage, and AI vendors are earning more subscription and token revenue.
More PRs are not necessarily more value. Without quality, delivery-time, and ROI metrics, the claimed productivity gain is largely meaningless.
Especially in conditions where AI assisted PRs are tracked for performance.
I had some extra usage on claude yesterday and put several ultra workflows to look for code cleanup opportunities. It made -40 tasks and PRs just like that. This is 10% of all PRs/commits in the project history, but the before and after is arguably meaningless.
Exactly. Once AI-assisted PR count becomes a performance metric, the metric is no longer merely incomplete—it actively creates an incentive to generate more tasks and PRs
>281 billion tokens. Using the least expensive version of Claude Opus 4.6, which costs $5 for every million tokens, that one
user alone could have cost Meta more than $1.4 million.
And that is based on the pricing that they sell a dollar for less than a dollar so the costs may be much much more.
I really wish this study had chosen to focus on more useful endpoints like revenue, profitability, customer retention, conversion rate, etc....at a minimum, don't look at PRs merged without also looking at bugs in production code, number of incident reports, features shipped, debt log canceled, etc. Even if you only want to consider merged PRs, you should weigh the productivity of the reviewer and whether they are also 24% boosted or else it's just an accounting trick.
The problem that I'm finding in my own work on figuring out the impact of AI is that there's just no reliable way to connect things directly to AI usage. Most of the tooling does things like "The user used AI on the same day that they opened this PR, therefore we'll assume the AI was used to write the code in the PR." In a mature AI-driven org that might be true, but in the rollout phase of an AI experiment it absolutely isn't.
Until there's a good way to fix that gap in the data any measure of AI impact is going to be horribly flawed.
Completely disable the usage of AI for time period t. Compare the impact on whatever before t and during t.
In any complex organization that might take several quarters if not more, though.
Possibly but this is not related to AI
How can you compare something when you can't see or measure the outcome for an undetermined amount of time?
The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.
24% of increased productivity (yes, this is assuming of course that the “proxy” of merged PRs reflects productivity) is actually a pretty big deal. Given the salary of developers, this translates to tens of thousands of dollars per year, per developer.
My guess is they used # of PRs as a measure as it’s easy to obtain, while other measures are hard, may be due to other factors, etc.
FWIW I saw a similar number for myself, around 30% more PRs in the last 6 months, compared to the 6 months before that (I picked up agentic coding around at the start of the year). And a similar increase for closed issues.
In my case this clearly doesn’t translate to as much value for the organization, or rather, it’s hard to say, as many of those PRs were things I wouldn’t even have done without AI support. This means they were low priority. However, many were of the cleanup/refactor type, so they might result in speedups later.
I've also seen an increase in merged PRs, but it coincided with developers opening smaller PRs. In other words, AI made devs break work down more so they opened more PRs for the same work. That's still good, because smaller increments are better, but there' no actual increase in coding productivity, and it means the context-switching burden from review work went up e.g developers slowed down in a different area.
I'm yet to be convinced that number of PRs as a productivity metric is any less flawed than counting lines of code.
I can believe - easily - that there's a real uplift here, but attaching any meaning to the 24% number at all is a massive overstep.
24% PRs isn't 24% more productivity. Lines of code isn't productivity, and neither is CLs landed. What's the feature velocity of the team? How much time is being spent on rollbacks, outage responses, etc.?
Here's a quick hack to triple your PRs landed: Land a PR, then land ANOTHER PR undoing that one when you realize it was full of bugs, then land the PR again once you realize management doesn't care about quality, they just care about the number of PRs landed.
The 24% more PRs was merged, not «landed». Presumably after human code review (but I didn’t check the paper…) Not sure about this particular study, but I «think» my own productivity (coding wise) has improved by more than 30%. In many ways I wouldn’t even have started on many of the things I’ve completed these last six months because I would have viewed their effort to be insurmountable within my time budget as a professor (with all my other duties).
Given that Microsoft's overhead to actually landing features has, from what I've heard, long been more about dodging the cross-org and inter-org "guns" famously depicted in [0] and [1] than coding time... a 24% increase in merged PRs is massive!
[0] https://static.ma.nu/publications/images/2013.07.12_new_york...
[1] https://newsletter.pragmaticengineer.com/i/138252015/drawing...
> In my case this clearly doesn’t translate to as much value for the organization, or rather, it’s hard to say, as many of those PRs were things I wouldn’t even have done without AI support.
That's not necessarily bad. It could be a sign of effective prioritization. If you're good at working on the most important thing, and suddenly find yourself 24% more productive, what extra are you working on? The things that wouldn't have quite made the cut before.
To me the biggest gain I see is that you take the programmers out of the loop. Instead of formulating your ideas to start a project and then acquiring the resources to do a single iteration on it which may take months if not years, now many specialists "just do it" and do several such iterations in a single day. Afterwards they may still go the ordinary route via programmers but on a completely different level and a lot of fruitless work is frontloaded and 100x cheaper. This doesn't show up in any such statistic.
So, the question is not "Does it make our programmers more productive?" but "Does it make our organization faster?".
What kind of metric would you trust for measuring organization gains?
I think number of features released to customers (not behind a feature flag or still being rolled out, but fully rolled out). And number of bug fixes (only those reported by customers).
Also just in general, customer satisfaction, acquisition, conversion, retention, etc.
Number of completed org-level roadmap items, org-level goals achievement rate, and so on.
I also think a good one would be seeing an increase in meeting estimation, like if project was estimated to take X days with Y devs, does the use of AI increased how often you met or beat those estimates in actual time/dev effort?
And you'd want to compare that against prior years, where no AI was used, within the same org, or try going 1 quarter without AI and another with and compare quarter to quarter.
I also think something along these lines is the correct answer. It can be hard to pin down an exact metric because once you start optimizing for a metric it tends to not be a good measure of the original thing anymore. But in general I think it comes down to some measure of feature velocity combined with a counter metric on support/maintenance burden.
"Number of PRs merged" seems like "number of lines of code" wearing a trenchcoat, and I thought we all agreed back in the 90s that number of lines of code was a terrible measure of software productivity...
Feature velocity is another that's extremely easy to game. My company is trying this right now: instead of measuring PRs or lines of code, we are measuring number of customer facing features shipped. Well guess what? Everything is now a customer-facing feature. You did a big internal code refactor and data migration? Well guess what, that's a customer feature now because it unlocks future such and such. Deploy a new piece of infra? Customer feature. Dev tool improvement? Customer feature.
IMO trying to measure productivity gains is a fools' errand. The only thing that matters is CSAT, escaped defects, retention, cost per contact, and other metrics that measure actual business outcomes.
Incredibly hard problem, but METR had a good method. They had people estimate how long a task would take (before knowing whether AI would be used), and then randomly assigned each task to “with AI” or “without AI.” When the data was in, they compared actual/estimate ratios of the two populations.
(Presumably, they used a t-test that only compared people against themselves.)
Interestingly, for that study (released in 2025), participants self-rated themselves as 20% more productive, but were measured as being 19% less productive.
Microsoft itself has a system for measuring this which outlines a few example metrics: https://queue.acm.org/detail.cfm?id=3819080
There was a nice talk about this by one of the author's at this year's BUILD conference: https://build.microsoft.com/en-US/sessions/BRK210
Great read, thanks for sharing!
It'll probably take a really good product built by a profitable company evangelizing an AI workflow with reproducible examples dating back a few years.
We use a tool called Weave (I believe YC 25?) that analyzes PRs for "expert units of work" and shows lift from AI tools. My understanding is they have their own proprietary model that assesses the difficulty of each PR. I find the organization level view and pivots useful and aligned with intuitive expectations.
This sounds like voodoo.
Engineering headcount.
Yeah welcome to the state of the art in measuring AI impact. I have contacts at a few different larger tech companies that are fully AI pilled (the one I work at included) and every single one has forgotten the last 50 years of lessons in measuring dev productivity and hyperfocused on PR throughput and token usage.
Fun fact: all the data I've seen suggests at most a 50% uplift in those metrics. And that's at the top percentiles. Its very clear that the already high performers see the greatest uplift but anyone in that meaty middle will only see incremental gains.
To be fair they do acknowledge this directly in the abstract. Kinda hard to have a good heuristic for this stuff, what would you propose?
more features doesn't mean anything if they don't translate into economic value, refactoring prs and dependency updates are easily automated now and it saves alot of productivity, but if you compare the costs and economic value gained it might not actually justify enterprise token spending.
Highly doubt. Pace for merging requests have not improved and teams at MSFT are terrible at reviewing said PRs. Longer PRs and more frequent requests were clearly creating more friction.
Dependabot PR merges :rocket: :rocket:
My personal experience was that I saw AI adopters opening and merging more PRs but I also spent more of my day reviewing PRs than I did in the past, in order to keep up with the flood of new stuff. Reconcile that with MSFT's repeated layoffs however you like.
The number of merged PRs increased. But what does that actually tell us?
There is no comparison of the complexity, or depth of those PRs with work previously done without agents. The authors themselves acknowledge that PR count rewards small, frequent PRs and that whether the additional output justifies its cost remains unanswered. We do not know whether end-to-end feature delivery became faster, whether the additional output justified the cost, or what happened to defects, maintainability, review effort, and rework.
The conclusion seems to be little more than: Microsoft engineers are using AI tools, Microsoft is spending more on AI usage, and AI vendors are earning more subscription and token revenue.
More PRs are not necessarily more value. Without quality, delivery-time, and ROI metrics, the claimed productivity gain is largely meaningless.
Especially in conditions where AI assisted PRs are tracked for performance.
I had some extra usage on claude yesterday and put several ultra workflows to look for code cleanup opportunities. It made -40 tasks and PRs just like that. This is 10% of all PRs/commits in the project history, but the before and after is arguably meaningless.
Exactly. Once AI-assisted PR count becomes a performance metric, the metric is no longer merely incomplete—it actively creates an incentive to generate more tasks and PRs
>281 billion tokens. Using the least expensive version of Claude Opus 4.6, which costs $5 for every million tokens, that one user alone could have cost Meta more than $1.4 million.
And that is based on the pricing that they sell a dollar for less than a dollar so the costs may be much much more.