We are Changing our Developer Productivity Experiment Design - METR

Authors: Joel Becker, Nate Rush, Tom Cunningham, David Rein, Khalid Mahamud Date: 2026-02-24 URL: https://metr.org/blog/2026-02-24-uplift-update/

Abstract / Summary

METR follow-up study on developer productivity with late-2025 AI tools. Notes severe selection bias: developers refusing to work in AI-disallowed condition, 30-50% selective task submission, pay rate dropped from $150/hr to $50/hr. Original developer subset showed raw -18% completion time (speedup) but confidence interval spanned -38% to +9%. Newly recruited devs showed -4% (-15% to +9%). Concludes data is only very weak evidence for true effect size.