"Google, on the one hand, is telling us that their engineers are about 10% more productive as a result of using AI. They are, of course, a very engineering productivity-focused organization in the first place. However, we also had this now infamous METR Study, that had some problems with it. It was a small sampling of engineers, but it was still a pretty well-executed study that showed a 19% decrease in overall productivity in this particular experiment."
"Again, problems with this study. There were some engineers who had never used Cursor before, which was the tool that they were using in this study. What I think is pretty interesting is that every engineer in this study felt like they were more productive, like their qualitative data actually showed, no, I think I actually am getting more done. The data bore out that that wasn't true."
"We need to manage perceptions and reality. We have to measure, and we have to really be diligent about how this technology is working for us."
AI’s impact on engineering productivity is unclear and varies across evidence. Google reports about 10% higher productivity for engineers using AI, while the METR study found a 19% decrease in overall productivity in a controlled experiment. The METR study had limitations, including a small sample and engineers who had never used Cursor before. Despite these issues, qualitative feedback from participants indicated they felt more productive and believed they were getting more done. Measured outcomes did not match those perceptions, creating a need to manage expectations and to measure performance diligently to understand how AI works in specific environments.
Read at InfoQ
Unable to calculate read time
Collection
[
|
...
]