The experiment was conducted by GitHub and Microsoft by Sid Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer in 2022 and published on the GitHub blog and Microsoft Research.
Aim of the study: Can GitHub Copilot - a generative AI tool designed to assist in software development tasks - make developers more efficient? In particular, the study explores how the tool impacts the speed and quality of task completion among professional software developers.
Methodology:
- Candidates: A controlled experiment was conducted to measure how long it took 95 professional developers recruited via Upwork
- Task: Participants were asked to implement an HTTP server in JavaScript. They were divided into two groups: one group completed the implementation with the assistance of GitHub Copilot and the other didn’t.
- Process: The group that was using GitHub Copilot received a brief video tutorial on using it. Performance metrics included task success rate and completion time. These parameters were evaluated using GitHub Classroom to ensure accurate timing and assessment.
Results and findings:
- Productivity: The group that used GitHub Copilot was able to complete the task 56% faster than the one that didn’t use the AI tool.
- Task completion rate: The group using GitHub Copilot achieved a higher task completion rate of 78%, compared to the 70% completion rate of the group that didn’t use AI assistance.
- Sense of fulfillment: 60-75% of AI users had higher job satisfaction and were less frustrated when completing the task or coding. This sense of accomplishment allowed them to focus more on the work.
- Bridging the skill gap: During the heterogeneous treatment effects, it was revealed that developers with less programming experience, those dedicating more hours to coding daily, and older developers, experienced significantly greater benefits from AI assistance (GitHub Copilot).