TL;DR
A landmark MIT study (Noy & Zhang, Science 2023) had hundreds of college-educated professionals do real work-style writing tasks, with half given ChatGPT. The AI group finished about 40% faster and scored 18% higher on quality. Lower performers gained the most, narrowing the skill gap. The catch: drafting time shrank, editing time grew, and fact-checking still falls on you.
A study conducted by MIT researchers explored the productivity effects of generative AI (ChatGPT) on professional writing tasks. Findings support the potential of generative AI to augment individual productivity and address broader issues of inequality in workplace performance.
Methodology of the experiment
To assess the impact of ChatGPT on productivity and the quality of professional writing tasks, 444 college-educated professionals were recruited to participate in this study. The participants came from diverse backgrounds including marketing, consulting, and management. They were assigned two writing tasks related to their occupation, ensuring that professional scenarios from their work were replicated. Each task was for 20 - 30 minutes and included - writing cover letters for grant applications, emails about organizational restructuring, and plans for customer engagement strategies.
How was the experiment conducted and evaluated?
Participants were randomly divided into two groups. The control group completed their tasks using traditional methods, while the treatment group used ChatGPT only for their second task. The treatment group signed up for ChatGPT and was instructed on how to use it. To ensure unbiased and rigorous evaluation, the assessment was done by blind evaluation, i.e., the evaluators were not aware of which submissions were completed with AI assistance. The tasks were rated based on various quality metrics, including clarity, innovation, and overall effectiveness. Also, to measure any gains in efficiency, the time spent on each task was meticulously recorded. In addition to objective outcomes, the experiment also assessed participants' satisfaction and how they perceived AI impacting their work through follow-up surveys. This well-rounded approach provided a deeper understanding of how AI integration affects professional environments. Some questions asked to participants were:
- The treatment group was asked how much they would willingly pay for ChatGPT per month.
- To rank themselves between 1 to 3 on their skills in communication (writing and speaking), problem-solving, and creativity.
Source
Results
Here are the key findings from the experiment:
- Time efficiency: The treatment group was able to complete their assignments 10 minutes faster, reducing the average time spent on tasks by about 37% - demonstrating a substantial efficiency gain.
- Quality improvement: An 18% improvement in the quality of work produced by the treatment group suggested that ChatGPT can speed up the writing process as well as enhance the quality of responses.
- Reduction in productivity inequality: Lower-performing individuals from the first task showed greater improvements when using ChatGPT for the second task, indicating that ChatGPT may serve as a leveling tool in professional settings.
- Sustained use and perceptions: A follow-up survey revealed continued use of ChatGPT among participants, with 33% of those in the treatment group incorporating ChatGPT into their regular professional tasks. Furthermore, participants reported an increase in job satisfaction and a positive shift in their attitudes toward AI and automation technologies.
Takeaway- 1: Effect of ChatGPT on task structure
The integration of ChatGPT notably altered the structure of writing tasks and time allocated:
- Before treatment: Initially, participants allocated approximately 25% of their time to brainstorming, 50% to drafting, and the remaining 25% to editing their work.
- After treatment: With ChatGPT, the time spent on drafting was more than halved, while editing time more than doubled.
This shift indicates that while the drafting phase became significantly less time-consuming due to AI assistance, participants chose to allocate more time to refining and editing the response - suggesting a shift in task focus from creation to modification.
Takeaway - 2: Benefits of ChatGPT and skill demand
ChatGPT has the potential to help those with poor writing skills but strong idea-generation skills, expand their occupational choices.
Findings:
- ChatGPT subscription: Regardless of the participants’ written skills, they were willing to pay 0.5% of their monthly salary for the ChatGPT subscription.
- Grade gains: Improvement in written skill was not significant. This means people with poor writing skills do not experience exceptionally high levels of grade gains with ChatGPT.
Explore how to use ChatGPT to create a marketing strategy.
Takeaway - 3: Perception of automation after using ChatGPT
Amongst the treatment group, before the study, 30% of them had never heard of ChatGPT and 70% had never used it, making this experiment their first encounter with the AI tool. After using ChatGPT for writing tasks, we surveyed their views on how AI impacts their professions on a scale of 1 to 10. The results showed:
- Concerns about AI and jobs: A slight increase in worry about AI replacing human jobs, with concern rising by 0.26 standard deviations (p = 0.006).
- Optimism about AI and productivity: Participants showed increased optimism about AI's ability to enhance productivity, with excitement rising by 0.39 standard deviations (p < 0.001).
- General outlook on AI: Overall optimism about the future of AI increased moderately, by about 0.20 standard deviations (p = 0.037).
Key learnings
Overall, there is no doubt that ChatGPT and other generative AI technologies can improve productivity and work quality in professional writing tasks. It helps low-skilled workers reduce the time spent on completing tasks. This allows high-skilled employees maintain their work quality and finish work faster.
Even business leaders, hiring managers, and candidates are increasingly accepting of AI-generated content. ChatGPT can aid in brainstorming and drafting, yet it necessitates more time for editing, indicating a shift towards enhancing writers' editing skills within organizations. Furthermore, the content produced by language models requires factual verification, pointing to the need for roles specializing in fact-checking and research. There are also concerns about job security with AI, however, optimism about AI improving productivity and future advancements remains strong. This nuanced perspective on AI in the workplace suggests a thoughtful integration of AI technologies, balancing efficiency gains with the broader implications for employment and workplace dynamics. As AI continues to evolve, ongoing research will be essential to maximize its benefits while addressing potential challenges. Do you agree with the findings of this study? Let us know your thoughts.
Sources
Frequently Asked Questions
How much does generative AI actually improve professional writing quality?
In the MIT study, professionals who used ChatGPT scored about 18% higher on quality than those who did not, graded by blind evaluators on clarity, writing, and content. That gain came alongside a roughly 40% drop in time spent. So it is not a trade-off of speed versus quality; both moved in the right direction at once.
Does ChatGPT make writers faster, and by how much?
Yes. The treatment group finished their writing tasks roughly 40% faster, about 10 to 11 minutes quicker on a 20 to 30 minute task. Most of the saving came from drafting: rough-draft time was more than halved because ChatGPT produced a usable first pass almost instantly.
Does AI replace writing skill or just speed things up?
The study found ChatGPT mostly substituted for effort rather than upgrading underlying skill. Time shifted away from drafting and toward editing, which more than doubled, so the human job moved from creation to refinement. People with weak writing skills did not suddenly become great writers; they just got to a decent draft faster and spent their time improving it.
Does generative AI reduce the gap between strong and weak writers?
It narrows it. Lower-performing participants in the first task improved the most when given ChatGPT for the second, which compressed the quality distribution across workers. The researchers describe this as ChatGPT acting as a leveling tool, lifting the floor more than the ceiling in professional writing.
Did using ChatGPT change how workers feel about AI taking jobs?
Both worry and optimism rose at once. Concern about AI replacing human jobs increased modestly, while excitement about AI boosting productivity rose more sharply and overall optimism about AI ticked up. Workers exposed to ChatGPT were also about twice as likely to report still using it in their real job two weeks later.
