Nature Retracts Controversial Study Claiming ChatGPT Boosts Student Learning

Nature Retracts Study on ChatGPT’s Educational Benefits

Nature has retracted a paper that asserted AI, specifically ChatGPT, had a positive impact on student learning outcomes. The original study, titled "The effect of ChatGPT on students’ learning performance, learning perception, and higher-order thinking: insights from a meta-analysis," was published in May 2025 by researchers Jin Wang and Wenxiang Fan from Hangzhou Normal University in China.

The paper was a meta-analysis, synthesizing data from 51 research studies published between November 2022 and February 2025. It claimed to find that ChatGPT had a "large or moderately positive impact" on students’ learning performance, learning perception, and higher-order thinking.

Reasons for Retraction

Nature issued a retraction note stating,

"The Editor has decided to retract this paper owing to concerns regarding discrepancies in the meta-analysis. These issues ultimately undermine the confidence the Editor can place in the validity of the analysis and resulting conclusions. The authors have not responded to correspondence regarding this retraction."

As of the retraction, the researchers had not provided a public response to requests for comment.

Rapid Rise and Fall of the Study

Ben Williamson, a senior lecturer in digital education at the University of Edinburgh, noted in an email that he first observed the paper shortly after its publication on May 6, 2025. He explained that it quickly gained traction on social media platforms like LinkedIn, where it was widely shared as evidence supporting the benefits of AI in education.

Within a month, the paper had been accessed online nearly 400,000 times and achieved an Altmetric score of 365. Its dissemination was amplified by influential individuals sharing it on platforms such as X (formerly Twitter) and Bluesky, positioning it as credible evidence for promoting AI tools in educational settings.

Methodological Flaws Highlighted Before Retraction

The retraction note did not elaborate on the specific discrepancies in the meta-analysis. However, a 2025 study published in the European Journal of Education Policy and Practice had already identified flaws in Wang and Fan’s methodology. The study, titled "What counts as evidence in AI ED: Towards Science-for-Policy 3.0," was authored by Ilkka Tuomi.

Tuomi’s research argued that meta-analyses in AI education often rely on peer-reviewed papers without adequately assessing their quality or the robustness of their findings. He stated,

"Existing empirical evidence on AIED [AI in education] suggests some positive effects, but a closer look reveals methodological and conceptual problems and leads to the conclusion that existing evidence should not be used to guide policy or practice."

Tuomi further criticized the heterogeneity of studies included in such meta-analyses, noting that variations in study quality and data integrity render quantitative results unreliable. He specifically referenced another viral study about ChatGPT’s impact on learning, stating,

"Despite its apparent methodological quality and apparent rigour, the heterogeneity of the analysed studies makes the quantitative results of the [referenced] meta-analysis meaningless."

Tuomi’s critique underscores broader concerns about the reliability of research claiming to demonstrate AI’s benefits in education, particularly when such studies are rapidly disseminated and widely cited.