Thirty-nine meta-analyses claim artificial intelligence improves education, yet an audit of 14 found fundamental flaws—errors that peer review missed despite their publication in top journals and thousands of citations. The evidence supporting AI’s effectiveness in classrooms is far less reliable than it appears.
Meta-analyses, which aggregate research to identify trends, are supposed to provide trustworthy conclusions. Instead, they have delivered a distorted picture of AI’s role in education. Justin Reich, an associate professor at MIT and director of its Teaching Systems Lab, described the findings as “not trustworthy evidence.” His assessment follows a detailed review by Patrick O’Neill, a community college professor at Ivy Tech in Indiana, who examined 14 of the 39 meta-analyses published by Springer Nature, Wiley, SAGE, and Taylor & Francis. O’Neill uncovered recurring errors in methodology and reporting.
These flawed studies have already influenced policy and classroom practice. One meta-analysis, published in a Nature journal in May 2025, was retracted nearly a year later after Norwegian scholars identified issues. The audit revealed “impossible statistics,” “hallucinated references,” and conclusions drawn from studies that did not measure learning outcomes.
Some studies compared students using AI tools to those receiving no instruction—a comparison too flawed to assess effectiveness. Others measured outcomes like reduced anxiety instead of academic progress. One experiment found nursing students performed better after three hours of AI-assisted study than peers who had no study sessions at all, a comparison that proved nothing about AI’s educational value.
Related Post: Libraries remain trusted oases amid fading public trust
Beth Tipton, a meta-analysis expert at Northwestern University, dismissed the broad approach as “crazy.” Tipton compared it to conducting a meta-analysis of math education, asking, “Is math education effective?” She emphasized that the question must be narrowed to specific approaches, such as whether spatial learning interventions improve math skills, and similarly, AI’s use must be specified before conducting a meta-analysis of it. She asked, “Do we mean people using [educational] apps and programs that include AI in them, like Khan Academy? Do we mean giving students access to Claude?” The question is too broad.
How meta-analyses skewed AI’s education hype
The audit also exposed publication bias: two-thirds of the meta-analyses explicitly sought studies showing AI’s benefits, excluding neutral or negative findings. O’Neill’s own rigorous meta-analysis found that 44 well-designed experiments produced results ranging from helpful to harmful, with an average effect near zero.
Even this neutral outcome raises concerns. The impact of AI depends entirely on implementation, yet most high-quality studies do not take place in K-12 schools. Susanna Loeb, a Stanford education professor, noted that her team reviewed nearly 900 studies but found only 20 of sufficient quality for inclusion in a recent summary. The rest were either flawed or irrelevant to real-world classrooms.
Education research lacks medicine’s rigorous oversight
The lack of oversight in education research creates a gap that medicine’s regulatory systems fill. The FDA evaluates drug trials before approval, and insurers assess treatment efficacy before coverage. Education has no equivalent, leaving teachers and policymakers to rely on studies that may lack rigor. Tipton receives daily requests to review meta-analyses but lacks the time to properly vet them. Peer review in education often lacks the expertise or resources to scrutinize statistical methods.
Related Post: UK transnational education faces quality recognition issues
O’Neill’s examination uncovered hallucinated references, citations to AI research published before the technology was publicly available or links redirecting to unrelated studies. In one case, a meta-analysis cited a paper on generative AI that did not exist, yet the error went unnoticed during peer review. These mistakes appeared across multiple studies, suggesting a lack of rigorous vetting at the journal level.
Peer reviewers in education often lack the time or expertise to verify whether underlying research measured what it claimed. Tipton, who fields daily requests to evaluate these studies, said she typically spends hours confirming whether experiments defined “learning” properly. Some measured engagement or satisfaction instead of academic performance, while others used flawed comparisons, such as AI-assisted groups versus no-intervention controls. For example, a study showing nursing students outperforming peers who had no study time proved nothing about AI’s educational value, yet such comparisons were common in the reviewed meta-analyses.
Only 20 studies meet real-world classroom standards
Loeb’s team at Stanford’s AI Hub for Education identified only 20 studies of sufficient quality out of nearly 900 reviewed, with just 32 additional high-quality studies added since. Most research on AI in education occurs outside U.S. K-12 classrooms, limiting real-world applicability. Even well-designed studies often focus on narrow applications, such as tutoring apps or chatbots, rather than broad AI integration.
