Research watchEducationNew
University trial finds students offered a GPT-4o study assistant got lower grades, in unreviewed paper
A randomized trial with 2,379 University of Maryland undergraduates found that offering a course-integrated AI tutor lowered final grades and course-platform activity, but few students used it and the working paper has not been peer reviewed.
Evidence: 4 sources · 2 independent reports · 2 research papers · 3 open questions
What happened
- The working paper, dated 30 September, was posted in the EdWorkingPapers series hosted by Brown University's Annenberg Institute.
- In fall 2025, 30 instructors teaching 2,379 undergraduates were randomly assigned to offer or not offer a course-grounded study assistant built on OpenAI's GPT-4o.
- Within matched sections of the same course, access was associated with final grades 0.37 standard deviations lower, about 4 points on a 100-point scale.
- Participation on the course learning platform fell by 0.90 standard deviations, and estimated grade losses were larger for first-generation students.
Why it matters
It is one of the largest randomized tests of an AI tutor in real university courses, and its result runs against smaller trials that found gains from carefully designed tutors.
What is confirmed and what is claimed
Confirmed
- The study is a working paper by Jing Liu, Tracy Sweet and colleagues, posted on EdWorkingPapers and SSRN; it has not been peer reviewed. 2
- The trial randomized instructors, not individual students, across 30 instructors and 2,379 undergraduates in fall 2025 (working paper; also reported by EdTech Innovation Hub). 3
- The authors report the clearest negative grade effect in the sample of identical courses; across the broader sample the evidence on grades is weaker.
Claimed, not independently verified
- The authors suggest the tool may have displaced existing coursework rather than adding to it; this is their interpretation, not a tested finding.
- Secondary summaries report that only about 15% of students offered the tool ever used it and that most requests asked for direct answers; we have not checked these figures against the full paper.
Researchers at the University of Maryland ran a randomized trial in fall 2025 in which some instructors offered students a "Virtual Study Assistant" built on GPT-4o and grounded in each course's own materials, while comparable instructors did not. The working paper reports that students in sections with access ended the term with lower final grades and less activity on the course learning platform.
The design matters for how to read the result. Randomization was at the instructor level, so the comparison is between whole sections. The authors' strongest estimate comes from sections of identical courses; across the full sample, commentators note, the grade effect is less certain, while the drop in platform participation is clearer.
Context
Earlier randomized studies, including a Harvard physics trial published in Scientific Reports, found gains from AI tutors built to guide students step by step rather than give answers. The Maryland tool was a general study assistant, and uptake was low. Lead author Jing Liu has said the results do not show that purpose-built tutoring tools are ineffective. Taken together, the studies suggest design and how a tool fits into a course may matter more than whether AI is present at all.
This is a working paper, not a peer-reviewed article. See our guides to preprints and peer review and what the evidence says about AI in education.
What to watch
- Whether the paper is revised or accepted by a journal, and whether the uptake figures are confirmed.
- Replications at other universities, especially with tutors that withhold direct answers.
- Whether universities that license campus-wide chatbots begin measuring learning outcomes rather than usage.
What we don’t know yet
- Whether the result holds up in peer review and in courses or institutions beyond one university.
- How a tool used by so few students could lower average grades, and which mechanism explains it.
- Whether a tutor designed to withhold answers, rather than a general study assistant, would produce a different result.
Evidence and sources
- Preprint (not peer reviewed) The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial (EdWorkingPaper 26-1598)
- Preprint (not peer reviewed) The Effects of Course-integrated AI Tutoring on Student Performance and Engagement (SSRN listing)
- Independent reporting AI tutor trial finds lower grades and student engagement
- Independent reporting AI tutoring lowers student performance and engagement. Or does it?