Study Suggests More Guidance Needed in Use of AI Course Assistants

As colleges nationwide seek to integrate artificial intelligence into education, a new University of Maryland study suggests that greater understanding is needed about how to employ AI tools for teaching and learning before the technology can take on an increased role in classrooms.

The study, led by College of Education and Division of Information Technology researchers, found some negative effects after a trial of course-integrated virtual study assistants (VSAs) was conducted at UMD during the Fall 2025 semester. Posted this month as a working paper, it is the first large-scale, randomized study of how such chatbot-based tools impact learning across multiple disciplines in higher education. The study included 30 instructors and about 2,500 undergraduate students from 11 of 12 schools and colleges at UMD. 

Forty courses currently have access to these tools; instructors in all of them have been informed of the study findings, and provided with guidance on best practices for the use of AI tools in instruction to help support effective teaching and learning. 

Among the study’s most significant findings: Access to the VSA was associated with lower final grades and substantially lower interaction with course materials through ELMS-Canvas, with larger estimated effects for first-generation students. Student engagement with the VSAs was also low — only about 15% who had access used them. The findings also suggest that introducing the VSA may have reduced student attendance at office hours and time spent meeting with instructors and peers. 

Students mostly turned to the tools to ask for answers, said lead author Jing Liu, an associate professor of education policy. As UMD developed the tools, the expectation had been that students would also take advantage of tutoring capabilities like creation of test quizzes and getting feedback, rather than primarily using it for fact finding. That might make study time seem superficially more efficient, but may bypass the “productive struggle” students need for learning, said Liu, who is also director of UMD’s Center for Educational Data Science and Innovation.

While the paper raises some new questions, its senior author, Senior Vice President and Provost Jennifer King Rice, said its findings emphasize the need for instructors to maintain a “human-centered” approach to teaching, supplemented by the effective use of technology.  

Maryland Today sat down with Rice to discuss the study and where UMD goes from here:

As an institution that's committed to the responsible and ethical use of AI, we're determined to employ it in ways that can effectively support teaching and learning. We believe deeply in evidence-based practice and won’t rush AI technology into general use in the classroom until we know it works.

So we decided that rather than roll this tool out to everyone just because we could, we should slow down and take the time to study it first. Last summer, I reached out to faculty members who have expertise in program evaluation and AI applications in education, and pulled them in alongside the team from DIT. We designed a study that used randomization to better understand the impact of the virtual study assistant on student learning and student engagement in course materials.

I was certainly concerned that our study found that having access to the VSA was associated with lower grades and less engagement with the course materials available through ELMS-Canvas. Those effects were found to be more pronounced for first-generation students, which speaks to the possibility that AI tools could exacerbate inequities in learning outcomes — that is particularly troubling to me. 

And, of course, I am concerned that VSA access was associated with lower attendance at office hours and time spent meeting with instructors and peers. Those social interactions are an important part of learning and development. Finally, it didn’t appear to matter if students actually used the VSA, because most students in the classes given access did not — but they still were subject to these effects. It may be the case that giving access to the VSA signaled an acceptance of AI tools more broadly, which prompted students to use other available AI agents.

It’s clear from these results that simply giving people access to these tools doesn't mean they will work as intended. This study indicates a need for more evidence to inform how AI tools can be designed, deployed and used in ways that effectively support student learning, and how their effects may vary across students from different backgrounds. 

I don’t think that’s true, or that the study says the AI chatbots have an inherently negative effect on student learning and success. What it does tell us is that making AI tools available without providing faculty or students with clear guidance was not effective. The study suggests that how students use AI matters — in this case students tended to use AI for fact-finding rather than for tutoring and engaged learning activities. The finding that students with VSA access were less engaged with course materials also suggests that direct use of those learning resources have value, and that AI should supplement not supplant them. 

Ultimately, the study doesn’t tell us not to use AI tools, but gives us data-based results that can inform our next steps to best support our students. 

The study results provide us with some clear considerations, as well as some questions to ask. For example, we should consider ensuring that the VSA provides regular feedback to  instructors on how their students are using the tool, so they can make adjustments to their instruction if needed. 

We should also ask ourselves how we can deploy the VSA so students are not simply using it to retrieve quick answers, but are instead using it to engage more intentionally with the course materials, peers and instructors to maximize learning and success. And providing evidence-based guidance to instructors and students on the use of AI in teaching and learning is important. We hope our study contributes to that evidence base.

We have a responsibility in higher education to take the time to carefully evaluate new technologies and understand their impact. Rather than trying to keep up with the rapid pace of AI adoption across sectors, we need to be willing to pause, study the tools, look carefully at the evidence, and ask whether what we’re doing is actually achieving what we intended – in this case, supporting student learning.