Research involving a Georgia State University course using a chatbot to keep students connected showed improved grades and retention rates.

Chatbot boosts student performance at Georgia State

Research involving a Georgia State University course using a chatbot to keep students connected showed improved grades and retention rates

Page and colleagues Katharine Meyer of Brown and Tyler Walsh of Georgia State presented the findings last week, in a talk titled “Let’s Chat: Chatbot Nudging for Improved Course Performance,” during a conference of the Association for Education Finance and Policy in Denver.

The study was conducted in coordination with Michael Evans, senior lecturer in political science at Georgia State and the faculty coordinator for the course, taken by about 3,000 students each semester to satisfy degree requirements in American government. Students participating in the fall 2021 trial were taking the course fully online.

Evans said the course typically has a relatively high rate of what are known as DFWs (students receiving a grade of D, F or withdrawing) — around 15 percent prior to the COVID-19 pandemic and as high as 25 percent since the onset.

“It’s a challenge because a lot of times students are just learning how to be college students,” Evans said. “We get every type of student, in terms of their preparation.”

Students already get email reminders about assignments, quizzes and exams, but Evans said students receiving text messages from the “PolsPounce” chatbot were getting an extra, direct reminder as well as the opportunity to text questions back, through PolsPounce, to their professor. Some reminders were targeted to students who didn’t have much time left to complete a reading assignment by the deadline. Students were also reminded when an exam was coming up and given an opportunity to take a short practice quiz to gauge their preparedness.

“Clearly this was having a bigger impact than just the emails, and part of that was because they were receiving the message,” Evans said. “I did not expect we’d get this overwhelming evidence that the chatbot is this effective.”

According to Page’s study results, students who received the PolsPounce messages earned grades of B or above at a rate 16 percent higher than those not on the chatbot.

The average final grade in the class for first-generation students in the control group was 64, while similar students receiving the messages earned a final grade an average of 11 points higher.

Among students who were not in their first year of college, those who received the chatbot messages were more likely to pass the class and less likely to drop their other Georgia State classes. They also did better in their other classes, earning fall term grade point averages 0.28 points higher.

Students participating in the study who displayed lower academic performance in high school were more likely to attend supplemental instruction, more likely to earn a B or higher in the class and complete more credits for the semester if they received the chatbot messages.

“While we’ve seen that students from all kinds of different demographic groups benefit, what we’ve seen consistently is that students who struggle the most overall benefit the most from the chatbot intervention,” Renick said. “Students who are low income, the first in their families to go to college or from underrepresented minorities often lack a support system at home or familiarity with how to get help in college. They are the ones to benefit the most by having this tool in their hand 24/7.”

Laura Ascione