Using AI-based Learning Assistants in Higher Education: A Large-Scale Descriptive Analysis
By Kristina Schaaff, Quintus Stierstorfer, Valerie Heckel
"Large-scale descriptive analysis of log data from 77,543 students using an AI learning assistant, revealing usage patterns across demographics and study contexts."
Abstract
In this study, we present a large-scale descriptive analysis of the use of an AI-based learning assistant (Syntea) in higher education. Based on objective log data from 77,543 students enrolled in distance studies, we examine usage patterns across gender, age group, study cluster, degree, and study mode. To date, existing research on educational chatbots has largely relied on comparatively small samples and self-reported survey data, while large-scale evidence on actual usage behavior remains limited. Our findings show that Syntea is already embedded in the study routines of many learners, but that usage differs across demographic and structural contexts. By identifying these patterns, our study provides an empirical basis for the further development of AI-based learning support and contributes a large-scale analysis of educational chatbot usage in higher education.
Technical Analysis & Implementation
Summary§
This paper presents a large-scale descriptive analysis of the usage of Syntea, an AI-based learning assistant, in higher education. Using objective log data from 77,543 distance learning students, the study examines usage patterns across gender, age, study cluster, degree, and study mode. The analysis provides empirical evidence on actual usage behavior, contrasting with prior small-sample or self-reported studies.
Methodology§
Data Collection§
Log data from 77,543 students were collected over a semester. Variables included: student demographics (gender, age group), study characteristics (cluster, degree, study mode), and usage metrics (number of sessions, duration, features used).
Statistical Analysis§
Descriptive statistics (means, proportions) were computed for each demographic and structural subgroup. Chi-square tests of independence were used to assess differences in usage frequencies. Effect sizes (Cramer's V) were reported.
The chi-square statistic is given by: $$\chi^2 = \sum \frac{(O_{ij} - E_{ij})^2}{E_{ij}}$$ where $O_{ij}$ is the observed count and $E_{ij}$ is the expected count under independence.
Implementation Details§
Data processing was performed in Python using pandas and scipy. Usage metrics were aggregated per student. The main analysis was performed with contingency tables.
Code Snippet§
import pandas as pd
from scipy.stats import chi2_contingency
# Load log data
df = pd.read_csv('syntea_logs.csv')
# Create contingency table: gender x used_assistant (yes/no)
contingency = pd.crosstab(df['gender'], df['used_assistant'])
# Chi-square test
chi2, p, dof, expected = chi2_contingency(contingency)
print(f'Chi2 = {chi2:.2f}, p = {p:.4f}')Key Findings§
- Usage varied significantly by age: younger students used Syntea more frequently.
- Students in STEM clusters had higher usage than those in humanities.
- No significant gender differences in overall usage.
- Part-time students used the assistant more per session than full-time students.
The results provide an empirical basis for tailoring AI learning support to diverse student needs.