Reimagining Student Success Prediction: Applying LLMs in Educational AI with XAI

Since the conception of Large Language Models (LLMs), their areas of application have increased significantly over time. This is due to their nature of being able to perform natural language processing (NLP) tasks (like question answering, text generation, text summarization, text classification etc.), which gives them flexibility in a multitude of spaces, including in Educational AI (EdAI). Despite their incredible wide range of use, LLMs are typically applied to generative AI, from text to image generation.This paper aims to apply LLMs for a classification task in EdAI, by reproposing the original PreSS (Predicting Student Success) model which makes use of more traditional Machine Learning (ML) algorithms for predicting CS1 students at risk of failing or dropping out. There are two main goals for this work: the first is to identify the best and most accurate method to re-purpose LLMs for a classification task; the second is to explore and access the explainability of the model outputs. For the former we investigate different techniques for using LLMs like Few-Shot Prompting, Fine-Tuning and Transfer Learning using Gemma 2B as base model along with two different kind of prompting techniques. For the latter we focus on attention scores of LLMs transformers, aiming to understanding what are the most important features that the model considers for generating the response. The obtained results are then compared with the previous PreSS model to evaluate whether LLMs can outperform traditional ML algorithms: this paper finds that Naïve Bayes still outperforms all the others, once again confirmed as the best algorithm for predicting student success.