Educators' Evaluation of Explanation Types in XAI for Higher Education Dropout

First-year student dropout rates represent a global challenge, particularly in Computer Science programs. In response, the Predict Student Success (PreSS) tool was developed over two decades using Naïve Bayes, a black box machine learning model, to identify students at risk of early withdrawal or academic failure. Given growing concerns about the use of black box models in high-stakes domains such as education, this study builds on former research by comparing the Naïve Bayes model to three glass box AI models: Decision Tree, Explainable Boosting Machine, and Automatic Piecewise Linear Regression. These models were used to generate four types of explainable AI visualisations: Feature Importance, Similar and Contrastive Examples, Decision Rules, and Counterfactuals. Educators evaluated these visualisations through a qualitative survey. Results indicate that glass box models can perform competitively with Naïve Bayes, and that contrastive explanation types were most preferred among educators. Open-ended responses underscore the importance of clear user guidance when implementing such tools.