LLMs in Open and Closed Book Examinations in a Final Year Applied Machine Learning Course (Early Findings).

This research has three prongs, with each comparing open- and closed-book exam questions across six years (2017-2023) in a final year undergraduate applied machine learning course. First, the authors evaluated the performance of numerous LLMs, compared to student performance, and comparing open and closed book exams. Second, at a micro level, the examination questions and categories for which LLMs were most and least effective were compared. This level of analysis is rarely if ever, discussed in the literature. The research finally investigates LLM detection techniques, specifically their efficacy in identifying replies created wholly by an LLM. It considers both raw LLM outputs and LLM outputs that have been tampered with by students, with an emphasis on academic integrity. This study is a staff-student research collaboration, featuring contributions from eight academic professionals and six students.