Optimizing Deep Q-Learning Experience Replay with SHAP Explanations: Exploring Minimum Experience Replay Buffer Sizes in Reinforcement Learning

Explainable Reinforcement Learning (xRL) faces challenges in debugging and interpreting Deep Reinforcement Learning (DRL) models. A lack of understanding for internal components like Experience Replay, which samples and stores data from the environment, risks burdening resources. This paper presents an xRL-based Deep Q-Learning (DQL) system using SHAP (SHapley Additive exPlanations) to explain input feature contributions. Data is sampled from Experience Replay, creating SHAP Heatmaps to understand how it influences the neural network Q-value approximator's actions. The xRL-based system aids in determining the smallest Experience Replay size for 23 simulations of varying complexities. It contributes an xRL optimization method, alongside traditional approaches, for tuning the Experience Replay size hyperparameter. This visual and creative approach achieves over 40% reduction in Experience Replay size for 18 of the 23 tested simulations, smaller than the commonly used sizes of 1 million transitions or 90% of total environment transitions.