Profit-Sensitive Benchmarking of Post-hoc Explainability in Customer Churn

Customer churn models on tabular data often favour boosted ensembles for predictive performance, but their adoption depends on explanations that are reliable and actionable. We benchmark post-hoc explainability for churn by applying LIME and SHAP to high-performing tree ensembles and comparing them against inherently interpretable baselines (logistic regression, decision tree) on two public telco datasets. Evaluation combines standard predictive metrics (accuracy, F1, AUC) with business- and profit-sensitive measures (top-decile lift, Expected Maximum Profit (EMP), and customer-level e-Profits), while explanation quality is assessed using four quantitative metrics: fidelity, robustness, simplicity, and consistency. Results show that SHAP, particularly with boosted ensembles, achieves near-perfect fidelity and stable robustness, whereas LIME yields simpler but less faithful explanations. Profit-based metrics reorder model rankings relative to accuracy and F1, with ensembles that do not lead on F1 still delivering higher e-Profits, underscoring profitability as a key selection criterion. Overall, Gradient Boosting emerges as a balanced choice across predictive and profit-sensitive measures, and SHAP provides dependable explanations for such models, offering a practical alternative to relying solely on inherently interpretable classifiers. This study delivers a compact benchmark that reports model performance and explanation assessment side-by-side, integrating technical and business perspectives without conflation. Source code: https://github.com/Awaismanzoor/aics-xai.