Interpretability is essential for trust and accountability in machine learning, particularly in sensitive domains such as healthcare, finance, and environmental decision-making. However, most explanation techniques provide only pointwise feature attributions and neglect epistemic uncertainty, limiting their reliability for high-stakes use. This article presents DSExplainer, a framework that integrates SHAP values with Dempster–Shafer theory (DST) to represent explanatory evidence as intervals of belief (
) and plausibility (
), distinguishing what a model knows from what it considers possible. The method maps SHAP contributions into basic probability assignments, separates the direction of influence, and fuses evidence across bootstrap replicas to produce signed intervals for explanatory hypotheses that capture both magnitude and epistemic reliability. Evaluation on three canonical tabular datasets demonstrates that DSExplainer preserves SHAP’s additive interpretability while augmenting it with explicit uncertainty information. Explanations with high
and narrow
intervals correspond to robust insights, while wide intervals reveal ambiguity. By incorporating uncertainty as a first-class component of explanations, DSExplainer advances explainable AI toward an uncertainty-aware and auditable paradigm and can be integrated into existing interpretability workflows. Source code will be released upon publication.
