A Protocol for Continual Explanation of SHAP

Continual Learning trains a model on a stream of data, with the aim of learning new information without forgetting previous knowledge. We study the behavior of SHAP values explanations in Continual Learning and propose an evaluation protocol to robustly assess the change of explanations in Class-Incremental scenarios. We observed that, while Replay strategies enforce the stability of SHAP values in feedforward/convolutional models, they are not able to do the same with fully-trained recurrent models. We show that alternative approaches, like randomized recurrent models, are more effective in keeping the explanations stable over time.