​

Feature importance along an ALS patient’s time series. The border between the red shade (output increasing features) and the blue shade (output decreasing features) represents the model’s output for each timestamp.

I’ve been working on interpreting recurrent neural networks, having made some changes on the SHAP package to adapt them to this type of model, on PyTorch. In order to share this, I’ve recently posted an article on Medium explaining the core concepts and showing examples of how it works on multivariate time series data. You can read it here: https://towardsdatascience.com/interpreting-recurrent-neural-networks-on-multivariate-time-series-ebec0edb8f5a

Also, feel free to ask me any questions, or to give some suggestions, if you’re interested in this topic 🙂