Hi,

I’m making an LSTM that takes a list of same-size vectors as input. These vectors are encodings of frames in a video, and I want the LSTM to output an encoding of the entire video. To get this encoding, I am just taking the last hidden state and feeding it through a linear layer.

My issue is the hidden state seems to be converging on some fixed vector after a couple of time steps. It seems like the LSTM is forgetting previous states and entering a loop. What could cause this behavior? Is there a nice way to fix this?

Thanks