I have this phenomenon that I don’t understand.

My loss is the cross entropy of the softmax prediction with one-hot encoded labels. The accuracy is a simple argmax comparison.

While the accuracy stays the same, the loss is heavily increasing. I would guess the reason is that all outputs get more equal but the correct label stays the highest values.

But what could be the reason for such a behavior? Why is that happening? What can I do about it?