I am aware of the famous example of Embedding(King) – Embedding(Man) + Embedding(Woman) = Embedding(Queen). From this example, we can say that the characteristic of “royalty” has been understood.

I guess in a way I am trying to interpret the hidden layer neurons which might not always have meaning.

I have looked into techniques like SHAP and LIME but I’m still to plug the concepts together.