Since the vocab file has over 30,000 entries a softmax vector would be gigantic.

But a similarity lookup would recquire embedding the vocab file myself, correct? In a way similar to the BERT embedding, which we can’t get because the entire point of BERT is an embedding depndent on the text itself.