https://medium.com/@xlnet.team/a-fair-comparison-study-of-xlnet-and-bert-with-large-models-5a4257f59dc0

We are the authors of XLNet. We conducted a fair comparison study of XLNet and BERT with large models. In this study, we ensure that almost every possible hyperparameter is the same for the training recipes of both BERT and XLNet, using the same training data.

We have the following interesting observations among others:

1. Trained on the same data with an almost identical training recipe, XLNet outperforms BERT by a sizable margin on all the datasets.
2. The gains of training on 10x more data are smaller than the gains of switching from BERT to XLNet on 8 out of 11 benchmarks.