Skip to main content

Blog

Learn About Our Meetup

5000+ Members

MEETUPS

LEARN, CONNECT, SHARE

Join our meetup, learn, connect, share, and get to know your Toronto AI community. 

JOB POSTINGS

INDEED POSTINGS

Browse through the latest deep learning, ai, machine learning postings from Indeed for the GTA.

CONTACT

CONNECT WITH US

Are you looking to sponsor space, be a speaker, or volunteer, feel free to give us a shout.

[R] A Fair Comparison Study of XLNet and BERT with Large Models

https://medium.com/@xlnet.team/a-fair-comparison-study-of-xlnet-and-bert-with-large-models-5a4257f59dc0

We are the authors of XLNet. We conducted a fair comparison study of XLNet and BERT with large models. In this study, we ensure that almost every possible hyperparameter is the same for the training recipes of both BERT and XLNet, using the same training data.

We have the following interesting observations among others:

  1. Trained on the same data with an almost identical training recipe, XLNet outperforms BERT by a sizable margin on all the datasets.
  2. The gains of training on 10x more data are smaller than the gains of switching from BERT to XLNet on 8 out of 11 benchmarks.

submitted by /u/kimiyoung
[link] [comments]