[D] NeurIPS 2019 Bengio Schmidhuber Meta-Learning Fiasco
The recent reddit post Yoshua Bengio talks about what’s next for deep learning links to an interview with Bengio. User u/panties_in_my_ass got many upvotes for this comment:
Spectrum: What’s the key to that kind of adaptability?***
Somewhere, on some laptop, Schmidhuber is screaming at his monitor right now.
because he introduced meta-learning 4 years before Bengio:
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: The meta-meta-… hook. Diploma thesis, Tech Univ. Munich, 1987.
Then Bengio gave his NeurIPS 2019 talk. Slide 71 says:
Meta-learning or learning to learn (Bengio et al 1991; Schmidhuber 1992)
What a childish slight… The Schmidhuber 1987 paper is clearly labeled and established and as a nasty slight he juxtaposes his paper against Schmidhuber with his preceding it by a year almost doing the opposite of giving him credit.
GANs were actually mentioned in the Turing laudation, it’s both funny and sad that Yoshua Bengio got a Turing award for a principle that Jurgen invented decades before him
and that section 3 of Schmidhuber’s post on their miraculous year 1990-1991 is actually about his former student Sepp Hochreiter and Bengio:
(In 1994, others published results [VAN2] essentially identical to the 1991 vanishing gradient results of Sepp [VAN1]. Even after a common publication [VAN3], the first author of reference [VAN2] published papers (e.g., [VAN4]) that cited only his own 1994 paper but not Sepp’s original work.)
So Bengio republished at least 3 important ideas from Schmidhuber’s lab without giving credit: meta-learning, vanishing gradients, GANs. What’s going on?