[P] AdamWR Full Keras + TF-Keras Implementation Available
A followup to original post (pasted shortened below), with major changes; release v1.1:
- Run-based weight decay normalization scheme, normalizing over arbitrary # of iterations independent of LR scheduler (e.g. over all epochs)
- Full compatibility with TensorFlow 2.0.0 and Keras 2.3.0 (
- Full compatibility with TensorFlow 1.14.0 and Keras 2.2.5 (
- Also compatible w/ TF 1.13.0 & 1.15.0, Keras 2.2.3-2.2.4
The latest Lookahead optimizer paper, co-authored by Geoffrey Hinton, used AdamW as its base optimizer, and noted it performing superior to plain Adam.
NadamW and SGDW included, along their WR (Warm Restart) counterparts – with cosine annealing learning rate schedule, and per layer learning rate multipliers (useful for pretraining). All optimizers are well-tested, and for me have yielded 3-4% F1-score improvements in already-tuned models for seizure classification.