[P] StyleGAN – understanding the learning rate values
In the original StyleGAN implementation, the learning rates are set to the following values (see line 52 here):
- 0.001 from 4 to 128 pixels
- 0.0015 for 256 pixels
- 0.002 for 512 pixels
- 0.003 for 1024 pixels
One thing I don’t understand is why the learning rate increases with the pixel size… are the two somehow correlated? Also, is there a rule of thumb to choose how to scale the learning rates with the batch size? Thanks!