GANs are usually trained on pretty high variance datasets (Imagenet, CIFAR, etc). Intuitively, these datasets are hard to model – they encompass a wide range of classes, and even within these classes, there’s a ton of variance between and within images.

A lower variance distribution, however, seems like it would be a strictly easier distribution to model. It seems natural to me that if I had a low variance dataset, I could take off-the-shelf GAN architectures and parameters that are able to model high-variance datasets, plug in my own data, and get nice outputs. For instance, let’s say I had thousands of overhead views of a crop field, which changed in minor ways based on the season, crop quality, etc – I’m intuiting that the variance of this data would be much smaller and consequently easier to model (this is not my actual problem – it’s even lower variance between+within images – but let’s just use this as an example).

I’m finding that in practice, however, this isn’t really true. Something about modeling this low variance dataset is proving to be hard for these off-the-shelf GANs. It could be a quality of the dataset aside from the low-variance, but some other tests I’ve run (artificially increasing the variance of the images) have me thinking otherwise.

I’m thinking that perhaps the methods/architectures to optimize GAN training are probably different across different dataset types – specifically, low vs high variance datasets. Any thoughts?