In a recent project I’ve found that I need to use gradient clipping to stop my model from suddenly NaNing out after training for a while, and it also seems to have a positive regularizing effect. I’ve read that the clipping cutoff should be trained as a hyperparameter, but is there any rule of thumb or results on setting the learning rate when using gradient clipping? It seems like if the gradient clipping cutoff is too low, most gradient steps will be of the same magnitude.