[R] Piecewise Strong Convexity of Neural Networks
Video summary: https://www.youtube.com/watch?v=z89BTMQGVn
Earlier related work: https://arxiv.org/abs/1607.04917 (piecewise convexity)
I am not the author. This paper will be presented at NeurIPS this month and exposes some convexity results about piece-wise linear nns under the least squares loss – namely piecewise strong-convexity & the non-existance of differentiable local maxima. The approach is a spectral analysis of the Hessian and weights of the nn. The result is a relatively attractive convergence estimate for sgd.
I guess this provides some more motivation for studying techniques like ADMM which have convergence properties for some classes of piece-wise functions and can exploit lipschitz cts gradients. Nice work!
submitted by /u/i-heart-turtles