Emergent outlier subspaces in high-dimensional stochastic gradient descent

Abstract:  It has been empirically observed that the spectrum of neural network Hessians after training have a bulk concentrated near zero, and a few outlier eigenvalues. Moreover, the eigenspaces associated to these outliers have been associated to a low-dimensional subspace in which most of the training occurs, and this implicit low-dimensional structure has been used as a heuristic for the success of high-dimensional classification. We will describe recent rigorous results in this direction for the Hessian spectrum over the course of the training by SGD in high-dimensional classification tasks with one and two-layer networks. We focus on the separation of outlier eigenvalues from the bulk, and subsequent crystallization of the outlier eigenvectors. Based on joint work with Ben Arous, Huang, and Jagannath.

Bio: Reza Gheissari is an assistant professor of mathematics at Northwestern University. Prior to joining Northwestern, he obtained his Ph.D. at NYU’s Courant Institute and was a Miller Postdoctoral Fellow at UC Berkeley. His research area is probability theory, with particular interest in out-of-equilibrium behavior of Markov chains, and relations to sampling, optimization, and learning problems in high dimensions.