Abstract:
Training data determines what neural networks can learn—but can we predict the geometry of learned representations directly from data statistics?

We  present a framework that addresses this question for sufficiently large, well-trained neural networks. The key idea is a coarse but predictive abstraction of such networks as log-bilinear softmax models, whose implicit regularization we can analyze. Within this framework, we show how label imbalance shapes representation geometry and, for language models, how word and context representations organize into structures characterized by a sparse-plus-low-rank decomposition of co-occurrence statistics.

Log-bilinear softmax models arise as a canonical non-convex extension of well-understood convex linear models, yet their gradient-descent implicit bias had until now remained unknown. We describe recent progress in characterizing this bias, enabled by an inconspicuous Hadamard-based initialization that effectively diagonalizes the softmax nonlinearity and yields tractable reduced dynamics. This provides the first definitive link between implicit bias theory and neural collapse geometries.

Bio:
Christos Thrampoulidis is an Associate Professor at the Department of Electrical and Computer Engineering (ECE) at the University of British Columbia (UBC). From July 2018 until December 2020, he served as an Assistant Professor (tenure-track) at the ECE Department at the University of California, Santa Barbara. Prior to that, he spent two years as a Postdoctoral Researcher at the Research Laboratory of Electronics (RLE) at MIT. Dr. Thrampoulidis received a M.Sc. and a Ph.D. degree in EE in 2012 and 2016, respectively, both from Caltech, with a minor in Applied and Computational Mathematics. In 2011, he received a Diploma in ECE from the University of Patras, Greece.