Abstract:
Compute scaling has dominated the conversation with modern language models, leading to an impressive array of algorithms that optimize performance for a given training (and sometimes inference) compute budget. But as compute has grown cheaper and more abundant, data is starting to become a bottleneck, and our ability to exchange computing for data efficiency may be crucial to future model scaling. In this talk, I will discuss some of our recent work on synthetic data and algorithmic approaches to data efficiency, and show that in both cases, classical statistical perspectives based on nonparametric modeling and ensembling bring new insights and empirical benefits to modern questions of scaling and data efficiency.

Biography: 
Tatsunori Hashimoto is an Assistant Professor in the Computer Science Department at Stanford University. Work from his group spans many areas within statistical machine learning and language models including language model post-training, uncertainty quantification, and data selection. He received his Ph.D. at MIT under the supervision of Tommi Jaakkola and David Gifford, and is the recipient of the NSF CAREER, Samsung AI researcher of the year award, a Kavli fellowship as well as best paper awards at ICML, ICLR, and CHI.

Abstract:
Privacy enhancing technologies, such as differentially private stochastic gradient descent (DP-SGD), allow us to access private data without worrying about leaking sensitive information. This is crucial in the modern era of data-centric AI, where all public data has been exhausted and the next frontier models rely on access to high-quality data. A central component in these technologies is private statistical estimation, such as mean estimation and linear regression. We present a series of results where robust statistics and stable algorithms have played critical roles in advancing the state-of-the-art in differentially private statistical estimation. Focusing only on statistical efficiency, we will start with the High-dimensional Propose-Test-Release algorithm (HPTR) that gives optimal sample complexity for a broad range of problems but takes exponential time. Next, we will present how to achieve such an optimal sample complexity in linear-time, for an example of linear regression, with the Insufficient Statistics Perturbation (ISP) algorithm. 

Bio: Sewoong Oh is a Professor at the Paul G. Allen School of Computer Science & Engineering at the University of Washington. Previous to joining University of Washington in 2019, he was at the department of Industrial and Enterprise Systems Engineering at University of Illinois at Urbana-Champaign since 2012. He received his PhD from the department of Electrical Engineering at Stanford University in 2011. Following his PhD, he worked as a postdoctoral researcher at Laboratory for Information and Decision Systems (LIDS) at MIT. Sewoong’s research interest is in foundations of machine learning in topics including private, secure, and robust machine learning and data-centric AI. He was co-awarded the ACM SIGMETRICS best paper award in 2015, NSF CAREER award in 2016, ACM SIGMETRICS rising star award in 2017, and GOOGLE Faculty Research Awards in 2017 and 2020.

TBD