Evaluating a black-box algorithm: stability, risk, and model comparisons
November 15 @ 11:00 am - 12:00 pm
Rina Foygel Barber, University of Chicago
E18-304
Abstract:
When we run a complex algorithm on real data, it is standard to use a holdout set, or a cross-validation strategy, to evaluate its behavior and performance. When we do so, are we learning information about the algorithm itself, or only about the particular fitted model(s) that this particular data set produced? In this talk, we will establish fundamental hardness results on the problem of empirically evaluating properties of a black-box algorithm, such as its stability and its average risk, in the distribution-free setting.
Bio:
Rina Foygel Barber is the Louis Block Professor in the Department of Statistics at the University of Chicago. Before starting at U of C, she was a NSF postdoctoral fellow in the Department of Statistics at Stanford University, and received her PhD in Statistics at the University of Chicago. Her research focuses on the theoretical foundations of statistical problems in estimation, prediction, and inference. In many modern settings, classical methods may not be reliable due to high dimensionality, failure of model assumptions, or other challenges. She works on distribution-free inference methods such as conformal prediction, and on developing hardness results to establish what types of questions can or cannot be solved with distribution-free methods. She is also interested in multiple testing methods, in algorithmic stability, and shape-constrained inference. She also collaborates on modeling and optimization problems in image reconstruction for medical imaging.
