How good is your model? Guilt-free interactive data analysis.

Reliable tools for model selection and validation are indispensable in almost all applications of machine learning and statistics. Decades of theory support a widely used set of techniques, such as holdout sets, bootstrapping and cross validation methods. Yet, much of the theory breaks down in the now common situation where the data analyst works interactively with the data, iteratively choosing which methods to use by probing the same data many times. A good example are data science competitions in which hundreds of analysts repeatedly score their models on the same holdout set with the danger of overfitting to the holdout itself.
In this talk, we first discuss that, in general, it is computationally hard to prevent overfitting in interactive data analysis. Achieving what is possible in light of the hardness result, we will then see a powerful reusable holdout method that can be used many times without losing the guarantees of fresh data. We conclude with a simple and practical algorithm for maintaining a provably accurate leaderboard in machine learning competitions and demonstrate its practical utility through experiments on real submission files from a Kaggle competition.

Based on joint works with Avrim Blum, Cynthia Dwork, Vitaly Feldman, Toni Pitassi, Aaron Roth, Omer Reingold and Jon Ullman.