Testing the I.I.D. assumption online

Abstract: Mainstream machine learning, despite its recent successes, has a serious drawback: while its state-of-the-art algorithms often produce excellent predictions, they do not provide measures of their accuracy and reliability that would be both practically useful and provably valid. Conformal prediction adapts rank tests, popular in nonparametric statistics, to testing the IID assumption (the observations being independent and identically distributed). This gives us practical measures, provably valid under the IID assumption, of the accuracy and reliability of predictions produced by traditional and recent machine-learning algorithms. An interesting application of conformal prediction is the existence of _exchangeability martingales_, i.e., random processes that are martingales under any exchangeable probability measure. In particular, they are martingales whenever the observations are IID. In this talk I will discuss the construction of exchangeability martingales and their use for different kinds of change detection, including detecting a point at which the IID assumption becomes violated and detecting concept shift. This may be useful for deciding when a prediction algorithm should be retrained.

Bio: Vladimir Vovk is Professor of Computer Science at Royal Holloway, University of London. His research interests include machine learning and the foundations of probability and statistics. He was one of the founders of prediction with expert advice, an area of machine learning avoiding making any statistical assumptions about the data. In 2001 he and Glenn Shafer published a book (“Probability and Finance: It’s Only a Game”, New York: Wiley) on new game-theoretic foundations of probability; the sequel (“Game-theoretic Foundations for Probability and Finance”, Hoboken, NJ: Wiley) appeared in 2019. His second book (“Algorithmic Learning in a Random World”, New York: Springer, 2005), co-authored with Alex Gammerman and Glenn Shafer, is the first monograph on conformal prediction, method of machine learning that provides provably valid measures of confidence for their predictions. His current research centres on the theory and applications of conformal prediction and applications of game-theoretic probability to statistics, machine learning, and finance.