Hypothesis testing with information asymmetry

Abstract:
Contemporary scientific research is a distributed, collaborative endeavor, carried out by teams of researchers, regulatory institutions, funding agencies, commercial partners, and scientific bodies, all interacting with each other and facing different incentives. To maintain scientific rigor, statistical methods should acknowledge this state of affairs. To this end, we study hypothesis testing when there is an agent (e.g., a researcher or a pharmaceutical company) with a private prior about an unknown parameter and a principal (e.g., a policymaker or regulator) who wishes to make decisions based on the parameter value. The agent chooses whether to run a statistical trial based on their private prior and then the result of the trial is used by the principal to reach a decision. We show how the principal can conduct statistical inference that leverages the information that is revealed by an agent’s strategic behavior — their choice to run a trial or not. In particular, we show how the principal can design a policy to elucidate partial information about the agent’s private prior beliefs and use this to control the posterior probability of the null. One implication is a simple guideline for the choice of significance threshold in clinical trials: the type-I error level should be set to be strictly less than the cost of the trial divided by the firm’s profit if the trial is successful.

This talk is based on two papers: primarily “Incentive-Theoretic Bayesian Inference for Collaborative Science” and also parts of “Principal-Agent Hypothesis Testing“.  This is joint work with Michael I. Jordan, Michael Sklar, and Jake Soloff.

Bio:
Stephen Bates is an Assistant Professor of AI and Decision-making in the MIT EECS department, where he works to understand uncertainty and reliable decision-making with data. In particular, he develops tools for statistical inference with AI models, data impacted by strategic behavior, and settings with distribution shift. In addition, he works on applications in the life sciences and sustainability.

Previously, he was a postdoctoral researcher with Michael I. Jordan in the UC Berkeley Statistics and EECS departments. He completed his Ph.D. in the Stanford Department of Statistics, advised by Emmanuel Candès, where he recieved the Theodore W. Anderson Theory of Statistics Dissertation Award.