What Works for Whom? A Bayesian Approach to Channeling Big Data Streams for Public Program Evaluation

Abstract

In the coming years, public programs will capture even more and richer data than they do now, including data from web-based tools used by participants in employment services, from tablet-based educational curricula, and from electronic health records for Medicaid beneficiaries. Program evaluators seeking to take full advantage of these data streams will require novel statistical methods, such as Bayesian approach. A Bayesian approach to randomized program evaluations efficiently identifies what works for whom. The Bayesian approach design adapts to accumulating evidence: Over the course of an evaluation, more study subjects are allocated to treatment arms that are more promising, given the specific subgroup from which each subject comes. We identify conditions under which there is more than a 90% chance that inference from the Bayesian adaptive design is superior to inference from a standard design, using less than one third the sample size.