On Overfitting in Analysis of Symmetrically Predicted Endogenous Subgroups From Randomized Experimental Samples

Abstract

Using exogenous characteristics to identify endogenous subgroups, the approach discussed in this method note creates symmetric subsets within treatment and control groups, allowing the analysis to take advantage of an experimental design. In order to maintain treatment–control symmetry, however, prior work has posited that it is necessary to use a prediction subsample, separate from the subsample used for impact estimation in order to prevent overfitting from affecting impact estimates. Doing so diminishes sample size—both for prediction and analysis—and so has costs. This article delves into this topic to consider the conditions under which overfitting occurs and to characterize the effects of overfitting in terms of bias and variance. It suggests a strategy for preserving the full sample size in all phases of the analysis. The research uses Monte Carlo simulation to directly measure overfitting, identify the circumstances that should concern us, and to explore possible recommended practices and future research implications.