Stress testing of econometric results using archived code for replication

Abstract

Replications merely check whether the results reported by authors are independently verifiable, not whether they are reliable, robust and stable. Statistical inference deals with specification and sampling errors whereas subject matter knowledge is needed to avoid errors in interpretation of the model. Vinod and Ullah [24] suggested perturbing the data beyond the available digits to evaluate the numerical stability of model results. This paper extends the idea into a simple algorithm to create random perturbations for checking perturbation sensitivity (=α $ _{p}$ ) of a model, its software and interpretations. We illustrate the proposed algorithm with archived replication examples from the Journal of Money Credit and Banking (JMCB) and Journal of Applied Econometrics. Some journals (against our sentiment) still allow some parts of the code to be hidden in a black box. Unfortunately, black boxes may also hide shortcuts and dishonesty, unless all black box users are required to post the α $_{p}$ developed here. There is a new requirement at Econometrica to report the sensitivity of empirical results, and other journals may soon follow. Ourα $ _{p}$ provides a tool for meeting such requirements.