Secondary Analysis of a Major Societal Experiment

Abstract

This work presents a methodological critique and secondary analysis of a large scale societal experiment performed by Kellaghan, et al. (1982) that studied the educational and psychological effects of the introduction of standardized achievement/ability testing in elementary schools in Ireland. The author considers the study by Kellaghan et al. to be of high quality; however, given the potential importance of this study, a methodological critique and secondary analysis are appropriate to provide an independent assessment of validity of findings, confirmation or disconfirmation of results, and a checkfor important effects that may have been missed by the original investigators. Some points of methodological criticism discussed involve the questionable validity of cross-lagged correlation analyses, spillover of treatments, insufficient measurement of treatment effects, and Hawthorne effects. Secondary analyses not performed by the original researchers found evidence of effects not indicated in the original report for the study. Some of the more important reanalyses found evidence of positive effects of immediate and delayed test feedback on student ability and achievement, and indications of positive effects of testing and test feedback on group variance of achievement possibly due to labeling processes and differential motivational effects on high versus low initial scorers.