Rickles, Heppen, Allensworth, Sorensen, and Walters (2018) compare two credit recovery programs with a null hypothesis of equal effects. Upon failing to reject the null, they implicitly accept the null. However, not rejecting the null is not equivalent to accepting the null, a confusion common to a range of research, which I highlight.
Bacher-HicksA.ChinM. J.KaneT. J.StaigerD. O. (2017). An evaluation of bias in three measures of teacher quality: Value-added, classroom observations, and student surveys (Working Paper No. 23478). Cambridge, MA: National Bureau of Economic Research.
2.
BettingerE. P.LongB. T. (2010). Does cheaper mean better? The impact of using adjunct instructors on student outcomes. The Review of Economics and Statistics, 92(3), 598–613. doi:10.1162/REST_a_00014
3.
BloomH. S.MichalopoulosC.HillC. J.LeiY. (2002). Can nonexperimental comparison group methods match the findings from a random assignment evaluation of mandatory welfare-to-work programs? (MDRC Working Papers on Research Methodology). Retrieved from https://eric.ed.gov/?id=ED471814
4.
ChettyR.FriedmanJ. N.RockoffJ. E. (2014). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review, 104(9), 2593–2632. doi:10.1257/aer.104.9.2593
5.
JacobR.SomersM.-A.ZhuP.BloomH. (2016). The validity of the comparative interrupted time series design for evaluating the effect of school-level interventions. Evaluation Review, 40(3), 167–198. doi:10.1177/0193841X16663414
6.
JohnsonS. D.AragonS. R.ShaikN.Palma-RivasN. (2000). Comparative analysis of learner satisfaction and learning outcomes in online and face-to-face learning environments. Journal of Interactive Learning Research, 11(1), 29–49.
7.
KaneT. J.StaigerD. O. (2008). Estimating teacher impacts on student achievement: An experimental evaluation (Working Paper No. 14607). Cambridge, MA: National Bureau of Economic Research.
8.
RicklesJ.HeppenJ. B.AllensworthE.SorensenN.WaltersK. (2018). Online credit recovery and the path to on-time high school graduation. Educational Researcher, 47(8), 481–491. doi:10.3102/0013189X18788054
9.
SteinbergM. P.GarrettR. (2016). Classroom composition and measured teacher performance: What do teacher observation scores really measure?Educational Evaluation and Policy Analysis, 38(2), 293–317. doi:10.3102/0162373715616249
10.
SteinerP. M.WongV. C. (2018). Assessing correspondence between experimental and nonexperimental estimates in within-study comparisons. Evaluation Review, 42(2), 214–217. doi:10.1177/0193841X18773807
11.
XuZ.OzekU.CorritoreM. (2012). Portability of teacher effectiveness across school settings (Working Paper 77). Retrieved from https://eric.ed.gov/?id=ED533217
