Restricted accessResearch articleFirst published online 2022-8
Regression Discontinuity Designs With an Ordinal Running Variable: Evaluating the Effects of Extended Time Accommodations for English-Language Learners
Regression discontinuity (RD) designs are commonly used for program evaluation with continuous treatment assignment variables. But in practice, treatment assignment is frequently based on ordinal variables. In this study, we propose an RD design with an ordinal running variable to assess the effects of extended time accommodations (ETA) for English-language learners (ELLs). ETA eligibility is determined by ordinal ELL English-proficiency categories of National Assessment of Educational Progress data. We discuss the identification and estimation of the average treatment effect (ATE), intent-to-treat effect, and the local ATE at the cutoff. We also propose a series of sensitivity analyses to probe the effect estimates’ robustness to the choices of scaling functions and cutoff scores and remaining confounding.
BaileyP.EmadA.HuoH.LeeM.LiaoY.LishinskiA.NguyenT.XieQ.YuJ.ZhangT.BuehleE.C’deBacaR. (2020). EdSurvey: Analysis of NCES education survey and assessment data [Computer software manual]. Retrieved fromhttps://CRAN.R-project.org/package=EdSurvey (R package version 2.5.0)
3.
BatesD.MächlerM.BolkerB.WalkerS. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01
4.
BergnerY.von DavierA. A. (2018). Process data in NAEP: Past, present, and future. Journal of Educational and Behavioral Statistics, 44(6), 706–732. https://doi.org/10.3102/1076998618784700
5.
BradleyR.KattiS.CoonsI. J. (1962). Optimal scaling for ordered categories. Psychometrika, 27(4), 355–374. https://doi.org/10.1007/BF02289644
6.
CrockerL.AlginaJ. (2006). Introduction to classical and modern test theory. Cengage Learning.
7.
de BreyC.MusuL.McFarlandJ.Wilkinson-FlickerS.DilibertiM.ZhangA.BranstetterC.WangX. (2019). Status and trends in the education of racial and ethnic groups 2018 (NCES 2019-038). National Center for Education Statistics. Retrieved fromhttps://nces.ed.gov/pubs2019/2019038.pdf
8.
ElwertF. (2013). Graphical causal models. In MorganS. L. (Ed.), Handbook of causal analysis for social research (pp. 245–273). Springer. https://doi.org/10.1007/978-94-007-6094-3_13
9.
GreggN.NelsonJ. M. (2012). Meta-analysis on the effectiveness of extra time as a test accommodation for transitioning adolescents with learning disabilities: More questions than answers. Journal of Learning Disabilities, 45(2), 128–138. https://doi.org/10.1177/0022219409355484
10.
HahnJ.ToddP.van der KlaauwW. (2001). Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica, 69(1), 201–209. https://doi.org/10.1111/1468-0262.00183
11.
HongG.RaudenbushS. W. (2006). Evaluating kindergarten retention policy: A case study of causal inference for multilevel observational data. Journal of the American Statistical Association, 101(475), 901–910. https://doi.org/10.1198/016214506000000447
JonsonJ. L.TranthamP.Usher-TateB. J. (2019). An evaluative framework for reviewing fairness standards and practices in educational tests. Educational Measurement: Issues and Practice, 38(3), 6–19. https://doi.org/10.1111/emip.12259
15.
KimY. Y.CirciR. (2018). The extended time accommodation (ETA) and performance of students with ETA. Paper presented at the annual meeting of the National Council on Measurement in Education, New York, USA.
16.
KimY. Y.CirciR. (2019). Effects of the extended time accommodation on performance in NAEP mathematics. Paper presented at the annual meeting of the National Council on Measurement in Education, Toronto, Canada.
17.
KolesárM.RotheC. (2018). Inference in regression discontinuity designs with a discrete running variable. American Economic Review, 108(8), 2277–2304. https://doi.org/10.1257/aer.20160945
LeeD. S.LemieuxT. (2010). Regression discontinuity designs in economics. Journal of Economic Literature, 48(2), 281–355. https://doi.org/10.1257/jel.48.2.281
20.
LiF.MercatantiA.MäkinenT.SilvestriniA. (2021). A regression discontinuity design for ordinal running variables: Evaluating central bank purchases of corporate bonds. The Annals of Applied Statistics, 15(1), 304–322. https://doi.org/10.1214/20-AOAS1396
MeulmanJ. J. (1998). Optimal scaling methods for multivariate categorical data analysis. SPSS White Paper.
23.
MeulmanJ. J.van der KooijA. J.DuistersK. L. W. (2019). ROS regression: Integrating regularization with optimal scaling regression. Statistical Science, 34(3), 361–390. https://doi.org/10.1214/19-STS697
24.
MorganS. L.WinshipC. (2014). Counterfactuals and causal inference: Methods and principles for social research. Cambridge University Press. https://doi.org/10.1017/CBO9781107587991
25.
National Center for Education Statistics. (2017a). 2017 Mathematics grades 4 and 8 assessment report cards: Summary data tables for national and state sample sizes, participation rates, proportions of SD and ELL students identified, and types of accommodations. Retrieved fromhttps://www.nationsreportcard.gov/math_2017/files/2017_Technical_Appendix_Math_State.pdf
National Research Council. (2002). Reporting test results for students with disabilities and English-language learners: Summary of a workshop. National Academies Press.
28.
NeymanJ. S. (1923). On the application of probability theory to agricultural experiments: essay on principles. Section 9 (with discussion). Statistical Science, 4, 465–480.
29.
OranjeA.KolstadA. (2019). Research on psychometric modeling, analysis, and reporting of the National Assessment of Educational Progress. Journal of Educational and Behavioral Statistics, 44(6), 648–670. https://doi.org/10.3102/1076998619867105
R Core Team. (2020). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved fromhttps://www.R-project.org/
34.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. https://doi.org/10.1080/01621459.1986.10478355
35.
RubinD. B. (1986). Comment: Which ifs have causal answers. Journal of the American Statistical Association, 81(396), 961–962. https://doi.org/10.2307/2289065
36.
SalesA. C.HansenB. B. (2020). Limitless regression discontinuity. Journal of Educational and Behavioral Statistics, 45(2), 143–174.
VanderWeeleT. J.ArahO. A. (2011). Unmeasured confounding for general outcomes, treatments, and confounders: Bias formulas for sensitivity analysis. Epidemiology, 22(1), 42–52. https://doi.org/10.1097/EDE.0b013e3181f74493
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
