An Investigation Into the Impact of Test Session Disruptions for At-Home Test Administrations

Abstract

In response to the closures of test centers worldwide due to the COVID-19 pandemic, several testing programs offered large-scale standardized assessments to examinees remotely. However, due to the varying quality of the performance of personal devices and internet connections, more at-home examinees likely suffered “disruptions” or an interruption in the connectivity to their testing session compared to typical test-center administrations. Disruptions have the potential to adversely affect examinees and lead to fairness or validity issues. The goal of this study was to investigate the extent to which disruptions impacted performance of at-home examinees using data from a large-scale admissions test. Specifically, the study involved comparing the average test scores of the disrupted examinees with those of the non-disrupted examinees after weighting the non-disrupted examinees to resemble the disrupted examinees along baseline characteristics. The results show that disruptions had a small negative impact on test scores on average. However, there was little difference in performance between the disrupted and non-disrupted examinees after removing records of the disrupted examinees who were unable to complete the test.

Keywords

test disruptions remote testing entropy balancing

Get full access to this article

View all access options for this article.

References

Ford

D. Y.

Helms

J. E.

(2012). Overview and introduction: Testing and assessing African Americans: “Unbiased” tests are still unfair. The Journal of Negro Education, 81(3), 186–189. https://doi.org/10.7709/jnegroeducation.81.3.0186

Guo

Fraser

M. W.

(2015). Propensity score analysis: Statistical methods and applications (2nd ed.). Sage Publications, Inc.

Haberman

S. J.

(2015). Pseudo-equivalent groups and linking. Journal of Educational and Behavioral Statistics, 40(3), 254–273. https://doi.org/10.3102/1076998615574772

Hainmueller

(2011). Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Political Analysis, 20(1), 25–46. https://doi.org/10.2139/ssrn.1904869

Hill

(2013). An analysis of the impact of interruptions on the 2013 administration of the Indiana Statewide Testing for Education Progress-Plus (ISTEP+). The National Center for the Improvement of Educational Assessment.

Martineau

Dadey

(2016). Final report on online interruptions of the Spring 2015 Smarter Balanced assessment administration in Montana, Nevada, and North Dakota. The National Center for the Improvement of Educational Assessment.

Martineau

Domaleski

Egan

Patelis

Dadey

(2015). Recommendations for addressing the impact of test administration interruptions and irregularities. The National Center for the Improvement of Educational Assessment.

McCaffrey

D. F.

Ridgeway

Morral

A. R.

(2004). Propensity Score Estimation With Boosted Regression for Evaluating Causal Effects in Observational Studies. Psychological Methods, 9(4), 403–425. https://doi.org/10.1037/1082-989X.9.4.403

Michel

R. S.

(2020). Remotely proctored K-12 high stakes standardized testing during COVID-19: Will it last? Educational Measurement: Issues and Practice. 39(3), 28–30. https://doi.org/10.1111/emip.12364

10.

R Core Team . (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. URL https://www.R-project.org/

11.

Rosenbaum

P. R.

Rubin

D. B.

(1983). The central role of the propensity score in observational studies for causal effects. Biometrika 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41

12.

Sinharay

Wan

Whitaker

Kim

D.-I.

Zhang

Choi

S. W.

(2014). Determining the overall impact of interruptions during online testing. Journal of Educational Measurement, 51(4), 419–440. https://doi.org/10.1111/jedm.12052

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.13 MB

0.17 MB