Educational assessments require periodic administration changes, such as transitioning from paper to digital administration. During such transitions, a linking function relating the metrics of the new and previous administration is estimated, but uncertainty in this estimation introduces variance into comparisons between administrations. Similarly, different assessments provided to overlapping populations may be linked, introducing linking or equating error. We introduce new generally applicable variance estimation methods, generalize prior methods to be more widely applicable, and confirm the validity of the methods via simulation. Our methods account for dependencies between linking and other sources of error, complex sampling, and nonlinear linking functions, while applying to a wide range of score comparisons and statistics such as means, standard deviations, percentiles, and differences.
BattauzM. (2017). Multiple equating of separate IRT calibrations. Psychometrika, 82(3), 610–636.
2.
DoransN. J. (2004). Using subpopulation invariance to assess test score equity. Journal of Educational Measurement, 41(1), 43–68.
3.
DoransN. J.MosesT. P.EignorD. R. (2011). Equating test scores: Towards best practices. In von DavierA. A. (Ed.), Statistical models for test equating, scaling, and linking (pp. 21–42). Springer.
4.
DoransN. J.PommerichM.HollandP. W. (Eds.). (2007). Linking and aligning scores and scales (pp. 135–159). Springer.
5.
EfronB.TibshiraniR. J. (1993). An introduction to the bootstrap. Chapman & Hall.
6.
EignorD. R. (2007). Linking scores derived under different modes of test administration. In DoransN. J.PommerichM.HollandP. W. (Eds.), Linking and aligning scores and scales (pp. 135–159). Springer.
7.
FishbeinB.MartinM. O.MullisI. V.FoyP. (2018). The TIMSS 2019 item equivalence study: Examining mode effects for computer-based assessment and implications for measuring trends. Large-scale Assessments in Education, 6(1), 1–23.
8.
HabermanS. J. (2015). Pseudo-eq52 groups and linking. Journal of Educational and Behavioral Statistics, 40(3), 254–273.
9.
HabermanS. J.LeeY.QianJ. (2009). Jackknifing techniques for evaluation of equating accuracy (ETS Research Report RR09-39). ETS.
10.
JewsburyP. A. (2019). Error variance in common population linking bridge studies (Research Report No. RR-19-42). Educational Testing Service.
11.
JewsburyP. A. (2023). Educational surveys: Methodological foundations. In TierneyR.RizviF.ErcikanK. (Eds.), International encyclopedia of education: Quantitative research/educational measurement (4th ed., pp. 359–368). Elsevier.
JiaY.PhillipsG.WiseL. L.RahmanT.XuX.WileyC.DiazT. E. (2014). 2011 NAEP-TIMSS linking study: Technical report on the linking methodologies and their evaluations (NCES 2014-461). National Center for Education Statistics.
14.
JohnsonE. G. (1998). Linking the National Assessment of Educational Progress (NAEP) and the Third International Mathematics and Science Study (TIMSS): A technical report (NCES 98-499). Government Printing Office.
15.
JohnsonE. G.CohenJ.ChenW.-H.JiangT.ZhangY. (2005). 2000 NAEP–1999 TIMSS linking report (NCES 2005-01). National Center for Education Statistics.
16.
JohnsonE. G.RustK. F. (1992). Chapter 5: Population inferences and variance estimation for NAEP data. Journal of Educational Statistics, 17, 175–190.
17.
JohnsonE. G.ZwickR. (1990). Focusing the new design: The NAEP 1988 technical report (Report No. 19-TR-20, pp. 3–9). Educational Testing Service.
18.
KolenM. J.BrennanR. L. (2014). Test equating, scaling, and linking: Methods and practices (3rd ed.). Springer-Verlag.
19.
KuhfeldM.SolandJ.LewisK.RuzekE.JohnsonA. (2022). The COVID-19 school year: Learning and recovery across 2020–2021. AERA Open, 8(1), 1–15.
20.
LiD. (2022). Assessing the impact of equating error on group means and group mean differences. Journal of Educational Measurement, 59(1), 62–79.
21.
LiuB.JiaY.MazzeoJ. (2022). Variance estimation for random-groups linking in large-scale survey assessments. In NgH. K. T.HeitjanD. F. (Eds.), Recent advances on sampling methods and educational statistics: In honor of S. Lynne Stokes. Springer.
22.
MazzeoJ.DonoghueJ. R.LiuB.XuX. (2018). Estimating standard errors for NAEP that incorporate random-groups linking error for the transition from paper-based to digital-based assessments. Paper presented at the annual meeting of the National Council on Measurement in Education, New York, NY.
23.
MislevyR. J. (1991). Randomization-based inference about latent variables from complex samples. Psychometrika, 56(2), 177–196.
24.
MislevyR. J.JohnsonE. G.MurakiE. (1992). Chapter 3: Scaling procedures in NAEP. Journal of Educational Statistics, 17, 131–154.
25.
MosesT.ZhangW. (2011). Standard errors of equating differences: Prior developments, extensions, and simulations. Journal of Educational and Behavioral Statistics, 36(6), 779–803.
OranjeA.KolstadA. (2019). Research on psychometric modeling, analysis, and reporting of the national assessment of educational progress. Journal of Educational and Behavioral Statistics, 44(6), 648–670.
PhillipsG. W. (2007). Expressing international educational achievement in terms of U.S. performance standards: Linking NAEP achievement levels to TIMSS. American Institutes for Research.
31.
PhillipsG. W.JiangT. (2019). Measurement error and equating error in power analysis. Practical Assessment, Research, and Evaluation, 21(1), 9.
32.
PokropekA.BorgonoviF. (2020). Linking via pseudo-eq53 group design: Methodological considerations and an application to the PISA and PIAAC assessments. Journal of Educational Measurement, 57(4), 527–546.
33.
ReardonS. F.KalogridesD.HoA. D. (2021). Validation methods for aggregate-level test scale linking: A case study mapping school district test score distributions to a common scale. Journal of Educational and Behavioral Statistics, 46(2), 138–167.
34.
RobitzschA. (2023). Linking error in the 2PL model. J, 6(1), 58–84.
35.
RobitzschA.LüdtkeO. (2019). Linking errors in international large-scale assessments: Calculation of standard errors for trend estimation. Assessment in Education: Principles, Policy & Practice, 26(4), 444–465.
36.
RobitzschA.LüdtkeO. (2023). Comparing different trend estimation approaches in country means and standard deviations in international large-scale assessment studies. Large-Scale Assessments in Education, 11(1), 26.
37.
RogersA.De SantoA.SikaliE. (2021). National Assessment of Educational Progress (NAEP) 2018 Civics, Geography, and U.S. History Grade 8 assessments restricted-use data files data companion. National Center for Education Statistics.
38.
RustK. (2014). Sampling, weighting, and variance estimation. In RutkowskiL.von DavierM.RutkowskiD. (Eds.), Handbook of international large-scale assessment (pp. 117–153). CRC Press.
39.
PerieM.MoranR.LutkusA. D. (2005). NAEP 2004 trends in academic progress: Three decades of student performance in reading and mathematics (NCES 2005-464). Government Printing Office.
40.
von DavierA. A. (2013). Observed-score equating: An overview. Psychometrika, 78, 605–623.
41.
von DavierA. A.HollandP. W.ThayerD. T. (2004). The kernel method of test equating. Springer.
42.
von DavierM.KhorramdelL.HeQ.ShinH. J.ChenH. (2019). Developments in psychometric population models for technology-based large-scale assessments: An overview of challenges and opportunities. Journal of Educational and Behavioral Statistics, 44(6), 671–705.
43.
von DavierM.SinharayS.OranjeA.BeatonA. (2006). The statistical procedures used in National Assessment of Educational Progress: Recent developments and future directions. In RaoC. R.SinharayS. (Eds.), Handbook of statistics: Vol 26, psychometrics (pp. 1039–1055). Elsevier.
44.
von DavierM.von DavierA. A. (2007). A unified approach to IRT scale linking and scale transformations. Methodology, 3(3), 115–124.
45.
von DavierM.YamamotoK.ShinH. J.ChenH.KhorramdelL.WeeksJ.DavisS.KongN.KandathilM. (2019). Evaluating item response theory linking and model fit for data from PISA 2000–2012. Assessment in Education: Principles, Policy & Practice, 26(4), 466–488.
46.
WuM. (2010). Measurement, sampling, and equating errors in large-scale assessments. Educational Measurement: Issues and Practice, 29(4), 15–27.
47.
XuX.von DavierM. (2010). Linking errors in trend estimation in large-scale surveys: A case study (ETS Research Report RR10-10). ETS.
48.
YamamotoK.MazzeoJ. (1992). Item response theory scale linking in NAEP. Journal of Educational Statistics, 17, 155–173.
49.
YamashitaT.SmithT. J.CumminsP. A. (2021). A practical guide for analyzing large-scale assessment data using Mplus: A case demonstration using the program for international assessment of adult competencies data. Journal of Educational and Behavioral Statistics, 46(4), 501–518.
50.
ZhangZ. (2022). Estimating standard errors of IRT true score equating coefficients using imputed item parameters. The Journal of Experimental Education, 90(3), 760–782.
51.
ZhangZ.ZhaoM. (2019). Standard errors of IRT parameter scale transformation coefficients: Comparison of bootstrap method, delta method, and multiple imputation method. Journal of Educational Measurement, 56(2), 302–330.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
