When equating two test forms, the equated scores will be biased if the test groups differ in ability. To adjust for the ability imbalance between nonequivalent groups, a set of common items is often used. When no common items are available, it has been suggested to use covariates correlated with the test scores instead. In this article, we reduce the covariates to a propensity score and equate the test forms with respect to this score. The propensity score is incorporated within the kernel equating framework using poststratification and chained equating. The methods are evaluated using real college admissions test data and through a simulation study. The results show that propensity scores give an increased equating precision in comparison with the equivalent groups design and a smaller mean squared error than by using the covariates directly. Practical implications are also discussed.
AbadieA.ImbensG. W. (2006). Large sample properties of matching estimators for average treatment effects. Econometrica, 74, 235–267.
2.
AnderssonB.BränbergK.WibergM. (2013). Performing the kernel method of test equating using the R package kequate. Journal of Statistical Software, 55, 1–25.
3.
AustinP. C. (2008). Goodness-of-fit diagnostics for the propensity score model when estimating treatment effects using covariate adjustment with the propensity score. Pharmacoepidemiology and Drug Safety, 17, 1202–1217.
4.
BränbergK.HenrikssonW.NyquistH.WedmanI. (1990). The influence of sex, education and age on the scores on the Swedish Scholastic Aptitude Test. Scandinavian Journal of Educational Research, 34, 189–203.
5.
BränbergK.WibergM. (2011). Observed score linear equating with covariates. Journal of Educational Measurement, 48, 419–440.
6.
CookL. L.EignorD. R.SchmittA. P. (1990). Equating achievement tests using samples matched on ability (College Board Report 90–2). New York, NY: College Entrance Examination Board.
7.
DoransN. J.FeigenbaumM. D. (1994). Equating issues engendered by changes to the SAT and PSAT/NMSQT (ETS Research Memorandum No. RM-94-10). Princeton, NJ: ETS.
8.
GonzálezJ.WibergM. (2017). Applying test equating methods using R. Cham, Switzerland: Springer.
9.
HabermanS. J. (2015). Pseudo-equivalent groups and linking. Journal of Educational and Behavioural Statistics, 40, 254–273.
10.
HäggströmJ.WibergM. (2014). Optimal bandwidth selection in kernel equating. Journal of Educational Measurement, 51, 201–211.
11.
HollandP. W.ThayerD. T. (2000). Univariate and bivariate loglinear models for discrete test score distributions. Journal of Educational and Behavioural Statistics, 25, 133–183.
12.
HsuT.WuK.YuJ.LeeM. (2009). Exploring the feasibility of collateral information test equating. International Journal of Testing, 2, 1–14.
KolenM. J. (1990). Does matching in equating work? A discussion. Applied Measurement in Education, 3, 23–39.
15.
LiouM.ChengP. E.LiM. (2001). Estimating comparable scores using surrogate variables. Applied Measurement in Education, 25, 197–207.
16.
LivingstonS. A.DoransN. J.WrightN. K. (1990). What combination of sampling and equating methods works best?Applied Measurement in Education, 3, 73–95.
17.
LongfordN. T. (2015). Equating without an anchor for nonequivalent groups of examinees. Journal of Educational and Behavioral Statistics, 40, 227–253.
18.
LyrénP.-E.HambletonR. K. (2011). Consequences of violated the equating assumptions under the equivalent group design. International Journal of Testing, 36, 308–323.
19.
MosesT.DengW.ZhangY.-L. (2010). The use of two anchors in the nonequivalent groups with anchor test (NEAT) equating (ETS research report RR-10-23). Princeton, NJ: Educational Testing Service.
20.
PaekI.LiuJ.OhH. J. (2006). Investigation of propensity score matching on linear/nonlinear equating method for the P/N/NMSQT (Report SR-2006-55). Princeton, NJ: ETS.
21.
PowersS. J. (2010). Impact of matched samples equating methods on equating accuracy and the adequacy of equating assumptions (PhD thesis). University of Iowa. Retrieved fromhttp://ir.uiowa.edu/etd/875
22.
QuenetteM. A.NicewanderW. A.ThomassonG. L. (2006). Model-based versus empirical equating of test forms. Applied Psychological Measurement, 30, 167–182.
23.
R Core Development Team. (2016). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for statistical computing. Retrieved fromhttp://www.R-project.org/
24.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
25.
RosenbaumP. R.RubinD. B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79, 516–524.
26.
SchwarzG. E. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464.
27.
SinharayS.HollandP. W. (2010). A new approach to comparing several equating methods in the context of the NEAT design. Journal of Educational Measurement, 47, 261–285.
28.
SungwornN. (2009). An investigation of using collateral information to reduce equating biases of the post-stratification equating method (PhD thesis). Michigan State University.
29.
von DavierA. A. (2013). Observed-score equating: An overview. Psychometrika, 78, 605–623.
30.
von DavierA. A.HollandP. W.ThayerD. T. (2004a). The chain and post-stratification methods for observed-score equating: Their relationship to population invariance. Journal of Educational Measurement, 41, 15–32.
31.
von DavierA. A.HollandP. W.ThayerD. T. (2004b). The kernel method of test equating. New York, NY: Springer.
32.
WangT.LeeW.-C.BrennanR. B.KolenM. J. (2006). A comparison of the frequency estimation and chained equipercentile methods under the common-item non-equivalent groups design. Applied Psychological Measurement, 32, 632–651.
33.
WibergM.BränbergK. (2015). Kernel equating under the non-equivalent groups with covariates design. Applied Psychological Measurement, 39, 349–361.
34.
WibergM.GonzálezJ. (2016). Statistical assessment of estimated transformations in observed-score equating. Journal of Educational Measurement, 53, 106–125.
35.
WrightN. K.DoransN. J. (1993). Using the selection variable for matching or equating (ETS Research Report RR-93–04). Princeton, NJ: Educational Testing Service.
36.
YuL.LivingstonS. A.LarkinK. C.BonettJ. (2004). Investigating differences in examinee performance between computer-based and handwritten essays (ETS Research Report RR-04-18). Princeton, NJ: Educational Testing Service.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
