The nonequivalent groups with anchor test (NEAT) design is widely used in test equating. Under this design, two groups of examinees are administered different test forms with each test form containing a subset of common items. Because test takers from different groups are assigned only one test form, missing score data emerge by design rendering some of the score distributions unavailable. The partially observed score data formally lead to an identifiability problem, which has not been recognized as such in the equating literature and has been considered from different perspectives, all of them making different assumptions in order to estimate the unidentified score distributions. In this article, we formally specify the statistical model underlying the NEAT design and unveil the lack of identifiability of the parameters of interest that compose the equating transformation. We use the theory of partial identification to show alternatives to traditional practices that have been proposed to identify the score distributions when conducting equating under the NEAT design.
AngoffW. H. (1984). Scales, norms, and equivalent scores. Educational Testing Service, Princeton.
2.
AngoffW. H. (1987). Technical and practical issues in equating: A discussion of four papers. Applied Psychological Measurement, 11(3), 291–300.
3.
BhideA.ShahP. S.AcharyaG. (2018). A simplified guide to randomized controlled trials. Acta Obstetricia et Gynecologica Scandinavica, 97(4), 380–387.
4.
BlundellR.GoslingA.IchimuraH.MeghirC. (2007). Changes in the distribution of male and female wages accounting for employment composition using bounds. Econometrica, 75, 323–363.
5.
BolsinovaM.MarisG. (2016). Can IRT solve the missing data problem in test equating?Frontiers in Psychology, 6, 1956.
6.
BraunH.HollandP. W. (1982). Observed-score test equating: A mathematical analysis of some ETS equating procedures. In HollandP. W.RubinD. B. (Eds.), Test equating (pp. 9–49). Academic Press.
7.
BrennanR. L.KolenM. J. (1987). A reply to angoff. Applied Psychological Measurement, 11(3), 301–306.
8.
CarnapR. (1962). Logical foundations of probability. The University of Chicago Press.
9.
EmbrechtsP.HofertM. (2013). A note on generalized inverses. Mathematical Methods of Operations Research, 77(3), 423–432.
10.
FisherR. A. (1922). On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London. Series A, 222, 309–368.
11.
FisherR. A. (1973). Statistical Methods for Research Workers. Hafner Publishing, New York, USA.
12.
FlorensJ. P.MouchartM. (1982). A note on noncausality. Econometrica, 50(3), 583–591.
13.
FlorensJ. P.MouchartM.RolinJ.-M. (1990). Elements of Bayesian statistics. Marcel Dekker.
14.
GonzálezJ.WibergM. (2017). Applying test equating methods using R. Springer.
15.
GundersenC.KreiderB. (2009). Bounding the effects of food insecurity on children’s health outcomes. Journal of Health Economics, 28, 971–983.
16.
HollandP. W.DoransN. J. (2006). Linking and equating. Educational Measurement, 4, 187–220.
17.
HollandP. W.SinharayS.von DavierA. A.HanN. (2008). An approach to evaluating the missing data assumptions of the chain and post-stratification equating methods for the neat design. Journal of Educational Measurement, 45(1), 17–43.
18.
HornC.SantelicesM.V.AvendañoX.C. (2014). Modeling the impacts of national and institutional financial aid opportunities on persistence at an elite Chilean university. Higher Education, 68(3), 471–488.
19.
ItôK. (1984). An introduction to probability theory. Cambridge University Press.
20.
KarrA. F. (1993). Probability. Springer.
21.
KolenM. J. (2007). Data collection designs and linking procedures. In DoransN. J.PommerichM.HollandP. W. (Eds.), Linking and aligning scores and scales (pp. 31–55). Springer.
22.
KolenM. J.BrennanR. L. (2014). Test equating, scaling, and linking: Methods and practices (3rd ed.). Springer.
23.
KolmogorovA. N. (1950). Foundations of the theory of probability. Chelsea Publishing Company.
24.
KoopmansT. C. (1949). Identification problems in economic model construction. Econometrica, 125–144.
25.
KoopmansT. C.ReiersolO. (1950). The identification of structural characteristics. The Annals of Mathematical Statistics, 21(2), 165–181.
LiouM.ChengP. E. (1995). Equipercentile equating via data–imputation technique. Psychometrika, 60, 119–136.
28.
LittleR. J.RubinD. B. (1994). Test equating from biased samples, with application to the armed services vocational aptitude battery. Journal of Educational and Behavioral Statistics, 309–335.
29.
LordF. M. (1950). Notes on Comparable Scales for Test Scores (Tech. Rep.). Educational Testing Service.
30.
LordF. M.NovickM. (1968). Statistical theories of mental test scores. Addison-Wesley.
31.
MaddalaG. (1983). Qualitative and limited dependent variable models in econometrics. Cambridge University Press.
32.
ManskiC. F. (2007). Identification for prediction and decision. Harvard University Press.
33.
ManskiC. F. (2013). Diagnostic testing and treatment under ambiguity: Using decision analysis to inform clinical practice. Proceedings of the National Academy of Sciences, 110, 2064–2069.
34.
ManskiC. F.NaginD. S. (1998). Bounding disagreements about treatment effects: A case study of sentencing and recidivism. Sociological Methodology, 28(1), 99–137.
35.
MiyazakiK.HoshinoT.MayekawaS.-i.ShigemasuK. (2009). A new concurrent calibration method for nonequivalent group design under nonrandom assignment. Psychometrika, 74(1), 1.
36.
MolinariF. (2010). Missing treatments. Journal of Business & Economic Statistics, 28, 82–95.
37.
Odgaard-JensenJ.VistG. E.TimmerA.KunzR.AklE. A.SchünemannH.…OxmanA. D. (2011). Randomisation to protect against selection bias in healthcare trials. Cochrane Database of Systematic Reviews, 2011(4), MR000012. https://doi.org/10.1002/14651858.MR000012.pub3
38.
PepperJ. (2000). The intergenerational transmission of welfare receipt: A nonparametric bounds analysis. The Review of Economics and Statistics, 82, 472–488.
39.
PielkeR. A.Jr, (2007). The honest broker: Making sense of science in policy and politics. Cambridge University Press.
40.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.
41.
San MartínE.GonzálezJ.TuerlinckxF. (2015). On the unidentifiability of the fixed-effects 3PL model. Psychometrika, 80, 450–467.
42.
SinharayS.HollandP. W. (2010). The missing data assumptions of the NEAT design and their implications for test equating. Psychometrika, 75, 309–327.
43.
StephensonJ.ImrieJ. (1998). Why do we need randomised controlled trials to assess behavioural interventions?BMJ, 316(7131), 611–613.
TamerE. (2010). Partial identification in econometrics. Annual Review of Economics, 2(1), 167–195.
46.
van der LindenW. J. (2019). Lord’s equity theorem revisited. Journal of Educational and Behavioral Statistics, 44(4), 415–430.
47.
van der LindenW. J.BarrettM. D. (2016). Linking item response model parameters. Psychometrika, 81(3), 650–673.
48.
von DavierA. A.HollandP.ThayerD. (2004). The kernel method of test equating. Springer.
49.
WunschG.MouchartM.RussoF. (2014). Functions and mechanisms in structural-modelling explanations. Journal for General Philosophy of Science, 45(1), 187–208.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
