Applications of multidimensional forced choice (MFC) testing have increased considerably over the last 20 years. Yet there has been little, if any, research on methods for linking the parameter estimates from different samples. This research addressed that important need by extending four widely used methods for unidimensional linking and comparing the efficacy of new estimation algorithms for MFC linking coefficients based on the Multi-Unidimensional Pairwise Preference model (MUPP). More specifically, we compared the efficacy of multidimensional test characteristic curve (TCC), item characteristic curve (ICC; Haebara, 1980), mean/mean (M/M), and mean/sigma (M/S) methods in a Monte Carlo study that also manipulated test length, test dimensionality, sample size, percentage of anchor items, and linking scenarios. Results indicated that the ICC method outperformed the M/M method, which was better than the M/S method, with the TCC method being the least effective. However, as the number of items “per dimension” and the percentage of anchor items increased, the differences between the ICC, M/M, and M/S methods decreased. Study implications and practical recommendations for MUPP linking, as well as limitations, are discussed.
BakerF. B.Al‐KarniA. (1991). A comparison of two procedures for computing IRT equating coefficients. Journal of Educational Measurement, 28(2), 147–162. https://doi.org/10.1111/j.1745-3984.1991.tb00350.x
2.
BrooksS. P.GelmanA. (1998). General methods for monitoring convergence of iterative simulations. Journal of Computational & Graphical Statistics, 7(4), 434–455. https://doi.org/10.1080/10618600.1998.10474787
3.
BrownA.BartramD. (2009). Development and psychometric properties of OPQ32r (Supplement to the OPQ32 technical manual). Ditton, UK: ThamesSHL Group Limited.
4.
ByrdR. H.LuP.NocedalJ.ZhuC. (1995). A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16(5), 1190–1208. https://doi.org/10.1137/0916069
5.
CandellG. L.DrasgowF. (1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12(3), 253–260. https://doi.org/10.1177/014662168801200304
6.
CaoM.DrasgowF. (2019). Does forcing reduce faking? A meta-analytic review of forced- choice personality measures in high-stakes situations. Journal of Applied Psychology, 104(11), 1347–1368. https://doi.org/10.1037/apl0000414
7.
ChalmersR. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06
8.
ChoS.DrasgowF.CaoM. (2015). An investigation of emotional intelligence measures using item response theory. Psychological Assessment, 27(4), 1241–1252. https://doi.org/10.1037/pas0000132
9.
DoornikJ. A. (2009). An object-oriented matrix language: Ox 6. Timberlake Consultants Press.
10.
DrasgowF.StarkS.ChernyshenkoO. S.NyeC. D.HulinC. L.WhiteL. A. (2012). Development of the tailored adaptive personality assessment system (TAPAS) to support army personnel selection and classification decisions. Drasgow consulting group URBANA Il.
11.
GelmanA.RubinD. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–472. https://doi.org/10.1214/ss/1177011136
12.
HaebaraT. (1980). Equating logistic ability scales by a weighted least squares method. Japanese Psychological Research, 22(3), 144–149. https://doi.org/10.4992/psycholres1954.22.144
13.
HansonB. A.BéguinA. A. (2002). Obtaining a common scale for item response theory item parameters using separate versus concurrent estimation in the common-item equating design. Applied Psychological Measurement, 26(1), 3–24. https://doi.org/10.1177/0146621602026001001
14.
JooS. H.LeeP.StarkS. (2017). Evaluating anchor-item designs for concurrent calibration with the GGUM. Applied Psychological Measurement, 41(2), 83–96. https://doi.org/10.1177/0146621616673997
15.
JooS. H.LeeP.StarkS. (2018). Development of information functions and indices for the GGUM‐RANK multidimensional forced choice IRT model. Journal of Educational Measurement, 55(3), 357–372. https://doi.org/10.1111/jedm.12183
16.
JooS. H.LeeP.StarkS. (2020). Adaptive testing with the GGUM-RANK multidimensional forced choice model: Comparison of pair, triplet, and tetrad scoring. Behavior Research Methods, 52(2), 761–772. https://doi.org/10.3758/s13428-019-01274-6
17.
JooS. H.LeeP.StarkS. (2023). Modeling multidimensional forced choice measures with the Zinnes and Griggs pairwise preference item response theory model. Multivariate Behavioral Research, 58(2), 241–261. https://doi.org/10.1080/00273171.2021.1960142
18.
KimS. H.CohenA. S. (1998). A comparison of linking and concurrent calibration under item response theory. Applied Psychological Measurement, 22(2), 131–143. https://doi.org/10.1177/01466216980222003
19.
KimS. H.CohenA. S. (2002). A comparison of linking and concurrent calibration under the graded response model. Applied Psychological Measurement, 26(1), 25–41. https://doi.org/10.1177/0146621602026001002
20.
KoenigJ.RobertsJ. S. (2007). Linking parameters estimated with the generalized graded unfolding model: A comparison of the accuracy of characteristic curve methods. Applied Psychological Measurement, 31(6), 504–524. https://doi.org/10.1177/0146621606297315
MorilloD.LeenenI.AbadF. J.HontangasP.de la TorreJ.PonsodaV. (2016). A dominance variant under the multi-unidimensional pairwise-preference framework: Model formulation and Markov chain Monte Carlo estimation. Applied Psychological Measurement, 40(7), 500–516. https://doi.org/10.1177/0146621616662226
27.
RobertsJ. S.BaoH.HuangC. W.GagneP. (2003). Exploring alternative characteristic curve approaches to linking parameter estimates from the generalized partial credit model. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
28.
RobertsJ. S.DonoghueJ. R.LaughlinJ. E. (2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24(1), 3–32. https://doi.org/10.1177/01466216000241001
29.
RobertsJ. S.DonoghueJ. R.LaughlinJ. E. (2002). Characteristics of MML/EAP parameter estimates in the generalized graded unfolding model. Applied Psychological Measurement, 26(2), 192–207. https://doi.org/10.1177/01421602026002006
30.
RobertsJ. S.FangH.CuiW.WangY. (2006). GGUM2004: A windows-based program to estimate parameters in the generalized graded unfolding model. Applied Psychological Measurement, 30(1), 64–65. https://doi.org/10.1177/0146621605280141
31.
SeybertJ.StarkS.ChernyshenkoO. S. (2014). Detecting DIF with ideal point models: A comparison of area and parameter difference methods. Applied Psychological Measurement, 38(2), 151–165. https://doi.org/10.1177/0146621613508306
32.
StarkS.ChernyshenkoO. S.DrasgowF. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi- unidimensional pairwise-preference model. Applied Psychological Measurement, 29(3), 184–203. https://doi.org/10.1177/0146621604273988
33.
StarkS.ChernyshenkoO. S.DrasgowF.WhiteL. A. (2012). Adaptive testing with multidimensional pairwise preference items: Improving the efficiency of personality and other noncognitive assessments. Organizational Research Methods, 15(3), 463–487. https://doi.org/10.1177/1094428112444611
34.
StarkS.ChernyshenkoO. S.DrasgowF.WilliamsB. A. (2006). Examining assumptions about item responding in personality assessment: Should ideal point methods be considered for scale development and scoring?Journal of Applied Psychology, 91(1), 25–39. https://doi.org/10.1037/0021-9010.91.1.25
35.
StarkS. E. (2002). A new IRT approach to test construction and scoring designed to reduce the effects of faking in personality assessment: The generalized graded unfolding model for multi-unidimensional paired comparison responses. University of Illinois at Urbana-Champaign.
36.
StockingM. L.LordF. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7(2), 201–210. https://doi.org/10.1177/014662168300700208
37.
SunT.FraleyR. C.DrasgowF. (2020). Matches made with information: Fitting measurement models to adult attachment data. Assessment, 28(7), 1828–1847. https://doi.org/10.1177/1073191120971847
38.
TayL.DrasgowF.RoundsJ.WilliamsB. A. (2009). Fitting measurement models to vocational interest data: Are dominance models ideal?Journal of Applied Psychology, 94(5), 1287–1304. https://doi.org/10.1037/a0015899
39.
TuN.JooS.LeeP.StarkS. (2022). Comparison of parameter estimation approaches for multi-unidimensional pairwise preference tests. Behavior Research Methods, 55(6), 2764–2786. https://doi.org/10.3758/s13428-022-01927-z
40.
TuN.ZhangB.AngraveL.SunT. (2021). Bmggum: An R package for Bayesian estimation of the multidimensional generalized graded unfolding model with covariates. Applied Psychological Measurement, 45(7-8), 553–555. https://doi.org/10.1177/01466216211040488
41.
TuN.ZhangB.AngraveL.SunT.NeumanM. (2023). Estimating the multidimensional generalized graded unfolding model with covariates using a bayesian approach. Journal of Intelligence, 11(8), 163.
42.
UsamiS.SakamotoA.NaitoJ.AbeY. (2016). Developing pairwise preference-based personality test and experimental investigation of its resistance to faking effect by item response model. International Journal of Testing, 16(4), 288–309. https://doi.org/10.1080/15305058.2016.1145123
43.
WetzelE.FrickS.BrownA. (2020). Does multidimensional forced-choice prevent faking? Comparing the susceptibility of the multidimensional forced-choice format and the rating scale format to faking. Psychological Assessment, 33(2), 156–170. https://doi.org/10.1037/pas0000971
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
