An IRT Approach to Constructing and Scoring Pairwise Preference Items Involving Stimuli on Different Dimensions: The Multi-Unidimensional Pairwise-Preference Model

Abstract

This article proposes an item response theory (IRT) approach to constructing and scoring multidimensional pairwise preference items. Individual statements are administered and calibrated using a unidimensional single-stimulus model. Tests are created by combining multidimensional items with a small number of unidimensional pairings needed to identify the latent metric. Trait scores are then obtained using a multidimensional Bayes modal estimation procedure based on a mathematical model called MUPP, which is illustrated and tested here using Monte Carlo simulations. Simulation results show that the MUPP approach to test construction and scoring provides accurate parameter recovery in both one- and two-dimensional simulations, even with relatively few (say, 15%) unidimensional pairings. The implications of these results for constructing and scoring fake-resistant personality items are discussed.

Keywords

IRT pairwise preference paired comparison forced choice ipsative multidimensional IRT personality assessment faking

Get full access to this article

View all access options for this article.

References

Andrich, D. (1995). Hyperbolic cosine latent trait models for unfolding direct responses and pairwise preferences. Applied Psychological Measurement, 19, 269-290.

Andrich, D. , & Luo, G. (1993). A hyperbolic cosine latent trait model for unfolding dichotomous single-stimulus responses. Applied Psychological Measurement, 17, 253-276.

Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397-472). Reading, MA: Addison-Wesley.

Chernyshenko, O. S. (2002). Applications of ideal point approaches to scale construction and scoring in personality measurement: The development of a six-faceted measure of conscientiousness. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign.

Chernyshenko, O. S. , Stark, S. , Chan, K. Y. , Drasgow, F. , & Williams, B. A. (2001). Fitting item response theory models to personality data. Multivariate Behavioral Research, 36, 523-562.

Conn, S. , & Rieke, M. L. (Eds.). (1994). The 16PF fifth edition technical manual. Champaign, IL: Institute for Personality and Ability Testing.

Drasgow, F. , Levine M. V. , Tsien, S. , Williams, B. A. , & Mead, A. D. (1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement, 19, 143-165.

Hicks, L. E. (1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin, 74, 167-184.

Levine, M. V. , & Williams, B. A. (1993). Nonparametric models for polychotomously scored item responses: Analysis and integration. Unpublished manuscript.

10.

Press, W. H. , Flannery, B. P. , Teukolsky, S. A. , & Vetterling, W. T. (1990). Numerical recipes: The art of scientific computing. New York: Cambridge University Press.

11.

Roberts, J. S. , Donoghue, J. R. , & Laughlin, J. E. (2000a). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24, 3-32.

12.

Roberts, J. S. , Donoghue, J. R. , & Laughlin, J. E. (2000b). GGUM2000: A DOS-based program for unfolding ideal point responses [Computer program]. Department of Measurement and Statistics, University of Maryland.

13.

Stark, S. (2001). MODFIT: A computer program for model-data fit. Urbana-Champaign: University of Illinois at Urbana-Champaign. Retrieved from http://work.psych.uiuc.edu/irt

14.

Stark, S. (2002). A new IRT approach to test construction and scoring designed to reduce the effects of faking in personality assessment. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign.

15.

Stark, S. , Chernyshenko, O. S. , Chan, K. Y. , Lee, W. C. , & Drasgow, F. (2001). Effects of the testing situation on item responding: Cause for concern. Journal of Applied Psychology, 86, 943-953.

16.

Stark, S. , Chernyshenko, O. S. , & Drasgow, F. (in press). Examining assumptions about item responding in personality assessment: Should ideal point methods be considered for scale development and scoring? Journal of Applied Psychology.

17.

Stark, S. , & Drasgow, F. (2002). An EM approach to parameter estimation for the Zinnes and Griggs paired comparison ideal point IRT model. Applied Psychological Measurement, 26, 208-227.

18.

Waters, L. K. (1965). A note on the “fakability” of forced-choice scales. Personnel Psychology, 18, 187-191.

19.

White, L. A. , & Young, M. C. (1998, August). Development and validation of the Assessment of Individual Motivation (AIM). Paper presented at the annual meeting of the American Psychological Association, San Francisco.