Recently, the use of technology in assessment for personnel selection has increased dramatically. An important consideration is whether test scores obtained via Internet administration are psychometrically equivalent to those obtained by the more traditional paper-and-pencil format. Our results suggest that there is comparability of scores for many personality constructs, including conscientiousness. However, invariance was not found for some scales between persons allowed to choose formats and those not given a choice of formats. As testing-format preference may be related to membership in federally protected demographic groups, this latter finding was somewhat troubling. Additionally, we illustrate the use of an experimental laboratory design to investigate possible causes of a lack of measurement invariance in Internet and paper-and-pencil comparisons.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing . Washington, DC: American Psychological Association.
2.
Anderson, J.C., & Gerbing, D.W. (1984). The effect of sampling error on convergence, improper solutions, and goodness-of-fit indices for maximum likelihood confirmatory factor analysis. Psychometrika, 49, 155—173.
3.
Anderson, J.C., & Gerbing, D.W. (1988). Structural equation modeling in practice: A review and recommended two stage approach. Psychological Bulletin,103, 411—423.
4.
Bagozzi, R.P., & Edwards, J.R. (1998). A general approach for representing constructs in organizational research. Organizational Research Methods , 1, 45—87.
5.
Bandalos, D.L. (2002). The effects of item parceling on goodness-of-fit and parameter estimate bias in structural equation modeling. Structural Equation Modeling, 9, 78—102.
6.
Bandalos, D.L., & Finney, S.J. (2001). Item parceling issues in structural equation modeling . In G. A. Marcoulides (Ed.), New developments and techniques in structural equation modeling (pp. 269—296). Mahwah, NJ: Lawrence Erlbaum.
7.
Barrett, P., Kline, P., Paltiel, L., & Eysenck, H.J. (1996). An evaluation of the psychometric properties of the concept 5.2 Occupational Personality Questionnaire. Journal of Occupational and Organizational Psychology, 69, 1—19.
8.
Barrick, M.R., & Mount, M.K. (1991). The Big Five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1—26.
9.
Bartram, D., & Brown, A. (2004). Information exchange article: Online testing: Mode of administration and the stability of OPQ 32i scores. International Journal of Selection and Assessment, 12, 278—284.
10.
Beaujouan, Y.M. (2000). The convergence of the factorial structure of the OPQ (OccupationalPersonality Questionnaire) towards the Five Factor model, in seven countries. European Review of Applied Psychology/Revue Europeenne de Psychologie Appliquee, 50, 349—357.
11.
Booth-Kewley, S., Edwards, J.E., & Rosenfeld, P. (1992). Impression management, social desirability, and computer administration of attitude questionnaires: Does the computer make a difference?Journal of Applied Psychology, 77, 562—566.
12.
Borman, W.C., Penner, L.A., Allen, T.D., & Motowidlo, S.J. (2001). Personality predictors of citizenship performance. International Journal of Selection and Assessment, 9, 52—69.
13.
Brehm, S.S., & Brehm, J.W. (1981). Psychological reactance: A theory of freedom and control. New York: Academic Press .
14.
Byrne, B.M., Shavelson, R.J., & Muthen, B. (1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin,105, 456—466.
15.
Chan, D. (2000). Detection of differential item functioning on the Kirton Adaptation-Innovation Inventory using multiple-group mean and covariance structure analyses. Multivariate Behavioral Research, 35, 169—199.
16.
Chapman, D.S., & Webster, J. (2003). The use of technologies in the recruiting, screening, and selection processes for job candidates. International Journal of Selection and Assessment, 11, 113—120.
17.
Cooper, J., & Weaver, K.D. (2003). Gender and computers: Understanding the digital divide. Mahwah, NJ: Lawrence Erlbaum .
18.
Cortina, J.M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98—104.
19.
Ding, L., Velicer, W.F., & Harlow, L.L. (1995). Effects of estimation methods, number of indicators per factor, and improper solution on structural equation modeling fit indices . Structural Equation Modeling, 2, 119—144.
20.
Donovan, M.A., Drasgow, F., & Probst, T.M. (2000). Does computerizing paper-and-pencil job attitude scales make a difference? New IRT analyses offer insight. Journal of Applied Psychology, 85, 305—313.
21.
Drasgow, F., & Kanfer, R. (1985). Equivalence of measurement in heterogeneous populations . Journal of Applied Psychology, 70, 662—680.
22.
Dulewicz, V. (1995). A validation of Belbin's team roles from 16PF and OPQ using bosses' ratings of competence. Journal of Occupational and Organizational Psychology, 68, 81—99.
23.
Dwight, S.A., & Fiegelson, M.E. (2000). A quantitative review of the effect of computerized testing on the measurement of social desirability . Educational and Psychological Measurement, 60, 340—360.
24.
Ellingson, J.E., Smith, D.B., & Sackett, P.R. (2001). Investigating the influence of social desirability on personality factor structure. Journal of Applied Psychology , 86, 122—133.
25.
Facteau, J.D., & Craig, S.B. (2001). Are performance appraisal ratings from different rating sources comparable?Journal of Applied Psychology, 86, 215—227.
26.
Ferguson, E., Payne, T., & Anderson, N. (1994). Occupational personality assessment: Theory, structure and psychometrics of the OPQ FMX5-Student. Personality and Individual Differences, 17, 217—225.
27.
Golembiewski, R.T., Billingsley, K., & Yeager, S. (1975). Measuring change and persistence in human affairs: Types of change generated by OD designs. Journal of Applied Behavioral Science, 12, 133—157.
28.
Hacker, K.L., & Steiner, R. (2002). The digital divide for Hispanic Americans. Howard Journal of Communications, 13, 267—283.
29.
Harris, M.M. (2001). Trends and issues in I-O Psychology: A glimpse into the crystal ball. The Industrial-Organizational Psychologist , 38(4), 70—73.
30.
Harris, M.M. (2003). Speeding down the information highway: Internet recruitment and testing. The Industrial-Organizational Psychologist , 41(2), 103—106.
31.
Hogan, R., Hogan, J., & Roberts, B.W. (1996). Personality measurement and employment decisions . American Psychologist, 51, 469—477.
32.
Horn, J.L., & McArdle, J.J. (1992). A practical and theoretical guide to measurement invariance in aging research. ExperimentalAging Research , 18(3-4), 117—144.
33.
Hu, L.T., & Bentler, P.M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424—453.
34.
Jackson, L.A., Eye, A.V., Barbatsis, G., Biocca, F., Zhao, Y., & Fitzgerald, H.E. (2003). Internet attitudes and Internet use: Some surprising findings from the HomeNetToo project. International Journal of Human-ComputerStudies, 59, 355—382.
35.
Jöreskog, K.G. (1993). Testing structural equation models. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models. Newbury Park, CA: Sage.
36.
Jöreskog, K., & Sörbom, D. (1996). LISREL 8: Users reference guide. Chicago: Scientific Software International.
37.
Lautenschlager, G.J. (1989). ALPHATST: Testing for differences in coefficient alpha. Applied Psychological Measurement, 13, 284.
38.
Lautenschlager, G.J., & Flaherty, V.L. (1990). Computer administration of questions: More desirable or more social desirability?Journal of Applied Psychology, 75, 310—314.
39.
Lievens, F., & Harris, M.M. (2003). Research on Internet recruitment and testing: Current status and future directions. In C. L. Cooper & I. T. Robertson (Eds.), International review of industrial and organizational psychology (Vol. 18). New York: Wiley.
40.
Lievens, F., Van Dam, K., & Anderson, N. (2002). Recent trends and challenges in personnel selection . Personnel Review, 31, 580—601.
41.
Loges, W.E., & Jung, J.Y. (2001). Exploring the digital divide: Internet connectedness and age. Communication Research, 28, 536—562.
42.
Martin, C.L., & Nagao, D.H. (1989). Some effects of computerized interviewing on job applicant responses. Journal of Applied Psychology, 74, 72—80.
43.
Matthews, G., & Stanton, N. (1994). Item and scale factor analyses of the Occupational Personality Questionnaire. Personality and Individual Differences , 16, 733—743.
44.
McBride, J.R. (1998). Innovations in computer-based ability testing: Promise, problems, and perils. In M. D. Hakel (Ed.), Beyond multiple choice: Evaluating alternatives to traditional testing for selection (pp. 23-39). Mahwah, NJ: Lawrence Erlbaum.
45.
McFarland, L.A., Ryan, A.M., & Paul, K.B. (1998, April). Equivalence of an organizational attitude survey across administration modes. Paper presented at the 13th annual Society for Industrial and Organizational Psychology conference, Dallas, TX.
46.
Meade, A. W., & Kroustalis, C. M. (in press). Problems with item parceling for confirmatory factor analytic tests of measurement invariance. Organizational Research Methods.
47.
Meade, A.W., & Lautenschlager, G.J. (2004). A Monte-Carlo study of confirmatory factor analytic tests of measurement equivalence/invariance . Structural Equation Modeling, 11, 60—72.
48.
Meade, A.W., Lautenschlager, G.J., Michels, L.C., & Gentry, W.A. (2003, April). Equivalence of online and paper and pencil assessments. In A.W. Meade (Chair), Applications of IRT to measurement in organizations. Symposium presented at the annual Society for Industrial and OrganizationalPsychology conference, Orlando, FL.
49.
Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525—543.
50.
Michels, L.C., Meade, A.W., Lautenschlager, G.J., & Gentry, W.A. (2003, May). The effect of test format and respondent choice on socially desirable responding. Paper presented at the 15th annual meeting of the American Psychological Society, Atlanta , GA.
51.
Mount, M.K. & Barrick, M.R. (1998). Five reasons why the “Big Five” article has been frequently cited. Personnel Psychology, 98, 849—858.
52.
Park, D., Smith, A., Lautenschlager, G.J., Earles, J., Frieske, D., Zwahr, M., & Gaines, C. (1996). Mediators of long-term memory performance across the life span. Psychology and Aging, 11, 621—637.
53.
Pew Internet and AmericanLife Project. (2000 a). Who's not online: 57% of those without Internet access say they do not plan to log on. Washington, DC: Pew Foundation. Retrieved from http://www.pewInternet.org/.
54.
Pew Internet and American Life Project. (2000 b). African Americans and the Internet. Washington, DC: Pew Foundation. Retrieved from http://www.pewInternet.org/.
55.
Ployhart, R.W., & Oswald, F.L. (2004). Applications of mean and covariance structure analysis: Integrating correlational and experimental approaches. Organizational Research Methods, 7, 27—65.
56.
Ployhart, R.E., Weekley, J.A., Holtz, B.C., & Kemp, C. (2003). Web-based and paper-and-pencil testing of applicants in a proctored setting: Are personality, biodata, and situational judgment tests comparable?Personnel Psychology, 56, 733—752.
57.
Potosky, D., & Bobko, P. (1997). Computer versus paper-and-pencil mode and response distortion in noncognitive selection tests. Journal of Applied Psychology, 82, 293—299.
58.
Potosky, D., & Bobko, P. (2004). Selection testing via the Internet: Practical considerations and exploratory empirical findings. Personnel Psychology, 57, 1003—1034.
59.
Rensvold, R.B., & Cheung, G.W. (2000, August). Beyond two-group comparisons: Identifying sets of invariant groups. Paper presented at the annual Academy of Management meeting, Toronto, Ontario, Canada.
60.
Richman, W.L., Kiesler, S., Weisband, S., & Drasgow, F. (1999). A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires, and interviews . Journal of Applied Psychology, 84, 754—775.
61.
Rigdon, E.E. (1996). CFI vs. RMSEA: A comparison of two fit indexes for structural equation modeling. Structural Equation Modeling , 3, 369—379.
62.
Riordan, C.M., & Vandenberg, R.J. (1994). A central question in cross-cultural research: Do employees of different cultures interpret work-related measures in an equivalent manner?Journal of Management, 20, 643—671.
63.
Robert, C., Probst, T.M., Martocchio, J.H., Drasgow, F., & Lawler, J.J. (2000). Empowerment and continuous improvement in the United States, Mexico, Poland, and India: Predicting fit on the basis of the dimensions of power distance and individualism. Journal of Applied Psychology, 85, 643—658.
64.
Rosenfeld, P., Doherty, L.M., Vicino, S.M., Kantor, J., & Greaves, J. (1989). Attitude assessment in organizations: Testing three microcomputer-based survey systems. Journal of General Psychology, 116, 145—154.
65.
Salgado, J.F. (1996). Personality and job competences: A comment on the Robertson & Kinder (1993) study. Journal of Occupational and Organizational Psychology, 69, 373—375.
66.
Salgado, J.F., & Moscoso, S. (2003). Internet-based personality testing: Equivalence of measures and assessees' perceptions and reactions. International Journal of Selection and Assessment, 11, 194—205.
67.
Saville, P., Sik, G., Nyfield, G., Hackston, J., & Maciver, R. (1996). A demonstration of the validity of the Occupational Personality Questionnaire (OPQ) in the measurement of job competencies across time and in separate organizations. Applied Psychology: An International Review, 45, 243—262.
68.
Schmit, M.J., & Ryan, A.M. (1993). The Big Five in personnel selection: Factor structure in applicant and nonapplicant populations. Journal of Applied Psychology, 78, 966—974.
69.
Sharf, J. (2000). As if “g-loaded” adverse impact isn't bad enough, Internet recruiters can expect to be accused of “e-loaded” impact. The Industrial-Organizational Psychologist, 38(2), 156.
Smith, D.B., Hanges, P.J., & Dickson, M.W. (2001). Personnel selection and the five-factor model: Reexamining the effects of applicant's frame of reference. Journal of Applied Psychology, 86, 304—315.
72.
Spera, S.D., & Moye, N.A. (2001, April). Paper and Web survey methods in a multinational company. Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, San Diego, CA.
73.
Stanton, J.M. (1998). An empirical assessment of data collection using the Internet. Personnel Psychology, 51, 709—726.
74.
Steiger, J.H. (2000). Point estimation, hypothesis testing, and interval estimation using the RMSEA: Some comments and a reply to Hayduck and Glaser . Structural Equation Modeling, 7, 149—162.
75.
Taris, T.W., Bok, I.A., & Meijer, Z.Y. (1998). Assessing stability and change of psychometric properties of multi-item concepts across different situations: A general approach . Journal of Psychology, 132, 301—316.
76.
Vandenberg, R.J., & Lance, C.E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4—69.
77.
West, S.G., Finch, J.F., & Curran, P.J. (1995). Structural equation models with nonnormal variables . In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications. Thousand Oaks, CA: Sage.
78.
Williams, L.J., & Holahan, P.J. (1994). Parsimony-based fit indices for multiple-indicator models: Do they work?Structural Equation Modeling, 1, 161—189.
79.
Yost, P.R., & Homer, L.E. (1998, April). Electronic versus paper surveys: Does the medium affect the response? Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, Dallas, TX.
80.
Young, S.A., Daum, D.L., Robie, C., & Macey, W.H. (2000). Paper versus Web survey administration: Do different methods yield different results? Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, New Orleans, LA.
