Two quasi-experimental field studies were conducted to evaluate the psychometric equivalence of computerized and paper-and-pencil job satisfaction measures. The present research extends previous work in the area by providing better control of common threats to validity in quasi-experimental research on test mode effects and by evaluating a more comprehensive measurement model for job attitudes. Results of both studies demonstrated substantial equivalence of the computerized measure with the paper-and-pencil version. Implications for the practical use of computerized organizational attitude surveys are discussed.
Bartram, D. , & Bayliss, R. (1984). Automated testings: Past, present, and future . Journal of Occupational Psychology, 57, 221-237 .
2.
Bentler, P.M. (1988). Causal modeling via structural equation systems . In J. R. Nesselroade & R. B. Cattell (Eds.), Handbook of multivariate experimental psychology. Perspectives on individual differences (2nd ed., pp. 317-335). New York: Plenum .
3.
Bentler, P.M. , & Bonett, D.G. (1980). Significance tests and goodness of fit in the analysis of covariance structures . Psychological Bulletin,88, 588-606 .
4.
Bollen, K.A. (1989). Structural equations with latent variables. Oxford, UK: John Wiley .
Booth-Kewley, S. , Edwards, J.E. , & Rosenfeld, P. (1992). Impression management, social desirability, and computer administration of attitude questionnaires: Does the computer make a difference ? Journal of Applied Psychology, 77, 562-566 .
7.
Burke, M.J. (1992). Computerized psychological testing: Impacts on measuring predictor constructs and future job behavior. In N. Schmitt & W. C. Borman (Eds.), Personnel selection in organizations (pp. 203-238). San Francisco: Jossey-Bass .
8.
Byrne, B.M. , Shavelson, R.J. , & Muthen, B. (1989). Testing for the equivalence of factor covariance and mean structures. The issue of partial measurement invariance . Psychological Bulletin, 105, 456-466 .
9.
Cheung, G.W. , & Rensvold, R.B. (1999). Testing factorial invariance across groups: A reconceptualization and proposed new method . Journal of Management, 25, 1-27 .
Cook, T.D. , & Campbell, T.L. (1979). Quasi-experimentation: Design & analysis issues for field settings. Boston : Houghton-Mifflin .
12.
Davis, C. , & Cowles, M. (1989). Automated psychological testing: Method of administration, need for approval, and measures of anxiety . Educational and Psychological Measurement, 49, 311-320 .
13.
Davis, R.N. (1999). Web-based administration of a personality questionnaire: Comparison with traditional methods . Behavior Research Methods, Instruments & Computers, 31, 572-577 .
14.
Dieffendorff, J.M. , Silverman, S.B. & Greguras, G.J. (2005). Measurement equivalence and multisource ratings for non-managerial positions: Recommendations for research and practice . Journal of Business and Psychology , 19, 399-425 .
15.
Donovan, M.A. , Drasgow, F. , & Probst, T.M. (2000). Does computerizing paper-and-pencil job attitude scales make a difference? New IRT analyses offer insight . Journal of Applied Psychology, 85, 305-313 .
16.
Drasgow, F. (1984). Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables are the central issues . Psychological Bulletin, 95, 134-135 .
17.
Drasgow, F. , & Kanfer, R. (1985). Equivalence of psychological measurement in heterogeneous populations . Journal of Applied Psychology, 70, 662-680 .
18.
Dwight, S.A. , & Feigelson, M.E. (2000). A quantitative review of the effect of computerized testing on the measurement of social desirability . Educational and Psychological Measurement, 60, 340-360 .
19.
Erdman, H. , Klein, M.H. , & Greist, J.H. (1983). The reliability of a computer interview for drug-use abuse information . Behavior Research Methods & Instrumentation , 15, 66-68 .
20.
Fox, S. , & Schwartz, D. (2002). Social desirability and controllability in computerized and paper-and-pencil personality questionnaires . Computers in Human Behavior, 18, 389-410 .
21.
Henson, R.K. (2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha . Measurement & Evaluation in Counseling & Development, 34, 177-189 .
22.
Horn, J.L. , & McArdle, J.J. (1992). A practical and theoretical guide to measurement invariance in aging research. ExperimentalAging Research , 18, 117-144 .
23.
Hu, L. , & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives . Structural Equation Modeling, 6, 1-55 .
24.
Joinson, A. (1999). Social desirability, anonymity, and Internet-based questionnaires . Behavior Research Methods, Instruments & Computers, 31, 433-438 .
25.
Kantor, J. (1991). The effects of computer administration and identification on the job descriptive index (JDI) . Journal of Business and Psychology, 5, 309-323 .
26.
Katz, L. , & Dalby, J.T. (1981). Computer and manual administration of the Eysenck Personality Inventory . Journal of Clinical Psychology, 37, 586-588 .
27.
Kiesler, S. , & Sproull, L.S. (1986). Response effects in the electronic survey . Public Opinion Quarterly, 50, 402-413 .
28.
King, W.C. , & Miles, E.W. (1995). A quasi-experimental assessment of the effect of computerizing noncognitive paper-and-pencil measurements: A test of measurement equivalence . Journal of Applied Psychology, 80, 643-651 .
29.
Kline, R.B. (1998). Principles and practice of structural equation modeling. New York: Guilford .
30.
Knapp, H. , & Kirk, S.A. (2003). Using pencil and paper, Internet and touch-tone phones for self-administered surveys: Does methodology matter? Computers in Human Behavior, 19, 117-134 .
31.
Lautenschlager, G.J. , & Flaherty, V.L. (1990). Computer administration of questions: More desirable or more social desirability?Journal of Applied Psychology, 75, 310-314 .
32.
Little, T.D. (2000). On the comparability of constructs in cross-cultural research. A critique of Cheung and Rensvold . Journal of Cross-Cultural Psychology, 31, 213-219 .
33.
Locke, S.D. , & Gilbert, B.O. (1995). Method of psychological-assessment, self-disclosure, and experiential differences. A study of computer, questionnaire, and interview assessment formats . Journal of Social Behavior and Personality , 10, 255-263 .
34.
Lushene, R.E. , O'Neil, H.F. , & Dunn, T. (1974). Equivalent validity of a completely computerized MMPI . Journal of Personality Assessment, 38, 353-361 .
35.
Martin, C.L. , & Nagao, D.H. (1989). Some effects of computerized interviewing on job applicant responses . Journal of Applied Psychology, 74, 72-80 .
36.
Mead, A.D. , & Drasgow, F. (1993). Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis . Psychological Bulletin , 114, 449-458 .
37.
Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance . Psychometrika, 58, 525-543 .
38.
Millstein, S.G. (1987). Acceptability and reliability of sensitive information collected via computer interview . Educational and Psychological Measurement, 47, 523-533 .
Niehoff, B.P. , & Moorman, R.H. (1993). Justice as a mediator of the relationship between methods of monitoring and organizational citizenship behavior . Academy of Management Journal, 36, 527-556 .
41.
Ployhart, R.E. , Weekley, J.A. , Holtz, B.C. , & Kemp, C. (2003). Web-based and paper-and-pencil testing of applicants in a proctored setting: Are personality, biodata, and situational judgment tests comparable ? Personnel Psychology, 56, 733-752 .
42.
Potosky, D. , & Bobko, P. (1997). Computer versus paper-and-pencil administration mode and response distortion in noncognitive selection tests . Journal of Applied Psychology, 82, 293-299 .
43.
Reise, S.P. , Widaman, K.F. , & Pugh, R.H. (1993). Confirmatory factor analysis and item response theory: Two approaches for exploring measurement invariance . Psychological Bulletin,114, 552-566 .
44.
Richman, W.L. , Kiesler, S. , Weisband, S. , & Drasgow, F. (1999). A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires, and interviews . Journal of Applied Psychology, 84, 754-775 .
45.
Riordan, C.M. , & Vandenberg, R.J. (1994). A central question in cross-cultural research: Do employees of different cultures interpret work-related measures in an equivalent manner?Journal of Management, 20, 643-671 .
46.
Rosenfeld, P. , Doherty, L.M. , & Carroll, L. (1987). Microcomputer-based organizational survey assessment: Applications to training . Journal of Business and Psychology , 2, 182-193 .
47.
Rosenfeld, P. , Doherty, L.M. , Vicino, S.M. , Kantor, J. , & Greaves, J. (1989). Attitude assessment in organizations—Testing three microcomputer-based survey-systems . Journal of General Psychology, 116, 145-154 .
48.
Rosenfeld, P. , Giacalone, R.A. , Knouse, S.B. , Doherty, L.M. , Vicino, S.M. , Kantor, J. , et al. (1991). Impression management, candor, and microcomputer-based organizational surveys—an individual-differences approach. Computers in Human Behavior, 7, 23-32 .
49.
Ryan, A.M. , Chan, D. , Ployhart, R.E. , & Slade, L.A. (1999). Employee attitude surveys in a multinational organization: Considering language and culture in assessing measurement equivalence . Personnel Psychology, 52, 37-58 .
50.
Schuldberg, D. (1988). The MMPI is less sensitive to the automated testing format than it is to repeated testing: Item and scale effects . Computers in Human Behavior, 4, 285-298 .
51.
Shadish, W.R. , Cook, T.D. , & Campbell, T.L. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton-Mifflin .
52.
Smith, P.C. , Kendall, L.M. , & Hulin, C.L. (1969). The measurement of satisfaction in work and retirement: A strategy for the study of attitudes. Oxford , UK: Rand McNally .
53.
Stanton, J.M. (1998). An empirical assessment of data collection using the Internet . Personnel Psychology, 51, 709-725 .
54.
Steenkamp, J.-B.E.M. , & Baumgartner, H. (1998). Assessing measurement invariance in cross-national consumer research . Journal of Consumer Research , 25, 78-90 .
55.
Steiger, J.H. (1990). Structural model evaluation and modification: An interval estimation approach . Multivariate Behavioral Research , 25, 173-180 .
56.
Synodinos, N.E. , & Brennan, J.M. (1990). Evaluating microcomputer interactive survey software . Journal of Business and Psychology, 4, 483-493 .
57.
Synodinos, N.E. , Papacostas, C.S. , & Okimoto, G.M. (1994). Computer-administered versus paper-and-pencil surveys and the effect of sample selection . Behavior Research Methods, Instruments, & Computers, 26, 395-401 .
58.
Turner, C.F. , Ku, L. , Rogers, S.M. , Lindberg, L.D. , Pleck, J.H. , & Sonenstein, F.L. (1998). Adolescent sexual behavior, drug use, and violence: Increased reporting with computer survey technology . Science, 280, 867-873 .
59.
Vandenberg, R.J. , & Lance, C.E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research . Organizational Research Methods, 3, 4-69 .
60.
van de Vijver, F.J.R. , & Poortinga, Y.H. (1997). Towards an integrated analysis of bias in cross-cultural assessment . European Journal of Psychological Assessment, 13, 29-37 .
61.
Vansickle, T.R. , Kimmel, C. , & Kapes, J.T. (1989). Test retest equivalency of the computer-based and paper pencil versions of the Strong-Campbell Interest Inventory . Measurement and Evaluation in Counseling and Development, 22, 88-93 .
62.
Weiss, D.J. , Dawis, R.V. , England, G.W. , & Lofquist, L.H. (1967). Manual for the Minnesota Satisfaction Questionnaire. Minneapolis: University of Minnesota Press .
63.
White, D.M. , Clements, C.B. , & Fowler, R.D. (1985). A comparison of computer administration with standard administration of the MMPI. Computers in Human Behavior, 1, 153-162 .
64.
Widaman, K.F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data . Applied Psychological Measurement , 9, 1-26 .
65.
Wilkerson, J.M. , Nagao, D.H. , & Martin, C.L. (2002). Socially desirable responding in computerized questionnaires: When questionnaire purpose matters more than the mode . Journal of Applied Social Psychology, 32, 544-559 .
66.
Wilson, F.R. , Genco, K.T. , & Yager, G.G. (1985). Assessing the equivalence of paper-and-pencil vs. computerized tests: Demonstration of a promising methodology. Computers in Human Behavior, 1, 265-275 .
