Multigroup confirmatory factor analysis was applied to the responses of 4, 909 employees of a multinational organization with locations in 50 countries to examine the measurement equivalence of otherwise identical Web-based and paper-and-pencil versions of 20 items comprising the transformational leadership component of Bass and Avolio's Multifactor Leadership Questionnaire. The results supported configural, metric, scalar, measurement error, and relational equivalence across administration modes, indicating that the psychometric properties of the 20 items were similar whether administered as a paper-and-pencil or Web-based measure. Although caution is always advised when considering multiple modes of administration, the results suggest that there are minimal measurement differences for well-developed, psychometrically sound instruments applied using either a paper-and-pencil or an online format. Thus, the results open a methodological door for survey researchers wishing to (a) assess transformational leadership with a Web-based platform and (b) compare or combine responses collected with paper-and-pencil and Web-based applications.
American Psychological Association. (2002). Ethical principles of psychologists and code of conduct. American Psychologist, 57, 1060-1073.
2.
Antonakis, J., Avolio, B. J., & Sivasubramaniam, N. (2003). Context and leadership: An examination of the nine-factor full-range leadership theory using the multifactor leadership questionnaire. Leadership Quarterly, 14, 261-295.
3.
Arbuckle, J. L. (1996). Full information estimation in the presence of missing data. In G. A. Marcoulides & R. E. Schumaker (Eds.), Advanced structural equation modeling: Issues and techniques (pp. 243-277). Mahwah, NJ: Erlbaum.
4.
Arbuckle, J. L., & Wothke, W. (1999). Amos 4.0 user's guide. Chicago: Small Waters.
5.
Asci, F. H., Eklund, R. C., Whitehead, J. R., Kirazci, S., & Koca, C. (2005). Use of the CY-PSPP in other cultures: A preliminary investigation of its factorial validity for Turkish children and youth. Psychology of Sport and Exercise, 6, 33-50.
6.
Avolio, B. J., Bass, B. M., & Jung, D. I. (1999). Re-examining the components of transformational and transactional leadership using the Multifactor Leadership Questionnaire. Journal of Occupational and Organizational Psychology, 72, 441-462.
7.
Bass, B. M. (1985). Leadership and performance beyond expectations. New York: Free Press.
8.
Bass, B. M. (1997). Does the transactional-transformational leadership paradigm transcend organizational and national boundaries?American Psychologist, 52, 130-139.
9.
Bass, B. M., & Avolio, B. J. (2000). Multifactor Leadership Questionnaire: Technical report, leader form, rater form, and scoring key for MLQ form 5x-short(2nd ed.). Redwood City, CA: Mindgarden.
10.
Booth-Kewley, S., Edwards, J. E., & Rosenfeld, P. (1992). Impression management, social desirability, and computer administration of attitude questionnaires: Does the computer make the difference?Journal of Applied Psychology, 77, 562-566.
11.
Brannick, M. T. (1995). Critical comments on applying covariance structure modeling. Journal of Organizational Behavior, 16, 201-213.
12.
Buchanan, T. (2002). Online assessment: Desirable or dangerous?Professional Psychology: Research and Practice, 33, 148-154.
13.
Buchanan, T., Johnson, J. A., & Goldberg, L. R. (2005). Implementing a five-factor personality inventory for use on the Internet. European Journal of Psychological Assessment, 21, 115-127.
14.
Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, 125-144.
15.
Bycio, P., Hackett, R. D., & Allen, J. S. (1995). Further assessments of Bass's (1985) conceptualization of transactional and transformational leadership. Journal of Applied Psychology, 80, 468-478.
16.
Byrne, B. M. (2001). Structural equation modeling with AMOS: Basic concepts, applications, and programming. Mahwah, NJ: Erlbaum.
17.
Byrne, B. M., Shavelson, R. J., & Muthén, B. (1989). Testing for equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105, 456-466.
18.
Carless, S. A. (1998). Assessing the discriminant validity of transformational leadership behavior as measured by the MLQ. Journal of Occupational and Organizational Psychology, 71, 353-358.
19.
Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246.
20.
Chen, F., Bollen, K. A., Paxton, P., Curran, P. J., & Kirby, J. B. (2001). Improper solutions in structural equation models: Causes, consequences, and strategies. Sociological Methods and Research, 29, 468-508.
21.
Cheung, G. W., & Rensvold, R. B. (1999). Testing factorial invariance across groups: A reconceptualization and proposed new method. Journal of Management, 25, 1-27.
22.
Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255.
23.
Cohen, R. J., Swerdlik, M. E., & Phillips, S. M. (1996). Psychological testing and assessment (3rd ed.). Mountain View, CA: Mayfield.
24.
Den Hartog, D. N., House, R. J., Hanges, P. J., Ruiz-Quintanilla, S. A., & Dorfman, P. W. (1999). Culture specific and cross-culturally generalizable implicit leadership theories: Are attributes of charisma/transformational leadership universally endorsed?Leadership Quarterly, 10, 219-256.
25.
Den Hartog, D. N., Van Muijen, J. J., & Koopman, P. L. (1997). Transactional versus transformational leadership: An analysis of the MLQ. Journal of Occupational and Organizational Psychology, 70, 19-34.
26.
Dillon, W. R., Kumar, A., & Mulani, N. (1987). Offending estimates in covariance structure analysis: Comments on the causes of and solutions to Heywood cases. Psychological Bulletin, 101, 126-135.
27.
Donovan, M. A., Drasgow, F., & Probst, T. M. (2000). Does computerizing paper-and-pencil job attitude scales make a difference? New IRT analyses offer insight. Journal of Applied Psychology, 85, 305-313.
28.
Drasgow, F. (1984). Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables are the central issue. Psychological Bulletin, 95, 134-135.
29.
Eaton, J., & Struthers, C. W. (2002). Using the internet for organizational research: A study of cynicism in the workplace. CyberPsychology and Behavior, 5, 305-313.
30.
Epstein, J., Klinkenberg, W. D., Wiley, D., & McKinley, L. (2001). Insuring sample equivalence across Internet and paper-and-pencil assessments. Computers in Human Behavior, 17, 339-346.
31.
Evan, W. M., & Miller, J. R. (1969). Differential effects on response bias of computer vs. conventional administration of a social science questionnaire: An exploratory methodological experiment. Behavioral Science, 14, 216-227.
32.
Fouladi, R. T., McCarthy, C. J., & Moller, N. P. (2002). Paper-and-pencil or online? Evaluating mode effects on measures of emotional functioning and attachment. Assessment, 9, 204-215.
33.
French, D. J., & Tait, R. J. (2004). Measurement invariance in the General Health Questionnaire-12 in young Australian adolescents. European Child and Adolescent Psychiatry, 13, 1-7.
34.
Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should we trust Web-based studies? A comparative analysis of six preconceptions about Internet questionnaires. American Psychologist, 59, 93-104.
35.
Herrero, J., & Meneses, J. (in press). Short Web-based versions of the Perceived Stress (PSS) and Center for Epidemiological Studies–Depression (CESD) scales: A comparison to pencil-and-paper responses among Internet users. Computers in Human Behavior.
36.
Hogg, A. (2002). Conducting online research. Burk Incorporated White Paper Series, 3(2). Retrieved July 8, 2004, from http://www.burke.com/whitepapers/PDF/B.WhitePaperVol3-2002-Iss2.pdf
37.
Hollenbeck, J. R., Klein, H. J., O'Leary, A. M., & Wright, P. M. (1989). Investigation of the construct validity of a self-report measure of goal commitment. Journal of Applied Psychology, 74, 951-956.
38.
Hu, L., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424-453.
39.
King, W. C., Jr., & Miles, E. W. (1995). A quasi-experimental assessment of the effect of computerizing noncognitive paper-and-pencil measurements: A test of measurement equivalence. Journal of Applied Psychology, 80, 643-651.
40.
Knapp, H., & Kirk, S. A. (2003). Using pencil and paper, Internet, and touch-tone phones for self-administered surveys: Does methodology matter?Computers in Human Behavior, 19, 117-134.
41.
Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M. (2004). Psychological research online: Report of board of scientific affairs' advisory group on the conduct of research on the Internet. American Psychologist, 59, 105-117.
42.
Lautenschlager, G. J., & Flaherty, V. L. (1990). Computer administration of questions: More desirable or more social desirability?Journal of Applied Psychology, 75, 310-314.
43.
Lawley, D. N., & Maxwell, A. E. (1971). Factor analysis as a statistical method(2nd ed.). London: Butterworth.
44.
Lievens. F., & Anseel, F. (2004). Confirmatory factor analysis and invariance of an organizational citizenship behavior measure across samples in a Dutch-speaking context. Journal of Occupational and Organizational Psychology, 77, 299-306.
45.
Little, T. D. (1997). Mean and covariance structures (MACS) analyses of cross-cultural data: Practical and theoretical issues. Multivariate Behavioral Research, 32, 53-76.
46.
Lowe, K. B., Kroeck, K. G., & Sivasubramaniam, N. (1996). Effectiveness correlates of transformation and transactional leadership: A meta-analytic review of the MLQ literature. Leadership Quarterly, 7, 385-425.
47.
Mantzicopoulos, P., French, B. F., & Maller, S. J. (2004). Factor structure of the pictorial scale of perceived competence and social acceptance with two pre-elementary samples. Child Development, 75, 1214-1228.
48.
Marsh, H. W., Balla, J. R., & McDonald, R. P. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103, 391-410.
49.
Meade, A. W., & Lautenschlager, G. J. (2004a). A comparison of item response theory and confirmatory factor analytic methodologies for establishing measurement equivalence/invariance. Organizational Research Methods, 7, 361-388.
50.
Meade, A. W., & Lautenschlager, G. J. (2004b). A Monte-Carlo study of confirmatory factor analytic tests of measurement equivalence/invariance. Structural Equation Modeling, 11, 60-72.
51.
Millsap, R. E., & & Kwok, O. (2004). Evaluating the impact of partial factorial invariance on selection in two populations. Psychological Methods, 9, 93-115.
52.
Mullen, M. R. (1995). Diagnosing measurement equivalence in cross-national research. Journal of International Business Studies, 26, 573-596.
53.
Naglieri, J. A., Drasgow, F., Schmit, M., et al. (2004). Psychological testing on the Internet: New problems, old issues. American Psychologist, 59, 150-162.
54.
Ployhart, R. E., & Oswald, F. L. (2004). Applications of mean and covariance structure analysis: Integrating correlational and experimental approaches. Organizational Research Methods, 7, 27-65.
55.
Ployhart, R. E., Wiechmann, D., Schmitt, N., Sacco, J. M., & Rogg, K. (2003). The cross-cultural equivalence of job performance ratings. Human Performance, 16, 49-79.
56.
Potosky, D., & Bobko, P. (1997). Computer versus paper-and-pencil administration mode and response distortion in noncognitive selection tests. Journal of Applied Psychology, 82, 293-299.
57.
Richman, W. L., Kiesler, S., Weisband, S., & Drasgow, F. (1999). A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires, and interviews. Journal of Applied Psychology, 84, 754-775.
58.
Riggs, M. L., & Knight, P. A. (1994). The impact of perceived group success-failure on motivational beliefs and attitudes: A causal study. Journal of Applied Psychology, 79, 755-766.
59.
Riordan, C. M., & Vandenberg, R. J. (1994). A central question in cross-cultural research: Do employees of different cultures interpret work-related measures in an equivalent manner?Journal of Management, 20, 643-671.
60.
Riordan, C. M., & Weatherly, E. W. (1999). Defining and measuring employees' identification with their work groups. Educational and Psychological Measurement, 59, 310-324.
61.
Rosenfeld, P., Booth-Kewley, S., Edwards, J. E., & Thomas, M. D. (1996). Responses on computer surveys: Impression management, social desirability, and the Big Brother syndrome. Computers in Human Behavior, 12, 263-274.
62.
Ryan, A. M., Chan, D., Ployhart, R. E., & Slade, L. A. (1999). Employee attitude surveys in a multinational organization: Considering language and culture in assessing measurement equivalence. Personnel Psychology, 52, 37-58.
63.
Scandura, T., & Dorfman, P. (2004). Leadership research in an international and cross-cultural context. Leadership Quarterly, 15, 277-307.
64.
Simsek, Z., & Veiga, J. F. (2001). A primer on Internet organizational surveys. Organizational Research Methods, 4, 218-235.
65.
Stanton, J. M. (1998). An empirical assessment of data collection using the Internet. Personnel Psychology, 51, 709-725.
66.
Stanton, J. M., & Rogelberg, S. G. (2001). Using Internet/Intranet Web pages to collect organizational research data. Organizational Research Methods, 4, 200-217.
67.
Steenkamp, J. B. E. M., & Baumgartner, H. (1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25, 78-90.
68.
Synodinos, N. E., & Brennan, J. M. (1990). Evaluating microcomputer interactive survey software. Journal of Business and Psychology, 4, 483-493.
69.
Thompson, L. F., Surface, E. A., Martin, D. L., & Sanders, M. G. (2003). From paper to pixels: Moving personnel surveys to the Web. Personnel Psychology, 56, 197-227.
70.
Vandenberg, R. J. (2002). Toward a further understanding of and improvement in measurement invariance methods and procedures. Organizational Research Methods, 5, 139-158.
71.
Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-70.
72.
Whitener, E. M., & Klein, H. J. (1995). Equivalence of computerized and traditional research methods: The roles of scanning, social environment, and social desirability. Computers in Human Behavior, 11, 65-75.
73.
Wothke, W. (2000). Longitudinal and multigroup modeling with missing data. In T. D. Little, K. U. Schnable, & J. Baumert (Eds.), Modeling longitudinal and multilevel data(pp. 219-240). Mahwah, NJ: Erlbaum.
74.
Zaccaro, S. J., Rittman, A. L., & Marks, M. A. (2001). Team leadership. Leadership Quarterly, 12, 451-483.
