An Exploratory and Confirmatory Factor-Analytic Investigation of Item Wording Effects on the Obtained Factor Structures of Survey Questionnaire Measures

Abstract

Item wording effects were investigated using scenarios depicting a fictitious leaders behavior, 496 respondents, and a questionnaire containing regular, polar opposite, negated polar opposite, and negated regular item versions. Oblique-rotated exploratory factor-analytic results showed clear item wording factors; confirmatory factor-analytic results showed the item formats to yield separate wording factors but that the regular items had substantially more trait variance than did the other item formats. Implications for future research are discussed.

Get full access to this article

View all access options for this article.

References

Anastasi, A. (1982). Psychological resting, 5th ed. NY: Macmillan.

Benson, J. & Hocevar, D. (1985). The impact of item phrasing on the validity of attitude scales for elementary school children. Journal of Educational Measurement, 22: 213-240.

Bentler, P.M. (1990). Comparative tit indexes in structural models. Psychological Bulletin, 107: 238-246.

Bentler, P.M. & Bonett, D.G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88: 588-606.

Bollen, K.A. (1989). Structural equations with latent variables. New York: Wiley.

Braylield, A.H. & Rothe, H.F. (1951). An index of job satisfaction. Journal of Applied Psychology, 35: 307-311.

Campbell, D.T. & Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56: 81-105.

Campbell, N.O. & Grissom, S. (1979). Influence of item direction on student responses in attitude assessment. Paper presented at the 63rd annual meeting of the American Educational Research Association, San Francisco, CA, April.

Cattell, R.B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1: 245-276.

10.

Cordery, J.L. & Sevastos, P.P. (1993). Responses to the original and revised job diagnostic survey: Is education a factor in responses to negatively worded items? Journal of Applied Psychology, 78: 141-143.

11.

Cudeck, R. (1989). Analysis of correlation matrices using covariance structure models. Psychological Bulletin, 105: 317-327.

12.

Cuttance, P. (1987). Issues and problems in the application of structural equation models. Pp. 241-279 in P. Cuttance & R. Ecob (Eds.), Structural modeling by example. New York: Cambridge University Press.

13.

Fornell, C. & Larcker, D.F. (1981). Evaluating structural equation models with observable variables and measurement error. Journal of Marketing Research, 18: 39-50.

14.

Gerbing, D.W. & Anderson, J.C. (1992). Monte Carlo evaluations of goodness of fit indices for structural equation models. Sociological Methodr and Research, 21: 132-160.

15.

Goffin, R.D. (1993). A comparison of two new indices for the assessment of fit of structural equation models. Multivariate Behavioral Research, 28: 205-214.

16.

Hackman, J.R. & Oldham, G.R. (1975). Development of the Job Diagnostic Survey. Journal of Applied Psychology, 60: 159-170.

17.

Hair, J.F. Jr. , Anderson, R.E. & Tatham, R.L. (1987). Multivariate data analysis, 2nd ed. New York: Macmillan.

18.

Harman, H.H. (1976). Modern factor analysis, 3rd ed. Chicago: University of Chicago Press.

19.

Harris, M.M. (1991). Role conflict and role ambiguity as substance versus artifact: A confirmatory factor analysis of House, Schuler, and Levanoni’s (1983) scales. Journal of Applied Psychology, 76: 122-126.

20.

Harvey, R.J. , Billings, R.S. & Nilan, K.J. (1985). Confirmatory factor analysis of the Job Diagnostic Survey: Good news and bad news. Journal of Applied Psychology, 70: 461-468.

21.

Hayduk, L.A. (1987). Structural equation modeling with LISREL: Essentials and advances. Baltimore: John Hopkins University Press.

22.

House, R.J. , Schuler, R.S. & Levanoni, E. (1983). Role conflict and ambiguity scales: Reality or artifacts? Journal of Applied Psychology, 68: 334-337.

23.

Idaszak, J.R. , Bottom, W.P. & Drasgow, F. (1988). A test of the measurement equivalence of the revised Job Diagnostic Survey: Past problems and current solutions. Journal of Applied Psychology, 73: 647-656.

24.

Idaszak, J.R. & Drasgow, F. (1987). A revision of the Job Diagnostic Survey: Elimination of a measurement artifact. Journal of Applied Psychology, 72: 69-74.

25.

Joreskog, K.G. & Sorbom, D. (1989). LISREL 7: Users reference guide. Mooresville, IN: Scientific Software, Inc.

26.

Kelloway, E.K. & Barling, J. (1990). Item content versus item wording: Disentangling role conflict and role ambiguity. Journal of Applied Psychology, 75: 738-742.

27.

Kulik, C.T. , Oldham, G.R. & Langer, P.H. (1988). Measurement of job characteristics: Comparison of the original and revised Job Diagnostic Survey. Journal of Applied Psychology, 73: 462-466.

28.

MacCallum, R.C. (1986). Specification searches in covariance structure modeling. Psychological Bulletin, 100: 107-120.

29.

Marsh, H.W. , Balla, J.R. & McDonald, R.P. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103: 391-410.

30.

McDonald, R.P. & Marsh, H.W. (1990). Choosing a multivariate model: Noncentrality and goodness of fit. Psychological Bulletin, 107: 247-255.

31.

McGee, G.W. , Ferguson, C.E. & Seers, A. (1989). Role conflict and role ambiguity: Do the scales measure these two constructs? Journal of Applied Psychology, 74: 815-818.

32.

Medsker, G.J. , Williams, L.J. & Holahan, P.J. (1994). A review of current practices for evaluating causal models in organizational behavior and human resources management research. Journal of Management, 20: 439-464.

33.

Meyer, J.P. & Allen, N.J. (1984). Testing the “side bet theory” of organizational commitment: Some methodological considerations. Journal of Applied Psychology, 69: 372-378.

34.

Miller, T.E. & Cleat-y, T.A. (1993). Direction of wording effects in balanced scales. Educational and Psychological Measurement, 53: 51-60.

35.

Mulaik, S.A. , James, L.R. , Van Alstine, J. , Bennett, N. , Lind, S. & Stilwell, C.D. (1989). Evaluation of goodness-of-fit indices for structural equation models. Psychological Bulletin, 10.5: 430-445.

36.

Nunnally, J.C. (1978). Psychometric theory, 2nd ed. New York: McGraw-Hill.

37.

Pilotte, W.J. & Gable, R.K. (1990). The impact of positive and negative item stems on the validity of a computer anxiety scale. Educational and Psychological Measurement, 50: 603-610.

38.

Porter, L.W. , Steers, R.M. , Mowday, R.T. & Boulian, P.V. (1974). Organizational commitment, job satisfaction, and turnover among psychiatric technicians. Journal of Applied Psychology, 59: 603-609.

39.

Rizzo, J.R. , House, R.J. & Lirtzman, S.E. (1970). Role conflict and ambiguity in complex organizations. Administrative Science Quarterly, 15: 150-163.

40.

Rorer, L.G. (1965). The great response style myth. Psychological Bulletin, 63: 129-156.

41.

Rummel, R.J. (1970). Applied factor analysis. Evanston, IL: Northwestern University Press.

42.

Schmitt, N. & Coyle, B.W. (1976). Applicant decisions in the employment interview. Journal of Applied Psychology, 61: 184-192.

43.

Schmitt, N. & Stults, D.M. (1985). Factors defined by negatively keyed items: The result of careless respondents? Applied Psychological Measurement, 9: 367-373.

44.

Schmitt, N. & Stults, D.M. (1986). Methodology review: Analysis of multitrait-multimethod matrices. Applied Psychological Measurement, 10: 1-22.

45.

Schriesheim, C.A. , Eisenbach, R.J. & Hill, K.D. (1991). The effect of negation and polar opposite item reversals on questionnaire reliability and validity: An experimental investigation. Educational and Psychological Measurement, 51: 67-78.

46.

Schriesheim, CA. & Hill, K.D. (1981). Controlling acquiescence response bias by item reversals: The effect on questionnaire validity. Educational and Psychological Measurement, 41: 1101-1114.

47.

Siegel, S.M. & Kaemmerer, W.F. (1978). Measuring the perceived support for innovation in organizations. Journal of Applied Psychology, 63: 553-562.

48.

Simpson, R.D. , Rentz, R.R. & Shrum J.W. (1976). Influence of instrument characteristics on student responses in attitude assessment. Journal of Research in Science Teaching, 13: 275-281.

49.

Smith, P.C. , Kendall, L.M. & Hulin, C.L. (1969). The measurement of satisfaction in work and retirement. Chicago: Rand-McNally.

50.

Stogdill, R.M. (1963). Manual for the leader behavior description questionnaire-Form XII. Columbus: Bureau of Business Research, Ohio State University.

51.

Tanaka, J.S. (1993). Multifaceted conceptions of lit in structural equation models. Pp. 10-39 in K.A. Bollen & J.S. Long (Eds.), Testing structural equation models. Newbury Park, CA: Sage.

52.

Tracy, L. & Johnson, T.W. (1981). What do the role conflict and role ambiguity scales measure? Journal of Applied Psychology, 55: 464-489.

53.

Widaman, K.F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9: 1-26.

54.

Winkler, J.D. , Kanouse, D.E. & Ware, J.E. Jr. (1981). Controlling for acquiescence response set in scale development. Paper presented at the 90th annual meeting of the American Psychological Association, Los Angeles, CA, August.