The study investigated whether the change of response order in a Likert-type scale altered participant responses and scale characteristics. Response order is the order in which options of a Likert-type scale are offered. The sample included 490 college students and 368 junior high school students. Scale means with different response orders were compared. Structural equation modeling was used to test the invariance of interitem correlations, covariances, and factor structure across scale formats and educational levels. The results indicated that response order had no substantial influence on participant responses and scale characteristics. Motivating participants and avoiding ambiguous items may minimize possible effects of scale format on participant responses and scale properties.
Get full access to this article
View all access options for this article.
References
1.
Belson, W. A. (1966).The effects of reversing the presentation order of verbal rating scale. Journal of Advertising Research, 6,30-37.
2.
Bendig, A. W. (1954).Reliability and the number of rating scale. Journal of Applied Psychology, 23,323-331.
3.
Bentler, P. M. , & Wu, E.J.C. (1995). EQS for windows users’ guide. Encino, CA: Multivariate Software, Inc.
4.
Champney, H. , & Marshall, H. (1939).Optimal refinement of the rating scale. Journal of Applied Psychology, 23,323-331.
5.
Chan, C. (1986). Grade, sex role, relationship orientation, and empathy. Unpublished master’s thesis, National Chengchi University, Taipei, Taiwan.
6.
Chan, J. C. (1991).Response-order effects in Likert-type scales. Educational and Psychological Measurement, 51,531-540.
7.
Chang, L. (1994).A psychometric evaluation of 4-point and 6-point Likert-type scales in relation to reliability and validity. Applied Psychological Measurement, 18,205-215.
8.
Fan, X. , Thompson, B., & Wang, L. (1999).Effects of sample size, estimation methods, and model specification on structural equation modeling fit indexes. Structural Equation Modeling, 6,56-83.
9.
French-Lazovik, G. , & Gibson, C. L. (1984).Effects of verbally labeled anchor points on the distributional parameters of rating measures. Applied Psychological Measurement, 8,49-57.
10.
Halpin, G. , Halpin, G., & Arbet, S. (1994).Effects of number and type of response choices on internal consistency reliability. Perceptual and Motor Skills, 79,928-930.
11.
Hancock, G. R. , & Klockars, A. J. (1991).The effect of scale manipulations on validity: Targeting frequency rating scales for anticipated performance levels. Applied Ergonomics, 22,147-154.
12.
Hu, L.-T. , & Bentler, P. M. (1999).Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6,1-55.
13.
Jenkins, G. D., Jr. , & Taber, T. D. (1977).A Monte Carlo study of factors affecting three indices of composite scale reliability. Journal of Applied Psychology, 62,392-398.
14.
Johnson, J. D. (1981).Effects of the order of presentation of evaluative dimensions for bipolar scales in four societies. Journal of Social Psychology, 113,21-27.
15.
Jöreskog, K. G. , & Sörbom, D. (1996). LISREL8: User’s reference guide. Chicago: Scientific Software International, Inc.
16.
Klockars, A. J. , & Hancock, G. R. (1993).Manipulations of evaluative rating scale to increase validity. Psychological Report, 73,1059-1066.
17.
Klockars, A. J. , & Yamagishi, M. (1988).The influence of labels and positions in rating scales. Journal of Educational Measurement, 25,85-96.
18.
Komorita, S. S. , & Graham, W. K. (1965).Number of scale points and the reliability of scales. Educational and Psychological Measurement, 15,987-995.
19.
Krosnick, J. A. , & Alwin, D. F. (1987).An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51,201-219.
20.
Krosnick, J. A. , & Alwin, D. F. (1988).A test of the form-resistant correlation hypothesis: Ratings, rankings, and the measurement of values. Public Opinion Quarterly, 52,526-538.
21.
Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22, 5-55.
22.
Likert, R. , Roslow, S., & Murphy, G. (1934).A simplified and reliable method of scoring the Thurstone attitude scales. Journal of Social Psychology, 5,228-238.
23.
Likert, R. , Roslow, S., & Murphy, G. (1993).A simplified and reliable method of scoring the Thurstone attitude scales. Personnel Psychology, 46,689-690.
24.
Matell, M. S. , & Jacoby, J. (1971).Is there an optimal number of alternatives for Likert-scale items? Study I: Reliability and validity. Educational and Psychological Measurement, 31,657-674.
25.
Mathews, C. O. (1927).The effect of position of children’s answers to questions in two-response types of tests. Journal of Educational Psychology, 18,445-457.
26.
Mathews, C. O. (1929).The effect of the printed response words on an interest questionnaire. Journal of Educational Psychology, 30,128-134.
27.
McClendon, M. J. (1986).Response-order effects for dichotomous questions. Social Science Quarterly, 67,205-211.
28.
Newstead, S. E. , & Arnold, J. (1989).The effect of response format on ratings of teaching. Educational and Psychological Measurement, 49,33-43.
29.
Spector, P. E. (1976).Choosing response categories for summated rating scales. Journal of Applied Psychology, 61,374-375.
30.
Weng, L.-J. , & Cheng, C.-P. (1996).On incremental fit indexes and estimation methods in structural equation modeling. Survey Research, 2,89-109.
31.
Weng, L.-J. , & Cheng, C.-P. (1997).Why might relative fit indices differ between estimators?Structural Equation Modeling, 4, 121-128.
32.
Wildt, A. R. , & Mazis, M. B. (1978). Determinants of scale response: Label versus position. Journal of Marketing Research, 15,261-267.
33.
Wong, C.-S. , Tam, K.-C., Fung, M.-Y., & Wan, K. (1993).Differences between odd and even number of response scale: Some empirical evidence. Chinese Journal of Psychology, 35,75-86.
