Semantic incompatibility, an error in constructing measuring instruments for rating oneself, others, or objects, refers to the extent to which item wordings are incongruent with, and hence inappropriate for, scale labels and vice versa. This study examines the effects of semantic incompatibility on rating responses. Using a 2 × 2 factorial design with semantic compatibility and scale packedness as the between-subjects factors, 160 university students were randomly assigned to four treatment conditions. Analysis of their responses to a 10-item academic ability self-assessment rating scale shows a significant difference in means between positive-packed and equal-interval conditions when item wordings and scale labels are semantically compatible. The semantically compatible conditions also show smaller variability and a slightly higher internal consistency of responses than the semantically incompatible conditions. The authors conclude that when rating scales are semantically incompatible, respondents tend to ignore the scale labels, use a greater variety of strategies to generate responses, and produce less reliable responses.
Bandalos, D.L., & Enders, C.K. (1996). The effects of nonnormality and number of response categories on reliability. Applied Measurement in Education , 9, 151-160.
2.
Bendig, A.W. (1953). The reliability of self-ratings as a function of the amount of verbal anchoring and of the number of categories on the scale . Journal of Applied Psychology, 37, 38-41.
3.
Bendig, A.W. (1954a). Reliability and the number of rating scale categories . Journal of Applied Psychology, 38, 38-40.
4.
Bendig, A.W. (1954b). Reliability of short rating scales and the heterogeneity of the rated stimuli. Journal of Applied Psychology, 38, 167-170.
5.
Bendig, A.W. (1955). Rater reliability and the heterogeneity of the scale anchors. Journal of Applied Psychology, 39, 37-39.
6.
Chang, L. (1997). Dependability of anchoring labels of Likert-type scales. Educational & Psychological Measurement, 57, 800-808.
7.
Childers, T.L., Houston, M.J., & Heckler, S.E. (1985). Measurement of individual differences in visual versus verbal information processing. Journal of Consumer Research, 12, 125-134.
8.
Cox, J. (1996). Your opinion, please!: How to build the best questionnaires in the field of education. Thousand Oaks, CA: Corwin Press.
9.
Desimone, L.M., & Floch, D.C.L. (2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis, 26, 1-22.
10.
Dixon, P.N., BoBo, M., & Stevick, R.A. (1984). Response differences and preferences for all-category-defined and end-defined Likert formats. Educational and Psychological Measurement, 44, 61-66.
11.
Dunham, T.C., & Davison, M.L. (1991). Effects of scale anchors on student ratings of instructors. Applied Measurement in Education, 4, 23-35.
12.
Feldt, L.S. (1969). A test of the hypothesis that Cronbach's alpha or Kuder-Richardson coefficient twenty is the same for two tests. Psychometrika, 34, 363-373.
13.
Fowler, F.J. (1992). How unclear terms can affect survey data. Public Opinion Quarterly, 56, 218-231.
14.
Fowler, F.J. (1995). Improving survey questions: Design and evaluation (Vol. 38). Thousand Oaks, CA : Sage.
15.
French-Lazovik, G., & Gibson, C.L. (1984). Effects of verbally labeled anchor points on the distributional parameters of rating measures. Applied Psychological Measurement, 2, 49-57.
16.
Frey, B.F., & Edwards, L.M. (2001, April). Strong words or moderate words: A comparison of the reliability and validity of responses on attitude scales. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA.
17.
Furuhata, T. (2003). The influence of labels and positions in rating scales of Japanese students. Unpublished doctoral dissertation, University of Washington.
18.
Gable, R.W., & Wolf, M.B. (1993). Instrument development in the affective domain (2nd ed.). Boston: Kluwer.
19.
Graesser, A.C., Bommareddy, S., Swamer, S., & Golding, J.M. (1996). Integrating questionnaire design with a cognitive computational model of human question answering. In N. Schwartz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 143-174). San Francisco: Jossey-Bass.
20.
Hancock, G.R., & Klockars, A.J. (1991). The effect of scale manipulations on validity: Targeting frequency rating scales for anticipated performance levels. Applied Ergonomics, 22, 147-154.
21.
Hippler, H.-J., Schwartz, N., & Sudman, S. (1988). Social information processing and survey methodology . New York: Springer-Verlag.
22.
Holleman, B. (1999). Wording effects in survey research: Using meta-analysis to explain the forbid/allow asymmetry. Journal of Quantitative Linguistics, 6, 29-40.
23.
Jenkins, G.D., Jr., & Taber, T.D. (1977). A Monte Carlo study of factors affecting three indices of composite scale reliability. Journal of Applied Psychology, 63, 392-398.
24.
Jobe, J., & Mingay, D.J. (1991). Cognition and survey measurement: History and overview. Applied Cognitive Psychology, 54, 940-952.
25.
Jobe, J.B., & Mingay, D.J. (1989). Cognitive research improved questionnaires. American Journal of Public Health, 79, 1053-1055.
26.
Klockars, A.J., & Hancock, G.R. (1993). Manipulations of evaluative rating scales to increase validity. Psychological Reports, 73, 1059-1066.
27.
Klockars, A.J., & Yamagishi, M. (1988). The influence of labels and positions in rating scales. Journal of Educational Measurement, 25, 85-96.
28.
Krosnick, J.A. (1999). Survey research. Annual Review of Psychology, 50, 537-567.
29.
Krosnick, J.A., Narayan, S., & Smith, W.R. (1996). Satisficing in surveys: Initial evidence. New Directions for Evaluation, 70, 29-44.
30.
Lam, T.C.M. (1997). Issues in rating scale construction. Unpublished manuscript. Toronto, Canada: University of Toronto.
31.
Lam, T.C.M., & Klockars, A.J. (1982). Anchor point effects on the equivalence of questionnaire items. Journal of Educational Measurement, 19, 317-322.
32.
Lam, T.C.M., & Stevens, J.J. (1994). Effects of content polarization, item wording, and rating scale width on rating response. Applied Measurement in Education, 7, 141-158.
33.
Likert, R.A. (1932). A technique for the measurement of attitudes. Archives of Psychology, 40-52.
34.
Lissitz, R.W., & Green, S.B. (1975). Effects of the number of scale points on reliability: A Monte Carlo approach. Journal of Applied Psychology, 60, 10-13.
35.
Newstead, S.E., & Arnold, J. (1989). The effect of response format on ratings of teaching . Educational and Psychological Measurement, 49, 33-43.
36.
Nunnally, J.C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.
37.
Ostrom, T.M., & Gannon, K.M. (1996). Exemplar generation. In N. Schwartz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 293-318). San Francisco: Jossey-Bass.
Rotter, G.S. (1972). Attitudinal points of agreement and disagreement . Journal of Social Psychology, 86, 211-218.
40.
Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. New York: Academic.
41.
Schwartz, N. (1999). Self-reports: How the questions shape the answers . American Psychologist, 54, 93-105.
42.
Schwarz, N., Knauper, B., Hippler, H.-J., Noelle-Neumann, E., & Clark, L. (1991). Rating scales: Numeric values may change the meaning of scale labels. Public Opinion Quarterly, 55, 570-582.
43.
Schwartz, N., & Sudman, S. (1996). Answering questions: Methodology for determining cognitive and communicative processes in survey research. San Francisco: Jossey-Bass.
44.
Sirken, M.G., Herrmann, D.J., Schechter, S., Schwarz, N., Tanur, J.M., & Tourangeau, R. (1999). Cognition and survey research . New York: John Wiley.
45.
Smith, T.W. (1987). That which we call welfare by any other name would smell sweeter: An analysis of the impact of question wording on response patterns . Public Opinion Quarterly, 51, 75-83.
46.
Spector, P.E. (1976). Choosing response categories for summated rating scales. Journal of Applied Psychology, 61, 374-375.
47.
Spector, P.E. (1980). Ratings of equal and unequal response choice intervals . Journal of Social Psychology, 112, 115-119.
48.
Spector, P.E. (1992). Summated rating scale construction: An introduction . Newbury Park, CA: Sage.
49.
Tanur, J.M. (1992). Questions about questions: Inquiries into the cognitive bases of surveys. New York: Russell Sage Foundation.
50.
Thomas, S.J. (2004). Using Web and paper questionnaires for data-based decision making: From design to interpretation of the results. Thousand Oaks, CA: Corwin Press.
51.
Tourangeau, R., & Rasinski, K. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103, 299-314.
52.
Tourangeau, R., Rips, L.J., & Rasinski, K. (2000). The psychology of survey response. Cambridge, UK: Cambridge University Press.
53.
Viswanathan, M. (1993). Measurement of individual differences in preference for numerical information. Journal of Applied Psychology, 78, 741-752.
54.
Weng, L. (2004). Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educational and Psychological Measurement, 64, 956-972.
55.
Willis, G.B. (1999). Cognitive interviewing: A ``how to'' guide. Retrieved July 29, 2005, from http://www.appliedresearch.cancer.gov/areas/cognitive/interview.pdf
56.
Willis, G.B. (2004). Cognitive interviewing revisited: A useful technique, in theory? In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, et al. (Eds.), Methods for testing and evaluating survey questionnaires (pp. 23-43). New Jersey: John Wiley.
57.
Willis, G.B. (2005). Cognitive interviews: A tool for improving questionnaire design. Thousand Oaks, CA: Sage .
58.
Wyatt, R.C., & Meyers, L.S. (1987). Psychometric properties of four 5-point Likert-type response scales. Educational and Psychological Measurement , 47, 27-35.
