Although agree–disagree (AD) rating scales suffer from acquiescence response bias, entail enhanced cognitive burden, and yield data of lower quality, these scales remain popular with researchers due to practical considerations (e.g., ease of item preparation, speed of administration, and reduced administration costs). This article shows that if researchers want to use AD scales, they should offer 5 answer categories rather than 7 or 11, because the latter yield data of lower quality. This is shown using data from four multitrait-multimethod experiments implemented in the third round of the European Social Survey. The quality of items with different rating scale lengths were computed and compared.
AlthauserRobert P.HeberleinThomas A.ScottRobert A.. 1971“A Causal Assessment of Validity: The Augmented Multitrait-Multimethod Matrix.” Pp. 374–99 in Causal Models in the Social Sciences, edited by Blalock H. M.Jr. Chicago, IL: Aldine.
2.
AlwinDuane F. 1974. “Approaches to the Interpretation of Relationships in the Mutlitrait-Multimethod Matrix.”Pp. 79–105 in Sociological Methodology 1973-74, edited by CostnerH. L.. San Francisco, CA: Jossey-Bass.
3.
AlwinDuane F.1992. “Information Transmission in the Survey Interview: Number of Response Categories and the Reliability of Attitude Measurement.” Pp. 83–118 in Sociological Methodology, Vol. 22, edited by MarsdenPeter V.. Washington, DC: American Sociological Association.
4.
AlwinDuane F.1997. “Feeling Thermometers versus 7-point Scales: Which Are Better?”Sociological Methods and Research25:318.
5.
AlwinDuane F. 2007. Margins of Errors: A Study of Reliability in Survey Measurement. Wiley-Interscience. Hoboken, New Jersey: Wiley and Sons, Inc.
6.
AlwinDuane. F.2011“Evaluating the Reliability and Validity of Survey Interview Data Using the MTMM Approach.” Pp. 265–95 in Question Evaluation Methods, edited by MadansJenniferMillerKristenMaitlandAaronWillisGordon. John Wiley. Hoboken, New Jersey: Wiley and Sons, Inc.
7.
AlwinDuane. F.KrosnickJon A.. 1991. “The Reliability of Survey Attitude Measurement.”Sociological Methods and Research20:139–81.
8.
AndrewsFrank. 1984. “Construct Validity and Error Components of Survey Measures: A Structural Modeling Approach.” Public Opinion Quarterly 46:409–42 Reprinted inSarisW. E.van MeursA.. 1990. Evaluation of Measurement Instruments by Metaanalysis of Multitrait Multimethod Studies. Amsterdam, the Netherland: North-Holland.
9.
BillietJaak B.DavidovEldad. 2008. “Testing the Stability of an Acquiescence Style Factor Behind Two Interrelated Substantive Variables in a Panel Design.”Sociological Methods and Research36:542–62.
10.
BrowneMichael W.1984. “The Decomposition of Multitraitmultimethod Matrices.”British Journal of Mathematical and Statistical Psychology37:1–21.
11.
CampbellDonald T.FiskeDonald W.. 1959. “Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix.”Psychological Bulletin6:81–105.
12.
CarpenterPatricia A.JustMarcel A.. 1975. “Sentence Comprehension: A Psycholinguistic Processing Model of Verification.”Psychological Review82:45–73.
13.
ClarkHerbert H.ClarkEve V.. 1977. Psychology and Language. New York: Harcourt Brace.
14.
CoendersGermàSarisWillem E.. 1995. “Categorization and Measurement Quality. The Choice between Pearson and Polychoric Correlations.” Pp. 125–144 in The MTMM Approach to Evaluate Measurement Instruments, Chapter 7, edited by SarisW. E.. Budapest: Eötvös University Press.
15.
CortenIrmgard W.SarisWillem E.CoendersGermà M.VeldWilliam M. van derAalbertsChris E.KornelisCharles. 2002. “Fit of Different Models for Multitrait-Multimethod Experiments.”Structural Equation Modeling9:213–32.
16.
CudeckRobert. 1988. “Multiplicative Models and MTMM Matrices.”Journal of Educational Statistics13:131–47.
17.
DawesJohn. 2008. “Do Data Characteristics Change According to the Number of Points Used? An Experiment Using 5-point, 7-point and 10-point Scales.”International Journal of Market Research50:61–77.
18.
FowlerFloyd J.1995. “Improving Survey Questions: Design and Evaluation.”Applied Social Research Methods Series38:56–57.
19.
GarnerWendell R.1960. “Rating Scales, Discriminability, and Information Transmission.”Psychological Review67:343–52.
20.
GoldbergLewis R.1990. “An Alternative ‘Description of Personality’: The Big-Five Factor Structure.”Journal of Personality and Social Psychology59:1216–29.
21.
JöreskogKarl G.. 1970. “A General Method for the Analysis of Covariance Structures.”Biometrika57:239–51.
22.
JöreskogKarl G. 1971. “Statistical Analysis of Sets of Congeneric Tests.”Psychometrika36:109–33.
23.
JöreskogKarl G.SörbomDag. 1991. LISREL VII: A Guide to the Program and Applications. Chicago: SPSS.
24.
KöltringerRichard. 1993. Messqualität in der sozialwissenschaftlichen Umfrageforschung. Endbericht Project P8690-SOZ des Fonds zur Förderung der wissenschaftlichen Forschung (FWF), Wien, Austria.
25.
KrosnickJon A.1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.”Applied Cognitive Psychology5:213–36.
26.
KrosnickJon A.BerentMatthew K.. 1993. “Comparisons of Party Identification and Policy Preferences: The Impact of Survey Question Format.”American Journal of Political Science37:941–64.
27.
LenskiGerhard E.LeggettJohn C.. 1960. “Caste, Class, and Deference in the Research Interview.”American Journal of Sociology65:463–67.
28.
LikertRensis. 1932. “A Technique for the Measurement of Attitudes.”Archives of Psychology140:1–55.
29.
MarshHerbert W.1989. “Confirmatory Factor Analyses of Multitrait-Multimethod Data: Many Problems and a Few Solutions.”Applied Psychological Measurement13:335–61.
30.
OberskiDanielSarisWillem E.HagenaarsJacques. 2007. “Why Are There Differences in the Quality of Questions across Countries?” Pp. 281–299 in Measuring Meaningful Data in Social Research, edited by LoosveldtGeertSwyngedouwMarcCambreBart. Leuven, Belgium: Acco.
31.
QuirogaAna M.1992. Studies of the Polychoric Correlation and Other Correlation Measures for Ordinal Variables. PhD thesis, Uppsala, Sweden.
32.
SarisWillem E.AalbertsChris. 2003. “Different Explanations for Correlated Disturbance Terms in MTMM Studies.”Structural Equation Modeling: A Multidisciplinary Journal10:193–213.
33.
SarisWillem E.AndrewsFrank M.. 1991. “Evaluation of Measurement Instruments Using a Structural Modeling Approach.” Pp. 575–97 in Measurement Errors in Surveys, edited by BiemerPaul P.GrovesRobert M.LybergLarsMathiowetzNancySudmanSeymour. New York: John Wiley.
34.
SarisWillem E.GallhoferIrmtraud. 2007. Design, Evaluation, and Analysis of Questionnaires for Survey Research. New York: John Wiley.
35.
SarisWillem E.RevillaMelanieKrosnickJon A.ShaefferEric M.. 2010. “Comparing Questions with Agree/Disagree Response Options to Questions with Construct-specific Response Options.”Survey Research Methods4:61–79.
36.
SarisWillem E.SatorraAlbertCoendersGerma. 2004. “A New Approach to Evaluating the Quality of Measurement Instruments: The Split-ballot MTMM Design.”Sociological Methodology.
37.
SarisWillem E.SatorraAlbertVan der VeldWilliam M.. 2009. “Testing Structural Equation Models or Detection of Misspecifications?”Structural Equation Modeling: A Multidisciplinary Journal34:311–347.
38.
SarisWillem E.WijkTheresia vanScherpenzeelAnnette C.. 1998. “Validity and Reliability of Subjective Social Indicators: The Effect of Different Measures of Association.”Social Indicators Research45:173–99.
39.
ScherpenzeelAnnette C.1995. A Question of Quality: Evaluating Survey Questions by Multitrait-Multimethod Studies. Amsterdam, the Netherlands: Nimmo.
40.
ScherpenzeelAnnette C.SarisWillem E.. 1997. “The Validity and Reliability of Survey Questions. A Meta-analysis of MTMM Studies.”Sociological Methods & Research25:341–83.
41.
TourangeauRogerRipsLance J.RasinskiKenneth. 2000. The Psychology of Survey Response. Cambridge, England: Cambridge University Press.
42.
TrabassoTomRollinsHowardShaughnesseyEdward. 1971. “Storage and Verification Stages in Processing Concepts.”Cognitive Psychology2:239–89.
43.
Van der VeldWilliam M.SarisWillem E.SatorraAlbert. 2008. Judgment Aid Rule Software. Jrule 2.0: User manual (Unpublished Manuscript, Internal Report). Radboud University Nijmegen, the Netherlands.
44.
Van MeursLex SarisWillem E.. 1990. “Memory Effects in MTMM Studies.” Pp. 134–146 in Evaluation of Measurement Instruments by Meta-analysis of Multitrait-Multimethod Studies, edited by WillemE. SarisMeursLex van. Amsterdam, the Netherlands: North Holland.
WinshipChristopherMareRobert D.. 1984. “Regressions Models with Ordinal Variables.”American Sociological Review49:512–25.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
