This article examines the possible dependency of composite reliability on presentation format of the elements of a multi-item measuring instrument. Using empirical data and a recent method for interval estimation of group differences in reliability, we demonstrate that the reliability of an instrument need not be the same when polarity of the response options for its individual components differs across administrations of the instrument. Implications for empirical educational, behavioral, and social research are discussed.
AlwinD. F.KrosnickJ. A. (1991). The reliability of survey attitude measurement: The influence of question and respondent attributes. Sociological Methods & Research, 20, 139-181. doi:10.1177/0049124191020001005
2.
BornmannL.DanielH.-D. (2000). Reliabilität und Konstruktvalidität des Kurzfragebogens für Kontrollüberzeugungen (FKK). Überprüfung der Testgütekriterien im Rahmen einer Mehrthemenbefragung unter Studierenden [Reliability and construct validity of a short questionnaire of control beliefs. Examining test criteria within multiple construct survey of students]. Empirische Pädagogik, 14, 391-407.
3.
CrockerL.AlginaJ. (1986). Classical and modern test theory. Baton Rouge, FL: Harcourt Brace Jovanovich.
4.
FeldtL. (1969). A test of the hypothesis that Cronbach’s alpha or Kuder-Richardson coefficient is the same for two tests. Psychometrika, 30, 357-370.
5.
GarnerW. R. (1960). Rating scales, discriminability, and information transmission. Psychological Review, 67, 343-352. doi:10.1037/h0043047
6.
JöreskogK. G. (1971). Statistical analysis of sets of congeneric tests. Psychometrika, 36, 109-133.
7.
KrebsD. (2012). The impact of response format on attitude measurement. In SalzbornS.DavidovE.ReineckeJ. (Eds.), Methods, theories, and empirical applications in the social sciences (pp. 105-113). Wiesbaden, Germany: VS Verlag für Sozialwissenschaften.
8.
KrosnickJ. A.BerentM. K. (1993). Comparisons of party identification and policy preferences: The impact of survey question format. American Journal of Political Science, 37, 941-964.
9.
KrosnickJ. A.PresserS. (2009). Question and questionnaire design. In MarsdenP. V.WrightJ. D. (Eds.), Handbook of survey research (pp. 263-313). Bingley, England: Emerald.
10.
LeungS.-O. (2011). A comparison of psychometric properties and normality in 4-, 5-, 6-, and 11-point Likert scales. Journal of Social Service Research, 37, 412-421. doi:10.1080/01488376.2011.580697
11.
LozanoL. M.García-CuetoE.MuñizJ. (2008). Effect of the number of response categories on the reliability and validity of rating scales. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 4(2), 73-79. doi:10.1027/1614-2241.4.2.73
12.
MatellM. S.JacobyJ. (1971). Is there an optimal number of alternatives for Likert scale items? Study I: Reliability and validity. Educational and Psychological Measurement, 31, 657-674. doi:10.1177/001316447103100307
13.
MenoldN.KaczmirekL.LenznerT.NeusarA. (2014). How do respondents attend to verbal labels in rating scales?Field Methods, 26, 21-39. doi:10.1177/1525822X13508270
14.
McDonaldR. P. (1999). Test theory. A unified treatment. Mahwah, NJ: Erlbaum.
15.
MuthénB. O. (2002). Beyond SEM: General latent variable modeling. Behaviormetrika, 29, 81-117.
16.
MuthénL. K.MuthénB. O. (2014). Mplus user’s guide. Los Angeles, CA: Muthén & Muthén.
17.
O’MuircheartaighC. A.GaskellG.WrightD. B. (1995). Weighing anchors: Verbal and numeric labels for response scales. Journal of Official Statistics, 11, 295-307.
18.
ParkerR. I.VannestK. J.DavisJ. L. (2013). Reliability of multi-category rating scales. Journal of School Psychology, 51, 217-229. doi:10.1016/j.jsp.2012.12.003
19.
PrestonC. C.ColmanA. M. (2000). Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent preferences. Acta Psychologica, 104, 1-15. doi:10.1016/S0001-6918(99)00050-5
20.
RaykovT. (2012). Scale construction and development using structural equation modeling. In HoyleR. (Ed.), Handbook of structural equation modeling (pp. 472-492). New York, NY: Guilford.
21.
RaykovT.MarcoulidesG. A. (2006). A first course in structural equation modeling. Mahwah, NJ: Erlbaum.
22.
RaykovT.MarcoulidesG. A. (2011). Introduction to psychometric theory. New York, NY: Taylor & Francis.
23.
RaykovT.MarcoulidesG. A. (2012). Basic statistics. An introduction with R. New York, NY: Rowman & Littlefield.
24.
RaykovT.MarcoulidesG. A. (2015). Scale reliability evaluation with heterogeneous populations. Educational and Psychological Measurement. Advance online publication. doi:10.1177/0013164414558587
25.
RaykovT.MarcoulidesG. A.MillsapR. E. (2013). Examining factorial invariance: A multiple testing procedure. Educational and Psychological Measurement, 73, 713-727.
26.
SarisW. E.GallhoferI. N. (2007). Estimation of the effects of measurement characteristics on the quality of survey questions. Survey Research Methods, 1, 31-46.
27.
SchaefferN. C.PresserS. (2003). The science of asking questions. Annual Review of Sociology, 29, 65-88. doi:10.1146/annurev.soc.29.110702.110112
28.
SchutzH. G.RuckerM. H. (1975). A comparison of variable configurations across scale lengths: An empirical study. Educational and Psychological Measurement, 35, 319-324. doi:10.1177/001316447503500210
29.
SchwarzN.KnäuperB.HipplerH.-J.Noelle-NeumannE.ClarkL. (1991). Rating scales: Numeric values may change the meaning of scale labels. Public Opinion Quarterly, 55, 570-582.
30.
StruminskayaB.KaczmirekL.SchaurerI.BandillaW. (2014). Assessing representativeness of a probability-based online panel in Germany. In CallegaroM.BakerR.BethlehemJ.GöritzA.KrosnickJ. A.LavrakasP. J. (Eds.), Online panel research: A data quality perspective (pp. 61-84). New York, NY: Wiley.
31.
WakitaT.UeshimaN.NoguchiH. (2012). Psychological distance between categories in the Likert scale: Comparing different numbers of options. Educational and Psychological Measurement, 72, 533-546. doi:10.1177/0013164411431162
32.
WengL.-J. (2004). Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educational and Psychological Measurement, 64, 956-997. doi:10.1177/0013164404268674
