Measuring values in sociological research sometimes involves the use of ranking data. A disadvantage of a ranking assignment is that the order in which the items are presented might influence the choice preferences of respondents regardless of the content being measured. The standard procedure to rule out such effects is to randomize the order of items across respondents. However, implementing this design may be impractical and the biasing impact of a response order effect cannot be evaluated. We use a latent choice factor (LCF) model that allows statistically controlling for response order effects. Furthermore, the model adequately deals with the known issue of ipsativity of ranking data. Applying this model to a Dutch survey on work values, we show that a primacy effect accounts for response order bias in item preferences. Our findings demonstrate the usefulness of the LCF model in modeling ranking data while taking into account particular response biases.
AllisonPaul D.ChristakisNicholas A.. 1994. “Logit Models for Sets of Ranked Items.” Sociological Methodology24:199–228.
2.
AlwinDuane F.KrosnickJon A.. 1985. “The Measurement of Values in Surveys: A Comparison of Ratings and Rankings.” Public Opinion Quarterly49:535–52.
3.
BeckerSam L.1954. “Why an Order Effect.” Public Opinion Quarterly18:271–78.
4.
CampbellDonald T.MohrPhillip J.. 1950. “The Effect of Ordinal Position upon Responses to Items in a Check List.” Journal of Applied Psychology34:62–67.
5.
ChanWaiBentlerPeter M.. 1993. “The Covariance Structure Analysis of Ipsative Data.” Sociological Methods & Research22:214–47.
6.
CheungMike W. L.2004. “A Direct Estimation Method on Analyzing Ipsative Data with Chan and Bentler’s (1993) Method.” Structural Equation Modeling: A Multidisciplinary Journal11:217–43.
7.
CroonMarcel A.1989. “Latent Class Models for the Analysis of Rankings.” Pp. 99–121 in New Developments in Psychological Choice Modeling, edited by de SoeteG.FegerH.KlauerK. C.. Amsterdam, the Netherlands: Elsevier Science.
8.
DeCarloLawrence T.LutharSuniya S.. 2000. “Analysis and Class Validation of a Measure of Parental Values Perceived by Early Adolescents: An Application of a Latent Class Model for Rankings.” Educational and Psychological Measurement60:578–91.
9.
DillmanDon A.ChristianLeah Melani. 2005. “Survey Mode as a Source of Instability in Responses across Surveys.” Field Methods17:30–52.
10.
FuchsMarek. 2005. “Children and Adolescents as Respondents: Experiments on Question Order, Response Order, Scale Effects and the Effect of Numeric Values Associated with Response Options.” Journal of Official Statistics21:701–25.
11.
GalesicMirtaTourangeauRogerCouperMick P.ConradFrederick G.. 2008. “Eye-tracking Data: New Insights on Response Order Effects and Other Cognitive Shortcuts in Survey Responding.” Public Opinion Quarterly72:892–913.
12.
InglehartRonald. 1977. The Silent Revolution: Changing Values and Political Styles among Western Publics. Princeton, NJ: Princeton University Press.
13.
JacksonDavid J.AlwinDuane F.. 1980. “The Factor Analysis of Ipsative Measures.” Sociological Methods & Research9:218–38.
14.
KleinMarkusDulmerHermannOhrDieterQuandtMarkusRosarUlrich. 2004. “Response Sets in the Measurement of Values: A Comparison of Rating and Ranking Procedures.” International Journal of Public Opinion Research16:474–83.
15.
KnoopRobert. 1994. “Work Values and Job Satisfaction.” Journal of Psychology128:683–90.
16.
KohnMelvin L.1969. Class and Conformity: A Study in Values. Homewood, IL: The Dorsey Press.
17.
KrosnickJon A.1992. “The Impact of Cognitive Sophistication and Attitude Importance on Response-order and Question-order Effects.” Pp. 203–18 in Context Effects in Social and Psychological Research, edited by SchwarzNorbertSudmanSeymour. New York: Springer-Verlag.
18.
KrosnickJon A.2000. “The Threat of Satisficing in Surveys: The Shortcuts Respondents Take in Answering Questions.” Survey Methods Newsletter20:4–8.
19.
KrosnickJon A.AlwinDuane F.. 1987. “An Evaluation of a Cognitive Theory of Response-order Effects in Survey Measurement.” Public Opinion Quarterly51:201–19.
20.
KrosnickJon A.PresserStanley. 2010. “Question and Questionnaire Design.” Pp. 263–313 in Handbook of Survey Research, edited by MarsdenP. V.WrightJ. D.. Bingley, UK: Emerald Group Publishing Limited.
21.
KrosnickJon A.SchumanHoward. 1988. “Attitude Intensity, Importance, and Certainty and Susceptibility to Response Effects.” Journal of Personality and Social Psychology54:940–52.
22.
MagidsonJayEagleTom C.VermuntJeroen K.. 2003. “New Developments in Latent Class Choice Models.” Pp. 89–112 in Proceedings of the Sawtooth Software Conference, San Antonio, TX, April 15-17, 2003 (Vol. 10). Sequim, WA: Unknown Publisher.
23.
MaioGregory R.RoeseNeal J.SeligmanCliveKatzAlbert. 1996. “Rankings, Ratings, and the Measurement of Values: Evidence for the Superior Validity of Ratings.” Basic and Applied Social Psychology18:171–81.
24.
McCartyJohn A.ShrumL. J.. 2000. “The Measurement of Personal Values in Survey Research: A Test of Alternative Rating Procedures.” Public Opinion Quarterly64:271–98.
25.
McClendonMcKee J.1986. “Response-order Effects for Dichotomous Questions.” Social Science Quarterly67:205–11.
26.
McClendonMcKee J.1991. “Acquiescence and Recency Response-order Effects in Interview Surveys.” Sociological Methods & Research20:60–103.
27.
McFaddenDaniel. 1986. “The Choice Theory Approach to Marketing Research.” Marketing Science5:275–97.
28.
McFaddenDanielTrainKenneth. 2000. “Mixed MNL Models for Discrete Response.” Journal of Applied Econometrics15:447–70.
29.
MoorsGuyVermuntJeroen. 2007. “Heterogeneity in Post-materialist Value Priorities. Evidence from a Latent Class Discrete Choice Approach.” European Sociological Review23:631–48.
30.
OvadiaSeth. 2004. “Ratings and Rankings: Reconsidering the Structure of Values and their Measurement.” International Journal of Social Research Methodology7:403–14.
31.
RafteryAdrian E.1995. “Bayesian Model Selection in Social Research.” Sociological Methodology25:111–63.
32.
RokeachMilton. 1973. The Nature of Human Values. New York: The Free Press.
33.
RosMariaSchwartzShalom H.SurkissShoshana. 1999. “Basic Individual Values, Work Values, and the Meaning of Work.” Applied Psychology: An International Review48:49–71.
34.
SacchiStefan. 1998. “The Dimensionality of Postmaterialism: An Application of Factor Analysis to Ranked Preference Data.” European Sociological Review14:151–75.
35.
SchumanHowardPresserStanley. 1996. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. Thousand Oaks, CA: Sage.
36.
SchwartzShalom H.BilskyWolfgang. 1987. “Toward a Universal Psychological Structure of Human Values.” Journal of Personality and Social Psychology53:550–62.
37.
SiminskiPeter. 2008. “Order Effects in Batteries of Questions.” Quality & Quantity42:477–90.
38.
SternMichael J.DillmanDon A.SmythJolene D.. 2007. “Visual Design, Order Effects, and Respondent Characteristics in a Self-administered Survey.” Survey Research Methods1:121–38.
39.
VermuntJeroen K.MagidsonJay. 2005a. “Factor Analysis with Categorical Indicators: A Comparison between Traditional and Latent Class Approaches.” Pp. 41–62 in New Developments in Categorical Data Analysis for the Social and Behavioral Sciences, edited by van der ArkL. A.CroonMarcel A.SijtsmaKlaas. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
