Randomized paired comparisons (RPC) for social values have various advantages over a matrix format of multiple items; however, their use cannot exhaust all possible pairs if there are too many items to compare one-to-one. This article proposes (1) applying a dimension reduction method, structural topic modeling (STM), to RPC survey data by restructuring answers into ordered pairs to estimate latent answering patterns, (2) visualizing them into directed graphs, and (3) interpreting them as respondents’ preference structures among social values. For empirical validation, we randomly divided 920 respondents into RPC and matrix-format groups and asked about the seriousness of ten social problems. Our STM from the RPC group revealed five preference structures beyond a linear order among the 10 items, which are interpretable and incorporate statistical tests with respondents’ traits as covariates. We also discuss how to improve topic modeling with RPC and contribute to various research streams, such as cultural value networks and gamification, by pairwise wiki survey.
AiroldiEdoardo M.BischofJonathan M.. 2016. “Improving and Evaluating Topic Models and Other Models of Text.” Journal of the American Statistical Association111(516):1381–403.
2.
AndrewsMarkViglioccoGabriella. 2010. “The Hidden Markov Topic Model: a Probabilistic Model of Semantic Representation.” Topics in Cognitive Science2(1):101–13.
3.
AwadEdmondDsouzaSohanKimRichardSchulzJonathanHenrichJosephShariffAzimBonnefonJean-FrançoisRahwanIyad. 2018. “The Moral Machine Experiment.” Nature563(7729):59–64.
4.
BaldassarriDeliaGoldbergAmir. 2014. “Neither Ideologues nor Agnostics: alternative Voters’ Belief System in an Age of Partisan Politics.” American Journal of Sociology120(1):45–95.
5.
BleiDavidLaffertyJohn. 2006. “Correlated Topic Models.” Advances in Neural Information Processing Systems18(2006):147.
BoutylineAndreiSoterLaura K.. 2021. “Cultural Schemas: what They are, how to Find Them, and What to do Once You’ve Caught one.” American Sociological Review86(4):728–58.
8.
BoutylineAndreiVaiseyStephen. 2017. “Belief Network Analysis: a Relational Approach to Understanding the Structure of Attitudes.” American Journal of Sociology122(5):1371–447.
9.
BowerAmandaBalzanoLaura. 2020. “Preference Modeling with Context-Dependent Salient Features.” In International Conference on Machine Learning PMLR; 1067-1077.
10.
Boyd-GraberJordanMimnoDavidNewmanDavid. 2014 “Care and Feeding of Topic Models: Problems, Diagnostics, and Improvements.” Pp. 225-255 in Handbook of Mixed Membership Models and Their Applications, edited by E. M. Airoldi, D. Blei, E. A. Erosheva, S. E. Fienberg. Boca Raton, FL, USA: CRC Press.
11.
BradleyRalph AllanTerryMilton E.. 1952. “Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons.” Biometrika39(3/4):324–45.
12.
CattelanManuela.2012. “Models for Paired Comparison Data: a Review with Emphasis on Dependent Data.” Statistical Science27(3):412–33.
13.
CauseurDavidHussonFrançois. 2005. “A 2-Dimensional Extension of the Bradley–Terry Model for Paired Comparisons.” Journal of Statistical Planning and Inference135(2):245–59.
ChangJonathanGerrishSeanWangChongBoyd-GraberJordanBleiDavid. 2009. “Reading Tea Leaves: how Humans Interpret Topic Models.” Advances in Neural Information Processing Systems22:1–9.
16.
ChenShuoJoachimsThorsten. 2016. “Modeling Intransitivity in Matchup and Comparison Data.” In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, 227–36.
17.
ChenXielingZouDiChengGaryXieHaoran. 2020. “Detecting Latent Topics and Trends in Educational Technologies Over Four Decades Using Structural Topic Modeling: a Retrospective of All Volumes of Computers & Education.” Computers & Education151:103855.
18.
ConversePhilip E.2006. “The Nature of Belief Systems in Mass Publics (1964).” Critical Review18(1-3):1–74.
19.
DellaPostaDaniel.2020. “Pluralistic Collapse: the “Oil Spill” Model of Mass Opinion Polarization.” American Sociological Review85(3):507–36.
20.
DellaPostaDanielShiYongrenMacyMichael. 2015. “Why Do Liberals Drink Lattes?” American Journal of Sociology120(5):1473–511.
21.
DennyMatthew J.SpirlingArthur. 2018. “Text Preprocessing for Unsupervised Learning: Why It Matters, When It Misleads, and What to Do About It.” Political Analysis26(2):168–89.
22.
DeSimoneJustin A.HarmsPeter D.. 2018. “Dirty Data: the Effects of Screening Respondents Who Provide Low-Quality Data in Survey Research.” Journal of Business and Psychology33:559–77.
23.
DiMaggioPaul.2011 “Cultural Networks.” In The Sage Handbook of Social Network Analysis (pp. 286–300). Thousand Oaks, CA: Sage.
24.
DingWeicongIshwarPrakashSaligramaVenkatesh. 2015. “A Topic Modeling Approach to Ranking.” Artificial Intelligence and Statistics PMLR: 214–22.
25.
EvansJoel R.MathurAnil. 2005. “The Value of Online Surveys.” Internet Research15(2):195–219.
26.
FinchW. HolmesHernández FinchMaria E.McIntoshConstance E.BraunClaire. 2018. “The Use of Topic Modeling with Latent Dirichlet Analysis with Open-Ended Survey Items.” Translational Issues in Psychological Science4(4):403-24.
27.
FishmanNicDavisNicholas T.. 2022. “Change we can Believe in: structural and Content Dynamics Within Belief Networks.” American Journal of Political Science66(3):648–63.
GoldbergAmir.2011. “Mapping Shared Understandings Using Relational Class Analysis: the Case of the Cultural Omnivore Reexamined.” American Journal of Sociology116(5):1397–436.
30.
GrimmerJustin.2010. “A Bayesian Hierarchical Topic Model for Political Texts: measuring Expressed Agendas in Senate Press Releases.” Political Analysis18(1):1–35.
31.
GrimmerJustinRobertsMargaret E.StewartBrandon M.. 2022. Text as Data: A New Framework for Machine Learning and the Social Sciences. Princeton, NJ, USA: Princeton University Press.
32.
GuerreiroJoãoRitaPauloTrigueirosDuarte. 2016. “A Text Mining-Based Review of Cause-Related Marketing Literature.” Journal of Business Ethics139:111–28.
33.
HerrmannChristophRheinSebastianSträterKatharina Friederike. 2022. “Consumers’ Sustainability-Related Perception of and Willingness-to-Pay for Food Packaging Alternatives.” Resources, Conservation and Recycling181:106219.
34.
HoldenRonald R.PasseyJennifer. 2009. “Social Desirability.” Pp. 441-454 in Handbook of Individual Differences in Social Behavior edited by M. R. Leary, R. H. Hoyle. New York, NY, USA: The Guilford Press.
KanchanapibulMaturosLackaEwelinaWangXiaojunChanHing Kai. 2014. “An Empirical Investigation of Green Purchase Behaviour among the Young Generation.” Journal of Cleaner Production66:528–36.
KwonY. J.KwonK. N.. 2013. “Cultural Omnivores’ Consumption: strategic and Inclusively Exclusive.” International Journal of Marketing and Distribution5(1):118–27.
39.
LacyDean.2001a. “Nonseparable Preferences, Measurement Error, and Unstable Survey Responses.” Political Analysis9(2):95–115.
40.
LacyDean.2001b. “A Theory of Nonseparable Preferences in Survey Responses.” American Journal of Political Science45(2):239–58.
41.
LeeHoyeopKwakJueunSongMinKimChang Ouk. 2015. “Coherence Analysis of Research and Education Using Topic Modeling.” Scientometrics102:1119–37.
42.
LefèvreBriceOhlFabien. 2012. “Consuming Sports: distinction, Univorism and Omnivorism.” Sport in Society15(1):44–63.
43.
LiXimingLiChangchunChiJinjinOuyangJihong. 2018. “Short Text Topic Modeling by Exploring Original Documents.” Knowledge and Information Systems56(2):443–62.
44.
LoomisJohnBrownThomasLuceroBeatricePetersonGeorge. 1997. “Evaluating the Validity of the Dichotomous Choice Question Format in Contingent Valuation.” Environmental and Resource Economics10:109–23.
45.
LouviereJordan J.CarsonRichard T.BurgessLeonieStreetDeborahMarleyAnthony AJ. 2013. “Sequential Preference Questions Factors Influencing Completion Rates and Response Times Using an Online Panel.” Journal of Choice Modelling8:19–31.
46.
LucasChristopherNielsenRichard A.RobertsMargaret E.StewartBrandon M.StorerAlexTingleyDustin. 2015. “Computer-assisted Text Analysis for Comparative Politics.” Political Analysis23(2):254–77.
47.
MaierDanielNieklerAndreasWiedemannGregorStoltenbergDaniela. 2020. “How Document Sampling and Vocabulary Pruning Affect the Results of Topic Models.” Computational Communication Research2(2):139–52.
48.
ManghamLindsay J.HansonKaraMcPakeBarbara. 2009. “How to Do (or Not to Do)… Designing a Discrete Choice Experiment for Application in a Low-Income Country.” Health Policy and Planning24(2):151–8.
49.
MartinFionaJohnsonMark. 2015. “More Efficient Topic Modelling Through a Noun Only Approach.” In Proceedings of the Australasian Language Technology Association Workshop, pp. 111–5.
50.
McCutcheonAllan L. 1987. “Latent Class Analysis.” Pp. 7–64 in Sage University Paper Series on Quantitative Applications in the Social Sciences. Newbury Park, CA, USA: Sage Publications.
51.
MimnoDavidMcCallumAndrew. 2012. “Topic Models Conditioned on Arbitrary Features with Dirichlet-Multinomial Regression.” arXiv preprint arXiv:1206.3278.
52.
MostellerFrederick.1951. “Remarks on the Method of Paired Comparisons. I. The Least Squares Solution Assuming Equal Standard Deviations and Equal Correlations. II. The Effect of an Aberrant Standard Deviation When Equal Standard Deviations and Equal Correlations are Assumed. III. A Test of Significance for Paired Comparisons When Equal Standard Deviations and Equal Correlations are Assumed.” Psychometrika16(3-9):203–18.
53.
NaultJean-FrançoisBaumannShyonChildressClaytonRawlingsCraig M.. 2021. “The Social Positions of Taste Between and Within Music Genres: from Omnivore to Snob.” European Journal of Cultural Studies24(3):717–40.
NelsonLaura K.BurkDerekKnudsenMarcelMcCallLeslie. 2021. “The Future of Coding: a Comparison of Hand-Coding and Three Types of Computer-Assisted Text Analysis Methods.” Sociological Methods & Research50(1):202–37.
56.
PetersonRichard A.KernRoger M.. 1996. “Changing Highbrow Taste: from Snob to Omnivore.” American Sociological Review61(5):900–7.
57.
PhillipsDerek L.ClancyKevin J.. 1972. “Some Effects of “Social Desirability” in Survey Studies.” American Journal of Sociology77(5):921–40.
58.
RamageDanielRosenEvanChuangJasonManningChristopher D.McFarlandDaniel A.. 2009. “Topic Modeling for the Social Sciences.” In NIPS 2009 Workshop on Applications for Topic Models: Text and Beyond5(27):1–4.
59.
RobertsMargaret E.StewartBrandon M.AiroldiEdoardo M.. 2016. “A Model of Text for Experimentation in the Social Sciences.” Journal of the American Statistical Association111(515):988–1003.
60.
RobertsMargaret E.StewartBrandon M.TingleyDustin. 2019. “STM: An R Package for Structural Topic Models.” Journal of Statistical Software91(2):1–40.
61.
RobertsMargaret E.StewartBrandon M.TingleyDustinLucasChristopherLeder-LuisJetsonGadarianShana KushnerAlbertsonBethanyRandDavid G.. 2014. “Structural Topic Models for Open-Ended Survey Responses.” American Journal of Political Science58(4):1064–82.
62.
SalganikMatthew J. 2019. Bit by Bit: Social Research in the Digital Age. Princeton, NJ, USA: Princeton University Press.
63.
SalganikMatthew J.LevyKaren E.C.. 2015. “Wiki Surveys: open and Quantifiable Social Data Collection.” PLoS One10(5):e0123483.
64.
SalomonJoshua A.VosTheoHoganDaniel R.GagnonMichaelNaghaviMohsenMokdadAliBegumNazma, et al.2012. “Common Values in Assessing Health Outcomes from Disease and Injury: Disability Weights Measurement Study for the Global Burden of Disease Study 2010.” The Lancet380(9859):2129–43.
65.
SbalchieroStefanoEderMaciej. 2020. “Topic Modeling, Long Texts and the Best Number of Topics. Some Problems and Solutions.” Quality & Quantity54(4):1095–108.
66.
SonikRajan AnthonyCreedonTimothy B.ProgovacAna MariaCarsonNicholasDelmanJonathanDelmanDeborahLê CookBenjamin, et al.2020. “Depression Treatment Preferences by Race/Ethnicity and Gender and Associations Between Past Healthcare Discrimination Experiences and Present Preferences in a Nationally Representative Sample.” Social Science & Medicine253:112939.
67.
TangJian.MengZhaoshiNguyenXuanlongMeiQiaozhuZhangMing. 2014. “Understanding the Limiting Factors of Topic Modeling via Posterior Contraction Analysis.” International Conference on Machine Learning PMLR32(1):190–8.
68.
ThurstoneLouis L.1927. “The Method of Paired Comparisons for Social Values.” Journal of Abnormal and Social Psychology21(4):384–400.
69.
TomzMichaelVan HouwelingRobert P.. 2009. “The Electoral Implications of Candidate Ambiguity.” American Political Science Review103(1):83–98.
70.
TrninićViktorijaJelaskaIgorŠtalecJanez. 2012. “Appropriateness and Limitations of Factor Analysis Methods Utilized in Psychology and Kinesiology: part I.” Fizička Kultura66(2):77–87.
71.
UsamiSatoshi.2010. “Individual Differences Multidimensional Bradley–Terry Model Using Reversible Jump Markov Chain Monte Carlo Algorithm.” Behaviormetrika37(2):135–55.
72.
Van NoordJochemTurner-ZwinkelsFelicity M.KesbergRebekkaBrandtMark J.EasterbrookMatthew J.KuppensToonSpruytBram. 2025. “The Nature and Structure of European Belief Systems: exploring the Varieties of Belief Systems Across 23 European Countries.” European Sociological Review41(1):143–61.
73.
VeenstraGerry.2015. “Class Position and Musical Tastes: a Sing-off Between the Cultural Omnivorism and Bourdieusian Homology Frameworks.” Canadian Review of Sociology /Revue Canadienne de Sociologie52(2):134–59.
74.
VeldwijkJorienLambooijMattijs S.de Bekker-GrobEsther W.SmitHenriëtte A.Ardine de WitG.. 2014. “The Effect of Including an Opt-Out Option in Discrete Choice Experiments.” PloS One9(11):e111805.
75.
VilenchikDanYichyeBarakAbutbulMaor. 2019. “To Interpret or not to Interpret PCA? This is our Question.” In Proceedings of the International AAAI Conference on Web and Social Media13:655–8.
76.
WardenaarKlaas.2021. Latent Profile Analysis in R: A Tutorial and Comparison to Mplus. 10.31234/osf.io/wzftr.
77.
WellerBridget E.BowenNatasha K.FaubertSarah J.. 2020. “Latent Class Analysis: a Guide to Best Practice.” Journal of Black Psychology46(4):287–311.
78.
WoldSvanteEsbensenKimGeladiPaul. 1987. “Principal Component Analysis.” Chemometrics and Intelligent Laboratory Systems2(1-3):37–52.
79.
WoodDustinHarmsPeter D.LowmanGraham H.DeSimoneJustin A.. 2017. “Response Speed and Response Consistency as Mutually Validating Indicators of Data Quality in Online Samples.” Social Psychological and Personality Science8(4):454–64.
80.
ZouLuyiSongWilliam Wei. 2016. “LDA-TM: A two-Step Approach to Twitter Topic Data Clustering.” In 2016 IEEE International Conference on Cloud Computing and Big Data Analysis (ICCCBDA), pp. 342–7.
81.
ZuoYuanWuJunjieZhangHuiLinHaoWangFeiXuKeXiongHui. 2016. “Topic Modeling of Short Texts: a pseudo-Document View.” In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 2105–-14.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
