Organizational research increasingly relies on online review data to gauge perceived valuation and reputation of organizations and products. Online review platforms typically collect ordinal ratings (e.g., 1 to 5 stars); however, researchers often treat them as a cardinal data, calculating aggregate statistics such as the average, the median, or the variance of ratings. In calculating these statistics, ratings are implicitly assumed to be equidistant. We test whether star ratings are equidistant using reviews from two large-scale online review platforms: Amazon.com and Yelp.com. We develop a deep learning framework to analyze the text of the reviews in order to assess their overall valuation. We find that 4 and 5-star ratings, as well as 1 and 2-star ratings, are closer to each other than 3-star ratings are to 2 and 4-star ratings. An additional online experiment corroborates this pattern. Using simulations, we show that the distortion by non-equidistant ratings is especially harmful in cases when organizations receive only a few reviews and when researchers are interested in estimating variance effects. We discuss potential solutions to solve the issue with rating non-equidistance.
AmbleeN.BuiT. (2011). Harnessing the influence of social proof in online shopping: The effect of electronic word of mouth on sales of digital microproducts. International Journal of Electronic Commerce, 16(2), 91–114. https://doi.org/10.2753/JEC1086-4415160205
2.
AndersonN. H. (1982). Methods of information integration theory (Vol. 2). Academic Press.
3.
ArchakN.GhoseA.IpeirotisP. G. (2011). Deriving the pricing power of product features by mining consumer reviews. Management Science, 57(8), 1485–1509. https://doi.org/10.1287/mnsc.1110.1370
4.
Babić RosarioA.SotgiuF.De ValckK.BijmoltT. H. (2016). The effect of electronic word of mouth on sales: A meta-analytic review of platform, product, and metric factors. Journal of Marketing Research, 53(3), 297–318. https://doi.org/10.1509/jmr.14.0380
5.
BaoT.ChangT.-l. S. (2014). Finding disseminators via electronic word of mouth message for effective marketing communications. Decision Support Systems, 67, 21–29. https://doi.org/10.1016/j.dss.2014.07.006
6.
BassB. M.CascioW. F.O'connorE. J. (1974). Magnitude estimations of expressions of frequency and amount. Journal of Applied Psychology, 59(3), 313–320. https://doi.org/10.1037/h0036653
7.
BergerJ. (2014). Word of mouth and interpersonal communication: A review and directions for future research. Journal of Consumer Psychology, 24(4), 586–607. https://doi.org/10.1016/j.jcps.2014.05.002
8.
BirnbaumM. H. (1974). Using contextual effects to derive psychophysical scales. Perception & Psychophysics, 15(1), 89–96. https://doi.org/10.3758/BF03205834
9.
BirnbaumM. H. (1980). Comparison of two theories of “ratio” and “difference” judgments. Journal of Experimental Psychology: General, 109(3), 304–319. https://doi.org/10.1037/0096-3445.109.3.304
10.
BloemJ. R. (2021). How much does the cardinal treatment of ordinal variables matter? An empirical investigation. Political Analysis, 30(2), 1–17. https://doi.org/10.1017/pan.2020.55
11.
BondT. N.LangK. (2019). The sad truth about happiness scales. Journal of Political Economy, 127(4), 1629–1640. https://doi.org/10.1086/701679
12.
BudescuD. V.ChenE. (2015). Identifying expertise to extract the wisdom of crowds. Management Science, 61(2), 267–280.
13.
BusemeyerJ. R.JonesL. E. (1983). Analysis of multiplicative combination rules when the causal variables are measured with error. Psychological Bulletin, 93(3), 549. https://doi.org/10.1037/0033-2909.93.3.549
14.
CasperW.EdwardsB. D.WallaceJ. C.LandisR. S.FifeD. A. (2020). Selecting response anchors with equal intervals for summated rating scales. Journal of Applied Psychology, 105(4), 390–409. https://doi.org/10.1037/apl0000444
15.
ChakrabortyI.KimM.SudhirK. (2022). Attribute sentiment scoring with online text reviews: Accounting for language structure and attribute self-selection. Journal of Marketing Research, 59(3), 600–622. https://doi.org/10.1177/00222437211052500
16.
ChevalierJ. A.MayzlinD. (2006). The effect of word of mouth on sales: Online book reviews. Journal of Marketing Research, 43(3), 345–354. https://doi.org/10.1509/jmkr.43.3.345
17.
CholletF. (2018). Deep learning with Python. Manning.
18.
ClemonsE. K.GaoG. G.HittL. M. (2006). When online reviews meet hyperdifferentiation: A study of the craft beer industry. Journal of Management Information Systems, 23(2), 149–171. https://doi.org/10.2753/MIS0742-1222230207
DaiW. D.JinG.LeeJ.LucaM. (2018). Aggregation of consumer ratings: An application to Yelp. com. Quantitative Marketing and Economics, 16(3), 289–339. https://doi.org/10.1007/s11129-017-9194-9
21.
DeSimoneJ. A.HarmsP. (2018). Dirty data: The effects of screening respondents who provide low-quality data in survey research. Journal of Business and Psychology, 33, 559–577. https://doi.org/10.1007/s10869-017-9514-9
22.
DevlinJ.ChangM.-W.LeeK.Tb outanovaK. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
23.
EdwardsA. L.KenneyK. C. (1946). A comparison of the Thurstone and Likert techniques of attitude scale construction. Journal of Applied Psychology, 30(1), 72–83.
24.
FisherM.NewmanG. E.DharR. (2018). Seeing stars: How the binary bias distorts the interpretation of customer ratings. Journal of Consumer Research, 45(3), 471–489. https://doi.org/10.1093/jcr/ucy017
25.
FrakeJ. (2016). Selling out: The inauthenticity discount in the craft beer industry. Management Science, 63(11), 3930–3943. https://doi.org/10.1287/mnsc.2016.2517
26.
FrijtersP.Haisken-DeNewJ. P.ShieldsM. A. (2004). Money does matter! Evidence from increasing real income and life satisfaction in East Germany following reunification. American Economic Review, 94(3), 730–740. https://doi.org/10.1257/0002828041464551
27.
GaltonF. (1907). One vote, one value. Nature, 75, 414–414.
28.
GigerenzerG.GoldsteinD. G. (1996). Reasoning the fast and frugal way: Models of bounded rationality. Psychological Review, 103(4), 650–669. https://doi.org/10.1037/0033-295X.103.4.650
29.
GöbR.McCollinC.RamalhotoM. F. (2007). Ordinal methodology in the analysis of Likert scales. Quality & Quantity, 41(5), 601–626. https://doi.org/10.1007/s11135-007-9089-z
HartM. (1996). Improving the dissemination of SERVQUAL by using magnitude scaling. In Total quality management in action (pp. 267–270). Springer.
32.
HsuG.KovácsB.KoçakÖ (2019). Experientially diverse customers and organizational adaptation in changing demand landscapes: A study of US cannabis markets, 2014–2016. Strategic Management Journal, 40(13), 2214–2241. https://doi.org/10.1002/smj.3078
33.
HuN.KohN. S.ReddyS. K. (2014). Ratings lead you to the product, reviews help you clinch it? The mediating role of online review sentiments on product sales. Decision Support Systems, 57, 42–53. https://doi.org/10.1016/j.dss.2013.07.009
34.
KahnemanD.TverskyA. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47(2), 363–391. https://doi.org/10.2307/1914185
35.
KaiserC.VendrikM. (2020). How threatening are transformations of happiness scales to subjective wellbeing research? SSRN.
36.
KaramanH. (2021). Online review solicitations reduce extremity bias in online review distributions and increase their representativeness. Management Science, 67(7), 3985–4642. https://doi.org/10.1287/mnsc.2020.3758
37.
KhareA.LabrecqueL. I.AsareA. K. (2011). The assimilative and contrastive effects of word-of-mouth volume: An experimental examination of online consumer ratings. Journal of Retailing, 87(1), 111–126. https://doi.org/10.1016/j.jretai.2011.01.005
38.
KovácsB.CarrollG. R.LehmanD. W. (2014). Authenticity and consumer value ratings: Empirical tests from the restaurant domain. Organization Science, 25(2), 458–478. https://doi.org/10.1287/orsc.2013.0843
LeeY.NgS.ShevlinT.VenkatA. (2021). The effects of tax avoidance news on employee perceptions of managers and firms: Evidence from Glassdoor.com ratings. The Accounting Review, 96(3), 343–372. https://doi.org/10.2308/TAR-2019-0148
42.
LehmanD. W.KovácsB.CarrollG. R. (2014). Conflicting social codes and organizations: Hygiene and authenticity in consumer evaluations of restaurants. Management Science, 60(10), 2602–2617. https://doi.org/10.1287/mnsc.2014.1903
43.
Le MensG.KovácsB.AvrahamiJ.KareevY. (2018). How endogenous crowd formation undermines the wisdom- of-the-crowd. Psychological Science, 29(9), 1475–1490. https://doi.org/10.1177/0956797618775080
44.
LikertR. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 140–155.
45.
LiuX.LeeD.SrinivasanK. (2019). Large-scale cross-category analysis of consumer review content on sales conversion leveraging deep learning. Journal of Marketing Research, 56(6), 918–943. https://doi.org/10.1177/0022243719866690
LucaM. (2016). Reviews, reputation, and revenue: The case of Yelp. com. Working paper, Harvard Business School.
48.
MacCallumR. C.ZhangS.PreacherK. J.RuckerD. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7(1), 19. https://doi.org/10.1037/1082-989X.7.1.19
49.
MayzlinD.DoverY.ChevalierJ. (2014). Promotional reviews: An empirical investigation of online review manipulation. American Economic Review, 104(8), 2421–2455. https://doi.org/10.1257/aer.104.8.2421
50.
McAuleyJ.PandeyR.LeskovecJ. (2015). Inferring networks of substitutable and complementary products. Paper presented at the Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
51.
NegroG.LeungM. D. (2013). Actual” and perceptual effects of category spanning. Organization Science, 24(3), 684–696. https://doi.org/10.1287/orsc.1120.0764
52.
NunnallyJ. C.BernsteinI. H. (1994). Psychometric theory. McGraw-Hill.
53.
OswaldA. J. (2008). On the curvature of the reporting function from objective reality to subjective feelings. Economics Letters, 100(3), 369–372. https://doi.org/10.1016/j.econlet.2008.02.032
54.
PamphileV. D.RuttanR. L. (2023). The (bounded) role of stated-lived value congruence and authenticity in employee evaluations of organizations. Organization Science, 34(6), 2332–2351. https://doi.org/10.1287/orsc.2022.1578
55.
PostonR. S.SpeierC. (2005). Effective use of knowledge management systems: A process model of content ratings and credibility indicators. MIS Quarterly, 29(2), 221–244. https://doi.org/10.2307/25148678
56.
PuranamD.KadiyaliV.NarayanV. (2021). The impact of increase in minimum wages on consumer perceptions of service: A transformer model of online restaurant reviews. Marketing Science, 40(5), 985–1004. https://doi.org/10.1287/mksc.2021.1294
57.
PutnamR. D. (2000). Bowling alone: The collapse and revival of American community. Simon and Schuster.
58.
RansbothamS.LurieN. H.LiuH. (2019). Creation and consumption of Mobile word of mouth: How are mobile reviews different?Marketing Science, 38(5), 773–792. https://doi.org/10.1287/mksc.2018.1115
59.
RiveraL. A.TilcsikA. (2019). Scaling down inequality: Rating scales, gender bias, and the architecture of evaluation. American Sociological Review, 84(2), 248–274. https://doi.org/10.1177/0003122419833601
60.
RocklageM. D.RuckerD. D.NordgrenL. F. (2018). The Evaluative Lexicon 2.0: The measurement of emotionality, extremity, and valence in language. Behavior Research Methods, 50(4), 1327–1344. https://doi.org/10.3758/s13428-017-0975-6
61.
RussellC. J.BobkoP. (1992). Moderated regression analysis and Likert scales: Too coarse for comfort. Journal of Applied Psychology, 77(3), 336. https://doi.org/10.1037/0021-9010.77.3.336
62.
RussellC. J.PintoJ. K.BobkoP. (1991). Appropriate moderated regression and inappropriate research strategy: A demonstration of information loss due to scale coarseness. Applied Psychological Measurement, 15(3), 257–266. https://doi.org/10.1177/014662169101500305
SharkeyA.KovácsB.HsuG. (2023). Expert critics, rankings, and review aggregators: The changing nature of intermediation and the rise of markets with multiple intermediaries. Academy of Management Annals, 17(1), 1–36. https://doi.org/10.5465/annals.2021.0025
65.
SharkeyA.PontikesE.HsuG. (2022). The impact of mandated pay gap transparency on firms’ reputations as employers. Administrative Science Quarterly, 67(4), 1136–1179. https://doi.org/10.1177/00018392221124614
66.
SmithA.AndersonM. (2016). Online shopping and e-commerce (Vol. 2016). Pew Research Center: Pew Research Center.
ThurstoneL. L. (1927). Psychophysical analysis. The American Journal of Psychology, 38(3), 368–389. https://doi.org/10.2307/1415006
72.
TimoshenkoA.HauserJ. R. (2019). Identifying customer needs from user-generated content. Marketing Science, 38(1), 1–20. https://doi.org/10.1287/mksc.2018.1123
73.
Van PraagB. M. (1991). Ordinal and cardinal utility: An integration of the two dimensions of the welfare concept. Journal of Econometrics, 50(1–2), 69–89. https://doi.org/10.1016/0304-4076(91)90090-Z
74.
VellemanP.WilkinsonL. (1993). Nominal, ordinal, interval, and ratio typologies are misleading. The American Statistician, 47(1), 65–72. https://doi.org/10.2307/2684788
75.
VerhaalJ. C.KhessinaO. M.DobrevS. D. (2015). Oppositional product names, organizational identities, and product appeal. Organization Science, 26(5), 1466–1484. https://doi.org/10.1287/orsc.2015.1000
76.
WestP. M.BroniarczykS. M. (1998). Integrating multiple opinions: The role of aspiration level on consumer response to critic consensus. Journal of Consumer Research, 25(1), 38–51. https://doi.org/10.1086/209525
77.
YeQ.LawR.GuB. (2009). The impact of online user reviews on hotel room sales. International Journal of Hospitality Management, 28(1), 180–182. https://doi.org/10.1016/j.ijhm.2008.06.011
