Quantitative performance ratings are ubiquitous in modern organizations—from businesses to universities—yet there is substantial evidence of bias against women in such ratings. This study examines how gender inequalities in evaluations depend on the design of the tools used to judge merit. Exploiting a quasi-natural experiment at a large North American university, we found that the number of scale points used in faculty teaching evaluations—whether instructors were rated on a scale of 6 versus a scale of 10—significantly affected the size of the gender gap in evaluations in the most male-dominated fields. A survey experiment, which presented all participants with an identical lecture transcript but randomly varied instructor gender and the number of scale points, replicated this finding and suggested that the number of scale points affects the extent to which gender stereotypes of brilliance are expressed in quantitative ratings. These results highlight how seemingly minor technical aspects of performance ratings can have a major effect on the evaluation of men and women. Our findings thus contribute to a growing body of work on organizational practices that reduce workplace inequalities and the sociological literature on how rating systems—rather than being neutral instruments—shape the distribution of rewards in organizations.
AbelMillicent H.MeltzerAndrea L. 2007. “Student Ratings of a Male and Female Professors’ Lecture on Sex Discrimination in the Workforce.” Sex Roles57(3–4):173–80.
2.
AguirreAdalbertoJr.2000. Women and Minority Faculty in the Academic Workplace. San Francisco, CA: Jossey-Bass.
3.
AiChunrongNortonEdward C. 2003. “Interaction Terms in Logit and Probit Models.” Economics Letters80(1):123–29.
4.
AlonSigal. 2009. “The Evolution of Class Inequality in Higher Education: Competition, Exclusion, and Adaptation.” American Sociological Review74(5):731–55.
5.
ArbuckleJulianneWilliamsBenne D. 2003. “Students’ Perceptions of Expressiveness: Age and Gender Effects on Teacher Evaluations.” Sex Roles49(9/10):507–16.
6.
BaldwinTamaraBlattnerNancy. 2003. “Guarding Against Potential Bias in Student Evaluations: What Every Faculty Member Needs to Know.” College Teaching51(1):27–32.
7.
BarbezatDebra A.HughesJames H. 2006. “Salary Structure Effects and the Gender Pay Gap in Academia.” Research in Higher Education46(6):621–40.
8.
BasowSusan A.1995. “Student Evaluations of College Professors: When Gender Matters.” Journal of Educational Psychology87(4):656–65.
9.
BasowSusan A.2000. “Best and Worst Professors: Gender Patterns in Students’ Choices.” Sex Roles43(5/6):407–17.
10.
BenjaminBeth A.PodolnyJoel M. 1999. ‘‘Status, Quality, and Social Order in the California Wine Industry.’’Administrative Science Quarterly44(3):563–89.
11.
BennettSheila. 1982. “Student Perceptions of and Expectations for Male and Female Instructors: Evidence Relating to the Question of Gender Bias in Teaching Evaluation.” Journal of Educational Psychology74(2):170–9.
12.
BergerJosephFişekM. HamitNormanRobert Z.ZelditchMorrisJr1977. Status Characteristics and Social Interaction: An Expectation States Approach. New York: Elsevier.
13.
BianLinLeslieSarah-JaneCimpianAndrei. 2017. “Gender Stereotypes about Intellectual Ability Emerge Early and Influence Children’s Interests.” Science355(6323):389–39.
14.
BiernatMonicaFuegenKathleen. 2001. “Shifting Standards and the Evaluation of Competence: Complexity in Gender-Based Judgment and Decision Making.” Journal of Social Issues57(4):707–24.
15.
BiernatMonicaTocciM. J.WilliamsJoan C. 2012. “The Language of Performance Evaluations: Gender-Based Shifts in Content and Consistency of Judgment.” Social Psychological and Personality Science3(2):186–92.
16.
BohnetIrisvan GeenAlexandraBazermanMax. 2015. “When Performance Trumps Gender Bias: Joint vs. Separate Evaluation.” Management Science62(5):1225–34.
17.
BoringAnne. 2017. “Gender Biases in Student Evaluations of Teaching.” Journal of Public Economics145:27–41.
18.
BoringAnneOttoboniKellieStarkPhilip B. 2016. “Student Evaluations of Teaching (Mostly) Do Not Measure Teaching Effectiveness.” ScienceOpen Research (doi: 10.14293/S2199-1006.1.SOR-EDU.AETBZC.v1).
19.
BowersAnnePratoMatteo. 2018. “The Structural Origins of Unearned Status: How Arbitrary Changes in Categories Affect Status Position and Market Impact.” Administrative Science Quarterly63(3):668–99.
20.
BrandtnerChristof. 2017. “Putting the World in Orders: Plurality in Organizational Evaluation.” Sociological Theory35(5):200–227.
21.
BrewerMarilyn. 1995. “In-Group Favoritism: The Subtle Side of Intergroup Discrimination.” Pp. 101–17 in Behavioral Research and Business Ethics, edited by MessickD.TenbrunselA. New York: Russell Sage.
22.
CastillaEmilio J.2008. “Gender, Race, and Meritocracy in Organizational Careers.” American Journal of Sociology113(6):1479–526.
23.
CorrellShelley J.BenardStephenPaikIn. 2007. “Getting a Job: Is There a Motherhood Penalty?”American Journal of Sociology112(5):1297–339.
24.
CorrellShelley J.RidgewayCecilia L. 2003. “Expectation States Theory.” Pp. 29–51 in The Handbook of Social Psychology, edited by DelamaterJ. New York: Kluwer Academic Press.
25.
DanaJasonDawesRobynPetersonNathanial. 2013. “Belief in the Unstructured Interview: The Persistence of an Illusion.” Judgement and Decision Making8(5):512–20.
26.
DobbinFrank. 2009. Inventing Equal Opportunity. Princeton, NJ: Princeton University Press.
27.
DobbinFrankSchrageDanielKalevAlexandra. 2015. “Rage against the Iron Cage: The Varied Effects of Bureaucratic Personnel Reforms on Diversity.” American Sociological Review80(5):1014–44.
28.
EaglyAlice H.KarauSteven J. 2002. “Role Congruity Theory of Prejudice toward Female Leaders.” Psychological Review109(3):573–98.
29.
EdelmanLauren B.2016. Working Law: Courts, Corporations, and Symbolic Civil Rights. Chicago: University of Chicago Press.
30.
EnglandPaula. 2010. “The Gender Revolution: Uneven and Stalled.” Gender & Society24(2):149–66.
31.
EspelandWendy N.SauderMichael. 2016. Engines of Anxiety: Academic Rankings, Reputation, and Accountability. New York: Russell Sage.
32.
EspelandWendy N.StevensMitchell L. 1998. “Commensuration as a Social Process.” Annual Review of Sociology24:313–43.
33.
EspelandWendy N.StevensMitchell L. 2008. “A Sociology of Quantification.” European Journal of Sociology49(3):401–36.
34.
FoschiMartha. 1996. “Double Standards in the Evaluation of Men and Women.” Social Psychology Quarterly59(3):237–54.
35.
FoschiMarthaLaiLarissaSigersonKirsten. 1994. “Gender and Double Standards in the Assessment of Job Applicants.” Social Psychology Quarterly54(4):326–39.
36.
FurrR. MichaelBacharachVerne R. 2014. Psychometrics: An Introduction. Thousand Oaks, CA: Sage.
37.
GouldRoger. 2003. Collusion of Wills: How Ambiguity About Social Rank Breeds Conflict. Chicago: University of Chicago Press.
38.
HeilmanMadeline E.1995. “Sex Stereotypes and Their Effects in the Workplace: What We Know and What We Don’t Know.” Journal of Social Behavior and Personality10(6):3–26.
39.
HeilmanMadeline E.2001. “Description and Prescription: How Gender Stereotypes Prevent Women’s Ascent up the Organizational Ladder.” Journal of Social Issues57(4):657–74.
40.
HuiC. HarryTriandisHarry C. 1989. “Effects of Culture and Response Format on Extreme Response Style.” Journal of Cross-Cultural Psychology20(3):296–309.
41.
JacobsJerry A.WinslowSarah E. 2004. “Overworked Faculty: Jobs, Stresses, and Family Divides.” Annals of the American Academy of Political and Social Science596(1):104–29.
42.
JencksChristopherPhillipsMeredith, eds. 1998. The Black-White Test Score Gap. Washington, DC: Brookings Institution Press.
43.
KalevAlexandraDobbinFrankKellyErin. 2006. “Best Practices or Best Guesses? Assessing the Efficacy of Corporate Affirmative Action and Diversity Policies.” American Sociological Review71(4):589–617.
44.
KarabelJerome. 2005. The Chosen: The Hidden History of Admission and Exclusion at Harvard, Yale, and Princeton. Princeton, NJ: Princeton University Press.
45.
KiersteadDianeD’AgostinoPattiDillHeidi. 1988. “Sex Role Stereotyping of College Professors: Bias in Students’ Ratings of Instructors.” Journal of Educational Psychology80(3):342–4.
46.
KlinePaul. 2013. The New Psychometrics: Science, Psychology and Measurement. New York: Routledge.
47.
Knobloch-WesterwickSilviaGlynnCarroll J. 2013. “The Matilda Effect: Role Congruity Effects on Scholarly Communication.” Communication Research40(1):3–26.
48.
Knobloch-WesterwickSilviaGlynnCarroll J.HugeMichael. 2013. “The Matilda Effect in Science Communication: An Experiment on Gender Bias in Publication Quality Perceptions and Collaboration Interest.” Science Communication35(5):603–25.
49.
LamontMichèle. 2012. “Toward a Comparative Sociology of Valuation and Evaluation.” Annual Review of Sociology38:201–21.
50.
LamontMichèleMolnárVirag. 2002. “The Study of Boundaries in the Social Sciences.” Annual Review of Sociology28:167–95.
51.
LeslieSarah-JaneCimpianAndreiMeyerMeredithFreelandEdward F. 2015. “Expectations of Brilliance Underlie Gender Distributions across Academic Disciplines.” Science16(January):262–5.
52.
LynessKaren S.HeilmanMadeline E. 2006. “When Fit Is Fundamental: Performance Evaluations and Promotions of Upper-Level Female and Male Managers.” Journal of Applied Psychology91(4):777–85.
53.
MacNellLillianDriscollAndreaHuntAdam. 2015. “What’s in a Name: Exposing Gender Bias in Student Ratings of Teaching.” Innovative Higher Education40(4):291–303.
54.
MaderaJuan M.HeblMichelle R.MartinRandi C. 2009. “Gender and Letters of Recommendation for Academia: Agentic and Communal Differences.” Journal of Applied Psychology94(6):1591–99.
55.
MalinakDanielPowersRyanWalterBarbara F. 2013. “The Gender Citation Gap in International Relations.” International Organization67(4):889–922.
56.
McPhersonMichael A.JewellR. ToddKimMyungsup. 2009. “What Determines Student Evaluation Scores? A Random Effects Analysis of Undergraduate Economics Classes.” Eastern Economic Journal35(1):37–51.
57.
MengelFriederikeSauermannJanZölitzUlf. 2017. “Gender Bias in Teaching Evaluations.” IZA Institute of Labor Economics Working Paper 11000.
58.
Moss-RacusinCorinne A.DovidioJohn F.BrescollVictoria L.GrahamMark J.HandelsmanJo. 2012. “Science Faculty’s Subtle Gender Biases Favor Male Students.” Proceedings of the National Academy of Sciences109(41):16474–79.
59.
MurphyKevin R.ClevelandJeannette. 1995. Understanding Performance Appraisal. New York: Sage.
60.
MurrayHarry G.1984. “The Impact of Formative and Summative Evaluation of Teaching in North American Universities.” Assessment and Evaluation in Higher Education9(2):117–32.
61.
NewmanLeonard S.TanMingxuanCaldwellTracy L.DuffKimberley J.WinerE. Samuel. 2018. “Name Norms: A Guide to Casting Your Next Experiment.” Personality and Social Psychology Bulletin44(10):1435–48.
PernaLaura W.2006. “Sex Differences in Faculty Tenure and Promotion: The Contribution of Family Ties.” Research in Higher Education46(3):277–30.
64.
PosseltJulie R.2016. Inside Graduate Admissions: Merit, Diversity, and Faculty Gatekeeping. Cambridge, MA: Harvard University Press.
65.
QuadlinNatasha. 2018. “The Mark of a Woman’s Record: Gender and Academic Performance in Hiring.” American Sociological Review83(2):331–60.
66.
RidgewayCecilia L.1997. “Interaction and the Conservation of Gender Inequality: Considering Employment.” American Sociological Review62(2):218–35.
67.
RidgewayCecilia L.2006. “Gender as an Organizing Force in Social Relations: Implications for the Future of Inequality.” Pp. 265–87 in The Declining Significance of Gender? edited by BlauF. D.BrintonM. C.GruskyD. B. New York: Russell Sage.
68.
RidgewayCecilia L.2011. Framed by Gender: How Gender Inequality Persists in the Modern World. New York: Oxford University Press.
69.
RidgewayCecilia L.CorrellShelley J. 2004. “Motherhood as a Status Characteristic.” Journal of Social Issues60(4):683–700.
70.
RiveraLauren A.2015. Pedigree: How Elite Students Get Elite Jobs. Princeton, NJ: Princeton University Press.
71.
RiveraLauren A.2017. “When Two Bodies Are (Not) a Problem: Gender and Relationship Status Discrimination in Academic Hiring.” American Sociological Review82(6):1111–38.
72.
RiveraLauren A.TilcsikAndrás. 2016. “Class Advantage, Commitment Penalty: The Interplay of Social Class and Gender in an Elite Labor Market.” American Sociological Review81(6):1097–131.
73.
RothLouise Marie. 2006. Selling Women Short: Gender and Money on Wall Street. Princeton, NJ: Princeton University Press.
74.
RuddElizabethMorrisonEmorySadrozinskiRenateNeradMaresiCernyJoseph. 2008. “Equality and Illusion: Gender and Tenure in Art History Careers.” Journal of Marriage and Family70(1):228–38.
75.
RudmanLaurie A.GlickPeter. 2001. “Prescriptive Gender Stereotypes and Backlash Toward Agentic Women.” Journal of Social Issues57(4):743–62.
76.
RustJohnGolombokSusan. 2009. Modern Psychometrics: The Science of Psychological Assessment. New York: Routledge.
77.
SambleJennifer N.2008. “Female Faculty: Challenges and Choices in the United States and Beyond.” New Directions for Higher Education143:55–62.
78.
SarsonsHeather. 2017. “Gender Differences in Recognition for Group Work.” Working paper, University of Toronto, Toronto, Canada.
79.
SauderMichael. 2006. “Third Parties and Status Systems: How the Structures of Status Systems Matter.” Theory & Society35(3):299–321.
80.
SauderMichaelLynnFredaPodolnyJoel M. 2012. “Status: Insights from Organizational Sociology.” Annual Review of Sociology37:267–83.
81.
SchmaderToniWhiteheadJessicaWysockiVicki H. 2007. “A Linguistic Comparison of Letters of Recommendation for Male and Female Chemistry and Biochemistry Job Applicants.” Sex Roles57(7–8):509–14.
82.
SidaniusJimCraneMarie. 1989. “Job Evaluation and Gender: The Case of University Faculty.” Journal of Applied Social Psychology19(2):174–97.
83.
SteinpreisRhea E.AndersKatie A.RitzkeDawn. 1999. “The Impact of Gender on the Review of the Curricula Vitae of Job Applicants and Tenure Candidates: A National Empirical Study.” Sex Roles 41(7–8):509–28.
84.
StevensMitchell L.2007. Creating a Class: College Admissions and the Education of Elites. Cambridge, MA: Harvard University Press.
StorageDanielHorneZacharyCimpianAndreiLeslieSarah-Jane. 2016. “The Frequency of ‘Brilliant’ and ‘Genius’ in Teaching Evaluations Predicts the Representation of Women and African Americans across Fields.” PLOS One11(3):e0150194.
TrixFrancesPsenkaCarolyn. 2003. “Exploring the Color of Glass: Letters of Recommendation for Female and Male Medical Faculty.” Discourse & Society14(2):191–220.
89.
Van der LeeRomyEllemersNaomi. 2015. “Gender Contributes to Personal Research Funding Success in The Netherlands.” Proceedings of the National Academies of Science112(40):12349–53.
90.
WagnerNataschaRiegerMatthiasVoorveltKatherine. 2016. “Gender, Ethnicity, and Teaching Evaluations: Evidence from Mixed Teaching Teams.” Economics of Education Review54:79–94.
91.
WebsterMurrayJr.FoschiMartha. 1988. Status Generalization: New Theory and Research. Stanford, CA: Stanford University Press.
92.
WenneråsChristineWoldAgnes. 1997. “Nepotism and Sexism in Peer-Review.” Nature387:341–43.
93.
WilliamsJoan C.2014. “Hacking Tech’s Diversity Problem.” Harvard Business Review92(10):94–100.
94.
WyerRobert S.CarlstonDonal E. 1979. Social Cognition, Inference, and Attribution. Hillsdale, NJ: Lawrence Erlbaum Associates.
95.
YoderJanice D.1991. “Rethinking Tokenism: Looking Beyond Numbers.” Gender and Society5(2):178–92.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
