Bias,Fairness,and Validity in Graduate-School Admissions: A Psychometric Perspective

Abstract

As many schools and departments are considering the removal of the Graduate Record Examination (GRE) from their graduate-school admission processes to enhance equity and diversity in higher education, controversies arise. From a psychometric perspective, we see a critical need for clarifying the meanings of measurement “bias” and “fairness” to create common ground for constructive discussions within the field of psychology, higher education, and beyond. We critically evaluate six major sources of information that are widely used to help inform graduate-school admissions decisions: grade point average, personal statements, resumes/curriculum vitae, letters of recommendation, interviews, and GRE. We review empirical research evidence available to date on the validity, bias, and fairness issues associated with each of these admission measures and identify potential issues that have been overlooked in the literature. We conclude by suggesting several directions for practical steps to improve the current admissions decisions and highlighting areas in which future research would be beneficial.

Keywords

graduate admissions validity test bias fairness discrimination higher education

Get full access to this article

View all access options for this article.

References

Aguinis

Culpepper

S. A.

Pierce

C. A.

(2010). Revival of test bias research in preemployment testing. Journal of Applied Psychology, 95(4), 648–680. https://doi.org/10.1037/a0018714

Ahmad

A. S.

Sabat

Trump-Steele

King

(2019). Evidence-based strategies for improving diversity and inclusion in undergraduate research labs. Frontiers in Psychology, 10, Article 1305. https://doi.org/10.3389/fpsyg.2019.01305

Aisch

Buchanan

Cox

Quealy

(2017, January 18). Some colleges have more students from the top 1 percent than the bottom 60. Find yours. The New York Times. https://www.nytimes.com/interactive/2017/01/18/upshot/some-colleges-have-more-students-from-the-top-1-percent-than-the-bottom-60.html

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.

Arneson

J. J.

Sackett

P. R.

Beatty

A. S.

(2011). Ability-performance relationships in education and employment settings: Critical tests of the more-is-better and the good-enough hypotheses. Psychological Science, 22(10), 1336–1342.

Babal

J. C.

Gower

A. D.

Frohna

J. G.

Moreno

M. A.

(2019). Linguistic analysis of pediatric residency personal statements: Gender differences. BMC Medical Education, 19(1), Article 392. https://doi.org/10.1186/s12909-019-1838-x

Banerjee

Reitz

J. G.

Oreopoulos

(2018). Do large employers treat racial minorities more fairly? An analysis of Canadian field experiment data. Canadian Public Policy, 44(1), 1–12. https://doi.org/10.3138/cpp.2017-033

Bangera

Brownell

S. E.

(2014). Course-based undergraduate research experiences can make scientific research more inclusive. CBE—Life Sciences Education, 13(4), 602–606. https://doi.org/10.1187/cbe.14-06-0099

Barrick

M. R.

Shaffer

J. A.

DeGrassi

S. W.

(2009). What you see may not be what you get: Relationships among self-presentation tactics and ratings of interview and job performance. Journal of Applied Psychology, 94(6), 1394–1411.

10.

Baxter

J. C.

Brock

Hill

P. C.

Rozelle

R. M.

(1981). Letters of recommendation: A question of value. Journal of Applied Psychology, 66(3), 296–301.

11.

Belasco

A. S.

Rosinger

K. O.

Hearn

J. C.

(2015). The test-optional movement at America’s selective liberal arts colleges: A boon for equity or something else? Educational Evaluation and Policy Analysis, 37(2), 206–223. https://doi.org/10.3102/0162373714537350

12.

Berry

C. M.

(2015). Differential validity and differential prediction of cognitive ability tests: Understanding test bias in the employment context. Annual Review of Organizational Psychology and Organizational Behavior, 2(1), 435–463. https://doi.org/10.1146/annurev-orgpsych-032414-111256

13.

Berry

C. M.

Clark

M. A.

McClure

T. K.

(2011). Racial/ethnic differences in the criterion-related validity of cognitive ability tests: A qualitative and quantitative review. Journal of Applied Psychology, 96(5), 881–906. https://doi.org/10.1037/a0023222

14.

Bertrand

Mullainathan

(2004). Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. American Economic Review, 94(4), 991–1013. https://doi.org/10.1257/0002828042002561

15.

Betts

J. R.

Morell

(1999). The determinants of undergraduate grade point average: The relative importance of family background, high school resources, and peer group effects. The Journal of Human Resources, 34(2), 268–293. https://doi.org/10.2307/146346

16.

Binning

J. F.

Barrett

G. V.

(1989). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74(3), 478–494. https://doi.org/10.1037/0021-9010.74.3.478

17.

Bleske-Rechek

Browne

(2014). Trends in GRE scores and graduate enrollments by gender and ethnicity. Intelligence, 46, 25–34. https://doi.org/10.1016/j.intell.2014.05.005

18.

Borsboom

Mellenbergh

G. J.

van Heerden

(2004). the concept of validity. Psychological Review, 111(4), 1061–1071. https://doi.org/10.1037/0033-295X.111.4.1061

19.

Bosco

F. A.

Aguinis

Singh

Field

J. G.

Pierce

C. A.

(2015). Correlational effect size benchmarks. Journal of Applied Psychology, 100(2), 431–449. https://doi.org/10.1037/a0038047

20.

Braun

H. I.

Jones

D. H.

(1984). Use of empirical Bayes methods in the study of the validity of academic predictors of graduate school performance. ETS Research Report Series, 1984(2): i-83. https://doi.org/10.1002/j.2330-8516.1984.tb00074.x

21.

Briggs

D. C.

(2002). SAT coaching, bias and causal inference [Unpublished doctoral dissertation]. University of California, Berkeley

22.

Brunswik

(1956). Perception and the representative design of psychological experiments (2nd ed.). University of California Press.

23.

Buckley

M. R.

Jackson

K. A.

Bolino

M. C.

Veres

J. G.

III Feild

H. S.

(2007). The influence of relational demography on panel interview ratings: A field experiment. Personnel Psychology, 60(3), 627–646.

24.

Burmeister

J. M.

Kiefner

A. E.

Carels

R. A.

Musher-Eizenman

D. R.

(2013). Weight bias in graduate school admissions. Obesity, 21(5), 918–920. https://doi.org/10.1002/oby.20171

25.

Burton

N. W.

Wang

(2005). Predicting long-term success in graduate school: A collaborative validity study. ETS Research Report Series, 2005(1), i–61. https://doi.org/10.1002/j.2333-8504.2005.tb01980.x

26.

Campion

M. A.

Palmer

D. K.

Campion

J. E.

(1997). A review of structure in the selection interview. Personnel Psychology, 50(3), 655–702.

27.

Carpi

Ronan

D. M.

Falconer

H. M.

Lents

N. H.

(2017). Cultivating minority scientists: Undergraduate research increases self-efficacy and career ambitions for underrepresented students in STEM. Journal of Research in Science Teaching, 54(2), 169–194. https://doi.org/10.1002/tea.21341

28.

Centers for Disease Control and Prevention. (2020, April 30). Communities, schools, workplaces, & events. https://www.cdc.gov/coronavirus/2019-ncov/community/health-equity/race-ethnicity.html

29.

Chapell

M. S.

Blanding

Z. B.

Silverstein

M. E.

Takahashi

Newman

Gubi

McCann

(2005). Test anxiety and academic performance in undergraduate and graduate students. Journal of Educational Psychology, 97(2), 268–274. https://doi.org/10.1037/0022-0663.97.2.268

30.

Chapman

D. S.

Zweig

D. I.

(2005). Developing a nomological network for interview structure: Antecedents and consequences of the structured selection interview. Personnel Psychology, 58(3), 673–702.

31.

Chari

Potvin

(2019). Understanding the importance of graduate admissions criteria according to prospective graduate students. Physical Review Physics Education Research, 15(2), Article 023101. https://doi.org/10.1103/PhysRevPhysEducRes.15.023101

32.

Chin

M. J.

Quinn

D. M.

Dhaliwal

T. K.

Lovison

V. S.

(2020). Bias in the air: A nationwide exploration of teachers’ implicit racial attitudes, aggregate bias, and student outcomes. Educational Researcher, 49(8), 566–578. https://doi.org/10.3102/0013189X20937240

33.

Cleary

T. A.

(1968). Test bias: Prediction of grades of Negro and White students in integrated colleges. Journal of Educational Measurement, 5(2), 115–124. https://doi.org/10.1111/j.1745-3984.1968.tb00613.x

34.

Cohn

Balch

D. C.

Bradley

(2004). Determinants of undergraduate GPAs: SAT scores, high-school GPA and high-school rank. Economics of Education Review, 23(6), 577–586. https://doi.org/10.1016/j.econedurev.2004.01.001

35.

Colella

Hebl

King

(2017). One hundred years of discrimination research in the Journal of Applied Psychology: A sobering synopsis. Journal of Applied Psychology, 102(3), 500–513. https://doi.org/10.1037/apl0000084

36.

Conway

J. M.

Jako

R. A.

Goodman

D. F.

(1995). A meta-analysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80(5), 565–579.

37.

Coronado

G. D.

Shuster

Ulrich

Anderson

Loest

(2012). Strategies for diversifying the pool of graduate students in biomedical sciences. Journal of Cancer Education, 27(3), 436–442. https://doi.org/10.1007/s13187-012-0374-8

38.

Cortina

J. M.

Goldstein

N. B.

Payne

S. C.

Davison

H. K.

Gilliland

S. W.

(2000). The incremental validity of interview scores over and above cognitive ability and conscientiousness scores. Personnel Psychology, 53(2), 325–351.

39.

Dahlke

J. A.

Sackett

P. R.

Kuncel

N. R.

(2019). Effects of range restriction and criterion contamination on differential validity of the SAT by race/ethnicity and sex. Journal of Applied Psychology, 104(6), 814–831. https://doi.org/10.1037/apl0000382

40.

Dawes

R. M.

Faust

Meehl

P. E.

(1989). Clinical versus actuarial judgment. Science, 243(4899), 1668–1674. https://doi.org/10.1126/science.2648573

41.

de Ayala

R. J.

(2009). The theory and practice of item response theory. Guilford Press.

42.

Dovidio

J. F.

Fiske

S. T.

(2012). Under the radar: How unexamined biases in decision-making processes in clinical interactions can contribute to health care disparities. American Journal of Public Health, 102(5), 945–952. https://doi.org/10.2105/AJPH.2011.300601

43.

Drasgow

(1984). Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables are the central issues. Psychological Bulletin, 95(1), 134–135. https://doi.org/10.1037/0033-2909.95.1.134

44.

Drasgow

(1987). Study of the measurement bias of two standardized psychological tests. Journal of Applied Psychology, 72(1), 19–29. https://doi.org/10.1037/0021-9010.72.1.19

45.

Educational Testing Service. (n.d.-a). About the GRE General test (for test takers). Retrieved October 6, 2021, from https://www.ets.org/gre/revised_general/about

46.

Educational Testing Service. (n.d.-b). GRE fee reduction program (for test takers). Retrieved October 6, 2021, from https://www.ets.org/gre/revised_general/about/fees/reductions

47.

Educational Testing Service. (2012). GRE General test score information by ethnicity/racial groups, 2009-2010. https://www.ets.org/s/gre/pdf/gre_general_test_score_information_by_ethnicity_2009_2010.pdf

48.

Educational Testing Service. (2019). A snapshot of the individuals who took the GRE General test July 2014-June 2019. https://www.ets.org/s/gre/pdf/snapshot_test_taker_data_2019.pdf

49.

Einhorn

H. J.

Hogarth

R. M.

(1978). Confidence in judgment: Persistence of the illusion of validity. Psychological Review, 85(5), 395–416. https://doi.org/10.1037/0033-295X.85.5.395

50.

Fischer

F. T.

Schult

Hell

(2013). Sex-specific differential prediction of college admission tests: A meta-analysis. Journal of Educational Psychology, 105(2), 478–488. https://doi.org/10.1037/a0031956

51.

Fletcher

Tienda

(2010). Race and ethnic differences in college achievement: Does high school attended matter? Annals of the American Academy of Political and Social Science, 627(1), 144–166. https://doi.org/10.1177/0002716209348749

52.

Friedman

Fang

C. H.

Hasbun

Han

Mady

L. J.

Eloy

J. A.

Kalyoussef

(2017). Use of standardized letters of recommendation for otolaryngology head and neck surgery residency and the impact of gender: Gender and letter of recommendations. The Laryngoscope, 127(12), 2738–2745. https://doi.org/10.1002/lary.26619

53.

Funder

D. C.

(1995). On the accuracy of personality judgment: A realistic approach. Psychological Review, 102(4), 652–670. https://doi.org/10.1037/0033-295X.102.4.652

54.

Gale

Ooms

Grant

Paget

Marks-Maran

(2016). Student nurse selection and predictability of academic success: The Multiple Mini Interview project. Nurse Education Today, 40, 123–127. https://doi.org/10.1016/j.nedt.2016.01.031

55.

Goho

Blackman

(2006). The effectiveness of academic admission interviews: an exploratory meta-analysis. Medical Teacher, 28(4), 335–340. https://doi.org/10.1080/01421590600603418

56.

Grimm

L. J.

Redmond

R. A.

Campbell

J. C.

Rosette

A. S.

(2020). Gender and racial bias in radiology residency letters of recommendation. Journal of the American College of Radiology, 17(1, Part A), 64–71. https://doi.org/10.1016/j.jacr.2019.08.008

57.

Grove

W. M.

Zald

D. H.

Lebow

B. S.

Snitz

B. E.

Nelson

(2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12(1), 19–30.

58.

Hall

J. D.

O’Connell

A. B.

Cook

J. G.

(2017). Predictors of student productivity in biomedical graduate school applications. PLOS ONE, 12(1), Article e0169121. https://doi.org/10.1371/journal.pone.0169121

59.

Harsh

J. A.

Maltese

A. V.

Tai

R. H.

(2012). A perspective of gender differences in chemistry and physics undergraduate research experiences. Journal of Chemical Education, 89(11), 1364–1370. https://doi.org/10.1021/ed200581m

60.

Hasher

Goldstein

Toppino

(1977). Frequency and the conference of referential validity. Journal of Verbal Learning & Verbal Behavior, 16(1), 107–112. https://doi.org/10.1016/S0022-5371(77)80012-1

61.

Henderson

M. C.

Kelly

C. J.

Griffin

Hall

T. R.

Jerant

Peterson

E. M.

Rainwater

J. A.

Sousa

F. J.

Wofsy

Franks

(2018). Medical school applicant characteristics associated with performance in multiple mini-interviews versus traditional interviews: A multi-institutional study. Academic Medicine: Journal of the Association of American Medical Colleges, 93(7), 1029–1034. https://doi.org/10.1097/ACM.0000000000002041

62.

Hersch

(2019). Catching up is hard to do: Undergraduate prestige, elite graduate programs, and the earnings premium (No. 12609). Institute of Labor Economics (IZA).

63.

Highhouse

Kostek

J. A.

(2013). Holistic assessment for selection and placement. In Geisinger

K. F.

Bracken

B. A.

Carlson

J. F.

Hansen

J.-I. C.

Kuncel

N. R.

Reise

S. P.

Rodriguez

M. C.

(Eds.), APA handbook of testing and assessment in psychology, Vol. 1: Test theory and testing and assessment in industrial and organizational psychology (pp. 565–577). American Psychological Association. https://doi.org/10.1037/14047-031

64.

Holder

W. B.

(1962). Letters of recommendation. American Psychologist, 17, 506–507.

65.

Houser

Lemmons

(2018). Implicit bias in letters of recommendation for an undergraduate research internship. Journal of Further and Higher Education, 42(5), 585–595. https://doi.org/10.1080/0309877X.2017.1301410

66.

Hoyt

W. T.

(2000). Rater bias in psychological research: When is it a problem and what can we do about it? Psychological Methods, 5(1), 64–86. https://doi.org/10.1037/1082-989X.5.1.64

67.

J. C.

(2020). Online GRE test heightens equity concerns. Science, 368(6498), 1414. https://doi.org/10.1126/science.368.6498.1414

68.

Huffcutt

A. I.

Arthur

(1994). Hunter and Hunter (1984) revisited: Interview validity for entry-level jobs. Journal of Applied Psychology, 79(2), 184–190. https://doi.org/10.1037/0021-9010.79.2.184

69.

Hughey

A. W.

(1995). Observed differences in graduate record examination scores and mean undergraduate grade point averages by gender and race among students admitted to a master’s degree program in college student affairs. Psychological Reports, 77(3, Suppl.), 1315–1321. https://doi.org/10.2466/pr0.1995.77.3f.1315

70.

International Taskforce on Assessment Center Guidelines. (2015). Guidelines and ethical considerations for assessment center operations. Journal of Management, 41(4), 1244–1273. https://doi.org/10.1177/0149206314567780

71.

Jaschik

(2008, May 22). Non-cognitive qualities join the GRE. Inside Higher Ed. https://www.insidehighered.com/news/2008/05/22/non-cognitive-qualities-join-gre

72.

Jaschik

(2019a, October 7). Brown follows Princeton in letting departments make the choice on the GRE. Inside Higher Ed. https://www.insidehighered.com/admissions/article/2019/10/07/brown-follows-princeton-letting-departments-make-choice-gre

73.

Jaschik

(2019b, March 18). A look at the many legal ways wealthy applicants have an edge in admissions. Inside Higher Ed. https://www.insidehighered.com/admissions/article/2019/03/18/look-many-legal-ways-wealthy-applicants-have-edge-admissions

74.

Jencks

Phillips

(Eds.). (1998). The black–white test score gap. Brookings Institution Press.

75.

Jones

P. E.

Roelofsma

P. H. M. P.

(2000). The potential for social contextual and group biases in team decision-making: Biases, conditions and psychological mechanisms. Ergonomics, 43(8), 1129–1152. https://doi.org/10.1080/00140130050084914

76.

Keiser

H. N.

Sackett

P. R.

Kuncel

N. R.

Brothen

(2016). Why women perform better in college than admission scores would predict: Exploring the roles of conscientiousness and course-taking patterns. Journal of Applied Psychology, 101(4), 569–581. https://doi.org/10.1037/apl0000069

77.

Khwaileh

F. M.

Zaza

H. I.

(2011). Gender differences in academic performance among undergraduates at the University of Jordan: Are they real or stereotyping? College Student Journal, 45(3), 633–648.

78.

Kim

J.-K.

(2017). Inequality in the scientific community: The effects of cumulative advantage among social scientists and humanities scholars in Korea. Higher Education, 73(1), 61–77. https://doi.org/10.1007/s10734-015-9980-9

79.

Kim

K. A.

Fann

A. J.

Misa-Escalante

K. O.

(2011). Engaging women in computer science and engineering: Promising practices for promoting gender equity in undergraduate research experiences. ACM Transactions on Computing Education, 11(2), 1–19. https://doi.org/10.1145/1993069.1993072

80.

Kim

Kyllonen

P. C.

(2006). Rasch rating scale modeling of data from the standardized letter of recommendation. ETS Research Report Series, 2006(2), i–22. https://doi.org/10.1002/j.2333-8504.2006.tb02038.x

81.

Kim

Y. K.

Sax

L. J.

(2009). Student–faculty interaction in research universities: Differences by student gender, race, social class, and first-generation status. Research in Higher Education, 50(5), 437–459. https://doi.org/10.1007/s11162-009-9127-x

82.

Kuncel

Tran

Zhang

(2020). Measuring student character: Modernizing predictors of academic success. In Oliveri

Wendler

(Eds.), Higher education admissions practices: An international perspective (pp. 276–302). Cambridge University Press. https://doi.org/10.1017/9781108559607.016

83.

Kuncel

N. R.

Hezlett

S. A.

(2007). Assessment: Standardized tests predict graduate students’ success. Science, 315(5815), 1080–1081. https://doi.org/10.1126/science.1136618

84.

Kuncel

N. R.

Hezlett

S. A.

Ones

D. S.

(2001). A comprehensive meta-analysis of the predictive validity of the Graduate Record Examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127(1), 162–181. https://doi.org/10.1037/0033-2909.127.1.162

85.

Kuncel

N. R.

Klieger

D. M.

Connelly

B. S.

Ones

D. S.

(2013). Mechanical versus clinical data combination in selection and admissions decisions: A meta-analysis. Journal of Applied Psychology, 98(6), 1060–1072. https://doi.org/10.1037/a0034156

86.

Kuncel

N. R.

Kochevar

R. J.

Ones

D. S.

(2014). A meta-analysis of letters of recommendation in college and graduate admissions: Reasons for hope: Letters of recommendation. International Journal of Selection and Assessment, 22(1), 101–107. https://doi.org/10.1111/ijsa.12060

87.

Kuncel

N. R.

Wee

Serafin

Hezlett

S. A.

(2010). The validity of the Graduate Record Examination for master’s and doctoral programs: A meta-analytic investigation. Educational and Psychological Measurement, 70(2), 340–352. https://doi.org/10.1177/0013164409344508

88.

Kutcher

E. J.

Bragger

J. D.

(2004). Selection interviews of overweight job applicants: Can structure reduce the bias? Journal of Applied Social Psychology, 34(10), 1993–2022.

89.

Kyllonen

Walters

A. M.

Kaufman

J. C.

(2005). Noncognitive constructs and their assessment in graduate education: A review. Educational Assessment, 10(3), 153–184. https://doi.org/10.1207/s15326977ea1003_2

90.

Landy

F. J.

(1986). Stamp collecting versus science: Validation as hypothesis testing. American Psychologist, 41(11), 1183–1192. https://doi.org/10.1037/0003-066X.41.11.1183

91.

Lang

(1987). Equality, prestige, and controlled mobility in the academic hierarchy. American Journal of Education, 95(3), 441–467. https://doi.org/10.1086/444314

92.

Larkin

(2018, October 24). Harvard has become more racially diverse, but most of its students are still really rich. WBUR. https://www.wbur.org/edify/2018/10/24/harvard-diverse-wealth

93.

Lei

S. A.

Chuang

N.-K.

(2009). Undergraduate research assistantship: A comparison of benefits and costs from faculty and students’ perspectives. Education, 130(2), 232–240.

94.

Lin

S. K.

Gordon

L. K.

Pineles

S. L.

Rosenberg

J. B.

Tsui

(2019). Gender-based differences in letters of recommendation written for ophthalmology residency applicants. BMC Medical Education, 19(1), Article 476. https://doi.org/10.1186/s12909-019-1910-6

95.

Ling

Buzick

Belur

(2020). The GRE and students with disabilities: A validity study at 10 universities. Exceptional Children, 86(2), 193–208. https://doi.org/10.1177/0014402919872688

96.

Liu

O. L.

Minsky

Ling

Kyllonen

(2009). Using the standardized letters of recommendation in selection: Results from a multidimensional Rasch model. Educational and Psychological Measurement, 69(3), 475–492. https://doi.org/10.1177/0013164408322031

97.

Lopatto

(2007). Undergraduate research experiences support science career decisions and active learning. CBE—Life Sciences Education, 6(4), 297–306. https://doi.org/10.1187/cbe.07-06-0039

98.

Lumb

A. B.

Homer

Miller

(2010). Equity in interviews: Do personal characteristics impact on admission interview scores? Medical Education, 44(11), 1077–1083. https://doi.org/10.1111/j.1365-2923.2010.03771.x

99.

Lunneborg

P. W.

Lillie

(1973). Sexism in graduate admissions: The letter of recommendation. American Psychologist, 28, 187–189.

100.

Macan

(2009). The employment interview: A review of current studies and directions for future research. Human Resource Management Review, 19, 203–218.

101.

Madera

J. M.

Hebl

M. R.

Dial

Martin

Valian

(2019). Raising doubt in letters of recommendation for academia: Gender differences and their impact. Journal of Business and Psychology, 34(3), 287–303. https://doi.org/10.1007/s10869-018-9541-1

102.

Madera

J. M.

Hebl

M. R.

Martin

R. C.

(2009). Gender and letters of recommendation for academia: Agentic and communal differences. Journal of Applied Psychology, 94(6), 1591–1599. https://doi.org/10.1037/a0016539

103.

Mattern

K. D.

Patterson

B. F.

(2013). Test of slope and intercept bias in college admissions: A response to Aguinis, Culpepper, and Pierce (2010). Journal of Applied Psychology, 98(1), 134–147. https://doi.org/10.1037/a0030610

104.

Melchers

K. G.

Lienhardt

Von Aarburg

Kleinmann

(2011). Is more structure really better? A comparison of frame-of-reference training and descriptively anchored rating scales to improve interviewers’ rating quality. Personnel Psychology, 64(1), 53–87.

105.

Messick

(1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741

106.

Milkman

K. L.

Akinola

Chugh

(2015). What happens before? A field experiment exploring how pay and representation differentially shape bias on the pathway into organizations. Journal of Applied Psychology, 100(6), 1678–1712. https://doi.org/10.1037/apl0000022

107.

Miller

Credé

Sotola

L. K.

(2021). Should research experience be used for selection into graduate school: A discussion and meta-analytic synthesis of the available evidence. International Journal of Selection and Assessment, 29(1), 19–28.

108.

Miller

C. W.

Zwickl

B. M.

Posselt

J. R.

Silvestrini

R. T.

Hodapp

(2019). Typical physics Ph.D. admissions criteria limit access to underrepresented groups but fail to predict doctoral completion. Science Advances, 5(1), Article eaat7550. https://doi.org/10.1126/sciadv.aat7550

109.

Miller

McCarthy

Fant

Li-Sauerwine

Ali

Kontrick

(2019). The standardized letter of evaluation narrative: Differences in language use by gender. Western Journal of Emergency Medicine, 20(6), 948–956. https://doi.org/10.5811/westjem.2019.9.44307

110.

Moneta-Koehler

Brown

A. M.

Petrie

K. A.

Evans

B. J.

Chalkley

(2017). The limitations of the GRE in predicting success in biomedical graduate school. PLOS ONE, 12(1), Article e0166742. https://doi.org/10.1371/journal.pone.0166742

111.

Morgan

W. B.

Elder

K. B.

King

E. B.

(2013). The emergence and reduction of bias in letters of recommendation: Bias in letters of recommendation. Journal of Applied Social Psychology, 43(11), 2297–2306. https://doi.org/10.1111/jasp.12179

112.

Murphy

M. J.

Nelson

D. A.

Cheap

T. L.

(1981). Rated and actual performance of high school students as a function of sex and attractiveness. Psychological Reports, 48(1), 103–106. https://doi.org/10.2466/pr0.1981.48.1.103

113.

Murphy

S. C.

Klieger

D. M.

Borneman

M. J.

Kuncel

N. R.

(2009). The predictive power of personal statements in admissions: A meta-analysis and cautionary tale. College & University, 84(4), 83–86.

114.

National Center for Education Statistics. (2017). Table 101.20. Estimates of resident population, by race/ethnicity and age group: Selected years, 1980 through 2017. https://nces.ed.gov/programs/digest/d17/tables/dt17_101.20.asp?referer=raceindicators

115.

National Council on Measurement in Education. (2019). Statement on admissions test–NCME. https://www.ncme.org/resources-publications/position-statements/college-admissions

116.

Nickerson

R. S.

(1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. https://doi.org/10.1037/1089-2680.2.2.175

117.

Nicklin

J. M.

Roch

S. G.

(2008). Biases influencing recommendation letter contents: Physical attractiveness and gender. Journal of Applied Social Psychology, 38(12), 3053–3074. https://doi.org/10.1111/j.1559-1816.2008.00425.x

118.

Niessen

A. S. M.

Meijer

R. R.

(2017). On the use of broadened admission criteria in higher education. Perspectives on Psychological Science, 12(3), 436–448. https://doi.org/10.1177/1745691616683050

119.

Nisbett

R. E.

Wilson

T. D.

(1977). The halo effect: Evidence for unconscious alteration of judgments. Journal of Personality and Social Psychology, 35(4), 250–256. https://doi.org/10.1037/0022-3514.35.4.250

120.

Norcross

J. C.

Kohout

J. L.

Wicherski

(2005). Graduate study in psychology: 1971-2004. American Psychologist, 60(9), 959–975. https://doi.org/10.1037/0003-066X.60.9.959

121.

Nunnally

J. C.

Bernstein

I. H.

(1994). Psychometric theory (3rd ed.). McGraw-Hill.

122.

O’Donnell

Botelho

Brown

González

G. M.

Head

(2015). Undergraduate research and its impact on student success for underrepresented students: Undergraduate research and its impact. New Directions for Higher Education, 2015(169), 27–38. https://doi.org/10.1002/he.20120

123.

Osman

N. Y.

Schonhardt-Bailey

Walling

J. L.

Katz

J. T.

Alexander

E. K.

(2015). Textual analysis of internal medicine residency personal statements: Themes and gender differences. Medical Education, 49(1), 93–102. https://doi.org/10.1111/medu.12487

124.

Outtz

J. L.

Newman

D. A.

(2010). A theory of adverse impact. In Outtz

J. L.

(Ed.), Adverse impact: Implications for organizational staffing and high stakes selection (pp. 53–94). Routledge/Taylor & Francis Group.

125.

Pashak

T. J.

Handal

P. J.

Ubinger

(2012). Practicing what we preach: How are admissions decisions made for clinical psychology graduate programs, and what do students need to know? Psychology, 3(1), 1–6. https://doi.org/10.4236/psych.2012.31001

126.

Pennock-Román

(1993). Differences among racial and ethnic groups in mean scores on the GRE and SAT: Longitudinal comparisons (GRE Board Professional Report No. 86-09bP; ETS Research Report 91-14). Educational Testing Service. https://onlinelibrary.wiley.com/doi/epdf/10.1002/j.2333-8504.1991.tb01380.x

127.

Petersen

S. L.

Erenrich

E. S.

Levine

D. L.

Vigoreaux

Gile

(2018). Multi-institutional study of GRE scores as predictors of STEM PhD degree completion: GRE gets a low mark. PLOS ONE, 13(10), Article e0206570. https://doi.org/10.1371/journal.pone.0206570

128.

Powers

D. E.

Fowles

M. E.

(1997). The personal statement as an indicator of writing skill: A cautionary note. Educational Assessment, 4(1), 75–87.

129.

Powers

D. E.

Rock

D. A.

(1999). Effects of coaching on SAT I: Reasoning Test scores. Journal of Educational Measurement, 36(2), 93–118. https://doi.org/10.1111/j.1745-3984.1999.tb00549.x

130.

Pruitt

A. S.

Isaac

P. D.

(1985). Discrimination in recruitment, admission, and retention of minority graduate students. The Journal of Negro Education, 54(4), 526–536. https://doi.org/10.2307/2294713

131.

Quillian

Pager

Hexel

Midtbøen

(2017). Meta-analysis of field experiments shows no change in racial discrimination in hiring over time. Proceedings of the National Academy of Sciences, USA, 114(41), 10870–10875. https://doi.org/10.1073/pnas.1706255114

132.

Roth

P. L.

Bobko

(2000). College grade point average as a personnel selection device: Ethnic group differences and potential adverse impact. Journal of Applied Psychology, 85(3), 399–406. https://doi.org/10.1037/0021-9010.85.3.399

133.

Roth

P. L.

Van Iddekinge

C. H.

Huffcutt

A. I.

Eidson

C. E.

Jr. Bobko

(2002). Corrections for range restriction in structured interview ethnic group differences: The values may be larger than researchers thought. Journal of Applied Psychology, 87(2), 369–376.

134.

Russell

S. H.

Hancock

M. P.

McCullough

(2007). The pipeline: Benefits of undergraduate research experiences. Science, 316(5824), 548–549. https://doi.org/10.1126/science.1140384

135.

Sacco

J. M.

Scheu

C. R.

Ryan

A. M.

Schmitt

(2003). An investigation of race and sex similarity effects in interviews: A multilevel approach to relational demography. Journal of Applied Psychology, 88(5), 852–865.

136.

Sackett

P. R.

Kuncel

N. R.

(2018). Eight myths about standardized admissions testing. In Buckley

Letukas

Wildavsky

(Eds.), Measuring success: Testing, grades, and the future of college admissions (pp. 13–39). Johns Hopkins University Press.

137.

Sackett

P. R.

Kuncel

N. R.

Beatty

A. S.

Rigdon

J. L.

Shen

Kiger

T. B.

(2012). The role of socioeconomic status in SAT-grade relationships and in college admissions decisions. Psychological Science, 23(9), 1000–1007. https://doi.org/10.1177/0956797612438732

138.

Santelices

M. V.

Wilson

(2012). On the relationship between differential item functioning and item difficulty: An issue of methods? Item response theory approach to differential item functioning. Educational and Psychological Measurement, 72(1), 5–36. https://doi.org/10.1177/0013164411412943

139.

Scherbaum

C. A.

Goldstein

H. W.

(2008). Examining the relationship between race-based differential item functioning and item difficulty. Educational and Psychological Measurement, 68(4), 537–553. https://doi.org/10.1177/0013164407310129

140.

Schmader

Whitehead

Wysocki

V. H.

(2007). A linguistic comparison of letters of recommendation for male and female chemistry and biochemistry job applicants. Sex Roles, 57(7–8), 509–514. https://doi.org/10.1007/s11199-007-9291-4

141.

Schmidt

F. L.

Hunter

J. E.

(1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262–274.

142.

Schwartz

(1982). Reinforcement-induced behavioral stereotypy: How not to teach people to discover rules. Journal of Experimental Psychology: General, 111(1), 23–59. https://doi.org/10.1037/0096-3445.111.1.23

143.

Shaw

Holbrook

Bourke

(2013). Student experience of final-year undergraduate research projects: An exploration of ‘research preparedness.’ Studies in Higher Education, 38(5), 711–727. https://doi.org/10.1080/03075079.2011.592937

144.

Sheard

(2009). Hardiness commitment, gender, and age differentiate university academic performance. British Journal of Educational Psychology, 79(1), 189–204. https://doi.org/10.1348/000709908X304406

145.

Shewach

O. R.

McNeal

K. D.

Kuncel

N. R.

Sackett

P. R.

(2019). Bunny Hill or Black Diamond: Differences in advanced course-taking in college as a function of cognitive ability and high school GPA. Educational Measurement: Issues and Practice, 38(1), 25–35. https://doi.org/10.1111/emip.12212

146.

Sireci

S. G.

(2016). On the validity of useless tests. Assessment in Education: Principles, Policy & Practice, 23(2), 226–235. https://doi.org/10.1080/0969594X.2015.1072084

147.

Snyder

J. A.

(2020, July 6). Inequities in American society go well beyond testing (opinion). Inside Higher Ed. https://www.insidehighered.com/admissions/views/2020/07/06/inequities-american-society-go-well-beyond-testing-opinion

148.

Society for Industrial and Organizational Psychology. (2018). Principles for the validation and use of personnel selection procedures (5th ed.). Industrial and Organizational Psychology: Perspectives on Science and Practice, 11(Suppl 1), 2–97. https://doi.org/10.1017/iop.2018.195

149.

Sonnert

Fox

M. F.

(2012). Women, men, and academic performance in science and engineering. Journal of Higher Education, 83(1), 73–101.

150.

Stasser

Titus

(1985). Pooling of unshared information in group decision making: Biased information sampling during discussion. Journal of Personality and Social Psychology, 48(6), 1467–1478. https://doi.org/10.1037/0022-3514.48.6.1467

151.

Syverson

S. T.

Franks

V. W.

Hiss

W. C.

(2018). Defining access: How test-optional works. National Association for College Admission Counseling. https://www.nacacnet.org/news–publications/Research/Defining-Access/

152.

Terenzini

P. T.

Cabrera

A. F.

Bernal

E. M.

(2001). Swimming against the tide: The poor in American higher education (ERIC Number ED562879). Education Resources Information Center, Institute of Education Sciences, U.S. Department of Education. https://eric.ed.gov/?id=ED562879

153.

Terregino

C. A.

McConnell

Reiter

H. I.

(2015). The effect of differential weighting of academics, experiences, and competencies measured by multiple mini interview (MMI) on race and ethnicity of cohorts accepted to one medical school. Academic Medicine: Journal of the Association of American Medical Colleges, 90(12), 1651–1657. https://doi.org/10.1097/ACM.0000000000000960

154.

Tyson

(2014, June 16). STEM graduate programs place too much emphasis on GRE scores, physicists say. Inside Higher Ed. https://www.insidehighered.com/news/2014/06/16/stem-graduate-programs-place-too-much-emphasis-gre-scores-physicists-say

155.

University of California Academic Senate. (2020). Report of the UC Academic Council Standardized Testing Task Force (STTF). https://senate.universityofcalifornia.edu/_files/committees/sttf/sttf-report.pdf

156.

U.S. Equal Employment Opportunity Commission. (n.d.). Race/color discrimination. Retrieved on October 6, 2021, from https://www.eeoc.gov/racecolor-discrimination

157.

Voyer

S. D.

(2014). Gender differences in scholastic achievement: A meta-analysis. Psychological Bulletin, 140(4), 1174–1204. https://doi.org/10.1037/a0036620

158.

Walpole

(2003). Socioeconomic status and college: How SES affects college experiences and outcomes. The Review of Higher Education, 27(1), 45–73. https://doi.org/10.1353/rhe.2003.0044

159.

Walpole

Burton

N. W.

Kanyi

Jackenthal

(2002). Selecting successful graduate students: In-depth interviews with GRE users. ETS Research Report Series, 2002(1), i–29. https://doi.org/10.1002/j.2333-8504.2002.tb01875.x

160.

Weissman

M. B.

(2020). Do GRE scores help predict getting a physics Ph.D.? A comment on a paper by Miller et al. Science Advances, 6(23), Article eaax3787. https://doi.org/10.1126/sciadv.aax3787

161.

Wendler

Bridgeman

(2014). The research foundation for the GRE revised General Test: A compendium of studies. Educational Testing Service. https://www.ets.org/gre/compendium

162.

White

S. W.

Xia

Edwards

(2021). Race, gender, and scholarly impact: Disparities for women and faculty of color in clinical psychology. Journal of Clinical Psychology, 77(1), 78–89. https://doi.org/10.1002/jclp.23029

163.

Williams

L. A.

Writer

C. S.

(2019). More than half of grad student survey respondents report discrimination. The Harvard Crimson. https://www.thecrimson.com/article/2019/11/7/gsc-half-discrimination/

164.

Zajonc

R. B.

(1968). Attitudinal effects of mere exposure. Journal of Personality and Social Psychology, 9(2, Pt. 2), 1–27. https://doi.org/10.1037/h0025848

165.

Zhang

Kuncel

N. R.

(2020). Moving beyond the Brag Sheet: A meta-analysis of biodata measures predicting student outcomes. Educational Measurement: Issues and Practice, 39(3), 106–121. https://doi.org/10.1111/emip.12313

166.

Zieky

(2003). A DIF primer. Educational Testing Service. https://www.ets.org/s/praxis/pdf/dif_primer.pdf

167.

Zwick

(2012). A review of ETS differential item functioning assessment procedures: Flagging rules, minimum sample size requirements, and criterion refinement. ETS Research Report Series, 2012(1), i–30. https://doi.org/10.1002/j.2333-8504.2012.tb02290.x