Are Students on Track? Comparing the Predictive Validity of Administrative and Survey Measures of Cognitive and Noncognitive Skills for Long-Term Outcomes

Abstract

Education leaders need valid metrics to predict students’ long-term success. We use a unique data set with cognitive skills, self-regulation, behavior, course performance, and test scores for eighth-grade students from a Northeast school district. We link these data to students’ high school outcomes, college enrollment, persistence, and on-time degree completion. Survey-based cognitive and self-regulation measures predict high school and college outcomes. However, these relationships become small and lose statistical significance when test scores, grade point average, and an absences-suspensions index are included in the predictive models. For leaders hoping to identify the best on-track indicators for college completion, the information collected in student longitudinal data systems better predicts both short- and long-term educational outcomes than the survey-based self-regulation and cognitive measures.

Keywords

achievement cognitive processes/development computerized testing educational policy experimental research longitudinal studies measurements regression analyses

Get full access to this article

View all access options for this article.

References

Almlund

Duckworth

A. L.

Heckman

Kautz

(2011). Personality psychology and economics. In Hanushek

E. A.

Machin

Woessmann

(Eds.), Handbook of the economics of education (pp. 1–181). Elsevier.

Ang

(2021). The effects of police violence on inner-city students. The Quarterly Journal of Economics, 136(1), 115–168, https://doi.org/10.1093/qje/qjaa027

Austin

Figlio

Goldhaber

Hanushek

Kilbride

Koedel

Lee

J. S.

Lou

Özek

Parsons

Rivkin

Sass

Strunk

(2020). Where are initially low-performing students the most likely to succeed? A multi-state analysis of academic mobility (CALDER Working Paper No. 227–0220). https://wes-austin.com/files/CALDER_academic_mobility.pdf

Bowen

W. G.

Chingos

M. M.

McPherson

M. S.

(2009). Crossing the finish line: Completing college at America’s public universities. Princeton University Press.

Brown

Sherbenou

R. J.

Johnsen

S. K.

(2010). Test of nonverbal intelligence—Fourth edition (Version A). The Psychological Corp.

Camara

Kimmel

Scheuneman

Sawtell

E. A.

(2003). Whose grades are inflated? College Entrance Examination Board, New York. https://files.eric.ed.gov/fulltext/ED563203.pdf

Canbolat

(2024). Early warning for whom? Regression discontinuity evidence from the effect of early warning system on student absence. Educational Evaluation and Policy Analysis. Advance online publication. https://doi.org/10.3102/01623737231221503

Case

Midian Kurland

Goldberg

(1982). Operational efficiency and the growth of short-term memory span. Journal of Experimental Child Psychology, 33, 386–404.

Chetty

Friedman

J. N.

Hilger

Saez

Whitmore Schanzenbach

Yagan

(2011). How does your kindergarten classroom affect your earnings? Evidence from Project STAR. The Quarterly Journal of Economics, 126(4), 1593–1660.

10.

Chetty

Friedman

J. N.

Rockoff

J. E.

(2014). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review, 104(9), 2593–2632.

11.

Connelly

B. S.

Ones

D. S.

(2010). Another perspective on personality: Meta-analytic integration of observers’ accuracy and predictive validity. Psychological Bulletin, 136(6), 1092–1122.

12.

Cowan

(2005). Working-memory capacity limits in a theoretical context. In Izawa

Ohta

(Eds.), Human learning and memory: Advances in theory and applications. The 4th Tsukuba International Conference on Memory (pp. 155–175). Erlbaum.

13.

Dee

T. S.

West

M. R.

(2011). The noncognitive returns to class size. Education Evaluation and Policy Analysis, 33(1), 23–46.

14.

Duckworth

A. L.

Quinn

P. D.

(2009). Development and validation of the Short Grit Scale (Grit-S). Journal of Personality Assessment, 91, 166–174.

15.

Duckworth

A. L.

Peterson

Matthews

M. D.

Kelly

D. R.

(2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92, 1087–1101.

16.

Duckworth

A. L.

Quinn

P. D.

Tsukayama

(2012). What No Child Left Behind leaves behind: The roles of IQ and self-control in predicting standardized achievement test scores and report card grades. Journal of Educational Psychology, 104(2), 439–451. https://doi.org/10.1037/a0026280

17.

Duckworth

A. L.

Yeager

D. S.

(2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44(4), 237–251.

18.

Dweck

(2006). Mindset: The new psychology of success. Random House.

19.

Easton

J. Q.

Johnson

Sartain

(2017). The predictive power of ninth-grade GPA. University of Chicago Consortium on School Research.

20.

Farrington

C. A.

Roderick

Allensworth

Nagaoka

Keyes

T. S.

Johnson

D. W.

Beechum

N. O.

(2012). Teaching adolescents to become learners: The role of noncognitive factors in shaping school performance–A critical literature review. Consortium on Chicago School Research.

21.

Finn

A. S.

Kraft

M. A.

West

M. R.

Leonard

J. A.

Bish

C. E.

Martin

R. E.

Sheridan

M. A.

Gabrieli

C. F. O.

Gabrieli

J. D. E.

(2014). Cognitive skills, student achievement tests, and schools. Psychological Science, 25(3), 736–744. https://doi.org/10.1177/0956797613516008

22.

Fry

A. F.

Hale

(1996). Processing speed, working memory, and fluid intelligence: Evidence for a developmental cascade. Psychological Science, 7(4), 237–241. http://www.jstor.org/stable/40062952

23.

Galla

B. M.

Shulman

E. P.

Plummer

B. D.

Gardner

Hutt

S. J.

Goyer

J. P.

D’Mello

S. K.

Finn

A. S.

Duckworth

A. L.

(2019). Why high school grades are better predictors of on-time college graduation than are admissions test scores: The roles of self-regulation and cognitive ability. American Education Research Journal, 20(10), 1–39.

24.

Geiser

Santelices

M. V.

(2007). Validity of high-school grades in predicting student success beyond the freshman year: High-school record vs. standardized tests as indicators of four-year college outcomes. Center for Studies in Higher Education.

25.

Goldhaber

Wolff

Daly

(2020). Assessing the accuracy of elementary school test scores as predictors of students’ high school outcomes (CALDER Working Paper No. 235–0520). https://files.eric.ed.gov/fulltext/ED605735.pdf

26.

Heckman

J. J.

Humphries

Veramendi

(2018). Returns to education: The causal effects of education on earnings, health, and smoking. Journal of Political Economy, 126(S1), S197–S246.

27.

Heckman

J. J.

Mosso

(2014). The economics of human development and social mobility. Annual Review of Economics, 6(1), 689–733.

28.

Jackson

(2018). What do test scores miss? The importance of teacher effects on non-test score outcomes. Journal of Political Economy, 126, 2072–2107.

29.

Jacob

Rothstein

(2016). The measurement of student ability in modern assessment systems. Journal of Economic Perspectives, 30(3), 85–108.

30.

John

O. P.

Srivastava

(1999). The Big Five trait taxonomy: History, measurement, and theoretical perspectives. In Pervin

L. A.

John

O. P.

(Eds.), Handbook of personality: Theory and research (2nd ed., pp. 102–138). The Guilford Press.

31.

Kanopka

Claro

Loeb

West

Fricke

(2024). Are changes in reported social-emotional skills just noise? The predictive power of longitudinal differences in self-reports. AERA Open, 10. https://doi.org/10.1177/23328584241233277

32.

Kautz

T. D.

Zanoni

(2014). Measuring and fostering noncognitive skills in adolescence: Evidence from Chicago Public Schools and the OneGoal Program [Unpublished manuscript]. Department of Economics, University of Chicago.

33.

Kuncel

N. R.

Hezlett

S. A.

(2007). Standardized tests predict graduate students’ success. Science, 315(5815), 1080–1081. https://doi.org/10.1126/science.1136618

34.

Lleras

(2008). Do skills and behaviors in high school matter? The contribution of noncognitive factors in explaining differences in educational attainment and earnings. Social Science Research, 37(3), 888–902.

35.

Liu

Kuhfeld

Lee

(2025). Noncognitive factors and student long-run success: Comparing the predictive validity of observable academic behaviors and social-emotional skills. Educational Policy, 39(1), 131–169. https://doi.org/10.1177/08959048231209262

36.

I. S.

Wang

Mount

M. K.

(2011). Validity of observer ratings of the five-factor model of personality traits: A meta-analysis. Journal of Applied Psychology, 96(4), 762–773.

37.

Poropat

A. E.

(2014). Other-rated personality and academic performance: Evidence and implication. Learning and Individual Differences, 34, 24–32.

38.

Sackett

P. R.

Borneman

M. J.

Connelly

B. S.

(2008). High stakes testing in higher education and employment: Appraising the evidence for validity and fairness. American Psychologist, 63(4), 215–227. https://doi.org/10.1037/0003-066X.63.4.215.

39.

Soland

Zamarro

Cheng

Hitt

(2019). Identifying naturally occurring direct assessments of social-emotional competencies: The promise and limitations of survey and assessment disengagement metadata. Educational Researcher, 48(7), 466–478.

40.

Tsukayama

Duckworth

A. L.

Kim

(2013). Domain-specific impulsivity in school-age children. Developmental Science, 16, 879–893.

41.

Wechsler

(2003). Wechsler Intelligence Scale for Children—Fourth edition. The Psychological Corp.

42.

West

M. R.

Kraft

M. A.

Finn

A. S.

Martin

R. E.

Duckworth

A. L.

Gabrieli

C. F. O.

Gabrieli

J. D. E.

(2016). Promise and paradox: Measuring students’ noncognitive skills and the impact of schooling. Education Evaluation and Policy Analysis, 38(1), 148–170.

43.

West

M. R.

Pier

Fricke

Hough

Loeb

Meyer

R. H.

Rice

A. B.

(2020). Trends in student social-emotional learning: Evidence from the first large-scale panel student survey. Educational Evaluation and Policy Analysis, 42(2), 279–303.

44.

Willingham

W. W.

Pollack

J. M.

Lewis

(2002). Grades and test scores: Accounting for observed differences. Journal of Educational Measurement, 39(1), 1–37.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.29 MB