How Readability Factors Are Differentially Associated With Performance for Students of Different Backgrounds When Solving Mathematics Word Problems

Abstract

The link between reading and mathematics achievement is well known, and an important question is whether readability factors in mathematics problems are differentially impacting student groups. Using 20 years of data from the National Assessment of Educational Progress and the Trends in International Mathematics and Science Study, we examine how readability factors—such as length, word difficulty, and pronouns—interact with student background characteristics—such as race/ethnicity, mathematics achievement, and socioeconomic status. Textual features that make problems more difficult to process appear to differentially negatively impact struggling students, while features that make language easier to process appear to differentially positively impact struggling students. It is critical that readability along various dimensions be considered when designing instruction and assessment.

Keywords

achievement gap language comprehension/development NAEP mathematics education

Get full access to this article

View all access options for this article.

References

Abedi

Hofstetter

Baker

Lord

(2001). NAEP mathematics performance and test accommodations: Interactions with student language background (CSE Technical Report 536). Retrieved from http://cresst.org/wp-content/uploads/newTR536.pdf

Abedi

Leon

Mirocha

(2001). Validity of standardized achievement tests for English language learners. Paper presented at the American Educational Research Association Conference, Seattle, WA.

Abedi

Lord

(2001). The language factor in mathematics tests. Applied Measurement in Education, 14, 219–234.

Abedi

Lord

Hofstetter

(2001). Impact of selected background variables on students’ NAEP mathematics performance (NCES Working Paper Series 2001-11). Washington, DC: U.S. Department of Education, National Center for Education Statistics.

Abedi

Lord

Hofstetter

Baker

(2000). Impact of accommodation strategies on English language learners’ test performance. Educational Measurement: Issues and Practice, 19(3), 16–26.

Adams

A. E.

Pegg

Case

(2015). Anticipation guides: Reading for mathematics understanding. Mathematics Teacher, 108, 498–504.

Arnold

J. E.

Losongco

Wasow

Ginstrom

(2000). Heaviness vs. newness: The effects of structural complexity and discourse status on constituent ordering. Language, 76, 28–55. doi:10.2307/417392

August

Shanahan

(Eds.). (2008). Developing reading and writing in second-language learners: Lessons from the report of the National Literacy Panel on Language-Minority Children and Youth. New York, NY: Routledge.

Bailin

Grafstein

(2001). The linguistic assumptions underlying readability formulae: A critique. Language & Communication, 21(3), 285–301.

10.

Ball

A. F.

(1996). Expository writing patterns of African American students. The English Journal, 85(1), 27–36.

11.

Barth

A. E.

Tolar

T. D.

Fletcher

J. M.

Francis

(2014). The effects of student and text characteristics on the oral reading fluency of middle-grade students. Journal of Educational Psychology, 106, 162–180.

12.

Bates

Maechler

Bolker

Walker

(2014). lme4: Linear mixed-effects models using Eigen and S4. R package version 1.1-7. Retrieved from https://cran.r-project.org/web/packages/lme4/lme4.pdf

13.

Bauer

L. M.

Altarriba

(2008). An investigation of sex differences in word ratings across concrete, abstract, and emotion words. The Psychological Record, 58, 465–474.

14.

Benjamin

R. G.

(2012). Reconstructing readability: Recent developments and recommendations in the analysis of text difficulty. Educational Psychology Review, 24, 63–88.

15.

Benjamini

Hochberg

(1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57, 289–300.

16.

Bernhardt

E. B.

Kamil

M. L.

(1995). Interpreting relationships between L1 and L2 reading: Consolidating the linguistic threshold and the linguistic interdependence hypotheses. Applied Linguistics, 16(1), 15–34.

17.

Bertram

Newman

(1981). Why readability formulas fail (Reading Education Report No. 28). Urbana, IL: Illinois University, Urbana Center for the Study of Reading.

18.

Carpenter

Fennema

Franke

Levi

Empson

(1999). Children’s mathematics: Cognitively guided instruction. New York, NY: Heinemann.

19.

Carpenter

Moser

(1984). The acquisition of addition and subtraction concepts in grades one through three. Journal for Research in Mathematics Education, 15(3), 179–202.

20.

Charity

A. H.

Scarborough

H. S.

Griffin

D. M.

(2004). Familiarity with school English in African American children and its relation to early reading achievement. Child Development, 75, 1340–1356.

21.

Chatterji

(2006). Reading achievement gaps, correlates, and moderators of early reading achievement: Evidence from the Early Childhood Longitudinal Study (ECLS) kindergarten to first grade sample. Journal of Educational Psychology, 98, 489–507.

22.

Chinn

(2000). A simple method for converting an odds ratio to effect size for use in meta-analysis. Statistics in Medicine, 19(22), 3127–3131.

23.

Clinton

Seipel

van den Broek

McMaster

K. L.

Kendeou

Carlson

Rapp

D. N.

(2014). Gender differences in inference generation by fourth-grade students. Journal of Research in Reading 37, 356–374.

24.

Clinton

van den Broek

(2012). Interest, inferences, and learning from texts. Learning and Individual Differences, 22, 650–663.

25.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

26.

Coltheart

(1981). MRC psycholinguistic database user manual: Version 1. New York, NY: Springer.

27.

Common Core State Standards Initiative. (2010). Common Core State Standards for English Language Arts. Washington, DC: National Governors Association Center for Best Practices and the Council of Chief State School Officers.

28.

Cooper

Harries

(2005). Making sense of realistic word problems: Portraying working class “failure” on a division with remainder problem. International Journal of Research & Method in Education, 28, 147–169.

29.

Crawford

Tindal

Stieber

(2001). Using oral reading rate to predict student performance on statewide achievement tests. Educational Assessment, 7(4), 303–323.

30.

Cummins

D. D.

Kintsch

Reusser

Weimer

(1988). The role of understanding in solving word problems. Cognitive Psychology, 20, 405–438.

31.

De Groot

Keijzer

. (2000). What is hard to learn is easy to forget: The roles of word concreteness, cognate status, and word frequency in foreign-language vocabulary learning and forgetting. Language Learning, 50(1), 1–56.

32.

Delpit

Dowdy

J. K.

(Eds.). (2002). The skin that we speak: Thoughts on language and culture in the classroom. New York, NY: The New Press.

33.

Department for Education and Skills. (2007). Gender and education: The evidence on pupils in England (Reference No. 00389-2007BKT-EN). Nottingham, UK: DfES Publications

34.

Durik

Harackiewicz

(2007). Different strokes for different folks: How individual interest moderates effects of situational factors on task interest. Journal of Educational Psychology, 99, 597–610.

35.

Frankenstein (2009). Developing a criticalmathematical numeracy through real real-life word problems. In B.

Greer

Verschaffel

Van Dooren

Mukhopadhyay

(Eds.) Word and worlds: Modelling verbal descriptions of situations (pp. 111–130). Rotterdam, Netherlands: Sense Publishers.

36.

Fry

(2007). How far behind in mathematics and reading are English language learners? Retrieved from http://www.pewhispanic.org/2007/06/06/how-far-behind-in-math-and-reading-are-english-language-learners/

37.

Fuchs

L. S.

Fuchs

Prentice

Hamlett

C. L.

Finelli

Courey

S. J.

(2004). Enhancing mathematical problem solving among third-grade students with schema-based instruction. Journal of Educational Psychology, 96, 635–647.

38.

Fuentes

(1998). Reading comprehension in mathematics. The Clearing House, 72(2), 81–88.

39.

Graesser

A. C.

McNamara

D. S.

Louwerse

M. M.

Cai

(2004). Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36, 193–202.

40.

Gerofsky

(2009). Genre, simulacra, impossible exchange, and the real: How postmodern theory problematises word problems. In Greer

Verschaffel

Van Dooren

Mukhopadhyay

(Eds.), Word and worlds: Modelling verbal descriptions of situations (pp. 39–54). Rotterdam, the Netherlands: Sense Publishers.

41.

Halpern

D. F.

Benbow

C. P.

Geary

D. C.

Gur

R. C.

Hyde

J.S.

Gernsbacher

M. A.

(2007). The science of sex differences in science and mathematics. Psychological Science in the Public Interest, 8(1), 1–51.

42.

Hartocollis

(2016, February 9). New, reading-heavy SAT has student worried. The New York Times, p. A1.

43.

Hecht

S. A.

Torgesen

J. K.

Wagner

R. K.

Rashotte

C. A.

(2001). The relations between phonological processing abilities and emerging individual differences in mathematical computation skills: A longitudinal study from second to fifth grades. Journal of Experimental Child Psychology, 79, 192–227.

44.

Hegarty

Mayer

R. E.

Monk

C. A.

(1995). Comprehension of arithmetic word problems: A comparison of successful and unsuccessful problem solvers. Journal of Educational Psychology, 87, 18–32.

45.

Helwig

Rozek-Tedesco

M. A.

Tindal

Heath

Almond

P. J.

(1999). Reading as an access to mathematics problem solving on multiple-choice tests for sixth-grade students. The Journal of Educational Research, 93(2), 113–125.

46.

Hill

C. J.

Bloom

H. S.

Black

A. R.

Lipsey

M. W.

(2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), 172–177.

47.

Inoue

(2005). The realistic reasons behind unrealistic solutions: The role of interpretive activity in word problem solving. Learning and Instruction, 15(1), 69–83.

48.

Jiban

C. L.

Deno

S. L.

(2007). Using mathematics and reading curriculum-based measurements to predict state mathematics test performance are simple one-minute measures technically adequate? Assessment for Effective Intervention, 32(2), 78–89.

49.

Jin

Y. S.

(1990). Effects of concreteness on cross-language priming in lexical decisions. Perceptual and Motor Skills, 70, 1139–1154.

50.

Kazemi

(2002). Exploring test performance in mathematics: The questions children’s answers raise. The Journal of Mathematical Behavior, 21(2), 203–224.

51.

Kelly

Nord

C. W.

Jenkins

Chan

J. Y.

Kastberg

(2013). Performance of US 15-year-old students in mathematics, science, and reading literacy in an international context. First look at PISA 2012 (NCES 2014-024). Washington, DC: National Center for Education Statistics.

52.

Khisty

L. L.

(1995). Making inequality: Issues in language and meanings in mathematics teaching with Hispanic students. In Secada

W. G.

Fennema

Adajian

L. B.

(Eds.), New directions for equity in mathematics instruction (pp. 279–297). Cambridge, UK: Cambridge University Press.

53.

Kim

Jiang

Song

(2015). The effects of interest and utility value on mathematics engagement and achievement. In Renninger

Nieswandt

Hidi

(Eds.), Interest in mathematics and science learning (pp. 63–78). Washington, DC: American Educational Research Association.

54.

Kintsch

(1998). Comprehension: A paradigm for cognition. Cambridge, UK: Cambridge University Press.

55.

Kintsch

Greeno

J. G.

(1985). Understanding and solving word arithmetic problems. Psychological Review, 92(1), 109–129.

56.

Kirkwood

K. J.

Wolfe

R. G.

(1980). Matching students and reading materials: A cloze-procedure method for assessing the reading ability of students and the readability of textual materials. Toronto, Canada: Ontario Government Bookstore.

57.

Koedinger

Nathan

(2004). The real story behind story problems: Effects of representations on quantitative reasoning. Journal of the Learning Sciences, 13(2), 129–164.

58.

Ladsen-Billings

(1995). Making mathematics meaningful in multicultural contexts. In Secada

(Ed.), For equity in mathematics education (pp. 126–145). Cambridge, UK: Cambridge University Press.

59.

Ladsen-Billings

(1997). It doesn’t add up: African American students’ mathematics achievement. Journal for Research in Mathematics Education, 28, 697–708.

60.

LaRoche

Joncas

Foy

(2016). Sample design in TIMSS 2015. In Martin

Mullis

Hooper

(Eds.) Methods and Procedures for TIMSS 2015 (pp. 3.1–3.38). Chesnut Hill, MA: Boston College.

61.

Lerkkanen

M. K.

Rasku-Puttonen

Aunola

Nurmi

J. E.

(2005). Mathematical performance predicts progress in reading comprehension among 7-year-olds. European Journal of Psychology of Education, 20(2), 121–137.

62.

Lesaux

N. K.

Kieffer

M. J.

(2010). Exploring sources of reading comprehension difficulties among language minority learners and their classmates in early adolescence. American Educational Research Journal, 47, 596–632.

63.

Lesh

Harel

(2003). Problem solving, modeling, and local conceptual development. Mathematical Thinking and Learning, 5(2/3), 157–190.

64.

Logan

Johnston

(2010). Investigating gender differences in reading. Educational Review, 62(2), 175–187.

65.

Loveless

Fennel

Williams

Ball

Banfield

(2008). Chapter 9: Report of the Subcommittee on the National Survey of Algebra I Teachers. Foundations for Success: Report of the National Mathematics Advisory Panel. Retrieved from https://www2.ed.gov/about/bdscomm/list/mathpanel/report/final-report.pdf

66.

Martiniello

(2008). Language and the performance of English-language learners in mathematics word problems. Harvard Educational Review, 78, 333–368.

67.

Mayer

(2001). Multimedia learning. Cambridge, UK: Cambridge University Press.

68.

McNamara

D. S.

Graesser

A. C.

McCarthy

P. M.

Cai

(2014). Automated evaluation of text and discourse with Coh-Metrix. New York, NY: Cambridge University Press.

69.

McNamara

D. S.

Louwerse

M. M.

Cai

Graesser

(2013). Coh-Metrix version 3.0. Retrieved from http://cohmetrix.com

70.

Miller

J. R.

Kintsch

(1980). Readability and recall of short prose passages: A theoretical analysis. Journal of Experimental Psychology: Human Learning and Memory, 6(4), 335–353.

71.

Mislevy

R. J.

Beaton

A. E.

Kaplan

Sheehan

K. M.

(1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29(2), 133–161.

72.

Montague

Warger

Morgan

T. H.

(2000). Solve it! Strategy instruction to improve mathematical problem solving. Learning Disabilities Research & Practice, 15(2), 110–116.

73.

Mullis

I. V.

Martin

M. O.

Foy

(2013). The impact of reading ability on TIMSS mathematics and science achievement at the fourth grade: An analysis by item reading demands. In Martin

M. O.

Mullis

I. V. S.

(Eds.), TIMSS and PIRLS 2011: Relationships among reading, mathematics, and science achievement at the fourth grade—Implications for early learning (pp. 67–108). Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College and IEA.

74.

Mullis

I. V.

Martin

M. O.

Foy

Drucker

K. T.

(2012). PIRLS 2011 international results in reading. Amsterdam, the Netherlands: International Association for the Evaluation of Educational Achievement.

75.

Mullis

I. V.

Martin

M. O.

Ruddock

G. J.

O’Sullivan

C. Y.

Preuschoff

(2009). TIMSS 2011 assessment frameworks. Amsterdam, the Netherlands: International Association for the Evaluation of Educational Achievement.

76.

Nasir

Hand

Taylor

E. V.

(2008). Culture and mathematics in school: Boundaries between “cultural” and “domain” knowledge in the mathematics classroom and beyond. Review of Research in Education, 32(1), 187–240.

77.

Nathan

Kintsch

Young

(1992). A theory of algebra-word-problem comprehension and its implications for the design of learning environments. Cognition and Instruction, 9, 329–389.

78.

National Center for Educational Statistics, U.S. Department of Education. (2009). The nation’s report card: An overview of procedures for the NAEP assessment (NCES 2009–493). Washington, DC: U.S. Government Printing Office.

79.

National Center for Educational Statistics, U.S. Department of Education (2013). NAEP 2012: Trends in academic progress. Retrieved from http://nces.ed.gov/nationsreportcard/subject/publications/main2012/pdf/2013456.pdf

80.

Neidorf

T. S.

Binkley

Gattis

Nohara

(2006). Comparing mathematics content in the National Assessment of Educational Progress (NAEP), Trends in International Mathematics and Science Study (TIMSS), and Program for International Student Assessment (PISA) 2003 assessments (NCES 2006-029). Washington, DC: National Center for Education Statistics.

81.

Newman

Groom

C. J.

Handelman

L. D.

Pennebaker

J. W.

(2008). Gender differences in language use: An analysis of 14,000 text samples. Discourse Processes, 45, 211–236.

82.

Noble

K. G.

Farah

M. J.

McCandliss

B. D.

(2006). Socioeconomic background modulates cognition-achievement relationships in reading. Cognitive Development, 21, 349–368.

83.

Organisation for Economic Cooperation and Development. (2010). PISA 2009 results: What students know and can do. Paris: OECD Publications.

84.

Pekrun

Goetz

Titz

Perry

R. P.

(2002). Academic emotions in students’ self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist, 37(2), 91–105.

85.

Pennebaker

J. W.

Chung

C. K.

Ireland

Gonzales

Booth

R. J.

(2007). The development and psychometric properties of LIWC2007. Austin, TX: LIWC.Net.

86.

Perry

McConney

(2010). Does the SES of the school matter? An examination of socioeconomic status and student achievement using PISA 2003. The Teachers College Record, 112(4), 7–8.

87.

Pimm

(1987). Speaking mathematically: Communication in mathematics classrooms. New York, NY: Routledge.

88.

Pimm

(1994). Spoken mathematical classroom culture: Artifice and artificiality. In Lerman

(Ed.), Cultural perspectives on the mathematics classroom (pp. 133–147). Norwell, MA: Kluwer Academic Publishers.

89.

Provasnik

Kastberg

Ferraro

Lemanski

Roey

Jenkins

(2012). Highlights from TIMSS 2011: Mathematics and science achievement of us fourth-and eighth-grade students in an international context (NCES 2013-009). Washington, DC: National Center for Education Statistics.

90.

Reardon

S. F.

(2011). The widening academic achievement gap between the rich and the poor: New evidence and possible explanations. In Duncan

G. J.

Murnane

R. J.

(Eds.), Whither opportunity: Rising inequality, schools, and children’s life chances (pp. 91–116). New York, NY: Russell Sage Foundations.

91.

Robinson

J. P.

Lubienski

S. T.

(2011). The development of gender achievement gaps in mathematics and reading during elementary and middle school examining direct cognitive assessments and teacher ratings. American Educational Research Journal, 48, 268–302.

92.

Saxe

(1988). Candy selling and mathematics learning. Educational Researcher, 17(6), 14–21.

93.

Schulz

R .A.

(1981). Literature and readability: Bridging the gap in foreign language reading. The Modern Language Journal, 65, 43–53.

94.

Shaftel

Belton-Kocher

Glasnapp

Poggio

(2006). The impact of language characteristics in mathematics test items on the performance of English language learners and students with disabilities. Educational Assessment, 11(2), 105–126.

95.

Shanahan

(2008). Teaching disciplinary literacy to adolescents: Rethinking content-area literacy. Harvard Educational Review, 78(1), 40–59.

96.

Sirin

S. R.

(2005). Socioeconomic status and academic achievement: A meta-analytic review of research. Review of Educational Research, 75(3), 417–453.

97.

Snijders

T. A. B.

Bosker

(1999). Introduction to multilevel analysis. London: Sage.

98.

Sweller

van Merrienboer

Paas

(1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251–296.

99.

Tate

W. F.

(1994). Race, retrenchment, and the reform of school mathematics. The Phi Delta Kappan, 75, 477–484.

100.

Taylor

(2005). Low income first and second grade students’ engagement in currency exchange: The relationship to mathematical development. Berkeley, CA: University of California, Berkeley.

101.

Twist

Schagen

Hodgson

(2007). Readers and reading: The National Report for England 2006 PIRLS (Progress in International Reading Literacy Study). Slough, UK: NFER.

102.

Vilenius-Tuohimaa

Aunola

Nurmi

(2008). The association between mathematical word problems and reading comprehension. Educational Psychology, 28, 409–426.

103.

Von Davier

Gonzalez

Mislevy

. (2009). What are plausible values and why are they useful. IERI Monograph Series, 2, 9–36.

104.

Walkington

(2013). Using learning technologies to personalize instruction to student interests: The impact of relevant contexts on performance and learning outcomes. Journal of Educational Psychology, 105(4), 932–945.

105.

Walkington

(in press). Design research on personalized problem-posing. In Galindo

Newton

(Eds.), Proceedings of the 39th annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education. Indianapolis, IN: Hoosier Association of Mathematics Teacher Educators.

106.

Walkington

Clinton

Ritter

Nathan

M. J.

(2015). How readability and topic incidence relate to performance on mathematics story problems in computer-based curricula. Journal of Educational Psychology, 107(4), 1051–1074.

107.

Walkington

Clinton

Shivraj

Yovanoff

(2015, April). Association between readability and topic of mathematics word problems and performance on large-scale assessments. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL.

108.

Walkington

Cooper

Howell

(2013). The effects of visual representations and interest-based personalization on solving percent problems. In Martinez

Castro Superfine

(Eds.), Proceedings of the 35th annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education (pp. 533–536). Chicago, IL: University of Illinois at Chicago.

109.

Walkington

Sherman

Petrosino

(2012). “Playing the game” of story problems: Coordinating situation-based reasoning with algebraic representation. Journal of Mathematical Behavior, 31(2), 174–195.

110.

Wolf

M. K.

Leon

(2009). An investigation of the language demands in content assessments for English language learners. Educational Assessment, 14(3–4), 139–159.

111.

(2010). Comparing the similarities and differences of PISA 2003 and TIMSS (OECD Working Papers No. 32). Paris: OECD Publishing.

112.

Yore

Pimm

Tuan

(2007). The literacy component of mathematical and scientific literacy. International Journal of Science and Mathematics Education, 5, 559–589.

113.

Zamanian

Heydari

(2012). Readability of texts: State of the art. Theory and Practice in Language Studies, 2, 43. doi:10.4304/tpls.2.1.43-53

114.

Zygouris-Coe

(2012). Disciplinary literacy and the common core state standards. Topics in Language Disorders, 32(1), 35–50.