Abstract
The stagnation of National Assessment of Educational Progress (NAEP) Writing scores demonstrates the need for research-based instruction that improves writing for all students, especially English learners. In this article, we synthesize the literature on effective instructional practices for this diverse group of learners and describe how these strategies are leveraged in a teacher professional development program that has been previously shown to improve students’ argument writing. Then, we share results of a study that focuses on distinct subgroups of secondary English learners students to (a) determine their needs and challenges and (b) examine the impact of a cognitive strategies approach on rhetorical and linguistic aspects of writing at posttest. Results show English learners have considerable challenges with higher-order tasks involved in writing literary arguments and with the linguistic demands of academic writing before receiving the intervention. However, after receiving the intervention, using descriptive statistics and multiple hierarchical linear regression, we show that these students grew in the areas of presentation of ideas, organization, evidence use, and language use. For example, students designated as reclassified English learners (RFEP [Reclassified Fluent English Proficient]) and students who have even more limited English proficiency (designated as EL [English learner] here) show improvements in many aspects of writing, especially in their ability to write claims and use evidence. In contrast, improvements in language use components were more limited for both groups of learners. Moreover, some of the gains due to being in the treatment were significant enough to bring the average EL student close to parity or beyond their EO (English Only) / IFEP (Initial Fluent English Proficient) peers in the control condition at posttest. We conclude by discussing pedagogical implications for English learners.
Argument writing is essential for success in institutes of higher education as well as in professional environments (Council of Writing Program Administrators, 2011). The importance of argument writing for college and career readiness is emphasized by George Hillocks (2011), who contends that forming arguments is the crux of critical thinking and that argument writing is a skill necessary for success in college and in life. It is also critical to participating in a democratic society. In his words, “Argument is not simply a dispute, as when people disagree with one another or yell at each other. Argument is about making a case in support of a claim in everyday affairs–in science, in policy making, in courtrooms, and so forth” (p. 1). Gerald Graft (2003) concurs that “argument literacy” is fundamental to being educated and suggests that students must be adept at engaging in both oral and written argumentation about complex issues when they enter college. Similarly, a 2009 ACT, Inc., U.S. curriculum survey of postsecondary literacy faculty found that writing “to argue or persuade readers” and writing “to convey information” were the two most important writing skills expected of incoming college students.
When discussing the pivotal role of argument writing in school contexts, the Common Core State Standards for English Language Arts & Literacy in History/Social Studies, Science, and Technical Subjects (CCSS) explain:
An argument is a reasoned, logical way of demonstrating that the writer’s position, belief, or conclusion is valid. English Language Arts students make claims about the worth or meaning of a literary work or works. They defend their interpretations or judgments with evidence from the text(s) they are writing about. In history/social studies, students analyze evidence from multiple primary and secondary sources to advance a claim that is best supported by the evidence, and they argue for a historically or empirically situated interpretation. In science, students make claims in the form of statements or conclusions that answer questions or address problems. (National Governors Association Center for Best Practices & Council of Chief State School Officers, 2010, Appendix A, p. 23)
Hence, in its vision of what it means to be literate in the 21st century, the CCSS and other state standards prioritize the ability to “read closely to determine what the text says explicitly and to make logical inferences from it” (p. 10) and to “write arguments to support claims in an analysis of substantive topics or texts, using valid reasoning and relevant and sufficient evidence” (p. 18).
Despite its importance, most students find argument writing challenging (Salahu-Din et al., 2008). Perie and colleagues (2005), for instance, found in their study that only 15% of the 12th-grade students who scored proficient in writing could write essays containing a clear arguable claim and consistent supporting evidence. Several researchers (Biancarosa & Snow, 2004; Graham & Perin, 2007; Langer, 2002) have also noted the difficulties learners have mastering the advanced reading skills, which entail synthesizing multiple texts with similar and competing perspectives, necessary for critical literacy and argumentative, evidence-based writing. Other researchers report that students are challenged by (a) developing warrants that explain why or how their evidence supports their claims, (b) adapting writing to various purposes and audiences, and (c) acknowledging and refuting potential criticisms of their positions (Kuhn, 1991, 2005; Persky et al., 2003). Further, results on state and national assessments indicate that students have difficulty writing at a proficient or advanced level. For example, on the 2011 administration of the National Assessment of Educational Progress (NAEP) for writing assessed argument writing as 35% of its sample in 8th grade and 40% of its sample in 12th grade. On that assessment, only 27% of all 12th-graders, 35% of White students, 11% of Hispanic students, and 9% of Black students scored at the level of proficient or above in writing (National Center for Education Statistics, 2012). More recent results of the NAEP have not been released; however, the technical report indicates that students performed lower than their 2011 peers (NCES, 2017).
The challenge of developing proficiency in argument writing is even greater for English learners (ELs) in a secondary education context. Meeting the College and Career Readiness Anchor Standards for Reading and Writing in Grades 6-12 can be especially demanding for these students. Even English learners with moderate to strong levels of English proficiency face particular challenges when learning argument writing. Many have not been asked to compose argument essays before, and their experiences, backgrounds, and previous practices may not have prepared them to follow the genre and disciplinary-specific conventions of argument writing. As Bunch (2013) explain, the argument is “grounded in particular socially and culturally developed values and practices that may or may not align with those of students from different backgrounds” (p. 10). As a result, students may not recognize relevant information required to support their perspective, acknowledge or rebut viewpoints inconsistent with their own, or consider the merits of other views. Further, many English learners face an additional cognitive load when writing in English as they are in process of learning the language features and rhetorical styles of English that have yet to be internalized (Fitzgerald, 2017; Scarcella, 2003). These factors may explain why only 1% of English learners at Grades 8 and 12 scored proficient or above on the 2011 NAEP.
Given that English learners in Grades 7-12 are the fastest-growing segment of the K-12 student population (Francis et al., 2006), the disparities in academic and argument writing proficiency are especially worrisome. A recent study by Steiss et al. (2022) found that English learners, compared to their English-fluent peers, had significantly lower performance in using evidence, developing and structuring their ideas, and using language to convey their ideas in their argument writing. Without an effective instructional approach that attends to developing proficiency in argument writing, the gaps and disparity in English learners’ argument writing will likely persist. As Kanno and Cromley (2015) point out, academic preparation in high school is a major predictor of college success, yet a lack of preparation in academic and argument writing is pervasive among English learners. Not only are they less likely than their English-fluent peers to graduate from high school, they also enroll in college and graduate from college at far lower rates (Nuñez et al., 2016). Further, studies have shown that academic writing is “the linguistic challenge that plagues” English learners the most at the college level (Kanno & Cromley, 2015). Because writing is a gatekeeper for college access and persistence and a “threshold skill” for hiring and promotion for salaried workers (National Commission on Writing for America’s Families, Schools, and Colleges, 2004), failure to close these opportunity gaps in academic and argument writing will have serious social and economic consequences.
Currently, English learners in the United States constitute approximately 10.4% of the total K-12 population (NCES, 2022). Although English learners in the United States speak more than 350 languages, 73% speak Spanish as their first language (Batalova & McHugh, 2010), 40% have origins in Mexico (Hernandez et al., 2008), and 60% of English learners in Grades 6 through 12 come from low-income families (Batalova et al., 2007). As English learner enrollments have increased in U.S. public schools, researchers and policymakers have highlighted large literacy gaps based on students’ English language proficiency. English learners in the United States are referred to by many different terms, including “English language learners,” “limited English proficient students,” “language-minority students,” “second language learners,” and “multilingual learners.”
English learners’ exposure to and experiences of language and literacy development is variable across the nation, with some who are “stuck” in designated English language development programs while others are in mainstream classes (Olsen, 2010). A common thread, however, is that these students are not provided with adequate support and ample opportunities to develop their academic literacy and language skills. These students are often classified as “lower achievers” and receive instruction that places a premium on the “transmission of information, providing very little room for the exploration of ideas, which is necessary for the development of deeper understanding” (Applebee et al., 2003, p. 689). Consequently, they are often limited in their knowledge of academic registers and literacy as many of them have been given “elementary school curricula and materials” in segregated program models rather than being provided with an opportunity to engage in comprehensive, literacy-rich curricula (Olsen, 2010, p. 2). While many of these students are mainstreamed into regular English language arts classrooms, they often lack additional support to develop their language and literary skills.
Effective Instructional Practices for English Learners
Given the many demands of academic and argument writing and the few opportunities to practice, English learners need access to high-quality curricula, explicit instruction, and ongoing support as they strive to become college and career ready. Current literature based on research and practice calls for contextualized, literacy-rich activities. These activities should be integrated into curricula that focus on developing the higher-level interpretive and analytical aspects of writing. For instance, Olsen (2010) recommended a comprehensive program that integrates “instruction in the academic uses of English, high-quality writing, extensive reading of relevant texts” (p. 33). Similarly, Walqui and Bunch (2019) argue for the amplification of the curriculum, rather than reduction and simplification, and emphasize the importance of enacting “stimulating, demanding, well-supported lessons to transform what is currently offered to many English learners” (p. 21).
One approach that has been empirically proven effective is comprehensive, cognitive strategies instruction that focuses on strategy instruction, interpretation, and knowledge transformation alongside the development of language skills necessary to express higher-level thinking effectively (Kim et al., 2011; Olson et al., 2012, 2017, 2020; Olson & Land, 2007; 2008). Numerous reports from policy centers and blue-ribbon panels “implicate poor understanding of cognitive strategies as the primary reason why adolescents struggle with reading and writing” (Conley, 2008, p. 84). Further, research conducted over the past 15 years on the content of college courses and instructor expectations indicates that cognitive strategy use is the key to college and career readiness (Conley, 2013).
Recent instructional frameworks and recommendations also support approaches that incorporate strategy instruction to advance English learners’ development of English (Calderón et al., 2011; Francis et al., 2006; Goldenberg, 2008; Schleppegrell, 2009). Short and Fitzsimmons (2007) hypothesize that strategy instruction is especially effective for English learners because it provides them with an explicit focus on language, increases their exposure to academic texts, makes the texts they read comprehensible, gives them multiple opportunities to affirm or correct their understanding and use of language, assists them in retrieving and using new language features for academic purposes, and provides them with the means of learning language on their own, outside of class. They further hypothesize that adolescent English learners of an intermediate level of English proficiency benefit from strategy instruction because they possess the language proficiency required to use cognitive strategies and engage in higher order cognitive reading and writing tasks (Echevarria et al., 2012). In short, explicitly teaching strategic reading and writing behaviors to English learners can help them engage with complex texts and convey those interpretations in well-reasoned essays to meet the CCSS-ELA (Bunch et al., 2012; Francis et al., 2006; Goldenberg, 2008).
The Present Study and RQs
Though prior research has demonstrated the efficacy of instructional practices that use cognitive strategies using broad literacy outcomes, we also see value in examining how discrete components of student writing improved through the use of a cognitive strategies approach. This is particularly important given the unique challenges English learners experience as writers. In this study, we use a more analytical lens to examine students’ pretest and posttest writing assessments from a previous randomized controlled field trial (Olson et al., 2020) that aimed to improve student writing. The following research questions guide this study:
What are the specific challenges that English learners must address when composing literary arguments?
How does the Pathway to Academic Success intervention program impact student growth in writing controlling for students’ language status?
Introduction to the Pathway to Academic Success Cognitive Strategies Intervention
Developed by a university-based site of the National Writing Project, Pathway supports teachers by providing comprehensive curriculum materials and evidence-based instructional practices designed to prepare all students, but specifically mainstreamed English learners, many of whom are classified as long-term English learners, in high-need schools to (1) develop proficiency in interpretive reading and text-based argument writing, (2) meet their state-adopted English language arts standards, and (3) graduate from high school and become college-bound and career ready. The goal of Pathway is to address the literacy and opportunity gaps (Olson et al., 2017) between English learners and their native English-speaking peers in the areas of argument writing and academic literacy. The treatment is an intensive 46-hour PD program (via six full-day meetings interspersed throughout the school year and five 2-hour after-school sessions) in which secondary teachers learn how to integrate cognitive strategy instruction into analytical reading and writing instruction. Teachers in the study were observed three times during the academic year by trained literacy coaches who completed a fidelity protocol and also provided teachers with in-person and written feedback on the observations.
Pathway’s Efficacy: Results from Previous Studies
Pathway began with an 8-year quasi-experimental longitudinal study in a large, urban district (98% Latinx, 84% Free and Reduced Price Lunch, 88% mainstreamed English learners) that yielded an average effect size of .34 across the eight years of implementation (Olson & Land, 2007). The project then received Institute of Education Sciences (IES) funding to conduct a Goal 3 Efficacy cluster randomized controlled trial in the same district. Year 1 of that RCT (Kim et al., 2011) yielded an effect size of .35. Year 2 of the RCT yielded an effect size of .67 (Olson et al., 2012). In both years of the study, there were statistically significant effects on the writing subtest of the California Standards Test (d = .10). The project applied for and received an Office of English Language Acquisition (OELA) grant to conduct an RCT to replicate the project in a neighboring school district. This study also yielded significant and positive results (Olson et al., 2017—Year 1, d = .48; Year 2, d = .60). Tenth-grade English learners in the treatment group in Year 2 also passed the California High School Exit Exam (CAHSEE) at 20 percentage points higher than the state pass rate (treatment = 57.9%; State = 38%). This strong record of statistically significant results positioned the project for an i3 Validation/Expansion award that funded a scale-up study. In this study, the intervention was expanded to include three sites of the National Writing Project in California, all with large urban partner districts. Results were positive and statistically significant for not only the holistic score (d = .32), but all four of the analytic scores: content (d = .31), structure (d = .29), fluency (d = .27), and conventions (d = .32) (Olson et al., 2020). To date, five studies that were conducted to test the efficacy of Pathway have received approval from the IES What Works Clearinghouse, one without reservations (Kim et al., 2011) and four with reservations (Olson & Land, 2008; Olson et al., 2012, 2017, 2020).
A Description of the Pathway Cognitive Strategies Approach and Intervention Strategies
A key component of Pathway is the use of cognitive strategies to develop students’ argument writing skills, improve their academic language and literacy, and help them move from “knowledge telling” to “knowledge transformation” (Bereiter & Scardamalia, 1987). The use of cognitive strategy instruction is grounded in a wide body of research on what experienced readers and writers do when they construct meaning from and with texts. Countless studies demonstrate the efficacy of cognitive strategy use in reading (Block & Pressley, 2002; Paris et al., 1991; Tierney & Pearson, 1983; Tierney & Shanahan, 1991). Similarly, Graham and Perin (2007) indicate that strategy instruction is the most effective of 11 key elements of writing instruction (d = .82) for all students, and particularly for students who find writing challenging.
Drawing from cognitive, sociocognitive, and sociocultural theory (Englert et al., 2006; Flower & Hayes, 1981; Langer, 1991; Tierney & Pearson, 1983), the Pathway intervention focuses on cognitive strategy use as a vehicle for higher-level thinking, uses an apprentice model where the teacher serves as a senior member of a learning community, provides a wide array of mental, linguistic, and physical tools to promote cognitive strategy use, and promotes collaboration among teachers, between teachers and students, and among students. The theory of action underlying the Project is that research-based guidance delivered via various curriculum materials is delivered as outputs over two full years and implemented by treatment teachers.
This focused pedagogy results in increased instructional time on writing, teacher knowledge of evidence-based practices, expertise in teaching reading and writing strategies as well as more frequent explicit instruction in strategies for reading and writing, modeling, guided practice, and more frequent revisions of argument writing. An intermediate outcome includes increased student text-based argument writing ability. Ultimately, the long-term outcomes are increased English language arts test scores for all students (Olson et al., 2012), but especially high-need students, including English learners, and increased high school graduation rates for those students (Olson et al., 2017). Figure 1 shows major components of Pathway that have been shown to help students, especially English learners, improve their argument writing and literacy skills. A more detailed description of the key strategies of Pathway’s intervention follows.
Research underlying the key components of Pathway.
Cognitive Strategies Approach and Tool Kit
Strategy instruction in Pathway occurred within the context of teaching reading and writing as a process and involved prereading, during reading, and postreading activities as well as prewriting, planning, drafting, sharing, revising, and editing activities. First, teachers were introduced to a model of the cognitive strategies that make up a readers’ and writers’ tool kit. Teachers used the following analogy to explain the tool kit to students:
When we read, we have thinking tools or cognitive strategies inside our heads that we access to construct meaning. Researchers say that when we read, we’re composing, just as when we write. So, when you think of yourself as a reader or writer, think of yourself as a craftsman, but instead of reaching into a metal tool kit for a hammer or a screwdriver to construct or build tangible or real objects you can actually see, you’re reaching into your mental tool kit to construct meaning from or with words.
In support of the instruction based on the cognitive strategies approach, Palincsar and Brown (1987) noted that explicit instruction of the strategies that tap into students’ metacognition to assist them “to identify and enlist strategies to promote and monitor learning” should be “an integral part of teaching activity” (p. 73). Such instruction can make the invisible thinking process visible.
Cognitive Strategies Bookmarks
Students also received laminated bookmarks in English and Spanish that illustrate what goes on in the mind of a reader or writer during the act of meaning construction. For example, a sentence starter for revising meaning is “At first I thought, but now I. . .” and a starter for reflecting and relating is “This is relevant to my life because. . . .” Students used these bookmarks to annotate texts and to write metacognitive reflections about their processes of meaning construction as readers and writers.
DO/WHAT Chart
To help students navigate how to respond to a prompt, they were taught a planning and goal-setting strategy that involved the creation of a DO/WHAT Chart, which enabled them to deconstruct a prompt and create a roadmap for composing. To complete a DO/WHAT chart, students used green and blue highlighters to mark all of the verbs in the prompt which instructed a student to do something in green and underline the task words that tell the student what to do in blue. For example, they had to write (green) an essay (blue) and make (green) a claim (blue) about the main point, lesson, or message of the text. They then transferred those words onto a T-chart below the prompt. This activity helped the students to clarify and visualize what is expected, plan and goal set, organize information, and evaluate the criteria for a successful response to the prompt.
HoT-SC-T
Students were taught a planning and organizing information strategy called HoT-SC-T (hook, TAG, summary statement/conflict, and thesis) to help them write their introductions. Although this strategy might be perceived as too formulaic for advanced writers, novice essay writers need to be exposed to form-making before engaging in form-breaking. Students were also introduced to transition words to develop coherence and use as a bridge to their main body and conclusion and practiced inserting appropriate transition words into cloze exercises to become familiar with the use and position of words like in addition, however, nevertheless, whereas, on the other hand, and so forth.
Dear Student Feedback Letter
The centerpiece of the Project focuses on having students revise their pretest essay into a multiple draft essay. Pretest essays were read and responded to by graduate students and teachers affiliated with the National Writing Project after 3 hours of training that stressed giving writers a “glow” before providing the “grow” feedback such as clearly stating a claim, using evidence, providing commentary linking the evidence to the claim, etc. Students consulted these letters as their teachers guided them through the revision process.
Color Coding for Revision
Students were taught a color-coding strategy to make their thinking visible in their pretest on-demand essays. Teachers first designated three colors for the types of assertions that comprise a literary argument and said the following:
Plot summary reiterates what is obvious and known in a text. Plot summary is yellow because it’s like the sun. It makes things as plain as day. Commentary is blue like the ocean because the writer goes beneath the surface of things to look at the deeper meaning to offer opinions, interpretations, insights, and “Ah-Ha’s.” Supporting detail is green because like the color, it brings together the facts of the text (yellow) with your interpretation of it (blue). It’s your evidence to support your claims, including quotations from the text.
After modeling the color-coding process with both less and more effective essay samples, students color-coded their own pretests with a partner to determine what aspects of their writing should be improved during revision.
Language Use Mini-Lessons
Students were introduced to a variety of mini-lessons to increase their awareness and command of language use. These included converting informal/conversational English to more academic written English to address using a register suited to a formal audience; sentence variety activities based on Noden’s (2011) Image Grammar; practice quoting from the text, using reporting verbs, and punctuating appropriately; and sentence command activities such as “Yes twice, comma splice,” a strategy to detect comma splice run-on sentences.
These intervention strategies were introduced and then returned to throughout the year to provide students multiple opportunities to practice and internalize them.
Description of Business-as-Usual Professional Development Activities
Control teachers in all of the RCTs conducted business as usual using the district English language arts textbook and core novels for teaching. Both treatment and control teachers attended PD sessions held by the districts for ELA teachers. Several districts also conducted PD on district benchmark assessments and prepared students to take standardized tests.
Methodology
Participants and Writing Corpus
Data used were 398 essays written by 199 students (49% female) in Grade 7-12 (ranging from 29 to 39 students in each grade) that were a subset of students in a larger writing intervention study Olson et al., 2020). Essays come from 56 English Language Arts classrooms in nine schools that were part of a large, urban public school district in the southwest region of the United States. The school district uses state language classification based on the California English Language Development Test (CELDT). The designation EL is given to students who score at the 50th percentile or below on CELDT, while RFEP students are reclassified from English learner status to fluent English proficient status based on four criteria: subsequent English language assessment, teacher evaluation, parental recommendation, and strong academic performance (California Department of Education, 2022). Each student’s language status was provided by the district at the beginning of the larger study (Olson et al., 2020). Students with RFEP designation were reclassified as fluent English proficient prior to the study. In the present study, students designated as English learner (EL) and RFEP are considered as English learners. Both groups are considered to be developing proficiency in English and in need of cognitive strategy instruction. During the data collection year, the percentages of students by language status were the following in the district: 20% ELs (English Learners), 8% RFEP (Reclassified Fluent English Proficient), 47% IFEP (Initial Fluent English Proficient), and 25% EO (English Only). The students and their families had consented to participation and all guidelines for human research protection continued to be followed in the present study.
Of the 398 essays used in the present study, 199 of the essays are pretests written by students before they were randomly assigned to experimental conditions. The other 199 essays are posttests taken at the end of the academic year, with half of these written by students in the experimental conditions and half by students in the control condition. Between pre- and posttest, students in the experimental condition received Pathway curriculum as described in the previous section, whereas students in the control conditions received “business as usual” instruction, following district pacing guides and instruction. To avoid crossover effects, teachers were instructed not to share Pathway instruction or materials with control teachers, and a previous study of the intervention found no evidence of crossover effects (Olson et al., 2020).
Stratified Random Sampling Procedure
Because we examine the unique challenges faced by English learners (students with EL and RFEP designations), we used stratified random sampling to ensure an adequate number of students designated as EL or RFEP were included in the data corpus. In this study, 32% of the students were designated as EO, 12% were designated as IFEP, 25% were designated as RFEP, and 28% of the students were designated as EL. For all subsequent analyses, we combine students who are designated as EO and IFEP into a single group as both designations include students who are initially fluent in English. These students perform similarly in language-based assessments given their proficiency in English (Kim et al., 2011). This group of students is used as a comparison group for their EL and RFEP peers. Additionally, the racial/ethnic composition of the sample students is as follows: 72% Hispanic, 13% Caucasian, 2% African American, and 2% Asian.
Measures
Writing task
In the previous study (Olson et al., 2020), both treatment and control teachers administered pre- and posttests to students. At the pretest (October), students read the text and wrote an essay in response to one of two text-based analytical writing prompts (see Appendix A), which were administered across two 50-minute class periods. At posttest (May), students wrote to the other prompt. By counterbalancing the two writing tasks across both treatment and control groups, we control for order effects and account for writing growth due to Pathway treatment status, not the assignment of a specific writing prompt. Each prompt asked students to read a text and to write an essay that analyzed the text based on a central idea or theme. The prompt required students to make a claim about the author’s message, analyze the author’s use of figurative language, and discuss the author’s purpose. This type of text-based analytical writing is an argument of interpretive analysis (Smith et al., 2012) or literary judgment (Hillocks, 2011).
Analytic coding
In a prior set of studies, Pathways was established as effective in improving students’ holistic writing (Kim et al., 2011; Olson & Land, 2008; Olson et al., 2012, 2017, 2020). In this article, we take a closer look at students’ writing and examine rhetorical and linguistic elements that the intervention improves. To answer the research questions, we developed an analytic framework to independently measure the quality of discrete components of text-based argument writing that together comprise holistic quality. For example, evaluators using the NWP-AWC analytic rubric (National Writing Project, 2010) give scores on the “Content” of writing by weighing multiple subcomponents of writing’s content, including the presentation of a thesis and the quality of the analysis of textual evidence. In this study, we use separate items for each of these subcomponents and assign each item a unique score (e.g., “How well does the writing present a claim that addresses the prompt?”).
To create a reliable and valid framework, we first generated a list of items reflecting holistic writing quality in this genre. The research team drew on extant writing rubrics (National Writing Project, 2010), literature (De La Paz et al., 2012; MacArthur, 2019), and input from subject matter experts in generating items. These items were continuous and were scored on a scale of 1-7, with a 1 indicating “not evident” and a 7 indicating “highly effective.” Using this scale, we code not only for the existence of these features, but also the quality of these features in advancing an argument of literary analysis.
After the independent generation of items, the research team assessed the representativeness of items, before taking steps to reduce items that were redundant or inapplicable. Next, we applied prototypical analytic frameworks to a sample of student papers. After several iterations of testing the framework, generating items, and reducing items, the tool was shared with 10 subject matter experts in the field of secondary writing research who provided critique and feedback on the analytic framework, especially as it relates to content validity (Anastasi & Urbina, 1997). Subject matter experts include Tanya Baker, the director of national programs for the National Writing Project, multiple authors of the IES Educator’s Practice Guide for Teaching Secondary Students to Write Effectively (Graham et al., 2016; https://ies.ed.gov/ncee/wwc/practiceguide/22), researchers with experience teaching and evaluating secondary writing, researchers with specific expertise in the writing of English learners, and researchers with peer-reviewed articles relating to the measurement of writing in secondary schools.
Approximately 15% of the essays (n = 64) were double-coded (Gallagher et al., 2017). Interrater agreement rates were calculated for each item of the framework; agreement within a score point (on a 7-point scale) was considered acceptable (Gallagher et al., 2017; MacArthur, 2019). The average agreement within 1 point for all items was 90%, and all agreement rates were above 75%. Raters include both professors with expertise in writing evaluation and Graduate Student Researchers. All raters engaged in multiple rounds of coding together, resolving discrepancies between raters and making detailed notes in a working codebook to ensure the criteria for scoring were clear, valid, and reliably administered. A description of items used in the analytic rubric can be found in Appendix B.
Data-Analytic Strategies
Our first research question aims to identify the challenges English learners must address when composing arguments of literary analysis. To answer this research question, we used a text-analytic approach (Sanders & Schilperoord, 2006) on a selected sample (n = 109) of pretest essays written by students designated as EL and RFEP, since this research question focuses on the specific challenges of students who are still in process of developing their English language proficiency. The text-analytic approach involves manually coding each sentence of a text for its syntactic form and rhetorical function. Two raters (one of whom is the second author) with expertise in text analysis and extensive experience of teaching university composition courses double-coded all sentences in 23% of the essays. The raters had 92.8% agreement in coding sentences. The second author manually coded each sentence for the remaining essays.
In using a text-analytic approach, we focus on both rhetorical and linguistic features of the texts, drawing on rhetorical structure theory (RST; Mann & Thompson, 1988) and the systemic functional linguistic (SFL) approach (Halliday, 1994). While RST centers on the macro-level features of the text including text structure, organization, and rhetorical relations such as restatement and summary, interpretation and evaluation, and integration of evidence, SFL deals with micro-level linguistic features based on the idea that certain patterns of language use within a specific discursive context can be detected through a systematic approach and functional analysis of text construction. Since we intended to identify rhetorical and linguistic challenges English learners encounter, we systematically analyzed each text in the selected sample using a text-analytic approach to answer our first research question.
To answer our second research question regarding Pathway’s impact on rhetorical and linguistic aspects of writing controlling for language status, we first present descriptive statistics showing growth on specific items in the analytic framework for students designated EL and RFEP. Then, multiple hierarchical regression analyses were conducted with the full writing sample to examine the effects of treatment on the specific components of writing as measured in our analytic framework controlling for EL status and pretest scores (Acock, 2008; Olson et al., 2017). A series of hierarchical regression models were run for each item in the analytic framework (e.g., How well does the student use source material as evidence?”). In each series of regressions, Step 1 used treatment condition as a predictor of students’ posttest writing score. Step 2 included a variable indicating EL or RFEP classification in order to examine differences between these students and their EO/IFEP peers at posttest. Finally, a third step (Step 3) controlled for a student’s pretest scores, given the lower average performance of ELs and RFEPs on the pretest measures and language status. By controlling for pretest scores, we show how much ELs and RFEPs gained relative to their peers because of being in the treatment condition. Racial/ethnic backgrounds and gender were not included because of the low sample size, and preliminary analyses showed no significant differences in outcomes.
Results
Challenges ELs Encounter When Composing Literary Arguments
The results of the systematic analysis of pretest essays (n = 109) revealed that English learners (students with EL and RFEP designations) have considerable challenges with both higher-order tasks involved in writing arguments of literary analysis and the linguistic demands of academic writing that requires students to analyze, interpret, synthesize, and explain complex ideas. These results are consistent with and correspond to what Shaughnessy (1997) contends in Errors and Expectations that developing writers’ unfamiliarity with certain features of academic writing may interfere with and impinge on their writing quality. This, particularly, may be the case for students who have not learned the genre- and discipline-specific conventions of argument writing and who have not internalized the language features of the formal codes that govern academic writing. In what follows, we discuss the rhetorical and linguistic demands of writing arguments of literary analysis, as well as the challenges students face in both areas.
Complex Rhetorical Problem Solving
Analytical writing demands logical modes of argument and organization that are different from narrative and summary writing; students need to move beyond what Bereiter and Scardamalia (1987) referred to as “knowledge telling,” which relies primarily on retelling or summary. Rather, they need to compose at an advanced level of “knowledge transformation” (Bereiter & Scardamalia, 1987) as they analyze, argue, interpret, and use evidence to support ideas. As Bereiter and Scardamalia (1987) pointed out, inexperienced writers often rely on knowledge telling or a simplified idea generation and summarization process while experienced writers go beyond mere retelling and engage in knowledge transformation that involves complex problem solving and analytical skills. However, the analysis of our pretest data points to several challenges students must address when composing arguments of literary analysis. We identified common challenges in three broader macro-level rhetorical features: (1) presentation of content and ideas, (2) organizational elements of argument-based writing, and (3) the use of evidence and analysis.
Presentation of ideas
Students were particularly challenged in presenting content that aligns with the genre-specific features of text-based argument writing. Although the writing prompt was developed to direct students’ thinking to analysis and argument construction, students had difficulty addressing all aspects of the prompt, formulating clear and compelling claims, and focusing on substantiating an interpretable claim. For example, only 6% of the selected sample of pretest essays addressed all aspects of the prompt. Forty-nine percent of the essays partially addressed the prompt, discussing some aspects of the prompt but neglecting other important tasks specified in the prompt. Thirty-nine percent of the essays minimally addressed the prompt, discussing only one or two aspects of the prompt without going in depth, and 6% did not address the prompt at all. The majority of the essays that partially and minimally addressed the prompt exhibited superficial treatment of elements they addressed and showed difficulty in both presenting a compelling claim and sustaining focus in substantiating the claims.
In analyzing the claims made about the theme of the literary nonfiction text, we found that students often confused the theme with character, plot, or topic. The two examples from student essays are presented below to show a contrast between a claim that only identifies a topic and a claim that presents a theme.
Claim 1: In the article “The Earth Is Cruel” by Leonard Pitts, the message is about
Claim 2: Leonard Pitt’s main point and the theme of his article is that
While students could identify the specific topic (e.g., vulnerability, courage, sacrifice) related to the text, they had difficulty formulating a theme statement that presents an interpretable claim that is an important part of constructing an argument of literary analysis.
Organization of literary arguments
The analysis of the pretest sample indicated challenges in making effective and purposeful rhetorical moves to introduce the topic (introduction), develop the argument (body), and provide closure (conclusion). In the area of organizing the arguments of literary analysis, we found two rhetorical features—paragraphing and logical modes of argument development—that need to be addressed.
In terms of paragraphing, students had difficulty with both paragraphing based on idea units and employing rhetorical moves to develop ideas within paragraphs. In the selected sample, the majority of the essays (61%) consisted of one or two paragraphs with no clear division of paragraphs based on ideas and text features including introduction, body, and conclusion. Even among the essays that have several paragraphs (39% of the essays), the idea units within paragraphs were not often distinct and clear, showing weak command of organization and argument development structure. Only 15% of the essays in the selected sample were rated as having effective organization—how well the writing is globally organized.
Besides paragraphing, students also had a challenge in following logical modes of argument development. We found a general tendency to rely on summary formats even though the prompt explicitly asked students to analyze by making a claim about the theme of the literary nonfiction text and supporting their claim with evidence from the source text. There were three distinct patterns that showed students’ reliance on summary format. The most common pattern was a pure summary of the text with no theme statement and explication or interpretations (42% of the essays). The second pattern was a brief statement of a theme followed by a general summary (19% of the essays). In this pattern, essays started with a theme statement. However, instead of developing the theme statement with reasoning and evidence from the source, students often resorted to generally summarizing the text without a connection to the theme. The third pattern was a general summary of the article followed by a short theme statement (16% of the essays). Essays that fall under this pattern started with a general summary of the article and ended with a short (often 1-3 sentences) statement of either the article’s theme or author’s message/purpose, again without a clear link between the two moves.
Use of evidence and analysis
In the area of evidence use, our analysis of pretest essays pointed to challenges in integrating evidence from the source in their own writing and balancing purposeful summary, textual evidence, and commentary. High-quality student texts exhibited more sophisticated use of source material with purposeful summary and paraphrasing and quotations that are both embedded into students’ own sentences and integrated with introductory phrases and reporting verbs. However, essays that received low scores on analytic coding in the category of students’ use of source material as evidence (73% of the essays) featured verbatim-copied sentences, floating or stand-alone quotations, and infrequent paraphrasing.
Language Demands of Academic Writing and Linguistic Constraints
Our analysis of pretest data indicated that English learners experience considerable linguistic challenges when composing arguments of literary analysis. These challenges point to the need for students to develop not only lexical capacity and knowledge but also knowledge of sentence structures, types, and functions.
Lexical challenges
The results of the systematic analysis indicated that the texts students produced prior to the intervention exhibited less lexical diversity, more repetition of content and function words, misuse of academic expressions, a lower percentage of words from the Academic Word List (AWL; Coxhead, 2000) and frequent use of informal diction. These lexical challenges were captured in the results of analytic coding, with the majority of the pretest essays (77%) scoring 3 and below in command of diction and word choice. The following excerpt from student writing demonstrates several issues, including dependence on basic and frequently used words, informal diction, and repetitive use of content words (people, jumping, saving).
People where actually jumping to save people because those people that were in the water were drowning and people in helicopters and guys with suits were jumping and saving the people.
In the example above, 83.3% of the words fall into the K1 word category, which indicates the first 1,000 most frequently used words based on the General Service List (GSL; West, 1953). Out of these words, 53% are function words and 30% are content words. The word frequency measures have been used to analyze lexical sophistication based on the notion that less proficient learners use more frequent words. None of the words in the sentence represent words covered by AWL, indicating a low capacity to use academic words. The example also shows the use of an informal word (guys). In addition, unnecessary repetitions of words (in this case, the word people used 5 times in a single sentence) not only lead to less lexical diversity but signal the writers’ lack of lexico-grammatical awareness. These features were pervasive in the pretest essays we analyzed.
Syntactic challenges
As an academic writing genre, argument writing demands complexity and variety in sentence structure and conventionalized clause organization to show complex relationships between ideas. Our pretest data pointed to several challenges students face in areas related to sentence structure, flow, and variety.
The pretest essays exhibited patterns of unconventional sentences (run-ons, fragments, and faulty sentences that had semantic and structural problems), reliance on simple sentences and simpler noun clause structures, repeated use of basic and commonly used coordinating (but, and, so) and subordinating (because, when, that) conjunctions, underuse of varied and sophisticated subordination and clause embedding that are common in academic writing. Knowledge of syntax allows students to avoid sentence boundary issues and choose between a variety of structures to convey complex ideas and construct coherent texts. Our analytic coding included two subcategories related to syntactic features: (a) how the essay demonstrates sentence fluency and flow and (b) how the essay demonstrates syntactic variety and style. Based on the results of the analytic coding, 74% of the essays in the selected sample were scored 3 and below in the subcategory of sentence fluency and flow. In syntactic variety and style, 91% of the essays received a score of 3 and below, while 5 was the highest score given (5% of the essays).
Putting it all together, we summarize the key findings of the specific challenges English learners must address when composing arguments of literary analysis (Table 1). In general, the pretest essays written by students designated as ELs and RFEPs in our sample exhibited difficulties with both higher-order tasks involved in writing arguments of literary analysis and linguistic demands of argument writing that require students to analyze, interpret, synthesize, and explain complex ideas.
Specific Challenges English Learners Must Address When Composing Arguments of Literary Analysis.
Addressing the challenges English learners have when engaging in a complex argument writing task is an important pedagogical goal. Given these challenges, we now turn to the efficacy of Pathway’s writing intervention and discuss how the intervention impacted the specific components of writing.
The Impact of Pathway on Rhetorical and Linguistic Elements
To answer our second research question, we first examined descriptive statistics that show differences between treatment and control students classified as ELs and RFEPs. We then conducted multiple hierarchical regression analyses to examine the effects of treatment on writing quality and specific components of writing as measured in our analytic framework. The following sections show the effect of treatment on specific components of writing controlling for language status and students’ pretest. First though, Table 2 shows students at pretest.
Scores on Pretest Analytic Items by Language Status.
Note. Ninety students were classified as English Only (EO) or Initially Fluent English Proficient (IFEP), 57 students were classified as English Language Learner (EL), and 52 students were classified as Reclassified Fluent English Proficient (RFEP).
Overall, RFEP students performed lower than their EO/IFEP peers, and EL-classified students scored even lower, on average, across all items in the analytic framework. This performance gap is observed widely in research. Rather than focusing on this gap, presently we describe student growth by EL status when receiving high-quality instruction.
Figure 2 shows how students classified as EL improved from pretest to posttest on specific components of writing.
Mean score improvement for English learners on writing components.
The figure shows gains in writing components from 0.41 points (Use of appropriate tone) to 1.63 points (Present a clear and compelling claim) measured on a 7-point scale. The improvement of treatment students in components related to the presentation of ideas, organization, and the use of evidence is significant and reflects comprehensive growth in text-based analytical writing. These items reflect Pathway’s impact in moving students from knowledge-telling to knowledge-transformation and developing students’ capacities to make and support interpretive claims with evidence and reasoning. However, less improvement is seen for students in both treatment and control for sentence fluency, command of conventions, and use of appropriate tone.
Figure 3 shows how students designated as RFEP improved from pretest to posttest on specific components of writing.
Students designated as RFEP show improvement in many components of writing over the year, with treatment students improving more, on average, than control students except for organization. RFEPs in the treatment condition improved significantly on items related to the presentation of ideas, organization, and use of evidence, similar to the growth exhibited by their English learner peers. Looking closely at skills related to evidence use, Figure 3 shows that treatment students improved by 1.67 points in the use of source material as evidence, 1.52 points on the use of commentary to interpret source material, and 1.7 points in the balance of evidence, purposeful summary, commentary, and summary. Given that literary analysis requires students to analyze and interpret texts and develop complex ideas, these results show notable improvement in this genre.
Mean score improvement for Reclassified English learners on writing components.
These improvements will be further examined in the following sections, which compare each subgroup’s gains in the rhetorical and linguistic writing components that we explored in this study. We also include outcomes for students designated IFEP and EO as a reference group, to contextualize the findings of EL and REFP students relative to their peers in the intervention.
Effect of Treatment on Presentation of Ideas
Students in the treatment condition scored significantly higher than students in the control condition for all items related to the presentation of ideas in their writing. For example, students in the treatment group scored 0.84 points higher, on average (on a 7-point scale), for the presentation of a clear and compelling claim. There were no significant differences on these items between students designated RFEP and students designated EO/IFEP. Students designated as EL did score significantly lower than average compared to students designated EO/IFEP. For example, Table 3 shows that being in the treatment condition predicted scoring 0.54 points higher on addressing all aspects of the prompt and being an EL predicted scoring 1.05 points lower than average. This means ELs in the treatment condition still underperformed relative to their EO/IFEP peers in the control condition on this item. This relation still held even after controlling for pretest performance for addressing all aspects of the prompt and presenting a clear and compelling claim.
Effect of Treatment on Presentation of Ideas at Posttest Controlling for EL Status and Pretest (n = 199).
Note. All regression coefficients presented are nonstandardized (scale 1-7). EL = English Language Learner; RFEP = Reclassified Fluent English Proficient. Students designated English Only (EO) or Initially Fluent English Proficient (IFEP) are used as the control groups for ELs and RFEPs. Standard errors are in parentheses.
p < .05.
In the following tables, we show the effect of the Pathway treatment on each component in the analytic framework using multiple hierarchical regression. Step 1 shows the effect of the treatment on the writing component (e.g., “Address prompt”). The next two models show the effect of the treatment controlling for EL status (Step 2) and EL status and pretest scores (Step 3), respectively. For all models, nonstandard coefficients are reported and the R-squared shows how each step changes the explained variation in posttest scores. Overall, models featuring EL status and pretest scores as controls better explained writing outcomes (see Table 3).
Effect of treatment on organization
Students in the treatment condition scored significantly higher than students in the control condition for all items related to the organization of their writing. Similar to the presentation of ideas, there were no significant differences on these items between students designated RFEP and students designated EO/IFEP, but ELs performed worse than EOs/IFEPs on organization, presentation of an introduction, presentation of a conclusion, and number of paragraphs. Moreover, some of the gains due to being in the treatment were significant enough to bring the average EL student close to parity or beyond their EO/IFEP peers in the control condition at posttest. In fact, this was true for all items except the conclusion. Though students in the treatment group performed 0.49 points higher than the control group controlling for pretest in their conclusions, EL students performed 0.92 points lower than EO/IFEP students controlling for pretest scores (see Table 4 for more detail).
Effect of Treatment on Organization at Posttest Controlling for EL Status and Pretest (n = 199).
Note. All regression coefficients presented are nonstandardized (scale 1-7). EL = English Language Learner; RFEP = Reclassified Fluent English Proficient. Students designated English Only (EO) or Initially Fluent English Proficient (IFEP) are used as the control groups for ELs and RFEPs. Standard errors are in parentheses.
p < .05.
Effect of treatment on evidence use
On the posttest, students in the treatment condition scored higher on the use of evidence as source material (0.64); the use of commentary to interpret and analyze source material (0.55); and the balance of evidence, commentary, and summary (0.62). The regression models also showed a significant underperformance of ELs relative to their EO/IFEP peers. For example, ELs performed -1.47 points lower on the use of commentary compared with EO/IFEP students. After controlling for pretest scores and the relatively lower starting point for ELs in these discrete components of writing, being an EL in the treatment condition still predicted a −1.01 lower score on commentary compared with EOs and IFEPs in the treatment. The model also shows these EL students would score lower in commentary than EOs/IFEPs in the control condition. A similar pattern is observed for evidence use and balance, though ELs made relatively more gains in these areas. See Table 5 for more details.
Effect of Treatment on Evidence Use at Posttest Controlling for EL Status and Pretest (n = 199).
Note. All regression coefficients presented are nonstandardized (scale 1-7). EL = English language learner; RFEP = reclassified fluent English proficient. Students designated English Only (EO) or Initially Fluent English Proficient (IFEP) are used as the control groups for ELs and RFEPs. Standard errors are in parentheses.
p < .05.
Effect of treatment on language use
Findings related to the effects of treatment on language use were more varied and complex than previous components of writing. For example, there was no significant difference between treatment and control groups for the use of conventions on posttest. Differences between ELs and EOs/IFEPs were significant across all language use items. Looking specifically at the use of diction or word choice, ELs performed −1.03 points lower than their peers. However, after controlling for the pretest, there was no significant effect, suggesting the lower performance of ELs on diction is mostly attributable to lower scores on the pretest, on average. This result is similar for syntactic variety and style and use of academic tone but is not the case for sentence fluency, with the underperformance of ELs persisting even after controlling for pretest (see Table 6).
Effect of Treatment on Language Use at Posttest Controlling for EL Status and Pretest (n = 199).
Note. All regression coefficients presented are nonstandardized (scale 1-7). EL = English language learner; RFEP = reclassified fluent English proficient. Students designated English Only (EO) or Initially Fluent English Proficient (IFEP) are used as the control groups for ELs and RFEPs. Standard errors are in parentheses.
P < .05.
Discussion
Argumentation is a necessary skill not only for college and career success but also for democratic and civic participation. The complexity of argument writing and the shifting instructional focus toward argument literacy in the CCSS, and other state standards adopted since 2010, have placed greater responsibility on teachers to find effective ways to support all students, but especially English learners, in developing argumentation skills. An important goal of Pathway is to support secondary teachers in effectively addressing the challenges English learners encounter when writing text-based arguments.
This study aimed to identify challenges that English learners from two subgroups must address when composing literary arguments and to examine specific writing components that the Pathway program had the most impact on. We now discuss findings for both research questions jointly, examining the challenges various English learners face and the extent to which the students were able to address these challenges through the intervention. Based on decades of experience as project developers and the reports of teachers and students who participated in the intervention, we also present some hypotheses regarding which intervention strategies may have moved the needle at posttest in the areas of the presentation of ideas, organization, evidence use, and language use. The results of our analyses suggest that the majority of English learners face considerable challenges in writing text-based analytical essays and that these challenges encompass both macro-level rhetorical features and higher-order tasks required for argument development and micro-level linguistic features of academic writing. Our analyses also indicate that the intervention had a positive impact on many elements of students’ text-based argument writing across students of different language groups.
Presentation of Ideas
Genre features of effective argumentation include presenting and developing a clear claim. Our results show Pathway significantly improves students’ abilities to establish a clear claim that addresses the prompt and helps students focus on proving this claim throughout their writing. Though English learners are generally challenged by presenting a claim and substantiating the claim, we find that by controlling for pretest scores, our subgroup of English learners improve as much as their EO/IFEP peers in presenting a clear and compelling claim and focusing on substantiating this claim. Further, there were no differences between the more advanced group of English learners (i.e., RFEPs) and EOs/IFEPs. This shows Pathway to be effective across all language groups in helping students craft and support an interpretive claim in response to the prompt. Because claims are interpretive, this indicates Pathway’s effectiveness in providing instructional scaffolds to move students from knowledge telling to knowledge transformation (Bereiter & Scardamalia, 1987).
We hypothesize that a key component of the intervention that leads to improved presentation of ideas is the cognitive strategies tool kit and bookmarks which help students to deepen their thinking and to move from literal comprehension to interpretation as readers (Conley, 2013; Langer, 2002; Tierney & Pearson, 1983). Sentence starters for asking questions, making connections, forming interpretations, reflecting and relating, evaluating, etc., prompt students to transition from reading to interpretation to criticism (Scholes, 1985).
Organization
An important feature of argument writing is advancing logical modes of organization (claim, reasoning, evidence, connection) with ideas organized into distinct paragraphs based on their rhetorical functions. This was an area that posed a considerable challenge for all of our English learners. The tendency to summarize and retell using chronological order and narrative style rather than analyzing, reasoning, and interpreting indicates challenges with argument structure. At pretest, many students tended to write their essays as one extended paragraph, failed to acknowledge the text they were writing about, neglected to provide context, and were limited in the rhetorical moves they employed to introduce, develop, and provide closure.
Pathway produced gains from pre- to posttest for both English learners and RFEPs for many organization items that brought them to parity with their EO/IFEP peers in the control condition. One Pathway tool that intends to improve organization is the DO/WHAT Chart (Graham et al., 2016), which is a procedural scaffold to help students “unpack” the writing prompt so students can organize their ideas based on the prompt. Further, the HoT-SC-T strategy (Olson et al., 2020) provided a way for students to structure their introduction. Given the gains in the quality of the introduction and overall organization, we see these two strategies as impactful as they help students explicitly identify key features of a text-based argument writing and scaffold the use of various rhetorical moves (Graham & Perin, 2007).
However, findings show that the English learners who had the lowest proficiency in English continue to struggle to write a conclusion. Given lower overall scores for this item across all students, we think it is plausible this finding is partially due to the time constraints of an on-demand writing task. Many students, especially English learners, are hard-pressed to reach the conclusion in a timed condition. It appears though that English learners are more affected by this time constraint and though English learners in the treatment condition outperformed English learners in the control, they did not make gains on parity with EO/IFEPs, or RFEPs on this outcome. This also suggests that teachers should provide explicit instruction on writing conclusions, as well as strategies for writing under time constraints. Additional practice in these areas could provide students opportunities to show growth in these areas.
Evidence Use
To develop their argument, students must select relevant details and examples from the source text that serve as evidence to support their claims, integrate source materials effectively into their writing, and provide commentary and interpretation explaining why and how the evidence supports the claim. The improvements in evidence use, commentary, and balance are notable for both English learners and RFEP students in Pathway, with RFEPs scoring on parity with their EO and IFEP peers in the control condition (Olson et al., 2017, finds they even outperform in writing using different measures). Controlling for pretest scores, we also find RFEPs generally had more improvement in these skills than English learners, despite having a higher starting point than their English learner peers. The use of commentary to interpret or elaborate on source material continues to be a difficult skill for English learners. Though English learners made some significant gains in the quality of their commentary, our analyses show that, on average, their posttest scores were lower than EO/IFEP students in the control group. Overall, though, Pathway is effective in improving these skills. For example, after controlling for their lower starting point in the use of evidence at pretest, score gains on evidence use at posttest were similar between English learners in the treatment group and EOs/IFEPs in the control group.
One tool that supports students in balancing summary, evidence, and commentary is the color-coding strategy (Graham et al., 2016; Olson & Land, 2007). This strategy helped students to visibly see whether they had simply summarized or offered evidence and commentary. Further, mini lessons on quoting from the text not only focused on how to use introductory phrases and reporting verbs to introduce quotations but provided sentence stems such as “This quote . . . signifies” or “These words suggest that . . . ” to present commentary discussing the significance of the evidence and linking evidence to claims. While English learners did grow in the use of evidence, balance, and commentary, suggesting they can use color coding to distinguish between summary and commentary in their writing, our results suggest more attention should be given to helping English learners develop commentary. In addition, the use of formative feedback from trained readers is also an element of Pathway that encourages students to move from knowledge telling, which was evident before instruction, to the complex process of knowledge transformation involving analysis and interpretation (Bereiter & Scardamalia, 1987). Students whose text production system corresponds to the knowledge-telling model “need more than encouragement to revise” (p. 156). Bereiter and Scardamalia (1987) advocate providing students with modeling, lessons in planning and writing for an audience, and “insight into their own composing processes” (p. 165). To help English learners grow in developing commentary, more guided practice with experienced writers, frequent opportunities to practice, the use of mentor texts, and opportunities to set goals, plan, and reflect upon their own writing progress may help English learners continue their growth in interpretive writing (Chung et al., 2021; Harris & Graham, 2017).
Language Use
Students who lack knowledge of specialized language use in academic written discourse, formal register, and discursive conventions are constrained as they engage in complex writing tasks that require higher-order analytical skills (Scarcella, 2003; Schleppegrell, 2009). English learners in general face the dual challenge of learning how to write in an academic context while developing proficiency in the English language. Developing proficiency in academic language takes time and is dependent on various factors, including their exposure to academic language and formal instruction, previous development of linguistic knowledge, and the extent to which students pay attention to words, phrases, and sentence structures (Maamuujav et al., 2021).
The improvements in language use (e.g., fluency, syntax, diction, conventions, and tone) are the smallest in our analyses. This is understandable as it takes many years to become proficient in a second language, a point emphasized by smaller gains both for treatment and control students on items related to language use. However, it does appear that the Image Grammar strategy (Noden, 2011; Olson et al., 2020) may have had a positive impact on improving sentence variety and style and that mini lessons that dealt with academic English and adopting a formal “school” register and appropriate tone had some salutary effects. However, regression analyses show this effect was greater for RFEPs. In our analysis, there was no statistically significant impact on the control of conventions, and English learners continued to struggle with the use of conventions relative to their EO/IFEP and RFEP peers. RFEPs, controlling for pretest scores, did improve more than their EO/IFEP peers at posttest, showing the capacity for growth in language use.
Another potential explanation for the rigidity of some language-specific features of writing, such as conventions, could be the difficulties of writing in an on-demand situation and self-regulating the revision process. Checking writing for correct and accurate language conventions often comes after writing a substantial portion of the essay, and time constraints may have interfered with students’ attention to such online revisions. In this way, productivity, self-regulating revision while writing, and revising for sentence fluency could be important areas for instruction for students designated as English learners.
Conclusion
The What Works Practice Guide Teaching Secondary Students to Write Effectively (Graham et al., 2016) presents the following evidence-based recommendations to improve student writing:
Explicitly teach appropriate writing strategies using a Model-Practice-Reflection instructional cycle
Integrate writing and reading to emphasize key writing strategies
Use assessments of student writing to inform instruction and feedback.
The Practice Guide acknowledges that most of the 15 studies that met WWC standards (including four Pathway studies) involved multicomponent interventions, pointing out, “Studies of these interventions typically cannot identify whether the effects of the intervention are due to one of the practices or all of the practices implemented together” (p. 3). We concur with Graham and colleagues that the Pathway to Academic Success is, first and foremost, a multicomponent intervention based on a cognitive strategies approach to reading and writing instruction. The use of integrated cognitive strategies is crucial in developing students’ strategic thinking and analytical reasoning skills, which ultimately improves students’ text-based argument writing.
We see that all components of writing can be improved through a comprehensive, cognitive strategies instruction approach and all students can improve their writing regardless of language status. However, specific elements of writing may need more time for English Learners in general to develop, such as developing a conclusion, using commentary to interpret textual evidence, and specific aspects of language use. Despite many improvements, English learners, including those who have been reclassified as English proficient on standardized tests, still need additional support, specifically in developing commentary, writing conclusions, acquiring academic language, and building sentence fluency and lexical capacity.
Like all students, English learners, regardless of designated label in school, need a comprehensive instructional approach that focuses on higher-level interpretive and argumentative aspects of writing, alongside the development of language skills necessary to express their higher-level thinking effectively. Our work shows both English learners and RFEPs benefit from rigorous instruction that prioritizes complex interpretive thinking and scaffolded instruction leading students to knowledge transformation. However, this type of instruction is often reserved for honors students (Applebee et al., 2003) or those deemed English proficient (Short & Fitzsimmons, 2007). The robust effects of the Pathway intervention indicate the capacity for all students to grow as argument writers, regardless of language proficiency, if given the opportunity, time, and scaffolded strategy instruction.
Footnotes
Appendix A
Appendix B
Analytic Framework Items to Measure Writing Quality.
| Areas of writing | Items | Descriptions | Interrater agreement (%) |
|---|---|---|---|
| Use of evidence | Evidence | How well does the student use source material as evidence? | 95 |
| Commentary | How well does commentary interpret and use the textual evidence (to support a claim)? | 98 | |
| Balance | How well does the writing balance purposeful summary, evidence, and commentary? | 93 | |
| Ideas | Address prompt | How well does the writing address all aspects of the prompt? | 100 |
| Present claim | How well does the writing present a clear and compelling claim | 100 | |
| Focus | How well is the writing focused on proving/substantiating an interpretable claim? | 100 | |
| Structure | Organization | How well is the writing globally organized? | 100 |
| Introduction | How well does the writing advance a strong introduction | 79 | |
| Conclusion | How well does the conclusion relate to claims made throughout and give the writing a sense of completeness? | 90 | |
| Language use | Fluency | How well does the essay demonstrate sentence fluency and sentence flow? | 78 |
| Syntax | How well does the essay demonstrate syntactic variety and style? | 90 | |
| Diction | How well does the essay demonstrate command of diction and word choice? (discount language “borrowed” from source text) | 84 | |
| Conventions | How well does the essay demonstrate control of language and standard grammar conventions including spelling, capitalization, and punctuation? | 84 | |
| Tone | How well does the writing adjust language and use tone appropriate to purpose, audience, and task? | 87 |
Note. Raters used a 7-point scale; reported agreement rates within 1 point.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding was provided by the Institute of Education Sciences (Grant No. R305C190007).