Sage Journals: Discover world-class research

Abstract

This paper provides an examination of the foundations, efficacy, and effectiveness of a set of practices associated with improved social studies and history learning and literacy outcomes for middle grade students, including students with varying learning needs (e.g., English learners, students with disabilities). This approach, Promoting Adolescents’ Comprehension of Text, has been the focus of multiple randomized controlled trials, including a large-scale effectiveness study. This paper has two foci: (a) to examine the evidence for the PACT intervention across settings, populations, and time and (b) to consider the design of these studies and the unit of randomization (i.e., within teacher or across school) to better understand why the typically expected differential effect sizes between efficacy and effectiveness studies were not evident.

Keywords

English language learners evidence-based practice literacy

National and international studies reveal that significant numbers of adolescents do not adequately understand complex texts, impeding their school success, access to postsecondary learning, and opportunities within our increasingly competitive work environment (Carnegie Council on Advancing Adolescent Literacy, 2010; Faggella-Luby et al., 2012; Vaughn et al., 2022). Annual student growth in reading achievement is greatest during elementary school and declines considerably over time, with high school students’ achievement growing the least (Bloom et al., 2008; Scammacca et al., 2015). Secondary students with reading difficulties are more likely to have general academic difficulties and are less likely to be provided the school-based support they need (Lipscomb et al., 2017). Many secondary teachers perceive that they are inadequately prepared to meet the literacy needs of students with reading difficulties (Hall, 2005; O’Brien et al., 2001). Yet the ability to understand complex text may differentiate students who are college ready from those who are not (ACT, 2009). Reports by ACT (2009, 2013) reveal that only 44% of ACT-tested students are ready to read and understand college-level text, with more students “on track” in 8th grade than in 12th grade. These reports are troubling because the students who take the ACT are self-identified as interested in postsecondary schooling.

One recommended avenue for improving the literacy and academic outcomes of all middle grade and secondary students is to provide literacy instruction across the content areas (e.g., Carnegie Council on Advancing Adolescent Literacy, 2010; Heller & Greenleaf, 2007; Kamil et al., 2008). However, content area general education teachers often struggle to incorporate reading instruction into their classes, perhaps considering text reading as a general task and not specific to the complexities and nuances of their subject area (Wineburg, 2001). These content area literacy needs suggest a vivid opportunity to develop an intervention to enhance both reading comprehension and content learning.

The Promoting Adolescents’ Comprehension of Text (PACT) intervention is one avenue to enhance content-focused reading comprehension skills in secondary students (Vaughn, Swanson, et al., 2013). PACT is a text- and discourse-based reading comprehension intervention that integrates effective literacy practices within social studies content instruction. Teachers use a set of instructional practices to engage students in the social studies content through vocabulary, text reading, and discussion with the goal of improving student understanding of the content. PACT aligns with current standards in content area literacy and history (e.g., Common Core State Standards, National Council for the Social Studies), which emphasize higher-level reasoning and thinking about text and content rather than simple acquisition of factual information.

Theoretical and Conceptual Foundations of PACT

PACT was based on a content approach to improving teachers’ facilitation of content knowledge and text use. Content approaches to reading instruction fit within a text-processing perspective (e.g., Kintsch, 1974; van den Broek et al., 1999) that characterizes written content as the vehicle for instruction, engaging students in building coherent representations of the ideas presented in print. Kintsch’s (1998) construction-integration model operationalizes the content approach. Broadly stated, students read text to construct and link understandings and then integrate these new ideas with previously learned content. Instructionally, this model includes a focus on discourse about text so that students can elaborate on important content, better integrate new ideas into their working understanding of the content, and more accurately recall elements of the newly learned content at subsequent points in time (Applebee et al., 2003; Beck & McKeown, 2006).

Several previously developed approaches to integrating reading comprehension practices within content area reading influenced PACT. First, McKeown et al. (2009) reported that a content approach outperformed a strategy-based approach on researcher-developed reading comprehension outcome measures. The content approach to instruction was designed to facilitate teachers’ knowledge and skills to focus student attention on the content of the text using open, meaning-based questions about the text. McKeown and colleagues contrasted this content approach with what they referred to as strategies instruction whereby students were taught procedures to guide their access to text during reading. Second, the Collaborative Strategic Reading model, in which secondary teachers implement selected literacy strategies within content area instructional texts (e.g., social studies, science) emphasized discourse through roles students enact within small groups (e.g., gist expert). Although comprehension reading strategies are taught (e.g., getting the gist), they are taught within the service of better understanding text. Findings reported statistically significant gains on a standardized reading comprehension measure (e.g., Vaughn et al., 2011). Third, the expert social studies teachers who participated in the PACT design experiments prior to the efficacy trials provided consistent support for ensuring that content was a focus of the approach to support knowledge acquisition and comprehension (Vaughn, Swanson, et al., 2013).

The instructional practices embedded in PACT were derived from research on literacy and learning practices associated with improved outcomes for individuals with learning difficulties (Edmonds et al., 2009; Scammacca et al., 2007, 2015). These practices included (a) purposeful activities with peers to facilitate learning, (b) advanced organizers that clarify themes necessary for learning, (c) instruction in vocabulary and concept development related to building background knowledge, (d) supported opportunities to read and use text to justify learning, and (e) integrated instructionally based assessment for ongoing feedback with review. The resources and materials developed promote student-focused practices emphasizing opportunities to respond with feedback and individual accountability (Gajria et al., 2007; Gersten et al., 2006; Swanson et al., 2012).

The practices were developed and refined through a series of design experiments with social studies teachers identified as expert teachers by school and district administrators, ensuring the literacy practices were feasible to implement and sustainable within the content area routines. Most other programs do not meet what Gersten et al. (2007) determined as necessary for sustained use of educational practices: to reflect the realities of implementation in today’s schools (be practical and useful) and have a high degree of specificity. Textbook-based curricula for social studies/history frequently cover content through isolated facts without clear links between events or review of related events and actions, making it difficult for many students to learn key aspects of the content (Gersten et al., 2006; Harniss et al., 2007; Villano, 2005). PACT was conceptualized to address these challenges by supporting differentiated instruction during content learning to meet the needs of a range of student learners.

PACT Components

The PACT intervention includes five instructional practices that are integrated into each unit of instruction: (a) comprehension canopy, (b) essential words, (c) critical reading of text, (d) team-based learning (TBL) comprehension checks, and (e) TBL knowledge application. Each component is described in more detail below.

Comprehension Canopy

The comprehension canopy is introduced at the beginning of a unit to build background knowledge and motivate students to learn new content. Students watch a brief, engaging video that introduces the topic. Prior to watching the video, teachers provide a purpose for viewing; afterward, student partners have a short discussion about the video. Teachers also introduce an overarching question for the unit topic to guide students’ learning throughout the unit.

Essential Words

Teachers introduce and review four to five high-utility words or concepts that are essential to comprehending the unit content. Instruction includes student-friendly definitions, visuals, related words, use of the words in context, and student discussion prompts to practice using the word and its meaning. Words are introduced on the first day of the unit; on several subsequent days throughout the unit, at least one essential word is reviewed in a warm-up activity that requires students to apply the meaning of the word. Students continue to receive exposure to the words in the texts, TBL comprehension checks, and TBL knowledge application activities.

Critical Reading of Text

Throughout the unit, students read and discuss information from high-quality primary and secondary texts. Readings can be completed with the whole class, in small groups, in student pairs, or individually. Each text is divided into stopping points with questions designed to check comprehension, extend thinking, and encourage discussion. At each stopping point, teachers facilitate discourse and note-taking to connect text to the overarching question, essential words, and previously learned material.

Team-Based Learning

Adapted from a university-level practice that promotes collaborative discourse, text-based discussions, and justifications for ideas (Michaelsen & Sweet, 2011), TBL provides structure for students to engage in peer problem-solving and discussion around the content. Students construct knowledge through discourse by thinking about the topic, expressing thoughts and perspectives, considering multiple perspectives, using evidence, and challenging ideas. TBL consists of four key elements: (a) heterogeneous, permanent teams of students; (b) a readiness-assurance process that incorporates individual and group accountability for content learning; (c) a peer-evaluation process for evaluating the team’s success; and (d) knowledge application activities in which teams complete a problem-solving task using the newly learned content.

TBL comprehension check

Short comprehension checks (i.e., the readiness-assurance process) are conducted twice during each unit, providing teachers with information on student understanding of the content. First, students complete five questions independently. Next, students complete the same questions as a team of three to four students, this time using their text and notes. Teams discuss and justify answers to each question. Scratch-off cards provide immediate feedback on accuracy of answers. If the answer selected is incorrect, the team discusses the question further until the correct answer is obtained. As students complete the group check, the teacher monitors progress, encourages discussion and active engagement, and prompts students to use text evidence. Finally, the teacher provides targeted instruction to address gaps in student understanding that were noted in the individual checks or team discussions.

TBL knowledge application

The TBL knowledge application activity concludes the unit. Students again meet with their assigned teams to integrate the content learned in the unit. Students discuss unit-level issues, make decisions, solve problems, or draw conclusions by applying knowledge gained in the unit and supplying evidence from texts and notes. Teams then present their conclusions, evidence, and reasoning to the class. To conclude the unit, teachers return to the comprehension canopy question with a student discussion. Teachers are then encouraged to evaluate the team process with the students.

Empirical Support for PACT

A series of efficacy studies and an effectiveness study have examined the impacts of PACT implementation in middle school social studies/history classrooms. The general education teachers were provided materials, resources, and professional development to implement PACT. Study classrooms were in several U.S. states (i.e., California, Indiana, Florida, Texas, Tennessee) and included a range of learners found in typical classes—students reading at or above grade level, students with reading difficulties, English learners (ELs), and students with disabilities.

Four experimental studies (three efficacy and one effectiveness) examining all learners in the classes met the What Works Clearinghouse evidence without reservation. Two quasi-experimental studies addressing students with disabilities met What Works Clearinghouse evidence with reservation (the highest rating possible for quasi-experimental studies).

Each of the studies included 1 to 2 days of professional development on the PACT practices before implementation and a 2- to 3-hr professional development session after the first unit to review the PACT elements and discuss areas to improve implementation. The efficacy studies also included in-class support during implementation that could include coaching and feedback, modeling of instruction, or assistance with lesson planning.

Measures of social studies content knowledge, reading comprehension in the content area, and general reading comprehension were administered to participating students to examine the effects in the studies. These measures are further described below.

Student Measures

Assessment of Social Studies Knowledge and Comprehension

The Assessment of Social Studies Knowledge and Comprehension (Vaughn, Swanson, et al., 2013) is an untimed measure of content knowledge in the intervention units that was developed through previous Institute of Education Sciences funding (Vaughn, Roberts, et al., 2013). The Content Acquisition subtest consists of 46 four-option, multiple-choice items. The measure is not overly aligned with the treatment, as items from state-level tests reflect knowledge of the content and both the treatment and business-as-usual (BaU) conditions receive content instruction in the three units (treatment uses the PACT intervention during content instruction; BaU uses typical practices). The items with known difficulty parameters were used with permission from released state tests in Texas and Massachusetts and from released advanced placement tests.

Subsamples of items were pilot tested in the population of students targeted by the interventions. Item-level confirmatory factor analysis was used to evaluate item parameters. The model represented the observed data well, χ² = 1,022.69, df = 989, p =.22, comparative fit index (CFI) = .97, root mean square error of approximation (RMSEA) = .009. Internal consistency is .89. The Assessment of Social Studies Knowledge and Comprehension is most informative in the average ability distribution. The peak of the test information function (information = .91) occurs at −0.25 on the ability distribution. The information was above .80 for ability levels ranging from −2.0 to 2.0, indicating greater precision of information near the average theta than toward the extremes. Test information indices indicate the precision of measurement for persons at different levels of the underlying latent construct, with higher information denoting more precision (Guion, 2011). This approach is an item response theory-based analog for estimates of reliability used in classical test models.

Modified Assessment of Social Studies Knowledge and Comprehension

The Modified Assessment of Social Studies Knowledge and Comprehension (Vaughn et al., 2015) consists of three reading passages (Lexile range from 1,090 to 1,140; word count ranges from 312 to 349) that participants have previously not read. Students read each passage silently and immediately answer seven four-option, multiple-choice questions about the passage. The items measure students’ ability to identify main ideas, understand vocabulary in context, identify cause and effect, and summarize. The model for the Modified Assessment of Social Studies Knowledge and Comprehension proximal reading comprehension test also fits the data extremely well: χ² = 249.97, df = 189, p = .24, CFI/Tucker–Lewis index (TLI) = .99, RMSEA = .028. Internal consistency is .85.

Gates-MacGinitie Reading Test–4th Edition

The Gates-MacGinitie Reading Test–4th Edition (MacGinitie et al., 2000) is a standardized, group-administered test used as an outcome measure in many secondary reading intervention studies. The average K-R 20 reliabilities are .90 or above. Test–retest reliabilities are .85 and above for the grade levels of interest. Internal consistency reliability ranges from .91 to .93, and alternate form reliability is reported as .80 to .87.

The findings from these PACT studies are discussed below and their effect sizes are reported in Tables 1 and 2.

Table 1.

Summary of Findings From Efficacy and Effectiveness Studies.

Study	Sample	Research design	Measures	Effect size (g)	Meets WWC standards
Vaughn, Roberts et al., 2013	419 eighth-grade students	Randomization at the class level; within-teacher experimental design; 16 PACT classes; 11 BaU classes	Content acquisition	0.17	Without reservation
			Content reading comprehension	0.29
			Broad reading comprehension	0.20
Vaughn et al., 2015	1,487 eighth-grade students	Replication of above study; within-teacher experimental design; 47 PACT classes; 38 BaU classes	Content acquisition	0.32	Without reservation
			4-week follow-up	0.29
			8-week follow-up	0.26
			Content reading comprehension	0.02
			Broad reading comprehension	0.01
Vaughn et al., 2017	1,629 eighth-grade students	Randomization at the class level; within-teacher experimental design; 49 PACT classes; 45 BaU classes; all classes had English learners	Content acquisition	0.40	Without reservation
			Content reading comprehension	0.20
			Broad reading comprehension	0.12
Roberts et al., 2023	6,302 eighth-grade students; 881 sixth-grade students	Randomization at the school level; between-school experimental design; 24 PACT schools; 24 BaU schools	Content acquisition	0.46	Without reservation expected
			9-week follow-up	0.40
			Content reading comprehension	0.15
			Broad reading comprehension	0.14

Note. WWC = What Works Clearinghouse; BaU = business as usual.

Table 2.

Summary of Findings From Quasi-Experimental Studies of Students With Disabilities.

Study	Sample	Research design	Measures	Effect size (g)	Meets WWC standards
Swanson et al., 2015	130 eighth-grade students with disabilities	Quasi-experimental design using data on students with disabilities from the Vaughn et al., 2013, and Vaughn et al., 2015, samples	Content acquisition	.26	With reservation
			Content reading comprehension	.34
			Broad reading comprehension	.09
Wanzek et al., 2016	148 eighth-grade students with disabilities	Quasi-experimental design using data on students with disabilities from the Vaughn et al., 2017, sample	Content acquisition	.51	With reservation
			Content reading comprehension	.04
			Broad reading comprehension	.02

Note. WWC = What Works Clearinghouse.

Efficacy Studies

Study 1, efficacy RCT: Eighth-grade social studies/history

Study 1 was an experimental study conducted in 27 eighth-grade social studies/history classes that were randomly assigned within teacher to either the PACT treatment condition (n = 16) or the BaU condition (n = 11; Vaughn et al., 2013). Because the intervention aligned with the history standards, and teachers’ classes were the unit of randomization, all students received the same history content during each unit. However, only students in the PACT condition learned the content using PACT components. In the BaU classes, teachers taught the content in their typical way. Fidelity data indicated that the PACT intervention was delivered with moderate to high levels of fidelity, suggesting that it was practical for teachers to implement. Fidelity data also indicated that specific PACT practices were generally not implemented in the teachers’ BaU classes, though some teachers in this study implemented text reading and note-taking as part of the content instruction.

Following three instructional units of implementation (approximately 6–8 weeks), the students in the PACT treatment classes outperformed students in the BaU classes at statistically significant levels on measures of content acquisition (g = 0.17), content reading comprehension (g = 0.29), and broad reading comprehension (g = 0.20).

Study 2, efficacy RCT replication: Eighth-grade social studies/history

Study 2 was a replication of Study 1 (Vaughn, Swanson, et al., 2013) conducted with 1,487 eighth-grade students in 85 classes that were randomly assigned within teacher to either the PACT treatment condition (n = 47) or the BaU condition (n = 38; Vaughn et al., 2015). Intervention fidelity was once again reported at moderate to high levels. The specific PACT elements were not observed in the BaU classes.

Following the three units of implementation for the study (6–10 weeks), the students in the treatment classes significantly outperformed the students in the BaU classes on the social studies content measure (g = 0.32). These effects were maintained at 4-week follow-up (g = 0.29) and 8-week follow-up (g = 0.26). Thus, the additional student learning in the treatment classes was retained over time. There were no differences between study conditions on the reading comprehension measures. In addition to the main effects, fidelity to the comprehension canopy, essential words, and TBL knowledge application elements mediated the effect of treatment on social studies content acquisition. Higher levels of fidelity in these areas served as an underlying mechanism for improved social studies content acquisition.

Study 3, efficacy RCT: ELs in eighth-grade social studies/history

Study 3 was a further replication with classes that contained ELs. As in the previous two studies, classes were randomized within teacher to either the PACT (n = 49) or BaU (n = 45) condition. Study 3 included 1,629 eighth-grade students in social studies classes that contained ELs (Vaughn et al., 2017). The proportion of ELs in the classes ranged from 14.5% to 44.5%. The existing PACT practices that already had many elements of instruction effective for ELs were enhanced with additional focus on academic vocabulary, visuals to accompany learning, and partner and small-group peer interactions and discussions. As with previous studies, PACT implementation was moderate to high in the treatment classes, and PACT practices generally were not observed during BaU instruction. As with Study 1, some text reading practices were observed in less than a fifth of the BaU observations but not with high levels of overlap with PACT practices.

Following implementation (6–8 weeks), the treatment students (both ELs and non-ELs) outperformed comparison students at a statistically significant level on measures of content acquisition (g = 0.40) and social studies content reading comprehension (g = 0.20) but not on general reading comprehension—though impact was in favor of students in the treatment condition (g = 0.12). The proportion of ELs in the class moderated outcomes on the social studies content acquisition measure. As the percentage of ELs in a class increased, performance declined on the social studies content acquisition measure for all students—with a more marked decline for ELs than non-ELs. The larger numbers of students with limited language and vocabulary use in English may reduce the effects of the discourse-based instruction. There was no moderating effect of class proportion of ELs on the social studies reading comprehension effects.

Similarly, in Study 3, Vaughn et al. (2017) reported that as the percentage of readers with reading comprehension difficulties (defined as students with a standard score of 85 or below on the Gates-MacGinitie Reading Comprehension subtest) in the class increased, student scores on the social studies content acquisition measure decreased. A total of 359 students with reading comprehension difficulties were in the treatment classes, and 331 students with reading comprehension difficulties were in the BaU classes. The moderating effect of class percentage of students with reading comprehension difficulties suggests that students are disadvantaged in classes with a large percentage of struggling readers.

Quasi-Experimental Studies: Disaggregated Data for Students With Disabilities

An additional investigation examined the effects of the PACT intervention on outcomes for the 130 eighth-grade students with disabilities served in the 112 classes from Study 1 and Study 2 (Swanson et al., 2015). Students with disabilities in the treatment condition (n = 72) scored statistically higher than students with disabilities in the BaU condition (n = 58) on measures of content acquisition (g = 0.26) and content reading comprehension (g = 0.34) but not broad reading comprehension (g = 0.09).

Similarly, an investigation examined the effects of PACT for students with disabilities (n = 148) in the classes that were a part of Study 3 (Wanzek et al., 2016). Again, students with disabilities in the treatment classes outperformed students with disabilities in the BaU classes on the measure of content acquisition (g = 0.51) and performed equally as well as students in the BaU condition on measures of content reading comprehension and broad reading comprehension. In addition, there were no differences in effects for EL and non-EL students with disabilities. Thus, evidence suggests that the PACT intervention benefits students with disabilities the same or more in content acquisition than students without disabilities when these students are included together in general education social studies classes.

Effectiveness Study

Overall, the experimental efficacy studies conducted in several states and multiple districts reveal a consistent pattern of improved content acquisition for students in social studies/history classes whose teachers implement PACT practices. In addition, the quasi-experimental studies examining effects for students with disabilities provide further evidence of the impacts of the PACT practices on content acquisition. Throughout all of the efficacy studies, researchers supported teachers with PACT implementation, raising questions about whether PACT would be impactful within an effectiveness study.

Study 4, effectiveness RCT: Middle school social studies/history

Study 4 examined the effectiveness of PACT and involved multiple cohorts of schools collected over several years with schools assigned randomly to the PACT or BaU condition (Roberts et al., 2023). The sample design, recruitment, and generalization of this effectiveness study are described in the Roberts et al. paper. In total, 48 schools and more than 7,000 students participated in the study, with 24 schools in the PACT condition and 24 schools in the BaU condition. Similar to the efficacy studies, data were collected before, during, and after teachers taught three units (Colonialism, Road to Revolution, and the American Revolution) to ensure that all students, regardless of condition, received the same content instruction.

As noted above, teacher preparation for implementing the PACT practices in the effectiveness study followed more typical school professional development models, with a 1-day workshop on PACT practices prior to PACT implementation and a 2- to 3-hr session after the first unit of instruction. The effectiveness study did not include in-class support during implementation. Teachers could contact research team members with questions about implementation if needed, but no other assistance was provided.

Following the three units of instruction, students in the treatment schools significantly outperformed students in the comparison schools on measures of content acquisition (g = 0.46) and content reading comprehension (g = 0.40). Students at treatment and comparison schools performed similarly on general reading comprehension—though impact was in favor of students in the treatment condition (g = 0.14).

These moderately strong impacts for students who participated in the effectiveness study were unexpected, as overall, the teachers received less support for PACT implementation. Typically, effectiveness studies yield significantly smaller effects than efficacy studies—largely because the research team has little or no involvement in fidelity or implementation and the expectation is that teachers will implement the program in a less consistent way that will reduce impact. For example, Vaden-Kiernan et al. (2008) conducted an effectiveness study of Open Court Reading, a reading program associated with improved outcomes for students in previous efficacy studies (Borman et al., 2008). Findings revealed overall effects on reading for the Open Court Reading treatment condition compared with BaU that were not statistically significant in a large sample of schools after 1 year. A small, negative effect was reported after 2 years. These types of findings for reading comprehension studies conducted as effectiveness trials are not uncommon (James-Burdumy et al., 2012) and often raise questions about whether and how to scale up reading comprehension practices (Pearson et al., 2020). The positive and statistically significant findings from the effectiveness study for PACT suggest that these practices within organized units of existing content may be feasible for teachers to integrate at sufficient levels into their existing content instruction. With respect to integrating into their existing content instruction, we reference that teachers utilized the units of study and the lessons within their extant content area instruction—in most cases replacing similar content taught in more traditional ways. These findings suggest that research and practice teams might consider designing from the beginning the instructional practices and interventions that are likely to be more readily adopted by teachers.

Efficacy and Effectiveness

As can be seen from the previous description and tables of findings, the PACT intervention demonstrates positive, significant, and consistent findings across settings, populations, and randomization practices (class level and school level). We think this evidence is robust and demonstrates a level of impact that warrants change in typical social studies/history practice. We think the findings from the effectiveness study are noteworthy because effectiveness studies are typically associated with null effects or even negative effects (Vaden-Kiernan et al., 2008), attributed in part to the challenges of asking teachers to change practices without adequate support for implementing these practices.

We also think it is worth discussing the potential power of randomizing classes within teacher to treatment and BaU conditions, as was done in the efficacy studies. This within-teacher design is not universally available (e.g., typically not available at the elementary level) and raises several questions about whether and how teachers are able to maintain their typical instruction for some classes and to alter their instruction for others. Developed by Francis (Francis & Vaughn, 2009), this design has been used in several other experimental studies including with collaborative strategic reading (Vaughn et al., 2011). In all cases, the studies met the What Works Clearinghouse evidence standards without reservation. This class randomization design involves delivering both treatment and control interventions using the same teachers across multiple sections of middle school students. By doing so, researchers are able to control for individual differences at the teacher level while still estimating variability within teachers in their delivery of instruction. Of course, the challenge in such settings is obtaining fidelity to treatment, both in terms of getting the treatment delivered in treatment sections and in keeping the treatment from bleeding over into the control sections. The study authors describe meeting these challenges by stressing the importance of treatment integrity in teachers’ professional development, aided by instructional materials to support the intervention that are not available in the BaU comparison sections. Researchers interested in educational effectiveness can gain a significant power advantage by using this within-teacher design when it is well suited.

We are aware of the potential threats of having the intervention “weep” into the BaU condition when random assignment is at the class level and that preventing this situation requires training all personnel, including participating teachers. The instructional data collection in each of the previous PACT studies, which included daily audio-recordings of instruction, indicated that no specific PACT practices were implemented in BaU classes. The previous PACT studies also include the extent to which treatment practices weep into BaU classes. Overlap was rare and occurred only to the extent that a teacher’s typical practice already included a subcomponent of a PACT practice (e.g., the teacher had students read text in a class; Wanzek & Vaughn, 2016).

So how might the impact of the effect sizes yielded from within-teacher class randomization for the efficacy studies be compared with the school randomization effectiveness study? The content acquisition in the effectiveness study was of moderate size, and this moderate impact was maintained 9 weeks later at follow-up. The effect sizes for the effectiveness study were hypothesized to be smaller than the effect sizes for the efficacy studies but were equivalent or higher on all measures. Why would this be the case? One possible explanation is that the efficacy studies used within-teacher designs, meaning that students in both the treatment and BaU conditions had the same teacher—the variation, of course, is that the teacher provides the PACT treatment in one class and typical instruction in the other class. Though you would expect that the researcher support for the efficacy trial would yield findings with more significant impact than an effectiveness study with less researcher support, findings across both design types were quite similar. It may be that a within-teacher design is sufficiently rigorous (e.g., controlling for teacher effects) that it yields findings in line with an effectiveness study. This additional rigor of a within-teacher design is that teacher effects are directly controlled by having the same teacher providing both BaU and treatment in separate classes. We are also not aware of the ways in which teachers in the effectiveness studies may have customized the PACT treatment to respond to their students in ways that made that treatment yield findings as robust as the efficacy studies.

Implications

Four RCTs (three efficacy studies and one effectiveness study) report robust and consistent findings for the impact of PACT on content learning and reading comprehension, with several studies reporting positive and sustained impact in follow-up studies. We think that these findings are impressive, considering that the studies have been implemented with approximately 10,000 students representing a range of learners and hundreds of teachers and schools across multiple states.

We recognize that influencing secondary students’ learning and comprehension is challenging, often yielding null or small impact (Goldman et al., 2016; Pearson et al., 2020). We offer two overarching implications from the PACT work to consider. First, for research teams, we think that the within-teacher design randomized at the class level may be useful, providing a relatively low-resource way to adequately power studies and establish a strong comparable BaU because the same teacher has students in both the treatment and control conditions. We recognize that within-teacher designs are not necessarily preferable to other types of randomized approaches (e.g., teacher, school level) but given the current challenges of recruiting large numbers of teachers and schools and the expense in so doing—the within-teacher design may be a feasible alternative. Second, we think that the PACT instructional practices may provide a useful set of heuristics to serve as an architectural framework guiding literacy and content learning—at least within social studies/history in the middle grades. We think it is important to note that we have utilized this set of practices within social studies/history units. There are several ways these instructional practices may be integrated into social studies/history instruction including: (a) integrating key practices like essential words, text-based activities, and TBL with a range of social studies/history topics, (b) encouraging publishers of social studies/history materials and resources to integrate these practices into their materials and resources, and (c) encouraging professional development providers and preservice teacher preparation to provide opportunities for teachers to learn to use and develop these practices within their content area instruction.

Footnotes

Associate Editor: Emily Solari

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This research was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R324A190072. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Institute of Education Sciences or the U.S. Department of Education.

ORCID iD

Sharon Vaughn

References

ACT. (2009). Measuring college and career readiness: The class of 2009. ACT.

ACT. (2013). The condition of college & career readiness. ACT.

Applebee

A. N.

Langer

J. A.

Nystrand

Gamoran

(2003). Discussion-based approaches to developing understanding: Classroom instruction and student performance in middle and high school English. American Educational Research Journal, 40(3), 685–730. https://doi.org/10.3102/00028312040003685

Beck

I. L.

McKeown

M. G.

(2006). Improving comprehension with questioning the author: A fresh and expanded view of a powerful approach. Scholastic.

Bloom

H. S.

Hill

C. J.

Black

A. R.

Lipsey

M. W.

(2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. Journal of Research on Educational Effectiveness, 1(4), 289–328. https://doi.org/10.1080/19345740802400072

Borman

G. D.

Dowling

N. M.

Schneck

(2008). A multisite cluster randomized field trial of Open Court Reading. Educational Evaluation and Policy Analysis, 30(4), 389–407. https://doi.org/10.3102/0162373708326283

Carnegie Council on Advancing Adolescent Literacy. (2010). Time to act: An agenda for advancing adolescent literacy for college and career success. Carnegie Corporation of New York.

Edmonds

M. S.

Vaughn

Wexler

Reutebuch

Cable

Tackett

K. K.

Schnakenberg

J. W.

(2009). A synthesis of reading interventions and effects on reading comprehension outcomes for older struggling readers. Review of Educational Research, 79(1), 262–300. https://doi.org/10.3102/0034654308325998

Faggella-Luby

M. N.

Graner

P. S.

Deshler

D. D.

Drew

S. V.

(2012). Building a house on sand: Why disciplinary literacy is not sufficient to replace general strategies for adolescent learners who struggle. Topics in Language Disorders, 32(1), 69–84. https://doi.org/10.1097/TLD.0b013e318245618e

10.

Francis

D. J.

Vaughn

(2009). Effective practices for English language learners in the middle grades: Introduction to the special issue. Journal of Research on Educational Effectiveness, 2(4), 289–296. https://doi.org/10.1080/19345740903217664

11.

Gajria

Jitendra

Sacks

Sood

(2007). Improving content area instruction for students with learning disabilities: A meta-analysis. The Journal of Learning Disabilities, 40, 210–225. https://doi.org/10.1177/00222194070400030301

12.

Gersten

Baker

S. K.

Shanahan

Linan-Thompson

Collins

Scarcella

(2007). Effective literacy and English language instruction for English learners in the elementary grades (NCEE 2007-4011). National Center for Education Evaluation and Regional Assistance.

13.

Gersten

Baker

S. K.

Smith-Johnson

Dimino

Peterson

(2006). Eyes on the prize: Teaching complex historical content to middle school students with learning disabilities. Exceptional Children, 72(3), 264–280. https://doi.org/10.1177/001440290607200301

14.

Goldman

S. R.

Snow

Vaughn

(2016). Common themes in teaching reading for understanding: Lessons from three projects. Journal of Adolescent & Adult Literacy, 60(3), 255–264. https://doi.org/10.1002/jaal.586

15.

Guion

(2011). Assessment, measurement, and prediction for personnel decisions (2nd ed.). Routledge.

16.

Hall

L. A.

(2005). Teachers and content area reading: Attitudes, beliefs and change. Teaching and Teacher Education, 21(4), 403–414. https://doi.org/10.1016/j.tate.2005.01.009

17.

Harniss

M. K.

Caros

Gersten

(2007). Impact of the design of U.S. history textbooks on content acquisition and academic engagement of special education students: An experimental investigation. Journal of Learning Disabilities, 40, 100–110. https://doi.org/10.1177/00222194070400020101

18.

Heller

Greenleaf

C. L.

(2007). Literacy instruction in the content areas: Getting to the core of middle and high school improvement. Alliance for Excellent Education.

19.

James-Burdumy

Deke

Gersten

Lugo-Gil

Newman-Gonchar

Dimino

Haymond

Liu

A. Y. H.

(2012). Effectiveness of four supplemental reading comprehension interventions. Journal of Research on Educational Effectiveness, 5(4), 345–383. https://doi.org/10.1080/19345747.2012.698374

20.

Kamil

M. L.

Borman

G. D.

Dole

Kral

C. C.

Salinger

Torgesen

(2008). Improving adolescent literacy: Effective classroom and intervention practices (NCEE 2008-4027). National Center for Education Evaluation and Regional Assistance.

21.

Kintsch

(1974). The representation of meaning in memory. Lawrence Erlbaum Associates.

22.

Kintsch

(1998). Comprehension: A paradigm for cognition. Cambridge University Press.

23.

Lipscomb

Hamison

Burghardt

Johnson

D. R.

Thurlow

(2017). Preparing for life after high school: The characteristics and experiences of youth in special education. Volume 2: Comparisons across disability groups (NCEE 2017-4018). National Center for Education Evaluation and Regional Assistance.

24.

MacGinitie

R. K.

Maria

Dreyer

L. G.

(2000). Gates-MacGinitie reading test (4th ed.). Riverside.

25.

McKeown

M. G.

Beck

I. L.

Blake

R. G.

(2009). Rethinking reading comprehension instruction: A comparison of instruction for strategies and content approaches. Reading Research Quarterly, 44(3), 218–253. https://doi.org/10.1598/RRQ.44.3.1

26.

Michaelsen

Sweet

(2011). Team-based learning: Small-group learning’s next big step. Wiley.

27.

O’Brien

D. G.

Moje

E. B.

Stewart

(2001). Exploring the context of secondary literacy: Literacy in people’s everyday school lives. In Moje

O’Brien

(Eds.), Construction of literacy: Studies of teaching and learning in and out of secondary schools (pp. 105–124). Routledge.

28.

Pearson

P. D.

Palincsar

A. S.

Biancarosa

Berman

A. I.

(2020). Reaping the rewards of the reading for understanding initiative. National Academy of Education.

29.

Roberts

Vaughn

Wanzek

Furman

Martinez

Sargent

(2023). Promoting Adolescents’ Comprehension of Text (PACT): A randomized control trial of its effectiveness. Journal of Educational Psychology, 115(5), 665–682. https://doi.org/10.1037/edu0000794

30.

Scammacca

N. K.

Roberts

Vaughn

Edmonds

Wexler

Reutebuch

C. K.

Torgesen

J. K.

(2007). Interventions for adolescent struggling readers: A meta-analysis with implications for practice. Center on Instruction.

31.

Scammacca

N. K.

Roberts

Vaughn

Stuebing

K. K.

(2015). A meta-analysis of interventions for struggling readers in grades 4–12: 1980–2011. Journal of Learning Disabilities, 48(4), 369–390. https://doi.org/10.1177/0022219413504995

32.

Swanson

Solis

Ciullo

McKenna

J. W.

(2012). Special education teachers’ perceptions and instructional practices in response to intervention implementation. Learning Disability Quarterly, 35(2), 115–126. https://doi.org/10.1177/0731948711432510

33.

Swanson

Wanzek

Vaughn

Roberts

Fall

A. M.

(2015). Improving reading comprehension and social studies knowledge among middle school students with disabilities. Exceptional Children, 81(4), 426–442. https://doi.org/10.1177/0014402914563704

34.

Vaden-Kiernan

Jones

D. H.

Rudo

Fitzgerald

Hartry

Chambers

Smith

Muller

Moss

M. A.

(2008). The National Partnership for Quality Afterschool Learning randomized controlled trial studies of promising afterschool programs: Summary of findings (SEDL Afterschool Research Consortium. Afterschool Research Brief no. 3). Southwest Education Laboratory.

35.

van den Broek

Young

Tzeng

Linderholm

(1999). The landscape model of reading: Inferences and the online construction of a memory representation. In van Oostendorp

Goldman

(Eds.), The construction of mental representations during reading (pp. 71–98). Lawrence Erlbaum Associates.

36.

Vaughn

Gersten

Dimino

Taylor

M. J.

Newman-Gonchar

Krowka

Kieffer

M. J.

McKeown

Reed

Sanchez

St Martin

Wexler

Morgan

Yañez

Jayanthi

(2022). Providing reading interventions for students in grades 4–9 (WWC 2022007). National Center for Education Evaluation and Regional Assistance.

37.

Vaughn

Klingner

J. K.

Swanson

E. A.

Boardman

A. G.

Roberts

Mohammed

S. S.

Stillman-Spisak

S. J.

(2011). Efficacy of collaborative strategic reading with middle school students. American Educational Research Journal, 48(4), 938–964. https://doi.org/10.3102/0002831211410305

38.

Vaughn

Martinez

L. R.

Wanzek

Roberts

Swanson

Fall

A. M.

(2017). Improving content knowledge and comprehension for English language learners: Findings from a randomized control trial. Journal of Educational Psychology, 109(1), 22–34. https://doi.org/10.1037/edu0000069

39.

Vaughn

Roberts

Klingner

J. K.

Swanson

E. A.

Boardman

Stillman-Spisak

S. J.

Mohammed

Leroux

A. J.

(2013). Collaborative strategic reading: Findings from experienced implementers. Journal of Research on Educational Effectiveness, 6(2), 137–163. https://doi.org/10.1080/19345747.2012.741661

40.

Vaughn

Roberts

Swanson

E. A.

Wanzek

Fall

A. M.

Stillman-Spisak

S. J.

(2015). Improving middle-school students’ knowledge and comprehension in social studies: A replication. Educational Psychology Review, 27(1), 31–50. https://doi.org/10.1007/s10648-014-9274-2

41.

Vaughn

Swanson

E. A.

Roberts

Wanzek

Stillman-Spisak

S. J.

Solis

Simmons

(2013). Improving reading comprehension and social studies knowledge in middle school. Reading Research Quarterly, 48(1), 77–93. https://doi.org/10.1002/rrq.039

42.

Villano

T. L.

(2005). Should social studies textbooks become history? A look at alternative methods to activate schema in the intermediate classroom. The Reading Teacher, 59(2), 122–130. https://doi.org/10.1598/RT.59.2.2

43.

Wanzek

Vaughn

(2016). Implementation of a text-based content intervention in secondary social studies classes. New Directions for Child and Adolescent Development, 2016(154), 31–48. https://doi.org/10.1002/cad.20176

44.

Wanzek

Vaughn

Scammacca

Gatlin

Walker

M. A.

Capin

(2016). Meta-analyses of the effects of Tier 2 type reading interventions in grades K–3. Educational Psychology Review, 28(3), 551–576. https://doi.org/10.1007/s10648-015-9321-7

45.

Wineburg

(2001). Historical thinking and other unnatural acts: Charting the future of teaching the past. Temple University Press.

Promoting Adolescents’ Comprehension of Text: Efficacy and Effectiveness

Abstract

Keywords

Theoretical and Conceptual Foundations of PACT

PACT Components

Comprehension Canopy

Essential Words

Critical Reading of Text

Team-Based Learning

TBL comprehension check

TBL knowledge application

Empirical Support for PACT

Student Measures

Assessment of Social Studies Knowledge and Comprehension

Modified Assessment of Social Studies Knowledge and Comprehension

Gates-MacGinitie Reading Test–4th Edition

Efficacy Studies

Study 1, efficacy RCT: Eighth-grade social studies/history

Study 2, efficacy RCT replication: Eighth-grade social studies/history

Study 3, efficacy RCT: ELs in eighth-grade social studies/history

Quasi-Experimental Studies: Disaggregated Data for Students With Disabilities

Effectiveness Study

Study 4, effectiveness RCT: Middle school social studies/history

Efficacy and Effectiveness

Implications

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iD

References