Abstract
Based on 52 studies with samples mostly from English-speaking countries, the current study used Bayesian network meta-analysis to investigate the intervention effectiveness of different reading comprehension strategy combinations on reading comprehension among students with reading difficulties in 3rd through 12th grade. We focused on commonly researched strategies: main idea, inference, text structure, retell, prediction, self-monitoring, and graphic organizers. Results showed (1) instruction of more strategies did not necessarily have stronger effects on reading comprehension; (2) there was no single reading comprehension strategy that produced the strongest effect; (3) main idea, text structure, and retell, taught together as the primary strategies, seemed the most effective; and (4) the effects of strategies only held when background knowledge instruction was included. These findings suggest strategy instruction among students with reading difficulties follows an ingredient-interaction model—that is, no single strategy works the best. It is not “the more we teach, the better outcomes to expect.” Instead, different strategy combinations may produce different effects on reading comprehension. Main idea, text structure, and retell together may best optimize the cognitive load during reading comprehension. Background knowledge instruction should be combined with strategy instruction to facilitate knowledge retrieval as to reduce the cognitive load of using strategies.
Keywords
Reading is important for individual development, not only as the foundation for educational attainment but also because it is vital for employment, health, and longevity (Kuncel & Hezlett, 2010; Wrulich et al., 2014). In particular, for 3rd grade and above, students at school are in the reading-to-learn stage, when they need to develop advanced comprehension skills so as to facilitate the development of reading and other subjects (e.g., math and science) to achieve school success (Swanson et al., 2014). However, reading is challenging for many, and it is estimated about 5% to 10% of the school-aged population are at risk for significant reading difficulties (RD; Anthony & Francis, 2005; Fletcher & Vaughn, 2009; Siegel, 2006). Thus, designing effective reading interventions for RD is one of the frontier topics in educational research and practice (Individuals with Disabilities Education Improvement Act, 2004; International Dyslexia Association, 2020). The current study used the Bayesian network meta-analysis (BNMA) to investigate the effects of different reading comprehension strategies and strategy combinations on reading comprehension among students with RD at 3rd grade through 12th grade. We hope our findings can deepen our understanding of reading comprehension strategies for RD and provide valuable information for teachers in choosing the optimal reading comprehension strategy or strategy combination in practice.
Theoretical Framework and Practice Considerations
Usually two approaches are used to improve reading comprehension among students with RD. One is through skill or knowledge-based instruction. Specifically, according to the simple view of reading (SVR, Hoover & Gough, 1990), reading comprehension, the ability to process text and understand its meaning, and integrate text information with what the reader already knows, is the ultimate outcome of reading. The two most important skills for reading comprehension are decoding, the ability to associate print with sounds accurately and/or fluently, and language or language comprehension, the ability to understand oral language in a word, sentence, or passage format. Based on SVR, instruction that is focused on decoding accuracy and fluency, vocabulary, oral language comprehension, and content knowledge are important for improving reading comprehension, which is widely researched and supported by studies and practice among students with RD, especially among English-speaking students (e.g., see reviews in Edmonds et al., 2009; Filderman et al., 2021; Solis et al., 2012; Stevens et al., 2017).
Another approach to improving reading comprehension is through instruction on reading comprehension strategy—cognitive or behavioral action that is enacted under particular contextual conditions, with the goal of improving some aspect of comprehension (Graesser, 2007). This is because reading comprehension strategies can help readers stay engaged and efficiently use the limited cognitive resources to organize information during reading comprehension (Duke et al., 2011; Gersten et al., 2001; McNamara, 2007). Specifically, the construction-integration model proposed by Kintsch (1988) emphasizes the important role of an interactive combination of top-down (knowledge-driven) and bottom-up (word-based) processes in reading comprehension. Yet, readers have a limited working memory capacity. Thus, reading comprehension strategies can serve as a bridge between the top-down and the bottom-up process to reduce the working memory load during reading comprehension. Kim (2017) proposed the direct and indirect effects model of reading (DIER), including components identified by the SVR (word reading and language comprehension) and components of text comprehension (i.e., working memory, knowledge, strategies). The component skills of reading comprehension have a hierarchical structure with direct and indirect relations among them. Word reading and language comprehension are the upper-level skills making direct contributions to reading comprehension. In contrast, cognitive skills (e.g., working memory) are the lower and foundational skills for all other component skills and can only indirectly influence reading comprehension via reading comprehension strategy use and background knowledge.
Although there is not a specific theoretical framework to define reading comprehension strategies (National Reading Panel, 2000), prior research in reading intervention suggested many evidence-based reading comprehension strategies. Most commonly used strategies for struggling readers include, but are not limited to, main idea (integrating ideas from the text information in a coherent way; National Reading Panel, 2000; Stevens et al., 2019), inference (integrating information within text and between the text and one’s general knowledge of the topic to understand ideas not explicitly stated in the text; O’Brien et al., 2015), text structure (recognizing the underlying structure of texts to help focus attention on key concepts and relationships, anticipate what is to come, and monitor their comprehension as they read; Hebert et al., 2016), retell (organizing information in order to provide a personal rendition of the text; Reed & Vaughn, 2012), prediction (utilizing what is already known from the text and background knowledge to hypothesize the content of the text and evaluate this hypothesis with the actual content; Nolan, 1991), self-monitoring (monitoring the understanding of the text through ways such as self-questioning and coherence check; Schwartz, 1997), and graphic organizers (making graphic representations of the material to assist comprehension; Merkley & Jefferies, 2000). In the current study, we focused on these strategies as they were the ones studied most in the reading comprehension interventions for older, at-risk readers (Filderman et al., 2021; Gersten et al., 2001; Hall, 2016).
Reading comprehension strategies are helpful when used alone but are considered to be more effective when used together (National Reading Panel, 2000). Thus, many reading intervention studies for students with RD often adopted a multistrategy approach tapping three or more strategies (e.g., Filderman et al., 2021; National Reading Panel, 2000). However, different studies focused on different strategies or different combinations of strategies. There is a lack of research to explore if there is the most important strategy, why certain strategies should go together, or if the more strategies we teach, the better reading comprehension outcomes we can expect. Answers to these questions may have several important theoretical and practical implications.
Cognitive Load of Strategy
Reading comprehension strategy instruction may be quite cognitively demanding and sometimes may not lead to desirable outcomes when readers have limited background knowledge and cognitive capacity (e.g., working memory; McNamara, 2007). For example, almost all reading comprehension strategies require working memory to simultaneously process and store text information and to integrate information extracted from the text with long-term memory knowledge (Kim, 2017). Moreover, students need to be flexible in their thinking about when to use what strategy during reading comprehension. However, students with RD often have insufficient background knowledge (e.g., Cain & Oakhill, 2006; Elleman et al., 2009; Filderman et al., 2021; Spencer et al., 2014) and cognitive deficits (e.g., deficits in working memory and flexibility; Booth et al., 2010; Kendeou et al., 2014; Meltzer, 2018; Peng & Fuchs, 2016). In addition, students with RD tend to have perceptual challenges (e.g., challenges in efficient visual and auditory process; Abadzi 2017; Gori & Facoetti, 2013, 2014; Liu et al., 2016), which may overload their working memory and prevent them from applying working memory to using reading comprehension strategies. Thus, although reading comprehension strategies are designed to reduce cognitive load and facilitate the reading comprehension process, the learning and application of these strategies themselves can be cognitively demanding for students with RD, especially when first taught or when there is not sufficient practice on the strategies (Graesser, 2007).
Moreover, each of these reading comprehension strategies, although considered independent, are conceptually related to one another as they all serve the purpose of schema construction during reading (Fuchs & Fuchs, 2007; Graesser et al., 1994). For example, although main idea strategy is supported by all the other strategies, main idea strategy can facilitate the use of other strategies such as retell (Stevens et al., 2019) and prediction (Fuchs et al., 2001; Nolan, 1991). Graphic organizers and retell in narrative texts often share a very similar structure and thus often mutually benefit each other (Douglas et al., 2011). Retell can facilitate the use the self-monitoring strategies to detect unclear and incoherent understanding (Brown et al., 2014). Inference and self-monitoring strategies are often applied together (van den Broek et al., 2001). Recognizing the text structure can help readers better understand when to use inference strategies, self-monitoring strategies, and graphic organizers for different types of text (Hebert et al., 2016; Meyer, 1987). Taken together, given the conceptual intercorrelations and the demanding cognitive load of learning and using reading comprehension strategies for students with RD, it is reasonable to assume that there may be an optimal strategy or strategy combination that can cover the function of most other reading comprehension strategies and reduce the repetitive instruction for overlapping concepts among strategies, while not tapping the cognitive load of learning and using these strategies too heavily.
Active Ingredient vs. Ingredient Interaction
Is there an “active-ingredient” or an “ingredient-interaction” theoretical framework in multiple-strategy reading comprehension instruction? Given the multicomponent nature of most reading interventions or programs, reading researchers often emphasize, or tend to implicate, an active-ingredient framework (Edmonds et al., 2009; Filderman et al., 2021; Wasik & Hindman, 2014). That is, instruction including a specific reading component may significantly increase the effects compared to instruction without such a component (e.g., Filderman et al., 2021; Wasik & Hindman, 2014). However, such an active-ingredient framework often assumes a null of the interaction effects among reading components. In contrast, with the “ingredient-interaction” theoretical framework, it is likely that a certain reading component exerts its biggest effects on reading only when it is combined with other reading components during instruction. For example, phonological processing is often considered at the core of decoding in English (e.g., Wagner & Torgesen, 1987; also see the debate in Castles & Coltheart, 2004), yet combining phonological processing with letter-sound instruction often produces better results on decoding than focusing on phonological processing alone (National Reading Panel, 2000), which is likely due to the interaction between phonological processing and letter-sound instruction. That is, letter-sound instruction can help alleviate the working memory load of applying phonological processing, boosting the effects of phonological processing on decoding in English (Ehri, 2003; National Reading Panel, 2000).
A similar logic may also apply to reading comprehension strategy instruction, given different reading comprehension strategies are conceptually related. Thus, reading comprehension strategy instruction for students with RD may be in line with the ingredient-interaction framework, and we may not expect a consistent effect for interventions including a specific strategy. Instead, we may expect to see a specific strategy combination that produces the best outcomes. However, if reading comprehension strategy intervention for students with RD follows an active-ingredient framework, we would expect to see interventions including a specific strategy, regardless of whether including other strategies or not, would almost always produce better reading comprehension outcomes than interventions without including this strategy. We aimed to examine these hypotheses in the current study.
Feasibility of Strategy Instruction in Practice
Investigating whether there is an optimal strategy or strategy combination may also have important implications for practice. Specifically, although multicomponent reading comprehension intervention is quite common nowadays, these interventions are often difficult to implement with fidelity by teachers in the classroom, which becomes one of the major factors contributing to the research-to-practice gap in (special) education (O’Donnell, 2008). Common reasons surveyed from teachers are the complexity and great length of many multicomponent interventions and differences between the way teachers perceive various interventions to be effective and the actual effectiveness of those interventions (e.g., McKenna et al., 2014). Thus, finding the optimal reading comprehension strategy or strategy combination can help teachers understand that not all strategies necessarily work well together from a cognitive load perspective. It is possible to save time by implementing some strategies under limited resources and time, which would greatly facilitate the implementation fidelity of the optimal strategies and strategy combination in the educational setting.
Prior Reviews
There are several reviews and meta-analyses suggesting the effectiveness of reading comprehension strategy instruction in the past decades. National Reading Panel (2000) focused on direct explanation strategies—which teachers instruct students to use independently—including comprehension monitoring, graphic organizer, story structure, question generation, summarization, multiple-strategy use, and interactive strategies that teachers and students apply together in reading activities, such as cooperative learning and teacher questioning. Findings suggested that strategy instruction leads to students’ awareness of strategies and use of strategies, which lead to improved reading, although the effects seemed to vary among different strategies (d = .32 ~ 1.15). Gersten et al. (2001) did a systematic review on studies of adopting a single reading comprehension strategy for students with learning disabilities. The authors suggested the potential effects of even a single strategy on improving reading comprehension but emphasized the need to explicitly teach multiple strategies together with background knowledge.
Hall (2016) did a synthesis of the effects of inference strategy for students in the elementary and intermediate grades. Findings suggested that compared to typically developing peers, struggling readers may benefit more from inference instruction for reading comprehension (g = −.03 ~ 1.96). Of note, some studies reviewed by Hall (2016) also taught students other strategies such as prediction and self-monitoring. Stevens et al. (2019) did a systematic review to examine the effects of main idea interventions among struggling readers in grades 3 to 12. The findings suggested main idea instruction may improve struggling readers’ main idea identification and reading comprehension (g = .97). Similar to Hall (2016), studies reviewed in Stevens et al. (2019) often tapped other strategies such as self-monitoring skills. Filderman et al. (2021) investigated interventions focused on either background knowledge, reading comprehension strategy, or both among struggling readers in grades 3 to 12. Findings demonstrated that these interventions were generally effective (g = .59), and the combination of background knowledge and strategy instruction or strategy instruction alone produced larger effects than background knowledge instruction only. Like Stevens et al. (2019) and Hall (2016), Filderman et al. (2021) also included studies with multiple strategies in the analysis to present the effectiveness of a specific strategy.
Taken together, all prior reviews on reading comprehension strategies adopted the active-ingredient framework and the traditional pairwise meta-analysis to code, analyze, and interpret their findings. These reviews did not consider the possible interaction effects among strategies when several strategies were taught together. That is, these reviews presented the effectiveness of a particular strategy from studies that included other strategies, completely neglecting the interactions among strategies. Moreover, these reviews did not attempt to rank/prioritize the most effective strategy or strategy combination that would be helpful for the field.
Bayesian Network Meta-analysis
In recent years, Bayesian network meta-analysis (BNMA) has been developed to help explore the possible interactions among intervention components (Greco et al., 2016; Miklowitz et al., 2021). BNMA builds on the mixed intervention comparisons rationale and utilizes both direct and indirect information for evidence synthesis, which addresses the limitations of traditional pairwise meta-analyses. That is, different interventions are compared one to another simultaneously by statistical inference methods rather than simply summing up studies that evaluated the same intervention compared to another intervention. For example, if a first trial compares the effects of strategy A to strategy B on the outcomes, showing that strategy A is significantly superior to strategy B, and a second trial investigates the same or a similar population comparing strategy B versus strategy C (demonstrating that strategy B is equivalent to strategy C), BNMA then allows us to infer that strategy A is also potentially superior to strategy C for this given population, even though there was no direct test of strategy A against strategy C. Thus, BNMA is often considered an alternative and optimal way to study interactions among intervention components when componential experimental designs are infeasible to implement for multicomponent interventions (Miklowitz et al., 2021). In our case, the effect of an intervention with one reading comprehension strategy or strategy combination on reading comprehension is incorporated in the network, which is used to more accurately estimate the effects of interventions with different strategies or strategy combinations on reading comprehension.
Moderators
Moreover, with BNMA, it is possible to detect the between-study heterogeneity and explore whether moderators can influence the magnitude of effect sizes found in the network (Harrer et al., 2021). In the current study, we included the following possible moderators: age or grade level, background knowledge instruction, text type, RD status, assessment type, intervention dosage, and study quality.
Specifically, it is suggested that older students with RD often tend to struggle most in reading because of the accumulated negative outcomes associated with low levels of reading with development (e.g., Vaughn et al., 2008) and reading intervention tended to yield smaller effects among older students with RD than younger students with RD (e.g., Torgesen et al., 2001). We also consider background knowledge instruction as another moderator. The rationale is that background knowledge is often considered an important component in reading comprehension interventions (e.g., Elleman et al., 2009; Filderman et al., 2021), because solid background knowledge can significantly increase the use of retrieval, which can reduce the cognitive load for the use of reading comprehension strategies during reading comprehension (Graesser, 2007). Thus, it is reasonable to assume that reading comprehension strategy interventions with a background knowledge component may yield larger effects than those without a background knowledge component. Regarding the text type, we explore whether using narrative texts or using expository texts or using both narrative and expository texts as intervention materials would exert an impact on reading comprehension strategy intervention. Compared to narrative texts, expository texts often involve less familiar text structures and are informationally dense with less-familiar academic or subject-specific vocabulary (Best et al., 2008; McNamara et al., 2012). Yet, there is also argument that narrative texts are equally challenging as expository texts, because narrative texts encountered by older students often have longer passages, more complex sentences, and less cohesion (Graesser et al., 2003; McNamara et al., 2012). Thus, text type may influence the cognitive load of strategy used in reading comprehension intervention. In addition, prior research suggests that different RD statuses may show different cognitive profiles (Cain & Oakhill, 2006; Carretti et al., 2009; Peng et al., 2018, 2022), which may interact with reading comprehension strategy that is cognitively demanding in general. Thus, to examine this possible RD status effect, we will further categorize RD into those identified by word reading test, by reading comprehension test, or by comprehensive reading tests or referred by school and special education teachers. We also looked at several methodological variables as potential moderators, including assessment type, intervention dosage, and study quality. Based on prior studies, we hypothesized that a higher percentage of standardized assessment to index outcomes (Clemens & Fuchs, 2021; Wanzek et al., 2016), lower intervention dosage (Roberts et al., 2021), and higher study quality (Austin et al., 2019) might lead to lower reading comprehension strategy instruction effects.
Aims
To sum, based on reading comprehension intervention studies and BNMA, we aimed to simultaneously compare, estimate, and rank different reading comprehension strategies and strategy combinations to answer two questions: (1) Is there an optimal reading comprehension strategy or strategy combination for improving reading comprehension among students with RD at 3rd ~ 12th grade? If yes, what is the strategy or strategy combination? (2) Do age or grade level, background knowledge instruction, text type, RD status, assessment type, intervention dosage, and study quality influence the effects of reading comprehension strategy? Based on the cognitive load theory of strategy use, we hypothesized that comprehension integration strategies such as main idea and retell tapping more cognitive resources may be more likely to be an active-ingredient strategy or appear in the optimal strategy combinations. Based on the conceptual interrelations hypothesis among strategies, strategies that share fewer concepts overlapping, when taught together, may utilize cognitive resources more efficiently and are more likely to produce synergistic effects than those sharing more overlapping concepts. However, as all strategies investigated in this study overlap with one another to some extent, we don’t have a specific hypothesis on such optimal strategy combinations.
Method
Literature Search
A computer search of EBSCOhost, including the databases of ERIC, MEDLINE, and PsycINFO for literature (including peer-reviewed articles, book chapters, and dissertations), was conducted. We used the earliest possible start date (1975) through December 2021 to do keywords search in the abstract. The primary search terms included reading OR text OR passage OR story. The secondary search terms included comprehen* OR meaning OR semantic* OR understand*. These terms were selected to identify studies focused on reading comprehension. The tertiary search terms included difficult* OR “learning disab*” OR “reading disab*” OR “at risk” OR “struggl* reader” OR dyslex* OR “special education” to identify studies investigating the effects of reading comprehension interventions for students with or at risk for RD. The final search terms captured the method of instruction and included instruct* OR interven* OR “pull out” OR teach* OR program* OR “tier 2” OR “tier 3” OR “small group” OR curricul* OR lesson OR strateg* OR “graphic organiz*” OR retell* OR “story map” OR “main idea ” OR inferenc* OR summar* OR predict* OR visualiz* OR vocabulary OR “background knowledge” OR paraphras* OR “text structure” OR “story grammar.” We also did the ancestry search and hand search via contacting potential authors for implementing possible reading comprehension strategy interventions. In addition, we searched in the abstracts of all available volumes of 15 relevant peer-reviewed journals, including Journal of Educational Research, Journal of Literacy Research, Journal of Research in Reading, Reading Research Quarterly, Journal of Learning Disabilities, Exceptional Children, Reading and Writing: An Interdisciplinary Journal, Reading and Writing Quarterly, Journal of Research on Educational Effectiveness, Remedial and Special Education, Research in the Teaching of English, Scientific Studies of Reading, American Educational Research Journal, Contemporary Educational Psychology, Learning Disability Quarterly, The Elementary School Journal, Journal of Education Psychology, and The Journal of Experimental Education.
The search yielded a total of 29,213 abstracts, with 26,470 abstracts remaining after duplicates were removed. The ancestry and hand search ended up with another 8 abstracts. We screened all abstracts and evaluated full texts of the studies that met inclusion criteria: (1) studies must be written in English but can include students speaking languages other than English; (2) studies must utilize experimental and quasi-experimental designs providing treatment and comparison to determine the experimental effect for struggling readers; (3) participants were identified as struggling readers in grades 3 through 12 (struggling readers were described by authors as at risk for or having reading disabilities / difficulties / problems / disorders); (4) studies included a dependent variable addressing reading comprehension (including text comprehension and vocabulary) on either a researcher-designed measure or a standardized measure; (5) studies included at least one reading comprehension strategy of main idea, text structure, prediction, retell, inference, graphic organizers, and self-monitoring; and (6) studies excluded reading interventions tapping skills other than reading comprehension, such as phonics and fluency, to avoid or reduce potential confounding effects of these reading interventions on reading comprehension strategy intervention effects. The first, third, and fourth author reached a 90% reliability (Agreements/(Agreements + Disagreements)] × 100) on the screening process before the third and fourth author independently screened abstracts and evaluated full texts of the studies that met inclusion criteria. The flow chart of literature screening is presented in Figure 1.
Flow chart of literature identification.
Coding Procedure and Inter-Rater Reliability
Based on the previously mentioned criteria, 52 studies were included in the final review (i.e., 12 dissertations and 40 peer-reviewed articles). Studies were coded according to the characteristics of participants and tasks used to measure reading. Not all studies provided sufficient information on the variables of interest for the present study. In case of insufficient information, authors were contacted to obtain the missing information. In addition to these variables, we also coded the number of participants (n) used to obtain each group.
Variables were discussed until a consensus was reached between the first and second authors. Then, the first and the fourth authors used this coding system to conduct the final coding of all studies (Table 1). Across the total variable matrix, the mean interrater agreement was .96, with acceptable values for all codes as follows: 1.00 for age or grade level, .95 for background knowledge instruction, .96 for text type, 1.00 for RD status, .92 for assessment type, .95 for intervention dosage, .93 for study quality, and .92 for reading comprehension strategies. We rated study quality based on research design, comparison-group instruction, and implementation fidelity, which is often adopted in prior special education meta-analyses (Austin et al., 2019; Filderman et al., 2021; Stevens et al., 2021). Specifically, we rated each study as high, medium, or low across each quality indicator. For research design, high indicates the use of a randomized design, medium indicates a quasi-experimental design with evidence of pretest equivalency, and low indicates a quasi-experimental design without evidence of pretest equivalency. For comparison-group instruction, high indicates that comparison-group instruction was provided and defined as small-group reading intervention, medium indicates that comparison-group instruction is typical school instruction with minimal description, and low indicates that no instruction was provided in the comparison condition. For fidelity of implementation, high indicates clear and replicable operational definitions of treatment procedures, with procedural fidelity ≥75% and interobserver reliability ≥.90; medium indicates operational definitions of treatment procedures were provided, with procedural fidelity ≥75% and interobserver reliability ≥.80; and low indicates the description of the intervention was insufficient such that the replication is impossible, with no reported procedural fidelity, or procedural fidelity <75%, and no reported interobserver reliability or interobserver reliability <.80. High equals 3 points, medium equals 1 point, and low equals 0 when constructing the overall quality of a study (i.e., mean score across three quality indicators). Any disagreements between raters were resolved by consulting the original article or by discussion. The description of the reviewed studies is presented in Table 2.
Description of codes
Description of studies
Note. G = Graphic organizers; I = inference; M = main idea; P = prediction; R = retell; S = self-monitoring; T = text structure.
Missing Data
After reaching out to the authors of reviewed studies, we still had some missing data on moderators, including age (1 missing), type of text used for intervention (11 missings), and intervention dosage (1 missing). Because currently, the BNMA method requires no missingness for the moderation analysis, we used the list-wise deletion for the studies that did not have the moderator value. Specifically, for moderation analysis on age and intervention dosage, we included 51 studies. For moderation analysis of the type of text, we included 41 studies.
Effect Size Calculation and Publication Bias
We extracted the mean and standard deviation of post-test reading comprehension outcomes (usually continuous variables), and sample size for both the treatment and control groups for each study. We then calculated the Hedges’s g between treatment and controls to indicate the effect size for each reading comprehension outcome and used all eligible effect sizes in each study. However, because currently, the BNMA method cannot handle dependency effects (only allows one effect size for group comparison [strategy vs. control or strategy vs. strategy] in one study), we synthesized multiple effect sizes within each group comparison into one effect size for either a specific strategy vs. control comparison or specific strategy vs. strategy comparison based on fixed effects within each study using metaan in Stata (Kontopantelis & Reeves, 2010).
For publication bias examination (the problem of selective publication, in which the decision to publish a study is influenced by its results), we used the method of Egger et al. (1997). We did not find significant publication bias based on Egger et al.’s (1997) publication bias statistics (i.e., the standard errors did not significantly predict effect sizes among studies with robumeta in Stata, p = .67). Thus, the original dataset was used in all reported BNMA analyses as described in the next section.
Bayesian Network Meta-Analysis (BNMA)
The BNMA approach (aka, mixed-treatment comparison meta-analysis; van Valkenhoef et al., 2012) is contrasted to the traditional meta-analysis method in its ability to more accurately estimate the effectiveness of multiple intervention strategies simultaneously and comparatively. More specifically, the BNMA pools all available information from a group of primary studies to formulate both direct and indirect evidence in one meta-analytical model. With BNMA, we are not only able to estimate indirect evidence among all the involved intervention strategies—which is otherwise impossible by traditional meta-analyses—but more importantly can increase the estimation precision of the true effect size of an intervention strategy with the inclusion of indirect evidence.
Very few studies can directly compare the effectiveness of two or more strategies (e.g., due to logistics limitations). Thus, BNMA allows us to use indirect evidence of the relative effectiveness of different strategies in different studies that involve a similar or the same control group to build a connected system—“network” (Figure 2). For example, if Strategy A is directly compared to a control group in one study, we may graphically illustrate the direct evidence in Figure 2a, where the black dot denotes Strategy A and the white dot denotes the control group, and the line linking the two dots represents the corresponding effect size. In the network graph theory, the dots are called nodes, the link is called an edge, and the strategy A - B link is called an arm. Similarly, in a separate study, Strategy B is compared to the same or a similar control group, and the direct evidence may be illustrated in Figure 2b. Because the studies involve the same or a similar control group, according to the transitivity assumption of the network meta-analysis method (Efthimiou et al., 2016), we may further combine the direct evidence shown in Figure 2a and b to create a complex network to formulate the indirect evidence for the comparative efficacy between Strategy A and Strategy B, as shown in Figure 2c, where the dotted line represents the indirect evidence. If there are more than two strategies for an intervention, the hypothetical network would be more complex, as shown in Figure 2d. Of note, to build a network, it is important to meet the transitivity assumption, which is defined as the extent to which the relative effect (e.g., Strategy A vs. control) based on direct evidence does not significantly differ from the one based on the indirect evidence (Schwarzer et al., 2015).
Illustration network meta-analysis transitivity and formulation: (a) the direct comparison between strategy A and control, (b) the direct comparison between strategy B and control, (c) the network with strategy A, B, and control, and (d) the network with strategy A, B, C, D, E, F, and control.
In the current study, we used the R package gemtc (van Valkenhoef et al., 2012)—under a Bayesian hierarchical framework—to perform the BNMA estimation. The Bayesian approach—contrasted to frequentist statistics—incorporates prior knowledge (i.e., what we have already known about an event) to make inferences and produce better estimates of parameters. In addition, the Bayesian approach does not have to require the estimated parameters to follow a normal distribution, making the estimation method more robust than the frequentist approach.
As the outcome variable was continuous in this study, the likelihood function in the BNMA model was specified as
Prior Distribution Choices for Heterogeneity and Sensitivity Analysis
The Bayesian approach to NMA estimates unknown parameters with a random effect related to a predefined prior distribution (Rosenberger et al., 2021). As noted by Rosenberger et al. (2021), the heterogeneity variance (
Parameter Estimation
To accurately estimate the parameters in our BNMA, we conducted a Markov Chain Monte Carlo simulation with four parallel chains by the Gibbs sampling algorithm implemented in the rjags R package (Plummer et al., 2019). Each of the Markov chains generated 50,000 iterations and discarded the first initial 10,000 iterations in the burn-in period. Convergence was first assessed by the trace plots of the historical trajectory and the corresponding density plots. Visual inspection revealed there were rapid up-and-down variations without long-term trends in the trace plots, and the density plots showed classic bell-shaped curves, both of which indicated the model converged. In addition, we further assessed convergence by the Brooks-Gelman-Rubin diagnostic technique (Brooks & Gelman, 1998), with the Gelman-Rubin plots showing the potential scale reduction factor (PSRF) scores that compared the variation within each Markov chain to the variation between Markov chains. The overall PSRF score was 1.0027, below the critical 1.05 value, suggesting model convergence was achieved. Further, the consistency was assessed by the node-split method developed by Dias et al. (2010). The node-split plot showed the effects of different comparisons when using (1) direct evidence only, (2) indirect evidence only, and (3) both direct and indirect evidence that was available in a network. All the p-values were above .05, indicating the transitivity assumption of our BNMA was met.
After the model convergence and transitivity were achieved, we produced results of the efficacy of different intervention strategies in terms of both the standard mean difference (SMD) along with the corresponding 95% confidence intervals, and the surface under the cumulative ranking curve scores (SUCRA; Salanti et al., 2011), which indicated how likely an intervention strategy would be evaluated as the most efficacious and ranked them from best (100%) to worst (0%). We also visualized the network of all direct and indirect evidence of the intervention strategies examined in the current network meta-analysis.
Because the current BNMA could not handle multiple moderators in one meta-regression model, we conducted the network meta-regression for each moderator (age or grade level, background knowledge instruction, text type, RD status, assessment types, intervention dosage, and study quality), respectively. Last, we ran correlations between the number of strategies in a strategy combination and its corresponding effect sizes to explore the question: whether the more strategy we teach, the better outcomes we expect. All the analyses involving MCMC were set with initial seeds to ensure reproducibility, and the analysis code and data have been uploaded to OSF1 to share with readers.
Results
We first checked the model fit with different informative priors. The model with noninformative priors generated DIC of 357.39; models of weakly informative priors generated DIC indices 129.77, 129.27, 129.54, 129.53, and 129.56 for priors uniform (0, 1), uniform (0, 2) uniform (0, 3), uniform (0, 4), and uniform (0, 5), respectively. The model with a strong informative prior generated a DIC of 129.67. Thus, based on the smallest DIC, the model from weakly informative priors uniform (0, 2) is considered the best-fitting model, and we reported the results from the prior uniform (0, 2) as the primary analysis in the following section. Technical information of B-MNA based on prior uniform (0, 2), such as sample trace plots, Gelman-Rubin plot for the potential scale reduction factor scores, and node-splitting analysis of inconsistency, were reported in Figures S1–S3 in the supplemental materials (online only). In addition, we ran the models with other noninformative and informative priors, all of which yielded similar results patterns to the model with prior uniform (0, 2).
The estimated amount of between-study heterogeneity in the network is small τᴧ2 = .03, which is in line with our transitivity test. This may be because we focus on a specific population, on only reading comprehension strategy intervention, and differentiated various combinations of reading comprehension strategies, which restricted the between-study heterogeneity. We then visualized the network evidence of all the 35 intervention strategies and the control group. As shown in Figure 3—where the size of the nodes denoted the number of studies for each intervention strategy or the control group, and the width of the lines (i.e., edges) connecting the nodes denoted the number of studies per pairwise comparison—and in Table 3, the most strategies or strategy combinations were graphic organizers (G; 7 arms, based on 26 effect sizes) and main idea-prediction-self monitoring combination (MPS; 6 arms, based on 43 effect sizes), followed by main idea (M; 5 arms, based on 42 effect sizes) and main idea–self monitoring combination (MS; 4 arms, based on 28 effect sizes), main idea–prediction combination (MP; 3 arms, 37 effect sizes), retell (R; 3 arms, 14 effect sizes), and main idea-text structure-prediction-self monitoring-graphic organizers combination (MTPSG; 3 arms, 8 effect sizes) involving 3 arms, as well as main idea-text structure-self monitoring-graphic organizers combination (MTSG; 2 arms, 10 effect sizes), main idea-text structure-graphic organizers combination (MTG; 2 arms, 22 effect sizes), main idea-inference-prediction combination (MIP; 2 arms, 7 effect sizes), text structure–graphic organizers combination (TG; 2 arms, 18 effect sizes), and main idea-inference-text structure-prediction-self monitoring combination (MITPS; 2 arms, 6 effect sizes) involving 2 arms. All other strategies or strategy combinations only involved 1 arm (ranging from 1 effect size to 6 effect sizes). On average, the sample size for each comparison ranged from 12 to 540, with about an average of 108 for each comparison. There were an average of 1.84 arms (ranged 1~7, with a median of 1) across all the strategies and strategy combinations, which provided adequate power in BNMA research with a relatively large number of components (Valentine et al., 2010).
Network of evidence of all the intervention strategies.
Efficacy of different intervention strategies compared to the control group
Note. 95Cl = 95% confidence interval; SUCRA = the surface under the cumulative ranking curve; G = graphic organizers; I = inference; M = main idea; P = prediction; R = retell; S = self-monitoring; T = text structure.
The Most Effective Strategy or Strategy Combination
Based on the results of the BNMA shown in Table 3, the main idea-text structure-retell (MTR) strategy combination was found to be the most effective (SMD = 1.72, 95CrI: 0.29, 3.15), followed by the main idea-text structure-self monitoring-graphic organizers combination (MTSG; SMD = 1.13; 95CrI: 0.04, 2.22), main idea strategy (M; SMD = 1.07; 95CrI: 0.39, 1.76), main idea-self monitoring combination (MS; SMD = 1.06; 95CrI: 0.31, 1.79), and graphic organizers combination (G; SMD = 0.83; 95CrI: 0.25, 1.43). All of the five strategies were statistically significant superior to the control group. These findings were also confirmed by the SUCRA scores, ranking them as 89%, 76%, 77%, 76%, and 63% most efficacious, respectively. In contrast, the combinations of IT (SMD = −0.61; 95CrI: −2.14, 0.93), RG (SMD = −0.03; 95CrI: -1.70, 1.63), MITPS (SMD = 0.05; 95CrI: −1.03, 1.14), TRS (SMD = 0.10; 95CrI: −1.36, 1.54), and MI (SMD = 0.13; 95CrI: −1.42, 1.67) seemed to be the least efficacious among the 35 intervention strategies and strategy combinations, with SUCRA scores of 15%, 32%, 31%, 35%, and 37%, respectively. In addition, we ran a correlation between the number of strategies in each strategy combination and the corresponding SUCRA and SMD, r = .29 and .04, ps > .05, indicating that the number of strategy instructions in an intervention did not influence the reading comprehension outcomes.
Bayesian Network Meta-Regression for Moderation Analysis
The Bayesian network meta-regression showed that the moderating effect of background knowledge instruction was statistically significant (B = 2.69; 95CrI: 0.24, 8.37), indicating that having background knowledge instruction significantly increased the strategy effects overall. We further conducted forest plots to show the SMD and the corresponding 95% credibility intervals for the 35 intervention strategies and strategy combinations between groups with vs. without background knowledge instruction. As shown in the left panel of Table 4, when there was background knowledge instruction in an intervention, many strategies or strategy combinations were found to be statistically effective. Specifically, the MTR combination was again revealed to be the most efficacious with background knowledge instruction (SMD = 3.27; 95CrI: 1.00, 9.75), followed by the MTSG combination (SMD = 3.07; 95CrI: 0.83, 10.15), the inference strategy (SMD = 2.85; 95CrI: 0.23, 10.45), the IG combination (SMD = 2.42; 95CrI: 0.29, 8.19), the MIP combination (SMD = 2.17; 95CrI: 0.20, 8.23), the MPS combination (SMD = 2.06; 95CrI: 0.34, 7.60), the MS combination (SMD = 2.05; 95CrI: 0.75, 5.85), the MP combination (SMD = 1.97; 95CrI: 0.32, 7.14), the main idea strategy (SMD = 1.96; 95CrI: 0.79, 5.31), the retell strategy (SMD = 1.80; 95CrI: 0.23, 6.67), the MTG combination (SMD = 1.79; 95CrI: 0.12, 6.61), and the graphic organizers strategy (SMD = 1.70; 95CrI: 0.61, 4.92). In contrast, when there was no background knowledge instruction in an intervention (see the right panel of Table 4), none of the strategies or strategy combinations showed statistically significant efficacy. In addition, we ran a correlation between the number of strategies in each strategy combination and the corresponding SMD with and without background knowledge instruction, r = .26 and .25, ps > .05, indicating that the number of strategy instructions in intervention does not influence the reading comprehension outcomes. Other than the background knowledge instruction, none of the other moderators examined in the primary analysis was statistically significant based on the results of the Bayesian network meta-regression models, including age or grade level (B = 1.42; 95CrI: −5.72, 12.45), assessment type (B = 2.89; 95CrI: −4.56, 8.97), study quality (B = 0.62; 95CrI: −0.74, 7.47), RD status (Bs = 4.18/0.01/−2.78, 95CrI: −0.81, 7.95 / 95CrI: −2.04, 1.25 / 95CrI: −9.90, 8.08—word reading screening vs. others / reading comprehension screening vs. others / comprehensive reading screening vs. others), text type (B = −1.25; 95CrI: −4.98, 0.87), and dosage (B = −0.93; 95CrI: −4.39, 0.23).
The moderating role of background knowledge instruction on the efficacy of different strategies/strategy combinations
Note. G = Graphic organizers, I = inference, M = main idea, P = prediction, R = retell, S = self-monitoring, T = text structure
Discussion
The current study applied BNMA to explore if there is an optimal reading comprehension strategy or strategy combination for students with RD from grade 3 through grade 12 and to explore possible moderators that could influence the strategy intervention effects. Based on BNMA, the main idea-text structure-retell combination, the main idea-text structure-self-monitoring-graphic organizers combination, and main idea were the top three most effective strategies and strategy combinations. With Bayesian network meta-regression, there seemed to be more effective strategies and strategy combinations when interventions included background knowledge instruction, with the top three most effective strategies and strategy combinations as the main idea-text structure-retell combination, the main idea-text structure-self monitoring-graphic organizers combination, and inference. Without the background knowledge instruction, the effects of reading comprehension strategies significantly decreased. Taken together, the main idea-text structure-retell combination seemed to be the most effective strategy combination. Yet, there seem to be unclear findings on the most effective single strategy across BNMA and the Bayesian network meta-regression. In addition, we did not find significant moderation effects for age or grade level, text type, RD status, assessment type, intervention dosage, and study quality. Moreover, the number of strategies in a combination did not correlate with the effect sizes of reading comprehension. Next, we discussed these findings in detail.
Cognitive Load of Strategy
Given reading comprehension strategies are generally cognitively demanding for struggling readers and the inter-correlations among different reading comprehension strategies, it was important to investigate a possible optimal reading comprehension strategy or strategy combination that could have the best effects on reading comprehension but not to tap the cognitive load of learning heavily. Yet, common componential experimental designs are often unfeasible for this question. Based on BNMA, our findings supported an optimal strategy combination hypothesis (Afflerbach, 1990; McNamara, 2007). That is, we found main idea, text structure, and retell, when learned and applied together as the primary strategies, produced the maximum effects on reading comprehension outcomes among struggling readers. One explanation is that main idea and retell may serve as umbrella strategies that often tap the functions of other strategies. For example, the application of retell often includes the use of prediction and graphic organizers. To summarize the main idea or retell, readers often need to apply inference strategies and self-monitoring to check the coherence of main idea and retell. The use of text structure can help readers better plan and organize their reading comprehension strategies to more efficiently use cognitive resources during reading comprehension (e.g., reducing switching and inhibition resources on nonrelevant information processing not aligned with the text structure).
Taken together, with the use of text structure, readers can better allocate their cognitive resources to important and cognitively demanding strategies such as main idea and retell to achieve comprehension (e.g., Hebert et al., 2016). That said, our findings do not suggest only teaching these strategies to students with RD. Instead, we think the findings are more interpretable or feasible in practice if main idea, text structure, and retell are the primary strategies emphasized during instruction, whereas other strategies could be taught as supplements if needed. For example, if students’ struggle in inference making interferes with their learning and application of main idea or retell, then inference making should be explicitly taught to facilitate the practice of main idea and retell.
We also noticed that many strategy combinations that included three or more strategies did not necessarily produce bigger effects than strategy combinations with a smaller number of strategies. We even noticed some five- or six-strategy combinations showed smaller effects on reading comprehension than a single-strategy intervention or intervention with fewer strategies instruction. Indeed, we did not find a significant correlation between the number of strategies in a combination and their corresponding effect sizes on reading comprehension. This finding, again, is in line with the cognitive load theory of strategy use. That is, teaching “too many” strategies in a reading comprehension intervention may bring extra cognitive load for struggling readers who often have limited cognitive capacity. Even if struggling readers can successfully learn many strategies in an intervention (as indicated by the relatively high-fidelity implementation data of many interventions reviewed in the current study), they may still have difficulties independently applying these strategies to reading comprehension tasks because their limited cognitive resources do not allow them to flexibly choose the right strategies and inhibit irrelevant ones (Booth et al., 2010; Peng & Fuchs, 2016). Alternatively, it is also possible that packaging “too many” strategies in an intervention or program may likely reduce the time of students practicing each strategy. The intervention dosage reported in the reviewed studies for those five- or six- strategy combination treatments ranged from 7.5 hours to 35 hours, with a mean of less than 23 hours. Maybe with sufficient practice on those strategies, the cognitive load of flexibly using different and appropriate strategies during reading comprehension may become smaller and thus lead to bigger effects on reading comprehension (Graesser, 2007). Future studies are needed to find this optimal intervention and practice dosage for those multistrategy reading comprehension interventions (e.g., Roberts et al., 2021).
Active Ingredient vs. Ingredient Interaction
Another important theoretical debate we aimed to address is whether strategy instruction follows an active-ingredient hypothesis or an ingredient-interaction hypothesis. Specifically, based on the active-ingredient hypothesis, interventions including a specific reading comprehension strategy would be more likely to produce bigger effects on reading comprehension than interventions without such a strategy. In contrast, the ingredient-interaction hypothesis suggests no single reading comprehension strategy can be considered as the best strategy. Due to interaction relations among strategies, different strategy combinations may produce different effects on reading comprehension. Our findings based on BNMA and Bayesian network meta-regression including background knowledge instruction reached a different conclusion on the most important single strategy (main idea in BNMA vs. inference in Bayesian network meta-regression). Moreover, we found those much less (or the least) effective strategy combinations also included main idea or inference. Taken together, reading comprehension strategy instruction is more aligned with the ingredient-interaction framework, not the active-ingredient framework. That is, there is no “the most important” single strategy. Rather, main idea, text structure, and retell, when combined, seem to produce the maximum interaction effects on reading comprehension among students with RD.
However, we also found that background knowledge was a significant moderator for our network. That is, for interventions without background knowledge instruction, we did not find any strategy or strategy combination as effective in improving reading comprehension. In contrast, more strategies or strategy combinations were significantly effective when background knowledge instruction was included in the interventions. This finding further contributed to the ingredient-interaction hypothesis of reading comprehension strategy instruction. For students with RD, there is a background knowledge-strategy interaction effect on reading comprehension improvement. Background knowledge may be an active ingredient for reading comprehension (e.g., Elleman et al., 2009; Filderman et al., 2021), whereas reading comprehension strategy instruction alone is not such an active ingredient. However, the effects of strategy instruction can be greatly enhanced when background knowledge instruction is included.
One explanation for such a background knowledge-strategy interaction may be the cognitive load of strategy use. Reading comprehension is the most comprehensive and structurally complex reading skill in the reading domain. Students need to use various sources of assistance to avoid overloading working memory during the reading comprehension process. Those with strong background knowledge often automatically and directly retrieve background knowledge or spend relatively less working memory on the direct retrieve of background knowledge for comprehension. In contrast, those with weak background knowledge often need to spend more working memory in making sense of different parts of a passage, such as word reading, grammar, and syntax for comprehension (e.g., Cain et al., 2004). Thus, it is not surprising that reading comprehension strategy use, which is already cognitively demanding, would be more efficient with the assistance of background knowledge but not so efficient or may even pose an extra cognitive load to reading comprehension if without the assistance of background knowledge.
That is, for struggling readers who have a limited cognitive capacity (Willingham, 2006), without background knowledge instruction, the strategy instruction may seem quite cognitively demanding and requires a lot of practice (e.g., longer intervention than reported in any of our reviewed studies) to be beneficial. Indeed, our findings from BNMA and Bayesian network meta-regression partly support this explanation. Based on BNMA, the main idea-text structure-self-monitoring-graphic organizers combination (SUCRA = 77%) seems inferior to the main idea-text structure-retell combination (SUCRA = 89%) that has fewer strategies. Yet, with Bayesian network meta-regression including the background knowledge instruction, the main idea-text structure-self-monitoring-graphic organizers combination and the main idea-text structure-retell combination seems more equivalent.
Other Moderators
We did not find age or grade level, text type, RD status, assessment types, intervention dosage, and study quality as significant moderators on the network. These insignificant moderators may indicate the rank order for the importance of different reading comprehension strategies and strategy combinations is relatively stable across age or grades, between narrative vs. expository texts, among different RD types, between standardized and researcher-developed measures, intervention dosages, and study quality. Moreover, these null moderation findings may suggest the reading comprehension strategies and strategy combinations are in general effective across different situations, in line with the findings from National Reading Panel (2000), and more importantly, highlight the unique role of background knowledge in enhancing the effectiveness of reading comprehension strategy instruction among struggling readers.
Limitations
Our findings should be explained with the following limitations in mind. First, the reading comprehension strategies chosen for the present study are primarily based on the prior literature without a specific framework (National Reading Panel, 2000). We included “strateg*” in our search term to catch all possible strategies so we can create more categories as needed. However, due to the sample restriction (only struggling readers in grades 3–12), we did not have all possible strategies included in our reviewed studies such as the mental imagery strategy (Pressley et al., 1989). In addition, we focused on reading comprehension interventions, excluding intervention studies that included other literacy components. Thus, studies with a primary focus on the “writing” component were excluded from our review. We did not exclude studies that used writing as a format for the use of reading comprehension strategies (e.g., main idea and graphic organizers; Graham & Hebert, 2011). However, we did not specifically code and study the writing strategies since the writing component is not systematically searched and reviewed, and it is difficult to figure out whether the study used writing strategies based on the limited description of the intervention. Thus, future studies including typically developing students, students with more severe or low-incidence disabilities, and students in lower grade levels may be able to include more diverse strategies, such as mental imagery and writing-oriented strategies.
Second, we coded the reading screening measures for our sample but could not specify the types of RD based on this screening information. That is, most studies used only one reading measure in one domain (e.g., reading comprehension or word reading) or a comprehensive reading test, which made it difficult to categorize struggling readers into those with only word-reading problems or those with only reading comprehension difficulties. Future studies may want to investigate whether the type of RD may influence our results.
Third, processing speed in general (Christopher et al., 2012) and in the reading context (decoding fluency and comprehension speed; Therrien, 2004) are important for reading comprehension. Slow processing speed is often a marker of students with RD (Peng et al., 2022), which may impede reading comprehension by introducing extra cognitive loads during reading comprehension. That is, slow word reading or comprehension speed may limit the amount of information from texts entering into working memory, decrease the efficiency of combining background knowledge with processed text information, and increase the possibility of decay of information held in working memory. Due to limited information, we could not investigate whether reading fluency influences our network results. Future studies can explore whether the optimal strategy combination from the current study also applies to slow readers.
Fourth, we did not limit our search for only English intervention, but we were only able to search in research written in English. Thus, the majority of studies reviewed were interventions on English. We have one study on German, one on Dutch, one on Spanish, and one on Greek. Because we had such a small number of studies in languages with relatively more transparent orthography than English, we were unable to run the moderation to test whether orthography transparency would influence our results. More studies from more transparent languages are needed to validate our results. It is likely our findings on effective reading comprehension strategies/combos may not apply to transparent orthographies, which may well facilitate comprehension because of greater predictability and lower cognitive load in working memory.
Alternative Methodological Considerations
We also noted several methodological limitations and considerations. Although we included studies from published and gray literature, we might have potentially missed some studies given that forward and backward citation search and the contact of relevant researchers on our topic was not conducted. In addition, currently BNMA does not handle the multiple-moderator model. We also tried to run sensitivity analyses by excluding studies with two or more low-quality indicators. However, after excluding five of these low-quality studies, our network model became disconnected (with some stand-alone strategy or strategy combinations not connected to controls or other strategies) and thus the network model could not converge. Moreover, the way we used the fixed effects model to synthesize mean effect sizes within each independent sample is not ideal, which may bring noise to our analyses, although we did not detect the effects of these noises (using the within-study heterogeneity as a covariate) in the present study.
Further, our BNMA approach may not provide the most robust inference about the main-effect and interaction-effect estimation. Specifically, the frequentist componential network meta-analysis (CNMA) is currently the only method, according to our knowledge, that can test the factorial design model in NMA (Rücker et al., 2020). There are two approaches in CNMA. One is to only estimate the main effect of each component in a network, thus called the additive CNMA. For example, for a study that involves four components: A, B, C, and D, to test the combined effect of all components in a factorial design, one only needs to estimate the main effects of A, B, C, and D, respectively. The combined effects of ABCD simply equals to the sum of A, B, C, and D. The other approach is the interaction CNMA, estimating the main effect of each component as well as the interaction effects of various component combinations. For example, for a study that involves four components—A, B, C, and D—to test the combined effect of all components in a factorial design, one needs to estimate all main effects, all the possible two-way and three-way interactions, and one four-way interaction. Thus, the effect of ABCD is A + B + C + D + A*B + A*C + A*D + B*C + B*D + C*D + A*B*C + A*B*D + B*C*D + A*C*D + A*B*C*D, which involves 15 effects in total.
We indeed ran the frequentist CNMA to compare three nested models: the standard model (treating each strategy combination as a unique node in the network, similar to our BNMA approach), the additive model, and the interaction model. Because of data limitations, we were only able to include the interaction CNMA with two-way interactions, omitting all three-way and higher interactions. The comparison between models is based on Q statistics that follow a chi-square distribution (Rücker et al., 2020). The standard model is the most complex one with the most parameters to estimate (df = 29), the interaction model is the less complex one (df = 49), and the additive model is the most parsimonious one (df = 57). Based on the Q-statistic, the standard model explained the data better than the additive model, ΔQ = 4119.49, Δdf = 28, p < .0001, and the interaction model ΔQ = 1567.55, Δdf = 20, p < .001. The interaction model explained the data better than the additive model, ΔQ = 2551.94, Δdf = 8, p < .0001. Taken together, based on the frequentist CNMA, the standard model explained our data the best. This finding suggested that the standard model, treating each strategy combination as an independent unit, is the most feasible approach to reflect possible interaction effects among strategies based on our data.
In comparison to the standard model based on the frequentist approach, the Bayesian approach we adopted is more powerful as we could incorporate prior knowledge (i.e., what we have already known about an event) to make inferences and produce better estimates of parameters. The Bayesian approach also does not have to require the estimated parameters to follow a normal distribution, making the estimation method more robust than the frequentist approach. Thus, with all these points taken into consideration, we decided to only report the findings from our Bayesian approach in the results section. That said, our current conclusion about the ingredient-interaction hypothesis may further warrant the interaction CNMA approach that requires a significantly larger number of studies in the future.
Theoretical and Practical Implications
With all those limitations in mind, the current study is the first one to apply the BNMA to investigate the effectiveness of different reading comprehension strategies and strategy combinations. Findings provided new and important information for our understanding of reading comprehension strategy instruction for students with RD. Specifically, we noted that in Filderman et al. (2021), the effects of individual strategies were reported on main idea, inferencing, text structure, and graphic organizer, with gs = .72, .56, .47, and .46, respectively. In comparison, the effects we estimated for these strategies were gs = 1.07, .94, .13, and .83, respectively. Based on these comparisons, ours are similar to but not completely aligned with Filderman et al. (2021). In Filderman et al. (2021), the estimation of effects for each individual strategy were based on studies that also tapped instruction of other strategies, completely ignoring the interaction effects among strategies. Thus, using the BNMA approach to differentiate each individual strategy from strategy combinations may provide more accurate estimates on the reading comprehension strategy effects.
In addition, there is no “the most important” or “active ingredient” reading comprehension strategy. Instead, different reading comprehension strategies are conceptually related and interact with one another to produce different effects based on different combinations. On the other hand, there seemed to be a background knowledge-strategy interaction on reading comprehension among students with RD. The effects of strategy learning and application can be greatly enhanced by including background knowledge instruction that can help reduce the cognitive load of strategy use. These two points, taken together, suggest that reading comprehension strategy instruction for struggling readers in practice does not follow “the more the better” principle; main idea, text structure, and retell together may best optimize the cognitive load of using these strategies and produce the optimal effects on reading comprehension. Yet, focusing on reading comprehension strategy instruction alone in a relatively short period of intervention time may not be ideal, because the inherent heavy cognitive load in learning and applying reading comprehension strategies may possibly impede reading comprehension. Thus, it is important to include background knowledge instruction to facilitate knowledge retrieval and reduce the cognitive load of using reading comprehension strategies for students with RD.
Supplemental Material
sj-docx-1-rer-10.3102_00346543231171345 – Supplemental material for The Active Ingredient in Reading Comprehension Strategy Intervention for Struggling Readers: A Bayesian Network Meta-analysis
Supplemental material, sj-docx-1-rer-10.3102_00346543231171345 for The Active Ingredient in Reading Comprehension Strategy Intervention for Struggling Readers: A Bayesian Network Meta-analysis by Peng Peng, Wei Wang, Marissa J. Filderman, Wenxiu Zhang and Lifeng Lin in Review of Educational Research
Footnotes
Funding
This research was supported by R324A220268 from the Institute of Education Sciences.
Notes
Authors
PENG PENG is an assistant professor in the Department of Special Education at The University of Texas at Austin, 1912 Speedway, Stop D5000, Austin, TX 78712, USA; email:
WEI WANG is an associate professor at the Graduate Center, City University of New York, Cuny Graduate Center, 365 Fifth Ave, New York, NY 10016; email:
MARISSA J. FILDERMAN is an assistant professor in the Department of Special Education at University of Alabama, 520 Colonial Dr, Tuscaloosa, AL 35401; email:
WENXIU ZHANG is a doctoral student in the Faculty of Education at Beijing Normal University, No. 19, Xinjiekou Outer Street, Haidian District, Beijing, P. R. China; email:
LIFENG LIN is an associate professor in the Department of Epidemiology and Biostatistics at the University of Arizona, 1295 N. Martin Ave. Tucson, AZ 85724; email:
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
