Abstract
The ability to read in a second language (L2) for academic purposes is essential for higher education students. Dutch colleges increasingly use materials in English or teach in English. This can be challenging for L2 readers, especially students entering higher education from vocational studies, who may have less experience with L2 academic reading. Teaching L2 reading programmes containing explicit instruction of reading strategies may benefit higher education students in L2 academic reading, particularly since reading strategies learned in the first language (L1) may not transfer to the L2. In this 7-week L2 reading strategy intervention, 801 first-year polytechnic students learned to use seven reading strategies that were effective according to a meta-analysis of L2 reading strategy studies. Data regarding students reading skills were collected over one academic year, from three treatment waves, using a regression discontinuity design. Three tests of equal difficulty were given to participants. In each wave students completed reading tests several weeks before the intervention, at the beginning and directly after the intervention. Results show that in all three waves the improvement in reading comprehension scores between the second and third measurement (due to the experimental course) significantly exceeded the increase between the first two measurement occasions. Although the intervention was shown to be effective, the effects were to some extent mediated by the previous education level. This study supports the explicit instruction of strategies in L2 reading for students in higher education and welcomes more research into L2 reading strategy interventions for students from vocational backgrounds.
Keywords
I Introduction
Being able to digest large amounts of academic text in a fast, efficient and effective manner is a crucial skill that not all students in higher education have yet mastered (Trudell, 2019). Especially so, when one bears in mind that the extensive demands of academic reading experienced by students in higher education differs a great deal from the reading demands faced by students in secondary education (Hermida, 2009). For example, college students are often required to digest and process large amounts of detailed and complex textual information within a short time-frame. Moreover, academic texts often tend to be written in a concise and compact style, so that even good readers must utilize a considerable amount of their reading comprehension abilities in order to extrapolate and comprehend their meaning (Lee & Spratley, 2010).
Furthermore, the demand for English as a second language (ESL) being used as a medium of academic information exchange is expanding within higher education, in what could be seen as a globalization of academic linguistic expression (Block, 2004). Certainly, there has been a notable growth in scholarly articles and papers in higher education offered in the English language, particularly within the north of Europe. The Netherlands and Scandinavia, for example, offer the highest percentage of higher education courses in English as a second language (Kuteeva & Airey, 2014). Increasingly, college textbooks, lecture notes and materials are also written in English, which can pose an extra reading challenge for non-native readers (Van Weijen, Tillema, & Van den Bergh, 2012). For this reason, instruction in second language (L2) reading comprehension at higher education level for ESL students is an important and relevant issue (Grabe & Stoller, 2011).
II Reading strategy instruction
Reading instruction that incorporates reading strategies and promote strategic reading behaviour has been found to be conducive and helpful in the development of a student’s academic reading ability (Mokhtari, Reichard & Sheorey, 2008; Pressley et al., 2006). However, as strategies learned in the first language (L1) do not automatically transfer to the L2 it is essential that instruction is extended, in formal education, to include L2 reading strategy instruction (Baddeley, Eysenck, & Anderson, 2009; Van Gelderen et al., 2007). Furthermore, due to the increase of the use of ESL reading within academia, it is all the more urgent to continue L2 reading strategy instruction beyond that of secondary school level (Holligan, 2018). Nevertheless, research into L2 reading strategies, and the role of L2 reading strategies in reading comprehension at college level has been scant and investigations into L2 reading strategy interventions in higher education have been lacking (Taylor, Stevens & Asher, 2006).
For the last 30 years, most L1 reading comprehension programmes in primary and secondary schools include reading strategies in some form (Duke & Pearson, 2002). Reading comprehension can be described as a complex and complicated process, in which the reader engages with a text in order to obtain meaning and understanding from it. Comprehension is achieved by employing the reader’s cognitive processes and metacognitive skills to understand the text’s meaning (Kintsch, 2002). The reader’s understanding, and resulting comprehension, is thus facilitated by the skills and strategies at his or her command. A reading strategy is therefore the mental tool that the reader may apply consciously, or semi-consciously, to monitor, repair or comprehend what they read (Afflerbach & Cho, 2009). Afflerbach, Pearson and Paris (2008) noted that reading strategies are in fact techniques the reader employs that over time may become an automatic part of the reading process but will require practice and frequent use “during reading in order to become so” (Afflerbach et al., 2008, p. 368).
Reading comprehension, therefore, is a skill that every teacher of each subject at all educational levels of instruction needs to be fully aware of (Strickland & Shanahan, 2004). Reading instruction can also be effective for older students (Anders, Hoffman & Duffy, 2000) and knowledge on how to use and apply reading strategies can be particularly helpful, especially since there is no guarantee of automatic transfer from the L1 (Snow, Griffin, & Burns, 2005).
Furthermore, the explicit instruction of reading strategies, for example, in the form of teacher modelling when and how to apply reading strategies may provide useful insights to student readers in helping them manage their metacognition and reading processes (De Milliano, Gelderen & van Sleegers, 2016). Grabe and Stoller (2011) found that reading strategy studies that had introduced reading programmes which incorporated cognitive and metacognitive reading strategies along with training in planning and self-monitoring, showed significant improvements in reading comprehension performance.
Reading strategies that help the reader determine what is important in the text, what may be implied but not explicitly stated, and which assist the reader to form a synthesis of the information presented, are considered to be effective strategies for active reading purposes (Harvey & Goudvis, 2007). Moreover, Pressley (2002) noted that reading strategies such as asking questions, making predictions and strategies aimed at inferential thinking enabled readers to think more strategically and to make more sense of what they were reading. After being taught metacognitive reading strategies that invite reflection of the reading process as a whole, students became more active in their reading, mimicking the successful strategic reading process which would normally be automatically employed by proficient readers (Pressley, 2002; Pressley et al., 2006).
Reading strategy training which places the focus on strategic reading, i.e. the planning, monitoring and evaluation of the reading process, can lead to significant improvements in L2 reading comprehension (Bimmel, van den Bergh & Oostdam, 2001; Cohen, 2014; Duffy, 2002). Moreover, structured training instructs students in using L2 reading strategies effectively, combined with the metacognitive understanding of why and how reading strategies can aid understanding, can assist in the monitoring and checking of the reading process (Paris, Wasik & Turner, 2016). Training such as this can also be helpful to less skilled readers in acquiring new reading comprehension skills in the L2 (Koda, 2007).
When considering the effectiveness of reading strategy interventions in improving reading comprehension it is important to consider the full variety of reading strategies available. Pressley (2002) and Guthrie and Humenick (2004) found that student reading performance improved significantly when instruction employed the widest range possible of reading strategies aimed at cognitive and metacognitive understanding.
For example, this was the case in The Benchmark School (Pressley et al., 2006) where 203 students aged between 12 to 16, all with a previous history of school failure and in particular difficulties with reading, were provided with an hour and a half of explicit literacy instruction every day, twice a day. The instruction of strategic reading skills occurred via direct explanation models, which provided students with explicit knowledge about why, when, where and how to use reading strategies. Reading strategy instruction included a focus on overt teacher modelling with teachers spending time explaining to students how to become aware of and monitor their own reading process. Students were taught reading strategies used by successful readers to comprehend and integrate new texts and ideas, and emphasis was placed on the importance of self-monitoring and self-evaluation of the reading process as a whole. The researchers praised the overall consistency in the delivery of the reading instruction, in which all teachers participated. The Benchmark School’s rigorous approach of placing reading instruction as the keystone of the daily curriculum resulted in 100% of their students graduating from high school.
Taraban, Kerr and Rynearson (2004) found support in their study for the view that students in higher education choose and utilize reading strategies that they believe orientate them towards their success in academic tasks. The researchers asked 575 college students to complete the Metacognitive Reading Strategy Questionnaire (MRSQ), 1 on their use of cognitive and metacognitive reading strategies to solve reading comprehension issues encountered during their studies. The researchers found that reading comprehension was not perceived by students to be an automatic process but one which required direct cognitive effort and the application of metacognitive input, by way of reading strategies. Students reported allocating significant attention to the checking, monitoring and evaluation of their reading process.
In a meta-analysis on the effectiveness of L2 reading strategies studies on reading comprehension performance, Yapp, De Graaff and Van den Bergh (in press) analysed 46 L2 reading strategy studies to ascertain which reading strategies and pedagogical approaches were the most effective in improving reading comprehension performance. The average effect size of the reading strategy studies analysed was found to be large (g = .91), meaning that reading strategy interventions in L2 reading comprehension are effective. Furthermore, the meta-analysis found that reading strategies that required explicit cognitive action to be taken by the reader produced significantly large effect sizes, for example: semantic mapping, paying attention to structure and signal words, activating background knowledge, asking questions while reading, and connecting new information to what is already known (for descriptions and effect sizes, see Appendix 1). For this reason, these strategies were deemed to be particularly effective in L2 reading comprehension and were therefore integrated into the present L2 reading strategy intervention.
Additionally, the reading strategies ‘activating background knowledge’ and ‘connecting new knowledge to what is already known’ were specifically selected from the meta-analysis to be used in this intervention due to the fact that at-risk students, i.e. those who have struggled with L2 reading in the past, or who enter higher education from senior vocational education, 2 may find the difficulty and complexity level of L2 texts daunting. These students have had little, or no experience at all, in dealing with complex academic texts in English, and have often not accumulated the reading expertise and background knowledge that would be expected from them to tackle their higher education reading demands (Beeker, 2012). Likewise, these students have not attained sufficient academic reading proficiency in their four years of vocational secondary school, compared to other undergraduate students with a non-vocational previous education.
Moreover, the meta-analysis tested the effectiveness of pedagogical approaches such as modelling and introducing strategies, as well as raising awareness of the reading strategies being taught and approaches such as collaborative and individual practice. It was observed that the effectiveness of the pedagogical approach analysed was dependent on which reading strategy was being taught. In other words, not all pedagogical approaches were found to be equally effective with all reading strategies taught but some approaches were more effective with certain reading strategies than other approaches. For example, teacher modelling, an approach used by the teacher which is used to demonstrate the application of a particular reading strategy (Amirabadi & Biria, 2016) was found to be significantly effective with the reading strategies of visualization and skimming and scanning, but less effective with the strategies semantic mapping and asking questions while reading. Therefore, in this present intervention, it was decided to employ a wide range of pedagogical approaches in combination with the most effective reading strategies.
Scientific investigation into the effectiveness of reading strategy instruction in ESL contexts in higher education has been lacking (Khalifa & Weir, 2009). This study is an attempt to address this research gap by, first, exploring whether a specifically designed L2 reading strategy method could be effective and could lead to improved levels of L2 reading comprehension performance among first-year college students. Second, due to the difficulties experienced in L2 reading for students from a previous education at a vocational level, we wanted to know whether the effect of the intervention would vary between the sub-populations of the student participants according to different previous educational levels.
III Method
1 The present study
An L2 reading strategy programme was designed, which lasted seven weeks and taught one strategy per week. This programme combined seven L2 reading strategies which had been found to be effective from our meta-analysis with the intention to improve L2 reading comprehension and promote students’ self-efficacy as long term successful L2 readers (Pinninti, 2016).
The reading strategies were taught with a variety of instructional pedagogy, such as awareness raising, introducing strategies, teacher modelling, collaborative and individual strategy use, feedback, scaffolding, teaching why the strategy was useful and when and how it could be used. The benefits of teaching students the why and how of reading strategies has been found to be beneficial in facilitating reading comprehension, rather than the instruction of reading strategies alone, without background explanation (Paris et al., 2016).
2 Participants and study institution
A complete year’s cohort of students from a faculty of Management and Governance took part in this study (N = 801). There was no selection for participation as a formal requirement from the management of the faculty had required that all first-year undergraduate students of that year’s cohort be allowed to follow treatment and should enjoy an equal opportunity to reap the benefits of any improvement experienced in their L2 reading comprehension performance. For this reason, the whole first year’s cohort participated in the study.
The participants were students in full-time education at a university of applied sciences located in the urban centre of The Netherlands. This institution began in 1981 as a higher technical school and has since grown to its current size of roughly 11,000 students. Most students are accepted into the institution on the basis of a five-year general secondary education diploma, 3 which constitutes approximately 70% of the student population at this institute. Some students (approximately 2%) enter after a six-year secondary school university preparatory education diploma which prepares students for a research university education. 4
Students with a senior vocational education are also accepted, after successful completion of their four-year vocational post-secondary study. These students make up approximately 30% of the entire student population. However, student attrition is higher among these students than in other groups. Their previous education is made up for a large part of vocational experience with little formal reading comprehension training in an L2 language such as English. In this respect, they are frequently the most disadvantaged when following English modules, due to their lack of L2 background knowledge and overall lack of exposure to English academic type texts.
As the student influx is heterogeneous in nature there is much variance in reading comprehension ability. English modules are part of the compulsory curriculum, and entry-level requirements expect students to have achieved an English language proficiency of B2, during the course of their polytechnic bachelor degree they are expected to extend this to B2 + / C1 (CEFR) 5 level in English. Moreover, other subjects may use textbooks or other academic materials written in academic English, which poses a particular problem for students arriving from vocational backgrounds. These students have completed their previous studies with either, at the lowest end, the level of A2 (CEFR) English proficiency, or, at the highest end, the level of B1. 6 In either case, this falls short of the minimum requirement of proficiency in English for institutes of higher education.
The participants of the study were full-time undergraduate students who agreed to participate in the study after completion of a signed informed consent form (N = 801). The students were enrolled in one of the following disciplines of study: Law, Communications, Commercial Economics, Human Management Resources, or Social Juridical Services and were in the first year of their bachelor degree course. To accommodate the large volume of students participating in the study, treatment occurred in three equivalent moments of treatment, i.e. waves, by using an intervention design that could accommodate this programme of treating large numbers of participants (see Design). For this reason, the specifically designed English reading programme was offered across the entire faculty and all students were required to follow the programme regardless of their study discipline. However, students were able to opt-out of data collection by indicating in the informed consent form that they did not give their consent to their data being used. For a breakdown of participants per wave, according to gender and previous education, see Appendix 2.
There were relatively more female students (n = 546) than male students (n = 255) in this sample, which is consistent with the ratio of females to males in these disciplines in Dutch higher education. Students with general secondary education as their previous education were the largest group (n = 493), 61.5% of the sample. This was to be expected as a university of applied sciences education is the conventional route through higher education for students with this type of secondary school diploma. The number of students with senior vocational education as their previous education was within normal range for this student population (n = 250) and comprised 31.1% of the sample. The number of students with university preparatory education as their previous education was relatively small (n = 28), 3.6% of the sample population. This was not unusual, as most students with this type of secondary school diploma pursue a research-orientated university education. The number of students with previous education unknown was small (n = 30), 3.8% of the sample. Ages of the student participants ranged between 17 years and 22 years of age, with a mean age of 19.22 (SD = 1.87).
Student attrition in higher education in The Netherlands is a large and common phenomenon. Most student attrition occurs within the first year of study; the institution where this study took place was no exception to this phenomenon. From our student sample 359 students did not complete all three tests, as they had either left the institution or, for other reasons, were no longer participating in their studies. Students tend to suspend their studies or transfer to other studies or institutions halfway through the year, or they may choose to discontinue their studies completely. To this end, an extra analysis was carried out to determine whether attrition caused any variation in results between student sub-populations.
Ten members of teaching staff participated in the study (8 females), with a mean age of 46.3 (SD = 10.54) and a mean of 12.0 years teaching experience in higher education (SD = 7.65, min. years’ experience = 3, max. = 25).
IV Design
The assumption made is that students’ reading performance will improve due to the experimental reading intervention. In order to make a distinction between ‘natural growth’ and the effect of the reading intervention followed by the students, three reading comprehension tests of equal difficulty were required to be completed on three subsequent measurement occasions (testing moments). The difference between the first two measurement occasions is indicative of ‘natural growth’ in reading skills, whereas the difference between the second and third measurement occasions reflects the effect of the experimental reading intervention. In this regression discontinuity data design, we wish to test whether the slope of the regression line changes due to the introduction of the reading intervention (Shadish, Cook & Campbell, 2002).
Not all students were able to follow the intervention at the same time, as the number of students exceeded the amount that could be accommodated within one academic period. Therefore, students were assigned to one of three waves of equivocal treatment, which coincides with the students’ departmental course curriculum. In each wave students completed the first measurement occasion (M1) ten weeks prior to the start of the reading intervention and the second measurement occasion (M2) at the start of the intervention. The third and last measurement occasion served as a post-test (M3) that was taken on completion of the intervention. Table 1 shows administration of measurement occasions and treatment under a Regression Discontinuity Data design where the experimental conditions are divided over four consecutive academic periods of ten weeks each.
Treatment and measurement occasions per wave.
Notes. CE = Commercial Economics. COM = Communications. HRM = Human Management Resources. SJD = Social Juridical Services.
The compulsory nature of the reading programme meant no student could be excluded from treatment, meaning all students must be allowed to follow treatment whether participating in the study or not. This coincides with our intention that all participants be able to benefit from any possible gains that might ensue from following the intervention.
In this intervention, each of the three waves followed an identical sequence in terms of treatment assignment, set-up and duration.
1 Treatment
The intervention ‘Effective reading for professional purposes’ was a seven-week L2 reading strategy instruction programme of two hours a week, in total 14 hours of instruction, offered to all students in periods two, three or four of the academic year. The students completed two of the measurement occasions during the intervention and one prior to starting the intervention.
Teachers met weekly for one-hour training sessions, on the implementation and teaching of the instructional principles of the intervention, over the complete academic year. Treatment fidelity was closely adhered to by conducting teacher observations during each treatment wave and through teacher logbooks, in which teachers reported to what extent they had followed the instructional principles for each week of the intervention. The reading strategy programme followed four distinct stages:
a Direct instruction of L2 reading strategies
Each week concentrated on one type of strategy use. For each strategy or strategies, explicit instruction was given on what the strategy was, and on how, when and why to use it (Veenman, Van Hout-Wolters, & Afflerbach, 2006). The purpose of this phase was to create awareness of the strategies, explain the importance of cognitive and metacognitive reading strategies, help students identify the strategies that they already use, and develop students’ metacognitive awareness of the relationship between their own mental processes and the promotion of effective reading behaviour.
b Teacher modelling
Teachers were trained during weekly implementation sessions on modelling the use of reading strategies, by explaining their mental processes out loud while reading a text (Schunk & Zimmerman, 2007). In class, the teacher explicitly explained the characteristics, usefulness, and applications of using the strategy, why to use it, when and gave specific examples. The teacher also explicitly illustrated his or her own strategy use by demonstrating a think-out loud approach to a reading task, for example, by examining unknown vocabulary items in a text by explaining their thought processes out loud and connecting the new information to what vocabulary or knowledge might already be known on the subject.
c Collaborative practice, scaffolding and individual practice
Students were encouraged to work together collaboratively to apply the reading strategies while solving a reading task. This was found to be an effective approach in the meta-analysis of L2 reading strategies (Yapp et al., in press). Collaborative practice among students has been found to promote empathy and communication and to bolster problem-solving skills (Chu, Tse, & Chow, 2011). Teachers were also encouraged to give feedback to the group or to scaffold where necessary. For example, if a group or individual student was having difficulty with the task, the teacher would either model the strategy again or scaffold until the student was able to continue with the task unaided. Students were asked to make a conscious effort to use the metacognitive strategies in combination with the reading task. Students were also shown how to recognize when one strategy was not working and how to move on to another strategy.
d Evaluation and expansion
The main purpose of this stage was to provide students with opportunities to evaluate their success in using reading strategies, thus developing metacognitive awareness of their learning processes (Cubukcu, 2008). Activities used to develop students’ self-evaluation insights included self-questioning and evaluative discussions after strategy practice, checklists of strategies used, and open-ended questionnaires on reading behaviour, in which students expressed their opinions about the usefulness of particular strategies. In the expansion phase, students were encouraged to use the strategies that they found most effective and apply these strategies to new contexts (Van Silfhout, Evers-Vermeul & Sanders, 2014). A variety of authentic texts were used in the method which highlighted topical or cultural issues interesting to this particular target group in order to increase student motivation to read (Guthrie & Wigfield, 2000).
Classes of approximately 25–28 students per class were comprised of students from the different study disciplines, which meant that texts and tasks had to be interesting and appealing to students studying from the five disciplines. Each class contained a core lesson that would be presented in a PowerPoint presentation with tasks and additional texts provided in a printed or digital handout.
Each class began with a brief recall of the previous reading strategies learned. students were encouraged to apply all the strategies they had learned so far during the reading tasks and to select the reading strategy that worked the best with any given reading task. Students were also invited to share, in an open discussion, what they found to be effective reading practices. In the last week of the intervention, students were given a practice reading exam so they could use the reading strategies that they had learned from the intervention, while the teacher was available for scaffolding or for questions. In the following week of the intervention all the students completed a final examination in the form of a formal reading comprehension test. This reading comprehension examination was used to determine their posttest result. The list of reading strategies based on the meta-analysis and taught in the intervention are listed below. For a short definition and calculated effect sizes for each reading strategy taught in the reading strategy programme, see Appendix 1.
Reading strategies taught per week in the programme:
Connecting new knowledge to what you already know
Asking oneself questions while reading
Making predictions while reading
Visualization
Paying attention to structure and signal words
Skimming
Scanning
2 Instruments
A crucial requirement of a Regression Discontinuity design is that the instruments of measurement must be of equal difficulty, due to the fact that we wish to rule out differences in difficulty as a possible explanation for observed differences. For this reason, we choose the Cambridge Advanced English (CAE) reading comprehension tests to be a suitable instrument for ESL comparison purposes, while also being a reading comprehension test that was relevant for higher education academic purposes. The argument behind this choice was based on the level of English reading proficiency of the participating students and the high reliability and validity of CAE reading tests. The extensive and worldwide use of Cambridge exams by comparable student populations within academia can be attributed to the fidelity and trustworthiness of these tests. Furthermore, during correspondence with the Cambridge test construction team, we were assured of a guarantee of equal test difficulty 7 (Vidakovic, Elliott & Sladden, 2015).
The University of Cambridge ESOL 8 examinations make use of an item banking system which uses the Rasch scale to determine the level of difficulty of items along a common scale of difficulty. Through item banking Cambridge explicitly guarantees that each CAE reading test is equal in difficulty and equivalent in its ‘predicted level of difficulty’ to all other CAE reading tests (Marshall, 2006; Vidakovic et al., 2015). All three measurement occasions used in this intervention were complete CAE reading tests with 30, 34 and 36 (respectively) multiple choice questions at B2–C1 level 9 (CEFR).
Each CAE reading test is divided into four parts: Part one consists of three short texts with a similar theme and a focus on reading for global understanding. This part contains two questions per text with six multiple-choice questions in total. Part two consists of one text that contains ‘gaps’ of missing paragraphs and the reader chooses which of the seven paragraphs fits best into the six gaps in the text; there is one extra paragraph that is not needed. This task focuses on understanding of textual structure, and knowledge of signal and linking words. Part three consists of one long text with seven to nine multiple-choice questions; this task focuses on reading for detailed understanding. Part four is a multiple matching task where the reader matches 5 or 6 short texts with 15–20 statements. These statements are matched to a specific part of the text and by reading the statements first, the reader must locate specific information in the text. Part three and four can vary in the total number of items they contain depending on the ‘predicted level of difficulty’ level of the items used.
The first measurement occasion (M1) was completed digitally, ten weeks prior to students following the intervention. The second measurement occasion (M2) was completed under written paper-and-pen exam conditions at the start of the intervention and the third measurement occasion (M3) was completed under similar exam conditions to the second test, after completion of the intervention. Students did not follow any English classes between the first and second measurement occasion. The three measurement occasions were tested for reliability using the Cronbach’s alpha coefficient; M1: ɑ = .84, M2: ɑ = .80, M3: ɑ = .67. These results indicate that our tests were satisfactory in terms of reliability. The data from our research suggests a comparable concurrent measure of reliability which corresponds with the findings of Cambridge ESOL examinations on the reliability of their examinations.
3 Treatment fidelity
In order to observe and evaluate whether teachers carried out the intervention according to the treatment plan, observations of teachers were necessary and logbooks were completed by teachers directly after each class. Teachers also met weekly, for an hour, for the entire year to discuss treatment implementation and any issues that might have arisen during the teaching of the intervention. Teachers also shared best practices and their own experiences in teaching the reading strategy intervention.
Teaching observations were conducted for all teachers between weeks two and six of each wave. An observational rubric was devised to examine if the teachers conducted the intervention according to the module description and the treatment plan, i.e. if the teacher adhered to the guiding principles of the intervention. Inter-rater reliability was calculated by means of observed agreement between two observers (the researcher and a senior colleague). An observer agreement of 92% was reached when teaching observations were compared. There were a small number of individual differences between teachers observed, for example; some teachers modelled more than others, but all teachers modelled. When all observations were compared it was found that teachers conducted the intervention according to the teaching guidelines of the programme, in other words, high fidelity was observed, as teachers taught the intervention in the manner in which was expected of them.
4 Procedure
Integral to this intervention was the use of three tests (see design). The first measurement occasion was completed via the electronic learning environment used by the institution. Participating students were sent an invitation to complete the test ten weeks prior to the period in which they would follow the intervention. The test was made available to students during ‘campus time’ 10 when most students would be on campus but not in scheduled classes. The participants took the test by logging in under their student number. Once completed the students would not be able to retake the test, and the test had to be completed in one sitting. The test did not contain any explanation of reading strategies other than the necessary instructions required for completion.
After completion, the participants’ score was calculated automatically, participants were informed of their score, without any further explanation. The researchers retrieved the completed test results by logging into the online learning environment. Students were requested not to use dictionaries while completing the test. There was no maximum time given as the participants completed this test digitally with minimum supervision; however, the researchers inspected the duration times of the test sessions and were able to verify that no student exceeded the maximum duration of two hours.
The second measurement occasion was administered during week one of the intervention, under formal test conditions with the intervention teacher. Students were not permitted to use dictionaries and were given two hours to complete the test. Score sheets were collected, marked, and returned to the students the following week. The decision to administer the first test digitally had to do with logistical factors. Students following course subjects, during an academic period, are assigned to class groups that may vary according to the particular classes that they are following for that period. However, the first and second measurement occasions were completed under, as far as possible, identical conditions, using equally difficult tests. Information and research data obtained from Cambridge ESOL, on examinations conducted by candidates via computer and on paper, found no indication of an influential factor between the two conditions of paper-based versus computer-based examinations, on candidate performance (Thurlow et al., 2010).
The third measurement occasion was administered under formal examination conditions, i.e. a passing grade would receive study credit, one week after completion of the intervention. In all three measurement occasions equally difficult tests were administered and students were given a maximum of two hours to complete the tests, dictionaries were not permitted.
5 Analysis
After the collection of all data, the dataset contained test results of 801 students. The reading scores of the three tests were analysed using SPSS mixed model procedures. In this study we are interested in the changes in slope of the regression line due to the introduction of the experimental reading intervention. Therefore, we expect to be able to observe that the student’s reading performance will improve more during the reading strategy intervention than could have been expected from ‘natural growth’ following business as usual.
In order to assess the effect of the experimental reading intervention, six models were formulated, which differ in fixed effects only. In all the model’s differences within and between students as well as between classes are estimated. In the first model, a so-called null model, it is assumed that there is neither an effect of the experimental reading intervention nor a (natural) increase in reading ability. In the second model for natural growth it is assumed that there is an increase in reading scores but the experimental intervention does not have an additive effect above and beyond natural change. In the third model, the experimental treatment effect model, it is assumed that there is an extra increase in reading scores which is due to the experimental reading strategy intervention. In the fourth model, the wave model, any differences in means between the three waves are allowed. In the fifth model, the natural growth * wave-model, we allow for differences in natural growth between the three waves. In the sixth model, the experimental effect * wave model, differences in effectiveness of the programme between waves are allowed
The fit of these six models is compared by means of the –2loglikelihood, as the difference between the –2loglikelihood of these (nested) models is chi-square distributed with the difference in number of parameters as degrees of freedom. The model can be extended with explanatory differences such as level of previous education, these differences will be taken into account, in order to estimate whether possible effects are attributable to these said differences.
Taking explanatory differences into account, any difference between these models could be attributed to the effect of the intervention. If the intervention influences reading comprehension performance, the increase in scores between the first and second measurement occasion will be less than the increase between the second and third measurement occasion.
V Results
In order to demonstrate the effect of the intervention, we measured if reading comprehension performance improves more between the second measurement occasion and the third measurement occasion than between the first measurement occasion and the second measurement occasion. The fit of the models was compared for changes in reading scores due to natural growth, wave and the experimental programme. The fit of the six models is presented in Table 2. Table 2 shows that the fit of the model increases if we allow for differences between measurement occasions due to natural growth; the increase between the first and second measurement occasions is the same as between the second and third measurement occasions: χ2(1) = 675.72; p < .001. If an extra effect of the experimental condition is allowed above and beyond natural growth the fit of the model increases significantly, in comparison between models 2 and 3: χ2(1) = 195.41; p < .001.
The –2loglikelihood fit (–2ll) and comparison of six models for changes in reading scores due to natural growth, wave and the experimental program.
Notes. m1 = measurement 1. m2 = measurement 2. m3 = measurement 3.
Comparison of the models 3 and 4 also shows that there are differences in reading scores between waves: χ2(2) = 74.33; p < .001; natural growth differs between the first and second measurement occasions, which differs between waves: χ2(2) = 14.05; p < .001, but the experimental effect cannot be shown to differ between waves: χ2(2) = .22; p = .90. Hence, there are some differences between waves in ‘natural growth’, but the effect of the intervention does not differ between waves. Irrespective of the effects of natural growth, the effects of the intervention are comparable between waves.
Figure 1 shows the mean reading scores per measurement occasions per wave. The figure clearly shows that there is a discontinuity. Natural growth between the first and second measurement occasions is smaller than between the second and third measurement occasions, hence the natural growth is smaller than growth due to intervention, despite there being small differences detected between waves. Moreover, the differences in the experimental effect did not differ significantly between waves. The confidence intervals in Figure 1 are relatively small; therefore, we are confident in assuming differences between measurement occasions and waves (see Figure 1). For numbers of students per wave, calculated means and standard deviations per measurement occasion and wave, see Appendix 3.

Estimated means per measurement occasion (M1, M2, M3) and per wave.
With respect to attrition, it was tested whether the average reading score of students who took the reading tests on all three measurement occasions differed from the average reading score of students who did not take all three tests. Results show that the average reading score of both groups of students did not differ significantly from each other (F (1, 1396.4) = 1.51; p = .22). Neither did the effect of attrition seem to be related to measurement occasion (F (2, 1363.0) = .11; p = .89), nor to wave (F (2, 868.1) = .02; p = .98). Hence, students who took all three reading tests do not (on average) systematically differ from those who missed one or two of the reading tests. Therefore, we may assume that L2 reading score is not related to attrition. For number of missing scores per wave see Appendix 4.
A difference in average reading scores, due to previous education was assessed (F (2, 722.89) = 80.37; p < .001). Pairwise comparison showed that the average reading scores of students from senior vocational education were lower than those of students from general secondary education (p < . 001) or with university preparatory education (p = .03). The difference in average reading scores between the latter two groups did not reach significance (p = .99).
The effect of previous education appeared to depend on measurement occasion as well (F (6, 1107.3) = 4.73; p < .001). Further analysis showed that the effect of previous education did not reach significance on the first two measurements (p ⩾ .52). The gain in mean reading scores between the second and third measurement occasion appeared to be lower for students from senior vocational education than those from general secondary education (t (1167.3) = 5.20; p < .001) which were comparable to the difference with those from a university preparatory education (t (1114.6) = 5.14; p < .001). The difference between general secondary education and university preparatory education students did not reach significance (t (1064.2) = 1.20; p = .23). However, the number of participants with a university preparatory education was relatively small (n = 28). Therefore this test may have lacked sufficient power. Hence, the learning gain due to the experimental reading course is (on average) lower for students with a background in senior vocational education than those with a general secondary or university preparatory education (see Figure 2).

Differences in mean reading performance per measurement occasion (M1, M2, M3) according to previous education.
Finally, we investigated whether our results depended on a student’s skills in English regardless of their previous education. In other words, whether a student with a previous vocational education with good reading skills in English, for example, would derive more benefit from the intervention than a student with a general secondary previous education, with poor English reading skills. The group of students in our sample with a previous education of general secondary education (n = 493) achieved the highest mean improvement score of 10.19. The group of students in our sample with a university preparatory education as their previous education (n = 28) had a mean improvement score of 8.22. The students with a senior vocational education as previous education (n = 250) had the lowest mean improvement score of 7.01.
We were interested in the growth between proficient and less proficient readers irrespective of previous level of education. In other words, whether growth is dependent on reading comprehension ability according to the differences between the three measurement occasions. To this end, two alternative models were analysed. In the first model the reading scores of the first measurement occasion (M1) were used as a covariate to predict the reading scores on the second measurement occasion (M2) and the third measurement occasion (M3). In the second model the effect of the covariate was allowed to differ between both measurement occasions. This last model clearly fits the data better than the first model (χ2(1) = 42.27; p < .001). An inspection of the regression weights showed that the effect of M1 on M2 (β = .92; se = .04) is much stronger than the effect of M1 on M3 (β = .55; se = .04). Both effects are shown in Figure 3. The left figure shows a strong relationship between M1 and M2 with a wider distribution along the M1 reading scale score. The right figure shows less proficient readers at M1 with an increase in reading scores, who are clustered along the M1 reading scale.

Relationship between measurement occasion M1 and measurement occasion M2 (shown left) and between measurement occasions M1 and M3 (shown right), including regression lines.
VI Discussion
This study set out to determine if an L2 reading strategy programme, specifically designed for higher education students, could improve first-year student’s English L2 reading comprehension performance. Our results indicated that the effect of the intervention was significantly effective. It was also found that previous education plays an influential role in the level of L2 reading comprehension improvement for the students in our sample. On average weaker readers improved (including those students with a senior vocational education as previous education) more than good readers from all levels. Indeed, on average, students with senior vocational education as previous education improved a great deal in their L2 reading comprehension. However, not all students from this level of education were poor readers before starting the intervention, correspondingly, not all students with a general secondary education as previous education were good readers at the start of the intervention. Nevertheless, we conclude that vocational students benefitted considerably from this intervention, and students with a general secondary education benefited even more so. We will return to this point when we examine our results in the context of our research questions.
Specifically, to our first research question, as to what extent does this L2 reading strategy programme improve undergraduate student reading performance, we conclude that student reading comprehension performance seemed to improve significantly between the second and third measurement occasions. In other words, the effect of experimental growth exceeded that of natural growth, which was true for all three waves and on average students improved in their reading comprehension scores. Our reading strategy intervention works.
In this respect our intervention corresponds with Duffy (2002) where the importance of teaching reading strategies through direct explanation was conducted in a large scale L2 study to young adults with good results. In this study teachers used a wide range of pedagogy to introduce strategies, explained when and how to use them, modelled, used scaffolding and practiced with the strategies, also explaining their use and implementation.
The results of our study also bear some resemblance to the results achieved at The Benchmark School (Pressley, 2006) our study showed the clear effects of explicit reading strategy instruction, even though our students are much older, both studies employed explicit reading strategy instruction in how, why and when to use reading strategies.
Our second research question focused on the intervention effect with regards to the role of previous education. To address this question fully we will divide our response into three parts. First, we present a general observation. Second, an observation based on the specific level of previous education. And third, our findings based on individual student gains.
Our general observation is that the L2 reading strategy intervention works. Students improved their average scores in L2 reading comprehension after following the intervention. Moreover, less proficient readers seemed to derive more benefit from the intervention than proficient readers, which is understandable, as readers who already achieved a high score in their first measurement occasion are most likely efficient readers already.
Our findings with regards to the influence of previous education level is that while this intervention was effective, students from a general secondary education or a university preparatory education seemed to improve more in their average L2 reading performance than students from a senior vocational education. This could be expected since most students with vocational backgrounds have had, on average, less experience in L2 reading comprehension and less exposure to complex academic texts in English. Furthermore, these students often lack general background knowledge in L2 reading. A general interest in reading and having possession of reading background knowledge were found by Strickland and Shanahan (2004) to be two determining factors which contributed to students being able to improve their reading comprehension beyond their currently held reading level. For some of our vocational students this intervention was their first exposure to this type of reading material. This lack of exposure was confirmed at the start of the intervention, as the M1 scores of vocational student’s reading comprehension were lower than those of the other two subgroups in this study population.
When we examine our results from an individual student level we observe that not all poor L2 readers were from vocational backgrounds, a number of students with a general secondary previous education achieved lower than expected scores in their M1 scores. Correspondingly, some students with a vocational background were exceptionally proficient L2 readers. This phenomenon accounts, to some extent, for the observation of the strong relationship between the first and second measurement occasions seen in the left diagram of Figure 3.
In some respects our study is similar to the results of Amirabadi and Biria (2016) where 50 Iranian higher education students from different study backgrounds, with ESL reading difficulties, were administered an ESL reading programme treatment via a reciprocal treatment approach. Students in the experimental group were taught to use a number of reading strategies through modelling, scaffolding and self-regulation activities, in order to improve their ESL reading comprehension. The experimental group outperformed the control group in reading comprehension performance and the study observed an average effect size of .84.
Our conclusion with regards to our second research question on previous education is that in the context of this reading strategy intervention previous education seems to have an influence on student L2 reading comprehension performance. This outcome gives room for concern. It is possible that lack of exposure in terms of level and, or, experience with, complex academic texts in English, during vocational type studies, has later repercussions for higher education students in their L2 reading. Alternatively, the type of L2 reading strategies offered in this intervention seemed to work particularly well for students who previously followed a general secondary level type of education. This point will be further explored in implications for future research.
Our conclusion with regards to individual improvements in L2 reading comprehension performance, as a result of this intervention, is that poor readers come from all levels of previous education and not solely from vocational backgrounds. For whatever reason, these students have not yet developed their reading skills sufficiently to deal with the ESL reading demands faced by students in Dutch higher education. To this end, it is essential that L2 reading strategy instruction programmes be extended past secondary school level to that of higher education. Anders, Hoffman and Duffy (2000) noted that the issue of how and when teachers should be teaching reading was one that had received little attention from the research community. We hope that the findings in this study have contributed to the interest and debate on this particular topic.
1 Limitations of this study
In this study only one post-test was administered. However, in order to determine whether the effects of treatment lasted beyond the post-test it would have been beneficial to have had participants complete a delayed post-test, such as used in the study of Duffy (2002). A delayed post-test would show whether improvements in student reading comprehension are still evident for a longer period after end of treatment; however, a delayed post-test was, not a feasible option, as it would have created an extra strain on the already over-burdened study curriculum.
Further, conducting the first test digitally and the subsequent two tests with paper and pen may have had some minimal influence on the outcome. While our results revealed virtually no difference in natural growth between measurement occasions one and two, and the testing conditions and difficulties of the two tests were, identical, as far as technically possible, we cannot exclude the possibility that completing a test on a computer may have had some small effect on the outcome.
On a positive note, the faculty made a note in their yearly report that the number of students who had passed English academic reading in the school year 2017–2018 had increased from 45% to 75%, which is a substantial increase in L2 reading comprehension performance.
2 Suggestions for further research
The results of our intervention point to the importance of L2 reading strategy instruction, within higher education, where reading strategies are taught, as well as explicit instruction and practice of why, where and when to use reading strategies. Duffy (2002) noted that readers must first be taught strategies in order to be able to have a choice in whether or not to use them. However, while our results are encouraging, they may to some extent be attributable to the individual characteristics of this particular faculty’s student population. This point notwithstanding, it is important to note that the effect of this intervention has been replicated twice, over the three equivalent treatment waves. Nevertheless, we would encourage similar L2 reading strategy interventions to be conducted in other higher education contexts.
Furthermore, while on average students showed improvement after treatment, students with a general secondary as previous education seemed to benefit more than students with a vocational previous education. This is an area where more research should be encouraged, especially since this vulnerable group of students comprises approximately a third of those currently in Dutch polytechnic higher education. Additional research would be especially welcome as to how future L2 higher education reading strategy interventions can better support students entering higher education from vocational backgrounds. This would also be invaluable for students who have little or no experience with L2 academic material or little background knowledge of academic reading in general. Students entering higher education, especially from vocational backgrounds, may derive benefit from a broad reading approach in reading in English as well as an ESL instructional approach that would help equip them with the necessary reading know-how and skills that their fellow students from non-vocational backgrounds are more likely to already possess.
3 Implications for educational practice
This higher education L2 reading strategy study has added to the existing research by exploring the possible benefits of explicit reading strategy instruction in the L2 reading programme. We believe that the rigorous and meticulous development of the treatment programme based on the outcomes of a meta-analysis on what was effective in L2 reading strategy studies contributed substantially to its positive outcome. It is therefore recommended as a useful tool in order to gain valuable prior insight into what is effective in the classroom.
Furthermore, this study leaned heavily on the professionalism and dedication of the intervention teachers who were responsible for the implementation of the intervention and maintaining experimental integrity over four academic periods. For future educational practice it is beneficial to know that L2 reading strategy interventions are effective and that L1 reading strategies do not automatically transfer to the L2. Lastly, we hope to see L2 reading strategy interventions be extended to more higher education institutions in the future.
Footnotes
Appendix 1
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This reading strategy study has been made possible with the support of the Netherlands Organization for Scientific Research (NWO), grant 023.006.052.
