Abstract
In today’s globalized world, teaching intercultural competence in educational contexts is of great importance. The potential of foreign language literary texts has been emphasized repeatedly. The present study investigates the value of a content and language integrated learning (CLIL) approach to the teaching of literature in foreign language classrooms for the simultaneous development of intercultural competence and foreign language reading proficiency. An intervention based on integrated intercultural literary pedagogy (IILP) was developed. This longitudinal quasi-experimental study examined the effects of the IILP treatment in a sample of 274 Dutch secondary school students (from seven schools) who were learning Spanish as a foreign language using a switch replication design with two conditions. Three Spanish novels were read in both conditions in two consecutive school years. For the experimental condition, students performed dialogical tasks based on IILP pedagogical principles; for the control condition, the tasks consisted of comprehensive reading questions. Multilevel analysis of the data revealed that the students in the experimental condition significantly improved their intercultural competence when compared with the students in the control condition. For reading proficiency, main effects were found in both conditions. The results demonstrate that a CLIL approach adds considerably to intercultural development; a broad inclusion of literary texts in foreign language teaching environments is therefore recommended.
Keywords
I Introduction
In today’s societies, ‘the ability to interact effectively with people from cultures that we recognize as being different from our own’ (Guilherme, 2000, p. 297) – commonly known as intercultural competence – is crucial. In times of migration, social polarization, and increased cultural diversity, the need for intercultural education has become evident, preparing adolescents for ‘interaction with people of other cultural backgrounds, teaching them skills and attitudes as well as knowledge’ (Byram & Wagner, 2018, p. 140). As might be expected, intercultural competence, one of the main objectives of secondary-level foreign language teaching, has gained importance (Council of Europe, 2018). The potential offered by literary texts in this regard has been repeatedly emphasized (Burwitz-Melzer, 2001; Matos & Melo-Pfeifer, 2020; Nemouchi & Byram, 2019). However, researchers have observed that its use has declined and, when literary texts are employed, the focus is on language learning (i.e. the use of literature) or literary analysis (i.e. the study of literature). Intercultural approaches to literature teaching at the secondary level are under-researched (Bloemert et al., 2016; Paran, 2008). In addition, several survey studies (Lázár, 2011; Sercu et al., 2005; Young & Sachdev, 2011) have shown that language teachers at the secondary level still define foreign language objectives mainly in linguistic terms, and that most class time is spent on training the four language skills – reading, listening, writing, and speaking – rather than fully integrating intercultural competence.
Intercultural competence can be developed in settings where L2 literature is used. Literature is generally considered to be ideal for the integration of language and cultural learning: in addition to an analytical approach, students can use the content of texts as a gateway to ‘other’ perspectives, products, and practices 1 and examine the features of the language through which this other culture is presented (Barrette et al., 2010). But what contributes even more than the text itself – and this is a key difference with informative texts – is that reading literature is an imaginative and interpretative process. As literature stimulates the imagination, it enables readers to ‘live other lives – by proxy’ (Kramsch, 1995, p. 85) and to bring their life experiences to a text (Bredella, 1996). They can ‘de-centre their own thinking by placing themselves in somebody else’s shoes’ (Porto & Zembylas, 2020, p. 358). When students engage in this way and reflect on the process with peers in heterogeneous foreign language classrooms (Kramsch, 1995, 2014), different cultures and languages are brought to the reading experience. This provides the opportunity for profound intercultural connections and the development of intercultural competence.
Several interculturalists (Byram, 2010; Porto, 2018) have argued that Content and Language Integrated Learning (CLIL), a pedagogy frequently applied in bilingual education and widely acclaimed as an effective approach to the integration of linguistic and content objectives (Coyle et al., 2010; Ruiz de Zarobe, 2015), offers possibilities for intercultural learning. The traditional use of the term CLIL refers to a bilingual programme where a foreign language, usually English, is used in subject classes such as history or biology (Van Kampen et al., 2018). The abbreviation has recently been more broadly interpreted as any type of pedagogical approach that integrates the teaching of content and language where the focus is on spoken and written output as a means of processing content (Mearns & de Graaff, 2018). We expected to find that a CLIL approach to literature teaching would develop intercultural competence. It focuses on content (the study of literature) and form (the use of literature) to understand culture but also takes into account the ability to discuss text through speaking and writing activities.
Coyle et al. (2010) use the 4Cs framework (culture, content, communication, and cognition) to describe CLIL, arguing that CLIL focuses on the interrelationship among content (subject matter), communication (language learning and use), cognition (learning and thinking processes), and culture (developing IC and global citizenship). The Language Triptych, which was developed using the framework, is a conceptual figure that identifies three types of language in each corner of a triangle:
the language of learning, which refers to the content-obligatory language related to understanding a subject (e.g. key phrases, expressions, lexis, and content-specific language);
language for learning, which refers to functional language required for enabling learning to take place in class (e.g. language for carrying out tasks and language for group work); and
language through learning, which refers to new language linked to deeper conceptual understanding on an individual level (e.g. language that arises during the learning process and that students need to articulate in order to reiterate their own learning (Coyle et al., 2010).
The triptych helps teachers to distinguish the various ‘languages’ in a CLIL-classroom and to encourage their students to use them in various contexts rather than learning vocabulary in isolation.
Departing from the idea of ‘language through learning’, more recent research (Coyle & Meyer, 2021; Meyer, Coyle, et al., 2015) foregrounds a pluriliteracies approach to CLIL, in which the understanding and development of subject-specific literacies is a key element. The Graz group argues that ‘subject-specific literacies are the key to deep learning and the development of transferable skills. Becoming literate in content subjects or topics will empower learners to successfully and appropriately communicate knowledge across cultures and languages’ (Meyer, Halbach, Coyle, 2015, p. 2). While much research has focused either on the benefits of CLIL for linguistic progression or gains in content knowledge, new directions should position pluriliteracies development as an end-goal, namely, ‘the growing ability to express/verbalize subject-specific concepts or conceptual knowledge in an appropriate style using the appropriate genre and genre moves for the specific purpose of the communication in a wide variety of modes’ (Meyer, Coyle, et al., 2015, p. 50). Approaching literature teaching from a pluriliteracies approach demands an emphasis on the development of students’ meaning-making repertoires, providing them with opportunities to develop ‘the skills involved in reading, synthesizing, constructing arguments, evaluating and demonstrating understanding; and the language and discourse required to engage in these processes’ (Coyle & Meyer, 2021, p. 154). The idea of developing literary literacy through a content- and language-integrated approach to literary texts is not new; the multiple literacies approach (Barrette et al., 2010; Paesani & Allen, 2020; Swaffar & Arens, 2005), which integrates the study of literature and culture, is employed in US higher education foreign language curricula. As CLIL is more deeply embedded in Europe and recommended at the policy level (Council of Europe, 2018), we were particularly interested in its use in the teaching of literature.
II Context and background
The Dutch national curriculum for foreign languages is based on the Common European Framework of Reference (Council of Europe, 2001) and contains core curriculum standards for the four language skills and literature (Meijer & Fasoglio, 2007). While the standards for reading, writing, listening, and speaking are well-defined and level-indicated, the three standards for literature are less clear. Students are required to report on their experiences of at least three literary works using clear arguments (standard 1: personal literary development), to distinguish text types and apply literary concepts (standard 2: literary terminology), and to place texts in a historical perspective (standard 3: literary history). These attainment targets encompass, respectively, reading for personal development (e.g. Maley & Duff, 2007), a literary analytical approach, and contextual historical perspectives (e.g. Barrette et al., 2010) but not intercultural or linguistic objectives. The dichotomy between the specifically defined programme for language skills based on the Common European Framework of Reference (CEFR; Council of Europe, 2001), on the one hand, and the literature component, on the other, can be seen in daily teaching practice: lessons focus either on language proficiency or literature, giving adolescents only limited opportunities to explore literary texts from an intercultural perspective.
Several survey studies in the Netherlands have found that Dutch foreign language teachers – in line with the results of earlier mentioned international surveys (e.g. Sercu et al., 2005) – consider linguistic objectives to be the main focus of their programmes. Regarding literature teaching, a prevalence was expressed for the study of literature through a literary analytical text approach (Bloemert et al., 2016) and the use of literary texts for extensive reading (Lehrner-te Lindert et al., 2018; Schat et al., 2018). The studies also revealed that teachers would like to spend more time on literature teaching with a focus on cultural reflection and productive language proficiency but that they seem to be unable to put this into practice for several reasons, for example, insufficient time, guidelines, and teaching materials.
In response, the present authors (Schat et al., 2021a) formulated a series of pedagogical principles and developed a CLIL-based literature programme following an educational design research approach (e.g. McKenney & Reeves, 2018). The principles underpinning our Integrated Intercultural Literary Pedagogy (IILP) have been proven feasible and easy to implement (Schat et al., 2021a), but the effectiveness of the approach had yet to be investigated. Given the ubiquity of foreign language teaching in secondary schools worldwide and the need to examine literary instruction empirically (Paran, 2008; Schrijvers et al., 2019b), it is important that the benefits of IILP be evaluated. The present study, therefore, aimed to implement an intervention based on the pedagogical principles of IILP in secondary education and evaluate its contribution to the student participants’ intercultural competence and reading proficiency. It is hoped that the findings will justify the greater use of literature, lessen the literature–language dichotomy, and reinforce the status of intercultural competence in secondary language learning environments.
III Theoretical framework
To investigate the effects of an IILP-based intervention, it is essential to have a solid understanding of the pedagogy. Thus, in the next subsections, we review the literature, describing four key curricular components of our pedagogy: rationale, objectives, content, and learning activities (see Van den Akker, 2006). First, we set out the theories that served as the foundations for our pedagogy and describe its aims. We then describe the pedagogical principles from our previous study on IILP (Schat et al., 2021a) along lines of the 4Cs framework (Coyle et al., 2010).
1 Rationale: Intercultural competence and dialogic literature teaching
To describe the rationale behind the IILP pedagogy, we first need to describe our conceptualizations of intercultural competence and literature education, as our study is positioned between them. We use the intercultural communicative competence model proposed by Byram (1997, 2021) and define our construct using the author’s five savoirs (attitudes, knowledge, skills of interpreting and relating, skills of discovery and interaction, and critical cultural awareness). We are aware that this model has been critiqued; for example, in its conceptualization of cultures in the context of ‘countries’ (Dervin, 2017; Matsuo, 2012; Risager, 2007) and its emphasis on us–them dichotomies through comparison, rather than taking into account unequal power relations in intercultural communication (Díaz & Dasli, 2016). Still, we choose to maintain the savoirs, as they are well-known categories, and convenient when taking a systematic approach towards mapping intercultural development. However, as the present study conceptualizes the five savoirs in the context of literature teaching, one specific critique must be elaborated upon. Hoff (2014, 2020) has argued that Byram’s model relies heavily on face-to-face communication between people from different national cultures and fails to take into account other types of communication. While we agree with this critique and recognize that the conceptualization of intercultural communication as an interaction between people from different countries is limited, and that ‘communication with peers’ or ‘communication with a text’ are other potential forms of intercultural communication, we argue that perceiving the five savoirs through a framework of dialogic literature education can help to overcome these limitations.
In our understanding, dialogic literature education encourages students to reflect on the world, themselves, and others. Reflection is stimulated through dialogue with the text and dialogue with others about the text. Through an internal dialogue with the text, students become aware of their thoughts, feelings, and ideas. Through external dialogues with others concerning personal responses to the text, students explore the other (Schrijvers et al., 2019a). In essence, a dialogic approach to literature education encourages students ‘to co-construct meaning by exploring, extending and constructively challenging one another’s ideas’ (Heggernes, 2021, p. 2), as opposed to formulating a correct answer to questions about literary texts. Dialogic literature education and intercultural competence are two strongly intertwined entities within CLIL-based literature education: the former aims to relate one’s feelings, ideas, and opinions to a text, and the latter involves looking at the relationships between one’s own and other cultures. In addition, the distinction between dialogue with a text and dialogue about a text can help us to focus on two aspects of intercultural communication within the literature classroom: the intercultural communication a student has with ‘the other world’ embodied by the text and intercultural interaction between peers in diverse classrooms. At a more general level, these dialogues may promote the kind of subject-specific literacies proposed by Meyer, Coyle, et al. (2015) and Meyer, Halbach, Coyle (2015).
Based on this dialogic conceptualization of CLIL-based literature education, we propose a model for intercultural literary competence (see Figure 1). Attitudes (1), knowledge (2), and skills of interpreting and relating (3) can be developed through an internal dialogue with the text. Engaging in external dialogue with others about literature stimulates the development of skills of discovery and interaction (4) and critical cultural awareness (5). Our construct hinges upon a student’s general target language proficiency. First, students need a certain level of reading ability in the foreign language to be able to engage in dialogue with L2 literature. Second, the success of external dialogue about the literature very much depends on the level of speaking and writing skills in the target language. We are aware that much of the research on dialogic literature education has focused on L1 classrooms (Schrijvers et al., 2019b), and that exploring and challenging one another’s ideas through authentic questions in the foreign language takes place at another level. However, we argue that this dialogic conceptualization of intercultural competence – stimulating reflection on the self through dialogue with and about a text in a foreign language – suits the CLIL paradigm, where content is processed through writing and speaking ‘and learning has to translate into the learners’ ability to articulate their knowledge and understanding appropriately’ (Meyer, Coyle, et al., 2015, p. 44).

Model for intercultural literary competence (ILC).
2 Aims and objectives: Intercultural literary competence (ILC)
a Dialogue with literature
Whereas savoir-être in Byram’s model refers to the student’s ability to approach intercultural learning with curiosity and openness and relativize the self and value the attitudes and beliefs of others, attitudes in the ILC model refer to the student’s willingness to read a literary text in a foreign language and the ability to engage with different cultural settings or characters. Savoirs in Byram’s model do not so much refer to fact-based knowledge about the foreign language culture as to knowledge of self and others, how interaction occurs, and the relationships between individuals and societies. As savoirs comprise reflexive and relational knowledge, which comes from reflecting on social structures and power relations, knowledge in the ILC model is defined as the ability to use a literary text to reflect on knowledge of social groups and the general processes of individual and societal interaction. Savoir comprendre refers to the ability to interpret and relate to information. In the case of ILC, this involves the skill of understanding textual elements of a literary text but also the ability to relate ‘the cultural’ in a text to one’s own life experiences.
b Dialogue about literature
Whereas Byram’s model describes savoir apprendre/faire – knowing how to interact and discover – as the student’s ability to make discoveries through personal involvement in social interaction, the ILC model stresses the ability to engage in dialogic talk and writing with others about the literary text and to explore not only the multiple voices in the text but also their peers’ voices. Savoir s’engager in the original model is defined as ‘an ability to evaluate, critically and on the basis of explicit, systematic process of reasoning, values present in one’s own and other cultures and countries’ (Byram, 2021, p. 66); it refers to the political consequences of education, that is, being critically aware of cultural behaviours. In the ILC model, savoir s’engager refers to the student’s ability to make informed critical evaluations of aspects of one’s own and other cultures and the literary text by writing a thorough analysis, identifying and interpreting its ideological values, and evaluating it based on systematic and conscious reasoning and with reference to their own ideological perspectives.
3 Culture and content: Critical interculturality
While the intercultural communicative competence model was developed as a reaction to the communicative paradigm – arguing that the domain of foreign language teaching became too functional because it focused overly on skills, thus missing a profound educational dimension (Byram, 2010, 2021) – researchers in the field of critical interculturality (Dervin, 2017; Díaz & Dasli, 2016; Hoff, 2020) have in turn denounced the model for serving merely utilitarian purposes and lacking awareness of power relations and conflict. They have argued for a more critical approach that includes content relating to ethical, social, and political issues. Theories of critical pedagogy (Guilherme, 2002) and intercultural citizenship (Porto et al., 2017), resound across the field of literary studies. Various scholars have argued that literary texts discussing controversial and social issues are apt to teach interculturality (Matos & Melo-Pfeifer, 2020). Texts dealing with social justice may stimulate not only perspective-taking and decentring but also students’ emotional engagement (Porto & Zembylas, 2020), as well as their ability to deal with differences in moral opinions. In our earlier IILP study we have formulated two design principles for text selection. Literary texts should be chosen that problematize culture and understand it in a non-essentialist way (DP1) and address themes of social justice to which students can relate directly (DP2). They should preferably involve a juvenile protagonist, as this facilitates identification (Schat et al., 2021a). The selection of these kinds of texts may stimulate the development of critical literacy, that is, the understanding that language and texts always contain within them ideological perspectives (Coyle & Meyer, 2021).
4 Communication and cognition: Subject-specific literacy
In response to the call for an intercultural literary pedagogy that stimulates dialogue with and about a text, we suggested in our earlier IILP study (Schat et al., 2021a) that students engage in tasks following the cycle of intercultural language learning as described by Liddicoat and Scarino (2013); that is, noticing, comparing, interacting, and reflecting. According to the third and fourth design principles of IILP (DP3 and DP4), students have to complete a spoken or written task in each lesson in which they are encouraged to notice cultural practices, perspectives, or products in the literary text and to write a spontaneous personal response to them. It is important that the teacher does not interfere at the noticing stage, as this is a dialogue between the text and the reader. The second step is to compare and explore one’s responses with others, in the text or in the classroom. At the third step, students are encouraged to interpret their experience and to interact, either in the written task or in dialogue with peers. At the last step, students reflect and discuss what they have learnt or what they will take away from the task in classroom discussions with their peers and the teacher. Because all steps in this cycle are carried out in the target language, and students perform a ‘variety of dialogues’ at a ‘variety of registers,’ IILP tasks stimulate ‘language’ subject-specific concepts and knowledge in an appropriate style using the appropriate genre moves for the specific purpose of the communication in a range of modes’ (Meyer, Coyle, et al., 2015, p. 52).
In the above IILP tasks, cognition and communication go hand in hand as students are prompted to articulate their learning and to use ‘language’ that bridges linguistic progression and conceptual knowledge-building. However, as new concepts are learnt, new language is not only acquired but also needed. In our earlier IILP study, which focused on the development and evaluation of materials (Schat et al., 2021a), we found that students lacked the necessary expression to talk and write about literature (the language of learning) and functional language to progress through the steps of this cycle (language for learning). We therefore provided scaffolds for the language of learning and language for learning in the tasks. From a pluriliteracies perspective, it is interesting to discover not only how ‘language through learning’ emerges through IILP teaching (by investigating the language students acquire to put into words their own intercultural learning processes), but also how ‘a deeper learning of literature challenges learners to move beyond the stage of surface learning – where texts are treated as stories – to reach a level of interpretation where texts are treated as complex cultural discourse’ (Coyle & Meyer, 2021, p. 156).
IV Current study and research questions
As this literature review has shown, IILP has great potential for intercultural development. While CLIL sounds promising on paper, studies that have investigated it in the context of L2 literature teaching are lacking, with the exception of Ballester-Roca & Spaliviero’s (2021) theory-based study and a study carried out in higher education using quantitative data (Rodríguez & Puyal, 2012). However, as no control groups were involved, a CLIL effect could not be demonstrated empirically in either instance. There is therefore a need for (quasi-)experimental research to illuminate whether and how CLIL-based literature teaching might foster students’ intercultural competence. As Dutch foreign language programmes in the upper forms are heavily oriented towards training reading skills (Michel et al., 2021), with literary texts used for extensive reading, we wanted to explore how IILP might contribute to general reading proficiency in the foreign language as well as the intercultural understanding of a text. Furthermore, studies investigating the effects of intercultural pedagogical classroom practices over an extended period are lacking (Zhang & Zhou, 2019). To fill this gap, we implemented and evaluated a longitudinal IILP intervention in a class for Spanish as a foreign language in the Netherlands for upper-secondary students aged 16–18 years. Because we aimed to investigate the benefits of this newly designed intervention for intercultural competence and reading proficiency, our research questions were as follows:
Research question 1: What are the effects of an Integrated Intercultural Literary Pedagogy intervention on students’ intercultural literary competence?
Research question 2: What are the effects of an Integrated Intercultural Literary Pedagogy intervention on students’ reading proficiency? 2
V Method
1 Research design
We adopted a quasi-experimental research approach. As difficulties in randomly assigning participants to the control and experimental conditions are inherent in school settings, we chose a switching replication design. A design with switching replications (Shadish et al., 2002) allows interventions to be implemented in two different groups but at different times. As the interventions are administered to both groups, a switching design is a more solid design than a regular pre–post quasi-experimental design in terms of internal validity; if the intervention is equally effective in both groups, one can rule out effects other than the pure intervention effect, such as teacher effects or group characteristics. It also adds to external validity; administering an intervention to different groups offers more information about the extent to which the results of a study can be generalized to and across other participants. It is also a more ethical design than a regular pre–post (quasi-) experimental design because all students, instead of merely a selection of them, can enjoy the intervention. As Table 1 shows, our longitudinal design consisted of two phases (the school years 2019–20 and 2020–21), two groups (group A and B), three measurement occasions (M1–M3), and two conditions: the experimental and the control condition. In the first phase of the study (2019–20), between M1 and M2, group A was the experimental group and group B was the control group. After M2, in the second phase of the study (2020–21), the groups switched conditions: group B moved to the experimental condition while group A served as the control group. Students in the experimental condition were exposed to an IILP intervention and students in the control condition were exposed to the text comprehension approach (TCA), 3 a common approach in foreign language teaching in the Netherlands.
Schematic representation of the research design.
2 Participants
Our intervention was aimed at secondary school students in their third and fourth year of Spanish-as-a-foreign-language studies. The target group comprised 15- to 18-year-old students in grades 10 and 11. Several teachers of Spanish at seven different schools volunteered to participate. Classes were randomly assigned to the experimental and control conditions, except at one school, where there was only one group. An extra control condition was set up at another, similar school. Thus, in the first phase, a total of 14 classes participated, of which seven were in the experimental condition (IILP) and seven were in the control condition (TCA). Because one school withdrew from the project after the 2019–20 school year (when the students moved to grade 11), a total of 11 classes participated in the second phase, of which six were in the experimental condition and five in the control condition. School leaders authorized the project, the parents were informed about it, and the students gave their informed consent. As the educational intervention was part of the regular educational programme for Spanish, all students participated. One student withheld consent regarding data collection, and four students were excluded due to poor attendance. Thus, a total of 10 teachers and 269 students participated in the first phase of the project. In the second phase, a total of eight teachers and 204 students participated. The project was approved by the ethics committee of Utrecht University.
a School and teacher demographics
All the participating schools were public schools: five were located in the west, one in the east, and one in the south. The sample consisted of one rural school, two suburban schools, and four urban schools. The selection of the seven schools was based on the teachers’ interest in participating. All 10 teachers were qualified and experienced. On average, the teachers of group A had 9.0 years of experience in teaching Spanish at the secondary level. The teachers of group B had 9.2 years of experience, so the difference was insignificant. The majority of the teachers were female (86%); this percentage did not differ significantly between groups.
b Student demographics
At the start of the project, the experimental condition consisted of 145 students (60% female), and the control condition, 124 (53.2% female). The distribution of boys and girls did not differ between the experimental and control groups: t(267) = −1.12, p = .265. The students’ ages ranged from 15 to 18 years, with an average of 16.34 years (SD = .56) at M1, which did not differ substantially between groups: t(267) = .21, p = .837. The most frequently spoken home languages apart from Dutch (90%) were Arabic (2%), English (1%), and Chinese (1%).
3 Intervention
a Selected reading materials for the intervention
Based on the pedagogical principles for content selection, as discussed in the theoretical framework (DP1 and DP2), we chose migration as an overarching topic for the intervention. Aside from being an intercultural experience in itself, migration is an important topic for the participants. In line with the core curriculum standards, we selected three texts that depicted migration in different parts of the Spanish-speaking world, ranging from level A2 to B1+, according to the CEFR (Council of Europe, 2018): Los Ojos de Carmen [‘The eyes of Carmen’] (Moscoso, 2020), Caravana al Norte [‘Caravan to the North’] (Argueta & Monroy, 2019), and Abdel (Páez, 2015).
b Intervention planning
For the school year 2019–20, we designed an eight-lesson sequence for Los Ojos de Carmen and a six-lesson sequence for Caravana al Norte. For the school year 2020–21, we designed a 12-lesson sequence for Abdel. The texts were not equally divided between phases 1 and 2, but we chose this division to enhance meaningful learning–teaching trajectories while taking into account the Dutch standards for literature education; planning two shorter texts in the semi-final year and one of greater length in the final year is more appropriate for foreign language programmes. In the first phase (November 2019 – January 2020), the teachers scheduled eight literature lessons of approximately 60 minutes for Los Ojos de Carmen and six lessons for Caravana al Norte. In the second phase (November 2020 – January 2021), teachers scheduled 12 literature lessons of approximately 60 minutes for Abdel.
c Tasks for the intervention
Two types of workbook were developed for the three texts: IILP workbooks and TCA workbooks. In both conditions, students filled out the tasks in the accompanying workbooks using the texts, though the treatment differed. In the workbooks for the experimental condition, the tasks were based on the IILP principles for task design. They contained a variety of dialogic tasks (DP3) for each chapter and were based on the four steps of noticing, comparing, interacting, and reflecting (DP4). The workbooks for the control condition comprised comprehensive questions about each chapter, as per a typical TCA approach. The difference in approaches is visualized in Appendix A in supplemental material. Appendix B in supplemental material presents an overview of all tasks for each IILP sequence.
4 Instrument and data collection
a Measures of intercultural literary competence
A self-evaluation instrument to measure ILC was developed and validated in an earlier study (Schat et al., 2021b). The intercultural literary competence questionnaire (ILCQ) is a digital self-assessment questionnaire based on the ILC model and its five dimensions. As Figure 2 shows, students assess themselves on a 7-point Likert scale comprising 16 ILC can-do statements. The ILCQ contains four items for attitudes (1, 4, 8 and 13), three for knowledge (5, 14, 15), three for skills of interpreting and relating (items 6, 9, 11), three for skills of discovery and interaction (7, 10, 12), and three for critical cultural awareness (2, 3, 16). Because the ILCQ is a new instrument, developed specifically for the target group and the particular educational context with the aim of mapping intercultural development through the reading of literary texts, we had evaluated its validity and reliability previously. We first performed exploratory factor analysis followed by confirmatory factor analysis. We used RMSEA (<.06), TLI (>.95), and CFI (>.95) as fit indices. The results showed that the ILCQ was a valid measure of ILC, as it measured five distinguishable but interrelated factors. After the fit of the questionnaire proved to be valid, the reliability of the instrument was examined in an implementation phase. With sufficient to high Cronbach’s alphas for all subscales (.74–.88) and significant correlations between the test and the retests (r = .69), the instrument’s reliability was also confirmed. The ILCQ had also been used in a pilot study (Schat et al., 2020), where it demonstrated its capacity to measure intercultural development. We used the ILCQ at three moments (ILCQ1, ILCQ2, and ILCQ3).

The intercultural literary competence questionnaire.
b Measures of reading proficiency
To measure reading proficiency, we used materials developed by the Central Institute for Test Development (CITO), an organization that specializes in educational measurement and is responsible for the Dutch national exams. The reading tests used for the present study comprised a selection of texts and items from past national exams. We used them at three moments: R1, R2, and R3. Each test contained four texts with 17 multiple-choice questions. Each test covered the four domains of language learning and contained a text on daily life, work, education, and public life. All tests had comparable mean p values for the texts used (R1 = 66.5; R2 = 66.5; R3 = 65.5) and mean word length (R1 = 1,685; R2 = 1,659; R3 = 1,687). We are confident that the tests are reliable and valid instruments, as will be demonstrated through psychometric analysis.
At the first measurement occasion (M1), the students completed the questionnaire (ILCQ1) and a reading test (R1) as pre-tests. At the end of phase 1, the students completed the post-tests (R2 and ILCQ2). For phase 2, the data collected at the second measurement (M2) were used as a pre-test, and at the end of phase 2 (M3), the students completed the third reading test (R3) and the questionnaire (ILCQ3) as post-tests. Because the reading tests were administered at three moments in two consecutive school years, the level of the tests gradually changed. R1 was at B1 level, R2 was at B1+ level, and R3 was at B2 level. This was in accordance with the CEFR (Council of Europe, 2018). Can-do statements on the ILCQ did not change but the 16 ILC items were presented in a different order at each measurement. Pre-tests were administered two weeks before the first lesson and post-tests three weeks after the lessons. Questionnaires were administered digitally and on paper. Reading tests were administered during regular hours by the teachers.
5 Implementation fidelity
To make sense of the effects of an intervention, it is important to describe measures of implementation fidelity. This is an estimate of the extent to which an intervention is implemented as intended. Due to difficulties in collecting the data, measures of implementation fidelity were of great importance in the present study. O’Donnell (2008) describes four criteria for measuring implementation fidelity:
adherence, which refers to whether the components of the intervention are being delivered as designed;
duration, which is a measure of the number, length, or frequency of sessions implemented;
quality of delivery, which touches upon how the implementer delivers the programme; and
participants’ responsiveness, which relates to the engagement and involvement of the participants.
To ensure implementation fidelity, we operationalized the construct using these criteria. The teachers filled out self-evaluation forms in which they evaluated the criteria, while the researchers observed one lesson at each school. The students’ workbooks were reviewed for direct assessment purposes.
a Teacher evaluations
The teachers in the experimental condition were asked to fill out an evaluation form during and after each intervention. They rated criteria 1 and 2 (adherence and the duration of their programs) on a scale of 1–3, where 1 was not sufficient, 2 was average, and 3 was good. For criteria 3 and 4 (quality of delivery and participants’ responsiveness), teachers rated themselves and their students on a scale of 1–10. They rated adherence as average: M = 2.13; SD = .35. They argued that they had difficulties organizing step 4. They rated the duration of the programmes as good: M = 2.50; SD = .53. Analysis of the data revealed that the teachers taught all the lessons in the workbooks. However, they found it difficult to teach the respective units in an hour. The quality of delivery was rated as moderate (M = 7.13; SD = .83), as was responsiveness (M = 7.10; SD = .77).
b Classroom observations and analysis of the students’ workbooks
We planned to carry out observations in the classrooms once for each intervention at every school. This objective was accomplished for the first intervention. A researcher sat at the back of the classroom, observing the full length of the lesson approximately halfway through phase 1. Due to Covid-19 regulations, from March 2020, school visits were no longer possible. During interventions 2 and 3, two digital lesson observations were carried out at most of the schools. After each intervention, we collected the workbooks to check for adherence and duration. We randomly reviewed the workbooks of two schools per intervention. The duration was operationalized in the number of lesson units students completed, and adherence was operationalized if the four-step tasks were accomplished sufficiently. Analysis indicated that duration was high: on average, the students conducted eight out of 10 lesson units. Adherence was lower: the last step in the four-step model was completed in only six out of 10 lesson units.
6 Data analysis
Because our measurements were nested within the students, and the students were nested within classes, and the classes were nested within schools, we performed a multilevel analysis. This takes into account the hierarchical structure of data. We used Linear Mixed Models in SPSS. To estimate the effect of the intervention, several models were analysed. The fit of the models was compared using a log likelihood test (−2LL). For the first model (the basic null model), we built a fixed intercept-only model to estimate the grand mean. To account for the nesting of repeated measures within the students and the classrooms, we included student ID and class as random effects. That is, ILCQ scores were allowed to vary within the students and the classes. To this basic null model, we successively added parameters that might explain the differences in intercultural competence. In the second model, time was added as a fixed effect to investigate whether change occurred over time, regardless of condition. In the third model, group was added as a fixed effect to check whether average scores differed between groups. In the fourth model, the interaction between group and time was estimated. This last model tested the effect of the intervention by examining whether change over time differed between groups. The results of phase 1 and phase 2 are presented separately. Because the sample in phase 2 involved fewer participants than phase 1, two separate data sets were used. There was sufficient power for two analyses.
7 Pre-analyses
The suitability of the data was assessed before they were analysed. To assess whether the data were normally distributed, we created box plots, P–P plots, and histograms for each dependent variable. We found several outliers for ILCQ. We did not remove outliers except for three responses at M1, as all items were rated 1. All data collected were included in the analysis (with the exception of the three aforementioned cases). The normality assumption was supported by normal probability plots representing a straight line and by values of skewness and kurtosis less than 1. To assess the reliability of our instruments, we calculated Cronbach’s alpha for the ILCQ at each measurement occasion for each condition (see Table 2). The reliability for the total scale was high at all moments for both conditions (.88–.91). All subscales also showed sufficient to high coefficients (.61–.84), except for the interpretation scale at M3 in the experimental condition (.59). We also calculated the reliability of the reading tests after each administration (R1, R2, R3). All three tests showed high coefficients (respectively α = .69, α = .72, and α = .70).
Reliability for the intercultural literary competence questionnaire (ILCQ) and its subscales.
VI Results
1 Effects of IILP on intercultural literary competence
Table 3 shows the results of the fit and comparison of the four models. There was a fixed effect for time (model 2 vs. model 1) in phase 1 [F(1, 255.00) = 18.66, p < .001] and phase 2 [F(1, 11.76) = 35.52, p < .001], indicating that average scores were not equal over time and that the mean scores of the post-tests were significantly higher than pre-test scores in both phases. The main effect for the group (model 3 vs. model 2) was statistically significant neither in phase 1 [F(1, 14.33) = .24, p = .637] nor in phase 2 [F(1, 11.76) = .20, p = .675], indicating that average scores were the same for students in groups 1 and 2. The interaction between time and group (model 4 vs. model 3) was significant in phase 1 [F(1, 255.63) = 3.93, p = .049] and phase 2 [F(1, 184.98) = 5.33, p = .024], indicating that there was a statistically significant effect for the intervention. Table 4 shows the average scores per condition per measurement moment of both phases. Inspection of the average scores of both groups at both time points shows that the students in the experimental condition made more progress (phase 1 = .32 and phase 2 = .58) than the students in the control condition (phase 1 = .11 and phase 2 = .24). Although the effect sizes were small (d = .17 and d = .28), the differences in mean scores over time and group were both significant, which means the intervention proved successful.
Model comparisons.
Average scores and standard deviations on the intercultural literary competence questionnaire (ILCQ).
Parameter estimates were also calculated for the five separate dimensions of ILC in both phases. In both phases, significant interaction effects were found for skills of discovery and interaction: F(1, 257.88) = 5.93, p = .016 and F(1, 185.94) = 13.33, p < .001, respectively. For critical cultural awareness, significant interaction effects were found in both phases: F(1, 259.02) = 4.59, p = .033 and F(1, 186.08) = 4.05, p = .046, respectively. In the second phase, significant interaction effects were also found for skills of interpreting and relating: F(1, 186,71) = 8.09, p = .005. We did not find significant interaction effects for attitude and knowledge, but significant main effects for time were found for knowledge in phase 1 [F(1, 255.43) = 6.72, p = .011] and phase 2 [F(1, 187.70) = 3.98, p = .048]. Figure 3 contains a graphical representation of the intervention effect for intercultural literary competence for the two phases.

Effects of integrated intercultural literary pedagogy (IILP) on intercultural literary competence (scale 1–7).
2 Effects of IILP on reading proficiency
Regarding reading proficiency, we also found a significant main effect in both phases respectively [F(1) = 70.95, p = <.001 and F(1) = 6.34, p = .013] but no significant interaction effect (p = .311 and p = .488, respectively). Table 5 shows the average scores per condition per measurement moment for both phases. Figure 4 contains a graphical representation of the intervention effect on reading proficiency for the two phases.
Average scores and standard deviations on reading scores.

Effects of integrated intercultural literary pedagogy (IILP) on reading proficiency (scale 1–10).
VII Discussion
We tested the effectiveness of an IILP-based programme developed for Dutch students of Spanish as a foreign language in the upper forms of secondary school. Teachers implemented the IILP intervention in their general education classrooms. In the classes in the control condition, the same literary texts were read but with a different approach; a text comprehension programme was implemented. This treatment was repeated the subsequent school year with switched conditions. The results showed that the IILP-based programme had a positive effect on the intercultural competence of the students. Intercultural development was significantly higher in the experimental conditions than in the control conditions in both phases of the project. The finding that the effect of the intervention was equally effective in both phases of the experiment also allowed us to show that it could be replicated and that it does not hinge upon this specific sample. In addition, the study demonstrated a maintenance effect for the IILP. Although intercultural development is not a linear process, we did not observe a relapse in the intercultural self-evaluations of the experimental group after the conditions were switched.
Focusing on the five dimensions of intercultural literary competence, one of our most important findings was that our intervention had a large effect on the students’ perceived skills of discovery and interaction and critical cultural awareness. In terms of the five savoirs of the theoretical model of intercultural literary competence, it is worth noting that both interaction effects (i.e. skills of discovery and interaction and critical cultural awareness) are related to ‘dialogue about literature’ as opposed to ‘dialogue with literature’. Although we have not demonstrated by means of direct assessment that students in the experimental conditions outperformed the students in the control conditions in their levels of speaking and writing proficiency, the finding that students in the IILP programme ranked themselves significantly higher on the savoirs for ‘dialogue about literature’ than students involved in the text-comprehension programme in both phases of the experiment suggests that they experienced direct benefits from the IILP lessons to engage in dialogic activities that involve talking and writing about literature.
Regarding research question 2, the results showed that both approaches to literature teaching were equally effective for reading proficiency. We did not find a significant interaction effect for reading proficiency, but we did find a significant main effect for time. These results demonstrate that for the development of reading comprehension, it makes no difference whether literature programmes place a strong on focus text comprehension or on teaching literature from an intercultural perspective. Nevertheless, the fact that we found a main effect leads us to conclude that using literary texts may be beneficial for improving reading proficiency. Although we cannot provide evidence for that statement, as we would have needed another research design with an extra control group that did not make use of literary texts at all, this finding adds to the growing corpus of research calling for the broad inclusion of literary texts in foreign language teaching (Heggernes, 2021; Paran, 2008; Swaffar & Arens, 2005).
1 A pluriliteracies perspective
One of our most important findings is that our intervention has a large effect on students’ perceived skills of discovery and interaction. This finding may tentatively suggest that a CLIL-approach adds to the development of literary literacy, the subject-specific language to engage in social interaction about literature. As it is argued in the pluriliteracies theory that literacy development is key to CLIL, this finding contributes to the emerging area of research that points to CLIL as beneficial for language as a tool which enables meaning-making and knowledge construction in communication. The students in this study learned language in such a way that it developed their meaning-making potential to communicate about literary and cultural concepts, ‘strengthening the connections between the conceptual continuum and the communication continuum’ (Meyer, Coyle, et al., 2015, p. 50). From a pluriliteracies perspective we can thus argue that the specific CLIL-approach to literature teaching added most to the ‘growing ability to express/verbalize subject-specific concepts or conceptual knowledge in an appropriate style using the appropriate genre and genre moves for the specific purpose of the communication in a wide variety of modes’(Meyer, Coyle, et al., 2015, p. 50).
Another remarkable finding is that we were not able to find a significant interaction effect for knowledge, while we did find significant effects for time in both conditions in both phases. This findings leads us to conclude that reading literary texts in foreign language class, regardless of which pedagogy is used, is beneficial for expanding one’s cultural knowledge and underpins the idea that a literary text is the ideal textual environment use for profound integration of language and cultural content knowledge within the foreign language domain (Barrette et al., 2010; Swaffar & Arens, 2005). In addition, this finding reveals that both approaches led to ‘progression on the knowledge pathway’ (in this case knowledge about culture and literature), but that this progression was only accompanied by ‘a progression in learners subject-specific literacies’ in the groups working with the CLIL-approach (Meyer, Coyle, et al., 2015, p. 50).
2 A critical interculturality perspective
A second important finding is that the IILP programme stimulated the development of critical cultural awareness in both phases of the experiment. This finding is in line with recent research that points to criticality as being fundamental to intercultural competence and supports the claim that critical cultural awareness ought to play an important role in intercultural education. Researchers have argued that critical cultural awareness should be regarded as an educational objective in itself and that its explicit promotion is paramount for language teaching (Guilherme & Sawyer, 2021). Scholars (Porto & Zembylas, 2020; Yulita, 2016) have stressed repeatedly that critical cultural awareness is essential to literature education and ‘should be consciously integrated into a pedagogical approach that aims at developing learners’ intercultural competence through reading literary texts’ (Matos, 2005, p. 68). The findings from the present study indicate that giving students ample opportunities to talk and write about social justice topics in their own cultures and other cultures, and to explore juxtaposing views, improves the capacity ‘to evaluate, critically and on the basis of an explicit, systematic process of reasoning, values present in one’s own and other cultures and countries’ (Byram, 2021, p. 66). The study also supports the idea that critical literacy can be developed through the use of literary texts (Coyle & Meyer, 2021). As ‘dialogue with’ the selected texts encouraged students to reflect on power relations, and ‘dialogue about’ literature fostered their ability to deal constructively with differences in moral opinions, IILP may be regarded as a practical way to integrate critical interculturality in secondary language classrooms (not only through critical content but also through criticality in tasks).
3 Limitations and implications for future research
The most important limitation of the present study is that it mapped intercultural competence development with a self-evaluation instrument, which is a form of indirect assessment. While many researchers have suggested assessing intercultural development using self-assessment instruments due to the difficulties of assessing someone else’s attitudes (Byram, 2021; Hoff, 2020), others have argued that these instruments have validity problems because appropriateness can be measured only through others’ perspectives (Deardorff, 2006). Therefore, the ILCQ should ideally be complemented by other more direct forms of assessment. As it was beyond the scope of the present study, we did not analyse the workbook writing tasks, though the data were collected. We did so in a follow-up study and investigated how critical cultural awareness was manifested in the students’ work (Schat et al., 2022). We also intend to demonstrate using direct assessment how IILP increases the students’ ability to communicate and reason critically, thus demonstrating their literary literacy in a foreign language and their capacity to relate literary content to political and social issues in the real world.
Another limitation of the present study is that we did not explore gains in productive language proficiency. We would therefore recommend that researchers examine how IILP might contribute to the development of speaking and writing proficiency. Notwithstanding these limitations, the study provided empirical evidence of the value of CLIL-based literature teaching for intercultural competence. Because reading proficiency improved in both groups, it has also indicated that the inclusion of literary texts may be beneficial for language development. Responding to the call for more empirical research on the use of literature in language classrooms (Paran, 2008; Schrijvers et al., 2019b), the results amply demonstrate the value of literature in language education, not only for language proficiency but also intercultural learning, which is a primary objective of foreign language teaching at the secondary level (Heggernes, 2021). In practical terms, the pedagogical principles underlying the intervention may be used as tools by practitioners in the field; they show that integrated language–literature instruction (Barrette et al., 2010; Swaffar & Arens, 2005) can be embodied in secondary schools and that CLIL – a pedagogy often used in bilingual education – can be applied to stimulate the development of literary and critical literacy in language learning environments.
Supplemental Material
sj-docx-1-ltr-10.1177_13621688231156391 – Supplemental material for Implementation of an integrated intercultural literary pedagogy intervention in Spanish-as-a-foreign-language classrooms in the Netherlands: An effect study at the secondary level
Supplemental material, sj-docx-1-ltr-10.1177_13621688231156391 for Implementation of an integrated intercultural literary pedagogy intervention in Spanish-as-a-foreign-language classrooms in the Netherlands: An effect study at the secondary level by Esther Schat, Ewout van der Knaap and Rick de Graaff in Language Teaching Research
Supplemental Material
sj-docx-2-ltr-10.1177_13621688231156391 – Supplemental material for Implementation of an integrated intercultural literary pedagogy intervention in Spanish-as-a-foreign-language classrooms in the Netherlands: An effect study at the secondary level
Supplemental material, sj-docx-2-ltr-10.1177_13621688231156391 for Implementation of an integrated intercultural literary pedagogy intervention in Spanish-as-a-foreign-language classrooms in the Netherlands: An effect study at the secondary level by Esther Schat, Ewout van der Knaap and Rick de Graaff in Language Teaching Research
Footnotes
Acknowledgements
The authors would like to thank the teachers and students who participated in this project.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is part of a PhD project funded by a Dudoc-Alfa Research Grant for teachers in the Netherlands (project code: DA2-2017-08).
Data availability
The data that support the findings of this study are available on request from the corresponding author, E. van der Knaap. The data are stored on a secured server of the Research Institute and shall remain restricted for access during ten years after completing the PhD project.
Supplemental material
Supplemental material for this article is available online.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
