Applying the Integration Dimensions of Quantitative and Qualitative Methods in Education Policy Research: Lessons Learned From Investigating Micro Policymaking in Norwegian Schools

Abstract

Qualitative research methods have been predominantly used to understand the complex phenomenon of educational policy enactment at the school level. This article contributes to the field of mixed methods research by applying three dimensions of integration—rationale, study purpose, and research design—to the study of micro-policymaking in schools. By examining these three dimensions, this article contributes to the limited body of mixed-methods models in this field. The findings show that the mixed methods approach provides unique insights into education policy scholarship, thus broadening the understanding of the relations and purposes of accountability and control in education.

Keywords

mixed methods integration multiphase educational policy rationale dimension

This article examines the use of mixed methods in education policy research by evaluating the following three dimensions of integration: rationale, study purpose, and research design. The multiphase research project primarily focuses on localized micro-policymaking, investigating micro-level enactments of national testing policies and examining the interactions between school principals, senior leaders, and teachers while using data from the national test results in Norwegian lower secondary schools. Following a brief introduction to the study, this paper examines the selected integration dimensions of the multiphase mixed methods design approach before summarizing and discussing the main findings. This article contributes to the field of mixed methods by providing the lessons learned from applying three dimensions of integration to the study of micro-policymaking in schools as part of education policy research. This paper therefore contributes to the discussion among education policy researchers who are flexible toward using mixed methods but uncertain about the benefits of doing so.

The Micro-Policymaking Project

This study develops new insights into national testing practices and regulations in Norway and determines how school principals, senior leaders, and teachers make use of the national test results to enhance individual student learning and school quality (Gunnulfsen, 2017). As national policy demands that the national test results are used to enhance individual student learning and school quality, I identify and discuss school professionals’ micro-policymaking in an educational system rooted in social democratic norms and values of trust and equity (Møller et al., 2007; Ottesen & Møller, 2016). I thus aim to answer the following research question: How and why do school professionals in a Norwegian social democratic context make use of the national test results? The Norwegian educational context is different from that of countries with longer traditions of education regulation based on large-scale student data and external accountability, such as England and North America. The concept of micro-policymaking is a theoretical perspective that has been developed building on Bowe et al. (1992), who claimed that, through social interaction, local school actors can be understood as contributors to policymaking and enactment.

The concept of “making use of” the national test results is explained as school professionals’ construction of discourses regarding the central government’s demands for national testing policies, how they deal with the aims of such policies, and how power and talk play into these policies. This project is particularly concerned with policymakers’ increasingly transnational emphasis on the relations between school quality and large-scale student assessment, and with how new modes of accountability influenced by new public management have entered the educational context of Norway, which has long been dominated by social democratic values (Gunnulfsen & Møller, 2017). National testing in the Norwegian context includes basic skills in reading, numeracy, and English reading. In Norway, the government views test results as tools for improving school quality and enhancing individual student learning (Norwegian Directorate for Education and Training, 2013).

This project uses a qualitative dominant design (Creswell & Clark, 2017) because two of the data collection phases are qualitative (school leader interviews and onsite observations/video recordings from meetings). The approach is described as a sequential exploratory design, with one form (e.g., quantitative data) building on the other (e.g., qualitative data) to identify important and unknown variables between the data (Creswell & Clark, 2017; Fetters et al., 2013; Johnson & Schoonenboom, 2016). The survey in Phase 3 was constructed on tentative conclusions, patterns, and generalizations based on the qualitative data obtained in Phases 1 and 2. The aim of combining these methods was to provide different perspectives on the school principals’ and teachers’ work with the national test results. At the time of this research, no equivalent studies had been conducted in the Norwegian education context on this topic. This article further accounts for the integration of mixed methods in education policy research and describes the phases, selection of participants, data collection, and analytical concepts used.

Integration of Mixed Methods in Education Policy Research

The discussion of integration in mixed methods research has grown substantially, especially within the fields of health studies, social psychology, and sociology (Bryman, 2007; Creswell, 2013; Guetterman et al., 2015). Integration means bringing the quantitative and qualitative strands of a mixed methods study together for analysis. According to Creswell (2009), most of the literature on mixed methods research can be assumed to be generic to any chosen field; however, developing methods literature in the context of a particular discipline will also help strengthen the field of mixed methods research. Existing international research on methods in education policy has focused mainly on qualitative data (e.g., Levinson et al., 2020; Owen, 2014; Saarinen, 2008), while research on education policy regarding accountability, data use, and the role of numbers focuses more on quantitative data (e.g., Gorard, 2001). Additionally, the use of mixed methods in education policy research has been discussed within the field of education program evaluation (Burch & Heinrich, 2015). Nevertheless, qualitative research methods have been predominantly used to understand the complex phenomenon of education, local school practices, and political aspects relating to educational policies, programs, and curricula (Ponce & Pagán-Maldonado, 2015).

Recent mixed methods research literature has explored several ways to combine qualitative and quantitative research as mixed methods (Bryman, 2012; Fiorini et al., 2016). According to Wolf (2010), mixing methods in comparative public policy research is promising in many contexts, but it is still not an obvious solution to all types of research questions. It can be demanding to discuss the challenges and opportunities of fully integrating quantitative and qualitative methods (Jones, 2017) and to identify the numerous typologies and dimensions of partially or fully mixed methods designs (Leech & Onwuegbuzie, 2009, p. 269). Further, practical, theoretical, and philosophical challenges to integration exist when conducting mixed methods research (Creswell & Clark, 2017; Fetters & Molina-Azorin, 2017). According to Fetters and Molina-Azorin (2017), recent advances in mixed methods show that researchers integrate methods throughout all dimensions of their research and that the integration dimensions need to be made clear to guide the emergent exercise of mixed methods research. Also, the sampling connection and timing give mixed methods researchers both opportunities and challenges for integration.

In this paper, I discuss as a backdrop the dimension of integration as integration through the researcher (Fetters & Molina-Azorin, 2017, p. 296). The researcher integration dimension means that the research could benefit from the researcher’s personal and professional experiences; in this case, the researcher’s experiences could provide a background for the critical evaluation of mixed methods in education policy research. As the researcher of this project, I worked for many years as a school principal and therefore have first-hand experience with increased accountability pressure, large-scale testing practices, and the external policy demands of enhancing school quality. These experiences as a school leader could thus be used to consider the qualitative, quantitative, and mixed methods procedures and to make sense of the complex and abstract phenomenon regarding school actors’ interpretations and negotiations of the national education policy demands at the local school level. The interactions between school leaders and teachers, between teachers and students, and between teachers, students, and parents contribute to the aim of educational researchers to deal with abstract social phenomena. The approach initiated by the researcher in designing the study will consequently play a crucial role in integrating mixed methods when investigating how and why school actors make use of the national test results.

In their integration trilogy, Fetters and Molina-Azorin (2017) provided 15 dimensions of integration for mixed methods research (pp. 294–295), which include the rationale, purpose, design, philosophical, theoretical, team, sampling, data collection, interpretation, rhetorical, dissemination, and research integrity dimensions. The high number of dimensions highlights the complexity and challenges involved in justifying and publishing considerations regarding mixed methods research. By building on the researcher integration dimension, this article has a delimited focus on three additional dimensions of integration relating to the study of education policy and micro-policymaking in schools: (a) the rationale dimension; (b) the study purpose, aims, and research dimension; and (c) the research design dimension (Fetters & Molina-Azorin, 2017). Table 1 presents the three dimensions outlined in the current project.

Table 1.

The Three Dimensions of Integration Presented Within the Scope of Education Policy Research and Micro Policymaking.

Dimension	Method of Integrating the Mixed Methods	Education Policy Research and Micro Policymaking in Schools
(a) Rationale	Citing a rationale for conducting an integrated mixed methods research study (e.g., offsetting strengths and weaknesses; and promoting social justice).	The rationale helps to balance strengths and weaknesses in policymakers’ desire to develop and control schools (Ozga & Lingard, 2007) with regard to the core purpose of education. The different relations and purposes of accountability and control create a rationale for conducting an integrated mixed method. An ideological dimension of the purpose of education is a compelling subject of the project.
(b) Study purpose, aims, and research questions	Composing an overarching mixed methods research purpose and stating qualitative, quantitative, and mixed methods aims or multiple mixed methods aims with quantitative aims and qualitative questions.	The overarching purpose of mixed methods research was to develop new insights into national testing practices and regulations in Norway. The purpose was followed by the overarching mixed (hybrid, integrated) research question: How and why do school professionals in a Norwegian social democratic context make use of national test results? This question is broken down into quantitative and qualitative sub-questions (Creswell & Clark, 2017) to answer in each phase of the study (see Table 3).
(c) Research design	Scaffolding the work in core (e.g., convergent, exploratory sequential, and explanatory sequential), advanced or emergent designs.	The study had a qualitative dominant design (Creswell & Clark, 2017) as two of the data collection phases were qualitative (school leader interviews and onsite observations/video recordings from meetings), with the third form (the survey) building on the others to identify important and unknown variables between the data (Creswell & Clark, 2017; Johnson & Christensen, 2014).

Note. Table contents adapted from Fetters and Molina-Azorin (2017, pp. 294–295).

Mixed Methods Design Approach

This study uses Bowe et al.’s (1992) model to “approach policy as discourse” (p. 13), based on the possibilities and impossibilities tied to knowledge on the one hand (analysis of problems and identification of solutions) and practice on the other (specifications of methods for achieving goals). I explore micro-policymaking in terms of value dispute processes and micro-policy discourses in lower secondary schools in one municipality, and I portray and analyze the processes of active interpretation and meaning-making that relate to policy texts and demands in practice. I therefore had to identify the resistance, adjustment, deception, and conformity within and between the arenas of practice and analyze the maneuvering of conflicts and disparities between continuous discourses in these arenas (Bowe et al., 1992). The multiphase mixed methods approach is typically guided by specific theoretical perspectives (Greene, 2007). In this study, policy sociology (e.g., Ball, 2008) and perspectives on policy enactment, policy actors, discursive roles (Ball et al., 2012), and crafting policy coherence (Honig & Hatch, 2004) inspired the methodological approach.

Multiphase Design

This multiple-case (Yin, 1994), multiphase study applies a mixed-methods approach to the empirical data obtained from 12 individual interviews with school principals and senior leaders from four lower secondary schools in the same municipality. Data were also obtained from onsite observations and informant-conducted video recordings of meetings of school principals and senior leaders and of teacher teams in two of the four lower secondary schools. In addition, a survey was administered to 176 lower secondary teachers in the same municipality. Figure 1 illustrates the design of the three phases, the kinds of data collected in each phase, and how the data influenced each other in relation to the aim of examining how school principals, senior leaders, and teachers make use of the national test results.

Figure 1.

The research design used to investigate schools’ micro-level enactment of national testing policies.

As two of the data collection phases were qualitative (school leader interviews and onsite observations/video recordings of meetings in Phases 1 and 2, respectively), this project has a predominantly qualitative design. The survey in Phase 3 was constructed based on tentative conclusions, patterns, and generalizations of the qualitative data obtained in Phases 1 and 2.

Phase 1: School principals’ and senior leaders’ perspectives. This phase obtained information about how teachers and school leaders make use of the national test results for reading and numeracy through individual interviews with school principals and senior leaders who had formal leadership responsibility. The interviews were semi-structured and conversational (Creswell, 2013; Kvale & Brinkmann, 2014), and each interview lasted 1–1.5 hours.

Phase 2: School leaders’ and teachers’ perspectives. Phase 2 involved obtained data through onsite observations and participant-conducted video recordings of the school principals, senior leaders, and teachers’ team meetings in which they discussed their negotiations and handling of national testing policies. The findings of Phases 1 and 2 were analyzed separately and compared. The findings of the analysis of Phase 1 narrowed the scope of Phase 2 to focus the school leaders’ and teachers’ discourses and negotiations on collaborative talk in meetings.

Phase 3: Teachers’ perspectives. Since dealing with policy expectations occurs at different levels of the school organization (Ball et al., 2012), it was important to compare the perspectives of formal school leaders (school principals and senior leaders) with the collaborative talk of school leaders and teachers from the teachers’ perspective. As the teachers were part of the school principals’ and senior leaders’ accounts of how national policy expectations were dealt with and negotiated, Phase 3 was designed as a quantitative descriptive analysis (Ary et al., 1990) of how teachers experience their work, the teacher teams’ work, and the principal’s facilitation of work with the national test results. Based on the data obtained from Phases 1 and 2, constructs of thematic areas were developed to create items for the teacher survey. The aim was to obtain a different perspective on the school principals’ and teachers’ work with the national test results.

Selecting the Participants

According to Ball et al. (2012), the notion of a school’s enactment of policy intentions includes a variety of policy actors. As a school cannot enact policy demands per se, policy actors might be local authorities, superintendents, parents, media, students, teachers, and school leaders (Ball, 2012; Ball et al., 2012). The sample includes school principals, senior leaders, and teachers from lower secondary schools in a municipality with average to high national test scores and excludes other policy actors, such as students and parents.

The selection criteria were created based on socioeconomic diversity, school size, and test scores. The 428 municipalities in Norway are required to provide 10 years of compulsory education at the primary and lower secondary school levels. These municipalities vary in size and average income. Based on national statistics concerning socioeconomic status, school size, and national test results, I selected the municipality Adde, which lies in the central eastern part of Norway. Adde is a medium to large-sized municipality that received a national and local reputation for its systematic work on national tests and average to high national test scores over the last three years. The following paragraphs describe the samples in the three phases, which have a multilevel, nested, and parallel relationship and are three of four major types of criteria used in mixed methods research (Johnson & Christensen, 2014, p. 272). The levels and relationships in this study are related to the central and local policy level, school principals, deputy managers, and teachers.

Sample 1: Many Norwegian schools are small. To protect the respondents’ anonymity, schools with more than 300 students were included, because using large schools makes it more difficult to expose the pupils’ identities. Six schools in Adde matched the criterion, with four immediately expressing a willingness to participate: Blue Mountain, Yellow Stone, Brown Hill, and Red Field. Their national test scores varied from average to high compared with the national average over the last three years. Table 2 presents the relevant characteristics of the schools and informants.

Table 2.

Overview of Participating Schools and Informants (Phase 1 and 2).

School	Informants	General Information	Average Test Score
Blue Mountain	Rebecca (principal)	400 students, eighth–10th grade	High
	Rosa (deputy manager)
	Ronny (deputy manager)
	Reese (deputy manager)
	Ninth grade teacher team including:
	Rihanna (teacher-team leader)
	Rusty (teacher)
	Roger (teacher)
Brown Hill	Gina (principal	350 students, eighth–10th grade	Medium/high
	Gary (deputy manager)
	Greta (deputy manager)
	Ninth grade teacher team including:
	Gorbert (teacher)
	Gail (teacher)
Yellow Stone	Brigitte (principal)	300 students, eighth–10th grade	Medium/high
	Brita (deputy manager)
	Barry (deputy manager)
Red Field	Hilary (principal)	320 students, eighth–10th grade	Medium
Red Field	Henry (deputy manager)	320 students, eighth–10th grade	Medium

Sample 2: To obtain different perspectives and investigate the relationship between the school principals, senior leaders, and teachers, I asked Brown Hill and Blue Mountain schools if I could observe their leader team and teacher team meetings. I selected these schools primarily because of the knowledge obtained about the two schools in Phase 1 and the different levels of their national test results. The two schools were seemingly similar with regard to organizational structure and meetings that formed part of the data obtained in Phase 1.

Sample 3: To obtain information about how teachers in Adde’s lower secondary schools experienced and perceived their principals regarding national testing policies and to compare the differences among the responses of school principals and senior leaders to the national test results observed onsite and in participant-conducted video recordings, I prepared and administered a teacher survey to 176 teachers from 12 lower secondary schools in Adde.

Data Collection

In Phase 1, data were collected through individual semi-structured interviews with principals and senior leaders of four different schools. The interviews obtained information about how and why the principals and senior leaders work with the national test results, when they work with the results, and who from the school works with the results. Further, the interviews elucidated their thoughts about school quality, student results, and school results in general, as well as the school’s capacity to enhance student achievement. The transcripts were coded openly according to questions in the interviews that addressed the ways in which the school leaders described micro-policymaking in relation to the national test results. The school leaders emphasized test results, practice, student learning, and school quality.

In Phase 2, data were obtained from onsite observations and participant-conducted video recordings of meetings from Blue Mountain and Brown Hill in addition to informal visits, gathering school documents, and taking field notes in both schools. The observed meetings were primarily concerned with improving student learning, making plans for the next teaching period and the whole school year, and working with the national test results. The observation data marked a shift between Phases 1 and 2, from collecting self-reporting data to integrating these data with observations. The observations were defined as “observer-as-participant” observations, which means that the researcher (as a former school principal) took on the role of the participating observer (Johnson & Christensen, 2014). Video footage was gathered in the fall when the schools were conducting the national tests and planning the rest of the school year. This footage showed how the school professionals talked about working with the national test results for reading and numeracy and gave instances in which micro-policymaking was prominent. These findings built on the data obtained in Phase 1, which influenced the choice of codes and categories, such as the terms “dealing with,” “negotiation,” and “making sense of national testing policies and basic skills.”

In Phase 3, the questionnaire shifted from Phases 1 and 2, which merged different types of qualitative data from four schools into a single municipality-based sample, to Phase 3, which built exclusively on quantitative survey data from all lower secondary teachers in the same municipality. A self-constructed questionnaire inspired by thematic areas was developed from the two first phases. The 27 core survey items used five-point Likert-type scales. The items represented a range of variables that were directed toward the concepts of “leadership,” “expectations,” “experiences,” and “attitudes.” According to Ary et al. (1990), descriptive research asks about the nature and distribution of variables and/or the relationships among these variables. In this case, items about the teachers’ practices with regard to the principal’s facilitation of work with the national test results were created and defined within the nature of micro-policymaking concerning national tests. See the teacher questionnaire in Online Appendix 1.

Analytical Concepts

This study used five overarching analytical concepts across and within the three phases: policy enactment, policy actors, discursive roles, crafting coherence, and micro-policy practice. As illustrated in Table 3, Phases 1 and 2 were designed to investigate school principals’, senior leaders’, and teachers’ micro-policymaking concerning national testing policies, while Phase 3 aimed to capture teachers’ experiences and attitudes toward leadership practices and micro-policymaking of these policies.

Table 3.

Overview of the Design Phases, Sub-Questions, Participants, Analytical Concepts, Credibility, and Main Findings.

	Phase 1 Interviews With School Leaders	Phase 2 Observation and Video Recordings of Leader Team and Teacher Meetings	Phase 3 Teacher Questionnaire
Methods	Qualitative	Qualitative	Quantitative
Main (sub) research question(s)	How do school leaders make sense of and respond to external demands related to a new policy context that emphasizes national test results for reading and numeracy?	How do school leaders and teachers position themselves in relation to each other in their interpretation and translation of national testing policies?	How do teachers perceive the principals’ facilitative work with national test results at the local school level?
			How do teachers and teacher teams use national test results, and what are teachers’ attitudes toward national tests? What relationship exists between the principals’ facilitative work with national test results and the teachers’ practices and attitudes?
		In what ways does school leaders’ and teachers’ language reflect their enactment of national testing policies?
	How do they use national test results in their local work with school and student development?

Participants	Sample 112 lower secondary school leaders (principals and senior leaders)	Samples 1 and 2Two leader teams (seven leaders, including the 12 school leaders from Sample 1), two teacher teams (16 teachers)	Samples 2 and 3176 teachers (including the 16 teachers from Sample 2)
Data	School principal and senior leader interviews	Video recordings and onsite observations of leader team and teacher team meetings	Teacher questionnaire
Analytical concepts	Policy enactment	School principals, senior leaders and teachers as policy actors	Teachers as policy actors
	Policy enactment		Crafting coherence and micro politics in schools
	School leadership and policy interpretation		Crafting coherence and micro politics in schools
	School leadership and policy interpretation		Relational and contextual aspects of micro-policy practices
	Policy enactment	Discursive roles in policy enactment
	Coupling mechanisms	Discursive roles in policy enactment
	Coupling mechanisms	Translation and negotiation of policy expectations
	Accountability	Translation and negotiation of policy expectations
	Accountability	School context
Main findings	This phase showed how test data has become a tool for symbolic responses and focuses on low-performing students. Internal accountability seems to trump external accountability.	This phase showed that, while responding to test data, school leaders serve as narrators and enthusiasts, senior leaders and teacher team leaders serve as messengers and enforcers, and teachers serve as critics and the preventers of “overburdening.”	This phase showed the different perceptions of 176 teachers toward their principals’ role in facilitative work with national test results. These differences correlated strongly with teachers’ practices and attitudes regarding their own and the teacher teams’ work with national test results.

Findings and Discussion

A Critical Examination of the Rationale Dimension

Following Fetters and Molina-Azorin (2017), the researcher integration dimension may enable a critical evaluation of the rationale dimension when applying mixed methods to education policy research. The researcher’s first-hand experiences with accountability pressure and the external policy demands of enhancing school quality based on the results from large-scale student testing motivated this research. The rationale dimension therefore connects closely with the researcher integration dimension because the researcher could benefit from the personal and professional background experiences of being a school leader. This connection also ties with the researcher’s experiences of the qualitative, quantitative, and mixed methods procedures to make sense of the complex and abstract phenomenon regarding school actors’ interpretations and negotiations of national education policy demands. However, the researcher integration dimension might also become a threat to the rationale dimension because of possible researcher bias, especially when analyzing the findings and determining their representation.

The purpose of his study was guided by the ideological dimension of the purpose of education, with a particular focus on the endangered Norwegian social democratic educational system, which has a long tradition of public trust, including free and equal education for all. The distinction between the demands for “external goods of effectiveness” and the “internal goods of excellence” was thus especially important in the rationale (Ranson, 2003, p. 461).

When investigating how and why school actors make use of the national test results, the findings in Phase 1 showed that the rationale dimension contributed to identifying strengths and weaknesses in policymakers’ desire to develop and control schools by student test results (Ozga & Lingard, 2007), and the ways in which professional school actors enacted the policies. The excerpt shows how the rationale dimension has contributed to identifying how policy intention meets local school policy enactment:

The individual feedback [the national test results] is meant to give support [policy intention], but it is meaningless to me! You know how the official information sounds: “A student at your level usually has difficulties with…” Well, this is not very interesting because we already have this information. (Gary, deputy manager at Brown Hill)

The analysis emphasizing low-performing students in the talk about the use of test results had a mitigating function in the study, perhaps because of the researchers’ values and beliefs regarding the purpose of the comprehensive education system in Norway. This argument, referring to the researcher integration dimension, connects with the rationale that the Norwegian education policy context is strongly rooted in institutional factors in terms of the ideologies and norms of equity that are linked to social democratic values (Møller et al., 2007; Ottesen & Møller, 2016). However, by analyzing the data and accentuating the findings about school professionals’ “gaming the system” with symbolic responses and non-conforming practices, values can also be regarded as a constraining factor in mixed methods research because the researcher may have omitted parts from the analysis. The transparency of the researcher dimension is imperative. Excerpts from statements from Phase 1 illuminate some of the non-conforming practices and internal beliefs and values that were found, perhaps leaving out the conforming practices:

Working with national test results is much about reading, and primarily the mother language [Norwegian] teachers have the main responsibility for monitoring and following up the results…But…what is the actual added value…what is the school’s contribution, right? […] So…we are leading a learning institution, and our mandate is to give students the most possible learning. That’s what we need to facilitate and that’s what we have to work toward. (Hilary, principal at Red Field)

Similarly, the rationale dimension was illuminated in the complementing of the Phase 1 results with the observation data of Phase 2. While responding to student test data, school professionals played different policy actor roles in the interpretation and translation of the national testing policies (Gunnulfsen, 2017). The rationale dimension within these results can be naturally linked to Ball et al.’s (2012) notion of micro-policy roles in the school organization. Micro-policy enactment of school professionals and the observation data illuminate policy enactment in practice more than do self-reports from the informants. On the one hand, the researcher integration dimension (Fetters & Molina-Azorin, 2017, p. 296) enables a critical evaluation of the rationale dimension. On the other hand, the choice of excerpts may restrain an objective discussion, where the researcher’s experiences with accountability policies and policy demands to develop and control schools may overshadow a valid critical discussion. The following excerpt in Table 4 shows one dimension: resistance.

Table 4.

The Criticism at Blue Mountain.

Transcript Line	Speaker	Talk/Text
(1)	Norbert (teacher)	Do we have to do this in our daily practice? Well, we can look at it, sure, but to use them [the results] in our teaching, well, I don’t know about that. (Leaning backward, with the arms crossed over the chest).
(2)	Hedvig (teacher)	But this will only be relevant regarding the students with extremely poor results, right? Regarding assignments that are extremely bad?
(3)	Åsmund (deputy)	(Pointing at the smart board, ignoring the question from Hedvig and Norbert.) Here, you can see a reversal of the prefix [looking into one particular student’s results in the numeracy test], we can see a poor understanding of equating, and at least it might be useful to have a retest. Plus, if we use the teacher’s guide…each one of you must check the results of your students and make a record of them…and then…we should spend more time sitting down with the class…but it could be that Gina plans differently than what she first said, but we should sit down and create questions to…eh…yes, for each student, and investigate at what level they are in reading and numeracy.
(4)	Norbert (teacher)	Do we have allocated time for this? We have other important things to do, if I may dare to say so…
(5)	Åsmund (deputy)	I don’t know. I have to ask Gina [the principal].

The conducting of a mixed methods study shows that the video observations influence the mutual strengths and balance the eventual weaknesses between Phases 1 and 2. The interviews illuminating school actors’ gaming of the system do not automatically connect to the observation data that teachers act as critics. However, the results must be understood in relation to the rationale, regarding different relations and purposes of accountability and control, and the core purpose of school, because the researcher needs to examine these overlapping facets (Collins et al., 2006; Greene, 2007). In this case, the observations built on the interviews. The rationale dimensions thus helped the researcher identify aspects that are linked to social democratic values in the Norwegian education system.

Expanding on the school professionals’ criticism in Phase 1 and the negotiations in Phase 2, the third phase revealed the views of 176 teachers on testing policies and their perceptions of their principals’ role in facilitative work with the national test results. The survey in Phase 3 was constructed based on the conclusions, patterns, and generalizations of the qualitative data obtained in Phases 1 and 2. Building on the first two phases, the items were divided into four themes:

Principals as facilitators of educational work, in general, and the national test results, specifically (seven items).

Teachers’ individual work with the national test results (six items).

Teachers’ use of the national test results in teams (five items).

Teachers’ personal attitudes toward the national tests and their results (nine items).

Phase 3 examined the different relations and purposes of accountability and control in the Norwegian education system. By conducting a teacher survey to investigate how and why school actors use the national test results and examine how leadership was performed in the matter, the distinction between the “external goods of effectiveness,” such as status and power, and the “internal goods of excellence,” such as the virtues of justice, courage, and friendship, might be visible in new ways. This distinction is especially important in the Norwegian educational system, which has a long tradition of including public trust, civic service, social justice, and equity as purposes that are central to education.

Table 5 shows the distribution of responses to the seven items used to measure the teachers’ perceptions of their principals as facilitators of educational work, in general, and of the national test results, specifically. These items were used to form a new construct: “Principals’ facilitation of work with the national test results.”

Table 5.

Distribution of Responses to Measure the Teachers’ Perceptions of Their Principals as Facilitators.

	Not At All	To a Small Extent	To Some Extent	To a Great Extent	To a Very Great Extent
1. The principal has confidence in my work with basic skills (N = 172).	0.0	0.3	4.7	51.0	44.0
2. The principal requests information about how I follow up on the national test results (N = 173).	8.6	21.4	35.3	26.6	8.1
3. The principal represents the school’s work with quality development in a good way with the wider community (N = 174).	0.0	0.6	21.2	43.1	35.1
4. The principal is in dialogue with teachers on the curriculum content (N = 174).	4.6	12.7	31.6	35.6	15.5
5. The principal is clear on what is expected of my work with the national test results (N = 172).	7.0	15.7	34.3	29.1	14.0
6. The principal expects that teachers will work systematically with the national test results (N = 172).	4.7	11.0	37.2	30.2	16.9
7. National tests are discussed at the appraisal meetings with the principal (N = 169).	33.1	27.3	27.8	7.7	4.1

As Table 5 shows, almost all the teachers perceived that their principals had confidence in their ability to work with basic skills (as part of national testing). The quantitative data confirm that the rationale dimension complements and even expands the viewpoints of the main research question, keeping the rationale of equity and social democratic schools in mind. The ideological dimension of the purpose of education has been a compelling rationale for the project.

It remains unclear whether the rationale and researcher integration dimensions affected the results. The weakness in the researcher dimension highlights a demand for transparency in the analysis process and a profound evaluation of the validity and reliability of the study. The researcher integration and the rationale dimension must both be applied at the time of the design of the study and not after the study is completed (Bryman, 2007).

Purpose, Aims, and Research Questions Dimension

The purpose of this mixed methods study was to develop new insights into national testing practices and accountability regulations in Norway. The aim was also to contribute to the understanding of how school actors make use of the national test results to enhance individual student learning and school quality in an educational system that has long been rooted in social democratic norms and values of trust and equity (Møller et al., 2007; Ottesen & Møller, 2016). I aimed to answer the mixed (hybrid, integrated) research question, “How and why do school professionals in a Norwegian social democratic context make use of the national test results?” by breaking it down into quantitative and qualitative sub-questions (Creswell & Clark, 2017) in line with each phase of the study (see Table 3). The research purpose and the overarching research question identified both qualitative, quantitative, and mixed methods aims because the main aim and question could be pursued by each method alone as well as in a mixed design.

The reported narratives from the interviews in Phase 1 and the collaborative talk observed in Phase 2 were dominated by methods of enhancing student learning by using methods that did not involve the national test results. Overall, the mixed methods approach highlighted the differences between the internal work with collective responsibilities, the shared purpose and values, and the external demand to improve the school’s test results. The overarching research question and the integrating dimension of purpose, which aimed to identify and discuss school professionals’ micro-policymaking in the Norwegian social democratic education system, is illuminated in examples from all three phases presented in the following excerpt and Tables 6 and 7:

But there is so much that we do here at Blue Mountain that cannot be measured, and that I think the students benefit a lot in the long run, too…When former students pay us visits to thank the teachers and the school for teaching them to be independent, to collaborate, and to believe in themselves. That means a lot, but it cannot be measured. (Reese, deputy manager at Blue Mountain)

Table 6 reveals how the school professionals discussed the value of all types of learning activities and how they enforced and advocated meanings about the national testing policies. The discussion took place in a teacher team meeting conducted in Phase 2.

Table 6.

The Messenger at Blue Mountain.

Transcript Line	Speaker	Talk/Text
(1)	Synne (team leader)	(Opens up her laptop and looks at Rosa as she begins to speak.) I think we need to discuss the parents meeting and what we want to present to the parents from the national test results. When it comes to the national tests, the information to teachers says: “national tests give us information about the school’s input from 8th grade.” […] This is about the anchor-tasks.
(2)	Rosa (deputy)	Yes?
(3)	Synne (team leader)	I wish that it [the information from the authorities] would rather say something like “the tests give us some information about the school’s input.” I mean this because it is important to illuminate and justify that we [the teachers and the school] also do other things that have most probably influenced the outcome [the national test results].
(4)	Rosa (deputy)	I totally agree! It is so good that we have more than one eye on these things. We mustn’t be influenced by these results only.

When turning to Phase 3, the results showed a strong correlation between the school’s common policy regarding the use of national test results and the principal’s facilitation, and that the use of the national test results was experienced as more affirmative.

Table 7, based on Phase 3, shows the items representing the teachers’ attitudes toward the national tests and the use of the results. In the analysis, the most prominent finding regarding school principals’ and senior leaders’ use of student test results was their disbelief that the information from the national tests could be beneficial for developing teachers’ instructional practice. However, the survey results in Phase 3 showed that school principals facilitated work with the national test results, even when they doubted the usability of the results. They even gamed the system regarding the policy demands for utilizing the tests.

Table 7.

Correlation of Teachers’ Attitudes and Principal’s Facilitation of Work With National Test Results.

	Principal’s Facilitation of Work With National Test Results
1. The national test results have little or no value in terms of individual student’s learning and development.	−.357^**
2. The national test results are useful tools for working with the individual student’s learning and development.	.271^**
3. The work with national tests takes too much time away from other, more important tasks in school.	−.254^**
4. I lack knowledge of how to use the national test results in reading in my subject(s).	−.294^**
5. I lack knowledge of how to use the national test results in numeracy in my subject(s).	−.191^**
6. National tests in reading have contributed to new knowledge about reading as a basic skill in subject(s).	.311^**
7. National tests in numeracy have contributed to new knowledge about numeracy as a basic skill in my subject(s).	.402^**
8. The school’s overall national test results are very important for the quality of our school’s work.	.572^**
9. Publication of the national test results has no value for improving the quality of our school’s work.	.031

** Correlation is significant at the 0.01 level (two-tailed).

The mixed methods design fulfilled the purpose of the integration dimension for obtaining insights into national testing practices and regulations in a Norwegian education policy context by highlighting the ways that the school professionals met the two distinct policy intentions of the national test results: (1) that the results should be used as tools to improve school quality by gathering accumulated data at a system level and (2) that formative assessment tools enhance individual student learning at the local level (Norwegian Directorate for Education and Training, 2011). The purpose dimension showed that school professionals did not express using the student test results as formative assessment tools in the qualitative data, but that this use came more to the fore in the quantitative data.

The national and local contexts are both important with regard to the purpose and aim of understanding why seemingly similar schools deal with policies differently (Ball et al., 2012). The analysis of the findings in all three phases revealed that the schools differed substantially in their accounts and collaborative talk and in their engagement with the national test results. The survey in Phase 3 supported these differences, showing significant variances between the schools relating to the thematic areas of principals’ facilitation and teachers’ attitudes and practices. However, these results did not explain the reasons for the differences and merely confirmed that school context and leadership practices affect the micro-policymaking of national testing policies. The findings from the three phases show the need for the researchers to reflect on integrating methods throughout all dimensions of the research design and that the integration dimensions need to be made clear to monitor the developing application of mixed methods research (Fetters & Molina-Azorin, 2017).

The Research Design Dimension

The research design dimension is about scaffolding the work at the core. The work at the core of this study has been to ask questions and obtain answers from informants and to learn from how informants act and what they think. Central contradictions in the quantitative and qualitative data invite a discussion related to the challenges involved in designing education policy research as mixed methods studies at the micro-policy level in schools. While the negotiation, power use, and communication to gain consensus about the policy demands for the use of the national test results were not visible from the survey results, the qualitative data were unambiguous regarding these issues and the distribution of leadership practices (Spillane, 2012; Spillane et al., 2004). However, the quantitative results gave strong indications of the effect of the principals on the teachers’ attitudes and practices with the national test results.

The presence and influence of the self in mixing methods demands recognition of the implications of the integration dimension of the researcher’s values and beliefs (Greene, 2012), which in turn affects the research design dimension. In this case, the quantitative data may have been affected by the researcher’s values in the design of the survey. In the qualitative data, the researcher’s bias and attitudes toward external policy demands and governing perspectives may affect the analysis of the data, such as the researcher’s decision to focus on the findings regarding low-performing students. The survey data did not illuminate the subtle practices observed in the observation data, which can be regarded as a challenge in the mixed methods approach in the study. The differences in the results might, however, also create opportunities to discuss and apply a mixed methods design in future studies of education policy enactment.

The two qualitative studies demonstrated how language and discursive talk were used and illuminated the power exercised through chains of events and individual principal practices; the quantitative part merely informed the study of the trust and relational aspects between the teachers and the principal. The central concerns of the project were power and conflict, which underlie almost all aspects of studying values, interests, and changes in organizational practices (Ball, 2012). Practices are permeated by the power seen in a relational and situated way, rather than as a top-down and linear phenomenon (Fairclough, 1992). The non-linear, multi-leveled phenomenon, consisting of external policy demands and school professionals’ translation and negotiation of the demands, characterizes the role that spatial representation can play in mixed methods analyses (Jones, 2017).

All three phases in the mixed methods design contributed to confirming the dominance of aspects of internal accountability in response to the main research question of how and why school professionals in a Norwegian social democratic context make use of the national test results, especially in the first qualitative part, where internal responsibility outperformed external accountability as an explanation of why they work with the results. The narratives that the school principals and senior leaders put forth in the interviews emphasized, criticized, and questioned the formative assessment intention of national testing.

The analysis showed how a mixed methods design can advance the investigation into the perspectives of internal values in schools and make it possible to observe and analyze the complexity in the contexts of micro-policy actors. This approach is particularly important at a time when external control and accountability practices are snowballing in educational contexts globally. A qualitative design would not have revealed the differences between the local schools in their micro-policymaking practices when they negotiated and interpreted the external policy demands of student data use. Neither would it have been possible to find and discuss the results of the individual leadership perspectives or the strong correlations between the principals’ facilitation of work and the teachers’ attitudes and practices. The research design dimension may contribute to advancing the knowledge of mixing methods in education policy research at the micro-policy level in schools. The mixed methods research design, which aims to understand the shared values and beliefs in the daily negotiations between school professionals, was made possible by the unknown variables that showed between the data. For example, the unknown variables of power, trust, autonomy, communication, and relations.

Contribution to the Field of Mixed Methods and Education Policy Research

The discussion suggests the considerable value of adopting a mixed methods approach within the complex field of education policy research and micro-policymaking in schools. An analysis of the integration dimensions in the three phases of the case study data revealed differences in how the school leaders and teachers enacted the central policy demand to make use of the national test results. Combined with the resemblance of the data across the schools, this variety suggests that leadership and teacher practice alone cannot explain the uneven understandings of national education policy intentions. The quantitative data also show that school principals facilitate large-scale data use in schools; however, further investigation into the teachers’ perceptions of this facilitation is needed to examine how schools respond to policymakers’ desire to develop and control schools through large-scale student testing.

The integration dimensions in the multiphase study led to meta-inferences about how and why school leaders and teachers make use of the national test results. An examination of the rationale, purpose, aims, and the research design dimension highlighted multiple perspectives that could not have been reached without using an integrated mixed methods approach. One perspective was that teachers’ attitudes toward the use of test results appeared more positive in combination with the principal’s facilitation in the quantitative data.

Despite the considerable work required in designing the mixed methods approach, this paper represents a small beginning to applying the integration dimensions of mixed methods research in education policy research. Further investigations are needed into education policy and micro-policymaking in schools. The researcher’s experience in applying a mixed methods design supports the need for mixed methods training and the value of being able to draw upon several types of knowledge.

Conclusion

This article has critically discussed the use of mixed methods in education policy research by evaluating the three integration dimensions of rationale, study purpose, and research design. The article contributes to the field of mixed methods research by applying these three dimensions of integration to the study of micro-policymaking in schools. This study also contributes to the discussion among education policy researchers who might want to use a mixed methods approach in their research of policy enactment at the grassroots level in schools but are uncertain about the benefits.

Guetterman and Fetters (2018) claimed that there is a constant need for support for the integration of mixed methods and case study designs. The findings of this study highlight the need for a better understanding of the integration dimensions to legitimize a dialogue about the many pathways that contribute to education policy scholarship. The analyses conducted in the three phases showed how and why a mixed methods approach contributes to the quality and reach of the study through multiple perspectives on school professionals’ micro-policymaking in schools. It remains unclear whether the mixed methods approach has more openly discussed the researcher integration dimension. However, the mixed methods approach allows for an open-minded approach to different types of wondering about different types of data (Greene, 2007). The mixed methods approach allowed mixed discussions about the visible, public, and accessible statements and results regarding school professionals’ values, attitudes, and practices. The discussions of the findings indicated both positive and negative attitudes toward large-scale student data use in schools. Having confidence in the design and its integration of a variety of data enables a thorough discussion of the findings and can even make it possible to ask national policymakers how and why they conduct national student testing. The paradoxes in the school actor roles, perceptions and practices of leadership, power relations, and time use are highly relevant issues for future mixed methods studies to examine in relation to micro-policymaking in schools.

Supplemental Material

Supplemental Material, sj-pdf-1-ijq-10.1177_16094069211028349 - Applying the Integration Dimensions of Quantitative and Qualitative Methods in Education Policy Research: Lessons Learned From Investigating Micro Policymaking in Norwegian Schools

Supplemental Material, sj-pdf-1-ijq-10.1177_16094069211028349 for Applying the Integration Dimensions of Quantitative and Qualitative Methods in Education Policy Research: Lessons Learned From Investigating Micro Policymaking in Norwegian Schools by Ann Elisabeth Gunnulfsen in International Journal of Qualitative Methods

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Ann Elisabeth Gunnulfsen

Supplemental Material

Supplemental material for this article is available online.

References

Ary

Jacobs

L. C.

Sorenson

Razavieh

(1990). Introduction to research in education (4th ed.). Holt, Rinehart, and Winston Inc.

Ball

S. J.

(2008). The education debate. Policy Press.

Ball

S. J.

(2012). The micro-politics of the school: Towards a theory of school organization. Routledge.

Ball

S. J.

Maguire

Braun

(2012). How schools do policy: Policy enactments in secondary schools. Routledge.

Bowe

Ball

S. J.

Gold

(1992). Reforming education and changing schools: Case studies in policy sociology (Vol. 10). Routledge.

Bryman

(2007). Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research, 1(1), 8–22. https://doi.org/10.1177/2345678906290531

Bryman

(2012). Social research methods (4th ed.). Oxford University Press.

Burch

Heinrich

C. J.

(2015). Mixed methods for policy research and program evaluation. Sage.

Collins

K. M.

Onwuegbuzie

A. J.

Sutton

I. L.

(2006). A model incorporating the rationale and purpose for conducting mixed methods research in special education and beyond. Learning Disabilities: A Contemporary Journal, 4(1), 67–100.

10.

Creswell

J. W.

(2009). Mapping the field of mixed methods research. Journal of Mixed Methods Research, 3(2), 95–108. https://doi.org/10.1177/1558689808330883

11.

Creswell

J. W.

(2013). Qualitative inquiry & research design. Choosing among five approaches (3rd ed.). Sage.

12.

Creswell

J. W.

Clark

V. L. P.

(2017). Designing and conducting mixed methods research. Sage.

13.

Fairclough

(1992). Discourse and social change. Polity Press.

14.

Fetters

M. D.

Curry

L. A.

Creswell

J. W.

(2013). Achieving integration in mixed methods designs—Principles and practices. Health Services Research, 48(6pt2), 2134–2156. https://doi.org/10.1111/1475-6773.12117

15.

Fetters

M. D.

Molina-Azorin

J. F.

(2017). The journal of mixed methods research starts a new decade: The mixed methods research integration trilogy and its dimensions. Journal of Mixed Methods Research, 11(3), 291–307. https://doi.org/10.1177/1558689817714066

16.

Fiorini

Griffiths

Houdmont

(2016). Mixed methods research in the health sciences: A review. Malta Journal of Health Sciences, 3(2), 37–45. https://doi.org/10.14614/MIXMETHRES/7/16

17.

Gorard

(2001). Quantitative methods in educational research: The role of numbers made easy. A&C Black.

18.

Greene

J. C.

(2007). Mixed methods in social inquiry (Vol. 9). John Wiley & Sons.

19.

Greene

J. C.

(2012). Engaging critical issues in social inquiry by mixing methods. American Behavioral Scientist, 56(6), 755–773. https://doi.org/10.1177/0002764211433794

20.

Guetterman

T. C.

Fetters

M. D.

(2018). Two methodological approaches to the integration of mixed methods and case study designs: A systematic review. American Behavioral Scientist, 62(7), 900–918. https://doi.org/10.1177/0002764218772641

21.

Guetterman

T. C.

Fetters

M. D.

Creswell

J. W.

(2015). Integrating quantitative and qualitative results in health science mixed methods research through joint displays. The Annals of Family Medicine, 13(6), 554–561. https://doi.org/10.1370/afm.1865

22.

Gunnulfsen

A. E.

(2017). School leaders’ and teachers’ work with national test results: Lost in translation? Journal of Educational Change, 18(4), 495–519. https://doi.org/10.1007/s10833-017-9307-y

23.

Gunnulfsen

A. E.

Møller

(2017). National testing: Gains or strains? School leaders’ responses to policy demands. Leadership and Policy in Schools, 16(3), 455–474. https://doi.org/10.1080/15700763.2016.1205200

24.

Honig

M. I.

Hatch

T. C.

(2004). Crafting coherence: How schools strategically manage multiple, external demands. Educational Researcher, 33(8), 16–30. https://doi.org/10.3102/0013189X033008016

25.

Johnson

R. B.

Christensen

(2014). Educational research: Qualitative, quantitative, and mixed approaches (5th ed.). Sage.

26.

Johnson

R. B.

Schoonenboom

(2016). Adding qualitative and mixed methods research to health intervention studies: Interacting with differences. Qualitative Health Research, 26(5), 587–602. https://doi.org/10.1177/1049732315617479

27.

Jones

(2017). Using a theory of practice to clarify epistemological challenges in mixed methods research: An example of theorizing, modeling, and mapping changing West African seed systems. Journal of Mixed Methods Research, 11(3), 355–373. https://doi.org/10.1177/1558689815614960

28.

Kvale

Brinkmann

(2014). Den kvalitativa forskningsintervjun [The qualitative research interview] . Studentlitteratur.

29.

Leech

N. L.

Onwuegbuzie

A. J.

(2009). A typology of mixed methods research designs. Quality & Quantity, 43(2), 265–275. https://doi.org/10.1007/s11135-007-9105-3

30.

Levinson

B. A.

Winstead

Sutton

(2020). An anthropological approach to education policy as a practice of power: Concepts and methods. In Fan

Popkewitz

(Eds.), Handbook of education policy studies (pp. 363–379). Springer.

31.

Møller

Eggen

Fuglestad

O. L.

Langfeldt

Presthus

A. M.

Skrøvset

Stjernstrøm

Vedøy

(2007). Successful leadership based on democratic values. In Day

Leithwood

(Eds.), Successful principal leadership in times of change (pp. 71–86). Springer Netherlands.

32.

Norwegian Directorate for Education and Training. (2011). Nasjonale prøver. Veiledning for skoleeiere og skoleledere [National tests. A guide for superintendents and school leaders]. Utdanningsdirektoratet.

33.

Norwegian Directorate for Education and Training. (2013). Nasjonale prøver [National tests]. http://www.udir.no/vurdering/nasjonale-prover/

34.

Ottesen

Møller

(2016). Organisational routines—The interplay of legal standards and professional discretion. European Educational Research Journal, 15(4), 428–446. https://doi.org/10.1177/1474904116638853

35.

Owen

G. T.

(2014). Qualitative methods in higher education policy analysis: Using interviews and document analysis. The Qualitative Report, 19(26), 1–19. https://www.nova.edu/ssss/QR/QR19/owen52.pdf

36.

Ozga

Lingard

(2007). Globalisation, education policy and politics. In Lindgard

Ozga

(Eds.), The RoutledgeFalmer reader in education policy and politics (pp. 75–92). Routledge.

37.

Ponce

O. A.

Pagán-Maldonado

(2015). Mixed methods research in education: Capturing the complexity of the profession. International Journal of Educational Excellence, 1(1), 111–135. https://doi.org/10.18562/ijee.2015.0005

38.

Ranson

(2003). Public accountability in the age of neo-liberal governance. Journal of Education Policy, 18(5), 459–480. https://doi.org/10.1080/0268093032000124848

39.

Saarinen

(2008). Position of text and discourse analysis in higher education policy research. Studies in Higher Education, 33(6), 719–728. https://doi.org/10.1080/03075070802457090

40.

Spillane

J. P.

(2012). Data in practice: Conceptualizing the data-based decision-making phenomena. American Journal of Education, 118(2), 113–141. https://doi.org/10.1086/663283

41.

Spillane

J. P.

Halverson

Diamond

J. B.

(2004). Towards a theory of leadership practice: A distributed perspective. Journal of Curriculum Studies, 36(1), 3–34. https://doi.org/10.1080/0022027032000106726

42.

Wolf

(2010). Enlightened eclecticism or hazardous hotchpotch? Mixed methods and triangulation strategies in comparative public policy research. Journal of Mixed Methods Research, 4(2), 144–167. https://doi.org/10.1177/1558689810364987

43.

Yin

(1994). Case study research. Design and methods. Sage.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.20 MB