PISA 2012: Examining the influence of prior knowledge,time-on-task,school-level effects on achievements in mathematical literacy processes

Abstract

Recent studies reiterate the importance of mathematical literacy and the identification of skills, knowledge and cognitive processes which contribute to composite test scores to facilitate targeted remediation and extension activities. To this end, the current article examines data from the 2012 cycle of the Programme for International Student Assessment (PISA), using multilevel modelling techniques to explore the relationship between selected student-level and teacher/school-level factors and the three processes of interpret, employ and formulate which were measured as the skills underlying mathematical literacy in that assessment. Results of the analyses indicate that boys outperform girls significantly (p < 0.001) in all three processes whereby formulate invokes relatively more inter- and intra-level influences compared with interpret. Apart from the relatively higher item-difficulties of formulate, an increase in the complexity of contextual effects at the student and the teacher/school-level emerges as mathematical processes move from interpret to employ to formulate. Findings also reveal that students taught by teachers who had mathematics as a major in their undergraduate studies and who work in relatively smaller classes or groups show higher performance in all three mathematical literacy processes. Use of ICT in mathematics lessons is negatively associated with the three mathematical literacy processes. The additional negative effect of mathematical extracurricular activities at school on the processes highlights the need to rethink how technology and extracurricular lessons are to be used, designed/structured and delivered to optimise the learning of mathematical processes, and ultimately improve mathematical literacy.

Keywords

Mathematical literacy processes – interpret employ and formulate PISA 2012 multilevel modelling student-level teacher-/school-level cross-level interactions

Introduction

The triennial Programme for International Student Assessment (PISA) assesses reading literacy, mathematical literacy and scientific literacy. The main assessment areas are rotated across cycles with mathematical literacy being the major domain of assessment in the 2012 cycle, meaning that a larger proportion of the assessment was devoted to mathematical literacy than to reading or scientific literacy. PISA operationalises mathematical literacy as ‘an individual’s capacity to formulate, employ, and interpret mathematics in a variety of contexts’ (OECD, 2013a, p. 25).

Mathematical literacy encompasses reasoning mathematically and using mathematical concepts, procedures, facts and tools in describing, explaining and predicting phenomena. The verbs ‘interpret’, ‘employ’ and ‘formulate’ indicate the three cognitive processes which students will engage with when completing mathematical literacy tasks (OECD, 2013b, p. 25).

This study examines the relationships between selected contextual factors at the student level as well as the teacher/school level and the cognitive processes associated with interpret, employ and formulate, and performance in mathematical literacy.

The focus on the three cognitive processes, which underpin mathematical literacy, has implications for learning, teaching and the mathematics curriculum (Thomson et al., 2013). This emphasis is explicit in Australia through the Victorian (VCCA, n.d.) and Queensland Education Department’s (Goos et al., 2015) directions.

Smith et al. (2018) warn Australia’s mathematics performance trend has stalled or declined in the National Assessment Program – Literacy and Numeracy (NAPLAN). They indicate the urgency to examine benefits of analytical skills and underlying cognitive skills and processes.

Our research study seeks to respond to Smith et al.’s challenges and to provide useful insights to mathematics educators and curriculum developers of mathematical literacy in Australia and beyond.

Synthesis of literature and foci of current study – Processes and contents underlying mathematical literacy

The 2012 cycle of PISA saw the broadening of the definition of mathematical literacy, with the integration of mathematical modelling (OECD, 2013a, p. 25). Figure 1 provides the progressions of learning as one proceeds through the stages (and processes) and the framework for the assessment of mathematical literacy and the relationships between contents, contexts and processes.

Figure 1.

A model of mathematical literacy in practice (OECD, 2013a, p. 26).

Three interrelated aspects of mathematical literacy were made explicit (OECD, 2013a, p. 27) and included the following:

mathematical processes describe what individuals do to connect the context of the problem with mathematics and thus solve the problem;

mathematical content targeted for use in the assessment items and

contexts in which the assessment items are located.

Of interest and scrutiny in this research are the mathematical processes as evidenced through student-level and school-level performance.

The OECD report (2013a) highlighted mathematical literacy, which begins with a problem in context, is an individual’s capacity to interpret, employ and formulate processes associated with mathematics in a variety of contexts. It includes reasoning mathematically and assists individuals to recognise the role that mathematics plays in the world and to make the well-founded judgements and decisions needed by constructive, engaged and reflective citizens. Importantly, ‘mathematical literacy is very closely related to mathematical modelling, because formulating mathematical models, employing mathematical knowledge and skills to work on the model and interpreting and evaluating the outcome are its essential processes’ (Stacey & Turner, 2015, p. 12).

Stacey (2001) aptly summarises the underlying processes in Figure 1. In the formulate process, a real-world problem is transformed into a mathematical problem (top arrow in Figure 1). The mathematical problem is solved by the employ process. The interpret process underpins how the mathematical results produced are translated into real-world terms and judged for adequacy.

In 2012, PISA mathematical literacy results were reported as overall score, scores for each of the four mathematical content categories and (new) scores for each of the three processes. The interpret scale should indicate how effectively students are able to reflect upon mathematical solutions or conclusions, interpret them in the context of a real-world problem and evaluate whether the results or conclusions are reasonable. Students’ facility at applying mathematics to problems and situations is dependent on all three of these processes, and an understanding of their effectiveness in each process can help inform both policy-level discussions and decisions being made closer to the classroom level.

Figure 2 illustrates the interpret–employ–formulate processes from the PISA 2012 mathematical literacy report (OECD, 2013a).

Figure 2.

Details illustrating the interpret – employ – formulate processes (OECD, 2013a).

The employ scale should indicate how well students are able to perform computations and manipulations and apply the concepts and facts they know to arrive at a mathematical solution to a problem formulated mathematically.

The PISA 2012 report (OECD, 2013c) also indicated scores on the formulate scale should show how effectively students are able to recognise and identify opportunities to use mathematics in problem situations and then provide the necessary mathematical structure needed to formulate that contextualised problem into a mathematical form.

Although employing is the application of mathematics knowledge to solve a problem, formulate can provide even higher difficulties if underlying interpret and employ processes are deficient. Alagumalai (1999) provided evidence that higher-order mathematical processes require fundamental knowledge and understanding of key mathematics concepts and skills. Moreover, he demonstrated the Rasch model provides support for the complexity of cognitive processes used in assessing understanding and applications. The more complex items, which require higher-order thinking and cognitive processes, had item difficulties higher than items involving recall and understanding. Both the interpretation and translation of word problems into its mathematical form is fundamental to success in mathematical problem-solving (Alagumalai, 1999, 2012).

Paralleling the linking of complexity of cognitive processes with difficulty of items associated to these processes, Stacey and Turner (2015, p. 17) argued that the empirical difficulty of cognitive demand increases when moving from ‘interpreting, applying and evaluating mathematical outcomes’, to ‘employing mathematical concepts, facts, procedures and reasoning’, to items which involve ‘formulating situations mathematically’. They further refer to the use of Rasch-based item response theory that the item difficulty increases with employ and formulate items, relative to those involving interpret. The item difficulty levels will not be tested in the current study. However, this research explores the various factors which associate and interact with items for the three cognitive processes. It is important to examine the stages of action in solving real problems and contextual factors which may facilitate the problem-solving processes (Stacey & Turner, 2015). Moreover, the PISA 2012 framework specifies that ‘about half of the mathematics items used in the PISA survey are classified as Employ and about one quarter are in each of the Formulate and Interpret categories’ (Stacey & Turner, 2015, p. 26). Hence, this research will provide understanding on how selected student and teacher/school-level factors influence the three cognitive processes uniquely and provide further insights beyond the aggregate mathematical literacy scores (and reported in nearly all research publications).

PISA provides information that allows comparison between jurisdictions and education systems to identify factors and examine trends that enhance performance in mathematics. Nguyen et al. (2020) highlight that contextual information in PISA enables the linking of background factors to student performance, and how student, home, teacher, school and organisational information may predict achievement in mathematical literacy. Good measures of students’ and families’ contexts are critically important when analysing large-scale assessment data so as to find which factors are associated with positive outcomes. It is pertinent to understand performance in processes associated with mathematical literacy and associations and relationships between the micro-student level and family factors, the meso-teacher/school-level factors and macro-country level factors (OECD, 2016a). As noted in the ‘Assessment and Analytical Framework’ (OECD, 2013a), a large section of the student questionnaire, school questionnaire and the international options, is devoted to contextual factors.

An important aim of PISA is to ‘assess the cumulative yield of education systems at an age where compulsory schooling is still largely universal’ (OECD, 2009, p. 11). The current study seeks to examine the factors (or variables) that contribute to student achievement in the three cognitive processes associated with mathematical literacy. As such, no comparisons between countries and/or teachers/schools are intended. Moreover, due to the complexities associated with missing data, complex sampling design, multilevel modelling and associated imputation methods (Allison, 2002, pp. 73–90; Raghunathan, 2015; Rutkowski et al., 2010), only complete datasets from participating jurisdictions have been used to identify statistically significant predictors to understand the contextual factors which contributed positively to performance in mathematical literacy.

Rationale for selection of variables for study

Variables that had been investigated in previous studies (Alivernini et al., 2010; Bando et al., 2017; Coyle et al., 2015; Ma et al., 2008; O’Dwyer et al., 2015; OECD, 2014c) were selected and included in the current analyses to provide a comprehensive picture of factors related to mathematical literacy. The categories, levels and variables nested under these levels included here follow on from the classifications articulated by Willms (2010):

. . . various factors that affect student achievement can be classified into three broad domains – curriculum, teaching practices and the context or learning environment. This classification helps to set some boundaries on what is meant by school context, and it provides a framework for considering different educational reforms. (p. 1010)

Willms (2010) summarised that ‘many of the important contextual factors, such as classroom disciplinary climate, are rooted in the classroom (teacher-level), not the school’ (p. 1012). Confounders can distort the association and demarcation between two variables. O’Dwyer et al. (2015) highlighted in the Trends in International Mathematics and Science Study the random sampling of one intact classroom in each school results in classroom effects can be confounded with school effects. Thus, the variance in achievement between classroom (or teachers) with schools and the between school variance in achievement are confounded. This study acknowledges the confounding nature of the teacher–school categorisation and therefore uses the term ‘teacher/school-level’ or ‘teacher/school variable’ or ‘teacher/school factor’.

Differences in the average performance of males and females in mathematics, mathematical literacy and mathematical sciences continue to draw research interests (Doris et al., 2012; Toh & Alagumalai, 1994). Strength in various cognitive skills, attitudes and motivation has been attributed to success in mathematics performance (Coyle et al., 2015; Hasan & Khan, 2015; Stoet & Geary, 2013). However, Reilly et al. (2016, p. 147) contend that ‘ . . . sex-role identity may have greater utility in explaining individual differences as demonstrated through performance in spatial skills and measures’. Ongoing debate about gendered differences in performance and associated equality issues provided the motivation to examine how gender is associated with performance in mathematical literacy in this study.

The variables identified in this study were guided by Plomp’s (1992) framework and rationale used in comparative educational research. He acknowledged the interconnectedness of curricular antecedents, curricular contexts and curricular contents, and their influences on the system, school/classroom/teachers and student levels.

Moreover, the OECD (2013a, 2016b) reports provided support for the selection of variables associated with students’ familiarity with mathematics, their attitudes towards mathematics and time-on-tasks (and completion of lessons, problems and tests). Namkung and Peng’s (2019) study reiterated the need to understand better the experiences and exposure to mathematical tasks. Curricular factors which influence school learning (Keeves, 1976) provided the basis for identifying school-level factors. Selected school-level factors were included to examine their importance for the three cognitive processes and in developing instructional strategies (Jitendra et al., 2016; Tambychik & Meerah, 2010). The variables selected for this study are presented in Table 1.

Table 1.

Home/student, classroom and school variables.

1. Level-1: Student level
1.1 Student background/home/contextual variables
	SEX* [ST04]	Sex of student [*Recoded: Male = 0; Female = 1]
1.2 Student’s experiences and familiarity (StuExp) [Constructs]
	EXAPPLM	Experience with Applied Mathematics Tasks at School
	EXPUREM	Experience with Pure Mathematics Tasks at School
	FAMCON	Familiarity with Mathematical Concepts
	FAMCONC	Familiarity with Mathematical Concepts (Signal Detection Adjusted)
	Exposure to ST73Q01RST73Q02R ST74Q01RST74Q02R ST75Q01RST75Q02R ST76Q01RST76Q02R	Maths Tasks and Test [Items]Opportunity to learn (OTL) – Algebraic Word Problem in Maths LessonOTL – Algebraic Word Problem in TestsOTL – Procedural Task in Maths LessonOTL – Procedural Task in TestsOTL – Pure Maths Reasoning in Maths LessonOTL – Pure Maths Reasoning in TestsOTL – Applied Maths Reasoning in Maths LessonOTL – Applied Maths Reasoning in Tests[**Recoded: Frequent + Sometimes = 1; Rarely + Never = 0]
1.2.3 Student Time-on-Task (StuToT)
	LMINS	Learning time (minutes per week) – <test lang>
	MMINS	Learning time (minutes per week) – Mathematics
	OUTHOURS	Out-of-School Study Time
	USEMATH	Use of ICT in Mathematics Lessons
2. Level-2: Teacher/school level
	ABGMATH	Ability grouping for mathematics classes
	ASSESS	Use of assessment
	MACTIV	Mathematics extracurricular activities at school
	MATHEXC	Mathematics extension course types offered
	PROPCERT	Proportion of certified teachers
	PROPMA5A	Proportion of Maths teachers with ISCED 5A (Maths major)
	PROPMATH	Proportion of Maths teachers
	PROPQUAL	Proportion of teachers with ISCED 5A
	SCHSIZE	Total enrolment at school
	SMRATIO	Maths teacher-to-student ratio
	STUDCLIM	Student-related factors affecting School Climate
	TCFOCST	Teacher focus
	TEACCLIM	Teacher-related factors affecting School Climate

Unless otherwise indicated, the indices above were scaled using a weighted likelihood estimate, using a one-parameter item response model (a partial credit model was used in the case of items with more than two categories) (OECD, 2013a, p. 191).

The Centre for Survey and Survey/Register Data (http://cssr.surveybanken.aau.dk/webview/) provide description for all variables used in the PISA2012 study. Moreover, Chapter 16: Scaling Procedures and Construct Validation of Context Questionnaire Data of the PISA2012 Technical Report (OECD, 2014b) provides details to scaling processes used.

Research questions and models

Figure 3 presents the conceptual framework adapted from the OECD report (2013a), the two-level proposed model for testing in this study, premised on what and how students learn and how teachers/schools support their learning and performance. Guided by this framework and the findings of previous studies discussed above, the research questions for this study were:

Figure 3.

HLM representation of possible effects and interactions (student- and school-level).

RQ1: What are the proportions of variance:

a) associated with the student and teacher/school levels in the interpret, employ and formulate processes, and

b) in student performance on the interpret, employ and formulate processes that are explained by the student and teacher/school factors?

RQ2: What student (Level-1) factors are significant predictors of student performance in the interpret, employ and formulate processes?

RQ3: What teacher/school (Level-2) factors are significant predictors of student performance in the interpret, employ and formulate processes?

RQ4: What is the nature of the cross-level interactions between teacher/school (Level-2) predictors and student (Level-1) predictors of student performance in the interpret, employ and formulate processes?

Methodology – Multilevel modelling in education

In education, students are nested within classrooms and classrooms are nested within schools. Students interact with the social and learning contexts within the class and in the school, and these broader ‘groupings’ (class and school) are influenced by the students and teachers within the system.

This investigation utilises the current developments in multilevel modelling and examines individual-level, school-level effects and the interactions between the variables in these levels. Multilevel modelling was used to estimate the effects of selected student background/home/contextual variables, student’s experiences and familiarity (StuExp), Student Time-on-Task (StuToT) and school-level factors on the three cognitive processes underlying mathematics literacy (achievement).

It was noted that several schools sampled and included in this study had ‘one mathematics classroom and with one mathematics teacher’. As indicated earlier, the classroom and school levels are confounded (O’Dwyer et al., 2015). The confounding between classroom, the teacher and school is acknowledged by including variables from these three categories at the second level, namely the Teacher/School-level category in the current study. A similar approach was applied by Alivernini et al. (2010), Mora (2020), O’Dwyer and Parker (2014) and OECD (2014d, 2015a).

Understanding the interactions between levels is important for examining policies in education. Raudenbush and Willms (1991) purport:

Interaction effects have important implications for policy, and one of the central purposes of multilevel analyses of educational data has been to explore the possibility that educational policies and practices may modify the distribution of outcomes within schools of classrooms. (p. 3)

To test the various effects and interactions, variables were entered into the model simultaneously, and only those that had statistically significant (p < 0.10) effects on the outcome variable were retained and reported. This article adopts Hox (2002), Raudenbush and Bryk (2002), as well as Snijders and Bosker’s (2012) heuristics of analysis and respective inference through examining (1) unconstrained (null) model, (2) random intercept model, (3) means as outcomes model and (4) interactions model. Figure 3 represents the conceptual framework for the hierarchical linear modelling (HLM) models for this study.

Sample and dataset

Students between the ages of 15 years 3 months and 16 years 2 months from 65 jurisdictions, including 34 OECD member countries, participated in the 2012 cycle of PISA (OECD, 2014a). In addition to completing items on mathematical literacy, scientific literacy and reading literacy, contextual information from students (including family background, motivation and instruction) and principals (schools) was also collected. Table 1 presents the student variables selected for inclusion at Level-1 as well as the teacher/school variables that were included as Level-2 variables in the models examined in this study.

Australia plays a key role in large-scale international studies through the Australian Council for Educational Research. Moreover, the focus on mathematical literacy (NAPLAN) and innovations in the learning and teaching of mathematics (Thomson et al., 2013) require scrutiny of the underlying analytical skills vis-à-vis through the three cognitive processes (Smith et al., 2018). It is acknowledged that although key variables were selected from PISA participating countries, the findings of this study are useful to inform policy and curriculum matters in Australia.

As the analyses for this study focussed primarily on examining broad teacher/school-level (Level-2) and student-level (Level-1) factors that predicted mathematics literacy, and not on the performance of individual schools, teachers or students, listwise deletion was used for Level-1 and Level-2 data. After removing all combinations of missing data (user-missing and system-missing values combined, see Appendix 1), the remaining sample for analysis included 46,518 students nested within 3001 schools/teachers.

This study acknowledges the correlations within and between compositional and contextual variables and its implications for multicollinearity and associated inferences. One suggestion to reduce multicollinearity in a regression model is re-specification through the removal of highly correlated variables. Moreover, Anderson (2012, p. 13) recommends that centring in HLM ‘has additional benefits, such as reducing issues of multicollinearity, which can ease estimation’. Hence, in this study, the following centring methods were used:

Group-mean centring: SEX, ST73Q01R, ST73Q02R, ST74Q01R, ST74Q02R, ST75Q01R, ST75Q02R, ST76Q01R and ST76Q02R.

Grand-mean centring: EXAPPLM, EXPUREM, FAMCON, FAMCONC, LMINS, MMINS, OUTHOURS and USEMATH.

Plausible values, which are computed approximations of proficiency estimates for mathematics literacy and the underlying processes, were used as outcome variables in the HLM analyses. Researchers (Adams & Wu, 2002; Monseur & Adams, 2009; Von Davier et al., 2009) have highlighted the advantages of plausible values over single individual (raw) scores, and the current features in HLM7 allow for their use in the current analyses.

This study adopted OECD’s (2010) recommended student final weights (W_FSTUWT) and school weights (W_FSCHWT) for Level-1 and Level-2, respectively, and in the analyses using HLM-8 software (Raudenbush et al., 2019).

The models tested for this study are the (1) unconstrained (null) model, (2) random intercept model, (3) means as outcomes model and (4) interactions model (see Figure 3).

Results and discussion

While the descriptive statistics of the variables included in the models are presented in Table 2, the multilevel analysis results are presented in Table 3. To address the first research question, ‘What are the proportions of variance (a) associated with the student and teacher/school levels in the interpret, employ and formulate processes, and (b) in student performance on the interpret, employ and formulate processes that are explained by the student and teacher/school factors?’ the results of the null model (Model 1) for each of the three processes were used. To obtain estimates of the available variance, the intraclass correlation coefficients (ICC) were calculated dividing the Level 2 variance by the sum of the Level 1 and Level 2 variances of the null model for each of the three processes. This produced ICCs of 0.46, 0.50 and 0.46 for interpret, employ and formulate, respectively, indicating that between 46% and 50% of the total variability in the three cognitive processes was associated with schools/teachers. These results regarding available variances is consistent with the findings of Nye et al. (2004), Sikora and Saha (2010) and United Nations Children's Fund (2012).

Table 2.

Descriptive statistics.

Variable name	N	Mean	SD	Minimum	Maximum^a
EXAPPLM	46,518	–0.07	0.95	–2.99	3.20
EXPUREM	46,518	0.13	0.92	–2.73	0.80
FAMCON	46,518	0.22	1.00	–4.57	4.44
FAMCONC	46,518	0.16	1.03	–5.16	5.18
LMINS	46,518	226.60	86.49	0.00	1620.00
MMINS	46,518	225.36	86.82	0.00	2200.00
OUTHOURS	46,518	10.67	9.73	0.00	180.00
USEMATH	46,518	–0.03	0.97	–0.77	2.80
PV1MAPE	46,518	508.38	94.35	125.88	868.68
PV2MAPE	46,518	508.34	94.39	126.35	877.56
PV3MAPE	46,518	508.25	94.42	125.88	922.74
PV4MAPE	46,518	508.43	94.36	145.51	957.48
PV5MAPE	46,518	508.28	94.34	164.68	879.66
PV1MAPF	46,518	507.46	109.58	12.94	1009.67
PV2MAPF	46,518	507.53	109.31	131.18	993.39
PV3MAPF	46,518	507.24	109.59	108.90	1100.33
PV4MAPF	46,518	507.64	109.43	45.65	1098.47
PV5MAPF	46,518	507.44	109.59	95.19	1039.42
PV1MAPI	46,518	510.51	98.75	135.93	904.51
PV2MAPI	46,518	510.50	98.62	103.14	878.10
PV3MAPI	46,518	510.50	98.64	110.93	897.73
PV4MAPI	46,518	510.67	98.44	95.19	974.61
PV5MAPI	46,518	510.62	98.62	94.57	875.69
SEX	46,518	0.51	0.50	0.00	1.00
ST73Q01R	46,518	0.85	0.36	0.00	1.00
ST73Q02R	46,518	0.76	0.43	0.00	1.00
ST74Q01R	46,518	0.93	0.26	0.00	1.00
ST74Q02R	46,518	0.87	0.33	0.00	1.00
ST75Q01R	46,518	0.76	0.43	0.00	1.00
ST75Q02R	46,518	0.68	0.47	0.00	1.00
ST76Q01R	46,518	0.62	0.49	0.00	1.00
ST76Q02R	46,518	0.51	0.50	0.00	1.00
Level-2 Descriptive statistics
ABGMATH	3001	2.11	0.74	1.00	3.00
ASSESS	3001	4.67	1.24	0.00	6.00
MACTIV	3001	2.83	1.16	1.00	5.00
MATHEXC	3001	2.48	0.62	1.00	3.00
PROPCERT	3001	0.84	0.28	0.00	1.00
PROPMA5A	3001	0.56	0.38	0.00	1.00
PROPMATH	3001	0.15	0.11	0.01	1.00
PROPQUAL	3001	0.84	0.27	0.00	1.00
SCHSIZE	3001	746.33	683.25	5.00	10,302.00
SMRATIO	3001	108.39	81.33	2.33	1471.71
STUDCLIM	3001	–0.02	1.03	–4.01	2.75
TCFOCST	3001	–0.20	0.99	–3.79	2.12
TEACCLIM	3001	–0.17	1.00	–3.59	2.85

For three mathematics process subscales interpret, employ and formulate, the plausible values variables are PV1MAPE to PV5MAPE, PV1MAPF to PV5MAPF and PV1MAPI to PV5MAPI, respectively.

^aOutliers were removed from the analyses.

Table 3.

Multilevel modelling results for mathematical literacy processes (N_student = 46,518; N_{school/teacher} = 3001).

	MAP – Interpret				MAP – Formulate				MAP – Employ
Parameter	Model 1 (Null)		Model 2 (Level-1 + Level-2)		Model 1 (Null)		Model 2 (Level-1 + Level-2)		Model 1 (Null)		Model 2 (Level-1 + Level-2)
Parameter	Coeff.	SE	Coeff.	SE	Coeff.	SE	Coeff.	SE	Coeff.	SE	Coeff.	SE
Fixed effect Intercept,							480.77***	2.91			479.02***	3.33
School/Teacher-Level 2 Predictors
ABGMATH			NS				NS				NS
ASSESS			NS				NS				NS
MACTIV			5.94**	2.51			12.98***	2.87			13.77***	3.45
MATHEXC			NS				–10.72**	4.61			–16.17***	5.40
PROPCERT			40.62***	7.14			59.63***	6.41			58.24***	6.54
PROPMA5A			16.85**	7.84			17.82**	7.61			17.47**	7.75
PROPMATH			–52.98***	17.56			–28.76*	15.27			–26.68*	16.11
PROPQUAL			NS				–14.54*	7.94			NS
SCHSIZE			0.04***	0.01			0.03***	0.01			0.03***	3.14
SMRATIO			–0.10**	0.05			NS				NS
STUDCLIM			11.30***	3.06			12.66***	2.95			16.41***	3.15
TCFOCST			–13.72***	3.78			–10.78***	3.49			–10.68***	3.96
TEACCLIM			8.37***	3.20			6.11**	3.03			NS
Student-Level 1 Predictors
SEX			–20.75***	2.05			–18.15***	2.01			–24.74***	2.22
LMINS			–0.05***	0.02			–0.06***	0.01			–0.08***	0.02
MMINS			0.05**	0.01			0.04***	0.02			0.05***	0.02
OUTHOURS			NS				NS				NS
USEMATH			–10.78***	1.11			–9.69***	1.09			–9.62***	1.14
EXAPPLM			–1.95*	1.12			NS				NS
EXPUREM			14.24***	1.04			14.39***	0.92			13.97***	1.28
FAMCON			22.96***	1.42			24.09***	1.34			27.85***	1.62
ST73Q01R^§			NS				NS				NS
ST73Q02R^§			NS				NS				NS
ST74Q01R^§			NS				8.16**	4.10			8.54**	4.13
ST74Q02R^§			9.63***	2.82			6.74**	3.00			8.10**	3.45
ST75Q01R^§			NS				–4.21*	2.23			NS
ST75Q02R^§			–8.85***	2.06			–6.93***	2.11			–10.03***	2.31
ST76Q01R^§			NS				NS				NS
ST76Q02R^§			–6.44**	2.48			–7.86***	2.31			–7.55**	2.57
Variance Component
Level 1 (σ²)	5371.36	82.97	4684.77	71.85	4653.62	61.13	3922.29	51.69	6218.09	99.30	5310.79	83.80
Level 2 (τ₀₀)	4518.23	138.33	3422.77	119.18	4576.19	129.58	3354.58	103.89	5356.21	153.96	3983.07	129.23
Percentage σ² explained			12.8%				15.7%				14.6%
Percentage τ₀explained			24.3%				26.7%				25.6%
Total variance explained at Level 1			7.0%				7.9%				7.8%
Total variance explained at Level 2			11.1%				13.2%				11.8%

R^§: Recoded: Never/Rarely = 0; Sometimes/Frequently = 1; NS: non-significant, p>0.10, *p<0.10, **p<0.05, ***p<0.01.

Table 3 also presents the total variance explained by factors in the models at Levels 1 and 2. The ability to formulate and employ has nearly equal variance explained at Level 1 (student level) compared to a relative lower variance explained at the same level for interpret. However, at Level 2, the variance explained is substantially higher. This confirms the importance of Level 2 predictors in terms of their contribution to explaining differences in higher-order thinking skills relevant to complete tasks associated with mathematical literacy. Thus, about 18% to 21% of the variance is explained by the model derived in Table 4, and about 79% (formulate) to 82% (interpret) is unexplained.

Table 4.

Interaction effects for the multilevel models explaining interpret, employ and formulate, respectively.

MAP – Interpret						MAP – Employ						MAP – Formulate
Fixed effect	Coefficient	Standard error	t-ratio	Approx. d.f.	p-value	Fixed effect	Coefficient	Standard error	t-ratio	Approx. d.f.	p-value	Fixed effect	Coefficient	Standard error	t-ratio	Approx. d.f.	p-value
For INTRCPT1, β₀						For INTRCPT1, β₀						For INTRCPT1, β₀
INTRCPT2, γ₀₀	489.058054	2.483066	196.957	2995	<0.001	INTRCPT2, γ₀₀	489.235156	2.226491	219.734	2995	<0.001	INTRCPT2, γ₀₀	488.129917	2.445614	199.594	2994	<0.001
PROPCERT, γ₀₁	50.560682	6.475320	7.808	2995	<0.001	MACTIV, γ₀₁	8.237407	2.142412	3.845	2995	<0.001	MACTIV, γ₀₁	11.519750	3.173010	3.631	618	<0.001
SCHSIZE, γ₀₂	0.031568	0.004882	6.466	807	<0.001	PROPCERT, γ₀₂	58.398026	6.223714	9.383	2995	<0.001	MATHEXC, γ₀₂	–15.951080	5.152592	–3.096	167	0.002
STUDCLIM, γ₀₃	8.093228	2.923809	2.768	2995	0.006	SCHSIZE, γ₀₃	0.027057	0.004065	6.655	2995	<0.001	PROPCERT, γ₀₃	58.830284	6.314360	9.317	1316	<0.001
TCFOCST, γ₀₄	–10.899580	3.164224	–3.445	738	<0.001	STUDCLIM, γ₀₄	11.288299	2.125537	5.311	2995	<0.001	SCHSIZE, γ₀₄	0.030977	0.004404	7.034	2481	<0.001
TEACCLIM, γ₀₅	8.041454	3.492014	2.303	755	0.022	TCFOCST, γ₀₅	–6.783757	3.224383	–2.104	1142	0.036	STUDCLIM, γ₀₅	11.032663	2.913136	3.787	2994	<0.001
For EXPUREM slope, β₁						For EXPUREM slope, β₁						TCFOCST, γ₀₆	–8.834872	3.625342	–2.437	2379	0.015
INTRCPT2, γ₁₀	14.068810	0.995893	14.127	363	<0.001	INTRCPT2, γ₁₀	15.065651	0.929948	16.201	200	<0.001	For EXPUREM slope, β₁
PROPCERT, γ₁₁	5.034221	2.038925	2.469	458	0.014	PROPCERT, γ₁₁	6.634085	2.019565	3.285	486	0.001	INTRCPT2, γ₁₀	15.075811	1.276848	11.807	28	<0.001
For FAMCON slope,						TCFOCST, γ₁₂	1.426201	0.828641	1.721	152	0.087	PROPCERT, γ₁₁	7.140783	2.467380	2.894	312	0.004
INTRCPT2, γ₂₀	22.507727	1.379258	16.319	65	<0.001	For FAMCON slope, β₂						For FAMCON slope, β₂
PROPCERT, γ₂₁	9.427383	2.737650	3.444	40509	<0.001	INTRCPT2, γ₂₀	23.200880	1.188761	19.517	229	<0.001	INTRCPT2, γ₂₀	27.231726	1.453194	18.739	226	<0.001
TCFOCST, γ₂₂	–2.284845	1.309650	–1.745	931	0.081	MACTIV, γ₂₁	1.806681	0.959968	1.882	123	0.062	MACTIV, γ₂₁	2.370406	1.120674	2.115	169	0.036
For LMINS slope, β₃						SCHSIZE, γ₂₂	0.003423	0.001689	2.027	13772	0.043	PROPCERT, γ₂₂	6.221314	3.193905	1.948	3709	0.052
INTRCPT2, γ₃₀	–0.024252	0.010147	–2.390	944	0.017	STUDCLIM, γ₂₃	3.091994	1.491298	2.073	410	0.039	SCHSIZE, γ₂₃	0.005017	0.002146	2.337	862	0.020
For USEMATH slope, β₄						TCFOCST, γ₂₄	–2.899474	1.215347	–2.386	1815	0.017	STUDCLIM, γ₂₄	2.933052	1.667790	1.759	193	0.080
INTRCPT2, γ₄₀	–11.136212	1.239460	–8.985	105	<0.001	For LMINS slope, β₃						TCFOCST, γ₂₅	–2.488820	1.539931	–1.616	838	0.106
For SEX slope, β₅						INTRCPT2, γ₃₀	–0.063682	0.016775	–3.796	657	<0.001	For LMINS slope, β₃
INTRCPT2, γ₅₀	–20.711180	2.044322	–10.131	42	<0.001	PROPCERT, γ₃₁	–0.061002	0.032780	–1.861	86	0.066	INTRCPT2, γ₃₀	–0.091954	0.016179	–5.683	155	<0.001
						For MMINS slope, β₄						MATHEXC, γ₃₁	0.024780	0.014436	1.717	2205	0.086
						INTRCPT2, γ₄₀	0.061382	0.017355	3.537	348	<0.001	PROPCERT, γ₃₂	–0.076783	0.032684	–2.349	164	0.020
						PROPCERT, γ₄₁	0.076380	0.035001	2.182	23	0.040	For MMINS slope, β₄
						For USEMATH slope, β₅						INTRCPT2, γ₄₀	0.066243	0.018956	3.495	97	<0.001
						INTRCPT2, γ₅₀	–9.896978	1.094499	–9.042	66	<0.001	PROPCERT, γ₄₁	0.072535	0.032877	2.206	108	0.029
						MACTIV, γ₅₁	–1.557475	0.843488	–1.846	33	0.074	For USEMATH slope, β₅
						For SEX slope, β₆						INTRCPT2, γ₅₀	–9.663735	1.123301	–8.603	70	<0.001
						INTRCPT2, γ₆₀	–17.630134	1.988485	–8.866	35	<0.001	MACTIV, γ₅₁	–1.860343	0.844413	–2.203	50	0.032
						PROPCERT, γ₆₁	9.486882	4.547731	2.086	164	0.039	For SEX slope, β₆
												INTRCPT2, γ₆₀	–24.207520	2.174651	–11.132	40	<0.001

To address Research Question 2, ‘What student (Level-1) factors are significant predictors of student performance in the interpret, employ and formulate processes?’ separate HLM models were run with each of the three processes as an outcome variable (see Table 3). The focus was on those student variables for which significant effects on each of the mathematical processes emerged in the respective Model 2.

Results for the student level predictors in Table 3 (Model 2) show that experience with pure mathematics tasks at school (EXPUREM) and familiarity with mathematical concepts (FAMCON) at the student level predicted mathematical literacy for all three cognitive processes when other factors were held constant. Although the positive relationship between EXPUREM and mathematical literacy has already been reported elsewhere (Koğar, 2015; OECD, 2013c), analyses in this study provide further insights of its relationships with the three cognitive processes. Thus, students who were familiar and had experiences with mathematical concepts and pure mathematics tasks at school did relatively better than students who had little or no experience or familiarity with associated concepts.

These findings parallel the findings reported by the New Zealand’s Ministry of Education (New Zealand – Ministry of Education, 2012). The report highlighted that ‘greater student familiarity with mathematical concepts and exposure to formal mathematics is related to higher mathematics achievement’ and that ‘greater perceived use of cognitive activation by students, where teachers encourage students to reflect on their learning in class, was related to higher mathematics achievement’ (New Zealand – Ministry of Education, 2012, pp. 50–51).

Use of Information and Communications Technology (ICT) in mathematics lessons (USEMATH) – a new scale created for PISA2012 – is a composite based on seven items (OECD, 2014b, p. 340). Table 3 provides evidence that USEMATH was significantly (p < 0.01) negatively related to all three outcome variables. In other words, greater use of ICT use during mathematics lessons was negatively linked to performance in interpret, employ and formulate. This is in line with findings of Bando et al. (2017), Rohaetia (2019), Skilling et al. (2020) and Watson (2001) who report that less use of ICT in mathematics lessons (USEMATH) provided better contexts for higher achievements in interpret items. It can be argued that the use of ICT to support mathematics needs careful planning and facilitation by appropriate pedagogy.

For all three processes, male students significantly outperformed girls which is noteworthy given that the modelling held all other effects constant.

Finally, for all three processes, the amount of time learning the language in which the test was administered in a particular country (LMINS) had a negative effect on performance while more mathematics learning time per week (MMINS) raised student’s performance.

Results to address Research Question 3 ‘What teacher/school (Level-2) factors are significant predictors of student performance in the interpret, employ and formulate processes?’ are also presented in Table 3.

Results show that some of the teacher/school-level effects were similar for all three processes. Thus, Table 3 highlights that students in schools which provided enrichment activities through extracurricular mathematics activities (MACTIV) had higher proportions of certified teachers (PROPCERT) and higher proportions of teachers who had studied mathematics as a major subject at university (PROPMA5A) showed higher performance in the three mathematical processes.

In contrast, more positive teacher-related factors affecting school climate (TEACCLIM) were associated with higher performance in interpret and employ but not in formulate. Noteworthy was also the finding that a lower Maths teacher-to-student ratio (SMRATIO) had a statistically significant (p < 0.05) effect, but only on interpret.

The additional negative effect of mathematical extracurricular activities at school (MACTIV) on employ and formulate through use of ICT in mathematics lessons (USEMATH) suggests that it might be beneficial to rethink how technology and extracurricular lessons are used, structured and delivered to optimise the learning of ‘reasoning’ of mathematical processes, and thus in improving achievement in mathematical literacy.

Results addressing Research Question 4 ‘What is the nature of the cross-level interactions between teacher/school (Level-2) predictors and student (Level-1) predictors of student performance in the interpret, employ and formulate processes?’ are presented in Table 4.

Interesting results emerged from the analysis of interaction effects of teacher/school-level variables on the relationship between student factors and performance. Thus, for larger schools (SCHSIZE) with more mathematics extracurricular activities (MACTIV) and with more positive student-related factors affecting school climate (STUDCLIM), an interaction could be observed with familiarity with mathematical concepts (FAMCON) which was associated with higher performance on the employ and formulate processes.

It was evident that the proportion of certified teachers (PROPCERT) in a school interacted positively with experience in pure mathematics tasks as school (EXPUREM) to enhance student performance on interpret items. Similar effects were noted for PROPCERT and familiarity with mathematical concepts (FAMCON).

Overall, Table 4 provides visual evidence of the increasing complexity of contextual effects as mathematical processes gradually increase in difficulty from interpret → employ → formulate in that more significant effects of teacher/school-level variables on the relationships between student-level variables and performance emerged. Perhaps the hypothesis of Stacey and Turner (2015) that higher-order cognitive processes (here formulate) are more difficult (relative to interpret and employ items), holds not only for the difficulty indices of items which involve formulate and employ but also for the relationships between factors having an effect on the cognitive processes involved. There is further evidence from Table 4 that complex cognitive processes in mathematics and subjects which require quantitative reasoning invoke more Level-1 and Level-2 factors. Familiarity of concepts (FAMCON), time learning the language of test administration (LMINS) and mathematics (MMINS) interact with relatively more factors in the models explaining differences in performance on the employ and formulate processes.

Conclusion and implications for research and practice

This study raises several issues that require urgent attention about composite contextual variables which predict performance in the three cognitive processes involved in mathematical literacy and specifically for the learning and teaching of mathematics. The importance of qualified mathematics teachers and their contributions through targeted school-based mathematics extracurricular activities is pivotal to success in mathematical literacy performance. The central role of qualified mathematics teachers is further supported through students’ familiarity with mathematical concepts and enrichment provided by quality learning time in the discipline. This becomes a necessity when students are required to utilise all three cognitive processes when solving problems in mathematics.

Gender interacts with the learning of mathematics and their differential performance in mathematical literacy warrants urgent attention. While many societal values and attitudes are likely to play a role in gender differences in mathematics, a highly granular analysis at the item or task level could provide understandings for the modification of test, curriculum and learning design.

In terms of what and how students learn, the frequency of procedural tasks, such as solving algebraic equations and finding the volume of abstract shapes in lessons (ST74Q01R) and tests (ST74Q02R), was positively related to achievement in mathematical literacy.

In contrast to procedural tasks, the frequency of pure mathematical reasoning, such as applying geometrical theorems to abstract shapes and answering conceptual questions about prime numbers in lessons (ST75Q01R) and tests (ST75Q02R), and the frequency of applied mathematical reasoning, such as solving problems from everyday life in tests (ST76Q02R), were negatively related to achievement in mathematical literacy. These findings warrant further investigation to scrutinise the reasons as to why an increased exposure to wider a variety of mathematics has a detrimental effect on students’ achievement. Although most research studies have examined the general performance in mathematical literacy, it is pertinent that higher-order thinking skills and cognitive processes are given specific attention and investigation.

One possibility is that students who had a greater exposure to procedural tasks (and consequently less exposure to mathematical reasoning) had a higher achievement in mathematical literacy because the literacy test may be biased towards those types of tasks. However, the PISA test incorporated a range of processes, content and contexts which suggests that a bias towards procedural tasks was unlikely. If anything, the definition of mathematical literacy used in this study is more focused on everyday applications of mathematics and the preparation of students for their post-school lives than is the case on other assessments.

Another hypothesis is that the teaching and learning centred on mathematical reasoning, vis-à-vis pure mathematics task, could be less robust than teaching and learning centred on procedural tasks. It may be that teachers were more adequately prepared and able to teach mathematical procedures and that teaching mathematical reasoning requires greater skill, understanding and mastery of the material.

The results in Tables 3 and 4 indicate that increased experience with pure mathematics tasks (EXPUREM) and familiarity with mathematics concepts (FAMCON) had positive effects on achievement in mathematical literacy. These results indicate that students benefited from exposure to mathematical concepts and procedures but did not benefit from exposure to reasoning. These results are of concern because students of all abilities will be expected and required not only to use mathematics, but more importantly to apply mathematical reasoning in novel situations. This necessity is articulated by Huang et al. (2016), for its centrality in curriculum design, and the professional development of teachers. They argued for the explicit teaching of mathematical reasoning and formalised as part of the learning for mathematics teachers.

Success in mathematical literacy tasks depends on students’ prior experiences of mathematics. If students have been mainly exposed to procedural tasks and mainly required to do rote learning and methodical practice of similar problems, they may struggle to cope when more mathematical reasoning, conceptualisation and more complex problems are introduced. Perhaps, as advanced by Rohaetia (2019), Schoenfeld (1992), Skilling et al. (2020) and Tall (2008, 2013), students were better equipped to respond to procedural tasks in mathematical literacy tests than to utilise the mathematical reasoning underlying each task/problem and step/sequence. The benefits of exposure to mathematical reasoning and its importance need to be emphasised in mathematics education. As emphasised by Jeannotte and Kieran (2017, p. 14), mathematical literacy involves mathematical reasoning and the latter includes processes of ‘the search for similarities and differences, and the search for validation’. They highlighted further that their mathematical reasoning model can be ‘applied to various mathematical content areas’ (Jeannotte & Kieran, 2017, p. 16).

Hence, the type of classroom activities, the type of mathematical thinking and reasoning need further scrutiny to optimise performance. The roles of theory, practice, pure mathematics, applied mathematics and ICT in the curriculum need further examination. This reiterates the recommendations of Thomson et al. (2013, p. 44) that ‘teachers can support students’ mathematics learning by providing direct and explicit instructions about strategies for understanding mathematics and tackling problems’.

Future research may want to consider associated mathematical processes beyond PISA’s – interpret, employ and formulate. Does exposure to procedural tasks improve students’ employment of procedures to solve problems? Does exposure to mathematical reasoning improve students’ interpretation of solutions? Although attitudinal variables were not included in this study, the variance explained by Level-1 predictors could be extended through their inclusion.

Quality time on meaningful mathematics tasks through teachers with optimal content and pedagogical knowledge is important. Although much professional development of teachers centres on pedagogical content knowledge, it is imperative that the focus and resources be committed to content knowledge, its renewal and extensions. The importance of content knowledge and associated pedagogies cannot be understated; this implies being mindful of the use of ICT in mathematics lessons and with pedagogy (and importantly the contents) in the driver seat (Brown et al., 2019; Fullan, 1991; Watson, 2001).

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Sivakumar Alagumalai

References

Adams

(Eds.). (2002). Programme for International Student Assessment (PISA): PISA 2000 technical report. OECD Publishing.

Alagumalai

(1999). Problem solving and AI as a tracking tool [Unpublished PhD thesis]. Flinders University.

Alagumalai

(2012). Ramanujan’s insights: Implications for mathematics education. National year of mathematics: Special issue. Experiments in Education Journal, 40(4), 109–138.

Alivernini

Palmerio

Vinci

Di Leo

(2010). An analysis of factors affecting pupils’ science achievement in Italy [Paper presentation]. Fourth IEA (International Association for the Evaluation of Educational Achievement) International Research Conference Proceedings, Gothenburg. http://www.iea.nl/fileadmin/user_upload/IRC/IRC_2010/Papers/IRC2010_Alivernini_Palmerio_etal.pdf

Allison

P. D.

(2002). Missing data. SAGE Publications.

Anderson, D. (2012). Hierarchical Linear Modeling (HLM): An Introduction to Key Concepts Within Cross-Sectional and Growth Modeling Frameworks. Technical Report # 1308. Behavioral Research and Teaching. University of Oregon. https://files.eric.ed.gov/fulltext/ED545279.pdf

Bando

Gallego

Gertler

Fonseca

D. R.

(2017). Books or laptops? The effect of shifting from printed to digital delivery of educational content on learning. Economics of Education Review, 61, 162–173.

Brown

R. E.

Weiland

Orrill

C. H.

(2019). Mathematics teachers’ use of knowledge resources when identifying proportional reasoning situations. International Journal of Science and Mathematics Education, 18(4), 1085–1104.

Coyle

T. R.

Snyder

A. C.

Richmond

M. C.

(2015). Sex differences in ability tilt: Support for investment theory. Intelligence, 50, 209–220.

10.

Doris

O’Neill

Sweetman

(2012). Gender, single-sex schooling and Maths achievement (IZA Discussion Paper No. 6917). Institute for the Study of Labor (IZA). http://ftp.iza.org/dp6917.pdf

11.

Fullan

(1991). The new meaning of educational change (2nd ed.). Cassell.

12.

Goos, M., Geiger, V., Bennison, A., & Roberts, J. (2015). Numeracy teaching across the curriculum in Queensland: Resources for teachers. Final Report. Brisbane: The University of Queensland. https://cdn.qct.edu.au/pdf/Numeracy_Teaching_Across_Curriculum_QLD.pdf

13.

Hasan

Khan

(2015). Academic achievement test in English and Mathematics of secondary school students in relation to gender differences. The International Journal of Indian Psychology, 2(3), 155–166.

14.

Hox

(2002). Multilevel analysis: Techniques and application. Lawrence Erlbaum Associates.

15.

Huang

Prince

(2016). Professional development of secondary mathematics teachers in mainland China. In Kaur

Kwon

D.N.

Leong

Y.H.

(Eds.), Professional development of Mathematics teachers: An Asian perspective (pp. 17–32). Springer.

16.

Jeannotte

Kieran

(2017). A conceptual model of mathematical reasoning for school mathematics. Educational Studies in Mathematics, 95(3), 1–16.

17.

Jitendra

A. K.

Nelson

Pulles

S. M.

Kiss

A. J.

Houseworth

(2016). Is mathematical representation of problems an evidence-based strategy for students with mathematics difficulties? Exceptional Children, 83(1), 8–25.

18.

Keeves

J. P.

(1976). Curricular factors influencing school learning. Studies in Educational Evaluation, 2(3), 167–183.

19.

Koğar

(2015). Examination of factors affecting PISA 2012 mathematical literacy through mediation model. Education and Science, 40(179), 45–55.

20.

L. L.

Bradley

K. D.

(2008). Using multilevel modeling to investigate school effects. In O’Connell

A. A.

McCoach

D. B.

(Eds.), Multilevel modeling of educational data (pp. 59–110). Information Age Publishing.

21.

Monseur

Adams

(2009). Plausible values: How to deal with their limitations. Journal of Applied Measurement, 10(3), 320–334.

22.

Mora

R. P.

(2020). Assessing the attitudes, aspirations, and gender differences of pre-service mathematics teachers. Journal of Business and Economics Review, 5(1), 18–27.

23.

Namkung

J. P.

Peng

X. L.

(2019). The relation between mathematics anxiety and mathematics performance among school-aged students: A meta-analysis. Review of Educational Research, 89(3), 459–496.

24.

New Zealand – Ministry of Education. (2012). PISA2012: New Zealand summary report. NZ-Ministry of Education - Comparative Education Research Unit.

25.

Nguyen

T. T.

Trinh

T. P. T.

Ngo

H. T. V.

Hoang

N. A.

Tran

Pham

H. H.

Bui

V. N.

(2020). Realistic mathematics education in Vietnam: Recent policies and practices. International Journal of Education and Practice, 8(1), 57–71.

26.

Nye

Konstantopoulos

Hedges

L. V.

(2004). How large are teacher effects? Educational Evaluation and Policy Analysis, 26(3), 237–257.

27.

O’Dwyer

L. M.

Parker

C. E.

(2014). Analyzing nested data using multilevel modeling in SPSS (REL 2014–046). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Central. http://ies.ed.gov/pubsearch/pubsinfo.asp?pubid=REL2015046

28.

O’Dwyer

L. M.

Wang

Shields

K. A.

(2015). Teaching for conceptual understanding: A cross-national comparison of the relationship between teachers’ instructional practices and student achievement in mathematics. Large Scale Assessments in Education, 3(1), 1–30.

29.

Organisation for Economic Co-operation and Development. (2009). PISA 2009 assessment framework: Key competencies in reading, mathematics and science. OECD Publishing.

30.

Organisation for Economic Co-operation and Development. (2010). Annex A5: Technical notes on analyses in Volume IV. In OECD (2010), PISA 2009 Results: What makes a school successful? Resources, policies and practices - Volume 4. OECD Publishing, pp. 142–151.

31.

Organisation for Economic Co-operation and Development. (2013a). PISA 2012 assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy. OECD Publishing.

32.

Organisation for Economic Co-operation and Development. (2013b). Ready to learn: Students’ engagement, drive and self-beliefs. Volume III – PISA2102. OECD Publishing.

33.

Organisation for Economic Co-operation and Development. (2013c). PISA 2012 results: What students know and can do: Student performance in mathematics, reading and science. Volume I – PISA2102. OECD Publishing.

34.

Organisation for Economic Co-operation and Development. (2014a). PISA 2012 results in focus: What 15-year-olds know and what they can do with what they know. OECD Publishing.

35.

Organisation for Economic Co-operation and Development. (2014b). PISA 2012 technical report. OECD Publishing.

36.

Organisation for Economic Co-operation and Development. (2014c). Schooling matters: Opportunity to learn in PISA 2012 (OECD Education Working Paper No. 95 - EDU/WKP[2013]10). OECD Publishing.

37.

Organisation for Economic Co-operation and Development. (2014d). Teacher remuneration in Latvia: An OECD perspective. OECD Publishing.

38.

Organisation for Economic Co-operation and Development. (2015a). A review of international large-scale assessments in education: Assessing component skills and collecting contextual data. OECD Publishing.

39.

Organisation for Economic Co-operation and Development. (2016a). PISA – Low performing students: Why they fall behind and how to help them succeed. OECD Publishing.

40.

Organisation for Economic Co-operation and Development. (2016b). Equations and inequalities: Making mathematics accessible to all, PISA. OECD Publishing.

41.

Plomp

(1992). Conceptualizing a comparative educational research framework. Prospects, 22(3), 278–288.

42.

Raghunathan

(2015). Missing data analysis in practice. CRC Press/Taylor Francis Group.

43.

Raudenbush

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). SAGE Publications.

44.

Raudenbush

S. W.

Bryk

A. S.

Cheong

Y. F.

Congdon

R. T.

Du Toit

(2019). HLM 8: Hierarchical linear and nonlinear modeling. Scientific Software International.

45.

Raudenbush

Willms

J. D.

(1991). The organisation of schooling and its methodological implications. In Raudenbush

S. W.

Willms

J. D.

(Eds.), Schools, classrooms, and pupils: International studies of schooling from a multilevel perspective (pp. 1–12). Academic Press.

46.

Reilly

Neumann

D. L.

Andrews

(2016). Sex and sex-role differences in specific cognitive abilities. Intelligence, 54, 147–158.

47.

Rohaetia

E. E.

(2019). Building students’ hard and soft skills through innovative teaching approaches to mathematics. Journal of Southwest Jiaotong University, 54(5), 1–16.

48.

Rutkowski

Gonzalez

Joncas

von Davier

(2010). International large-scale assessment data: Issues in secondary analysis and reporting. Educational Researcher, 39(2), 142–151.

49.

Schoenfeld

A. H.

(1992). Learning to think mathematically: Problem solving, metacognition, and sense-making in mathematics. In Grouws

(Ed.), Handbook for research on mathematics teaching and learning (pp. 334–370). MacMillan.

50.

Sikora

Saha

L. J.

(2010). New directions in national education policymaking: Student career plans in international achievement studies. In Wiseman

(Ed.), The impact of international achievement studies on national education policymaking (pp. 83–115). Emerald Group Publishing Ltd.

51.

Skilling

Bobis

Martin

A. J.

(2020). The “ins and outs” of student engagement in mathematics: Shifts in engagement factors among high and low achievers. Mathematics Education Research Journal. https://doi.org/10.1007/s13394-020-00313-2

52.

Smith, P., Ladewig, M., & Prinsley, R. (2018). Improving the mathematics performance of Australia's students. Office of the Chief Scientist. Canberra: Australian Government. https://www.chiefscientist.gov.au/2018/07/occasional-paper-improving-the-mathematicsperformance-of-australias-students/

53.

Snijders

T. A. B.

Bosker

R. J.

(2012). Multilevel analysis: An introduction to basic and advanced modeling (2nd ed.). Sage.

54.

Stacey

(2001). The international assessment of mathematical literacy: PISA 2012 Framework and items [Paper presentation]. 12th International Congress on Mathematical Education Program, 8 July – 15 July, 2012, COEX, Seoul, Korea, pp. 6–7. http://www.icme12.org/upload/submission/2001_f.pdf

55.

Stacey

Turner

(2015). The evolution and key concepts of the PISA mathematics frameworks. I n Stacey K & Turner R. (Eds.), Assessing Mathematical Literacy: The PISA Experience (pp. 5–33). Springer.

56.

Stoet

Geary

D. C.

(2013). Sex differences in mathematics and reading achievement are inversely related: Within- and across-nation assessment of 10 years of PISA data. PLoS One, 8(3), e57988.

57.

Tall

(2008). The transition to formal thinking in mathematics. Mathematics Education Research Journal, 20(2), 5–24.

58.

Tall

(2013). How humans learn to think mathematically: Exploring the three worlds of mathematics. Cambridge University Press.

59.

Tambychik

Meerah

T. S. M.

(2010). Students’ difficulties in mathematics problem solving: What do they say? Procedia Social and Behavior Science, 8, 142–151.

60.

Thomson

Hillman

De Bortoli

(2013). A teacher’s guide to PISA mathematical literacy. Australian Council for Educational Research.

61.

Toh

K. A.

Alagumalai

(1994). Gender differences: They don’t happen here. Journal of Science and Mathematics Education in South East Asia, 17(1), 54–60.

62.

United Nations Children's Fund. (2012). Equity in learning? A comparative analysis of the PISA 2009 results in central and Eastern Europe and the Commonwealth of Independent States. UNICEF Regional Office for Central and Eastern Europe and the Commonwealth of Independent States (CEE/CIS).

63.

Victorian Curriculum and Assessment Authority (VCAA). (n.d.). Victorian Curriculum Foundation–10: Mathematics. Maths curriculum - Rationale and aim. https://victoriancurriculum.vcaa.vic.edu.au/mathematics/introduction/rationale-and-aims

64.

Von Davier

Gonzalez

Mislevy

R. J.

(2009). What are plausible values and why are they useful? IERI Monograph Series, 2(1), 9–36.

65.

Watson

D. M.

(2001). Pedagogy before technology: Re-thinking the relationship between ICT and teaching. Education and Information Technologies, 6(4), 251–266.

66.

Willms

J. D.

(2010). School composition and contextual effects on student outcomes. Teachers College Record, 112(4), 1008–1037.

PISA 2012: Examining the influence of prior knowledge,time-on-task,school-level effects on achievements in mathematical literacy processes – Interpret,employ and formulate

Abstract

Keywords

Introduction

Synthesis of literature and foci of current study – Processes and contents underlying mathematical literacy

Rationale for selection of variables for study

Research questions and models

Methodology – Multilevel modelling in education

Sample and dataset

Results and discussion

Conclusion and implications for research and practice

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iD

References