Abstract
The factor structure of the measure of school engagement ‘Engagement Versus Disaffection with Learning: Teacher Report’ was investigated in the sample of 360 Swedish 6th graders. Confirmatory factor analyses showed no support for the suggested four-factor structure including behavioural engagement, emotional engagement, behavioural disaffection and emotional disaffection. Exploratory analyses resulted instead in solutions differentiating between indicators of engagement and indicators of emotional problems. A four-factor solution including factors of positive engagement, disengagement, internalized and externalized emotional problems had the best fit. Positive engagement and disengagement factors showed good internal consistency with omega coefficients exceeding 0.95 and are deemed suitable for further use. Several challenges of measuring the complex construct of school engagement are discussed.
Introduction
School engagement is an important aspect of educational experience that reflects the quality of education and drives children’s learning not only in academic, but also in social, behavioural and emotional domains (Reschly & Christenson, 2012). Various conceptualizations of this multifaceted construct have been proposed by different researchers (Finn & Zimmer, 2012). Building on Finn’s (1989) early conceptualization of engagement as consisting of participatory behaviours and identification with school, most contemporary conceptualizations of engagement include behavioural and affective components as separate dimensions of school engagement.
Behavioural engagement refers to paying attention and completing assignments in class, following classroom rules, positively interacting with teachers and peers, taking initiative and doing more than necessary or to participating in extracurricular activities (Finn & Zimmer, 2012). The affective component of engagement refers to children’s sentiments towards the school such as a sense of belonging and appreciation for school, or to affective reactions in the classroom, and is thought to predict behavioural engagement (Ladd & Dinella, 2009).
Additional aspects of engagement have been proposed in engagement literature. Cognitive engagement is often considered as a distinct component of engagement (Fredricks et al., 2004), academic engagement is used by some authors to specify the focus on academic tasks (Appleton et al., 2006), and students’ agency has been suggested as an additional relevant component of school engagement (Reeve & Tseng, 2011). Phenomenological studies indicate that teachers’ understanding of the concept is even more complex and layered (Harris, 2008).
Several authors argue that behavioural engagement presents a complex multidimensional expression of students’ perceptions, beliefs and motives and cannot be considered on a continuum from low to high engagement (Hospel et al., 2016; Skinner et al., 2009). It is important to distinguish between an unengaged student and a disengaged student prone to absenteeism, withdrawal and disruptive behaviours (Hospel et al., 2016; Portelli & McMahon, 2004). For these reasons, measures of engagement can include markers of engagement and disengagement, which are assumed to negatively correlate but are distinct dimensions of engagement (Finn & Zimmer, 2012).
Arguing that a comprehensive assessment of engagement should include markers of emotional and behavioural aspects of engagement, and a separate dimension of disaffection, Skinner et al. (2009) developed the questionnaire ‘Engagement Versus Disaffection with Learning: Teacher Report (EDL: TR)’, which consists of four subscales: emotional engagement, behavioural engagement, emotional disaffection and behavioural disaffection.
EDL: TR has previously been translated to Swedish (Elevernas engagemang i lärande; EEL; Ritoša et al., 2020) following the COSMIN checklist for cross-cultural validation of instruments (Mokkink et al., 2010). The item relevance, dimensional structure, convergent validity, test-retest reliability, and internal consistency of the questionnaire have been investigated on a sample of six to seven-year-old preschool children (Ritoša et al., 2020). Pre-schoolers’ scores on EDL: TR correlated highly with their scores on the Children’s Engagement Questionnaire (CEQ; McWilliams, 1991), a measure of engagement for preschool children. The study found no support for the four-factor structure proposed by Skinner et al. (2009). Exploratory factor analysis resulted in two factors, behavioural and emotional engagement, while behavioural and emotional disaffection items loaded negatively on the two corresponding engagement factors. Behavioural engagement factor showed good internal consistency (Ω = 0.96 [0.94, 0.97]) and test–retest reliability (r = .81, p < .001), while the emotional engagement factor showed good internal consistency (Ω = 0.94 [0.93, 0.96]) (Ritoša et al., 2020) and acceptable test–retest reliability (r = .71, p < .001).
The purpose of the current study was to investigate the factor structure of the school engagement questionnaire ‘Engagement Versus Disaffection with Learning: Teacher Report (EDL: TR)’ in a sample of Swedish school children. Although disaffection items did not make a separate factor in the preschool population, it is hypothesised that the four-factor structure differentiating between engaged behaviours, engaged emotions, disaffected behaviours and disaffected emotions will have more support in the sample of six-graders. As children mature, their motivation and interests become more complex, and their self-regulation skills improve (McClleland et al., 2015). For these reasons, a more complex picture of engagement dimensions is expected in six-graders in comparison to preschool class children. Developmental changes are also followed by increased educational demands; since the school environment is more structured in comparison to the preschool class environment, it is expected that teacher reports of engagement will distinguish between the elements of engagement and disaffection, as well as between behavioural and emotional aspects of engagement, as suggested by Skinner et al. (2009).
Method
Participants
Characteristics of Classrooms and Participant Teachers.
Note. The classroom with 48 pupils had 2 teachers.
a29% of the elementary school pupils had a right to educational support in their mother tongue in the school years 2019/20 and 2020/21 (Skolverket, 2021).
Data Collection
Data collection for this study took place from autumn 2019 to spring 2021, as a part of a follow-up research project. Children previously included in a research project that took place in Swedish preschools in 2012–2014, were now followed up in grade 6. This study uses survey data about 49 children that were followed up and 311 of their classmates in grade 6.
Organisation Statistics Sweden (Statistik centralbyrå, SCB) was employed to deliver information about the school placement for the children followed up from preschool. Researchers then contacted schools and informed the school principals and children’s class teachers about the study via phone. The followed-up children, their classmates and their parents were informed about the study via parental meetings, informing videos, written information, and verbally by the teachers. Passive consent to participation in the study was required from parents and children and they had a right to decline participation by informing the teacher. Teachers were offered compensation for the time they put in answering the questionnaires. Regional Ethical Board in Linköping has approved the data collection procedure (Dnr 2018/189-31). In the spring of 2020, communication between researchers and participants was adapted to follow the schools’ COVID-safety regulations. Information about the study was delivered to the parents through videos instead of parental meetings. Questionnaires were sent to teachers by post instead of being personally delivered by the researchers.
Materials
‘Engagement Versus Disaffection with Learning: Teacher Report’ (EDL: TR; Skinner et al., 2009) is a measure of children’s engagement and disaffection in learning activities in school. It consists of 25 items across 4 subscales. The behavioural engagement subscale consists of five items about investing effort, persistence and attention. The emotional engagement subscale consists of five items asking about energized emotional states such as enthusiasm, interest and enjoyment expressed during learning activities. Behavioural disaffection consists of five items tapping lack of effort and withdrawal. The emotional disaffection subscale consists of ten items about negative school-related emotions including boredom, frustration, anxiety, emotional withdrawal and alienation during learning activities. Although these disaffected emotions could be further distinguished in the student surveys, authors reported that they belong to a single dimension of emotional disaffection when analysing teacher surveys of pupils’ engagement (Skinner et al., 2009). An alternative three-factor model of the scale was reported by Immekus and Ingle (2019) who identified three distinct factors: behavioural engagement, emotional engagement and emotional disaffection.
Questionnaire items are presented in the Appendix. Items are to be answered by teachers, and answers are given on a 4-point Likert-type scale. In the Swedish version, the rating scale can be translated as ‘almost never happens’ (1), ‘sometimes happens’ (2), ‘happens quite often’ (3) and ‘happens very often’ (4). The administration is expected to take 5–10 minutes per child.
Data Analysis
Descriptive item-levels analysis and subsequent factor analyses were performed in the R programme (R Core Team, 2018) using psych (Revelle, 2019) and lavaan packages (Rosseel, 2019).
Score distributions of items, Spearman’s correlations between items, sampling adequacy and Bartlett’s test of sphericity were checked first to see if all items should be included in the subsequent factor analyses. Confirmatory factor analysis for ordinal data (WLSMW estimator) was then run on the factor structure proposed by Skinner et al. (2009). Additional exploratory factor analyses based on the minimum residual estimation method and polychoric correlation matrices appropriate for ordinal variables were planned in case CFA did not show support for the proposed model (Watkins, 2018).
Parallel analysis and interpretability of factors were the main criteria for deciding the number of factors in EFA (Henson & Roberts, 2006). Correlation between factors was allowed in the exploratory models as it was expected that different aspects of engagement would correlate. CFAs were run on the models resulting from exploratory analysis to assess their fit. Since the chi-square difference statistic can indicate a bad fit of the data to the model due to large sample size, root mean square error of approximation (RMSEA), comparative fit index (CFI), and Tucker–Lewis Index (TLI) indices of model fit were used to estimate the model fit. RMSEA values less than 0.06, and CFI and TLI values above 0.96, were used to indicate good model fit (Hu & Bentler, 1999). Lastly, the omega total coefficient was calculated for each subscale to assess reliability, as suggested by Mcneish (2018).
Results and Discussion
Item Descriptive Statistics.
Note. Minimum and maximum values were 1–4 across all items.
All the items significantly correlated with all the other items in the questionnaire, with Spearman’s rho ranging from .18 to .82. Four pairs of items correlated highly (>.8): doing more than necessary and working harder after experiencing failure (items 4 and 5), interest in the new content and enjoyment in classwork (item 8 and 9), enjoyment in classwork and fun in learning (items 9 and 10) and appearing bored and looking bored (items 16 and 17). Item 16 was excluded from the further analysis due to redundancy in meaning with item 17.
Sampling adequacy of the remaining items was good (KMO = 0.95), and Bartlett’s test of sphericity indicated that the correlation matrix is different from the identity matrix (χ2 = 7929.56, df = 276, p < .001).
CFA was run on the 24 remaining items to investigate if the four-factor structure suggested by Skinner et al. (2009) fits the data.
Fit indices (χ2, CFI, TLI and RMSEA values) for all the models tested with CFA.
Note. aFour-factor model (Skinner et al., 2009), bTwo-factor model (behavioural and emotional engagement), cTwo-factor model (engagement and disengagement), dOne-factor model (general engagement), eTwo-factor model resulting from EFA, fTwo-factor model resulting from EFA after removing items 22 and 23, gFour-factor model resulting from EFA.
*significant at p < .001 level.
A more suitable alternative factor structure was then explored. Parallel analysis suggested 4 factors and 2 components and two EFAs were run looking for two- and four-factor solutions.
Item Factor Loadings for the Two-Factor and Four-Factor EFA Solutions.
Note. Engage = engagement, IntEmoPr = internalized emotional problems, PosEng = positive engagement, Diseng = disengagement, ExtEmoPr = externalized emotional problems.
A four-factor solution was interpreted as ‘positive engagement’, ‘disengagement’, ‘internalized emotional problems’ and ‘externalized emotional problems’. The majority of the positively oriented items previously belonging to behavioural engagement and emotional engagement subscales loaded to the factor ‘positive engagement’, with the addition of one item from the behavioural disaffection subscale which needs to be recoded to have a positive loading. The second factor named “disengagement” included items describing lack of interest in school and cognitive absenteeism. The first two factors were in a high negative correlation (r = −.77). The third factor, ‘internalized emotional problems’, included four items from the emotional disaffection subscale and one from the emotional engagement subscale which needs to be recoded to have a positive loading to this factor. It was in a moderate correlation with the first two factors (−.50 and .52, respectively). The fourth factor titled ‘externalized emotional problems’ included only two items and was in low correlation with the first three factors (.02, .11 and. 00, respectively). Factor loadings are presented in Table 4. Although chi-square was significant, the other fit indices for the four-factor solution were good, as shown in Table 3.
After recoding negatively oriented items, Omega total coefficient for ordinal scales was calculated to check for internal consistency of the four factors. For the positive engagement factor, Omega was 0.98 [0.97, 0.98], for the disengagement factor Omega was 0.96 [0.96, 0.97], and for internalized emotional problems factor Omega was 0.92 [0.91, 0.93], indicating good internal consistency of the three factors. Spearman’s correlation between the two items in the fourth factor was 0.49.
In summary, the results indicated that teacher reports of behavioural and emotional features of school engagement are better represented by a single factor of positive engagement, while the disaffection dimension showed greater complexity than expected. Besides the general disengagement factor which captured indicators of behavioural and emotional disengagement, items capturing symptoms of internalized and externalized emotional problems loaded onto separate factors. This questionnaire previously showed good convergent validity in comparison to a measure of preschool engagement, and good test–retest reliability (Ritoša et al., 2020). The current study further supports the use of positive engagement and disengagement subscales with the school-aged children, while the remaining items seem to capture antecedents of engagement that are related, but distinct from the engagement construct. Emotional problems, along with self-regulation and motivation can be thought of as predictors of school engagement (McClelland et al., 2015). Although it is important to identify causes of children’s low school engagement, including these antecedents of engagement into engagement assessment can be misleading if the aim is to study engagement as a malleable mediator between the child and environmental characteristics and learning outcomes (Finn & Zimmer, 2012).
Conclusions about the construct of engagement are limited by the content of the questionnaire items used in this study (Fredricks & McColskey, 2012). As described in the introduction, literature on engagement recognizes additional aspects of the engagement concept not included in this questionnaire. Another major limitation of this study is the lack of children’s perspective. Student reports are likely to be more appropriate than proxy ratings for assessing internal aspects of school engagement as distinct from behavioural engagement.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Vetenskapsrådet, 2017-03659.
