Abstract
Student engagement, while widely recognized as both important to outcomes of higher education institutions and critical to student learning, is still missing a unifying model addressing students both inside and outside of the classroom. This mixed-methods paper will explore the existing models of student engagement and propose a model which better supports analysis from multiple perspectives. This model was validated within the context of Vietnamese higher education. The results indicate that student engagement, both in-class and out-of-class, can be modeled through the four sub-components of emotional, cognitive, participatory, and agentic engagement. Having these common elements allows analysis vertically, through these four dimensions, or horizontally, across two contexts, providing greater flexibility for future investigations.
Introduction
The topic of student engagement (SE) emerged near the end of the last century and has quickly become the latest concern for both researchers and practitioners in promoting teaching and learning in higher education (HE). In a broad sense, the concept of SE, without the name, has been around for quite some time. One of the very first mentions is the pioneering work of Ralph Tyler from the 1930s, which showed the positive relationship between time-on-task and learning (Merwin, 1969, cited by Kuh, 2009). Bloom (1974) confirmed similar results.
As the concept of SE developed, variations emerged with different definitions that contain similar, though not always consistent, components (Appleton et al., 2008; Bond, 2020; Fredricks et al., 2004; Fredricks & McColskey, 2012; Furlong et al., 2003). Throughout the literature, a common list of sub-components can be found as emotional, cognitive, and behavioral/participatory engagement. While certain definitions explicitly focus on the in-class context, some incorporate both in-class and out-of-class/campus environments.
In order to focus on the level of individual engagement and to cover both in-class and out-of-class contexts, this research is based on the concept proposed by Kuh et al. (2007), in which SE is conceptualized as students’ involvement, inside and outside of the classroom, in effective practices of learning leading to measurable outcomes.
Over the last decade, SE has stayed on top of meeting agendas and conference themes around the world (Leach, 2012; V. Trowler, 2010). More specifically, P. Trowler and Trowler (2010) concluded that there is no longer a question as to the value SE with supporting evidence from the (US) National Institute of Education’s Involvement in Learning Report. Kahu (2013, p. 758) also agreed that “Student engagement is widely recognized as an important influence on achievement and learning in higher education and as such is being widely theorized and researched.”
The reason why SE has been of increasing interest to researchers in higher education is its association with students’ academic achievements, school completion, well-being (Kahu, 2013), as well as other long-term outcomes, such as work success (Christenson et al., 2012). However, there is little consensus on the conceptualization of SE among researchers (Bond, 2020; Fredricks et al., 2004; Furlong et al., 2003), except that SE is a multifaceted, multidimensional meta-construct (Burch et al., 2015; Fredricks et al., 2004; Kahu, 2013).
One way to view SE is with two contexts: In-class and out-of-class (or campus) engagement (Gunuc & Kuzu, 2015). Another way to view SE is that it contains the four components of emotional, cognitive, behavioral/participatory, and agentic engagement (Reeve & Tseng, 2011). However, so far agentic engagement has only been considered within the in-class context. It has never been examined and measured in the out-of-class context. If agentic engagement can be confirmed as a separate element of out-of-class SE then it would be possible to have a unifying model of SE which includes the four dimensions of SE across the two contexts. While this would not include every context, unifying these two contexts begins to move the study of SE toward a holistic model.
As a result, this study aims to integrate agentic engagement as a distinct element of out-of-class SE by developing a unifying model using the elements of emotional, cognitive, behavioral/participatory, and agentic engagement across both in-class and out-of-class engagement. The seven-step procedure by Hinkin et al. (1997) was adopted to confirm the validity of all instruments in this proposed model on a sample of 312 undergraduate students of business and management in Vietnam.
Literature Review
Dimensions of Student Engagement
There are multiple ways of defining the dimensions of SE, based on (1) contexts of student engagement; and (2) components of student engagement.
Contexts of student engagement
The first approach in identifying the dimensions of SE is to contrast the contexts of SE.
Originally, the concept of SE was primarily developed within the classroom context (Astin, 1984; Bloom, 1974; Newmann, 1992) and was said to change with educational level (Gunuc & Kuzu, 2015). When research on SE in higher education emerged, the environment of SE was extended outside the classroom, showing a more complete educational environment for students (Audas & Willms, 2001; Finn & Voelkl, 1993; S. Hu & Kuh, 2002; Voelkl, 1997).
In the following model of SE, theorized by Gunuc and Kuzu (2015), there are two main contexts of in-class and out-of-class (campus) engagement, each with three sub-components (Figure 1).
Earlier theoretical model of student engagement.
In-class engagement
In the classroom context, SE is mostly seen as engagement with learning. This context is very popular for doing research with students from kindergarten through HE.
When it comes to HE, in-class engagement is described in terms of student participation in academic and educational activities, their efforts, interests, and their investment in class (Appleton et al., 2008, 2006), such as complying with classroom procedures and taking initiative in the classroom (Finn, 1989). In-class engagement therefore refers to an investment in learning, learning goals, planning, self-regulated learning, as well as using different learning strategies (Fredricks et al., 2004), their attitudes, interests, their relationships to the teacher/staff, peers, course, and content (Kahu, 2013), and their feelings of belonging to the class (Fredricks et al., 2004; Kember et al., 2001).
Disengaged students may experience boredom, anxiety, shame, sadness, or frustration (Finn, 1989), exhaustion, lack of concentration, apathy, inattention, amotivation, or may withdraw from learning tasks (Skinner & Pitzer, 2012).
What happens in the classroom, impacting SE, is vital as it actually leads to students’ learning and development (Appleton et al., 2008; Skinner & Pitzer, 2012). Therefore, SE in the classroom plays an important role contributing to the quality of students’ daily experiences while they are attending school (Skinner & Pitzer, 2012).
Out-of-class or campus engagement
In higher education, students have more opportunities to interact with the broader school community as they need to prepare for the adult environment, real-life. Therefore, SE at university is often associated with the school community (Bryson, 2010; Fullarton, 2002).
Out-of-class engagement is considered important for SE and quality learning outcomes (Gunuc & Kuzu, 2015; Hausmann et al., 2007). While Gunuc and Kuzu’s (2015) theoretical model of campus engagement included valuing, sense of belonging, and participation, these three elements map well to the widely accepted cognitive, emotional, and behavioral engagement, respectively. Valuing school is about the belief (cognition) that students have (Voelkl, 1996). The sense of belonging is about the student’s feelings (Voelkl, 1996; Willms, 2003). Participation represents student behavior (Finn, 1989).
Components of student engagement
The second approach in identifying the dimensions is to explore the various components of SE.
While earlier studies identified two components of SE, emotional and behavioral (Finn, 1993; Marks, 2000; Newmann, 1992; Willms, 2003), more recent studies have added a third component, cognitive (Appleton et al., 2008; Fredricks et al., 2004; Yazzie-Mintz, 2007) (Figure 2).
Three primary dimensions of engagement.
Most research studies before 2000 adopted a one- or two-element construct approach when studying SE and rarely chose to work with all three elements at the same time (Fredricks et al., 2005).
In an effort to continue to refine and complete the concept, some recent studies have proposed a fourth component of SE. A recent model introduced by Reeve and Tseng (2011) proposed adding agentic engagement since they believed students can actively participate and contribute to the education process, thereby affecting their own learning outcomes (Lawson & Lawson, 2013). Later studies have confirmed the validity of agentic engagement as a distinct and meaningful component of in-class SE (Jang et al., 2016; Reeve, 2012; Sinatra et al., 2015).
Next, each component of SE will be briefly reviewed.
Behavioral or participatory engagement
Behavioral engagement is drawn on the idea of participation. Finn (1993) proposed that behavioral engagement can be involvement in academic and social/extracurricular activities, and it is found to be correlated with achieving positive academic outcomes and preventing drop out.
Behavioral engagement is the first component of student engagement that many instructors and researchers paid attention to because it can be observed directly. It helps instructors know how they can motivate students and affect their performance during the learning process.
In this paper, behavioral/participatory engagement can be operationalized in two main contexts, in-class and out-of-class. For the classroom context, behavioral engagement can be observed and measured by attendance, participation, preparation for class (Appleton et al., 2006), positive conduct like effort, persistence, intensity, attention, absorption, and involvement (Skinner et al., 2008). For the out-of-class context, behavioral engagement is manifested in the participation in school-related activities such as athletics and clubs (Sinatra et al., 2015).
Behavioral engagement has been found to be positively correlated with academic achievement in numerous research studies (Finn, 1993; Fullarton, 2002; Klem & Connell, 2004; Marks, 2000). This association has been explained as when students participate in their own learning and become involved in school-related activities, they often begin to find themselves identifying with school resulting in higher retention, increased completion (Finn, 1993), and potentially to positive outcomes such as higher academic achievement (Marks, 2000).
Emotional engagement
Emotional engagement involves reactions to teachers, classmates, academics, and school and is presumed to create ties to an institution and influences willingness to do the work (Finn, 1993). The term emotional engagement is used to describe students’ social, emotional, and psychological attachment to school (Finn & Voelkl, 1993; Reschly & Christenson, 2012).
In the context of the classroom, emotional engagement is measured by a students’ levels of interest, enjoyment, happiness, boredom, and anxiety during academic activities (Appleton et al., 2008; Skinner et al., 2008) as well as their emotion and enjoyment toward the subjects they are taking (Sinatra et al., 2015). In the context outside the classroom, emotional engagement is represented by the student’s feeling toward the broader school community. Those feelings can be attachment, belonging, identification, and relatedness to the school overall (Finn & Voelkl, 1993; Reschly & Christenson, 2012).
Research has found that students’ feelings and emotional attachments are very important during their life at school. For example, students who are connected and attached to peers, teachers, and the school are more motivated to pursue and complete academic tasks than those who lack similar school attachments (Voelkl, 2012). More significantly, students without such emotional connections are found to be less engaged, and may experience negative behavior such as disciplinary problems and dropping out (Finn, 1989; Finn & Voelkl, 1993).
Cognitive engagement
Cognitive engagement draws on the idea of mental investment. It incorporates thoughtfulness and willingness to put in the effort needed to understand and digest complex ideas and acquire difficult skills (Fredricks et al., 2004).
In comparison to the above mentioned two types of engagement, cognitive engagement seems more difficult to precisely define. Cognitive engagement is not directly observable and its definition may overlap with emotional and behavioral engagement sometimes (Sinatra et al., 2015).
Cognitive engagement can be seen as motivational or self-regulatory effort initiating or sustaining learning actions (Ainley, 2012), setting learning goals, or persisting on challenging tasks (Zimmerman, 1990). In the in-class context, cognitive engagement refers to investment of effort when a student tries to understand complex definitions, spends time and effort to find appropriate learning strategies, and recall previous experiences to solve new problems. However, when conceptualized this way, certain dimensions of cognitive engagement (e.g., effort) overlap with dimensions of behavioral engagement (Sinatra et al., 2015).
Outside of the classroom context, cognitive engagement is operationalized in the motivation and planning to join extra-curricular activities, the cognitive valuing of the school, as well as understanding the importance of education at school and taking it seriously. Cognitive engagement is also seen as an internal process, such as students’ self-reflection and evaluation of effectiveness when participating in school-based activities (Cleary & Zimmerman, 2012).
Cognitive engagement therefore should be carefully measured, otherwise it may get confused with other aspects of SE (Sinatra et al., 2015).
Agentic engagement
While emotional, cognitive, and behavioral engagement are seen as students’ reactions to the structure of learning environments, described as compliance-oriented by Crick (2012), agentic engagement is about students’ active contribution to the flow of education (Reeve & Tseng, 2011). However, agentic engagement has not been studied as widely as the other components of SE (Bond, 2020). While Reeve and Tseng (2011) did produce an instrument to measure in-class agentic engagement for high school students, Montenegro (2019) pointed out that there is still no scale to measure agentic engagement in the classroom in the context of HE.
This element of SE is also supported by other researchers as it represents a proactive form of engagement (Brooks et al., 2012) rather than the reactive behavioral engagement. That is, while behavioral engagement happens where the student responds to the learning environment, agentic engagement occurs when a student actively contributes by shaping the educational environment.
Agentic engagement has been found to be a statistically distinct element of SE that relates to the other three dimensions (emotional, cognitive, behavioral/participatory) but independently predicts achievement and motivation (Jang et al., 2016; Reeve, 2012; Reeve & Tseng, 2011), and reflects a good relationship with peers and a negative association with psychological distress (Mameli & Passini, 2017).
Based on the original definition in the classroom context, agentic engagement can be operationalized as having a proactive approach to learning, giving suggestions to improve a lesson or class environment (Reeve & Tseng, 2011), self-directing one’s learning (Cleary & Zimmerman, 2012; Reeve, 2012), actively expressing one’s thoughts, opinions, and interests during activity, or when one engages communally, collectively, and critically with others (Lawson & Lawson, 2013).
While agentic engagement has been widely accepted as a construct of SE within the in-class context, few research studies have examined it in the wider educational context (Bond, 2020). However, one can visualize that, in the out-of-class (campus) context, a student who is agentically engaged will act to contribute to the academic community, willingly propose ideas or solutions to improve the campus environment, and involve themselves in extra-curricular activities.
Agentic engagement is the only proactive form of engagement as it is defined as student-initiated activity, which is different from the other three reactive forms of engagement (Montenegro, 2019). As agentic engagement manifests through student behavior, confusion with (reactive) behavioral engagement is possible. Therefore, in order to avoid any possible confusion between behavioral engagement and agentic engagement in this paper, behavioral engagement will be referred to as participatory engagement as suggested by Finn (1989) and Yazzie-Mintz (2007).
By unifying the terminology of these three components across the two contexts (in-class and out-of-class), we start to see the foundation of a unifying model. While this model does not include academic engagement, which encompasses SE in the context of students’ independent academic studies, we believe there is value in the unification of the two contexts included herein.
Methods
A mixed-methods approach was taken to explore the possibility of unifying these two SE contexts as a mixed-methods approach can provide greater insights (Creswell & Creswell, 2018, p. 294). This section will explain the methods in greater detail.
The literature review provides an overview of SE with different structures, approaches, and dimensions. What is widely agreed upon is that SE is a meta-construct that can be analyzed from different perspectives based on the researchers’ purpose. While there are four components of SE including emotional, cognitive, participatory, and agentic engagement, agentic engagement has only been considered in the in-class context. It has not been examined and measured in the out-of-class context. This research will explore integrating agentic engagement into a unifying model which includes all four dimensions of SE across the two contexts. Our proposed model is presented in Figure 3.
Proposed unifying model of student engagement.
To confirm the presence of agentic engagement in the proposed model, the authors followed the first six steps in the seven-step procedure by Hinkin et al. (1997) to develop a comprehensive SE model with appropriate instrumentation to be validated in the Vietnamese HE context. These six steps are detailed as follows.
Step 1: Item Generation
Step 1.1: Review existing instruments
A thorough review of the literature produced 14 existing instruments used to measure SE. Four of the existing questionnaires were developed for undergraduate use but none specifically for students in business and management. Eight of the 14 questionnaires used included both in-class and out-of-class SE. The other six considered only the in-class context, meaning that none explored solely out-of-class SE, which is the focus of this paper.
The Student Engagement Scale (Gunuc & Kuzu, 2015) was selected as the foundation for the new measurement instrument due to its theoretical construction in HE in the Global South. The items include: Valuing (five items) used for out-of-class cognitive, Sense of belonging (10 items) used for out-of-class emotional, and Participation (five items) used for out-of-class participatory engagement (Figure 4).
Student out-of-class engagement structure.
There is still one component in the proposed model of SE without measurement items, agentic engagement. As past research has provided reliability and validity evidence for in-class agentic engagement, this pilot test is to confirm this sub-construct in a new context (out-of-class) with students in the specific discipline of business and management. Those items will be developed in the next step of item generation through qualitative research.
All these items were translated into Vietnamese by an expert Vietnamese translator working in the field of education. After that, the translated version was reviewed by another Vietnamese university lecturer who is proficient in both English and Vietnamese. Minor adjustments were made to the translated list of items.
Step 1.2: Item generation through qualitative research
The theoretical unifying structure of SE used for this research has one new element, out-of-class agentic (OA) engagement, that has never been measured (with the exception of Finn’s (1989) related work). Some initial measurement items were formed based on the descriptions of the taxonomy from Finn (1989), such as a student’s active participation in extra-curricular activities and school governance.
Qualitative research was then undertaken to explore possible additional items in measuring the sub-construct of OA in the context of business and management students in Vietnam. The methods used for collecting qualitative data included in-depth interviews, expert interviews, and a focus group. During expert interviews, beside the exploration purpose, the face validity of the constructs as well as instrument measurement of SE were reviewed.
Selection of informants
Informants of this qualitative research were undergraduate students and lecturers in the field of business and management, as well as experts in the field of student behavior and student psychology.
The first group of students were introduced to the first author through the connection of the class facilitators. However, after the first two in-depth interviews, the interviewer decided to invite eight students to join a focus group. Seven students actually showed up and participated in the focus group, one was absent due to personal reasons.
All interviews and the focus group were hosted in the interviewer’s office with a friendly decor to ensure a quiet environment to facilitate audio recording. The students were informed of the choice of room in advance and all students consented. The length of student interviews were from 30 to 45 minutes and that of the focus group was 1 hour and 45 minutes.
Lecturers are those who have the most frequent interactions with students; therefore they have certain understandings and insights into student engagement. Four lecturers in business and management at a large public university were invited to in-depth interviews. The lecturers were chosen using convenience sampling and each had a minimum of 10 years of experience teaching undergraduate students in business and management programs. All interviews with lecturers were held in person in the aforementioned office, for the same reason as with students. The length of each interview with lecturers was about 1 hour.
In order to get expert opinions about student behavior, three experts in the field of psychology with a research focus on undergraduate students were invited to join individual in-depth interviews. Three researchers with PhDs in educational psychology were included, each from a different public university’s department of pedagogy. Due to the constraints of the experts’ availability and pandemic-related issues, all expert interviews were held via video call. A brief containing the research objectives and interview purpose was sent to each expert in advance for their preparation. The length of interview with each expert was between 45 and 60 minutes.
Implementation of data collection
At the beginning of each interview and focus group, the interviewer spent approximately five minutes explaining the purpose of the research and the importance of the informant’s participation and thanked them for their presence. The interviewer also explained about the confidentiality policy and informant’s right to stop providing information at any time. Informants showed their understanding by signing a consent form. Regarding the online interviews, consent forms were sent to interviewees in advance via email and all interviewee showed their consent via email or via oral acceptance at the beginning of their interview session.
All participants were informed that the interviews and focus group would be audio recorded and they all agreed. All audio recordings were transcribed within one week from the time of recording. Additional hand-written notes made during interviews supported the transcription process.
The interviews and focus group were all semi-structured asking participants about their perspectives on SE, along with how to measure it, both in-class and out-of-class. The topic of SE is not widely discussed in Vietnam so the early questions included “When you think of a student being engaged or connected to their school and studies, what does that mean to you?” Another example is “How can you tell if one student is engaged but another is not?” Follow up questions were asked to delve deeper into the participants’ conceptual understanding of SE with questions such as, “If an engaged student pays attention in class, what exactly does that look like?” As the meeting progressed, we explored exactly what questions should be asked to determine how engaged a student is.
The context was new both because it takes place in Vietnam (where the concept of SE is quite new) and because the subjects of the pilot questionnaire would be business and management students (a group of students not targeted by any existing measurement instruments). Accordingly, we discussed not only in-class and out-of-class contexts but also all aspects so that we could compare what participants thought against what is found in literature to look for conflicts, support, or simply additional concepts in order to produce the richest possible measurement instrument.
Data processing
After being transcribed into text files, the transcripts were then marked up, compared between documents, and compared with the proposed measurement instrument of SE.
As the proposed measurement instrument was extracted from various articles in the literature from other countries, with different cultures, there were three possibilities when comparing the instrument with the findings of the respondents. The first possibility is that the instrument matches with what has been reported in the interview transcripts. This situation will help to confirm the face validity of the instrument. The second is that one or more items in the proposed instrument does not appear in the interview transcripts. In this case, these items were highlighted for further face validity checks to ensure they belong in the instrument. The final, and most interesting, findings of these interviews would be anything in the transcripts which was not in the proposed measurement instrument. Such findings may serve as new items to measure student engagement in the current research context.
One key question of this research is to see if OA SE is a distinct factor of SE, which would allow for a model to be created that would unify in-class and out-of-class SE. However, because SE has not been measured in the current contexts (Vietnam and business and management HE students), the goal is to produce a complete measurement instrument to measure SE. This would be done by adapting existing instruments for in-class SE to the local context, by adapting existing instruments for OP (out-of-class participative), OC (cognitive), and OE (emotional), and by creating items to measure OA (agentic). Therefore, the interviews did ask about all aspects of SE, although there was a heavier focus on OA SE.
Findings from interviews
Most of the interviews with students showed that they were not familiar with the concept of SE. However, they had no difficulty in understanding or answering the interview questions. Lecturers were more familiar with the concept and had favorable attitudes toward it.
- I think the phenomenon that students love their school and have positive thoughts, attitudes, and activities toward the school, is a phenomenon that has existed for a long time, possibly different in their hidden or explicit form of demonstration. (Female, 25 years of teaching experience)
Below are the main findings extracted from the interviews:
Interactions between students and their friends, faculty, and school on social media was mentioned as an aspect of engagement. Those interactions also demonstrate various components and levels of engagement.
This has been repeatedly found in student interviews:
- I will share on Facebook interesting events of my university. I think that is how I make people know more about my study program. (Female, 2nd year student)
- I interact with my school in both direct communication and online interaction on social media. My school’s [fan]page is very active. I just feel that if we made more contacts, such as sharing, texting, tagging, more people will know about my school, and it will help boost awareness of my school and its reputation. (Female, 3rd year student)
- I chat with my friends in class, hang out with them after school time, and comment on each other’s Facebook. (Female, 2nd year student)
- I often share enrollment information of my university [on my Facebook], and if anyone comes in to ask I am willing to advise. (Male, 3rd year student)
Lecturers also suggested that students interact and engage with their school via social media:
- The bonding to the school can also be seen in actions such as protecting and showing affection toward the school or spreading awareness about the school’s brand name. It means it is outside the scope of the classroom. This made me recall a recent event at my university. My university initiated some volunteer activities…in a very short time, and then a few criticized them on social media about certain small details during the organization of the event. Later on, I learned that some of our students got together and spoke up to protect the image and reputation of the university… I think such an initiation-taking activity [of students] should be seen as a high-level of student interaction with their school. (Female, 20 years of teaching experience)
- On student forums, when someone asks about which university to study…, our students will answer and share related information such as information resources, information sessions… (Male, 14 years of teaching experience)
From expert interviews:
- I find students’ interactions on social media nowadays is also a very important aspect. (Female, 16 years of research experience)
- The scope of student engagement expands much broader now, not just among students in the classroom, but through social media like Facebook. (Male, 10 years of research experience)
- Now when it comes to students, it definitely goes with the foundation of social media. It’s their daily life as they spend thousands of hours there, display all different aspects of their life, including their engagement with school. (Male, 15 years of teaching and research experience)
This common finding from different interviews matches recent studies about digital technology as an integral and central aspect of higher education that significantly affects all aspects of the student experience (Barak, 2018; Henderson et al., 2017; Selwyn, 2016). This is also emphasized when students are referred to as “learners of the 21st century” (Gunuc & Kuzu, 2014) or “digital natives” (Prensky, 2001) and that digital technology and social media have become an indispensable part of their daily lives (Yu et al., 2010).
Among those opinions, there are different levels of demonstration of student engagement via social media, from (1) commenting on Facebook, (2) sharing events and spreading awareness, to (3) giving advice, and (4) protecting the reputation of the school. Based on the engagement level, those items will be put into two categories. The first two items belong to OP SE, and the last two belong to OA SE.
2. Active participation in extracurricular activities can be displayed in the form of membership in club management boards and event organizers.
- I think I am an active student. I participate in all the extracurricular activities of the program. I am a member of some club management boards and event organizers…and I have mentored freshmen to get familiar with the program in the last two years. (Male, 3rd year student)
This is a high level of engagement that can help measure OA SE, as only a few students can become members of club management boards. Such participation shows the proactiveness of students in their interaction with the campus environment. It also fits well with the highest level of participation in the taxonomy proposed by Finn (1989) in the form of school governance, involvement in club management boards, student associations, and other relevant student organization bodies.
In the end, the face validity of the construct, as well as the instrumentation of SE, were reviewed. One item from the Finn (1989) taxonomy and five new items were proposed to measure OA SE. Additionally, some items were added to the OP SE measure to reflect the interactions of students in the digital era.
Step 2: Content Adequacy
Step 2.1: Choosing a suitable scale
In student self-report measurement instruments, a seven-level Likert Scale is typically used, including in large surveys. Examples include the Motivation and Engagement Scale—University/College (MES-UC) (Martin, 2009) and the Student Satisfaction Survey used at Oxford Brooks University (Ghori, 2016), the University of Central England in Birmingham (Kane et al., 2008), and most other universities in the UK (Williams & Cappuccini-Ansfield, 2007).
Step 2.2: Data collection method
Among the methods used to measure SE, self-report measures are the most common (Fredricks & McColskey, 2012) and were chosen for that reason.
Step 2.3: Final format of the pilot questionnaire
The questionnaire used in this pilot test version has three parts for the main content. The first section of the questionnaire is an introduction providing an explanation of the research purpose, type of informants and information collection, confidentiality policy, statement of consent, and instructions on completing the questionnaire. At the end of this introduction, respondents will be asked to complete a “consent statement” where they show their voluntary consent of taking part in the survey.
The second section contains student engagement items, with eight items for OP engagement, 10 items for OE engagement, six items for OC engagement, and six items for OA engagement. The measurement of this part is a seven-point Likert scale, with which students show their level of agreement to each of the statements.
The final section was about the informant’s demographic and personal information, such as age, gender, year of study, major, number of subjects they are taking, and university name.
As the pilot version was long and took significant time for respondents to complete, no reversed item were designed into the form, so respondents could focus on the main content of the pilot test.
Step 2.4: Final instrument format review
In this step, the pilot questionnaire was put through an instrument content review (and pretest) to make sure that the Vietnamese version was clearly worded and ready to use. As there are two versions of the questionnaire, online and paper, the online version was sent to three lecturers and two students, the paper version was sent to one lecturer and four students.
Informants were told about the purpose of the research and asked to fill in the questionnaire, to record their completion time, and prompted to provide feedback on the wording or to raise any concerns with the Vietnamese translation. Most of the feedback was about the length of the questionnaire with a recommendation to give enough time for respondents to complete it.
Some minor changes were then made based on comments about meaning and wording consistency, the question arrangement of the paper version, as well as the branching setting of the online version.
Step 3: Questionnaire Administration
Step 3.1: Administration
The final list of items was put in two different versions: online and printed. The online questionnaire was sent to business students at a large national university in Vietnam. Only those who explicitly clicked to accept the consent statement proceeded to the main questionnaire.
The printed version, given to business students at a different large national university in Vietnam, had a similar format and content. However, in order to save time for the participants, the research purpose, privacy policy, and completion instructions were orally communicated.
Step 3.2: Determining sample size
In order to appropriately conduct the subsequent analysis, data must be collected from an adequate sample size (Hinkin et al., 1997). Many researchers agree that the number of variables or items to be assessed will determine the sample size needed to reach robust results. That means that as the number of items increases, the sample size should increase accordingly (Hinkin et al., 1997). Some earlier recommendations for item-to-response ratios ranged from 1:4 (Rummel, 1970, cited by Hinkin et al., 1997) to 1:10 (Schwab, 1980, cited by Hinkin et al., 1997) for each set of scales to be factor analyzed. Hair et al. (2009) emphasized that sample size affects results and suggested a desired ratio of 5 to 10 observations per variable. As there are 30 variables being analyzed for out-of-class SE, the range would be 150 to 300 responses. Similarly, general guidelines from Worthington and Whittaker (2006) state that a minimum of 300 respondents is adequate in most situations.
Step 4: Factor Analysis
Exploratory factor analysis (EFA) was performed to explore the components of the SE construct. The extraction method used was principal component analysis (eigenvalue greater than 1, Varimax rotation, with the absolute value of small coefficients to be suppressed being 0.45).
Confirmatory factor analysis (CFA) was run using AMOS Graphic version 20 to confirm the proposed model of SE.
Step 5: Internal Consistency
Cronbach’s alpha was used to verify internal consistency as it is the most frequently used measure for this purpose (Cho & Kim, 2015). While some researchers are concerned that alpha includes a non-trivial bias (Green & Yang, 2009) compared to structural equation modeling, a thorough meta-analysis exploring the presence of any such bias in practice shows that the bias is, practically speaking, insignificant (Peterson & Kim, 2013).
Step 6: Construct Validity
Construct validity was handled through checking if there is convergence (that all within-construct correlations are both high and of approximately the same magnitude) and differentiation (cross-correlations are high, uniform, and lower than the within-construct correlations) as recommended by Fornell and Larcker (1981).
Results and Discussion
As earlier research has provided reliability and validity evidence for in-class agentic engagement, the purpose of this pilot test is to confirm the sub-construct in a new context, with students in the specific discipline of business and management. This approach also fits with the difference in the nature of the contexts used to identify measurements for each set: in-class and out-of-class engagement.
There were 61 online and 264 printed responses collected, among which 312 responses were valid. The response rate was 58.7% for the printed version. Invalid responses included blank or half-blank responses, responses completed with a single choice across all questions, or completed with a purposefully ordered pattern (e.g., 1-2-3-4-5-6-7-6-5-4-3-2-1).
As the online and printed questionnaires were distributed through separate channels, there was no duplication of responses. The total sample size of 312 was satisfactory to the target range of 150 to 300 (Table 1).
Sample Information.
These gender ratios are typical for undergraduate students in business and management where female students are in the majority. The sample was fairly balanced across the years of study.
Identify Outliers
In order to prevent outliers from inappropriately influencing the analysis, it is important that any outliers are identified and eliminated (Hair, 2019). There were five observations found as outliers, based on Mahalanobis distance, which were removed from the data set, resulting in a final data set of 307 observations.
Factor Analysis
Exploratory factor analysis
An exploratory factor analysis (EFA) was used to analyze the components of the SE construct. EFA results for in-class SE confirmed the four sub-constructs of IE (in-class emotional), IC (in-class cognitive), IP (in-class participative), and IA (in-class agentic) engagement as found in earlier literature (Mameli & Passini, 2017; Montenegro, 2019).
Regarding out-of-class SE, EFA was used to explore the factor dimensions as well as to reduce the set of observed variables to a smaller set (Hinkin et al., 1997).
The initial results of the EFA for out-of-class engagement are as follows:
While there is debate over the reliability of the Guttman-Kaiser criterion (using eigenvalues above one) it has been found that the criterion works well when the number of factors is much less than the number of variables and communality is high (Yeomans & Golder, 1982). Still, a scree plot was used to visually confirm the optimum number of factors to be extracted (Hair et al., 2009). The point at which the curve begins to straighten out is considered the maximum number of factors to extract. As the slope of the scree plot is significantly reduced to make the curve flat after the fourth point (matching the earlier results), four factors should be extracted as the optimal result of this EFA. However, to be certain we must thoroughly explore the possibilities of either three or four factors and we must explore each of these options before deleting any items (Worthington & Whittaker, 2006) and performing CFA (Figure 5, Tables 2 and 3).
Scree plot.
KMO and Bartlett’s Test.
Note. The KMO value is 0.927 (>0.7) and the Bartlett’s test of sphericity is significant at p = .001.
Total Variance Explained and Pattern Matrix.
Note. Extraction method: principal component analysis.
The first step is performing EFA imposing three factors. This resulted in 56.5% of total variance explained. However, there were many items showing high cross-loadings. After removing items with high cross-loadings, the following matrix is produced (Table 4).
Rotated Component Matrix for Three Factors.
Note. Extraction method: principal component analysis. Rotation method: varimax with Kaiser normalization.
Rotation converged in five iterations.
Here we see some items from OP load on the same factor as some items from OC, which does not make sense conceptually. This leads us to generally rule out three factors.
The next step is performing EFA again imposing four factors. The initial result was an improved 59.0% of total variance explained but there were still some cross-loadings and loadings on the wrong conceptual component. After removing those items, the variance explained improved to 66.9%, an improvement over Gunuc and Kuzu’s (2015) model which explained only 59%. The final matrix is as follows (Table 5):
Rotated Component Matrix for Four Factors.
Note. Extraction method: principal component analysis. Rotation method: varimax with Kaiser normalization.
Rotation converged in five iterations.
Confirmatory factor analysis
In order to run a confirmatory factor analysis (CFA), as it is very sensitive to missing data, a function to impute missing data was applied to the data set. After, the fitness indices for the model for out-of-class SE were compared with norms resulting in the following evaluation:
Among these six indices, three (CFI, CMIN/DF, and RMSEA) were good, one was acceptable (SRMR), and two (GFI and 90% confidence interval for RMSEA) were only marginally good when compared with standard cutoff values (Table 6).
Goodness of Fit.
There are a few points to consider here. First is Moss’s (2009) recommendation that cutoff values only be used as guidelines and not hard and fast rules. Second is that others (Sharma et al., 2005) say that RMSEA is preferable over GFI. The third consideration is that, while GFI is below the standard of .90, it is very sensitive to small sample sizes (Fan et al., 1999), like the size we see in this pilot study. We can also see this in the work of Gunuc and Kuzu (2015), which formed an important foundation for this pilot study. Their GFI was only .80 with a larger sample (n = 464). Although, that study did have a better CFI (.96) and SRMR (.62). This can be compared with a study by Mameli and Passini (2017) which reported a CFI of .92 and RMSEA of .47. Unfortunately, given Mameli and Passini’s large sample size (n = 1,210), they did not report GFI.
With such a problem of fitting engagement data to a model, Samuelsen (2012) noted that some factors may not fit the data as neatly as we want or even fit at all. Betts (2012) also identified this challenge when stating that SE’s indicators often load as different factors or constructs when doing survey research.
Importantly, all six indices had better values for four factors than for three, providing reasonable evidence that there should be four factors and that OA is a distinct factor of SE. This helps to answer the question from Bond (2020) whether student engagement has three or four factors.
In full light of these facts, the confirmatory factor analysis of this model was considered moderately satisfactory and acceptable for a pilot test.
CFA was run to check the proposed unifying model. The component matrix of in-class and out-of-class engagement were put into Amos Graphics version 20. The result can be seen in Figure 6.
Standardized path diagram (4-factors, 2-dimensions).
Internal consistency assessment
In this step, the reliability of the scale will be tested based on Cronbach’s alpha (Table 7).
Item-Total Statistics.
Alphas of OE, OC, and OA were all above .8. OP was slightly below .7, which might be concerning to some; however, Vaske (2008) stated the general rule that anything above .65 was considered adequate and Hair (2019, p. 161) considered .6 to be acceptable during exploratory research like the current study. Additionally, the number of items in OP is very low (only three items) and alpha is very sensitive to this issue (Vaske et al., 2017). All Alpha if Deleted were below Cronbach’s alpha (except the case of OC3 where alpha may slightly be improved).
Additionally, all Corrected Item-Total Correlation >.4, meaning they are highly interrelated and likely to measure the same construct. Therefore, the items used to measure the components of SE can be considered reliable.
Construct Validity
Convergence implies that all within-construct correlations are (1) high and (2) of approximately the same magnitude. Differentiation is satisfied if the cross-correlations are (1) high, (2) uniform, and (3) lower than the within-construct correlations (Fornell & Larcker, 1981).
Below are the Pearson correlations of all components of SE. The correlation of all pairs of components are statistically significant at the .001 level. All correlations are higher than .3, and most of them are higher than .5.
The criterion-related validity of the SE model is satisfactory (Table 8).
Correlations.
Correlation is significant at the .01 level (2-tailed).
The above procedure shows that OA SE fits into the unifying model of SE as a separate and valid component. The EFA confirmed this factor based on the correlations among variables, while CFA presented a moderate fitness of this unifying model. Other tests of reliability and construct validity were carried out and all showed good results. Therefore, after six steps of developing the measurement instrument of out-of-class SE, which includes OA, the current instrument should be considered successful. However, further validation should be implemented to confirm the instrument in different contexts and communities.
Limitations
During the process of developing the measurement instrument for OA engagement, while the pilot test produced promising results with partial satisfaction of the CFA of the model, certain problems regarding the fitness of the model were encountered. These issues have been noted by other researchers (Betts, 2012; Samuelsen, 2012). Therefore, it is important to follow up this pilot study with a more complete investigation to fully validate the instrument.
Conclusion
After an extensive review of existing models of, and instruments to measure, student engagement (SE), its dimensions and components, mixed-methods research was conducted to explore the potential of unifying the way SE is modeled both inside and outside the classroom. Interviews and focus groups indicated this was possible by revising the existing models of campus engagement.
The model proposed in this paper includes the following components for both in-class and out-of-class (or campus) engagement: emotional, cognitive, participatory, and agentic. While in-class engagement had been previously modeled this way, existing models of out-of-class SE do not include agentic engagement. Therefore, the contribution of this research is to add agentic engagement to out-of-class SE, thus creating a common way to model SE inside or outside of the classroom.
EFA confirmed the four components and CFA gave satisfactory results for this pilot study. The instrument created to measure out-of-class SE is internally consistent when analyzed using Cronbach’s alpha.
The proposed unifying model provides a structure to represent these two contexts of SE. Since the authors preserved the most popular components of emotional, cognitive, and participatory engagement in the unifying model, researchers should be able to extend their investigations using this new model where appropriate.
As this model integrates the in-class and out-of-class contexts, it highlights the possible differences of the contextual environment regarding the components of SE. This is essential for researchers to analyze the impacts of SE and propose relevant interventions to drive desirable outcomes in higher education.
Researchers can now explore SE vertically or horizontally to meet their research purpose. By approaching the concept vertically, each of the four components of emotional, cognitive, participatory, and agentic engagement can be analyzed separately or together in any order. By approaching this concept horizontally, researchers can separate between the two contexts of the in-class and out-of-class environments. While this model does not include academic engagement, which encompasses SE in the context of students’ independent learning, it is still believed that there is great value in unifying the way in-class and out-of-class SE are structured.
This unifying structure of SE has many advantages and allows measurement of SE from a broad point of view as well as understanding SE in its complex environment. Thus, this proposed model improves the analytical options available to researchers of SE.
While not generalizable, we believe that there is value in the unification of the two contexts included herein and that there are underlying themes from which other researchers can learn.
Footnotes
Appendix
Student Engagement Questionnaire Wording.
| Code | Vietnamese version (revised) | English items | Source |
|---|---|---|---|
| In-class participative engagement | |||
| IP1 | Tôi là một sinh viên tích cực trong lớp. | I am an active student in class | Newmann et al. (1992) |
| IP7 | Tôi cố gắng hết sức để hoàn thành trách nhiệm của mình trong công việc nhóm. | I try to do my best regarding my responsibilities in group work | Newmann et al. (1992) |
| IP9 | Tôi tham gia tích cực trong thảo luận nhóm. | I contribute to class discussions | Gonyea et al. (2003) |
| IP10 | Tôi tham gia trong các bài tập lớn, dự án, hoặc thuyết trình với các sinh viên khác. | I work on a class assignment, project, or presentation with other students | Gonyea et al. (2003) |
| In-class emotional engagement | |||
| IE3 | Tôi tôn trọng bạn bè trong lớp. | I respect my classmates | Voelkl (1996) |
| IE4 | Các lớp học của tôi rất thoải mái. | My classes are entertaining | Fredricks et al. (2003) |
| IE5 | Tôi tôn trọng các thầy cô giáo của mình. | I respect my teachers | Voelkl (1996) |
| In-class cognitive engagement | |||
| IC1 | Tôi tự tạo động lực học tập cho bản thân. | I motivate myself to learn | Lehr et al. (2004) |
| IC2 | Tôi đặt ra các mục tiêu học tập của riêng mình. | I determine my own learning goals | Lehr et al. (2004) |
| IC3 | Tôi nỗ lực hết mình trong giờ học. | I try to do my best during classes | Gunuc and Kuzu (2015) |
| IC4 | Bên cạnh việc làm bài tập, tôi nghiên cứu thêm các nội dung liên quan đến bài học của mình. | Besides doing my lessons, I further study for my lessons | Fredricks et al. (2003) |
| IC8 | Tôi cố gắng làm bài tập về nhà theo cách tốt nhất. | I try to do my homework in the best way | Newmann et al. (1992) |
| IC9 | Tôi thích đón nhận những thử thách trong học tập. | I enjoy intellectual difficulties I encounter while learning | Appleton et al. (2006) |
| IC10 | Tôi dành đủ thời gian và nỗ lực để học tập. | I spend enough time and make enough effort to learn | Newmann et al. (1992) |
| In-class agentic engagement | |||
| IA1 | Trong giờ học, tôi đặt câu hỏi. | During class, I ask questions. | Reeve and Tseng (2011) |
| IA2 | Tôi nói với thầy cô về những gì tôi thích và không thích về giờ học. | I tell the teacher what I like and what I don’t like. | Reeve and Tseng (2011) |
| IA3 | Tôi nói cho thầy cô biết những gì làm tôi thấy hứng thú trong giờ học. | I let my teacher know what I’m interested in. | Reeve and Tseng (2011) |
| IA4 | Trong giờ học, tôi thể hiện các mong muốn và ý kiến của mình. | During class, I express my preferences and opinions. | Reeve and Tseng (2011) |
| IA5 | Tôi chủ động trao đổi với thầy cô khi thấy các vấn đề của lớp học. | I actively discuss with my teachers when there is concerns about the class. | Interviews |
| IA6 | Tôi đưa ra các đề xuất để làm cho lớp học tích cực hơn. | I offer suggestions about how to make the class better. | Reeve and Tseng (2011) |
| Out-of-class participative engagement | |||
| OP4 | Tôi được hưởng lợi ích từ các tiện ích sẵn có trong trường (căn-tin, thư viện, khu thể thao…) | I benefit from the facilities in campus (canteen, library, sports arenas, and so on) | Interviews |
| OP5 | Tôi tuân thủ các quy định trong khuôn viên trường. | I follow campus rules | Fredricks et al. (2003) |
| OP6 | Tôi tham gia các hoạt động ở trường một cách nghiêm túc. | I participate seriously | Finn (1993) |
| Out-of-class emotional engagement | |||
| OE1 | Tôi thích đến trường. | I look forward to going to campus | Willms (2003) |
| OE2 | Tôi cảm thấy mình là một phần của nhà trường. | I feel myself as a part of the campus | Finn (1993) |
| OE3 | Khuôn viên trường là một nơi thật thoải mái. | Campus is an entertaining place | Sutherland (2010) |
| OE4 | Tôi thích các hoạt động được tổ chức trong nhà trường. | I enjoy the activities carried out in campus | Voelkl (1996) |
| OE5 | Tôi cảm thấy hạnh phúc khi ở trường. | I feel happy in campus | Fredricks et al. (2003) |
| OE6 | Tôi thích dành thời gian ở trường. | I like spending time in campus | Fredricks et al. (2003) |
| Out-of-class cognitive engagement | |||
| OC1 | Tôi tin tưởng trường đại học mang lại giá trị và lợi ích cho tôi. | I believe university is beneficial for me | Finn (1993) |
| OC2 | Trường đại học có ý nghĩa quan trọng với cuộc sống của tôi. | University is of great importance in my life | Finn (1993) |
| OC3 | Tôi nghĩ rằng các quy định ở trường đại học là công bằng với tất cả sinh viên. | I think the rules at university are fair for everybody | Appleton et al. (2006) |
| OC4 | Tôi cố gắng không làm hỏng bất cứ thứ gì thuộc về nhà trường. | I try not to damage anything that belongs to the university | Gunuc and Kuzu (2015) |
| OC5 | Tôi hiểu tầm quan trọng của giáo dục đại học. | I give importance to university education and take it seriously | Finn (1993) |
| Out-of-class agentic engagement | |||
| OA1 | Tôi đóng góp tích cực cho các hoạt động ngoại khóa của lớp và/hoặc của trường. | I actively contribute to extra-curriculum activities of my class and/or school. | Interviews |
| OA2 | Tôi là thành viên chủ chốt của ít nhất 1 câu lạc bộ/hội/nhóm của trường. | I am a core member of at least one club/association/group of my school. | Interviews |
| OA3 | Tôi tham gia điều hành ít nhất 1 câu lạc bộ/hội/nhóm của trường. | Involved in the school governance or membership of student organizations. | Finn (1989) |
| OA4 | Tôi đóng góp tích cực trong các hoạt động cho Câu lạc bộ mà mình tham gia. | I actively contribute to my club’s activities. | Interviews |
| OA5 | Tôi chia sẻ các phong trào/sự kiện của Trường mình trên các mạng xã hội. | I share campaigns/events of my school on social networks. | Interviews |
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by National Economics University, Hanoi, Vietnam.
