Sage Journals: Discover world-class research

Abstract

Student engagement, while widely recognized as both important to outcomes of higher education institutions and critical to student learning, is still missing a unifying model addressing students both inside and outside of the classroom. This mixed-methods paper will explore the existing models of student engagement and propose a model which better supports analysis from multiple perspectives. This model was validated within the context of Vietnamese higher education. The results indicate that student engagement, both in-class and out-of-class, can be modeled through the four sub-components of emotional, cognitive, participatory, and agentic engagement. Having these common elements allows analysis vertically, through these four dimensions, or horizontally, across two contexts, providing greater flexibility for future investigations.

Keywords

student engagement campus engagement agentic engagement unifying model higher education Vietnam

Introduction

The topic of student engagement (SE) emerged near the end of the last century and has quickly become the latest concern for both researchers and practitioners in promoting teaching and learning in higher education (HE). In a broad sense, the concept of SE, without the name, has been around for quite some time. One of the very first mentions is the pioneering work of Ralph Tyler from the 1930s, which showed the positive relationship between time-on-task and learning (Merwin, 1969, cited by Kuh, 2009). Bloom (1974) confirmed similar results.

As the concept of SE developed, variations emerged with different definitions that contain similar, though not always consistent, components (Appleton et al., 2008; Bond, 2020; Fredricks et al., 2004; Fredricks & McColskey, 2012; Furlong et al., 2003). Throughout the literature, a common list of sub-components can be found as emotional, cognitive, and behavioral/participatory engagement. While certain definitions explicitly focus on the in-class context, some incorporate both in-class and out-of-class/campus environments.

In order to focus on the level of individual engagement and to cover both in-class and out-of-class contexts, this research is based on the concept proposed by Kuh et al. (2007), in which SE is conceptualized as students’ involvement, inside and outside of the classroom, in effective practices of learning leading to measurable outcomes.

Over the last decade, SE has stayed on top of meeting agendas and conference themes around the world (Leach, 2012; V. Trowler, 2010). More specifically, P. Trowler and Trowler (2010) concluded that there is no longer a question as to the value SE with supporting evidence from the (US) National Institute of Education’s Involvement in Learning Report. Kahu (2013, p. 758) also agreed that “Student engagement is widely recognized as an important influence on achievement and learning in higher education and as such is being widely theorized and researched.”

The reason why SE has been of increasing interest to researchers in higher education is its association with students’ academic achievements, school completion, well-being (Kahu, 2013), as well as other long-term outcomes, such as work success (Christenson et al., 2012). However, there is little consensus on the conceptualization of SE among researchers (Bond, 2020; Fredricks et al., 2004; Furlong et al., 2003), except that SE is a multifaceted, multidimensional meta-construct (Burch et al., 2015; Fredricks et al., 2004; Kahu, 2013).

One way to view SE is with two contexts: In-class and out-of-class (or campus) engagement (Gunuc & Kuzu, 2015). Another way to view SE is that it contains the four components of emotional, cognitive, behavioral/participatory, and agentic engagement (Reeve & Tseng, 2011). However, so far agentic engagement has only been considered within the in-class context. It has never been examined and measured in the out-of-class context. If agentic engagement can be confirmed as a separate element of out-of-class SE then it would be possible to have a unifying model of SE which includes the four dimensions of SE across the two contexts. While this would not include every context, unifying these two contexts begins to move the study of SE toward a holistic model.

As a result, this study aims to integrate agentic engagement as a distinct element of out-of-class SE by developing a unifying model using the elements of emotional, cognitive, behavioral/participatory, and agentic engagement across both in-class and out-of-class engagement. The seven-step procedure by Hinkin et al. (1997) was adopted to confirm the validity of all instruments in this proposed model on a sample of 312 undergraduate students of business and management in Vietnam.

Literature Review

Dimensions of Student Engagement

There are multiple ways of defining the dimensions of SE, based on (1) contexts of student engagement; and (2) components of student engagement.

Contexts of student engagement

The first approach in identifying the dimensions of SE is to contrast the contexts of SE.

Originally, the concept of SE was primarily developed within the classroom context (Astin, 1984; Bloom, 1974; Newmann, 1992) and was said to change with educational level (Gunuc & Kuzu, 2015). When research on SE in higher education emerged, the environment of SE was extended outside the classroom, showing a more complete educational environment for students (Audas & Willms, 2001; Finn & Voelkl, 1993; S. Hu & Kuh, 2002; Voelkl, 1997).

In the following model of SE, theorized by Gunuc and Kuzu (2015), there are two main contexts of in-class and out-of-class (campus) engagement, each with three sub-components (Figure 1).

Figure 1.

Earlier theoretical model of student engagement.

In-class engagement

In the classroom context, SE is mostly seen as engagement with learning. This context is very popular for doing research with students from kindergarten through HE.

When it comes to HE, in-class engagement is described in terms of student participation in academic and educational activities, their efforts, interests, and their investment in class (Appleton et al., 2008, 2006), such as complying with classroom procedures and taking initiative in the classroom (Finn, 1989). In-class engagement therefore refers to an investment in learning, learning goals, planning, self-regulated learning, as well as using different learning strategies (Fredricks et al., 2004), their attitudes, interests, their relationships to the teacher/staff, peers, course, and content (Kahu, 2013), and their feelings of belonging to the class (Fredricks et al., 2004; Kember et al., 2001).

Disengaged students may experience boredom, anxiety, shame, sadness, or frustration (Finn, 1989), exhaustion, lack of concentration, apathy, inattention, amotivation, or may withdraw from learning tasks (Skinner & Pitzer, 2012).

What happens in the classroom, impacting SE, is vital as it actually leads to students’ learning and development (Appleton et al., 2008; Skinner & Pitzer, 2012). Therefore, SE in the classroom plays an important role contributing to the quality of students’ daily experiences while they are attending school (Skinner & Pitzer, 2012).

Out-of-class or campus engagement

In higher education, students have more opportunities to interact with the broader school community as they need to prepare for the adult environment, real-life. Therefore, SE at university is often associated with the school community (Bryson, 2010; Fullarton, 2002).

Out-of-class engagement is considered important for SE and quality learning outcomes (Gunuc & Kuzu, 2015; Hausmann et al., 2007). While Gunuc and Kuzu’s (2015) theoretical model of campus engagement included valuing, sense of belonging, and participation, these three elements map well to the widely accepted cognitive, emotional, and behavioral engagement, respectively. Valuing school is about the belief (cognition) that students have (Voelkl, 1996). The sense of belonging is about the student’s feelings (Voelkl, 1996; Willms, 2003). Participation represents student behavior (Finn, 1989).

Components of student engagement

The second approach in identifying the dimensions is to explore the various components of SE.

While earlier studies identified two components of SE, emotional and behavioral (Finn, 1993; Marks, 2000; Newmann, 1992; Willms, 2003), more recent studies have added a third component, cognitive (Appleton et al., 2008; Fredricks et al., 2004; Yazzie-Mintz, 2007) (Figure 2).

Figure 2.

Three primary dimensions of engagement.

Most research studies before 2000 adopted a one- or two-element construct approach when studying SE and rarely chose to work with all three elements at the same time (Fredricks et al., 2005).

In an effort to continue to refine and complete the concept, some recent studies have proposed a fourth component of SE. A recent model introduced by Reeve and Tseng (2011) proposed adding agentic engagement since they believed students can actively participate and contribute to the education process, thereby affecting their own learning outcomes (Lawson & Lawson, 2013). Later studies have confirmed the validity of agentic engagement as a distinct and meaningful component of in-class SE (Jang et al., 2016; Reeve, 2012; Sinatra et al., 2015).

Next, each component of SE will be briefly reviewed.

Behavioral or participatory engagement

Behavioral engagement is drawn on the idea of participation. Finn (1993) proposed that behavioral engagement can be involvement in academic and social/extracurricular activities, and it is found to be correlated with achieving positive academic outcomes and preventing drop out.

Behavioral engagement is the first component of student engagement that many instructors and researchers paid attention to because it can be observed directly. It helps instructors know how they can motivate students and affect their performance during the learning process.

In this paper, behavioral/participatory engagement can be operationalized in two main contexts, in-class and out-of-class. For the classroom context, behavioral engagement can be observed and measured by attendance, participation, preparation for class (Appleton et al., 2006), positive conduct like effort, persistence, intensity, attention, absorption, and involvement (Skinner et al., 2008). For the out-of-class context, behavioral engagement is manifested in the participation in school-related activities such as athletics and clubs (Sinatra et al., 2015).

Behavioral engagement has been found to be positively correlated with academic achievement in numerous research studies (Finn, 1993; Fullarton, 2002; Klem & Connell, 2004; Marks, 2000). This association has been explained as when students participate in their own learning and become involved in school-related activities, they often begin to find themselves identifying with school resulting in higher retention, increased completion (Finn, 1993), and potentially to positive outcomes such as higher academic achievement (Marks, 2000).

Emotional engagement

Emotional engagement involves reactions to teachers, classmates, academics, and school and is presumed to create ties to an institution and influences willingness to do the work (Finn, 1993). The term emotional engagement is used to describe students’ social, emotional, and psychological attachment to school (Finn & Voelkl, 1993; Reschly & Christenson, 2012).

In the context of the classroom, emotional engagement is measured by a students’ levels of interest, enjoyment, happiness, boredom, and anxiety during academic activities (Appleton et al., 2008; Skinner et al., 2008) as well as their emotion and enjoyment toward the subjects they are taking (Sinatra et al., 2015). In the context outside the classroom, emotional engagement is represented by the student’s feeling toward the broader school community. Those feelings can be attachment, belonging, identification, and relatedness to the school overall (Finn & Voelkl, 1993; Reschly & Christenson, 2012).

Research has found that students’ feelings and emotional attachments are very important during their life at school. For example, students who are connected and attached to peers, teachers, and the school are more motivated to pursue and complete academic tasks than those who lack similar school attachments (Voelkl, 2012). More significantly, students without such emotional connections are found to be less engaged, and may experience negative behavior such as disciplinary problems and dropping out (Finn, 1989; Finn & Voelkl, 1993).

Cognitive engagement

Cognitive engagement draws on the idea of mental investment. It incorporates thoughtfulness and willingness to put in the effort needed to understand and digest complex ideas and acquire difficult skills (Fredricks et al., 2004).

In comparison to the above mentioned two types of engagement, cognitive engagement seems more difficult to precisely define. Cognitive engagement is not directly observable and its definition may overlap with emotional and behavioral engagement sometimes (Sinatra et al., 2015).

Cognitive engagement can be seen as motivational or self-regulatory effort initiating or sustaining learning actions (Ainley, 2012), setting learning goals, or persisting on challenging tasks (Zimmerman, 1990). In the in-class context, cognitive engagement refers to investment of effort when a student tries to understand complex definitions, spends time and effort to find appropriate learning strategies, and recall previous experiences to solve new problems. However, when conceptualized this way, certain dimensions of cognitive engagement (e.g., effort) overlap with dimensions of behavioral engagement (Sinatra et al., 2015).

Outside of the classroom context, cognitive engagement is operationalized in the motivation and planning to join extra-curricular activities, the cognitive valuing of the school, as well as understanding the importance of education at school and taking it seriously. Cognitive engagement is also seen as an internal process, such as students’ self-reflection and evaluation of effectiveness when participating in school-based activities (Cleary & Zimmerman, 2012).

Cognitive engagement therefore should be carefully measured, otherwise it may get confused with other aspects of SE (Sinatra et al., 2015).

Agentic engagement

While emotional, cognitive, and behavioral engagement are seen as students’ reactions to the structure of learning environments, described as compliance-oriented by Crick (2012), agentic engagement is about students’ active contribution to the flow of education (Reeve & Tseng, 2011). However, agentic engagement has not been studied as widely as the other components of SE (Bond, 2020). While Reeve and Tseng (2011) did produce an instrument to measure in-class agentic engagement for high school students, Montenegro (2019) pointed out that there is still no scale to measure agentic engagement in the classroom in the context of HE.

This element of SE is also supported by other researchers as it represents a proactive form of engagement (Brooks et al., 2012) rather than the reactive behavioral engagement. That is, while behavioral engagement happens where the student responds to the learning environment, agentic engagement occurs when a student actively contributes by shaping the educational environment.

Agentic engagement has been found to be a statistically distinct element of SE that relates to the other three dimensions (emotional, cognitive, behavioral/participatory) but independently predicts achievement and motivation (Jang et al., 2016; Reeve, 2012; Reeve & Tseng, 2011), and reflects a good relationship with peers and a negative association with psychological distress (Mameli & Passini, 2017).

Based on the original definition in the classroom context, agentic engagement can be operationalized as having a proactive approach to learning, giving suggestions to improve a lesson or class environment (Reeve & Tseng, 2011), self-directing one’s learning (Cleary & Zimmerman, 2012; Reeve, 2012), actively expressing one’s thoughts, opinions, and interests during activity, or when one engages communally, collectively, and critically with others (Lawson & Lawson, 2013).

While agentic engagement has been widely accepted as a construct of SE within the in-class context, few research studies have examined it in the wider educational context (Bond, 2020). However, one can visualize that, in the out-of-class (campus) context, a student who is agentically engaged will act to contribute to the academic community, willingly propose ideas or solutions to improve the campus environment, and involve themselves in extra-curricular activities.

Agentic engagement is the only proactive form of engagement as it is defined as student-initiated activity, which is different from the other three reactive forms of engagement (Montenegro, 2019). As agentic engagement manifests through student behavior, confusion with (reactive) behavioral engagement is possible. Therefore, in order to avoid any possible confusion between behavioral engagement and agentic engagement in this paper, behavioral engagement will be referred to as participatory engagement as suggested by Finn (1989) and Yazzie-Mintz (2007).

By unifying the terminology of these three components across the two contexts (in-class and out-of-class), we start to see the foundation of a unifying model. While this model does not include academic engagement, which encompasses SE in the context of students’ independent academic studies, we believe there is value in the unification of the two contexts included herein.

Methods

A mixed-methods approach was taken to explore the possibility of unifying these two SE contexts as a mixed-methods approach can provide greater insights (Creswell & Creswell, 2018, p. 294). This section will explain the methods in greater detail.

The literature review provides an overview of SE with different structures, approaches, and dimensions. What is widely agreed upon is that SE is a meta-construct that can be analyzed from different perspectives based on the researchers’ purpose. While there are four components of SE including emotional, cognitive, participatory, and agentic engagement, agentic engagement has only been considered in the in-class context. It has not been examined and measured in the out-of-class context. This research will explore integrating agentic engagement into a unifying model which includes all four dimensions of SE across the two contexts. Our proposed model is presented in Figure 3.

Figure 3.

Proposed unifying model of student engagement.

To confirm the presence of agentic engagement in the proposed model, the authors followed the first six steps in the seven-step procedure by Hinkin et al. (1997) to develop a comprehensive SE model with appropriate instrumentation to be validated in the Vietnamese HE context. These six steps are detailed as follows.

Step 1: Item Generation

Step 1.1: Review existing instruments

A thorough review of the literature produced 14 existing instruments used to measure SE. Four of the existing questionnaires were developed for undergraduate use but none specifically for students in business and management. Eight of the 14 questionnaires used included both in-class and out-of-class SE. The other six considered only the in-class context, meaning that none explored solely out-of-class SE, which is the focus of this paper.

The Student Engagement Scale (Gunuc & Kuzu, 2015) was selected as the foundation for the new measurement instrument due to its theoretical construction in HE in the Global South. The items include: Valuing (five items) used for out-of-class cognitive, Sense of belonging (10 items) used for out-of-class emotional, and Participation (five items) used for out-of-class participatory engagement (Figure 4).

Figure 4.

Student out-of-class engagement structure.

There is still one component in the proposed model of SE without measurement items, agentic engagement. As past research has provided reliability and validity evidence for in-class agentic engagement, this pilot test is to confirm this sub-construct in a new context (out-of-class) with students in the specific discipline of business and management. Those items will be developed in the next step of item generation through qualitative research.

All these items were translated into Vietnamese by an expert Vietnamese translator working in the field of education. After that, the translated version was reviewed by another Vietnamese university lecturer who is proficient in both English and Vietnamese. Minor adjustments were made to the translated list of items.

Step 1.2: Item generation through qualitative research

The theoretical unifying structure of SE used for this research has one new element, out-of-class agentic (OA) engagement, that has never been measured (with the exception of Finn’s (1989) related work). Some initial measurement items were formed based on the descriptions of the taxonomy from Finn (1989), such as a student’s active participation in extra-curricular activities and school governance.

Qualitative research was then undertaken to explore possible additional items in measuring the sub-construct of OA in the context of business and management students in Vietnam. The methods used for collecting qualitative data included in-depth interviews, expert interviews, and a focus group. During expert interviews, beside the exploration purpose, the face validity of the constructs as well as instrument measurement of SE were reviewed.

Selection of informants

Informants of this qualitative research were undergraduate students and lecturers in the field of business and management, as well as experts in the field of student behavior and student psychology.

The first group of students were introduced to the first author through the connection of the class facilitators. However, after the first two in-depth interviews, the interviewer decided to invite eight students to join a focus group. Seven students actually showed up and participated in the focus group, one was absent due to personal reasons.

All interviews and the focus group were hosted in the interviewer’s office with a friendly decor to ensure a quiet environment to facilitate audio recording. The students were informed of the choice of room in advance and all students consented. The length of student interviews were from 30 to 45 minutes and that of the focus group was 1 hour and 45 minutes.

Lecturers are those who have the most frequent interactions with students; therefore they have certain understandings and insights into student engagement. Four lecturers in business and management at a large public university were invited to in-depth interviews. The lecturers were chosen using convenience sampling and each had a minimum of 10 years of experience teaching undergraduate students in business and management programs. All interviews with lecturers were held in person in the aforementioned office, for the same reason as with students. The length of each interview with lecturers was about 1 hour.

In order to get expert opinions about student behavior, three experts in the field of psychology with a research focus on undergraduate students were invited to join individual in-depth interviews. Three researchers with PhDs in educational psychology were included, each from a different public university’s department of pedagogy. Due to the constraints of the experts’ availability and pandemic-related issues, all expert interviews were held via video call. A brief containing the research objectives and interview purpose was sent to each expert in advance for their preparation. The length of interview with each expert was between 45 and 60 minutes.

Implementation of data collection

At the beginning of each interview and focus group, the interviewer spent approximately five minutes explaining the purpose of the research and the importance of the informant’s participation and thanked them for their presence. The interviewer also explained about the confidentiality policy and informant’s right to stop providing information at any time. Informants showed their understanding by signing a consent form. Regarding the online interviews, consent forms were sent to interviewees in advance via email and all interviewee showed their consent via email or via oral acceptance at the beginning of their interview session.

All participants were informed that the interviews and focus group would be audio recorded and they all agreed. All audio recordings were transcribed within one week from the time of recording. Additional hand-written notes made during interviews supported the transcription process.

The interviews and focus group were all semi-structured asking participants about their perspectives on SE, along with how to measure it, both in-class and out-of-class. The topic of SE is not widely discussed in Vietnam so the early questions included “When you think of a student being engaged or connected to their school and studies, what does that mean to you?” Another example is “How can you tell if one student is engaged but another is not?” Follow up questions were asked to delve deeper into the participants’ conceptual understanding of SE with questions such as, “If an engaged student pays attention in class, what exactly does that look like?” As the meeting progressed, we explored exactly what questions should be asked to determine how engaged a student is.

The context was new both because it takes place in Vietnam (where the concept of SE is quite new) and because the subjects of the pilot questionnaire would be business and management students (a group of students not targeted by any existing measurement instruments). Accordingly, we discussed not only in-class and out-of-class contexts but also all aspects so that we could compare what participants thought against what is found in literature to look for conflicts, support, or simply additional concepts in order to produce the richest possible measurement instrument.

Data processing

After being transcribed into text files, the transcripts were then marked up, compared between documents, and compared with the proposed measurement instrument of SE.

As the proposed measurement instrument was extracted from various articles in the literature from other countries, with different cultures, there were three possibilities when comparing the instrument with the findings of the respondents. The first possibility is that the instrument matches with what has been reported in the interview transcripts. This situation will help to confirm the face validity of the instrument. The second is that one or more items in the proposed instrument does not appear in the interview transcripts. In this case, these items were highlighted for further face validity checks to ensure they belong in the instrument. The final, and most interesting, findings of these interviews would be anything in the transcripts which was not in the proposed measurement instrument. Such findings may serve as new items to measure student engagement in the current research context.

One key question of this research is to see if OA SE is a distinct factor of SE, which would allow for a model to be created that would unify in-class and out-of-class SE. However, because SE has not been measured in the current contexts (Vietnam and business and management HE students), the goal is to produce a complete measurement instrument to measure SE. This would be done by adapting existing instruments for in-class SE to the local context, by adapting existing instruments for OP (out-of-class participative), OC (cognitive), and OE (emotional), and by creating items to measure OA (agentic). Therefore, the interviews did ask about all aspects of SE, although there was a heavier focus on OA SE.

Findings from interviews

Most of the interviews with students showed that they were not familiar with the concept of SE. However, they had no difficulty in understanding or answering the interview questions. Lecturers were more familiar with the concept and had favorable attitudes toward it.

- I think the phenomenon that students love their school and have positive thoughts, attitudes, and activities toward the school, is a phenomenon that has existed for a long time, possibly different in their hidden or explicit form of demonstration. (Female, 25 years of teaching experience)

Below are the main findings extracted from the interviews:

Interactions between students and their friends, faculty, and school on social media was mentioned as an aspect of engagement. Those interactions also demonstrate various components and levels of engagement.

This has been repeatedly found in student interviews:

- I will share on Facebook interesting events of my university. I think that is how I make people know more about my study program. (Female, 2nd year student)

- I interact with my school in both direct communication and online interaction on social media. My school’s [fan]page is very active. I just feel that if we made more contacts, such as sharing, texting, tagging, more people will know about my school, and it will help boost awareness of my school and its reputation. (Female, 3rd year student)

- I chat with my friends in class, hang out with them after school time, and comment on each other’s Facebook. (Female, 2nd year student)

- I often share enrollment information of my university [on my Facebook], and if anyone comes in to ask I am willing to advise. (Male, 3rd year student)

Lecturers also suggested that students interact and engage with their school via social media:

- The bonding to the school can also be seen in actions such as protecting and showing affection toward the school or spreading awareness about the school’s brand name. It means it is outside the scope of the classroom. This made me recall a recent event at my university. My university initiated some volunteer activities…in a very short time, and then a few criticized them on social media about certain small details during the organization of the event. Later on, I learned that some of our students got together and spoke up to protect the image and reputation of the university… I think such an initiation-taking activity [of students] should be seen as a high-level of student interaction with their school. (Female, 20 years of teaching experience)

- On student forums, when someone asks about which university to study…, our students will answer and share related information such as information resources, information sessions… (Male, 14 years of teaching experience)

From expert interviews:

- I find students’ interactions on social media nowadays is also a very important aspect. (Female, 16 years of research experience)

- The scope of student engagement expands much broader now, not just among students in the classroom, but through social media like Facebook. (Male, 10 years of research experience)

- Now when it comes to students, it definitely goes with the foundation of social media. It’s their daily life as they spend thousands of hours there, display all different aspects of their life, including their engagement with school. (Male, 15 years of teaching and research experience)

This common finding from different interviews matches recent studies about digital technology as an integral and central aspect of higher education that significantly affects all aspects of the student experience (Barak, 2018; Henderson et al., 2017; Selwyn, 2016). This is also emphasized when students are referred to as “learners of the 21st century” (Gunuc & Kuzu, 2014) or “digital natives” (Prensky, 2001) and that digital technology and social media have become an indispensable part of their daily lives (Yu et al., 2010).

Among those opinions, there are different levels of demonstration of student engagement via social media, from (1) commenting on Facebook, (2) sharing events and spreading awareness, to (3) giving advice, and (4) protecting the reputation of the school. Based on the engagement level, those items will be put into two categories. The first two items belong to OP SE, and the last two belong to OA SE.

2. Active participation in extracurricular activities can be displayed in the form of membership in club management boards and event organizers.

- I think I am an active student. I participate in all the extracurricular activities of the program. I am a member of some club management boards and event organizers…and I have mentored freshmen to get familiar with the program in the last two years. (Male, 3rd year student)

This is a high level of engagement that can help measure OA SE, as only a few students can become members of club management boards. Such participation shows the proactiveness of students in their interaction with the campus environment. It also fits well with the highest level of participation in the taxonomy proposed by Finn (1989) in the form of school governance, involvement in club management boards, student associations, and other relevant student organization bodies.

In the end, the face validity of the construct, as well as the instrumentation of SE, were reviewed. One item from the Finn (1989) taxonomy and five new items were proposed to measure OA SE. Additionally, some items were added to the OP SE measure to reflect the interactions of students in the digital era.

Step 2: Content Adequacy

Step 2.1: Choosing a suitable scale

In student self-report measurement instruments, a seven-level Likert Scale is typically used, including in large surveys. Examples include the Motivation and Engagement Scale—University/College (MES-UC) (Martin, 2009) and the Student Satisfaction Survey used at Oxford Brooks University (Ghori, 2016), the University of Central England in Birmingham (Kane et al., 2008), and most other universities in the UK (Williams & Cappuccini-Ansfield, 2007).

Step 2.2: Data collection method

Among the methods used to measure SE, self-report measures are the most common (Fredricks & McColskey, 2012) and were chosen for that reason.

Step 2.3: Final format of the pilot questionnaire

The questionnaire used in this pilot test version has three parts for the main content. The first section of the questionnaire is an introduction providing an explanation of the research purpose, type of informants and information collection, confidentiality policy, statement of consent, and instructions on completing the questionnaire. At the end of this introduction, respondents will be asked to complete a “consent statement” where they show their voluntary consent of taking part in the survey.

The second section contains student engagement items, with eight items for OP engagement, 10 items for OE engagement, six items for OC engagement, and six items for OA engagement. The measurement of this part is a seven-point Likert scale, with which students show their level of agreement to each of the statements.

The final section was about the informant’s demographic and personal information, such as age, gender, year of study, major, number of subjects they are taking, and university name.

As the pilot version was long and took significant time for respondents to complete, no reversed item were designed into the form, so respondents could focus on the main content of the pilot test.

Step 2.4: Final instrument format review

In this step, the pilot questionnaire was put through an instrument content review (and pretest) to make sure that the Vietnamese version was clearly worded and ready to use. As there are two versions of the questionnaire, online and paper, the online version was sent to three lecturers and two students, the paper version was sent to one lecturer and four students.

Informants were told about the purpose of the research and asked to fill in the questionnaire, to record their completion time, and prompted to provide feedback on the wording or to raise any concerns with the Vietnamese translation. Most of the feedback was about the length of the questionnaire with a recommendation to give enough time for respondents to complete it.

Some minor changes were then made based on comments about meaning and wording consistency, the question arrangement of the paper version, as well as the branching setting of the online version.

Step 3: Questionnaire Administration

Step 3.1: Administration

The final list of items was put in two different versions: online and printed. The online questionnaire was sent to business students at a large national university in Vietnam. Only those who explicitly clicked to accept the consent statement proceeded to the main questionnaire.

The printed version, given to business students at a different large national university in Vietnam, had a similar format and content. However, in order to save time for the participants, the research purpose, privacy policy, and completion instructions were orally communicated.

Step 3.2: Determining sample size

In order to appropriately conduct the subsequent analysis, data must be collected from an adequate sample size (Hinkin et al., 1997). Many researchers agree that the number of variables or items to be assessed will determine the sample size needed to reach robust results. That means that as the number of items increases, the sample size should increase accordingly (Hinkin et al., 1997). Some earlier recommendations for item-to-response ratios ranged from 1:4 (Rummel, 1970, cited by Hinkin et al., 1997) to 1:10 (Schwab, 1980, cited by Hinkin et al., 1997) for each set of scales to be factor analyzed. Hair et al. (2009) emphasized that sample size affects results and suggested a desired ratio of 5 to 10 observations per variable. As there are 30 variables being analyzed for out-of-class SE, the range would be 150 to 300 responses. Similarly, general guidelines from Worthington and Whittaker (2006) state that a minimum of 300 respondents is adequate in most situations.

Step 4: Factor Analysis

Exploratory factor analysis (EFA) was performed to explore the components of the SE construct. The extraction method used was principal component analysis (eigenvalue greater than 1, Varimax rotation, with the absolute value of small coefficients to be suppressed being 0.45).

Confirmatory factor analysis (CFA) was run using AMOS Graphic version 20 to confirm the proposed model of SE.

Step 5: Internal Consistency

Cronbach’s alpha was used to verify internal consistency as it is the most frequently used measure for this purpose (Cho & Kim, 2015). While some researchers are concerned that alpha includes a non-trivial bias (Green & Yang, 2009) compared to structural equation modeling, a thorough meta-analysis exploring the presence of any such bias in practice shows that the bias is, practically speaking, insignificant (Peterson & Kim, 2013).

Step 6: Construct Validity

Construct validity was handled through checking if there is convergence (that all within-construct correlations are both high and of approximately the same magnitude) and differentiation (cross-correlations are high, uniform, and lower than the within-construct correlations) as recommended by Fornell and Larcker (1981).

Results and Discussion

As earlier research has provided reliability and validity evidence for in-class agentic engagement, the purpose of this pilot test is to confirm the sub-construct in a new context, with students in the specific discipline of business and management. This approach also fits with the difference in the nature of the contexts used to identify measurements for each set: in-class and out-of-class engagement.

There were 61 online and 264 printed responses collected, among which 312 responses were valid. The response rate was 58.7% for the printed version. Invalid responses included blank or half-blank responses, responses completed with a single choice across all questions, or completed with a purposefully ordered pattern (e.g., 1-2-3-4-5-6-7-6-5-4-3-2-1).

As the online and printed questionnaires were distributed through separate channels, there was no duplication of responses. The total sample size of 312 was satisfactory to the target range of 150 to 300 (Table 1).

Table 1.

Sample Information.

Gender	n	%
Male	112	35.9
Female	185	59.3
Missing	15	4.8
Total	312	100.0
Age
17–18	14	4.5
19	72	23.1
20	104	33.3
21	86	27.6
≥22	33	10.6
Missing	3	1.0
Total	312	100.0
Study year
Year 1	14	4.5
Year 2	78	25.0
Year 3	112	35.9
Year 4	102	32.7
≥Year 5	2	0.6
Missing	4	1.3
Total	312	100.0

These gender ratios are typical for undergraduate students in business and management where female students are in the majority. The sample was fairly balanced across the years of study.

Identify Outliers

In order to prevent outliers from inappropriately influencing the analysis, it is important that any outliers are identified and eliminated (Hair, 2019). There were five observations found as outliers, based on Mahalanobis distance, which were removed from the data set, resulting in a final data set of 307 observations.

Factor Analysis

Exploratory factor analysis

An exploratory factor analysis (EFA) was used to analyze the components of the SE construct. EFA results for in-class SE confirmed the four sub-constructs of IE (in-class emotional), IC (in-class cognitive), IP (in-class participative), and IA (in-class agentic) engagement as found in earlier literature (Mameli & Passini, 2017; Montenegro, 2019).

Regarding out-of-class SE, EFA was used to explore the factor dimensions as well as to reduce the set of observed variables to a smaller set (Hinkin et al., 1997).

The initial results of the EFA for out-of-class engagement are as follows:

While there is debate over the reliability of the Guttman-Kaiser criterion (using eigenvalues above one) it has been found that the criterion works well when the number of factors is much less than the number of variables and communality is high (Yeomans & Golder, 1982). Still, a scree plot was used to visually confirm the optimum number of factors to be extracted (Hair et al., 2009). The point at which the curve begins to straighten out is considered the maximum number of factors to extract. As the slope of the scree plot is significantly reduced to make the curve flat after the fourth point (matching the earlier results), four factors should be extracted as the optimal result of this EFA. However, to be certain we must thoroughly explore the possibilities of either three or four factors and we must explore each of these options before deleting any items (Worthington & Whittaker, 2006) and performing CFA (Figure 5, Tables 2 and 3).

Figure 5.

Scree plot.

Table 2.

KMO and Bartlett’s Test.

Kaiser-Meyer-Olkin measure of sampling adequacy	0.927
Bartlett’s test of sphericity
Approx. Chi-Square	5,957.145
df	435
Sig.	.001

Note. The KMO value is 0.927 (>0.7) and the Bartlett’s test of sphericity is significant at p = .001.

Table 3.

Total Variance Explained and Pattern Matrix.

Comp-onent	Initial eigenvalues			Extraction sums of squared loadings			Rotation sums of squared loadings
Comp-onent	Total	% of variance	Cumu-lative %	Total	% of variance	Cumu-lative %	Total	% of variance	Cumu-lative %
1	11.420	38.066	38.066	11.420	38.066	38.066	6.040	20.132	20.132
2	4.049	13.497	51.563	4.049	13.497	51.563	4.816	16.052	36.184
3	1.801	6.005	57.568	1.801	6.005	57.568	4.520	15.068	51.252
4	1.156	3.854	61.422	1.156	3.854	61.422	2.285	7.618	58.870
5	1.066	3.552	64.974	1.066	3.552	64.974	1.831	6.104	64.974

Note. Extraction method: principal component analysis.

The first step is performing EFA imposing three factors. This resulted in 56.5% of total variance explained. However, there were many items showing high cross-loadings. After removing items with high cross-loadings, the following matrix is produced (Table 4).

Table 4.

Rotated Component Matrix for Three Factors.

Rotated component matrix^a
Component
Item	1	2	3
OP1	0.782
OP2	0.742
OP4			0.424
OP5			0.705
OP7	0.751
OP8	0.601
OE1		0.741
OE2		0.749
OE3		0.681
OE5		0.862
OE6		0.840
OE8		0.588
OE9		0.525
OC1			0.722
OC3			0.584
OC4			0.773
OC5			0.778
OA1	0.713
OA2	0.809
OA3	0.721
OA4	0.846
OA5	0.769

Note. Extraction method: principal component analysis. Rotation method: varimax with Kaiser normalization.

Rotation converged in five iterations.

Here we see some items from OP load on the same factor as some items from OC, which does not make sense conceptually. This leads us to generally rule out three factors.

The next step is performing EFA again imposing four factors. The initial result was an improved 59.0% of total variance explained but there were still some cross-loadings and loadings on the wrong conceptual component. After removing those items, the variance explained improved to 66.9%, an improvement over Gunuc and Kuzu’s (2015) model which explained only 59%. The final matrix is as follows (Table 5):

Table 5.

Rotated Component Matrix for Four Factors.

Rotated component matrix^a
Component
	1	2	3	4
OP4				0.542
OP5				0.776
OP6				0.684
OE1	0.745
OE2	0.778
OE3	0.710
OE4	0.644
OE5	0.865
OE6	0.842
OE9	0.492
OC1			0.764
OC2			0.734
OC3			0.629
OC4			0.754
OC5			0.822
OA1		0.645
OA2		0.866
OA3		0.799
OA4		0.877
OA5		0.755

Note. Extraction method: principal component analysis. Rotation method: varimax with Kaiser normalization.

Rotation converged in five iterations.

Confirmatory factor analysis

In order to run a confirmatory factor analysis (CFA), as it is very sensitive to missing data, a function to impute missing data was applied to the data set. After, the fitness indices for the model for out-of-class SE were compared with norms resulting in the following evaluation:

Among these six indices, three (CFI, CMIN/DF, and RMSEA) were good, one was acceptable (SRMR), and two (GFI and 90% confidence interval for RMSEA) were only marginally good when compared with standard cutoff values (Table 6).

Table 6.

Goodness of Fit.

Criteria	3-factor model	4-factor model	Cutoff	Source	Model fitness
CMIN/DF	3.819	2.851	<5	Schumacker and Lomax (2004)	Good
CFI	.849	.924	≥.9	Byrne (1994)	Good
RMSEA	.096	.078	<.08	Hair et al. (2009)	Good
90% confidence interval for RMSEA	(.089, .103)	(.069, .087)	The upper confidence should not exceed .08	L.-T. Hu and Bentler (1995)	Marginally good
SRMR	.101	.088	<.08	L.-T. Hu and Bentler (1999)	Acceptable
GFI	.788	.878	≥.9	Hooper et al. (2008)	Marginally good

There are a few points to consider here. First is Moss’s (2009) recommendation that cutoff values only be used as guidelines and not hard and fast rules. Second is that others (Sharma et al., 2005) say that RMSEA is preferable over GFI. The third consideration is that, while GFI is below the standard of .90, it is very sensitive to small sample sizes (Fan et al., 1999), like the size we see in this pilot study. We can also see this in the work of Gunuc and Kuzu (2015), which formed an important foundation for this pilot study. Their GFI was only .80 with a larger sample (n = 464). Although, that study did have a better CFI (.96) and SRMR (.62). This can be compared with a study by Mameli and Passini (2017) which reported a CFI of .92 and RMSEA of .47. Unfortunately, given Mameli and Passini’s large sample size (n = 1,210), they did not report GFI.

With such a problem of fitting engagement data to a model, Samuelsen (2012) noted that some factors may not fit the data as neatly as we want or even fit at all. Betts (2012) also identified this challenge when stating that SE’s indicators often load as different factors or constructs when doing survey research.

Importantly, all six indices had better values for four factors than for three, providing reasonable evidence that there should be four factors and that OA is a distinct factor of SE. This helps to answer the question from Bond (2020) whether student engagement has three or four factors.

In full light of these facts, the confirmatory factor analysis of this model was considered moderately satisfactory and acceptable for a pilot test.

CFA was run to check the proposed unifying model. The component matrix of in-class and out-of-class engagement were put into Amos Graphics version 20. The result can be seen in Figure 6.

Figure 6.

Standardized path diagram (4-factors, 2-dimensions).

Internal consistency assessment

In this step, the reliability of the scale will be tested based on Cronbach’s alpha (Table 7).

Table 7.

Item-Total Statistics.

Item	Scale mean if item deleted	Scale variance if item deleted	Corrected item-total correlation	Cronbach’s alpha if deleted
OP4	11.21	5.377	.430	.617
OP5	10.32	6.506	.530	.498
OP6	10.97	5.708	.457	.565
OP Cronbach alpha				.653
OE1	28.32	53.485	.731	.888
OE2	28.47	52.923	.763	.884
OE3	27.93	54.050	.727	.888
OE4	28.30	54.075	.708	.890
OE5	28.52	53.035	.801	.880
OE6	28.73	52.507	.777	.882
OE Cronbach alpha				.912
OC1	22.96	16.688	.732	.800
OC2	23.03	17.297	.702	.809
OC3	23.24	17.862	.532	.857
OC4	22.50	18.114	.604	.834
OC5	22.61	17.094	.757	.796
OC Cronbach alpha				.850
OA1	13.53	47.243	.600	.871
OA2	14.73	40.291	.772	.831
OA3	15.21	43.830	.672	.855
OA4	14.29	38.682	.807	.821
OA5	13.89	41.533	.675	.856
OA Cronbach alpha				.875

Alphas of OE, OC, and OA were all above .8. OP was slightly below .7, which might be concerning to some; however, Vaske (2008) stated the general rule that anything above .65 was considered adequate and Hair (2019, p. 161) considered .6 to be acceptable during exploratory research like the current study. Additionally, the number of items in OP is very low (only three items) and alpha is very sensitive to this issue (Vaske et al., 2017). All Alpha if Deleted were below Cronbach’s alpha (except the case of OC3 where alpha may slightly be improved).

Additionally, all Corrected Item-Total Correlation >.4, meaning they are highly interrelated and likely to measure the same construct. Therefore, the items used to measure the components of SE can be considered reliable.

Construct Validity

Convergence implies that all within-construct correlations are (1) high and (2) of approximately the same magnitude. Differentiation is satisfied if the cross-correlations are (1) high, (2) uniform, and (3) lower than the within-construct correlations (Fornell & Larcker, 1981).

Below are the Pearson correlations of all components of SE. The correlation of all pairs of components are statistically significant at the .001 level. All correlations are higher than .3, and most of them are higher than .5.

The criterion-related validity of the SE model is satisfactory (Table 8).

Table 8.

Correlations.

	(OP) Out-of-class participatory engagement	(OE) Out-of-class emotional engagement	(OC) Out-of-class cognitive engagement
(OE) Out-of-class emotional engagement
Pearson correlation	.648**
(OC) Out-of-class cognitive engagement
Pearson correlation	.524**	.675**
(OA) Out-of-class agentic engagement
Pearson correlation	.718**	.440**	.373**

Correlation is significant at the .01 level (2-tailed).

The above procedure shows that OA SE fits into the unifying model of SE as a separate and valid component. The EFA confirmed this factor based on the correlations among variables, while CFA presented a moderate fitness of this unifying model. Other tests of reliability and construct validity were carried out and all showed good results. Therefore, after six steps of developing the measurement instrument of out-of-class SE, which includes OA, the current instrument should be considered successful. However, further validation should be implemented to confirm the instrument in different contexts and communities.

Limitations

During the process of developing the measurement instrument for OA engagement, while the pilot test produced promising results with partial satisfaction of the CFA of the model, certain problems regarding the fitness of the model were encountered. These issues have been noted by other researchers (Betts, 2012; Samuelsen, 2012). Therefore, it is important to follow up this pilot study with a more complete investigation to fully validate the instrument.

Conclusion

After an extensive review of existing models of, and instruments to measure, student engagement (SE), its dimensions and components, mixed-methods research was conducted to explore the potential of unifying the way SE is modeled both inside and outside the classroom. Interviews and focus groups indicated this was possible by revising the existing models of campus engagement.

The model proposed in this paper includes the following components for both in-class and out-of-class (or campus) engagement: emotional, cognitive, participatory, and agentic. While in-class engagement had been previously modeled this way, existing models of out-of-class SE do not include agentic engagement. Therefore, the contribution of this research is to add agentic engagement to out-of-class SE, thus creating a common way to model SE inside or outside of the classroom.

EFA confirmed the four components and CFA gave satisfactory results for this pilot study. The instrument created to measure out-of-class SE is internally consistent when analyzed using Cronbach’s alpha.

The proposed unifying model provides a structure to represent these two contexts of SE. Since the authors preserved the most popular components of emotional, cognitive, and participatory engagement in the unifying model, researchers should be able to extend their investigations using this new model where appropriate.

As this model integrates the in-class and out-of-class contexts, it highlights the possible differences of the contextual environment regarding the components of SE. This is essential for researchers to analyze the impacts of SE and propose relevant interventions to drive desirable outcomes in higher education.

Researchers can now explore SE vertically or horizontally to meet their research purpose. By approaching the concept vertically, each of the four components of emotional, cognitive, participatory, and agentic engagement can be analyzed separately or together in any order. By approaching this concept horizontally, researchers can separate between the two contexts of the in-class and out-of-class environments. While this model does not include academic engagement, which encompasses SE in the context of students’ independent learning, it is still believed that there is great value in unifying the way in-class and out-of-class SE are structured.

This unifying structure of SE has many advantages and allows measurement of SE from a broad point of view as well as understanding SE in its complex environment. Thus, this proposed model improves the analytical options available to researchers of SE.

While not generalizable, we believe that there is value in the unification of the two contexts included herein and that there are underlying themes from which other researchers can learn.

Footnotes

Appendix

Appendix.

Student Engagement Questionnaire Wording.

Code	Vietnamese version (revised)	English items	Source
In-class participative engagement
IP1	Tôi là một sinh viên tích cực trong lớp.	I am an active student in class	Newmann et al. (1992)
IP7	Tôi cố gắng hết sức để hoàn thành trách nhiệm của mình trong công việc nhóm.	I try to do my best regarding my responsibilities in group work	Newmann et al. (1992)
IP9	Tôi tham gia tích cực trong thảo luận nhóm.	I contribute to class discussions	Gonyea et al. (2003)
IP10	Tôi tham gia trong các bài tập lớn, dự án, hoặc thuyết trình với các sinh viên khác.	I work on a class assignment, project, or presentation with other students	Gonyea et al. (2003)
In-class emotional engagement
IE3	Tôi tôn trọng bạn bè trong lớp.	I respect my classmates	Voelkl (1996)
IE4	Các lớp học của tôi rất thoải mái.	My classes are entertaining	Fredricks et al. (2003)
IE5	Tôi tôn trọng các thầy cô giáo của mình.	I respect my teachers	Voelkl (1996)
In-class cognitive engagement
IC1	Tôi tự tạo động lực học tập cho bản thân.	I motivate myself to learn	Lehr et al. (2004)
IC2	Tôi đặt ra các mục tiêu học tập của riêng mình.	I determine my own learning goals	Lehr et al. (2004)
IC3	Tôi nỗ lực hết mình trong giờ học.	I try to do my best during classes	Gunuc and Kuzu (2015)
IC4	Bên cạnh việc làm bài tập, tôi nghiên cứu thêm các nội dung liên quan đến bài học của mình.	Besides doing my lessons, I further study for my lessons	Fredricks et al. (2003)
IC8	Tôi cố gắng làm bài tập về nhà theo cách tốt nhất.	I try to do my homework in the best way	Newmann et al. (1992)
IC9	Tôi thích đón nhận những thử thách trong học tập.	I enjoy intellectual difficulties I encounter while learning	Appleton et al. (2006)
IC10	Tôi dành đủ thời gian và nỗ lực để học tập.	I spend enough time and make enough effort to learn	Newmann et al. (1992)
In-class agentic engagement
IA1	Trong giờ học, tôi đặt câu hỏi.	During class, I ask questions.	Reeve and Tseng (2011)
IA2	Tôi nói với thầy cô về những gì tôi thích và không thích về giờ học.	I tell the teacher what I like and what I don’t like.	Reeve and Tseng (2011)
IA3	Tôi nói cho thầy cô biết những gì làm tôi thấy hứng thú trong giờ học.	I let my teacher know what I’m interested in.	Reeve and Tseng (2011)
IA4	Trong giờ học, tôi thể hiện các mong muốn và ý kiến của mình.	During class, I express my preferences and opinions.	Reeve and Tseng (2011)
IA5	Tôi chủ động trao đổi với thầy cô khi thấy các vấn đề của lớp học.	I actively discuss with my teachers when there is concerns about the class.	Interviews
IA6	Tôi đưa ra các đề xuất để làm cho lớp học tích cực hơn.	I offer suggestions about how to make the class better.	Reeve and Tseng (2011)
Out-of-class participative engagement
OP4	Tôi được hưởng lợi ích từ các tiện ích sẵn có trong trường (căn-tin, thư viện, khu thể thao…)	I benefit from the facilities in campus (canteen, library, sports arenas, and so on)	Interviews
OP5	Tôi tuân thủ các quy định trong khuôn viên trường.	I follow campus rules	Fredricks et al. (2003)
OP6	Tôi tham gia các hoạt động ở trường một cách nghiêm túc.	I participate seriously	Finn (1993)
Out-of-class emotional engagement
OE1	Tôi thích đến trường.	I look forward to going to campus	Willms (2003)
OE2	Tôi cảm thấy mình là một phần của nhà trường.	I feel myself as a part of the campus	Finn (1993)
OE3	Khuôn viên trường là một nơi thật thoải mái.	Campus is an entertaining place	Sutherland (2010)
OE4	Tôi thích các hoạt động được tổ chức trong nhà trường.	I enjoy the activities carried out in campus	Voelkl (1996)
OE5	Tôi cảm thấy hạnh phúc khi ở trường.	I feel happy in campus	Fredricks et al. (2003)
OE6	Tôi thích dành thời gian ở trường.	I like spending time in campus	Fredricks et al. (2003)
Out-of-class cognitive engagement
OC1	Tôi tin tưởng trường đại học mang lại giá trị và lợi ích cho tôi.	I believe university is beneficial for me	Finn (1993)
OC2	Trường đại học có ý nghĩa quan trọng với cuộc sống của tôi.	University is of great importance in my life	Finn (1993)
OC3	Tôi nghĩ rằng các quy định ở trường đại học là công bằng với tất cả sinh viên.	I think the rules at university are fair for everybody	Appleton et al. (2006)
OC4	Tôi cố gắng không làm hỏng bất cứ thứ gì thuộc về nhà trường.	I try not to damage anything that belongs to the university	Gunuc and Kuzu (2015)
OC5	Tôi hiểu tầm quan trọng của giáo dục đại học.	I give importance to university education and take it seriously	Finn (1993)
Out-of-class agentic engagement
OA1	Tôi đóng góp tích cực cho các hoạt động ngoại khóa của lớp và/hoặc của trường.	I actively contribute to extra-curriculum activities of my class and/or school.	Interviews
OA2	Tôi là thành viên chủ chốt của ít nhất 1 câu lạc bộ/hội/nhóm của trường.	I am a core member of at least one club/association/group of my school.	Interviews
OA3	Tôi tham gia điều hành ít nhất 1 câu lạc bộ/hội/nhóm của trường.	Involved in the school governance or membership of student organizations.	Finn (1989)
OA4	Tôi đóng góp tích cực trong các hoạt động cho Câu lạc bộ mà mình tham gia.	I actively contribute to my club’s activities.	Interviews
OA5	Tôi chia sẻ các phong trào/sự kiện của Trường mình trên các mạng xã hội.	I share campaigns/events of my school on social networks.	Interviews

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by National Economics University, Hanoi, Vietnam.

ORCID iD

John Andre

References

Ainley

(2012). Students’ interest and engagement in classroom activities. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 283–302). Springer.

Appleton

J. J.

Christenson

S. L.

Furlong

M. J.

(2008). Student engagement with school: Critical conceptual and methodological issues of the construct. Psychology in the Schools, 45(5), 369–386.

Appleton

J. J.

Christenson

S. L.

Kim

Reschly

A. L.

(2006). Measuring cognitive and psychological engagement: Validation of the student engagement instrument. Journal of School Psychology, 44(5), 427–445.

Astin

A. W.

(1984). Student involvement: A development theory for higher education. Journal of College Student Development, 40(5), 518–529.

Audas

Willms

J. D.

(2001). Engagement and dropping out of school: A life-course perspective. Citeseer.

Barak

(2018). Are digital natives open to change? Examining flexible thinking and resistance to change. Computers & Education, 121, 115–123.

Betts

(2012). Issues and methods in the measurement of student engagement: Advancing the construct through statistical modeling. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 783–803). Springer.

Bloom

B. S.

(1974). Time and learning. American Psychologist, 29(9), 682.

Bond

(2020). Mapping research in student engagement and educational technology in higher education: A systematic evidence map. International Journal of Educational Technology in Higher Education, 17, 2.

10.

Brooks

Goldstein

(2012). The power of mindsets: Nurturing engagement, motivation, and resilience in students. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 541–562). Springer.

11.

Bryson

(2010). Enhancing student integration and success through a holistic engagement approach [Paper presentation]. What Works? Student Retention & Success, Leeds, England, UK.

12.

Burch

G. F.

Heller

N. A.

Burch

J. J.

Freed

Steed

S. A.

(2015). Student engagement: Developing a conceptual framework and survey instrument. Journal of Education for Business, 90(4), 224–229.

13.

Byrne

B. M.

(1994). Structural equation modeling with EQS and EQS/Windows: Basic concepts, applications, and programming. SAGE.

14.

Cho

Kim

(2015). Cronbach’s coefficient alpha: Well known but poorly understood. Organizational Research Methods, 18(2), 207–230. https://doi.org/10.1177/1094428114555994

15.

Christenson

S. L.

Reschly

A. L.

Wylie

(2012). Handbook of research on student engagement. Springer Science & Business Media.

16.

Cleary

T. J.

Zimmerman

B. J.

(2012). A cyclical self-regulatory account of student engagement: Theoretical foundations and applications. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 237–257). Springer.

17.

Creswell

J. W.

Creswell

J. D.

(2018). Research design: Qualitative, quantitative, and mixed methods approaches. SAGE.

18.

Crick

R. D.

(2012). Deep engagement as a complex system: Identity, learning power and authentic enquiry. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 675–694). Springer.

19.

Fan

Thompson

Wang

(1999). Effects of sample size, estimation methods, and model specification on structural equation modeling fit indexes. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 56–83. https://doi.org/10.1080/10705519909540119

20.

Finn

J. D.

(1989). Withdrawing from school. Review of Educational Research, 59(2), 117–142.

21.

Finn

J. D.

(1993). School engagement & students at risk. National Center for Education Statistics.

22.

Finn

J. D.

Voelkl

K. E.

(1993). School characteristics related to student engagement. The Journal of Negro Education, 62(3), 249–268.

23.

Fornell

Larcker

D. F.

(1981). Structural equation models with unobservable variables and measurement error: Algebra and statistics. SAGE.

24.

Fredricks

J. A.

Blumenfeld

Friedel

Paris

(2003, March). School Engagement [Paper presentation]. Indicators of Positive Development Conference, Washington, DC, Unites States.

25.

Fredricks

J. A.

Blumenfeld

P. C.

Friedel

Paris

A. H

. (2005). School engagement. In Moore

K. A.

Lippman

L. H.

(Eds.), What do children need to flourish? (pp. 305–321). Springer.

26.

Fredricks

J. A.

Blumenfeld

P. C.

Paris

A. H.

(2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109.

27.

Fredricks

J. A.

McColskey

(2012). The measurement of student engagement: A comparative analysis of various methods and student self-report instruments. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 763–782). Springer.

28.

Fullarton

(2002). Student engagement with school: Individual and school-level influences (LSAY Research Reports, 31). Australian Council for Education Research.

29.

Furlong

M. J.

Whipple

A. D.

Jean

G. S.

Simental

Soliz

Punthuna

(2003). Multiple contexts of school engagement: Moving toward a unifying framework for educational research and practice. The California School Psychologist, 8(1), 99–113.

30.

Ghori

(2016). Deconstructing concepts of student satisfaction, engagement and participation in UK higher education. Oxford Brookes University.

31.

Gonyea

R. M.

Kish

K. A.

Kuh

G. D.

Muthiah

R. N.

Thomas

A. D.

(2003). College student experiences questionnaire: Norms for the fourth edition. College Student Experiences Questionnaire Assessment Program (NJ1).

32.

Green

S. B.

Yang

(2009). Commentary on coefficient alpha: A cautionary tale. Psychometrika, 74(1), 121–135. https://doi.org/10.1007/s11336-008-9098-4

33.

Gunuc

Kuzu

(2014). Factors influencing student engagement and the role of technology in student engagement in higher education: Campus-class-technology theory. Turkish Online Journal of Qualitative Inquiry, 5(4), 86–113. https://doi.org/10.17569/tojqi.44261

34.

Gunuc

Kuzu

(2015). Student engagement scale: Development, reliability and validity. Assessment & Evaluation in Higher Education, 40(4), 587–610.

35.

Hair

J. F.

(2019). Multivariate data analysis (8th ed.). Cengage.

36.

Hair

J. F.

Black

W. C.

Babin

B. J.

Anderson

R. E.

(2009). Multivariate data analysis (7th ed.). Pearson.

37.

Hausmann

L. R.

Schofield

J. W.

Woods

R. L.

(2007). Sense of belonging as a predictor of intentions to persist among African American and White first-year college students. Research in Higher Education, 48(7), 803–839.

38.

Henderson

Selwyn

Aston

(2017). What works and why? Student perceptions of ‘useful’ digital technology in university teaching and learning. Studies in Higher Education, 42(8), 1567–1579.

39.

Hinkin

T. R.

Tracey

J. B.

Enz

C. A.

(1997). Scale construction: Developing reliable and valid measurement instruments. Journal of Hospitality & Tourism Research, 21(1), 100–120.

40.

Hooper

Coughlan

Mullen

(2008). Structural equation modelling: Guidelines for determining model fit. ÙElectronic Journal of Business Research Methods, 6(1), 53–60.

41.

L.-T.

Bentler

P. M.

(1995). Evaluating model fit. In Hoyle

R. H.

(Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 76–99). SAGE.

42.

L.-T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55.

43.

Kuh

G. D.

(2002). Being (dis)engaged in educationally purposeful activities: The influences of student and institutional characteristics. Research in Higher Education, 43(5), 555–575.

44.

Jang

Kim

E. J.

Reeve

(2016). Why students become more engaged or more disengaged during the semester: A self-determination theory dual-process model. Learning and Instruction, 43, 27–38.

45.

Kahu

E. R.

(2013). Framing student engagement in higher education. Studies in Higher Education, 38(5), 758–773.

46.

Kane

Williams

Cappuccini-Ansfield

(2008). Student satisfaction surveys: The value in taking an historical perspective. Quality in Higher Education, 14(2), 135–155.

47.

Kember

Lee

(2001). Cultivating a sense of belonging in part-time students. International Journal of Lifelong Education, 20(4), 326–341.

48.

Klem

A. M.

Connell

J. P.

(2004). Relationships matter: Linking teacher support to student engagement and achievement. Journal of School Health, 74(7), 262–273.

49.

Kuh

G. D.

(2009). The national survey of student engagement: Conceptual and empirical foundations. Contemporary Educational Psychology, 2009(141), 5–20. https://doi.org/10.1002/ir.283

50.

Kuh

G. D.

Kinzie

Buckley

J. A.

Bridges

B. K.

Hayek

J. C.

(2007). Piecing together the student success puzzle: Research, propositions, and recommendations: ASHE higher education Report (1st ed.). Jossey-Bass.

51.

Lawson

M. A.

Lawson

H. A.

(2013). New conceptual frameworks for student engagement research, policy, and practice. Review of Educational Research, 83(3), 432–479.

52.

Leach

(2012). Bursting bubbles in higher education. In Coiffait

(Ed.), Blue skies: New thinking about the future of higher education (pp. 58–60). Pearson.

53.

Lehr

C. A.

Johnson

D. R.

Bremer

C. D.

Cosio

Thompson

(2004). Essential tools: Increasing rates of school completion: Moving from policy and research to practice. University of Minnesota, Institute on Community Integration, National Center on Secondary Education and Transit.

54.

Mameli

Passini

(2017). Measuring four-dimensional engagement in school: A validation of the student engagement scale and of the agentic engagement scale. TPM: Testing, Psychometrics, Methodology in Applied Psychology, 24(4), 527–541.

55.

Marks

H. M.

(2000). Student engagement in instructional activity: Patterns in the elementary, middle, and high school years. American Educational Research Journal, 37(1), 153–184.

56.

Martin

A. J.

(2009). Motivation and engagement across the academic life span: A developmental construct validity study of elementary school, high school, and university/college students. Educational and Psychological Measurement, 69(5), 794–824.

57.

Montenegro

(2019). Why are students’ self-initiated contributions important (?) A study on agentic engagement. International Journal of Sociology of Education, 8(3), 291–315.

58.

Moss

(2009). Fit indices for structural equation modeling. https://www.sicotests.com/newpsyarticle/Fit-indices-for-structural-equation-modeling

59.

Newmann

F. M.

(1992). Student engagement and achievement in American secondary schools. ERIC.

60.

Newmann

Wehlage

Lamborn

(1992). The significance and sources of student engagement. In Newmann

(Ed.), Student engagement and achievement in American secondary schools (pp. 11–39). Teachers College Press.

61.

Peterson

R. A.

Kim

(2013). On the relationship between coefficient alpha and composite reliability. Journal of Applied Psychology, 98(1), 194–198. https://doi.org/10.1037/a0030767

62.

Prensky

(2001). Digital natives, digital immigrants Part 1. On the Horizon, 9(5), 1–6. https://doi.org/10.1108/10748120110424816

63.

Reeve

(2012). A self-determination theory perspective on student engagement. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 149–172). Springer US. https://doi.org/10.1007/978-1-4614-2018-7_7

64.

Reeve

Tseng

C.-M.

(2011). Agency as a fourth aspect of students’ engagement during learning activities. Contemporary Educational Psychology, 36(4), 257–267.

65.

Reschly

A. L.

Christenson

S. L.

(2012). Jingle, jangle, and conceptual haziness: Evolution and future directions of the engagement construct. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 3–19). Springer.

66.

Samuelsen

K. M.

(2012). Part V commentary: Possible new directions in the measurement of student engagement. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 805–811). Springer.

67.

Schumacker

R. E.

Lomax

R. G.

(2004). A beginner’s guide to structural equation modeling. Psychology Press.

68.

Selwyn

(2016). Digital downsides: Exploring university students’ negative engagements with digital technology. Teaching in Higher Education, 21(8), 1006–1021.

69.

Sharma

Mukherjee

Kumar

Dillon

W. R.

(2005). A simulation study to investigate the use of cutoff values for assessing model fit in covariance structure models. Journal of Business Research, 58(7), 935–943. https://doi.org/10.1016/j.jbusres.2003.10.007

70.

Sinatra

G. M.

Heddy

B. C.

Lombardi

(2015). The challenges of defining and measuring student engagement in science. Educational Psychologist, 50(1), 1–13.

71.

Skinner

E. A.

Furrer

Marchand

Kindermann

(2008). Engagement and disaffection in the classroom: Part of a larger motivational dynamic? Journal of Educational Psychology, 100(4), 765.

72.

Skinner

E. A.

Pitzer

J. R.

(2012). Developmental dynamics of student engagement, coping, and everyday resilience. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 21–44). Springer.

73.

Sutherland

S. D.

(2010). Student and teacher perceptions of student engagement. Toronto University.

74.

Trowler

(2010). Student engagement evidence summary. The Higher Education Academy.

75.

Trowler

(2010). Student engagement literature review. The Higher Education Academy, 11, 1–15.

76.

Vaske

J. J.

(2008). Survey research and analysis: Applications in parks, recreation and human dimensions. Venture Publishing, Inc.

77.

Vaske

J. J.

Beaman

Sponarski

C. C.

(2017). Rethinking internal consistency in Cronbach’s alpha. Leisure Sciences, 39(2), 163–173. https://doi.org/10.1080/01490400.2015.1127189

78.

Voelkl

K. E.

(1996). Measuring students’ identification with school. Educational and Psychological Measurement, 56(5), 760–770.

79.

Voelkl

K. E.

(1997). Identification with school. American Journal of Education, 105(3), 294–318.

80.

Voelkl

K. E.

(2012). School identification. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 193–218). Springer.

81.

Williams

Cappuccini-Ansfield

(2007). Fitness for purpose? National and institutional approaches to publicising the student voice. Quality in Higher Education, 13(2), 159–172. https://doi.org/10.1080/13538320701629186

82.

Willms

J. D.

(2003). Student engagement at school: A sense of belonging and participation: Results from PISA 2000. OECD publishing.

83.

Worthington

R. L.

Whittaker

T. A.

(2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806–838. https://doi.org/10.1177/0011000006288127

84.

Yazzie-Mintz

(2007). Voices of students on engagement: A report on the 2006 high school survey of student engagement. Center for Evaluation and Education Policy, Indiana University. https://files.eric.ed.gov/fulltext/ED495758.pdf

85.

Yeomans

K. A.

Golder

P. A.

(1982). The Guttman-Kaiser criterion as a predictor of the number of common factors. The Statistician, 31(3), 221. https://doi.org/10.2307/2987988

86.

A. Y.

Tian

S. W.

Vogel

Kwok

R. C.-W.

(2010). Can learning be virtually boosted? An investigation of online social networking impacts. Computers & Education, 55, 1494–1503.

87.

Zimmerman

B. J.

(1990). Self-regulated learning and academic achievement: An overview. Educational Psychologist, 25(1), 3–17.

Student Engagement: Validating a Model to Unify In-Class and Out-of-Class Contexts

Abstract

Keywords

Introduction

Literature Review

Dimensions of Student Engagement

Contexts of student engagement

In-class engagement

Out-of-class or campus engagement

Components of student engagement

Behavioral or participatory engagement

Emotional engagement

Cognitive engagement

Agentic engagement

Methods

Step 1: Item Generation

Step 1.1: Review existing instruments

Step 1.2: Item generation through qualitative research

Selection of informants

Implementation of data collection

Data processing

Findings from interviews

Step 2: Content Adequacy

Step 2.1: Choosing a suitable scale

Step 2.2: Data collection method

Step 2.3: Final format of the pilot questionnaire

Step 2.4: Final instrument format review

Step 3: Questionnaire Administration

Step 3.1: Administration

Step 3.2: Determining sample size

Step 4: Factor Analysis

Step 5: Internal Consistency

Step 6: Construct Validity

Results and Discussion

Identify Outliers

Factor Analysis

Exploratory factor analysis

Confirmatory factor analysis

Internal consistency assessment

Construct Validity

Limitations

Conclusion

Footnotes

Appendix

Declaration of Conflicting Interests

Funding

ORCID iD

References