Relations between Students’ Study Approaches,Perceptions of the Learning Environment,and Academic Achievement in Flipped Classroom Learning: Evidence from Self-Reported and Process Data

Abstract

This study examined the extent to which relations between students’ perceptions of the learning environment, academic achievement, and study approaches measured by both self-reported and process data were consistent amongst 323 engineering students in a flipped classroom course. A hierarchical cluster analysis on four self-reported scales identified deep and surface study approaches. Students reporting deep approaches obtained significantly better marks than those reporting surface approaches. An agglomerative sequence clustering on sequences of students’ online interactions found four observed study approaches: two focused on content and two focused on assessment. Students adopting content-focused approaches performed significantly better than those employing assessment-focused ones. Two cross-tabulations revealed consistency of relational patterns between perceptions of the learning environment and study approaches by self-reported or process data. Amongst students with better perceptions, a significantly higher proportion of them reported deep approaches than surface approaches. Amongst students using content-focused approaches, significantly higher proportions of them held positive perceptions than negative perceptions. The study results suggest to teachers that fostering a better learning environment, helping students understand how online and in-person components are integrated, and equipping them with knowledge and principles of flipped classroom learning would be useful to enhance students’ learning experiences in flipped classroom courses.

Keywords

study approaches self-reported and process data perceptions of the learning environment academic achievement flipped classroom Australian engineering students

Over the last few decades, learning in higher education has undergone significant transformations and innovations. In particular, flipped classroom learning design has gradually gained popularity worldwide (Cho et al., 2021). Flipped classroom learning design is a specific type of blended learning design and requires students to engage in “interactive content focusing on key concepts prior to class thus allowing class time for collaborative activities that clarify concepts and contextualize knowledge through application, analysis, and planning and producing solutions” (Karanicolas et al., 2018, p. 1). Accordingly, university students’ experiences of learning in flipped classrooms are becoming increasingly complex as students are often required to move back and forth between in-class and on-line learning environments. In these experiences, students not only interact with the human elements (teaching staff and fellow students), but also with the non-human or material elements (Fenwick et al., 2015). For instance, students spend a significant proportion of time navigating online learning platforms where they interact with a variety of technology-enabled learning tools, engage in online discussion forums, and move across physical and online learning spaces (Fenwick & Dahlgren, 2015). Transforming traditional learning and teaching into flipped classroom learning has an urgent need to understand how students perceive their learning environment (perceptions of the learning environment), how they approach learning (their study approaches), and the relations between their perceptions, approaches, and their academic achievement, in flipped classroom learning (Karabulut-Ilgu et al., 2018).

Traditionally, investigations into student learning in higher education from research areas in educational psychology, learning sciences, and pedagogy and curriculum studies, have predominantly relied on collecting students’ self-reported data. Recent developments in the areas of learning analytics and educational data mining have the capacity to collect and analyse comprehensive process data in technology-mediated learning, which not only provides relatively objective descriptions of students’ online study approaches, but also reflects dynamic and nuanced differences in how students approach online learning (Baker & Siemens, 2014).

However, both self-reported and process data have limitations. The self-reported data has received criticism for lacking objectivity (Matcha et al., 2020; Zhou & Winne, 2012). In addition, the responses to self-reported questionnaires may be affected by careless answering and item nonresponse (Hitt et al., 2016; Zamarro et al., 2018). Reliance on process data without guidance from educational theories has also received criticism for being data-centric, which can result in erroneous interpretation due to a lack of meaningful theory and context (Buckingham Shum & Crick, 2012).

To address the above-mentioned drawbacks, researchers have started to examine consistencies between self-reported and process data to describe students’ learning (Gašević et al., 2017; Han & Ellis, 2020; Han et al., 2020; 2022). Such investigations not only enable a triangulation between different data sources to describe student learning, but also provide complementary information on students’ reporting of their motives for using certain study approaches and their actual use of study approaches (Reimann et al., 2014). The current study aims to extend this area of research by investigating the relations between students’ perceptions of the learning environment, their academic achievement, and their study approaches measured by self-reported data collected from a questionnaire and process data collected in the Learning Management System (LMS) in a flipped engineering course.

Literature Review

Students’ Perceptions of the Learning Environment and their Self-Reported Study Approaches

How students perceive the situational characteristics of their learning context, including the curriculum, course structure, and teaching and delivery methods, have been consistently identified as an important aspect in students’ learning experience (Ellis et al., 2006; Guo et al., 2017; 2022). To describe the interrelatedness of the key aspects of students’ learning experience and outcomes, Biggs (1989) have proposed a Presage-Process-Product model (known as 3P model).

The Presage stage consists of both students’ personal attributes and the situational characteristics of their current learning environment. The Process stage may include aspects such as, how students perceive the learning environment (their perceptions) and how they go about learning (their study approaches). The Product stage is concerned with various measures of students’ learning outcomes, such as the course marks and students’ post conceptions of the subject matter. The elements in the 3P model are relational and coexisting rather than indications of causal or linear relations. Past research has shown that in some learning contexts, the elements in the Process stage can mediate the relations between the Presage stage and the Product stage, whereas in other contexts, the elements in the Presage stage are directly associated with elements in the Product stage (Trigwell & Prosser, 2020).

The two key factors which have been recognized in the Process stage are perceptions of the learning environment and study approaches. Two broad categories of approaches (deep and surface study approaches) have been consistently identified in various academic disciplines. Deep study approaches are motivated by an attempt to understand the deep meaning of subject matter, and to link theoretical concepts in learning with contextualized applications. Deep study approaches are typically characterized by taking initiative, while also being independent and reflective of learning. On the other hand, surface study approaches are motivated by the aim to satisfy course requirements and learning tasks. They use strategies which are simplistic and mechanistic, rely heavily on textbooks and course notes, and are dependent on peers and teachers (Nelson Laird et al., 2014). The two categories of study approaches have also been extended to the domain of using online learning technologies. The deep approaches to using online learning technologies employ technologies with an aim to facilitate learning and to deepen understanding of the subject matter; whereas surface approaches to using online learning technologies adopt technologies mostly to fulfill practical purposes, such as downloading files (Ellis & Bliuc, 2016; 2019).

Study approaches are not personal traits and can be consciously chosen according to different learning contexts and situations (Entwistle & Peterson, 2004; Joshi & Lau, 2021; Rajaratnam, D’cruz, 2016; Ramsden, 2003). In particular, past research has consistently shown that students’ perceptions of the teaching and learning environment are associated with the study approaches they adopt. Deep study approaches tend to be adopted by students who perceive teaching as being of good quality and with clear aims, and assessment tasks being appropriate and coherent with the course aims. Students who perceive that the workload is inappropriate, the learning objectives are unclear, and teacher-student interactions are lacking, are more likely to employ surface study approaches (Guo et al., 2017; Lizzio et al., 2002; Wilson & Fowler, 2005).

Research has also shown that in blended course designs, when students perceive that face-to-face and online elements are well integrated, the online learning workload is appropriate, and online contributions are of value, they are more likely to adopt deep approaches to study and to using online learning technologies. Comparatively, when students do not see the relevance between face-to-face and online learning, and do not appraise the online learning, they tend to adopt surface approaches and limit their use of technologies in learning (Ellis & Bliuc, 2016; 2019). Coherent associations have also been found between students’ perceptions of the learning environment, study approaches, and their academic learning outcomes, such that: positive perceptions, deep approaches, and higher academic performance tend to be related; negative perceptions, surface approaches, and poorer academic performance tend to be associated (Entwistle & Peterson, 2004; Guo, 2018; Lonka et al., 2004).

However, the existing research on the relations between students’ perceptions of the learning environment and their study approaches has mainly employed self-reported measures to identify students’ study approaches. Apart from the above-mentioned limitations of lacking objectivity and issues with students’ careless responses, self-reported measures also lack capacities to represent the complex and dynamic nature of students’ study approaches in the blended course designs where students traverse back and forth between physical and virtual spaces and integrate ideas across learning activities, resources, and technologies (Hadwin et al., 2007; Han, 2022; Matcha et al., 2020). To improve insights into understanding contemporary university students’ experiences of learning, suggestions have been put forward to expand the current self-reporting methods by including other types of measurements to study student learning (Vermunt & Donche, 2017). In this regard, the process data frequently employed in learning analytics research seems to be promising as such data “provides both researchers and practitioners with the opportunity to monitor students’ strategic decisions in online environments in minute detail and in real time” (Richardson, 2017, p. 359).

Observed Students’ Study Approaches Measured by Process Data

The recent development of educational technology has produced prolific studies using process data, which enables researchers to collect rich and detailed digital traces of students’ interactions with a variety of online learning resources and activities. The process data has the advantages of not only offering descriptions of student learning behaviors more objectively but also in more granular details than using self-reported methods (Gašević et al., 2015; Siemens & Gašević, 2012). Process data has been increasingly employed in various domains in the higher education sector to fulfil multiple purposes, such as advising students’ career choice (Bettinger & Baker, 2014); detecting at risk students to reduce dropout rates (Krumm et al., 2014); providing personalised feedback in learning (Gibson et al., 2017); facilitating collaborative learning (Kaendler et al., 2015); monitoring students’ affect (Ocumpaugh et al., 2014); identifying patterns of study strategies and approaches (Chen et al., 2017); and predicting students’ academic learning outcomes (Romero et al., 2013).

Earlier research in learning analytics predominantly used frequency data to describe students’ online learning experiences. More advanced LMSs are able to collect many different types of process data, such as: what types of online learning actions students generate; the proportions of each type of learning action; the total duration of students’ online access; the time spent on a certain online learning task; and the sequential order of the learning actions with precise timestamps (Chen et al., 2017; Matcha et al., 2019; 2020). Using data mining techniques, such as Hidden Markov Model (HMM), agglomerative sequence clustering, and process mining algorithms; the complexity of students’ study approaches can be represented (Dawson et al., 2017; Fincham et al., 2019; Jovanović et al., 2017; Matcha et al., 2019; 2020).

In a number of studies, similar approaches have been taken to analyse the sequences of time-stamped online learning events by using a two-step method. While the first step focused on identifying common study strategies shared by all the students, the second step focused on identifying study approaches by categorizing sub-groups of students who shared similar study strategies. In the first step, the types of online learning events were processed by HMMs to generate a limited set of learning sequence types by similar sequential distributions of the online learning events, which represented students’ study strategies. In the second step, the generated learning sequences were processed by agglomerative sequence clustering on student population sample to identify students’ study approaches.

For instance, using the above-mentioned two step method, Jovanović et al. (2017) identified five different types of online study approaches amongst 290 computer science undergraduates:

• “the intensive learners”: they used a variety of study approaches;

• “the strategic learners”: they prioritized summative and formative assessment tasks;

• “the highly strategic learners”: they emphasized summative;

• “the selective learners”: they focused on summative activities with few reading activities;

• “the highly selective learners”: they only performed summative activities.

Jovanović et al. (2017) also compared students’ academic performance by their study approaches. Results showed that “the intensive learners”, “the strategic learners”, and “the highly strategic learners” obtained higher marks than “the selective learners” and “the highly selective learners” on both mid-term and final examinations.

Adopting a similar method, Han et al. (2022) investigated Australian’ engineering students’ study approaches using process data and identified four distinct approaches, namely: intensive theory application; moderate theory application; weak theory application and moderate theoretical testing; and weak reading and weak theory application. Similar to Jovanović et al. (2017), Han et al. found that students who used different study approaches also differed on their academic achievement. Students who employed intensive theory application strategy achieved the highest examination scores, whereas those who used weak reading and weak theory application scored the lowest examination scores. This study further examined the relations between students’ perceptions of the learning environment and their study approaches and found logical relations between perceptions and approaches. Amongst students who adopted the intensive theory application strategy, the proportion of students who self-reported better perceptions was significantly higher than those reporting poorer perceptions. In contrast, amongst students using the weak reading and weak theory application strategy, the proportion of students having poorer perceptions was significantly higher than those holding better perceptions.

The study by Han et al. (2022) addressed the limitation of predominant use of self-reports to examine students’ study approaches in the investigations of the relations between perceptions and approaches. However, it was not able to provide evidence as to whether the students’ study approaches measured by self-reported and process data are similarly or differently associated with students’ perceptions of the learning environment. The current study aims to fulfil this purpose. Moreover, the current study will also extend previous investigations in general blended course designs into a specific type of course design – the flipped classroom design. The current study addresses three research questions:

1. What are students’ study approaches in flipped classroom learning identified by self-reported and process data?

2. To what extent are students’ perceptions of the learning environment in flipped classroom learning are associated with their study approaches by self-reported and process data?

3. To what extent do students’ academic achievement in flipped classroom learning differ based on their study approaches by self-reported and process data?

Method

Participants and Recruitment Procedure

The participants of the study were 323 undergraduates recruited from a large metropolitan research-intensive Australian university. All the participants were first-year students who majored in Bachelor of Computing Engineering. The range of their age was from 17 to 31 years old, with an average of 20 years old. Of 323 students, 170 were males and 153 were females, accounting for 52.6% and 47.4% respectively. The percentages of male and female students showed a relatively equal gender distribution. The recruitment of the participants strictly followed the ethical requirements of the researchers’ institution. All the participants signed a written consent form.

Course Design

This engineering course, which lasted a full semester, was offered to first-year computing engineering students. The course covered foundational concepts of a computer system: including hardware and software components of a computer, principles of computer architecture, digital logic design, the Internet and servers, steps in program development and design, and programming style and documentation. Apart from content learning, the course also aimed to develop students’ graduate skills of inquiry and research competence in areas such as formulating appropriate questions, searching and synthesizing ideas, and evaluating and critically reflecting on evidence.

The course adopted a flipped classroom learning design. The online course site was hosted in a bespoke learning management system (LMS), which was comprised of the following learning activities:

• pre-lecture videos: covered the learning and teaching in the forthcoming week, including: 1) theoretical concepts and topics to be explained in the lectures; 2) the sample problem-solving sequences to be demonstrated in tutorials; and 3) the laboratory instructions and the tasks to be completed in the laboratory sessions;

• pre-lecture quizzes: consisted of quizzes for students to check their learning and understandings of the pre-lecture videos;

• post-lecture quizzes: consisted of quizzes of problem-solving sequences, which required students to apply theories to solve practical problems.

The in-person part of the course consisted of the following elements:

• 2-hour weekly lectures: recapped and expanded the concepts from pre-lecture videos.

• 2-hour weekly tutorials: theories to solve practical issues were first demonstrated by tutors; students were asked to work in pairs or in groups to go through problem-solving sequences, including calculations, model configurations, and mini case studies. The tutors facilitated the process by providing timely feedback and answering questions raised by the students.

• 3-hour weekly laboratory sessions: students worked in pairs or in groups to gain hands-on skills through completing design projects, such as design and configuration of an electric circuit.

Data and Instruments

Self-reported Data Collected by a Questionnaire

A 5-point Likert scale questionnaire was used to collect self-reported data. The questionnaire consisted of six scales. Two scales assessed students’ perceptions of the learning environment, two measured students’ approaches to learning through inquiry, and two evaluated students’ approaches to using online learning technologies. These scales have been used and validated in previous studies that investigated university students’ learning experiences (Ellis & Bliuc, 2016; 2019; Han & Ellis, 2020; Han et al., 2020). The descriptions of the six scales and coefficient H reliability are reported below:

• perceptions of the integrated learning environment scale (7 items, H¹ = .89): examined students’ perceptions of the level of integration between the online learning and the face-to-face learning components (e.g., The online activities in this course help with the work we do in class).

• perceptions of the online contributions scale (6 items, H = .89): assessed students’ perceptions of the value of online contributions in the course (e.g., Online contributions in this course motivated me to think about things more).

• deep approaches to learning through inquiry scale (4 items, H = .86): described approaches that had characteristics of taking initiative, being proactive, and undertaking reflection (e.g., I often take the initiative when pursuing a line of questioning in research.).

• surface approaches to learning through inquiry scale (7 items, H = .87): described approaches that were formulaic, highly dependent upon teaching staff, and without much reflection in the learning process (e.g., I only use the directions my teacher gives me when researching something for a task).

• deep approaches to using online learning technologies scale (6 items, H = .83): evaluated using online learning technologies in meaningful ways to facilitate learning, such as to deepen their understanding of the concepts in the course, or to facilitate their learning through inquiry process in the course (e.g., I spend time using the learning technologies in this course to connect key ideas to real contexts).

• surface approaches to using online learning technologies scale (5 items, H = .84): evaluated using online learning technologies in formulaic and mechanistic ways, such as downloading documents or fulfilling course requirements (e.g., I use learning technologies in this course mainly to download files).

Process Data Collected by LMS

The process data was collected by the learning analytic functions in the LMS, which recorded students’ unique identification codes and students’ interactions with different online learning activities with timestamps. The three types of timestamped online interactions were: 1) interactions with pre-lecture videos; 2) interactions with pre-lecture quizzes; and 3) interactions with post-lecture quizzes.

Academic Achievement

Students’ academic achievement was represented by the course mark with the highest possible score being 100. The course mark was an aggregated score of lecture attendance, performance in the tutorials, and a close-book examination. Lecture attendance was 10 best results out of 13 quizzes at the end of each lecture. The quizzes used a short-answer format, which tested important concepts covered in each lecture. Each quiz was scored between 0 and10. The maximum raw score of lecture attendance was 100, which was given 10% weights in the course mark. Performance in the tutorials was 10 best results of 13 quizzes at the end of each tutorial. The quizzes assessed students’ abilities to apply theories in tackling problems through problem-solving sequences. Each quiz was scored from 0 to 10. The maximum raw score of performance in the tutorials was 100, which was weighted 40% in the course mark. The close-book examination evaluated both students’ understandings of theoretical concepts using multiple-choice questions and their competence to use theories to solve practical problems using mini case studies. The maximum score of the close-book examination was 100, which was given 50% weights in the course mark.

Data Analysis Methods

To answer research question 1 – students’ self-reported study approaches, we performed a hierarchical cluster analysis using the means of deep and surface approaches to learning through inquiry scales; and deep and surface approaches to using online learning technologies scales. Because cluster analysis is exploratory and does not offer a single best solution, the appropriate number of clusters needs be determined by the researchers. We used the values of the Squared Euclidean Distance between different cluster solutions and the meaningfulness of the results to decide upon the number of clusters (Murtagh & Legendre, 2014). We then conducted one-way ANOVAs on the four approaches scales using the cluster membership as the independent variable and the Ms of the four approaches scales as the dependent variables. To provide robust statistical analyses of the data, we calculated and reported 95% robust confidence intervals for one-way ANOVAs. The results of cluster analysis and one-way ANOVAs were used to describe students’ self-reported study approaches. For students’ study approaches by process data, we used agglomerative sequence clustering to cluster students based on the sequences of the three types of timestamped online interactions. We used clustering quality measures calculated in TramineR (Gabadinho et al., 2011) and the interpretability of the results to determine number of clusters.

To answer research question 2 – the association between students’ perceptions of the learning environment and their study approaches by self-reported and process data, we first calculated the mean of the two perceptions scales and categorized students into better (above the mean) or poorer (below the mean) perceptions groups. Then we conducted two separate cross-tabulation analyses using groupings of perceptions of the learning environment and using either groupings of study approaches by self-reported data or groupings of study approaches by process data.

To answer the last research question – students’ academic performance based on study approaches by self-reported data or by process data, we carried out two separate one-way ANOVAs using either the cluster membership of study approaches by self-reported data or the cluster membership of study approaches by process data as the independent variable and students’ final mark as the dependent variable. We calculated and reported 95% robust confidence intervals for ANOVAs.

Results

Descriptive Statistics

The descriptive statistics of the six scales in the questionnaire and students’ course mark are presented in Table 1, including minimum and maximum values, Ms and SDs.

Table 1.

Descriptive Statistics of The Six Scales and Students’ Course Mark.

Variables	M	SD	M	SD
Perceptions of the integrated learning environment	1.75	5.00	3.51	0.65
Perceptions of the online contributions	1.43	5.00	2.81	0.61
Deep approaches to learning through inquiry	1.67	5.00	3.57	0.63
Surface approaches to learning through inquiry	1.00	5.00	2.67	0.80
Deep approaches to using online learning technologies	1.43	5.00	3.75	0.72
Surface approaches to using online learning technologies	1.00	5.00	3.18	0.89
Course mark	11.67	98.33	61.83	18.02

Results for Research Question 1 – Students’ Study Approaches by Self-reported Data

The hierarchical cluster analysis produced a range of two-cluster to four-cluster solutions. The values of Squared Euclidean Distance measure revealed a relatively large increase in the value of a two-cluster solution compared to three-cluster and four-cluster solutions, suggesting a two-cluster solution was more appropriate. The labelling of the two clusters was in accordance with the literature of students’ study approaches (Trigwell & Prosser, 2020).

From Table 2, we can see that cluster 1 and cluster 2 had 162 and 161 students respectively. One-way ANOVAs showed that the two clusters of students differed significantly on all four scales. Cluster 1 students reported using significantly more deep approaches to inquiry, F (1, 321) = 115.76, p < .01, η² = .27, 95% robust C.I. = [0.54, 0.79] and deep approaches to using online learning technologies, F (1, 321) = 15.87, p < .01, η² = .13, 95% robust C.I. = [-0.57, −0.32] than cluster 2 students; whereas cluster 2 students reported using significantly more surface approaches to inquiry, F (1, 321) = 89.04, p < .01, η² = .22, 95% robust C.I. = [0.46, 0.71] and surface approaches to using online learning technologies, F (1, 321) = 94.18, p < .01, η² = .23, 95% robust C.I. = [-0.91, −0.60] than cluster 1 students. According to the patterns of the results, cluster 1 and 2 were labelled as deep study approaches and surface study approaches respectively.

Table 2.

Self-reported Online Study Approaches.

Self-Reported Study Approaches	1 Deep (n = 162)		2 Surface (n = 161)		F	p	η²	95% Robust C.I.
Self-Reported Study Approaches	M	SD	M	SD	F	p	η²	Lower Bound	Upper Bound
DAI	3.84	0.49	3.17	0.61	115.76	.00	.27	0.54	0.79
SAI	2.58	0.54	3.03	0.59	49.32	.00	.13	−0.57	−0.32
DAT	3.86	0.51	3.28	0.60	89.04	.00	.22	0.46	0.71
SAT	2.29	0.66	3.05	0.74	94.18	.00	.23	−0.91	−0.60

Note: DAI = deep approaches to learning through inquiry, SAI = surface approaches to learning through inquiry, DAT = deep approaches to using online learning technologies, SAT = surface approaches to online learning technologies.

Results for Research Question 1 – Students’ Study Approaches by Process Data

Consulting with the clustering quality measures, we retained the four clusters used in the agglomerative sequence clustering. The four clusters represented four distinct study approaches by process data, which are visualized in Figure 1. Figure 1 schematically demonstrated the learning sequences, which consisted of the three types of timestamped online interactions with varying proportional distributions. Red colour represented students’ online interactions with pre-lecture videos, green colour represented students’ interactions with post-lecture quizzes, and blue indicated students’ interactions with post-lecture quizzes.

Figure 1.

The four observed online study approaches.

The four observed study approaches are described below:

• observed study approach 1 (n = 51): extensive pre-lecture videos watching, moderate pre-lecture quizzes, and few post-lecture quizzes;

• observed study approach 2 (n = 115): extensive pre-lecture videos watching, few pre-lecture quizzes, and moderate post-lecture quizzes;

• observed study approach 3 (n = 128): moderate pre-lecture videos watching, few pre-lecture quizzes, and moderate post-lecture quizzes;

• observed study strategy 4 (n = 29): few pre-lecture videos watching, few pre-lecture quizzes, and extensive post-lecture quizzes.

Results for Research Question 2 – Association between Students’ Self-reported Study Approaches and Their Perceptions of the Learning Environment

A 2 (deep vs. surface study approaches) x 2 (better vs. poorer perceptions) cross-tabulation analysis was conducted and the association was significant and moderate, χ² (1) = 22.83, p < .01, φ = .27. The proportions in Table 3 revealed that amongst students who held better perceptions of the learning environment, a significantly higher proportion of them adopted deep study approaches in the course (68.9%) relative to using surface study approaches (31.1%). In contrast, of students who held poorer perceptions, a significantly higher proportion of them used surface study approaches (59.4%), compared with employing deep approaches (40.6%).

Table 3.

Association between Self-Reported Study Approaches and Perceptions of the Learning Environment.

Grouping Variables	Count % within Perceptions	Deep Study Approaches	Surface Study Approaches	Total
Better perceptions	Count	73	33	106
Better perceptions	%	68.9%	31.1%	100.0%
Poorer perceptions	Count	88	129	217
Poorer perceptions	%	40.6%	59.4%	100.0%
Total	Count	161.00	162.00	323
	%	50.2%	49.8%	100.0%

Results for Research Question 2 – Association between Students’ Observed Study Approaches and Perceptions of the Learning Environment

A 4 (four study approaches) x 2 (better vs. poorer perceptions) cross-tabulation analysis was conducted and the results show a significant and moderate association between students’ observed study approaches and their perceptions of the learning environment, χ² (3) = 22.48, p < .01, Cramer’s V = .26). The proportions in Table 4 show that amongst 51 learners who adopted observed study approach 1 (extensive pre-lecture videos watching, moderate pre-lecture quizzes, and few post-lecture quizzes), a significantly higher proportion of them had better perceptions of the learning environment (41.0%) than poorer perceptions (24.5%). A similar pattern was also observed amongst 115 students who adopted observed study approach 2 (extensive pre-lecture videos watching, moderate pre-lecture quizzes, and moderate post-lecture quizzes), where the proportion of students having better perceptions (77.4%) was significantly higher than poorer perceptions (22.6%).

Table 4.

Association between Observed Study Approaches and Perceptions of the Learning Environment.

Grouping Variables	Count % within Study Approaches	Better Perceptions	Poorer Perceptions	Total
Observed study approach 1	Count	40	11	51
Observed study approach 1	%	78.4%	21.6%	100.0%
Observed study approach 2	Count	89	26	115
Observed study approach 2	%	77.4%	22.6%	100.0%
Observed study approach 3	Count	77	51	128
Observed study approach 3	%	60.2%	39.8%	100.0%
Observed study approach 4	Count	11	18	29
Observed study approach 4	%	37.9%	62.1%	100.0%
Total	Count	106	217	323
	%	67.2%	32.8%	100.0%

In contrast, of 128 students who adopted observed study approach 3 (moderate pre-lecture videos watching, few pre-lecture quizzes, and moderate post-lecture quizzes), a significantly higher proportion of them held poorer perceptions of the learning environment (60.2%) than better perceptions (39.8%). The same pattern was also found amongst students who adopted study approach 4 (few pre-lecture videos watching, few pre-lecture quizzes, and extensive post-lecture quizzes), where the proportion of students having poorer perceptions (62.1%) was significantly higher than those having better perceptions (37.9%).

Results for Research Question 3 – Students’ Academic Achievement by Their Self-reported Study Approaches

The results of the one-way ANOVA, F (1, 321) = 7.08, p < .01, η² = .02, 95% robust C.I. = [1.37, 9.21] showed that students who adopted more deep study approaches (M = 64.46, SD = 17.26) obtained significantly higher course marks than their peers who used more surface study approaches (M = 61.83, SD = 18.02).

Results for Research Question 3 – Students’ Academic Achievement by Their Observed Study Approaches

The results of the one-way ANOVA showed that students adopting different observed study approaches differed significantly on their academic achievement, F (3, 319) = 30.71, p < .01, η² = .22. To identify the differences, we conducted post-hoc analyses and the results are displayed in Table 5. According to Table 5, students using observed online learning approach 1 (extensive pre-lecture videos watching, moderate pre-lecture quizzes, and few post-lecture quizzes: M = 72.87), were followed by students adopting observed online learning approach 2 (extensive pre-lecture videos watching, moderate pre-lecture quizzes, and moderate post-lecture quizzes: M = 67.27). Students using observed study approach 3 (moderate pre-lecture videos watching, few pre-lecture quizzes, and moderate post-lecture quizzes: M = 56.88) were ranked third by their course marks and those employing observed study approach 4 (few pre-lecture videos watching, few pre-lecture quizzes, and extensive post-lecture quizzes: M = 42.66) scored the lowest for their academic performance.

Table 5.

Post-hoc Analyses of Students’ Academic Achievement.

Observed Study Approaches	Course Marks		p values [95% Robust C.I. [Lower Bound, Upper Bound]
Observed Study Approaches	M	SD	Approach 2	Approach 3	Approach 4
Observed study approach 1	72.87	15.84	.04 [0.32, 10.87]	.00 [10.80, 21.19]	.00 [22.92, 37.51]
Observed study approach 2	67.27	16.54	—	.00 [6.37, 14.43]	.00 [18.10, 31.14]
Observed study approach 3	56.88	15.09	—	—	.00 [7.77, 20.67]
Observed study approach 4	42.66	17.37	—	—	—

Discussion

This study investigated Australian first-year engineering students’ study approaches in flipped classroom learning using both self-reported data and process data. It also examined the relations between students’ perceptions of the learning environment, their academic achievement, and their study approaches measured by self-reported and process data.

Students’ Study Approaches Identified by Self-reported and Process Data

Consistent with previous findings in the general blended learning context, students studying in flipped classrooms could also be clearly distinguished by either adopting more deep approaches or adopting more surface approaches, which covered the ways they learned and used online learning technologies in the learning designs (Han et al., 2020; Ellis & Bliuc, 2016; 2019). When students’ motives for learning was to achieve understanding of subject matter, the ways they approached learning was proactive, reflective, and analytical, and used online learning technologies in meaningful ways to help them understand, make inquiry, and research. On the other hand, when their learning intents were to merely satisfy the course requirements, they approached learning through relatively simplistic activities of rote learning, reproducing, and duplication; and used online learning technologies in simplistic ways (Biggs, 1989; Biggs & Tang, 2011; Ramsden, 2003; Trigwell & Prosser, 2020). The results showed consistency in how students implemented inquiry-based learning strategies and how they used online learning technologies to facilitate them to inquire.

As for students’ observed study approaches, although students differed on how much they distributed their learning in different online activities, clearly students who adopted observed study approach 1 (extensive pre-lecture videos watching, moderate pre-lecture quizzes, and few post-lecture quizzes) and 2 (extensive pre-lecture videos watching, few pre-lecture quizzes, and moderate post-lecture quizzes) focused more on the pre-class preparation activities Comparatively, students who adopted observed study approach 3 (moderate pre-lecture videos watching, few pre-lecture quizzes, and moderate post-lecture quizzes) and 4 (few pre-lecture videos watching, few pre-lecture quizzes, and extensive post-lecture quizzes) placed greater emphasis on the post-class assessment activities.

Apart from emphasizing pre-class or post-class activities, study approaches 1 and 2 seemed to be content-focused approaches, whereas approaches 3 and 4 were assessment-focused. The former two approaches seemed to study the course to gain understandings and to focus on learning through watching pre-lecture videos. Despite this, the two approaches differed in whether the testing of understanding occurred before (approach 1) or after (approach 2) the in-person lectures. This kind of content-focused approach could be an in-depth approach rather than an assessment-focused one, which places greater emphasis on post-lecture quizzes. As the process data were not able to reveal students’ intents and motives behind what they actually did, students could complete post-lecture quizzes to help them consolidate their learning, while they could also undertake quizzes to fulfill the assessment requirements. If their objective was the former, clearly post-lecture quizzes seemed not to be as effective as pre-lecture quizzes, as reflected by the poorer academic achievement results of students who used approach 3 and 4. However, students’ study motives and intents need to be verified via interviews.

The study approaches identified by self-report and process data provided complementary information. Specifically, while the self-reported measures offered the information of students’ motives and purposes of using a particular strategy, the process data showed the distribution of different online learning activities and the stages (e.g., before or after the in-person lectures) in which these activities occurred in flipped classroom learning. Hence, the advantage of employing both self-reported and process measures is the capacity to equip teachers and researchers with richer information on the study approaches of students (Han, 2022; Ober et al., 2021; Reimann et al., 2014).

Patterns of Relations between Perceptions of the Learning Environment, Academic Achievement, and Study Approaches Measured by Self-Reported and Process Data

The results of the relations between students’ self-reported study approaches, perceptions of the learning environment, and academic achievement, largely corroborate previous findings that deep study approaches, better perceptions, and higher academic performance tend to be related, whereas surface approaches, poorer perceptions, and lower academic performance are more likely to be associated with traditional in-person delivery (Asikainen & Gijbels, 2017; Asikainen et al., 2022; Guo, 2018; Guo et al., 2017) and general blended course designs (Ellis & Bliuc, 2016; 2019; Han et al., 2020); but extend these patterns of the relations in the flipped classroom learning context. In our study, of students who reported using more deep study approaches in the course, a significantly higher proportion of them also perceived that the online learning materials and activities, including pre-lecture videos, and pre- and post-lecture quizzes, were well integrated with what they did in lectures, tutorials, and laboratory practice, while the online learning was also an important part in the whole course. In contrast, amongst students who employed more surface study approaches, a significantly higher proportion of them perceived that the online learning was fragmented from what they did in class and did not value the online learning. At the same time, students adopting more deep study approaches also obtained higher course marks than their peers using more surface study approaches.

As explained above, content-focused approaches involved learning at a more in-depth level than assessment-focused approaches, and for the observed study approaches we found that amongst students who adopted two content-focused approaches (approaches 1 and 2), there were higher proportions with better perceptions of the learning environment; whereas amongst students using assessment-focused approaches (approach 3 and 4), it was found that there were higher proportions of students holding poorer perceptions. Also, resembling the patterns observed for the relations between students’ self-reported study approaches and academic achievement, students who adopted content-focused approaches also performed more highly than their peers who used two assessment-focused approaches. These results suggest that the relations between students’ observed study approaches and perceptions of the learning environment were consistent with the relations between students’ self-reported study approaches and perceptions of the learning environment.

Implications for Teaching

The findings of our study may offer teachers a number of strategies to improve Australian engineering students’ learning in flipped classrooms. The significant associations between students’ perceptions of the learning environment and their use of study approaches suggest to teachers that fostering a better learning environment would be helpful for students to develop desirable study approaches. For instance, teaching staff may design online learning activities and assessment tasks which can reinforce the learning and teaching of disciplinary contents in classroom activities (Rotellar & Cain, 2016). Teachers may also explicitly signal to students how the online learning materials, activities, and assessment tasks, complement examples and demonstrations in lectures and tutorials in order for students to see how the online and in-person components of the course are integrated.

Furthermore, teachers could equip students with sufficient knowledge about, and principles of, flipped classroom learning at the commencement of the course, as research has shown that many students do not possess much understanding about learning in a flipped classroom upon entering college (Hao, 2016; Yilmaz, 2017). As our study results indicated the importance of pre-class learning in flipped classrooms, teachers may emphasize activities in the preparation phase, such as watching the pre-lecture videos and completing the pre-lecture quizzes (Long et al., 2017; Schwarzenberg et al., 2020). Inclusion of the pre-lecture quizzes as a compulsory assessment may also be useful in flipped classroom learning design. This would ensure that students achieve a certain level of conceptual understanding to be discussed and expanded in classroom teaching (Cho et al., 2021). The learning analytic functions built in the LMS may offer teachers tools to identify students’ study approaches early in the semester so that teachers can remind those who do not prepare them well before the classroom learning to change their preparation strategies and approaches (Viberg et al., 2018).

Limitations of the Study and Directions for Future Research

The limitations of the study should be noted and addressed in future research. First, the current study only investigated some important aspects in the Process (study approaches and perceptions of the learning environment) and Product (academic performance) stages in the 3P model. Future studies should also include important aspects in the Presage stage, such as students’ conceptions of learning and their motivation in learning, as these aspects have also been reported to relate to study approaches in previous research (Trigwell et al., 2013).

Moreover, the current study adopted a cross-sectional design, which did not reflect changes of students’ study approaches and perceptions of the learning environment in the course during the semester. Hence, fluctuations and changes of students’ study approaches and perceptions were not able to be reflected in the current study. Future research should use a longitudinal design to measure these aspects multiple times throughout the semester, which may help reveal dynamic relations between study approaches, perceptions, and academic performance.

Last but not least, while the perceptions of the learning environment covered both face-to-face and online learning components in this flipped classroom course, the study approaches measured by process data only demonstrated what students did online. Future studies should employ other data collection methods, such as video recording, to capture information on how students approach their learning tasks in-person, so that a more comprehensive picture of students’ observed study approaches can be captured.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Feifei Han

Note

Author Biography

Feifei Han currently is a Senior Research Fellow at the Institute for Learning Sciences and Teacher Education, Australia Catholic University. Her current research interests are in the areas of educational technology, learning analytics, and STEM education. She has over 100 publications, including a book, 26 book chapters, 61 journal articles, and 22 conference proceedings. Some of her publications appear in top-quality journals in educational technology, such as The Internet and Higher Education, Computers and Education, and IEEE Transactions on Learning Technologies. She serves as an associate editor for The Australasian Journal of Educational Technology and Frontiers in Psychology (the Educational Psychology section).

References

Asikainen

Gijbels

(2017). Do students develop towards more deep study approaches during studies? A systematic review on the development of students’ deep and surface study approaches in higher education. Educational Psychology Review, 29(2), 205–234. https://doi.org/10.1007/s10648-017-9406-6

Asikainen

Nieminen

J. H.

Häsä

Katajavuori

(2022). University students’ interest and burnout profiles and their relation to approaches to learning and achievement. Learning and Individual Differences, 93(1), 102105. https://doi.org/10.1016/j.lindif.2021.102105

Baker

Siemens

(2014). Educational data mining and learning analytics. In Sawyer

(Ed), The Cambridge handbook of the learning sciences (2nd ed. pp. 253–272). New York, NY: Cambridge University Press.

Bentler

P. M.

(2007). Covariance structure models for maximal reliability of unit-weighted composites. In Lee

(Ed), Handbook of computing and statistics with applications: Vol. 1. Handbook of latent variable and related models (pp. 1–19). New York, NY: Elsevier.

Bettinger

E. P.

Baker

R. B.

(2014). The effects of student coaching: An evaluation of a randomized experiment in student advising. Educational Evaluation and Policy Analysis, 36(1), 3–19. https://doi.org/10.3102/0162373713500523

Biggs

Tang

(2011). Teaching for quality learning at university: What the student does (4th. ed.). London: McGraw-Hill.

Biggs

J. B.

(1989). Approaches to the enhancement of tertiary teaching. Higher Education Research and Development, 8(1), 7–25. https://doi.org/10.1080/0729436890080102

Buckingham Shum

Crick

R. D.

(2012). Learning dispositions and transferable competencies: Pedagogy, modelling and learning analytics. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 92–101). Vancouver: ACM.

Chen

Resendes

Chai

C. S.

Hong

H. Y.

(2017). Two tales of time: Uncovering the significance of sequential patterns among contribution types in knowledge-building discourse. Interactive Learning Environments, 25(2), 162–175. https://doi.org/10.1080/10494820.2016.1276081

10.

Cho

H. J.

Zhao

Lee

C. R.

Runshe

Krousgrill

(2021). Active learning through flipped classroom in mechanical engineering: Improving students’ perception of learning and performance. International Journal of STEM Education, 8(1), 13–46. https://doi.org/10.1186/s40594-021-00302-2

11.

Dawson

Jovanović

Gašević

Pardo

(2017). From prediction to impact: Evaluation of a learning analytics retention program. In Proceedings of the 7th International Conference on Learning Analytics and Knowledge (pp. 474–478). Vancouver: ACM.

12.

Ellis

R. A.

Bliuc

A. M.

(2016). An exploration into first‐year university students' approaches to inquiry and online learning technologies in blended environments. British Journal of Educational Technology, 47(5), 970–980. https://doi.org/10.1111/bjet.12385

13.

Ellis

R. A.

Bliuc

A. M.

(2019). Exploring new elements of the student approaches to learning framework: The role of online learning technologies in student learning. Active Learning in Higher Education, 20(1), 11–24. https://doi.org/10.1177/1469787417721384

14.

Ellis

R. A.

Goodyear

Prosser

O’Hara

(2006). How and what university students learn through online and face‐to‐face discussion: Conceptions, intentions and approaches. Journal of Computer Assisted Learning, 22(4), 244–256. https://doi.org/10.1111/j.1365-2729.2006.00173.x

15.

Entwistle

Peterson

(2004). Learning styles and approaches to studying. Encyclopedia of Applied Psychology, 2, 537–542.

16.

Fenwick

Dahlgren

M. A.

(2015). Towards socio‐material approaches in simulation‐based education: Lessons from complexity theory. Medical Education, 49(4), 359–367. https://doi.org/10.1111/medu.12638

17.

Fenwick

Edwards

Sawchuk

(2015). Emerging approaches to educational research: Tracing the socio-material. New York, NY: Routledge.

18.

Fincham

Gašević

Jovanović

Pardo

(2019). From study tactics to learning strategies: An analytical method for extracting interpretable representations. IEEE Transactions on Learning Technologies, 12(1), 59–72. https://doi.org/10.1109/TLT.2018.2823317

19.

Gabadinho

Ritschard

Müller

N. S.

Studer

(2011). Analyzing and visualizing state sequences in R with TraMineR. Journal of Statistical Software, 40(4), 1–37. https://doi.org/10.18637/jss.v040.i04

20.

Gašević

Dawson

Siemens

(2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. https://doi.org/10.1007/s11528-014-0822-x

21.

Gašević

Jovanović

Pardo

Dawson

(2017). Detecting learning strategies with analytics: Links with self-reported measures and academic performance. Journal of Learning Analytics, 4(2), 113–128. https://doi.org/10.18608/jla.2017.42.10

22.

Gibson

Aitken

Sándor

Á.

Buckingham Shum

Tsingos-Lucas

Knight

(2017). Reflective writing analytics for actionable feedback. In Proceedings of the 7th International Conference on Learning Analytics and Knowledge (pp. 153–162). Vancouver: ACM.

23.

Guo

(2018). Building bridges to student learning: Perceptions of the learning environment, engagement, and learning outcomes among Chinese undergraduates. Studies in Educational Evaluation, 59(4), 195–208. https://doi.org/10.1016/j.stueduc.2018.08.002

24.

Guo

Yang

Shi

(2017). Effects of perceptions of the learning environment and approaches to learning on Chinese undergraduates’ learning. Studies in Educational Evaluation, 55(4), 125–134. https://doi.org/10.1016/j.stueduc.2017.09.002

25.

Guo

J. P.

Yang

L. Y.

Zhang

Gan

Y. J.

(2022). Academic self-concept, perceptions of the learning environment, engagement, and learning outcomes of university students: Relationships and causal ordering. Higher Education, 83(4), 809–828. https://doi.org/10.1007/s10734-021-00705-8

26.

Hadwin

A. F.

Nesbit

J. C.

Jamieson-Noel

Code

Winne

P. H.

(2007). Examining trace data to explore self-regulated learning. Metacognition and Learning, 2(2-3), 107–124. https://doi.org/10.1007/s11409-007-9016-7

27.

Han

(2022). Recent development in university student learning research in blended course designs: Combining theory-driven and data-driven approaches. http://doi.org/10.3389/fpsyg.2022.905592

28.

Han

Ellis

R. A.

(2020). Combining self-reported and observational measures to assess university student academic performance in blended course designs. Australasian Journal of Educational Technology, 36(6), 1–14. https://doi.org/10.14742/ajet.6369

29.

Han

Ellis

R. A.

Pardo

(2022). The descriptive features and quantitative aspects of students’ observed online learning: How are they related to self-reported perceptions and learning outcomes? IEEE Transactions on Learning technologies. https://doi.org/10.1109/TLT.2022.3153001

30.

Han

Pardo

Ellis

R. A.

(2020). Students’ self-report and observed learning orientations in blended university course design: How are they related to each other and to academic performance? Journal of Computer Assisted Learning, 36(6), 969–980. https://doi.org/10.1111/jcal.12453

31.

Hancock

G. R.

Mueller

R. O.

(2001). Rethinking construct reliability within latent variable systems. In Cudeck

du Toit

Sörbom

(Eds), Structural equation modeling: Present and future—a festschrift in honor of Karl Jöreskog (pp. 195–216). Lincolnwood, IL: Scientific Software International.

32.

Hao

(2016). Exploring undergraduates’ perspectives and flipped learning readiness in their flipped classrooms. Computers in Human Behavior, 59(7), 82–92. https://doi.org/10.1016/j.chb.2016.01.032

33.

Hitt

Trivitt

Cheng

(2016). When you say nothing at all: The predictive power of student effort on surveys. Economics of Education Review, 52(3), 105–119. https://doi.org/10.1016/j.econedurev.2016.02.001

34.

Joshi

Lau

S. K.

(2021). Effects of process-oriented guided inquiry learning on study approaches, long-term performance, and online learning outcomes: Interactive Learning Environments. https://doi.org/10.1080/10494820.2021.1919718

35.

Jovanović

Gašević

Pardo

Dawson

Mirriahi

(2017). Learning analytics to unveil study approaches in a flipped classroom. The Internet and Higher Education, 23(2), 74–85. http://doi.org/10.1016/j.iheduc.2017.02.001

36.

Kaendler

Wiedmann

Rummel

Spada

(2015). Teacher competencies for the implementation of collaborative learning in the classroom: A framework and research review. Educational Psychology Review, 27(3), 505–536. http://doi.org/10.1007/s10648-014-9288-9

37.

Karabulut‐Ilgu

Jaramillo Cherrez

Jahren

C. T.

(2018). A systematic review of research on the flipped learning method in engineering education. British Journal of Educational Technology, 49(3), 398–411. https://doi.org/10.1111/bjet.12548

38.

Karanicolas

Snelling

Winning

Chapman

Williams

A. M.

Douglas

M. T.

Lee

M. I.

(2018). Translating concept into practice: Enabling first-year health sciences teachers to blueprint effective flipped learning approaches. Adelaide: University of Adelaide.

39.

Krumm

Waddington

Teasley

Lonn

(2014). A learning management system-based early warning system for academic advising in undergraduate engineering. In Larusson

White

(Eds), Learning analytics: From research to practice (pp. 103–119). New York: Springer.

40.

Lizzio

Wilson

Simons

(2002). University students’ perceptions of the learning environment and academic outcomes: Implications for theory and practice. Studies in Higher Education, 27(1), 27–52. https://doi.org/10.1080/03075070120099359

41.

Long

Cummins

Waugh

(2017). Use of the flipped classroom instructional model in higher education: Instructors’ perspectives. Journal of Computing in Higher Education, 29(2), 179–200. https://doi.org/10.1007/s12528-016-9119-8

42.

Lonka

Olkinuora

Mäkinen

(2004). Aspects and prospects of measuring studying and learning in higher education. Educational Psychology Review, 16(4), 301–323.

43.

Matcha

Gasevic

Ahmad Uzir

Jovanovic

Pardo

Lim

Maldonado-Mahauad

Gentili

Perez-Sanagustin

Tsai

Y. S.

Tsai

Y. S.

(2020). Analytics of learning strategies: Role of course design and delivery modality. Journal of Learning Analytics, 7(2), 45–71. https://doi.org/10.18608/jla.2020.72.3

44.

Matcha

Gašević

Uzir

N. A. A.

Jovanović

Pardo

(2019). Analytics of study approaches: Associations with academic performance and feedback. In Proceedings of the 9th International Conference on Learning Analytics and Knowledge (pp. 461–470). Vancouver: ACM.

45.

McNeish

(2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412–433. https://doi.org/10.1037/met0000144

46.

Murtagh

Legendre

(2014). Ward’s hierarchical agglomerative clustering method: Which algorithms implement ward’s criterion? Journal of Classification, 31(3), 274–295. https://doi.org/10.1007/s00357-014-9161-z

47.

Nelson Laird

T. F.

Seifert

T. A.

Pascarella

E. T.

Mayhew

M. J.

Blaich

C. F.

(2014). Deeply affecting first-year students' thinking: Deep approaches to learning and three dimensions of cognitive development. The Journal of Higher Education, 85(3), 402–432. https://doi.org/10.1080/00221546.2014.11777333

48.

Ober

T. M.

Hong

M. R.

Rebouças-Ju

D. A.

Carter

M. F.

Liu

Cheng

(2021). Linking self-report and process data to performance as measured by different assessment types. Computers and Education, 167(8), 104188. https://doi.org/10.1016/j.compedu.2021.104188

49.

Ocumpaugh

Baker

Gowda

Heffernan

(2014). Population validity for educational data mining models: A case study in affect detection. British Journal of Educational Technology, 45(3), 487–501. https://doi.org/10.1111/bjet.12156

50.

Rajaratnam

D’cruz

S. M.

(2016). Learning styles and study approaches – are they different? Education for Health, 29(1), 59–60. https://doi.org/10.4103/1357-6283.178924

51.

Ramsden

(2003). Learning to teach in higher education (2nd ed.). London: Routledge.

52.

Reimann

Markauskaite

Bannert

(2014). e‐Research and learning theory: What do sequence and process mining methods contribute? British Journal of Educational Technology, 45(3), 528–540. http://doi.org/10.1111/bjet.12146

53.

Richardson

J. T. E.

(2017). Student learning in higher education: A commentary. Educational Psychology Review, 29(2), 353–362. https://doi.org/10.1007/s10648-017-9410-x

54.

Romero

López

M.-I.

Luna

J.-M.

Ventura

(2013). Predicting students’ final performance from participation in on-line discussion forums. Computers and Education, 68(9), 458–472. https://doi.org/10.1016/j.compedu.2013.06.009

55.

Rotellar

Cain

(2016). Research, perspectives, and recommendations on implementing the flipped classroom. American Journal of Pharmaceutical Education, 80(2), 34. https://doi.org/10.5688/ajpe80234

56.

Schwarzenberg

Navon

Pérez-Sanagustín

(2020). Models to provide guidance in flipped classes using online activity. Journal of Computing in Higher Education, 32(2), 282–306. https://doi.org/10.1007/s12528-019-09233-y

57.

Siemens

Gašević

(2012). Guest editorial: Learning and knowledge analytics. Journal of Educational Technology and Society, 15(3), 1–2.

58.

Trigwell

Ashwin

Millan

E. S.

(2013). Evoked prior learning experience and approach to learning as predictors of academic achievement. The British Journal of Educational Psychology, 83(Pt 3), 363–378. https://doi.org/10.1111/j.2044-8279.2012.02066.x

59.

Trigwell

Prosser

(2020). Exploring university teaching and learning: Experience and context. London: Palgrave Macmillan.

60.

Vermunt

J. D.

Donche

(2017). A learning patterns perspective on student learning in higher education: State of the art and moving forward. Educational Psychology Review, 29(2), 269–299. https://doi.org/10.1007/s10648-017-9414-6

61.

Viberg

Hatakka

Bälter

Mavroudi

(2018). The current landscape of learning analytics in higher education. Computers in Human Behavior, 89(12), 98–110. https://doi.org/10.1016/j.chb.2018.07.027

62.

Wilson

Fowler

(2005). Assessing the impact of learning environments on students’ study approaches: Comparing conventional and action learning designs. Assessment and Evaluation in Higher Education, 30(1), 85–99. https://doi.org/10.1080/0260293042003251770

63.

Yilmaz

(2017). Exploring the role of e-learning readiness on student satisfaction and motivation in flipped classroom. Computers in Human Behavior, 70(5), 251–260. https://doi.org/10.1016/j.chb.2016.12.085

64.

Zamarro

Nichols

Duckworth

D’Mello

(2018). Further validation of survey effort measures of relevant character skills: Results from a sample of high school students. EDRE working Paper No. 2018-07. http://doi.org/10.2139/ssrn.3265332

65.

Zhou

Winne

P. H.

(2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419. https://doi.org/10.1016/j.learninstruc.2012.03.004