Response of learning analytics to the online education challenges during pandemic: Opportunities and key examples in higher education

Abstract

Emerging technological advancements can play an essential role in overcoming challenges caused by the COVID-19 pandemic. As a promising educational technology field, Learning Analytics (LA) tools or systems can offer solutions to COVID-19 pandemic-related needs, obstacles, and expectations in higher education. In the current study, we systematically reviewed 20 papers to better understand the responses of LA tools to the online learning challenges that higher education students, instructors, and institutions faced during the pandemic. In addition, we attempted to provide key cases in which LA has been effectively deployed for various purposes during the pandemic in the higher education context. We found out several prominent challenges for stakeholders. Accordingly, learners needed of timely support and interaction, and experienced difficulty of time management. Instructors lacked pedagogical knowledge for online teaching. In particular, individual and collaborative assessment have been a challenge for them. Institutions have not been ready for a digital transformation and online teaching. In response to these challenges, LA tools have been deployed for the following opportunities: monitoring, planning online learning process, fostering learners’ engagement and motivation, facilitating assessment process; increasing interaction, improving retention, being easy to use. Understanding these promises can also give insight into future higher education policies.

Keywords

Learning analytics pandemic online learning policy making higher education

Introduction

The ongoing COVID-19 pandemic has dramatically disrupted higher education institutions all over the world. Therefore, as a response to the rapid spread of the pandemic, higher education institutions have suspended face-to-face education temporarily (Bao, 2020; Usher et al., 2021). For instance, at the end of April 2020, higher education institutions from approximately 180 countries closed their campuses (Marinoni et al., 2020). After the closure of the campus, the institutions had swiftly changed their instruction to online education. This is most probably because online education has been regarded as the safest and most responsible approach during the pandemic for keeping social isolation (Openo, 2020).

However, the rapid transition to online education has led to many challenges for all stakeholders including the institutions, teachers, and learners (Chetty et al., 2020; Hodges et al., 2020). For example, the institutions had to update and broaden their servers to meet the mass deployment of online learning environments (Balderas and Caballero-Hernández, 2020). In addition to these technical issues, the institutions had to struggle in providing the instructors with pedagogical support for online teaching. In fact, many instructors conceived online teaching and learning as “painful, worrisome, and anxiety-inducing” during the pandemic (Kimmons et al., 2020, p.2). For instance, the instructors failed to effectively monitor students’ behavioral, emotional, and cognitive aspects of learning because of the lack of an effectively designed system (Van der Spoel et al., 2020). Also, instructors were unable to assess and evaluate the learners due to students’ cheating behavior, which is hard to detect in online learning environments (Balderas and Caballero-Hernández, 2020). From the student perspective, the concern regarding cheating behavior was leading instructors to take immensely demanding exams (Lancaster and Cotarlan, 2021). It has been evidenced that higher education students also have dissatisfied with and challenged the online learning process (Ludlow, 2020). One possible reason for the dissatisfaction is regarded as non-stop online lectures throughout all day, causing Zoom fatigue in the students (Bonk, 2020). Therefore, pure online lectures that are far from meeting the psychological well-being, needs, and expectations of students may escalate the drop-out rate during pandemic (Baber, 2020; Diver and Martinez, 2015; Teuber et al., 2021). Furthermore, the pandemic process increased the inequality among the students. In this regard, Chetty et al. (2020) found out students with high socio-economic status outperform those of low socio-economic status during the pandemic. This is because students from low-income areas may lack the necessary technical equipment such as the Internet or personal computer; these students may receive less support from their parents (Zhang et al., 2021).

Emerging technological advancements can play an essential role in overcoming challenges caused by the pandemic (Richter, 2020). Indeed, this period might be a unique opportunity to better understand the implication of technological solutions in learning and teaching (Dwivedi et al., 2020; Sein, 2020). As an emerging technology field, Learning Analytics (LA) tools or systems can offer solutions to pandemic-related needs, obstacles, and expectations in higher education (O’Leary, 2020).

For example, dashboards as one of the most common LA tools can provide teachers with important information about their students, including time management, material use, and social interactions (Safsouf et al., 2021). Hence, LA dashboards can serve as a monitoring tool for students by creating and reporting several visualizations on student progress and learning outcomes during the pandemic (Conde et al., 2020). Instructors can straightforwardly recognize specific learning/support needs of students through LA dashboards; ultimately, this provides instructors with possibilities for timely and individualized interventions (He et al., 2021; Molenaar and Knoop van Campen, 2018). Moreover, with the help of machine learning algorithms, the dashboards can create predictive models based on student real-time data (Dabbebi et al., 2017; Diana et al., 2017). These models can be specified in the process of the pandemic, considering students’ needs and guidance (Safsouf et al., 2021). In addition to dashboards, LA tools might be utilized by instructors, for measuring and monitoring students’ individual and group performance and for assessing progress (Conde et al., 2020). Also, some LA tools have been developed for the detection of students’ cheating behavior during online exams (Balderas and Caballero-Hernández, 2020).

In order to better support learners with current and emerging technologies, there is a need to understand how LA has been utilized for mitigating pandemic-related challenges. In the current study, we systematically reviewed a selection of the papers to better understand the responses of LA tools to the online learning challenges that higher education students, instructors, and institutions faced during the pandemic. In addition, we attempted to provide key cases in which LA has been effectively deployed for various purposes during the pandemic in the higher education context. Having a positive understanding of the advantages regarding LA tools accelerates the adoption of LA (De Laet et al., 2020; Park and Jo, 2019). Then, it is likely that higher education stakeholders, who perceive LA as useful, can benefit from tools in future natural crises or emergency situations.

Learning Analytics in higher education

The extensive use of big data technologies enables us to collect a massive amount of data on the learning process (Gašević et al., 2016). This data has been regarded as a valuable asset and promising source of information to support and enhance effective learning behavior and played a role in the emergence of the field of LA (Siemens and Gasevic, 2012; Larrabee Sønderlund et al., 2019). LA is defined as the measurement, collection, analysis, and reporting of data about learners and their contexts to better understand and optimize the learning process (Siemens and Gasevic, 2012). Meanwhile, LA tools are designed to provide customized information for supporting the learning process by analysis of stored data when learners interact with the learning environment (Dyckhoff et al., 2012). Considering these definitions, online learning environments might be one of the most suitable platforms for the successful deployment of LA (Heath, 2021). In fact, students in online learning leave a large amount of data while interacting with course materials and assignments (Alexandron et al., 2019; Schwendimann et al., 2017). Therefore, these environments may contribute to data-driven decision-making in higher education, also in non-emergency days (Tsai et al., 2019; Zhang et al., 2021).

LA tools have provided possibilities and approaches to support learners in online and hybrid settings in higher education (e.g., Larsen et al., 2019). We argue that these tools could be used effectively to support higher education institutions, instructors and learners in post-pandemic higher education. To better understand the ways of how and the purposes to which these tools could be used, an overview of LA use during pandemic is needed. Understanding LA promises can also give insight for future higher education policies (Ifenthaler and Yau, 2020). In this context, the synthesis of the current study is important for higher education students, instructors, and policy-makers.

We addressed the following research questions (RQs) in the current study:

RQ1- What are the online education challenges that institutions, instructors, and students face during the COVID-19 pandemic?

RQ2- What are the advantages of LA with the purpose of optimizing online education during the COVID-19 pandemic?

RQ3- What are the key cases from the pandemic process in which LA tools have been effectively deployed in response to online education challenges during the COVID-19 pandemic?

Method

The manuscript search processes

We adopted the guidelines of systematic literature review procedure suggested by Kitchenham and Charters (2007). These guidelines start with a manuscript search process followed by employment of selection of criteria. Next, we performed analysis and synthesis of findings of the selected manuscripts. In the current study, four databases have been used for the manuscript search processes: ACM Digital-Library, IEEE XPLORE, Thomson Reuters’ Web of Science, Elsevier’s SCOPUS. These databases consist of several high-ranking journals (Shih, et al., 2008) including high-quality studies (Crescenzi-Lanna, 2020).

The search terms used comprised the following words and combinations: “Learning analytics,” “educational data mining,” “dashboard,” “visualization,” “remote teaching,” “remote learning,” “online learning,” “distance education,” “higher education,” “pandemic,” and “covid” (Table1). After accessing the articles, we employed a set of inclusion and exclusion criteria. Following the relevant criteria, we selected empirical (1) articles and conference papers that have been conducted in higher education context (2) during the COVID-19 pandemic (3). Lastly, one more inclusion criterion is the actual use of LA analytical tools as a response to online learning challenges. As the time period, the conference papers and articles (in English) published started from the first of January 2020 to first of September 2021. The editorials, reviews and the studies conducted in the K-12 context have been excluded. As a result of the application of the criteria, 20 articles and conferences remained for the further analysis and synthesis process. Table 1 indicates the search string for each database and the number of articles selected/accessed.

Table 1.

The search string used in each database and its results.

Data base	Search string (Selected/Accessed)
Web of Science TS = Topic. covers title, abstract and keywords	(((TS = (“learning analytics” or “analytic* " or “educational data mining")) AND TS=(“pandemic” OR “COVID”)) AND TS=(“remote teaching” or “remote learning” or “online learning” or “distance education")) AND TS=(“higher education” or “univers*") ((TS=(“automated feedback” or “dashboard” or “visualization")) AND TS=(“higher education")) AND TS=(“pandemic” or “covid”) (5/23)
Scopus TITLE-ABS-KEY = Title, abstract and keywords	TITLE-ABS-KEY ((((“learning analytics” or “analytic* “ or “educational data mining”) AND (“higher education” or “univers*”) AND (“remote teaching” or “remote learning” or “online learning” or “distance education”) AND (“pandemic” OR “covid”)))) (4/38)
ACM Digital-Library	“learning analytics” AND pandemic (6/52)
IEEE Xplore	“learning analytics” AND pandemic (5/45)

The data coding and analysis processes

In order to get an overview of the studies, we coded the selected articles in terms of data modality, analytical tool, country, and domain. Table 2 shows the overview of the studies on the response of LA to the online learning challenges during the COVID-19 pandemic. We performed the qualitative content analysis for these 20 studies. Considering that the preliminary or template coding scheme could unnecessarily direct researchers to the out of research aims (Şimşek and Yıldırım, 2011), we employed the open coding approach (Williamson, 2015) in the process of qualitative content analysis. For this, we firstly familiarized with the whole by picking a few of papers randomly and considered their basic meaning. Secondly, we defined preliminary thought in the margin, made a list of all thoughts, and grouped similar thoughts. In this process, we also made columns to differentiate key, unique, and leftover thoughts. Thirdly, after coding the text, we decided the most illustrative phrasing for our thoughts and turned them into categories. Fourthly, a certain abbreviation was decided for each category, to alphabetize these codes. Lastly, we incorporated the last code and employed initial analysis and recode, if necessary. The first author initially completed coding the studies individually and then shared the codes with the second and third authors. Disagreements were negotiated by checking the code list and the relevant studies. Some categories were merged and renamed. Finally, we recorded the 20 studies according to the latest code list (Table 2).

Table 2.

An analysis of studies on the response of Learning Analytics to the COVID-19 pandemic (n = 20).

Author(s)	Article title	Data modality	Analytic tool	Domain	Country
Dias et al. (2020)	DeepLMS: a deep learning predictive model for supporting online learning in the Covid-19 era	Discourse Log	Moodle	Engineering	Greece UAE
Liang et al. (2020)	Opportunities for Improving the Learning/Teaching Experience in a Virtual Online Environment	Video Discourse	Moodle Sutori	ComputerSci Engineering	China Japan
Lapitan et al. (2021)	An effective blended online teaching and learning strategy during the COVID-19 pandemic	Self-reported Log	Blackboard Moodle	Chemistry	Philippines
Campanyá et al. (2021)	Mixed Analysis of the Flipped Classroom in the Concrete and Steel Structures Subject in the Context of COVID-19 Crisis Outbreak. A Pilot Study	Self-reported Log	Blackboard Moodle	Engineering	Spain
Mata et al., (2021)	How to Teach Online? Recommendations for the assessment of online exams with University students in the USA in times of pandemic	Performance	Code of Conducts	Business	USA
Anteby et al. (2021)	Development and Utilization of a Medical Student Surgery Podcast During COVID-19	Log	Podcast	Medicine	Israel
Tabuenca et al., (2021)	Mind the gap: smoothing the transition to higher education fostering time management skills	Performance log self-reported	Moodle	ComputerSci	Spain
Farah et al. (2020)	Bringing computational thinking to non-STEM undergraduates through an integrated notebook application	Clickstream video	CodeApp Dashboard	Business	Switzerland
Prat et al. (2021)	A Methodology to Study the University’s Online Teaching Activity from Virtual Platform Indicators: The Effect of the Covid-19 Pandemic at Universitat Politècnica de Catalunya	Log Discourse	Moodle Atenea	Mixed	Spain
Zhang et al. (2021)	Measuring the Impact of COVID-19 Induced Campus Closure on Student Self-Regulated Learning in Physics Online Learning	Clickstream Log	e-Learning Modules	Physics	USA
Safsouf et al. (2021)	TABAT: Design and Experimentation of A Learning Analysis Dashboard for Teachers And Learners	Log	Moodle Dashboard	Engineering Business	Morocco
Günther (2021)	The impact of social norms on students’ online learning behavior: Insights from two randomized controlled trials	Log Discourse	Dashboard	ComputerSci	Germany
Balderas Caballero-Hernández (2020)	Analysis of Learning Records to Detect Student Cheating on Online Exams: Case Study during COVID-19 Pandemic	Video	Moodle Py-Cheat	Engineering ComputerSci	Spain
Conde et al. (2020)	A Learning Analytics tool for the analysis of students’ Telegram messages in the context of teamwork virtual activities	Discourse	Heroku	ComputerSci	Spain
Perez-Sanagustin et al. (2021)	Can Feedback based on Predictive Data Improve Learners’ Passing rates in MOOCs? A Preliminary Analysis	Log Clickstream	DapMOOC	ComputerSci	Chile
Zikas (2021)	Covid-19—VR Strikes Back: innovative medical VR training	Log	CVRSB	Medicine	Switzerland
Naidoo and Naidoo (2021)	Linking Formative and Summative Performance in an online L2 module: Insights from Learning Analytics	Log	Moodle	Mixed	S.Africa
Maher et al., (2020)	Learners on Focus: Visualizing Analytics Through an Integrated Model for Learning Analytics in Adaptive Gamified E-Learning	Log Discourse	Moodle	Mixed	Egypt
Rahmah et al. (2020)	Developing Distance Learning Monitoring Dashboard with Google Sheet: An Approach for Flexible and Low-Price Solution in Pandemic Era	Observation discourse assignment	Google Sheet	Mixed	Indonesia
Liu et al. (2020)	MapOnLearn: The Use of Maps in Online Learning Systems for Education Sustainability	Self-reported	MaponLearn Dashboard	English Math	China

According to Table 2, log data was found to be the main source of data for the LA tool. Also, Moodle and dashboards have been the most deployed tools for analytical purposes. LA tools have been commonly utilized in engineering and computer science departments.

Findings and discussion

Online distance education challenges (RQ1)

We found a number of reported challenges that stakeholders faced during online distance education in the COVID-19 pandemic. We addressed these challenges in terms of higher education stakeholders, namely, learners, instructors and institutions (see Table 3).

Table 3.

The challenges that stakeholders faced during online distance education.

Stakeholders	Challenges (f)
Learner (f = 19)	Lack of timely support (f = 9)
	Lack of interaction (f = 3)
	Having difficulty of time management (f = 3)
	Lack of motivation for engagement (f = 2)
	Lack of digital literacy (f = 1)
	Increasement of digital inequalities (f = 1)
Instructor (f = 9)	Individual and collaborative assessment (f = 4)
	Assessment reliability (f = 2)
	Lack of pedagogical knowledge for online teaching (f = 1)
	Stress from heavy dependence on technologies (f = 1)
	Less chance for real time monitoring of learners (f = 1)
Institutions (f = 6)	Not being ready for digital transformation (f = 2)
	Providing pedagogical support for online teaching (f = 2)
	Insufficient infrastructure (f = 2)

According to the results of our review, the most commonly reported problem for students was lack of timely support. For instance, Maher et al. (2020) revealed that students needed support for their learning but could not receive it sufficiently during online education. The support is needed because students were less familiar with online learning before pandemic (Landrum et al., 2021). Students had also struggled to understand course goals and instructor’s expectations; thus, they had a sense of ambiguity about the assessment process of their online learning (Platt et al., 2014). Similarly, lack of interaction appeared to be another challenge from the learner side. In fact, having less interaction chance is an important factor negatively affecting learners’ online learning satisfaction (Cole et al., 2014).

We also found that having difficulty with time management was a prominent challenge for students (e.g., Tabuenca et al., 2021). This result is similarly evidenced with another study conducted during pandemic (Barrot et al., 2021). During COVID-19 pandemic, empirical studies have shown that poor quality Internet access and less digital literacy skills were associated with lower use of the Internet for educational purposes (Van Deursen, 2020). Similarly, in the current review, we reported students’ digital inequalities to properly access learning content and lack of digital skills to utilize online environments as two important challenges (e.g., Günther, 2021; Rahmah et al., 2020).

From the instructor`s side, our review revealed that assessment was a challenging issue for online learning (Conde et al., 2020; Naidoo and Naidoo, 2021). Therefore, reliability of online exam results has been a conflicting issue for instructors. We also observed that instructors rarely had trust in the reliability level of the online assessment process (Balderas and Caballero-Hernández, 2020). For instructors, this may be due to the decreased chance/opportunity for real time monitoring of learners. Similar to students, higher education instructors were not ready for online learning; therefore, we found that instructors were far from delivering best quality online lecture pedagogically and technically (e.g., Rahmah et al., 2020).

It has been also found that the online learning process due to the pandemic situations lead to stressful situations for in the higher education instructors (Boyer-Davis, 2020). One of the reasons for this stress was depending on online education technologies without necessary digital skills. In the current review, we also observed that instructors had stress from heavy dependence on online learning technologies (Maher et al., 2020).

The advantages of LA for online distance education challenges (RQ2)

As a result of the qualitative analysis, nine main categories were formed for the opportunities: providing timely support (1), automated adaptive feedback (2), planning online learning process (3), fostering learners’ engagement and motivation (4), monitoring (5), facilitating assessment process (6), increasing interaction (7), improving retention (8), easy to be deployed (9). We then grouped each main category into separate sub-categories, based on where the reported opportunity naturally fit (see Table 4). It is important to note that some papers showed more than one opportunity. For such papers, more than one code was assigned for the opportunities. We also used “+” to indicate which stakeholders benefited from the relevant opportunity.

Table 4.

Opportunities of LA tools for online distance education challenges.

Opportunities		Relevant stakeholders
Categories (f)	Sub-categories (f)	Learners	Instructor	Institution
Monitoring (f = 12)	Instructor monitoring (f = 5)		+
	Early detection of students’ learning outcome (f = 3)	+	+
	Learner self-monitoring (f = 4)	+
Planning online learning process (f = 12)	resource allocation (f = 2)		+	+
	set up learning goals (f = 6)	+	+
	setting completion times and conditions (f = 3)	+	+
Fostering learners’ engagement and motivation (f = 7)		+
Facilitating assessment process (f = 6)	formative (f = 3)		+
	summative (f = 1)		+
	individual (f = 1)		+
	group (f = 1)		+
Increasing interaction (f = 2)	student-student (f = 1)	+
Increasing interaction (f = 2)	student-instructor (f = 1)	+	+
Improving retention (f = 2)		+	+	+
Being easy to use (f = 2)		+	+	+

According to results from the reviewed papers, two of the most remarkable opportunities of LA tools were that the tools supported learners and instructors in the planning the online learning process and monitoring. In the process of planning, learners and instructors can determine learning goals (Tabuenca et al., 2021) and set up completion times and conditions (Liang et al., 2020) by utilizing LA tools. This helps the learners with self-regulating their learning and the instructors with pedagogical planning (e.g., Zhang et al., 2021). Also, LA tools helped the institutions’ and instructors’ resource allocation during planning online learning (e.g., Anteby et al., 2021).

Instructors benefited from monitoring opportunities when they monitor their students during a lecture or the whole online course (Farah et al., 2020) or when they receive information of students’ early detected outcome (Dias et al., 2020). In addition, students can monitor their own online learning progress using LA tools (e.g., Lapitan et al., 2021). As a result of our review, providing timely support was found to be a crucial opportunity for LA tools (e.g., Liu et al., 2020). Given that one of the challenging issues for online learning is lack of support, LA tools can be a solution for this challenge. Similarly, the opportunity of automated adaptive feedback from LA tools can save teachers’ time and may help learners when they have a difficulty (Günther et al., 2021). This adaptive feedback promotes personalized learning.

According to our findings, LA tools might play an important role in fostering learners’ engagement and motivation in online learning environments. In their study, Safsouf et al., (2021) identified that students shared their positive experience when they used the LA dashboard. In particular, students might have positive feelings when they monitor their learning progress through various useful visualizations. Also, students are more satisfied with having a chance of setting up their learning goal through the reminder systems (Safsouf et al., 2021). When students face tough conditions such as pandemic, it is difficult for them to be highly motivated for learning and ultimately engage in the learning process (Mishra et al., 2020). For this, successful deployment of LA tools offer opportunities for supporting their engagement (Chen et al., 2021).

LA facilitated online learning assessment was identified as an important support for instructors during pandemic. Specifically, the formative (Lapitan et al., 2021) and/or summative (Mata et al., 2021) assessment can be performed with the help of LA tools. For instance, Naidoo and Naidoo (2021) utilized visualizations of students’ learning progress, which is beneficial for formative assessment. Further, it is possible to use LA individual (Balderas and Caballero-Hernández, 2020) and group assessment (Conde et al., 2020) in the process of online learning. Another LA opportunity is found to be the increment of student-student (Maher et al., 2020) and students-instructor interaction (Campanyà et al., 2021). One possible explanation for increment in students-instructor interaction is the advantages of LA tools to teachers for understanding students' learning progress. This contributes to timely support from teachers, eventually resulting in the students-instructor interaction (Hernández-Lara et al., 2019).

Some studies reported that LA tools have benefits for improving students' retention as a response to increased dropping-out risk (Perez-Sanagustin et al., 2021). Instructors generally have time limitation for detecting students at-risk and offering adaptive feedback for students (Günther, 2021). Students at-risk who cannot receive adaptive support are more likely to drop out the course (Martin et al., 2020). In such times, LA tools with the advantages of timely support and intervention can increase student retention (Perez-Sanagustin et al., 2021). It has been also reported that LA tools were perceived as easy to use (Maher et al., 2020; Liu et al., 2020). Hence, this could be regarded as an opportunity for deployment.

Key case examples regarding deployment of LA tools during the pandemic (RQ3)

Anteby et al. (2021), aimed at exploring the cost and effectiveness of implementation of a set of podcasts for medical students in online distance education during the COVID-19 pandemic. The podcast series were released on a website and well-known podcast platforms. A total of 10 podcast episodes were available for free download within 9 months starting from March 2020. Podcast analytics acquired from transistor.fm that is a hosting platform for measuring audience and interest. The measuring is based on a set of metrics such as the number of downloads per episode by the day. Utilizing podcast analytics, the researchers conducted a cost-analysis process of podcast-supported online teaching methods. As a result of this process, it has been estimated that the total cost to create podcast series is comparatively less than other educational modalities. Hence, podcast analytics as a LA tool might offer premises for decision-making on the efficacy and effectiveness of a teaching and content delivery method.

In a study by Balderas and Caballero-Hernández (2020), a LA tool has been developed to detect students’ cheating behavior when higher education students take online exams. For this, the students’ data on the Moodle platform has been obtained. Then, this data has been transformed into a spreadsheet format and analyzed using process mining (Disco) and cheating detection software (By-Cheat). After the analysis, a variety of reports and visualizations have been formed representing students’ information (grade, starting exam time, IP address from the student`s location, resources accessed during the exam, etc.). The LA tool has been deployed in the Degree in Computer Science and Engineering of the University of Cadiz (Spain). Results indicated that students have taken online exams in groups, so they have been seeking the answers to the questions among a few. It has been detected that some students who prefer taking the exam later, obtained higher grades within less time. This study highlighted that LA tools might be utilized for contributing to a valid and reliable assessment for distance education during or even after the pandemic.

Safsouf et al. (2021), utilized a LA dashboard entitled TaBAT that enabled instructors and students to monitor students’ learning process with numerous visualizations. Specifically, the dashboard provided instructors with descriptive and/or predictive graphs regarding students. These graphs and visualizations helped to understand students’ engagement, academic performance and probability of drop-out. This process led to timely interventions that might promote learners’ self-reflection, autonomy, and academic performance. In the development process of TaBAT, researchers used a set of indicators (e.g., course, participation, success) for the three types of notifications, namely, students in difficulty, students in risk of dropping-out, and recommendations. According to the authors, automated feedback and timely support offered by LA tools such as TaBAT dashboard might play an important role in reducing drop-out rate at online distance education.

In another study, a LA tool has been created and employed with the purposes of evaluation of students’ interactions (Conde et al., 2020). The tool has been tested in a computer animation course during the pandemic. The students worked in groups of four members for a collaborative task. It was possible to monitor and assess student participation and interaction by using the number of short and long messages. Further, the number of initiated conversations was considered as an indicator of student leadership. Therefore, the Telegram Instant Messaging Tool facilitated teachers to better understand the individual acquisition of teamwork competence. The results pointed out that the usage of the tools was associated with better academic performance. Higher education students were willing to use Telegram Instant Messaging Tool, since they were familiar with instant messaging software. According to Conde et al. (2020), the individual assessment from a collaborative work during online education requires online tools. This study indicated that LA tools revealed the frequency and contributions from students’ messages which can be an important asset for collaborative work evaluation.

Maher et al. (2020) integrated gamification principles into the LA dashboard. In particular, they suggested the Personalized Adaptive Gamied E-learning (PAGE) model as a response to online learning challenges such as learner’s lack of motivation and lack of enjoyment. The suggested model is designed as independent of any domain so that it fits any sort of course for learners with different backgrounds. These flexible options enable teachers to make a dynamic course plan or/and practice. In addition, the PAGE model supports learners in overcoming the miscommunication challenges caused by distance learning. The results presented that LA dashboard supported with gamification principles had the potential for personalizing the higher education courses to each individual learner. During pandemic times, it is necessary to make the learning process as enjoyable as possible especially when higher education students feel isolated. Hence, the study of Maher et al. (2020) offers empirical evidence that using gamificated-LA dashboard might support instructors for making the courses more interesting.

Future studies and implications for post-pandemic

Previous review studies indicated opportunities of LA in higher education (e.g., Viberg et al., 2018; Tsai and Gasevic, 2017). However, the educational needs and expectations of stakeholders are most likely to vary in crisis situations such as pandemic. Therefore, the deployment of LA in higher education can differentiate compared to regular life conditions. In this regard, it is crucial to understand the role of LA in mitigating the learning and teaching challenges related to pandemic. The current study is the first attempt to provide an overview on the usage of the LA tools for overcoming online education challenges during COVID-19 pandemic. The implications of the current study might be important for post-pandemic policies of the higher education institutions. In fact, successful deployment of LA can be inspiring for another educational context such as K-12 level. For instance, when pre-service teachers increase their awareness of LA benefits in the higher education, they can utilize these tools in their professional life, specifically in K-12 context.

Based on our findings students have faced the most difficulties during online learning in COVID-19 pandemic. Further, the instructors have been identified as the stakeholder group potentially most benefiting from LA services. Thus, this leads to believe that LA has the potential to provide solutions to the challenges that institutions and especially instructors face during crises now and in the future. However, the nature of students’ difficulties remains a challenge to address. From the student perspective, lack of timely support for learning activities, lack of human interaction and time management difficulties were the most prominent problems in the online education. As responses to these challenges, LA tools served as supportive tools for monitoring and planning online learning process and increasing their motivation for engagement. In light of this result, we suggest that the institutions and policy-makers should encourage the instructors and students to use LA tools effectively to overcome interaction, planning and time management challenges in particularly crisis time. According the results from our synthesis, instructors had difficulties in terms of individual and group assessment reliability of those assessments. In addition, they had less chance for real time monitoring of learners. For the difficulties that the instructors had, LA offers opportunities such as detecting students at-risk as a function of monitoring and facilitating assessment process. Moreover, the current study pointed out instructors might use some LA tools for more valid and reliable online exams. As seen from the current and previous studies, higher education institutions are not ready for digital transformation with insufficient infrastructure. This might lead to insufficient online learning management. Particularly, the current study showed that universities failed to provide the instructors with pedagogical support for online teaching during pandemic. We argue that preparing the instructors for giving online lecture should be agenda in the post-pandemic higher education policies. In this preparation, the opportunities of LA tools for online learning should be introduced to instructors. This can lessen instructors’ stress stem from intense use of technologies. However, online education challenges identified at the institutional level also indicated that similar initiatives are needed to support learners. For example, improvement of the data infrastructure and improving the readiness for digital transformation could involve actions such as providing students with the platforms that enable students’ study planning, monitoring and time management, or instructors’ online monitoring and reliable assessment of student work during online courses. These implications might be considerable for post-pandemic higher education.

The results of this study are in line with the previous findings of LA opportunities and challenges in higher education context. As identified in this study, previous studies have identified LA tools opportunities to support learners’ self-regulation of learning, including study planning and monitoring (e.g., Matcha et al., 2019), as well utilization of LA tools as an instructors support for pedagogical design, monitoring and assessment of online learning processes (e.g., Molenaar and Knoop van Campen, 2018). Additionally, institutional leadership and investment on the development of data infrastructure and development of LA tools has been identified as one of the key issues in implementing high quality LA supported practices and processes in educational institutions (Tsai et al., 2019).

Despite addressing a topical and highly relevant area of research, this study has several limitations. First, extant literature focuses on the topic of LA and challenges experienced by the stakeholders of higher education are analyzed in this context. Literature reviews with different foci might uncover a variety of other online learning challenges, which would be important to take into account for a holistic perspective needed to organize distance education in the future. Some challenges such as difficulties with time management experiences by students have so far been only addressed by LA to a very limited extent. We see further possibilities for exploration in this area, as problems with time management will likely remain a challenge after a pandemic as well, for example, in organized hybrid education. Other identified challenges, such as a lack of social interactions, need to be investigated as a complex problem of students’ well-being and the LA role in solving this issue has to be evaluated critically. We, however, see possibilities in this largely unaddressed area with involvement of other institutional resources, such as academic advisors, whose role in LA utilization has so far been underexplored. For future research, the impact of implementing LA tools in post-pandemic education as supporting teacher and student engagement, fluency of institutional practices and student learning should be explored.

Conclusions

According to the results of this study, LA as an emerging technology provides promising solutions to tackle the online education challenges faced by higher education institutions during COVID-19 pandemic. However, it is still unclear, what kind of educational practices higher education institutions will build for post-pandemic time. LA tools should be further investigated in the context of online and hybrid education practices. Key finding from our study is that while students were identified to experience the greatest number of challenges during distance education, instructors were potentially greatest beneficiaries from the LA tools. Key implication from this finding is the need for the institutions to initiate, supervise, and continually examine that investments into LA tools for teachers translates into support for students. Hence, the results of this study show that educational institutions have a responsible and critical role in understanding the challenges and needs of their stakeholders and in exploration of LA tools to address unique needs of these stakeholders in post-pandemic educational practices.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Ismail Celik

Anni Silvola

Author Biographies

Ismail Celik is a postdoctoral researcher in the Learning and Learning Processes Research Unit at the Faculty of Education, University of Oulu (Finland). He holds a PhD in educational technology. His research areas cover social media use, technology integration models in education, epistemological beliefs, artificial intelligence, and learning analytics.

Egle Gedrimiene is a PhD student in Faculty of Education, at the University of Oulu, Finland. She has a background in psychology and is currently involved in research project investigating technology supported transitions in education. Her research interests are career guidance, self-efficacy and learning analytics.

Anni Silvola is a PhD student at the Faculty of Education, at the University of Oulu, Finland. Her doctoral thesis focuses on researching how to support study engagement with learning analytics during university studies. Her research interests are study engagement and competence development during higher education studies, collaborative learning and learning analytics. Currently, she is working on a national learning analytics research and development project, funded by the Ministry of Culture and Education

Hanni Muukkonen is a professor in Educational Psychology at the Faculty of Education, University of Oulu (Finland). She holds a PhD in psychology. Her research has addressed collaborative learning and knowledge creation in higher education, design for learning and technology-mediated collaboration. Technology design and development of related knowledge practices has continued through multiple large R&D projects in educational technology and learning analytics.

References

*References marked with an asterisk show the articles used in the review

Alexandron

Yoo

Ruipérez-Valiente

, et al. (2019) Are MOOC learning analytics results trustworthy? With fake learners, they might not be! International Journal of Artificial Intelligence in Education 29(4): 484–506. DOI: 10.1007/s40593-019-00183-1.

* Anteby

Amiel

Cordoba

, et al (2021) Development and utilization of a medical student surgery podcast during COVID-19. Journal of Surgical Research 265: 95–99.

Baber

(2020) Determinants of students’ perceived learning outcome and satisfaction in online learning during the pandemic of COVID19. Journal of Education and E-Learning Research 7(3): 285–292. DOI: 10.20448/journal.509.2020.73.285.292 .

* Balderas

Caballero-Hernández

(2020) Analysis of learning records to detect student cheating on online exams: case study during COVID-19 Pandemic. In: The Eighth International Conference on Technological Ecosystems for Enhancing Multiculturality, 752–757. DOI: 10.1145/3434780.3436662 .

Bao

(2020) COVID ‐19 and online teaching in higher education: a case study of Peking university. Human Behavior and Emerging Technologies 2(2): 113–115. DOI: 10.1002/hbe2.191 .

Barro

Llenares II Del Rosario

(2021) Students’ online learning challenges During the pandemic and how they cope with them: the case of the Philippines. Education and Information Technologies online ahead of print, 1–18. DOI: 10.1007/s10639-021-10589-x .

Bonk

(2020) Pandemic ponderings, 30 years to today: Synchronous signals, saviors, or survivors? Distance Education 41(4): 589–599.

Boyer-Davis

(2020) Technostress in higher education: An examination of faculty perceptions before and during the COVID-19 pandemic. Journal of Business and Accounting 13(1): 42–58.

(2021) Mixed Analysis of the Flipped Classroom in the Concrete and Steel Structures Subject in the Context of COVID-19 Crisis Outbreak. A Pilot Study. Sustainability 13(11): 5826.

(2021) Promoting student engagement in online collaborative writing through a student‐facing social learning analytics tool. Journal of Computer Assisted Learning 1–17, DOI: 10.1111/jcal.12604 .

, et al. (2020) The Economic Impacts of COVID-19: Evidence from a New Public Database Built from Private Sector Data, Opportunity Insights

(2014) Online instruction, e-learning, and student satisfaction: a three-year study. The International Review of Research in Open and Distributed Learning 15(6): 111–131.

, et al. (2020) A learning analytics tool for the analysis of students’ Telegram messages in the context of teamwork virtual activities. In: The Eighth International Conference on Technological Ecosystems for Enhancing Multiculturality, 719–724.

15.

Crescenzi‐Lanna

(2020) Multimodal learning analytics research with young children: a systematic review. British Journal of Educational Technology 51(5): 1485–1504, DOI: 10.1111/bjet.12959 .

, et al. (2017) Towards adaptive dashboards for learning analytic: An approach for conceptual design and implementation. In: The 9th International Conference on Computer Supported Education. CSEDU, 120–131.

, et al. (2020) Adoption and impact of a learning analytics dashboard supporting the advisor-Student dialogue in a higher education institute in Latin America. British Journal of Educational Technology 51(4): 1002–1018. DOI: 10.1111/bjet.12962 .

, et al. (2017) An instructor dashboard for real-time analytics in interactive programming assignments. In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference, 272–279.

, et al. (2020) DeepLMS: a deep learning predictive model for supporting online learning in the Covid-19 era. Scientific Reports 10(1): 1–17. DOI: 10.1038/s41598-020-76740-9 .

20.

Diver

Martinez

(2015) MOOCs as a massive research laboratory: opportunities and challenges. Distance Education 36(1): 5–25. DOI: 10.1080/01587919.2015.1019968 .

, et al. (2012) Design and implementation of a learning analytics toolkit for teachers. Journal of Educational Technology and Soceity 15(3): 58–76.

(2020) Impact of COVID-19 pandemic on information management research and practice: Transforming education, work and life. International journal of information management 55: 102211.

, et al. (2020) Bringing computational thinking to non-STEM undergraduates through an integrated notebook application. In: The 15th European Conference on Technology Enhanced Learning. (No. CONF).

(2016) How do we start? State and directions of learning analytics adoption. 2016 ICDE Presidents Summit, December, 1–24. DOI: 10.13140/RG.2.2.10743.42401.

25.

Günther

(2021) The impact of social norms on students’ online learning behavior: insights from two randomized controlled trials. In: LAK21: 11th International Learning Analytics and Knowledge Conference, 12–21, DOI: 10.1145/3448139.3448141 .

26.

Zhang

(2021) Information technology solutions, challenges, and suggestions for tackling the COVID-19 pandemic. International Journal of Information Management 57: 102287, DOI: 10.1016/j.ijinfomgt.2020.102287 .

27.

Heath

(2021) Buried treasure or Ill-gotten spoils: the ethics of data mining and learning analytics in online instruction. Educational Technology Research and Development 69(1): 331–334.

(2019) Applying learning analytics to students’ interaction in business simulation games the usefulness of learning analytics to know what students really learn. Computers in Human Behavior 92: 600–612.

, et al. (2020) Remote teaching and online learning. Educause Review 1–15.

(2020) Utilising learning analytics to support study success in higher education: a systematic review. Educational Technology Research and Development 68(4): 1961–1990. DOI: 10.1007/s11423-020-09788-z.

(2020) What (some) faculty are saying about the shift to remote teaching and learning. Educause Review. https://er.educause.edu/blogs/2020/5/what-some-faculty-are-saying-about-the-shift-to-remote-teaching-and-learning.

32.

Kitchenham

Charters

(2007) Guidelines for Performing Systematic Literature Reviews in Software Engineering. Staffordshire, Durham: EBSE 2007-001 Technical Report. Keele University and Durham University Joint Report.

33.

* Lapitan

Jr Tiangco

Sumalinog

DAG

, et al. (2021) An effective blended online teaching and learning strategy during the COVID-19 pandemic. Education for Chemical Engineers 35: 116–131. DOI: 10.1016/j.ece.2021.01.012.

34.

Lancaster

Cotarlan

(2021) Contract cheating by STEM students through a file-sharing website: a Covid-19 pandemic perspective. International Journal for Educational Integrity 17(1): 1–16. DOI: 10.1007/s40979-021-00070-0 .

, et al. (2021) A class of one: students’ satisfaction with online learning. Journal of Education for Business 96(2): 82–88.

(2019) The efficacy of learning analytics interventions in higher education: a systematic review. British Journal of Educational Technology 50(5): 2594–2618. DOI: 10.1111/bjet.12720.

(2020) Opportunities for improving the learning/teaching experience in a virtual online environment. In: 2020 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE). IEEE, 243–250.

, et al. (2020) MapOnLearn: the use of maps in online learning systems for education sustainability. Sustainability 12(17): 7018. DOI: 10.3390/su12177018 .

. (2020) Learners on focus: visualizing analytics through an integrated model for learning analytics in adaptive gamified e-learning. IEEE Access 8: 197597–197616.

(2020) The impact of Covid-19 on higher education around the world. IAU global survey report 23.

(2020) A synthesis of systematic review research on emerging learning environments and technologies. Educational Technology Research and Development 68(4): 1613–1633. DOI: 10.1007/s11423-020-09812-2 .

42.

Mata

(2021) How to Teach Online? Recommendations for the assessment of online exams with University students in the USA in times of pandemic. IJERI: International Journal of Educational Research and Innovation 15(4): 188–202.

(2019) Detection of learning strategies: A comparison of process, sequence and network analytic approaches. In European conference on technology enhanced learning (pp. 525–540). Springer, Cham.

(2020) Online teaching-learning in higher education during lockdown period of COVID-19 pandemic. International Journal of Educational Research Open 1: 100012.

45.

Molenaar

Knoop-van Campen

(2018) How teachers make dashboard information actionable. IEEE Transactions on Learning Technologies 12(3): 347–355.

46.

* Naidoo

Naidoo

(2021) Linking Formative and Summative Performance in an online L2 module: Insights from Learning Analytics. In: 2021 9th International Conference on Information and Education Technology (ICIET). IEEE, 227–231.

47.

O’Leary

(2020) Evolving information systems and technology research issues for COVID-19 and other pandemics. Journal of Organizational Computing and Electronic Commerce. DOI: 10.1080/10919392.2020.1755790 .

48.

Openo

(2020) Education’s Response to the COVID-19 Pandemic Reveals Online Education’s Three Enduring Challenges. Canadian Journal of Learning and Technology 46(2). DOI: 10.21432/cjlt27981 .

49.

Park

I-H

(2019) Factors that affect the success of learning analytics dashboards. Educational Technology Research and Development 67(6): 1547–1571. DOI: 10.1007/s11423-019-09703-1 .

, et al. (2021) Can feedback based on predictive data improve learners’ passing rates in MOOCs? A preliminary analysis. In: Proceedings of the Eighth ACM Conference on Learning@ Scale, 339–342. DOI: 10.1145/3430895.3460991 .

, et al. (2021) A methodology to study the university’s online teaching activity from virtual platform indicators: the effect of the covid-19 pandemic at universitat politècnica de catalunya. Sustainability 13(9): 5177.

(2014) Virtually the same?: student perceptions of the equivalence of online classes to face-to-face classes. Journal of Online Learning & Teaching 10(3): 489–503.

, et al. (2020) Developing distance learning monitoring dashboard with google sheet: an approach for flexible and low-price solution in pandemic era. In: 2020 International Conference on ICT for Smart Society (ICISS). IEEE, 1–6. DOI: 10.1109/ICISS50791.2020.9307558 .

54.

Richter

(2020) Locked-down digital work. International Journal of Information Management Article 102157 55: 102157. DOI: 10.1016/j.ijinfomgt.2020.102157 .

(2021) TaBAT: design and experimentation of a learning analysis dashboard for teachers and learners. Journal of Information Technology Education: Research 20(1): 331–350. DOI: 10.28945/4820 .

, et al. (2017) Perceiving learning at a glance: a systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies 10(1): 30–41. DOI: 10.1109/TLT.2016.2599522 .

57.

Sein

(2020) The serendipitous impact of COVID-19 pandemic: A rare opportunity for research and practice. International Journal of Information Management Article 102164 55: 102164, DOI: 10.1016/j.ijinfomgt.2020.102164 .

(2008) Research and trends in the field of e-learning from 2001 to 2005: a content analysis of cognitive studies in selected journals. Computers & Education 51(2): 955–967.

59.

Siemens

Gasevic

(2021) Learning analytics special issue. Journal of Educational Technology & Society 15(3): 1–2, http://www.ifets.info/journals/15_3/ets_15_3.pdf.

60.

Şimşek

Yıldırım

(2011) Qualitative Research Methods in Social Sciences. Ankara: Seçkin Publishing.

61.

Tabuenca

Greller

Verpoorten

(2021) Mind the gap: smoothing the transition to higher education fostering time management skills. Universal Access in the Information Society 1–13.

62.

Teuber

Jia

Niewöhner

(2021) Satisfying students’ psychological needs during the covid-19 outbreak in German higher education institutions. Frontiers in Education 6: 679695.

63.

Tsai

Gasevic

(2017) Learning analytics in higher education---challenges and policies: a review of eight learning analytics policies. In: Proceedings of the seventh international learning analytics & knowledge conference, 233–242.

64.

Tsai

Poquet

Gašević

, et al. (2019) Complexity leadership in learning analytics: drivers, challenges and opportunities. British Journal of Educational Technology 50(6): 2839–2854.

65.

Usher

Hershkovitz

Forkosh‐Baruch

(2021) From data to actions: Instructors' decision-making based on learners' data in online emergency remote teaching. British Journal of Educational Technology. DOI: 10.1111/bjet.13108.

, et al. (2020) Teachers’ online teaching expectations and experiences during the Covid19-pandemic in the Netherlands. European Journal of Teacher Education 43(4): 623–638.

67.

van Deursen

. Digital inequality during a pandemic: quantitative study of differences in COVID-19-related internet uses and outcomes among the general population. Journal of Medical Internet Research 2020; 22(8): e20073.

68.

Williamson

. I Wasn’t Reinventing the Wheel Just Operating the Tools”: The Evolution of the Writing Processes of Online First-Year Composition Students (Unpublished Doctoral Dissertation). USA: Arizona State University, 2015.

69.