Sage Journals: Discover world-class research

Abstract

Discussion of the effectiveness of distance learning as a means of delivering higher education programs at classical universities has been ongoing for the past decade. The article presents the findings of a study of changes in academic performance of university students, covering the period from fall 2018 to fall 2023. This period included a rapid switch to online learning at Russian universities. Our study is based on a large dataset generated through the erp system of the national university of science and technology misis (nust misis). The dataset contains over 600,000 students’ academic performance entries covering every subject and practical class, including the final assessments. In addition to the abovementioned dataset, the study also used a database of students’ unified state examination (use) results. The results of use exams (taken during the final year of high school) determine the applicant’s eligibility for admission to university programs. We evaluate the impact of an individual student’s use results on their future academic performance at the undergraduate level. The study also analyzes the overall readiness of a university for distance learning at the time of the covid-19 pandemic outbreak. Excel pivot tables were used as a statistical analysis tool; shewhart charts were employed to analyze the academic performance trends. Our analysis confirmed the initial hypothesis—that there exists a potential negative impact of online learning on the academic performance of university students. The use of big data in learning analytics enhances the reliability of the results to the greatest extent possible.

Keywords

higher education distance learning academic performance digital learning tools teaching and learning management measurement and evaluation COVID-19

Introduction

The total transition to online education at universities around the world occurred in early 2020 as various countries implemented restrictions to combat the spread of COVID-19 (Edelhauser & Lupu-Dima, 2021; Khan et al., 2021; Lo et al., 2021). This led to a significant number of studies examining the transition to online learning in universities (Slykerman et al., 2022; Ukhurebor et al., 2024), effectiveness (Nuryatin et al., 2022) and the outcomes of such a transition (Gautam & Gautam, 2021; Radaev, 2022). A significant number of research studies were conducted in the 2 years following the outbreak of the pandemic. The main concerns addressed in these studies address university readiness for online teaching (Aithal & Aithal, 2016; Jena, 2020; Sinha, 2020; Xia et al., 2022), tools and techniques for organizing online learning (Ghilay, 2019; Morze & Trybulska, 2021; Moralista & Oducado, 2020), and student satisfaction with online learning (Martin & Bolliger, 2022; Moralista & Oducado, 2020; Olmos-Gómez et al., 2023). Topics receiving less attention and analysis in these studies include the academic performance of students after the transition to online education in a classical university format. We use the term “classical universities” to mean all accredited higher education institutions (Prokopev, 2000). The results of the latter group of studies are contradictory. Some studies have revealed an increase in academic performance during the transition to online learning (Boyarshinova & Karaseva, 2021; HSE Research, 2021; Pogrebnikov et al., 2021). Other studies, however, have reported a decrease in performance during the pandemic period (Batyrshin & Sosnin, 2023; Mbunge et al., 2021; Pokhorukova et al., 2021; Shcherbakova, 2023). It is worth noting that most of these studies were conducted between 2020 and 2022. During that period, the universities still used the online education format. Another distinguishing feature of most of these studies is the small sample size of students whose academic performance was analyzed (Batyrshin & Sosnin, 2023; Edelhauser & Lupu-Dima, 2021; Pokhorukova et al., 2021; Shcherbakova, 2023; Chan, 2020). In our opinion, the consequences of moving the educational process online due to the COVID-19 pandemic from the perspective of its impact on the academic performance of university students can be fully assessed only after a sufficient amount of time has passed (Sato et al., 2023), and a larger sample size has been analyzed. Therefore, this study is relevant at present.

The Theoretical and Conceptual Framework

The purpose of this study is to analyze the impact of the transition to online learning on the academic performance of university students. The statistical basis for our analysis is the academic performance data for every student at the National University of Science and Technology MISIS (NUST MISIS) for periods starting with the fall semester of 2018 and until the fall semester of 2023. We also use information on the students’ individual average scores for the Unified State Examination (USE). University applicants compete for admission to universities on the basis of their USE scores. We used other data accumulated by the MISIS university’s ERP (Enterprise Resource Planning) management system.

To achieve the objectives of our study it was necessary to conduct a statistical analysis of the academic performance of students in the period before and after the pandemic and identify the factors affecting student performance. It was also necessary to analyze the readiness and ability of university to implement online learning.

To conduct this study, data mining (for learning analytics, LA) was used (Liñán & Pérez, 2015). This approach uses statistical information accumulated by universities in their ERP systems. In 2018, MISIS University fully implemented and used the 1C ERP system: 1C is a Russian accounting system and ERP software automating organizational management (Volkov et al., 2022) to automate the university’s management process. The document flow management is almost paperless; information about students’ academic performance is accumulated automatically by the system. However, it cannot yet be analyzed in automatic mode. In addition to the university’s ERP system, which stores statistical information on the students’ academic performance, there is also a second digital platform for online education, used for interaction with the students. This type of software is collectively called a Learning Management System (LMS). Examples of LMSs include Canvas and Moodle. LMS can summarize information only within the context of a single discipline. For statistical analysis of student performance, data generated by the LMS cannot be processed automatically. However, it is important to analyze the use of such systems in universities, especially in online learning processes, as the use of the LMS in the educational process ensures quality of online education and blended learning, complements face-to-face interactions with students as confirmed by the findings of several studies (Ghilay, 2019; Hassan Rakha & Abdo Khalifa, 2024; Moralista & Oducado, 2020; Volkov et al., 2022). In our opinion, the degree of LMS use at a university during the transition to online education plays an important role in mitigating the difficulties of this transition and remains an important factor in ensuring the quality of educational programs. Therefore, this aspect has received special attention in our long-term research, which has been ongoing for more than 5 years.

Therefore, analysis of the effectiveness of online learning at universities would benefit from the use of Learning analytics (LA) and educational data mining (EDM) techniques. The effectiveness of combining approaches have been acknowledged by several authors (Liñán & Pérez, 2015; Toktarova & Popova, 2022). These two methods of analysis are interrelated. The primary aim of EDM is to obtain information from electronic interactions with students through LMS systems (such as the number of times a certain material was accessed, the student’s level of activity in interactions with faculty members, switching between linked servers, and ultimately—a final result in the form of the student’s academic performance (grade) in a given subject. All these data are accumulated by the system during the study course. The focus of LA is on processing statistical information and presenting it in a format that is convenient for making managerial decisions in education. Our research is based on learning analytics derived from big data, which is supplemented by information about the usage of the LMS at the university and the students’ USE examination performance prior to entering the university.

Notably, large-scale LA studies based on big data and centered on the evaluation of the effectiveness of online education were conducted from 2012 to 2014 at Harvard and MIT (Chuang & Ho, 2016; Ho et al., 2014; Ho et al., 2015). This research targeted massive open online courses (MOOCs) but largely unrelated to the analysis of online learning at classical universities. MOOCs are self-paced online courses that students can take at their convenience in an asynchronous mode without direct interaction with a teacher. For this format of online courses, research was conducted at classical universities as an experiment (Arias et al., 2018; HSE Research, 2021), when some students studied the same discipline face-to-face (F2F) and others—independently, in an online format, and the academic performance of these two groups was compared. In such studies, online education was also characterized by an asynchronous mode, that is, the absence of a class schedule and lack of interaction with faculty members on a regular basis (Zoncita & Norman, 2020). Therefore, it is important to define online education at a classical university.

Online education at a university is the process of delivering educational experience according to a predetermined schedule, with the personal involvement of a teacher, who utilizes one of several online interaction platforms to conduct classes (e.g., MS Teams, Google Meet, VKontakte). All materials required for studying the subject are available through the LMS, including lectures and materials for independent preparation for practical classes, tasks for individual completion, group projects, and tests. Final control examinations can also be held via an LMS using either an online control system (proctoring) (College of Education, Mindanao State University, General Santos City, Philippines & Cahapay, 2021), or in a face-to-face format (possible in the absence of pandemic restrictions). It is essential that a student has no choice of whether to study a particular subject online, in person, or not at all—if that subject is included in that student’s curriculum. Obtaining a final qualification in higher education (a diploma or a degree) is contingent on an acceptable level of academic performance in each subject (this factor does not apply to people learning via MOOCs). Research focusing on the use of MOOCs to analyze the number of students enrolled in a particular discipline and the number of students successfully completing the final assessment as one of their metrics (Chuang & Ho, 2016). Online study of disciplines at a university ends with taking the final control examination in those disciplines, and obtaining the final grade reflects the student’s overall academic performance in this area.

Once again, research into online learning at traditional universities has predominantly targeted small groups of participants. Typically, the sample size has been limited to a particular group defined by specified criteria, and surveys have been conducted within that group (Chen et al., 2023; Hamdanah et al., 2024; Toktarova & Popova, 2022). Statistical data have been analyzed less often, but also on a very limited sample (HSE Research, 2021). As a result, some studies consistently support an increase in academic performance during the overall transition to online learning during the pandemic, whereas other studies prove that there is a decrease in academic performance in the process of implementing distance learning. Even prior to the COVID-19 pandemic, studies have experimentally compared online university education to F2F education (Faidley, 2018; Paul & Jefferson, 2019; Pearcy, 2009; Slagoski, 2019), and these studies have yielded conflicting results. Our study analyzes the academic performance of 20,000 students with a dataset containing over 600,000 records of their academic performance, their average USE score, and the university readiness to switch to an online format through the level of implementation of the educational process in LMS.

Materials and Methods

The 1C ERP system for educational institutions, used by NUST MISIS since 2018, is a comprehensive software solution tailored to the specific needs of higher education establishments. The 1C system provides a wide range of operational and analytical functions designed to manage educational processes and administer various large-scale activities at a university.

The use of an ERP system allows the collection of grades from various courses, subjects, and academic years; this creates a repository with a wide range of data allowing for a more in-depth and comprehensive examination of the academic performance of NUST MISIS students from 2018 to the present. Since 2018, because of the implementation of the ERP system, NUST MISIS has collected more than 600,000 data points of student academic performance (Database 1). This database is the foundation of this research. The data on the admitted students’ USE exam results (Database 2) is also available (these results were used to select the applicants for admittance to the university).

The data collected by the ERP system are considered to be primary data for the purpose of statistical analysis. These data are accumulated within a single university and a restricted number of individuals have access to these data. For the purposes of our study, the data were depersonalized and normalized for use with a common grading scale. Qualitative descriptions of study outcomes were converted to corresponding numeric values (e.g., a “fail” examination result was converted to a grade of 2 points, which in a 5-point grading system is traditionally interpreted as “unsatisfactory”). Data points for students who failed to complete the course of studies (program dropouts) were deleted from the database. Therefore, the database used for our analysis is unique and could not be accessed by any other researchers. Database 1, a redacted sample of which is shown in Table 1, contains the following data: the student’s identification number; the academic subject (course) they were enrolled in; the department responsible for teaching that course; the academic year and semester this course was delivered; the degree program the student was taking; the grade received by the student, and certain other personal information for the student.

Table 1.

Configuration of Database 1 Fields (Example—Personal Data Redacted).

Id	Student departments	Degree Program	Year of study	Specialization (major)	Study year	Semester	Subject	Grade (w/o retakes)	Department teaching the subject
1x	Department of Computer Science	M.Sc.	2nd	Applied Computer Science	2019–2020	1st	Linux for Data Science	Satisfactory	Department of Computer Science
2x	Department of Physics	B.Sc.	4th	Applied Physics	2019–2020	2nd	Scientific research	Good	Department of Mathematics
3x	Department of Management	B.Sc.	3rd	Management	2019–2020	2nd	Economics	Excellent	Department of Economics
nx	…	…	…	…	…	…	…	…	…

The initial statistical analysis of Database 1 used Excel pivot tables and Shewhart charts based on multiple samples defined by specific criteria. The analysis of student performance is conducted over an 11-semester period, starting with data from the fall semester of 2018 and ending with the fall semester of 2023. This resulted in a timeframe comprising three prepandemic and seven postpandemic semesters. The onset of COVID-19 pandemic took place during the spring semester of 2020—this is referred to as the zero semester in our diagrams. During that semester, there was a transition to remote learning caused by the COVID-19 lockdown in Moscow. Shewhart charts were chosen as the tool for learning analytics (Holweg et al., 2018; Lloyd, 2019). Given that the data we analyze in our study are not related to industrial manufacturing (where notable deviations from specified parameters are commonly observed), the two-sigma criterion was selected for performance analysis. The presence of outlier data points exceeding three standard deviations is considered unlikely (in the educational context), as that would imply a significant flaw in the learning delivery system and in the adaptation of educational materials to meet the needs of modern-day students.

The average academic performance of students and their results on the USE exam were compared. It was important to identify which metrics of success at the admission stage (such as USE exam results) correlate with the students’ future academic performance at the university. To achieve this objective, a supplementary database was formed through the NUST MISIS ERP system. Database 2, presented in Table 2, contains information on the average USE exam results of enrolled students during and after the pandemic. It is important to note that the key limitation of this sample is that the results of the USE are presented for only a part of the student population. Students admitted to the university in 2020 and later on the basis of their USE results (which means only bachelor’s degree students) were included in the dataset. Master’s degree students and applicants enrolling at the university without taking the USE (such as applicants admitted under preferential quotas or based on the results of local admission tests (e.g., graduates of vocational schools) are not included in the sample. The total number of students in the sample is 5,405.

Table 2.

Configuration of Database 2 Fields (Example—Personal Data Redacted).

Id	Student department	Specialization (major) code and name	Year	The overall USE score	Overall USE score of all students applied
Id	Student department	Specialization (major) code and name	Year	The overall USE score	State-funded	Full-fee tuition
1x	Department of Computer Science	09.03.03 Applied Computer Science	2020	93.33	94.80	68.85
2x	Department of Physics	03.03.02 Physics	2019	75.65	88.47	66.06
3x	Department of Management	38.00.00 Economics and Management	2022	91.50	90.41	74.08
nx	…	…	…	…	…	…

Analysis of the relationship between students’ academic performance at NUST MISIS University and their average USE score was determined via statistical methods. Spearman’s rank correlation coefficient was calculated; this metric assesses the degree of correlation between two variables without assuming normal distribution of either variable (Mitsel & Cherniaeva, 2016). The average USE score for a student and that student’s average academic performance were ranked in ascending order. The sample consisted of 5,405 values (n = 5,405) of “X” and “Y,” respectively.

Spearman’s correlation coefficient (C) was calculated via the following formula (Liu, 2017): where n is the number of paired observations and d represents the difference in rank between two variables:

C = 1 - \frac{6 \times \sum d^{2}}{n \times (n^{2} - 1)},

(1)

The resulting Spearman’s correlation coefficient allowed us to evaluate the degree of monotonic relationship between students’ academic performance and their average USE score (used by the university in its admittance procedure). A value close to 1.00 indicates a strong positive correlation; in our model suggest that higher USE scores are typically associated with better academic performance at the university level. Conversely, a coefficient close to −1.00 indicates a strong negative correlation. A value near .00 implies a weak relationship (correlation) or none at all between the two variables analyzed.

Results

Academic Performance Charts

In this section, we present the results of analyzing the academic performance of students enrolled in bachelor’s and master’s degree programs for the period from 2018 to 2023. The results of the students studying for technical degrees (departments of technology, new materials, computer science, and mining) and humanities (departments of linguistics and economics and management) were analyzed separately. First, the academic performance of students in technical bachelor’s degree programs was analyzed (Table 3).

Table 3.

Academic Performance of Technical Bachelor’s Degree Students, NUST MISIS.

The semester since the start of the pandemic	Number of instances each grade was received:				The average grade for the period	Control line	Upper control line	Bottom control line
The semester since the start of the pandemic	2	3	4	5	The average grade for the period	Control line	Upper control line	Bottom control line
−3	5,306	2,644	4,678	7,097	3.688	3.429	4.038	2.820
−2	7,016	2,604	4,296	7,844	3.596	3.429	4.038	2.820
−1	5,887	2,682	4,460	6,992	3.627	3.429	4.038	2.820
0	8,385	2,609	4,339	8,498	3.543	3.429	4.038	2.820
1	9,381	2,465	4,030	6,945	3.374	3.429	4.038	2.820
2	12,006	2,677	4,293	8,352	3.329	3.429	4.038	2.820
3	13,215	2,345	4,007	6,983	3.179	3.429	4.038	2.820
4	15,513	2,369	3,908	7,951	3.144	3.429	4.038	2.820
5	11,995	3,409	4,956	7,967	3.314	3.429	4.038	2.820
6	15,293	3,380	5,017	9,200	3.247	3.429	4.038	2.820
7	7,876	5,659	6,900	11,219	3.678	3.429	4.038	2.820

Note. The Russian grading system has 5 points; traditionally grades from 2 to 5 are used, with 5 being “Excellent,” 4 being “Good,” 3 being “Satisfactory” and 2—“Unsatisfactory” or “Fail.”

Figure 1 illustrates the academic performance of technical bachelor’s degree students.

Figure 1.

The learning outcomes of technical bachelor’s degree students at NUST MISIS.

The diagram in Figure 1, which is a type of control chart used to monitor a process for stability, tracks the average grade value (quantifying the level of the students’ academic performance) over time. The diagram has three horizontal lines that represent the control limits. The central line represents the average value, while the upper and lower control lines show values that are two standard deviations above and below the average value. The broken line in the chart represents the average trend in academic performance. The vertical axis of the chart represents the average grade value, whereas the horizontal axis represents the semesters prior to and following the onset of the pandemic. The zero period on our chart is set to be the spring semester of 2020, during which the educational process had to switch to an online-only format. Figure 1 illustrates that after a transition to online learning, average academic performance decreases significantly, although remains within acceptable ranges.

Next, the academic performance of students in bachelor’s degree programs in the humanities was analyzed (Table 4). Figure 2 illustrates the results of the analysis of the academic performance of bachelor’s degree students in humanities, which broadly follows the trend observed among students pursuing a technical bachelor’s degree programs, albeit with fewer and less significant fluctuations.

Table 4.

Academic Performance of Humanities Bachelor’s Degree Students, NUST MISIS.

The semester since the start of the pandemic	Number of instances each grade was received:				The average grade for the period	Control line	Upper control line	Bottom control line
The semester since the start of the pandemic	2	3	4	5	The average grade for the period	Control line	Upper control line	Bottom control line
−3	2,432	1,380	2,644	4703	3.862	3.467	4.217	2.717
−2	3,276	1,203	1,848	4,420	3.690	3.467	4.217	2.717
−1	3,170	1,051	1,896	3,896	3.651	3.467	4.217	2.717
0	4,120	884	1,434	3,893	3.494	3.467	4.217	2.717
1	4,804	1,103	1,667	3,128	3.291	3.467	4.217	2.717
2	5,102	869	1,376	3228	3.258	3.467	4.217	2.717
3	5,180	1,060	1,727	3,244	3.271	3.467	4.217	2.717
4	5,435	924	1,584	3,541	3.281	3.467	4.217	2.717
5	5,242	1,203	1,835	3,536	3.310	3.467	4.217	2.717
6	6,011	1,003	1,560	3,366	3.191	3.467	4.217	2.717
7	2,757	1,798	2,945	5,234	3.837	3.467	4.217	2.717

Note. The Russian grading system has 5 points; traditionally grades from 2 to 5 are used, with 5 being “Excellent,” 4 being “Good,” 3 being “Satisfactory” and 2 –“Unsatisfactory” or “Fail.”

Figure 2.

The learning outcomes of humanities bachelor’s degree students at NUST MISIS.

The academic performance of students pursuing technical and humanities master’s degrees was analyzed separately. Data on the academic performance of students enrolled in the technical master’s degree programs is shown in Table 5 below.

Table 5.

Academic Performance of Technical Master’s Degree Students, NUST MISIS.

The semester since the start of the pandemic	Number of instances each grade was received:				The average grade for the period	Control line	Upper control line	Bottom control line
The semester since the start of the pandemic	2	3	4	5	The average grade for the period	Control line	Upper control line	Bottom control line
−3	1,339	712	2,123	5,585	4.225	4.024	4.537	3.511
−2	1,028	308	1,183	3,906	4.240	4.024	4.537	3.511
−1	1,685	722	1,959	5,380	4.132	4.024	4.537	3.511
0	1,260	313	957	3,897	4.166	4.024	4.537	3.511
1	2,081	536	1,902	5,460	4.076	4.024	4.537	3.511
2	1,358	292	1,051	3,905	4.136	4.024	4.537	3.511
3	2,377	471	1,565	4,830	3.957	4.024	4.537	3.511
4	1,775	316	904	3,305	3.911	4.024	4.537	3.511
5	2,842	590	1,475	4,487	3.810	4.024	4.537	3.511
6	2,270	394	949	3,420	3.785	4.024	4.537	3.511
7	3,729	731	1,889	6,069	3.829	4.024	4.537	3.511

On average, the academic performance of students in the technical master’s degree programs is higher than that of students in the technical bachelor’s degree programs, and the transition to remote learning made virtually has no impact on that performance. Academic performance of students enrolled in the technical master’s program is shown in Figure 3.

Figure 3.

The learning outcomes of technical master’s degree students at NUST MISIS.

In our opinion, master’s degree students are more motivated to acquire knowledge and the learning format is not as important for them. The online format may even be more convenient for them, as most master’s degree students combine study and work. Additionally, during the pandemic outbreak, master’s degree students had a sufficient level of technical knowledge acquired during their F2F studies for their bachelor’s degrees. We associate the delayed decline in academic results in technical master’s degree programs with a high level of student preparedness. The observed decrease in average grades could be explained by the admission to master’s degree programs of students who have studied remotely for their bachelor’s degree during the COVID-19 pandemic (and such remote studies could have negatively affected the level of their technical competencies).

The hypothesis regarding the convenience of an online learning format for master’s degree students was ultimately validated when analyzing the academic performance of humanities master’s degree students. The academic performance of such students tended to improve during the online-only study period. The data from the analysis of academic performance in humanities master’s degree programs are presented in Table 6 and Figure 4.

Table 6.

Academic Performance of Humanities Master’s Degree Students, NUST MISIS.

The semester since the start of the pandemic	Number of instances each grade was received:				The average grade for the period	Control line	Upper control line	Bottom control line
The semester since the start of the pandemic	2	3	4	5	The average grade for the period	Control line	Upper control line	Bottom control line
−3	77	103	239	472	4.241	4.407	4.780	4.034
−2	33	61	188	435	4.430	4.407	4.780	4.034
−1	116	81	261	505	4.199	4.407	4.780	4.034
0	28	35	126	359	4.489	4.407	4.780	4.034
1	80	32	145	420	4.337	4.407	4.780	4.034
2	32	8	57	313	4.588	4.407	4.780	4.034
3	68	21	122	389	4.387	4.407	4.780	4.034
4	24	15	61	267	4.556	4.407	4.780	4.034
5	54	49	144	517	4.471	4.407	4.780	4.034
6	43	30	141	405	4.467	4.407	4.780	4.034
7	142	74	243	729	4.312	4.407	4.780	4.034

Figure 4.

The learning outcomes of humanities master’s degree students at NUST MISIS.

It is significantly easier to implement distance learning in humanities master’s degree programs, as the educational process does not involve practical assignments requiring the use of laboratory equipment. It is nearly impossible to complete such assignments remotely, and practical assignments are a requirement in certain technical subjects. Therefore, the online format for master’s degree programs can be highly effective, particularly when combined with F2F practical classes in a number of technical subjects.

As a result, we can conclude that we have statistically proven that the transition to distance learning does make a negative impact on the academic performance of bachelor’s degree students. This influence was evident in the technical bachelor’s degree programs, which may subsequently have an impact on academic performance of students in the technical master’s degree programs. In general, the transition to online master’s degree programs did not have a significant negative impact on academic performance; in fact, improving in academic performance for some humanities programs in the postpandemic period. However, this trend is not consistent and requires further investigation. Of course, further research into fluctuations in student academic performance is required to identify other significant factors that may coinfluence the final result. Technical aspects such as the level of implementation of distance learning and the extent of use of LMS are likely to be relevant in this regard. In particular, we attribute the overall decline in academic performance during the 6th post-COVID semester to the fact that the university was forced to stop using the LMS Canvas platform as a result of the sanctions. The LMS Canvas was an essential component of learning experience at NUST MISIS. The university required additional time to implement a new learning management system (LMS Moodle). However, the assumption that the loss of the LMS Canvas was the sole cause for the decrease in academic performance in the sixth post-COVID semester requires further investigation, although we did investigate the level of preparedness of the university for distance learning during the pandemic as an important factor that could have directly affected student academic performance in the new learning format.

The Level of Preparedness of the University to Transition to a Remote Format of Education Process

A well-functioning LMS system is an important tool used at a university. Analysis of the history of LMS implementation at NUST MISIS allows us to assess the potential impact (both positive and negative) of LMS usage on student academic performance during the period of complete transition to online learning. In 2018, NUST MISIS underwent a comprehensive digital transformation, both in the area of university administration (implementation of the 1C ERP system) and in the educational process (the introduction of LMS Canvas for all subjects). The transition to the use of an LMS was not immediate. Faculty members were trained, and during the academic year, they had to start using the LMS Canvas in their teaching process. Coincidentally, by the time the COVID-19 pandemic began, most of the courses at NUST MISIS already had study materials available on the electronic system, making the transition to online-only teaching seamless as possible.

Notably, the educational department at NUST MISIS has systematically monitored faculty members’ use of the LMS Canvas. Since 2018, evaluating work on this platform has been one of the criteria for assessing teacher qualifications. Information on the quality of the materials presented on the LMS Canvas formed the basis for our research conducted by our team from 2020 to 2021. We analyzed 236 subjects offered during the spring semester of the 2020 to 2021 academic year, using online learning platforms such as the LMS Canvas and MS Teams. In 2021, during the period when online-only learning was still underway, the global adoption of electronic systems into the educational process at NUST MISIS is reflected in the interactive schedule, (available on the university’s website). Figure 5 shows a sample of a faculty member’s interactive timetable. Two mandatory links to courses in the LMS Canvas and MS Teams are displayed. A similar timetable is accessible for every student through their personal accounts.

Figure 5.

Interactive schedule fragment in the faculty member’s personal online cabinet, available at the NUST MISIS website (Volkov et al., 2022).

Therefore, we conclude that NUST MISIS University was adequately prepared for the complete transition of the entire educational process to an online-only format during the COVID-19 pandemic. This did not significantly affect the academic performance of the students.

The Correlation Between the Outcomes of the USE and the Academic Performance of Students

At the beginning of this study, we were aware that the quality of students may well have a decisive impact on the average academic performance of students at the university, because the applicants’ level of competence after high school graduation can serve as a factor in their further academic performance in the process of mastering higher education programs. Therefore, it was important for us to accept or reject this assumption to make the study as objective as possible. University admission campaigns are based on USE scores for bachelor’s and specialty programs in Russia. Leaving this factor outside of the scope of our research would be wrong, as it could distort the results of the study.

It was important to determine the level of correlation between the academic performance of NUST MISIS students and the grades they have obtained earlier on the USE exam. The variables for calculating Spearman’s rank correlation coefficient are presented in Table 7.

Table 7.

Variables for Calculating the Spearman Coefficient Using Formula 1.

Student Id	The average value of The USE scores (X)	The GPA at the university (Y)	Rank X (Rx)	Rank Y (Rx)	d (Rx-RY)	d^2
16001xx	78.000	2.6909	2,356	4,589	−2233	4,986,289
16010xx	86.667	2.2715	776	4,977	−4201	1,764,8401
16035xx	74.000	2.0000	3,133	5,109	−1976	3,904,576
16055xx	92.667	4.4592	210	1,387	−1177	1,385,329
17003xx	51.333	2.8571	5,361	4,339	1,022	1,044,484
17005xx	76.000	2.0000	2,759	5,109	−2,350	5,522,500
…	…	…	…	…		…
SUM						21,770,980,881

According to the results of the calculations performed, the Spearman’s correlation coefficient was 0.173; indicating a very weak positive relationship between the two parameters under consideration. This suggests that there is virtually no connection between students’ performance in the Unified State Exam and their academic performance. Therefore, it is possible to exclude the influence of students’ quality assessed by their performance on the USE exam from our analysis of the students’ academic performance during and after the pandemic.

On the basis of the abovementioned findings, the quality of a student’s education during school years is not a definitive predictor of their success at the university, at least in the context of the specific time period studied. This confirms that it was precisely the transition to online learning necessitated by the COVID-19 pandemic that had a significant impact on the trajectory of students’ academic achievement in university.

Discussion

Research has revealed a negative impact of the transition to online learning at university on the academic performance of bachelor’s and master’s students in technical fields has been statistically confirmed. However, an increase in academic achievement has been observed only among students pursuing master’s degrees in the humanities. Therefore, the results of research showing an increase in academic performance among bachelor’s degree students during the transition to online learning remain controversial (Boyarshinova & Karaseva, 2021; HSE Research, 2021; Pogrebnikov et al., 2021). In our opinion, such results were obtained primarily because the actual academic performance was not statistically studied, but based on data obtained from surveys (Batyrshin & Sosnin, 2023; Chan, 2020; Edelhauser & Lupu-Dima, 2021; Pokhorukova et al., 2021; Shcherbakova, 2023; Zhang et al., 2024). When conducting surveys among students regarding their academic performance, it is necessary to consider the fact that they may have not received a grade for the final assessment on the first attempt. Retaking the exam with the aim of obtaining a positive grade is considered an unsatisfactory outcome in our study. In statistical terms, repeated attempts to pass the final assessment with a positive result are not taken into account. However, the students interviewed may consider their final results to be positive in such cases.

Another potential issue with the reliability of statistical research is the possibility of limited sampling, as demonstrated in the HSE University study. To quote their approach, “students who received at least three grades in two modules were selected for the analysis of academic performance. The sample included grades obtained in subjects and grades for term papers. The average grade within the sample was 7.8 in the 2020 to 2021 academic year and 7.45 in the previous academic year. During 2020, the proportion of students receiving un-satisfactory grades (less than 4 points out of 10) decreased by half compared with the previous year, whereas the proportion of outstanding students increased by one-quarter. The percentages of good and satisfactory grades remained at the same levels (HSE Research, 2021). In our view, such a sample does not provide a valid indicator of the trend in overall student academic performance.

The significance of the level of digitalization in the educational process during the period of total transition to online education continues to be the topic of debate (Onah & Chikeleze, 2024; Mhlanga, 2024; Chowdhury et al., 2024). MISIS University had a high level of digitalization for all processes even as early as 2018 (Boboshko, 2020; Volkov, Rishko, Kostyukhin et al., 2023; Volkov, Rishko, & Vidmanova, 2023). This made it possible to design a learning process within the paradigm of shifting focus from teaching factual knowledge to “helping students figure out phenomena and design solutions to problems” (Greubel et al., 2024; Zeng et al., 2023). This allows for a fair and objective evaluation of students’ work, even in an online learning environment, as the results of the final assessment cannot be manipulated through unethical methods such as cheating.

Conclusion

The analysis of the impact of online learning on students’ academic performance does not allow us to reach a definitive conclusion regarding the nature of this impact. Inconsistency in research outcomes in this field can be attributed to a variety of factors. First, the context in which students are faced with distance learning can vary: voluntary self-improvement through the study of MOOCs; experiments with the simultaneous delivery of the same curriculum in both online and F2F formats; or, as in the case of our study, a complete, and mandatory transition to online learning at a classical university. Second, in our opinion, an important factor influencing the assessment of the impact of online education on student academic performance is the statistical database of such assessments: research could be based either on data obtained from student surveys or on statistical information automatically accumulated by ERP systems based on grades for every subject studied. The third factor is whether the study considers the level of an educational program (bachelor’s, master’s or PhD) of students and their field of study.

In our study, which was based on a large amount of data collected from the ERP system of a classical university, we have identified a statistically proven decline in the academic performance of bachelor’s degree program students during those semesters in which the university was required to conduct the entire educational process remotely. Academic performance steadily decreased until the bachelor’s degree students fully resumed their face-to-face studies. Moreover, the impact of online learning on the academic performance of master’s degree program students has not been conclusively established. Given the higher levels of motivation among master’s degree program students and their ability to engage in self-directed learning (both of these factors developed during their bachelor’s degree program studies), it is possible that the online or F2F format of study for certain master’s degree programs is not a decisive factor, and online learning may actually be more convenient for those students.

Footnotes

Acknowledgements

Not applicable.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethical Approval

Not applicable.

ORCID iDs

Yury Rishko

Diana Boboshko

Evgeniya Eliseeva

Aleksandr Malkin

Dmitrii Treistar

Data Availability Statement

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

References

Aithal

P. S.

Aithal

(2016). Impact of on-line education on higher education system. International Journal of Engineering Research and Modern Education, 1(1), 225–235. https://doi.org/10.5281/ZENODO.62029

Arias

J. J.

Swinton

J. R.

Anderson

K. M.

(2018). Online vs. face-to-face: A comparison of student outcomes with random assignment. e-Journal of Business Education and Scholarship of Teaching, 12, 1–23. https://www.semanticscholar.org/paper/Online-vs.-Face-to-Face%3A-A-Comparison-of-Student-Arias-Swinton/1cad83a84dd0cd31ca0fd99a2cd4bd3eb6a59b36

Chowdhury

Banerjee

S. D.

Mandal

Das

Maity

A. K.

Mete

J. K.

(2024). A review on the role of digital education in higher education. Educational Administration: Theory and Practice, 30(6), 2375–2381. https://doi.org/10.53555/kuey.v30i6.5737

Batyrshin

D. V.

Sosnin

E. A.

(2023). The impact of distance learning on the academic performance of FIT NI TSU. Collection of materials of the XIX International Conference of students, postgraduates and young scientists (vol. 1, pp. 465–468). https://www.elibrary.ru/item.asp?id=54249164&pff=1

Boboshko

D. Y.

(2020). Digitalization in small business tax administration. In Industry competitiveness: Digitalization, management, and integration: (Vol. 1, pp. 675–683). Springer International Publishing. https://doi.org/10.1007/978-3-030-40749-0_80

Boyarshinova

O. A.

Karaseva

M. G.

Sednina

M. A.

(2021). The impact of distance learning on student academic performance. Information And Communication Technologies in Management, Education, Science, 77–84. https://rep.bntu.by/handle/data/99049

Liñán

L. C.

Pérez

Á. A. J

. (2015). Educational data mining and learning analytics: Differences, similarities, and time evolution. Universities and Knowledge Society Journal, 12(3), 98–112. https://doi.org/10.7238/rusc.v12i3.2515

Chan

R. Y.

(2020). Studying coronavirus (COVID-19) and global higher education: Evidence for future research and practice. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3622751

Chen

Zhang

Hou

(2023). Investigating structural relationships between professional identity, learning engagement, academic self-efficacy, and university support: Evidence from tourism students in China. Behavioral Sciences, 14(1), 26. https://doi.org/10.3390/bs14010026

10.

Chuang

(2016). HarvardX and MITx: Four years of open online courses—Fall 2012-Summer 2016. SSRN Scholarly Paper. https://doi.org/10.2139/ssrn.2889436

11.

Cahapay

M. B.

(2021). Problems encountered by college students in online assessment amid COVID-19 crisis: A case study. International Journal of Computer Science and Information Technology for Education, 6(1), 1–10. https://doi.org/10.21742/IJCSITE.2021.6.1.01

12.

Edelhauser

Lupu-Dima

(2021). One year of online education in COVID-19 age, a challenge for the Romanian education system. International Journal of Environmental Research and Public Health, 18(15), 8129. https://doi.org/10.3390/ijerph18158129

13.

Faidley

J. K.

(2018). Comparison of learning outcomes from online and face-to-face accounting courses. Electronic Theses and Dissertations. Paper 3434. https://doi.org/10.13140/RG.2.2.15703.98724

14.

Gautam

D. K.

Gautam

P. K.

(2021). Transition to online higher education during COVID-19 pandemic: turmoil and way forward to developing country of South Asia-Nepal. Journal of Research in Innovative Teaching & Learning, 14(1), 93–111. https://doi.org/10.21203/rs.3.rs-59206/v1

15.

Ghilay

(2019). Effectiveness of learning management systems in higher education: Views of lecturers with different levels of activity in LMSs. Journal of Online Higher Education, 3(2), 29–50. https://www.academia.edu/40153219/Effectiveness_of_Learning_Management_Systems_in_Higher_Education_Views_of_Lecturers_with_Different_Levels_of_Activity_in_LMSs

16.

Greubel

Siller

H. S.

Hennecke

(2024). Assessing teachers’ capabilities to work with models and evaluate results in the context of a complex and authentic STEM problem. Education Sciences, 14(1), 104. https://doi.org/10.3390/educsci14010104

17.

Hamdanah

Rusmaniah

Rajiani

Muslimah

(2024). Continuance intention of digital education in traditional Indonesian higher education: Policy implication forward. Journal of Infrastructure, Policy and Development, 8(5), 3596. https://doi.org/10.24294/jipd.v8i5.3596

18.

Hassan Rakha

Abdo Khalifa

. (2024). Blended learning using Edmodo: Students’ performance and attitude in boxing during COVID-19. Sage Open, 14(2), 3743. https://doi.org/10.1177/21582440241253743

19.

Chuang

Reich

Coleman

Whitehill

Northcutt

Williams

Hansen

Lopez

Petersen

(2015). HarvardX and MITx: Two years of open online courses Fall 2012-Summer 2014. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2586847

20.

Reich

Nesterko

Seaton

Mullaney

Waldo

Chuang

(2014). HarvardX and MITx: The first year of open online courses, Fall 2012-Summer 2013. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2381263

21.

Holweg

Staats

B. R.

Upton

D. M.

(2018). making process improvements stick. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3240097

22.

HSE Research. (2021, February 17). HSE research: How has the student’s academic performance changed during the distance learning Period. HSE. https://www.hse.ru/our/news/444619547.html

23.

Jena

P. K.

(2020). Impact of Covid-19 on higher education in India. International Journal of Advanced Education and Research, 5(3), 77–81. https://doi.org/10.31235/osf.io/jg8fr

24.

Khan

M. M.

Rahman

S. M. T.

Islam

S. T. A.

(2021). Online education system in Bangladesh during COVID-19 pandemic. Creative Education, 12(2), 441–452. https://doi.org/10.4236/ce.2021.122031

25.

Liu

(2017). A short note on spearman correlation: Impact of tied observations. Academia. https://www.academia.edu/31867274/A_Short_Note_on_Spearman_Correlation_Impact_of_Tied_Observations

26.

Lloyd

(2019). Quality health care: a guide to developing and using indicators. Jones & Bartlett Learning. https://www.cec.health.nsw.gov.au/__data/assets/pdf_file/0011/605657/Understanding-Variation-with-Shewart-Charts.pdf

27.

A. W.

Stevens

Willems

S. P.

(2021). World of EdCraft: Challenges and opportunities in synchronous online teaching. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3793342

28.

Martin

Bolliger

D. U.

(2022). Developing an online learner satisfaction framework in higher education through a systematic review of research. International Journal of Educational Technology in Higher Education, 19(1), 50. https://doi.org/10.1186/s41239-022-00355-5

29.

Mbunge

Fashoto

S. G.

Olaomi

(2021). COVID-19 and online learning: Factors influencing students’ academic performance in first-year computer programming courses in higher education. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3757988

30.

Mhlanga

(2024). Digital transformation of education, the limitations and prospects of introducing the fourth industrial revolution asynchronous online learning in emerging markets. Discover Education, 3(1), 32. https://doi.org/10.1007/s44217-024-00115-9

31.

Mitsel

A. A.

Cherniaeva

N. V.

(2016). Analysis of the curriculum subjects correlation. Engineering Education, 19, 62–68. https://elibrary.ru/item.asp?id=27180329

32.

Moralista

R. B.

Oducado

R. M.

(2020). Faculty perception toward online education in a state college in the Philippines during the Coronavirus disease 19 (COVID-19) pandemic. Universal Journal of Educational Research, 8(10), 4736–4742. https://doi.org/10.13189/ujer.2020.081044

33.

Morze

Trybulska

E. S.

(2021). Web-based community-supported online education during the COVID-19 pandemic. International Journal of Web Based Communities, 17(1), 9. https://doi.org/10.1504/IJWBC.2021.112858

34.

Nuryatin

Mukhibad

Tusyanah

(2022). Effectiveness of online learning at universities: Do sociocultural differences matter? European Journal of Educational Research, 11(4), 2153–2166. https://doi.org/10.12973/eu-jer.11.4.2153

35.

Olmos-Gómez

M. D. C.

Luque-Suárez

Castán-García

Portillo-Sánchez

(2023). Validation of educational quality as a tool to measure the degree of satisfaction of university students. Education Sciences, 14(1), 2. https://doi.org/10.3390/educsci14010002

36.

Onah

C. C.

Chikeleze

F. O.

(2024). Information and communication technology (ICT) as a panacea for overcoming the challenges of COVID-19 pandemic: A study of academic staff performance in tertiary institutions in Enugu State, Nigeria. Sage Open, 14(2), 6341. https://doi.org/10.1177/21582440241256341

37.

Paul

Jefferson

(2019). A Comparative analysis of student performance in an online vs. face-to-face environmental science course from 2009 to 2016. Frontiers in Computer Science, 1, 7. https://doi.org/10.3389/fcomp.2019.00007

38.

Pearcy

(2009). Finding the perfect blend: A comparative study of online, face-to-face, and blended instruction. Academia. https://www.academia.edu/84979331/Finding_the_perfect_blend_A_comparative_study_of_online_face_to_face_and_blended_instruction

39.

Pogrebnikov

A. K.

Shestakov

V. N.

Yakunin

(2021). Impact of grading system on student academic performance during the COVID-19 pandemic. Perspectives of Science and Education, 52(4), 80–93. https://doi.org/10.32744/pse.2021.4.5

40.

Pokhorukova

M. Y.

Aghababyan

E. O

Varlamov

V. A.

(2021). The impact of distance learning on the quality of student academic performance. Modern Pedagogical Education, 11, 45–47. https://cyberleninka.ru/article/n/vliyanie-distantsionnogo-obucheniya-na-kachestvo-uspevaemosti-studentov/viewer

41.

Prokopev

V. P.

(2000). The characteristics of a classical university. Universitetskoe upravlenie: praktika I analiz, 2, 35–39. https://elar.urfu.ru/handle/10995/42829

42.

Radaev

V. V.

(2022). Transition to the online education in the content of pandemic: Initial outcomes. University Management: Practice and Analysis, 26(1), 6–17. https://doi.org/10.15826/umpa.2022.01.001

43.

Sato

S. N.

Condes Moreno

Rubio-Zarapuz

Dalamitros

A. A.

Yañez-Sepulveda

Tornero-Aguilera

J. F.

Clemente-Suárez

V. J.

(2023). Navigating the new normal: Adapting online and distance learning in the post-pandemic era. Education Sciences, 14(1), 19. https://doi.org/10.3390/educsci14010019

44.

Shcherbakova

I. K.

(2023). The distance learning impact on the quality of the educational process (according to the study results conducted at the State University of Management). Digital Sociology, 6(2), 89–98. https://doi.org/10.26425/2658-347X-2023-6-2-89-98

45.

Sinha

(2020). Post-COVID challenges and opportunities in the education sector. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3710846

46.

Slagoski

(2019). Professional learning practices and beliefs of an online community of English language teachers. Contemporary Issues in Technology and Teacher Education, 19(1), 97–116. https://www.learntechlib.org/primary/p/181934/

47.

Slykerman

R. F.

Mitchell

E. A.

(2022). Students’ experience of online university education during the COVID-19 pandemic: Relationships to psychological health. Student Success, 13(1), 32–40. https://doi.org/10.5204/ssj.2023

48.

Toktarova

V. I.

Popova

O. G.

(2022). An analysis of educational data on the correlation between learning success and students’ behavior in the university digital educational environment. Informatics and Education, 37(4), 54–63. https://doi.org/10.32517/0234-0453-2022-37-4-54-63

49.

Ukhurebor

K. E.

Oaihimire

I. E.

Wada

Z. B.

Jokthan

Inegbedion

Ofulue

Efanodor-Obeten

H. O.

Otsupius

A. I.

Tanglang

Mai-inji

A. Y.

Ashawa

(2024). A perspective overview of the influence of the COVID-19 pandemic on the educational sector. Journal of Infrastructure, Policy and Development, 8(5), 2955. https://doi.org/10.24294/jipd.v8i5.2955

50.

Volkov

Rishko

Boboshko

Eliseeva

Treistar

Kulapova

(2023). Multitrack educational programs as a method of educational process personalization at universities. Education Sciences, 14(1), 22. https://doi.org/10.3390/educsci14010022

51.

Volkov

Rishko

Kostyukhin

Sidorova

Boboshko

Savinova

Ershova

(2022). Using digital tools to teach soft skill-oriented subjects to university students during the COVID-19 pandemic. Education Sciences, 12(5), 335. https://doi.org/10.3390/educsci12050335

52.

Volkov

A. A.

Rishko

Y. I.

Vidmanova

A. N.

(2023). An innovative system for designing adaptive educational programs of higher education based on the harmonization of employers’ requests and applicants’ expectations. Innovation and investment, 10, 324–331. https://cyberleninka.ru/article/n/innovatsionnaya-sistema-proektirovaniya-adaptivnyh-obrazovatelnyh-programm-vysshego-obrazovaniya-na-osnove-garmonizatsii-zaprosov

53.

Xia

Yang

Lei

(2022). Challenges of online learning amid the COVID-19: College students’ perspective. Frontiers in Psychology, 13. https://doi.org/10.3389/fpsyg.2022.1037311

54.

Zeng

M. R.

Shin

Krajcik

(2023). Characterizing students’ performances for interactive instructional decisions making to meet individual needs. In Proceedings of the 17th international conference of the learning sciences-ICLS 2023, pp. 1913–1914. International Society of the Learning Sciences. https://doi.org/10.22318/icls2023.267425

55.

Zhang

Duffy

Zhang

Cai

(2024). Assessing the impact of the COVID-19 pandemic on graduate learning experiences in higher education: Insights and recommendations. Sage Open, 14(2), 9889. https://doi.org/10.1177/21582440241239889

56.

Zoncita

Norman

P. D.

(2020). Understanding the effect of distance learning vs. face-to-face learning experiences on students’ engagement in higher education. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3734764

Analysis of the Impact of the Transition to Online Education During the COVID-19 Pandemic on the Future Academic Performance of University Students

Abstract

Keywords

Introduction

The Theoretical and Conceptual Framework

Materials and Methods

Results

Academic Performance Charts

The Level of Preparedness of the University to Transition to a Remote Format of Education Process

The Correlation Between the Outcomes of the USE and the Academic Performance of Students

Discussion

Conclusion

Footnotes

Acknowledgements

Declaration of Conflicting Interests

Funding

Ethical Approval

ORCID iDs

Data Availability Statement

References