Abstract
There has been a rapid growth of massive open online courses (MOOCs) in the global education market in the last decade. Online learning technologies are becoming increasingly widespread in the non-formal education sector and in higher and supplementary vocational education. The use of MOOCs in Russia to support the delivery of educational programmes at university level opens opportunities in terms of expanding the educational choice for students, the development of virtual academic mobility, reduction in the cost of educational services, and improvement in the accessibility of education. However, the effectiveness of using different online learning technologies at university level, and the consequences of their widespread adoption, has not been sufficiently explored. In this research study, a comparative analysis is made of the effects of different online learning models on student educational outcomes in a university setting. A study was undertaken in which different groups of students at the Ural Federal University, Russia, were encouraged to study technical and humanities disciplines using a framework of blended learning, and online learning with tutoring support. The results of the study were compared with the results of a reference (control) group of students who studied the same disciplines in a traditionally taught model. It was found that both models (blended and online) of MOOC implementation demonstrated greater learning gains, in comparison with the traditional model. For engineering and technical disciplines, there was no statistically significant difference between blended or online learning technologies. For the humanities discipline, where the communicative component of the learning process was significant, the blended learning technology produced better results. Conclusions of this empirical research may be useful for heads of educational organizations and teachers in helping them to make strategic decisions about the modernization of university courses by increasing the effectiveness of the implementation of new educational technologies. The results of this research project will be used for implementing the State Priority Project, ‘The Modern Digital Educational Environment of the Russian Federation’.
Keywords
Introduction
This paper investigates student performance in a range of different massive open online course (MOOC) settings in a Russian University (the Ural Federal University (UrFU), Ekaterinburg, Russia). Continuing development of the information society, and associated technological and structural changes, have created new challenges for the national education system of Russia. The need for active, interesting, lifelong learning is increasing. This demand remains unsatisfied, despite the rapid spread of MOOCs and increased access to education. Reasons for this include problems with existing online courses and low levels of motivation of students for more traditional courses, particularly among those belonging to the so-called ‘Generation Z’ (Veen, 2007). It is argued that this generation, whose use of technology permeates almost all aspects of life, are accustomed to accessing, retrieving, and assimilating information quickly (Freitas et al., 2015), and have a need to be engaged with others (among other distinctive characteristics) (Gryaznova et al., 2016; Guo et al., 2014; Tyler-Smith, 2006). There is a gap between student expectations and actual experiences once actively engaged in higher education (Brown and Lally, 2017; Castano et al., 2016).
We argue that practitioners of the face-to-face approach to learning regard MOOCs, perhaps incorrectly, as a sequence of standard text blocks, forgetting that learning means not only access to information, but many processes of learning, as well as the acquisition of specific knowledge and skills (Lundvall and Borrás, 1997; Nonaka and Takeuchi, 2011). It could be argued that the emergence of MOOC-style online courses has resulted in the creation of the technological and methodological conditions for a new educational paradigm (Jansen and Schuwer, 2014; Kop et al., 2011). The following points indicate some possible trends in online education.
Shifting the focus of organizational activities related to the implementation of MOOCs in higher education from quantitative to qualitative indicators (Crouch and Mazur, 2001; Klimentiev and Klimentiev, 2015). This trend became especially noticeable in the last three years.
Extension of theory development and research with respect to both MOOCs and student perception of the online courses (Open Education Special Interest Group and SURFnet, 2015: 13).
European universities demonstrating an intent to move to independent monitoring of students’ knowledge, involving MOOCs as an instrument of assessing results of learning in higher education.
Desire to explain MOOC statistics using cognitive and psychological research (the study of data perception (Rosselle et al., 2014: 137); study of personal factors related to online course selection (Pellas and Kazanidis, 2014); student involvement analysis by means of surveys and interviews (Allen and Seaman, 2014).
Increasing diversification of the online education sphere (Open Education Special Interest Group and SURFnet, 2015: 18).
Increase in the number of programmes that consist of MOOCs in part or in whole. For example, in 2017 the number of MicroMasters programmes on international online learning platforms reached 35, the number of the XSeries programmes reached 40, and the number of vocationally certified programmes reached 17.
Use of interactive MOOC elements for blended learning to achieve educational goals (Israel, 2015).
Many university policies related to online courses are now aimed at expanding educational provision (Bangert, 2005; Larionova et al., 2016; Morris and Stommel, 2013), especially in relation to personal and communicative (Yin et al., 2017; Zheng and Weng, 2016) capabilities of students. The use of MOOCs allows universities to ‘reconfigure’ issues of staffing, methodology, and logistical support related to educational processes. Educational process mechanism changes may vary: one trend is the transference of educational process support and control functions to less qualified online course tutors to affect cost reduction; another is the provision of organizational and methodological student support within MOOC-based online learning, leading to the reduction of class workload for teachers. Despite the advantages and economic benefits that may possibly be accrued from the introduction of massive online learning technology, there are several internal and external barriers to be recognized and overcome. Key organizational, methodological, and psychological challenges encountered by universities when implementing e-learning, generally (Knyazeva, 2014; Komleva, 2014; Kostyuk et al., 2014; Leontiev, 2014; Lisitsyna et al., 2014; Solovov and Menshikova, 2015), may arise due to poor strategic implementation of online learning. Inappropriate pedagogy, insufficient support by university management, low levels of confidence by academics towards online learning, and lack of understanding of personalization of education could all lead to increased barriers for online courses. These problems are increasingly a reality for modern universities (Dmitrievskaya, 2014; Glotova et al., 2015; Swanberg and Martinsen, 2010; Yin et al., 2017; Zhang et al., 2016).
Universities may often behave quite conservatively and be slow to implement the structural and organizational changes required for a wide use of online technologies in education. The change management process affects staff composition in favour of tutors, assessors, and technicians, leading to a redistribution of educational expertise (Kostolányová, 2017; Martins and Nunes, 2016). Institutional educational inertia may lead to the desire to preserve classroom workload, maintaining a centric planning-administrative-management model, and is correlated with a lack of understanding of the online educational processes.
Of course, students’ capabilities must also be considered in the design of online programmes. Levels of student motivation for online learning are critical to the process; low student motivation to learn in an online environment may result in poor self-regulation, with associated inherent dangers posed by this issue (Van Laer and Elen, 2018).
It has been argued that flexibility, efficiency (Uchidiuno et al., 2016), and the creation of a new, up-to-date educational environment supporting online learning will not be achieved without overcoming these barriers. Success requires engagement of the institutional educational actors and stakeholders, and specific data generation and analysis for the objective, efficient evaluation of different online education models within the university.
This paper reports on the definition of five key approaches to the application of online courses to educational processes in universities, derived from an analysis of theoretical and empirical material on e-learning development (see, for example, Israel, 2015; Protsiv et al., 2016; Rosselle et al., 2014). This led to our underpinning quasi-hypothesis that the selection of the most appropriate online course model in a university depends upon an assessment of the educational goals and the characteristics of the student body. Different disciplines and student groups may require different models of online learning in order to be effective. In the following section, we summarize the organizational, financial, and functional aspects of these models as found in Russia.
Models of online learning
Model 1: ‘Use of the MOOC as a support resource’
In this case, the MOOC is only used to provide additional training and methodological material. It can be used by the teacher to prepare for classes and to organize self-study activities for the students. However, its use in the educational process is not mandatory. The teacher requires extra time to select and evaluate several online courses in advance, to ensure full compliance with the content and learning outcomes of a discipline or a module. The model involves neither additional financial, nor administrative costs for either university or student. It does not change the workload of the teacher, student tuition fees remain the same, and it does not involve any significant pedagogical changes.
Model 2: ‘Blended learning model using parts of the MOOC for mastering the discipline/module’ (‘flipped class’ model)
The MOOC is used to study a certain part of the discipline/module (for example, theoretical materials of the course). The model allows for the use of components of an online course, located on a public website in open access, and not requiring user identification. The teacher provides organizational and methodological support for e-learning, practical exercises, and timed activities with an in-class final assessment. The model does not require formal agreements with the educational organization that developed the online course. This enables cost savings in time, labour, and logistics for the university.
Model 3: ‘Blended learning model based on a MOOC with mid-term and final examinations online while preserving part of the face-to-face classes’
This model assumes partial transfer of the educational processes to the electronic education environment. Students gain access to electronic educational resources, including video lectures, text materials, training tasks, and tests. The resources support mastery of the discipline or module, using e-learning and independent control of the learning outcomes. The teacher provides organizational and methodological support for the students in class during some face-to-face classes and advisory hours. The educational institution establishes formal contracts for educational services from the other providers. The institution regulates and governs issues such as credit transfer, administration, and external support. Results obtained in the online course are credited to the main educational programme. The cost of teaching a student in a university that implements the educational programme does not change.
Model 4: ‘Online-learning model, using MOOCs, internal organizational and technical support by the tutors’
This model includes the complete transfer of the educational process to the electronic information and educational environment, whereby students obtain access to electronic educational resources. Tutors provide organizational and technical support to students on the course. The university may pay another university, based on a partnership agreement or agreement for additional educational services, to support remote students. The client university must develop formally agreed regulations to govern the process of e-learning. This model is efficient if the university lacks teachers to provide courses and/or subjects by choice. Other benefits include:
(a) expanding the number of courses to offer individual learning paths to students;
(b) supporting first-year students as they adapt to university life and learning requirements;
(c) the reduction of student costs;
(d) the elimination of academic differences in the curricula between educational organizations;
(e) the development of information and communication-related skills among the students;
(f) access to online courses from leading universities; and
(g) the reduction/redistribution of the workload of teachers.
The efficiency of this model is influenced, to a degree, by student preparedness and motivation for working in an online environment, concentration on results, and timely tutor assistance.
Model 5: ‘Exclusive online-learning using MOOC’
Within the framework of this model, the online course completely defines the methodology and technology of training, and determines the content of the module and the course of the learning process. The complete course and materials are used fully within the MOOC; the university using the online course of another university must agree a contract for the full provision and support of students. The contracted university is paid to teach the selected course to students. This model of exclusive e-learning offers universities an opportunity to save and redistribute resources for development in other areas and may suit those universities aiming at reducing teacher workloads. Students’ personality is very important for the success of this model; students are expected to self-organize, focus on the learning outcomes, and be able to work in an online environment.
All of these models are already in use in different Russian and foreign universities. For the purpose of examining their efficiency for realizing educational outcomes, the authors compared the learning outcomes of students who studied common disciplines in the different model formats. The work took place in UrFU (Ekaterinburg, Russia). UrFU is the biggest university in Russia, with more than 35,000 students. During 2015–2017, 25 MOOCs were created by teachers and scientists of UrFU for the National Open Education Platform (NOEP: www.openedu.ru). Given the particularities of UrFU as a technical university, academic subjects (modules) from the engineering educational programmes were chosen for this comparison. Preference was given to those online courses providing the core modules of the programmes, thus ensuring the maximum number of student participants. The following disciplines were chosen for the comparison: engineering mechanics, engineering graphics, and philosophy as a humanitarian discipline of the engineering educational programme.
Students
More than 800 Bachelor programme students of various majors were involved in this study. All participants were first-year students in the following engineering educational programmes: Engineering (18.2%), Software Engineering (11.0%), Design and Technology for Machine Building Industries (7.4%), Special Purpose Vehicles (3.8%), Automation of Technological Processes and Production (6.8%), Machinery and Equipment (5.6%), Land Transportation and Technological Complexes (5.6%), Radio Technics (4.9%), Technosphere Safety (4.2%), and a number of other fields, with proportions below 4%. The curricula of these educational programmes are similar for the first-year students.
In the framework of the study, three models of implementing a discipline were selected: extended traditional format (Model 1); blended learning with online examination (Model 3); and online format with tutor support (Model 4). The online learning models were compared with each other and with the traditional learning model.
Students were divided into groups in order to establish the different approaches to studying a discipline. A summary of the data on student batches is presented in Table 1.
General information on students studying using four models.
ThM-1 and EnG-1 (138 in total) groups took traditional face-to-face training without using the online courses. The EnG-2 (31) group used an online course while training in a traditional format. For the Ph-1, ThM-2, and EnG-3 groups (316 in total) training was conducted in blended format with part classwork and online mid-term and final testing. Online learning with organizational and technical support by tutors was implemented for 339 students in Ph-2 and ThM-3 groups.
Methodology and analysis
The following data were collected concerning participating students.
General student data from the university database, including their average score on the Unified State Exam (USE) and average performance for the period of previous study.
Data on current academic performance of students during the study, uploaded from the grading system of the university (for traditional models), or from the online platform (for blended and online models).
Data from student questionnaire.
Study data were processed using the following algorithm. In the first step, normality and homogeneity in terms of initial training level for different groups of students that participated in the different models were assessed using the Kolmogorov–Smirnov (K–S) test. No pretests were performed before the study, so the average score on USE was selected to characterize the initial training level of students. Since all students participating in the study were first-year students, their USE scores were an objective indicator of their initial training level. The data on academic performance of students for the previous semesters could not serve as an adequate parameter for assessing the homogeneity of the student sets, as it was subject to the influence of exogenous factors associated with the adaptation of the first-year students to university study. Factors such as changing living conditions, different levels of adaptation to university study, different levels of self-discipline, etc., affect students’ academic performance in the first semester.
In the second step, the descriptive statistical tools were used for processing the distributions of students’ average scores for their previous study at the university (for the first semester of the 2016–2017 academic year). Statistical distribution indicators were calculated, and frequency bar charts were plotted, allowing clustering of student groups according to their previous academic performance. Selected subgroups corresponded to different score ranges in accordance with the standard evaluation criteria used in the university grading system. Thus, the ‘honours students’ corresponded to the range of average scores above 80; ‘good students’, from 60 and less than 80 points; ‘weak students’, below 60 points. The average scores on the previous study served as input parameters for analysing the efficiency of different models and as a controlled variable when comparing models with each other.
In the third step, statistical indicators of the students’ final score distribution for a discipline in different learning models were analysed using descriptive statistics. For students who studied disciplines in the purely traditional format we used data from the university grading system. When students studied disciplines using the online course (Models 3 and 4), the progress of students was taken from the online learning platform. The frequency distributions of the students’ final scores served as an output parameter for analysing the efficiency of different models.
The distributions of the initial and final grades and their statistical indicators were compared. It was shown that the type of distributions of the students’ output scores was far from normal, which made it impossible to apply the methods of correlation–regression analysis or one-factor analysis of variance for processing the data and drawing unambiguous conclusions about the impact of learning technology on students’ learning outcomes.
For a more detailed comparison of the online learning models with each other and with the traditional model, a hypothesis was put forward that different teaching technologies have different effects on the performance of academically strong and weak students. Subgroups of students with different input scores were singled out, as previously indicated. Further, the analysis of the students’ final scores for each subgroup of students in all the models was carried out, and the percentage ratios between students who have lowered their academic performance, retained their previous level of performance, and increased it, were calculated. The integral assessment for all subgroups allowed a comparison of the efficiency of applying various learning technologies in teaching different disciplines.
Results
The engineering discipline selected for analysis in this study is ‘Theoretical Mechanics’, which is a compulsory subject for all engineering programmes. It was implemented in three different models during the second term of the 2016–2017 academic year.
Traditional training without using online course (control group: ThM-1).
Model 3 (the first experimental group: ThM-2).
Model 4 (the second experimental group: ThM-3).
The total number of students studying the discipline in each model exceeds the minimum requirements for sample size (at least 25 people per each group), calculated on the basis of reliability and validity criteria for statistical inference. The total number of participants was 472 students: 115 students in the control group, 148 students in the first ‘experimental’ group; 209 students in the second ‘experimental’ group.
To analyse the homogeneity of student sets for all the groups and the normality of the distribution according to the level of initial training, the K–S criterion was applied. The average scores on the USE, uploaded from the administrative database of the university, were used to determine the initial training level. The maximum value of difference between the empirical and normal distribution of average USE scores was calculated. For the control group of students, the maximum deviation value of the experimental distribution from the theoretical normal law is demp = 0.052, which is statistically insignificant: demp < min(dcr), where dcr is displayed in Table 2.
Critical value of Kolmogorov–Smirnov test of distribution normality.
Thus, the empirical distribution of the students’ average scores on USE is close to normal. Similarly, the same calculations of K–S criterion for the normality of distribution for two experimental groups were performed. For the blended learning model, the empirical value of deviation from the normal distribution was demp = 0.055, which is less than the critical value (see Table 2). For the model of online learning with tutor support, demp = 0.074, which confirms a statistically insignificant difference between the empirical and theoretical distributions.
To verify the homogeneity of the student groups (similarity in characteristics for all the models), the K–S test was also undertaken. To compare the distributions of the students’ average scores on USE for control and experimental groups, the calculations of the maximum difference between these distributions were performed.
The empirical value of the K–S test is equal to λemp = 1.199, which is less than λcr (p ≤ 0.05) = 1.36, thus providing the basis to conclude that the difference between the distributions of the average USE scores in the control and the first experimental groups is statistically insignificant. In the same way, the empirical differences were calculated when comparing the distributions for the 1st and 2nd experimental groups. The resulting value of the K–S test is λemp = 1.118, which is in the area of statistical insignificance.
Thus, we are dealing with three homogeneous distributions for the initial training level of students studying the discipline in three different technologies, which excludes the influence of the initial sampling parameters on the result of statistical processing and allows a comparison of the efficiency of all three models with each other.
Figure 1 shows the frequency distributions of the average USE scores of students who studied Theoretical Mechanics in three different models.
Frequency distribution of the average USE scores of students who studied Theoretical Mechanics in three different models: traditional model of learning (solid curve with squares); blended learning with current and final online testing (dashed curve with rhombs); and online learning model with tutor support (dotted curve with triangles).
The results of calculation by the method of descriptive statistics have shown that the statistical indices for these three groups are the same within the accuracy of calculations.
Minimum values of the average USE scores range from 44 to 47, and maximum ones, from 81 to 100. At the same time, the percentage of students with the average USE scores below 60 is between 13% and 22% for different teaching technologies; for the average USE score from 60 to 80, it is 76% to 80% of students; and for the share of ‘honour students’ (more than 80 points) in different samples, it ranges from about 1% to 8%.
Analysis of information from the administrative database of the university on the average USE scores have shown that most students in the groups (75%) are ‘good students’. This fact determines the average performance and uniformity of the sets. Proportions of the ‘honours’ and ‘weak’ students are significantly lower, and they are not essential to the statistical indices, except for the variance, which shows the range of values of the average USE scores from the expected value. For the control group of students, the value of the dispersion is minimal, and for the second experimental group it is maximum – this determines the difference in the mean square deviations in the groups.
Referring to the analysis of the average performance of students in the previous period of study, this is the input parameter when comparing the efficiency of learning in different models. The information was taken from the administrative database of the university. Results of the calculation of the statistical indicators for average students’ score distributions for the fall term of the 2016–2017 academic year are shown in Table 3.
Statistical indicators for average students’ score distributions for the fall term of 2016–2017 academic year.
An analysis of the distribution of the average students’ scores for the intermediate certificate students revealed an insignificant shift in the mean and median values of the previous performance to large values, as well as an increase in variance in all three samples. In this case, the shape of the distribution changed considerably, and the differences from the normal distribution of the groups were statistically significant. Figure 2 shows the frequency distribution of the average scores of intermediate certificate students from the different samples.
Frequency distribution of the average scores of intermediate certificate students belonging to different samples: traditional model of learning (solid curve with squares); blended learning with current and final online testing (dashed curve with rhombs); and online learning model with tutor support (dotted curve with triangles).
The most significant difference was observed for the students of the 1st and 2nd experimental groups: two maxima appeared on the frequency distribution of scores. For students of the 1st experimental group, one maximum arose at 60 points, and the second at 70, while for the students of the 2nd experimental group the maxima were at the level of 65 and 75 points. In addition, the proportion of ‘good’ students in the 1st and 2nd experimental groups decreased, and the share of ‘honours’ students increased to 18% and 32% in the 1st and 2nd groups, respectively (Table 4). For the control group, the percentage of ‘good’ students remained at the same level, but the share of ‘weak’ students decreased significantly and the share of ‘honours’ students increased.
Proportions of the total number of students in the samples with average scores in the following ranges: less than 60 (‘weak’); from 60 and less than 80 (‘good’); 80 and above (‘honours’).
Comparisons of the distribution of USE scores and the average performance of students in the previous semester are not useful, since the assessment technologies and the format of the exams vary considerably. In addition, as previously shown, the results of the first session can be influenced by the different level of adaptation of students to the new conditions, as well as different motivation and self-discipline.
In the next step of processing, the results were analysed by final grades, which were obtained from two sources: the progress of students on the online platform (for students studying the discipline in terms of the Model 4 online learning), and the score from university grading system (for students studying in the traditional format), and from both sources for students who studied in the framework of Model 3 (blended learning). Figure 3(a)–(c) show the frequency distribution of final grades in the subject (output frequency) compared to the average performance scores in the previous period (input frequency) for students that trained with the traditional model.
Frequency distributions of the final scores for the discipline (output frequency depicted by the dashed curve) and the average score of the intermediate certificate (input frequency depicted by the solid curve) for students trained within: (a) the traditional model; (b) blended learning technology with current and final examinations online; (c) the online learning model with tutor support.
As it can be seen from the Figure 3(a)–(c), the distribution of the final scores for the discipline differs significantly from the distribution of the scores of the average performance for the previous period of study. In all three models, the range of final scores is wider than for the previous performance of students. There is a redistribution of frequencies in the direction of lower points for all three models. This can be explained by the fact that Theoretical Mechanics is the first engineering discipline in the curriculum of the first-year students who study the engineering training programmes. It requires not only knowledge of the basics of physics, but also the skills of using mathematical analysis tools acquired by these students in the same semester. As a rule, the students’ progress in Theoretical Mechanics is lower than in other subjects. In order to support students in studying this complicated discipline and facilitate their understanding of the laws of mechanics, the teachers created an online course entitled ‘Engineering Mechanics’, using the NOEP and edX platform (www.edx.org), which contains not only video lectures, theoretical material, and control tasks, but also interactive training simulators of real mechanical systems.
Considering the range of low final grades in the discipline, the largest number of ‘weak’ students is observed in the traditional training format. The smallest number of students with low grades is found in the online learning model. The number of weak students increased slightly in the blended learning model. The two peaks on the curves correspond to ‘good’ and ‘honours’ students; the peaks are small for traditional learning, and significantly larger for the blended and online formats. The comparison of the final scores in the discipline for different models was based on statistical indicators calculated using the descriptive statistics tools. The calculation results are shown in Table 5.
Statistical indicators of the distribution of final grades for the discipline ‘Theoretical Mechanics’, for students studying in different models.
The average student scores in the traditional model are heavily skewed toward lower scores due to the large number of ‘weak’ students. The mean score in the other two models has also shifted to the left, but the median values remained the same as the previous study scores. The sample variance increased significantly as compared with the variance of the input scores and, accordingly, the standard deviation increased. Figure 4 shows the frequency distribution of the final points of the discipline for three different models of learning.
Frequency distribution of the final grades in the discipline, for three different models of learning: traditional model of learning (solid curve with squares); blended learning with current and final online testing (dashed curve with rhombs); and online learning model with tutor support (dotted curve with triangles).
Comparing models of online learning with one another and with the traditional format is difficult due to the complexity of the distribution function and the ambiguity of the conclusions about the success or failure of students for the entire aggregate sample. For further analysis, subgroups with different output scores were singled out from the total samples of students for different models. The principle of clustering was the same: students with a score below 60 (‘weak’); students with a score from 60 and less than 80 (‘good’); and students with a score of 80 and above (‘honours/excellent’) (Figure 5).
Distribution of students using the final score for different models of learning: traditional model of learning (diagonal hatching); blended learning with current and final online testing (horizontal hatching); and online learning model with tutor support (dotted hatching).
A direct comparison of the distribution of students by subgroups for the input and output scores yielded the following results (Table 6). In all the models under consideration, the proportion of students with low scores (less than 60) increased, while the share of ‘good’ students fell significantly, and only ‘excellent’ students increased in blended and online models.
Proportions of students in the samples that have input and output scores in the following ranges: less than 60; from 60 and less than 80; 80 and above.
For a more detailed analysis of the effectiveness of different teaching technologies and the identification of the characteristics of students with different entrance scores, the frequency distributions of students on input score were assessed for each range of final scores in the discipline. This led to a conclusion about which students showed low results in the discipline, and which were good and excellent in progress. Figure 6(a)–(c) show the frequency distribution of the average performance of students in the previous period for different final grade ranges and different learning models (percentages herein are calculated for the number of students in cohorts with low, medium, and high final score).
Distribution of the average performance of the previous period of study for subgroups of students who received different final scores in the discipline (solid curve with squares corresponds to 0 < score < 60, dashed curve with rhombs corresponds to 60 ≤ score < 80, dash-dotted curve with triangles corresponds to score ≤ 80) for the traditional learning model (a), the blended learning model with current and final testing online (b), and online learning model with tutor support (c).
An anomalous distribution is observed with the traditional model of training: most students received a low overall score in the discipline (the solid curve in Figure 6(a)), and this cohort included all ‘weak’, most ‘good’, and more than half of the ‘honours’ students. In the blended and online models, the subgroup with a low final score (the solid curves in Figure 6(b) and (c)) consisted mostly of ‘weak’ students, but a small proportion of ‘good’ and ‘honours’ students also satisfactorily coped with the testing in the discipline.
In the final score range ‘from 60 and less than 80’ (dashed curves in Figure 6(a)) there were ‘good’ students in all three models; that is, they were able to confirm their scores on previous studies. The presence of small ‘tails’ left from 60 points and to the right from 80 points indicate some ‘weak’ students close to the assessment of ‘good’, and a small proportion of the ‘honours’ students lowered their scores to ‘good’. A high score was achieved by students who had ‘excellent’ and ‘good’ grades in the previous period of study, in all three models.
Figure 7(a)–(c) present data on progress with different entry points of the previous studies. The proportions of students were calculated as the total number of ‘weak’ (Figure 7(a)), ‘good’ (Figure 7(b)), and ‘honours’ (Figure 7(c)) students according to the average performance in the previous period of study.
Final performance score of the students with different ranges of input score: (a) 0 < score < 60, (b) 60 ≤ score < 80, and (c) score ≤ 80, for groups of students studying the discipline in the following models: traditional model of learning (diagonal hatching); blended learning with current and final online testing (horizontal hatching); and online learning model with tutor support (dotted hatching).
Thus, the analysis of the progress of students who had different entry points, for different learning models, showed the following:
in the traditional training model, all students with low input scores showed poor performance;
14% and 6% of ‘weak’ students in the blended and online learning models, respectively, could improve their performance to ‘good’, while the rest confirmed their knowledge as ‘satisfactory’;
in the traditional model of education, only 10% of ‘good’ students confirmed their rating of ‘good’, 87% of ‘good’ students lowered their performance to ‘satisfactory’, and only 3% were able to get ‘excellent’;
in the blended learning model with the current and final testing online, 43% of ‘good’ students confirmed their rating, 48% of ‘good’ students lowered their performance to ‘satisfactory’, and 9% achieved ‘excellent’;
in the online learning model with tutor support, 44% of ‘good’ students confirmed their rating of ‘good’, 39% of them lowered their performance to ‘satisfactory’, and 17% obtained an ‘excellent’;
in the traditional model, 17% of ‘honours’ students showed excellent knowledge, 17% lowered their performance to ‘good’, and 66% of them obtained an assessment of ‘satisfactory’; and
in the blended model, 16.2% of ‘honours’ students confirmed their excellent knowledge of the subject, 19% lowered their performance to ‘good’, and 19% obtained ‘satisfactory’.
In the online learning model with tutor support, 68% of ‘honours’ students confirmed their excellent knowledge of the subject, 24% reduced their performance to ‘good’, and 8% rated ‘satisfactory’. The overall picture of student performance changes with respect to input scores for different training technologies, as presented in Figure 8.
Changes in the performance of students in the subject ‘Theoretical Mechanics’ with respect to the input points (average performance of the previous period of study) for different learning models: traditional model of learning (diagonal hatching); blended learning with current and final online testing (horizontal hatching); and online learning model with tutor support (dotted hatching).
Discussion and comparison with other disciplines
The detailed analysis of progress on the ‘Theoretical Mechanics’ subject showed that the discipline of engineering was rather difficult for students participating in the study. In all three models, there is a general decline in performance level. In the traditional model, only 21% of students retained their level of achievement in relation to their previous study, 76% showed a lower score, and only 3% increased academic performance. The reason for this may be excessive examination requirements in the traditional format. This will require further investigation. Students may also have received low levels of training in physics and mathematics (Treacy and Faulkner, 2015), which are the prerequisites of the study course ‘Theoretical Mechanics’. This may have also had an impact on the motivation of students. In the blended and online models, a positive dynamic of performance is observed, where more than half of students (54–55%) exhibited grades corresponding to the average score for their previous study, 8% of students in the blended learning technology group and 11% in the online technology group showed higher scores, but 37% and 33% in the blended and online groups, accordingly, reduced their performance.
These benefits may be associated with the learning models. In the specially designed online course students were able to access video recordings and methodical development at any time and from any location. They could use training simulators for the investigation of problems on mechanics. They also had access to intermediate testing for the training period, uniform teaching time, and strictly defined deadlines for tasks. In addition, they had the opportunity to attend face-to-face classes (in the blended learning group), as well as consultations (for both models) with the teacher during the semester. From the above, it would appear that the two models of online learning (blended learning with the current and final testing online, and the online learning model with tutor support) are effective in enhancing the learning performance of students in the discipline of engineering (Theoretical Mechanics). The difference in learning outcomes for students in these two models is statistically insignificant.
The same data processing algorithm was applied to the data that had been obtained for the two other disciplines (see Table 1). A humanitarian discipline ‘Philosophy’ was examined using the MOOC in the following online models:
Model 3 (the first experimental group: Ph-1); and
Model 4 (the second experimental group: Ph-2).
The total number of participants was 269 students: 139 students in the first experimental group; 130 students in the second experimental group. All students were studying the discipline in the second term of the 2016–2017 academic year.
The final results for the discipline ‘Philosophy’ are presented in Figure 9. As it can be seen from the figure in the blended learning model, 35% of the students retained their academic performance in relation to their previous studies, 55% showed higher scores, and only 10% decreased their performance. In the online learning model, the effect appears less, but still shows positive progress of students: more than half of the total number of students (52%) demonstrated scores corresponding to their average scores on previous studies, 32% showed higher scores, and about 16% decreased their academic performance.
Changes in the performance of students in ‘Philosophy’ with respect to their input grades from a previous period of study, for different learning models: blended learning with current and final online testing (horizontal hatching); and online learning model with tutor support (dotted hatching).
The analysis of the students’ progress showed that for humanities disciplines such as ‘Philosophy’, where communicative components of the educational process play an important role in the discipline, and in achieving learning outcomes, the use of blended learning models with some face-to-face teaching gives better results.
A detailed analysis was also carried out of the academic progress of students in ‘Engineering Graphics’ using three different models. As indicated in Table 1, three models of online learning were implemented:
traditional training without using an online course (control group: EnG-1);
traditional training with using an online course for additional material (the first experimental group: EnG-2); and
Model 3 (the second experimental group: EnG-3).
Eighty-three first-year students took part in this study, including 23 students who were taught in the traditional format; 31 students used the MOOC independently, and 29 students were involved in blended learning.
The analysis shows that using MOOC improved the progress of students and facilitated their understanding of the basic concepts of projecting a spatial figure on a plane. The online MOOC course ‘Descriptive Geometry and Engineering Graphics’, used in the learning process, contained interactive training tasks and simulators that develop spatial imagination and help students to gain practical skills for engineering drawings.
In all three models of training (see Figure 10), most students (from 59% to 68%) maintained their average scores compared to their previous period of study. At the same time, about 40% of students lowered their performance in the traditional model, about 20% lowered in the traditional format with MOOC, whereas only 7% under-performed in the blended learning model. Using the online course with current and final testing raised performance for 34% of students, whereas in the purely traditional format no student could raise their academic performance. In the blended model, 10% of students received higher grades in relation to their average scores from the previous studies.
Changes in the performance of students in ‘Philosophy’ with respect to the input grades from the previous period of study, for different learning models: traditional model of learning (diagonal hatching); traditional model with online course as an addition (vertical hatching); and blended learning with current and final online testing (horizontal hatching).
Concluding comments
In order to study the educational processes in online courses for engineering and technical majors in a university environment in Russia, the authors identified five models of online learning from the literature, and conducted a study aimed at comparing these different models of online learning with each other, and with the traditionally taught format. The results of the study may help in purposeful model selection for implementation in other university environments. This study considers the specific disciplines being undertaken by students, and the initial training level of students.
We consider the results of the study to be helpful and potentially significant. The results indicate that the use of online courses does not reduce the learning outcomes of students in these disciplines or result in lower grades. Moreover, the models using MOOCs, such as ‘blended learning’ and ‘online learning with tutor support’ demonstrate greater effectiveness in comparison with the traditional model. It was also shown that the use of the online additional materials in the traditionally taught format raised the students’ academic achievements. Related to this are other elements offered by the online learning models: specially developed online courses with free, unlimited access to video recordings and content; the possibility of using training simulators to solve problems; intermediate testing throughout the training period; uniformity of the workload distribution over time; and strictly defined time limits for the achievement of learning outcomes.
This empirical study also provides some evidence for the efficacy of MOOC-style learning in humanities disciplines, where the communicative component of the learning process is significant. In this study, the blended learning technology produces better educational outcomes for students. However, for engineering and technical disciplines, there is no statistically significant difference between blended or online learning technologies.
The results of this empirical study may be useful to heads of educational organizations and teachers when considering the strategic issues in modernizing educational programmes in universities, augmenting teaching methods, and leveraging the effectiveness of new educational technologies. The results of this research project will be used for implementing the State Priority Project ‘The Modern Digital Educational Environment of the Russian Federation’.
Footnotes
Acknowledgements
The authors would like to express their deepest gratitude to the Russian Scientific Foundation for the support of the research within the project number 15-18-10014, ‘Designing Optimal Social and Economic Systems in the Context of Turbulence of External and Internal Environment’.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.