Sage Journals: Discover world-class research

Abstract

This is the protocol for a Campbell systematic review. The objectives are as follows: To evaluate the effectiveness of online distance education compared to other approaches on K-12 students’ academic performance; to evaluate the effectiveness of online distance education compared to other approaches on K-12 students’ non-academic performance, including classroom participation, learning interest, learning pressure, social skills and social isolation, and; to evaluate the potential moderators between online distance education and students’ performance and other outcomes, such as grade, country, subject, study duration, teaching content and methods, and family circumstances.

Keywords

Systematic review K-12 education Online distance education Academic performance Non-academic performance

Background

The Problem, Condition, or Issue

Distance education is an institution-based, formal education where telecommunications systems, which may be interactive, are used to connect learners, resources, and instructors (Schlosser & Simonson, 2009). There are various forms of distance education, including correspondence courses, radio, television, telephone, and web-based education (Kaplan & Haenlein, 2016). However, now widely recognized in the narrow sense, distance education refers only to teaching in the form of live broadcasts or online videos. In recent years Massive Open Online Courses (MOOCs) have become popular. In this review, we restrict the scope to online distance education.

At present, online distance education is developing rapidly and is being applied increasingly widely worldwide. Therefore, there are many studies on the effect of online distance education, but there are also many differences in the conclusions of such studies. For example, Spitzer and Musslick (2021) find that students’ performance in mathematics improved in an online learning environment, with performance improvements being greatest for students categorized as low performers in the previous year (Spitzer & Musslick, 2021). However, Means et al. (2013) argue that there is no difference in the effectiveness of online learning compared to traditional face-to-face learning (Means et al., 2013), with a similar view being expressed by Allen et al. (2002). Furthermore, Yan et al. (2021) study showed that distance education reduced K-12 students’ academic performance and made them feel lonely and lacked learning motivation (Yan et al., 2021). Wendt and Rockinson-Szapkiw (2015) study reported similar results (Wendt & Rockinson-Szapkiw, 2015).

Many countries have implemented specific policies to advance distance education many years ago (German Federal Government, 1976; MEXT, 1990; U.S. Department of Education, 1998). But these policies are more targeted at higher education and correspondence education. In order to control the spread of COVID-19, some countries have initiated policies to accelerate the application of online distance education among K-12 students (The Central People’s Government of the People’s Republic of China, 2019; UNESCO, 2020; United States federal government, 2020). K-12 students refer to students from kindergarten to 12th grade, which is the stage before college. Compared to undergraduate, the vast majority of K-12 students are minors, so they have poor self-control or other aspects, which may lead to differences in the effectiveness of distance education (Wang et al., 2017). So it is very important to arrive at a comprehensive and reliable conclusion as to its effectiveness with this age group. This study will synthesize all the evidence to determine the effectiveness of online distance education among K-12 students, including their academic performance and non-academic performance. Academic performance is the total score or a subject score on the final exam and some standardized tests, which reflects the level of students’ mastery and application of knowledge in some subjects. Non-academic performance refers to other types of performance beyond academic performance, including classroom participation, learning interest, learning pressure, social skills and social isolation.

The Intervention

The earliest known distance education was held in 1728, when Caleb Phillips taught shorthand courses by mail (Kaplan & Haenlein, 2016). Since then, people have studied or taught various courses in various ways of distance education. The history of distance education can be divided into three stages: the printing era, television era and internet era. There are various forms of distance education, including mail, radio, television, telephone, and web-based education (Kaplan & Haenlein, 2016).

Compared with traditional education, distance education has four advantages: flexibility, content availability, low cost and study at home. Flexibility means that students can study anytime and anywhere without being restricted by the fixed environment and time schedule. Content availability means that students can keep a given piece of content for reference at any time in the course. Low cost means that the cost of distance education is lower than that of traditional education, both in terms of fees and other participation costs. Finally, students can study at home at any time, which has benefits for flexibility and cost and the ability to draw on family support but ambiguous effects on learning (Santana de Oliveira et al., 2018). For this reason, Belousova et al. (2022) found that students are very satisfied with this way of education and have a high level of enthusiasm (Belousova et al., 2022).

Online distance education is a popular and widely used form of distance education especially since the outbreak of COVID-19 (Lockee, 2021). This approach provides all the above advantages. Therefore, the distance education method described in this study involves online distance education, including synchronous distance education based on Zoom, VOOV Meeting and other conference live broadcast software and asynchronous distance education based on Udemy, Coursera and other online education websites or LMSes. In addition, Martin et al. (2020) proposed dual time distance education in 2020, which is defined as the blending of both asynchronous and synchronous online learning, where students can participate in anytime, anywhere learning during the asynchronous parts of the course but then participate in real-time activities for synchronous sessions (Martin et al., 2020). It should be emphasized that we only focused on online distance education in the school setting.

Conceptual Framework

The conceptual framework for this review is the theory of change that describes how distance education affects students’ academic achievement and learning autonomy. Figure 1 below demonstrates the conceptual framework through which the interventions are hypothesized to lead to the intended outcomes.

Figure 1

Conceptual Framework for Intervention and Outcomes of Distance Education

There are various forms of distance education. A high fit between distance education and the internet increases the possible advantages of these methods. The greatest advantage of distance education is its flexibility (Santana de Oliveira et al., 2018). Different tools for learning software platforms have been developed to make teachers’ teaching and students’ learning more diverse and flexible (Kuzminykh et al., 2022). Moreover, online distance education provides students with more freedom in space and time for learning, which may enhance their learning autonomy and learning interest (Alzahrani & Wright, 2016). This means that self-paced learning is possible, with the most able students tackling more advanced material, while others may focus on mastering the basics. Moreover, due to the non-face-to-face communication between teachers and students in distance education, the pressure felt by students from the environment will be reduced, thus increasing their desire for communication and classroom participation.

Distance education is also extremely convenient. The popularity of electronic devices and networks has made distance education inexpensive, and learning software has communication functions that make it very convenient for students and teachers in different places and for different students to communicate.

Moreover, the extensive storage and rapid processing of information on the internet make it easier for teachers to manage students’ information and carry out activities such as examinations and homework (Cooper et al., 2009).

Finally, teachers can upload teaching videos and other resources to the network so that students can watch them at any time or many times. In addition, the participation of students and the support of the family environment also have an impact on the process of distance education (Curtis & Werth, 2015).

In addition to students, distance education is also very helpful for teachers.

First, teachers can use a variety of teaching tools in software to enrich teaching content and make teaching activities more vivid and interesting to improve the efficiency of teaching (Menabò et al., 2022). And studies have shown that the diversity of teaching modes brought about by these diverse teaching tools can increase students’ learning motivation, indirectly enhancing and maintaining their learning interest (Fırat et al., 2018). At the same time, higher learning opportunities improve student attendance and course participation, especially in synchronous and dual time distance education (Göksu et al., 2021).

Second, teachers do not have to repeat important knowledge frequently, which can save time communicating with students and solving the problems encountered by students because students can watch videos repeatedly (Tzankova et al., 2023). Finally, teachers can use the network to quickly and timely understand the learning progress of students to change teaching plans in a timely manner, which can lead to the formulation of more complete and possibly personalized teaching plans and the provision of more high-quality teaching content (Aykan & Yildirim, 2022).

In addition, these benefits for teachers indirectly help students so that they can receive better courses and improve their learning efficiency. Although teachers can keep track of students’ learning progress in a timely manner through the internet, there is a lack of efficient feedback between teachers and students, and due to distance limitations, it is difficult to monitor students in real time (Habala, 2021). These two points reduce the amount of face-to-face interaction between teachers and students.

To sum up, distance education makes students have higher attendance, stronger learning motivation, more learning interest, higher learning efficiency and deeper understanding and memory of knowledge, so they will have stronger learning autonomy and improve their academic performance. However, due to less close face-to-face interaction with other people, students’ social isolation will increase, and their social skills may be worse than they would otherwise have been had they been in a traditional school environment (Gaidelys et al., 2022).

Why It Is Important to do This Review

The effectiveness of distance education for K-12 students remains controversial. Robert (1979) found that distance education is slightly better than face-to-face classroom teaching in terms of achievement but slightly lower than face-to-face classroom teaching in terms of attitude and memory (Robert, 1979). Means et al. (2013) reported that there is no difference in the effectiveness of online learning and traditional face-to-face learning (Means et al., 2013). Blended learning, which refers to a teaching mode that combines distance education with traditional face-to-face teaching, and their results showed that students were very satisfied with this way of education and had high enthusiasm (Poirier et al., 2019).

However, existing reviews have evaluated the effectiveness of distance education only with respect to students’ total test scores but have paid less attention to the differences between the scores of different subjects. In addition, the existing reviews focus more on high school students and college students but exclude lower-grade students. Moreover, compared to adults, minors have lower self-control (Wang et al., 2017), and self-control plays an important role in academic performance (Duckworth et al., 2019). The effectiveness of distance education may differ between K-12 students and college students. These studies also did not report non-academic outcomes, such as socioemotional development. The best way to resolve this controversy is to conduct a systematic review. Indeed, many teams have conducted systematic reviews with the theme of distance education. Therefore, based on the above studies, this study will integrate the existing evidence more comprehensively, conduct research specifically on K-12 students, and analyse and compare the differences among different subjects to better synthesize and supplement the existing studies.

The Contribution of This Review

Due to the flexibility and convenience of distance education, governments in many countries expect that distance education will become an important form of education and have formulated policies to promote the development of distance education, such as the Law on Distance Education and Lifelong Learning Revitalization Law of Japan (MEXT, 1990). Previous reviews have indeed proven the effectiveness of distance education (Bernard et al., 2004). However, they studied the students’ total scores but paid less attention to the differences between the scores of different subjects. In addition, the existing reviews focus more on high school students and college students but exclude lower-grade students.

Therefore, our study will draw a comprehensive and reliable conclusion to provide a basis for promoting the development of distance education, providing suggestions for teachers to effectively use the internet for teaching and improving distance education methods and formulating and perfecting laws related to distance education.

Objectives

The objectives of this study are the following:

(1) To evaluate the effectiveness of online distance education compared to other approaches on K-12 students’ academic performance.

(2) To evaluate the effectiveness of online distance education compared to other approaches on K-12 students’ non-academic performance, including classroom participation, learning interest, learning pressure, social skills and social isolation.

(3) To evaluate the potential moderators between online distance education and students’ performance and other outcomes, such as grade, country, subject, study duration, teaching content and methods, and family circumstances.

Methods

Criteria for Considering Studies for This Review

Types of Studies

We will include randomized controlled trials, non-randomized controlled trials and quasi-experimental studies with a comparison group. We included only studies with comparisons, including blank controls, placebo controls, positive controls, and crossover controls.

We will exclude qualitative studies, cross-sectional studies, cohort studies, and other observational studies. We will exclude before-after (pre-post) designs with no comparison group because they have no counterfactual thus do not tell us about effectiveness since other factors such as maturation or regression to the mean could explain differences (Ho et al., 2018).

Types of Participants

The participants of the systematic review are K-12 students. Due to the uneven age of enrollment in different countries, we use grade as the criteria rather than age. We don’t limit the countries and regions of included studies.

Types of Interventions

The interventions include distance education based on internet live sessions, online recorded videos, other online materials or a combination of these materials. The comparisons involve other educational methods, including traditional face-to-face classroom education and blended education, which combine the two approaches.

Types of Outcome Measures

We will extract the outcomes of all tests on students’ academic performance and non-academic performance, regardless of whether the outcome variable was binary or continuous.

The outcomes can be measured via examination or questionnaire. We don’t limit the examination methods. Standardized tests or papers compiled by researchers themselves can be used. For the questionnaire, we did not limit the type of questionnaire. The NRS can be a recognized standard scale or a questionnaire made by researchers themselves.

Primary Outcomes

The primary outcome is students’ academic performance, which included the total score or a subject score on the final exam and some standardized tests. We do not limit specific subjects which can be any subject such as reading, writing, mathematics, and science.

Secondary Outcomes

The secondary outcome is non-academic performance, which included classroom participation, learning interest, learning pressure, social skills and social isolation.

Duration of Follow-Up

There are no eligibility criteria based on the duration of follow-up. All follow-up periods of eligible studies were considered. During the analysis, studies with similar follow-up periods may be grouped together if there is a great degree of diversity in follow-up duration across studies (described below in more detail).

Types of Settings

We only include formal education in the school setting. This means that students must be enrolled in a specific school, and the course content must be within the regular curriculum.

Others

We only include studies in English or Chinese.

Due to the controversy over the quality of evidence for conference abstracts without peer-reviewed, for any potentially eligible conference abstracts, we will try our best to locate their full publications. If the full paper has been published, we will include, if not, we will exclude these conference abstracts.

Search Methods for Identification of Studies

We utilized the Campbell Searching for Studies Guide (Kugley et al., 2017) and collaborated with information retrieval expert Zhipeng Wei to formulate the following retrieval strategy.

Electronic Searches

The following databases will be searched from inception to present.

• APA PsychInfo (EBSCOhost)

• Education Source (EBSCOhost)

• ERIC (https://eric.ed.gov/)

• LearnTechLib (https://learntechlib.org/)

• Social Sciences Citation Index (Web of Science)

The following Chinese databases were also searched.

• CNKI (https://www.cnki.net/)

• VIP (https://qikan.cqvip.com/)

The specific search strategies are presented in Appendix 1, using Social Sciences Citation Index (Web of Science), ERIC, and CNKI as examples.

Searching Other Resources

Grey literature

We will consult the following sources of grey literature and search the websites of organizations devoted to education research to identify relevant unpublished studies and reports. The following grey literature resources are searched with the keywords “distance education”, “e-learning”, “distance learning”, “remote education”, or “remote instruction”.

• American Educational Research Association (https://www.aera.net/)

• Best Evidence Encyclopedia (https://www.bestevidence.org/)

• Education Endowment Foundation (https://educationendowmentfoundation.org.uk/)

• European Educational Research Association (https://www.eeraecer.de/)

• Open Grey (https://www.opengrey.eu/)

• What Works Clearinghouse (https://ies.ed.gov/ncee/wwc/)

Dissertations and Theses

We will search for master’s and doctoral theses through the following Dissertations & Theses databases.

• Open Access Theses and Dissertations (https://oatd.org/)

• ProQuest Dissertations & Theses Global (https://www.proquest.com/index)

Web Search Engines

We will also search Google Scholar with the keywords “intitle:” option: (“distance education”, “e-learning” or “distance learning” or “remote education” or “remote instruction”) and (K-12 or student), and we will stop the scan if there are 5 consecutive pages with no relevant studies.

Journals

We will perform an extensive hand search of topic-relevant renowned journals. We will browse at the tables of contents for all issues published within the last 5 years. We use a product called Journal Citation Reports from Web of Science to identify journals. The top ten journals based on their impact factor rankings, related to distance education and instructional technology in the category of EDUCTION&EDUCATIONAL RESEARCH have been identify.

• Computer Assisted Language Learning

• Computers and Education Open

• Computers & Education

• Distance Education

• Education and Information Technologies

• International Journal of Computer-Supported Collaborative Learning

• Journal of Computer Assisted Learning

• Journal of Computers in Education

• Journal of Research on Technology in Education

• Smart Learning Environments

Other Resources

In addition, we will conduct supplementary searches. First, we will manually search the lists of the references of the included studies and the citations of the included studies (using Google Scholar). Second, due to the unrestricted search for study design, we will annotate the systematic reviews related to distance education during screening and include their included studies as one of the sources for supplementary searches. Finally, we will contact and consult experts in distance education, such as John Daniel, Desmond Keegan and Tony Bates, to determine whether there are omissions in our search results.

This study will include only experimental studies; due to the nature of their research question, it is unlikely that government documents will be found; therefore, no such studies will be sought.

Data Collection and Analysis

Selection of Studies

We will use Rayyan to screen studies. All the retrieved studies will be independently screened by two evidence reviewers (Junjie Ren and Liping Guo) according to the inclusion and exclusion criteria. The whole screening process will be divided into two stages. In the first stage, the screening will be only conducted according to the title and abstract of the study. In the second round, the full text of the studies included in the first stage will be viewed and subsequently screened. We constructed a screening tool and applied it to the second stage of screening (Table 1). Two reviewers (Junjie Ren and Liping Guo) will check each item in order and exclude the literature as long as one item don’t meet the inclusion criteria, otherwise, it will be included. For studies in which only a subset of the sample is eligible for inclusion in the review, if the population eligible for inclusion in the review is reported separately, it will be included, otherwise, it will be excluded.

Table 1

Screening Tool

Study ID	Language	Study design	Population	Intervention	Decision
(An & Reigeluth, 2011)	Yes	Yes	No		Exclude
(Acar & Kayaoglu, 2020)	Yes	Yes	Yes	Yes	Include
(Arbogast et al., 2014)	Yes	No			Exclude
(Florit et al., 2012)	Yes	Yes	No		Exclude
(Lee et al., 2007)	Yes	No			Exclude
(Lewin, 2012)	Yes	No			Exclude
(Li & Zhao, 2021)	Yes	Yes	Yes	Yes	Include
(Lin et al., 2022)	Yes	Yes	Yes	Yes	Include
(Lombardi et al., 2017)	Yes	Yes	Yes	Yes	Include
(Mearns, 2012)	Yes	Yes	Yes	No	Exclude
(Moore, 2020)	Yes	Yes	No		Exclude
(Muhanna, 2012)	Yes	Yes	Yes	Yes	Include
(Palaigeorgiou et al., 2018)	Yes	Yes	Yes	No	Exclude
(Rézeau, 1991)	Yes	No			Exclude
(Smits et al., 2012)	Yes	Yes	No		Exclude
(Sultmann et al., 2021)	Yes	No			Exclude
(Tofel-Grehl et al., 2021)	Yes	No			Exclude
(Van Mensel et al., 2020)	Yes	Yes	Yes	No	Exclude
(Zarate & Gàndara, 2019)	Yes	No			Exclude
(Zheng et al., 2022)	Yes	No			Exclude

Any discrepancies between the two reviewers will be resolved through consultation. If the consultation fails to reach a consensus, the decision will be made by a third reviewer (Zhipeng Wei). The PRISMA study selection flow chart and a table of “characteristics of excluded studies” will be provided.

We conducted two rounds of pilot trials (Table 2). In the second pilot trial, the consistency rate of the results of the two reviewers’ screening studies reached 90%.

Table 2

Excluded Studies

Study	Reason for exclusion
(Mearns, 2012)	Wrong intervention
(Van Mensel et al., 2020)	Wrong intervention
(Tofel-Grehl et al., 2021)	Wrong study design
(Zarate & Gàndara, 2019)	Wrong study design
(Lewin, 2012)	Wrong study design
(An & Reigeluth, 2011)	Wrong population
(Smits et al., 2012)	Wrong population
(Florit et al., 2012)	Wrong population
(Rézeau, 1991)	Wrong study design
(Arbogast et al., 2014)	Wrong intervention
(Zheng et al., 2022)	Wrong study design
(Sultmann et al., 2021)	Wrong study design
(Palaigeorgiou et al., 2018)	Wrong intervention
(Moore, 2020)	Wrong population
(Lee et al., 2007)	Wrong study design

Data Extraction and Management

We will use Excel for data extraction and management. To ensure the consistency of the data extraction process, we developed and used standardized data collection forms (Table 3). The form is composed of four main parts. The first part includes bibliographic information, including title, first author, year of publication, journal and language. The second part includes sample characteristics, such as sample size, grade, region and number of males and females. The third part includes the intervention information, including the study design, subject, intervention, comparison condition, intervention duration and follow-up. The fourth part includes outcomes, including outcomes and their measurements, subgroups, subgroup variables, effect indices, durations of effect measurements, and sample sizes and effect sizes of the interventions and comparisons.

Table 3

Standardized Data Collection Form

Title

First author

Year of publication

Journal

Language

Sample characteristics: Age, grade, region, number of males

Intervention information: Study design, subject, intervention, comparison, intervention duration, follow up

Outcomes: Outcome, measurement, subgroup, subgroup variable, effect index, sample size, effect size

This process will be independently carried out by two reviewers (Junjie Ren and Liping Guo), and any differences will be resolved through discussion. If the discussion fails to reach a unified view, it will be decided by a third reviewer (Lingxia Zhao). Before the formal data extraction, we extracted 20 studies in two preextractions (Table 4). After two pilot rounds of data extraction, the consistency rate of the results of the two reviewers’ data extraction reached 90%.

Table 4

Data Extraction of Three Rounds Pilots

(Li & Zhao, 2021)
Basic information	- Title: An investigation into the influence of blended learning on oral English proficiency of senior high school students- First author: Li Xin- Year of publication: 2020- Journal: International education studies- Language: English
Sample characteristics	Age: NR; grade: 10th grade; region: China
Intervention information	- Study design: Non-randomized controlled trial- Subject: Oral English- Intervention: Blended education- Comparison: Traditional face-to-face education- Intervention duration: 5 months- Follow-up: NR
Outcomes	Outcome: Oral performance (measure: Oral test) - Subgroup: Total Effect index: Mean and SD Sample size = 60 Intervention: (Pre) n = 30, M = 72.16, SD = 0.08 (Post) n = 30, M = 79.00, SD = 11.99 Comparison: (Pre) n = 30, M = 72.08, SD = 0.08 (Post) n = 30, M = 73.00, SD = 11.01

(Acar & Kayaoglu, 2020)
Basic information	- Title: MOODLE as a potential tool for language education under the Shadow of COVID-19- First author: Aylin Acar- Year of publication: 2020- Journal: Eurasian journal of educational research- Language: English
Sample characteristics	Age: NR; grade: 11th grade; region: Turkey
Intervention information	- Study design: Quasi-experiment study- Subject: English- Intervention: Blended education- Comparison: NR- Intervention duration: 15 weeks- Follow-up: NR
Outcomes	Outcomes: Exam scores (measure: Non-standardized test) - Subgroup: Total Effect index: Mean and SD Sample size: 44 Intervention: (Pre) n = 22, M = 65.09, SD = 0.59 (Post) n = 22, M = 74.84, SD = 0.56 Comparison: (Pre) n = 22, M = 72.06, SD = 0.64 (Post) n = 22, M = 78.69, SD = 0.41

(Muhanna, 2012)
Basic information	- Title: The effectiveness of web-based curricula on seventh grade mathematics students in Jordan- First author: Wafa Muhanna- Year of publication: 2012- Journal: International education studies- Language: English
Sample characteristics	Age: NR; grade: 7th grade; region: Kingdom of Saudi Arabia
Intervention information	- Study design: Non-randomized controlled trial- Subject: Mathematics- Intervention: Distance education- Comparison: Traditional face-to-face education- Intervention duration: 8 weeks- Follow-up: NR
Outcomes	Outcomes: Academic achievement (measurement: Self designed non-standard test) - Subgroup: Total Effect index: Mean and SD Sample size: 148 Intervention: (Pre) n = 75, M = 54.52, SD = 7.83 (Post)n = 75, M = 85.88, SD = 7.82 Comparison: (Pre) n = 73, M = 54.78, SD = 7.19 (Post)n = 73, M = 78.40, SD = 10.59 - Subgroup: Gender (male) Effect index: Mean and SD Sample size: 70 Intervention: (Pre) n = 37, M = 64.16, SD = 6.12 (Post)n = 37, M = 84.59, SD = 7.82 Comparison: (Pre) n = 33, M = 56.13, SD = 7.36 (Post)n = 33, M = 76.94, SD = 9.60 - Subgroup: Gender (female) Effect index: Mean and SD Sample size: 78 Intervention: (Pre) n = 38, M = 54.87, SD = 9.33 (Post)n = 38, M = 87.13, SD = 6.98 Comparison: (Pre) n = 40, M = 53.65, SD = 6.95 (Post)n = 40, M = 79.60, SD = 11.32

(Lombardi et al., 2017)
Basic information	- Title: The impact of an online Transition curriculum on secondary student reading: A Multilevel examination- First author: Allison R. Lombardi- Year of publication: 2017- Journal: Career development and Transition for exceptional individuals- Language: English
Sample characteristics	Age: NR; grade: 9th, 10th, 11th grade; region: America
Intervention information	- Study design: Non-randomized controlled trial- Intervention: Blended education- Comparison: NR- Subject: Reading- Intervention duration: NR- Follow-up: NR
Measurement of outcomes	Outcomes: Reading (measurement: AIMS web Maze test for 8th grade reading prompts)- Subgroup: TotalEffect index: Mean and SDSample size: 307Intervention: (Pre) n = 223, M = 23.68, SD = 10.42(Post)n = 210, M = 26.03, SD = 11.11Comparison: (Pre) n = 115, M = 30.13, SD = 11.12(Post)n = 97, M = 31.18, SD = 12.61

(Lin et al., 2022)
Basic information	- Title: Using an online learning platform to show students’ achievements and attention in the video lecture and online practice learning environments- First author: Chih-Hung Lin- Year of publication: 2022- Journal: Educational technology & society- Language: English
Sample characteristics	Age: NR; grade: 11th, 12th grade; region: China
Intervention information	- Study design: Randomized controlled trial- Intervention: Distance education (video lecture)- Comparison: Distance education (online practice)- Subject: Mathematics- Intervention duration: NR- Follow-up: NR
Measurement of outcomes	Outcomes: Academic achievement (measurement: Achievement test from the Junyi Academy online platform) - Subgroup: Total Effect index: Mean and SD Sample size: 38 Intervention: (Pre) n = 19, M = 14.68, SD = 3.32 (Post)n = 19, M = 15.95, SD = 2.70 Comparison: (Pre) n = 19, M = 14.47, SD = 3.30 (Post)n = 19, M = 16.62, SD = 2.38 Outcomes: Attitude (measurement: Physiological signals) - Subgroup: Total Sample size: 38 Intervention: (Pre) n = 19, M = 40.59, SD = 15.20 (Post)n = 19, M = 55.37, SD = 5.32 Comparison: (Pre) n = 19, M = 45.57, SD = 11.75 (Post)n = 19, M = 49.34, SD = 4.90

For the controlled clinical trials, if there is no baseline data available, we will extract and synthesize only posttest data; if the pretest and posttest results are reported at the same time, we will extract only all the results to calculate double differences. The units of analysis issues are listed below for more details about the analysis issues.

Assessment of Risk of Bias in Included Studies

For randomized controlled trials, the Cochrane risk of bias tool will be used to assess quality and risk bias (Higgins et al., 2022). For nonrandomized controlled trials and quasiexperimental studies, the risk of bias in nonrandomized studies of interventions (ROBINS-I) will be used to assess quality and risk bias (Higgins et al., 2022). The quality and risk bias of all studies will be independently evaluated by two reviewers (Junjie Ren and Liping Guo). Any discrepancies between the two reviewers will be resolved through consultation. If the consultation fails to reach a consensus, the decision will be made by a third reviewer (Kehu Yang).

Measures of Treatment Effect

For continuous outcomes, we will use the mean and standard deviation to calculate the SMD and 95% CI. If the study doesn’t report the mean or standard deviation, the SMD will be calculated based on the F ratio, t value, χ2 value and correlation coefficient (Lipsey & Wilson, 2001). If we do not obtain enough information, we will contact the main researchers to request such information. Hedges’ g was used for estimating the SMD. Any standardized measures of student academic achievement (e.g., English and mathematics) are examples of relevant continuous outcomes in this review.

For dichotomous outcomes, we will calculate ORs with 95% confidence intervals (CIs). Classroom participation and social isolation are examples of possible relevant dichotomous outcomes in this review.

There are statistical approaches available to re-express dichotomous and continuous data to be pooled together (Sanchez-Meca et al., 2003). To calculate common metrics, odds ratios will be converted to SMD effect sizes using Cox transformation. We will only transform dichotomous effect sizes to SMD if appropriate; for example, as may be the case with, for example, classroom participation, which can be measured with binary and continuous data.

When effect sizes cannot be pooled, study-level effects will be reported in as much detail as possible. The software used for storing the data and performing the statistical analyses will be RevMan 5.4, Excel, and Stata 12.0.

Unit of Analysis Issues

We considered the unit of analysis of the studies to determine whether individuals were randomized into groups (i.e., cluster-randomized trials), whether individuals may have undergone multiple interventions, whether there were multiple treatment groups and whether several studies were based on the same data source.

• Clustered assignment of treatment

The cluster of randomized trials included in this review will be checked for consistency in the unit of allocation and the unit of analysis, as statistical analysis errors can occur when they are different. When appropriate analytic methods have been used, we will meta-analyze effect estimates and their standard errors (Higgins et al., 2022). In cases where study investors have not applied appropriate analysis methods that control for clustering effects, we estimate the intra-cluster correlation (Donner et al., 2001; Hedges, 2007) and correct standard errors.

• Multiple intervention groups and multiple interventions per individual

Studies with multiple intervention groups and different individuals may be included in this review, although only intervention and control groups that meet the eligibility criteria were used in the data synthesis. To avoid problems with dependence between effect sizes, we will apply robust standard errors (Hedges et al., 2010) and apply small sample adjustments to the estimator itself (Tipton, 2015). We will use the results of Tanner-Smith and Tipton tests to evaluate whether there are enough studies available for this method to consistently estimate the standard errors (Tanner-Smith & Tipton, 2014). The data synthesis procedure is described in more detail below (Tanner-Smith & Tipton, 2014).

If there are not enough studies, we will use a synthetic effect size to avoid dependence between effect sizes. This method provides an unbiased estimate of the mean effect size parameter but overestimates the standard error. Random effects models applied when synthetic effect sizes are involved actually perform better in terms of standard errors than do fixed effects models (Hedges, 2006). However, tests of heterogeneity when synthetic effect sizes are included are rejected less often than nominal tests.

If pooling is not appropriate (e.g., multiple interventions and/or control groups include the same individuals), only one intervention group will be coded and compared to the control group to avoid overlapping samples. The choice of which estimate to include will be based on our risk of bias assessment. We will choose the estimate that we judge to have the least risk of bias (primarily, confounding bias and, in the case of equal scoring, the missing outcome data domain will be used).

• Multiple studies using the same sample of data

In some cases, several studies may have used the same sample of data, or some studies may have used only a subset of a sample used in another study. We will review all such studies, but in the meta-analysis, we will only include one estimate of the effect from each sample of data. This will be done to avoid dependencies between the “observations” (i.e., the estimates of the effect) in the meta-analysis. The choice of which estimate to include will be based on our risk of bias assessment of the studies. We will choose the estimate from the study that we judge to have the least risk of bias (primarily, confounding bias). If two (or more) studies are judged to have the same risk of bias and one of the studies (or more) uses a subset of a sample used in another study (or studies), we will include the study using the full set of participants.

• Multiple time points

When the results are measured at multiple time points, each outcome at each time point will be analyzed in a separate meta-analysis with other comparable studies taking measurements at a similar time point. According to general guidelines, patients were grouped together according to the following criteria: 1) 0- to 1-year follow-up, 2) 1- to 2-year follow-up and 3) more than 2 years of follow-up. However, if the studies provide viable reasons for an adjusted choice of relevant and meaningful duration intervals for the analysis of outcomes, we will adjust the grouping.

• Multiple outcomes

When the primary studies reported results of multiple outcomes (both score of each subject and the overall), the overall score will be analyzed as the primary outcome, and each subject score will be analyzed with other comparable outcomes analysis as subgroups.

When the primary studies reported results of multiple outcomes (e.g., English and mathematics outcomes) without overall score, each outcome will be analyzed in a separate meta-analysis with other comparable outcomes as subject-specific outcome analysis.

Where studies report multiple measures of the same outcome, we average these effects to obtain one effect size per study. For example, if male and female scores are reported separately, we will average these scores to obtain the whole population effect size. However, we will retain the individual effects for gender subgroup analysis.

Dealing With Missing Data

If there are any missing data, we will contact the author for this information. If there is unavailable data, we will not analyze the data; rather, we will analyze only the available data. The study lacking data will be described in the results section. In addition, the potential impact of missing data on the results will be considered in the discussion section.

Assessment of heterogeneity

The chi-square test, Q test and I² statistic will be used to assess the heterogeneity of the studies. If I² ≤ 50%, we will consider the study to be homogeneous. If I²> 50%, we will consider the study to have great heterogeneity (Higgins & Thompson, 2002). We will use stratified analysis for subgroup analysis of categorical variables and meta-regression for continuous variables to explore the source of heterogeneity and explain it.

Assessment of Reporting Biases

If we include 10 or more studies, we will use a funnel plot and Egger’s test to evaluate publication bias (Robert, 1979). If the funnel is asymmetric and Egger’s test was p < 0.05, we will suspect publication bias. Otherwise, we will consider that there is no significant publication bias. When significant publication bias occurred, we will verify and correct the publication bias by trimming and filling (Duval & Tweedie, 2000). The possible reasons for this difference (e.g., nonreporting biases, poor methodological quality leading to spuriously inflated effects in smaller studies, true heterogeneity, artifacts, and chance) will be also considered (Higgins et al., 2022).

Data Synthesis

For continuous variables in studies, such as students’ academic performance, we will extract the mean and standard deviation (SD) and select the mean and 95% CI as the effect size indices. For categorical variables in studies, such as students’ academic performance, we will extract the respective numbers of outcomes for each category and select the odds ratio (OR) and 95% confidence interval (CI) as the effect size indices.

We will use Review Manager 5.4 software for data integration and assess the statistical heterogeneity by the Q test. Because different studies differ in terms of students, subjects, tools used to measure outcomes and other aspects, we choose to use a random effects model to synthesize effect quantities. The pooled estimates are presented in forest plots (Higgins & Thompson, 2002).

Planned Moderators

Gender

One study reported that the score of women is slightly lower than that of men (Mosqueda & Maldonado, 2013). Another study showed that the improvement in mathematics performance is greater for girls than for boys (Arroyo et al., 2013). Therefore, the impact of gender on the effectiveness of distance education is worth studying.

Learning Ability

One study found that students with poor grades improve more than students with good grades, which suggests that distance education may help reduce the differences between students’ grades (Spitzer & Musslick, 2021). Therefore, we will explore whether there are differences in the effectiveness of distance education for students with different learning abilities in this study.

Region

One study found that students from different regions in the same school have different grades (Crosnoe, 2005). Although distance education theoretically breaks the barrier of distance, it is still necessary to consider region as a potential regulatory factor.

Family Social Status

One study found that family social status was significantly associated with students’ academic grades (Bacete & Badenes, 2003). We believe that family factors may also have an impact on the effect of distance education.

Grade Level

One study found that students’ learning enthusiasm and the influence of the external environment differ according to grade level (Chen, 2008). Therefore, we suspect that different grades may also lead to different effects of distance education.

Subject

The existing systematic evaluation has reached different conclusions on the effect of distance education in different subject areas. Some studies believe that mathematics and science are different from arts and humanities in distance education (Bernard et al., 2004; Poirier et al., 2019). However, some studies believe that the subject field is not a statistically significant regulator of the effectiveness of distance education (Means et al., 2013). Therefore, we decided to explore again in this review whether there are differences in the effectiveness of distance education among different subjects.

Intervention Design Features

There are differences between synchronous distance education and asynchronous distance education in many aspects of intervention (Kuzminska et al., 2021). A study showed that they have different effects on academic performance, and the author says that perhaps combining the two may have better effectiveness (Demirtas & Türk, 2022). Therefore, we will analyze the effectiveness of different distance education models, including asynchronous distance education, synchronous distance education and bichronous distance education.

Study Time

During the COVID-19 pandemic, teachers did not have the time to develop or provide high-quality conventional internet distance education, nor could they be considered to have the same quality. Therefore, we will conduct a subgroup analysis based on the time of the study to compare the differences in the effectiveness of online distance education during the COVID-19 pandemic and before.

Sensitivity Analysis

Sensitivity analysis will be conducted by “Leave-one-out” to assess the impact of each study on the effect size of the meta-analysis (Cooper et al., 2009) to test the robustness of the meta-analysis.

Summary of Findings and Assessment of the Certainty of the Evidence

The Grades of Recommendation, Assessment, Development, and Evaluation Working Group (GRADE) approach will be used to document the certainty of evidence for each main outcome (Schunemann et al., 2011). In RevMan, the GRADEpro GDT function will support exporting yielded data to a “Summary of findings” table holding information on the following.

• A list of all important outcomes, both desirable and undesirable.

• A measure of the typical burden of these outcomes (e.g., illustrative risk, or illustrative mean, on control intervention).

• The absolute and relative magnitude of effect (if both are appropriate).

• Numbers of participants and studies addressing these outcomes.

• The grade of the overall quality of the body of evidence for each outcome (which may vary by outcome).

• Space for comments (see the GRADE handbook and Cochrane Handbook for Systematic Reviews of Interventions (Higgins et al., 2022)).

Treatment of Qualitative Research

We do not plan to include qualitative research because it does not involve the data needed for data synthesis.

Footnotes

ORCID iDs

Junjie Ren

Liping Guo

Kehu Yang

Author Contribution

Junjie Ren drafted the protocol, and all authors reviewed the draft and approved the final version. Howard, Kehu Yang, and Liping Guo provided guidance for the systematic review method, while Wei Zhipeng participated and led the information retrieval.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Major Project of the National Social Science Fund of China: Research on the Theoretical System, International Experience and Chinese Path of Evidence-based Social Science under Grant 19ZDA142.

Appendix

References

Acar

Kayaoglu

M. N.

(2020). MOODLE as a potential tool for language education under the shadow of COVID-19. Eurasian Journal of Educational Research, 20(90), 67–82. https://doi.org/10.14689/ejer.2020.90.4

Allen

Bourhis

Burrell

Mabry

(2002). Comparing student satisfaction with distance education to traditional classrooms in higher education: A meta-analysis. American Journal of Distance Education, 16(2), 83–97. https://doi.org/10.1207/s15389286ajde1602_3

Alzahrani

Wright

(2016). Design and management of a self-access language learning space integrated into a taught course. Studies in Self-Access Learning Journal, 7(2), 136–151. https://doi.org/10.37237/070203

Reigeluth

C. M.

(2011). Creating technology-enhanced, learner-centered classrooms. Journal of Digital Learning in Teacher Education, 28(2), 54–62. https://doi.org/10.1080/21532974.2011.10784681

Arbogast

Burke

R. V.

Muller

Ruiz

Knudson

M. M.

Upperman

J. S.

(2014). Randomized controlled trial to evaluate the effectiveness of a video game as a child pedestrian educational tool. Journal of Trauma and Acute Care Surgery, 76(5), 1317–1321. https://doi.org/10.1097/ta.0000000000000217

Arroyo

Burleson

Tai

M. H.

Muldner

Woolf

B. P.

(2013). Gender differences in the use and benefit of advanced learning technologies for mathematics. Journal of Educational Psychology, 105(4), 957–969. https://doi.org/10.1037/a0032748

Aykan

Yildirim

(2022). The integration of a lesson study model into distance STEM education during the COVID-19 pandemic: Teachers' views and practice. Technology, Knowledge and Learning, 27(2), 609–637. https://doi.org/10.1007/s10758-021-09564-9

Bacete

F. J. G.

Badenes

L. V.

(2003). Family and ability correlates of academic grades. Psychological Reports, 92(3), 858–860. https://doi.org/10.2466/pr0.92.3.858-860

Belousova

Mochalova

Tushnova

(2022). Attitude to distance learning of schoolchildren and students: Subjective assessments of advantages and disadvantages. Education Sciences, 12(1), 46. https://doi.org/10.3390/educsci12010046

10.

Bernard

R. M.

Abrami

P. C.

Lou

Borokhovski

Wade

Wozney

Wallet

P. A.

Huang

F. B.

Huang

(2004). How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74(3), 379–439. https://doi.org/10.3102/00346543074003379

11.

Chen

J. J.-L.

(2008). Grade-level differences relations of parental, teacher and peer support to academic engagement and achievement among Hong Kong students. School Psychology International, 29(2), 183–198. https://doi.org/10.1177/0143034308090059

12.

Cooper

Valentine

Hedges

(2009). Handbook of research synthesis and meta-analysis 2nd edition. Russell Sage Foundation. https://www.researchgate.net/publication/274565216_The_Handbook_of_Research_Synthesis_and_Meta-Analysis_Second_Edition.

13.

Crosnoe

(2005). The diverse experiences of Hispanic students in the American educational system. Sociological Forum, 20(4), 561–588. https://doi.org/10.1007/s11206-005-9058-z

14.

Curtis

Werth

(2015). Fostering student success and engagement in a K-12 online school. Journal of Online Learning Research, 1(2), 163–190. https://www.learntechlib.org/primary/p/150962/

15.

Demirtas

B. K.

Türk

(2022). Student performance under asynchronous and synchronous methods in distance education: A quasi-field experiment. International Review of Economics Education, 41(9), Article 100244. https://doi.org/10.1016/j.iree.2022.100244

16.

Donner

A. D.

Piaggio

Villar

(2001). Statistical methods for the meta-analysis of cluster randomized trials. Statistical Methods in Medical Research, 10(5), 325–338. https://doi.org/10.1191/096228001680678322

17.

Duckworth

A. L.

Taxer

J. L.

Eskreis-Winkler

Galla

B. M.

Gross

J. J.

(2019). Self-control and academic achievement. In Fiske

S. T.

(Ed.), Annual review of psychology. (70, pp. 373–399). Annual Reviews. https://doi.org/10.1146/annurev-psych-010418-103230

18.

Duval

Tweedie

(2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463. https://doi.org/10.1111/j.0006-341x.2000.00455.x

19.

Fırat

Kılınç

Yüzer

T. V.

(2018). Level of intrinsic motivation of distance education students in e-learning environments. Journal of Computer Assisted Learning, 34(1), 63–70. https://doi.org/10.1111/jcal.12214

20.

Florit

D. P.

Montaño

J. L. A.

Anes

J. A. D.

(2012). Distance learning and academic performance in accounting: A comparative study of the effect of the use of videoconferencing. Revista de Contabilidad, 15(2), 195–209. https://doi.org/10.1016/s1138-4891(12)70042-0

21.

Gaidelys

Ciutiene

Cibulskas

Miliauskas

Jukstaite

Dumciuviene

(2022). Assessing the socio-economic consequences of distance learning during the COVID-19 pandemic. Education Sciences, 12(10), 22. https://doi.org/10.3390/educsci12100685

22.

German Federal Government . (1976). Gesetz zum Schutz der Teilnehmer am Fernunterricht (Fernunterrichtsschutzgesetz - FernUSG). https://www.gesetze-im-internet.de/fernusg/BJNR025250976.html

23.

Göksu

İ.

Ergün

Özkan

Sakız

(2021). Distance education amid a pandemic: Which psycho-demographic variables affect students in higher education? Journal of Computer Assisted Learning, 37(6), 1539–1552. https://doi.org/10.1111/jcal.12544

24.

Habala

(2021). Teacher to student feedback in distance education. Teaching Mathematics and Its Applications: An International Journal of the IMA, 41(3), 256–265. https://doi.org/10.1093/teamat/hrab022

25.

Hedges

L. V.

(2006). 29 meta-analysis. In Rao

C. R.

(Ed.), The handbook of statistics (26, pp. 919–953). Elsevier. https://doi.org/10.1016/S0169-7161(06)26029-2

26.

Hedges

& L. V.

(2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32(4), 341–370. https://doi.org/10.3102/1076998606298043

27.

Hedges

L. V.

Tipton

Johnson

M. C.

(2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65. https://doi.org/10.1002/jrsm.5

28.

Higgins

J. P. T.

Thomas

Chandler

Cumpston

Page

M. J.

, & (Eds). (2022, 4 August, 2022). Cochrane handbook for systematic reviews of interventions version 6.3. https://www.handbook.cochrane.org

29.

Higgins

Thompson

S. G.

(2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558. https://doi.org/10.1002/sim.1186

30.

A. M. H.

Phelan

Mizubuti

G. B.

Murdoch

J. A. C.

Wickett

A. K.

Shyam

Gilron

(2018). Bias in before-after studies: Narrative overview for anesthesiologists. Anesthesia & Analgesia, 126(5), 1755–1762. https://doi.org/10.1213/ane.0000000000002705

31.

Kaplan

A. M.

Haenlein

(2016). Higher education and the digital revolution: About MOOCs, SPOCs, social media, and the Cookie Monster. Business Horizons, 59(4), 441–450. https://doi.org/10.1016/j.bushor.2016.03.008

32.

Kugley

Wade

Thomas

Mahood

Jørgensen

A.-M. K.

Hammerstrøm

Sathe

(2017). Searching for studies: A guide to information retrieval for Campbell systematic reviews. Campbell Systematic Reviews, 13(1), 1–73. https://doi.org/10.4073/cmg.2016.1

33.

Kuzminska

Morze

Mazorchuk

Barna

Dobriak

(2021). How to balance synchronous and asynchronous teaching and learning: A local study. E-learning [E-learning in the time of covid-19]. In: 13th annual international scientific conference on theoretical and practical aspects of distance learning - E-Learning in the time of COVID-19, electr network. October 11-12, Katowice-Sosnowiec, Poland.

34.

Kuzminykh

N. Y.

Pérez-Cota

González-Castro

M. R.

(2022). Comparative analysis of tools and solutions for online study. 17th Iberian Conference on Information Systems and Technologies (CISTI). January 22-25, Madrid, Spain.

35.

Lee

C.-H.

Yeh

Kung

R. J.

Hsu

C.-S.

(2007). The influences of learning portfolios and attitudes on learning effects in blended E-Learning for mathematics. Journal of Educational Computing Research, 37(4), 331–350. https://doi.org/10.2190/EC.37.4.a

36.

Lewin

(2012). Primary schools and ICT: learning from pupil perspectives [Book Review]. Educational Review, 64(4), 511–512. https://doi.org/10.1080/00131911.2011.619890

37.

Zhao

(2021). An investigation into the influence of blended learning on oral English proficiency of senior high school students. International Education Studies, 14(7), 27–35. https://doi.org/10.5539/ies.v14n7p27

38.

Lin

C. H.

W. H.

Lee

T. N.

(2022). Using an online learning platform to show students' achievements and attention in the video lecture and online practice learning environments. Educational Technology & Society, 25(1), 155–165. https://www.jstor.org/stable/48647037

39.

Lipsey

M. W.

Wilson

D. B.

(2001). Practical meta‐analysis. In Applied social research methods series (49). Sage Publications, Inc. https://psycnet.apa.org/record/2000-16602-000.

40.

Lockee

B. B.

(2021). Online education in the post-COVID era. Nature Electronics, 4(1), 5–6. [Editorial Material]. https://doi.org/10.1038/s41928-020-00534-0

41.

Lombardi

A. R.

Izzo

M. V.

Rifenbark

G. G.

Murray

Buck

Monahan

Gelbar

(2017). The impact of an online transition curriculum on secondary student reading: A multilevel examination. Career Development and Transition for Exceptional Individuals, 40(1), 15–24. https://doi.org/10.1177/2165143416681287

42.

Martin

Polly

Ritzhaupt

(2020). Bichronous online learning: Blending asynchronous and synchronous online learning. Educause Review. https://er.educause.edu/articles/2020/9/bichronous-online-learning-blending-asynchronous-and-synchronous-online-learning#

43.

Means

Toyama

Murphy

Baki

(2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record, 115(3), 1–47. https://doi.org/10.1177/016146811311500307

44.

Mearns

(2012). Using CLIL to enhance pupils’ experience of learning and raise attainment in German and health education: A teacher research project. Language Learning Journal, 40(2), 175–192. https://doi.org/10.1080/09571736.2011.621212

45.

Menabò

Skrzypiec

Sansavini

Brighi

Guarini

(2022). Distance education among Italian teachers: Differences and experiences. Education and Information Technologies, 27(7), 9263–9292. https://doi.org/10.1007/s10639-022-11008-5

46.

MEXT . (1990). Lifelong learning revitalization method. https://elaws.e-gov.go.jp/document?lawid=402AC0000000071

47.

Moore

A. B.

(2020). Increasing learning coach involvement to support cultivating student engagement in K-12 online classrooms (Publication No. 28378617) [Doctoral dissertation, The Johns Hopkins University]. ProQuest Dissertations & Theses Global. https://www.proquest.com/docview/2515592336

48.

Mosqueda

Maldonado

S. I.

(2013). The effects of English Language proficiency and curricular pathways: Latina/os' Mathematics achievement in secondary schools. Equity & Excellence in Education, 46(2), 202–219. https://doi.org/10.1080/10665684.2013.780647

49.

Muhanna

(2012). The effectiveness of web-based curricula on seventh grade mathematics students in Jordan. Journal of College Teaching & Learning, 9(4), 267–276. https://doi.org/10.19030/tlc.v9i4.7297

50.

Palaigeorgiou

Karakostas

Skenteridou

(2018). Touching and traveling on 3D augmented tangible maps for learning geography: The FingerTrips approach. Interactive Technology and Smart Education, 15(3), 279–290. https://doi.org/10.1108/ITSE-12-2017-0066

51.

Poirier

Law

J. M.

Veispak

(2019). A spotlight on lack of evidence supporting the integration of blended learning in K-12 education: A systematic review. International Journal of Mobile and Blended Learning, 11(4), 1–14. https://doi.org/10.4018/ijmbl.2019100101

52.

Rézeau

(1991). Evaluating computer‐assisted Language learning (CALL) as seen by pupils. Educational Media International, 28(1), 32–35. https://doi.org/10.1080/0952398910280108

53.

Robert

(1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638

54.

Sanchez-Meca

Marin-Martinez

Chacon-Moscoso

(2003). Effect-size indices for dichotomized outcomes in meta-analysis. Psychological Methods, 8(4), 448–467. https://doi.org/10.1037/1082-989x.8.4.448

55.

Santana de Oliveira

M. M.

Torres Penedo

A. S.

Pereira

V. S.

(2018). Distance education: Advantages and disadvantages of the point of view of education and society. Dialogia, 29, 139–152. https://doi.org/10.5585/Dialogia.n29.7661

56.

Schlosser

L. A.

Simonson

(2009). Distance education 3rd edition: Definition and glossary of terms. Information Age Publishing.

57.

Schunemann

Hill

Guyatt

Akl

E. A.

Ahmed

(2011). The GRADE approach and Bradford Hill's criteria for causation. Journal of Epidemiology & Community Health, 65(5), 392–395. https://doi.org/10.1136/jech.2010.119933

58.

Smits

P. B. A.

de Graaf

Radon

de Boer

A. G.

Bos

N. R.

van Dijk

F. J. H.

Verbeek

J. H. A. M.

(2012). Case-based e-learning to improve the attitude of medical students towards occupational health, a randomised controlled trial. Occupational and Environmental Medicine, 69(4), 280–283. https://doi.org/10.1136/oemed-2011-100317

59.

Spitzer

M. W. H.

Musslick

(2021). Academic performance of K-12 students in an online-learning environment for mathematics increased during the shutdown of schools in wake of the COVID-19 pandemic. PLoS One, 16(8), Article e0255629. https://doi.org/10.1371/journal.pone.0255629

60.

Sultmann

Lamb

Hall

Borg

(2021). Student perceptions of a trial Religious Education curriculum: Establishing baseline data. Journal of Beliefs & Values-Studies in Religion & Education, 42(2), 247–257. https://doi.org/10.1080/13617672.2020.1818923

61.

Tanner-Smith

E. E.

Tipton

(2014). Robust variance estimation with dependent effect sizes: Practical considerations including a software tutorial in Stata and SPSS. Research Synthesis Methods, 5(1), 13–30. https://doi.org/10.1002/jrsm.1091

62.

The Central People's Government of the People's Republic of China . (2019). China's educational modernization 2035. https://www.gov.cn/zhengce/2019-02/23/content_5367987.htm

63.

Tipton

(2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375–393. https://doi.org/10.1037/met0000011

64.

Tofel-Grehl

Ball

Searle

(2021). Making progress: Engaging maker education in science classrooms to develop a novel instructional metaphor for teaching electric potential. Journal of Educational Research, 114(2), 119–129. https://doi.org/10.1080/00220671.2020.1838410

65.

Tzankova

Compare

Marzana

Guarino

Di Napoli

Rochira

Calandri

Barbieri

Procentese

Gatti

Marta

Fedi

Aresi

Albanesi

(2023). Emergency online school learning during COVID-19 lockdown: A qualitative study of adolescents' experiences in Italy. Current Psychology, 42(15), 12743–12755. https://doi.org/10.1007/s12144-021-02674-8

66.

UNESCO . (2020). UNESCO’s support: Educational response to COVID-19. https://en.unesco.org/covid19/educationresponse/support

67.

United States federal government . (2020). Cpronavirus aid, Relief, and Economic Security Act. https://www.federalreserve.gov/supervisionreg/caletters/CARES_Act_Exam_Procedures_07.06.2020_final.pdf

68.

U.S. Department of Education . (1998). Learning anytime anywhere partnership. https://www2.ed.gov/programs/fipselaap/funding.html

69.

Van Mensel

Hiligsmann

Mettewie

Galand

(2020). CLIL, an elitist language learning approach? A background analysis of English and Dutch CLIL pupils in French-speaking Belgium. Language Culture and Curriculum, 33(1), 1–14. https://doi.org/10.1080/07908318.2019.1571078

70.

Wang

Fan

Tao

Gao

(2017). Age and gender differences in self-control and its intergenerational transmission. Child: Care, Health and Development, 43(2), 274–280. https://doi.org/10.1111/cch.12411

71.

Wendt

J. L.

Rockinson-Szapkiw

A. J.

(2015). The effect of online collaboration on adolescent sense of community in eighth-grade physical science. Journal of Science Education and Technology, 24(5), 671–683. https://doi.org/10.1007/s10956-015-9556-6

72.

Yan

L. X.

Whitelock-Wainwright

Guan

Q. L.

Wen

G. X.

Gasevic

D. G.

Chen

G. L.

(2021). Students' experience of online learning during the COVID-19 pandemic: A province-wide survey study. British Journal of Educational Technology, 52(5), 2038–2057. https://doi.org/10.1111/bjet.13102

73.

Zarate

M. E.

Gàndara

(2019). Can the LCFF improve teaching and learning for EL students? A review of the emerging research in California and directions for future implementation. Peabody Journal of Education, 94(2), 157–175. https://doi.org/10.1080/0161956X.2019.1598111

74.

Zheng

Zhang

Lau

E. N. S.

Zhang

P. K. H.

Yang

Mak

E. C. W.

Wong

S. Y. S.

(2022). Primary school students' online learning during coronavirus disease 2019: Factors associated with satisfaction, perceived effectiveness, and preference. Frontiers in Psychology, 13, Article 784826. https://doi.org/10.3389/fpsyg.2022.784826

Protocol: Effectiveness of Online Distance Education on Learning Outcomes Among K-12 Students: A Systematic Review

Abstract

Keywords

Background

The Problem, Condition, or Issue

The Intervention

Conceptual Framework

Why It Is Important to do This Review

The Contribution of This Review

Objectives

Methods

Criteria for Considering Studies for This Review

Types of Studies

Types of Participants

Types of Interventions

Types of Outcome Measures

Primary Outcomes

Secondary Outcomes

Duration of Follow-Up

Types of Settings

Others

Search Methods for Identification of Studies

Electronic Searches

Searching Other Resources

Grey literature

Dissertations and Theses

Web Search Engines

Journals

Other Resources

Data Collection and Analysis

Selection of Studies

Data Extraction and Management

Assessment of Risk of Bias in Included Studies

Measures of Treatment Effect

Unit of Analysis Issues

Dealing With Missing Data

Assessment of heterogeneity

Assessment of Reporting Biases

Data Synthesis

Planned Moderators

Gender

Learning Ability

Region

Family Social Status

Grade Level

Subject

Intervention Design Features

Study Time

Sensitivity Analysis

Summary of Findings and Assessment of the Certainty of the Evidence

Treatment of Qualitative Research

Footnotes

ORCID iDs

Author Contribution

Declaration of Conflicting Interests

Funding

Appendix

References