Evidence on Online Higher Education: The Promise of COVID-19 Pandemic Data

Abstract

Among the many disruptions caused by the COVID-19 Pandemic was the sudden move to online teaching in colleges and universities across the globe. In this paper, we provide a brief overview of existing literature on the effectiveness of online college programs relative to traditional in-person programs. We argue that pre-pandemic studies may have drawn overly pessimistic conclusions about online teaching in higher education. We highlight two important limitations of pre-pandemic studies, namely endogeneity bias and the use of older instructional technology. The data that will emerge from the forced shift to online instruction during the pandemic will help correct several of these biases and provide a more accurate picture of the hopes and challenges of online higher education. Finally, we also provide some preliminary evidence on virtual instruction and evaluation methods using a survey of online undergraduate and graduate classes. We find that large undergraduate classes benefitted greatly from the online format, while smaller graduate classes faced significant challenges. Empirical studies of post-pandemic data will help in identifying when and how online instruction can provide the effective instruction to students to address both the short-term goals of course and degree completion and long-term outcomes in the labor market.

Introduction

Online teaching has been the matter of much debate for the past two decades. Proponents of online instruction in higher education point to the promise of vastly expanding access to college education and of bending the higher education cost curve. At the same time, concerns have been raised about the quality of instruction and its poor track record in course and degree completion rates. This debate has, however, been temporarily rendered moot due to the circumstances created by the Covid-19 pandemic. Notwithstanding the incredible shock that this has thrown into the system, the shutdowns following the pandemic in March 2020 forced all educational institutions—especially higher education—to confront the challenges and opportunities of online teaching head-on. As and when the pandemic shows signs of receding, an important question that colleges and universities will face is whether to return to traditional in-person teaching or replace it with online teaching partially or completely. Crucial in answering this question will be the goldmine of evidence—both data and anecdotal—that will emerge from the last year and a half of online teaching across the globe about what teaching and evaluation strategies work best in an online setting and the challenges of scaling up online education.

Even before the pandemic, higher education was seeing a clear trend towards greater enrollment in online courses and online degree programs in the United States. With the reversal of the 50% rule of the Higher Education Act (HEA) by the US Congress in 2006, many for-profit online colleges came up. ¹ This was further followed by a novel experiment in online higher education in the form of Massive Open Online Courses or MOOCs provided by platforms such Coursera, Udacity and edX where college courses taught by faculty at elite universities like Harvard, Stanford and MIT are offered for free online. Over the years, using data from online courses and online programs, a growing body of research has developed on the effectiveness of online teaching. However, the conclusions drawn about online instruction from these studies have had limited validity due to empirical biases introduced by constraints posed by the available data pre-pandemic. The data that are now emerging in the years following the pandemic are likely to correct several of these biases. Our goal, in the current article is to discuss ways in which this new data can supersede pre-pandemic studies so that researchers in the fields of economics, business and education, armed with a natural experiment in the form of the pandemic, are in a good position to analyze the costs and value of online college programs. This will provide a clearer picture of its role in the future of higher education.

We start by describing the case for online learning in terms of expanding access and lowering costs. We then look at existing studies on online learning, describe their limitations and provide a guide to how post-pandemic data can address them. We conclude by providing a normative assessment of where we think online instruction can be beneficial and where there is a reason to be cautious based on our experience as instructors.

Access to Higher Education through Online Learning

Proponents of online teaching point to the potential for technology to greatly increase access to education. Online courses have the benefit of reaching large numbers of students without the restriction of physical space and geographical distances. The idea of “democratizing” higher education is epitomized by MOOCs where online learning platforms partner with the best faculty from major universities to provide courses that are practically free to vast numbers of students around the globe. Overall, there has been a clear trend towards greater enrollment in online courses. Between Fall of 2012 and Fall of 2018, online course enrollment in US post-secondary education increased by 29%, and in 2018–2019, about 79% of all colleges offered either online courses or entire online degrees (National Center for Education Statistics, 2020). However, in order to understand how this increase in enrollment influences access to higher education as a whole, we have to look at the interaction between online and in-person programs.

There are two ways in which online programs can expand access to higher education. First is a direct effect of making courses and programs available to individuals who would not have otherwise attended on-campus traditional college. An increased enrollment in online programs would suggest an expansion in the numbers of students receiving higher education. However, if online programs are simply substituting for on-campus programs, overall enrollment in college courses will not have increased. Goodman et al. (2017) found that this is not the case. Using enrollment data from an online MS in Computer Science degree offered by Georgia Institute of Technology, they show that the online degree option expanded the pool of students who would otherwise not have pursued the degree. Their study finds no overlap between the applicant pool of candidates in the online program and the in-person program. The applicants to the in-person program tended to be younger and more likely to be recent college graduates, those applying to online programs were mid-career professionals. This demonstrates that online programs have valuable features distinct from in-person classes that cater to the preferences of individuals who would not have applied to traditional campus programs.

Second, online programs also offer a competitive alternative to traditional brick and mortar colleges and hence enhance access to the latter. While the study by Goodman et al. points to online programs being distinct from campus programs, evidence that online colleges and programs exert competitive pressure on traditional colleges indicate that they are close substitutes. Deming et al. (2019) found that after the regulatory change to HEA that eased entry of online colleges in 2006, non-selective in-person colleges responded by increasing their expenditure on instructional spending suggesting an improvement in education quality. ²

Both the studies by Goodman et al. and Deming et al. provide a positive case for the expansion of online programs with respect to meeting unmet demand for higher education and competitive supply enhancing response from traditional colleges. However, the ultimate test of the desirability of expanded access in this manner is its contribution to student learning outcomes in the short-term and labour market outcomes in the long-term. As we discuss below, so far, pre-pandemic studies provide a pessimistic view of online programs in this regard.

Pre-pandemic Evidence on Student Outcomes at a Glance

Research generally shows that student outcomes are worse in online programs than their in-person counterparts. Bettinger et al. (2017) studied the undergraduate students at a large for-profit institution and found that, controlling for observed characteristics, students taking online courses had significantly lower grades than students taking in-person versions of the same course. Further, students who enrolled in online classes were less likely to stay enrolled in the degree program a year out. In keeping with this finding, Krieg and Henson (2016) showed that online students had lower grades in follow-up courses, thus pointing to downstream losses in the value of instruction received in an online course. Hart et al. studied the data from California community colleges to reach similar conclusions—online students had worse grades, were more likely to repeat the course and less likely to take follow up classes.

Studies have also shown that the performance gaps in grades and course completion was worse for low-performing students and students of color. Xu and Jaggars (2014) use administrative data from undergraduate students enrolled in community and technical colleges in Washington state. They found, in all cases, that online students had lower grades than their in-person counterparts and this difference was greater for male, Black and academically less well-prepared students. Figlio et al. (2010) conducted an experiment where students of a large introductory economics class were randomly assigned to in-person or online versions of the same class. All instructional resources were kept identical with the only difference being that in-person students received live instruction from the instructor while online students watched videos of the lecture online. They find that test scores were moderately higher for the live instruction students than online students. They also find that this difference was highest for male Hispanic students.

Labour market outcomes of online degree students also do not bode well for the quality of online programs. Hoxby (2018) finds that although an online college degree increases wage growth faster than for individuals without a degree it does not compensate for the cost of online education thus resulting in a negative rate of return on the educational investment. Audit studies with fictitious applicants to real job openings find that applicants with traditional degrees receive higher call backs from employers than otherwise observationally equivalent applicants with online degrees (Deming et al., 2016; Lennon, 2020).

While the existing data point a bleak picture of the value of online education, we believe that the conclusions from existing studies is limited in several ways. In the following section, we discuss the ways in which new data that will emerge after the widespread adoption of online teaching during the pandemic can better inform us of the value and costs on online learning.

Hope of Post-pandemic Data

The proliferation of online programs in higher education since 2006 has provided researchers with many years of data to evaluate their performance. However, there are several caveats to the conclusions that we can draw from the studies thus far.

First, most online programs that have existed so far have involved little to no instructor interaction. Online courses and degrees are simply digital versions of distance learning programs where students receive instructional material and are expected to learn independently. Given that distance learning programs have historically under-performed relative to campus programs, it is not surprising then to find that their online versions also suffer from similar problems (Merisotis & Phipps, 1999; Sherritt, 1996; Simonson et al., 2011). This has, however, changed in the last year as online learning platforms have incorporated video conferencing technology that support virtual classrooms.

Recent studies have shown that synchronous online teaching through virtual classrooms provide better outcomes for student learning than asynchronous classes (Cellini & Grueso, 2021). This method of instruction enabled by new technology has brought online instruction one step closer to in-person instruction in a way that was not possible before. Student interactions with the instructor as well as with other peers is now possible in a virtual format. Still, questions remain as to whether virtual classrooms provide the same experience as a physical classroom. For example, student surveys indicate that many students struggle with staying focused online (Business Wire, 2020). Additionally technological challenges with internet connections, access to devices, etc. create disparities in the quality of instruction received by economically disadvantaged students (The Hill, 2020). In our own experience as virtual instructors for the last few semesters, we have found that some of these challenges can be addressed by adopting a flipped classroom format where lecture videos are provided in advance and online classroom time is utilized for addressing student questions, discussions and problem solving. The ability to record online lectures and make them available to students provides another way of dealing with disruptions that may occur during class time. Further, closing the digital divide by providing easy access to devices and high-speed internet should be a priority for educational institutions as well as governments at the local, state and federal levels.

Given the scale at which the online teaching experiment was carried out during the pandemic, we suspect that there has been a wide variety of online teaching strategies adopted by different instructors. To the extent that these were codified in some way instructors, the data should tell us a great deal about what works best and where in online instruction.

The second major problem with most existing online studies is the self-selection bias in the data. Before Spring of 2020, students could choose whether to enroll in online courses, programs or colleges. This introduced an endogeneity bias in any empirical analysis of the relationship between instruction medium and student outcomes. For example, if it is the case that many students who opt for online programs are low-performing students, the correlation between low grades and online instruction does not tell us much about the causal relation between the two. Some of these biases are addressed in these studies by using distance from local college or course offerings as instruments. However, there are limitations to the validity of these instruments as well to the extent that students’ choice of location decisions and courses are correlated with their unobserved characteristics related to academic performance. The forced shut-down of in-person classes during the pandemic eliminates the need for such instruments by providing an exogenous natural experiment. All students moved online and hence we can track outcomes for both low and high performing students through their course progression.

Third, audit studies that point to potential labor market biases against online degree holders suggest that employers place a lower value online degrees than traditional degrees. There are two possible reasons for this. One, the self-selection of low performing students into online programs may act as a signal to employers about unobserved ability of the candidate. Alternatively, if online programs in fact provide lower levels of employment productivity for students, then the labour market is correctly pricing labor based on human capital value. We believe that online education is likely to see a correction in both these aspects of labour market acceptance. As online programs become more widely used by well-reputed universities, the pool of applicants with online degree will suffer less of a bias. Further, as online classes adopt better pedagogical innovations facilitated by technology, the social value of online education will converge with that of traditional education and may even surpass it. Above all, as discussed earlier, the pandemic will serve as an important tool in testing both the selection bias and relative student outcomes for online vs traditional learning methods.

As a final point, the disruption of the pandemic and the move to online teaching has called into question traditional methods of evaluating student performance in a course. In the absence of proctoring, in-class exams have been difficult to administer and monitor for instances of cheating. Some colleges adopted digital proctoring services; however, such surveillance software have raised important privacy concerns. Along with experiments on online teaching strategies, instructors have also tried creative ideas on evaluating students for a grade ranging from more personalized computer-based testing, scenario-based learning and collaborative assessments of students. It is likely that such evaluation methods will be used not just in online formats but in a traditional classroom as well (Vox, 2020). Emerging research in this area will further enhance our understanding of how to correctly evaluate student learning.

Some Preliminary Findings

Anecdotal and survey evidence from classes in the authors’ own department in the last year provides some markers that would benefit from more rigorous data analysis. A survey of students in online courses taught synchronously at the department of Economics and Business in the City College of City University of New York shows that more than 40% of students found it easier to participate in class. A majority of students found it easier to interact with the instructor online than in person. Evidence from experiences shared by students in the survey suggests that they highly valued the flexibility provided by online classes. This was especially the case for working students. On the negative side, more than 50% of the students found it more challenging to engage and socially interact with other classmates. About 40% of students also found it harder to focus online. Despite the negatives, overall, a majority of students preferred the online format to the in-person format.

Cutting the survey results by course suggests to us that online instruction may be more appropriate for some classes than others. Among the five courses included in the study were two large introductory classes with 100 or more students, one small elective class with 10 undergraduate students, one Masters’ level class and one PhD-level course in Financial Economics. Students in the large introductory classes showed the highest favorability towards the online format. This is likely because in-person teaching for a large class like this usually occurs in a large auditorium where there is no feasible way to incorporate student-instructor interaction. In contrast, the online format allowed students to ask questions and participate during the lecture through chats and virtual feedbacks like raising their hand. Further, the large asynchronous class enables students to interact with the material and instructor at their own pace and time (via chat and discussion board features). These features make it possible to use flipped classroom methodology even in case of large enrollments. On the other hand, the elective undergraduate class with 10 students reported less favorably to online teaching. The graduate classes were moderately positive in their endorsement of online teaching. There was a greater concern about being able to focus online. This may have been driven by the fact that the graduate classes were two-hours or longer in duration compared to the other classes that were an hour and fifteen minutes or less. The PhD class followed a seminar format. Generating robust discussions on research and future ideas was found to be challenging in a virtual environment. Appointing student leaders for each session and assigning responsibilities for each seminar paper helped mitigate some of these challenges, but the observed discussion quality remained below that of prior in-person sessions.

Our survey results suggests that online classes are likely to be more effective in some courses than others. Large introductory classes that traditionally require large physical spaces, but fewer classroom discussions are likely to benefit from a transition to online instruction. Smaller classes with more advanced and specialized instructional material appear to perform better under a traditional classroom setting. In such cases, a move to online teaching can be more effective only when technology can be used to preserve or even enhance interactivity among course participants.

We also found that traditional ways of evaluating students through individual exams and tests are problematic in online courses due to the difficulty of monitoring students. Using exams as a collaborative experience rather than individual testing was preferable from both learning and accountability perspectives, as was using long application-oriented questions rather than multiple or right-wrong questions. Case studies and blog/vlog posts that encouraged students to interact in small groups were also found to be incredibly useful. Requiring students to comment on others’ videos and blog posts ensured students kept pace with materials assigned to them.

Needless to say, our survey was conducted at a very small-scale and is perhaps applicable to a narrow field of departments where lectures can be moved online relatively easily. The task is likely to be more formidable for fields that require lab work and practical skill training. As technology continues to improve, it is not impossible that many of these programs can also incorporate remote options with virtual labs and remotely operated equipment (Waldrop, 2013).

Conclusion

We present a summary of the current research on the effectiveness of online instruction in a higher education setting. The shut-down of colleges and universities across the world provides a natural experiment to improve our understanding of online teaching techniques and its value relative to in-person teaching. We also provide some preliminary evidence of student preferences between virtual and in-person Economics classes from a survey of economics and finance classes offered in the City University of New York. Evidence shows that overall, students prefer the online format although the responses varied by class size, course material and course duration.

As the world emerges out of the pandemic in fits and starts, it seems inevitable that online instruction will be a crucial part of higher education pedagogy and is here to stay for the foreseeable future. As instructors and researchers, we look forward to new evidence on optimal teaching and evaluation strategies in both virtual and traditional classrooms.