Do MOOCs Make You More Marketable? An Experimental Analysis of the Value of MOOCs Relative to Traditional Credentials and Experience

Abstract

Massive open online courses (MOOCs) are marketed as opportunities for participants to improve their labor market outcomes, and tens of thousands of students have paid for career-focused MOOC certificate programs. However, there is limited and conflicting research on MOOCs and labor market outcomes. Using two randomized control trials, we test Amazon’s Mechanical Turk respondents’ preferences for hiring hypothetical freelance web developers. In the first experiment, we examine preferences for profiles with MOOCs as compared to profiles with traditional degrees (bachelor’s, associates, and community college certificates). Respondents preferred all traditional degrees over a MOOC; MOOCs do not serve as substitutes for traditional postsecondary credentials. In the second experiment, we examine preferences for MOOCs as compared to no listed credential. Results demonstrate a 61–percentage point preference for MOOC credentials compared to no MOOC credential; the preference does not vary across levels of professional experience.

Keywords

economics of education educational policy experimental design higher education postsecondary education regression analyses

Massive open online courses (MOOCs) are a relatively new way of offering postsecondary education and training to individuals around the world. Over 101 million students have enrolled in over 11,400 MOOCs and instructors at over 900 universities have offered these classes (Shah, 2018). MOOCs have shifted from entirely free courses to more paid content, and the number of paying users has increased each year (Shah, 2018).

Although many people take these courses for personal enrichment, participants commonly enroll with the intent of learning new skills to further their own career goals (Dillahunt et al., 2016; Kizilcec & Schneider, 2015). Indeed, many people who have completed MOOCs report that what they learned has positively affected their career performance (Zhenghao et al., 2015).

The professional purpose of MOOCs is reflected both in the proliferation of career focused credentials being offered by MOOC providers and in the ways they market their offerings. Dozens of microcredentials (groups of related courses) are offered across platforms, including Coursera’s Specializations, edX’s MicroMasters and Professional Certificates, and Udacity’s Nanodegrees. These microcredentials tend to be in professionally oriented domains, such as Coursera’s Digital Marketing Specialization and Udacity’s Nanodegree in Google AdWords. The marketing for these programs also reveals their professional orientations: “She learned . . . the skills that ultimately made her dreams a reality” (Watson, 2018) and “Master in-demand skills . . . Earn a valued credential. Launch your career in Data Science, . . . Be in demand” (Udacity, 2018). Online Supplemental Appendix A illustrates the labor market focus of MOOCs with screenshots from the landing pages from Udacity and Coursera.

Despite these marketing efforts, little is known about the effect of such credentials in various labor markets. Surveys of MOOC students indicate perceived career benefits but evidence from the perspective of those doing the hiring is thin and mixed. Surveys of hiring managers indicate that they would view a MOOC positively, but they also state that they would prefer traditional degrees to MOOCs (Radford et al., 2014; Rosendale, 2016). While these surveys provide useful attitudinal measures, they are limited for two reasons. First, general surveys conducted outside an actual hiring context are unlikely to provide the most valid measures of the value of MOOC credentials. Second, MOOCs are popular for workers who participate in less formal labor markets, such as piece-rate work online (Perera, 2017; Taylor, 2017). Such labor markets often operate differently from traditional application-based labor markets, so the applicability of surveys from hiring managers at firms who are far removed from these labor markets is unclear.

Our study improves on the existing survey evidence by offering the first causal evidence of the effect of MOOC certification on hireability and provides the first evidence on how MOOC credentials are viewed in one common, informal labor market: hiring a freelance web developer. We use two experiments to causally assess how online respondents view MOOC credentials relative to having traditional educational credentials and relative to having no MOOC credential. We also consider whether the value of a MOOC credential varies at different levels of worker experience. Our specific research questions are as follows:

Research Question 1: How does the effect of a MOOC credential compare to the effect of traditional postsecondary credentials on stated preferences for hiring in a freelance setting?

Research Question 2: Does the presence of a MOOC credential on a freelance worker’s profile, as compared to no stated postsecondary credentials, influence potential clients’ stated preferences for hiring him? Does this effect vary by professional experience?

Research Question 3: How does the effect of MOOC credential compare to the effect of professional experience on preferences for hiring in a freelance setting?

To answer our research questions, we present online respondents with pairs of similar hypothetical freelance workers’ profiles and randomly vary the educational credentials and years of professional experience listed on the profiles. Respondents choose which candidate they would recommend hiring.

Our results demonstrate that respondents prefer the three traditional postsecondary credentials we tested (BA [bachelor of arts], AA [associate in arts], and community college certificate) relative to a MOOC credential. However, when no other education is explicitly mentioned, respondents prefer candidates with a MOOC credential relative to no MOOC credential. We also conclude that the effect of the MOOC on hypothetical hiring preferences does not vary by the number of years of work experience the fictional candidate has.

These experimental results provide valuable information about a growing sector of the economy (Kässi & Lehdonvirta, 2018) and align with the two extant surveys of hiring managers in more formal labor markets. Our work extends the survey evidence by quantifying the value of MOOCs relative to other credentials. Although the presence of a MOOC may appear as a positive signal of motivation and a desire for continued learning or reveal a greater ability, these credentials are not viewed as adequate replacements for traditional degrees.

Literature Review

MOOC providers promote their courses as education that builds marketable skills. The kinds of certificates and credentials offered by large MOOC providers reflect the modal student profile of a highly educated worker and reflect that many MOOC students state that they are looking to change career fields or retool for their current job (Dillahunt et al., 2016; Ho et al., 2015; Kizilcec & Schneider, 2015). For example, Coursera offers certificates for completing groups of courses (“specializations”). These specializations, which award official certificates, require the completion of several courses (usually 4–6), take approximately 3 to 6 months to complete, conclude with a capstone project or exam, and cost around $250 (Coursera, 2014; Pickard, 2018). In 2020, Google announced a plan to offer three new certificate programs via Coursera, which will be taught by Google employees and will be treated as equivalent to college degrees within the company (Hess, 2020). Udacity has similar offerings, called Nanodegrees, that offer multicourse project-based degrees. These take an average of 3 to 5 months to complete, and cost around $900 (Pickard, 2018). EdX offers a similar microcredential called a Professional Certificate. Students who successfully complete such certificates can add a badge to their LinkedIn profile, signaling their credentials to potential employers.¹

While the number of new users enrolling in MOOCs has declined slightly in the past few years (20 million new users in 2018, as compared to 23 million new users in 2017), the number of paying users continues to increase. There are over 600 microcredentials offered across platforms. Coursera offers 310 MOOC credentials and is the largest provider (Shah, 2018).

Our study focuses on identifying the value of these MOOC credentials in an informal labor market, both relative to traditional postsecondary credentials and experience and relative to having no education explicitly stated. The limited existing research on the effects of MOOCs on marketability indicates two seemingly contradictory narratives. Surveys of MOOC completers indicate positive effects on job prospects, but surveys of hiring managers suggest MOOCs are viewed unfavorably. Below, we consider evidence of both perspectives.

Students appear to derive value from their MOOC education in the labor market. A large survey of MOOC completers found that, among the roughly half of respondents who primarily took the MOOC for career goals, 87% reported the MOOC provided a benefit to their career (Zhenghao et al, 2015). These benefits include improving skills for a current job (62%), successfully finding a new job (26%), and even starting their own business (9%).

Although students’ responses suggest highly positive effects of MOOCs, there are several concerns with relying on such self-reports. First, an individual may not be able to accurately assess whether a specific course or credential is responsible for labor market outcomes. Indeed, a different survey found that while 47% of respondents said they were taking a MOOC for an employment related reason, only 10% of these respondents were able to list the specific ways in which the MOOC had helped their chances of employment (Dillahunt et al., 2016). This divergence may be because a large portion of MOOC participants, including many respondents to these surveys, were already employed and highly educated. Second, there is likely substantial selection bias into taking a MOOC, as more motivated and ambitious workers are more likely to pursue these credentials. Therefore, it is hard to identify the MOOC as the cause of career success. Additionally, these surveys do not test of the value of taking these courses in lieu of traditional degree programs.

An alternative to student self-reports is eliciting hiring managers’ perceptions of online credentials via surveys. Rosendale (2016) investigated 202 employers’ perceptions of MOOCs and other degrees using an online survey. Overall, hiring managers stated that they preferred job candidates with traditional degrees as opposed to candidates with equivalent training through MOOCs. About half of the respondents did not know what a MOOC was. Open-ended responses indicated that employers generally believed that a MOOC education might result in less training in communication, collaboration, and teamwork skills than an education from a traditional college. Additionally, many managers doubted the legitimacy of the MOOC providers and responded that they believed that MOOC credentials require less commitment and are less rigorous than traditional higher education degrees.

Radford et al. (2014) similarly find that MOOCs are not considered adequate substitutes for traditional credentials. In a survey of 103 human resources professionals, they found that while most hiring managers would view a MOOC favorably on a resume (85% of those who had heard of MOOCs reported that they would view a MOOC “positively” or “very positively”), the vast majority of those surveyed said that they were less likely to think that a MOOC demonstrated a specific skill or ability as compared to a traditional credential. These findings comport with Kizilcec et al. (2019) who find that survey respondents believe that online degree programs are less legitimate, less rigorous, and less well-respected than traditional degrees.

The hiring manager surveys resolve some of the self-reporting issues associated with student responses but do not overcome the selection bias issues of who takes MOOCs. Furthermore, asking for general preferences of hiring managers devoid of any context may not elicit accurate information and preferences about hiring decisions.

One increasingly common way to elicit preferences for hiring decisions in authentic settings is to use a resume audit design. In such studies, experimentally manipulated resumes are sent to job postings and the number of call-backs is recorded for each fictional resume. In these studies, some characteristics, such as race, gender, and years of experience, can be held constant, while others, such as college attended, can be experimentally manipulated. Such designs have been used to examine the effect of race, country of origin, college selectivity, and for-profit college attendance, on a candidate’s probability of being hired (Bertrand & Mullainathan, 2004; Darolia et al., 2015; Deming et al., 2016; Oreopoulos, 2011). To our knowledge, no resume audit studies have been published that examine the effect of MOOC credentials on labor market outcomes.

There are two general problems with resume audit studies and surveys of hiring managers. First, they ignore less traditional, but very common, labor markets such as those for freelance work. MOOCs are popular among workers who participate in such labor markets (Perera, 2017; Taylor, 2017). Second, the labor markets in which MOOCs are common often use hiring practices that diverge significantly from more traditional processes. Many of these less-formal labor markets are supply- rather than demand-centered; instead of applicants responding to posted listings, employers (e.g., people looking for a web designer, editor, babysitter, etc.) can browse posted applicant profiles. Many large employment sites (e.g., Care.com for nannies, Upwork for a variety of tech-related jobs, and TaskRabbit for household work) are organized around posted profiles that potential employers can browse. Thus, in attempting to estimate the effects of various factors on hireability in these scenarios, traditional resume audit studies are not an appropriate approach.

Our study situates the consideration of the labor market value of a MOOC in a novel context: hiring a freelance web developer via an online freelance platform. This setting of making a hiring decision about a short-term web development job is realistic, as there is a rich and growing market for freelance web developers (Agrawal et al., 2015). The demand for online freelance work is growing approximately 20% each year; tens of millions of workers have posted profiles on platforms such as freelancer.com and upwork.com (Kässi & Lehdonvirta, 2018; Kuek et al., 2015), and the majority of posted jobs are in technology (Kässi & Lehdonvirta, 2018). For example, a search for freelancers using the search term “web develop*” on the popular website Upwork² returns over 5,000 active freelancers looking for hourly work.

Past work has found that the perceived skill-set of freelance workers is one of the attributes that is most strongly correlated with hiring decisions in online labor marketplaces (Kokkodis et al., 2015). This is probably especially true for web developers, as they produce a product whose quality can be measured, and the job relies relatively little on social and interpersonal skills. Thus, examining how potential hirers make judgments about perceived skills or motivation in freelance markets is a necessary advance in this research literature.

In summary, our study extends the literature by making several concrete contributions. Through our experiments, we causally estimate the magnitude of the difference between a MOOC and traditional credential (or no stated credential) in being the preferred candidate for a job. We also test the value of the MOOC at different levels of professional experience and compare the value of a MOOC relative to differences in years of experience. Finally, we assess the value of a MOOC in the specific context of comparing two hypothetical candidates for a freelance web development job as opposed to generally asking respondents about their perceptions of MOOC credentials.

Theoretical Framework

Fugate et al. (2004) define employability as several person-centered dimensions: career identity (how a person defines oneself in relation to a career), personal adaptability (a person’s ability to remain productive and attractive to employers in a changing work environment), social capital (the professionally beneficial ties within one’s social network), and human capital (skills and knowledge that are valuable in the labor market) (Becker, 1962). These dimensions capture a person’s ability to identify and achieve career opportunities in a changing economy, including moving between jobs.

MOOCs, like other forms of postsecondary education, could affect each of these dimensions of employability. First, taking a MOOC could affect a person’s conception of their career trajectory or potential jobs. Second, a MOOC could expand a person’s professional social network. Simply by being in class with fellow students interested in the same content, MOOC participants may learn about job opportunities. Because our study relies on profiles of fictional employees, we can rule out any effects due to an employee’s perceptions of their career identity/trajectory and any effects of social capital.

Third, a MOOC could signal to employers that a worker is capable of and interested in remaining productive in a changing work environment. Finally, a MOOC can increase a worker’s skills and knowledge. The application of these aspects of the theory in our scenario is clear: A MOOC credential in web design that teaches specific skills might enable the worker to be more productive, and the MOOC sends signals regarding the worker’s interest and commitment.

Given this framework, when comparing MOOCs to traditional postsecondary credentials, we hypothesize that a 4-year bachelor’s degree is likely to both teach a greater number of skills and provide a stronger signal of productivity than a five course MOOC credential, and hence will be preferred in the hiring process. We predict the same is true, though to a lesser extent, for associate degrees.

The comparison with community college certificates is less clear and respondent hirers could prefer either traditional credentials or MOOC credentials. Traditional community college credentials are widely understood, generally well regarded in the labor market, and do not suffer from the same stereotypes and stigma that credentials earned online do (Columbaro & Monaghan, 2009). However, MOOCs may have some advantages such as nimbly adapting to changing labor markets and teaching skills that are of more immediate use to employers than community college certificates. This is likely especially true in the rapidly evolving context of computer science.

Based on Fugate et al.’s framework, we hypothesize that the profile with the MOOC credential should be preferred to a similar profile with no MOOC credential and that this preference should be stronger at lower levels of experience. At higher levels of experience, we hypothesize that employers are less likely to value a credential. However, we also acknowledge that MOOCs (and online courses generally) are not universally valued (e.g., Kizilcec et al., 2019), and some employers may have a negative view of these courses such that their addition to a job profile may lower the preference for that employee. Hence, measuring the magnitude of the preference, on average, is valuable.

Method

Our study reports on the results of two experiments that seek to determine if, and to what extent, MOOC credentials are valued in a labor market over other traditional credentials or no stated education credential. In each experiment, Amazon’s Mechanical Turk (MTurk) respondents were asked to assist a friend in hiring a freelance web developer by choosing between two similar fictional candidates. We randomly assigned the candidate profiles different educational credentials, which allows us to determine the causal difference between credentials on respondent preferences for hiring. We describe the experiments and analysis in more detail below.

Experimental Procedures

To test preferences for hiring freelance workers with different credentials, we embed a randomized control trial into an online Qualtrics survey offered to MTurk respondents. Each study began with a vignette telling respondents that their friend needs a website for her small business, has narrowed her choice down to two freelance workers found on a job website, and has asked for help selecting one to hire. The survey then asked the respondent to read the profiles of the two different freelance web developers (Brad and Todd) and indicate their preference for which one to hire. After selecting the preferred profile, respondents completed a short set of demographic questions.

In each experiment, respondents were randomly assigned to a comparison that presented two distinct profiles (“Brad” and “Todd”), each of which contained a randomly selected set of experience and education credentials. The comparisons are summarized in Table 1. In the first experiment, MOOC versus traditional credentials, we test the comparison of MOOCs to three traditional credentials (BA degree, AA degree, and a certificate from a community college). In the second experiment, MOOC versus no MOOC, we test the comparison between a profile that lists a MOOC credential and one which lists no credential.

Table 1

Experimental Comparisons

MOOC Versus Traditional Credentials
Comparison	Profile A	Profile B
1	MOOC certificate	BA degree
2	MOOC certificate	AA degree
3	MOOC certificate	Community college certificate
4	Community college certificate	BA degree
5	Community college certificate	AA degree
6	AA degree	BA degree
MOOC Versus no MOOC
Comparison	Profile A	Profile B
7	1 year of experience, MOOC certificate	1 year of experience, no certificate
8	3 years of experience, MOOC certificate	3 years of experience, no certificate
9	5 years of experience, MOOC certificate	5 years of experience, no certificate
10	1 year of experience, no certificate	3 years of experience, no certificate
11	1 year of experience, no certificate	5 years of experience, no certificate
12	3 years of experience, no certificate	5 years of experience, no certificate

Note. This table presents the comparisons studied in each experiment. Additional conditions (for a total of 24 in each experiment) were created by randomly varying the order of the profile (which was listed first) and the name on the profile (Brad or Todd). MOOC = massive open online course; BA = bachelor of arts degree; AA = associate in arts degree.

To control for any potential bias caused by the order in which the profiles were presented and the name and specific wording attached to a given profile, we randomly varied which profile (Brad or Todd) received each education credential and/or experience and the order in which the profiles were presented (e.g., MOOC on the first or second profile).

MOOC Versus Traditional Credentials

In this experiment, the main variation between the two profiles (Brad and Todd) was different educational credentials in web development (Research Question 1). Comparisons 1 to 3 in Table 1 tested respondents’ preferences for a five course Coursera MOOC specialization in Web Development and Design (MOOC credential) as compared to traditional education credentials: a community college certificate in Web Development and Design from Memorial Community College (CC certificate), an associate degree in Web Development and Design from Memorial Community College (AA), and a bachelor’s degree in Web Development and Design from Hillside College (BA).³ Because education credentials are likely to be more important for early career professionals, we conduct this experiment with worker profiles having only 1 year of professional work experience.

To provide a comparison, we also measure preferences between the three traditional credentials by providing some respondents with choices between two traditional credentials (Comparisons 4–6 in Table 1). To obtain more precise estimates on the effect of MOOC credentials, we overassigned respondents to the MOOC/traditional credential comparisons (Comparisons 1–3) as opposed to the comparisons in which neither profile had a MOOC credential (Comparisons 4–6).

MOOC Versus No MOOC

We consider Research Questions 2 and 3 by testing respondents’ preferences for MOOC credential as compared to no explicitly stated credentials. We test the value of the MOOC credential at three different levels of professional experience: 1, 3, and 5 years. Comparisons 7, 8, and 9 in Table 1 presented each respondent with one profile with a Coursera 5-course certificate in Web Development and Design (MOOC credential) and one profile without such a credential.

We also included three additional comparisons (10, 11, and 12), which asked respondents to indicate a preference between profiles that had different years of experience with neither profile having a MOOC credential. These comparisons provide a measure of respondents’ preferences for experience and allow us to compare the magnitude of the preference for years of experience to the magnitude of the preference for a MOOC credential. Again, to obtain more precise estimates on the effect of MOOC credentials, we overassigned respondents to the MOOC/no MOOC comparisons.

Profiles

The profiles were modeled after a popular site for hiring freelance workers and include a short description from the applicant that includes information about his experience, education (when applicable), and MOOC credential (when applicable). Details on the design and content of the profiles and example profiles are included in online Supplemental Appendix B.

Respondents

In each of the two experiments, we recruited approximately 1,000 respondents via Mturk, a website that allows researchers to hire workers to complete discrete tasks for monetary reward.⁴ MTurk service allows location restrictions, and we restricted our sample to respondents in the United States.⁵ Respondents completed our study online. Respondents were compensated $0.25 for their participation and took an average of 3.5 minutes to complete the study. Demographic characteristics of our sample are included in Table 2 in which we see diversity across gender, race, education level, and working full-time. There are slight differences in these characteristics between respondents in our two experiments, but we control for these observable characteristics in our preferred model. In our preferred model, we drop respondents without complete control variables (4%–6% of our sample). We examine whether the listwise deletion of respondents without complete controls changes our results. Our main results table (Table 3) presents models that (1) include all respondents who selected a preferred candidate (all respondents except two in the first experiment and four in the second), (2) reduce our sample to those with complete control variables but without including those controls in the model, and (3) use the reduced sample and include the complete set of control variables in the model. Across all models, the significance and general magnitude of the coefficients do not change. These additional models are included in the online Supplemental Appendix D.

Table 2

Summary Statistics for Covariates

Variable	MOOC Versus Traditional Credentials		MOOC Versus No MOOC
Variable	M	SD	M	SD
Female	0.569	0.495	0.469	0.499
Age (years)	35.887	13.084	35.831	11.778
Hispanic	0.058	0.235	0.044	0.205
Asian	0.075	0.264	0.096	0.295
White	0.690	0.463	0.735	0.441
Black	0.102	0.303	0.065	0.247
Other race	0.032	0.177	0.016	0.126
Multiple race	0.038	0.190	0.044	0.205
College degree	0.657	0.475	0.666	0.472
Works full-time	0.581	0.494	0.569	0.496
Makes hiring decisions	0.260	0.439	0.197	0.398
Self-employed	0.230	0.421	0.190	0.392
Manages employees	0.309	0.474	0.309	0.463
Has hired work online	0.337	0.473	0.275	0.447
Has taken an online class	0.725	0.447	0.750	0.433
Has heard of Coursera	0.313	0.464	0.342	0.474
	N = 958		N = 937

Note. All variables except age are binary. Participants were dropped from the analytic sample in each experiment due to incomplete covariates (66 participants in MOOC vs. traditional credentials and 63 participants in MOOC vs. no MOOC). MOOC = massive open online course.

Table 3

Preference for MOOCs Versus Traditional Credentials

	1: Choose MOOC Versus CC Certificate			2: Choose MOOC Versus AA			3: Choose MOOC Versus BA
MOOC	−0.145* (0.066)	−0.147* (0.069)	−0.165* (0.073)	−0.323*** (0.065)	−0.354*** (0.067)	−0.353*** (0.072)	−0.273*** (0.064)	−0.272*** (0.066)	−0.273*** (0.068)
Sample
Full sample	Yes			Yes			Yes
Reduced sample		Yes	Yes		Yes	Yes		Yes	Yes
Controls
Respondent			Included			Included			Included
Control mean	0.501	0.521	0.521	0.658	0.667	0.667	0.637	0.637	0.637
N	225	211	211	214	195	195	226	212	212
R ²	0.025	0.022	0.072	0.107	0.128	0.177	0.079	0.078	0.118

Note. Heteroscedastic robust standard errors are in parentheses. We use linear probability models to predict the binary outcome. The binary outcome “Choose” was created from a Likert-type scale asking the degree of preference for Profile A versus Profile B. Respondent controls include gender, age, race/ethnicity, and binary indicators for the following: college degree, full-time work status, job making hiring decisions, self-employed, job with managerial role, whether they have hired a company or worker online, whether they have taken an online class, and whether they had heard of Coursera prior to this study. MOOC = massive open online course; BA = bachelor of arts degree; AA = associate in arts degree; CC = community college.

p < .05. ^**p < .01. ^***p < .001.

Dependent Variable

The primary outcome measure for both experiments was a question that asked, “Who would you recommend to your friend?” The response options were a 1 to 6 Likert-type scale, with 1 equal to “definitely Profile A” and 6 equal to “definitely Profile B.” An example of this scale is included in the online Supplemental Appendix E. For the purpose of our analyses, the scale was dichotomized to report whether a specific profile was chosen.

Although we employ linear probability models of this binary outcome throughout our analyses for ease of interpretation, we consider the full data of the underlying ordered responses in histograms and in alternate specifications. Results from logit models provide similar results (same sign and significance) across all main analyses. When treating our outcome variable as a continuous 6-point measure of preference using ordinary least squares, our results also provide similar results across all main analyses. We also model the outcome using an ordered logit, which yields similar results, further demonstrating robustness. We also include models that only consider the two strongest answer choices (“definitely” and “probably,” but not “slightly”) to be a choice in favor of a particular profile. Finally, we present results from an ordered logit model in which we collapse the six options into three: (1) definitely or probably prefer the MOOC profile, (2) “slightly” prefer either the MOOC or non-MOOC profile, and (3) definitely or probably prefer the non-MOOC profile. This model approximates allowing respondents to express indifference between the two profiles. The results of these supplementary analyses are included in online Supplemental Appendix F. Across all comparisons, these models yield similar results maintaining the same general pattern and significance.

Analytic Methods

Because of the randomized control trial nature of the experiment, any difference in average preferences across different educational credentials can be interpreted as the causal effect of one credential over the other credential (or no credential) on the preference for being hired. We describe the estimation methods for each experiment below.

MOOC Versus Traditional Credentials

We first compare the effect of MOOCs to the effect of traditional educational credentials on preferences for hiring. In these analyses, we estimate three effects, one for each paired comparison of MOOC and a traditional education credential. Because we randomly assigned profile order, name, and certification, we can identify the effect of MOOC credential on being the preferred candidate through the following regression model:

Y_{i} = β_{0} + β_{1} M O O C C r e d e n t i a l_{i} + β_{2} B r a d_{i} + θ_{i} δ + ϵ_{i},

(1)

where $Y_{i}$ is a dummy variable indicating that respondent i chose Profile A (defined as definitely, probably, or slightly preferred). $M O O C C r e d e n t i a l_{i}$ is a dummy variable indicating that the first profile (Profile A) has the MOOC certificate. Because the MOOC credential was randomly assigned between profiles, $β_{1}$ indicates the effect of having a MOOC credential relative to the traditional credential on respondents’ stated preferences for hiring and is our coefficient of interest. $B r a d_{i}$ is a dummy variable indicating that Profile A was the Brad profile, and $θ_{i}$ is a vector of individual level controls gathered from the demographic survey of respondents including gender, age, race/ethnicity, and binary indicators for the following: college degree, full-time work status, job making hiring decisions, self-employed, job with managerial role, whether they have hired a company or worker using an online service, whether they have taken an online class, and whether they had heard of Coursera prior to this study. We run this model separately for Comparisons 1, 2, and 3 for the MOOC credential versus traditional credentials experiment in Table 1. To estimate the effect of different traditional credentials against each other (Comparisons 4, 5, and 6 in Table 1), we replace $M O O C C r e d e n t i a l_{i}$ in Equation 1 with a dummy variable defined as the lower of the two credentials in each comparison.

MOOC Versus No MOOC

In the second set of analyses, we estimate the effect of the presence of a MOOC credential relative to not having a MOOC credential on the preference for being hired (Research Question 2). For this analysis, we use data from the respondents who saw two profiles that each had the same number of years of experience (1, 3, or 5) and one of which had a MOOC credential (Comparisons 7, 8, and 9 in Table 1). We estimate effects using linear probability model regressions of the following form:

\begin{array}{l} Y_{i} = β_{0} + β_{1} M O O C_{i} + β_{2} 3 Y r E X P_{i} + β_{3} 5 Y r E X P_{i} \\ + β_{4} B r a d_{i} + θ_{i} δ + ϵ_{i}, \end{array}

(2)

Where $Y_{i}$ is a binary variable equal to 1 if respondent i chose Profile A, $M O O C_{i}$ is a dummy variable indicating that Profile A had a MOOC credential listed on his profile, $3 Y r E X P_{i}$ and $5 Y r E X P_{i}$ are dummy variables for the profiles having 3 or 5 years of experience with 1 year of experience serving as the omitted category, and the other variables are defined as above. We pool the (randomly assigned) respondents from Comparisons 7, 8, and 9 in Table 1 for this analysis. Our coefficient of interest is $β_{1}$ as it indicates the difference in probability of a respondent preferring Profile A if Profile A has a MOOC credential.

We also investigate whether the value of the MOOC credential varies across levels of experience. We accomplish this by adding interaction terms between MOOC and 3 and 5 years of experience to Equation 2. The coefficient on the interaction terms, which indicate the difference in the effect of the MOOC credential for profiles with 3 or 5 years of experience, as compared to profiles with 1 year of experience, are the coefficients of interest in this analysis. For these analyses, we include Comparisons 7, 8, and 9 in Table 1.

Finally, in an effort to compare the magnitude of the MOOC effect against the magnitude of effects of years of professional experience (Research Question 3), we examined the effects of different years of experience on the likelihood that a respondent preferred Profile A. For this analysis, we rely on respondents who were randomly assigned Comparisons 10, 11, and 12 in Table 2, and estimate a separate effect for each comparison. For each of these comparisons the respondent saw two profiles with different levels of experience (i.e., profiles with 1 and 3 years of experience, 1 and 5 years of experience, or 3 and 5 years of experience), neither of which had a MOOC credential. We again use a binary measure of preferring the first profile as the outcome and estimate Equation 3 using a linear probability model:

Y_{i} = β_{0} + β_{1} M o r e E x p_{i} + β_{2} B r a d_{i} + θ_{i} δ + ϵ_{i} .

(3)

In these regressions, $M o r e E x p_{i}$ is an indicator for the first profile having the greater level of experience and captures the difference in preference between the higher and lower levels of experience among profiles that did not have a MOOC credential. The results of these three regressions allow us to examine, respectively, the effect of having 3 years of experience as compared to 1 year of experience, the effect of 5 years of experience as compared to 1 year of experience, and the effect of 5 years of experience as compared to 3 years of experience.

Results

MOOC Versus Traditional Credentials

Balance Check

Before discussing the results of our experiment, we address balance in our sample. Because we have 24 unique treatment arms (four conditions resulting from the randomization of order and profile of each of the first six comparisons in Table 2), we assess balance in two ways. First, we collectively test the balance of our respondent covariates across all 24 treatment arms using a multinomial logit model to assess if the covariates are distributed equally across conditions. This model considers the treatment arm as an unordered outcome taking 24 possible values. We regress this outcome on the full set of respondent covariates and use an F test to assess the equality of relative risk ratios of all variables across the treatment arms. Our multinomial logit model produces a p value of the F test of 0.99.

As our main interest is in estimating the effect of MOOC credential on respondents’ ratings, we are especially interested in determining if different groups of respondents were equally likely to see the profile with a MOOC credential first. To examine this, we also conduct a simpler balance check in which we predict each of the 16 respondent covariates using one predictor: if the respondent saw the profile with a MOOC first (“treatment”). The results of 16 different regressions regressing a binary treatment indicator on each covariate produces 16 coefficients, none of which are statistically significant at the 5% level. The results of this balance test are presented in online Supplemental Appendix G. These two methods for assessing balance in our sample together suggest that we achieved proper random assignment and any differences in outcomes across the groups can be attributed to the experimental conditions.

Preferences for MOOCs Versus Traditional Credentials

We first provide unconditional, descriptive histograms of the preferences for MOOCs versus other educational credentials. Figure 1 indicates clear preferences for BA and AA degrees over MOOCs, but the preference for community college certificates over MOOCs is less clear.

Figure 1.

Respondents’ unconditional preferences for MOOCs versus traditional education credentials.

We formalize the descriptive findings in regressions to control for respondent characteristics and for any potential effects of profile order or names due to unlucky randomization. We provide results of estimating Equation 1 in Table 3 using dichotomized preferences for hiring. When a MOOC is compared to a CC certificate in Web Development and Design, respondents preferred the CC certificate by 16.5 percentage points (β = −0.165, p < .05; shown in Table 3, Model 3). In every case, traditional educational credentials are preferred over the MOOC credential. The AA degree is preferred to a MOOC by 35.3 percentage points, and the BA is preferred by 27.3 percentage points. The magnitude of these preferences does not indicate a clear pattern; there is a stronger preference for an associate degree over a MOOC credential than there is for a bachelor’s degree over a MOOC credential, although this difference is not statistically significant. It is possible this preference arises out of the unique context of web development credentials.

Comparing Different Educational Credentials

In order to provide a comparison for our MOOC findings, we also compared traditional credentials with each other. Figure 2 shows descriptive histograms for our comparisons between the three traditional educational credentials. The histograms suggest expected preferences for BA degrees over AA and CC certificates. The histogram showing preference for AA degrees over community college certificates is less clear. Formal analyses of these results, which include our controls, are shown in Table 4. The preference for BA degrees over community college certificates and AA degrees is about the same at 34 and 35 percentage points, respectively. Both of these are significant at the p < .001 level. The magnitude of this effect is roughly equivalent to the largest preference we observed when comparing a traditional credential to a MOOC in Table 3 (the preference for an AA over a MOOC). The comparison between community college certificates and AA degrees is not statistically significant, so we conclude the value of any traditional credential over a MOOC is larger than the labor market value of the difference between an AA and CC certificate for web development at 1 year of professional experience.

Figure 2.

Respondents’ unconditional preferences between different traditional education credentials.

Table 4

Preference Between Traditional Educational Credentials

Credential	1: Choose CC Certificate Versus AA	2: Choose AA Versus. BA	3: Choose CC Certificate Versus BA
CC certificate	−0.109 (0.101)		−0.337*** (0.089)
AA degree		−0.351*** (0.100)
Controls
Individual	Included	Included	Included
Control M	0.535	0.711	0.678
N	114	111	115
R ²	0.077	0.283	0.293

Note. Heteroscedastic robust standard errors are in parentheses. We use linear probability models to predict the binary outcome. The binary outcome “Choose” was created from a Likert-type scale asking the degree of preference for Profile A versus Profile B. Respondent controls include gender, age, race/ethnicity, and binary indicators for the following: college degree, full-time work status, job making hiring decisions, self-employed, job with managerial role, whether they have hired a company or worker online, whether they have taken an online class, and whether they had heard of Coursera prior to this study. CC = community college; AA = associate of arts degree; BA = bachelor of arts degree.

p < .05. ^**p < .01. ^***p < .001.

MOOC Versus No MOOC

We next focus on our final two research questions: Research Question 2: Does the presence of a MOOC credential on a freelance worker’s profile, as compared to no stated postsecondary credentials, influence potential clients’ stated preferences for hiring him? Does this effect vary by professional experience? Research Question 3: How does the effect of a MOOC credential compare to the effect of professional experience on preferences for hiring in a freelance setting?

Balance Checks

Before discussing the results of our experiment, we again address balance in our sample. To address balance in our sample for the MOOC versus no MOOC experiment, we follow a similar method to the MOOC versus traditional credentials experiment discussed above. First, we collectively test the balance of our respondent covariates across all 24 treatment arms using a multinomial logit model to assess if the covariates are distributed equally across conditions. The p value of this F test is 0.9, suggesting balance across treatment arms.

We also conduct the simpler balance check, in which we combine all treatment arms and consider the difference in individual covariates in being assigned to seeing the MOOC credential on the first profile. The results of 16 different regressions regressing a binary treatment indicator on each covariate produces 16 coefficients, none of which are statistically significant at the 5% level. The results of this balance test are presented in online Supplemental Appendix G. The conclusions from both tests is that respondents were successfully randomly balanced across treatment arms.

Effect of MOOCs on Ratings

We first present unconditional, descriptive histograms in Figure 3, depicting respondents’ preference ratings for MOOC credentials for candidates with 1, 3, and 5 years of experience. In each of these histograms, definitely choosing the profile with the MOOC credential is on the right and definitely choosing the profile without it is on the left. From each of these figures, it is clear that respondents had a strong preference for MOOC credentials. Across all three experience levels, candidates with MOOC credentials were much more likely to be preferred to candidates with no MOOC credentials.

Figure 3.

Respondents’ unconditioned preferences for MOOC credentials versus no MOOC credential at each level of experience.

We formalize these descriptive findings in Table 5 Model 1, which presents the results of estimating Equation 2. The coefficient on the MOOC term is the magnitude of the preference for the first profile when the first profile is randomly assigned the MOOC credential. It is interpreted as the difference in probability of selecting the MOOC credential profile over the no MOOC credential profile. The descriptive results shown in Figure 3 are apparent in these results. Respondents strongly, and significantly, preferred profiles with MOOC credentials. Profiles with a MOOC credential were 61 percentage points more likely to be preferred than those without. We should be careful about attributing this total effect to a preference for MOOC credentials themselves as the MOOC profile also simply had an additional component of information. It is possible that a subset of respondents chose the profile with more information (i.e., the MOOC credential) as opposed to selecting it because they value the MOOC credential.

Table 5

Preference for MOOC Versus No MOOC

MOOC	0.611*** (0.032)	0.613*** (0.055)
3 years’ experience	0.002 (0.038)	−0.012 (0.052)
5 years’ experience	−0.004 (0.039)	0.012 (0.053)
MOOC * years’ experience		0.028 (0.077)
MOOC * years’ experience		−0.030 (0.080)
Controls
Respondent	Included	Included
Control M	0.196	0.196
N	626	626
R ²	0.400	0.400

p < .05. ^**p < .01. ^***p < .001.

Differential Effect of MOOCs on Ratings by Years of Experience

We next examine if the effect of having a MOOC credential varies by experience level. If we compare the histograms in Figure 3, it appears that across experience levels, the preference for MOOC credentials appears consistent. Model 2 in Table 5 presents a formal test of these comparisons. The coefficients on the interaction terms indicate if the effect of MOOCs credential on preference for hiring differs for candidates with different levels of experience. We do not see any evidence that the effect differs as none of the coefficients on the interaction terms are statistically significant.

Effect of Experience on Ratings

To provide comparisons, we also tested respondents’ preferences for years of professional experience in the absence of any MOOC certification. Table 6 presents the results of Equation 3. Respondents strongly, and significantly, preferred profiles with more experience. Profiles with 3 (5) years of experience, instead of 1 year, were 63 percentage points (78 percentage points) more likely to be chosen. In line with diminishing marginal returns, we see suggestive evidence that the marginal effect of two additional years of experience decreases as candidates get more experience; the difference in stated preference between 3 and 5 years of experience is less than the difference between 1 and 3 years of experience.

Table 6

Preference Between Years of Experience

Experience	1: 1 Versus 3 years	2: 1 Versus 5 years	3: 3 Versus 5 years
3 years’ experience	0.630*** (0.079)
5 years’ experience		0.779*** (0.074)	0.546*** (0.092)
Controls
Individual	Included	Included	Included
Control M	0.173	0.126	0.232
N	105	103	103
R ²	0.532	0.590	0.388

Note. Heteroscedastic robust standard errors are in parentheses. We use linear probability models to predict the binary outcome. The binary outcome “Choose” was created from a Likert-type scale asking the degree of preference for Profile A versus Profile B. Respondent controls include gender, age, race/ethnicity, and binary indicators for the following: college degree, full-time work status, job making hiring decisions, self-employed, job with managerial role, whether they have hired a company or worker online, whether they have taken an online class, and whether they had heard of Coursera prior to this study.

p < .05. ^**p < .01. ^***p < .001.

We can use these preferences for experience as comparisons for our effects of having a MOOC (vs. no MOOC). The effect of having a MOOC is similar in magnitude to the effect of having 3 years, rather than 1 year, of experience (shown in Table 6) and larger than our estimated effect of having 5, rather than 3 years of experience. It appears that, all else being equal, the effect of having a MOOC credential on being the preferred job candidate is roughly equivalent to having two additional years of professional experience for candidates that have relatively little labor market experience.

Discussion

MOOCs have developed a reputation as an affordable and widely accessible way of providing education and career retooling in high-demand, high-growth fields. However, for these courses to be an effective means of providing career-advancing training, they must be recognized as acceptable signals of knowledge, skills, or desirable personal traits in the labor markets.

Our article assessed whether MOOCs are viewed favorably by people making hiring decisions, both in relation to other traditional credentials and in relation to having no stated credential. Past work has shown that MOOC participants are generally quite positive about the ability of MOOCs to help learners find new jobs or advance in their careers (Dillahunt et al., 2016; Zhenghao et al, 2015), and hiring managers generally feel that MOOCs are a positive signal of an applicant’s motivation and persistence. However, past work has also found that such credentials are generally considered by people making hiring decisions to be inferior to credentials earned from traditional colleges (Radford et al., 2014; Rosendale, 2016).

Our results corroborate and quantify prior survey findings that hiring managers prefer traditional credentials to MOOCs in an important and novel context. We hypothesized that both bachelor’s and associate degrees would be preferred to a MOOC, but we did not have a prediction about MOOCs compared to community college certificates. Empirically, we find that when compared to three different traditional educational credentials in the same field, MOOCs are never favored. This finding corroborates past work (Radford et al., 2014; Rosendale, 2016).

We also examine preferences across profiles that listed no traditional education, but in which one had a MOOC credential. We hypothesized that when compared to a similar profile with no MOOC credential, the profile with a MOOC credential would be preferred, and this hypothesis was supported. We find that respondents who were asked to state a preference between two fictional job seekers looking for freelance web development work preferred profiles with MOOC credentials to those without MOOC credentials. We conclude that when two candidates have the same (implied) level of traditional education, MOOCs are perceived as a positive signal.

We also hypothesized that the preference for MOOCs would differ depending on the experience level of the profiles (e.g., with more experience, the preference would diminish), but this hypothesis was not supported. Preferences for candidates with a MOOC credential versus no MOOC credential appeared to be consistent across experience levels, although we only tested 5 years of experience or less. We estimate that the size of the preference for MOOCs is similar to the effect of two additional years of professional experience.

Our study cannot directly identify whether the value of MOOCs is a signal to potential employers of an applicant’s inherent skills or MOOCs are seen as building human capital by teaching marketable skills. It is possible respondents see MOOCs as positive signals about an applicant’s skills, knowledge, or personal traits (such as a desire to stay current with knowledge in the field, curiosity, or persistence), but the signal is not as strong as the signal from traditional education credentials. It is equally plausible that respondents view MOOCs as teaching them useful skills but not as many as those in traditional degree programs. We can definitively say that respondents do not view MOOCs as an acceptable substitute for traditional education, but our MOOC versus no MOOC experiment demonstrates that MOOCs might be valued as a complement to traditional education.

Future studies could build on our findings by examining the perceived value of a MOOC credential when it is listed in addition to various traditional credentials. Additionally, our findings raise an interesting question regarding why all traditional credentials, including certificates from community colleges, are substantially preferred to a MOOC credential. As MOOCs often carry the imprimatur of an elite college, and certificates from a community college do not, this could indicate that respondents are distrustful of online delivery. Future work could explore how MOOCs are compared to other types of credentials, such as certificates or 2- or 4-year degrees, that are earned exclusively online from nonprofit or for-profit providers. Finally, tracking people’s impressions of MOOCs over time could be interesting, as MOOCs are becoming increasingly common and well-known.

There are some notable limitations to our study. First, we surveyed respondents in a hypothetical hiring environment. While this scenario improves on existing surveys that ask about general attitudes without context, respondents’ answers were not binding and carried no consequences. While we are confident of the internal validity of our experimental methodology, we cannot be sure that our results reflect how people respond to freelance hiring profiles in actual hiring situations, nor how hiring managers would respond in more traditional hiring environments. To partially examine if our results might vary for people who actually hire online, we include a series of subgroup analyses in online Supplemental Appendix H. In these analyses, we examine if our estimate effects are different for respondents who have experience hiring or for respondents who are familiar with online courses. While we think these exploratory analyses are important for the reasons we noted above, we do note that the context of our study in which people hire workers online in informal labor markets is increasingly common.

Similarly, our examination focuses only on one aspect of a worker’s profile, educational training. Apart from a comparison with the effect of years of experience, we do not compare the magnitude of this effect to other criteria that hirers would use to make a decision, such as samples of work, references from trusted sources, or hourly wage. The effect we find might be dominated by these other considerations in practice.

Our study also does not test differences in human capital development between the various credentials we include in our experiment. We provide no information about how effective these different forms of education are in increasing actual labor market productivity. Future studies that examine job performance and retention could shed light on the ability of these programs to increase students’ skills and knowledge.

Finally, we test the effects of MOOCs for (fictional) freelance workers with names intended to connote a White male identity. Given evidence of discrimination, by race/ethnicity or immigrant status, in the labor market (e.g., Bertrand & Mullainathan, 2004; Oreopoulos, 2011), it is not clear that these results would generalize to all job seekers.

Our results have several clear implications for students and institutions. Although valuable relative to no MOOC credential, the value of MOOC credentials for students appears to be less than credentials from traditional sources. We might not expect a Coursera specialization to be equivalent to the value of a 2- or 4-year degree, but survey respondents also found it less valuable than a community college certificate. Although MOOCs provide a clear advantage in flexibility of location, pace, and time, as well as a very low cost, students must assess the tradeoffs between these advantages and the costs and benefits of commuting to a community college to complete a similar certificate or degree.

From the perspective of institutions of higher education, MOOCs provide a form of competition for certain sectors of traditional higher education. Although MOOC students cannot earn a liberal arts degree, they can gain specific skills and credentials, often in technical fields. As we have seen in numerous other industries, disruptive innovation from a competitor that offers a lower cost product can reshape an industry, and traditional institutions of higher education should consider how these MOOC offerings will force changes in recruitment, curriculum, and pedagogies.

Our results also hold implications for MOOC providers such as Coursera, Udacity, and EdX. Although their websites are filled with confident promises and success stories of graduates whose MOOC courses and credentials ostensibly improved their labor market outcomes, the findings from this study suggest that these credentials are not perceived to be equivalent to traditional higher education credentials. However, our results suggest that perhaps MOOC credentials could be valuable in serving as a signal of continued curiosity and dedication. In addition, MOOC providers may need to work with employers to convince them that the skills built in MOOCs and the signal these courses and credentials provide are related to actual productivity.

Supplemental Material

sj-docx-1-ero-10.1177_2332858420973577 – Supplemental material for Do MOOCs Make You More Marketable? An Experimental Analysis of the Value of MOOCs Relative to Traditional Credentials and Experience

Supplemental material, sj-docx-1-ero-10.1177_2332858420973577 for Do MOOCs Make You More Marketable? An Experimental Analysis of the Value of MOOCs Relative to Traditional Credentials and Experience by Mariela J. Rivas, Rachel B. Baker and Brent J. Evans in AERA Open

Footnotes

1.

For literature on digital badges, see Gibson et al. (2013) and .

2.

Search on upwork.com conducted on October 25, 2018.

3.

The schools were fictional. We chose to use fictitious schools in order to avoid overly positive or overly negative feelings of either school that could influence respondents’ views. In a pilot study, we tested whether respondents were familiar with the schools and what they thought about the quality of the schools. Results show that the majority (about 60%) of respondents had never heard of either school, and 60% rated the quality of each of the schools as neutral (neither good nor bad), with comparable mean quality between the two fictitious schools.

4.

Research into the MTurk platform has compared samples and data outcomes with traditional methods (other kinds of internet samples and American college student samples) and demonstrated that MTurk samples are more demographically diverse (older population, respondents from all 50 states, and more racially diverse) and equally psychometrically reliable (test–retest reliability and Cronbach’s alphas were compared; Buhrmester et al., 2011; Follmer et al., 2017). Additionally, recent evidence suggests that the treatment effects obtained from MTurk and nationally representative population-based samples are comparable, though the authors note that while convenience samples such as MTurk can play an important role in the generation and testing of new hypotheses, replication is, as always, necessary (Mullinix et al., 2015).

5.

Recently, researchers have raised concerns over respondent samples obtained on the MTurk platform, with some evidence suggesting that respondents could be using multiple accounts to take surveys more than once, and/or using software to mask their location (so that they appear to be in the United States when they are actually elsewhere; Bai, 2018; Dennis et al., 2018; TurkPrime, 2018). In order to address these concerns in our current sample, present results from models excluding respondents with repeat IP addresses, repeat geolocations, and/or geolocations outside the United States. Our main results hold for all of these sample restrictions, which indicates that any potential issues with repeat takers or non–U.S. respondents are minimal and do not invalidate our results and main conclusions.

Authors

MARIELA J. RIVAS is a doctoral candidate at the University of California, Irvine’s School of Education. She studies teaching and learning in higher education.

RACHEL B. BAKER is an assistant professor at the University of California, Irvine’s School of Education. She studies institutional and state policies that can improve access to and success in higher education for traditionally underrepresented groups.

BRENT J. EVANS is an associate professor of public policy and higher education at Peabody College, Vanderbilt University. He employs quantitative methods to conduct research on student success in higher education.

References

Agrawal

Horton

Lacetera

Lyons

(2015). Digitization and the contract labor market: A research agenda. In Goldfarb

Greenstein

Tucker

C. E.

(Eds.), Economic analysis of the digital economy (pp. 219–250). University of Chicago Press. https://doi.org/10.7208/chicago/9780226206981.003.0008

Bai

(2018, August 8). Evidence that a large amount of low quality responses on MTurk can be detected with repeated GPS coordinates. https://www.maxhuibai.com/blog/evidence-that-responses-from-repeating-gps-are-random

Becker

G. S.

(1962). Investment in human capital: A theoretical analysis. Journal of Political Economy, 70(5, Part 2), 9–49. https://doi.org/10.1086/258724

Bertrand

Mullainathan

(2004). Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. American Economic Review, 94(4), 991–1013. https://doi.org/10.1257/0002828042002561

Buhrmester

Kwang

Gosling

S. D.

(2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5. https://doi.org/10.1177/1745691610393980

Columbaro

N. L.

Monaghan

C. H.

(2009). Employer perceptions of online degrees: A literature review. Online Journal of Distance Learning Administration, 12(1).

Coursera. (2014). Coursera specializations: Focused programs in popular fields. http://coursera.tumblr.com/post/73994272513/coursera-specializations-focused-programs-in

Darolia

Koedel

Martorell

Wilson

Perez-Arce

(2015). Do employers prefer workers who attend for-profit colleges? Evidence from a field experiment. Journal of Policy Analysis and Management, 34(4), 881–903. https://doi.org/10.1002/pam.21863

Deming

D. J.

Yuchtman

Abulafi

Goldin

Katz

L. F.

(2016). The value of postsecondary credentials in the labor market: An experimental study. American Economic Review, 106(3), 778–806. https://doi.org/10.1257/aer.20141757

10.

Dennis

S. A.

Goodson

B. M.

Pearson

(2018). MTurk workers’ use of low-cost ‘virtual private servers’ to circumvent screening methods: A research note. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3233954

11.

Dillahunt

T. R.

Fiesta

Wang

(2016, February). Do massive open online course platforms support employability? In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing (pp. 233–244). ACM. https://doi.org/10.1145/2818048.2819924

12.

Follmer

D. J.

Sperling

R. A.

Suen

H. K.

(2017). The role of MTurk in education research: Advantages, issues, and future directions. Educational Researcher, 46(6), 329–334. https://doi.org/10.3102/0013189X17725519

13.

Fugate

Kinicki

A. J.

Ashforth

B. E.

(2004). Employability: A psycho-social construct, its dimensions, and applications. Journal of Vocational Behavior, 65(1), 14–38. https://doi.org/10.1016/j.jvb.2003.10.005

14.

Gibson

Ostashewski

Flintoff

Grant

Knight

(2013). Digital badges in education. Education and Information Technologies, 20(2), 403–410. https://doi.org/10.1007/s10639-013-9291-7

15.

Hess

(2020, July 13). Google announces 100,000 scholarships for online certificates in data analytics, project management and UX. https://www.cnbc.com/2020/07/13/google-announces-certificates-in-data-project-management-and-ux.html

16.

A. D.

Chuang

Reich

Coleman

Whitehill

Northcutt

Williams

J. J.

Hansen

Lopez

Petersen

(2015). HarvardX and MITx: Two years of open online courses (HarvardX Working Paper No. 10). https://doi.org/10.2139/ssrn.2586847

17.

Kässi

Lehdonvirta

(2018). Online labour index: Measuring the online gig economy for policy and research. Technological Forecasting and Social Change, 137, 241-248. https://doi.org/10.1016/j.techfore.2018.07.056

18.

Kizilcec

Davis

Wang

(2019). Online degree stigma and stereotypes: A new instrument and implications for diversity in higher education. https://doi.org/10.2139/ssrn.3339768

19.

Kizilcec

R. F.

Schneider

(2015). Motivation as a lens to understand online learners: Toward data-driven design with the OLEI scale. ACM Transactions on Computer-Human Interaction (TOCHI), 22(2), 6. https://doi.org/10.1145/2699735

20.

Kokkodis

Papadimitriou

Ipeirotis

P. G.

(2015, February). Hiring behavior models for online labor markets. In Proceedings of the Eighth ACM International Conference on web search and data mining (pp. 223–232). ACM. https://doi.org/10.1145/2684822.2685299

21.

Kuek

S. C.

Paradi-Guilford

Fayomi

Imaizumi

Ipeirotis

Pina

Singh

(2015). The global opportunity in online outsourcing. World Bank Group.

22.

Mullinix

K. J.

Leeper

T. J.

Druckman

J. N.

Freese

(2015). The generalizability of survey experiments. Journal of Experimental Political Science, 2(2), 109–138. https://doi.org/10.1017/XPS.2015.19

23.

Oreopoulos

(2011). Why do skilled immigrants struggle in the labor market? A field experiment with thirteen thousand resumes. American Economic Journal: Economic Policy, 3(4), 148–171. https://doi.org/10.1257/pol.3.4.148

24.

Perera

(2017). How to learn the Top 5 freelance skills with online courses. https://freelancinghacks.com/learn-freelance-skills-with-online-courses/

25.

Pickard

(2018). Analysis of 450 MOOC-based microcredentials reveals many options but little consistency. Class Central. https://www.class-central.com/report/moocs-microcredentials-analysis-2018/

26.

Radford

A. W.

Robles

Cataylo

Horn

Thornton

Whitfield

(2014). The employer potential of MOOCs: A mixed-methods study of human resource professionals’ thinking on MOOCs. International Review of Research in Open and Distance Learning, 15(5), 1–25. https://doi.org/10.19173/irrodl.v15i5.1842

27.

Raish

Rimland

(2016). Employer perceptions of critical information literacy skills and digital badges. College & Research Libraries, 77(1), 87–113. https://doi.org/10.5860/crl.77.1.87

28.

Rosendale

J. A.

(2016). Valuing non-degree, online training: An examination of hiring managers’ perceptions of MOOCs [Unpublished doctoral dissertation]. Indiana University of Pennsylvania.

29.

Shah

(2018). By the numbers: MOOCs in 2018. Class Central. https://www.classcentral.com/report/mooc-stats-2018/

30.

Taylor

(2017, October 24). Training for freelancers: Free and cheap options. https://www.homeworkingclub.com/training-for-freelancers/

31.

TurkPrime. (2018, September 18). After the bot scare: Understanding what’s been happening with data collection on MTurk and how to stop it. https://blog.turkprime.com/after-the-bot-scare-understanding-whats-been-happening-with-data-collection-on-mturk-and-how-to-stop-it

32.

Udacity. (2018). Get job read. https://www.udacity.com/nanodegree

33.

Watson

(2018, January 19). Find your passion and build your dream career. Udacity. https://blog.udacity.com/2018/01/find-passion-build-dream-career.html

34.

Zhenghao

Alcorn

Christensen

Eriksson

Koller

Emanuel

E. J.

(2015, September 22). Who’s benefiting from MOOCs, and why. Harvard Business Review. https://hbr.org/2015/09/whos-benefiting-from-moocs-and-why

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.31 MB