Sage Journals: Discover world-class research

Abstract

This design-based research (DBR) study had both local and general goals. Its local goal was to increase active learning in the online courses offered at a large research university in the midwestern United States. Its larger goal was to define active learning design principles for online courses in general, so that they might be used to improve the learning experience for wider audiences. While the principles of active learning can be applied to courses in any mode of delivery: flipped classroom, blended, or fully online, the importance of active learning in online courses is highlighted because active learning course design requires numerous upfront considerations. Moreover, because the pedagogical model is structured throughout the online learning environment and thus is made visible, online courses present a unique opportunity to review what is core to the principles of active learning. The design intervention (an innovative course review method) incorporated the principles of authentic e-learning. The study addressed two major research questions: (1) “To what extent does the intervention—a new course review method—indicate the extent to which active learning is present in the design of an online course?” and (2) “How do the principles of authentic e-learning incorporated in the new course review method need to be refined?” To address the first question, the online course review intervention was used to evaluate the learning activities and assessments of 75 undergraduate online courses against these authentic e-learning principles, resulting in an active learning (AL) score for each course. To address the second question, we surveyed the learners in these 75 courses about what made learning meaningful and coded the learners’ feedback in reference to the active learning design principles. The practical outcome of this DBR study is a pedagogical course review that quantifies active learning in online courses. The theoretical outcomes of this DBR study are refined active learning design principles that can serve instructors, designers, teacher educators, and administrators in enhancing the design of online courses. The findings of this study affirm that the authentic task principles as well as newly identified learner-centered design principles together can serve as evidence-based principles to define and refine active learning in online courses.

Keywords

active learning design-based research design principles learning design online course review

Introduction to the problem and overview of related literature

Active learning is associated with the broad family of social-constructivist approaches that challenge educators to move from a direct instruction model toward more inquiry-based approaches, such as problem-based, project-based, and authentic learning opportunities (Chi, 2009). Some scholars criticize active learning for lacking strong evidence of effectiveness in comparison to traditional instructional methods whereas others have reported research support for active learning (Michael, 2006; Prince, 2004). Unfortunately, active learning as a quality standard of online course design (Lowenthal and Hodges, 2015) remains largely undefined. With online courses having become a much more widely accepted mode of instruction, especially during the pandemic that began in 2020, it is imperative to better define what active learning in online courses is and to provide evidence-based design principles for designers and teachers. The research reported in this paper was conducted to do exactly that.

Too few online courses incorporate active learning pedagogies in the design of learning tasks, and too many online courses still place the learner in the role of passive consumer of disciplinary content rather than in the central role of tackling authentic tasks that foster active knowledge construction (Herrington et al., 2010). Recorded lectures, preselected chapter readings, artificial discussions, and automated assessments too often make up the primary discourse of online courses instead of taking advantage of the pedagogical innovations made possible through active learning tasks mediated by technology (Falkner and Sheard, 2019).

Active teaching practices in face-to-face delivery do not seamlessly become active design practices for online delivery. There is a need to define clear design principles when developing the activities and assessments that will eventually guide the learners’ active participation online. The capacity of online courses to instantiate a particular pedagogical approach within the learning environment and automate this approach for ongoing use and reuse highlights the importance of upfront design considerations for active learning in online courses. We must reduce the persistent gap between what is known about the science of learning and what is practiced in online courses (Baum and McPherson, 2019; Berliner, 2008). Defining active learning in the design of online courses thus becomes a matter of quality for online delivery because without clear principles of design, vague definitions regarding active learning in online courses will continue to be the only available indicators of quality.

While multiple quality assurance frameworks for online learning in K-12 [INACOL] (2011), Bakken and Bridges (2011), [VLLA] (2019c) and higher education [Quality Matters] (2015), [NSQ] (2019), [OSCQR] (2020) exist, hardly any frameworks specifically mention active learning as a standard. Even fewer attempt to define active learning, and none make active learning measurable. In short, what exactly is meant by active learning in online courses remains a vague concept, and frequently a misunderstood goal. For example, the Quality Matters rubric (Shattuck, 2010), which has become a widely adopted set of peer-review standards in the United States and beyond (Lee et al., 2020), recommends in Standard 5.2 that learning activities provide opportunities for interaction that support active learning, but the standard does not go beyond the statement to articulate what precisely is meant by active learning.

Clearly, it is time to define active learning in online courses through more precise design principles (Carr et al., 2015). This 3-year design-based research (DBR) study documented one design team’s intervention to address this research-practice gap.

The first guiding principle of design-based research (DBR) is its “focus on a persistent problem of practice from multiple stakeholders’ perspectives” (Penuel et al., 2011: 332). The problem that this DBR addressed is the paucity of active learning pedagogy evident in online courses despite extensive research in the learning sciences providing empirical evidence that a learner-centered design using active learning methods fosters engagement and deepens learning (Freeman et al., 2014; Ghilay and Ghilay, 2015; Pallas et al., 2017; Prince, 2004). Specifically, this study addressed the lack of a definition of QM Standard 5.2 as its primary problem and conducted a systematic inquiry into what active learning in online courses looks like and how active learning indicators can quantify the extent to which active learning is present in the design of an online course.

What defines active approaches to learning is that they require the learner to exhibit a degree of agency and self-regulation that is not usually fostered when learners remain passive recipients of content knowledge transmitted to them by lecturers, readings, etc. It is when learners engage in an authentic activity, reflect on the new knowledge inherent in completing an authentic task, and share their findings in a polished product or performance, that knowledge is actively constructed (Herrington et al., 2010). Active learning course design begins with an understanding that “learning is an activity carried out by the learner” (Schneider and Stern, 2010: 82) and that knowledge is built based on previous learning (National Research Council, 1999). As such, traditional knowledge such as the laws of classical mechanics, the Cartesian coordinate system, or the mechanisms of photosynthesis remain inert until the learner employs this knowledge in order to solve a real-world task and create meaning. “The more connections a learner sees between the educational world of learning environments and the outside world, the easier the knowledge transfer will be” (Schneider and Stern, 2010: 83).

An active learning course design provides thoughtfully scaffolded and designed learning experiences that carefully guide the learners’ knowledge construction through action, reflection, abstraction, and application (Swan, 2004). Active learning requires connecting academic knowledge beyond the disciplinary focus to consider how this knowledge will impact real-world contexts in which social systems, culture, and complexity apply. The constructive, situated, self-regulated, and collaborative nature of learning is well represented by the principles for authentic e-learning (Herrington et al., 2010) (see Table 1). The design principles in Table 1 were used as the foundation for a new course review method described in the next section of this paper.

Table 1.

The principles of authentic eLearning (Herrington et al., 2010).

Principle	Definition	Applied in learning design
Real-world Relevance	Learning rises to the level of authenticity when it asks students to work actively with abstract concepts, facts, and formulas inside a realistic and highly social context mirroring the practices of the disciplinary culture.	In a Construction Management course, learners are asked to visit a job site and identify the concept of lean construction waste and make recommendations for improvements that would reduce this type of construction waste.
Ill-defined Problems	Challenges cannot be solved easily by the application of an existing algorithm; instead, authentic activities are relatively undefined and open to multiple interpretations, requiring students to identify for themselves the tasks and subtasks needed to complete the major task.	In a Business Communications course, learners research and analyze a real-world problem in an organization of their choosing and produce an analytical report that communicates the problem, the analysis of findings, as well as recommendations for solving the client’s problem.
Sustained Investigations	Problems cannot be solved in a matter of minutes or even hours. Instead, authentic activities comprise complex tasks to be investigated by students over a sustained period of time, requiring significant investment of time and intellectual resources.	In a Sociology course, learners apply sociology methods by conducting a semester-long ethnographic study of their own neighborhood. Learners identify a sociological theme to investigate (e.g. public health), identify data sources, conduct interviews, write field notes, use statistical software, and finally produce a digital story about their neighborhood supported by data.
Interdisciplinary Perspectives	Relevance is not confined to a single domain or subject matter specialization. Instead, authentic activities have consequences that extend beyond a particular discipline, encouraging students to think in interdisciplinary terms.	In a Sociology course, learners explore the strengths and weaknesses of particular methodological approaches (interview, survey, ethnography) to explore complex societal problems and debate the promises and perils of engaging in Public Sociology.
Collaboration	Success is not achievable by an individual learner working alone. Authentic activities make collaboration integral to the task, both within the course and in the real world.	In a Mathematics for Design course, groups explore weekly applications of mathematics to design problems that require learners to discuss the assigned problem, collaboratively come to a consensus of how to approach and solve the problem mathematically, and present the solution.
Reflection	Authentic activities enable learners to make choices and reflect on their learning, both individually and as a team or community.	In a Management course, learners are asked to reflect on the learned materials from the perspective of a manager or organization. An activity encourages learners to reflect creatively through formats such as blog, digital story, e-portfolio, travel log, poetry, film script, musical lyrics, presentation, YouTube Live event, etc. Reflection makes a personal and meaningful connection to the course concepts.
Articulation	Learning activities enable presentation and defense of arguments.	In a Physics course, learners are given physics problems to solve on their own, then collaborate within a group forum to articulate how they arrived at their solution. By the end of the week the group has to come to consensus and submit their agreed upon solutions for a single shared grade.
Integrated Assessments	Assessment is not merely summative in authentic activities but is woven seamlessly into the major task in a manner that reflects real-world evaluation processes.	In a Digital Marketing course, groups of students collaboratively develop a digital marketing campaign that integrates customer research, brand value proposition, content strategy and content development, UX design, and social media strategy, and campaign pitch for a real client.
Polished Products	Conclusions are not merely exercises or sub-steps in preparation for something else. Authentic activities culminate in the creation of a whole product, valuable in its own right.	In a music course about the historical origins of Rock Music, learners are tasked to explore the history of this era through a music compilation they themselves curate. They are prompted to compile a Spotify “best of” compilation that creates and communicates a distinct media message through 20th century rock music.
Multiple Interpretations and Outcomes	Rather than yielding a single correct answer obtained by the application of rules and procedures, authentic activities allow for diverse interpretations and competing solutions.	In a sociology course on the topic of Law, Crime, and Punishment, learners are asked to create a polished audio podcast (think This American Life) that integrates the theory of “legal consciousness” about the role and effects of law in everyday life by interviewing attorneys and clients.
Multiple Sources and Perspectives	Learners are not given a list of resources. Authentic activities provide the opportunity for students to examine the task from a variety of theoretical and practical perspectives, using a variety of resources, and requires students to distinguish relevant from irrelevant information in the process.	In a History course, learners collaboratively create an interactive digital timeline to bring a chosen time period to life. To construct the timeline, groups of learners select a theme, research relevant historical events, movements, and cultural artifacts to contribute to the collective timeline.

Methodology

This DBR study encompassed one design team’s investigation into the learning design of online courses by systematically reviewing the learning activities and assessments that learners engaged in against the principles of authentic e-learning as articulated by Herrington et al. (2010). Over a 5-year period (2014–2019), the design team implemented, revised, and refined its course review method and reviewed over 100 unique online courses. The first reviews were done with only the Quality Matters standards, but during the last 2 years (2017–2019) the reviews utilized the expanded active learning design principles. Complete data were available for 75 of the 100 online courses, and these 75 courses comprise the data set reported in this study. The goal of this design intervention was to make active learning visible in the design of online courses and to explore the extent to which the intervention outcome—an active learning score—was able to quantify active learning in the design of online courses. The individual phases of the DBR are described in the findings section of this paper to help readers follow each iteration and its outcomes.

DBR must be substantiated by existing theories and frameworks in order to articulate a close relationship between the high-level conjecture, the mediating processes, and the design intervention. The high-level conjecture of this design intervention speculated that online courses that score high in the principles of authentic e-learning provide more active learning opportunities than online courses that score low. The mediating processes are social-constructivist pedagogies such as problem-based, project-based, and student-led inquiry in support of curricular goals. The tangible intervention designed and refined during this DBR study was a course review method by which to evaluate the level of active learning in online courses for the purpose of guiding educators toward a course design that maximizes active learning. Specifically, this DBR study addressed the following two questions:

Q1: To what extent does the intervention—a new course review method—indicate the extent to which active learning is present in the design of an online course?

Q2: How do the principles of authentic e-learning incorporated in the new course review method need to be refined?

One strength of this study is that the principles of authentic e-learning were applied across a wide variety of disciplines. The online courses that were reviewed as a part of this study ranged in disciplines from the natural sciences to the social sciences, humanities, and the applied professions. As such, active learning was examined in direct relationship to diverse disciplinary outcomes, diverse disciplinary ways of thinking, and diverse disciplinary content, yet with the consistent goal to deepen the learning of the varied disciplinary curriculum through active engagement. If active learning tasks could successfully work through such a diverse set of disciplinary goals, then the resulting principles would be generalizable and as such suitable to define active learning in online courses.

Context

This DBR study was led by the first author of this paper when she was a Ph.D. student at a large public university in the midwestern US. The second author served as an external doctoral dissertation committee member. The student researcher simultaneously served as the director of a department that provided instructional design services for online courses at the university, hereafter referred to as the Office of Online and Educational Services (OES). This study was approved by the university Institutional Review Board and categorized as exempt.

OES initially based its course review process on the Quality Matters fourth edition rubric (Shattuck, 2010, 2014), a widely accepted standard of online course quality in higher education in the U.S. and other countries. This review included eight general standards of online course design: (1) Course Overview Introduction, (2) Learning Objectives, (3) Assessment and Measurement, (4) Instructional Materials, (5) Course Activities and Learner Interaction, (6) Course Technology, (7) Learner Support, and (8) Accessibility and Usability. While the QM rubric covered many important organizational and usability standards that improved the learning experience, the rubric lacked pedagogical guidance on what was meant by active learning in online courses. General Standard 5 of the QM rubric, Course Activities and Learner Interaction, suggests that “meaningful interactions (instructor to student, among students, and student to course materials) are to be employed to motivate students and foster active learning, intellectual commitment, and personal development” (Shattuck, 2014), but how active learning was defined, and consequently how active learning would be instantiated in the learning design of online courses remained a vague concept.

In spring 2016, OES conducted a semester-long professional learning initiative on the topic of authentic e-learning, guided by A Guide to Authentic e-Learning (Herrington et al., 2010). This in-depth exploration of evidence-based design principles focused the team’s discussion on both how to design for active learning using these principles but also how to review existing courses against these principles so that there would be a feedback loop that would communicate whether or not active learning was indeed taking place. As such, the principles of authentic e-learning (Table 1) presented an opportunity to grapple with the core question of how to make active learning more tangible and recognizable for all stakeholders.

Findings

Research question 1

The first major research question, “To what extent does the intervention—a new course review method—indicate the extent to which active learning is present in the design of an online course?” was addressed in four DBR phases. Phase 1 involved incorporating the authentic e-learning principles (Table 1) into the OES course review form and adopting them as OES design standards. The revised course review allowed OES to evaluate the learning activities and assessments against each of the authentic task principles (on a scale of 0 = absent, 1 = underemphasized, 2 = emphasized, 3 = maximized) resulting in an active learning (AL) score in the course review summary. In this iteration of the course review, the principles of authentic eLearning were integrated into the organizational structure of the existing course review which followed the eight general standards of the Quality Matters (QM) rubric. This had the effect of distributing the 11 authentic task principles among the QM general standard categories. While this first phase successfully integrated the principles of authentic e-learning into the OES online course review, the active learning (AL) score resulting from the aggregate of all 11 principles was difficult to interpret since the principles were spread across the QM categories.

Phase 2 of this DBR consisted of OES examining the first set of active learning (AL) course reviews and confronting the challenge of evaluating the principles consistently among members of the design team. For example, some designers took into consideration not only the task design but also the weight in grading that a learning activity or assessment was assigned to in a given course. This prompted the collaborative development of an Authentic Task Rubric by the instructional designers in which multiple courses were used to arrive at the articulation of the rubric criteria in which both the time dedicated to a learning activity and the percent of the grade assigned to the activity were taken into consideration when scoring a particular learning activity (see Table 2 as an example of the rubric for one of the authentic e-learning principles).

Table 2.

Example rubric criteria.

	Rubric level descriptions
	Absent (0)	Underemphasized (1)	Emphasized (2)	Maximized (3)
General Level Description	Learning activities and assessments do not evidence the principle.	Learning activities and assessments do evidence the principle but without emphasis in quantity: little time is allocated to the task and little to no grade points are allocated to the task.	Learning activities and assessments evidence the principle in a noticeable amount of time and the grade points allocated to the task reflect the time allocated to the task.	Learning activities and assessments evidence the principle in a significant amount of time and the majority of the grade is determined by authentic tasks.
Real-world Relevance	Learning activities and assessments are purely academic and are presented/assessed without any real-world context.	Learning activities and assessments are largely presented without a real-world context or context is provided but activities are under-emphasized in the grade scheme.	Learning activities and assessments match or simulate the real-world tasks for several of the activities and an appropriate amount of time and grade points are allocated to the tasks.	Learning activities and assessments mirror real world tasks of professionals or make clear how the knowledge is useful in the world for the majority of the activities and account for a significant portion of the final grade.

During Phase 3 of the study, it became apparent that while the 11 principles of authentic e-learning were capturing the majority of attributes of the learning activities and assessments that learners engaged in, there was a core component that they were not able to capture. This component was tied to the research on the dynamic interplay of emotion, motivation, and cognition (Dumont et al., 2010). Students’ interests, hobbies, and passions, their values, and their goals in life contributed to the complex interplay of cognition and motivation. While what motivates individual students can be widely varied, courses with a high active learning (AL) score often accommodated these individual differences by allowing learners some degree of choice in the execution of their assignments. Recognizing this gap, the design team added an additional principle to the course review, the principle of Personal Relevance. The final course review included 12 principles, with a total possible score of 36.

Phase 4 of the study consisted of OES restructuring the layout of the course review general standards to reunite the principles of authentic e-learning under a single new general standard called Learning Design. The term Learning Design, previously called Instructional Design, denotes the switch in focus from teaching to learning and the orientation toward a more learner-centered paradigm. This reorganization of the standards not only elevated active learning from its place of sub-standard in the Quality Matters scheme; it placed Learning Design as the first and central element of the course review. This standard was given such a central position because in online courses, the Learning Design of the activities and assessments that learners engaged in required upfront pedagogical considerations to create the best environmental conditions for learning. The active learning (AL) scores resulting from the course review consequently also became easier to review since they were no longer spread across the QM categories (see Supplemental Material for full set of authentic learning design principles).

Subsequently, OES undertook the first level of data analysis by reviewing the active learning (AL) scores of 75 online courses across a wide range of disciplines. The courses were grouped into three categories: low, medium, and high according to their AL scores (See Table 3).

Table 3.

Courses by low, medium, high active learning ranges.

AL level	AL score	Courses	# of courses per AL score
Low	0	LAMP 41xx	1
	1–4		0
	5	MM 300xxW	1
	6–9		0
	10	CMGT 30xxW, CMGT 40xx	2
	11	ABUS 41xx, CMGT 40xx	2
	12	ABUS 41xx, ABUS 41xx, CMGT 40xx	3
	13	CSCL 33xx, HORT 10xx	2
	14	ABUS 33xx, MATH 12xx, MATH 12xx	3
Medium	15	ENGW 31xx, CMGT 43xx, ABUS 42xx, ECON 11xx	4
	16	MKTG 30xx, MATH 10xx, ABUS 42xx	3
	17	ABUS 45xx, ABUS 40xxW, ABUS 40xx, SOC 11xx, HIST 37xx, PUBH 30xx	6
	18	CMGT 40xx, HORT 10xx, CMGT 4073, HSM 45xx	4
	19	ABUS 35xx, ABUS 40xx, CMGT 43xx, ABUS 40xxW, MM 40xx, ENGL 30xxW, MUS 10xx, MUS 10xx	8
	20	AFRO 31xx, HSM 35xx, CMGT 44xx, ABUS 40xx, MATH 11xx	5
	21	SOC 37xx, ABUS 45xx, SOC 10xx, TRIN 31xx, ENGL 30xxW, COMM 34xx, MATH 10xx	7
High	22	ABUS 30xx, ABUS 47xx, ABUS 47xx, ENGL 30xx, ECON 11xx	5
	23		0
	24	INET 10xx, MATH 22xx, ENGL 30xx	3
	25	CMGT 40xx, ABUS 45xx, MGMT 30xx, SOC 42xx	4
	26	ABUS 41xx, SOC 30xx, MATH 10xx, HIST 38xx	4
	27	INET 40xx	1
	28		0
	29	ABUS 47xx	1
	30	SPAN 10xx, MDS 30xxW, COMM 54xx	3
	31	ABUS 45xxW, HSM 30xx, SPAN 10xx	3
	32	ABUS 40xxW	1
	33–34		0
	35	HSM 45xx	1
	36		0

An active learning (AL) score range between 0 and 14 was classified as low, meaning that a redesign of the learning activities and assessments could increase active learning opportunities. An AL score of 15–21 was classified as medium, which meant that active learning opportunities were somewhat emphasized in the course. Finally, an active learning (AL) score of >21 was classified as high, which meant that active learning was fully realized within the design constraints of the curriculum, and that learning activities and assessments provided significant active learning opportunities. The ranges that classified courses into low (0–14), medium (15–21), and high (22–36) were chosen with the middle range smaller than the other two ranges because there was as much variety in the 15–21 range as there was on the low and high end. Most courses (37 out of 75) fell into the medium active learning range.

Only a single online course scored zero on the active learning scale. This course was designed around weekly lectures introducing students to the subject of pathology through various body systems. While the content lectures were of high quality, the learning activities consisted exclusively of weekly quizzes and multiple-choice high stakes summative midterm and final exams. From a learning sciences perspective, providing digital content and automated assessments provides an unacceptably shallow form of engagement whereas active learning aims to provide learners with multiple processes to construct relationships between the academic content and the learners’ prior knowledge with the goal of building relationships that transfer to long-term memory. Learning is not the same as just exposure to content but rather a process that requires the learners to engage and construct relationships between prior and new knowledge.

While the design principles of active learning applied to courses across the disciplines (mathematics, the sciences, the humanities, professional studies), each curriculum had its own unique opportunities and bespoke solutions. Courses tasked with introducing foundational disciplinary knowledge concepts required active learning processes as a means for motivation and engagement whereas higher-level courses or courses with a more divergent-type curriculum more naturally offered opportunities for authentic engagement.

None of the courses reached the total score of 36/36 which would require for each of the 12 authentic task principles to be maximized. The highest active learning score (35/36) was received by an online course on the topic of innovation. Higher-level courses or courses that included divergent course objectives (innovation, creativity) more naturally presented opportunities for authentic engagement whereas introductory courses burdened with introducing learners to specific disciplinary concepts required imagination on how to incorporate active learning methods in order to motivate and engage learners. This said, the data of this study (as shown in Table 4) showed that active learning was achievable at all levels (1000/2000, 3000, 4000) of the undergraduate journey and independent of the disciplinary subject matter.

Table 4.

Distribution of low, medium, high active learning courses by undergraduate level.

Undergraduate level	0–14 (low)	5–21 (medium)	22–36 (high)
1000/2000	3	9	6
3000	4	13	8
4000	7	16	12

To illustrate how the differences in AL scores played out in the Learning Design of different courses, details are provided below on three freshman 1000-level math courses, one each at the low level, medium level, and high level (see Table 5).

Table 5.

Comparison of a low, medium, and high active learning math course.

Authentic task principles	Course
Authentic task principles	Math 12xx	Math 11xx	Math10xx
Personal Relevance	0	1	2
Real-world Relevance	2	2	2
Ill-defined Problem	1	1	2
Sustained Investigation	1	2	3
Interdisciplinary Perspectives	1	0	1
Collaboration	3	3	1
Reflection	0	2	3
Coaching and Scaffolding	3	3	1
Integrated Assessment	1	2	3
Polished Product	2	3	2
Multiple Interpretations and Outcomes	0	1	3
Multiple Sources and Perspectives	0	1	3
Total AL Score	14	21	26

MATH 12xx with an AL score of 14 scored in the low active learning range. This course used a publisher textbook with an associated publisher course pack to introduce calculus theorems through textbook exercises and homework problems. It assessed learners’ comprehension through weekly quizzes, one midterm, and a final exam. Learners worked individually on all activities except for the weekly homework problems which required them to collaborate, reflect on their own solutions in relation to classmates’ solutions, and revise their work before submitting the final solution. Through the collaborative homework problems, the course maximized the principles of Collaborative Construction of Knowledge and Coaching and Scaffolding, but scored low on the remaining principles as learners continued to work individually on math problems and were assessed exclusively through summative quizzes, midterms, and final exams.

MATH 10xx with an AL score of 21 scored in the medium AL range. This course required a traditional midterm and final exam (a math department requirement), but also provided extensive knowledge building exercises, collaborative activities, and a reciprocal teaching project for students to demonstrate their mathematical understanding. Learners completed online problems individually that they could resubmit until they got the correct answer. Learners were also prompted to use the discussion forum to discuss practical mathematical topics and to complete collaborative group homework problems. Additionally, learners were asked to choose a calculus problem with real-world application and to create a recorded video describing the mathematical concepts and techniques needed to solve the problem. After selecting the calculus to demonstrate, learners used the discussion forum to receive feedback from other students and the instructor on their solution method. Through the weekly collaborative homework problems, the course maximized the principles of Collaborative Construction of Knowledge and Coaching and Scaffolding. Additionally, as learners worked independently on preparing a math demonstration video over several modules, the course emphasized the principles of Sustained Investigation, Real-world Relevance, Integrated Assessment, Reflection, and Polished Products.

MATH 10xx with an active learning score of 26 scored in the high AL range. In this course, students were assigned individual homework problems, as well as weekly real-world design problems for learners to solve in teams. Additionally, teams were tasked with creating a final project worth 30% of their grade in which they connected a real-world design problem with a mathematical concept introduced in the course. The team project which was to be completed over multiple modules required learners to select a design challenge that could be solved mathematically, create an elevator speech outlining the problem’s importance, and as a team create a presentation to the class that demonstrated their mathematical approach and solutions. As part of solving the problem, learners were asked to consider the situation in which the math concept would be used, to investigate if there was a historical basis or story behind solving and applying the math principles, and which perspective they used to approach the problem. Additionally, learners were asked individually to demonstrate their understanding of mathematical concepts through written reflections and exams. Through the weekly team-based design problems and the team project, the course maximized the principles of Collaboration, Sustained Investigation, Integrated Assessment, Multiple Interpretations and Outcomes, and Multiple Sources and Perspectives, Interdisciplinary Perspectives while also emphasizing the principles of Real-world Relevance, Ill-defined Problems, Learner-Relevant, Learner-Choice, and Polished Product. It is noteworthy that the Math 10xx course was originally designed for face-to-face delivery but included a companion site complete with all the learning resources and learning activities required to complete the course. When the COVID-19 pandemic necessitated the fully online delivery of Math 10xx, very little needed to be changed in the learning management system to accommodate the change in delivery to online. The exact same pedagogical learning design worked for face-to-face, blended, and fully online delivery. The idea that a well-designed active learning course site can accommodate multiple modes of delivery (face-to-face, blended, and fully online) with minimal change is critical to being able to flexibly adapt the course delivery to evolving circumstances.

Research question 2

The second major research question asked, “How do the principles of authentic e-learning incorporated in the new course review method need to be refined?” Learner survey data was used to answer this question. We asked learners to complete a voluntary, ungraded Online Course Evaluation consisting of 16 questions on three topics:

Student Readiness and Access to Learn Online

Course Design

Instructor Interaction

The survey made clear to learners that it was solely intended to provide feedback for improving their online learning experience and distinctly different from the traditional Student Rating of Teaching (SRT) which students were asked to complete separately. The purpose of a separate Online Course Evaluation was to be able to specifically address aspects of online learning that are not covered in a traditional Student Rating of Teaching (SRT).

In the Course Design section of the survey, questions 13 and 14 surveyed learners about their learning experience:

Q13: The course provides the opportunity for me to create a meaningful project, presentation, or authentic task. Students were able to respond on a five point scale ranging from strongly disagree to strongly agree.

Q14: If the course provided the opportunity for you to create a meaningful project, presentation, or authentic tasks, what made it meaningful and how? In not, describe a type of assignment that would have made it meaningful. Students were able to respond in long text format.

The first level of survey data analysis compared the learner survey ratings from over 75 unique online courses (267 course sections) against the active learning scores assigned to courses by the design team. Of 5836 students in these courses, 1586 learners responded to the survey (27%). Table 6 shows learner survey ratings in low, medium, and high active learning courses.

Table 6.

Learner survey ratings of the item: “This course provided the opportunity to create a meaningful project, presentation, or authentic task.” in low, medium, and high active learning courses.

AL range	Total responses	Strongly disagree (%)	Disagree (%)	Neutral (%)	Agree (%)	Strongly agree (%)
High	537	3	3	16	37	41
Medium	722	2	4	14	45	35
Low	327	3	3	15	40	38

Interestingly, the percentage of students agreeing or strongly agreeing that they had the opportunity to create a meaningful project, presentation, or authentic task did not significantly differ between courses in the low, mid, or high AL range. It is likely that students were thinking about a single activity in their response, whereas the AL score looked at the entire course and was an aggregate of all the learning activities and assessments. Thus, it was important to use the qualitative data provided by students to better understand their experiences.

The next level of analysis of the learner surveys coded nearly 1000 qualitative survey responses to the question: “If the course provided the opportunity for you to create a meaningful project, presentation or authentic task, what made it meaningful and how? If not, describe a type of assignment that would have made it meaningful.” Table 7 shows samples of how statements from the Learner Survey were coded by authentic task principles. A view of all learner comments coded by these principles is available in the Appendix of the dissertation from which this paper was derived (Reilly, 2020).

Table 7.

Learner survey statements coded by authentic task principles.

No. of learners	Authentic task principle with a sample quote
153	Real-world Relevance. “The final Applied Management Research Paper allowed students to analyze the activities of a real-world manager in the light of the management theory presented in our textbook. Rather than simply relying on some examples from a textbook, we were able to evaluate the decisions made by real world managers based on the theory that we learned. This gave us a chance to draw conclusions about how theory is applied when managing an actual organization.”
84	Collaboration. “The most meaningful aspect of the various projects assigned in class, such as the group project, was the ability to work with other students and share ideas and information on the texts that we read. I found this to be helpful when completing my assignments, such as the essays.”
67	Integrated Assessment. “The final allowed me to create something that represented all we had learned and I really liked that.”
47	Reflection. “Weekly reflections & the occasional mini papers forced me to utilize and apply my learnings. It made me reflect on the different leadership concepts we’ve learned and how they’re connected. Doing these assignments have helped me prepare for real life leadership experiences.”
20	Articulation. “It allowed us to create demonstration videos and for our classmates to view them which allowed us to learn the problems further in depth and be able to communicate them.”
15	Polished Products. “The Spotify assignment helped me think about the subject material cumulatively and made a tangible thing to show my knowledge to others.”
12	Multiple Sources and Perspectives. “I think that Essay C was very meaningful because it allowed me to connect with a different generation and see how their opinions impacted their perception of the same music that I heard. It was meaningful because I can apply this knowledge outside of the course and remember that other’s different experiences will make their perception of an event different from mine.”
11	Multiple Interpretations and Outcomes. “I thought that the playlist assignment was very meaningful. It allowed us to make a compilation of the most important songs and I thought it was really cool to see that everyone’s playlist was very different.”
6	Sustained Investigation. “It was meaningful to be able to create a project based on our course materials and work on it throughout the semester.”
2	Interdisciplinary Perspectives. “The timeline project gave us an opportunity to study the historical context of the literature for that week. In my opinion, knowing the history made the readings more interesting and engaging.”
1	Ill-Defined. “I really liked how the projects were long and tough, which made it more appealing because we had to figure out on our own how to complete it”.

The authentic task principle with the highest frequency was Real-world Relevance (153). Learners repeatedly stated that being able to make the connection between what they were learning and how it applied in the real world helped them understand the course work and made what they were learning memorable and meaningful. Many of them described that being able to apply what they were learning to the real world helped them see the usefulness of the information. Others explained being able to compare what they learned in class with the real world helped them gain further insights into the course material as well as allowed them to imagine what it would be like to work in that field.

The authentic task principle with the second highest frequency count was Collaboration (84). Learners stated that they enjoyed learning and working with their classmates which speaks to collaboration meeting some of the learners’ social needs, namely, to meet new people, see what other students are involved in, and having the opportunity for learners to get to know each other and develop meaningful peer relationships. The majority of learners selecting Collaboration also spoke of the academic benefits of working with their classmates. They stated that working together on a project pooled their best ideas and produced a better result than working independently. They remarked that receiving feedback from classmates helped them understand the class material better and improved their own work. As this learner stated:

The group homework discussions were integral to developing meaningful relationships with my peers along with helping to build a stronger foundation on topics that were difficult to understand.

Others commented that discussions with classmates helped them see additional viewpoints and exposed diverse perspectives that enriched their thinking. This was particularly true when learners were asked to share their personal views on a subject matter. By sharing their own perspectives, learners added nuance and complexity to the disciplinary subject matter beyond the material that was presented. Learners further commented that listening to the perspectives of their classmates helped them understand that there often is no single right answer but rather a continuum of perspectives. This realization was new to some students whose main coursework came from disciplines in which right answers were routinely a part of their course assessment. When group work had been successful, learners made positive statements about having become a team and expressed pride about having accomplished something together. Students also commented that they valued collaboration as a skill in and of itself (in particular the skills derived from coordinating and collaborating fully online) because learners were aware that teamwork was going to be valued in their future workplace.

But not all collaboration comments were positive. Of the 84 learners who stated that collaboration with others helped them learn, 13 learners commented that they struggled with one or the other aspect when learning in groups due to challenges in having to coordinate with others, group members not doing their share, the make-up of the group, or the assignment not lending itself to collaborative work. The negative comments about collaboration provide an important insight into the need to purposefully design collaborative assignments for individual accountability and positive interdependence between the group members and the need for online instructors to pay attention to group culture and to proactively set expectations on how to collaborate successfully (Oh and Reeves, 2015).

The authentic task principle with the third highest frequency was Integrated Assessment (67) though learners did not use those exact words. Instead, they described the activities as synthesizing assignments that helped them bring together/culminate/apply/connect/make sense of everything they learned that semester. Many of the learning activities that learners referred to as culminating or synthesizing fell into the category of inquiry and project-based learning and were mentioned in learner statements as final assignments, final presentations, final papers, final projects, etc.

The authentic task principle with the fourth highest frequency was Reflection (47) which learners stated helped them look back at what they were learning, build connections, and deepen their understanding. Learners especially spoke of Reflection being a powerful tool for learning when it involved self-reflection. Being able to look back and articulate their understanding through self-reflection, namely being able to articulate how they personally related to or understood the materials, helped them build stronger relationships to the course material.

Discussion

This DBR study addressed the lack of a definition of active learning as a quality standard for online course design, and through its literature review established the principles of authentic eLearning (Herrington et al., 2010) as a well-suited heuristic for active learning. While it is important to keep in mind that the findings were derived at one institution and based on the learning designs that were available in these courses, this DBR demonstrated through its design intervention that active learning in online courses can indeed be quantified. By applying these evidence-based principles along with the added learner-centered principles (see Supplemental Material) the high-level conjecture of this design intervention, namely that online courses that score high in the principles of authentic e-learning provide more active learning opportunities than courses that score low, was supported by the data.

Courses in the low active learning range (<15) offered minimal opportunities for learners to build relationships between the academic content and the real-world, and offered fewer complex use cases that would clarify the purpose of the knowledge for the learner. They also provided minimal opportunities to connect the academic content to the learners’ prior knowledge. This paucity of connection opportunities ignores the situated nature of learning (Lave and Wenger, 1991). Courses in the low active learning range further underemphasized the social nature of learning so core to human cognition (Brown et al., 1989) and as such did not leverage the potential that collaboration has to offer to enhance learning. Finally, the learning activities and assessments in low active learning courses lacked opportunities for learners to self-regulate their learning which is important for learners to practice, so that they develop the meta-cognitive abilities that lead to learning independently and successfully (Broadbent, 2017).

Courses in the medium active learning range showed strong representation of some of the principles but these courses did not truly create authentic contexts. Learners may have been provided “suitable examples from real-world situations to illustrate the concept or issue being taught,” such as a case study, “but they did not go as far as providing the purpose and motivation for learning, nor did they provide a sustained and complex learning environment that can be explored at length” (Herrington et al., 2010: 19).

Courses that fell in the high active learning range (>21) operationalized cognitive science empirical findings that learning is constructed by a learner through interactions with others and with the world (Bandura, 1989). These online courses were creatively designed to bring the curriculum alive by infusing the course with opportunities for learners to build connections to real-world contexts, their own interests, and through rich exchanges and collaboration with classmates. These courses provided opportunities for learners to construct their own learning by being tasked with sustained investigations into a subject matter that required them to research, collaborate, reflect, articulate, and produce a product that demonstrates their learning.

Courses in the high active learning range consistently exhibited a creative use of technology guided by the pedagogical understanding of learning as a constructive, self-directed, situated, and collaborative process (De Corte, 2010; Morris, 2019). The learning design was focused on the learning experience and the use of technology was determined by the alignment of technological affordances to pedagogical needs. Moreover, highly active learning courses had reordered the disciplinary content in support of the learning experience. These courses wove the entire course content into a narrative structure organized around the learning experience and effectively scripted the course material into a storyline organized around how the learner would experience the sequence. Courses that followed a narrative structure received the highest ratings of personal relevance which was frequently accomplished by allowing learners some degree of choice of how to execute their assignments. The products that learners created to demonstrate their learning were unique and evaluated via a rubric to assure that they nevertheless met all of the disciplinary goals articulated for the course. Not only did these high active learning courses meet all of the disciplinary outcomes articulated for a course, but they frequently elevated the levels of learning because by their very nature, active learning tasks operate at an applied or higher cognitive level with learners engaged in the constructive process of building their own mental models and other types of higher order learning.

Some educational researchers position constructivism in opposition to direct instruction (Kirschner and Hendrick, 2020). Nothing could be further from the truth. Direct instruction and constructivist approaches complement one another. They each hold an important role in the design of effective learning environments. Learning as an active and constructive process does not at all imply that students’ construction of their knowledge should not be guided and mediated through appropriate modeling, coaching, and scaffolding by teachers, peers, and educational media (Collins et al., 1989). Nor does the importance of active learning diminish the value of a carefully curated curriculum that scaffolds basic elements of disciplinary knowledge such as foundational terminology, concepts, and frameworks into the more complex knowledge structures of the discipline. Engaging learners in active learning experiences does not imply that learners should start from scratch and rediscover knowledge structures that took decades to develop independently and without guidance. These knowledge structures can be provided through direct instruction focused on the major ideas and frameworks in the disciplinary content, but active learning provides the necessary opportunities to apply and refine that knowledge.

The point is to consider learning design beyond the content per se and to focus design considerations on learning experiences that allow learners to build the real-world, personally relevant, interdisciplinary, relationships to the subject matter (Hokanson et al., 2015). Learning design must increase opportunities for reflection, articulation, and collaboration with peers on projects for a sustained amount of time that require all of the learners’ capabilities: cognitive (thinking), affective (feeling), and conative (doing) so that learners can construct these relationships. The Active Learning principles derived from this DBR provide the guidance that learning designers need to provide learners with these critical opportunities.

Admittedly, it requires creativity to design online courses that weave together the existing curriculum into something that the learner can experience. This is where the design principles become useful. The design principles implemented in the active learning (AL) course review serve as generators that power-up the learning design of an online course to increase the likelihood for learners to engage and build connections. The investment of time and effort in redesigning an existing online course for active learning, however, is well worth it because a well-designed active learning online course provides teachers with the flexibility to change the modality of learning from flipped classroom to blended to fully online rapidly and as such future proofs the course in the case of catastrophic event like a pandemic or simply when more flexibility is needed. Moreover, the principles of authentic learning that define active learning hold even when teaching a course in the classroom.

The COVID pandemic which forced teachers world-wide to convert their face-to-face courses to online delivery highlights the importance of supporting teachers in designing for active learning online. Active learning is as much about deepening comprehension as it is about engaging and motivating learners. The authentic e-Learning principles while relevant for all modes of delivery will have a lasting impact for both learner engagement and learning outcomes now that they can be made visible. Future research should examine interactions between disciplines and the dimensions of active learning.

Active learning is not new, and neither are the principles of authentic e-learning, not even the learner-centered principles. What is new as a result of this DBR is that these evidence-based principles can now be applied during a course review to reveal the extent of active learning incorporated into any given course. As such, the active learning (AL) course review lends itself to support teachers in reflective practice similar to the peer review method in which the Quality Matters rubric is currently being applied. Furthermore, the AL review could serve to support teacher professional learning in how to design for active learning online because the scaffolding of authentic learning experiences such as effective online collaboration requires careful consideration of team dynamics, positive interdependence, and individual accountability in the learning design. Hopefully in time, being able to quantify a minimum degree of active learning course design will become a standard of online quality similar to the way that online courses today need to demonstrate regular and substantive interaction between instructor and student to qualify for federal financial aid in the United States.

Supplemental Material

sj-docx-1-alh-10.1177_14697874221096140 – Supplemental material for Refining active learning design principles through design-based research

Supplemental material, sj-docx-1-alh-10.1177_14697874221096140 for Refining active learning design principles through design-based research by Christiane Reilly and Thomas C Reeves in Active Learning in Higher Education

Footnotes

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Christiane Reilly

Supplemental material

Supplemental material for this article is available online.

Author biographies

Christiane Reilly, Ph.D. is a design researcher focused on the pedagogical use of technology to advance teaching and learning. Her current research interests focus on learning designs that foster learner agency, learner creativity, empathy, cultural humility, metacognition, and other higher order skills.

Thomas C. Reeves, Ph.D. is a Professor Emeritus of Learning, Design, and Technology in the College of Education at The University of Georgia. His research interests include effective applications of technology to support teaching and learning around the world, educational design research, and evaluation of medical and public health education.

References

Bakken

Bridges

(2011) National Standards for Quality Online Courses (Version 2). Vienna, VA: International Association for K-12 Online Learning iNACOL. Available at: http://www.aurora-institute.org/wp-content/uploads/national-standards-for-quality-online-courses-v2.pdf (accessed 5 January 2022).

Bandura

(1989) Human agency in social cognitive theory. American Psychologist 44(9): 1175–84.

Baum

McPherson

(2019) The human factor: The promise & limits of online education. Daedalus 148(4): 235–54.

Berliner

(2008) Research, policy, and practice: The great disconnect. In: Lapan

Quartaroli

(eds) Research Essentials: An Introduction to Designs and Practices. San Francisco, CA: Jossey-Bass, pp.295–326.

Broadbent

(2017) Comparing online and blended learner’s self-regulated learning strategies and academic performance. The Internet and Higher Education 33: 24–32.

Brown

Collins

Duguid

(1989) Situated cognition and the culture of learning. Educational Researcher 18(1): 32–42.

Carr

Palmer

Hagel

(2015) Active learning: The importance of developing a comprehensive measure. Active Learning in Higher Education 16(3): 173–86.

Chi

(2009) Active-constructive-interactive: A conceptual framework for differentiating learning activities. Topics in Cognitive Science 1(1): 73–105.

Collins

Brown

Newman

(1989) Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In: Resnick

(ed.) Knowing, Learning and Instruction: Essays in Honor of Robert Glaser. Hillsdale, NJ: Lawrence Erlbaum Associates, pp.453–94.

10.

De Corte

(2010) Historical developments in the understanding of learning. In: Hanna

David

Francisco

(eds) The Nature of Learning. Using Research to Inspire Practice. Paris: OECD Publishing, pp.35–67.

11.

Dumont

Istance

Benavides

(eds) (2010) The Nature of Learning: Using Research to Inspire Practice. Paris: OECD Publishing.

12.

Falkner

Sheard

(2019) Pedagogic approaches. In: Fincher

Robins

(eds) The Cambridge Handbook of Computing Education Research. Cambridge: Cambridge University Press, pp.445–80.

13.

Freeman

Eddy

McDonough

, et al. (2014) Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences of the United States of America 111(23): 8410–5.

14.

Ghilay

(2015) TBAL: Technology-based active learning in higher education. Journal of Education and Learning 4(4): 10–8.

15.

Herrington

Reeves

Oliver

(2010) A Guide to Authentic E-learning. London: Routledge.

16.

Hokanson

Clinton

Tracey

(eds) (2015) The Design of Learning Experience: The Future of Educational Technology. New York, NY: Springer-Verlag.

17.

Kirschner

Hendrick

(2020) How Learning Happens: Seminal Works in Educational Psychology and What They Mean in Practice. London: Routledge.

18.

Lave

Wenger

(1991) Situated Learning: Legitimate Peripheral Participation. Cambridge: Cambridge University Press.

19.

Lee

Recker

Yuan

(2020) The validity and instructional value of a rubric for evaluating online course quality: An empirical study. Online Learning 24(1): 245–63.

20.

Lowenthal

Hodges

(2015) In search of quality: Using quality matters to analyze the quality of massive, open, online courses (MOOCs). The International Review of Research in Open and Distributed Learning 16(5): 83–101.

21.

Michael

(2006) Where’s the evidence that active learning works? Advances in Physiology Education 30: 159–67.

22.

Morris

(2019) Self-directed learning: A fundamental competence in a rapidly changing world. International Review of Education 65(4): 633–53.

23.

National Research Council (1999) How People Learn: Bridging Research and Practice. Washington, DC: National Academies Press.

24.

Reeves

(2015) Collaborating online: A logic model of online collaborative group work for adult learners. International Journal of Online Pedagogy and Course Design 5(3): 47–61.

25.

Pallas

Neumann

Campbell

(2017) Policies and Practices to Support Undergraduate Teaching Improvement. Cambridge, MA: American Academy of Arts and Sciences.

26.

Penuel

Fishman

Haugan Cheng

, et al. (2011) Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher 40(7): 331–7.

27.

Prince

(2004) Does active learning work? A review of the research. Journal of Engineering Education 93(3): 223–31.

28.

Reilly

(2020) Defining active learning in online courses through principles for design. Doctoral Dissertation, University of Minnesota, Minneapolis, MN.

29.

Schneider

Stern

(2010) The cognitive perspective on learning: Ten cornerstone findings. In: Dumont

(ed.) The Nature of Learning: Using Research to Inspire Practice. Paris: OECD Publishing, pp.69–90.

30.

Shattuck

(2010) Quality matters: A faculty-centred program to assure quality in online course design. Collected Essays on Learning and Teaching 3: 49–53.

31.

Shattuck

(ed.) (2014) Assuring Quality in Online Education: Practices and Processes at the Teaching, Resource, and Program Levels. Sterling, VA: Stylus Publishing, LLC.

32.

Swan

(2004) Relationships Between Interactions and Learning in Online Environments. Needham, MA: The Sloan Consortium.

33.

Virtual Learning Leadership Alliance and Quality Matters (2019) National standards for quality online teaching. Available at: https://www.nsqol.org/the-standards/quality-online-teaching/ (accessed 5 January 2022).

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB