Pedagogical Agent in Math versus Art Lessons: Impact on Learning,Perceived Workload,and Motivation

Objectives

Pedagogical agents (PAs) are virtual anthropomorphic characters designed to enhance digital learning environments and have emerged as a frequent tool in the field of computer-aided education. PAs are used to add a social aspect to the learning experience. They have been shown to increase student learning and motivation and to decrease cognitive load ( Atkinson, 2002 ; Dinçer & Doğanay, 2017 ; Moreno et al., 2001 ). Despite the growing body of research demonstrating the effectiveness of PAs, a noticeable disparity exists in the focus of these studies. STEM (Science, Technology, Engineering, and Mathematics) disciplines are often represented in such studies, but the humanities and other non-STEM fields have received little attention.

This disproportionate research focus provides reason to question whether the demonstrated effectiveness of PAs applies to all academic domains. While the learning of STEM subjects has been shown to benefit from the guidance provided by PAs ( Kizilkaya & Askar, 2008 ; Moreno et al., 2001 ), it remains unclear whether these benefits translate equally to the humanities. For instance, art, a humanities subject, presents different pedagogical challenges and learning objectives compared to math. It is therefore possible that these subjects require distinct approaches to achieve optimal instructional design. A meta-analytical study ( Schroeder et al., 2013 ) provided preliminary support for differential impacts of PAs in mathematics—a field characterized by structured and objective problem solving—and the humanities, which typically engage learners in interpretative and subjective thinking. However, empirical research directly comparing the effectiveness of PAs in these two domains is notably lacking.

Understanding where and how PAs can be most effectively employed is important for maximizing educational outcomes. The current investigation was conducted to bridge this gap in the literature by empirically examining the role of PAs in two contrasting domains: mathematics lessons and art lessons. Specifically, this study addresses the following research question: What is the effect of a PA on learning outcomes, perceived workload, and motivation in a mathematics lesson versus an art lesson?

Approach

The reported findings are an analysis of data from a larger study conducted to compare learning outcomes among instructional design conditions that differed in the presentation of a PA.

Participants (N = 48) were recruited from two sources: undergraduate students in an introductory psychology course at the Georgia Institute of Technology and undergraduate college students registered on the online recruitment platform Prolific. During a 2 hr session, participants watched one math video lesson and one art video lesson. For each lesson, they completed a pre-test, a post-test, and self-report measures of perceived workload and motivation. The math lesson covered either multinomial distribution or Bernoulli trials. The art lesson covered either the elements of art or the principles of design. The math and art topics were randomly assigned, as was the order of lesson presentation. The instructional design condition for each of the two lessons was also randomly assigned under the constraint that a participant could not receive the same design condition for both.

Data from participants who were assigned to a PA condition for one or both of their lessons (n = 29) was analyzed. Thus, a one-way design where the independent variable, Lesson, had two levels (math and art) was used. Three dependent variables were measured: perceived workload, motivation, and learning. Workload and motivation were measured by the NASA Task Load Index (NASA-TLX; Hart & Staveland, 1988 ) and the Reduced Instructional Materials Motivation Survey (RIMMS; Loorbach et al., 2015 ), respectively. Learning was operationalized as the change in participant scores from pre-test to post-test.

Findings

Linear mixed-effects model analyses were conducted with 32 observations from 29 participants to compare the effects of math versus art lessons within conditions involving a PA. Differences in learning, perceived workload, and motivation were examined.

Participants reported higher perceived workload after the math lessons than after the art lessons, with significantly higher NASA-TLX composite scores for math (β = 4.65, SE = 1.47, t(6.69) = 3.16, p = .02). The model estimated that the perceived workload scores for math lessons were, on average, 4.65 points higher than for art lessons. The average composite NASA-TLX scores following the math lessons and the art lessons can be seen in Figure 1.

Figure 1.

Average composite NASA-TLX score by lesson.

A significant difference in self-reported motivation between the math and art lessons was also found; composite RIMMS scores were lower for the math lessons than the art lessons (β = −.69, SE = 0.29, t(19.75) = −2.39, p = .03), indicating lower motivation regarding the math lessons. The average composite RIMMS scores following the math lessons and the art lessons can be seen in Figure 2.

Figure 2.

Average composite RIMMS score by lesson.

The critical finding of the present work is that, despite the heightened workload and reduced motivation found for the math lessons, participants achieved significantly higher learning in the math lessons than in the art lessons (β = 22.58, SE = 7.56, t(7.07) = 2.99, p = .02). The average score improvement for the math lessons and the art lessons can be seen in Figure 3. Such a finding has implications regarding the varying impact of PAs across domains, considerably contributing to the body of literature on PAs.

Figure 3.

Average score improvement by lesson.

Takeaways

The outcomes of this study facilitate a more thorough understanding of how PAs impact learning in different subject areas. The finding that participants had higher learning in the math lessons than art lessons directly supports the insights from a prior meta-analysis ( Schroeder et al., 2013 ) that the magnitude of a PA’s pedagogical value could depend on the academic discipline in which it is deployed; for instance, favoring disciplines like mathematics over the humanities. Unique to this study is the demonstration that improved math learning persists despite higher workload and lower motivation. By analyzing workload and motivation, the present analysis offers a more comprehensive view of student experiences with PAs, suggesting that their learning benefits could potentially outweigh the increased workload and motivational challenges associated with complex subjects like math.

For educators and instructional designers, these findings emphasize the value of integrating PAs strategically within digital learning environments, particularly in subjects traditionally seen as challenging. This study also demonstrates a promising area for future research. Namely, investigations of potential interactions between instructional designs (e.g., the inclusion of PAs) and academic disciplines to reveal optimal pairing principles. Future work could also more conclusively examine whether PAs truly combat the heightened workload and decreased motivation found in complicated subjects such as math.

In summary, this study underscores the importance of continuing to investigate the complex relationship between computer-based instructional methods, subject matter, and learning outcomes. Such investigations offer the potential for a better understanding of the nuances of digital learning, particularly in the context of PA-enhanced environments.