Abstract
Teachers interested in implementing problem-based learning (PBL) in classrooms encounter many challenges. Research suggests learning analytics (LA) can provide insights into students’ problem-solving process and offer teachers valuable information about when to provide necessary scaffolding. Yet, LA research in K-12 is lacking. There is also a need to explore how K-12 students’ self-efficacy may impact their problem-solving performance and learning behaviors. This study examined 418 middle school students’ self-efficacy, their learning performance and behavioral patterns after using a multimedia-enriched PBL program, and explore the relationships among these factors. Using a mixed-methods design, this study showed students’ different levels of self-efficacy were correlated with their success rates in solving problems. Students with higher self-efficacy used more appropriate tools at the appropriate times than students with lower self-efficacy. Students’ different levels of content knowledge also play a role in how students determine when and what tools to use to facilitate their problem-solving.
Keywords
Research has demonstrated the benefits of problem-based learning (PBL) for K-12 students, yet it has also indicated that teachers encounter challenges when they implement PBL in their classrooms. Recent research suggests that techniques of learning analytics (LA) can provide insight into students’ learning processes in open-ended environments such as PBL and offer teachers valuable information about when to provide necessary scaffolding. However, research with LA in K-12 is lacking. Positive correlations have been found between students’ self-efficacy (SE) and performance, but there is a need to explore how K-12 students’ self-efficacy may impact their problem-solving performance and learning behaviors. In this study, we use LA to examine middle school students’ self-efficacy and performance in a PBL context, and we explore relationships between these students’ self-efficacy and their learning performance and behaviors.
Literature Review
PBL in K-12 Education
PBL is a student-centered approach in which students learn by solving problems and reflecting on their learning experiences (Barrows & Tamblyn, 1980). It shares similarities with project-based learning in that both are student-centered and aim for promoting higher-order thinking skills (Savery, 2006). However, in project-based learning learners are usually given specific requirements to create an artifact or product as the outcome (Thomas, 2000), PBL, the focus of this study, emphasizes on the problem-solving process and students developing solutions to an ill-defined problem. PBL has been employed at different grade levels and in various subjects, including math, science, social studies, and literacy (Merritt et al., 2017), and a number of studies have shown that PBL positively impacts students’ academic achievement. Günter and Alpat (2017) found that students using PBL had higher academic achievement than those in traditional teaching. Spires et al. (2011) indicated that problem-solving tasks helped middle school students gain knowledge of science content, and their effective selection of hypotheses within PBL was a key to successful learning. PBL encourages learners’ control over the learning process, such that students can freely utilize their prior knowledge and experience within the PBL environment (Kubiatko & Vaculová, 2011). However, the influence of prior knowledge on students’ final learning achievement needs more research. Some have argued that knowledge gained in PBL does not necessarily result from PBL itself, because other factors may be involved: teachers’ manners of implementation, the composition of student groups, and technological infrastructure (Kokotsaki et al., 2016).
Research has also indicated that PBL has the potential to improve students’ higher-order thinking skills, attitudes about learning, and self-efficacy. Lapuz and Fulgencio (2020) found PBL effective in enhancing critical thinking among middle school students. Valdez and Bungihan (2019) found that PBL improved high school students’ problem-solving in chemistry. There is also empirical evidence that K-12 students’ attitudes toward science can be improved with PBL (e.g., Akınoğlu & Tandoğan, 2007), and PBL has increased students’ self-efficacy significantly (e.g., Duman & Özçelik, 2018).
Nevertheless, despite the many benefits of adopting PBL in K-12 education, some teachers and students consider its implementation in the classroom to be a challenge (e.g., Wijnen et al., 2017). In PBL, students are intended to be responsible for their own learning, play a more active role, and think critically about their learning processes (Bereiter & Scardamalia, 2000). Some researchers have pointed out that this high demand makes young K-12 students struggle with tasks, which can negatively influence their learning experiences (Belland et al., 2017). The depth and breadth of the required learning in PBL also create obstacles for young learners (Botelho et al., 2018). Al-Kloub et al. (2014) have noted that while students are in PBL, they can experience uncertainty and confusion, a large workload, and a lack of confidence. Some researchers have also found that PBL can be difficult for some students, especially those who lack confidence in analyzing data to solve problems (Nijhuis et al., 2005). Yuan et al. (2011) showed that students felt that PBL involved a lack of guidance and was time consuming, making them stressed and frustrated.
In implementing PBL, teachers face challenges as well. The literature on teachers’ perceptions of adopting PBL suggests that they may struggle with allowing students the freedom to explore content (Maxwell et al., 2005), with assessing the open-ended nature of the PBL environment (Hung et al., 2019), and with tracking students’ behavior as they progress in problem-solving (Chen et al., 2021). In PBL, teachers must provide students with optimal scaffolding (Kim et al., 2018), move students through the stages of problem-solving, and monitor group processes (Hmelo-Silver, 2004). Thus, teachers need to monitor the students’ learning throughout. But some teachers are unsure about monitoring students’ progress in learning: they find it difficult to identify relevant issues and problems (Ng et al., 2014), and they need assistance (Chen et al., 2021). For example, students in Tawfik et al. (2021) stated that they initially received too much guidance or unnecessary support in the middle of problem-solving, which redirected them away from their own directions. This difficult balance highlights a challenge to teachers’ provision of appropriate scaffolding at the right times (Chen et al., 2016). We believe that recent advancements in LA, with the analysis of the data produced during learning, can offer teachers insightful perspectives into students’ learning processes (El Alfy et al., 2019), because LA can capture the trajectory of each student's problem-solving patterns (Foster & Siddle, 2020).
LA and Students’ Learning Performance
LA consists of collecting, analyzing, and reporting data to better understand an environment where learning takes place and optimize it (Siemens & Baker, 2012). Understanding factors that influence students’ academic performance has become a common topic in LA, owing to the availability of abundant data for students’ learning within specific contexts (Tempelaar et al., 2015). Researchers have investigated learning environments, habits, skills, and personal characteristics in relation to students’ learning performance in order to enhance their learning outcomes (Credé & Kuncel, 2008).
In LA, the characteristics of learners are important for potentially optimizing the learning process to support students’ learning performance (Avella et al., 2016). LA can promote educators’ understanding of students’ learning experience with technology such as games or other online tools, which enrich students’ learning experience (e.g., Hung & Zhang, 2012). LA can also provide valuable insights into students’ learning behaviors and thus help researchers and educators provide personalized learning solutions tailored to students’ learning processes (e.g., DiCerbo, 2014). Because students’ characteristics (e.g., self-efficacy) vary in scale, it is meaningful to examine how differences in levels of personal characteristics can impact students’ learning performance.
LA has been applied less in K-12 than in higher education (Kovanovic et al., 2021). There is a need to apply LA to gain insights for optimal learning, given K-12 students’ various characteristics and behavioral patterns. Kostagiolas et al. (2019), for example, has pointed out that the literature on the relationship between students’ self-efficacy and performance from the perspective of LA is limited. Especially in K-12, there is a need for the use of LA to support students’ self-efficacy to improve students’ learning outcomes.
Self-Efficacy Associated With Learning Outcomes, Learning Behaviors, and Perceptions
Self-efficacy, as defined by Bandura (1994), pertains to an individual's perception of their ability to achieve a desired outcome. In contrast to growth mindset, which posits that an individual's capabilities can be developed through persistent effort (Claro et al., 2016), self-efficacy is specific to particular domains and linked to their prior learning experiences in that particular area (Bai et al., 2019). The self-efficacy of students may differ across domains due to variations in their success, mastery experiences, and setbacks in each task (Pajares, 2006). According to current literature, self-efficacy is a significant predictor of students’ performance (e.g., Tan et al., 2021; Yadav et al., 2021), and self-efficacy positively affects students’ learning outcomes and performance (Amini et al., 2019; Yadav et al., 2021). Students with higher self-efficacy have better learning outcomes than those with lower self-efficacy (Amini et al., 2019). These effects have been examined in various disciplines. Tan et al. (2021), for example, found significant positive correlations between conceptions of learning science and self-efficacy for learning science in 431 undergraduate students. Yadav et al. (2021) found that self-efficacy was a major predictor of learning outcomes among university students in computer science.
Researchers have also examined the effects of self-efficacy on learning behaviors. Self-efficacy can be an important mediator for students’ problem-solving (Fadillah et al., 2021; Fitriani et al., 2020; Hwang & Oh, 2021). Students with high self-efficacy are more willing to tackle challenging problems, whereas students with low self-efficacy are more likely to become frustrated in solving problems and simply give up (e.g., Demirören et al., 2016). In problem-solving and self-directed learning among nursing students, Hwang and Oh (2021) found that self-efficacy played a mediating role: students with higher self-efficacy had higher problem-solving ability and higher self-directed learning. In examining junior high school students’ metacognition in solving math problems, Fadillah et al. (2021) found that students with higher self-efficacy were more effective in planning and monitoring their learning.
Research also shows positive relationships between self-efficacy and students’ attitudes toward and perceptions of learning. Hong et al. (2016), for example, found that students with high self-efficacy were more interested in their current learning and had higher learning satisfaction. Lee and List (2021) found that self-efficacy was a significant predictor of students’ perceptions of task difficulty in the relationship between students’ self-efficacy and their perceptions of their training: students with a strong sense of competency considered the learning task easier to complete. Similarly, Jia et al. (2014) examined whether self-efficacy influenced learners’ perceptions of the effectiveness of learning activities and found that individuals with higher self-efficacy were more likely to describe learning activities as enjoyable. However, although many researchers have examined the importance of self-efficacy, we lack research exploring how K-12 students’ varying levels of self-efficacy may impact their performance in solving problems and their learning behaviors.
In this study, we build on our previous research, in which we investigated the impact of PBL on performance and explored the relationships among self-efficacy, attitudes toward science, and learning outcomes in middle school students in a multimedia-enriched PBL program (Liu et al., 2021). The results showed that students’ science knowledge was increased significantly by the PBL program and that students’ self-efficacy was a significant predictor of their science knowledge. Self-efficacy mediated between students’ attitudes and science knowledge. However, we did not examine the students’ behavioral patterns in problem-solving with PBL, and we did not thoroughly evaluate the role of varying levels of students’ self-efficacy and science knowledge in problem-solving. This present study intends to further investigate the relationships among these factors using LA.
Research Questions
The goal of this study is to examine the role of students’ self-efficacy and science knowledge in problem-solving. To examine the role of self-efficacy, we ask the following research questions:
What is the relationship between students’ self-efficacy and their behavioral patterns? Are there any differences in students’ science knowledge and problem-solving skills corresponding to their different self-efficacy levels, and what are students’ perceptions of the PBL program? What is the relationship between students’ science knowledge and their behavioral patterns? What is the relationship between the changes in students’ science knowledge and their behavioral patterns? What do pathway analyses reveal regarding students’ behavioral patterns during the different stages of problem-solving, given different levels of self-efficacy and science knowledge?
To examine the role of students’ science knowledge, we ask the following research questions:
In addition, we ask the following research question:
Method
Research Context
A total of 418 sixth graders (male = 211, female = 207) from three middle schools in the western and southwestern United States participated in this study. Alien Rescue, a 3D immersive multimedia-enriched PBL program, was used to examine problem-solving in PBL. In this program, sixth graders learn about space science by rescuing six different alien species and finding new suitable homes for them. Several multimedia tools (see Table 1 and Figure 1) are built into the program to promote students’ problem-solving with instructional scaffolding. These tools enable students to engage in cognitive activities to solve problems, and they are designed to support students’ scientific inquiry and help them develop self-efficacy (Liu et al., 2021).
Screenshots of technology tools in Alien Rescue to support scientific inquiry. (a) Alien Information Center. (b) Solar System Database. (c) Probe Design Center. (d) Notebook. (e) Solution Form. (f) Mission Control Center.
Technology-Enriched Tools in the Problem-Based Learning Program.
The PBL program includes several problem-solving activities. Students research the features of each alien and planet to determine which planets are suitable for each alien species. Databases (e.g., Alien Information Center, Solar System Database) are designed to provide information about the aliens and planets. To test their hypotheses about the aliens’ suitability for living on specific planets, students design and launch probes in the Probe Design Center, then collect data from the probes in the Mission Control Center. To perform these problem-solving activities, students must plan, gather relevant information, and determine how to use multimedia tools effectively.
Data Sources and Analysis
This is a mixed-methods study. First, eight items from the Motivated Strategies for Learning Questionnaire (Pintrich et al., 1993) were used to measure students’ self-efficacy (Cronbach's alpha = 0.92 for this sample). Second, a 20-item multiple-choice science knowledge test was used to measure students’ understanding of key science concepts introduced in the PBL program. This test was used previously to study the same program with similar age groups (Liu et al., 2019, 2021) with Cronbach's alpha = 0.84 for the present sample. The students completed the self-efficacy survey and science knowledge test before and after using Alien Rescue. Third, log data (over 30,000 lines in the form of time- and date-stamped entries) were a major data source, showing how each student used the tools to solve the problem within the PBL program. These data captured the just-in-time trajectory of students’ navigation patterns, which can aid in understanding students’ problem-solving processes. The log data were cleaned and used to calculate the frequency (how many times each tool is accessed) and duration (how long a student used each tool in minutes) of each student's tool use. In addition, students responded to the following two open-ended questions after they completed the program: (a) What have you learned from Alien Rescue? (b) Did you like or NOT like Alien Rescue? Why?
Both statistical and visualization techniques were used to investigate students’ activity logs to answer our five research questions. The students’ learning behavioral patterns were measured with the frequency and duration of using each tool and the counts of sending probes. Students were divided into three groups based on their post-self-efficacy test scores: low, with scores below the 25th percentile; medium, with scores between the 25th and 75th percentiles; and high, with scores above the 75th percentile. The same method was used for the post-science knowledge test scores. Kruskal–Wallis tests were used to examine possible differences among the high, medium, and low groups, followed by post hoc Mann–Whitney tests with Bonferroni-adjusted alpha (Bland & Altman, 1995) to control for inflated familywise Type I error in multiple comparisons.
Responses to the open-ended questions were analyzed with the constant comparative method (Creswell & Poth, 2018). Researchers independently coded 25% of student responses and generated initial codes and definitions; each response was reviewed by two researchers. The whole team then reviewed, modified, and consolidated the codes to create a codebook. Next, half of the remaining students’ responses were coded by two researchers using the codebook, and the other two researchers used the codebook to code the other half. Within each subteam, coding was completed until 100% agreement was reached.
To further investigate students’ self-efficacy and science knowledge in relation to students’ problem-solving, we examined the behavioral patterns of students who were initially in the low test score group but subsequently in the high test score group after they had participated in the PBL program (the low-high group). We also wanted to determine whether this group's behavioral patterns differed from those who remained in the low group after participating in the program (the low-low group). For this purpose, we conducted two four-stage navigation pathway analyses. One was between the low-low and low-high science knowledge test score groups; the other was between the low-low and low-high self-efficacy groups. The pathways were visualized, and percentages of sequential patterns at each stage were calculated to show the frequency of each pattern.
Results
Students’ Self-Efficacy and Their Performance in the PBL Program
Relationship Between Students’ Self-Efficacy and Their Behavioral Patterns
In our previous study (Liu et al., 2021), self-efficacy was a significant mediator between students’ attitudes toward science and their science knowledge. In the present study, we further investigated the role played by self-efficacy when students were engaged in problem-solving and how it related to students’ behavioral patterns in using different tools in the program. We used Kruskal–Wallis tests to investigate differences among self-efficacy groups based on their post-PBL program self-efficacy scores. Significant differences were found in the duration and frequency of tool use such as the Solar System Database and Spectra, as well as in duration for the Periodic Table (Table 2).
Findings from Kruskal–Wallis Tests Examining Overall Behavioral Patterns Grouped by Post-Self-Efficacy Groups.
*p < .05, **p < .02, ***p < .01.
We then conducted post-hoc Mann–Whitney U tests on the significant variables to detect differences among the three self-efficacy groups. Significant differences were found between high and low self-efficacy for Solar System Database duration (U = 5,095, p < .01) and frequency (U = 5,131.5, p < .01); for Periodic Table duration (U = 5,393, p < .01) and frequency (U = 5,473.5, p < .02); and for Spectra duration (U = 5,160.5, p <.01) and frequency (U = 5,099.5, p < .01) (Table 3). Between the high and medium self-efficacy groups, significant differences were found for Solar System Database duration (U = 7,855, p < .01) and frequency (U = 8,014, p < .01) and for Spectra duration (U = 8,154.5, p <.02). No statistically significant difference was found between the medium and low self-efficacy groups. That is, the high self-efficacy group used the Solar System Database significantly longer and more frequently than the other two groups. At the same time, they used Spectra and the Periodic Table significantly longer and more frequently than did the low self-efficacy group, and they used Spectra longer than the medium self-efficacy group.
Findings From Mann–Whitney U Tests Between the Behavioral Patterns Among the High, Medium, and Low Self-Efficacy Groups.
*p < .05, **p < .02, ***p < .01.
Designing and launching probes in this PBL program enable students to collect relevant data about planets so that they can test hypotheses about whether a planet is an appropriate place for an alien (Liu et al., 2021). Hence, sending probes is a problem-solving behavior. Students are asked to provide a rationale for why they plan to launch a probe and what data the probe is to gather. Their probe justifications were categorized into five types (i.e., random input, vague inquiry, vague claim, specific inquiry, and reasoning; Liu et al., 2022). Even though the Mann–Whitney U tests showed no significant differences among the three self-efficacy groups in the total numbers of probes launched, significant differences were found in the two high-level types of probe justification (specific inquiry and reasoning; see Table 4). The high self-efficacy group sent significantly more probes for reasoning than did the medium self-efficacy group (U = 5,601.5, p < .01) and the low self-efficacy group (U = 5,601.5, p < .01). They also launched significantly more probes for specific inquiry than did the medium self-efficacy group (U = 5,754.5, p < .02) and the low self-efficacy group (U = 5,754.5, p < .02). The medium self-efficacy group launched significantly more probes for specific inquiries than did the low self-efficacy group (U = 10,310, p < .01). There were no statistically significant differences across the three self-efficacy groups for the three lower levels of probe justification (vague inquiry, vague claim, and random input). In addition, although there was no significant difference among the numbers of solutions submitted by the three groups, the high self-efficacy group had a significantly greater number of successful solutions than their medium (U = 5,653.5, p < .01) and low (U = 5,653.5, p < .01) counterparts. Thus, we may conclude that launching more probes with reasoning and specific inquiry helped those in the high self-efficacy group as they collected specific information about the planets and found successful solutions to problems.
Findings From Mann–Whitney U Tests Between the Probe Justification, Solution, Probe Count of Students Grouped by High, Medium, and Low Self-Efficacy Levels.
*p < .05, **p < .02, ***p < .01.
Knowledge and Skills Learned by Students With Different Self-Efficacy Levels, and Their Perceptions
To further explore students’ self-efficacy and their science learning, we gathered qualitative data. Analysis of the students’ responses to the open-ended question “What have you learned from using Alien Rescue?” revealed that the most common theme across the three self-efficacy groups was “science concepts learning” (Figure 2). Students in all self-efficacy groups responded that science concepts were what they learned most after engaging in the program. For the high and medium self-efficacy groups, the second most frequently common theme was “problem-solving skills”—for example, “I learned about what instruments were used for different tasks,” and “I also learned how to identify a problem and solve it.” In contrast, the most common theme for the low self-efficacy students was “others” (13%)—for example, “aliens are cool,” which was not related to the conceptual knowledge and skills we expected them to acquire in the PBL program.
Findings for “what have you learned From using Alien Rescue.”
We also asked students whether they enjoyed using the PBL program to learn science and solve problems. In the high self-efficacy group, 99.99% of response codes showed a positive attitude toward Alien Rescue; in the medium self-efficacy group, 84.71% of codes revealed a positive attitude, 14.65% were negative, and 0.64% were neutral. In the low self-efficacy group, although the majority of codes (51.61%) showed a positive attitude, 34.41% were negative, and 13.98% were neutral. That is, the high self-efficacy students were more likely to have a favorable perception of their PBL learning. In contrast, the low self-efficacy students were more likely to have a negative attitude toward PBL learning.
Regarding reasons for liking or disliking the PBL program, the dominant positive reasons were similar across self-efficacy groups. “Fun” and “learning something new” were the top two positive reasons (Table 5). There were seven negative reasons why students did not like using Alien Rescue. The top two negative reasons in the medium self-efficacy group were “challenging” (30.43%) and “not fun” (30.43%); in the low self-efficacy group, “not fun” (38.24%) was the main reason, followed by “other negative” (26.47%) and “challenging” (17.65%). Only one theme in the high self-efficacy group was related to a negative reason, “challenging.”
Reasons for Liking or Disliking Alien Rescue by Self-Efficacy Groups.
Students’ Science Knowledge
Relationship Between Students’ Science Knowledge and Their Behavioral Patterns
The science knowledge test was given before and after students used the PBL program. We were interested in connecting this external measure with the students’ behavioral patterns during the program. All students were categorized into low, medium, and high knowledge groups, given their post-science knowledge test scores. Kruskal–Wallis tests revealed significant differences in frequency of tool usage for the Concept Database, Spectra, and Notebook. For tool usage duration, significant differences were found for the Concept Database, Periodic Table, Spectra, Notebook, and Alien Information Center (Table 6).
Findings From Kruskal–Wallis Tests Examining Overall Behavioral Patterns Grouped by Post-Science Knowledge Test Scores.
*p < .05.
Mann–Whitney U tests revealed that for tool frequency, significant differences were found between high and low knowledge groups for the Concept Database (U = 5,249.5, p < .01), Spectra (U = 5,281, p < .01), and Notebook (U = 5,105, p < .02) (Table 7). Moreover, the medium and low knowledge groups differed significantly for Concept Database (U = 10878, p < .01) and Notebook (U = 10700, p < .02). However, no significant differences were found between the high and medium science knowledge groups.
Findings From Mann–Whitney U Tests Between Students Grouped by High, Medium, and Low Groups of Post-PBL Science Knowledge Scores.
*p < .05, **p < .02, ***p < .01.
For tool duration, there were significant differences between the high and low science knowledge groups for Concept Database (U = 5,415.5, p < .01), Periodic Table (U = 4,917, p < .01), Spectra (U = 5,426, p < .01), and Notebook (U = 5,145, p < .01). Significant differences were also found for Concept Database (U = 11,136, p < .01) and Notebook (U = 10,684, p < .02) between the medium and low knowledge groups, as well as for Periodic Table (U = 14,098, p <.02) and Alien Information Center (U = 14,202, p < .02) between the high and medium knowledge groups.
Students in the low science knowledge group spent significantly more time using the Concept Database, Periodic Table, Spectra, and Notebook. They also used the Concept Database, Spectra, and Notebook more frequently in problem-solving than the high science knowledge group did. In this PBL program, the Concept Database provides instructional modules on ten scientific concepts to facilitate conceptual understanding. When students began using the PBL program, some were already familiar with some of those science concepts, whereas others may not have been, depending on when a school used the program as its earth science unit. The Concept Database is designed to support students who feel a need to learn or review science concepts, enabling just-in-time learning. The Periodic Table provides a periodic table of the elements, and the Spectra tool offers information to help students interpret spectra. All of these tools are designed to facilitate students’ science learning, depending on their prior knowledge, and to help students solve problems. Hence, longer and more frequent use of those tools by students with low science knowledge scores suggests that those students needed more support with regard to conceptual science knowledge within the PBL program. Unlike their high science knowledge peers, who may have had some conceptual science knowledge, the low science knowledge group depended more on the tools within the program to get the science information that they needed to solve the problem in the PBL program. Similarly, the low science knowledge group also spent more time and used the Concept Database more frequently than the medium science knowledge group.
Relationship Between the Changes in Students’ Science Knowledge and Their Behavioral Patterns
While students’overall science knowledge increased significantly from pre- to post-PBL, we were interested in investigating the behavioral patterns of the groups who had significant changes in their science knowledge from pre- to post-PBL: (a) low-high (n = 32), those students whose science knowledge improved from low on the pre-PBL test to high post-PBL; (b) low-low (n = 29), those whose science knowledge was low on both tests; and (c) high-high (n = 14), those whose science knowledge was high on both tests. Significant differences were discovered among these three groups of students in tool usage frequency and duration for the Solar System Database, Notebook, Probe Design Center, Mission Control Center, and Concept Database (Table 8). In addition, the number of probes launched, successfully launched probes, and different probe justifications (i.e., vague claim probes and random input probes) were also found to differ significantly among the three groups.
Findings of Kruskal–Wallis Tests Examining Overall Behavioral Patterns Grouped by Their Pre- and Post-PBL Science Knowledge Test Scores.
*p < .05, **p < .02, ***p < .01.
Additional Mann–Whitney U tests on those variables showed no significant differences between the low-low and low-high science knowledge students (Table 9). Between the low-low and high-high science knowledge groups, significant differences were found in tool usage frequencies for the Solar System Database (U = 321, p < .01), Notebook (U = 96, p < .01), Probe Design Center (U = 296.5, p < .01), and Mission Control Center (U = 300, p < .01). Between the low-low and high-high groups significant differences were also found in tool usage duration for the Solar System Database (U = 318, p < .01), Notebook (U = 101, p < .01), and Mission Control Center (U = 307, p < .01). Furthermore, significant differences were found between the low-high and high-high groups for all variables except the frequency of using the Solar System Database. In addition, the high-high group sent significantly more successful probes than the other two groups.
Findings From Mann–Whitney U Tests Between the Students as Grouped by Low-Low, Low-High, and High-High Levels of Pre- & Post-PBL Science Knowledge Scores.
*p < .05, **p < .02, ***p < .01.
The high-high science knowledge group used some tools (e.g., the Solar System Database) for a shorter period of time and less frequently but spent longer time and used the probe design-related tools more often than others. As a result, they launched greater numbers of probes and had more successful probes. This finding highlights the role of probe design in the high-high group's problem-solving.
In short, our results show that self-efficacy and science knowledge impacted students’ behavioral patterns in the PBL program. However, it is still unclear why the low-low and low-high groups had great differences in post-science knowledge test scores, even though no significant differences were found between these groups in tool usage. Test scores provide information about only one captured moment of performance (e.g., pre-PBL or post-PBL) but not the progress of students’ navigation. Therefore, we also investigated how students navigated their way with the different tools at each problem-solving stage within the program, using pathway analysis.
Pathway Analyses
Previous studies with similar student populations (e.g., Liu et al., 2016) have suggested four stages of problem-solving in this PBL program. In Stage 1 (Days 1–2), students’ primary purpose is to understand the problem; in Stage 2 (Days 3–5), students identify, gather, and organize information; in Stage 3 (Days 6–8), they integrate information and conduct hypothesis testing; and in Stage 4 (Days 9–10), they evaluate the process and submit their final solutions. Following this guideline, we divided students’ pathways within the program into four stages for further examination.
Comparison of Pathways Between the Low-Low and Low-High Self-Efficacy Groups
The patterns of learning behavior between students who improved in self-efficacy and those who did not are presented as pathways as low-high and low-low self-efficacy in Figure 3. The low-low self-efficacy students showed a greater variety of tool use patterns in the first and second stages. Noticeably, the low-low self-efficacy students began designing and launching probes earlier (in the first stage) and did not focus on probe design in the third stage. Conversely, the low-high self-efficacy students focused on the Alien Information Center and Notebook during the first and second stages. They also used other tools, such as Spectra and the Periodic Table, at a higher frequency. In particular, during the third stage, the low-high self-efficacy students tended to navigate between the Probe Design Center and Mission Control more often than the low-low group, indicating that they put more emphasis on testing their hypotheses in this stage. In the last stage, the low-high self-efficacy group visited the Communication Center more frequently to submit their solutions. They also reviewed their ideas by revisiting Notebook, the Alien Information Center, the Solar System Database, and sending probes. Overall, the tool usage pathway of the low-high self-efficacy group aligned more closely with the suggested stages of problem-solving than the tool usage pathway of the low-low self-efficacy group (Liu et al., 2016).
Comparison of tool usage in four stages by the low-low and the low-high self-efficacy groups. (a) 1st stage low-high self-efficacy group. (b) 1st stage low-low self-efficacy group. (c) 2nd stage low-high self-efficacy group. (d) 2nd stage low-low self-efficacy group. (e) 3rd stage low-high self-efficacy group. (f) 3rd stage low-low self-efficacy group. (g) 4th stage low-high self-efficacy group. (h) 4th stage low-low self-efficacy group.
Comparison of Pathways Between the Low-Low and Low-High Science Knowledge Groups
We also analyzed students’ pathways in the low-low and low-high science knowledge groups. As shown in Figure 4, in the first stage, the most frequent patterns of the low-high group were between Alien Information Center and Notebook, indicating that students in this phase spent most of the time understanding the information about the aliens and taking notes. Also, they explored different tools, such as the Solar System Database, Spectra, and the Periodic Table. Similarly, in this phase, the low-low science knowledge students frequently went back and forth between the Alien Information Center and Notebook. However, the low-low group visited the Mission Control Center and the Probe Design Center, which are the places to design and launch probes and are not expected at this early stage (Liu et al., 2022). In the second stage, the most frequent patterns in the low-high group was Alien Information Center ←→ Notebook, and frequent patterns of Notebook ←→ Concept Database, and Notebook ←→ Solar System Database were also observed. The Alien Information Center, Concept Database, and Solar System Database provide information about the aliens and the planets, share cognitive load, and support cognitive processes (Liu & Bera, 2005). The most frequent patterns in the low-low group was Solar System Database ←→ Notebook, and frequent patterns of Alien Information Center ←→ Notebook, Notebook ←→ Spectra, Notebook ←→ Mission Database were also observed. These patterns showed both groups were engaged in conducting research. In the third stage, both groups visited the Probe Design Center and launched probes to test their hypotheses. The frequent patterns in this stage were Notebook ←→ Solar System Database and Alien Information Center ←→ Notebook for both groups, indicating that students revisited the Alien Information Center and Solar System Database to check probes’ results. Moreover, the students in the low-low group started to submit their final solutions by visiting the Communication Center. In the last phase, both groups visited the Notebook ←→ Solar System Database the most and visited the Communication Center to submit final solutions. To summarize, pathway analyses revealed that the low-high science knowledge group spent more time collecting relevant information before designing probes. In contrast, the low-low group started exploring probe design much earlier, without collecting enough information. Compared to their low-high peers, they rushed to submit their final solutions during the third stage.
Comparison of tool usage in four stages by the low-low and the low-high science knowledge groups. (a) 1st stage low-high science knowledge group. (b) 1st stage low-low science knowledge group. (c) 2nd stage low-high science knowledge group. (d) 2nd stage low-low science knowledge group. (e) 3rd stage low-high science knowledge group. (f) 3rd stage low-low science knowledge group. (g) 4th stage low-high science knowledge group. (h) 4th stage low-low science knowledge group.
Discussion and Implications
The Role of Self-Efficacy
In this study, we have explored the role of self-efficacy in the PBL program by examining the relationship between students’ self-efficacy and their patterns of learning behaviors. We found that the high self-efficacy group used the Solar System Database significantly more frequently and used the Solar System Database and Spectra significantly longer than the medium and low self-efficacy groups. In this PBL program, the Solar System Database provides information on selected planets and moons, and Spectra's function is to help students interpret spectra related to aliens. These tools are key in identifying, interpreting, and gathering information. Thus, the high self-efficacy students were more aware of what learning tools to use to solve problems. Fadillah et al. (2021) have pointed out that students with strong self-efficacy can better manage and control their learning as they solve problems; they understand what information and learning strategies they need. In addition, the high self-efficacy group sent significantly more probes with reasoning and specific inquiries than the other two groups. The high self-efficacy group also submitted a significantly greater number of successful solutions than their medium and low counterparts, despite no significant difference in the number of solutions submitted by the three groups. Sending more probes with reasoning and specific inquiries showed that high self-efficacy students understood the information they collected and the goal they were expected to achieve, so they were more likely to come up with the correct solutions. With more reasoning and specific inquiry probes, the high self-efficacy group could get more specific information, which yielded a significantly greater number of successful solutions. According to Alt et al. (2022), students with high self-efficacy often strive to succeed and understand the learning material in order to reach specific learning goals. We found that students’ levels of self-efficacy were positively correlated with their success rates in solving problems, which aligns with previous literature indicating self-efficacy is an important mediator of students’ performance and learning behaviors (Amini et al., 2019; Fadillah et al., 2021).
Our pathway analysis demonstrated different patterns of learning behavior between the low-low and low-high self-efficacy groups. The low-low group showed greater variety in tool use patterns in the first and second stages of problem-solving and began designing and launching probes in the first stage. However, the low-high self-efficacy group initially focused on the Alien Database, Notebook, and Solar System Database in the first stage and sent probes in the third stage. Thus, the low-low self-efficacy group began sending probes to evaluate their hypotheses too early, without spending enough time and effort in research to develop hypotheses. As Fitriani et al. (2020) found that students’ self-efficacy improved during a process of investigation in PBL, our findings show that the low-high self-efficacy group may have improved their self-efficacy by following PBL procedures. In a previous study with Alien Rescue (Liu & Bera, 2005), those sixth-graders who followed the identification and understanding of the problem and then collected related information to support their hypotheses before testing were more likely to solve the problem successfully. The present study confirms that the low-high students also enhanced their self-efficacy by following the four stages of problem-solving.
As shown from the qualitative data, the high and medium self-efficacy groups reported more often that they had learned about problem-solving skills than the low self-efficacy students. “Problem-solving skills” was the second most frequent theme in the high (13.42%) and medium (20.84%) self-efficacy groups. In contrast, only 6% of the low self-efficacy students reported that they had learned problem-solving skills in the program. Because self-efficacy represents one's belief in successfully completing tasks and achieving objectives (Bandura, 1988), it is not surprising that the high self-efficacy students were more confident and positive in their learning. Hong et al. (2016) has pointed out that self-efficacy is linked with students’ interests in learning and it plays an essential role in learning tasks that require more independence and initiative. The high and medium self-efficacy groups were more likely to believe that they were learning problem-solving skills, perhaps because self-efficacy drove them to be more motivated to solve problems. Thus, they were more willing to practice, leading to greater problem-solving skills than their peers with low self-efficacy. Our study also shows that the greater the self-efficacy, the more positive the perception of the PBL program. The students in the medium and low self-efficacy groups felt that the program was difficult. Previous studies have also found that self-efficacy is associated with students’ perceptions of task difficulty (e.g., Lee & List, 2021) and their perceived efficacy in learning (e.g., Jia et al., 2014). Students’ different levels of self-efficacy may explain their different attitudes toward the PBL program. For low self-efficacy students, lower levels of perceived ability to successfully complete tasks make their perceptions of a given task challenging (Li et al., 2007), whereas the high self-efficacy students were more confident in their ability, thus resulting in a different attitude toward PBL.
Our findings highlighted the importance of self-efficacy in PBL regarding learning behavior, performance, and perceptions, which has profound implications for teaching. For teachers, it is important to increase students’ self-efficacy when implementing PBL. It is beneficial for teachers to monitor their students’ development in problem-solving skills as part of PBL, especially among students with low self-efficacy and difficulty in managing their learning. In addition, the results revealed the importance of following the PBL procedures in improving students’ self-efficacy. Teachers should spend more time guiding students to prepare for PBL at the beginning of the PBL process to help lead them toward success. Encouraging students to take on challenges and making them feel confident and positive can also help develop their self-efficacy. While we only compared the learning behaviors of students with different levels of self-efficacy in this study, given the positive findings a follow-up study should examine if better learning behaviors would lead to improvement in students’ self-efficacy.
The Role of Students’ Science Knowledge
With respect to the relationship between students’ science knowledge and their behavioral patterns within PBL, the students with low science knowledge spent significantly more time and used some tools more frequently (e.g., the Concept Database and Spectra) to solve problems than those with high science knowledge. The low science knowledge group was not as familiar with some key science concepts as their higher science knowledge peers, so they needed to use those tools more often to gather relevant information. The longer and more frequent use of the tools suggests that the low science knowledge group lacked sufficient prior knowledge and needed additional support and content scaffolding to solve the problem. This finding also has implications for teachers’ PBL implementation. Teachers may need to make an extra effort with students who lack sufficient knowledge of content to solve the problem, and teachers should monitor their progress more frequently in order to provide support (Kim et al., 2018). Providing teachers with adaptive dashboard systems that show students’ progress during PBL can be helpful (Tissenbaum & Slotta, 2019; van Leeuwen et al., 2022). We are pursuing this effort in our research.
Our pathway analyses showed that the low-low science knowledge group was not as efficient in using crucial tools in the initial phase of problem-solving in comparison with their low-high peers. The low-low science knowledge students wasted time and effort using tools that would be necessary only in the later phases of problem-solving. This result supports the findings of Li and Stylianides (2018) and Park and Ertmer (2008), who have argued for the importance of scaffolding to guide students through PBL. Providing just-in-time feedback to the low-low students might have led them to develop more knowledge and problem-solving strategies to move through the problem-solving phases in a timely manner.
The findings for the high-high group also have implications. These students spent more time with probe design-related tools (i.e., the Mission Control Center and Probe Design Center) and used them more frequently than the other two groups. However, they spent significantly less time on the Solar System Database, which provides conceptual scientific facts. This suggests the importance of testing hypotheses in problem-solving, which aligns with Spires et al.'s (2011) assertion of a positive relationship between the effects of middle-grade students’ hypothesis-testing strategies and learning outcomes in PBL. The high-high group's greater time spent in probe design, and their greater number of trials in launching probes understandably increased their chances of success. The task of designing and launching probes to test their hypotheses played a crucial role. Furthermore, the high-high group's lesser utilization of the Solar System Database and their focused utilization of probe-related tools imply that these students already had adequate science knowledge to proceed to the next phase in problem-solving. Students who lacked science knowledge spent time with tools that taught them the science knowledge needed to proceed, resulting in less time to finish problem-solving. On the other hand, students with sufficient science knowledge before PBL had more time to focus on hypotheses testing by saving time and effort in the initial problem-solving phase.
In conclusion, through detailed analyses of students’ learning behavior patterns using a LA approach, we have shown that self-efficacy plays an important role in students’ problem-solving. Science knowledge also plays a role in how students determine when and what tools to use to facilitate their problem-solving. Such findings offer insights for researchers in PBL and LA who are interested in designing technology-enriched tools to support K-12 teachers in classroom implementation.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
