Understanding Middle School Students’ Self-Efficacy and Performance in a Technology-Enriched Problem-Based Learning Program: A Learning Analytics Approach

Abstract

Teachers interested in implementing problem-based learning (PBL) in classrooms encounter many challenges. Research suggests learning analytics (LA) can provide insights into students’ problem-solving process and offer teachers valuable information about when to provide necessary scaffolding. Yet, LA research in K-12 is lacking. There is also a need to explore how K-12 students’ self-efficacy may impact their problem-solving performance and learning behaviors. This study examined 418 middle school students’ self-efficacy, their learning performance and behavioral patterns after using a multimedia-enriched PBL program, and explore the relationships among these factors. Using a mixed-methods design, this study showed students’ different levels of self-efficacy were correlated with their success rates in solving problems. Students with higher self-efficacy used more appropriate tools at the appropriate times than students with lower self-efficacy. Students’ different levels of content knowledge also play a role in how students determine when and what tools to use to facilitate their problem-solving.

Keywords

multimedia-enriched problem-based learning learning analytics self-efficacy problem-solving K-12 mixed-methods

Research has demonstrated the benefits of problem-based learning (PBL) for K-12 students, yet it has also indicated that teachers encounter challenges when they implement PBL in their classrooms. Recent research suggests that techniques of learning analytics (LA) can provide insight into students’ learning processes in open-ended environments such as PBL and offer teachers valuable information about when to provide necessary scaffolding. However, research with LA in K-12 is lacking. Positive correlations have been found between students’ self-efficacy (SE) and performance, but there is a need to explore how K-12 students’ self-efficacy may impact their problem-solving performance and learning behaviors. In this study, we use LA to examine middle school students’ self-efficacy and performance in a PBL context, and we explore relationships between these students’ self-efficacy and their learning performance and behaviors.

Literature Review

PBL in K-12 Education

PBL is a student-centered approach in which students learn by solving problems and reflecting on their learning experiences (Barrows & Tamblyn, 1980). It shares similarities with project-based learning in that both are student-centered and aim for promoting higher-order thinking skills (Savery, 2006). However, in project-based learning learners are usually given specific requirements to create an artifact or product as the outcome (Thomas, 2000), PBL, the focus of this study, emphasizes on the problem-solving process and students developing solutions to an ill-defined problem. PBL has been employed at different grade levels and in various subjects, including math, science, social studies, and literacy (Merritt et al., 2017), and a number of studies have shown that PBL positively impacts students’ academic achievement. Günter and Alpat (2017) found that students using PBL had higher academic achievement than those in traditional teaching. Spires et al. (2011) indicated that problem-solving tasks helped middle school students gain knowledge of science content, and their effective selection of hypotheses within PBL was a key to successful learning. PBL encourages learners’ control over the learning process, such that students can freely utilize their prior knowledge and experience within the PBL environment (Kubiatko & Vaculová, 2011). However, the influence of prior knowledge on students’ final learning achievement needs more research. Some have argued that knowledge gained in PBL does not necessarily result from PBL itself, because other factors may be involved: teachers’ manners of implementation, the composition of student groups, and technological infrastructure (Kokotsaki et al., 2016).

Research has also indicated that PBL has the potential to improve students’ higher-order thinking skills, attitudes about learning, and self-efficacy. Lapuz and Fulgencio (2020) found PBL effective in enhancing critical thinking among middle school students. Valdez and Bungihan (2019) found that PBL improved high school students’ problem-solving in chemistry. There is also empirical evidence that K-12 students’ attitudes toward science can be improved with PBL (e.g., Akınoğlu & Tandoğan, 2007), and PBL has increased students’ self-efficacy significantly (e.g., Duman & Özçelik, 2018).

Nevertheless, despite the many benefits of adopting PBL in K-12 education, some teachers and students consider its implementation in the classroom to be a challenge (e.g., Wijnen et al., 2017). In PBL, students are intended to be responsible for their own learning, play a more active role, and think critically about their learning processes (Bereiter & Scardamalia, 2000). Some researchers have pointed out that this high demand makes young K-12 students struggle with tasks, which can negatively influence their learning experiences (Belland et al., 2017). The depth and breadth of the required learning in PBL also create obstacles for young learners (Botelho et al., 2018). Al-Kloub et al. (2014) have noted that while students are in PBL, they can experience uncertainty and confusion, a large workload, and a lack of confidence. Some researchers have also found that PBL can be difficult for some students, especially those who lack confidence in analyzing data to solve problems (Nijhuis et al., 2005). Yuan et al. (2011) showed that students felt that PBL involved a lack of guidance and was time consuming, making them stressed and frustrated.

In implementing PBL, teachers face challenges as well. The literature on teachers’ perceptions of adopting PBL suggests that they may struggle with allowing students the freedom to explore content (Maxwell et al., 2005), with assessing the open-ended nature of the PBL environment (Hung et al., 2019), and with tracking students’ behavior as they progress in problem-solving (Chen et al., 2021). In PBL, teachers must provide students with optimal scaffolding (Kim et al., 2018), move students through the stages of problem-solving, and monitor group processes (Hmelo-Silver, 2004). Thus, teachers need to monitor the students’ learning throughout. But some teachers are unsure about monitoring students’ progress in learning: they find it difficult to identify relevant issues and problems (Ng et al., 2014), and they need assistance (Chen et al., 2021). For example, students in Tawfik et al. (2021) stated that they initially received too much guidance or unnecessary support in the middle of problem-solving, which redirected them away from their own directions. This difficult balance highlights a challenge to teachers’ provision of appropriate scaffolding at the right times (Chen et al., 2016). We believe that recent advancements in LA, with the analysis of the data produced during learning, can offer teachers insightful perspectives into students’ learning processes (El Alfy et al., 2019), because LA can capture the trajectory of each student's problem-solving patterns (Foster & Siddle, 2020).

LA and Students’ Learning Performance

LA consists of collecting, analyzing, and reporting data to better understand an environment where learning takes place and optimize it (Siemens & Baker, 2012). Understanding factors that influence students’ academic performance has become a common topic in LA, owing to the availability of abundant data for students’ learning within specific contexts (Tempelaar et al., 2015). Researchers have investigated learning environments, habits, skills, and personal characteristics in relation to students’ learning performance in order to enhance their learning outcomes (Credé & Kuncel, 2008).

In LA, the characteristics of learners are important for potentially optimizing the learning process to support students’ learning performance (Avella et al., 2016). LA can promote educators’ understanding of students’ learning experience with technology such as games or other online tools, which enrich students’ learning experience (e.g., Hung & Zhang, 2012). LA can also provide valuable insights into students’ learning behaviors and thus help researchers and educators provide personalized learning solutions tailored to students’ learning processes (e.g., DiCerbo, 2014). Because students’ characteristics (e.g., self-efficacy) vary in scale, it is meaningful to examine how differences in levels of personal characteristics can impact students’ learning performance.

LA has been applied less in K-12 than in higher education (Kovanovic et al., 2021). There is a need to apply LA to gain insights for optimal learning, given K-12 students’ various characteristics and behavioral patterns. Kostagiolas et al. (2019), for example, has pointed out that the literature on the relationship between students’ self-efficacy and performance from the perspective of LA is limited. Especially in K-12, there is a need for the use of LA to support students’ self-efficacy to improve students’ learning outcomes.

Self-Efficacy Associated With Learning Outcomes, Learning Behaviors, and Perceptions

Self-efficacy, as defined by Bandura (1994), pertains to an individual's perception of their ability to achieve a desired outcome. In contrast to growth mindset, which posits that an individual's capabilities can be developed through persistent effort (Claro et al., 2016), self-efficacy is specific to particular domains and linked to their prior learning experiences in that particular area (Bai et al., 2019). The self-efficacy of students may differ across domains due to variations in their success, mastery experiences, and setbacks in each task (Pajares, 2006). According to current literature, self-efficacy is a significant predictor of students’ performance (e.g., Tan et al., 2021; Yadav et al., 2021), and self-efficacy positively affects students’ learning outcomes and performance (Amini et al., 2019; Yadav et al., 2021). Students with higher self-efficacy have better learning outcomes than those with lower self-efficacy (Amini et al., 2019). These effects have been examined in various disciplines. Tan et al. (2021), for example, found significant positive correlations between conceptions of learning science and self-efficacy for learning science in 431 undergraduate students. Yadav et al. (2021) found that self-efficacy was a major predictor of learning outcomes among university students in computer science.

Researchers have also examined the effects of self-efficacy on learning behaviors. Self-efficacy can be an important mediator for students’ problem-solving (Fadillah et al., 2021; Fitriani et al., 2020; Hwang & Oh, 2021). Students with high self-efficacy are more willing to tackle challenging problems, whereas students with low self-efficacy are more likely to become frustrated in solving problems and simply give up (e.g., Demirören et al., 2016). In problem-solving and self-directed learning among nursing students, Hwang and Oh (2021) found that self-efficacy played a mediating role: students with higher self-efficacy had higher problem-solving ability and higher self-directed learning. In examining junior high school students’ metacognition in solving math problems, Fadillah et al. (2021) found that students with higher self-efficacy were more effective in planning and monitoring their learning.

Research also shows positive relationships between self-efficacy and students’ attitudes toward and perceptions of learning. Hong et al. (2016), for example, found that students with high self-efficacy were more interested in their current learning and had higher learning satisfaction. Lee and List (2021) found that self-efficacy was a significant predictor of students’ perceptions of task difficulty in the relationship between students’ self-efficacy and their perceptions of their training: students with a strong sense of competency considered the learning task easier to complete. Similarly, Jia et al. (2014) examined whether self-efficacy influenced learners’ perceptions of the effectiveness of learning activities and found that individuals with higher self-efficacy were more likely to describe learning activities as enjoyable. However, although many researchers have examined the importance of self-efficacy, we lack research exploring how K-12 students’ varying levels of self-efficacy may impact their performance in solving problems and their learning behaviors.

In this study, we build on our previous research, in which we investigated the impact of PBL on performance and explored the relationships among self-efficacy, attitudes toward science, and learning outcomes in middle school students in a multimedia-enriched PBL program (Liu et al., 2021). The results showed that students’ science knowledge was increased significantly by the PBL program and that students’ self-efficacy was a significant predictor of their science knowledge. Self-efficacy mediated between students’ attitudes and science knowledge. However, we did not examine the students’ behavioral patterns in problem-solving with PBL, and we did not thoroughly evaluate the role of varying levels of students’ self-efficacy and science knowledge in problem-solving. This present study intends to further investigate the relationships among these factors using LA.

Research Questions

The goal of this study is to examine the role of students’ self-efficacy and science knowledge in problem-solving. To examine the role of self-efficacy, we ask the following research questions:

What is the relationship between students’ self-efficacy and their behavioral patterns?

Are there any differences in students’ science knowledge and problem-solving skills corresponding to their different self-efficacy levels, and what are students’ perceptions of the PBL program?

To examine the role of students’ science knowledge, we ask the following research questions:

What is the relationship between students’ science knowledge and their behavioral patterns?

What is the relationship between the changes in students’ science knowledge and their behavioral patterns?

In addition, we ask the following research question:

What do pathway analyses reveal regarding students’ behavioral patterns during the different stages of problem-solving, given different levels of self-efficacy and science knowledge?

Method

Research Context

A total of 418 sixth graders (male = 211, female = 207) from three middle schools in the western and southwestern United States participated in this study. Alien Rescue, a 3D immersive multimedia-enriched PBL program, was used to examine problem-solving in PBL. In this program, sixth graders learn about space science by rescuing six different alien species and finding new suitable homes for them. Several multimedia tools (see Table 1 and Figure 1) are built into the program to promote students’ problem-solving with instructional scaffolding. These tools enable students to engage in cognitive activities to solve problems, and they are designed to support students’ scientific inquiry and help them develop self-efficacy (Liu et al., 2021).

Figure 1.

Screenshots of technology tools in Alien Rescue to support scientific inquiry. (a) Alien Information Center. (b) Solar System Database. (c) Probe Design Center. (d) Notebook. (e) Solution Form. (f) Mission Control Center.

Table 1.

Technology-Enriched Tools in the Problem-Based Learning Program.

Tool name	Tool functions
Available at tool bar
Solar System Database	Provides information on selected planets and moons within our solar system. Data are intentionally incomplete to support the ill-structured nature of the problem-solving environment and foster the need for hypothesis testing.
Missions Database	Provides information on past NASA missions, including detailed descriptions of probes used on these missions.
Concept Database	Provides instructional modules on 10 selected scientific concepts designed to facilitate conceptual understanding. This tool is for real-time learning. Students access this tool when they encounter a science concept that they are unfamiliar with but that is needed in Alien Rescue.
Periodic Table	Provides a periodic table of the elements.
Spectra	Provides information to help students interpret spectra found in the Alien Information Center.
Notebook	A notebook to store students’ notes about their research findings. This tool has a notes-comparison feature to facilitate comparing alien needs and planet requirements.
Available using 4 arrows
Alien Information Center	Provides information via 3D imagery and text, on the aliens’ home planet, their journey, species characteristics, and habitat requirements.
Communication Center	Serves as a home for the program and also houses the message tool. Students receive welcome messages when they log into the program as well as submit their final solutions using the message tool.
Probe Design Center	Provides information on scientific equipment used in past and future probe missions. Students construct probes by deciding on probe type, communication, power source, and instruments.
Mission Control Center	Displays the data collected by the probes. Students analyze and interpret this data in order to develop a solution. Equipment malfunction can occur, and poor planning may lead to mission failure and budget waste.

The PBL program includes several problem-solving activities. Students research the features of each alien and planet to determine which planets are suitable for each alien species. Databases (e.g., Alien Information Center, Solar System Database) are designed to provide information about the aliens and planets. To test their hypotheses about the aliens’ suitability for living on specific planets, students design and launch probes in the Probe Design Center, then collect data from the probes in the Mission Control Center. To perform these problem-solving activities, students must plan, gather relevant information, and determine how to use multimedia tools effectively.

Data Sources and Analysis

This is a mixed-methods study. First, eight items from the Motivated Strategies for Learning Questionnaire (Pintrich et al., 1993) were used to measure students’ self-efficacy (Cronbach's alpha = 0.92 for this sample). Second, a 20-item multiple-choice science knowledge test was used to measure students’ understanding of key science concepts introduced in the PBL program. This test was used previously to study the same program with similar age groups (Liu et al., 2019, 2021) with Cronbach's alpha = 0.84 for the present sample. The students completed the self-efficacy survey and science knowledge test before and after using Alien Rescue. Third, log data (over 30,000 lines in the form of time- and date-stamped entries) were a major data source, showing how each student used the tools to solve the problem within the PBL program. These data captured the just-in-time trajectory of students’ navigation patterns, which can aid in understanding students’ problem-solving processes. The log data were cleaned and used to calculate the frequency (how many times each tool is accessed) and duration (how long a student used each tool in minutes) of each student's tool use. In addition, students responded to the following two open-ended questions after they completed the program: (a) What have you learned from Alien Rescue? (b) Did you like or NOT like Alien Rescue? Why?

Both statistical and visualization techniques were used to investigate students’ activity logs to answer our five research questions. The students’ learning behavioral patterns were measured with the frequency and duration of using each tool and the counts of sending probes. Students were divided into three groups based on their post-self-efficacy test scores: low, with scores below the 25th percentile; medium, with scores between the 25th and 75th percentiles; and high, with scores above the 75th percentile. The same method was used for the post-science knowledge test scores. Kruskal–Wallis tests were used to examine possible differences among the high, medium, and low groups, followed by post hoc Mann–Whitney tests with Bonferroni-adjusted alpha (Bland & Altman, 1995) to control for inflated familywise Type I error in multiple comparisons.

Responses to the open-ended questions were analyzed with the constant comparative method (Creswell & Poth, 2018). Researchers independently coded 25% of student responses and generated initial codes and definitions; each response was reviewed by two researchers. The whole team then reviewed, modified, and consolidated the codes to create a codebook. Next, half of the remaining students’ responses were coded by two researchers using the codebook, and the other two researchers used the codebook to code the other half. Within each subteam, coding was completed until 100% agreement was reached.

To further investigate students’ self-efficacy and science knowledge in relation to students’ problem-solving, we examined the behavioral patterns of students who were initially in the low test score group but subsequently in the high test score group after they had participated in the PBL program (the low-high group). We also wanted to determine whether this group's behavioral patterns differed from those who remained in the low group after participating in the program (the low-low group). For this purpose, we conducted two four-stage navigation pathway analyses. One was between the low-low and low-high science knowledge test score groups; the other was between the low-low and low-high self-efficacy groups. The pathways were visualized, and percentages of sequential patterns at each stage were calculated to show the frequency of each pattern.

Results

Students’ Self-Efficacy and Their Performance in the PBL Program

Relationship Between Students’ Self-Efficacy and Their Behavioral Patterns

In our previous study (Liu et al., 2021), self-efficacy was a significant mediator between students’ attitudes toward science and their science knowledge. In the present study, we further investigated the role played by self-efficacy when students were engaged in problem-solving and how it related to students’ behavioral patterns in using different tools in the program. We used Kruskal–Wallis tests to investigate differences among self-efficacy groups based on their post-PBL program self-efficacy scores. Significant differences were found in the duration and frequency of tool use such as the Solar System Database and Spectra, as well as in duration for the Periodic Table (Table 2).

Table 2.

Findings from Kruskal–Wallis Tests Examining Overall Behavioral Patterns Grouped by Post-Self-Efficacy Groups.

Variable	Kruskal–Wallis test
Tool frequency
Solar System Database	9.2253***
Mission Database	1.5865
Concept Database	2.4623
Periodic Table	5.1597*
Spectra	8.7244*
Notebook	1.8202
Alien Information Center	0.4009
Probe Design Center	0.8146
Mission Control Center	2.5046
Tool duration
Solar System Database	10.007***
Mission Database	1.2692
Concept Database	0.8559
Periodic Table	6.0384*
Spectra	8.3808*
Notebook	0.7404
Alien Information Center	3.7224
Probe Design Center	0.4070
Mission Control Center	1.6023

*p < .05, **p < .02, ***p < .01.

We then conducted post-hoc Mann–Whitney U tests on the significant variables to detect differences among the three self-efficacy groups. Significant differences were found between high and low self-efficacy for Solar System Database duration (U = 5,095, p < .01) and frequency (U = 5,131.5, p < .01); for Periodic Table duration (U = 5,393, p < .01) and frequency (U = 5,473.5, p < .02); and for Spectra duration (U = 5,160.5, p <.01) and frequency (U = 5,099.5, p < .01) (Table 3). Between the high and medium self-efficacy groups, significant differences were found for Solar System Database duration (U = 7,855, p < .01) and frequency (U = 8,014, p < .01) and for Spectra duration (U = 8,154.5, p <.02). No statistically significant difference was found between the medium and low self-efficacy groups. That is, the high self-efficacy group used the Solar System Database significantly longer and more frequently than the other two groups. At the same time, they used Spectra and the Periodic Table significantly longer and more frequently than did the low self-efficacy group, and they used Spectra longer than the medium self-efficacy group.

Table 3.

Findings From Mann–Whitney U Tests Between the Behavioral Patterns Among the High, Medium, and Low Self-Efficacy Groups.

	High vs. Low		High vs. Medium		Medium vs. Low
	Median		Median		Median
	High	Low	High	Medium	Medium	Low
Variable	(n = 103)	(n = 128)	(n = 103)	(n = 187)	(n = 187)	(n = 128)
Tool frequency
Solar SystemDatabase	225***	148.5	225***	53.417	53.42	148.5
PeriodicTable	8**	6	8	8	8	6
Spectra	14***	8	14	10	10	8
Tool duration
Solar SystemDatabase	74.17***	53.53	74.17***	180	180	53.53
PeriodicTable	2.08***	0.81	2.08	1.63	1.63	0.81
Spectra	4.92***	2.78	4.92**	4.07	4.07	2.78

*p < .05, **p < .02, ***p < .01.

Designing and launching probes in this PBL program enable students to collect relevant data about planets so that they can test hypotheses about whether a planet is an appropriate place for an alien (Liu et al., 2021). Hence, sending probes is a problem-solving behavior. Students are asked to provide a rationale for why they plan to launch a probe and what data the probe is to gather. Their probe justifications were categorized into five types (i.e., random input, vague inquiry, vague claim, specific inquiry, and reasoning; Liu et al., 2022). Even though the Mann–Whitney U tests showed no significant differences among the three self-efficacy groups in the total numbers of probes launched, significant differences were found in the two high-level types of probe justification (specific inquiry and reasoning; see Table 4). The high self-efficacy group sent significantly more probes for reasoning than did the medium self-efficacy group (U = 5,601.5, p < .01) and the low self-efficacy group (U = 5,601.5, p < .01). They also launched significantly more probes for specific inquiry than did the medium self-efficacy group (U = 5,754.5, p < .02) and the low self-efficacy group (U = 5,754.5, p < .02). The medium self-efficacy group launched significantly more probes for specific inquiries than did the low self-efficacy group (U = 10,310, p < .01). There were no statistically significant differences across the three self-efficacy groups for the three lower levels of probe justification (vague inquiry, vague claim, and random input). In addition, although there was no significant difference among the numbers of solutions submitted by the three groups, the high self-efficacy group had a significantly greater number of successful solutions than their medium (U = 5,653.5, p < .01) and low (U = 5,653.5, p < .01) counterparts. Thus, we may conclude that launching more probes with reasoning and specific inquiry helped those in the high self-efficacy group as they collected specific information about the planets and found successful solutions to problems.

Table 4.

Findings From Mann–Whitney U Tests Between the Probe Justification, Solution, Probe Count of Students Grouped by High, Medium, and Low Self-Efficacy Levels.

	High vs. Low		High vs. Medium		Medium vs. Low
	Median		Median		Median
	High	Low	High	Medium	Medium	Low
Variable (count of)	(n = 103)	(n = 128)	(n = 103)	(n = 187)	(n = 187)	(n = 128)
Reasoning	0.86***	0.54	0.86***	0.49	0.49	0.54
Specific inquiry	1.11**	0.01	1.11**	0.73	0.73***	0.01
Vague inquiry	0.5	0.625	0.5	0.54	0.54	0.625
Vague claim	0.82	0.91	0.82	0.97	0.97	0.91
Random input	0.83	1.14	0.83	1.5	1.5	1.14
Solution	1.61	0.69	1.61	0.91	0.91	0.69
Successful solution	1.17	0.41	1.17***	0.56	0.56	0.41
Probe	4.02	3.95	4.02	4.6	4.6	3.95
Successful probe	2.85	2.93	2.85	3.2	3.2	2.93

*p < .05, **p < .02, ***p < .01.

Knowledge and Skills Learned by Students With Different Self-Efficacy Levels, and Their Perceptions

To further explore students’ self-efficacy and their science learning, we gathered qualitative data. Analysis of the students’ responses to the open-ended question “What have you learned from using Alien Rescue?” revealed that the most common theme across the three self-efficacy groups was “science concepts learning” (Figure 2). Students in all self-efficacy groups responded that science concepts were what they learned most after engaging in the program. For the high and medium self-efficacy groups, the second most frequently common theme was “problem-solving skills”—for example, “I learned about what instruments were used for different tasks,” and “I also learned how to identify a problem and solve it.” In contrast, the most common theme for the low self-efficacy students was “others” (13%)—for example, “aliens are cool,” which was not related to the conceptual knowledge and skills we expected them to acquire in the PBL program.

Figure 2.

Findings for “what have you learned From using Alien Rescue.”

We also asked students whether they enjoyed using the PBL program to learn science and solve problems. In the high self-efficacy group, 99.99% of response codes showed a positive attitude toward Alien Rescue; in the medium self-efficacy group, 84.71% of codes revealed a positive attitude, 14.65% were negative, and 0.64% were neutral. In the low self-efficacy group, although the majority of codes (51.61%) showed a positive attitude, 34.41% were negative, and 13.98% were neutral. That is, the high self-efficacy students were more likely to have a favorable perception of their PBL learning. In contrast, the low self-efficacy students were more likely to have a negative attitude toward PBL learning.

Regarding reasons for liking or disliking the PBL program, the dominant positive reasons were similar across self-efficacy groups. “Fun” and “learning something new” were the top two positive reasons (Table 5). There were seven negative reasons why students did not like using Alien Rescue. The top two negative reasons in the medium self-efficacy group were “challenging” (30.43%) and “not fun” (30.43%); in the low self-efficacy group, “not fun” (38.24%) was the main reason, followed by “other negative” (26.47%) and “challenging” (17.65%). Only one theme in the high self-efficacy group was related to a negative reason, “challenging.”

Table 5.

Reasons for Liking or Disliking Alien Rescue by Self-Efficacy Groups.

	Self-efficacy levels
	High	Medium	Low
Positive reasons (n codes)	102	167	76
Authenticity	7.84%	8.38%	6.58%
Can communicate with peers	0.00%	0.60%	0.00%
Can work collaboratively	6.86%	3.59%	7.89%
Fun	31.37%	28.14%	28.95%
Game-based	1.96%	3.59%	3.95%
General positive	8.82%	5.39%	9.21%
Imaginary/futuristic/sci-fiction Genre	1.96%	0.60%	0.00%
Innovative learning environment	9.80%	7.19%	2.63%
Learning something new	23.53%	28.14%	28.95%
Other (positive)	2.94%	4.79%	6.58%
Self-directed learning	1.96%	4.19%	0.00%
Technology-enriched	2.94%	5.39%	5.26%
Negative reasons (n codes)	1	23	34
Challenging	100%	30.43%	17.65%
Not fun		30.43%	38.24%
Inconvenient environment design		4.35%	0.00%
Too much work		17.39%	2.94%
Not authentic		0.00%	8.82%
Technical issues		0.00%	5.88%
Others negative		17.39%	26.47%

Students’ Science Knowledge

Relationship Between Students’ Science Knowledge and Their Behavioral Patterns

The science knowledge test was given before and after students used the PBL program. We were interested in connecting this external measure with the students’ behavioral patterns during the program. All students were categorized into low, medium, and high knowledge groups, given their post-science knowledge test scores. Kruskal–Wallis tests revealed significant differences in frequency of tool usage for the Concept Database, Spectra, and Notebook. For tool usage duration, significant differences were found for the Concept Database, Periodic Table, Spectra, Notebook, and Alien Information Center (Table 6).

Table 6.

Findings From Kruskal–Wallis Tests Examining Overall Behavioral Patterns Grouped by Post-Science Knowledge Test Scores.

Variable	Kruskal–Wallis test
Tool frequency
Solar System Database	2.9826
Mission Database	0.9229
Concept Database	7.8226*
Periodic Table	3.3164
Spectra	7.0318*
Notebook	5.8617*
Alien Information Center	3.5829
Probe Design Center	0.1830
Mission Control Center	0.6835
Tool duration
Solar System Database	1.1804
Mission Database	1.2361
Concept Database	10.705*
Periodic Table	4.9126*
Spectra	8.784*
Notebook	6.183*
Alien Information Center	5.8323*
Probe Design Center	0.0788
Mission Control Center	1.4015

*p < .05.

Mann–Whitney U tests revealed that for tool frequency, significant differences were found between high and low knowledge groups for the Concept Database (U = 5,249.5, p < .01), Spectra (U = 5,281, p < .01), and Notebook (U = 5,105, p < .02) (Table 7). Moreover, the medium and low knowledge groups differed significantly for Concept Database (U = 10878, p < .01) and Notebook (U = 10700, p < .02). However, no significant differences were found between the high and medium science knowledge groups.

Table 7.

Findings From Mann–Whitney U Tests Between Students Grouped by High, Medium, and Low Groups of Post-PBL Science Knowledge Scores.

	High vs. Low		High vs. Medium		Medium vs. Low
	Median		Median		Median
	High	Low	High	Medium	Medium	Low
Variable	(n = 80)	(n = 107)	(n = 80)	(n = 231)	(n = 231)	(n = 107)
Tool frequency
ConceptDatabase	2***	4	2	5	5***	4
Spectra	6***	13	6	12	12	13
Notebook	20.5**	74	20.5	54	54**	74
Tool duration
ConceptDatabase	0.1***	0.72	0.1	0.73	0.73***	0.72
PeriodicTable	1.18***	2.07	1.18	1.32**	1.32	2.07
Spectra	2.1***	5.12	2.1	3.97	3.97	5.12
Notebook	4.69***	21.87	4.69	12.8	12.8**	21.87
AlienInformationCenter	61.74	64.35	61.74	51.7**	51.7	64.35

*p < .05, **p < .02, ***p < .01.

For tool duration, there were significant differences between the high and low science knowledge groups for Concept Database (U = 5,415.5, p < .01), Periodic Table (U = 4,917, p < .01), Spectra (U = 5,426, p < .01), and Notebook (U = 5,145, p < .01). Significant differences were also found for Concept Database (U = 11,136, p < .01) and Notebook (U = 10,684, p < .02) between the medium and low knowledge groups, as well as for Periodic Table (U = 14,098, p <.02) and Alien Information Center (U = 14,202, p < .02) between the high and medium knowledge groups.

Students in the low science knowledge group spent significantly more time using the Concept Database, Periodic Table, Spectra, and Notebook. They also used the Concept Database, Spectra, and Notebook more frequently in problem-solving than the high science knowledge group did. In this PBL program, the Concept Database provides instructional modules on ten scientific concepts to facilitate conceptual understanding. When students began using the PBL program, some were already familiar with some of those science concepts, whereas others may not have been, depending on when a school used the program as its earth science unit. The Concept Database is designed to support students who feel a need to learn or review science concepts, enabling just-in-time learning. The Periodic Table provides a periodic table of the elements, and the Spectra tool offers information to help students interpret spectra. All of these tools are designed to facilitate students’ science learning, depending on their prior knowledge, and to help students solve problems. Hence, longer and more frequent use of those tools by students with low science knowledge scores suggests that those students needed more support with regard to conceptual science knowledge within the PBL program. Unlike their high science knowledge peers, who may have had some conceptual science knowledge, the low science knowledge group depended more on the tools within the program to get the science information that they needed to solve the problem in the PBL program. Similarly, the low science knowledge group also spent more time and used the Concept Database more frequently than the medium science knowledge group.

Relationship Between the Changes in Students’ Science Knowledge and Their Behavioral Patterns

While students’overall science knowledge increased significantly from pre- to post-PBL, we were interested in investigating the behavioral patterns of the groups who had significant changes in their science knowledge from pre- to post-PBL: (a) low-high (n = 32), those students whose science knowledge improved from low on the pre-PBL test to high post-PBL; (b) low-low (n = 29), those whose science knowledge was low on both tests; and (c) high-high (n = 14), those whose science knowledge was high on both tests. Significant differences were discovered among these three groups of students in tool usage frequency and duration for the Solar System Database, Notebook, Probe Design Center, Mission Control Center, and Concept Database (Table 8). In addition, the number of probes launched, successfully launched probes, and different probe justifications (i.e., vague claim probes and random input probes) were also found to differ significantly among the three groups.

Table 8.

Findings of Kruskal–Wallis Tests Examining Overall Behavioral Patterns Grouped by Their Pre- and Post-PBL Science Knowledge Test Scores.

Variable	Kruskal–Wallis test
Tool frequency
Solar System Database	10.401***
Mission Database	3.689
Concept Database	5.310*
Periodic Table	0.57
Spectra	1.224
Notebook	13.289***
Alien Information Center	0.841
Probe Design Center	11.738***
Mission Control Center	13.834***
Tool duration
Solar System Database	9.915***
Mission Database	4.257
Concept Database	6.385*
Periodic Table	0.928
Spectra	1.537
Notebook	12.595***
Alien Information Center	4.312
Probe Design Center	12.073***
Mission Control Center	13.768***
Other
Probe count	9.572***
Successful probe count	12.378***
Solution count	0.333
Successful solution count	4.087
Wrong solution count	5.708
Vague inquiry count	1.439
Vague claim count	11.61***
Random input count	14.4***
Reasoning count	4.506
Specific inquiry count	0.806

*p < .05, **p < .02, ***p < .01.

Additional Mann–Whitney U tests on those variables showed no significant differences between the low-low and low-high science knowledge students (Table 9). Between the low-low and high-high science knowledge groups, significant differences were found in tool usage frequencies for the Solar System Database (U = 321, p < .01), Notebook (U = 96, p < .01), Probe Design Center (U = 296.5, p < .01), and Mission Control Center (U = 300, p < .01). Between the low-low and high-high groups significant differences were also found in tool usage duration for the Solar System Database (U = 318, p < .01), Notebook (U = 101, p < .01), and Mission Control Center (U = 307, p < .01). Furthermore, significant differences were found between the low-high and high-high groups for all variables except the frequency of using the Solar System Database. In addition, the high-high group sent significantly more successful probes than the other two groups.

Table 9.

Findings From Mann–Whitney U Tests Between the Students as Grouped by Low-Low, Low-High, and High-High Levels of Pre- & Post-PBL Science Knowledge Scores.

	Low-Low vs. Low-High		Low-Low vs. High-High		Low-High vs. High-High
	Median		Median		Median
	L-L	L-H	L-L	H-H	L-H	H-H
Variable	(n = 29)	(n = 32)	(n = 29)	(n = 14)	(n = 32)	(n = 14)
Tool frequency
Solar System Database	229	128.5	229	59***	128.5	59
Concept Database	3	5	3	0	5	0***
Notebook	49	100.5	49	1***	100.5	1***
Probe Design Center	85	54.5	85	249***	54.5	249***
Mission Control Center	39	18.5	39	120***	18.5	120***
Tool duration
Solar System Database	68.717	60.425	68.717	20.4***	60.425	20.4***
Concept Database	0.45	0.6	0.45	0	0.6	0***
Notebook	11.733	34.083	11.733	0.042***	34.083	0.042***
Probe Design Center	15.3	5.017	15.3	35.475	5.017	35.475**
Mission Control Center	4.7	4.325	4.7	35.192***	4.325	35.192***
Other
Probe count	3	1	3	6***	1	6***
Successful probe count	1	0	1	5.5***	0	5.5***
Vague claim count	0	0	0	1.5**	0	1.5***
Random input count	0	0	0	1.5***	0	1.5***

*p < .05, **p < .02, ***p < .01.

The high-high science knowledge group used some tools (e.g., the Solar System Database) for a shorter period of time and less frequently but spent longer time and used the probe design-related tools more often than others. As a result, they launched greater numbers of probes and had more successful probes. This finding highlights the role of probe design in the high-high group's problem-solving.

In short, our results show that self-efficacy and science knowledge impacted students’ behavioral patterns in the PBL program. However, it is still unclear why the low-low and low-high groups had great differences in post-science knowledge test scores, even though no significant differences were found between these groups in tool usage. Test scores provide information about only one captured moment of performance (e.g., pre-PBL or post-PBL) but not the progress of students’ navigation. Therefore, we also investigated how students navigated their way with the different tools at each problem-solving stage within the program, using pathway analysis.

Pathway Analyses

Previous studies with similar student populations (e.g., Liu et al., 2016) have suggested four stages of problem-solving in this PBL program. In Stage 1 (Days 1–2), students’ primary purpose is to understand the problem; in Stage 2 (Days 3–5), students identify, gather, and organize information; in Stage 3 (Days 6–8), they integrate information and conduct hypothesis testing; and in Stage 4 (Days 9–10), they evaluate the process and submit their final solutions. Following this guideline, we divided students’ pathways within the program into four stages for further examination.

Comparison of Pathways Between the Low-Low and Low-High Self-Efficacy Groups

The patterns of learning behavior between students who improved in self-efficacy and those who did not are presented as pathways as low-high and low-low self-efficacy in Figure 3. The low-low self-efficacy students showed a greater variety of tool use patterns in the first and second stages. Noticeably, the low-low self-efficacy students began designing and launching probes earlier (in the first stage) and did not focus on probe design in the third stage. Conversely, the low-high self-efficacy students focused on the Alien Information Center and Notebook during the first and second stages. They also used other tools, such as Spectra and the Periodic Table, at a higher frequency. In particular, during the third stage, the low-high self-efficacy students tended to navigate between the Probe Design Center and Mission Control more often than the low-low group, indicating that they put more emphasis on testing their hypotheses in this stage. In the last stage, the low-high self-efficacy group visited the Communication Center more frequently to submit their solutions. They also reviewed their ideas by revisiting Notebook, the Alien Information Center, the Solar System Database, and sending probes. Overall, the tool usage pathway of the low-high self-efficacy group aligned more closely with the suggested stages of problem-solving than the tool usage pathway of the low-low self-efficacy group (Liu et al., 2016).

Figure 3.

Comparison of tool usage in four stages by the low-low and the low-high self-efficacy groups. (a) 1st stage low-high self-efficacy group. (b) 1st stage low-low self-efficacy group. (c) 2nd stage low-high self-efficacy group. (d) 2nd stage low-low self-efficacy group. (e) 3rd stage low-high self-efficacy group. (f) 3rd stage low-low self-efficacy group. (g) 4th stage low-high self-efficacy group. (h) 4th stage low-low self-efficacy group.

Comparison of Pathways Between the Low-Low and Low-High Science Knowledge Groups

We also analyzed students’ pathways in the low-low and low-high science knowledge groups. As shown in Figure 4, in the first stage, the most frequent patterns of the low-high group were between Alien Information Center and Notebook, indicating that students in this phase spent most of the time understanding the information about the aliens and taking notes. Also, they explored different tools, such as the Solar System Database, Spectra, and the Periodic Table. Similarly, in this phase, the low-low science knowledge students frequently went back and forth between the Alien Information Center and Notebook. However, the low-low group visited the Mission Control Center and the Probe Design Center, which are the places to design and launch probes and are not expected at this early stage (Liu et al., 2022). In the second stage, the most frequent patterns in the low-high group was Alien Information Center ←→ Notebook, and frequent patterns of Notebook ←→ Concept Database, and Notebook ←→ Solar System Database were also observed. The Alien Information Center, Concept Database, and Solar System Database provide information about the aliens and the planets, share cognitive load, and support cognitive processes (Liu & Bera, 2005). The most frequent patterns in the low-low group was Solar System Database ←→ Notebook, and frequent patterns of Alien Information Center ←→ Notebook, Notebook ←→ Spectra, Notebook ←→ Mission Database were also observed. These patterns showed both groups were engaged in conducting research. In the third stage, both groups visited the Probe Design Center and launched probes to test their hypotheses. The frequent patterns in this stage were Notebook ←→ Solar System Database and Alien Information Center ←→ Notebook for both groups, indicating that students revisited the Alien Information Center and Solar System Database to check probes’ results. Moreover, the students in the low-low group started to submit their final solutions by visiting the Communication Center. In the last phase, both groups visited the Notebook ←→ Solar System Database the most and visited the Communication Center to submit final solutions. To summarize, pathway analyses revealed that the low-high science knowledge group spent more time collecting relevant information before designing probes. In contrast, the low-low group started exploring probe design much earlier, without collecting enough information. Compared to their low-high peers, they rushed to submit their final solutions during the third stage.

Figure 4.

Comparison of tool usage in four stages by the low-low and the low-high science knowledge groups. (a) 1st stage low-high science knowledge group. (b) 1st stage low-low science knowledge group. (c) 2nd stage low-high science knowledge group. (d) 2nd stage low-low science knowledge group. (e) 3rd stage low-high science knowledge group. (f) 3rd stage low-low science knowledge group. (g) 4th stage low-high science knowledge group. (h) 4th stage low-low science knowledge group.

Discussion and Implications

The Role of Self-Efficacy

In this study, we have explored the role of self-efficacy in the PBL program by examining the relationship between students’ self-efficacy and their patterns of learning behaviors. We found that the high self-efficacy group used the Solar System Database significantly more frequently and used the Solar System Database and Spectra significantly longer than the medium and low self-efficacy groups. In this PBL program, the Solar System Database provides information on selected planets and moons, and Spectra's function is to help students interpret spectra related to aliens. These tools are key in identifying, interpreting, and gathering information. Thus, the high self-efficacy students were more aware of what learning tools to use to solve problems. Fadillah et al. (2021) have pointed out that students with strong self-efficacy can better manage and control their learning as they solve problems; they understand what information and learning strategies they need. In addition, the high self-efficacy group sent significantly more probes with reasoning and specific inquiries than the other two groups. The high self-efficacy group also submitted a significantly greater number of successful solutions than their medium and low counterparts, despite no significant difference in the number of solutions submitted by the three groups. Sending more probes with reasoning and specific inquiries showed that high self-efficacy students understood the information they collected and the goal they were expected to achieve, so they were more likely to come up with the correct solutions. With more reasoning and specific inquiry probes, the high self-efficacy group could get more specific information, which yielded a significantly greater number of successful solutions. According to Alt et al. (2022), students with high self-efficacy often strive to succeed and understand the learning material in order to reach specific learning goals. We found that students’ levels of self-efficacy were positively correlated with their success rates in solving problems, which aligns with previous literature indicating self-efficacy is an important mediator of students’ performance and learning behaviors (Amini et al., 2019; Fadillah et al., 2021).

Our pathway analysis demonstrated different patterns of learning behavior between the low-low and low-high self-efficacy groups. The low-low group showed greater variety in tool use patterns in the first and second stages of problem-solving and began designing and launching probes in the first stage. However, the low-high self-efficacy group initially focused on the Alien Database, Notebook, and Solar System Database in the first stage and sent probes in the third stage. Thus, the low-low self-efficacy group began sending probes to evaluate their hypotheses too early, without spending enough time and effort in research to develop hypotheses. As Fitriani et al. (2020) found that students’ self-efficacy improved during a process of investigation in PBL, our findings show that the low-high self-efficacy group may have improved their self-efficacy by following PBL procedures. In a previous study with Alien Rescue (Liu & Bera, 2005), those sixth-graders who followed the identification and understanding of the problem and then collected related information to support their hypotheses before testing were more likely to solve the problem successfully. The present study confirms that the low-high students also enhanced their self-efficacy by following the four stages of problem-solving.

As shown from the qualitative data, the high and medium self-efficacy groups reported more often that they had learned about problem-solving skills than the low self-efficacy students. “Problem-solving skills” was the second most frequent theme in the high (13.42%) and medium (20.84%) self-efficacy groups. In contrast, only 6% of the low self-efficacy students reported that they had learned problem-solving skills in the program. Because self-efficacy represents one's belief in successfully completing tasks and achieving objectives (Bandura, 1988), it is not surprising that the high self-efficacy students were more confident and positive in their learning. Hong et al. (2016) has pointed out that self-efficacy is linked with students’ interests in learning and it plays an essential role in learning tasks that require more independence and initiative. The high and medium self-efficacy groups were more likely to believe that they were learning problem-solving skills, perhaps because self-efficacy drove them to be more motivated to solve problems. Thus, they were more willing to practice, leading to greater problem-solving skills than their peers with low self-efficacy. Our study also shows that the greater the self-efficacy, the more positive the perception of the PBL program. The students in the medium and low self-efficacy groups felt that the program was difficult. Previous studies have also found that self-efficacy is associated with students’ perceptions of task difficulty (e.g., Lee & List, 2021) and their perceived efficacy in learning (e.g., Jia et al., 2014). Students’ different levels of self-efficacy may explain their different attitudes toward the PBL program. For low self-efficacy students, lower levels of perceived ability to successfully complete tasks make their perceptions of a given task challenging (Li et al., 2007), whereas the high self-efficacy students were more confident in their ability, thus resulting in a different attitude toward PBL.

Our findings highlighted the importance of self-efficacy in PBL regarding learning behavior, performance, and perceptions, which has profound implications for teaching. For teachers, it is important to increase students’ self-efficacy when implementing PBL. It is beneficial for teachers to monitor their students’ development in problem-solving skills as part of PBL, especially among students with low self-efficacy and difficulty in managing their learning. In addition, the results revealed the importance of following the PBL procedures in improving students’ self-efficacy. Teachers should spend more time guiding students to prepare for PBL at the beginning of the PBL process to help lead them toward success. Encouraging students to take on challenges and making them feel confident and positive can also help develop their self-efficacy. While we only compared the learning behaviors of students with different levels of self-efficacy in this study, given the positive findings a follow-up study should examine if better learning behaviors would lead to improvement in students’ self-efficacy.

The Role of Students’ Science Knowledge

With respect to the relationship between students’ science knowledge and their behavioral patterns within PBL, the students with low science knowledge spent significantly more time and used some tools more frequently (e.g., the Concept Database and Spectra) to solve problems than those with high science knowledge. The low science knowledge group was not as familiar with some key science concepts as their higher science knowledge peers, so they needed to use those tools more often to gather relevant information. The longer and more frequent use of the tools suggests that the low science knowledge group lacked sufficient prior knowledge and needed additional support and content scaffolding to solve the problem. This finding also has implications for teachers’ PBL implementation. Teachers may need to make an extra effort with students who lack sufficient knowledge of content to solve the problem, and teachers should monitor their progress more frequently in order to provide support (Kim et al., 2018). Providing teachers with adaptive dashboard systems that show students’ progress during PBL can be helpful (Tissenbaum & Slotta, 2019; van Leeuwen et al., 2022). We are pursuing this effort in our research.

Our pathway analyses showed that the low-low science knowledge group was not as efficient in using crucial tools in the initial phase of problem-solving in comparison with their low-high peers. The low-low science knowledge students wasted time and effort using tools that would be necessary only in the later phases of problem-solving. This result supports the findings of Li and Stylianides (2018) and Park and Ertmer (2008), who have argued for the importance of scaffolding to guide students through PBL. Providing just-in-time feedback to the low-low students might have led them to develop more knowledge and problem-solving strategies to move through the problem-solving phases in a timely manner.

The findings for the high-high group also have implications. These students spent more time with probe design-related tools (i.e., the Mission Control Center and Probe Design Center) and used them more frequently than the other two groups. However, they spent significantly less time on the Solar System Database, which provides conceptual scientific facts. This suggests the importance of testing hypotheses in problem-solving, which aligns with Spires et al.'s (2011) assertion of a positive relationship between the effects of middle-grade students’ hypothesis-testing strategies and learning outcomes in PBL. The high-high group's greater time spent in probe design, and their greater number of trials in launching probes understandably increased their chances of success. The task of designing and launching probes to test their hypotheses played a crucial role. Furthermore, the high-high group's lesser utilization of the Solar System Database and their focused utilization of probe-related tools imply that these students already had adequate science knowledge to proceed to the next phase in problem-solving. Students who lacked science knowledge spent time with tools that taught them the science knowledge needed to proceed, resulting in less time to finish problem-solving. On the other hand, students with sufficient science knowledge before PBL had more time to focus on hypotheses testing by saving time and effort in the initial problem-solving phase.

In conclusion, through detailed analyses of students’ learning behavior patterns using a LA approach, we have shown that self-efficacy plays an important role in students’ problem-solving. Science knowledge also plays a role in how students determine when and what tools to use to facilitate their problem-solving. Such findings offer insights for researchers in PBL and LA who are interested in designing technology-enriched tools to support K-12 teachers in classroom implementation.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Min Liu

Author Biographies

Min Liu is a Professor of Learning Technologies at the University of Texas at Austin. Her teaching and research interests center on educational uses of new media and other emerging technologies, particularly the impact of such technologies on teaching and learning; and the design of new media enriched interactive learning environments for learners at all age levels. She has published over 80 peer-reviewed articles in leading educational technology research journals, 20 peer-reviewed book chapters, and presents regularly at national and international technology conferences. Her current R&D projects include studying the design and effectiveness of immersive, rich media environments on learning and motivation; learning analytics in serious game and other online environments; and understanding MOOCs as an emerging online learning tool.

Ying Cai is a doctoral student in the Learning Technologies program at the University of Texas at Austin. She received her master's degree in Chinese language and literature from the University of Massachusetts Amherst. Before her study at UT, she had taught Mandarin Chinese in the United States and intensive Chinese language programs in China. Her research interests include game-based learning, learning analytics, second language acquisition, and integration of technology into language learning.

Songhee Han is a doctoral candidate in the Learning Technologies program at the University of Texas at Austin. She received a master's degree in curriculum and instruction from the University of Texas at Austin in 2019. Her current projects include the adoption of chatbot technology in online courses as well as the influence of game-based learning on students' achievement in science learning. Her work has been published in many prestigious journals, such as Computers & Education, International Journal of Artificial Intelligence in Education, Online Learning, and more.

Peixia Shao is a doctoral student in the Learning Technologies program at the University of Texas at Austin. Her interests focus on learning analytics, online education and multimedia project based learning. She works with the Alien Rescue research team (https://alienrescue.education.utexas.edu/) to investigate students' self-efficacy, attitude, learning behaviors and patterns, and science learning performance in an online problem-based 3D immersive learning environment.

References

Akınoğlu

Tandoğan

RÖ

(2007). The effects of problem-based active learning in science education on students’ academic achievement, attitude and concept learning. Eurasia Journal of Mathematics, Science & Technology Education, 3(1), 71–81. https://doi.org/10.12973/ejmste/75375

Al-Kloub

M. I.

Salameh

T. N.

Froelicher

E. S.

(2014). Nursing students evaluation of problem based learning and the impact of culture on the learning process and outcomes: A pilot project. Nurse Education in Practice, 14(2), 142–147. https://doi.org/10.1016/j.nepr.2013.06.013

Alt

Weinberger

Heinrichs

Naamati-Schneider

(2022). The role of goal orientations and learning approaches in explaining digital concept mapping utilization in problem-based learning. Current Psychology, 1–16. Advance online publication. https://doi.org/10.1007/s12144-021-02613-7

Amini

Setiawan

Fitria

Ningsih

(2019). The difference of students learning outcomes using the project-based learning and problem-based learning model in terms of self-efficacy. Journal of Physics: Conference Series, 1387(1), Article 012082. https://doi.org/10.1088/1742-6596/1387/1/012082

Avella

J. T.

Kebritchi

Nunn

S. G.

Kanai

(2016). Learning analytics methods, benefits, and challenges in higher education: A systematic literature review. Online Learning, 20(2), 13–29. https://doi.org/10.24059/olj.v20i2.790

Bai

Chao

G. C. N.

Wang

(2019). The relationship between social support, self-efficacy, and English language learning achievement in Hong Kong. Tesol Quarterly, 53(1), 208–221. https://doi.org/10.1002/tesq.439

Bandura

(1988). Self-efficacy conception of anxiety. Anxiety Research, 1(2), 77–98. https://doi.org/10.1080/10615808808248222

Bandura

(1994). Self-efficacy. In Ramachaudran

V. S.

(Ed.), Encyclopedia of human behavior (Vol. Vol. 4, pp. 71–81). Academic Press.

Barrows

H. S.

Tamblyn

R. M.

(1980). Problem-based learning: An approach to medical education. Springer Publishing Company.

10.

Belland

B. R.

Walker

A. E.

Kim

N. J.

Lefler

(2017). Synthesizing results from empirical research on computer-based scaffolding in STEM education: A meta-analysis. Review of Educational Research, 87(2), 309–344. https://doi.org/10.3102/0034654316670999

11.

Bereiter

Scardamalia

(2000). Commentary on part I: Process and product in problem-based learning (PBL) research. In Evensen

D. H.

Hmelo

C. E.

(Eds.), Problem-based learning: A research perspective on learning interactions (pp. 185–195). Lawrence Erlbaum Associates.

12.

Bland

J. M.

Altman

D. G.

(1995). Multiple significance tests: The Bonferroni method. BMJ, 310(6973), 170. https://doi.org/10.1136/bmj.310.6973.170

13.

Botelho

M. G.

Lam

O. L. T.

Watt

R. M.

Leung

D. Y. P.

Kember

(2018). Evaluation of peer-generated MCQs to assess and support learning in a problem-based learning programme. European Journal of Dental Education, 22(3), e358–e363. https://doi.org/10.1111/eje.12304

14.

Chen

Kolmos

(2021). Forms of implementation and challenges of PBL in engineering education: A review of literature. European Journal of Engineering Education, 46(1), 90–115. https://doi.org/10.1080/03043797.2020.1718615

15.

Chen

Hogaboam

Hmelo-Silver

C. E.

Lajoie

S. P.

Wiseman

Bodnar

Kazemitabar

Chan

L.-K.

(2016). Instructional dashboards to support deep learning in an online problem-based learning environment. In Wang

Kirschner

P. A.

Bridges

S. M.

(Eds.), Proceedings of the workshop on computer-based learning environments for deep learning in inquiry and problem-solving contexts (pp. 6–12). Laboratory for Knowledge Management & E-Learning, University of Hong Kong. https://www.mcgill.ca/atlas-lab/files/atlas-lab/peter_et_al._howard.pdf

16.

Claro

Paunesku

Dweck

C. S.

(2016). Growth mindset tempers the effects of poverty on academic achievement. Proceedings of the National Academy of Sciences of the United States of America, 113(31), 8664–8668. https://doi.org/10.1073/pnas.1608207113

17.

Credé

Kuncel

N. R.

(2008). Study habits, skills, and attitudes: The third pillar supporting collegiate academic performance. Perspectives on Psychological Science, 3(6), 425–453. https://doi.org//10.1111/j.1745-6924.2008.00089.x

18.

Creswell

J. W.

Poth

C. N.

(2018). Qualitative inquiry and research design: Choosing among five approaches (4th ed.). Sage Publications.

19.

Demirören

Turan

Öztuna

(2016). Medical students’ self-efficacy in problem-based learning and its relationship with self-regulated learning. Medical Education Online, 21(1), Article 30049. https://doi.org/10.3402/meo.v21.30049

20.

DiCerbo

K. E.

(2014). Game-based assessment of persistence. Journal of Educational Technology & Society, 17(1), 17–28. https://doi.org/https://www.jstor.org/stable/jeductechsoci.17.1.17

21.

Duman

Özçelik

(2018). The effect of the creative drama-supported problem-based learning approach on the self-efficacy ability in geometry. Universal Journal of Educational Research, 6(12), 2918–2924. https://eric.ed.gov/?id=EJ1198883 https://doi.org/10.13189/ujer.2018.061227

22.

El Alfy

Gómez

J. M.

Dani

(2019). Exploring the benefits and challenges of learning analytics in higher education institutions: A systematic literature review. Information Discovery and Delivery, 47(1), 25–34. https://doi.org/10.1108/IDD-06-2018-0018

23.

Fadillah

Budiyono & Nurhasanah

(2021). The analysis of students’ metacognition in solving math problems based on self-efficacy. Journal of Physics: Conference Series, 1808(1), Article 12062. https://doi.org/10.1088/1742-6596/1808/1/012062

24.

Fitriani

Zubaidah

Susilo

Al Muhdhar

M. H. I.

(2020). The effects of integrated problem-based learning, predict, observe, explain on problem-solving skills and self-efficacy. Eurasian Journal of Educational Research, 2020(85), 45–64. https://eric.ed.gov/?id=EJ1242444 https://doi.org/10.14689/ejer.2020.85.3

25.

Foster

Siddle

(2020). The effectiveness of learning analytics for identifying at-risk students in higher education. Assessment & Evaluation in Higher Education, 45(6), 842–854. https://doi.org/10.1080/02602938.2019.1682118

26.

Günter

Alpat

S. K.

(2017). The effects of problem-based learning (PBL) on the academic achievement of students studying “electrochemistry.”. Chemistry Education Research and Practice, 18(1), 78–98. https://doi.org/10.1039/C6RP00176A

27.

Hmelo-Silver

C. E.

(2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235–266. https://doi.org/10.1023/B:EDPR.0000034022.16470.f3

28.

Hong

J.-C.

Hwang

M.-Y.

Szeto

Tsai

C.-R.

Kuo

Y.-C.

Hsu

W.-Y.

(2016). Internet cognitive failure relevant to self-efficacy, learning interest, and satisfaction with social media learning. Computers in Human Behavior, 55(Part A), 214–222. https://doi.org/10.1016/j.chb.2015.09.010

29.

Hung

Zhang

(2012). Examining mobile learning trends 2003–2008: A categorical meta-trend analysis using text mining techniques. Journal of Computing in Higher Education, 24(1), 1–17. https://doi.org/10.1007/s12528-011-9044-9

30.

Hung

Dolmans

D. H. J. M.

van Merriënboer

J. J. G.

(2019). A review to identify key perspectives in PBL meta-analyses and reviews: Trends, gaps and future research directions. Advances in Health Sciences Education, 24(5), 943–957. https://doi.org/10.1007/s10459-019-09945-x

31.

Hwang

(2021). The relationship between self-directed learning and problem-solving ability: The mediating role of academic self-efficacy and self-regulated learning among nursing students. International Journal of Environmental Research and Public Health, 18(4), Article 1738. https://doi.org/10.3390/ijerph18041738

32.

Jia

Bhatti

Nahavandi

(2014). The impact of self-efficacy and perceived system efficacy on effectiveness of virtual training systems. Behaviour & Information Technology, 33(1), 16–35. https://doi.org/10.1080/0144929X.2012.681067

33.

Kim

N. J.

Belland

B. R.

Axelrod

(2018). Scaffolding for optimal challenge in K–12 problem-based learning. The Interdisciplinary Journal of Problem-Based Learning, 13(1), Article 3. https://doi.org/10.7771/1541-5015.1712

34.

Kokotsaki

Menzies

Wiggins

(2016). Project-based learning: A review of the literature. Improving Schools, 19(3), 267–277. https://doi.org/10.1177/1365480216659733

35.

Kostagiolas

Lavranos

Korfiatis

(2019). Learning analytics: Survey data for measuring the impact of study satisfaction on students’ academic self-efficacy and performance. Data in Brief, 25, Article 104051. https://doi.org/10.1016/j.dib.2019.104051

36.

Kovanovic

Mazziotti

Lodge

(2021). Learning analytics for primary and secondary schools. Journal of Learning Analytics, 8(2), 1–5. https://doi.org/10.18608/jla.2021.7543

37.

Kubiatko

Vaculová

(2011). Project-based learning: Characteristic and the experiences with application in the science subjects. Energy Education Science and Technology Part B: Social and Educational Studies, 3(1), 65–74.

38.

Lapuz

A. M.

Fulgencio

M. N.

(2020). Improving the critical thinking skills of secondary school students using problem-based learning. International Journal of Academic Multidisciplinary Research, 4(1), 1–7. https://doi.org/http://ijeais.org/wp-content/uploads/2020/01/IJAMR200101.pdf

39.

Lee

H. Y.

List

(2021). Examining students’ self-efficacy and perceptions of task difficulty in learning from multiple texts. Learning and Individual Differences, 90, Article 102052. https://doi.org/10.1016/j.lindif.2021.102052

40.

H.-C.

Stylianides

A. J.

(2018). An examination of the roles of the teacher and students during a problem-based learning intervention: Lessons learned from a study in a Taiwanese primary mathematics classroom. Interactive Learning Environments, 26(1), 106–117. https://doi.org/10.1080/10494820.2017.1283333

41.

Lee

Solmon

(2007). The role of perceptions of task difficulty in relation to self-perceptions of ability, intrinsic value, attainment value, and performance. European Physical Education Review, 13(3), 301–318. https://doi.org/10.1177/1356336X07081797

42.

Liu, M., & Bera, S. (2005). An analysis of cognitive tool use patterns in a hypermedia learning environment. Educational Technology Research and Development, 53(1), 5–21.

43.

Liu, M., Cai, Y., Han, S., & Shao, P. (2022). Understanding student navigation patterns in game-based learning. Journal of Learning Analytics, 9(3), 50–74.

44.

Liu, M., Lee, J., Kang, J., & Liu, S. (2016). What we can learn from the data: A multiple-case study examining behavior patterns by students with different characteristics in using a serious game. Technology, Knowledge and Learning, 21(1), 33–57. https://doi.org/10.1007/s10758-015-9263-7

45.

Liu, M., Liu, S., Pan, Z., Zou, W., & Li, C. (2019). Examining science learning and attitude by at-risk students after they used a multimedia-enriched problem-based learning environment. Interdisciplinary Journal of Problem-Based Learning, 13(1). https://doi.org/10.7771/1541-5015.1752

46.

Liu, M., Pan, Z., Cai, Y., Shao, P. & Han, S. (2021). The effect of a multimedia-enriched problem-based learning environment on socioeconomically disadvantaged middle school students' science learning: Examining the relationship among self-efficacy, attitude, and performance. Journal of Educational Multimedia and Hypermedia, 30(4), 359–391.

47.

Maxwell

N. L.

Mergendoller

J. R.

Bellisimo

(2005). Problem-based learning and high school macroeconomics: A comparative study of instructional methods. The Journal of Economic Education, 36(4), 315–329. https://doi.org/10.3200/JECE.36.4.315-331

48.

Merritt

Lee

M. Y.

Rillero

Kinach

B. M.

(2017). Problem-based learning in K–8 mathematics and science education: A literature review. Interdisciplinary Journal of Problem-Based Learning, 11(2), Article 3. https://doi.org/10.7771/1541-5015.1674

49.

M. L.

Bridges

Law

S. P.

Whitehill

(2014). Designing, implementing and evaluating an online problem-based learning (PBL) environment—A pilot study. Clinical Linguistics & Phonetics, 28(1–2), 117–130. https://doi.org/10.3109/02699206.2013.807879

50.

Nijhuis

J. F. H.

Segers

M. S. R.

Gijselaers

W. H.

(2005). Influence of redesigning a learning environment on student perceptions and learning strategies. Learning Environments Research, 8(1), 67–93. https://doi.org/10.1007/s10984-005-7950-3

51.

Pajares

(2006). Self-efficacy during childhood and adolescence: Implications for teachers and parents. In Pajares

Urdan

(Eds.), Self-efficacy beliefs of adolescents (Vol. Volume 5, pp. 339–367). Information Age Publishing.

52.

Park

S. H.

Ertmer

P. A.

(2008). Examining barriers in technology-enhanced problem-based learning: Using a performance support systems approach. British Journal of Educational Technology, 39(4), 631–643. https://doi.org/10.1111/j.1467-8535.2008.00858.x

53.

Pintrich

P. R.

Smith

D. A. F.

Garcia

McKeachie

W. J.

(1993). Reliability and predictive validity of the motivated strategies for learning questionnaire (MSLQ). Educational and Psychological Measurement, 53(3), 801–813. https://doi.org/10.1177/0013164493053003024

54.

Savery

J. R.

(2006). Overview of problem-based learning: Definitions and distinctions. Interdisciplinary Journal of Problem-Based Learning, 1(1), 9–20. Available at: https://doi.org/10.7771/1541-5015.1002

55.

Siemens

Baker

R. S. J. D.

(2012). Learning analytics and educational data mining: Towards communication and collaboration. In LAK ‘12: Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 252–254). Association for Computing Machinery. https://doi.org/10.1145/2330601.2330661.

56.

Spires

H. A.

Rowe

J. P.

Mott

B. W.

Lester

J. C.

(2011). Problem solving and game-based learning: Effects of middle grade students’ hypothesis testing strategies on learning outcomes. Journal of Educational Computing Research, 44(4), 453–472. https://doi.org/10.2190/EC.44.4.e

57.

Tan

A.-L.

Liang

J.-C.

Tsai

C.-C.

(2021). Relationship among high school students’ science academic hardiness, conceptions of learning science and science learning self-efficacy in Singapore. International Journal of Science and Mathematics Education, 19(2), 313–332. https://doi.org/10.1007/s10763-019-10040-1

58.

Tawfik

A. A.

Gish-Lieberman

J. J.

Gatewood

Arrington

T. L.

(2021). How K-12 teachers adapt problem-based learning over time. Interdisciplinary Journal of Problem-Based Learning, 15(1), 1–18. https://doi.org/10.14434/ijpbl.v15i1.29662

59.

Tempelaar

D. T.

Rienties

Giesbers

(2015). In search for the most informative data for feedback generation: Learning analytics in a data-rich context. Computers in Human Behavior, 47, 157–167. https://doi.org/10.1016/j.chb.2014.05.038

60.

Thomas

J. W.

(2000). A review of research on project-based learning. Autodesk Foundation.

61.

Tissenbaum

Slotta

(2019). Supporting classroom orchestration with real-time feedback: A role for teacher dashboards and real-time agents. International Journal of Computer-Supported Collaborative Learning, 14(3), 325–351. https://doi.org/10.1007/s11412-019-09306-1

62.

Valdez

J. E.

Bungihan

M. E.

(2019). Problem-based learning approach enhances the problem solving skills in chemistry of high school students. Journal of Technology and Science Education, 9(3), 282–294. https://doi.org/10.3926/jotse.631

63.

van Leeuwen

Teasley

S. D.

Wise

A. F.

(2022). Teacher and student facing learning analytics. In Lang

Siemens

Wise

A. F.

Gašević

Merceron

(Eds.), Handbook of learning analytics (pp. 130–140). Society for Learning Analytics Research.

64.

Wijnen

Loyens

S. M. M.

Smeets

Kroeze

van der Molen

H. T.

(2017). Students’ and teachers’ experiences with the implementation of problem-based learning at a university law school. Interdisciplinary Journal of Problem-Based Learning, 11(2), Article 5. https://doi.org/10.7771/1541-5015.1681

65.

Yadav

Mayfield

Moudgalya

S. K.

Kussmaul

H. H.

(2021). Collaborative learning, self-efficacy, and student performance in cs1 pogil. In SIGCSE ‘21: Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (pp. 775–781). Association for Computing Machinery. https://doi.org/10.1145/3408877.3432373.

66.

Yuan

H. B.

Williams

B. A.

Yin

Liu

Fang

J. B.

Pang

(2011). Nursing students’ views on the effectiveness of problem-based learning. Nurse Education Today, 31(6), 577–581. https://doi.org/10.1016/j.nedt.2010.10.009