Sage Journals: Discover world-class research

Abstract

Despite the growing emphasis on integrating collaborative problem-solving (CPS) into science, technology, engineering, and mathematics (STEM) education, a comprehensive understanding of the critical factors that affect the effectiveness of this educational approach remains a challenge. This study aims to identify effective strategic and behavioral factors in course design and assess how these factors contribute to students’ learning performance. This study collected data from 106 students enrolled in seventh-grade science classes by using a mixed-method approach. First, the t-test results indicate that students’ learning performance was improved through CPS-based STEM learning. A path analysis shows that CPS awareness and several behavioral factors had direct effects, while several strategic factors had indirect effects on the improvement of learning performance. Finally, a dialog analysis indicates that students’ integrative use of CPS skills, especially task regulation skills used along with other skills, helped improve learning performance. This study not only bridges the gap in understanding the effectiveness of CPS in STEM education but also provides specific suggestions for improving instructional design.

Plain Language Summary

Exploring how learning behaviors and learning strategies affect secondary students’ learning in collaborative science lessons

This study looked at how certain teaching strategies and student behaviors can make a difference in learning. We checked out 106 seventh-graders in science classes, using different methods to gather data. As the result, when students working together to solve problems, their grades get better. We found that knowing how to collaborate and certain behaviors directly improve learning, while some behind-the-scenes strategies also play an important role. Specifically, being good at organizing tasks and use it with other skills have positive effects for their grades

Keywords

learning analytics learning behavior collaborative problem solving STEM education learning strategy

Introduction

Science, technology, engineering, and mathematics (STEM) education has received great attention worldwide, driven by the changing global workforce. The aim of STEM education is to diffuse knowledge and skills in integrated fields to solve real-world problems by linking scientific knowledge and skills, technology, engineering design, mathematical thinking, and analysis (Kelley & Knowles, 2016).

It has been found that students benefit more from solving problems collaboratively than from solving them individually in STEM education (Hogan, 1999) due to the integrative nature and complexity of the problems involved (Kelley & Knowles, 2016). Students facing situations requiring solutions to complex problems and new solution ideas are expected to perform cognitive processes collaboratively, which requires skills associated with collaborative problem solving (CPS; Andrews-Todd & Forsyth, 2020).

In CPS, students’ activities and contributions are closely intertwined, related, and mutually affected (Care et al., 2016; Hesse et al., 2015). Thus, CPS awareness is considered to mutually affect other psychological factors, such as perceptions on how to use specific cognitive strategies (Chen et al., 2019), as well as certain behavioral factors, such as individual problem-solving behaviors and interactions with peers during online programming learning (Hwang et al., 2012).

The learning process in CPS-based STEM education is integrative, involving CPS skills, STEM learning methods, and specific behaviors. Analyze this integrated learning process to identify key factors that affect learning performance is crucial. Moreover, given the importance of putting learning strategies into practice through behaviors (Zimmerman, 2002), learning strategies and behaviors are closely related, leading traditional approaches like surveys and classroom observations inadequate. A more comprehensive approach of data collection and analysis is necessary to fully understand and enhance learning performance.

In CPS, learners’ CPS skills of how to solve problems and collaborate with others can be observed only through collaboration (Hesse et al., 2015), such as how to share their knowledge or understanding of a problem, method, or solution during group work. However, it is difficult to observe and measure all learning processes during learning activities. For example, the cognitive process that occurs during individual learning, an important factor in the preparation for CPS learning (OECD, 2017), cannot be measured during the collaboration process.

The learning analytics (LA) approach is considered helpful for exploring educational data regarding students’ learning processes and behaviors and incorporating these data and analyses into instructional design (Dunbar et al., 2014). As Ifenthaler et al. (2021) emphasized, LA focuses on “analytics for learning,” which aims to enhance learning, teaching, and assessment and improve learning design. Learning analytics provides opportunities to see how learning design impacts not only outcomes but also learning processes. It is more effective and efficient to use the LA approach to explore students’ learning behavioral patterns than to use more labor-intensive methods, such as observing and coding learning behaviors displayed in class (Zarzour et al., 2020). For example, learning logs can be recorded via e-book readers to provide data on the learning behaviors and cognitive and metacognitive processes students display when reading digital learning materials (Ogata et al., 2017; Yen et al., 2018).

This study aims to clarify the complex learning process and the relationships among strategic factors including using CPS skills and conducting SETM learning, and behavioral factors. It focuses on identifying effective factors improving learning performance in this integrative framework. We collected and analyzed students’ learning data regarding learning performance, learning behaviors and strategical factors of conducting CPS and STEM learning, and their applications of CPS skills. The study addressed three research questions (RQs):

RQ1: What is the difference in learning performance between the pre- and post-tests in CPS-based STEM lessons?

RQ2: What are the strategic and behavioral factors that improve the learning performance in CPS-based STEM lessons?

RQ3: How do students use CPS skills to improve their understanding in CPS-based STEM lessons?

Literature Review

Strategy of Using CPS Skills in CPS-Based STEM Education

CPS-based STEM education integrates science, technology, engineering, and mathematics (STEM) disciplines through a “joint activity where dyads or small groups execute several steps in order to transform a current state into a desired goal state” (Hesse et al., 2015, p. 39). The primary aim of this approach is to improve students’ learning performance and the development of CPS skills in authentic and interdisciplinary contexts.

The CPS framework involves social and cognitive domains, in which the integration of these two domains is important (Hesse et al., 2015). In the social domain, students manage social interactions, awareness, and participant contributions to achieve a common goal, while in the cognitive domain, they apply reasoning skills to deal with problems (Care et al., 2016).

Several studies have shown that the CPS learning approach benefits students’ learning performance in STEM-related subjects based on instructors’ scaffolding and intervention (e.g., Lin et al., 2020) and students’ learning strategies (e.g., Kim & Tawfik, 2023). In STEM education, students are required to solve problems in real-world settings that are complex and ill-defined; the students must examine and apply theories and knowledge to improve their problem-solving skills and integrate their comprehension and application of knowledge in STEM fields (Lou et al., 2011; Savery, 2006). Integrating CPS in STEM education encourages students to collaborate and use collection knowledge to acquire inadequate knowledge and solve ill-defined problems (Gu et al., 2015). Effective integration of CPS skills with STEM learning methodologies is crucial for achieving success in CPS-based STEM education.

Hesse et al.’s (2015) framework divided CPS skills into smaller and teachable units within social (participation, perspective taking, and social regulation) and cognitive (task regulation, learning, and knowledge building) domains, making it easier to instruct and evaluate. The framework is designed to be interdisciplinary, allowing for its application in integrated learning such as STEM.

Regarding learning performance improvement, by engaging in structured activities and developing specific CPS skills, this framework help students apply theoretical knowledge to authentic problems, enhancing recall and understanding, and leading to improved learning performance. It also promotes reflective thinking and metacognition, which helps them adapt their strategies to optimize learning performance.

Based on the insights from previous studies and Hesse et al.’s (2015) framework, we formulate the first hypothesis: Integrating CPS framework into STEM education improves students’ CPS skills, consequently positively affecting their learning performance.

STEM Learning Strategy in CPS-Based STEM Education

STEM education comprises scientific inquiry, technology literacy, mathematical thinking, and engineering design (Kelley & Knowles, 2016), which need structured classroom environment and well-designed instructional approaches, including problem-based approaches, open-ended tasks, and technological support (Herro et al., 2019). In STEM education, students deal with authentic and open-ended problems using various knowledge and skills. Open-ended tasks are under-described, forcing students to use various strategies and approaches to develop content knowledge in the learning process. Appropriate technology is integrated into instructional design to support problem-solving, whereby students search for information, create products, or conduct mathematical analysis. Moreover, technology makes it easier to track learners’ cognitive actions and behaviors and analyze them objectively than in paper-and-pencil learning environments (Graesser et al., 2022). Students need to use effective learning strategies to understand STEM content and transfer knowledge to solve authentic problems.

Due to its integrative nature, STEM learning strategy is enhanced by under difference course conditions. Hora and Oleson (2017) explored undergraduate students’ use of strategies in STEM learning, focusing on the use of specific strategies under different course conditions and course structures. Although many students tend to utilize ineffective learning strategies, effective learning strategies can be encouraged and improved through appropriate instructional designs, such as the course structure (Hora & Oleson, 2017), or the sequence of teaching activities in STEM education (Kranzfelder et al., 2019). Therefore, identifying and applying effective STEM learning strategies is important for optimizing instructional design in STEM education.

CPS-based STEM lessons include specific CPS activities, teachers’ instructions, and other individual learning activities in and out of class. It is necessary to consider collaborative and individual learning activities in CPS-based STEM lessons. Griese et al. (2015) developed a framework for assessing individual learning strategies for STEM students that comprises cognitive and metacognitive strategies, motivation, and collaborative behaviors. The factors in this framework can be fostered by interventions, such as organizing and summarizing important points using organizing strategies, connecting new knowledge to earlier knowledge using an elaborating strategy, and collaborating with others to construct a shared understanding using the peer learning strategy.

When integrated within CPS-based STEM lessons, these strategies contribute to both the development of CPS skills and the improvement of learning performance. For example, cognitive strategies support students in better understanding and retaining STEM content, while metacognitive strategies help them reflect on and regulate their learning process. Applying Griese et al.’s (2015) framework in CPS-based STEM education ensures that individual learning strategies are effectively taught and measured, connecting these strategies to improve learning performance.

Therefore, we adopted Griese et al.’s (2015) framework in this study and formulated the second hypothesis: Identifying and implementing effective STEM learning strategies improve students’ learning performance in CPS-based STEM education.

Learning Behaviors and Learning Analytics

Many studies indicated that learning behaviors were important indicators of learning performance in CPS learning. For example, Gu et al. (2015) conducted observations to understand students’ learning behaviors and specialize their CPS skills by analyzing the group discourse. Collecting and analyzing discourse and video recordings are standard methods for understanding learners’ learning behaviors, actions, and processes. However, it is difficult to understand students’ usual learning situations by conducting qualitative analysis merely due to the labor-intensive and time-consuming features, which limits its practical use for regular lessons.

More studies have started to utilize learning log data to identify students’ learning behaviors in online courses. For example, Lowes et al. (2015) identified students’ online behaviors by collecting the learning log data on a learning management system (LMS) and found the relationship between higher levels of online behaviors, attendance, and student-student interactions with higher learning performance (Lowes et al., 2015). As one example of a large-scale study, Greiff et al. (2015) conducted a log-file analysis by capitalizing on the computer-based problem-solving assessment in the PISA 2012 assessment to understand students’ behavior when working on problem-solving items. Their results provided great implications for researchers and teachers to understand students’ cognitive behaviors and identify the hidden patterns of behaviors and strategies.

In these previous studies, the LA was the primary approach adopted. LA involves the “measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Learning Analytics & Knowledge [LAK], 2011). LA-related research is interested in exploring the data for more interpretable and understandable models by human-led methods and looking for ways to inform instructors and learners of the results (Baker & Siemens, 2014). LA can be used to understand how learning design impacts not only on outcomes but also on learning processes (Ifenthaler et al., 2021). This study adopted the LA approach to collect and analyze students’ learning behaviors, represent their general learning process during the whole course, and explore the expected instructional design framework for instructors and learners.

In online courses using e-book systems, LA shows great potential to capture user-system interactions, such as automatically collecting users’ reading behaviors with specific strategies, such as critical reading (Majumdar et al., 2021). Some functional tools in the e-book system, such as markers and bookmarks, can support students’ reading strategies (Ogata et al., 2017). For example, a marker has always been considered an effective tool for promoting comprehension and learning performance (Bernacki et al., 2012) since highlighting contents can engage students in more cognitive processing, including deciding what contents to focus on or which contents require a special effort to understand (Yue et al., 2015).

LA approach was also adopted to understand students’ CPS learning behaviors, especially their cognitive processes. For example, Kwon et al. (2018) identified students’ learning behavior patterns during inquiry tasks and explored the relationship between learning behaviors and students’ domain knowledge. They identified the important inquiry behaviors that should be encouraged during the problem-solving process. Andrews-Todd and Forsyth (2020) sought to clarify cognitive and collaborative processes via a qualitative analysis that assessed students’ online chat contents and used the LA approach to explore problem-solving processes during online simulation-based tasks.

CPS-based STEM lessons included CPS activities as well as individual learning, including previewing, reviewing, and listening to the teacher’s instruction. In CPS-based STEM education, understanding and analyzing students’ learning behaviors is essential for improving learning performance. While previous studies focused on cognitive processes using the LA approach, this study extends the focus to a broader learning activities and processes. By collecting students’ learning data during the whole course, this study aims to explore the effective behavioral factors that affect learning performance.

Therefore, the third hypothesis is: Students’ learning behaviors affect their learning performance in CPS-based STEM education.

Moreover, since it is important to put learning strategies into practice to achieve success in learning (Zimmerman, 2002), learning behaviors, as external manifestations of students’ mastery of strategies, are considered related to learning strategies. Thus, we establish the fourth hypothesis: In CPS-based STEM education, students’ learning strategies and behaviors are correlated and have combined effects on learning performance.

Methods

Participants

This study was conducted in a seventh-grade science class at a private junior high school in Shanghai, China, originally with 114 students aged 14 to 15 years. The students came from four classes but were allocated to the same science teacher. The teacher, all the students and their parents signed the consent forms, except for one student (whose data were not collected). Ultimately, data from 106 students were collected for the study due to the absence of several students and missing questionnaire data. Among the participants, 50 were male (47.2%) and 56 were female (52.8%). This study passed an ethical review conducted by the author’s university.

Design of CPS-Based STEM Course

Course Theme

The theme of the course was “Solutions Around Us,” which included four sub-themes. Three sub-themes, (a) The Formation of Solutions, (b) Various Types of Solutions, and (c) Using Solutions Safely, were designed according to the paper-based learning materials in students’ usual classes. A sub-theme, (d) Limnic Eruption, was added based on related content. The sub-themes were addressed on a weekly basis.

The learning materials and curriculum outline of this multidisciplinary science course are unified in Shanghai. The “Solutions Around Us” theme is the first unit in the seventh-grade science course. This theme, which was in line with the teaching schedule of the school, was used to help students understand and construct scientific knowledge using examples from their daily lives. In the Limnic Eruption sub-theme, students had to determine the mechanism of a natural disaster, clarify the relationship between the natural environment and human beings, and reflect on how to protect themselves and others when facing such unavoidable disasters. This theme was included in our previous studies (Chen et al., 2019, 2020) and was revised according to students’ proficiency levels.

Design of CPS-Based STEM Lessons Based on CPS Framework

The lesson procedure was as follows. First, the students were required to read the learning materials in advance as their preview work to help them conduct the individual thinking needed to understand the contents. During group work, students were instructed to solve problems following the CPS framework.

The CPS framework, proposed by Hesse et al. (2015), aims to develop learners’ CPS skills at a teachable level based on the distinction between social and cognitive skills as well as the interventive use of those skills in two domains. The CPS framework includes Identifying the problem, Representing the problem, Planning and executing, and Monitoring and Reflecting. The CPS processes in STEM lessons are presented in Figure 1.

Figure 1.

CPS processes in STEM lessons.

First, students used cognitive skills to analyze the elements of and hidden information in a problem and identify the discrepancy between the current and expected states. The purpose of this process was to inform other members of this discrepancy using social skills. The students then used graphs, tables, or simple words to represent the information to the group using cognitive skills. The students used social skills to communicate or negotiate to make the individual representations similar across the groups.

After identifying and representing the problem, the students clarified the goal and established a shared plan by managing information and resources. During this process, they needed to share the information within the group and ensure that each member had the same knowledge and understanding. Finally, the group evaluated its progress and results, revised or reformulated the plan if necessary, and decided how to proceed.

Functional Tools for Supporting STEM Learning Strategy

Before the course began, the following STEM learning strategy (Griese et al., 2015) was taught through the BookRoll system in order to help students construct knowledge and understanding in integrative STEM education:

Organizing: Organizing and summarizing the important points.

Elaborating: Connecting new scientific facts with earlier ones or practical applications.

Repeating: Learning and remembering scientific facts through repetition.

Attention: Concentration on learning science and solving problems.

Time management: Conducting individual learning or group work according to a schedule.

Peer learning: Collaborating with others when learning science and solving problems.

Using references: Using references for additional information.

Digital learning materials were provided through the BookRoll system along with the following functional tools: Next, Previous, and Page Jump buttons (for changing pages), a Marker tool (for highlighting contents), an Annotation tool (for adding annotations), and a Bookmark tool (for posting bookmarks; Ogata et al., 2017).

Students were instructed to use these functional tools based on STEM learning strategies during both individual learning and group work. For example, during individual learning, students could use the Next and Previous tools to read the learning materials repeatedly (Repeating strategy), the Marker/Annotation/Bookmark tools to organize and summarize the important points (organizing strategy), connect new facts with earlier ones (elaborating strategy), and references provided by the system (using references strategy). During group work, students were asked to share the contents to which they had added markers, annotations, or bookmarks, which aimed to help them understand the state of other members and the group. They were also asked to delete markers, annotations, or bookmarks when they changed their ideas, which aimed to help them conduct reflection. The descriptions of functional tools related to learning strategies are presented in Table 1.

Table 1.

Descriptions of Functional Tools Related to Learning Strategies.

Functional tool	Learning behavior	Learning strategy
Next	Turning to next page	Repeating (passive rehearsal strategy)
Previous	Turning to previous page to read the contents again
Page Jump	Jumping to a specific page for specific information
Add marker	Highlighting the texts when they are deemed important or not understood	Organization, Elaboration (active rehearsal strategy)
Del marker	Deleting the highlighted texts after reflection	Reflection (metacognitive strategy)
Add Annotation	Summarizing the important points or taking notes about interpretations of them	Organization, Elaboration
Delete Annotation	Deleting and adding new annotations after reflection	Reflection
Add Bookmark	Posting a bookmark on a page when it is considered important	Organization, Elaboration
Delete Bookmark	Deleting bookmarks after reflection	Reflection

In order to ensure that students have understood and mastered how to use these strategies, in the first week, students were required to practice using these functional tools according to the teacher’s instructions. During this week, students were asked to use the BookRoll system to preview the scientific contents as well as discuss the contents during group work. Students’ appropriate use of learning strategies and functional tools was supported by the teacher and their group members.

The STEM learning strategies were considered as a part of CPS-based STEM lessons in addition to the CPS framework. Considering the integrative nature of CPS-based STEM lessons, it is necessary to integrate the CPS framework and STEM learning strategies when conducting instructional designs. In this study, we collected students’ learning data regarding how to use CPS skills and learning strategies to explore what factors and how these factors affected the effectiveness of the lessons and to construct the expected instructional design framework that considered the integrative effects of these factors.

Research Procedure

The study consisted of a four-week course in September 2019. Before the course, students were required to respond to a pre-questionnaire regarding their prior awareness of CPS and their use of STEM learning strategies when they participated in typical science lessons, as well as participate in a pre-test that checked their prior knowledge of the topics about which they were about to learn.

Since it was the first time the students accessed the BookRoll system, the activities in the first week were designed to familiarize them with the system. Thus, the data collected during this week were not included in the final data analysis and results. The students took their CPS-based STEM lessons from the second to the fourth week. After each weekly lesson, students took a comprehension test to check for changes in their learning performance. After the course, students were required to take the same CPS and STEM questionnaires as their post-questionnaires. The procedure used in this study is presented in Figure 2.

Figure 2.

Research procedure.

Data Collection and Analysis

In CPS, the success or failure of the application of knowledge to solve problems is related to students’ cognitive and collaborative strategies use (Hesse et al., 2015), their prior knowledge, their ability to connect what they do and do not know, and the cognitive pattern of their learning processes (Dinsmore et al., 2014). In this study, we sought to identify the factors affecting learning performance in CPS-based STEM lessons by examining variables reflecting the students’ subjective data, such as CPS awareness and use of STEM strategy, objective data, such as cognitive learning behaviors collected by the system, and dialog data collected during discussions. The data were collected from tests, questionnaires, learning logs, and dialogs.

Learning Performance

The pre-test, comprising tests 1 to 4, examined students’ prior knowledge of the contents they would learn over the next four weeks. After each week’s lesson, students were required to take post-tests 1 to 4. The questions in tests 1 to 3 were drawn from the science workbook used by the experimental school, which aimed to assess the student’s knowledge about the formation, characteristics, and application of solutions. The items in test 4, designed in collaboration with the teacher of the course, included conceptual questions related to the nature of the solutions and solubility of CO₂ and two application questions that required students to solve problems related to the solubility of CO₂ and disaster-reduction consciousness. All the items were discussed with and checked by the teacher based on the students’ proficiency levels. For example, one application question in test 4 was “What do you think we should do from an individual standpoint to reduce the damage caused by natural disasters?” which required students to think about the formation and nature of CO₂, as well as reflect on the influence of natural disasters.

The pre- and post-test scores used in this study were the sum of all questions in both tests. The differences in scores between the pre- and post-tests were calculated to indicate the changes in students’ understanding of the contents and their ability to use it to solve problems. Since the activities in the first week were designed to familiarize students with the system, pre-post 1 was not analyzed. There were 13 questions (58 full marks) in test 2, eight questions (31 full marks) in test 3, and three questions (11 full marks) in test 4.

CPS Awareness

Students’ awareness of the quality and processes of their collaborative and cognitive learning activities was examined by the CPS questionnaire, which was designed based on the framework of CPS processes (Hesse et al., 2015). The CPS pre and post questionnaires contained two dimensions, composed of five factors and 17 items: the social skills dimension, which included participation (three items), perspective taking (three items), and social regulation (three items); and the cognitive skills dimension, which included task regulation (four items), and learning and knowledge building (four items).

Our previous studies indicated that the CPS questionnaire has good reliability (Chen et al., 2019, 2020). We also assessed its reliability in this study. The overall Cronbach’s α value of the pre-CPS questionnaire was .95, and the reliability values of participation, perspective taking, social regulation, task regulation, and learning and knowledge building were .83, .87, .81, .72, and .80, respectively. The overall Cronbach’s α value of the post-CPS questionnaire was 0.96, and the reliability values of the five factors were 0.87, 0.87, 0.84, 0.71, and 0.86, respectively.

Use of STEM Learning Strategy

Students’ perceptions of how to utilize their learning strategies during STEM learning were measured using the STEM Learning Strategy Questionnaire (SLS Questionnaire); this was developed by Griese et al. (2015), who proved its reliability. The pre and post SLS questionnaires contained 9 factors and 36 items: organizing (three items), elaborating (five items), repeating (six items), effort (five items), attention (three items), time management (three items), learning environment (five items), peer learning (three items), and using references (three items).

Our previous study indicated that the SLS questionnaire has good reliability (Chen et al., 2019). We translated this questionnaire into Chinese and made minor changes to its items to make them more suitable for the study’s student participants. The translated questionnaires were checked by an expert and the teacher who conducted the lessons. The reliability of the SLS questionnaire was examined. The overall Cronbach’s α value of the pre-SLS Questionnaire was .96, and the reliability values of organizing, elaborating, repeating, effort, attention, time management, learning environment, peer learning, and using references were .86, .86, .88, .78, .84, .84, .84, .84, and .79, respectively. The overall Cronbach’s α value of the post-SLS questionnaire was .96, and the reliability values of the nine factors were .86, .86, .90, .85, .71, .84, .84, .84, and .77, respectively. Both the CPS and SLS questionnaires had a five-point response scale (“strongly agree” = 5; “agree” = 4; “neutral” = 3; “disagree” = 2; and “strongly disagree” = 1).

Learning Behaviors

Nine types of learning logs on the BookRoll system were collected to represent the following learning behaviors. The descriptions of learning behaviors are listed in Table 1 in 3.2.3.

Dialog Analysis

Before the lecture, a voice recorder was put in the middle of each group, and students were informed about the recording. In order to reduce the interference of recording their discussion, the voice recorders were put in inconspicuous places such as behind students’ computers. All dialog threads were coded by the first author, based on the discussion with one expert (the fourth author in this study), into five CPS sub-skills according to Hesse et al.’s (2015) framework.

Data Analysis Process

This study used a mixed-method approach, including quantitative and qualitative analysis. First, a paired sample t-test was used to assess the learning effectiveness of CPS-based STEM lessons by examining the changes in learning performance between pre- and post-tests 2 to 4, worth 100 marks (RQ1). Second, we conducted path analysis to examine the contribution of cognitive learning behaviors and changes in STEM learning strategy and CPS awareness, as well as how these factors affected learning performance (RQ2). Finally, we conducted a dialog analysis to investigate how students used their CPS skills to improve understanding and solve problems (RQ3).

Results

The Improvement of Learning Performance

Several students were absent from one or more lessons during the course. Although these students completed the pre- and post-tests, their test scores were excluded from the data analysis. As Table 2 shows, the results of the paired sample t-test indicate that the post-test score (M = 64.75, SD = 15.40) showed a statistically significant improvement over the pre-test (M = 24.22, SD = 10.33), t(105) = 29.63, at a significance value of .000 < .001. The effect size was d = 2.88, at a 95% confidence interval (CI) [2.69, 3.49].

Table 2.

Paired Sample t-Test Results of Pre- and Post-Tests.

Pre-test		Post-test		df	t value	p Value
Mean	SD	Mean	SD	df	t value	p Value
24.22	10.33	64.75	15.40	105	29.63^***	.000

Note. N = 106; d = 2.88; ***p < .001.

Path Analysis

We conducted a path analysis to explore the factors affecting learning performance improvement. First, the indicators of model fitting were evaluated. The indicators χ²(46) = 57.807, CFI = 0.974, TLI = 0.967, RMSEA = 0.049, and AIC = 10,830.493 indicated that the model fit the data well. All paths met the significance level of p < .05.

Each factor of CPS awareness and STEM learning strategy was the average difference between the pre and post questionnaires. The exogenous variables, which are independent variables and are not affected by other variables, are changes in STEM strategy use and learning behaviors. The endogenous variables include the mediator and the dependent variable. The endogenous mediator is the change in CPS awareness observed between changes in STEM strategy use and learning performance improvement, and the dependent variable is learning performance improvement. As Figure 3 shows, the results indicate that the factors of learning behaviors and changes in CPS awareness had direct effects on learning performance improvement, and changes in STEM strategy use had indirect effects on learning performance improvement, which was mediated by changes in CPS awareness. The statistical result for each significant variable is presented in Table 3.

Figure 3.

Path analysis results of the behavioral and strategic effects on learning performance.

Table 3.

Statistical Result of the Path Analysis.

Dependent variable	Independent variable	Standarized coefficient	SE	z-Value	p-Value
Direct effect
Learning performance improvement	Task_cps	0.17	0.08	2.00	.046
	Next	0.24	0.09	2.67	.008
	Add Marker	−0.31	0.12	−2.69	.007
	Delete marker	0.25	0.12	2.16	.031
	Add bookmark	0.37	0.13	2.72	.007
	Delete bookmark	−0.42	0.14	−3.02	.003
Indirect effect
Participation_cps	Perspective_cps	0.35	0.09	4.03	.000
	Social_cps	0.40	0.09	4.60	.000
	Effort_sls	0.21	0.06	3.50	.000
Task_cps	Social_cps	0.29	0.09	3.34	.001
	Knowledge_cps	0.37	0.08	4.36	.000
	Peer_sls	0.29	0.08	3.83	.000
Peer_sls	Environment_sls	0.66	0.05	13.80	.000
Next	Previous	0.89	0.02	59.54	.000

Direct Effects of CPS Awareness on Learning Performance

First, regarding the effects of the changes in CPS awareness, the results showed that an increase in task regulation (Task_cps) awareness promoted the improvement of learning performance, meaning that students’ learning performance was improved if their task regulation in CPS awareness increased after the course. However, other factors of CPS awareness had no direct effect on learning performance improvement. Among the five factors, the increase in social regulation (Social_cps) and learning and knowledge building (Knowledge_cps) showed indirect positive effects on learning performance improvement, mediated by an increase in task regulation. Dialog analysis was conducted to explore why task regulation awareness affected learning performance improvement and how students used CPS skills. The results are discussed in RQ3.

Behavioral Factors Affecting Learning Performance

The descriptive data for learning behaviors is presented in Table 4. Next and Previous were the two most frequent behaviors, followed by Add Marker and Delete Marker. Compared with the marker tool, students used the bookmark tool less frequently. The learning behaviors Next, Delete Marker, and Add Bookmark showed direct positive effects on learning performance improvement. When students conducted these behaviors more frequently, their test scores were more likely to improve. Moreover, Previous showed a positive effect on learning performance improvement when mediated by Next. On the other hand, the results also revealed that Add Marker and Delete Bookmark had negative effects on learning performance improvement, suggesting that frequently adding markers or deleting bookmarks may have been harmful to the students’ test scores.

Table 4.

Descriptive Data for Learning Behaviors.

Learning behavior	M	SD	Minimum	Maximum
Next	161.84	81.28	19	490
Previous	78.08	64.60	1	335
Page Jump	10.75	12.49	0	98
Add Marker	42.09	38.66	5	217
Delete Marker	15.54	26.08	0	180
Add Annotation	8.79	7.14	0	30
Delete Annotation	0.56	1.03	0	5
Add Bookmark	3.95	7.10	0	55
Delete Bookmark	1.92	4.07	0	28

We investigated why the two pairs of learning behaviors, Add Marker and Delete Marker, Add Bookmark, and Delete Bookmark showed different effects on learning performance through dialog analysis. Regarding the use of the marker tool during their discussion, when students used task regulation skills to collect information on the learning materials, they tended to highlight the texts first and then show them to others. However, many of the contents they had highlighted were judged as “wrong” or “meaningless” by other students. For example, in the following dialog, S1’s behavior of highlighting important contents was judged as “meaningless” by S2, and S1 was advised to reconsider the contents:

14. S1: I highlighted all the contents on several pages. All of them are important.

15. S2: No, this is meaningless. You should understand the contents first. It should be noted that this is nitrogen oxide. Do you understand that?

As for the use of the bookmark tool, students tended to post bookmarks on pages they wanted to save to share information with others or to save the pages they thought would be useful for the presentation. For example, in the following dialog, S4 used a bookmark to find the specific content they searched for and shared it with S3:

68. S3: I remember I have seen the contents somewhere.

69. S4: I remember I have seen the contents two. Let me check the bookmark. Here, I have found the page.

Strategic Factors That Affect Learning Performance Mediated by CPS

Changes in peer learning (Peer_sls) strategy positively affected on learning performance improvement, mediated by changes in task regulation awareness. Peer learning strategy is the act of collaborating during STEM learning (Griese et al., 2015), which is similar to social regulation in CPS. This result was consistent with the direct effects in the path analysis, which showed that task regulation awareness could be enhanced by the effective use of cognitive and metacognitive strategies through collaboration.

Dialog Analysis

The path analysis was conducted to explore how students used their CPS skills to solve problems and to examine whether students’ subjective awareness in path analysis was consistent with their objective actions. The results revealed that the changes in task regulation in CPS awareness directly affected learning performance improvement. Dialog analysis was thus conducted to identify the use of CPS skills. In this study, we focused on the mutual effects related to task regulation skill use (see Figure 4) to determine why task regulation directly affected learning performance and how it mediated other factors.

Figure 4.

CPS processes in STEM lessons.

Any two factors can have a causal relationship, sequential relationship, or display integrative use. In a causal relationship, a student would use one CPS skill for a specific purpose and then use another skill to achieve the same purpose. In a sequential relationship, a student would use two or more skills in one activity. In an integrative use, a student would use two or more skills simultaneously or repeatedly. For example, in task regulation and participation, students used participation first and then used task regulation (sequential relationship), or they used both skills (integrative use). In task regulation and perspective taking, students used task regulation to analyze the problem and then used perspective taking to achieve the same purpose (causal relationship).

Among these relationships, it was found that the integrative use of task regulation and learning and knowledge building skills and participation skills were effective in fostering understanding. Three examples of using task regulation skill with learning and knowledge building skill and participation skill are presented to show how the students constructed knowledge using these skills and strategies.

Table 5 describes the elements of task regulation, learning, and knowledge building, and participation skills. However, we did not code for the element of action in participation. The action element includes all activities in a collaborative environment; therefore, it is difficult to identify the relationship between the action element and other elements.

Table 5.

Descriptions of Three CPS Sub-Skills and Elements.

Sub-skill	Description of sub-skill	Element	Description of element
Task regulation (Cognitive domain)	Ability to analyze the problem, manage resources, set clear goals, collect information, and seek various solutions to complex situations.	*Problem analysis	Analyzing and describing a problem in familiar language
		Sets goals	Setting a clear goal and sub-goals for a task
		*Resource management	Managing resources or people to complete a task
		Flexibility and ambiguity	Accepting or adapting ambiguous situations (for example, multiple solutions for a problem)
		*Collects elements of information	Exploring and understanding elements of the task, what information is useful, and how to use this information
		Systematicity	Implementing possible solutions to a problem and monitoring progress
Learning and knowledge building (Cognitive domain)	Ability to identify relationships between pieces of information, integrate knowledge from other fields or subjects, monitor outcomes, reflect on processes.	*Relationships	Identifying relationships between and among elements of knowledge.
	*Rules: “If…then”		Setting a plan based on an understanding of the cause and effect of knowledge.
	Hypothesis “what if…”		Setting and testing hypotheses using a “What if…?” approach.
Participation (Social domain)	Engagement with the tasks, the extent to which they attach importance to others’ opinions and interact with others.	Action	The activities in the collaborative environment. For example, learners conducting activities in familiar or unfamiliar contexts actively.
		*Interaction	Responding to others’ contributions and interacting with others during a collaborative activity.
		Task completion	Conducting and completing a task or part of a task by attempting different strategies.

Indicates the element identified

Example 1: Task regulation→Learning and knowledge building:

17. S3: What is acid rain? Is it related to its pH value? It is written here. pH < 5.6, right?

18. S1: Yes, I think it’s difficult, in winter, the acid rain.

19. S3: I think the news is fake. At that time, the Icelandic volcano erupted and formed the, what was that called? That is, when it reaches 7,000 to 10,000 meters in the atmosphere, I think the pH value can’t reach 5.6.

20. S2: Yes, so it can’t form the acid rain.

21: S3: On the other hand, even the acid rain is formed, when you go out, your skin might be infected by the virus. No, I mean, the skin disease.

22. S2: So we shouldn’t go out.

23: S3: If there such acid rain happens, I suggest not to go out. But I don’t know, whether the acid rain will be formed. I think we should examine the pH value, to see whether it is 5.6. If the result shows it isn’t acid rain, or won’t cause such disease, it’s ok. But it is better to take protection when going out. It is said that the probability of skin diseases may be very high. It’s the probability, we don’t know, so we should examine it.

This dialog occurred on the theme of (c) Using Solutions Safely, dealing with the problem of the harmfulness of acid rain. This example shows the relationships between the use of the “Problem analysis” skill and “Collects elements of information” skill in task regulation and the “Rules: If…then” skill in learning and knowledge building.

In Line 17, S3 analyzed the main point of the problem and confirmed the definition of acid rain according to the information in the learning materials. Then, S3 provided the subsequent explanations based on the definition of acid rain (“Problem analysis” and “Collects elements of information” skills).

In Line 19, S3 inferred that the pH value could not reach 5.6, also based on information in the learning materials (“Collects elements of information”).

In Line 23, S3 understood the main points of the problem: measuring the pH value of the rain, the causes and effects of the pH value, and the harmfulness of acid rain. Therefore, S3 set a plan to examine the pH value first and then judge whether the rain was harmful and what kind of diseases it would cause.

In this case, students used the task regulation skill to analyze the points and knowledge of the problem. They used learning and knowledge building to construct knowledge based on their understanding of the problem (Rules: “If…then”).

Example 2: Task regulation↔Learning and knowledge building:

44. S1: Why did carbon dioxide erupt?

45. S2: Just like the Cola. The carbon dioxide in the magma is released into the air, causing the water in the lake to be pressed out, and then carbon dioxide erupts into the air, making people unable to breathe.

46. S1: If there is no change in the lake, it won’t erupt, right?

47. S2: Oh, I got it, I got it. That is, there are volcanic lakes below it, and then there is magma below it. And then the magma released carbon dioxide gas, and then because there was a rock above it, it is said that the gas cannot be released. It bursts out. Then this lake which is dissolved a large amount of carbon dioxide erupted.

48. S1: So it’s because of the earthquake? The carbon dioxide in the magma.

49. S3: What earthquake?

50. S2: It is written here [the learning material].

51. S1: It is said that may be an earthquake occurred.

52. S2: So, then, the carbon dioxide in the magma would be full of the lake, because the top is closed. That is, the hole, the rock, I mean the eruption port, are closed. Because the hole for the eruption was closed, just like you put the cap on a Cola bottle, the earthquake is just like shaking the bottle, and then it will erupt.

53. S1: Yeah, that’s what I mean.

This dialog occurred on the theme of (d) Limnic Eruption. In this example, students repeatedly used the “Collects elements of information” skill in task regulation and the “Relationships” skill in learning and knowledge building.

In Lines 45 and 47, S2 explained the Limnic Eruption mechanism by using the example of the Cola, which they were familiar with (“Relationships”).

In Lines 48 to 51, the students connected the information from the learning material with S2’s explanation (“Collects elements of information”). In Line 52, S2 supplemented his explanation based on the information they had discussed (“Relationships”).

In this case, the students used task regulation to seek information and learning and knowledge building to make connections between the current information and familiar knowledge repeatedly to understand the mechanism of the Limnic Eruption.

Example 3: Task regulation↔Participation:

32. S1: Let me find the information for you.

33. S3: Yes, you open this page [of a learning material] on your screen, I’ll open another one, then let’s discuss.

34. S1: I have found it for you. Look, here [content of the learning material].

35. S3: Great.

36. S1: [Read the content] At the bottom of Lake Nyos, a large number of magma deposits have accumulated.

37. S3: So, so it killed people.

38. S1: But it didn’t kill people. Look at here.

39. S3: It is carbon dioxide poisoning.

40. S1: But it didn’t kill people. Look at here. It only killed animals, here. It only killed animals, right? Look at here. It only killed animals.

41. S3: It also killed people, and it was because of carbon dioxide poisoning.

This dialog also occurred on the theme of (d) Limnic Eruption. In this example, students used “Resource management” and “Collects elements of information” skills in task regulation and “Interaction” in participation.

In Lines 32 to 34, S1 and S3 conducted a simple discussion to decide how to open the provided learning materials to find the relevant information (“Resource management”). In this activity, they did not just divide their labor in opening the learning materials but also discussed how to display it, ensure the readability of the information, and help each other to find the necessary information (“Interaction”).

In Lines 40 to 41, after managing the available resources, S1 found an inconsistency between one element of the problem (the problem described that the lake killed people) and the information in the learning material (S1 found only the information about the animals). S3 recognized S1’s incorrect understanding of the problem and the provided information and corrected it (“Resource management” and “Collects elements of information”).

In this case, the students first used task regulation and participation to decide how to manage the available materials for problem solving within the group. They then used task regulation to analyze and understand the information in the learning materials.

The above analysis identified the integrative use of task regulation with other CPS sub-skills for understanding, indicating that the students used several CPS sub-skills as a set or used CPS skills integratively rather than separately.

The three dialog examples primarily focus on CPS skills, while the influence of STEM strategies use was also observed. For example, in the first example, while S3 was employing Task regulation and Learning and knowledge building skills, he/she also relied on the conversation with others to proceed the task, which is related to Peer learning in STEM strategy. The second example presents an integration use of CPS skills and STEM strategies, where S2 organized the thoughts while answering questions to others and paid attention to the importance of learning materials. In the third example, the group discussion placed significant emphasis on the role of learning materials (related to the Environment in STEM strategy). We will further discuss these relationships in the subsequent Discussion section.

Discussion

This study investigated the effects of a CPS-based STEM course on student’ learning performance. Data from pre- and post-tests validated the hypothesis of learning performance improvement (RQ1). Further investigation through path analysis explored the complex causal relationships between strategic and behavioral factors in learning performance improvement (RQ2). Additionally, dialog analysis (RQ3) provided evidence on the integrative utilization of CPS skills and STEM strategies, delineating a comprehensive framework. In the discussion, we interpreted the results, analyzed the implications and reflected the methodologies.

What Is the Difference in Learning Performance Between the Pre- and Post-Tests in CPS-Based STEM Lessons?

Learning performance was significantly improved after the CPS-based STEM lessons, indicating the effectiveness of students’ acquisition of STEM knowledge.

Students were required to take pre-tests before the course and to take each post-test immediately after the weekly lesson. The students were also required to make a presentation on their group’s solution as their final activity. They then received feedback from the teacher and other students, which allowed them to revise errors in their knowledge and understanding. This ensured that their learning performance improved, which was one possible reason for the large effect size. However, considering the integrative and complex nature of CPS and STEM, it was not clear what instructional design factors affected learning performance and whether the effects were positive or negative.

This study integrated some social, cognitive, and metacognitive strategies related to CPS and STEM into the lessons, which were considered effective in knowledge acquisition. For example, cognitive strategies of Repeating, Organizing, and Elaborating can help students to remember the knowledge, understand the contents, and apply a new principle to sample problems by repeating reading, comparing similar materials, and connecting different concepts and information (Weinstein et al., 2011); metacognitive strategies of Time Management could help students to set plans and monitor the learning activities (Claessens et al., 2007); collaborative strategy of Peer Learning could help students to conduct cognitive and metacognitive strategies by teaching the materials to others or discussing and analyzing materials with others (Weinstein et al., 2011).

However, it is difficult to distinguish the effects of CPS and STEM learning strategies on the improvement of learning performance. Therefore, we conducted path analysis to explore the factors affecting learning performance improvement, including CPS awareness about how to use CPS skills and how to use STEM learning strategies and the learning behaviors during CPS-based STEM lessons.

What Are the Strategic and Behavioral Factors That Improve the Learning Performance in CPS-Based STEM Lessons?

The learning behaviors Next, Delete Marker, and Add Bookmark showed positive effects, while Add Marker and Delete Bookmark had negative effects on learning performance improvement.

Turning to the next or previous pages implies that students are repeating their reading, which is related to the repeating and passive rehearsal strategies (Weinstein et al., 2011). In this study, students who conducted these behaviors to read and memorize the contents repeatedly tended to achieve better learning performance.

Highlighting content was more effective than simply reading and memorizing it in promoting comprehension and learning performance (Bernacki et al., 2012). This behavior can be related to an active rehearsal strategy and elaboration strategy, in which students use more cognitive processing, including deciding what part of the content they should focus on and what part they should make a special effort to understand (Nesbit et al., 2006; Weinstein et al., 2011; Yue et al., 2015).

However, in this study, Add Marker behavior negatively affected students’ learning performance improvement, which was inconsistent with previous studies. Some situations in which the marker tool is ineffective were considered. For example, if students need help identifying what content to highlight, this simple act of highlighting is not beneficial for cognitive processing (Yik et al., 2018). In this study, Add Marker behavior occurred more often in students’ individual learning (preview and review works), wherein students were instructed to use marker tools to identify important and unfamiliar contents. If students in this study could not identify whether the contents were important or not, or whether the contents deserved to be discussed, the marker tools were not helpful during the preview and review work. Delete Marker behavior occurred most often in two situations. First, the students added markers to particular contents and deleted markers they had added incorrectly. Second, during the discussion, students were required to share the contents they had highlighted and their annotations and to discuss the contents. Students deleted markers after taking other students’ contributions into consideration, and reflected on their former understanding. In both situations, students’ deleting marker behaviors were related to their monitoring and reflecting, which were crucial processes in CPS and STEM.

Another cognitive tool, the bookmark, was also proved to be positively associated with students’ learning scores (Van Horne et al., 2016). The bookmark tool was easier to use than the marker because students did not need to decide the key points of the content. We found that many students chose to add bookmarks on the pages the teacher said were important. Therefore, the teacher’s instruction was one explanation for the students’ effective use of the bookmark tool.

The descriptive data for learning behaviors shows that students used the marker tool most frequently among all the functional tools, especially the Add Marker behavior. We analyzed students’ discussions to understand why adding marker and deleting bookmark behaviors showed negative effects. The results indicated that students tended to add markers to the meaningless content because they had no idea what to highlight or just thought this behavior was interesting. However, they deleted markers more carefully than they added markers. Students usually deleted markers after they accepted others’ opinions and revised their own understanding, which could be related to the reflection strategy. It was found that sharing and discussing their initial understanding of the scientific contents during the students’ group discussions was effective for new knowledge building.

As for the bookmark, students tended to post it on pages that they wanted to share with others or for the final group presentation. Using bookmarks is an effective way to locate and organize specific information during discussions. However, since students tended to add bookmarks following others’ advice or the teacher’s instructions, their reflection was limited, which could explain the negative effect of deleting bookmarks. In this study, the frequency of Delete Bookmark is much lower than Delete Marker, which proved students’ less reflective behavior when using bookmarks.

Regarding the strategic factors, changes in peer learning strategy showed a positive effect on learning performance improvement, mediated by changes in task regulation awareness. Planning is a core activity in task regulation, which is related to the cognitive domain; this includes problem analysis, goal setting, resource management, and reflection to achieve adequate planning complexity (Hesse et al., 2015; OECD, 2017). This activity initiates the problem-solving process, in which students need to use several cognitive and metacognitive strategies in the shared problem-solving space. Moreover, the students’peer learning strategy was enhanced if they were more willing to study in environments where it was easier to concentrate or find references related to the learning environment of STEM strategy (Griese et al., 2015).

The above results provide new insights about which factors of learning behaviors and STEM learning strategies are most effective and should be considered in CPS-based STEM lessons. Considering the indirect effects of STEM strategies, teaching STEM learning strategies incorporating collaborative and cognitive processes is more effective than teaching STEM learning strategies independently.

How Do Students Use CPS Skills to Improve Their Understanding in CPS-Based STEM Lessons?

In RQ1 and RQ2, we found that students’ learning performance was significantly improved, and the improvement was affected by some behaviors and strategic factors either directly or mediated by CPS awareness. In RQ3, we explored students’ actual use of CPS skills to solve problems and found the effectiveness of using task regulation skills on students’ knowledge building and understanding. It is found that students used task regulation skills, along with the other four sub-skills as a set, to understand the information and then construct new knowledge. Compared with the results of path analysis, the dialog analysis indicated that the use of CPS skills showed mutual effects on each other rather than a simple one-way causal relationship in the path analysis results.

According to the result of path analysis, the element of “collects elements of information” of task regulation was used in all three examples in dialog analysis. This element focused on selecting and utilizing the information, including the information provided through the learning materials and new information students searched for from other resources. In task regulation, planning is one of the core activities (Gunzelmann & Anderson, 2003). In this study, planning activities involve discussing the problem and specific information to understand the expected state of the problem and the missing information and dividing the labor within a group toward a common goal. During this process, students also conducted metacognitive activities to reflect on their cognitive processes (Hayes-Roth & Hayes-Roth, 1979). ÿ£¢Monitoring and reflection are important for cyclical problem solving behaviors, which can help students be aware of the sufficient and insufficient parts of the group’s progress. In this study, to understand the problem and its related contents, students also used other cognitive strategies to understand the elements of knowledge and collaborative strategies of interaction with others. The results indicated that the interactive use of cognitive, collaborative, and metacognitive strategies was effective in knowledge acquisition.

In addition, the influence of STEM learning strategies on using CPS skills was also observed. In these three examples, the peer learning STEM strategy is primarily utilized to respond to peers’ questions and discuss learning materials and the environment STEM strategy is mainly reflected in their focus on instructional resources. In the second example, S2 exhibited a more comprehensive integration of CPS skills and STEM strategies use. When S2 received questions from another student, S2 used learning and knowledge building CPS skill to answer the questions, which helped S2 be inspired to clarify and further refine his/her thoughts. Thus, although S2 was the primary user of CPS skills in this dialog, the effectiveness of these skills was closely related to questions and interactions initiated by peers. Moreover, when the group discussed the problem using task regulation of CPS skill, S2 specifically cited content from the learning materials, highlighting the importance of the resource. This indicated the influence of environment and peer learning STEM strategies on using task regulation skill.

Conclusions

This study designed a CPS-based STEM course for seventh-grade students to help them improve their learning performance and CPS skills. First, the results of the pre- and post-tests show that students’ learning performance improved after the course (RQ1). This indicates that the design of the CPS-based STEM course effectively improved learning performance.

Second, path analysis was conducted to identify the behavioral and strategic factors affecting the improvement in learning performance (RQ2). The results reveal the direct effects of changes in task regulation awareness in CPS and several behavioral factors, including changing pages, deleting markers, and adding bookmarks after accepting others’ opinions, on learning performance improvement. Among the strategic factors, peer learning improved learning performance, mediated by task regulation awareness. Thus, these factors should be considered in CPS-based STEM design, and students should be provided with appropriate guidance.

Third, a dialog analysis conducted to investigate how students used their CPS skills (RQ3) indicates that students tended to use CPS skills integratively when seeking an understanding and solving problems. The path and dialog analyses show that task regulation is an important factor in improving learning performance when other social and cognitive CPS skills are used.

This study has several implications. First, it shows the possibilities of using the LA approach to understand students’ learning processes across the whole CPS-based STEM lessons, including cognitive problem solving, collaborative learning, as well as individual cognitive and metacognitive learning. Second, the complex and integrative nature of CPS learning and STEM fields makes it difficult for teachers to consider all the essential factors in CPS-based STEM course design. This study integrated the CPS and STEM strategies, refined the effective factors, and provided an expected instructional design involving the whole learning process.

Overall, this study shows the effectiveness of using the LA approach to understand and enhance student learning by incorporating integrative instructional theory and data analysis insights. The versatility of LA allows its application across various learning environments and materials. For example, in more interactive or technologically advanced classrooms, LA could be used to tailor teaching methods to individual learning styles detected through data patterns. Even in traditional educational settings, LA insights might guide the strategic use of traditional materials to maximize learning outcomes. Our findings indicate potential generalizability in other contexts, however, the effectiveness of the generalizability depends on the understanding of underlying LA principles. LA approach can be customized to meet the needs of different educational environments, such as designing personalized learning, identifying learning gaps, or maximizing collaborative learning opportunities.

Our study has several limitations. First, this study was conducted over only a month due to the experimental school’s contraints. This short duration limits understanding students’ long-term changes effects of CPS-based STEM lessons. Long-term studies are needed for a more comprehensive examination of the effects. Additional, despite a week of orientation to system functions, challenges remained in effectively using cognitive tools. Future work should focus on enhancing instruction in cognitive strategy use to improve engagement and proficiency. For example, interactive tutorials and exercises tailored to promote strategic thinking may address this challenge.

Second, we only collected learning logs from the BookRoll system, which limited the fully understanding of students’ broader collaborative and cognitive behaviors. Future study should collect data from various learning management systems and interactive platforms to cover a wider range of student interactions. Furthermore, the implementation of multimodal data collection, including recordings and teacher observations, and integration of quantitative and qualitative insights, will ensure a comprehensive and multidimensional understanding of the integrative learning environment.

Third, the dialog analysis primarily focused on identifying CPS skills, which limited identification and interpretation of STEM strategies, learning behaviors, and their relationships. Future studies should conduct other statistical analysis (such as correlation analysis or Time Series Analysis) to examine these dynamics. Furthermore, dialog analysis should broaden its scope from CPS skills to include STEM strategies and learning behaviors, to enhance the explanatory power of this model.

Fourth, while previous studies recognized the critical role of teacher involvement in CPS (e.g., Gu et al., 2015) and STEM learning (e.g., Sergis et al., 2019), this study lacked an in-depth analysis of teacher practices. Future study should include the analysis of teacher strategies, decision-making processes, and teacher-student interactions. Analyzing both teacher and student activities will reveal how teaching methods have impact on students’ engagement and performance in CPS-based STEM education.

Finally, although we identified critical factors regarding CPS and STEM strategies that improved learning performance, distinguishing their separate effects within integrative CPS-based STEM framework is challenging. This distinction is important for developing focused strategies and interventions of CPS and STEM, thus maximizing students’ learning effectiveness. Future work should distinct effects of CPS framework and STEM learning strategies through experimental designs, such as using control groups or different strategies across different lesson phases.

Footnotes

Acknowledgements

We would like to acknowledge the teachers and students who participated in this study.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Drs. Chen, Taniguchi, Yamada, and Shimada received research grants from JSPS for this research project. Dr. Shimada received research grants from JST.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by Japan Society for the Promotion of Science (JSPS; grant number JP21K18134, JP22H00552, JP24K16759, and JP21KK0184).

ORCID iD

Li Chen

Data Availability Statement

All data generated or analyzed during this study are included in this published article.

References

Andrews-Todd

Forsyth

C. M.

(2020). Exploring social and cognitive dimensions of collaborative problem solving in an open online simulation-based task. Computers in Human Behavior, 104, 105759. https://doi.org/10.1016/j.chb.2018.10.025

Baker

Siemens

(2014). Educational data mining and learning analytics. In Sawyer

(Ed.), The Cambridge handbook of the learning sciences. Cambridge Handbooks in Psychology (pp. 253–272). Cambridge University Press.

Bernacki

M. L.

Byrnes

J. P.

Cromley

J. G.

(2012). The effects of achievement goals and self-regulated learning behaviors on reading comprehension in technology-enhanced learning environments. Contemporary Educational Psychology, 37(2), 148–161. https://doi.org/10.1016/j.cedpsych.2011.12.001

Care

Scoular

Griffin

(2016). Assessment of collaborative problem solving in education environments. Applied Measurement in Education, 29(4), 250–264. https://doi.org/10.1080/08957347.2016.1209204

Chen

Yoshimatsu

Goda

Okubo

Taniguchi

Konomi

Shimada

Ogata

Yamada

(2019). Direction of collaborative problem solving-based STEM learning by learning analytics approach. Research and Practice in Technology Enhanced Learning, 14(24). https://doi.org/10.1186/s41039-019-0119-y

Chen

Inoue

Goda

Okubo

Taniguchi

Konomi

Ogata

Yamada

. (2020). Exploring factors that influence collaborative problem solving awareness in science education. Technology, Knowledge and Learning, 25, 337–366. https://doi.org/10.1007/s10758-020-09436-8

Claessens

Van Eerde

Rutte

C. G.

Roe

R. A.

(2007). A review of the time management literature. Personnel Review, 36(2), 255–276. https://doi.org/10.1108/00483480710726136

Dinsmore

D. L.

Baggetta

Doyle

Loughlin

S. M.

(2014). The role of initial learning, problem features, prior knowledge, and pattern recognition on transfer success. The Journal of Experimental Education, 82, 121–141. https://doi.org/10.1080/00220973.2013.835299

Dunbar

R. L.

Dingel

M. J.

Prat-Resina

(2014). Connecting analytics and curriculum design: process and outcomes of building a tool to browse data relevant to course designers. Journal of Learning Analytics, 1(3), 223–243. https://doi.org/10.18608/jla.2014.13.26

10.

Graesser

Sabatini

(2022). Educational psychology is evolving to accommodate technology, multiple disciplines, and twenty-first-century skills. Annual Review of Psychology, 73(1), 547–574. https://doi.org/10.1146/annurev-psych-020821-113042

11.

Greiff

Wüstenberg

Avvisati

(2015). Computer-generated log-file analyses as a window into students’ minds? A showcase study based on the PISA 2012 assessment of problem solving. Computers & Education, 91, 92–105. https://doi.org/10.1016/j.compedu.2015.10.018

12.

Griese

Lehmann

Roesken-Winter

(2015). Refining questionnaire-based assessment of STEM students’ learning strategies. International Journal of STEM Education, 2, 12. https://doi.org/10.1186/s40594-015-0025-9

13.

Chen

Zhu

Lin

(2015). An intervention framework designed to develop the collaborative problem-solving skills of primary school students. Educational Technology Research and Development, 63(1), 143–159. https://doi.org/10.1007/s11423-014-9365-2

14.

Gunzelmann

Anderson

J. R.

(2003). Problem solving: Increased planning with practice. Cognitive Systems Research, 4, 57–76. https://doi.org/10.1016/S1389-0417(02)00073-6

15.

Hayes-Roth

(1979). A cognitive model of planning. Cognitive Science, 3, 275–310. https://doi.org/10.1207/s15516709cog0304_1

16.

Herro

Quigley

Cian

(2019). The challenges of STEAM instruction: Lessons from the field. Action in Teacher Education, 41(2), 172–190. https://doi.org/10.1080/01626620.2018.1551159

17.

Hesse

Care

Buder

Sassenberg

Griffin

(2015). A framework for teachable collaborative problem solving skills. In Griffin

McGaw

Care

(Eds.), Assessment and teaching of 21st century skills (pp. 37–56). Springer.

18.

Hogan

(1999). Thinking aloud together: A test of an intervention to foster students’ collaborative scientific reasoning. Journal of Research in Science Teaching, 36(10), 1085–1109. https://doi.org/10.1002/(SICI)1098-2736(199912)36:10<1085::AID-TEA3>3.0.CO;2-D

19.

Hora

M. T.

Oleson

A. K.

(2017). Examining study habits in undergraduate STEM courses from a situative perspective. International Journal of STEM Education, 4(1), 1. https://doi.org/10.1186/s40594-017-0055-6

20.

Hwang

W. Y.

Shadiev

Wang

C. Y.

Huang

Z. H.

(2012). A pilot study of cooperative programming learning behavior and its relationship with students’ learning performance. Computers & Education, 58(4), 1267–1281. https://doi.org/10.1016/j.compedu.2011.12.009

21.

Ifenthaler

Gibson

Prasse

Shimada

Yamada

(2021). Putting learning back into learning analytics: Actions for policy makers, researchers, and practitioners. Educational Technology Research and Development, 69(4), 2131–2150. https://doi.org/10.1007/s11423-020-09909-8

22.

Kelley

T. R.

Knowles

J. G.

(2016). A conceptual framework for integrated STEM education. International Journal of STEM Education, 3(1), 11. https://doi.org/10.1186/s40594-016-0046-z

23.

Kim

Tawfik

A. A.

(2023). Different approaches to collaborative problem solving between successful versus less successful problem solvers: Tracking changes of knowledge structure. Journal of Research on Technology in Education, 55(4), 628–645. https://doi.org/10.1080/15391523.2021.2014374

24.

Kranzfelder

A. T.

Melloy

M. P.

Walker

L. E.

Warfa

A. R. M.

(2019). Instructional practices in reformed undergraduate STEM learning environments: A study of instructor and student behaviors in biology courses. International Journal of Science Education, 41(14), 1944–1961. https://doi.org/10.1080/09500693.2019.1649503

25.

Kwon

Shin

Brush

T. A.

Glazewski

K. D.

Edelberg

Park

S. J.

Khlaif

Nadiruzzaman

Alangari

(2018). Inquiry learning behaviors captured through screencasts in problem-based learning. Interactive Learning Environments, 26(6), 839–855. https://doi.org/10.1080/10494820.2017.1419496

26.

Learning Analytics & Knowledge (LAK). (2011). What is learning analytics? Retrieved November 9, 2021, from http://https://www.solaresearch.org/about/what-is-learning-analytics/

27.

Lin

K. Y.

K. C.

Hsiao

H. S.

Chang

Y. S.

Chien

Y. H.

(2020). Effects of web-based versus classroom-based STEM learning environments on the development of collaborative problem-solving skills in junior high school students. International Journal of Technology and Design Education, 30, 21–34. https://doi.org/10.1007/s10798-018-9488-6

28.

Lou

S. J.

Liu

Y. H.

Shih

R. C.

Tseng

K. H.

(2011). The senior high school students’ learning behavioral model of STEM in PBL. International Journal of Technology and Design Education, 21(2), 161–183. https://doi.org/10.1007/s10798-010-9112-x

29.

Lowes

Lin

Kinghorn

(2015). Exploring the link between online behaviours and course performance in asynchronous online high school courses. Journal of Learning Analytics, 2(2), 169–194. https://doi.org/10.18608/jla.2015.22.13

30.

Majumdar

Bakilapadavu

Majumder

Chen

M. A.

Flanagan

Ogata

(2021). Learning analytics of humanities course: Reader profiles in critical reading activity. Research and Practice in Technology Enhanced Learning, 16, 25. https://doi.org/10.1186/s41039-021-00164-w

31.

Nesbit

J. C.

Winne

P. H.

Jamieson-Noel

Code

Zhou

Allister

K. M.

Bratt

Wang

Hadwin

(2006). Using cognitive tools in gStudy to investigate how study activities covary with achievement goals. Journal of Educational Computing Research, 35(4), 339–358. https://doi.org/10.2190/H3W1-8321-1260-1443

32.

Ogata

Taniguchi

Suehiro

Shimada

Okubo

Yamada

Kojima

(2017). M2B system: A digital learning platform for traditional classrooms in university. Proceedings of the Seventh International Learning Analytics & Knowledge Conference [Practitioner Track Proceedings], pp. 155–162, Simon Fraser University, Vancouver.

33.

Organisation for Economic Cooperation and Development. (2017). PISA 2015: Collaborative problem-solving framework. Retrieved July 23, 2020, from https://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Collaborative%20Problem%20Solving%20Framework%20.pdf

34.

Savery

J. R.

(2006). Overview of problem-based learning: Deﬁnitions and distinctions. Interdisciplinary Journal of Problem-Based Learning, 1(1), 9–20. https://doi.org/10.7771/1541-5015.1002

35.

Sergis

Sampson

D. G.

Rodríguez-Triana

M. J.

Gillet

Pelliccione

de Jong

(2019). Using educational data from teaching and learning to inform teachers’ reflective educational design in inquiry-based STEM education. Computers in Human Behavior, 92, 724–738. https://doi.org/10.1016/j.chb.2017.12.014

36.

Van Horne

Russell

J. E.

Schuh

K. L.

(2016). The adoption of mark-up tools in an interactive e-textbook reader. Educational Technology Research and Development, 64(3), 407–433. https://doi.org/10.1007/s11423-016-9425-x

37.

Weinstein

C. E.

Acee

T. W.

Jung

(2011). Self-regulation and learning strategies. New Directions for Teaching and Learning, 126, 45–53.

38.

Yen

M. H.

Wang

C. Y.

Chang

W. H.

Chen

Hsu

Y. S.

Liu

T. C.

(2018). Assessing metacognitive components in self-regulated reading of science texts in e-based environments. International Journal of Science and Mathematics Education, 16(5), 797–816. https://doi.org/10.1007/s10763-017-9818-2

39.

Yik

N. C.

L. X.

Somadam

S. T. N.

Amirudin

A. E. B.

Ananthan

(2018). Effect of highlighting text on concentration, memory and attention among undergraduate medical students: A randomized controlled trial. American Journal of Educational Science, 4(4), 149–158.

40.

Yue

C. L.

Storm

B. C.

Kornell

Bjork

E. L.

(2015). Highlighting and its relation to distributed study and students’ metacognitive beliefs. Educational Psychology Review, 27, 69–78. https://doi.org/10.1007/s10648-014-9277-z

41.

Zarzour

Bendjaballah

Harirche

(2020). Exploring the behavioral patterns of students learning with a Facebook-based e-book approach. Computers & Education, 156, 103957. https://doi.org/10.1016/j.compedu.2020.103957

42.

Zimmerman

B. J.

(2002). Becoming a self-regulated learner: An overview. Theory Into Practice, 41(2), 64–70. https://doi.org/10.1207/s15430421tip4102_2

Exploring Behavioral and Strategic Factors Affecting Secondary Students’ Learning Performance in Collaborative Problem Solving-Based STEM Lessons

Abstract

Plain Language Summary

Keywords

Introduction

Literature Review

Strategy of Using CPS Skills in CPS-Based STEM Education

STEM Learning Strategy in CPS-Based STEM Education

Learning Behaviors and Learning Analytics

Methods

Participants

Design of CPS-Based STEM Course

Course Theme

Design of CPS-Based STEM Lessons Based on CPS Framework

Functional Tools for Supporting STEM Learning Strategy

Research Procedure

Data Collection and Analysis

Learning Performance

CPS Awareness

Use of STEM Learning Strategy

Learning Behaviors

Dialog Analysis

Data Analysis Process

Results

The Improvement of Learning Performance

Path Analysis

Direct Effects of CPS Awareness on Learning Performance

Behavioral Factors Affecting Learning Performance

Strategic Factors That Affect Learning Performance Mediated by CPS

Dialog Analysis

Discussion

What Is the Difference in Learning Performance Between the Pre- and Post-Tests in CPS-Based STEM Lessons?

What Are the Strategic and Behavioral Factors That Improve the Learning Performance in CPS-Based STEM Lessons?

How Do Students Use CPS Skills to Improve Their Understanding in CPS-Based STEM Lessons?

Conclusions

Footnotes

Acknowledgements

Declaration of Conflicting Interests

Funding

ORCID iD

Data Availability Statement

References