Explore,Exploit,and Prune in the Classroom: Strategic Resource Management Behaviors Predict Performance

Abstract

Self-regulated learners strategically manage physical, technological, online, and social resources for learning—by selecting resources that could be useful, reflecting on how useful these resources have been, and adjusting resource use accordingly. We propose a model that conceptualizes resource management as learners’ intentional, self-reflective decisions to explore new resources, exploit (continue to use) previously useful resources, and prune (stop using) previously ineffective resources. We modeled 4,766 students’ reported exploration, exploitation, and pruning between three class exams among four cohorts of an organic chemistry class (i.e., more than 100,000 discrete data points of resource use). Each of these behavioral mechanisms of resource management predicted students’ academic achievement: The more students reported exploring, exploiting, and pruning between their exams, the higher they performed on their subsequent exams, controlling for prior performance. These findings enrich self-regulated learning theories by concretizing the behavioral mechanisms of resource management by which learners take control of their learning.

Keywords

self-regulated learning resource management achievement explore exploit prune

Effective self-regulated learning involves managing multiple, various learning resources available to develop mastery, and getting better at this over time through self-reflection (Brown et al., 1996; Chen, 2020; Makara & Karabenick, 2013). These resources could be physical, online, or social sources of information, such as textbooks, course pack readings, online websites, discussion forums, peer study groups, or mentors, all of which exist as tools in the environment to leverage for learning. Specifically, the psychological and behavioral process of strategically choosing, using, and adjusting the use of resources to achieve one’s learning goals is called “resource management” (Karabenick, 2014; Pintrich et al., 1991; Zimmerman, 2008).

Scoring high on resource management means that students are strategic in choosing what resources would be useful for building mastery, they reflect on how effective the resources have been, and they adjust their resource use accordingly (Chen et al., 2017; Pintrich et al., 1991). Prior research has shown that college students who report managing their resources more strategically tend to be more intrinsically motivated to learn, they report using more effective cognitive learning strategies, and they perform better in their classes (Karabenick, 2003; Kosnin, 2007; Lynch, 2006; Pintrich et al., 1991, 1993). An intervention that guided students to strategize and plan out their resource use before exams significantly raised students’ class performance (Chen et al., 2017). These findings indicate that resource management can be correlated and causally associated with better learning and performance.

Despite these many benefits, many students do not make such strategic decisions about their resource use (Pintrich et al., 1991). Poor resource management matters, because it can hold back the individual’s development of mastery and performance. At the societal level, ineffective resource management could have aggregated implications—potentially wasting substantial investments by governments and educational institutions in better resources for education (Organisation for Economic Co-operation and Development, 2018; World Bank, 2018). Therefore, it is important to understand how learners can be strategic in managing their resources and to help them improve this decision-making process, in order to become more effective.

In this article, we ask and address a question that is one step toward this goal: Are there identifiable, context-general mechanisms of strategic resource management that reliably align with academic achievement? Identifying such behavioral processes could provide one step toward creating interventions to promote these adaptive behaviors, such as direct guidance for students and faculty on effectively navigating or developing learning resources. To address this, we (a) defined and operationalized key behaviors that reflect a strategic decision-making process by which students manage their resource use; and (b) we tested how well each of these resource management behaviors tracked students’ actual exam performance.

Resource Management in a Class Resource Ecology

The hallmark of resource management as a self-regulated behavior is the agentic use of multiple, various resources for learning that is conducted in a self-reflective manner (Zimmerman, 1990, 2008). Consider the example of Fiona, a student in a typical college class. Like her peers, Fiona has a variety of resources that she could use for learning—a class resource ecology—that includes attending class lectures, going to office hours, completing homework assignments, participating in peer study groups, working with a tutor, and so on. How does she manage her use of each resource to learn effectively, and how does she do so over time as she gathers information about their usefulness?

Each learner in the class faces this challenge of navigating the buffet spread of resources, deciding which resources to use, and changing their resource use over time, to achieve their learning goals. There could be heterogeneity among different learners’ subjective judgments of the usefulness of each resource: For example, Fiona may find attending instructor office hours effective for her learning, but her classmate Mike may not; conversely, Mike may find engaging in a peer study group useful, but Fiona could have similarly tried it out and realized that it was too distracting for her. Strategic resource management, therefore, involves a psychological and behavioral process of trial, reflection, and adjustment of resource use over time, based on what the learner decides matters for their own learning (Brown et al., 1996). This theoretical perspective suggests that each student may have resources that they find more useful, rather than treating specific resources as intrinsically “better” or “worse” for all learners (or the average learner). How might we capture this in a model of resource management?

Prior Literature on Self-Regulated Resource Management

Many components of self-regulated learning have vast literatures, including those such as motivation and use of cognitive learning strategies. Comparatively speaking, little attention has been paid to understanding the general principles of resource management. Existing theories of resource use have focused on classifying different kinds of resources (e.g., time and study environment management, peer learning, help-seeking; Pintrich et al., 1991, 1993), testing how well various components of resource management correlate with one another and with academic performance (e.g., Karabenick & Knapp, 1991; Pintrich et al., 1993), explaining who (or what) people go to for help when they encounter problems (Makara & Karabenick, 2013), and examining the profiles of students who are more or less likely to ask for help (e.g., Karabenick & Dembo, 2011; Ryan et al., 2001). These studies have been immensely valuable for classifying various resources and highlighting the important implications of resource management. Yet they do not explain how learners adjust their resource use over time based on their reflections of how useful (or not) the resources have been. To add to existing studies on resource management, our research aims to provide a model that describes the general self-regulatory mechanisms by which most learners choose and update their resource use in a self-reflective manner over time.

A similar gap can be observed from a measurement perspective: Existing scales that measure resource management have asked students to self-report on generic resource management strategies (e.g., “I attend class regularly,” “I try to identify students in this class whom I can ask for help if necessary”; Pintrich et al., 1991; Weinstein et al., 2002) or to describe how they would respond to hypothetical learning challenges (e.g., Makara & Karabenick, 2013). These measurement methods are valuable in their own right, but they do not focus on tracking students’ decisions to use and change resources in a self-reflective manner based on what works (or does not work) for their own learning over time. In this work, we build on prior methods in a new way—by computing indices of how students’ resource use has or has not changed over time (as measures of their strategic resource management), and testing to what extent each of these indices track academic performance.

In our view, which is consistent with self-regulated learning theories (Karabenick, 2014; Pintrich et al., 1991; Winne & Hadwin, 2012), a model of the self-regulatory mechanisms behind resource management would focus on the processes by which learners choose and update their resource use in a self-reflective manner over time. That is, it should capture the process of self-reflective trial and updating of resource use. To the best of our knowledge, there is no such model at present that captures these general mechanisms of resource management. We introduce one plausible model that comprises three key resource management behaviors, and we test how well each of these behaviors predicts academic performance.

Modeling Self-Reflective Choice and Change in Resource Use Through Learners’ Exploration, Exploitation, and Pruning Behaviors

To complement previous work, our model of resource management focuses on learners’ intentional, self-reflective decisions to explore new resources, exploit previously useful resources, and prune previously ineffective resources. Exploration is defined as using a resource that was not previously used; exploitation as continuing to use a resource that was previously used and considered useful; and pruning as discontinuing the use of a resource that was previously used and perceived as not useful for learning.

To illustrate this, imagine that Fiona takes time between her exams to reflect on how effective it had been for her to participate in a study group, attend her teaching assistant’s office hours, or work on practice exam questions. Based on how useful she previously found each of these resources, she may choose to continue their use (exploit), discontinue their use (prune), or even try out new resources (explore) when studying for her subsequent exam. We consider these behaviors concrete, quantifiable units of resource management, which represent how students are intentionally choosing to use resources based on their prior resource use, and how subjectively useful that had been.

These processes have been conceptualized as strategic behaviors in other fields, such as evolutionary biology, decision making, and neuroscience. But to our knowledge, they have not been applied to human learning processes. Like prior research in other fields, our current definitions reflect the strategic nature of exploiting, exploring, and pruning. For example, behavioral models in evolutionary biology and decision making define resource optimization as the process of maximizing resource-use payoff by balancing the trade-off between exploiting a known resource and exploring a new resource (e.g., Charnov, 1976; Cohen et al., 2007). Such trade-offs have been studied in the domain of foraging animals that have to decide how long to continue exploiting a berry patch versus when to leave in search of greener pastures (Krebs et al., 1974; Krebs et al., 1978); or gamblers who have to decide when to switch from a one-armed bandit to another (Daw et al., 2006). However, applying these resource trade-off models wholesale to human academic learning may be too simplistic, because they typically examine an agent’s use of one resource at a time (Charnov, 1976; Cohen et al, 2007; Daw et al., 2006). In contrast, during human learning, learners are often free to choose from among multiple resource options and they can use more than one resource at the same time. Because people can continue to exploit previously valuable resources and simultaneously choose to explore new resources, exploration and exploitation do not always require a trade-off. Therefore, our model considers them resource-use choices that learners can make at the same time on different resources during learning.

On top of exploring and exploiting their learning resources, learners can also exercise a third type of resource management: “pruning”—or discontinuing their use of previously nonbeneficial resources. We borrow the term pruning from neuroscience, where pruning in the human brain refers to the weakening or elimination of neuronal connections that encode information no longer of relevance to the organism (e.g., Buller & Hardcastle, 2000; Casey et al., 2005; Goswami, 2004).¹ As in neuroscience, we consider such pruning behavior potentially adaptive because it involves reducing further investment in resources that are not paying off.

Integrating these three key processes and applying them to resource management, we propose exploration, exploitation, and pruning as concrete, measurable mechanisms of resource management when humans learn. Because these are defined as strategic, learners’ use of each of these self-regulatory processes should predict their academic performance. In other words, we hypothesize and test that exploring new resources, exploiting resources that were previously considered useful, and pruning resources that were previously considered not useful would each be positively correlated with academic performance. What this model of exploring, exploiting, and pruning can potentially contribute to the self-regulated learning literature is to define, operationalize, and track how human learners manage their use of various resources for learning.

Characteristics of the Explore, Exploit, Prune Model

We offer an analytical approach that is neutral on the question of which specific resource is being used, or what resources are more or less popular, unlike some theories that prioritize the identification and the relative use of specific resources. As mentioned earlier, this is important in light of the heterogeneity in resource choice, and the subjectivity in judging the effectiveness of various resources. For example, Fiona might find reading the textbook more useful for reviewing the course content, whereas Mike may find reading his lecture notes more useful; Mike might find a study group useful when he needs to ask others for help, but Fiona may find going to her teaching assistant’s office hours more useful. Because there is such individual heterogeneity of resource use, what matters to us is describing and modeling the general self-regulatory mechanisms by which learners choose and update their resource use in a self-reflective manner over time. Our approach to studying general principles of self-regulated learning is commonly used in the educational psychology literature to better understand the benefits and mechanisms of self-regulated learning (e.g., Pintrich et al., 1991; Weinstein et al., 1988; Winne & Hadwin, 2012). For instance, “study[ing] in a place where I can concentrate on my course work” is a generally important resource management behavior that, on average, positively relates to academic performance (Pintrich et al., 1991)—at the same time, it allows for different learners to choose different study venues that afford them such concentration. This kind of approach is meant to complement, not to replace, studies on individual differences in resource use (e.g., Karabenick & Dembo, 2011; Ryan et al., 2001).

The self-regulated learning literature has traditionally defined resource management as employing “regulatory strategies for controlling other resources besides cognition” (Pintrich et al., 1993, p. 803). In line with this, empirical research has mostly focused on the management of external resources when describing resource management—specifically, time management (e.g., how one plans their revision time), choice of study environment (e.g., selecting a conducive environment to study), peer learning (e.g., forming a study group), and help-seeking (e.g., asking an instructor to clarify a concept; Pintrich et al., 1991). While we acknowledge that some resources can also be internal in nature (e.g., effort, attention), in this research, we focus on how students regulate their external resources for learning, which is consistent with prior resource management research (Brown et al., 1996; Karabenick & Knapp, 1991; Makara & Karabenick, 2013; Pintrich et al., 1991).

The Present Study

In this study, our goals were to (a) define exploration, exploitation, and pruning in the context of human learning; (b) model how much students carried out these processes between multiple class exams; and (c) test how well each process of exploration, exploitation, and pruning related to their exam performance over time. If exploration, exploitation, and pruning of resources are indeed strategic in the context of human learning, then we would expect students’ reported practice of each of these behaviors to positively relate to their exam performance, even after controlling for their prior performance. To our knowledge, this research is the first to define and model these three resource management mechanisms in the context of human learning, and to test how well these behaviors predict academic achievement.

Method

We chose to begin by studying these processes in a controlled class setting that had well-defined learning goals; a standardized set of multiple, varied resources; and learner autonomy to navigate these resources. We invited students in four consecutive cohorts of an organic chemistry college course to voluntarily participate in this study before each of their three class exams. Organic chemistry gateway courses like this are highly challenging courses that stretch most students (Szu et al., 2011). Students taking organic chemistry are generally motivated to do well, because the course is an important prerequisite for the chemistry, biomedical engineering, and chemical engineering majors, as well as a premedical and other prehealth track requirement.

We chose this organic chemistry class because all students had the same set of class resources, learning objectives, and evaluations: All students could access the same class resources (see Supplemental Online Materials [SOM] Appendix A for the full list of resources and their descriptions), they were offered complete autonomy to manage these resources, and their exam performance was directly tied to demonstrating content mastery of the subject. In this class, none of the resources were explicitly emphasized or enforced by the instructors as compulsory for students to use—that is, students had the autonomy to choose what resources to use for their learning and how they wanted to use them. The teaching and content of the course was kept consistent across the 4 years of our study.

The course exams were valid assessments of the organic chemistry knowledge gained, for the following reasons: One, all exams involved answering open-ended questions that required interpretation, extrapolation, and generation to demonstrate mastery of the organic chemistry concepts (examples of which we have included in SOM Appendix B). Low-demand multiple choice and recall questions were not used. Two, the exams went through an internal validation process: They were developed by a team of organic chemistry professors at a top R1 (research-intensive) institution, who had many years of experience teaching the course, and they were also reviewed by other organic chemistry professors who commonly taught the course in different semesters (see SOM Appendix B for details). New exam questions were created every semester, and the questions were not recycled. Three, every exam was objectively graded by teams of trained and supervised teaching staff, and standardized across all students taking the course at the same time. For these reasons, this class context was well-suited to modeling students’ resource management, and how that tracks their learning outcomes.

Participants

We recruited a total of 4,766 students (N_total = 4,766; N_Cohort1 = 1,172; N_Cohort2 = 1,071; N_Cohort3 = 1,347; N_Cohort4 = 1,176) from a public Midwestern university in the United States. Aggregating across the four cohorts, 51.8% of our participants were female, 48.0% were male, and the remainder did not report their gender; 59.4% self-identified as White, 4.6% as Black/African American, 19.3% as Asian, 3.7% as Hispanic, 0.9% as Native American, and the remainder did not report their race/ethnicity (online Supplemental Table S1 shows the demographic breakdown by cohort). Response rates were high across the four cohorts (proportions of students who participated in at least one of the surveys: Cohort 1: 99.8%, Cohort 2: 80.2%, Cohort 3: 93.7%, Cohort 4: 86.3%) and across the three time points within each cohort (see Table 1).

Table 1

Numbers and Percentages of Students Who Participated Across the Four Consecutive Cohorts

Variable	Cohort 1	Cohort 2	Cohort 3	Cohort 4
Total class enrollment	1,172	1,336	1,438	1,392
Exam 1, n (%)	1,136 (96.9)	992 (74.3)	1,265 (87.97)	1,064 (76.4)
Exam 2, n (%)	1,119 (95.5)	994 (74.4)	1,300 (90.4)	1,096 (78.7)
Exam 3, n (%)	1,123 (95.8)	898 (67.2)	1,286 (89.4)	1,104 (79.3)
Participation in at least one survey that year	1,170 (99.8)	1,071 (80.2)	1,347 (93.7)	1,201 (86.3)

Procedure

Prior to the first exam in the course, the research team described the study, both in person and through electronic communication, to students. A copy of the consent form was provided in advance for the students’ inspection, along with opportunities to ask questions or make clarifications. Students were told that they would be asked to fill out an optional short survey in the 5 minutes preceding each of their class exams and that they had the right not to participate.

In the 5 minutes preceding the start of each exam, students voluntarily filled out our “Resource Use and Usefulness” survey. The survey took about 3 minutes or less for students to complete. We measured their self-reported resource use right before each of their three exams in the class, rather than much later after their exams, to minimize the possibility that they might forget how they had studied for the exam. We sought participating students’ permission to associate their survey responses with their exam performance and their demographic data from the school registrar.

Measure of Resource Use and Usefulness

Our survey presented students with a comprehensive list of 12 different kinds of resources available to all learners in the class, such as lectures, textbook, course pack problems, graduate student instructor–led discussions, graduate student instructor office hours, and peer study groups (SOM Appendix A presents and describes these 12 resources). We identified the 12 kinds of resources used in our survey through prior discussions with the course instructors and student focus groups. Although we did offer an “Other” open-ended response option for students to write any additional, unlisted resources that were not included in our list, there were no other kinds of resources that at least 5% of student participants described using on any given exam. In the survey, students were asked to indicate (a) whether or not they had used each of the 12 different kinds of resources to study for the exam (yes/ no) and (b) if they had used a particular resource, how useful they had found it for learning (1–5 scale from extremely useless to extremely useful; Brown et al., 1996).

Our questions about students’ resource use were straightforward and easy to understand—which we validated with cognitive interviews conducted with a separate group of 13 students, who were not part of this study (see SOM “Survey Validation” for details). The questions were neither sensitive in nature nor did they have strong social desirability. Moreover, students themselves were in the best position to report whether they had actually used a resource or not to study, and how useful they had found it. Students knew that the data would be analyzed in a manner that protected their anonymity. Test–retest reliability of the measure conducted 5 days apart with a separate, smaller sample of 60 students was high (students’ reported use of the 12 resources showed an average 86% agreement rate between the two time points; see SOM “Survey Validation”). For these reasons, students’ reports about their resource use appeared valid (Bradburn et al., 2004; Tourangeau et al., 2000).

Our survey method was a brief and scalable way of allowing us to longitudinally track how students used their learning resources over time, as a function of the perceived usefulness of each resource. This allowed us to compute indices of the extent to which they explored, exploited, and pruned their resources, based on our aforementioned definitions of these constructs. Moreover, our data collection approach minimized variation-in-context issues (i.e., differing circumstances under which the same survey is taken by respondents), because all our participants took the surveys within the same time window before their exams (Tourangeau et al., 2000). This strengthens the reliability of our results.

Modeling Instances of Exploration, Exploitation, and Pruning

We modeled exploration, exploitation, and pruning as transformations derived from students’ self-reported resource use and usefulness ratings in the Resource Use and Usefulness survey. We operationalized exploration as trying a resource that students had reported not using on the previous exam, exploitation as continuing to use a resource that they had reported as useful on the previous exam, and pruning as ceasing to use a resource that they had used and reported as useless to their learning on the previous exam. An instance of exploration occurred for each resource that was reported as used for the current exam and had not been used on the exam prior; an instance of exploitation occurred for each resource that was reported as used again on the current exam, after having been used and rated as either “useful” or “extremely useful” on the exam prior; an instance of pruning occurred for each resource that was reported as not used on the current exam, and which had been used and rated as either “useless” or “extremely useless” on the exam prior.²

If Use_t is a binary variable indicating whether a given resource was used at time t, and Usefulness_t−1 (from 1 = extremely useless to 5 = extremely useful) was the rated usefulness at time t−1, then the coding for an individual learner and a given resource is,

\begin{matrix} \begin{array}{l} Resource Management {Behavior}_{t} \\ = {\begin{array}{l} Explore, & if {Use}_{t - 1} = 0 and {Use}_{t} = 1 \\ Exploit, & if {Usefulness}_{t - 1} > 3 and {Use}_{t} = 1 \\ Prune, & if {Usefulness}_{t - 1} < 3 and {Use}_{t} = 0 \end{array} \end{array} \end{matrix}

(1)

Online Supplemental Table S2 provides a detailed breakdown on how resource management behaviors were coded and the percentages in each category.

Importantly, our operationalizations are based on theory about what it means to be strategic (e.g., we do not consider continued use of a resource previously found useless as exploitation).³ This allows us to make strong a priori hypotheses that each of these three reported behaviors will be adaptive for performance. We made a theory-driven choice to have the reported behavior coded based on the choice to use the resource on the current exam (Use_t), but not on whether or not students found it useful on the current exam (i.e., not Usefulness_t). This is because the strategic regulatory choice to use a resource or not precedes using the resource and finding out its actual usefulness.

For each individual learner, we calculated three separate indices of exploring, exploiting, and pruning between Exams 1 and 2, and another three indices between Exams 2 and 3. Table 2 summarizes the proportions of learners’ decisions to explore, exploit, and prune their resources, out of the total number of resources available for them to explore, exploit, and prune between their exams. On average, students explored 16.0% of the resources that they had not used on the prior exam. Out of the total number of resources that students had used and found useful on their prior exam, students continued to exploit, on average, 90.1% of these resources for their current exam. Out of the (small number of) resources that students had used and considered useless on the exam prior, students pruned (or stopped using) 42.3% of these resources when studying for their current exam. To assuage concerns about correlation among these indices, we note that explore, exploit, and prune are mutually exclusive behaviors (e.g., a resource-use behavior categorized as explore cannot possibly also be categorized as exploit). Empirically, our explore, exploit, and prune indices (i.e., the number of resource-use behaviors counted as explore, exploit, and prune) are negatively correlated with each other only to a small degree (explore and exploit r = −.24; explore and prune r = −.02; exploit and prune r = −.19)—well below thresholds that warrant concerns about multicollinearity.

Table 2

Descriptive Frequencies of Resource Use on the Prior Exam (Exams 1 and 2) and Percentages of Resources Explored, Exploited, and Pruned Out of Those Possible (on the Subsequent Exams 2 and 3, Respectively), Aggregating Across Cohorts

Variable	Exam 2	Exam 3	Aggregating exams
Explore
Number of resources not used on prior exam	4.13	4.62	4.39
Percentage of resources explored out of resources available to explore (%)	16.9	14.0	16.0
Exploit
Number of resources rated useful on prior exam	6.17	6.00	6.07
Percentage of resources exploited out of resources available to exploit (%)	90.1	90.4	90.1
Prune
Number of resources rated useless on prior exam	0.40	0.34	0.37
Percentage of resources pruned out of resources available to prune (%)	40.9	37.8	42.3

Note. The numbers of resources reflect the mean numbers per student, averaged across all students per exam.

Analysis

All analyses were done in R (4.0.1), using RStudio (1.3.959) and package lme4 (1.1-23).

Missing Data

We used all available data, and response rates are given in Table 1. Note that the calculation of the explore, explore, and prune indices (Equation 1) requires that a student responded at two consecutive time points (out of three total time points).

Resource Use Over Time

To first analyze how students’ resource use changed over time, we fit a simple linear model predicting the number of resources used at a given time point, assuming a linear coefficient on time, an integer-valued variable (t = 1, 2, 3). We applied the same model to analyze how students’ reported mean usefulness ratings changed over time. These two models were estimated with random effects of individual students nested within cohort (year):

\begin{matrix} N u m R e s o u r c e s U s e d_{i, t, y} = b_{0} + b_{1} t + u_{i, y} + u_{y} + ϵ_{i, t, y} \end{matrix}

(2a)

\begin{matrix} M e a n U s e f u l n e s s_{i, t, y} = b_{0} + b_{1} t + u_{i, y} + u_{y} + ϵ_{i, t, y} \end{matrix}

(2b)

where u_i,y is the student-specific random intercept nested within year, u_y is a random intercept for year, and ε_{i,t y} is the residual error term. In R syntax, the model in Equation 2a is lmer(numResources ~ time + (1|studentID:Year) + (1|Year)).

Predicting Exam Performance

Ultimately, we were interested in whether students’ resource management behaviors between exams were associated with their exam performance. In a mixed-effects linear model, we regressed students’ current exam performance on their reported exploration, exploitation, and pruning behaviors, controlling for students’ performance on the prior exam as fixed effects. We added random intercepts by student nested within year, and time point nested within year. Exam scores, the dependent variable, were converted into percentage scores out of 100 for all exams. Effect sizes (unstandardized b coefficients) can be interpreted in units of percentage points. Means and standard deviations (SDs) of the three class exam scores are presented in online Supplemental Table S3. Thus, for student i at time t in year y, we estimate the following model:

\begin{array}{l} E x a m_{i, t, y} = b_{0} + b_{1} N u m E x p l o r e_{i, t, y} + b_{2} N u m E x p l o i t_{i, t, y} \\ + b_{3} N u m P r u n e_{i, t, y} + b_{4} E x a m_{i, t - 1, y} \\ + u_{i, y} + u_{t, y} + u_{y} + ε_{i, t, y} \end{array}

(3)

where Exam_i,t,y denotes the exam score of student i at time t in year y, u_i,y gives the random intercept of student nested within year, u_t,y gives the random intercept of exam nested within year, and u_y gives the random intercept by year. Note that controlling for the exam performance at the previous time point is conservative and allows us to test if the resource regulatory behaviors (done between exams) explain exam performance over and above prior performance. In R syntax, this model is:

lmer(currentScore ~ sum_explore + sum_exploit + sum_prune + pastScore + (1|studentID:Year) + (1|examNum:Year) + (1|Year))

Results

Aggregate Patterns of Resource Use and Usefulness

We begin by describing how students in our study interacted with their resources. Students showed a decreasing linear trend in the number of resources that they used to study for each of their three exams (Equation 2a). Across all four cohorts, students started off using an average of 7.9 (SD = 1.7) resources to study for Exam 1, 7.4 (SD = 1.7) resources for Exam 2, and 7.0 (SD = 1.7) resources for Exam 3, linear trend b = −.44, 95% CI = [−.47, −.42], p < .001. We illustrate students’ use and usefulness ratings for the 12 kinds of resources in Figure 1, aggregated across cohorts and categorized by exam. On average, some resources (e.g., attending the lecture and using the coursepack) were used more than others (e.g., lecture podcasts, Science Learning Center)—not surprisingly, these tended to also be the resources that a large proportion of students rated as “extremely useful” for their learning.

Figure 1.

Percentage of students who rated each of the 12 kinds of resources at different levels of usefulness, aggregated across cohorts and categorized by exam.

While this decreasing trend of the number of resources used could be interpreted as decreasing motivation, our evidence suggests otherwise: Over the same period, students’ mean ratings of how useful their resources had been exhibited a positive linear trend over time—increasing from 4.09 (SD = 0.47) on Exam 1, to 4.17 (SD = 0.53) on Exam 2, and 4.21 (SD = 0.58) on Exam 3, linear trend b = .06, [.05, .07], p < .001 (Equation 2b). A score of 4 on our 5-point scale corresponds to “useful.” We inferred that, rather than necessarily being less invested, students were, on average, becoming possibly more focused and effective in their resource use over time. Evidence from our cognitive interviews with a randomly selected sample of students in the class who were not part of this study (see SOM “Survey Validation” for details) supports this idea that some students were strategically changing their resource use over time in the class. For example, one student shared that, “By exam 3 I wasn’t using [textbook problems]. I used those for the first exam, but found them not as helpful, so I stopped using those.” Next, we turn to how students increased their resource-use effectiveness by managing their resource use wisely from exam to exam.

Predicting Exam Performance

We analyzed the effect of exploring, exploiting, and pruning behaviors on exam performance (Equation 3). Consistent with our hypotheses, the extent to which students engaged in each of these resource management behaviors positively predicted their exam performance. This was true for exploration (b = 0.85, [0.52, 1.18], p < .001), for exploitation (b = 0.91, [0.74, 1.08], p < .001), and for pruning (b = 0.75, [0.03, 1.48], p = .042). Table 3 presents the full statistical results. Following recent recommendations in psychological science and statistics to move away from Null Hypothesis Statistical Testing (e.g., Wasserstein et al., 2019) and toward an “estimation” framework (e.g., Cumming, 2014), we provide p values for completeness, but we focus on interpreting the effect sizes, which are directly interpretable in terms of exam performance. Exploring one new resource was associated with an average of 0.85 percentage points increase in students’ performance on the current exam; exploiting one additional resource that was considered useful on the previous exam was associated with an average of 0.91 percentage points increase in students’ performance on the current exam; and pruning one additional resource that was found to be useless on the previous exam was associated with an average of 0.75 percentage points increase in students’ performance on the current exam.

Table 3

Results of the Mixed-Effects Model Predicting Students’ Exam Performance on the Current Exam, Controlling for Their Score on the Previous Exam, Including Random Effects for Individual Student Nested Within Cohort, and Exam Nested Within Cohort

Variable	b	SE	95% CI	p
Sum of resources explored on current exam	0.848	0.170	[0.515, 1.18]	<.001
Sum of resources exploited on current exam	0.911	0.087	[0.741, 1.08]	<.001
Sum of resources pruned on current exam	0.752	0.370	[0.027, 1.48]	.042
Score on previous exam	0.825	0.010	[0.806, 0.843]	<.001
Model Conditional R²	0.601

Note. We report unstandardized b values, standard errors, 95% confidence intervals, and p values. Conditional R² was calculated using the definition provided in Nakagawa & Schielzeth (2013) using the MuMIn library in R, which estimates the proportion of variance explained by both the fixed and random effects.

Figure 2 visually illustrates how empirically observed combinations of exploration, exploitation, and pruning related to changes in students’ exam performance. We observed that greater resource management was associated with larger changes in students’ performance on subsequent exams: Starting from the origin and moving out along each of the three axes, as learners report practicing more exploration, exploitation, and pruning, we see that their exam performance improves. Our findings underscore the adaptive, strategic nature of learners’ decisions to explore new resources, exploit previously useful resources, and prune previously useless resources between one exam to the next.

Figure 2.

Graphical representation of the frequency of exploring, exploiting and pruning in relation to changes in performance.

Robustness Checks

In addition, we replicated our results when controlling for student engagement with the course resources (i.e., the total number of resources they used). Controlling for the number of resources students reported using at the beginning of the course (before Exam 1) as a proxy of their course engagement, as well as prior performance (i.e., adding total number of resources initially used as additional covariate to Equation 3), we find that greater exploration (b = 0.93, [0.59, 1.28], p < .001), exploitation (b = 0.85 [0.65, 1.05], p < .001), and pruning (b = 0.68 [−0.06, 1.43], p = .072) between exams still predicted students’ subsequent exam performance. The effect sizes in this analysis were relatively similar in magnitude (although we note that the coefficient on pruning is no longer statistically significant at the .05 level), suggesting that strategic resource management behaviors of exploring, exploiting, and pruning offer predictive value above and beyond a proxy of students’ sheer use of more course resources.

Finally, our definition of exploration only required that students had not used a resource on the preceding exam, but did not consider if they had not used it on all previous exams. We repeated all our analyses using a stricter definition of exploration: trying a resource that students had not used on any previous exam, rather than just the preceding exam. Our results replicated, and the effect size on exploration was, in fact, stronger (b = 1.18 [0.78, 1.57], p < .001). However, we choose to retain our current (conservative) operationalization, using only the previous exam’s (non-)use, to be consistent with how we operationalized exploiting and pruning. We note that exploitation and pruning are theoretically well-defined even using only the previous exam.

Discussion

As a start to illuminating some important mechanisms of strategic resource management, we defined and tested three primary resource management processes: exploring resources that students had not tried on the prior exam, exploiting resources that they had previously tried and found useful, and pruning resources that they had previously tried but found ineffective. Evidence from 4,766 college students and more than 100,000 instances of resource management collected longitudinally suggests that the more students explored, exploited, and pruned their resources between exams, the better they performed on their subsequent exams—even when we controlled for their prior performance (a well-established, strong correlate of current performance), and their sheer amount of resource use.

What does this mean in practical effect sizes? Imagine that Alice is a strategic, self-regulated learner in our class. Based on her first exam, if she had, by the second exam, (a) explored one new resource that she had not previously tried, (b) exploited one resource that she had previously considered useful, and (c) pruned one resource that she had previously found useless, she could be expected to score 2.84 percentage points higher on her second exam, relative to what she would have achieved had she not enacted these behaviors. Sustaining such self-regulation (one explore, one exploit, one prune) before her third exam, she could be expected to score 5.68 percentage points higher than a counterfactual, nonregulated Alice, who did not strategically regulate her resource use during the course. Practically, in the specific organic chemistry class that we studied, such sustained improvements could translate into performing about one-third of a letter grade higher in the course (e.g., getting a B+ instead of a B, or an A− instead of a B+). Although this example of Alice may be a simplified illustration, it underscores just how much of a practical difference strategic resource management from one exam to the next could potentially make for learners’ exam performance. This effect size is approximately comparable with the effect size found in a recent intervention study targeting strategic resource use, which produced an average of one-third of a letter grade boost in students’ course grades (Chen et al., 2017).

These findings provide evidence, in a controlled learning context with a standardized set of resources and performance metrics, that reported exploring, exploiting, and pruning resources over time are indeed strategic for learning. To our knowledge, this research is the first to define and model exploration, exploitation, and pruning of resource use in these ways, and to show that these self-reported strategic behaviors are indeed associated with academic performance. This research offers a new approach to understanding and modeling important behavioral mechanisms of self-regulated learning—putting flesh on the bones of what Pintrich et al. (1991, p. 23) generically referred to as learners’ “fine-tuning and continuous adjustment” of cognition and behavior during self-regulated learning.

Theoretical Implications

Our methods and findings invite psychologists and educators to reconsider how we conceptualize and operationalize resource management. We need to go beyond classifying and measuring resource use as only adaptive or maladaptive for most learners, to capturing resource management as an active process of trial and error, reflection, and change for every individual—that is, the general self-regulatory processes by which each individual navigates the buffet spread of resources available by selecting, monitoring, and adjusting their resource use according to what benefits their own learning. Here, we focused on students’ self-reported exploration, exploitation, and pruning, rather than which specific resources are used, which are most popular, or which might objectively be the most useful—questions that others have long been pursuing (Karabenick & Dembo, 2011; Karabenick & Knapp, 1991; Makara & Karabenick, 2013). These diverse scientific perspectives can effectively complement one another to provide a more comprehensive understanding of how people effectively manage their resource use toward mastery.

Prior research tends to focus on estimating the effects of introducing new resources, including interventions, as “treatments” on the seemingly passive average learner, who is construed as having no choice in their resource use (e.g., Cuban & Cuban, 2009; Dunlosky et al., 2013; Hanushek, 1997). While this provides valuable perspective on the utility of individual resources, it should also be complemented with an appreciation that, within a classroom resource ecology, each educational tool or intervention is just one resource among many that learners could adopt. The self-regulated learning perspective, one that we align with, has urged scientists and practitioners to consider the larger resource ecology within which the new resource is contextualized, and the autonomy of the learner to choose whether and how to make use of the learning resources available to them (Chen et al., 2017; Feldon, 2010; Karabenick & Knapp, 1991). Indeed, our findings show that even within the same class, different students may find the same resources useful to different extents—importantly, however, what matters is how they use the resources to make their own learning effective.

Our definitions of learners’ decisions to explore, exploit, and prune are independent of how objectively useful or useless the resources actually turn out to be in contributing to their performance. This is an important point because the objective usefulness of a resource in contributing to exam performance depends on a combination of factors, including some out of people’s control (e.g., the type and difficulty of the task they had to accomplish, and whether the resource changed over time). What often matters in predicting a student’s autonomous choice to use a particular resource is their subjective construal of how useful the resource would be, based on any prior knowledge they have about its use. For example, if they perceive the resource to have been previously useful, they might decide to use it again; if they perceive the resource to have been previously unhelpful, they might no longer want to use it. Therefore, we based our operationalizations of exploration, exploitation, and pruning on students’ subjective evaluations of the resource’s usefulness and their subsequent choices to use or not to use the resources again.

Limitations and Future Directions

As a start in this line of work on exploring, exploiting, and pruning mechanisms, we assessed students’ self-reports of their resource use at three time points in the class. Self-report measures certainly have their pros and cons—one evident limitation being that they rely on students’ recollection and honest reporting of their past behavior. As mentioned earlier, we addressed these concerns by measuring resource use right before the exam, phrasing our questions in a straightforward manner, and making sure our questions were not sensitive or socially desirable and were easy to understand and answer. Nevertheless, future studies could supplement (and further validate) our self-report surveys with behavioral measures of actual resource use in situ with greater granularity, such as by monitoring students’ naturalistic resource-use behaviors in online learning environments (e.g., Thille et al., 2014).

Another important future direction would be to develop and experimentally test interventions that promote exploration, exploitation, and pruning behaviors. Random assignment could rule out other possible factors (e.g., intrinsic motivation, mastery orientation) that may have contributed to our results. At the same time, future intervention studies could additionally provide (a) guidance on how to use various resources effectively, and (b) metacognitive training to help students monitor and evaluate their own resource use better. These steps could potentially make interventions more useful for students who do not naturally practise such strategic behaviors, students who may not know how to make effective use of certain resources, and also students who lack the awareness to detect when their resource use is ineffective (Dunning, 2011; Kruger & Dunning, 1999).

The ideal scenario may be a point where (a) the learner is maximally exploiting useful resources, (b) the utility of all available resources has already been explored, and (c) they are no longer using any useless resources (i.e., maximal exploitation, but no further exploration or pruning needed). How long learners take to get to this optimal point may depend on the number of resources available, the opportunities and costs of exploration, the frequency of feedback provided in the context, and individual differences in self-regulation. Future work could investigate how some individuals learn and reach this optimal state more efficiently than others, and how these skills can be taught effectively.

Could students’ sheer engagement with the course resources explain our results? No, we argue that exploration, exploitation, and pruning are conceptually and empirically dissociable from simply using more resources. Our indices of exploration, exploitation, and pruning represent how students strategically change (or do not change) in their reported resource use from one exam to the next, as a function of how useful the resources have previously been. These resource management behaviors are not simply equivalent to using more resources—in fact, pruning actually means using less resources (that were previously perceived as ineffective). Moreover, students’ degree of exploration, exploitation, and pruning still predicted their exam performance, even when we controlled for the sheer number of resources they reported using and their prior performance.

Given the multiple cohorts and large samples tested within this particular class context, we can infer with confidence that our findings should replicate among the average learner in this class. How well these results generalize to other learning contexts is an exciting empirical question for future research. It is possible that, although class structures, resources available, and learner profiles may differ by context, the value of self-reflective, strategic resource management could be generalizable—as prior literature on self-regulated learning processes have found for resource management and other forms of self-regulated learning (Pintrich et al., 1991; Weinstein et al., 1988; Winne & Hadwin, 2012). At the same time, there are plausible boundary conditions worth noting: Our organic chemistry course context allowed learners to have completely free and open choice among an array of resources, which is arguably representative of how many college classes work. Yet such resource-use autonomy may not generalize as well to all classrooms, such as those with younger learners, who often need more structure. Future replications should also take note that applying these models directly to instructional contexts that have more rigid, structured teaching approaches, or those that offer extrinsic rewards for resource use (e.g., class credit, forced requirements), may introduce such confounding factors.

Practical Implications

Learning occurs in an ecology of resources, and achievement depends on how effectively people manage this ecology for mastery (Brown et al., 1996; Chen et al., 2017; Zimmerman, 1990). To these ends, billions in government spending worldwide are invested in providing people with new educational resources in larger numbers or of higher quality (Organisation for Economic Co-operation and Development, 2018; The World Bank Group, 2018). However, if recipients do not use the resources effectively, these expensive solutions can turn out to be unproductive (Cuban & Cuban, 2009; Hanushek, 1997; Hattie, 2008).

Our findings offer insights for educators and policy makers: Providing an abundance and variety of resources is valuable. But it is also crucial to consider how effectively recipients are making use of their resources, and to help them use the resources more effectively over time. Instead of conceiving “education” as intervening on the passive learner, we should guide learners to cultivate thoughtful self-regulation of their resource use for mastery (Chen et al., 2020). These mental processes include recognizing all the resources available to them, seeking out new resources, weighing the pros and cons of each, selecting and using resources with purpose, and improving their resource use over time through self-reflection (Chen et al., 2017; Karabenick, 2012). Effective resource management is a psychological process that can be scaffolded and honed. Specifically, we can encourage more self-reflective exploration, exploitation, and pruning among learners. At the same time, it could be fruitful to also complement such advice with guidance on how to use the resources effectively and metacognitive training to help students who are poor judges of their own resource use, as mentioned earlier.

We also offer important takeaways for learners: Resources in our environments exist as tools that we can choose to leverage or not. Resource management is not a one-shot, fixed endeavor, but it entails a process of continual refinement over time. As learners, we can adaptively regulate our resource use by consciously self-reflecting on what we have been doing, considering how effective our actions have been, and then changing our resource use to gradually become more effective over time (Chen et al., 2020). The more we regulate our resource use in adaptive ways—by exploring new resources, exploiting previously useful ones, and pruning ineffective resources—the more efficiently our efforts will be channeled toward our goals.

Supplemental Material

sj-docx-1-ero-10.1177_2332858420986180 – Supplemental material for Explore, Exploit, and Prune in the Classroom: Strategic Resource Management Behaviors Predict Performance

Supplemental material, sj-docx-1-ero-10.1177_2332858420986180 for Explore, Exploit, and Prune in the Classroom: Strategic Resource Management Behaviors Predict Performance by Patricia Chen, Desmond C. Ong, Jessica C. Ng and Brian P. Coppola in AERA Open

Footnotes

Acknowledgements

We would like to thank L. Bricker, E. Brummelman, X. H. Chua, and the members of the University of Michigan Chemistry Department who supported this research over the years. We particularly acknowledge early contributions from E. Rothman, A. L. Kiste, J. P. Lomont, L. Zhang, and D. Childers.

Authors’ Note

Patricia Chen and Desmond C. Ong shared first authorship through equal contribution.

ORCID iD

Patricia Chen

Notes

Authors

PATRICIA CHEN is an assistant professor in the Department of Psychology at the National University of Singapore, where she directs the Motivation and Self-Regulation Lab. Her research focuses on the mind-sets and behaviors that help people achieve their learning goals, better performance, and well-being.

DESMOND C. ONG is an assistant professor in the Department of Information Systems and Analytics, School of Computing, at the National University of Singapore. He is a cognitive scientist who studies how people and computers reason about how humans think and feel.

JESSICA C. NG is a PhD student in the Department of Psychology at the National University of Singapore. She studies how people’s mind-sets and motivation facilitate their academic performance and how psychological interventions can help students succeed.

BRIAN P. COPPOLA is the Arthur F. Thurnau Professor of Chemistry in the Department of Chemistry at the University of Michigan, Ann Arbor. His research and teaching focus on promoting self-regulated learning, discipline-centered teaching and learning, and the globalization of science education.

References

Bradburn

Sudman

Wansink

(2004). Asking questions: The definitive guide to questionnaire design—for market research, political polls, and social and health questionnaires. Jossey-Bass.

Brown

M. I.

Doughty

G. F.

Draper

S. W.

Henderson

McAteer

(1996). Measuring learning resource use. Computers & Education, 27(2), 103–113. https://doi.org/10.1016/0360-1315(96)00017-6

Buller

D. J.

Hardcastle

(2000). Evolutionary psychology, meet developmental neurobiology: Against promiscuous modularity. Brain and Mind, 1(3), 307–325. https://doi.org/10.1023/a:1011573226794

Campanelli

(1997). Testing survey questions: New directions in cognitive interviewing. Bulletin of Sociological Methodology, 55(1), 5–17. https://doi.org/10.1177/075910639705500103

Campanelli

(2008). Testing survey questions. In de Leeuw

E. D.

Hox

J. J.

Dillman

D. D.

(Eds.), International handbook of survey methodology (pp. 176–200). Taylor & Francis.

Casey

Tottenham

Liston

Durston

(2005). Imaging the developing brain: What have we learned about cognitive development? Trends in Cognitive Sciences, 9(3), 104–110. https://doi.org/10.1016/j.tics.2005.01.011

Charnov

E. L.

(1976). Optimal foraging, the marginal value theorem. Theoretical Population Biology, 9(2), 129–136. https://doi.org/10.1016/0040-5809(76)90040-x

Chen

(2020). The strategic resource use intervention. In Waltion

G. M.

Crum

A. J.

(Eds.), Handbook of wise interventions: How social psychology can help people change (pp. 166–190). Guilford Press.

Chen

Chavez

Ong

D. C.

Gunderson

(2017). Strategic resource use for learning: A self-administered intervention that guides self-reflection on effective resource use enhances academic performance. Psychological Science, 28(6), 774–785. https://doi.org/10.1177/0956797617696456

10.

Chen

Powers

J. T.

Katragadda

K. R.

Cohen

G. L.

Dweck

C. S.

(2020). A strategic mindset: An orientation toward strategic behavior during goal pursuit. Proceedings of the National Academy of Sciences of the U S A, 117(25), 14066–14072. https://doi.org/10.1073/PNAS.2002529117

11.

Cohen

J. D.

McClure

S. M.

A. J.

(2007). Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1481), 933–942. https://doi.org/10.1098/rstb.2007.2098

12.

Cuban

(2009). Oversold and underused: Computers in the classroom. Harvard University Press. https://doi.org/10.2307/j.ctvk12qnw

13.

Cumming

(2014). The new statistics: Why and how. Psychological Science, 25(1), 7–29. https://doi.org/10.1177/0956797613504966

14.

Daw

N. D.

O’doherty

J. P.

Dayan

Seymour

Dolan

R. J.

(2006). Cortical substrates for exploratory decisions in humans. Nature, 441(7095), 876–879. https://doi.org/10.1038/nature04766

15.

Dunlosky

Rawson

K. A.

Marsh

E. J.

Nathan

M. J.

Willingham

D. T.

(2013). Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1), 4–58. https://doi.org/10.1177/1529100612453266

16.

Dunning

(2011). The Dunning–Kruger effect: On being ignorant of one’s own ignorance. Advances in Experimental Social Psychology, 44, 247–296. https://doi.org/10.1016/B978-0-12-385522-0.00005-6

17.

Esposito

Malerba

Semeraro

Kay

(1997). A comparative analysis of methods for pruning decision trees. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(5), 476–491. https://doi.org/10.1109/34.589207

18.

Feldon

D. F.

(2010). Why magic bullets don’t work. Change: The Magazine of Higher Learning, 42(2), 15–21. https://doi.org/10.1080/00091380903563043

19.

Forsyth

B. H.

Lessler

J. T.

(2004). Cognitive laboratory methods: A taxonomy. In Biemer

P. P.

Groves

R. M.

Lyberg

L. E.

Mathiowetz

N. A.

Sudman

(Eds.), Measurement errors in surveys (pp. 393–418). Wiley.

20.

Goswami

(2004). Neuroscience and education. British Journal of Educational Psychology, 74(1), 1–14. https://doi.org/10.1348/000709904322848798

21.

Hanushek

E. A.

(1997). Assessing the effects of school resources on student performance: An update. Educational Evaluation and Policy Analysis, 19(2), 141–164. https://doi.org/10.3102/01623737019002141

22.

Hattie

(2008). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Routledge. https://doi.org/10.4324/9780203887332

23.

Huys

Q. J.

Eshel

O’Nions

Sheridan

Dayan

Roiser

J. P.

(2012). Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Computational Biology, 8(3). https://doi.org/10.1371/journal.pcbi.1002410

24.

Karabenick

S. A.

(2003). Seeking help in large college classes: A person-centered approach. Contemporary Educational Psychology, 28(1), 37–58. https://doi.org/10.1016/s0361-476x(02)00012-7

25.

Karabenick

S. A.

(2012). Strategic help seeking: Implications for learning and teaching. Routledge. https://doi.org/10.4324/9781410602701

26.

Karabenick

S. A.

(2014). The past, present and future of help seeking as a self-regulated learning strategy [Paper presentation]. Annual Meeting of the American Educational Research Association, Philadelphia, PA, United States.

27.

Karabenick

S. A.

Dembo

M. H.

(2011). Understanding and facilitating self-regulated help seeking. New Directions for Teaching and Learning, 2011(126), 33–43. https://doi.org/10.1002/tl.442

28.

Karabenick

S. A.

Knapp

J. R.

(1991). Relationship of academic help seeking to the use of learning strategies and other instrumental achievement behavior in college students. Journal of Educational Psychology, 83(2), 221–230. https://doi.org/10.1037/0022-0663.83.2.221

29.

Kosnin

A. M.

(2007). Self-regulated learning and academic achievement in Malaysian undergraduates. International Education Journal, 8(1), 221–228.

30.

Krebs

J. R.

Kacelnik

Taylor

(1978). Test of optimal sampling by foraging great tits. Nature, 275(5675), 27–31. https://doi.org/10.1038/275027a0

31.

Krebs

J. R.

Ryan

J. C.

Charnov

E. L.

(1974). Hunting by expectation or optimal foraging? A study of patch use by chickadees. Animal Behaviour, 22(Pt 4), 953–964. https://doi.org/10.1016/0003-3472(74)90018-9

32.

Kruger

J. M.

Dunning

(1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121–1134. https://doi.org/10.1037/0022-3514.77.6.1121

33.

Lynch

D. J.

(2006). Motivational factors, learning strategies and resource management as predictors of course grades. College Student Journal, 40(2), 423–429.

34.

Makara

K. A.

Karabenick

S. A.

(2013). Characterizing sources of academic help in the age of expanding educational technology: A new conceptual framework. In Karabenick

S. A.

Puustinen

(Eds.), Advances in help seeking research and applications: The role of information and communication technologies (pp. 37–72). Information Age.

35.

Nakagawa

Schielzeth

(2013). A general and simple method for obtaining R² from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2), 133–142. http://doi.org/10.1111/j.2041-210x.2012.00261.x

36.

Organisation for Economic Co-operation and Development. (2018). Education spending. https://data.oecd.org/eduresource/education-spending.htm#indicator-chart

37.

Pintrich

P. R.

Smith

D. A.

Garcia

McKeachie

W. J.

(1991). A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). University of Michigan, National Center for Research to Improve Postsecondary Teaching and Learning. http://eric.ed.gov/?id=ED338122

38.

Pintrich

P. R.

Smith

D. A.

Garcia

McKeachie

W. J.

(1993). Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ). Educational and Psychological Measurement, 53(3), 801–813. https://doi.org/10.1177/0013164493053003024

39.

Ryan

A. M.

Pintrich

P. R.

Midgley

(2001). Avoiding seeking help in the classroom: Who and why? Educational Psychology Review, 13(2), 93–114. https://doi.org/10.1023/a:1009013420053

40.

Szu

Nandagopal

Shavelson

R. J.

Lopez

E. J.

Penn

J. H.

Scharberg

Hill

G. W.

(2011). Understanding academic performance in organic chemistry. Journal of Chemical Education, 88(9), 1238–1242. https://doi.org/10.1021/ed900067m

41.

Thille

Schneider

Kizilcec

R. F.

Piech

Halawa

S. A.

Greene

D. K.

(2014). The future of data-enriched assessment. Research & Practice in Assessment, 9, 5–16.

42.

Tourangeau

Rips

L. J.

Rasinski

(2000). The psychology of survey response. Cambridge University Press. https://doi.org/10.1017/cbo9780511819322

43.

Wasserstein

R. L.

Schirm

A. L.

Lazar

N. A.

(2019). Moving to a world beyond “p < 0.05.” The American Statistician, 73(Suppl. 1), 1–19. https://doi.org/10.1080/00031305.2019.1583913

44.

Weinstein

C. E.

Palmer

D. R.

Shulte

(2002). Learning and Study Strategies Inventory (LASSI). H&H.

45.

Weinstein

C. E.

Zimmermann

S. A.

Palmer

D. R.

(1988). Assessing learning strategies: The design and development of the LASSI. In Weinstein

C. E.

Goetz

E. T.

Alexander

P. A.

(Eds.), Learning and study strategies (pp. 25–40). Academic Press.

46.

Winne

P. H.

Hadwin

A. F.

(2012). The weave of motivation and self-regulated learning. In Schunk

D. H.

Zimmerman

B. J.

(Eds.), Motivation and self-regulated learning (pp. 297–314). Routledge.

47.

The World Bank Group. (2018). Government expenditure on education, total (% of GDP). https://data.worldbank.org/indicator/SE.XPD.TOTL.GD.ZS

48.

Zimmerman

B. J.

(1990). Self-regulated learning and academic achievement: An overview. Educational Psychologist, 25(1), 3–17. https://doi.org/10.1207/s15326985ep2501_2

49.

Zimmerman

B. J.

(2008). Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45(1), 166–183. https://doi.org/10.3102/0002831207312909

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.21 MB