Early Preparation and Inspiration for STEM Careers: Preliminary Report of the Epidemiology Challenge Randomized Intervention,2014-2015

Abstract

Objectives:

Early Preparation and Inspiration for Careers in the Biomedical Sciences (EPIC) is a university–high school partnership for increasing high school student interest and persistence in the biomedical sciences. EPIC includes a year-long, project-based learning intervention, the Think Like an Epidemiologist Challenge (Epi Challenge). We describe the main components of the Epi Challenge and report on short-term changes in scientific literacy and science-related motivations and beliefs.

Methods:

From June 2014 through June 2015, a randomized sample of students with above-median interest in science from 5 high schools in Pennsylvania completed baseline and midyear assessments of scientific self-efficacy, beliefs regarding acquisition of scientific knowledge (personal scientific epistemology), and personal interest in science using 5-point Likert-type scales (with higher scores indicating stronger or more sophisticated beliefs).

Results:

Of 984 students completing baseline assessments, 110 enrolled in the Epi Challenge, and 84 remained at midyear. At midyear, mean scores for scientific self-efficacy (change = 0.26, P < .001) and personal scientific epistemology (change = 0.19, P = .004) increased significantly, but personal interest in science (change = −0.17, P = .06) did not. Increases in personal scientific epistemology were greatest for African American (change = 0.47, P = .005), free/reduced-price lunch (change = 0.35, P = .001), underrepresented minorities in science (change = 0.27, P = .002), and female (change = 0.26, P = .01) students.

Conclusions:

Epi Challenge participation was associated with improvement in high school students’ scientific self-efficacy and sophistication of epistemologic beliefs. Long-term follow-up of this cohort may shed light on whether such changes will be sustained and shape college major and career decisions.

Keywords

epidemiology STEM epistemology project-based learning science literacy

In 2015, the National Science Board noted concerns about the lack of racial/ethnic diversity in science, technology, engineering, and mathematics (STEM) education.¹ In 2016, students from underrepresented racial/ethnic minority groups, including African American, Hispanic, Native Hawaiian/other Pacific Islander, and American Indian/Alaska Native populations, constituted only 23% of first-time enrollees in US graduate programs.² Despite improvements since 1995 in proportions of people from racial/ethnic minority groups among STEM graduates, in 2014, people from underrepresented racial/ethnic minority groups earned just 20.3%, 14.5%, and 8.0% of science and engineering bachelor’s, master’s, and doctoral degrees, respectively.³

The shortage of STEM graduates has roots in elementary and high school.⁴ In 2016, only 20% of American College Testing–tested US high school students attained the benchmark or higher for STEM scores. Among the subset of students (48%) who indicated an overall interest in STEM subjects and careers, only 26% met the benchmark or higher, and percentages were considerably lower for those in underrepresented racial/ethnic minority groups.⁴

Some experts have suggested that epidemiologic principles could be used in kindergarten through 12th-grade education to address the gaps in STEM education “by fostering opportunities for multidisciplinary learning around specific problems and by enabling students to grasp the relevance of real-world applications of mathematics and science.”⁵ In 2013, our research team established Early Preparation and Inspiration for Careers in the Biomedical Sciences (EPIC), a National Institutes of Health–funded, university–high school partnership that uses epidemiologic principles in STEM education. EPIC combines a career interest assessment and counseling program with a year-long, project-based learning intervention starting the summer after ninth grade. As part of EPIC, we implemented an intervention called the Think Like an Epidemiologist Challenge (hereinafter, Epi Challenge) from June 2014 through June 2015. In this article, we describe the main components of the Epi Challenge and report on short-term changes in the scientific literacy and science-related motivations and beliefs of students who participated.

Methods

Members of our research team have been involved since 2000 in various efforts to provide epidemiology education in grades 6 through 12. These efforts have included participation in science competitions such as the Young Epidemiology Scholars⁶ and New Jersey Science Olympiad,⁷ development of epidemiology curricula for younger students,^8
–10 and preparing teachers to teach epidemiology.^11,12 From 2011 through 2014, we engaged regional school districts in Pennsylvania in 4 annual 1-day workshops, called Disease Scene Investigators, for high school teachers and students. The positive community experience with these workshops, combined with career assessment and counseling offered in Pennsylvania schools,¹³ led to the opportunity to investigate the impact of an epidemiology intervention on decisions about STEM majors and careers.

In consultation with area school districts, we invited 5 high schools to partner in EPIC if they had participated in at least 3 of the 4 Disease Scene Investigator workshops, had available administrative support, and had sufficient diversity to ensure that at least 50% of participants in a stratified random sample of students would be classified as an underrepresented racial/ethnic minority group in science or as living in a low-income household (URM-S/LI).

We classified students as being URM-S/LI if they self-reported being (1) in an underrepresented minority in science (ie, Hispanic and all nonwhite/non-Asian racial/ethnic minority groups) or (2) a recipient of free or reduced-price lunch (hereinafter, free/reduced lunch), a surrogate for low income. We also classified students as URM-S/LI if they reported being in a racial/ethnic minority group and the low-income category. Students who did not qualify as URM-S/LI were designated as non–URM-S/LI. Coaches for EPIC in each school included at least 1 of its science educators and at least 1 student from a nearby university graduate program in public health or education. The research team held 40 hours of training for participating coaches.

Assessments of Scientific Literacy and Interest in Science

To assess EPIC students’ scientific literacy and their motivations and beliefs, we used the discipline-neutral Science Literacy Assessment (SLA) instrument, which was developed and validated by Fives et al¹⁴ in collaboration with the National Institutes of Health–Science Education Partnership Award program.¹⁵ The SLA has 2 parts: the SLA-Demonstrated Ability (SLA-D), which uses a 19-item multiple-choice instrument to assess demonstrated scientific literacy, including critical thinking and problem solving; and the SLA-Motivation and Beliefs (SLA-MB), which addresses motivations and beliefs with regard to science.

The SLA-MB is based on adaptations of 3 existing scaled measures (Table 1). The first measure is Value of Science, which is one’s beliefs in the value of science; it has 6 items (eg, “For me, being good in science is important”). The second measure is Scientific Self-Efficacy, which measures one’s beliefs in one’s capabilities to achieve a scientific goal or outcome; it has 8 items (eg, “I know how to use the scientific method to solve problems”). The third measure is Personal Scientific Epistemology, which is one’s beliefs about the sources and certainty of scientific knowledge; it has 11 items (eg, “If you read something in a science book, you can be sure it is true”). We scored questions for each measure on a 5-point Likert-type scale, whereby a higher score indicated stronger or more sophisticated beliefs in the value of science, in one’s capabilities to achieve a goal or outcome, and in the acquisition of knowledge, respectively, for the 3 scales. We developed a fifth instrument to measure Personal Interest in Science, which we scored on a 5-point Likert-type scale (where 1 = definitely yes and 5 = definitely no); this instrument consisted of 5 statements about interest in spending time working on science (eg, “I would like to spend less time in school studying science”).

Table 1.

Characteristics of instruments and domains used to gauge science literacy and personal interest in science among ninth-grade students at 5 schools in the EPIC^a Epi Challenge,^b Pennsylvania, June 2014 to June 2015

Characteristic	Instrument
Characteristic	Science Literacy Assessment–Demonstrated Ability^c,d	Science Literacy Assessment–Motivation and Beliefs^c			Personal Interest in Science^e
Sub-instrument scale	No sub-instrument	Value of Science^f	Scientific Self-Efficacy^g	Personal Scientific Epistemology^h	No sub-instrument
No. of items in instrument	19	6	8	11	5
Domains	Role of science Scientific thinking and doing Science and society Mathematics in science	Single domain of valuing science	Single domain of self-efficacy in science	Source (5 items) Certainty (6 items)	Interest in spending time on science
Sample items	You notice an advertisement on the internet selling green tea that claims, “Green tea makes you smart.” Choose which of the following kinds of evidence would help to scientifically assess this claim: (a) A list of famous smart people who drink green tea. (b) A comparison of intelligence tests from people who do and do not drink green tea. (c) An article by a man who said his intelligence increased by 30% after drinking green tea every day for 1 year. (d) None of the preceding could help to scientifically assess this claim.	For me, being good in science is important.	I know how to use the scientific method to solve problems.	If you read something in a science book, you can be sure it is true.	Science is one of my favorite subjects. I would like a job that uses science. I would like to spend less time in school studying science. I would be interested in participating in an after-school science club. When I hear the word science, I have a feeling of dislike.
Measurement range/scale	Multiple-choice items (0-19 correct)	Likert (1-5)	Likert (1-5)	Likert (1-5)	Likert (1-5)
Source	Fives et al (2014)¹⁴	Wigfield and Eccles (2000),¹⁶ Fives et al (2014)¹⁴	Ketelhut (2011)¹⁷	Conley et al (2004)¹⁸	Huebner et al (2014)^e

^aEarly Preparation and Inspiration for Careers in the Biomedical Sciences (EPIC) is a National Institutes of Health–funded, university–high school partnership that uses epidemiologic principles in science, technology, engineering, and mathematics education.

^bEpi Challenge, the Think Like an Epidemiologist Challenge, is a year-long, project-based learning intervention offered by EPIC.

^cThe Science Literacy Assessment instrument was developed and validated by Fives et al¹⁴ in collaboration with the National Institutes of Health Science Education Partnership Award program.¹⁵

^dScience Literacy Assessment–Demonstrated Ability assesses demonstrated scientific literacy, including critical thinking and problem solving.

^ePersonal Interest in Science assesses level of interest in science, scored on a 5-point Likert-type scale ranging from 1 = definitely yes to 5 = definitely no (eg, “I would like to spend less time in school studying science”). The instrument was developed by one of the authors (W.H., unpublished, 2014).

^fValue of Science assesses beliefs in the value of science, scored on a 5-point Likert-type scale ranging from 1 = not at all important to 5 = very important (or interesting/not interesting) (eg, “For me, being good in science is important”).

^gScientific Self-Efficacy assesses beliefs in one’s capabilities to achieve a scientific goal or outcome, scored on a 5-point Likert-type scale ranging from 1 = strongly disagree to 5 = strongly agree (eg, “I know how to use the scientific method to solve problems”).

^hPersonal Scientific Epistemology assesses beliefs about the sources and certainty of scientific knowledge, scored on a 5-point Likert-type scale ranging from 1 = strongly disagree to 5 = strongly agree (eg, “If you read something in a science book, you can be sure it is true”).

Sampling and Randomization

Before the Epi Challenge was implemented at the 5 high schools, all ninth-grade students at the schools were offered an opportunity to respond to all 5 parts of the SLA and Personal Interest in Science instruments during the latter part of the school year. Using these baseline scores, we conducted a stepwise, stratified, random-sampling process to select potential participants in the Epi Challenge intervention. Students were initially placed into 1 of 4 groups in each school: URM-S/LI girl, URM-S/LI boy, non–URM-S/LI girl, and non–URM-S/LI boy. Then, to ensure a minimum baseline interest in science among Epi Challenge participants, we chose for randomization only those students in each of the 4 groups who were at the median or higher for the Personal Interest in Science instrument.

We assigned a random number to each student, listed the students in numerical order in each of the 4 groups, and then blindly selected every other student on each list in order and placed them into the Epi Challenge intervention group. To accommodate an expected 35% overall dropout rate and sustain a final sample size of 80 in the intervention, we aimed to fill 26 participant spots from each of the 5 schools (130 students in total), leaving approximately 800 or more students who had responded to the assessment instruments but were not included in the Epi Challenge intervention.

Epi Challenge participants had a year-long experience beginning in the summer after ninth grade and concluding in the summer after 10th grade (June 2014 through June 2015). To Epi Challenge participants only, we offered the chance to respond again to the assessment instruments in the middle of their sophomore year, creating midyear scores. Both Epi Challenge participants and nonparticipants responded again to all 5 instruments at the end of high school (in spring 2017) (results not reported).

Epi Challenge Framework and Components

We used the 8 practices in the National Research Council’s Framework for K-12 Science Education ¹⁹ to guide the Epi Challenge experience. As the Framework document states, “Students cannot comprehend scientific practices, nor fully appreciate the nature of scientific knowledge, without directly experiencing these practices themselves.”¹⁹ We developed 5 major activity components of the Epi Challenge.

Intensive summer training

Rising 10th-grade students attended a 1-week, 40-hour summer program that included lectures, games, and hands-on and team-building activities. Lecture topics included current and emerging public health issues, principles of epidemiology, and roles of epidemiologist teams. We developed a website for students, teachers, and parents to access training materials and other related information during the school year.²⁰

Team participation

Educator coaches assigned students to teams of 3-5 members from the same school. Team members were instructed in basic elements of the National Institutes of Health code of conduct for collaborative research teams.²¹ Coaches encouraged team members to identify leaders for each section of the research plan rather than appoint 1 overall delegator/leader.

Simulated research project

During the summer training and ensuing school year, we asked teams to make a list of health-related behaviors of interest and to develop a hypothesis about a health behavior and its association with an outcome of interest (eg, “Students who spend more time on social media are more likely to be sleep deprived compared with students who spend less time on social media”). After they developed a research proposal for a cross-sectional study that had to be approved by the school principal, teams proceeded with the ethical conduct, analysis, interpretation, and communication of their investigations. We set up the Epi Challenge to give students an authentic inquiry experience “in which the conclusion is not obvious, the issues are murky, and the situation is ambiguous in terms of what knowledge and skills are required.”²²

Project-based learning

We instructed Epi Challenge coaches in the principles of project-based learning, a teaching method in which students gain knowledge and skills by investigating and responding to a complex question or problem. We emphasized the importance of enhancing student interest and ownership without immediately supplying an answer or conveying the solution and by encouraging students to explore, think critically, and problem solve as a team.²³

Interactions with stakeholders

Student teams presented their research proposals to other students and local and regional public health professionals at a midyear research symposium. At year’s end, students practiced presenting their investigations in oral and poster sessions at a 2-day summer session followed by a final symposium event that involved parents and guardians, school administrators, experts in the biomedical sciences, and public health practitioners.

Statistical Methods

To assess reliability of our scaled measures, we calculated standardized and deleted variables analysis Cronbach’s alpha coefficients for the 3 SLA-MB and Personal Interest in Science scores, to determine if multiple items about a particular measure (eg, Value of Science) were closely related as a group. We also assessed correlations among all 5 instrument scores, calculating Spearman (nonparametric) correlation coefficients (r) and P values, because the scores were not normally distributed. We considered P < .05 to be significant.

We assessed differences in the baseline distributions of Epi Challenge participants and nonparticipants and of those who remained in and those who dropped out of the Epi Challenge, stratified by their school, sex, race/ethnicity, Hispanic origin, free/reduced lunch, and URM-S/LI status using the Pearson χ² test. When cell populations were ≤5 students, we used the Fisher exact test to assess these differences.

For the SLA-D instrument, we classified missing answers as incorrect and then recorded the total correct answers (of a possible 19) for each student. For the 4 instruments that used 5-point Likert-type scales (3 SLA-MB scales and Personal Interest in Science), we required that at least 80% of questions be answered to include a student’s score in the baseline mean scores for Epi Challenge participants and nonparticipants and for analysis of the other demographic subgroups. To prevent overrepresentation of information from entries that were less than 100% complete, we then weighted the mean scores for these groups and subgroups according to the total number of questions that were answered in each instrument. We also revised the scoring of some scales so that the highest ranking (ie, 5 on a 1-5 scale) always reflected the most positive outcome.

For the Personal Interest in Science instrument, because a score of 3 on the 5-point Likert-type scale reflected uncertainty about this measure, we used baseline and midyear scores to measure the increase in certainty that occurred during the project, which we called PSI_certain. We defined PSI_certain as the absolute value of moving from uncertainty toward either higher interest (eg, a score of 4 or 5) or lower interest (eg, a score of 1 or 2) in science.

We assessed differences in the baseline mean SLA-D, SLA-MB, and Personal Interest in Science scores (and differences in these scores by demographic characteristics) between Epi Challenge participants and nonparticipants, and between those who remained in and those who dropped out of the Epi Challenge, using the nonparametric Wilcoxon rank-sum test. Then, for Epi Challenge participants, we determined pairwise differences among the baseline to midyear changes in Scientific Self-Efficacy, Personal Scientific Epistemology, and Personal Interest in Science mean scores, using the Wilcoxon signed rank test. Also, we conducted a sensitivity analysis of pairwise differences to determine the possible changes in change values if Epi Challenge participants who dropped out had stayed in the study but reported no changes from their baseline scores. We conducted all 2-sided statistical tests using SAS version 9.4.²⁴

Results

A total of 984 of 1303 ninth-grade students in the 5 schools completed the baseline assessment; 497 of 621 (80.0%) girls and 487 of 682 (71.4%) boys responded. Of 207 students invited to join the Epi Challenge, which required year-long participation including 2 summer sessions, 110 (53.1%) accepted and enrolled in the Epi Challenge (range at the 5 schools: 33.3%-84.0%). By midyear, 84 of 110 (76.4%) students were still in the program (range at the 5 schools: 25.5% to 96.2%). Of the 26 students who dropped out, 7 moved or started online school, 2 lost interest, and 17 did not indicate a reason for dropping out.

We found no significant differences at baseline between Epi Challenge participants and nonparticipants for race/ethnicity, Hispanic origin, free/reduced lunch, URM-S/LI status, SLA-D, and Personal Scientific Epistemology (Table 2). Epi Challenge participants were significantly more likely than nonparticipants to be girls (P = .01) and to score higher on Value of Science, Scientific Self-Efficacy, and Personal Interest in Science (all Ps < .001). Compared with the 84 students who remained enrolled in the Epi Challenge at midyear, the 26 students who dropped out of the Epi Challenge were significantly more likely to be in the URM-S/LI group (P = .02) and to have lower scores for SLA-D (P = .005), Value of Science (P = .003), Scientific Self-Efficacy (P = .04), and Personal Interest in Science (P = .006) (Table 3).

Table 2.

Demographic characteristics and baseline mean science literacy and interest scale scores of ninth-grade students enrolled and not enrolled in the EPIC^a Epi Challenge,^b Pennsylvania, June 2014 to June 2015

	Enrolled in Epi Challenge	Not Enrolled in Epi Challenge
Variable	(N = 110)	(N = 874)	P Value^c
School, No. (%)^d			<.001
A	26 (23.6)	331 (37.9)
B	12 (10.9)	115 (13.2)
C	21 (19.1)	71 (8.1)
D	25 (22.7)	230 (26.3)
E	26 (23.6)	127 (14.5)
Sex, No. (%)^d			.01
Male	42 (38.2)	445 (50.9)
Female	68 (61.8)	429 (49.1)
Race/ethnicity, No. (%)^d
White	63 (57.3)	569 (65.1)	.11
African American	24 (21.8)	163 (18.6)	.43
Asian	3 (2.7)	27 (3.1)	.99
Other race	20 (18.2)	115 (13.2)	.15
Hispanic origin, No. (%)^d	17 (15.5)	119 (13.6)	.60
Free or reduced-price lunch, No. (%)^d
Yes	39 (35.5)	291 (33.3)	.07
No	53 (48.2)	502 (57.4)	.29
Don’t know	18 (16.4)	81 (9.3)	.11
URM-S/LI,^e No. (%)^d			.18
Yes	62 (56.4)	433 (49.5)
No	48 (43.6)	441 (50.5)
Science literacy and interest scales, mean score (SD) [No. tested]
Demonstrated Ability^f	9.7 (3.8) [110]	9.2 (4.3) [874]	.18
Value of Science^g	3.8 (0.7) [109]	3.3 (0.9) [852]	<.001
Scientific Self-Efficacy^h	3.8 (0.7) [109]	3.5 (0.9) [850]	<.001
Personal Scientific Epistemologyⁱ	3.9 (0.7) [109]	3.9 (0.8) [847]	.57
Personal Interest in Science^j	3.8 (0.7) [110]	3.0 (1.0) [845]	<.001

Abbreviation: URM-S/LI, underrepresented minority in science/low-income classification.

^bEpi Challenge, the Think Like an Epidemiologist Challenge, is a year-long, project-based learning intervention offered by EPIC.

^c P values for differences in demographic characteristics were determined by using the Pearson χ² test, or the Fisher exact test when cell populations were ≤5. For differences in science literacy and interest scales, the nonparametric Wilcoxon rank-sum test was used. Significance was set at α <.05.

^dNot all percentages total to 100.0 because of rounding.

^eURM-S/LI includes Hispanic and all nonwhite/non-Asian racial/ethnic minority groups, those classified as eligible for free or reduced-price lunch, or those who have both characteristics (racial/ethnic minority and low income).

^fScience Literacy Assessment–Demonstrated Ability assesses demonstrated scientific literacy, including critical thinking and problem solving.

^gValue of Science assesses beliefs in the value of science, scored on a 5-point Likert-type scale ranging from 1 = not at all important to 5 = very important (or interesting/not interesting) (eg, “For me, being good in science is important”).

^hScientific Self-Efficacy assesses beliefs in one’s capabilities to achieve a scientific goal or outcome, scored on a 5-point Likert-type scale ranging from 1 = strongly disagree to 5 = strongly agree (eg, “I know how to use the scientific method to solve problems”).

ⁱPersonal Scientific Epistemology assesses beliefs about the sources and certainty of scientific knowledge, scored on a 5-point Likert-type scale ranging from 1 = strongly disagree to 5 = strongly agree (eg, “If you read something in a science book, you can be sure it is true”).

^jPersonal Interest in Science assesses level of interest in science, scored on a 5-point Likert-type scale ranging from 1 = definitely yes to 5 = definitely no (eg, “I would like to spend less time in school studying science”). The instrument was developed by one of the authors (W.H., unpublished, 2014).

Table 3.

Demographic characteristics and midyear mean scientific literacy and interest in science scores of ninth-grade students who remained in and those who dropped out of the EPIC^a Epi Challenge,^b Pennsylvania, June 2014 to June 2015

	Remained in Epi Challenge	Dropped Out of Epi Challenge		Started in Epi Challenge (Baseline)
Category	(n = 84)	(n = 26)	P Value^c	(N = 110)
School, No. (%)^d
A	25 (29.8)	1 (3.9)	.007	26 (23.6)
B	3 (3.6)	9 (34.6)	<.001	12 (10.9)
C	13 (15.5)	8 (30.8)	.08	21 (19.1)
D	23 (27.4)	2 (7.7)	.04	25 (22.7)
E	20 (23.8)	6 (23.1)	.94	26 (23.6)
Sex, No. (%)^d
Male	36 (42.9)	6 (23.1)	.07	42 (38.2)
Female	48 (57.1)	20 (76.9)	.07	68 (61.8)
Race/ethnicity, No. (%)^d
White	56 (66.7)	7 (26.9)	<.001	63 (57.3)
African American	15 (17.9)	9 (34.6)	.07	24 (21.8)
Asian	3 (3.6)	0 (0.0)	.33	3 (2.7)
Other race	10 (11.9)	10 (38.5)	.002	20 (18.2)
Hispanic origin, No. (%)^d	12 (14.3)	5 (20.0)	.49	17 (15.6)
Free or reduced-price lunch, No. (%)^d
Yes	29 (34.5)	10 (38.5)	.71	39 (35.5)
No	46 (54.8)	7 (26.9)	.01	53 (48.2)
Don’t know	9 (10.7)	9 (34.6)	.004	18 (16.4)
URM-S/LI,^e No. (%)^d
Yes	42 (50.0)	20 (76.9)	.02	62 (56.4)
No	42 (50.0)	6 (23.1)	.02	48 (43.6)
Science literacy and interest scales, mean score (SD)
Demonstrated Ability^f	10.3 (3.4)	7.7 (4.3)	.005	9.7 (3.8)
Value of Science^g	3.9 (0.6)	3.5 (0.8)	.003	3.8 (0.7)
Scientific Self-Efficacy^h	3.8 (0.8)	3.6 (0.6)	.04	3.8 (0.7)
Personal Scientific Epistemologyⁱ	3.9 (0.7)	3.8 (1.0)	.64	3.9 (0.7)
Personal Interest in Science^j	3.9 (0.6)	3.4 (0.8)	.006	3.8 (0.7)

Abbreviation: URM-S/LI, underrepresented minority in science/low-income classification.

^bEpi Challenge, the Think Like an Epidemiologist Challenge, is a year-long, project-based learning intervention offered by EPIC.

^c P values for differences in demographic characteristics were determined by using the Pearson χ² test, or the Fisher exact test when cell populations were ≤5. For differences in scientific literacy and interest scales, the nonparametric Wilcoxon rank-sum test was used. Significance was set at α <.05.

^dNot all percentages total to 100.0 because of rounding.

^fScience Literacy Assessment–Demonstrated Ability assesses demonstrated scientific literacy, including critical thinking and problem solving.

SLA-D scores were significantly correlated with baseline scores for all 4 scaled measures (r range: 0.31-0.47; all Ps < .001) (Table 4). We observed the strongest correlations between Value of Science and Scientific Self-Efficacy (r = 0.66; P < .001) and between Value of Science and Personal Interest in Science (r = 0.72, P <.001). Personal Scientific Epistemology was not strongly correlated with the other 3 scaled measures (r ≤ 0.13). Cronbach’s alpha for each of the 4 scaled measures was >.80 for all groups investigated by sex, race/ethnicity (African American and Hispanic), URM-S/LI, and free/reduced lunch.

Table 4.

Spearman correlation coefficient (r) matrix for scale scores of high school students responding to scientific literacy and interest instruments before the EPIC^a Epi Challenge,^b Pennsylvania, spring 2014

Instrument	Demonstrated Ability^c (n = 984), Spearman r	Spearman r (P Value)^d
Instrument	Demonstrated Ability^c (n = 984), Spearman r	Value of Science^e (n = 961)	Scientific Self-Efficacy^f (n = 959)	Personal Scientific Epistemology^g (n = 956)	Personal Interest in Science^h (n = 955)
Demonstrated Ability^c	1.00	0.31 (<.001)	0.47 (<.001)	0.36 (<.001)	0.32 (<.001)
Value of Science^e	—	1.00	0.66 (<.001)	–0.02 (.46)	0.72 (<.001)
Scientific Self-Efficacy^f	—	—	1.00	0.13 (<.001)	0.56 (<.001)
Personal Scientific Epistemology^g	—	—	—	1.00	0.04 (.26)
Personal Interest in Science^h	—	—	—	—	1.00

^bEpi Challenge, the Think Like an Epidemiologist Challenge, is a year-long, project-based learning intervention offered by EPIC.

^cScience Literacy Assessment–Demonstrated Ability assesses demonstrated scientific literacy, including critical thinking and problem solving.

^dCorrelations and P values among all 5 instrument scores were calculated using Spearman’s (nonparametric) correlation coefficients (r). Significance was set at P < .05.

^eValue of Science assesses beliefs in the value of science, scored on a 5-point Likert-type scale ranging from 1 = not at all important to 5 = very important (or interesting/not interesting) (eg, “For me, being good in science is important”).

^fScientific Self-Efficacy assesses beliefs in one’s capabilities to achieve a scientific goal or outcome, scored on a 5-point Likert-type scale ranging from 1 = strongly disagree to 5 = strongly agree (eg, “I know how to use the scientific method to solve problems”).

^gPersonal Scientific Epistemology assesses beliefs about the sources and certainty of scientific knowledge, scored on a 5-point Likert-type scale ranging from 1 = strongly disagree to 5 = strongly agree (eg, “If you read something in a science book, you can be sure it is true”).

^hPersonal Interest in Science assesses level of interest in science, scored on a 5-point Likert-type scale ranging from 1 = definitely yes to 5 = definitely no (eg, “I would like to spend less time in school studying science”). The instrument was developed by one of the authors (W.H., unpublished, 2014).

Of the 84 students who remained in the Epi Challenge at midyear, we found a significant increase from baseline to midyear in the mean Scientific Self-Efficacy score for all students (change = 0.26, P < .001), and this increase was largest for non-African American students (change = 0.34, P = .03), students not receiving free/reduced lunch (change = 0.35, P = .004), non–URM-S/LI students (change = 0.37, P = .008), and boys (change = 0.34, P = .006) (Table 5). The mean Personal Scientific Epistemology score for all Epi Challenge students also increased (change = 0.19, P = .004); however, increases in these scores were highest for African American students (change = 0.47, P = .005), students receiving free/reduced lunch (change = 0.35, P = .001), URM-S/LI students (change = 0.27, P = .002), and girls (change = 0.26, P = .01). In contrast with Scientific Self-Efficacy and Personal Scientific Epistemology, the mean Personal Interest in Science score for all Epi Challenge students decreased, but not significantly (change = –0.17, P = .06), although the decrease in mean score (change = –0.32, P = .02) for students receiving free/reduced lunch was significant. For students with a Personal Interest in Science score of 3 at baseline, mean PSI_certain results at midyear suggested a nonsignificant trend toward certainty (change = 0.08, P = .14).

Table 5.

Changes in mean Scientific Self-Efficacy,^a Personal Scientific Epistemology,^b and Personal Interest in Science^c scores from ninth-grade year-end (baseline) to 10th-grade midyear among 84 high school students enrolled in the EPIC^d Epi Challenge,^e Pennsylvania, June 2014 to June 2015

Category	No. of Students	Mean Score From Baseline to Midyear (Change) [P Value^f]
Category	No. of Students	Scientific Self-Efficacy^a	Personal Scientific Epistemology^b	Personal Interest in Science^c
School
A	25	3.78-4.17 (0.39) [.09]	3.80-3.96 (0.17) [.30]	4.02-3.78 (–0.23) [.12]
B	3	3.45-3.54 (0.08) [.50]	2.85-3.45 (0.61) [.25]	3.47-2.80 (–0.67) [.75]
C	13	3.86-3.76 (–0.10) [.40]	4.01-4.28 (0.26) [.14]	3.88-3.54 (–0.34) [.04]^f
D	23	3.92-4.21 (0.29) [.06]	4.13-4.24 (0.11) [.26]	3.87-3.81 (–0.06) [.89]
E	20	3.88-4.21 (0.33) [.01]^f	3.91-4.11 (0.20) [.11]	3.82-3.80 (–0.02) [.99]
P value^g		.13	.47	.20
Sex
Male	36	3.71-4.05 (0.34) [.006]^f	3.95-4.05 (0.10) [.17]	3.78-3.56 (–0.24) [.07]
Female	48	3.94-4.14 (0.21) [.07]	3.89-4.15 (0.26) [.01]^f	3.95-3.85 (–0.11) [.09]
P value^g		.82	.40	>.99
URM-S/LI^h
Yes	42	3.74-3.90 (0.16) [.08]	3.81-4.09 (0.27) [.002]^f	3.84-3.69 (–0.20) [.09]
No	42	3.94-4.30 (0.37) [.008]^f	4.02-4.13 (0.11) [.30]	3.93-3.80 (–0.13) [.37]
P value^g		.17	.03^g	.88
Free or reduced-price lunch
Yes	29	3.68-3.87 (0.18) [.21]	3.78-4.13 (0.35) [.001]^f	3.88-3.55 (–0.32) [.02]^f
No	46	3.95-4.30 (0.35) [.004]^f	4.04-4.09 (0.06) [.56]	3.94-3.84 (–0.10) [.50]
Don’t know	9	3.78-3.85 (0.07) [.55]	3.73-4.08 (0.34) [.07]	3.67-3.67 (–0.02) [.91]
P value^g		.21	.05	.84
African American
Yes	15	3.81-3.71 (–0.10) [.37]	3.65-4.13 (0.47) [.005]^f	3.81-3.56 (–0.25) [.29]
No	69	3.85-4.19 (0.34) [<.001]^f	3.97-4.10 (0.13) [.08]	3.90-3.75 (–0.15) [.15]
P value^g		.03^g	.02^g	.77
Hispanic
Yes	12	3.93-4.09 (0.17) [.17]	3.98-4.13 (0.14) [.57]	3.97-3.83 (–0.13) [.23]
No	72	3.83-4.11 (0.28) [.003]^f	3.90-4.10 (0.20) [.005]^f	3.88-3.70 (–0.17) [.09]
P value^g		.41	.71	.83
All students	84	3.84-4.10 (0.26) [<.001]^f	3.92-4.11 (0.19) [.004]^f	3.89-3.72 (–0.17) [.06]

Abbreviation: URM-S/LI, underrepresented minority in science/low-income classification.

^aScientific Self-Efficacy assesses beliefs in one’s capabilities to achieve a scientific goal or outcome, scored on a 5-point Likert-type scale ranging from 1 = strongly disagree to 5 = strongly agree (eg, “I know how to use the scientific method to solve problems”).

^bPersonal Scientific Epistemology assesses beliefs about the sources and certainty of scientific knowledge, scored on a 5-point Likert-type scale ranging from 1 = strongly disagree to 5 = strongly agree (eg, “If you read something in a science book, you can be sure it is true”).

^cPersonal Interest in Science assesses level of interest in science, scored on a 5-point Likert-type scale ranging from 1 = definitely yes to 5 = definitely no (eg, “I would like to spend less time in school studying science”). The instrument was developed by one of the authors (W.H., unpublished, 2014).

^dEarly Preparation and Inspiration for Careers in the Biomedical Sciences (EPIC) is a National Institutes of Health–funded, university–high school partnership that uses epidemiologic principles in science, technology, engineering, and mathematics education.

^eEpi Challenge, the Think Like an Epidemiologist Challenge, is a year-long, project-based learning intervention offered by EPIC.

^fThe significance of the degree of change of each of the 3 scales for each demographic subgroup (eg, Scientific Self-Efficacy for school A) was evaluated using the nonparametric Wilcoxon signed rank test to assess paired differences between baseline and time 1 (midyear). Significance was set at α <.05.

^gThe significance of the degree of change of each of the 3 scales within a demographic category (eg, school) was evaluated using the nonparametric Wilcoxon rank-sum test. Significance was set at α <.05.

^hURM-S/LI includes Hispanic and all nonwhite/non-Asian racial/ethnic minority groups, those classified as eligible for free or reduced-price lunch, or those who have both characteristics (racial/ethnic minority and low income).

At midyear, mean Scientific Self-Efficacy scores were significantly lower for URM-S/LI students (3.90 vs 4.30, P < .001) and for African American students (3.71 vs 4.19, P = .004) than for non–URM-S/LI and non-African American students, respectively (midyear P values not shown). In contrast, mean Personal Scientific Epistemology and Personal Interest in Science scores did not differ significantly in any of the demographic categories at midyear.

In the sensitivity analysis, which assumed that students who dropped out of the Epi Challenge had stayed in the study and had no change from their baseline scores, increases in mean Scientific Self-Efficacy (change = 0.20, P = .001) and Personal Scientific Epistemology (change = 0.14, P = .004) scores remained significant, and changes in the mean Personal Interest in Science score (change = –0.13, P = .06) remained nonsignificant.

Discussion

We conducted a midyear analysis to assess early impacts of the Epi Challenge program while students were in the midst of their project work. We found significant increases in scores for Scientific Self-Efficacy and Personal Scientific Epistemology. To date, only a few studies involving project-based learning, nonrandomized STEM interventions among high school students have reported increased achievement in mathematics among Hispanic students,²⁵ changes in career aspirations,²⁶ and benefits to lower performing students compared with higher achieving students.²⁷ Other observational or descriptive studies have found an association between authentic research experiences in high school and an increased likelihood of completing a STEM major in college²⁸ and an increase in collaborative behavior.²⁹

Scientific Self-Efficacy

We found that Scientific Self-Efficacy was strongly correlated (0.66) with Value of Science in our data. This finding is consistent with expectancy-value research that reports that students often indicate that a task is valuable if they feel confident about succeeding (or vice versa).¹⁴ The significant correlation between Scientific Self-Efficacy and SLA-D was somewhat lower (0.47) but noteworthy. Increases in Scientific Self-Efficacy may lead to increased student proficiency in STEM. Although the hypothesis of whether involving students in project-based learning in high school leads to increased self-efficacy and subsequently increased proficiency in the sciences is only beginning to be more fully investigated, it should be pointed out that this logic underlies major policy statements in science and education (ie, hands-on learning in the sciences increases proficiency in the sciences).^19,30

–33 Further in-depth investigation is needed to explain other important factors related to self-efficacy in science.

Personal Scientific Epistemology

Our results on students’ epistemologic beliefs are consistent with earlier work reporting significant changes in beliefs about knowledge of science after a hands-on intervention. In 2004, Conley et al¹⁸ reported that elementary school students developed more sophisticated beliefs about the source and certainty of knowledge of science after a 9-week hands-on science unit, indicating a movement toward maturation of epistemologic beliefs. Involving students in study design and in data collection and analysis may promote epistemologic thinking, perhaps by helping students understand that answers to questions do not always come from authorities but can come from ongoing investigations and new discoveries.^18,34 Our findings at baseline are consistent with the findings of Conley et al,¹⁸ who reported significantly lower scores among students with lower socioeconomic status. By midyear, we found temporal increases in Personal Scientific Epistemology scores to be highest among Epi Challenge participants who were African American, low income, and female. Overall, baseline differences among the 5 schools disappeared by midyear.

More sophisticated or mature epistemologic beliefs are associated with higher levels of achievement and learning,³⁵ although the mechanism for this association is unclear. One hypothesis is that achievement and learning could be related to an increased awareness that scientific knowledge is not owned by a particular group but, instead, developed through constant review, debate, and revision. Interestingly, we observed almost no association between epistemologic beliefs and the other 2 SLA-MB scales (value and self-efficacy), both with correlation coefficients <0.15, which indicates that the various components of the SLA-MB instrument do indeed measure different domains of scientific literacy and beliefs.

Personal Interest in Science

We developed the Personal Interest in Science 5-item scale for this study to evaluate the willingness of students to invest personal time and resources into scientific inquiry. Our findings that the Cronbach’s alpha coefficients for Personal Interest in Science were all >0.80 and correlated positively with each of the 3 SLA-MB scores suggest reasonable construct validity for the Personal Interest in Science instrument. On a separate note, we observed a temporal decrease in Personal Interest in Science scores among Epi Challenge participants that was significant for students receiving free/reduced lunch and nonsignificant decreases across all other subgroups. When we retest Epi Challenge participants and nonparticipants in their senior year, we will assess overall trends in interest in science and possible differences between the 2 groups.

Demonstrated Scientific Literacy and Ability

The National Research Council identified scientific literacy as a key outcome to be measured when assessing the quality of K-12 science education.¹⁹ The SLA-D instrument has been initially validated as a way to measure scientific literacy.¹⁴ In our study, students correctly answered a mean of 9 of the 19 questions in the SLA-D. Although no published norms for the SLA-D exist, these results fall within the expected range of performance for the instrument.¹⁴ In addition, we found that SLA-D scores correlated moderately with the 3 SLA-MB scales, similar to results published by Fives et al¹⁴ and suggestive of a relationship between demonstrated knowledge and motivations and beliefs.

Limitations

This study had several limitations. Although we identified and evaluated students at baseline who did not participate in the Epi Challenge, this study provided interim midyear results for only Epi Challenge participants. So, although we observed significant changes in the scientific literacy of Epi Challenge participants, our analyses were limited by the lack of a control group. In addition, this interim analysis did not assess impacts of other possibly important factors, such as parental engagement, student absenteeism, previous hands-on STEM experience, and past levels of scientific achievement. Also, although we selected students at random to participate in the Epi Challenge (from those with above-median Personal Interest in Science scores), a proportion of students declined the invitation (with its considerable time commitment) or dropped out during the intervention, resulting in a form of self-selection that could have biased our results. However, our sensitivity analysis showed that our results would have been similar even if all Epi Challenge participants had remained in the program. In addition, when we performed a comparison of baseline data of Epi Challenge participants with the 874 nonparticipants, we found no significant difference in baseline SLA-D.

Conclusions

Using epidemiologic principles for STEM education can increase interest and ability in related fields, as students experience authentic scientific inquiry through practicing team science, hypothesis development, and critical evaluation of scientific evidence. In the short term, exposure to the Epi Challenge intervention was associated with improvements in our measures of self-efficacy and sophistication of students’ epistemologic beliefs. Follow-up of this cohort may shed light on whether the intervention will increase the number of students who choose a STEM college major or career.

Footnotes

Acknowledgments

The authors thank the following people who were instrumental in conducting this research: Sieta Achampong, Trent Baker, Lori Beaver, Michael Benner, Allison Blumling, Trea Buck, Michael Carnes, Vernon M. Chinchilli, Steff Chortanoff, Ralph Cordell, Jenn Crager, Nadine J. Crone, Jenn Fasick, Cheryl Friedman, Sarah Goodman, Jessica Gray, Betty Holmboe, John Huebner, Jennifer Jenkins, Dawn Koons, Chris Kroft, Sherrie LaPorta, Craig Livelsberger, Laura Luongo, Mark McCauslin, Jody McClymont, Todd Neuhard, Terri O’Neil, Diane J. Pague, Robin Perry-Smith, Audrey Plassio, Sunada Roberts, Ken Roseberry, Stacey Rossi, Eugene Spells, Shwetha Srikanth, Andrea Stennett, Emilie B. Tekely, Michael Thompson, Stacy Thorpe, Abbey Wascovich, and Jamal Wells.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by the National Institutes of Health NIGMS grant to RTW (1R01GM107715-01).

References

National Science Board. Revisiting the STEM Workforce: A Companion to Science & Engineering Indicators 2014. Arlington, VA: National Science Foundation; 2015. https://nsf.gov/nsb/publications/2015/nsb201510.pdf. Accessed October 30, 2017.

Okahana

Zhou

. Graduate Enrollment and Degrees: 2006-2016. Washington, DC: Council of Graduate Schools; 2017. http://cgsnet.org/graduate-enrollment-and-degrees. Accessed October 30, 2017.

National Center for Science and Engineering Statistics. Women, Minorities, and Persons With Disabilities in Science and Engineering. Arlington, VA: National Science Foundation; 2017. https://www.nsf.gov/statistics/2017/nsf17310. Accessed October 30, 2017.

American College Testing. The condition of STEM 2016. http://www.act.org/content/dam/act/unsecured/documents/STEM2016_52_National.pdf. Published 2016. Accessed December 22, 2016.

Stroup

Thacker

. Epidemiology and education: using public health for teaching mathematics and science. Public Health Rep. 2007;122(3):283–291.

Robert Wood Johnson Foundation and the College Board. Young Epidemiology Scholars competition. http://www.yes-competition.org/yes/index.html . Accessed November 7, 2017.

Epidemiology Education Movement. Think Like an Epidemiologist Challenge (Epi Challenge): New Jersey Science Olympiad Trial Event 2010-2013. http://www.epiedmovement.org/DOCS/NJSO.pdf. 2014. Accessed November 7, 2017.

Kaelin

Huebner

. Epidemiology, health literacy, and health education. Am J Health Educ. 2002;33(6):362–364.

Kaelin

Huebner

Nicolich

Kimbrough

. Field test of an epidemiology curriculum for middle school students. Am J Health Educ. 2007;38(1):16–31.

10.

Epidemiology Education Movement. Epidemiology curricula developed for grades 6-12. http://www.epiedmovement.org. 2007. Revised October 2015. Accessed November 7, 2017.

11.

Kaelin

Huebner

Cordell

Szklarczuk

. Professional development for prospective epidemiology teachers in grades 6-12. Public Health Rep. 2008;123(suppl 2):5–11.

12.

Robert Wood Johnson Foundation. Teach epidemiology. http://www.teachepidemiology.org. Published 2009. Accessed November 7, 2017.

13.

Pennsylvania Department of Education. Pennsylvania career education and work standards. http://www.education.pa.gov/K-12/PACareerStandards/Pages/default.aspx#tab-1 . Accessed November 7, 2017.

14.

Fives

Huebner

Birnbaum

Nicolich

. Developing a measure of scientific literacy for middle school students. Sci Educ. 2014;98(4):549–580.

15.

National Institutes of Health. Science Education Partnership Award. http://www.nihsepa.org. Accessed January 18, 2017.

16.

Wigfield

Eccles

. Expectancy-value theory of achievement motivation. Comtemp Educ Psychol. 2000;25(1):68–81.

17.

Ketelhut

. Assessing gaming, computer and scientific inquiry self-efficacy in a virtual environment. In: Annetta

Bronack

, eds. Serious Educational Game Assessment: Practical Methods and Models for Educational Games, Simulations and Virtual Worlds. Amsterdam, The Netherlands: Sense Publishers; 2011:1–18.

18.

Conley

Pintrich

Vekiri

Harrison

. Changes in epistemological beliefs in elementary science students. Contemp Educ Psychol. 2004;29(2):186–204.

19.

National Research Council. A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. Washington, DC: National Academies Press; 2012.

20.

Huebner

Wilson

. Epi Challenge for you: an authentic science research experience for high school students. http://epiedmovement.org/EPIChallenge.html . Published 2014. Accessed November 7, 2017.

21.

Bennett

Gadlin

Levine-Finley

. Collaboration and Team Science: A Field Guide. Washington, DC: National Institutes of Health; 2010.

22.

Wiggins

McTighe

. Understanding by Design. Alexandria, VA: Association for Supervision and Curriculum Development; 1998.

23.

Krajcik

McNeill

Reiser

. Learning-goals-driven design model: developing curriculum materials that align with national standards and incorporate project-based pedagogy. Sci Educ. 2008;92(1):1–32.

24.

SAS Institute, Inc. SAS Version 9.3. Cary, NC: SAS Institute, Inc; 2011.

25.

Han

Capraro

. How science, technology, engineering, and mathematics project based learning affects high-need students in the US. Learn Individ Diff. 2016;51:157–166.

26.

Duran

Höft

Lawson

Medjahed

Orady

. Urban high school students’ IT/STEM learning: findings from a collaborative inquiry- and design-based afterschool program. J Sci Educ Technol. 2014;23(1):116–137.

27.

Han

Capraro

. How science, technology, engineering and mathematics (STEM) project-based learning (PBL) affects high, middle, and low achievers differently: the impact of student factors on achievement. Int J Sci Math Educ. 2015;13(5):1089–1113.

28.

Subotnik

Tai

Almarode

. Study of the impact of selective SMT high schools: reflections on learners gifted and motivated in science and mathematics. Paper presented for the Committee on Highly Successful STEM Schools or Programs for K-12 STEM education. Washington, DC: National Research Council; 2011.

29.

Clark

Ernst

. A model for the integration of science, technology, engineering, and mathematics. Tech Teach. 2006;66(4):24–26.

30.

American Academy for the Advancement of Science. Benchmarks for Science Literacy. New York, NY: Oxford University Press; 1993.

31.

National Research Council. National Science Education Standards. Washington, DC: National Academies Press; 1996.

32.

Organisation for Economic Co-operation and Development. Assessing Scientific, Reading, and Mathematical Literacy: A Framework for Pisa 2006. Paris, France: OECD Publishing; 2006.

33.

National Assessment Governing Board. Science Framework for the 2011 National Assessment of Educational Progress. Washington, DC: US Department of Education; 2010. https://www.nagb.org/content/nagb/assets/documents/publications/frameworks/science/2011-science-framework.pdf. Accessed January 18, 2017.

34.

Solomon

Scott

Duveen

. Large-scale exploration of pupils’ understanding of the nature of science. Sci Educ. 1996;80(5):493–508.

35.

Hofer

Pintrich

. The development of epistemological theories: beliefs about knowledge and knowing and their relation to learning. Rev Educ Res. 1997;67(1):88–140.