What Happens When Trained Graduate Student Instructors Switch to an Open Textbook? A Controlled Study of the Impact on Student Learning Outcomes

Abstract

Open educational resources (OER) are increasingly attractive options for reducing educational costs, yet controlled studies of their efficacy are lacking. The current study addressed many criticisms of past research by accounting for course and instructor characteristics in comparing objective student learning outcomes across multiple sections of General Psychology taught by trained graduate student instructors at a large research-intensive university. We found no evidence that use of the OER text impeded students' critical thinking compared to use of a traditional textbook, even after accounting for instructor characteristics. To the contrary, we found evidence of a slight increase in content knowledge when using an OER text. Importantly, this effect was driven by improvements from both our lowest-performing students and our highest-performing students. Moreover, student learning outcomes were not influenced by instructor experience, suggesting even novice instructors fared well with OER materials. Finally, students from traditionally underserved populations reported the lower cost of the book had a significantly higher impact on their decision to enroll in and remain enrolled in the course.

Keywords

General Psychology graduate student instructors open textbooks

Instructors face a dilemma in selecting course textbooks. On one hand, they are important for students' success (Darwin, 2011; Skinner & Howes, 2013), and may even influence student learning more than the teaching methods themselves (Svickni & McKeachie, 2014). On the other hand, students across disciplines report using required textbooks far less than expected (e.g., Clump, Bauer, & Breadley, 2004; Podolefsky & Finkelstein, 2006), with around 60% of students avoiding the purchase altogether (Kinzie, 2006), likely due to prohibitive costs (see Redden, 2011).

Textbook costs have increased at sharper rates than tuition, fees, or housing (U.S. Bureau of Labor Statistics, 2016), which, together, increases demand for financial aid (Ma, Baum, Pender, & Welch, 2017; Popken, 2015). Given that earning a college degree is associated with greater income, health, relationship stability, and overall quality of life (Hout, 2012), increasing access to higher education by reducing costs—without sacrificing educational quality—is a social justice imperative. Indeed, many have challenged the notion that loan-based approaches are effective at ensuring upward mobility for low-income families and are instead calling for other forms of cost reduction (Goldrick-Rab, Kelchen, Harris, & Benson, 2016).

The most effective approach to reducing textbook costs is arguably the adoption of open educational resources (OERs), which are any educational material found in the public domain or introduced with an open publishing license (Butcher, 2015). Freely accessible to the student, OERs can be copied, printed, modified, and distributed at little to no cost (Bissell, 2009), and can be accessed and updated indefinitely. Furthermore, the digital form of most OERs includes many of the same benefits as e-texts (McFall, 2005, but cf. Singer & Alexander, 2017). Accordingly, OERs have become celebrated as a viable alternative that can drastically reduce financial burdens on students.

Despite these advantages and plentiful options (MERLOT, 2015), OERs still show low adoption rates by faculty, likely due to two key factors: low awareness of availability (Allen & Seaman, 2014), and avoidance due to expectations of low quality (Pitt, 2015). Interestingly, preference ratings for OERs are as good or better than traditional alternatives according to instructors' reviews (Allen & Seaman, 2014; Hilton, 2016; Jhangiani, Pitt, Hendricks, Key, & Lalonde, 2016), particularly within social sciences (Fischer, Ernst, & Mason, 2017). Students seem to prefer OERs, as well (Jhangiani & Jhangiani, 2017), and use of OERs has been associated with fewer course withdrawals (Clinton, 2018).

In synthesizing the results of a meta-analysis, Hilton (2016) concluded that “a strong majority of students and teachers believe that OERs are as good or better than traditional textbooks” (p. 585). Of course, as good or better perceptions of quality are far less important than as good or better learning outcomes. Here, the evidence is less clear, particularly within psychology. Some evidence indicates few, if any, differences in student learning between OER and traditional textbooks (Clinton, 2018; Hilton, 2016). However, much of the research examining the effects of OERs on student learning outcomes has been criticized (Griggs & Jackson, 2017a, 2017b), particularly regarding a failure to control for a range of potential confounds.

The current study addresses these criticisms. First, to address the criticism that past research has failed to control for instructor characteristics (Griggs & Jackson, 2017b), we used instructor experience and students' ratings of instructor characteristics (e.g., authoritative, creative, respectful) to account for differences in teaching proficiency between instructors. Further, we used multiple course sections taught by similarly trained graduate student instructors, a previously unstudied population of instructors, allowing us to average effects across instructors to control for instructor variability. Second, to address the criticism that past research has failed to control for course characteristics (such as course difficulty; Griggs & Jackson, 2017b), we used identical outcome measures across all course sections and textbook formats as opposed to course grades. Third, to address the criticism that past research has not been representative of the general college student population and thus lacks external validity (Griggs & Jackson, 2017b), we collected data from a large Highest Research Activity (Carnegie Classification of Institutions of Higher Education, 2017) public university.

We report data collected as part of a long-standing curriculum assessment program before and after a switch to an OER text in General Psychology. Our standard assessment involves measures of content knowledge and critical thinking—key learning outcomes for both the course itself (Gurung et al., 2016; Jhangiani & Hardin, 2015) and the university's General Education curriculum—in all sections of the course twice each semester. This created an opportunity to assess the impact of the switch to an OER text on student learning in ways that address many of the criticisms of prior work. In addition, we examined whether the cost of the assigned text influenced enrollment decisions, particularly among historically underserved students.

Method

Participants

Participants included 3,678 students enrolled in General Psychology during the Fall 2016 (n = 1,896) and Fall 2017 (n = 1,782) semesters at a large public university in the southeastern United States. However, as noted below, most of our analyses relied on smaller subsets of students. All students enrolled in Fall 2016 used the Lilienfeld, Lynn, Namy, and Woolf (2013) traditional textbook and all students enrolled in Fall 2017 used the OpenStax (2017) OER textbook. In both years, the majority of non-honors sections were taught by graduate teaching assistants (GTAs) in a traditional 150-minutes per week face-to-face format. In Fall 2016, this included two late afternoon or evening sections with 54 and 59 students, and nine other sections with 148–150 students each. In Fall 2017, there were six GTA-taught sections, with 146–166 students per section. There were also two to three smaller (25–30 students) honors sections each semester, taught by either experienced GTAs or faculty. Finally, in both semesters there was a larger (n ∼ 380) faculty-taught section that met twice a week for 50 minutes each, with a smaller (n ∼ 30), once a week graduate-student-led discussion section. In Fall 2017, there were also two larger (n = 521) faculty-taught sections that utilized a hybrid format with several online assignments replacing one of the three 50-minute class sessions, along with 14 GTAs who facilitated smaller within-class cohorts within the larger face-to-face class sessions. As described below, because of these differences in format, most of our analyses focus only on the GTA-taught sections.

Table 1 shows the numbers of students taught by novice (those in their first semester as the sole instructor of record) and experienced (those who had been sole instructor of record in at least one prior semester) graduate student instructors each semester. Of the Fall 2016 students, 1,418 (94.3% of the 1,504 students enrolled in sections taught by a graduate student instructor) completed at least some of the measures at the beginning of the semester (T1) and 1,141 (75.9%) completed measures at the end of the semester (T2). In Fall 2017, 878 (87.6% of the 1,002 students enrolled in sections taught by a graduate student instructor) completed at least some of the measures at T1 and 697 (69.6%) completed the measures at T2. Most participants were in their first year of college, White / European American, female, with at least one parent who had completed a four-year degree (see Table 1). Consistent with the course serving as one of the most popular social science general education courses at the university, students reported 128 different majors, with only 6.6% being psychology majors.

Table 1.

Number of Students by Each Instructor Type, Semester, Course Type, and Demographics.

Sample characteristics	Instructor type
	Fall 2016 (traditional textbook)					Fall 2017 (OER textbook)
	GTA					GTA
	Novice (n = 7)	Exp'd (n = 6)	All (n = 13)	Faculty (n = 2)	All	Novice (n = 3)	Exp'd (n = 5)	All (n = 8)	Faculty (n = 3)	All
Course type
110	769	557	1326	301	1627	439	417	856	691	1547
117	0	46	46	23	69	0	56	56	25	81
Gender
Female	430	379	809	203	1012	226	275	501	360	861
Male	303	198	501	101	602	132	131	263	257	520
Other	2	0	2	0	2	1	0	1		1
Race / ethnicity
European Am.	573	467	1040	248	1288	295	310	605	448	1053
AfA / Black	51	40	91	26	117	21	28	49	57	106
Latino(a)	27	12	39	7	46	12	11	23	19	42
Asian Am. / PI	27	26	53	15	68	16	28	44	33	77
Other	2	4	6	1	7	1	1	2	4	6
Native Am.	2	1	3	0	3	0	0	0	6	6
Multiracial	35	19	54	5	59	8	21	29	27	56
Parental college education
None	83	57	140	28	168	31	35	66	77	143
Some	81	80	161	39	200	53	50	103	67	170
2-year degree	51	51	102	20	122	30	28	58	39	97
4-year degree +	516	384	900	216	1116	245	294	539	425	964
Unsure	8	5	13	1	14	2	1	3	4	7
Year
First	580	428	1008	230	1238	271	295	566	462	1028
Sophomore	101	100	201	45	246	57	70	127	95	222
Junior	31	35	66	22	88	23	32	55	42	97
Senior	28	15	43	9	52	11	11	22	11	33
Other	5	2	7	1	8	2	1	3	4	7

Notes: GTA = graduate teaching assistant; Novice = instructor in first semester as sole instructor of record; Exp'd = experienced instructor with at least one prior semester as sole instructor of record; European Am. = European American / White; AfA / Black = African American / Black; Asian Am. / PI = Asian American / Pacific Islander; Native Am. = Native American / American Indian.

Table 2.

Content Knowledge and Critical Thinking Scores as a Function of Time, Semester, Course Type, and Instructor Experience for GTA-taught Students.

Course characteristics	Student learning outcome measure
	Content knowledge				Critical thinking
	Beginning of semester		End of semester		Beginning of semester		End of semester
	N	M (SD)	N	M (SD)	N	M (SD)	N	M (SD)
Fall 2016 (trad'l book)
Course type
Honors	46	0.57 (0.13)	44	0.64 (0.19)	14	0.57 (0.17)	42	0.59 (0.19)
Non-honors	1281	0.44 (0.18)	1078	0.48 (0.19)	426	0.43 (0.18)	1073	0.43 (0.18)
All	1327	0.44 (0.18)	1122	0.48 (0.20)	440	0.44 (0.18)	1115	0.44 (0.18)
Instructor experience
Novice	746	0.45 (0.18)	618	0.48 (0.20)	260	0.44 (0.18)	613	0.43 (0.18)
Experienced	581	0.44 (0.18)	504	0.49 (0.19)	180	0.43 (0.17)	502	0.44 (0.19)
All GTA	1327	0.44 (0.18)	1122	0.48 (0.20)	440	0.44 (0.18)	1115	0.44 (0.18)
Fall 2017 (OER book)
Course type
Honors	20	0.58 (0.17)	52	0.76 (0.17)	16	0.59 (0.14)	27	0.66 (0.15)
Non-honors	227	0.42 (0.18)	630	0.54 (0.21)	252	0.45 (0.18)	305	0.44 (0.18)
All	247	0.44 (0.18)	682	0.56 (0.21)	268	0.46 (0.18)	332	0.45 (0.18)
Instructor experience
Novice	114	0.41 (0.18)	340	0.53 (0.21)	128	0.44 (0.18)	165	0.42 (0.18)
Experienced	133	0.45 (0.18)	342	0.58 (0.21)	140	0.47 (0.17)	167	0.49 (0.18)
All GTA	247	0.44 (0.18)	682	0.56 (0.21)	268	0.46 (0.18)	332	0.45 (0.18)

Note: GTA = graduate teaching assistant.

Table 3.

Correlation Matrix.

	M (SD)	1	2	3	4	5	6	7	8
1. Content knowledge T1	4.41 (1.78)	1
2. Content knowledge T2	5.11 (2.05)	.37**	1
3. TBC	1.39 (.55)	.01	−.86**	1
4. Critical thinking T1	4.89 (1.94)	.33**	.33**	−.03	1
5. Critical thinking T2	4.83 (2.02)	.32**	.39**	−.02	.43**	1
6. Book enroll	1.87 (2.74)	.11	−.02	.01	−.12	−.09	1
7. Book stay	2.43 (3.08)	.07	−.03	.03	−.16	−.04	.76**	1
8. Book satisfaction	6.45 (2.58)	.08	.14**	−.16**	−.02	.20**	.19**	.22**	1

**p < .01; *p < .05

Notes: TBC = teacher behaviors checklist (Keeley, Smith, & Buskist, 2006); CT = critical thinking; book enroll = book role in decision to enroll in course; book stay = book role in decision to stay in course.

In Fall 2016, the course used an “Inclusive Access” model in which all students were automatically charged for and given access to the required traditional textbook in an electronic book (eBook) format, unless they opted out; 13.1% of students did so. Of course, it is unknown if any of these students obtained access to the book in some other way. In Fall 2017, 9.8% of students who completed the end-of-semester assessment reported never accessing the OER text, despite having free access. The most common reason had to do with feeling the textbook was not needed to do well in the class. Given that we are less interested in the question of how “those who had access” performed than in “when OER is assigned, how do students perform?”, we include all students in our analyses, regardless of whether they reported using the textbook.

Procedure

To encourage completion of the program assessment measures, students received nominal course credit (≤1% of the overall grade) for each of the two assessments; data are used for research purposes following IRB-approved procedures. In each semester, Time 1 (T1) occurred just after the semester add / drop deadline, typically ∼2 weeks after the start of the semester, and Time 2 (T2) occurred at the start of the final week of classes. At each time point, students received a personalized link to the online program evaluation assessments and had one week to complete them in order to earn credit toward their course grade. All students first completed questions about their reasons for enrolling in the course (T1) and suggestions for improving the course (T2) before the content and critical thinking measures described below (in random order), followed by an opportunity to provide feedback on their instructor (T2 only). Measures were administered in slightly different ways to reduce burden on students. In Fall 2016, at both T1 and T2, all students received the content and critical thinking items. In Fall 2017, all students completed the critical thinking items at T1, with half randomly assigned to complete them again at T2; for the content items, half of students were randomly assigned to complete the items at T1 and all students completed them at T2.

Measures

Content knowledge

A 24-item multiple choice test developed by the first author for program evaluation purposes has been used to assess learning outcomes over the previous 10 semesters. These items were developed without reference to any specific textbook and cover the breadth of content instructors were required to cover in the course, including research methods, neuroscience, sensation and perception, learning, memory, child development, social psychology, cognitive biases, disorders, and stress. The majority of the items required students to apply knowledge learned from the course. For example, Pat and Chris broke up, and Pat has been feeling sad ever since. Which of the following would be important in determining whether Pat is experiencing normal sadness after a break-up versus something more serious, like clinical depression? Another example: Which of these is the better example of confirmation bias? A copy of the items is available on request from the first author.

The same 24 items were used in both semesters and at both time points. As part of our efforts to reduce the time required for students to complete the assessment battery, at the end of the semester in Fall 2017, the online survey tool used to deliver the assessment battery randomly selected 10 of the 24 items for each student. Students completed all 24 items at the other time points. To facilitate comparisons across semesters, we randomly selected 10 items for each participant for Fall 2016 T1, Fall 2016 T2, and Fall 2017 T1 and used all 10 items for Fall 2017 T2. Content knowledge scores were calculated based on only these 10 items. Thus, for all time points, we summed the number of correctly answered questions and divided by 10 to yield a percent correct total score that may range from 0 to 1.

Critical thinking

Critical thinking was measured using 11 multiple-choice items that were originally developed by the first author while at another institution to assess learning outcomes in social sciences general education courses. Originally, 20 items were written to assess students' abilities to Identify and critique alternative explanations for claims about social issues and human behavior (10 items) and to Demonstrate knowledge of the appropriate methods, technologies, and data that social and behavioral scientists use to investigate the human condition (10 items). The items were initially developed based on Lawson's (1999) list of questions that reflect basic critical thinking skills in psychology, and then broadened to be relevant to all social and behavioral sciences disciplines. The items were intentionally designed to capture common misunderstandings to help inform curriculum delivery. An example item stem is: You read that researchers studying the effects of divorce on children found that children raised in single-parent homes are more likely to get into trouble at school than children raised by two parents. The researchers conclude that children from single-parent families should be considered at-risk for problems in school. What, if anything, is wrong with this conclusion?

Unpublished data collected by the first author in 2011 showed that students enrolled in senior-level social science capstone courses outside of psychology (n = 31) scored significantly higher than first-year students (n = 40) enrolled in General Psychology. Moreover, 12 of the 20 items distinguished scores on a commercially available and well-validated measure of critical thinking skills; for each of the 12 items, students who correctly answered the item scored significantly higher on the commercially available test than students who did not correctly answer the item. These unpublished data were used to select the 11 critical thinking items used in the program evaluation assessments and reported here. The number of correct answers is summed and divided by 11 to yield a percent correct score that may range from 0 to 1. A copy of the items is available on request from the first author.

Instructor characteristics

Students completed the teacher behaviors checklist (TBC; Keeley, Smith, & Buskist, 2006) for their General Psychology instructor as part of the end-of-semester assessment. Students rated their instructor on 28 characteristics (e.g., sensitive and persistent) with behavioral indicators (e.g., “Makes sure students understand material before moving to new material, holds extra study sessions, repeats information when necessary, and asks questions to check student understanding,” p. 85) using a 5-point scale from 1 (My instructor never exhibits / has exhibited these behaviors reflective of this quality) to 5 (My instructor always exhibits / has exhibited these behaviors reflective of this quality). Responses were averaged to yield a total score (α = .97 in the current sample) that was used as an indicator of instructor characteristics in our analyses.

OER book impact

In Fall 2017 only, participants were asked how they accessed the textbook (hard copy, online platform, downloaded PDF, or not at all). Students were also asked to rate their level of satisfaction with the book (0 = extremely dissatisfied, 10 = extremely satisfied). In spring and summer 2017, the psychology department distributed fliers to advisors across the university to advertise the General Psychology course, highlighting the fact that we were moving to an OER text for Fall 2017. The end-of-course assessment in Fall 2017 also asked students about the extent to which the cost of the textbook influenced their decision to enroll or remain enrolled in the course (0 = none at all, 5 = a moderate amount, 10 = a great deal).

Demographics

Students were asked to report their major, race / ethnicity, gender, and parent education level. A student was operationalized as a first-generation college student (FGCS) if neither primary caregiver obtained any formal education after high school. Racial minorities were operationalized as students who did not identify as White.

Results

Book Impact

Students (n = 1180) in Fall 2017 reported a moderate level of satisfaction with the OER textbook, M = 6.83, SD = 2.53. The cost of the book had little impact on students' decision to enroll in the course, N = 864, M = 1.95, SD = 2.82, and a moderately low, but significantly higher, impact on their decision to stay enrolled in the course, n = 843, M = 2.56, SD = 3.21, t (797) = −6.34, p < .001, d = 0.22. However, first-generation college students (FGCS; n = 68) reported that the cost of the book had a significantly higher impact on their decision to enroll in the course (M = 2.71, SD = 3.44) compared to students who were not FGCS, n = 659, M = 1.86, SD = 2.74, t (81.13) = −2.01, p < .05, d = 0.27, 95% CI [0.01, 1.68].¹ The decision to remain enrolled was not significantly different for FGCS (M = 3.03, SD = 3.57) and non-FGCS, M = 2.46, SD = 3.16, t (710) = −1.40, p > .21. Additionally, students who did not identify as White reported that the cost of the book had a significantly higher impact on decisions to enroll in the course (n = 166, M = 2.45, SD = 3.02) and remain enrolled (n = 159, M = 2.97, SD = 3.41), compared to students who identified as White, n = 584, M = 1.75, SD = 2.71, t (245.58) = 2.69, p < .01, d = 0.24, 95% CI [0.19, 1.21]; and n = 573, M = 2.34, SD = 3.11, t (235.75) = 2.11, p < .05, d = 0.19, 95% CI [0.04, 1.22], respectively. It should be noted that students who dropped the course did not complete these items and the means for all groups are relatively low, so the results should be interpreted with caution.

Learning Outcomes

Analyses of learning outcomes are limited to those students enrolled in a course section taught by a GTA to avoid confounding effects of book with effects of course format. Preliminary analyses indicated there were no univariate outliers in either content knowledge or critical thinking, but there were 15 univariate outliers in the TBC total scores (defined as scores more than 4 SD from the mean and confirmed through inspection of histograms). Although the overall pattern of results is the same whether the outliers are included or excluded, we report the more conservative results from analyses involving the TBC that excluded these 15 outliers. Chi-square analyses indicated no differences in the composition of the Fall 2016 and Fall 2017 samples by race: X² (7) = 5.90, p > .55, gender: X² (2) = 3.04, p > .22, or class year: X² (4) = 6.33, p > .18.

Supporting the validity of our outcome measures, students enrolled in honors sections scored significantly higher than students enrolled in non-honors sections on both content knowledge, t (106.93) = −10.12, p < .001, d = 1.04, 95% CI [1.64, 2.44], and critical thinking, t (101.76) = −8.67, p < .01, d = 0.91, 95% CI [0.13, 0.20] at the end of the semester. Given that honors students scored higher on our outcome measures and were taught exclusively by experienced instructors, and in light of their small numbers, we exclude honors students from our analyses below to ensure any effects are not being inadvertently driven by honors students. TBC total scores were correlated with content scores, r = .09, p < .001, indicating students who rated their instructors as more frequently exhibiting the characteristics included on the TBC scored higher on the content knowledge measure. Surprisingly, novice instructors scored significantly higher on TBC total scores, M = 4.65, SD = .50, than experienced instructors, M = 4.59, SD = .52, t (1299.53) = −2.21, p < .05, d = 0.11, 95% CI [-0.11, −0.01]. This difference was driven by novice instructors scoring higher than experienced instructors on the Caring subscale, Ms = 4.62 and 4.53, respectively, t (1237.77) = −3.21, p = .001, d = 0.17, 95% CI [−0.16, −0.04].

Differences in average scores

To determine if student learning outcomes varied as a function of instructor experience and text book type, end-of-semester critical thinking and content scores were analyzed using separate 2 (book type: traditional, OER) × 2 (instructor experience: novice, experienced) univariate ANCOVAs, with TBC scores as the covariate.² No student had scores on both of these outcomes in Fall 2017, so we could not conduct an omnibus MANCOVA. Descriptive statistics are presented in Table 2; correlations are presented in Table 3.

In addition to a very small but significant effect of TBC scores, F (1, 1494) = 5.87, p < .02, $η_{p}^{2} = . 004$ , there was a small but significant main effect of book type on content knowledge, F (1, 1494) = 36.96, p < .001, $η_{p}^{2} = . 02$ : Students scored significantly higher when the OER text was used (n = 630, EMM = 0.55, SE = 0.10) than when the traditional text was used (n = 1078, EMM = 0.48, SE = 0.01), 95% CI [0.47, 0.92]. Neither the main effect of instructor experience F (1, 1494) = 1.82, p > .17, $η_{p}^{2} = . 001$ , nor the interaction of book type by instructor experience was significant, F (1, 1494) = 1.44, p > .23, $η_{p}^{2} = . 001$ . There were no significant main or interaction effects on the critical thinking measure, Fs < 0.22, ps > .64, $η_{p}^{2} s < . 001$ . See Figure 1.

Figure 1.

Estimated marginal means of end-of-semester content and critical thinking scores as a function of instructor experience level and textbook.

Differences in distribution of scores

We examined the distribution of end-of-semester scores to determine if differences in means were apparent across the entire sample, or for just the lowest-performing students. We coded participants' end-of-semester scores into quartiles based on possible scores (i.e., scores of 25% or lower were coded in the bottom quartile, scores of 75% or higher were coded in the top quartile). There were no differences in quartile distribution of scores on the critical thinking measure, X² (3) = 0.18, p > .98. However, there were significant differences in the distribution of content knowledge scores as a function of book type, X² (3) = 37.75, p < .001. When the traditional text was used, 13.1% of students scored in the bottom quartile, 52.1% scored in the second-lowest quartile, 26.7% scored in the second-highest quartile, and only 8.1% scored in the top quartile. However, when the OER text was used, these percentages were 9.2%, 43.0%, 31.9%, and 15.9%, respectively. In other words, fewer students scored less than 50% and more students scored above 50% when the OER text was used, with nearly twice as many students scoring 75% or above. Thus, the mean difference in end-of-semester scores is driven both by decreases in low scores and increases in high scores.

Repeated measures analyses

To rule out the possibility that students may have differed on either content knowledge or critical thinking at the start of the semester, as well as to explore changes in time across the semester, we also conducted 2 (time: beginning of semester, end of semester) × 2 (book type: traditional, OER) × 2 (instructor experience: novice, experienced) repeated measures ANCOVAs, with time as a within-subjects factor, and TBC total scores as the covariate.³ Only students who provided matched data at both time points are included.

There were no significant main effects or interactions for the critical thinking items (all ps > .44. $η_{p}^{2}$ ≤ .001). For content knowledge, neither the main effect of instructor experience F (1, 1166) = 0.38, p > .53, $η_{p}^{2}$ < .001, nor the interaction of instructor experience and book type was significant, F (1, 1166) = 1.47, p > .22, $η_{p}^{2}$ < .001. However, there was a very small but significant main effect of book type, F (1, 1166) = 4.58, p < .05, $η_{p}^{2} = . 004$ : scores were higher for students during the semester in which the OER book was used (n = 150, EMM = 0.49, SE = .013) than in the semester in which the traditional book was used (n = 1021, EMM = 0.46, SE = .013), 95% CI [0.003, 0.058].

Most importantly, there was a small but significant time by book type interaction, F (1, 1166) = 14.92, p < .001, $η_{p}^{2} = . 01$ . Post-hoc tests showed that students appeared to learn more over the course of the semester in Fall 2017, when the OER book was used, than in Fall 2016, when the traditional text was used. Specifically, although there were no differences at the beginning of the semester in content knowledge scores between the traditional (n = 1046, EMM = 0.44, SE = 0.006) and OER (n = 180, M = 0.43, SE = 0.015), 95% CI [-0.024, 0.039] texts, p > .63, students scored significantly higher at the end of the semester when the OER book was used (EMM = 0.55, SE = .017) than the semester in which the traditional text was used, EMM = 0.481, SE = 0.006), 95% CI [0.03, .10], p < .001.

Discussion

We examined the effects of an OER versus traditional textbook among students enrolled in multiple sections of General Psychology. The present study controlled for instructor characteristics and relied on objective indicators of content knowledge acquisition and critical thinking that were developed without reference to any specific textbook and were identical across all course sections. We found no evidence that use of the OER text impeded learning on any outcome, and some evidence that use of the OER text was in fact associated with a small increase in content knowledge acquisition. We also found that under-represented students (i.e., first-generation college students and racial / ethnic minority students) were more likely than non-FGCS and European American students, respectively, to report that the use of the OER text had affected their decision to enroll in the course and, for minority students, to remain enrolled in the course.

Consistent with past research finding no evidence that OER impede learning (e.g., Hilton, 2016), there were no differences in critical thinking as a function of textbook. However, students learned slightly more (in terms of basic content knowledge) during the semester in which the OER text was used than during the semester in which the traditional text was used, regardless of their ratings of instructors' competence and supportiveness or of the instructors' experience level itself. Moreover, this effect was not simply due to decreasing the proportion of students who scored especially low on the content knowledge measure, but also to increasing the proportion of students who scored especially high. In other words, use of the OER text did not simply decrease the number of students who did not learn very much; it increased the number of students who did learn. The fact that these findings are based on the same criterion measure for all students—a measure that was designed to assess student learning regardless of instructor, textbook, grading policies, or teaching methods—is particularly compelling.

Moreover, our data were collected from students taught by graduate student instructors, all of whom were new to teaching from an OER text. To our knowledge, ours is the first study to explore the use of OER texts by graduate student instructors, as well as the first to include instructor experience as a factor. We did not find any evidence that novice instructors have difficulty effectively using OER texts. However, it is important to note that all of the graduate student instructors had participated in a robust instructor training program. All but one GTA (an experienced transfer student) in both semesters had completed a 1-credit didactic course on college teaching, taught by the first author, followed by a 2-credit “bootcamp” course focused specifically on preparing students to teach General Psychology. The 1-credit course is typically completed by all graduate students in their first semester, and focuses on course design (e.g., articulating learning objectives from a backward design perspective), ethics, diversity and inclusion, assessment of student learning, technology in teaching, course delivery (lecture, discussion, activities), and professional development. Students who were assigned as the instructor of record for a course completed the 2-credit “bootcamp” course in the mini-term, typically just days or weeks before co-teaching the course to a small summer section before solo teaching a large fall section. Students in the mini-term course delivered two practice presentations, including a complete 50-minute practice class, and discussed teaching ideas for all required content. In both 2016 and 2017, the first author shared her own lecture slides, exams, and other teaching materials. The primary difference between the May 2016 and May 2017 course was that in May 2016, discussion focused on how students would choose which content to focus on from the comprehensive traditional text, whereas in May 2017, discussion focused on how students would supplement content from the OER text. Otherwise, both courses were similar in format and content for new instructors in both semesters. Our results indicate that under these circumstances, use of an OER text was associated with comparable or better outcomes for students.

Limitations and Future Directions

Our results, which address many of the criticisms of past work, add to the growing evidence that the use of OER textbooks in psychology increases student access to course materials and, as a result, increases student success—at least when used by trained and / or well-supported instructors. The extensive training the graduate student instructors in this study received is in some ways both a strength and a limitation of this work. It is a strength insofar as our sample is consistent with recent calls for robust training for instructors of General Psychology (Gurung et al., 2016; Jhangiani & Hardin, 2015); it is a limitation in that this training, although increasingly common, is still not representative of the experience of many new instructors. Future research should examine the extent to which OER textbooks are associated with comparable, if not improved, outcomes in other contexts where instructors receive less training and support.

Our results also relied on comparisons between a traditional and OER text that were both primarily accessed online. This again is both a strength and limitation of our work. It is a strength insofar as we largely controlled for format, rather than confounding medium (print versus digital) with textbook type. This is important, given mixed evidence about the effects of medium on learning outcomes (see Singer & Alexander, 2017). It is a limitation, however, to the extent that comparing a traditional print textbook to a digital OER textbook better reflects the choices students and instructors are typically faced with. Future research could provide an even stronger test by using a 2 (medium: print versus digital) x 2 (type: traditional versus OER) design.

Given the contentious presidential election that occurred in the United States during the Fall 2016 semester when the traditional book was used, it is possible that history effects could have affected our results. Such effects were unavoidable, given that data were collected from two different cohorts of students one year apart. However, historical context / cohort effects are difficult to avoid in such research, and there is at least anecdotal evidence that students' academic performance elsewhere was not affected by the election (Ballard & Wang, 2016). Still, future research that collected data from students using a traditional or OER in the same semester would provide stronger evidence.

Although textbook cost had minimal effect on students' decisions to enroll in or remain enrolled in the course, it is still noteworthy that the use of the OER text was rated as significantly more important to underserved students. Many institutions are working to improve retention of underserved students, and our results suggest that adoption of OER textbooks may be one tool for addressing inequalities in retention rates. Future research should also explore the extent to which use of an OER text might affect course enrollment over time. One factor influencing our decision to adopt an OER text was that the General Psychology course had faced declining enrollments for several years. Although the lower enrollment in Fall 2017 compared to Fall 2016 was consistent with this multi-year trend, we are unable to rule out the possibility of differential attrition from the course because we were unable to obtain institutional data on drop rates in the course each semester. However, past research has found that OER are associated with fewer course withdrawals (Clinton, 2018), so it seems unlikely that our results can be explained by more low-performing students withdrawing from the course in Fall 2017 than in 2016. Still, future research should directly examine whether enrollment and retention patterns shift as textbook choices shift.

Future research should also continue to use objective measures of learning outcomes that are not tied to specific instructors or derived from specific textbooks. Jhangiani and Hardin (2015) identified applying content-based knowledge and using scientific reasoning and critical thinking—both of which are clearly tied to the learning goals and outcomes for psychology majors (American Psychological Association, 2013)—as the two essential skills that all General Psychology courses should foster in students. We included measures of each of these skills; however, it is important for future research to include other psychometrically sound measures that explicitly assess students' ability to apply content knowledge and think scientifically.

Finally, the effect of textbook on students' content knowledge was small. Students who used the OER text correctly answered, on average, fewer than one more item out of 10 than students who used the traditional text, and still only answered 54.1% of the items correctly, on average (versus 47.6% for the traditional text). Given that our courses do not use required comprehensive final exams, these low scores are consistent with past research that has found low retention of course material (58%) among introductory psychology students (Landrum & Gurung, 2013). However, to the extent that these learning gains are representative of students' learning and performance more generally in the course, the small gains may still be quite meaningful, as they represent more than half a letter grade difference (i.e., 6.5% increase in final score).

Conclusions

As institutions grapple with how to increase the persistence and success of all students, but especially those underserved, questions of cost continue to be important. The accessibility of higher education is a social justice issue. Our results suggest that traditional textbooks and their associated costs do not necessarily ensure greater academic success in General Psychology courses relative to low-cost OER, at least in the hands of well-trained instructors. Institutions committed to increasing access and decreasing costs should consider devoting resources to instructor training and support in the adoption of OER textbooks, which allow a greater number of individuals access to course material without negatively affecting those who could otherwise afford a traditional textbook.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Author biographies

Erin E. Hardin is professor and director of Undergraduate Studies in the psychology department at the University of Tennessee in Knoxville, TN. Her favorite courses to teach are Introductory Psychology and College Teaching seminars to train graduate student instructors, both of which she has taught for two decades at three different institutions. Her research has focused on cultural differences in the self and understanding individuals’ career development in their unique cultural contexts. Most recently, her work has examined the recruitment and retention of under-represented individuals in STEM. She has also been immersed in thinking about the purpose, value, and implementation of General Education curricula.

Bret Eschman is a doctoral student in the Experimental Psychology (Developmental track) PhD program at the University of Tennessee, Knoxville. During his time as a graduate student, he has taught Research Methods, Introductory Psychology, Honors General Psychology, and Child Psychology. He is a member of the Infant Visual Cognition Lab and his research interests include using psychophysiological measures to explore individual differences in working memory and attention in both infants and adults.

Elliot S. Spengler is a doctoral student in the Counseling Psychology PhD program at the University of Tennessee, Knoxville. During his time in the program, he has been a teaching assistant for Theories of Personality and Abnormal Psychology as well as a discussion section leader for Introductory Psychology. His research and clinical interests incorporate the intersection of science, clinical practice, and social justice advocacy through topics such as clinical judgment, structural stigma, exploring and intervening within the minority stress model for LGBT+ individuals, infusing cultural competency into training and the community, and mindfulness.

J. Alex Grizzell is a doctoral student in the Experimental Psychology (Biological track) PhD program at the University of Tennessee, Knoxville. He has served as an instructor of record for Learning and Behavior, Psychology of Motivation, Behavioral Neuroscience, and Advanced Topics of Neuroscience. His research interests include a focus on the neurobiological substrates of stress resilience with a particular focus on the role of inflammatory processes and the role of neural circuitry involving the prefrontal cortex.

Anahvia Taiyib Moody is a doctoral student in the Counseling Psychology PhD program at the University of Tennessee, Knoxville. During her time in the program, she has been a graduate teaching assistant for Introductory Psychology and African American Psychology. She also enjoys teaching an exploratory psychology course with the KidsU program at UT, providing a fun and interactive one-week immersion experience to introduce high school students to the field. Her research and clinical interests focus on Black women’s experiences with trauma, as well as the incorporation of more holistic and affirming models of trauma into research and therapy.

Shannon Ross-Sheehy is an assistant professor of psychology, and director of the Infant Visual Cognition Lab at the University of Tennessee Knoxville. Her undergraduate courses include Introductory Psychology, Child Development, Cognition and the Brain, and Cognitive Development. Graduate courses include Infant Sensation, Perception and Cognition, as well as Lifespan Development. Her research explores individual differences in the co-development of visual attention, perception, and working memory in both typically and atypically developing infants.

Kevin M. Fry is a doctoral student in the Counseling Psychology PhD program at the University of Tennessee, Knoxville. During his time in the program, he has taught General Psychology and Honors General Psychology. He has also been a graduate teaching assistant for Multicultural Psychology, Group Facilitation in Intergroup Dialogue, and Introduction to Psychology Research Methods. His research interests include lay beliefs about sexual orientation and their impacts on attitudes and behaviors towards sexual minorities. His clinical interests include working with LGBTQ+ clients in therapy, as well as clients who hold any marginalized identity.

References

Allen, I. E., & Seaman, J. (2014). Opening the curriculum: Open educational resources in US Higher Education, 2014. Babson Survey Research Group. Retrieved from https://www.onlinelearningsurvey.com/reports/openingthecurriculum2014.pdf.

American Psychological Association. (2013). APA guidelines for the undergraduate psychology major: Version 2.0. Retrieved from http://www.apa.org/ed/precollege/undergrad/index.aspx.

Ballard, A., & Wang, K. (2016, November 17). Academic performance unaffected by election, students say. Yale Daily News. Retrieved from https://yaledailynews.com/blog/2016/11/17/academic-performance-unaffected-by-election-students-say/.

Bissell

A. N.

(2009) Permission granted: Open licensing for educational resources. Open Learning 24: 97–106. doi: https://doi.org/10.1080/02680510802627886.

Butcher N. (2015). A basic guide to open educational resources (OER). Paris: Commonwealth of Learning (COL). Retrieved from http://unesdoc.unesco.org/images/0021/002158/215804e.pdf.

Carnegie Classification of Institutions of Higher Education. (2017). Institution lookup. Retrieved from http://carnegieclassifications.iu.edu.

Clinton, V. (2018). Savings without sacrifice: A case report on open-source textbook adoption, Open Learning: The Journal of Open, Distance and e-Learning, DOI: 10.1080/02680513.2018.1486184.

Clump

M. A.

Bauer

Breadley

(2004) The extent to which psychology students read textbooks: A multiple class analysis of reading across the psychology curriculum. Journal of Instructional Psychology 31: 227–232.

Darwin

(2011) How much do study habits, skills, and attitudes affect student performance in introductory college accounting courses?. New Horizons in Education 59(3): 1–15.

10.

Delacre

Lakens

Leys

(2017) Why psychologists should by default use Welch's t-test instead of Student's t-test. International Review of Social Psychology 30: 92–101. doi: https://doi.org/10.5334/irsp.82.

11.

Fischer

Ernst

Mason

S. L.

(2017) Rating the quality of open textbooks: How reviewer and text characteristics predict ratings. The International Review of Research in Open and Distributed Learning 18: 142–154. doi: https://doi.org/10.19173/irrodl.v18i4.2985.

12.

Goldrick-Rab

Kelchen

Harris

D. N.

Benson

(2016) Reducing income inequality in educational attainment: Experimental evidence on the impact of financial aid on college completion. American Journal of Sociology 121: 1762–1817. doi: https://doi.org/10.1086/685442.

13.

Griggs

R. A.

Jackson

S. L.

(2017a) Open introductory psychology textbooks: Prose and qualms. Teaching of Psychology 44: 193–202. doi: https://doi.org/10.1177%2F0098628317711283.

14.

Griggs

R. A.

Jackson

S. L.

(2017b) Studying open versus traditional textbook effects on students' course performance: Confounds abound. Teaching of Psychology 44: 306–312. doi: https://doi.org/10.1177%2F0098628317727641.

15.

Gurung

R. A. R.

Hackathorn

Enns

Frantz

Cacioppo

J. T.

Loop

Freeman

J. E.

(2016) Strengthening introductory psychology: A new model for teaching the introductory course. American Psychologist 71: 112–124. doi: https://doi.org/10.1037/a0040012.

16.

Hilton

(2016) Open educational resources and college textbook choices: A review of research on efficacy and perceptions. Educational Technology Research and Development 64: 573–590. doi: https://doi.org/10.1007/s11423-016-9434-9.

17.

Hout

(2012) Social and economic returns to college education in the United States. Annual Review of Sociology 38: 379–400. doi: https://doi.org/10.1146/annurev.soc.012809.102503.

18.

Jhangiani

Hardin

E. E.

(2015) Skill development in Introductory Psychology. Scholarship of Teaching and Learning in Psychology 1: 362–376. doi: https://doi.org/10.1037/stl0000049.

19.

Jhangiani

R. S.

Jhangiani

(2017) Investigating the perceptions, use, and impact of open textbooks: A survey of post-secondary students in British Columbia. The International Review of Research in Open and Distributed Learning 18: 172–192. DOI: http://dx.doi.org/10.19173/irrodl.v18i4.3012.

20.

Jhangiani, R. S., Pitt, R., Hendricks, C., Key, J., & Lalonde, C. (2016). Exploring faculty use of open educational resources at British Columbia post-secondary institutions. BCcampus Research Report. Victoria, BC: BCcampus. Retrieved from http://viuspace.viu.ca/handle/10613/2705.

21.

Keeley

Smith

Buskist

(2006) The Teacher Behaviors Checklist: Factor analysis of its utility for evaluating teaching. Teaching of Psychology 33: 84–91. doi: https://doi.org/10.1207%2Fs15328023top3302_1.

22.

Kinzie, S. (2006, January 23). Swelling textbook costs have college students saying ‘Pass.’ Washington Post. Retrieved from http://www.washingtonpost.com/wp-dyn/content/article/2006/01/22/AR2006012201290.html?noredirect=on.

23.

Landrum

R. E.

Gurung

(2013) The memorability of introductory psychology revisited. Teaching of Psychology 40: 222–227. https://doi.org/10.1177/0098628313487417 .

24.

Lawson

T. J.

(1999) Assessing psychological critical thinking as a learning outcome for psychology majors. Teaching of Psychology 26: 207–209.

25.

Lilienfeld

S. O.

Lynn

S. J.

Namy

L. L.

Woolf

N. J.

(2013) Psychology: From inquiry to understanding, Hoboken, NJ: Allyn & Bacon.

26.

Baum

Pender

Welch

(2017) Trends in college pricing, New York, NY: The College Board. Trends in Higher Education Series Retrieved from https://trends.collegeboard.org/sites/default/files/2017-trends-in-college-pricing_1.pdf.

27.

McFall

(2005) Electronic textbooks that transform how textbooks are used. The Electronic Library 23: 72–81. doi: https://doi.org/10.1108/02640470510582754.

28.

MERLOT. (2015). Programs and projects. Retrieved from http://info.merlot.org/merlothelp/index.htm#merlot_collection.htm.

29.

Miller

G. M.

Chapman

J. P.

(2001) Misunderstanding analysis of covariance. Journal of Abnormal Psychology 110: 40–48. doi: https://doi.org/10.1037//0021-843x.110.1.40.

30.

OpenStax College. (2017). Psychology. Houston, TX: OpenStax College. Retrieved from https://openstax.org/details/books/psychology.

31.

Pitt

(2015) Mainstreaming open textbooks: Educator perspectives on the impact of OpenStax College open textbooks. The International Review of Research in Open and Distributed Learning 16: 133–155. doi: https://doi.org/10.19173/irrodl.v16i4.2381.

32.

Podolefsky

Finkelstein

(2006) The perceived value of college physics textbooks: Students and instructors may not see eye to eye. Physics Teacher 44: 338–342. doi: https://doi.org/10.1119/1.2336132.

33.

Popken, B. (2015, August 6). College textbook prices have risen 1,041 percent since 1977. NBC News. Retrieved from https://www.nbcnews.com/feature/freshman-year/college-textbook-prices-have-risen-812-percent-1978-n399926.

34.

Redden M. (2011). 7 in 10 students have skipped buying a textbook because of its cost, survey finds. Chronicle of Higher Education. Retrieved from https://www.chronicle.com/article/7-in-10-Students-Have-Skipped/128785.

35.

Singer

L. M.

Alexander

P. A.

(2017) Reading on paper and digitally: What the past decades of empirical research reveal. Review of Educational Research 87: 1007–1041. doi: https://doi.org/10.3102/0034654317722961.

36.

Skinner

Howes

(2013) The required textbook—Friend or foe? Dealing with the dilemma. Journal of College Teaching & Learning (TLC) 10: 133–142. doi: https://doi.org/10.19030/tlc.v10i2.7753.

37.

Svickni

McKeachie

(2014) Teaching tips: Strategies, research and theory for college and university teachers, Belmont, CA: Wadsworth, Cengage.

38.

U.S. Bureau of Labor Statistics (2016). College tuition and fees increase 63 percent since January 2006. Retrieved from https://www.bls.gov/opub/ted/2016/college-tuition-and-fees-increase-63-percent-since-january-2006.htm.