Students' Use and Perceptions of the Relevance and Quality of Open Textbooks Compared to Traditional Textbooks in Online and Traditional Classroom Environments

Abstract

The bulk of previous research on students' perceptions of open educational resources have lacked a control group of students rating traditional textbooks. Moreover, few studies have examined differences in the perceptions of online students and those taking classes in the classroom. A 2 × 2 cross-sectional design was used in which 925 students, assigned either a traditional textbook or an open textbook in either an online or classroom environment, were recruited to complete an online survey. Students assigned open textbooks were almost twice as likely to report using their textbooks, they used them more frequently, and for more time per week overall. Students assigned open textbooks also perceived a greater degree of overlap between the textbook, lecture, and quiz material than did students assigned traditional textbooks. Finally, ratings of the open textbooks were significantly higher than ratings of the traditional textbooks overall and on 11 of 15 different dimensions. Few differences in the online and classroom students were detected, suggesting both groups experienced similar benefits of the open textbooks. These findings demonstrate that replacing traditional textbooks with open textbooks may help to offset some of the financial hardships students face while improving students' engagement and satisfaction with their assigned textbook.

Keywords

OER online students use relevance quality

Introduction

Escalating textbook prices have led most students to forgo purchasing one or more textbooks needed to support their learning (Florida Virtual Campus, 2016; Martin, Belikov, Hilton, Wiley, & Fischer, 2017). One potential solution to this problem is open educational resources (OER), which are openly licensed educational materials (e.g., open textbooks, videos, lectures) that are free to distribute, edit, and use (United Nations Educational, Scientific and Cultural Organization (UNESCO), 2017). However, only two studies have examined whether students are more likely to obtain assigned OER than they are to purchase assigned textbooks, and the results were mixed. Feldstein, Martin, Hudson, Hilton, and Wiley (2012) found that a higher percentage of Business students downloaded an open textbook (76%) than had previously purchased traditional textbooks (47%). In contrast, Lawrence and Lester (2018) found that students in an American Government course were no more likely to report obtaining an open textbook than students in a previous semester were to report purchasing a traditional textbook.

While only the two aforementioned studies have examined rates of obtaining the required textbook, additional studies have examined the frequency with which students' use OER. Results of these studies indicate that students in a variety of disciplines self-report using OER at a similar (Bliss, Hilton, Wiley, & Thanos, 2013; Bliss, Robinson, Hilton, & Wiley, 2013; California Open Educational Resources Council, 2016; Hendricks, Reinsberg, & Rieger, 2017; Illowsky, Hilton, Whiting, & Ackerman, 2016) or higher (Delimont, Turtle, Bennett, Adhikari, & Lindshield, 2016) frequency as they use traditional textbooks in a typical course. Two additional studies included a control group of students using and rating the frequency with which they use a traditional textbook. The results of one indicated no significant differences in the self-reported use of open versus traditional textbooks for a variety of purposes in Introductory Psychology classes (Clinton, 2018). Similarly, the other found no significant differences in the amount of time that students in Introductory Psychology courses assigned open versus traditional textbooks spent reading their textbook or in their proportion of completed weekly readings; however, students assigned traditional textbooks spent more time studying than did those assigned open textbooks (Jhangiani, Dastur, Le Grand, & Penner, 2018).

There is a paucity of research on students' perceptions of the relevance of OER or the need for OER to support learning in the course, perhaps because many researchers have been focused on other important aspects of OER (e.g., impacts on grades, perceived quality, etc.). Nevertheless, it is important to understand these perceptions, as students may be more likely to purchase and use textbooks that they perceive to be more necessary for the successful completion of their coursework. Hilton, Gaudet, Clark, Robinson, and Wiley (2013) found that the majority of students using OER in a Math course reported that it adequately supported their work inside and outside of class. Similarly, Gil, Candelas, Jara, Garcia, and Torres (2013) reported that most students in a Computer Network course provided neutral or positive ratings to questions pertaining to whether the OER materials were essential to learning. Finally, Jhangiani and Jhangiani (2017) reported that students using open textbooks in a variety of disciplines estimated that 50–75% of the exam questions were drawn from their textbook and they perceived a high degree of overlap between the textbook and lecture material.

According to a recent survey of a large nationally representative sample of faculty from diverse disciplines, one of the top five barriers to the adoption of OER are concerns about the quality of the materials, with 28% of faculty endorsing statements that OER are not high quality (Seaman & Seaman, 2017). Nevertheless, research on faculty who have adopted OER and on students using OER indicates that the vast majority perceive OER to be of equal or superior quality to traditional textbooks (see Hilton, 2016 for a review). Three additional studies, utilizing a control group of students rating a traditional textbook, produced somewhat mixed results. Specifically, one revealed that students' ratings of the quality of an open textbook on American Government were significantly lower than another cohort of students' ratings of the quality of a traditional textbook on the same topic (Lawrence & Lester, 2018). The results of the second revealed only one difference in students' ratings of seven dimensions of the quality of their assigned Introductory Psychology textbook, with students using an open textbook rating the writing significantly higher than those assigned a traditional textbook (Clinton, 2018). Finally, the results of the third indicated that students assigned a print format of a digital open Introductory Psychology textbook rated seven of 16 dimensions of the quality of that textbook higher than did students assigned a traditional Introductory Psychology textbook. However, no differences were found in the textbook quality ratings of students assigned a digital open textbook and those assigned either a printed open textbook or traditional textbook (Jhangiani et al., 2018).

Finally, while research has examined use and perceptions of OER in online and traditional classroom environments (e.g., Hendricks, Reinsberg, & Rieger, 2017; Hilton, Fischer, Wiley, & William, 2016; Lindshield & Adhikari, 2013), only one previous study has directly compared students taking classes online with those taking classes in the classroom. The results of that study indicated that online students used a Human Nutrition OER more frequently and rated it as higher quality than students taking classes in the classroom (Lindshield & Adhikari, 2013). However, without a group of students assigned a traditional textbook, it is unclear whether those findings are unique to OER or would extend to traditional textbooks. It is important to consider potential differences in these cohorts of students as online courses are a rapidly expanding market (Allen & Seaman, 2009) and online students typically possess different demographic characteristics that may impact their perceptions and use of open textbooks. For instance, online students tend to be older (Johnson, 2015; Ke & Xie, 2009; Kummerow, Miller, & Reed 2012; Moore & Kearsley, 1996; U.S. Department of Education, 2014), predominantly female (Ke & Xie 2009; U.S. Department of Education, 2014), work in outside employment (Colorado & Eberle, 2010), and take courses part-time (Kim, Kwon, & Cho, 2011; U.S. Department of Education, 2014). Given the lack of research comparing these groups, it is unclear whether online students would use or perceive open textbooks differently than students in a traditional classroom environment.

In summary, while there is an existing body of research on students' use of open textbooks and their perceptions of the relevance and quality of these textbooks, few studies have focused on these aspects in relation to psychology textbooks specifically. Moreover, given the inherent difficulties of conducting controlled research in learning environments where researchers have less than perfect control, very few studies have included a control group of students using and evaluating traditional textbooks. Therefore, the present study was guided by the following three research questions:

Do psychology students report using/reading open textbooks more or less than traditional textbooks?

Are there differences in the perceived need for open versus traditional psychology textbooks to complete various aspects of coursework?

Are various dimensions of the quality of open psychology textbooks rated differently than traditional psychology textbooks?

Moreover, the present study sought to compare each of these outcomes in students taking classes online with those taking classes in a classroom environment.

Method

Design and Procedure

Undergraduate students were asked to complete a brief (10–15 min) online survey in the Fall 2017 and Spring 2018 semesters. Students in sections with a response rate of 75% or higher were given a small number of bonus points to encourage a high response rate. A 2 × 2 between-subjects cross-sectional design was used to compare the responses of students assigned two different types of textbooks (traditional vs. open) in two different types of classroom environments (classroom vs. online). As shown in Table 1, this convenience sample of students was drawn from 15 different sections of four courses taught by eight different instructors using one of three open textbooks or one of three traditional textbooks. Although some attempts were made to equate the difficulty-level and popularity/interest-level of the courses used in the open and traditional textbook conditions, due to the lack of control over course instructor assignments and lack of availability of open textbooks for several of the courses, the courses and instructors were not matched across conditions. Nevertheless, there is little systematic relationship among courses, instructor, and textbook type and as such these variables should be considered extraneous, rather than true confounding, variables. It should also be noted that students taking classes online at this university are rarely permitted to take classes in the classroom and those taking classes in the classroom are rarely permitted to take classes online. As such, the online and classroom students represent different student bodies. The Office of Research Assurances found this study to be exempt from the need for Institutional Review Board review.

Table 1.

Courses, Environment, Textbook, Instructor, and Course Designer.

Course/Term	Environment	Textbook	Instructor	Course designer
312F	Classroom	Open	A	A
312S	Classroom	Open	A	A
312S	Online	Open	B	A
312F	Online	Open	C	A
312S	Online	Open	D	A
210F	Online	Traditional	E	B
210F	Online	Traditional	F	B
210S	Online	Traditional	F	B
333F	Classroom	Traditional	A	A
333S	Classroom	Traditional	A	A
333F	Online	Traditional	G	B
333S	Online	Open	G	B
328S	Online	Open	B	B
328S	Online	Open	H	B
328S	Classroom	Open	B	B

Note. 312 is a Research Methods course, 210 is an Introduction to Research Methods and Critical Thinking course, 333 is an Abnormal Psychology course, and 328 is a Behavior Modification course. Instructors and course designers are identified by letter with instructor A and course designer A representing the same individual and instructor B and course designer B representing the same individual.

Materials

Demographic and financial characteristics

Participants were asked to indicate their age, gender, minority status, and status as a first-generation student. They were also asked to indicate their year of university, the number of courses they were currently taking, and their grade point average. Participants were further asked to indicate whether or not they have a student loan from a bank or government, receive financial assistance from the university, and/or receive financial assistance from their family. They were asked to input the total amount they owe in student loans, the average amount they spend on textbooks in a typical semester, and their average weekly income. Finally, participants were asked to indicate how many hours they work per week using a scale ranging from 0 to 8, where 0 indicated 0 h, 10 indicated more than 35 h, and scale points 1–7 represented the 5-h intervals in between (e.g., 1 = 1–5 h).

Use of textbook

Participants were asked to indicate how often they read/use the required textbook in a typical course using the following response scale: 0 = Never, 1 = once per semester, 2 = 2–3 times per semester, 3 = 2–3 times per month, 4 = 1–3 times per week, 5 = 4–6 times per week, 6 = daily. They were also asked whether or not they had read/used the textbook for the course they were being surveyed about. Those who responded in the affirmative were asked to indicate how often they read/used the assigned textbook for their surveyed course using the same 0–6 scale described above. They were further asked to indicate how much time per week they devoted to reading the textbook for their surveyed course (a) overall, (b) when studying for exams, (c) when studying for quizzes, and (d) when completing assignments, using the following response scale: 0 = 0 h, 1 = 1–2 h, 2 = 3–4 h, 3 = 5–6 h, 4 = 7–8 h, 5 = more than 8 h.

Relevance of textbook

Participants were asked to input the overall percentage of textbooks they have been required to purchase that were not used enough to justify their cost. They were also asked to indicate the percentage of (a) quiz, (b) exam, and (c) assignment questions that could be answered using the textbook in their surveyed course, using the following response scale: 0 = 0–25%, 1 = 26–50%, 2 = 51–75%, 3 = 76–100%. Finally, they were asked to use the same scale to indicate the percentage of overlap between the textbook and lecture material in their surveyed course.

Quality of textbook

An abbreviated version of the Textbook Assessment and Usage Scale (Gurung & Martin, 2011) was used to assess quality of the textbooks. Participants were shown 15 statements pertaining to various dimensions of the quality of their textbook and for each were asked to rate the assigned textbook they were being surveyed about using a 1 (not at all) to 7 (very much) scale. The complete items are shown in Table 5; they assess various dimensions of quality including aesthetic quality (e.g., how visually appealing is the textbook?), simplicity (e.g., how easy are the figures to understand?), and utility (e.g., how well do the research examples help you understand the material?). Cronbach's alpha was 0.97 in the present study, suggesting a high degree of internal consistency. Finally, participants were asked to rate the overall quality of their assigned textbook using the following scale: 0 = very poor, 1 = below average, 2 = average, 3 = above average, 4 = excellent.

Participants

Undergraduate students enrolled in psychology classes at Washington State University (N = 1133) were invited to complete the online survey and 925 of them completed it (response rate = 81.6%). The sample ranged in age from 18 to 65 (M = 24.45, SE = 0.25) and was predominantly female (82.2%) and Caucasian (74.3%). Students were taking an average of 4.80 (SE = 0.83) courses and the average self-reported GPA was 3.25 (SE = 0.02).

Of the 925 students who completed the survey, 363 were in a traditional classroom environment and were assigned a traditional textbook, 203 were in a traditional classroom and were assigned an open textbook, 181 were in an online class and were assigned a traditional textbook, and 178 were in an online class and were assigned an open textbook.

Demographic, financial, and other characteristics for each of the four groups are provided in Table 2. As shown in the table, a series of logistic regression analyses and 2 × 2 analyses of variance revealed a number of differences across the groups. Specifically, there was a higher percentage of females and first-generation students in online classes than in the classroom and fewer students taking classes online reported receiving financial assistance from their families relative to those taking classes in the classroom. Online students were also significantly older (with a slightly larger discrepancy in the ages of the two open textbook groups than the two traditional textbook groups), reported spending more money on textbooks, worked significantly more hours, earned a higher income, were enrolled in fewer courses (with a stronger effect in those assigned open textbooks), and were more advanced in their education, than the students taking classes in the classroom. Finally, students assigned open textbooks were significantly more advanced in their undergraduate education than those assigned traditional textbooks. Given these potentially confounding differences, gender, first-generation status, family financial assistance, age, year in university, number of currently enrolled courses, hours of employment, and weekly income will be entered as covariates in subsequent analyses involving comparisons of the four groups.

Table 2.

Demographic and Financial Characteristics of the Four Groups.

	Classroom		Online		Main Effects and Interactions
	Traditional	Open	Traditional	Open	Main Effect Textbook			Main Effect Classroom
	N = 363	N = 203	N = 181	N = 178	χ²	p	OR	χ²	p	OR
% Female	79.3%	80.1%	87.3%	85.3%	0.41	0.84	0.96	6.62	0.01	0.61
% Female	79.3%	80.1%	87.3%	85.3%	Interaction			0.28	059	1.23
% Caucasian	76.3%	70.2%	75.7%	73.7%	2.21	0.14	0.79	0.17	0.68	0.94
% Caucasian	76.3%	70.2%	75.7%	73.7%	Interaction			0.40	0.53	0.82
% First generation	29.3%	32.5%	49.7%	38.6%	0.64	0.42	1.12	18.43	<0.001	1.84
% First generation	29.3%	32.5%	49.7%	38.6%	Interaction			4.43	0.04	0.55
Financial assistance (university)	59.5%	58.1%	54.1%	53.9%	0.04	0.53	1.10	1.90	0.17	0.77
Financial assistance (university)	59.5%	58.1%	54.1%	53.9%	Interaction			0.12	0.73	0.12
Financial assistance (family)	68.9%	69.0%	18.9%	23.6%	0.92	0.34	0.85	32.23	<0.001	0.28
Financial assistance (family)	68.9%	69.0%	18.9%	23.6%	Interaction			0.11	0.75	0.11
Financial assistance (other)	60.9%	60.1%	69.6%	68.0%	0.64	0.42	1.13	0.83	0.36	1.20
Financial assistance (other)	60.9%	60.1%	69.6%	68.0%	Interaction			0.03	0.86	0.03
					Main effect textbook			Main effect classroom
					F	p	$η_{p}^{2}$	F	p	$η_{p}^{2}$
Age	20.31 (0.30)	20.97 (0.40)	30.96 (0.42)	30.07 (0.42)	0.09	0.78	0.000	650.73	<0.001	0.42
Age	20.31 (0.30)	20.97 (0.40)	30.96 (0.42)	30.07 (0.42)	Interaction			4.05	0.04	0.004
University year	2.92 (0.05)	3.29 (0.07)	3.13 (0.08)	3.48 (0.08)	26.54	<0.001	0.03	8.37	0.004	0.009
University year	2.92 (0.05)	3.29 (0.07)	3.13 (0.08)	3.48 (0.08)	Interaction			0.03	0.87	0.000
Number of courses	5.10 (0.13)	5.94 (0.17)	3.79 (0.18)	3.91 (0.18)	8.58	0.003	0.009	103.40	<0.001	0.10
Number of courses	5.10 (0.13)	5.94 (0.17)	3.79 (0.18)	3.91 (0.18)	Interaction			4.80	0.03	0.005
Grade point average	3.26 (0.03)	3.21 (0.04)	3.28 (0.04)	3.24 (0.04)	1.09	0.30	0.001	0.59	0.44	0.001
Grade point average	3.26 (0.03)	3.21 (0.04)	3.28 (0.04)	3.24 (0.04)	Interaction			0.004	0.95	0.000
Student loans	$26,394.41 (2346.92)	$24,581.48 (3032.43)	$24,674.70 (3093.01)	$24,153.53 (3150.13)	0.19	0.66	0.000	0.09	0.76	0.000
Student loans	$26,394.41 (2346.92)	$24,581.48 (3032.43)	$24,674.70 (3093.01)	$24,153.53 (3150.13)	Interaction			0.06	0.81	0.000

					Main effect textbook			Main effect classroom
					F	p	$η_{p}^{2}$	F	p	$η_{p}^{2}$
Textbook $/ semester	$339.64 (10.04)	$338.48 (12.95)	$386.790 (14.71)	$380.21 (14.44)	0.11	0.75	0.000	7.73	0.006	0.009
Textbook $/ semester	$339.64 (10.04)	$338.48 (12.95)	$386.790 (14.71)	$380.21 (14.44)	Interaction			0.05	0.82	0.000
Hours worked/week	1.73 (0.16)	2.15 (0.20)	5.17 (0.23)	4.98 (0.22)	0.43	0.51	0.000	159.44	<0.001	0.15
Hours worked/week	1.73 (0.16)	2.15 (0.20)	5.17 (0.23)	4.98 (0.22)	Interaction			2.69	0.10	0.003
Income/week	$150.04 (33.13)	$217.81 (42.62)	$431.91 (48.76)	$473.31 (47.93)	1.86	0.17	0.002	25.63	<0.001	0.03
Income/week	$150.04 (33.13)	$217.81 (42.62)	$431.91 (48.76)	$473.31 (47.93)	Interaction			0.11	0.74	0.000

Note. Bolded statistics indicate a significant effect. Percentages are presented for nominal variables. Means are presented with standard errors in parentheses for continuous variables.

N: sample size; χ²: Pearson's chi-squared; p: p-value; OR: odds ratio; $η_{p}^{2}$ : partial eta-squared.

Statistical Analyses

In order to examine the main effects of classroom type and textbook type as well as potential classroom × textbook interactions on dependent variables measured on interval or ratio scales (for which means could be computed), 2 × 2 analyses of covariance (ANCOVA) were used. ANCOVA rather than more traditional analyses of variance were used in order to control for the confounds identified above. Multivariate analyses of covariance (MANCOVA) was used to examine main effects of classroom and textbook types and their interactions for outcomes with multiple dependent variables (i.e., time spent using the textbooks for various purposes, ratings of 15 different dimensions of quality), while controlling for confounds. Significant MANCOVAs were followed up with a series of ANCOVAs examining specific effects on each dependent variable. Finally, hierarchical logistic regression was used to examine main effects of classroom type and textbook type and their interaction for binary outcome variables (e.g., use vs. no use of textbook). Confounding group differences were entered into the first step of these analyses to statistically control for them, the primary variables of interest (classroom type, textbook type) were entered into the second step, and finally classroom × textbook interactions were entered into the third and final step of these analyses.

Results

Use of Textbooks

The average rating on the survey item asking students to indicate the frequency with which they read/use textbooks in a typical course was 3.94 (SE = 06), which roughly translates to 1–3 times per week. A 2 × 2 ANCOVA was conducted to examine group differences in typical textbook usage while controlling for all confounding demographic and financial variables. The results revealed no significant interaction; however, there was a significant main effect of classroom type, F = 55.80 p < 0.001, $η_{p}^{2} = 0.06$ , with online students reporting reading/using their textbooks in typical courses significantly more (M = 4.12, SE = 0.08) than those taking classes in the classroom (M = 3.22, SE = 0.06). There was also a significant main effect of textbook type, F = 8.97, p = 0.003, $η_{p}^{2} = 0.01$ , with students in classes assigned open textbooks reporting using textbooks in typical courses significantly more (M = 3.79, SE = 0.06) than those assigned traditional textbooks (M = 3.55, SE = 0.05). As such, this variable will be added as a covariate in subsequent analyses.

Overall 74.1% of the students reported reading/using the textbook they were assigned for their surveyed course. Breaking this down by group, only 52.9% of students taking classes in a classroom with a traditional textbook reported reading/using the textbook, 74.4% of students taking classes in a classroom with an open textbook reported using it, 96.1% of online students assigned a traditional textbook used it, and 94.4% of online students assigned an open textbook reported using it. Hierarchical logistic regression was used to compare these percentages while controlling for all potentially confounding variables. The results revealed an overall significant effect of textbook type, χ² = 10.79, p < 0.001, with students assigned an open text being 1.89 times more likely to use their textbook than those assigned a traditional text. There was also a significant effect of classroom type, χ² = 8.94, p = 0.003, with online students being 3.07 times more likely to read/use their textbook than students in the classroom. The interaction between textbook type and classroom type was not significant.

To determine the effects of classroom type and textbook type on the frequency of textbook use across the four groups while controlling for the confounding variables, a 2 × 2 ANCOVA was conducted using only the 74.1% of students who reported reading/using their assigned textbook in their surveyed course. There was a significant interaction between classroom type and textbook type on overall frequency of textbook use, F = 4.84, p = 0.03, $η_{p}^{2} = 0.008$ . Follow-up tests indicated that for students taking classes in the classroom, the frequency of reading the open textbook was significantly higher than the frequency of reading the traditional textbooks (p < 0.001). There was a similar trend that was not statistically significant in the online students (p = 0.054).

A 2 × 2 × 4 MANCOVA was used to examine the effects of classroom type and textbook type on the average amount of time students spent using their assigned textbook in their surveyed course (a) overall, (b) to study for exams, (c) to study for quizzes, and (d) to complete assignments, while controlling for all confounding variables. The results revealed no significant interaction. However, there was a significant main effect for textbook type, Wilks' λ = 0.98, F = 2.85, p = 0.02,

η_{p}^{2} = 0.02

and classroom type, Wilks' λ = 0.90, F = 15.88, p < 0.001,

η_{p}^{2} = 0.10

. As shown in Table 3, follow-up ANCOVAs revealed that students assigned an open textbook reported using it for significantly more time overall compared to those assigned a traditional textbook. In contrast, the differences between the amount of time spent using the textbook for exams, quizzes, and assignments were not significantly different in students assigned open versus traditional textbooks. Further, students taking courses online were found to spend significantly more time using their assigned textbook overall, to study for exams, to study for quizzes, and to complete assignments.

Table 3.

Ratings of Time Spent Reading/Using the Textbook.

	Classroom		Online		Main Effects and Interactions
	Traditional N = 179	Open N = 138	Traditional N = 155	Open N = 150	Main Effect Textbook			Main Effect Classroom
	Traditional N = 179	Open N = 138	Traditional N = 155	Open N = 150	F	p	$η_{p}^{2}$	F	p	$η_{p}^{2}$
Overall time/week	1.32 (0.07)	1.47 (0.8)	2.25 (.09)	2.40 (0.09)	4.56	0.03	0.007	5.03	0.03	0.008
Overall time/week	1.32 (0.07)	1.47 (0.8)	2.25 (.09)	2.40 (0.09)	Interaction			0.01	0.93	0.000
To study for exams	1.62 (0.08)	2.19 (0.09)	2.44 (0.09)	2.35 (0.09)	0.18	0.67	0.000	23.29	<0.001	0.04
To study for exams	1.62 (0.08)	2.19 (0.09)	2.44 (0.09)	2.35 (0.09)	Interaction			0.64	0.42	0.001
To study for quizzes	1.43 (0.09)	1.86 (0.10)	2.39 (0.10)	2.35 (0.09)	0.71	0.40	0.001	8.64	0.003	0.01
To study for quizzes	1.43 (0.09)	1.86 (0.10)	2.39 (0.10)	2.35 (0.09)	Interaction			1.00	0.32	0.002
To complete assignments	1.95 (0.09)	2.33 (0.09)	2.04 (0.09)	2.22 (0.09)	2.08	0.15	0.003	11.44	0.001	0.02
To complete assignments	1.95 (0.09)	2.33 (0.09)	2.04 (0.09)	2.22 (0.09)	Interaction			0.00	0.99	0.000

Note. Bolded statistics indicate a significant effect. Means (with effects of covariates controlled) are presented with standard errors in parentheses. The scale used to provide responses was as follows: 0 = 0 h, 1 = 1–2 h, 2 = 3–4 h, 3 = 5–6 h, 4 = 7–8 h, 5 = more than 8 h.

N: sample size; F: F-test; p: p-value; $η_{p}^{2}$ : partial eta-squared.

Relevance of Textbooks

Overall students reported that 51.68% (SE = 31.14) of the textbooks they have been required to purchase in the past were not used enough to justify their cost. A 2 × 2 ANCOVA comparing the four groups, while controlling for all confounds, revealed no significant interaction. However, there were main effects of textbook type, F = 4.26, p = 0.04, $η_{p}^{2} = 0.005$ and classroom type, F = 26.76, p < 0.001, $η_{p}^{2} = 0.03$ . Overall students in the classroom reported a significantly higher percentage of underutilized textbooks (M = 57.44, SE = 1.38) than online students (M = 43.20, SE = 1.90), and students assigned an open textbook reported a significantly higher percentage of underutilized textbooks (M = 52.20, SE = 1.38) than those assigned a traditional textbook (M = 48.44, SE = 1.21).

A 2 × 2 × 4 MANCOVA was used to compare the percentage of quiz, exam, and assignment questions that students in the four groups reported could be answered using the textbook assigned in their surveyed course and the percentage of overlap between the lecture and textbook material in their surveyed course. Once again only the 74.1% of students who reported using their assigned textbook were included and confounding variables were entered as covariates. The results revealed a significant interaction between classroom type and textbook type, Wilks' λ = 0.96, F = 5.91, p < 0 .001,

η_{p}^{2} = 0.04

. As shown in Table 4, follow-up tests revealed significant classroom type × textbook type interactions on the reported percentages of exam and assignment questions that could be answered using the assigned textbook as well as significant main effects of textbook type on the number of quiz questions that could be answered using the assigned textbook and the percentage of overlap between the textbook and lecture material. Follow-up tests of the two significant interactions revealed significant simple effects, indicating that students taking courses in the classroom (but not those taking courses online) reported that a significantly higher percentage of exam, F = 22.01, p < 0.001,

η_{p}^{2} = 0.07

, and assignment questions F = 8.96, p = 0.003,

η_{p}^{2} = 0.03

, could be answered using the open textbook compared to the traditional textbook.

Table 4.

Ratings of Percentage of Questions that Could be Answered Using Text and Overlap Between Textbook and Lecture Material.

	Classroom		Online		Main Effects and Interactions
	Traditional N = 181	Open N = 138	Traditional N = 154	Open N = 151	Main Effect Textbook			Main Effect Classroom
	Traditional N = 181	Open N = 138	Traditional N = 154	Open N = 151	F	p	$η_{p}^{2}$	F	p	$η_{p}^{2}$
Quiz questions	2.26 (0.08)	2.54 (0.9)	2.25 (.09)	2.40 (0.09)	7.84	0.005	0.01	0.42	0.52	0.001
Quiz questions	2.26 (0.08)	2.54 (0.9)	2.25 (.09)	2.40 (0.09)	Interaction			0.71	0.40	0.001
Exam questions	1.62 (0.08)	2.19 (0.09)	2.44 (0.09)	2.35 (0.09)	10.05	0.002	0.02	19.66	<0.001	0.03
Exam questions	1.62 (0.08)	2.19 (0.09)	2.44 (0.09)	2.35 (0.09)	Interaction			20.03	<0.001	0.03
Assignment questions	1.43 (0.09)	1.86 (0.10)	2.39 (0.10)	2.35 (0.09)	5.53	0.02	0.009	34.76	<0.001	0.05
Assignment questions	1.43 (0.09)	1.86 (0.10)	2.39 (0.10)	2.35 (0.09)	Interaction			8.85	0.003	0.01
Textbook/lecture overlap	1.95 (0.09)	2.33 (0.09)	2.04 (0.09)	2.22 (0.09)	12.56	<0.001	0.02	0.01	0.93	0.000
Textbook/lecture overlap	1.95 (0.09)	2.33 (0.09)	2.04 (0.09)	2.22 (0.09)	Interaction			1.68	0.19	0.003

N: sample size; F: F-test; p: p-value; $η_{p}^{2}$ : partial eta-squared.

Quality of Textbooks

Students who reported using/reading the assigned textbook were once again selected and a 2 × 2 × 15 MANCOVA was used to compare the four groups' ratings of the 15 dimensions of quality of the textbook assigned in their surveyed course, while statistically controlling for all confounding variables. The interaction between classroom type and textbook type was not significant, nor was the main effect of classroom type. However, there was a significant overall main effect of textbook type, Wilks' λ = 0.85, F = 3.79, p < 0.001,

η_{p}^{2} = 0.15

. As shown in Table 5, follow-up one-way ANCOVAs examining the main effect of textbook type on each item indicated that the open textbooks received significantly higher ratings on 11 of the 15 dimensions and equivalent ratings on the remaining four dimensions. Consistent with these results, a 2 × 2 ANCOVA on students' ratings of the overall quality of the textbook assigned in their surveyed course revealed no interaction and no main effect of classroom type. However, there was a significant main effect of textbook type, F = 63.50, p < 0.001,

η_{p}^{2} = 0.11

, with students assigned open textbooks providing significant higher ratings (M = 3.90, SE = 0.07) than those assigned traditional textbooks (M = 3.03, SE = 0.08).

Table 5.

Ratings of Quality of the Assigned Textbooks Across the Four Groups.

	Classroom		Online
	Traditional N = 55	Open N = 94	Traditional N = 102	Open N = 89	Main Effect Textbook
	Traditional N = 55	Open N = 94	Traditional N = 102	Open N = 89	F	p	$η_{p}^{2}$
How much do the figures help you understand the text?	4.58 (0.22)	5.39 (0.17)	5.44 (0.16)	5.58 (0.17)	8.13	0.005	0.02
How easy are the figures to understand?	5.09 (0.20)	5.53 (0.16)	5.49 (0.14)	5.72 (0.15)	5.03	0.03	0.02
How easy are the tables to understand?	5.26 (0.20)	5.61 (0.15)	5.59 (0.14)	5.84 (0.15)	4.08	0.04	0.01
How relevant are the photographs in relation to the material?	4.92 (0.21)	5.28 (0.17)	5.34 (0.16)	5.67 (0.16)	4.70	0.03	0.01
How well do the research examples explain the material?	5.14 (0.17)	5.77 (0.14)	5.86 (0.13)	6.12 (0.14)	12.15	0.001	0.04
How much do the research examples help you understand the material?	4.84 (0.19)	5.81 (0.15)	5.90 (0.13)	6.08 (0.14)	16.98	<0.001	0.04
How well do the everyday life examples explain the material?	5.36 (0.18)	5.75 (0.14)	6.09 (0.13)	6.10 (0.14)	2.38	0.12	0.007
How much do the everyday life examples help you understand the material?	5.30 (0.18)	5.87 (0.14)	6.07 (0.13)	6.12 (0.14)	5.35	0.02	0.02
How relevant are the everyday life examples?	5.29 (0.18)	5.70 (0.14)	5.93 (0.13)	5.97 (0.14)	2.69	0.10	0.008
How adequate is the number of study aids?	5.16 (0.20)	5.52 (0.16)	5.47 (0.15)	5.60 (0.16)	2.64	0.11	0.008
How helpful are the study aids?	4.83 (0.22)	5.54 (0.17)	5.14 (0.16)	5.49 (0.17)	10.71	0.001	0.03
How visually appealing is the textbook?	4.78 (0.23)	5.08 (0.18)	5.35 (0.17)	5.14 (0.18)	0.10	0.76	0.000
How engaging/interesting is the writing?	4.32 (0.22)	5.21 (0.17)	5.26 (0.16)	5.45 (0.16)	11.11	0.001	0.03
How understandable/clear is the writing?	5.10 (0.19)	5.84 (0.15)	5.59 (0.14)	5.98 (0.15)	15.61	<0.001	0.05
How helpful is the textbook overall?	4.95 (0.18)	6.01 (0.14)	5.78 (0.13)	6.18 (0.14)	29.91	<0.001	0.08

Note. Bolded statistics indicate a significant effect. Means (with effects of covariates controlled) are presented with standard errors in parentheses. Responses were provided using a 1 (not at all) to 7 (very much) scale.

N: sample size; F: F-test; p: p-value; $η_{p}^{2}$ : partial eta-squared.

Discussion

The results of this survey of a large sample of undergraduate psychology students at a major U.S. university indicate that more than 50% of the textbooks that they have been required to purchase in the past were not used enough to justify their cost. This finding is consistent with previous research (Jhangiani & Jhangiani, 2017), but further extends upon it by demonstrating that students taking classes in the classroom report significantly higher rates of underutilized textbooks than those taking classes online. Moreover, over 25% of the students surveyed reported that they had not used the textbook assigned for the class they were being surveyed about. This rate was particularly high (approaching 50%) in students taking classes in the classroom using a traditional textbook. In contrast, the vast majority (>95%) of students taking classes online reported using their assigned textbook.

Of greater interest, the results pertaining to textbook usage revealed that students assigned an open textbook were 1.89 times more likely to report using that textbook compared to students assigned a traditional textbook. Similarly, students assigned open textbooks reported using them more frequently and they reported spending more overall time per week reading them. These results conflict with the findings of Lawrence and Lester (2018) and Jhangiani and colleagues (2018) described in the Introduction, but are consistent with the finding of Feldstein et al. (2012) that more students downloaded the OER than had previously purchased the traditional textbook. This suggests that one benefit of OER may be increased engagement with the material.

One reason why the students assigned open textbooks may use those textbooks more is that they perceive a greater need for/relevance of their textbook relative to those assigned traditional textbooks. More specifically, compared to students assigned traditional textbooks, those assigned open textbooks reported that a significantly higher percentage of quiz questions could be answered using the open textbook and they perceived a significantly greater percentage of overlap between the open textbook and lecture material. The mean rating for the percentage of overlap between the open textbook and lecture material was just over 2, which translates to 50–75% and is consistent with what has been reported previously by students using OER (Jhangiani & Jhangiani, 2017). Additionally, for students taking classes in the classroom, those assigned open textbooks reported a significantly higher percentage of exam and assignment questions that could be answered using that textbook compared to those assigned traditional textbooks. These results may be, at least partially, due to the fact that the course designers created the open textbooks specifically for these courses and/or utilized the Creative Commons licenses to update and customize the open textbooks for the courses. Students assigned unadulterated, existing OER may not share these perceptions. While comparisons of adapted OER to traditional textbooks have been criticized as introducing a confounding effect (Griggs & Jackson, 2017), the ability to customize OER is one of their strongest distinguishing features. Our results indicate that use of the Creative Commons license to adapt OER may better align them with course goals, thereby better suiting the needs of instructors and students. Nevertheless, future research is needed to compare open textbooks that have not been modified with traditional textbooks (that cannot be modified).

While one of the top barriers faculty report to adopting OER is concern about their quality relative to traditional textbooks, the results of the present study are consistent with previous research showing that students rate open textbooks of equivalent or superior quality to traditional textbooks (see Hilton, 2016 for a review). More specifically, students assigned open textbooks rated them significantly higher on 11 of 15 different dimensions and they gave significantly higher ratings of their overall quality compared to students assigned traditional textbooks. The remaining four dimensions were rated comparably across these two groups. These results are consistent with those of Jhangiani and colleagues (2018) who found that students assigned a print format of an open Introductory Psychology textbook rated 7 out of 16 dimensions of the quality of that textbook higher than students assigned a traditional Introductory Psychology textbook. These results build on the existing literature by demonstrating superior ratings of several open textbooks using a control group of students rating multiple traditional textbooks, which should improve the internal and external validity of these findings.

One motivation for conducting this study was to examine whether online students differ from classroom students in their use and perceptions of open versus traditional textbooks. Only a few differences emerged; the only substantial difference was that although classroom students perceived that a significantly higher percentage of exam and assignment questions could be answered using the open textbooks than traditional textbooks, online students assigned the two different types of textbooks reported no significant differences in the number of quiz or assignment questions that could be answered using open versus traditional textbooks.

Further comparisons of students taking classes online with those taking classes in the classroom revealed that online students were over three times more likely to report using their assigned textbook in their surveyed courses and they reported using their assigned textbooks (both traditional and open) significantly more than classroom students overall, to study for exams, to study for quizzes, and to complete assignments. These findings are consistent with those of Lindshield and Adhikari (2013), who found that online students reported using an OER significantly more than students taking classes in the classroom; but the present study expands upon these findings by demonstrating that this effect is not limited to OER and extends to traditional textbooks. It is likely that the self-directed nature of online classes and the lack of “live” lectures and office hours increase the need for a textbook in the online learning environment. Finally, the two groups gave comparable ratings of the quality of the textbooks, which contrasts with the results of one previous study that found that online students gave higher textbook quality ratings than classroom students (Lindshield & Adhikari, 2013).

The primary limitations of this study are the exclusive use of self-report instruments and the cross-sectional design employing a non-equivalent control group. Since students could not practically be randomly assigned to courses, environments, or instructors, causal conclusions cannot be conclusively made, and results should be interpreted as associations with open versus traditional textbooks rather than true cause-and-effect relationships. The inability to control for which instructors used open versus traditional textbooks is a notable limitation that could be addressed by future research that matches instructors across open versus traditional textbook groups. Nevertheless, the inclusion of a control group of students assigned a traditional textbook represents an improvement upon the bulk of the existing literature that has failed to include any control group. Moreover, numerous potentially confounding variables were identified and statistically controlled for by entering them as covariates in the statistical comparisons of the groups. This process would rule out these variables as competing explanations for the findings.

Of course, it is possible that there are other confounding variables that were not considered, measured, and statistically controlled. For instance, since the vast majority of students used digital format for the open textbooks but print format for the traditional textbooks, format of the textbooks represents an important confounding effect that could not be controlled for in the present study. Given that most students report a preference for printed textbooks (e.g., Jhangiani et al., 2018), it is unlikely that this difference contributed to the finding that the open textbooks garnered higher ratings of quality or use than the traditional textbooks.

Limitations in internal validity are often offset by enhancements in external validity and the use of multiple open and traditional textbooks in several different courses with numerous different instructors should improve the ability to generalize the findings to other textbooks, courses, and instructors. The large sample and high response rate should also further increase external validity.

Conclusions

Many students report forgoing purchasing required textbooks due to exorbitant cost and OER are increasingly being endorsed as a potential solution to this problem. However, few previous studies have empirically examined the assumption that OER will increase access to course materials and the results of those studies are equivocal. The present study provides support for this assumption by demonstrating that compared to those assigned traditional textbooks, students assigned open textbooks were almost twice as likely to report reading/using those textbooks. Further, results from the present study indicate that students assigned open textbooks reported using them more frequently and for more time per week overall. It is possible that this increased use of the open textbooks was related to students' perceived need for the textbook to complete coursework as students assigned open textbooks also reported that more quiz questions could be answered using the open textbooks than those assigned traditional textbooks, and they perceived a greater degree of overlap between the open textbooks and lecture material. This may reflect the customizability of open textbooks, specifically the ability to edit and adapt them to suit the needs of the course. One notable barrier that has prevented faculty from adopting OER is concerns about the quality of the materials. The present study extends upon a growing body of research indicating that OER are not perceived to be lower in quality than traditional textbooks. Specifically, in the present study, ratings of the open textbooks were significantly higher than ratings of the traditional textbooks overall and on 11 of 15 different dimensions of quality. Finally, although online students were significantly more likely to report reading/using their textbook at all and to report using their textbook more frequently, there were few substantive differences in the perceptions and quality ratings of online and classroom students—suggesting that both groups experienced similar benefits of open textbooks. Collectively, these findings extend upon a burgeoning literature demonstrating that replacing traditional textbooks with open textbooks may help to offset some of the financial hardships that students face while improving student engagement and satisfaction with their assigned textbook.

Footnotes

Acknowledgements

The author thanks the William and Flora Hewlett Foundation and the Open Education Group for sponsoring the OER research fellowship that supported this work, Lee W. Daffin Jr. for his assistance with collecting data for this study and for reviewing a draft of this manuscript, and John Hilton III for his guidance and for reviewing a draft of this manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Carrie Cuttler is an assistant professor at Washington State University. She received her Ph.D. at the University of British Columbia and has been teaching courses in psychology for over a decade at the University of British Columbia, Concordia University, and now at Washington State University. She currently teaches abnormal psychology and graduate-level statistics courses. She is currently an OER Research Fellow with the Open Education Group and serves on her university's OER Visioning Group and Course Materials Value and Effectiveness Committee. She co-authored an open textbook entitled Research Methods in Psychology: 3rd American Edition and edited an open textbook entitled Essentials of Abnormal Psychology.

References

Allen, I. E., & Seaman, J. (2009). Learning on demand: Online education in the United States. Retrieved from https://onlinelearningconsortium.org/survey_report/2009-learning-demand-online-education-united-states/.

Bliss, T., Hilton, J., Wiley, D., & Thanos, K. (2013a). The cost and quality of open textbooks: Perceptions of community college faculty and students. First Monday, 18. Retrieved from https://doi:10.5210/fm.v18i1.3972.

Bliss T., Robinson, T. J., Hilton, J., & Wiley, D. (2013b). An OER COUP: College teacher and student perceptions of open educational resources. Journal of Interactive Media in Education. Retrieved from https://doi:10.5334/2013-04.

California Open Educational Resources Council. (2016). OER adoption study: Using open educational resources in the college classroom. Retrieved from http://tinyurl.com/WPOERAdoption040116.

Clinton, V. (2018). Savings without sacrifices: A case study of open-source textbook adoption. Open Learning: The Journal of Open, Distance, and e-learning. Retrieved from: https://www.researchgate.net/publication/324088320_Savings_without_sacrifice_A_case_report_on_open-source_textbook_adoption.

Colorado

J. T.

Eberle

(2010) Student demographics and success in online learning environments. Emporia State Research Studies 46: 4–10. Retrieved from http://academic.emporia.edu/esrs/vol46/colorado.pdf.

Delimont

Turtle

E. C.

Bennett

Adhikari

Lindshield

B. L.

(2016) University students and faculty have positive perceptions of open/alternative resources and their utilization in a textbook replacement initiative. Research in Learning Technology24Retrieved from https://doi:10.3402/rlt.v24.29920.

Feldstein, A., Martin, M., Hudson, A., Hilton, J., & Wiley, D. (2012). Open textbooks and increased student access and outcomes. European Journal of Open, Distance and E-learning, n2. Retrieved from https://eric.ed.gov/?id=EJ992490.

Florida Virtual Campus. (2016). 2016 Florida Student Textbook and Course Materials Survey. Retrieved from https://florida.theorangegrove.org/og/items/3a65c507-2510-42d7-814c-ffdefd394b6c/1/.

10.

Gil

Candelas

F. A.

Jara

C. A.

Garcia

G. J.

Torres

(2013) Web-based OERs in computer networks. International Journal of Engineering Education 29: 1537–1550.

11.

Griggs

R. A.

Jackson

S. L.

(2017) Studying open versus traditional textbook effects on students' course performance: Confounds abound. Teaching of Psychology 44: 306–312.

12.

Gurung

R. A. R.

Martin

R. C.

(2011) Predicting textbook reading: The textbook assessment and usage scale. Teaching of Psychology 38: 22–28.

13.

Hendricks, C., Reinsberg, S. A., & Rieger, G. W. (2017). The adoption of an open textbook in a large physics course: An analysis of cost, outcomes, use, and perceptions. The International Review of Research in Open and Distributed Learning, 18. Retrieved from https://files.eric.ed.gov/fulltext/EJ1146231.pdf.

14.

Hilton

(2016) Open educational resources and college textbook choices: A review of research on efficacy and perceptions. Educational Technology Research and Development 64: 573–590.

15.

Hilton III, J. L., Fischer, L., Wiley D., & William, L. (2016). Maintaining momentum toward graduation: OER and the course throughput rate. The International Review of Research in Open and Distributed Learning, 17. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2686.

16.

Hilton

J. L.

Gaudet

Clark

Robinson

Wiley

(2013) The adoption of open educational resources by one community college math department. The International Review of Research in Open and Distance Learning 14: 37–50.

17.

Illowsky

B. S.

Hilton

Whiting

Ackerman

J. D.

(2016) Examining student perception of an open statistics book. Open Praxis 8: 265–276.

18.

Jhangiani, R., Dastur, F. N., Le Grand, R., & Penner, K. (2018). As good or better than commercial textbooks: Students perceptions and outcomes from using open digital and open print textbooks. The Canadian Journal for the Scholarship of Teaching and Learning, 9. Retrieved from https://doi:10.5206/cjsotl-rcacea.2018.1.5.

19.

Jhangiani, R., & Jhangiani, S. (2017). Investigating the perceptions, use, and impact of open textbooks: A survey of post-secondary students in British Columbia. The International Review of Research in Open and Distributed Learning, 18. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/3012/4214.

20.

Johnson, G. M. (2015) On-campus and fully-online university students: Comparing demographics, digital technology use and learning characteristics. Journal of University Teaching & Learning Practice, 12. Retrieved from http://ro.uow.edu.au/jutlp/vol12/iss1/4.

21.

Xie

(2009) Toward deep learning for adult students in online courses. The Internet and Higher Education 12: 136–145.

22.

Kim

Kwon

Cho

(2011) Investigating factors that influence social presence and learning outcomes in distance higher education. Computers & Education 57: 1512–1520.

23.

Kummerow

Miller

Reed

(2012) Baccalaureate courses for nurses online and on campus: A comparison of learning outcomes. American Journal of Distance Education 26: 50–65.

24.

Lawrence, C. N., & Lester, J. A. (2018). Evaluating the effectiveness of adopting open educational resources in an introductory American government course. Journal of Political Science Education. Retrieved from https://doi: 10.1080/15512169.2017.1422739.

25.

Lindshield

Adhikari

(2013) Online and campus college students like using an open educational resource instead of a traditional textbook. Journal of Online Learning & Teaching 9: 1–7.

26.

Martin

M. T.

Belikov

O. M.

Hilton

Wiley

Fischer

(2017) Analysis of student and faculty perceptions of textbook costs in higher education. Open Praxis 9: 79–91.

27.

Moore

M. G.

Kearsley

(1996) Distance education: A systems view, Belmont, CA: Wadsworth.

28.

Seaman, J. E., & Seaman, J. (2017). Opening the textbook: Educational resources in U.S. higher education, 2017. Babson Survey Research Group. Retrieved from http://www.onlinelearningsurvey.com/reports/oer2017/2017Opening%20the%20Textbook_format.html.

29.

United Nations Educational, Scientific and Cultural Organization (UNESCO) (2017). What are open educational resources (OERs)? Retrieved from http://www.unesco.org/new/en/communication-and-information/access-to-knowledge/open-educational-resources/what-are-open-educational-resources-oers/.

30.

U.S. Department of Education (2014). Number and percentage of undergraduate students taking distance education or online classes and degree programs, by selected characteristics: Selected years, 2003–04 through 2011–12. National Center for Education Statistics. Retrieved from https://nces.ed.gov/programs/digest/d15/tables/dt15_311.22.asp.