Abstract
In the current study we examined students' course performance, perceptions, and self-reported use of an open textbook compared to a traditional publisher's textbook in an introductory psychology course. Sixty students from two course sections used either an open textbook or a traditional textbook. To minimize the effect of confounding variables, each section was taught by the same instructor, at the same time of day, using identical in-class materials, syllabi, and sequencing, on alternating days of the week. Course performance was measured by analyzing exam scores and final course grades. A 22-item survey was utilized at the end of the term to measure students' textbook perceptions and use. Findings revealed no significant differences in student course performance or textbook use across the two sections. Participants in both groups perceived cost, weight, and convenience as the biggest advantages of an open textbook and perceived ease of reading, convenience, ability to highlight and take notes, ability to quickly find a topic, and ability to keep as a reference as the biggest advantages of a traditional textbook. Finally, when asked to select which type of textbook they preferred, students in each section selected the textbook they were using. The implications of these findings are discussed.
Keywords
Many instructors, including those in psychology, are making the shift to open educational resources (OER), including open textbooks, because of their perceived cost savings for students, flexibility, and increased access to course material (Hilton, 2016). This shift increases the need to address questions regarding the efficacy and perceptions of the texts among students. A growing body of research has examined these qualities of open resources, including cost, outcomes, use, and perceptions, also known as the COUP model (Bliss, Robinson, Hilton, & Wiley, 2013). Findings from these studies suggest that there are few, if any differences, between outcomes (e.g., Fischer, Hilton, Robinson, & Wiley, 2015; Hilton, 2016) and perceptions (e.g., Bliss, Hilton, Wiley, & Thanos, 2013; Hilton, 2016; Hilton, Gaudet, Clark, Robinson, & Wiley, 2013) of open textbooks as compared to traditional textbooks.
The current study examines students' course performance, perceptions, and self-reported use of an open textbook compared to a traditional publisher's textbook in an introductory psychology course. It is nearly impossible to attain true experimental conditions in a classroom setting, but our aim was to control several common extraneous variables that can lessen internal validity, including instructor, semester, and time of course. We compared three outcome variables–course performance, perceptions, and self-reported textbook use–between two introductory psychology courses–one that used an open textbook and one that used a traditional textbook. Both courses were taught by the same instructor, in the same room, at the same time of day, using identical in-class materials, syllabi and sequencing, during the same semester. The same instructional techniques and activities were also used for both course sections. By controlling these factors, we hoped to gain a better understanding of how student performance, perceptions, and use might differ according to the textbook being used.
Literature Review
One of the main drivers of open textbook adoption is the rise of textbook costs. College students spend an average of $1200 per year on textbooks and supplies (Ma, Baum, Pender, & Bell, 2015), a cost that exceeds other education costs (e.g., tuition, housing) over the past decade (Bureau of Labor Statistics, 2016). To reduce cost, instructors have adopted open (free) materials. These materials are typically available under a Creative Commons license, which allow users to retain, reuse, revise, remix, and redistribute the materials (Wiley, n.d.). Free materials have unquestionably saved students money, but do they provide an effective alternative to traditional textbooks in terms of student performance?
Efficacy of Open Materials
Research addressing the efficacy of open materials on student course performance has been mixed, but most publications in this area suggests that open materials typically yield equivalent, if not in some cases, better, student course performance outcomes than traditional textbooks (Hilton, 2016). These results span multiple disciplines, including psychology (Clinton, 2018; Fischer et al., 2015; Hilton & Laman, 2012), history (Grewe & Davis, 2017), business (Feldstein, Martin, Hudson, Hilton III, & Wiley, 2012), biology (Fischer et al., 2015), math (Chiorescu, 2017; Fischer et al., 2015; Hilton et al., 2013; Pawlyshyn, Braddlee, Casper, & Miller, 2013), computer networks (Gil, Candeles, Jara, Garcia, & Torres, 2013), statistics (Bowen, Chingos, Lack, & Nygren, 2012), physics (Hendricks, Reinsberg, & Rieger, 2017), and chemistry (Allen, Guzman-Alvarez, Molinaro, & Larsen, 2015).
In psychology, Hilton and Laman (2012) found that 690 students using an online, open psychology textbook had higher course grades, final exam grades, and higher retention rates than students who did not use an open text in the previous semester. Fischer et al. (2015) compared performance across 15 courses, four of which were psychology, in a multi-institutional study. They measured course completion, passing grades (C or better), and course grades across courses that used OER and ones that used a traditional textbook. Of these measured outcomes in the psychology courses, students who used an open text in one course (Psych 100) had a higher course grade than students who used a traditional text. The other results did not reveal any other group differences.
Although these studies have illustrated the impact OER have on students, to our knowledge none have controlled for instructor, instructional strategies, semester, room, course, and time of day in a single study. We controlled these factors within the present study in the hope of better understanding the impact of OER on its users.
Student Perceptions of Open Materials
Research on student perceptions of open compared to traditional learning materials reveal similar results–students largely view OER similarly, or at times more positively, than a traditional textbook (e.g., Bliss et al., 2013a; Bliss et al., 2013b; Delimont, Turtle, Bennett, Adhikari, & Lindshield, 2016; Feldstein et al., 2012; Gil et al., 2013; Hendricks et al., 2017; Hilton et al., 2013; Illowsky, Hilton, Whiting, & Ackerman, 2016; Ozdemir & Hendricks, 2017; Watson, Domizi, & Clouser, 2017). Findings regarding students' positive perceptions of OER have also been found among students taking online classes (Lindshield & Adhikari, 2013).
As most OER are online, including those used in the current study, it is important to also consider students' perceptions of online materials as compared to print. The majority of previous research in this area suggests that students prefer print over online materials (e.g., McGowan, Stephens, & West, 2009; Shepperd, Grace, & Koch, 2008; Terpend, Gattiker, & Lowe, 2014; Vernon, 2006). Despite this preference, research suggests using online materials does not translate into lower course performance (Rockinson-Szapkiw, Courduff, Carter, & Bennett, 2013; Shepperd et al., 2008; Siebenbruner, 2011; Terpend et al., 2014).
Student Use of Open Materials
In terms of students using and accessing open materials, findings have been mixed. Bliss et al. (2013a; 2013b) found no differences between students' use of open textbooks and traditional textbooks. Conversely, Lindshield and Adhikari (2013) found that students in online and face-to-face courses reported using their open text more often than traditional texts. Likewise, Allen et al. (2015) found that students spent more time using the free online chemistry site, ChemWiki, than traditional learning materials. In that study, Allen and colleagues acknowledged that the reason for increased use of OER is unclear. Indeed, there are many factors that influence student use of materials, and although being “open” may seemingly increase student use because of free, open access to the materials, existing research is not clear whether this is the case. The current study does not necessarily address the core of the question of how OER might increase use, but it does further explore the use of an open textbook as compared to a traditional textbook.
Current Study
Research to-date suggests that open materials yield similar course outcomes and are generally deemed to possess positive characteristics when compared to traditional course materials. Several of these studies, however, did not closely control several factors, including semester, course instructor, and time of day, which may have influenced the outcome of the studies (Griggs & Jackson, 2017). The present study attempts to minimize these factors by examining outcome variables across two course sections taught by the same instructor, using the same instructional strategies, in a single semester, in the same room, at the same time of day, using identical in-class materials, syllabi and sequencing, on alternating days. Our specific research questions were as follows: RQ1: Are there differences in student performance between students who use an open textbook and those who use a traditional textbook? RQ2: What are student perceptions of an open textbook compared to a traditional textbook? RQ3: Are there differences in student use of an open textbook compared to a traditional textbook?
Method
Participants
Sixty students from two introductory psychology course sections at a small, public university in the southeastern United States participated in this study (n[open text users] = 31; n[traditional text users] = 29). This included 23 males and 37 females, with an average age of 19.34 years (SD = 1.84 years) across the two course sections. All students provided informed consent for their completion of the perceptions survey, and IRB approval was acquired.
We did not conduct any pre-test to measure students' pre-existing knowledge about psychology. Students were largely from first-year except for a sophomore (n = 1), a junior (n = 1), a senior (n = 1), and two students who did not report their class standing. Students represented more than ten majors from across campus.
Materials
Textbooks
Students in the open-textbook course section used the Introduction to Psychology: Version 1.0 (Stangor, 2010) textbook. This textbook was selected because it was the only open introductory-level psychology textbook available when textbooks were selected for the course. Students could download the textbook for free in a .pdf format from the Saylor Foundation website (http://saylor.org). A link to the textbook was also provided in the course management platform, Desire2Learn. The instructor did not make any modifications to the textbook.
Students in the traditional publisher's textbook course section used Psychology in Everyday Life, 3rd edition (Myers & DeWall, 2014). This text was selected because the instructor had used this book in previous semesters, and it was used by other faculty in the department. This book retailed for approximately US$100 at the school bookstore. Students in both course sections were required to have their respective textbook.
Survey
We administered a 22-item survey to measure students' perceptions (including preferences, perceived advantages, and characteristics) and self-reported use of the open and traditional textbook. The survey was modified for each course section, so that students answered questions regarding the material they used. To clarify the text in the open-textbook group, that survey used the phrase “free, online textbook,” whereas the phrase “traditional paper textbook” was used in the traditional publisher's textbook survey. Except for Questions 20 and 21, which were about the perceived advantages of each type of text, and Question 22 about textbook preference, students' ratings were only for the textbook they were using.
Questions 1–4 pertained to students' ideas about the textbook they were using (e.g., “I would like to have a free online textbook to use in this class in addition to the traditional paper textbook”) and were scored on a 7-point Likert scale (1- strongly disagree; 7- strongly agree). Questions 5–17 pertained to students' perceptions of the textbook quality (“Rate the overall level of quality of the textbook you used in this course”) and were also scored on a 7-point Likert scale (1- very poor quality; 7-very good quality). Questions 18 and 19 asked students about their use of the textbook (“Compared to your experience with textbooks in other courses, how often do you use the textbook in this course?” and “How frequently do you use the textbook in this course?”). Both were scored on a 7-point Likert scale (respectively, 1- much less often, 7- much more often; and 1- never, 7- more than three times a week). Question 20 asked students “What do you see as the advantage(s) of a free online textbook? (check all that apply).” Question 21 asked students “What do you see as the advantage(s) of a traditional paper textbook? (check all that apply).” The advantages listed, for both Questions 20 and 21, were “cost,” “ease of reading,” “weight,” “convenience,” “ability to highlight and take notes,” “ability to quickly find a topic,” “can keep it as a reference book for future use,” and “other (please specify).” Students could select more than one advantage for each type of text. For Question 22 (“If given the choice between a traditional paper text and a free, online text, which would you choose?”), students were asked to choose between either a “free, online text” or “traditional paper text.”
Except for Question 22, which was developed by the researchers, all other questions were adapted from textbook scales used in previous research regarding textbook quality, use, and satisfaction (Gurung & Landrum, 2012; Lindshield & Adhikari, 2013; McGowan et al., 2009), including the Textbook Assessment and Usage Scale (TAUS; Gurung & Martin, 2011).
Procedure
Textbook implementation
Textbook implementation and data collection were carried out in spring 2015. The two course sections were randomly assigned an open or traditional publisher's textbook. As open resources had not been adopted within the department before, this was an opportunity for faculty in the department to better understand the implementation process, the strengths and weaknesses of using an open textbook, and to contribute to the extant body of literature on open educational resources in higher education.
Students using the open textbook could access the textbook online at no cost. The instructor guided the students using the open textbook through the steps of accessing the textbook. Students could access the textbook from a hyperlink in the course syllabus and through the course learning management system.
Students using the traditional textbook were allowed to purchase, rent, or use a free-copy that was made available at the school library (for two-hour intervals). The same instructor (one of the authors) taught both course sections in the same room, and each course was offered at the same time but on alternating days of the week during the same semester. Thus, instructor, location, time, and semester were controlled across groups. Course sequencing and instructional strategies were held constant across each course section, with the instructor using the same slide presentations, lecture notes, and class activities in both sections.
Student course performance
To measure student course performance, we analyzed five exam scores, using identical exams in each course, as well as final course grades. Course grades comprised the five exams (worth 76% of the course grade), a final paper (worth 19%), and a research participation requirement (worth 5%). Exam questions were created by the instructor and drawn from the test-bank of the traditional text. All questions were textbook agnostic, meaning they only covered information that was covered in each textbook. These included definitions of terms that were covered in each text and everyday examples that were not covered specifically in either text. Exams were not cumulative and were given on consecutive days.
Student perceptions and use
We administered surveys to students in each course section at the end of the semester to capture students' perceptions of open and traditional textbooks. The surveys were printed, stapled, and given to students to complete by hand during class. The surveys took approximately 20 minutes to complete. Students earned a small amount of extra credit (5 points, or approximately 1% of the total possible course points) for participating. Overall participant response rate was 90.91%, with group response rates as 87.88% for the traditional textbook section (n = 29) and 93.94% (n = 31) for the open textbook section.
Results
Student Performance
Our first research question was whether there are differences in student performance between those who use an open textbook and those who use a traditional textbook. We first compared the total number of exam points across the two groups. The same five exams were given to students in both courses. Each exam was worth 80 points, so students could earn up to 400 exam points.
Open-textbook users earned an average of 273.26 total exam points (SD = 79.37) compared to 295.91 total exam points (SD = 67.68) among traditional textbook users. An independent group t-test revealed that this difference was not statistically significant, t(64) = −1.329, p = .188.
Student Performance on Five Exams.
Note. Mean scores were out of a possible 80 points.
Last, we compared final grades, which included the five exams (worth 76% of the final grade), a paper (worth 19%), and a research requirement (worth 5%). Traditional textbook users' final grades were M = 73.72; SD = 18, and open textbook users' final grades were M = 70.82; SD = 21. These differences were not statistically significant, t(62) = −.611, p = .544.
Student Perceptions of Textbooks
Our second research question was “what are student perceptions of an open textbook compared to a traditional textbook?” To address this question, we first examined students' ideas about the textbook (Questions 1–4), and perceptions of the characteristics of the textbook they were using (Questions 5–17). Next, we asked about students' textbook use (Questions 18–19). Third, students listed advantages of both open and traditional textbooks (Questions 20 and 21). The last question (Question 22) asked students which type of text they preferred most (open or traditional). Results for each of these questions are listed below.
Students using the open textbook were more likely to like the idea of their text (Question 1), t(58) = −3.39, p = .001, d = .868, 95% CI [−1.62, −.42], prefer using their text (Question 2), t(58) = −2.48, p = .016, d = .634, 95% CI [−2.18, −.23], and like the idea of (not) buying their textbook (Question 3), t(58) = −5.22, p < .000, d = 1.336, 95% CI [−3.06, −1.36]. For Question 4, students in traditional textbook group were more likely to want a “free, online” textbook in addition to the traditional textbook they were using compared to students in the open textbook group, who were less likely to want a traditional textbook in addition to the text they were using, t(58) = 2.50, p = .015, d = .649, 95% CI [.25, 2.28]. Finally, students in the traditional textbook group were more likely to rate their textbook as visually appealing (Question 15) than students in the open textbook group, t(58) = 3.93, p < .000, d = 1.025, 95% CI [.58, 1.77].
Figure 1 illustrates students' perceived advantages of an open and traditional textbook. Students in both courses (n[traditional textbook users] = 29; n[open textbook users] = 31) perceived cost, weight, and convenience as the biggest advantages of an open text. Students perceived ease of reading, convenience, ability to highlight and take notes, ability to quickly find a topic, and ability to keep as a reference the biggest advantages of traditional textbooks.
Students' perceived advantages of open and traditional textbooks.
Textbook Use
Perceptions of Traditional Textbooks among Traditional Textbook Users (n = 29) and Open Textbooks among Open Textbook Users (n = 31).
Note. All responses were scored on 7-point Likert scales with higher values representing stronger responses (e.g., strongly agree, very helpful, very often).
Question 19 asked, “How frequently do you use the textbook in this course?” It was scored on a seven-point scale (1- never; 7- more than three times a week). Traditional textbook users scored a mean of 3.31 (SD = 1.58), and open textbook users scored a mean of 3.16 (SD = 1.34). These differences were not significant, t(64) = 0.42, p = .680, 95% CI [−.91, 61] (Table 2). Essentially, it appears that regardless of what textbook condition the students were in, it did not influence their reports of textbook use.
Discussion
Overall, our study provided a much-needed empirical look at student perceptions, self-reported use, and performance when using an open psychology textbook compared to a traditional, publisher's textbook through a quasi-experimental approach. In our study, we controlled for instructor, instructional strategies, semester, room, time of day, syllabi, sequencing of in-class content, and course to maximize internal validity. This was the first study to our knowledge that took all of these factors into consideration within a single study.
Our findings revealed no significant differences between groups in terms of performance and self-reported textbook use. This finding is consistent with previous research, which demonstrates similar student performance, regardless of the modality of the text (e.g., Allen et al., 2015; Bowen et al., 2012; Chiorescu, 2017; Clinton, 2018; Feldstein et al., 2012; Fischer et al., 2015; Gil et al., 2013; Grewe & Davis, 2017; Hendricks et al., 2017; Hilton & Laman, 2012; Hilton et al., 2013; Pawlyshyn et al., 2013). Despite the instructional strategies being tailored more to the traditional than the open textbook, performance across the two sections did not differ as the instructor had used the traditional textbook in previous semesters. This illustrates that even at a potential instructional disadvantage, students using the open textbook did as well as those using the traditional textbook.
Open textbook users also tended to like the idea of an open textbook more than traditional textbook users liked the idea of a traditional textbook. Traditional textbook users also wanted a “free, online textbook” in addition to the traditional paper one they were using. These findings were in line with prior research that suggests students tend to prefer open textbooks or at least view them similarly in relation to traditional textbooks (Bliss et al., 2013a; Bliss et al., 2013b; Delimont et al., 2016; Feldstein et al., 2012; Gil et al., 2013; Hendricks et al., 2017; Hilton et al., 2013; Illowsky et al., 2016; Ozdemir & Hendricks, 2017; Watson et al., 2017).
Previous research suggests that students prefer printed materials over online materials (e.g., McGowan et al., 2009; Shepperd et al., 2008; Terpend et al., 2014; Vernon, 2006); however, our findings differ. It is not clear why this may be the case. One possibility is the wording of our questions. Another possibility is students' prior experiences with each type of textbook. Both possibilities are further discussed in the Limitations section.
When asked to make a choice about their preference between an open and a traditional paper textbook, students in both conditions selected the text they were using. Students' preference for whichever text they had, online or paper, may be explained by behavioral economics, a tendency for people to like what they already possess (Ariely, 2008). Despite the admitted advantages of the free, open online text, students who had purchased the paper text experience ownership and consequently, value in what they possessed. Ariely (2008) refers to this as the “Ikea effect.” Thus, actual value and perceived value can be quite different.
In terms of advantages, students from both groups perceived ease of reading, convenience, ability to highlight and take notes, ability to quickly find a topic, and ability to keep as a reference as the biggest advantages of traditional textbooks, and cost, weight, and convenience as the biggest advantages of an open text. These findings are similar to previous research on open online resources as compared to traditional print ones (e.g., Hilton, 2016).
Limitations
There were several limitations in the current study. First, the study was quasi-experimental, so it is not possible to draw causal conclusions; however, both course sections were taught by the same instructor, using the same instructional strategies, in the same room, at the same time of day, using identical in-class materials, syllabi and sequencing, on alternating days of the week, which minimized the effect that these factors may have had on the dependent measures. Next, our sample size was limited (N = 60), which affected the power of our statistical analyses, and results only included one semester, which limits the scope of the findings. We also did not collect demographic data on race or ethnicity, nor did we collect prior test scores (e.g., ACT, SAT), which could have served as covariates in our statistical analyses. We recommend future studies collect this information if possible.
Question wording was also an issue. The wording on the survey did not include the terms “open textbook,” and instead emphasized that the text was “free” and “online.” This may have affected the way students responded to the questions. Further, Question 3 read, “I like the idea of buying a free textbook.” This likely caused confusion since a free textbook cannot be bought.
In terms of our measures, we modified questions that had been used previously in research studies examining student perceptions and use of textbooks (Gurung & Landrum, 2012; Lindshield & Adhikari, 2013; McGowan et al., 2009), including the Textbook Assessment and Usage Scale (TAUS; Gurung & Martin, 2011). Our survey questions did not undergo control for validity and reliability, so it is possible that our measure was lacking in these areas. Future studies will need to carefully modify the language used in each question and use questions for which validity and reliability standards have been met.
Additionally, our comparison of open online text versus a traditional paper text could indicate that rather than comparing open versus traditional textbooks, it was more of a comparison between online and paper textbooks. Prior research shows that students respond differently to online compared to paper texts (e.g., McGowan et al., 2009; Shepperd et al., 2008; Terpend et al., 2014; Vernon, 2006). Furthermore, students in the paper text condition may not have any experience with open or online texts, which could have skewed their perceptions about these texts. In future studies, researchers should ask students about their prior experience with open or online textbooks.
Finally, although we randomly assigned conditions to each class, we did not measure students' prior knowledge about psychology. Therefore, we do not know if one class had more existing knowledge than the other about the topic. This is a potentially large flaw in the design of the study; however, it should be noted that this course was an introductory level psychology course, and most students were taking it for the first time. If any students had prior knowledge of the subject before taking the course, we believe this would have a minimal effect on our findings. In the future, researchers should measure pre-existing knowledge through a pre-test in order to eliminate the possibility of this knowledge having an effect on group differences.
Conclusions
Prior research on open psychology textbooks have offered valuable insights into the effect these resources have on student performance, perceptions, and textbook use. It is important that this research continue, while more carefully minimizing possible confounds through more closely controlled research studies (Griggs & Jackson, 2017). Open textbooks and other OER have the potential to make learning more accessible to students by providing a free, downloadable alternative to traditional publishers' texts in a variety of disciplinary areas, including psychology. Our findings, although with limitations, have implications for students, educators, and administrators. By using OER, students can not only save on textbook costs, but also have less restricted access to course materials; educators can use the materials in ways that best benefit their own course needs and teaching practices; and because of the decreased financial burden students face, administrators in higher education may find OER appealing because of the possible increase in retention, progression, and graduation rates. At this point, the body of empirical evidence is growing, and we call on others to continue addressing these questions through their own research. By doing so, the body of literature on OER will continue to grow and improve, helping to further inform best teaching practices.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.