Sage Journals: Discover world-class research

Abstract

Sonography programs continuously explore and evaluate methods to improve quality. The purpose of this mixed-methods study was to investigate the use of open-book examinations (OBEs) in an accredited sonography program. Participants included sonography students enrolled in an academic health sciences center who used an OBE format in two ultrasound physics courses. OBE mean examination scores were compared to students who used a closed-book examination (CBE) format. No statistically significant difference was seen in mean examination scores between the OBE and CBE cohorts with the exception of Spring Examination 1 (P = .0026). All OBE students (n = 17) participated in focus groups. Qualitative analysis showed examination preparation and text usage changed over time. Seventy percent of OBE students reported reduced examination tension and stress. OBEs do not have a significant effect on examination grades, and examination anxiety and stress are reported to be reduced. The OBE cohort noted a preference of OBEs over CBEs.

Keywords

research educational educational measurement open-book examinations education ultrasonography medical education undergraduate

An educator’s goal is to develop students who, at a minimum, are entry-level competent in the cognitive, psychomotor, and affective learning domains. One method for evaluation of cognitive competence is through assessment. Traditional, closed-book assessment is the most common assessment method.¹ Closed-book examinations (CBEs) do not allow the learner to use educational resources during the examination. This may encourage study habits of “cramming” the night before the examination and “data dumping” the day of the examination, with little long-term recall of knowledge. The memorization of facts and the ability to recall these facts is a key focus in CBEs.^2–6 In contrast, open-book examinations (OBEs) allow the learner to use educational resources such as books and notes during the examination. There are two basic types of OBEs: the restricted type, which allows the learner to bring to the examination only “instructor-approved” texts and study material, and the unrestricted or free type, which allows the learner to bring in any type of resource to the examination.⁷

Consistent themes have emerged in the literature related to OBEs. Data from student surveys and focus groups find that most students prefer OBEs to CBEs.^2,3,5,8–10 Students approach open-book testing with greater optimism; they work out their answers in a more relaxed way and feel more confident about their overall performance.⁹

OBEs improve study habits and examination preparation.¹⁰ Theophilides and Koutselini⁹ found that students who are aware the course assessment method is OBE are more attentive throughout the course, use study activities that promote a deeper learning of the material, and become more involved in their learning process. Learning and examination preparation are enhanced by the process of students organizing their notes for the examination and rewriting the information in a format that makes the most sense to them.^3–5,8,11 Students must be familiar with the textbook structure, which will enable them to search for answers quickly.^12,13 Divya⁷ and Broyles et al.⁵ note that students must study differently for OBEs, understanding that not all information can be looked up, and they must have a solid understanding of the overall concepts. Although students may spend more time preparing for the OBE, information retention is not always improved over closed-book testing.^8,12,13

OBEs may reduce cramming and rote memorization the night before the examination, although some students may initially study less for the OBE.^3–9 Feller⁶ advocated the use of OBEs, which correlate more closely to real-life situations and provide an opportunity to use resources to answer questions or develop solutions. He argued that in the CBE environment, examination preparation and studying is open book, but the examination is closed book, which promotes memorization and lack of reasoning.⁶ Theophilides and Koutselini⁹ relate this to students relying less on memorization and more on applying analytical thinking and synthesizing of information during the examination process.

Most studies report examination scores that do not improve in the OBE environment.^3,8,10,14,15 This could be for a number of reasons. First, OBE questions may be written at a higher level than CBE questions, allowing for assessment of Bloom’s Taxonomy skills of analyzing, synthesizing, and evaluating course material.³ Questions may also be written so that direct answers will not be found in the notes or the text. In addition, lower scores may result from students underestimating the difficulty of the examination and trying to “wing it” by looking up the answers during the examination. This strategy will be unsuccessful for examinations that have higher-order questions or that are timed. In examinations where students could look up a specific fact, scores were still lower, leaning to lack of preparation.⁸

One of the biggest benefits linked to OBE is the reduction in students’ stress and test anxiety.^4–6,9,10 Stress has been shown to negatively affect higher-order thinking. Dale et al.⁴ note that a more relaxed examination environment can lead to an improvement in higher-order brain function and allow the students to take a deeper approach to their learning. OBEs create a more relaxed study environment with less pressure to memorize every fact. The examination environment may also be viewed as more relaxed or safe when students are allowed to fall back on or access their learning resources.⁹

The purpose of this study was to explore the effect of implementing OBEs into sonography education by using the testing method in two ultrasound physics courses. The following research questions were explored in this OBE cohort:

Does the OBE environment result in improved examination scores compared to a CBE environment?

How do students prepare for an open-book format of a multiple-choice examination?

How do students use the text during an OBE?

Are tension and stress reduced by the knowledge of an open-book setting?

How do students feel about the process of learning and assessment with open-book testing?

Materials and Methods

The open-book subject population consisted of 17 students enrolled in a diagnostic medical sonography program at a Midwestern academic health sciences center over a period of two years (n = 8 in academic year [AY] 2016–2017 and n = 9 in AY 2017–2018). Examination scores for these students were compared to students in AY 2015–2016 (n = 10) who tested in the traditional, closed-book format. All participants had completed 21 hours of college prerequisites and an accredited radiography program, and they were certified in radiography by the American Registry of Radiologic Technology (ARRT). All had completed at least one postsecondary physics course, either a general college physics course with lab or a radiography physics course. The sonography program culminated in either a Bachelor of Science (BS) in medical imaging and therapeutic sciences or a postbaccalaureate certificate (PBC) in sonography depending on the individual’s undergraduate background and radiography program. Breakdown by year for educational background included one BS student and nine PBC students in 2015–2016, one BS student and seven PBC students in 2016–2017, and two BS students and seven PBC students in 2017–2018.

All students admitted to the program (baccalaureate or postbaccalaureate level) followed the same curriculum. All courses in the 12-month curriculum were cross-listed at the undergraduate and graduate levels. The OBE study was implemented in the Ultrasound Physics I course offered in the fall semester and the Ultrasound Physics II course offered in the spring semester. Each course was one semester credit hour, met one hour per week each semester, and was taught by an experienced sonography faculty member. The required textbook and course format remained consistent from AY 2015–2016 through AY 2017–2018. Each physics course was divided into three units with an examination at the end of each unit. The fall semester course ended in a comprehensive final examination over all of the fall semester content. The spring semester course ended in a comprehensive final examination over both fall and spring semester material. Examinations were proctored and administered using an offline, secure testing platform called Examplify. All questions were multiple choice with a total examination time based on one minute/question (60 questions = 60 minutes). Only grammatical or spelling errors changes were made in examination questions from AY 2015–2016 to AY 2017–2018. During program orientation, all students received the same introductory training to the Examplify testing platform. The OBE cohorts, 2016–2017 and 2017–2018, also received a tip sheet with recommendations for open-book testing (Figure 1)5. The OBE cohorts were allowed to use only their physics textbook during each examination, although they were allowed to add notes in the book margins and tabs denoting important content that they may want to access during the examination.

Figure 1.

Best practices in preparing for an open-book examination (OBE).

Descriptive statistics, including means and standard deviations for examination results and counts and percentages for student characteristics, were used to summarize the examination score data. The independent sample t test was used to compare grade point averages (GPAs) and mean test scores by testing type. A P value of <.05 was considered statistically significant. All analysis was done using SAS Version 9.4 software (SAS Institute, Cary, North Carolina).

Qualitative data were collected from the OBE cohorts only. Three focus group sessions were held: one after the first fall examination, one after the fall comprehensive final examination, and one after the spring comprehensive final examination. Focus group questions were developed based on the current literature, and each focus group lasted approximately 30 minutes. OBE question topics included examination preparation, text usage during the examination, examination stress levels, and opinion of the OBE format. Focus group sessions were conducted by an imaging sciences faculty member outside the sonography program. Sessions were audiotaped, transcribed, and reviewed following Creswell’s procedure¹⁶ for data analysis and representation. This involved the organization of the data, a preliminary read through, coding and organizing of themes, data representation, and finally interpretation.¹⁶ This study was approved by the institutional review board as exempt.

Results

The two testing groups were comparable in demographics and GPA information. All students in the CBE group were female. The OBE group included 16 females and 1 male. The average age of the OBE group (24.9 years; SD = 0.053) was slightly older than the CBE group (23.1 years; SD = 1.0). No statistically significant difference was found in prerequisite GPA (CBE = 3.5, OBE = 3.6,P = .23) or radiography GPA (CBE = 3.8, OBE = 3.76,P = .55) between the two groups. Seven out of eight course examination grades showed no statistically significant difference between the testing groups with the exception of Spring Examination 1 with a mean examination average of 87.0 (SD 6.5) for the CBE group compared to 77.7 (SD 9.4) for the OBE group (P = .0026) (Figure 2).

Figure 2.

Descriptive statistics comparing the open-book examination and closed-book examination cohorts. GPA, grade point average.

Data collected from the OBE focus groups showed student examination preparation time and method varied over the two semesters (Figure 3). For the first fall physics examination, the majority of students (59%, n = 10/17) began examination preparation one week prior to the examination. Limited textbook use was reported, with the majority of students only skimming the text and using most of their time to review lecture notes and PowerPoints. They used the same examination preparation for CBE. Two students admitted to underpreparing for the first examination due to the open-book format. At the end of the fall semester, the majority of students (65%, n = 11/17) admitted to using the cramming method for examination preparation, but 50% of the students (n = 9/17) altered their examination preparation methods during the course of the semester. For these students, more time was dedicated to modifying the textbook for use during the examination. In-depth notes and drawings were added to text margins to clarify content. Color-coded tabs were used to organize concepts and to allow rapid location of material during the examination. Students focused on learning the general ideas and layout of the text versus memorization of facts. End of spring semester results showed a move away from the cramming method (35%, n = 6/17) reported at the end of fall semester and a shift to the week prior preparation time (59%, n = 10/17). Only one student (6%) consistently reported studying and editing the text immediately after the material was presented in class. Students listed the open-book format, other course priorities, outside time constraints, and procrastination as factors affecting the time they allotted for examination preparation. Of note, almost all studying was done independently versus in a group setting.

Figure 3.

Data demonstrating how examination preparation time fluctuated over the two semesters for the open-book examination (OBE) cohort.

When describing how they used the text during the actual examination, students were given three options: option A, text was opened at the beginning of the examination and used for all questions; option B, text was used to look up unknown questions when encountered during the examination; and option C, the entire examination was completed without opening the text and flagged questions were looked up at the end of the examination. The majority of students (53%, n = 9/17) selected option B when asked this question. Over the duration of the two semesters, this percentage increased slightly (Figure 4). Students felt that this was the most efficient option, which is important in a timed examination. Option A and option C were also reported in all three OBE focus group sessions but at a much lower rate. Those preferring option A looked up every answer on the examination until time became a factor and then switched to option B or C. Confirmation of answers early in the examination improved their confidence and offered peace of mind. Option C students viewed the examination as if it was a CBE, completing the examination without the use of the text. Students using this option preferred a CBE format and challenged themselves not to depend on the text, which would not be available during the Sonographic Principles and Instrumentation (SPI) credentialing examination.

Figure 4.

Data depicting how open-book examination students used the text during the examination. Option A: text opened at the beginning of the examination and used for all questions. Option B: text used to look up unknown questions when encountered during the examination. Option C: entire examination completed without opening the text and flagged questions looked up at the end of the examination.

Drawing on past testing experience, almost 60% (n = 10/17) of the OBE students reported reduced anxiety and stress at the first examination compared to examination in other courses. This improved over time, with 70% (n = 12/17) of students reporting reduced examination anxiety and stress over the course of the two semesters. Going into an examination, the students tied the stress reduction to knowing they had a “fall back” in case they went blank on material. Two unique stressors arose during the examination related to the timed OBE format. Stress stemmed from the need to look up every answer but the lack of time to do so and the amount of time that should be allotted to look up an answer before moving on.

Sonography students who were part of the OBE cohort consistently expressed a strong preference for the open-book format, with over 75% (n = 13/17) of the students stating this method of testing was preferred over CBE. Students felt they were able to focus on the big picture of information and were able to gain a higher level of thinking versus simply memorizing facts, possibly allowing them to retain the information longer. Although students expressed a preference for the OBE format, many stated they would not prefer it for other sonography courses linked to patient care and the clinical environment. Only two to three students (12%–18%) in each focus group session reported they favored CBE.

Discussion

This study was unique in that it is the first study to report on the use of OBE in a sonography program. The quantitative findings support the literature that the OBE format does not result in higher examination scores compared to those testing in the CBE setting.^3,8,10,14,15 Seven of the eight examinations showed no statistically significant difference between the groups with the exception of Spring Examination 1. This decrease in the mean examination score could be related to the OBE cohort’s overconfidence and failure to prepare for the first examination of the spring semester or to a greater number of high-level questions on this particular examination that required analysis and reasoning and could not be found directly in the text. Although not statistically significant (P = .16), the OBE mean examination score was higher than the CBE mean (92.5 vs. 89.70) on Spring Examination 2. This examination evaluated the students’ knowledge of color Doppler. The higher score may be attributed to a greater number of questions that could be referenced directly from the book or more exposure to color Doppler at their clinical sites.

In this study, students were allowed to bring only their physics book to the examinations. It was the instructor’s goal to promote a student’s personalization of his or her textbook with notes, diagrams, and other resources. Although students were provided with best practices for OBE during course orientation, examination preparation had limited text usage and mimicked practices used for CBE courses on the first examination. Examination preparation did evolve after Fall Examination 1 with a turn to personalization of the text and less memorization of facts, a theme that was seen in other OBE studies.^3–5,8,11 Examination preparation time fluctuated over the two semesters and may have been influenced by external factors such as the number of other course examinations during finals week and the upcoming SPI examination. Findings were similar to other studies with students being overconfident for the first examination, not preparing enough, and underestimating the difficulty of the examination.^3–9 Instructors who move to this testing format should educate students on how to prepare for an OBE, stressing that it will require at least the same amount of preparation as a CBE.⁴

All examinations in this study were timed to prepare students for upcoming credentialing examinations. This time limit did influence the student’s use of the textbook. Students realized during the first examination that looking up every question was not possible, and it was essential to have a good understanding of the textbook layout. This most likely was one factor that changed their text use for examination preparation after Fall Examination 1. All students testing in the OBE format did use the entire allotted examination time, either to complete the examination or to go back and verify answers with the text.

Reduction of examination stress and anxiety is seen in OBEs,^{4–6, 9,10} and this was a consistent finding in this study. The reassurance of a textbook in the examination setting helped relax the students. Some OBE students did report a different type of stress linked directly to the timed OBE format, a finding not reported in other OBE literature. This did improve over the course of the two semesters and may be associated with becoming more comfortable with the OBE format.

Study findings also support the literature in that students prefer an OBE environment to that of a CBE environment.^2,3,5,8–10 In this study, students referenced themes of improved organization, reduced memorization, and reduced fear of forgetting key concepts as benefits associated with the OBE format.

Although the research questions did not address the effect of OBEs on SPI pass rates, it should be noted that all students, CBE and OBE, had a 100% first-time pass rate on the American Registry for Diagnostic Medical Sonography’s SPI examination. All students from both testing formats took the SPI examination within 14 days of completing their spring final physics examination. This program has consistent 100% first-time pass rates for the SPI examination and cannot attribute the use of OBEs to improved registry pass rates as reported in the literature.^5,8 In the final focus group session held after the SPI credentialing examination, some OBE students remarked that this testing format promoted course information organization, which improved the efficiency of the registry preparation process. This benefit has not been previously reported in the literature.

Limitations

This study has limitations that are associated with the research design. In addition, sonography programs tend to be smaller, and the number of students enrolled in the program limits the overall sample size. The study was also limited to one institution and one imaging program, and this adversely affects the generalizability of these results. There are future plans to add additional sonography courses and other medical imaging programs to the OBE study. At least one student in each examination had technology issues with the examination software. This may have affected examination grades and stress levels. For future studies, it would be valuable to include CBE cohorts in the focus group sessions. There is also opportunity to explore the effect of higher-level examination questions on OBE scores.

Conclusion

Overall, results of this study may support the continued use of OBEs for ultrasound physics courses. Statistically significant differences were not found between the mean examination averages for the OBE format compared to the CBE format except for one examination. After the first examination, OBE students did modify how they used the textbook during examination preparation and during the examination. Students using the OBE format reported a reduction in examination tension and stress compared to other classes using a CBE format. Most students testing in the OBE format preferred it over the CBE format for ultrasound physics.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Kim Michael

References

Gharib

Phillips

Mathew

: Cheat sheet or open-book? A comparison of the effects of exam types on performance, retention, and anxiety. Psychol Res 2012;2(8):469–478.

Williams

Wong

: The efficacy of final examinations: a comparative study of closed-book, invigilated exams and open-book, open-web exams. Br J Educ Technol 2009;40(2):227–236.

Brightwell

Daniel

Stewart

: Evaluation: is an open book examination easier? Biosci Educ 2004;3(1):1–10.

Dale

VHM

Weland

Pirkelbauer

Nevel

: Value and benefits of open-book examinations as assessment for deep learning in a post-graduate animal health course. J Vet Med Educ 2009;36(4):403–410.

Broyles

Cyr

Korsen

: Open book tests: assessment of academic learning in clerkships. Med Teacher 2005;27(5):456–462.

Feller

: Open-book testing and education for the future. Stud Educ Eval 1994;20:235–238.

Divya

: Open book vs closed book exams. https://www.eoneducation.com/open-book-vs-closed-book-exams/. Accessed June 25, 2018.

Conan

: Cheat sheet or open book: putting tests to the test [radio broadcast episode]. December 13, 2012. https://www.npr.org/2012/12/13/167180405/open-or-closed-book-exams-which-style-works-best. Accessed June 25, 2018.

Theophilides

Koutselini

: Study behavior in the closed-book and the open-book examination: a comparative analysis. Educ Res Eval 2000;6(4):379–393.

10.

Loi

Chuah

: The impact of open book examinations on student learning. http://www.hkta1934.org.hk/NewHorizon/abstract/1999/page34.pdf. Accessed June 25, 2018.

11.

Pelech

: Comparing the effectiveness of closed-notes quizzes with open-notes quizzes: blending constructivist principles with action research to improve student learning. Inquiry Educ 2016;8(1):8–18.

12.

Agarwal

: Test expectancy and transfer of knowledge with open-book and closed-book tests. https://openscholarship.wustl.edu/cgi/viewcontent.cgi?article=1492&context=etd. Accessed June 25, 2018.

13.

Agarwal

Karpicke

Kang

SHK

, et al: Examining the testing effect with open- and closed-book tests. Appl Cognit Psychol 2018;22:861–876.

14.

Cade

Riou

Richard

Passirani

Letertre

Fayolle

: Comparison of open and closed book test for admission in medical school. https://www.mededpublish.org/manuscripts/1416/v1. Accessed June 25, 2018.

15.

Johanns

Dinkens

Moore

: A systematic review comparing open-book and closed-book examinations: Evaluating effects on development of critical thinking skills. Nurse Educ Pract 2017;27:89–94.

16.

Creswell

: Qualitative Inquiry & Research Design: Choosing Among Five Approaches. 3rd ed. Los Angeles, CA, Sage, 2013.

Open-Book Examinations (OBEs) in an Ultrasound Physics Course: A Good Idea or a Bad Experiment?