Abstract
The National Institute for Occupational Safety and Health (NIOSH) and several university programs have collaborated on a large effort to expand and improve occupational safety and health content in Wikipedia using a platform developed by Wiki Education. This article describes the initiative, student contributions, and evaluations of this effort by instructors from two universities between 2016 and 2020. The Wiki Education platform allowed instructors to set timelines and track students’ progress throughout the semester while students accessed training to best expand health content in Wikipedia. Students chose topics in occupational health based on their interests and by a set of topics deemed as a priority by the “WikiProject Occupational Safety and Health.” Students’ contributions were peer-reviewed by instructors, NIOSH Wikipedians-in-Residence, and traditional Wikipedians. Students presented their projects in class at the end of the semester. Students from both schools expanded 55 articles, created 8 new articles, and translated 2 articles to Spanish, adding 1270 references; these articles were viewed over 8 million times by May 2020. Feedback received from the implementation suggested that students learned about science communication and digital literacy—providing valuable content on occupational health while reducing misinformation in the public domain. The process of identifying and addressing gaps in occupational health in Wikipedia requires participation and engagement toward improving access to information that otherwise would be restricted to the scientific literature, often behind a paywall. The Wikipedia assignment proved to be an engaging approach for instruction and information literacy. It helped students improve their science communication skills and digital literacy, tools that are likely to be critical for successful communication of science in their future careers.
Keywords
Summary
The National Institute for Occupational Safety and Health (NIOSH) collaborates with Wiki Education and academic programs in expanding and improving occupational safety and health content. In this paper we describe Wikipedia-based writing assignments in occupational health graduate courses. The Wikipedia assignments were effective in teaching students about occupational health, scientific writing, and science communication.
Introduction
Since 2012, NIOSH collaborated with several organizations within the Wikimedia movement (Wikipedia, Wikidata, Wikimedia Commons, Wikiversity, and Wiki Education, a spin-off of the Wikimedia Foundation) for the goal of expanding and improving occupational safety and health content on their platforms.1-5 NIOSH is one of the first U.S. federal agencies to contribute data, content, images, and videos of the latest research to these platforms.6,7 The breadth and accuracy of Wikipedia science and public health coverage vary widely. 8 Medical content has consistently improved as a result of an effort by subject-matter experts who contribute to Wikipedia. 9 The NIOSH goals are not only to freely share information, aligned with Wikipedia’s goals, but also to translate research into usable information. NIOSH’s digital analytics indicated that an average of ten thousand individuals visit the NIOSH website within a typical month because of redirection from a Wikipedia page, based on The Digital Download monthly report of NIOSH’s web presence.
While there are numerous avenues for researchers and academics in occupational safety and health to publish professionally, there are legitimate reasons to expand the scientific content in Wikipedia.10-12 In medicine, for example, Wikipedia has been praised to expand the reach of health information13,14 and has been successfully used in teaching assignments.15,16 Wikipedia-based writing assignments have been suggested to be a pedagogy appropriate for graduate education as it provides a disciplinary praxis and teacher training while providing opportunities for authentic writing. 17 In traditional course assignments, students spend time writing papers that only the instructors have access to. The rationale for writing into Wikipedia includes (1) providing a public service by contributing solid, verifiable knowledge about occupational health to the general public, (2) eliminating misinformation, and (3) collecting outcome metrics reporting on the reach of the dissemination efforts. In addition, it constitutes a unique opportunity for experts to rapidly and collaboratively write on an extra-organizational, international effort. Since 2010, Wiki Education has been improving Wikipedia content through collaborations with colleges and universities. 18 Instructors and students get technical support for the use of a dashboard platform they can use to assign their students a Wikipedia project as part of their coursework and provide training resources and track their progress. 19 By integrating the development of Wikipedia content into the course curriculum, students review course-related topics that are missing or underrepresented in Wikipedia, synthesize the available literature and use the Wiki Education’s free tools and trainings to add relevant information to Wikipedia. 20 Using this approach, students also improve their science communication skills and digital literacy, tools that will equip them for their future careers.11,21
In 2015, NIOSH reached out to the University of Cincinnati to encourage the adoption of the Wiki Education Dashboard tool in the classroom to increase the availability of occupational safety and health content and to tap into students’ enthusiasm to make real-world contributions. Students in a graduate-level course in occupational epidemiology were tasked with adding content to Wikipedia. Based on this experience, NIOSH partnered with the Harvard T.H. Chan School of Public Health for the Introduction to the Work Environment graduate-level course in 2017 and 2018. The outcomes reported in conference presentations and in a NIOSH Science Blog,1-7 as well as the value proposition of contributing to a free knowledge activity, attracted the interest of instructors from other disciplines who also joined the partnership from different universities. The objectives of this article are to (1) provide detailed information about the process of creating a teaching assignment in Wikipedia for others to follow in occupational health, (2) share article contributions resulting from using the Wiki Education platform as a teaching tool in graduate courses that were focused specifically on occupational health, and (3) summarize feedback from students and instructors.
Methods
We involved the active participation of the students and their instructors in a process of expanded review that is Wikipedia’s public review. This assignment consisted of graduate students becoming co-authors in the collective production of knowledge regarding their subject matter expertise and putting into practice the new knowledge they acquired on how to edit an encyclopedia.
Guideline for Instructors to Set Up a Wikipedia Assignment in Their Course
Before the course started, instructors from both universities worked with the NIOSH team to create a step-by-step Wikipedia document for instructors. The aim of this document (Supplemental Information Document 1) is to assist any instructor in occupational health in setting a Wiki Education Dashboard for their class with a customized timetable. This document was also designed to anticipate key milestones and provide resources during the course, such as helping students identify project topics, and suggestions to facilitate grading including an example rubric and recommendations from WikiEdu. 21 Instructors can check a student’s Wikipedia sandbox (i.e., content before publishing) or page history so that is easy to know what part of the student’s work was accepted or not in Wikipedia and be able to grade the students for their effort. Alternatively, students can submit a literature review in a Word format to the instructor, before uploading it to Wikipedia, to facilitate grading.
Guideline for Students for the Final Wiki Project
Instructors developed a document to guide students on creating a login in the Wiki Education Dashboard, choosing a project topic, and providing a class timetable with clear deliverables throughout the semester. The guideline for students included advice developed by Wiki Education for instructors but tailored for the particular class. For example, best Wikipedia topics for a student project are often those that are (1) missing or underrepresented in Wikipedia (i.e., easy to add content without being redundant), (2) well studied (i.e., easy to find references), and (3) non-controversial (i.e., to avoid being edited out).
The Dashboard also allows instructors to track students’ progress throughout the semester. Students were able to access weekly training modules, discussions, exercises, and milestones that teach them to develop Wikipedia content in incremental steps and complete their assignments by the end of the semester. The Dashboard also provided students access to Wiki Education resources in a variety of topics including guides for writing articles using medical and scientific communication approaches, how to navigate Wikipedia tools, and avoiding plagiarism. A guideline for students’ example from the 2018 Harvard Chan course is found in Supplemental Information Document 2.
A Semester-Long Writing Wiki Project
During the first day of class, instructors introduced the final project and discussed the relevance of sharing technical content in Wikipedia and how to choose a good topic for the assignment. Instructors invited the NIOSH team to make a presentation at different times during the semester to provide further guidance to students about adding occupational health content to Wikipedia and to answer questions. Students at Harvard Chan were tasked to create a health and safety section for an existing Wikipedia page, while students at the University of Cincinnati created or updated a health and safety section in an existing topic, developed new articles, or translated articles into a foreign language.
Detailed information on the courses (timeline, number of students, topics they worked on and what their contributions were, and number of views) are available in the following Wiki Education dashboards: • The University of Cincinnati students worked independently: • The Harvard Chan students in the 2017 class worked in groups: https://dashboard.wikiedu.org/courses/Harvard_University/EH_262_Introduction_to_the_Work_Environment_(Fall_2017) • The Harvard Chan students in the 2018 class workedindependently: https://dashboard.wikiedu.org/courses/Harvard_University/Harvard_Chan_EH262_Fall_2018_-_Intro_to_the_Work_Environment_(Fall_2018)
Topics were chosen by students based on their interest and assisted by a set of topics Wikipedians deemed as priorities in the “WikiProject Occupational Safety and Health.” 22 Students were also encouraged to contribute to pages with a lower quality score, which is a score provided by independent reviewers to assess a set of principles described by Wikipedia in terms of what it means to be a good or feature article. The topic, outline, and content developed by students were revised by instructors before content was entered in Wikipedia using the Wiki Education platform. NIOSH researchers and Wikipedians-in-Residence offered instructors and students technical and topical expertise support throughout the course. Students were able to get technical assistance related to the platform from the Wiki Education staff as well. Students had the opportunity to obtain peer-reviews from the instructors, teaching assistants, and the NIOSH Wikipedian-in-Residence, and public review from Wikipedia editors who offered immediate feedback. Students presented their final Wikipedia pages to the class at the end of the semester.
Evaluation of the Final Wiki Project Experience
To evaluate the Wikipedia assignment, we used feedback provided by instructors and NIOSH Wikipedians-in-Residence as well as students’ comments to instructors and the evaluations of the courses. During the final presentations, students shared their opinions with the class regarding the project experience, their challenges, and rewards. Also, the instructors provided a 20-question in-class survey to the students at the Harvard Chan 2017 course. The other courses just collected feedback with other evaluation efforts during and after the semester.
Results and Discussion
Occupational Epidemiology Course at the University of Cincinnati in 2016, 2018, and 2020
The occupational epidemiology course at the University of Cincinnati is a graduate-level course that focused on the concepts of epidemiologic methods as they are applied in occupational health. The course included a mixture of lectures, directed readings, and classroom exercises that cover epidemiologic study designs, issues of validity, measurement of exposure, ascertainment of health outcomes, approaches to analysis, and special considerations for studying the health of a working population. Case studies on a variety of occupational hazards were used to illustrate these issues. The Wikipedia assignment brought the information they were studying to one of the most widely read websites in the world. The Wikipedia article and final presentation counted toward 35%–40% of the final grade.
Wikipedia Graduate Student Projects in Occupational Health.
Totals registered at the end of the semester; note that the tracking period was not the same for each article, which does not allow for a precise comparison on the interest from readers on a certain topic. Topics that were not related to the class subject matter were not included.
*Views are from the course dashboard accessed in May 2020.
There were 10 graduate students in the 2016 course. Five of the students created new Wikipedia pages, such as the health effects of coal ash. One student translated the page on occupational asthma into Spanish, not only improving the content from the one in English but also providing links and references to other Wikipedia pages in Spanish. At the end of the semester, the quality score provided by Wikipedia of the occupational asthma article in Spanish was higher than that of corresponding articles in English and French. In total, the pages edited were viewed 5.31 million times during the time tracked by Wiki Education (for approximately up to a month following the class’s conclusion). As Spanish is the second most spoken language in the world, and several occupational groups at risk for occupational asthma comprise workers who often only speak Spanish, these contributions can be particularly impactful in terms of outreach.
The 2018 course included 11 graduate students. Twenty-one existing Wikipedia articles were improved. These included one on occupational health concerns of cannabis use based on one student’s medical practice experience and updates to the page on NIOSH’s National Occupational Research Agenda. During this semester, students added a total of 179 references, and their pages had 4 million views by the end of the semester. The course used the same platform in 2020, and progress made by the seven enrolled students can be tracked on the course website.
Introduction to the Work Environment Course at Harvard Chan in 2017 and 2018
Both in 2017 and 2018, the overall objective of this graduate-level course was to develop an understanding of the anticipation, recognition, evaluation, and control of workplace hazards. The course includes a mixture of lectures, directed readings, and classroom exercises and case studies to understand workplace hazards in the context of the fundamental principles of environmental health and safety and to learn to read critical industrial hygiene literature.
The 2017 class had 11 students divided into three Wikipedia topics. Each of the three groups included students from one of the different programs they were registered in. Six of the students were physicians in the Master of Public Health (MPH) program, of which four were doing a residency in the Occupational & Environmental Medicine Residency (OEMR) program. The other students were doctoral or master’s students. The 2018 class had seven students participating in individual Wikipedia projects. Five students were physicians in the MPH program or the OEMR program.
Table 1 lists the contributions of Harvard Chan students in the years 2017 and 2018. Information on a wide range of occupational health topics was covered in the two semesters. In 2017, while working in groups, 11 students edited three Wikipedia articles, contributed three media files to Wikimedia Commons, the Wikipedia media repository, and added 92 references. In 2018, seven students edited over nine Wikipedia articles and added 467 references and 11 media files. Collectively, these additions to Wikipedia have been viewed over 822,000 times by May 2020.
Topic selection was often inspired by personal and professional experiences. For example, several of the students who worked on the microgravity (micro-g) environment Wikipedia article were from the US Air Force (USAF) and inspired by experiences collaborating with the US National Aeronautics and Space Administration (NASA). Another physician student updated the hydrazine Wikipedia page, inspired by his experience in the USAF.
Wikipedia’s expeditious review process was evident by the prompt feedback offered to some of the students’ contributions. For example, an occupational medicine student added a new health and safety section to the Wikipedia article on casinos, but the Wikipedia public peer review edited his extensive work down to two sentences within a day. He then received guidance to create a completely new article focused on the hazards and best practices for casino employees—even modeling for photos to illustrate the concepts (photos that have now been removed by Wikipedia editors). Another occupational medicine student, inspired by his experience as a resident on an oncology ward, added occupational health content to the Wikipedia page on chemotherapy. A student from China was moved by the electronic waste issues in her home country and updated the electronic waste page in the English Wikipedia, which had been created by a student who took the 2016 University of Cincinnati course.
Students’ Evaluation of the Final Wiki Project Experience
Student Responses from Harvard Chan 2017 Course Regarding the Wiki Project Experience (n = 10).
From talking with students during and after the semesters about their experiences and based on the feedback shared during the final presentations, we learned that many of the students were being challenged for the first time with the process of scientific writing. The prompt Wiki Education support and useful training resources helped students understand key principles of scientific writing, such as avoiding plagiarism, selecting and citing sources, and using neutral language. They also learned about the challenges of choosing unbiased supporting evidence from the health literature. Students were appreciative of the support provided by the NIOSH Wikipedians, who offered personalized classes and follow-up, provided potential article ideas, edited their articles, and infused overall motivation for their efforts. Students who needed assistance with selecting topics for their projects found the information available on the “WikiProject Occupational Safety and Health” page to be helpful. Students expressed enthusiasm toward their project topics and about showcasing their science communication skills.
Other Feedback From the Final Wiki Project Experience
Evaluation of Wikipedia Student Projects at two Universities’ Occupational Health Graduate Courses, 2016–2020.
The NIOSH team also noticed how some students in both schools went beyond what was required by the assignment, in a few cases, developing new Wikipedia articles. Another example is from a student who not only added content to the article on chloroprene but also uploaded the HAZCOM pictogram to Wikimedia Commons. This allowed the pictogram to be incorporated into the article and available for anyone to use with attribution.
Efforts from the 2018 class were highlighted in a Harvard Chan news article.23,24 The Wikipedia efforts with students were highlighted by the Dean of Dr. Harvard Chan Dean Michelle A. Williams, who said, “Dr. Diana Ceballos teaches a class called Introduction to the Work Environment, and her students aren’t just learning about occupational hazards—they’re spreading their knowledge in the most effective way possible: Wikipedia,” during her convocation address in 2019. 24 Larger efforts related to the use of Wikipedia at NIOSH in collaboration with partners were recognized by the Bullard–Sherwood r2p Awards—Knowledge category, Reaching Our Audience Where They Are: NIOSH work with Wikipedia. 25
Takeaways From Feedback Received From the In-Class Final Wiki Projects in Occupational Health
Students improved both scientific and plain English writing
The online training provided by Wiki Education on citations was relevant to students regarding writing for both an expert audience and the general public. The training helped students understand the value of including appropriate references for scientific writing while acknowledging the challenges of identifying unbiased supporting evidence from the health literature. Students often had to re-examine the accuracy and flow of the contributed scientific content. Several students were alerted by Wikipedia editors who found their contributions to be lengthy, complicated, or with poor spelling or grammar. This required students to write more clearly and succinctly, a necessary skill for health communication.
Students got familiar with the peer review process
Publishing in Wikipedia involves review and feedback by other contributors, a process which has been referred to as “public review.” 26 Some of the students’ contributions were initially erased from Wikipedia because they lacked references but were later reinstated once appropriate references were added. The much-expanded public review process provided by the Wikipedia community was at times challenging to the students but ultimately appreciated for being dynamic and transparent. Stylistic and technical feedback was also provided by the instructors and teaching assistants to help students improve their articles within the timeline of the class. In this joint effort, instructors, along with the Wikipedia peer review process, helped improve articles and contributions both technically and structurally. Furthermore, instructors benefited from the external feedback being provided to students during the semester because it legitimized the instructor’s guidance—as getting similar feedback from different sources makes a better argument to the students of the need to address a weakness in their writing.
Wikipedia as a science translation teaching tool
Students valued the opportunity to write knowing that their contributions were reaching the general public and contributing to occupational safety and health literacy. They also appreciated choosing a topic for their Wikipedia final class project. Students learned about the process and the value of translating science to the public. The goal of learning a topic well enough to be able to write in a style accessible to lay readers is challenging and requires more research than a traditional assignment. Finally, the articles the students worked on were viewed more than 8 million times among the different courses, which was rewarding for both instructors and students. Vetter 17 argued that “Wikipedia-based education allows for direct and transparent observation of practices and concepts related to the writing process, research, social collaboration, and digital rhetoric while also providing opportunities for authentic writing situations.” Vetter 17 also suggested that Wikipedia “empowers a more transparent and collaborative peer production towards knowledge equity.” Our experiences support these claims. The process of identifying and addressing gaps in a selected discipline in Wikipedia explicitly requires participation and engagement toward improving access to information that otherwise would be restricted to the scientific literature, often behind a paywall. 27
Wikipedia metrics useful for grading and tracking public health impact
The Wiki Education Dashboard offers detailed metrics which facilitate the assessment of student learning. One can access which Wikipedia articles were edited and by whom and review the extent and the text of the edit, including the number of words and references they added, and if the students’ contributions remain in Wikipedia in time. One can also determine contributions from students vs those from others that engaged to further develop the page. Furthermore, the number of views since the time of the student’s contributions is a powerful tool to understand the impact of the topic and content reach to the masses—and even if a specific student’s contribution was to be deleted in time, it contributed to the development of a better page overall and the effort is not lost.
The systems used to evaluate scholarly impact today recognize the importance of a publication appearing in Wikipedia. Many journals and agencies, like the Centers for Disease Control and Prevention, adopted bibliometric platforms to complement traditional citation metrics in the evaluation of research impact. Altmetric is one of the platforms that measure the influence and online reach of scholarly output through readership counts; mentions by news sites and blog posts; and citation manager bookmarks such as Mendeley, social media, and Wikipedia citations. 28 Contributing information to Wikipedia boosts not only the dissemination of knowledge in public health 29 but also its scholarly recognition. This effort of incorporating Wikipedia assignments can be and has been tailored and replicated in other classes of a variety of subject matters.
Conclusion
The Wikipedia-based writing assignments and the use of the Wiki Education platform and tools were effective in teaching students about occupational health, scientific writing, health communication, and science. The resulting Wikipedia articles enhance online learning tools and do not only represent great examples of participatory education and the future of education but also help NIOSH realize its mission to disseminate occupational health knowledge. Articles filled public health content gaps and widely disseminated scientific knowledge on Wikipedia.
Future work should involve utilizing WikiEdu platform metrics in more detail to do an in-depth analysis across different classes, and to do tracking of the contributed student content to understand if it was kept in time in the Wikipedia articles. Beyond a student-focused study, an in-depth evaluation is also needed to understand the impact of these contributions in occupational safety and health knowledge translation and comparison to other online network methods such as those studied by Rhebergen et al.30-32
The joint efforts of university programs with Wiki Education and NIOSH greatly expanded Wikipedia’s content in occupational health through contributions from students and instructors. We are encouraged that other programs teaching students in occupational health around the country are trying this approach. 3
Supplemental Material
sj-pdf-1-inq-10.1177_00469580211035735 – Supplemental Material for Expanding Reach of Occupational Health Knowledge: Contributing Subject-Matter Expertise to Wikipedia as a Class Assignment
Supplemental Material, sj-pdf-1-inq-10.1177_00469580211035735 for Expanding Reach of Occupational Health Knowledge: Contributing Subject-Matter Expertise to Wikipedia as a Class Assignment by Diana M. Ceballos, Robert F. Herrick, Tania Carreón, Vy T. Nguyen, MyDzung T. Chu, John P. Sadowski, Helaine Blumenthal and Thais C. Morata in INQUIRY: The Journal of Health Care Organization, Provision, and Financing
Footnotes
Acknowledgments
The authors wish to thank the students for their contributions to Wikipedia, the Wikipedians, and Ian Ramjohn from Wiki Education for their assistance. We also thank Chuck Kardous and Albeliz Santiago-Colón, from the National Institute for Occupational Safety and Health, Dr. Stuart Batterman from the University of Michigan, and Laura Jenemann, from Boston University for their helpful critiques of initial versions of this manuscript. The National Institute for Occupational Safety and Health Educational Research Centers (ERC) at the University of Cincinnati and Harvard Chan supported many of the students in the classes described in this publication.
Author Contributions
DC, TC, and TM designed the work, developed class guidelines, and wrote the manuscript. RH, VN, MC, JS, and HB assisted in the implementation of the Wikipedia assignments described in this manuscript and reviewed the manuscript (i.e., RH was a co-instructor, VN and MC were teaching assistants, and JS and HB were external support). All authors approved of this manuscript.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article. The authors do not declare any funding source, with the exception of salaries for the faculty and support for open access publication fees by the National Institute for Occupational Safety and Health. The NIOSH Educational Research Centers (ERC) at Harvard Chan and the University of Cincinnati supported many of the students in the classes described in this publication for their graduate studies.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
