Abstract
Nurses are subject to considerable stress, arising from the physical, psychological, and social aspects of their work. Widely used nursing occupational stress measures were developed decades ago and may not adequately capture contemporary sources of stress. The study purpose was to evaluate the content validity of items from existing measures and newly developed items to identify a nursing stress scale that can be used to measure occupational stress in current practice. A three-phase approach for development and evaluation of occupational stress items was conducted. Phase 1: item generation; Phase 2: (
Keywords
Certain occupations are significantly more stressful than others and nursing is widely regarded as one such profession (Moustaka & Constantinidis, 2010; Roberts & Grubb, 2014; Safi Keykaleh et al., 2018). Stress contributes to numerous detrimental outcomes for both individual nurses (Mayo Clinic, 2021) and healthcare organizations (Johnson et al., 2018), highlighting the ongoing need to conduct stress related research. However, widely used occupational stress measures are decades old, limiting nurse researcher’s ability to conduct accurate and reliable research, and further hindering the advancement of stress management and nurse wellness.
Stress and Stress Measurement in Nursing
Nurses are subject to considerable stress, which occurs as a result of the physical, psychological, and social stressors inherent in their work (Moustaka & Constantinidis, 2010; Roberts & Grubb, 2014). Hospital based nurses are particularly vulnerable to occupational stressors due to their work environment, shift schedules, and working conditions (Safi Keykaleh et al., 2018). Specifically, workload, staffing ratios, exposure to infectious diseases, high levels of responsibility, a real or perceived lack of adequate preparation for the nursing role, and communication within the healthcare team contribute to occupational stress (Moustaka & Constantinidis, 2010; Safi Keykaleh et al., 2018; Sarafis et al., 2016). Additionally, many nurses may experience other occupational stressors that are unique to their hospital unit, patient population, or healthcare setting.
Stress is an ongoing occupational and health hazard within the nursing profession that results in detrimental consequences for individuals and healthcare organizations. On an individual level, stress can result in numerous negative health outcomes that can manifest as physical and or psychological symptoms, and may be acute or chronic in nature. These serious negative health outcomes impact the health of the nurse and can result in negative organizational outcomes (Johnson et al., 2018). Within healthcare organizations, stress can result in higher rates of turnover, absenteeism, burnout, errors, inefficiencies, dissatisfaction, and poor quality of healthcare delivery (Johnson et al., 2018; Sarafis et al., 2016).
In order to effectively study occupational stress in contemporary nursing, valid and reliable up to date measures are needed. In addition to the importance of accurate measurement for research, measurement is also foundational to efforts to address healthcare workforce safety. The National Academies of Sciences, Engineering, and Medicine (2019) published a report on clinician burnout, which identifies routine measurement of occupational stress and burnout, using validated measurement tools, as a key component in monitoring and improving clinician well-being. Additionally, the Institute for Healthcare Improvement (2021) identifies measurement as a foundational component for improving workplace safety.
In studying and exploring occupational stress in nursing, the Nursing Stress Scale (NSS) is the best known and most widely used measurement tool among nurse researchers (French et al., 2000). A review of the literature indicates extensive use of the NSS over the last 40 years (Bautista et al., 2020; Bené & Foxall, 1991; Healy & McKay, 1999; Purcell et al., 2011). The NSS was developed to measure the frequency and major sources of stress experienced by hospital-based nurses (Gray-Toft & Anderson, 1981). The NSS is a 34-item questionnaire that provides a total stress score as well as scores for seven subscales, with higher scores representing increased frequency of perceived stress (Gray-Toft & Anderson, 1981). Studies have demonstrated reliability of the NSS with Cronbach alpha values reported at .82 (Hayes et al., 2015) and .89 with a test-retest reliability coefficient of .81 (Abuatiq & Borchardt, 2021). Validity of the NSS has been demonstrated, including content validity ratios >0.8 (Ezenwaji et al., 2019; Movahedi et al., 2020). While studies have indicated that the NSS is reliable and valid, it is unclear whether the NSS remains a comprehensive and accurate measure of occupational stress in current nursing practice.
In 2000, a study was conducted to review the NSS and develop an Expanded Nursing Stress Scale (ENSS) to measure the sources and frequency of stress perceived by nurses (French et al., 2000). The authors concluded that the NSS needed revision to accurately measure occupational stress among nurses, including the identification of 25 new sources of stress, two new subscales, modification of two NSS items, movement of items within subscales, and removal of two NSS items (French et al., 2000). The ENSS is a 57-item questionnaire with 9 subscales (French et al., 2000). Reliability of the ENSS has been demonstrated with alpha coefficients ≥.95 and construct validity has been established using confirmatory factor analysis (Alkrisat & Alatrash, 2017; French et al., 2000; Sarafis et al., 2016). While reliability and validity of the ENSS have been reported, the measure should be reassessed for use in current practice (French et al., 2000).
Nursing practice has changed substantially over the past 20 years, which presents new potential sources of occupational stress. Most notably, technology has transformed nursing practice and healthcare delivery, nursing has integrated the use of the electronic health record into daily patient care, new nursing specialties have been introduced, and some nursing roles have been entirely redesigned (Lucena et al., 2021; National Academies of Sciences, Engineering, & Medicine, 2021; The Journal of mHealth, 2019). While the aforementioned changes in nursing practice are known, and these changes have been shown to contribute to stress in nursing, it is not clear if these occupational stressors are adequately captured in existing stress measures, specifically the NSS and ENSS, that were developed decades ago.
Purpose
The purpose of this study was to identify a comprehensive and accurate occupational stress measure that can be used in current nursing practice. To achieve this, the content validity of items from the NSS, the ENSS, and newly developed items reflecting contemporary stressors identified in the literature and from content experts was evaluated. While there are multiple forms of validity, determining content validity is identified as a prerequisite for other forms of validity and should receive the highest priority during instrument development (Zamanzadeh et al., 2015). According to the process for establishing instrument validity by Polit and Beck (2006), determining the content validity of a measure is a necessary first step in order to draw conclusions about the scale’s quality. Furthermore, determining content validity can improve the instrument by providing information on the representativeness and clarity of items through recommendations from a panel of experts (Zamanzadeh et al., 2015).
Methods
Design
A three-phase approach for development and evaluation of occupational stress items was conducted. Phase 1 was conducted to generate and identify potential new items that represent occupational stressors in current practice that were not included, or adequately addressed, in the existing stress measures. Phase 2 was conducted to determine the item content validity index (I-CVI) of each item in the NSS, the ENSS, and proposed new items, by evaluating the clarity, relevance, and importance of items as they relate to the concept of occupational stress. Phase 3 was conducted to provide a second round of expert review on select items. The study was approved by the local Institutional Review Board.
Phase 1—Literature Review and Item Generation
There is agreement in the methodologic literature that content validity is largely a matter of judgment that involves two distinct phases: a priori efforts to enhance content validity through careful conceptualization and item analysis, and a posteriori efforts to evaluate the scale’s content through assessment by content experts (Polit & Beck, 2006). Prior to the assessment of content validity of items, the literature was reviewed to identify contemporary occupational stressors in nursing literature that may not be adequately represented in the existing nursing stress measures (NSS and ENSS). Three databases (CINAHL Plus, PubMed, and PsycInfo) were searched from 2000 to 2020 with the following terms: hospital-based nurses, nurses, stress, occupational stress, workplace stress, work-related stress, job stress, technology, electronic health record, nursing roles, compensation, and transition to practice. The lead author reviewed the literature to identify contemporary occupational stressor themes which were used to develop new items to be included in Phase 2.
Phase 2—Content Validity
Among nurse researchers, the content validity index is the most widely reported measure of content validity (Polit & Beck, 2006). More specifically, the I-CVI represents the proportion of agreement among expert reviewers regarding specific factors that contribute to content validity, and can be used to guide researchers in revising, removing, or substituting items in a measurement scale (Polit & Beck, 2006; Zamanzadeh et al., 2015). In this study, I-CVI values were calculated for three factors: clarity, relevance, and importance.
Sample
Eleven Registered Nurses were recruited via convenience and snowball sampling to participate and serve as content experts. Eligibility criteria included: (1) holding licensure as a Registered Nurse; (2) working at a hospital in the United States; (3) providing direct patient care in their current role; and (4) working in a setting that provides direct patient care 24 hours per day, 7 days per week. Various practice areas were recruited to encourage diverse representation of nurses within hospitals.
Instruments
Survey items consisted of demographics, organizational characteristics, and occupational stress items. Stress items included 34 items from the NSS, 25 items from the ENSS, and 10 new items that were identified from the literature, for a total of 69 items. Each item was rated using a 4-point Likert scale to assess the clarity: not clear (1), somewhat clear (2), quite clear (3), very clear (4); relevance: not relevant (1), somewhat relevant (2), quite relevant (3), very relevant (4); and importance: not important (1), somewhat important (2), quite important (3), very important (4). Additionally, three open-ended questions were included at the end of the survey, including: (1) if any items should be re-worded, (2) if there were additional items or situations that were not listed that contribute to occupational stress, and (3) if there were any additional questions, comments, or thoughts that they wanted to provide.
Procedure
The survey was sent to participants via a link to a self-administered online Qualtrics (2020) survey. Consent was obtained electronically at the beginning of the survey, prior to data collection. A 10-dollar Amazon gift card incentive was electronically sent to participants after completion of the study. Data collection took place between March 2020 and May 2020.
Analyses
The focus of Phase 2 analysis was on discovering which items to retain, revise, or remove, and soliciting input from content experts to determine if aspects of the construct are adequately represented or whether additional items are needed (Polit et al., 2007). An I-CVI was calculated for each factor (clarity, relevance, and importance) for each item, by dichotomizing the ordinal scale into positive ratings (scores of 3 &4) and negative ratings (scores of 1 & 2), and then dividing the total number of positive ratings by the total number of experts (
Items were retained as written: Guideline #1: If all three factors had I-CVI values ≥0.78. Guideline #2: If the item was both clear and relevant (I-CVIs ≥ 0.78) and had an importance I-CVI ≥ 0.73. Guideline #3: If the item was both clear and important (I-CVIs ≥ 0.78) and had a relevance I-CVI ≥ 0.73. Decisions for Guideline #2 and Guideline #3 were made as the item was clear, met the recommended ≥0.78 value for relevance or importance, and 8 of 11 experts positively scored the item in the other factor (relevance or importance), demonstrating support of the item.
Guideline #4: Items were removed if they were determined to be clear (I-CVI ≥ 0.78), but neither important nor relevant (both “importance” and “relevance” I-CVIs ≤ 0.64).
Items were identified as needing additional review by experts and potential revision: Guideline #5: If the “clarity” I-CVI was <0.78. Items cannot be adequately assessed for importance and relevance if they were not clearly understood by the participant. Guideline #6: Items that were clear (I-CVI of ≥0.78), had either an “importance” or “relevance” I-CVI value ≥0.73, and the other factor I-CVI value ≤0.55, were identified as needing further review as these values indicate some disagreement among content experts, but at least moderate support for item importance or relevance.
Phase 3—Second Review of Select Items
Sample
In Phase 3, a subsample of
Instruments
An electronic survey was sent to participants via email that included 25 items needing additional review based on Phase 2 results. Participants were asked to respond to two survey questions to solicit feedback on the clarity, relevance, and importance of these items. For clarity, participants were asked to respond to the prompt, “Please make any comments or suggestions for improving the clarity of the following items. Please be as specific as possible.” For relevance and importance, participants were asked to respond to the prompt, “When thinking about each item, please comment on why you think the item is or is not relevant and/or important in relation to workplace stress in nursing. Please be as specific as possible.”
Procedure
Data collection took place between June and July 2020. Participants were informed that they could provide responses to the survey electronically or they could verbally discuss their responses with the lead author via a phone call. All four participants provided written responses and one participant provided additional verbal responses via a phone call. The lead author compiled and reviewed all responses provided by content experts.
Analyses
The focus of Phase 3 analysis was to solicit more in-depth and detailed feedback from a subsample of content experts to clarify and understand discrepancies in the I-CVI ratings from Phase 2. Expert feedback was used to guide revision, removal, or retention of the 25 items under review. The lead author carefully reviewed all expert responses and compared comments to determine if there was discordance or consensus among experts. When consensus was identified, items were removed or retained as appropriate. Suggestions to revise items were reviewed by the lead author and one content expert to determine revisions to select items.
Results
Phase 1—Item Generation
Five themes of occupational stressors were identified in the literature. Identified themes were then used to develop new items to be included in the evaluation of content validity by content experts. The first two themes related to the integration of the electronic health record and other technologies into nursing practice. Specifically, nurses experience occupational stress, exhaustion, and burnout, related to the adoption and use of the electronic health record due to inadequate training, the cumbersome and time-consuming nature of charting, increased cognitive workload, and the multiple steps and time needed to find and input data (Gephart et al., 2015; Schenk et al., 2016; Yen et al., 2019). Nursing skill sets have needed to evolve in response to emerging technologies to successfully implement new technology into practice (Lucena et al., 2021; Schenk et al., 2016).
In addition to the impact of technology, stressors related to workload, compensation, and expectations also emerged as themes in the literature. Changes in workload were identified as the third theme contributing to occupational stress. An increase in technology, additional job demands and responsibilities, and managing the number of roles that nurses feel they need to fill can often lead to an unreasonable workload, which results in an increase in occupational stress (Moustaka & Constantinidis, 2010; Salmond & Echevarria, 2017). With the increased workload and additional occupational stressors that nurses reported, nurses believe that they are not adequately compensated for the work that they do, which presented as the fourth theme in the literature (Satoh et al., 2017). This dissonance between perceived workload and inadequate pay further contributes to nurses’ stress and dissatisfaction. Lastly, the fifth theme was identified as the misalignment between what new graduates experience as they enter the profession of nursing, compared to what they expected the profession of nursing to be (Moustaka & Constantinidis, 2010).
The occupational stress themes that were identified in the literature were compared to the existing NSS and ENSS items to determine if they were adequately represented in the existing measures. The lead author determined that the identified themes were not clearly reflected or adequately represented in the existing items in the NSS and ENSS, therefore, 10 new items were generated to capture these occupational stressors (Table 1).
I-CVI Values & Decision Guidelines for Each Survey Item (
The lead author also carefully reviewed the wording of each item in the NSS and ENSS to ensure that the language used is in alignment with current practice. Upon review, both instruments consistently used the term “physician” or “doctor” in numerous items; however, many healthcare organizations have other members of the healthcare team leading and directing patient care. The term “provider” is more inclusive as it recognizes other individuals that may be identified as the leader of the healthcare team. Therefore, items that used the term “physician” or “doctor” were revised to use the term “provider” to better reflect the varied structure of healthcare teams in current practice.
Phase 2—Content Validity
Eleven content experts had the following characteristics: mean age was 32.8 years (min-max, 23–55 years old), 100% were female, 10 were White and 1 identified their race as “other,” highest degrees were; Associate Degree in Nursing (1), Bachelor of Science in Nursing (9), Master’s of Science in Nursing (1), years of experience ranged from 1.5 year to 27 years with a mean of 9.5 years. Content experts were employed by various hospital systems in one US region. Various practice areas included Medical-Surgical, Float, Oncology, Labor & Delivery/Maternal, Pediatrics, and the Emergency Department. Day, evening/PM, and night shifts were represented in the sample and 8- and 12-hour shifts were also represented. Five content experts reported working 21 to 30 hours per week, five reported working 36 to 40 hours per week, and one content expert worked per diem. Various hospital types were included: critical access, non-profit, academic, governmental, rural, and investor-owned for profit.
The I-CVI values were calculated for each of the 69 survey items for all three factors (clarity, relevance, and importance), which resulted in a total of 207 I-CVI values as displayed in Table 1. A single expert response was missing for six factors; to account for the missingness, the denominator for these six factors was adjusted to 10.
Of the total 69 survey items, 34 items were retained as written as all I-CVI values were ≥0.78. Thirty-five items had an I-CVI < 0.78 for at least one factor. The pre-established guidelines were then applied to these 35 items to determine if items should be retained, removed, or revised (Table 1). Of these 35 items, 5 items (2 items from the NSS, 1 item from the ENSS, and 2 items from the literature) met criteria for Guideline #4 and were removed. Nine items (7 from the NSS, 1 from the ENSS, and 1 new item) met criteria for Guideline #2 and Guideline #3 and were retained. Fifteen items met criteria for Guideline #5 and 5 items met criteria for Guideline #6, resulting in 20 items needing further expert review.
One exception to the established guidelines was made as one item that met criteria for removal was instead included for further expert review as it was deemed important in the literature and by anecdotal reference. This item was from the NSS, “Difficulty in working with a particular nurse (or nurses) on the unit.” In total, 21 items required a second round of expert review. Open-ended questions and free text responses in the Phase 2 survey also generated 4 new survey items reflecting occupational stressors that participants felt were not addressed in the reviewed items. These expert-generated items included, NEW11: “The risk of infectious work exposures that may impact myself, my family, or my loved ones,” NEW12: “Working in a healthcare system where I am not valued as an individual,” NEW13: “The use of patient satisfaction rating scales being applied to an extremely complex enterprise which minimizes the complexity of the actual job itself,” and NEW14: “Lack of adequate support and debriefings for high stress situations (i.e., suicide, death of a child, trauma, etc.)..” With these items, there were 25 items that were included in a second round of expert review.
Phase 3—Second Review of Select Items
Participants had the following characteristics: mean age was 31 years (min-max, 24–37 years old), 100% were female, 3 were White and 1 identified as “other,” 3 had their Bachelor of Science in Nursing and 1 had their Master’s of Science in Nursing, years of experience ranged from 1.5 years to 12 years with a mean of 6.7 years. Content experts were employed by various hospital systems in one US region. Practice areas included Medical-Surgical, Oncology, Pediatrics, and the Emergency Department. Day, evening/PM, and night shifts were represented and 8- and 12-hour shifts were also represented in the sample. Various hospital types were included to ensure adequate representation: critical access, non-profit, academic, and investor-owned for profit.
Comparison of feedback from all four experts resulted in a decision to retain 21 items and remove 2 items (from the ENSS). Two items (1 from NSS and 1 new proposed item) underwent minor revision by content experts and the lead author to improve clarity and reflect language used in current practice (Table 2).
Summary of Items Retained, Removed, and Revised.
The review and data analysis process of both study phases is depicted in Figure 1. In total, of the 73 items (comprised of 69 original items from Phase 2 and 4 new items generated by content experts in Phase 2) that were reviewed in this study, 64 items were retained without revision, 7 items were removed, and 2 items were revised and retained (Table 2). Following this review for content validity, a total of 66 items were retained to form a Revised Nursing Stress Scale.
Results based on two phases of analysis (Phase 2:
Discussion
The purpose of this study was to identify a comprehensive and accurate occupational stress measure that can be used in current nursing practice. Twelve new items were identified and recommended for inclusion in future occupational stress measures and five items from existing stress measures were deemed to no longer be relevant in current nursing practice. In total, review of all items resulted in a set of 66 items to form a Revised Nursing Stress Scale.
Nine of the 12 new items aligned with the following themes: the use of technology, workload, and not feeling valued. These findings were not surprising as recent advances in technology have significantly changed nursing practice and contributed to additional job demands and responsibilities, which consequently leads to higher levels of stress (Lucena et al., 2021; Moustaka & Constantinidis, 2010; Yen et al., 2019). Studies have also reported an association between nurses not feeling valued, lower levels of satisfaction, and higher levels of occupational stress (Prasad et al., 2021; Waltz et al., 2020). While the item, “Nursing not being the job that I thought it would be” initially did not meet established I-CVI guidelines for retention, after a second round of expert review, it met criteria and was retained. This item may have been rated lower in Phase 2 as this occupational stressor is closely linked with transition to practice and the mean years of experience for participants was 9.5 years. As a result, participants may not have scored this item as relevant or important as their transition to practice occurred a number of years ago and this item may not relate to their current experience of occupational stress. The item “lack of adequate support and debriefings for high stress situations (i.e., suicide, death of a child, trauma, etc.)” was not anticipated based on findings from the literature; however, participants were clear that support and debriefings are either inadequate, or inconsistent in practice which can contribute to increased stress. Lastly, the item “The risk of infectious work exposures that may impact myself, my family, or loved ones” likely reflects issues in current practice as the study was conducted during the COVID-19 pandemic. While the ENSS had a related item, “Being exposed to health and safety hazards,” content experts clearly indicated the need to include an item that represents the risk to their family and loved ones. As the COVID-19 pandemic evolved and more nurses began caring for critically ill patients with a highly transmittable infectious disease, the associated occupational risk became a significant stressor for many. No longer were the occupational risks limited to the nurse, now their loved ones were at risk as well (Spoorthy et al., 2020).
The themes of the seven items that were deemed to no longer be relevant to occupational stress in nursing varied. Two items from the ENSS were removed as participants felt the items were antiquated as they related to a patient classification system and organizing providers’ work. Two newly proposed items that represent the use of the electronic health record were rejected, which was not expected as the literature extensively cites the electronic health record as a significant contributing factor to occupational stress (Gephart et al., 2015; Schenk et al., 2016; Yen et al., 2019). Content experts indicated that while the electronic health record does increase some of the complexity of their work, it also has notable benefits and is an ingrained part of their work that is generally accepted as a part of nursing practice.
While the findings of this study contribute to improving the measurement of occupational stress in nursing, limitations must also be noted. While the content validity index is extensively used to estimate content validity, this index does not consider the possibility of inflated values due to chance agreement (Polit & Beck, 2006). Validation of an instrument is a lengthy process, and while the first step is to determine content validity, additional analyses (i.e., reliability, internal structure validity, and relationships to other variables) should be conducted with a larger and more diverse sample of hospital based nurses (Zamanzadeh et al., 2015). Lastly, this study cannot ensure that all aspects of occupational stress in nursing were considered for inclusion in the Revised Nursing Stress Scale. However, open-ended questions were included to elicit feedback on missing items, two rounds of expert review were conducted, and the literature was reviewed to help minimize this limitation.
Perceived stress is a result of the complex interplay between the individual’s unique characteristics, the environment, and the individual’s appraisal of current demands and available resources (Folkman & Lazarus, 1984). Therefore, due to these complex and ever-changing factors, perceived stressors are also always changing. Study findings support that occupational stress is no different, and occupational stressors have also evolved in response to the changing role of the professional nurse, the occupational demands that they encounter, and the environment in which they work. An essential part of organizational safety and nurse well-being is ensuring that tools exist that accurately reflect and measure occupational stress, so organizations can evaluate the impact of stress and intervene accordingly. As a result, it is essential that stress measures are updated to include known occupational stressors in order to more accurately measure occupational stress as experienced by nurses in current practice.
The findings from this study contribute to nursing knowledge as we continue to research occupational stress among nurses and take action to develop accurate and comprehensive stress measures to improve nurse wellness. Specifically, findings from this study have contributed to a Revised Nursing Stress Scale that can be used to guide research and evaluate occupational stress among hospital-based nurses in contemporary nursing practice.
Footnotes
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: All authors report no conflicts of interest for the publication of this manuscript. Funding for the study was provided from a Research and Scholarship Award from the University of Wisconsin-Madison School of Nursing and from a scholarship from the Beta Eta at Large Chapter of the Sigma Honor Society of Nursing. These sponsors had no role in the study design; collection, analysis, and interpretation of data; writing of the report; and the decision to submit the article for publication.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by a Research and Scholarship Award from the University of Wisconsin-Madison School of Nursing and a scholarship from the Beta Eta at Large Chapter of the Sigma Honor Society of Nursing.
Ethics Statement
Study approval was obtained from the University of Wisconsin-Madison Institutional Review Board (ID 2019-1562).
