Abstract
Objectives:
The overall purpose of this study was to evaluate the validity and reliability of the Caring Assessment Tool-Administration survey. Three specific aims were to (1) evaluate construct validity of the Caring Assessment Tool-Administration survey by testing the hypothesized eight-factor structure of staff nurses’ perceptions of nurse manager caring behaviors, (2) estimate the internal consistency, and (3) conduct item reduction analysis.
Methods:
A 94-item Caring Assessment Tool-Administration designed to assess nurse manager caring behaviors appeared in the literature but lacked robust psychometric testing. Using a foundational theory and a cross-sectional descriptive design, the Caring Assessment Tool-Administration was evaluated for reliability and construct validity. Using convenience sampling, 1143 registered nurses were recruited from acute care hospitals in three states located in the Midwestern, Mid-Atlantic, and Southern Regions of the United States.
Results:
Psychometric testing of the Caring Assessment Tool-Administration was conducted using confirmatory analysis to determine the dimensionality of the construct, nurse manager caring behavior. The null hypothesis was an eight-factor solution fitting the theoretical model being tested. The null hypothesis was rejected because none of the measures examined for goodness of fit indicated the model fit the data. Confirmatory factor analysis did not support the hypothesized structure; however, exploratory factor analysis supported a one-factor solution that was conceptually labeled caring behaviors. To decrease subject burden, the 94-item survey was reduced to 25 items using item reduction analysis including assessing minimum factor loadings of ≥0.60 and evaluating survey item-total correlation and alpha. The Cronbach’s alpha of the new 25-item survey was 0.98.
Conclusion:
The new 25-item Caring Assessment Tool-Administration survey provides hospital administrators, nurse managers, and researchers with a sound, less burdensome instrument to collect valuable information about nurse manager caring behaviors.
Keywords
Introduction
The current acute care health systems are complex, chaotic, and rapidly changing, generating unprecedented uncertainty.1,2 This uncertainty stems from the nearly complete restructuring of American healthcare, which has placed immense pressure on leaders of acute care organizations to make changes that affect employees at all levels. 3 Among the many challenges presented to hospital administrators at all levels, work must be done to keep their institutions financially viable, which often requires difficult and unpopular decisions related to reorganization, consolidation and elimination of services, and workforce reduction. Balancing the demands of fiscal management and, at the same time, providing effective leadership to employees is difficult in times of uncertainty. 4 More often, priority is given to the financial needs of the institution over the needs of employees.
Inattention to employees’ needs adds additional strain to an already stressed organization plagued by change and growing complexity that adversely affects the work environment. The nature of nurse work has also changed in recent years because nurses have been asked to simultaneously care for sicker patients with fewer resources, implement new “smart” technologies, launch electronic medical record programs, and deal with value-based reimbursement models. These changes, coupled with administrative pressure to focus on fiscal cutbacks, can frustrate nurses and lead to job dissatisfaction, burnout, fatigue, anxiety, and inability to provide the quality of care expected 5 and desired. The consequences of work-related frustration contribute to unhealthy work environments that are associated with excessive absences and the intention to resign, which further adds to an unstable workforce.6–10
There is considerable agreement in the literature about the relationship between a healthy work environment and staff nurses’ satisfaction, professional empowerment, and retention.5,7,8,11–18 Cara et al. 19 suggested that a supportive and caring work environment could be enhanced by caring relationships between nurse managers and staff nurses. Expanding the relational role of the nurse manager is a viable way to create, enhance, and sustain a healthy work environment.6,14,20,21
Caring relationships between nurse managers and staff nurses support the idea that nurses who feel cared for by their managers are more motivated to develop caring relationships with patients. 19 According to Duffy’s 22 Quality-Caring Model® (QCM), caring relationships are instrumental and expressive behaviors that when used in the context of shared professional interactions facilitate understanding, learning, comfort, human dignity, security, self-confidence, hope, and encouragement. Examples of behaviors used by nurse managers that demonstrate caring are listening, being accessible, being encouraging, maintaining safe working environments, and offering support. When staff nurses feel cared for, positive patient, nurse, and system outcomes are theoretically proposed to occur. 22
The measurement of staff nurses’ perceptions of nurse managers’ caring behaviors has not been extensively studied. The lack of a psychometrically tested tool has contributed to the lack of evidence supporting more relational approaches to leadership. For example, the Caring Assessment Tool-Administration (CAT-Adm©) survey originally developed by Duffy 23 had not undergone comprehensive psychometric testing,22,23 and no other instruments were found that adequately measured this phenomenon.
Thus, the overall purpose of this study was to evaluate the validity and reliability of the CAT-Adm survey. Three specific aims were to (1) evaluate construct validity of the CAT-Adm survey by testing the hypothesized eight-factor structure of the measure of staff nurses’ perceptions of nurse manager caring behaviors, (2) estimate the internal consistency, and (3) conduct item reduction analysis to reduce administrative and participant burden.
Theoretical framework
Duffy’s 22 QCM served as the theoretical framework for this study because it informs development of the items on the CAT-Adm Survey and specifically identifies the critical nature of nurse managers’ caring behaviors. The QCM is a middle-range theory originally described as a blended model with major concepts related to quality and caring. 24 The model was revised in 2013 to include the influence of complex adaptive systems. 22 Relationship-centered professional encounter is a major tenet of the QCM and is described as a process between individuals that when conducted in a mutually reciprocal manner promotes “feeling cared for” in the recipients which then leads to progression or advancement. Relationship-centered professional encounters can be expressed through the attitudes, behaviors, and skills of professional nurses and their managers. Duffy originally proposed that eight factors comprised the concept: mutual problem-solving, attentive reassurance, human respect, encouraging manner, appreciation of unique meanings, healing environment, affiliation needs, and basic human needs. This study hypothesized that an eight-factor structure comprised this aspect of the QCM.
Methodology
Design
A cross-sectional, descriptive design was used to evaluate the specific aims of this study.
Setting/sample
To maximize the potential for recruiting the study sample, hospitals and hospital systems with greater than 500 beds were selected as possible recruitment locations. Hospital administrators from 10 hospital/hospital systems were approached for inclusion as recruitment sites. Of the 10 hospital/hospital systems, five hospital administrators chose to participate in the study. These five hospital/hospital system administrators represented the seven hospitals from which data were collected (see Table 1). Staff nurses were recruited from these seven hospitals located in three states in the Midwestern, Mid-Atlantic, and Southern Regions of the United States (see Table 1). These seven hospitals included suburban community, urban academic/teaching, and rural institutions and included both magnet- and non-magnet-certified facilities. Permission from chief nurse executives (CNEs) and approval from the university and each site’s institutional review board were obtained prior to beginning the study.
Demographics and sample characteristics a .
RN: registered nurse; PPM: professional practice model.
Sample characteristics were based on 703 staff nurses who met inclusion criteria and completed all survey items.
DeVellis 25 suggested using a ratio of 5–10 participants per survey item as a basis for determining sample size. Therefore, a sample of 940 participants were sought. Larger sample sizes are needed in factor analysis to determine the factor structure particularly depending on the number of items to be factored. 26 A convenience sample of 1143 acute care hospital staff nurses (registered nurses (RNs)) were recruited. To be included in this study, staff nurses must have been permanent staff members for a minimum of 6 months on the unit in which they were employed. Six months duration of unit-specific employment was chosen because this allowed the participants time to establish and reflect upon relationships that had developed with the nurse manager. Staff nurses with less than 6 months of experience working on the unit or were temporary or traveling nurses were excluded.
Instruments
Study data were collected using two instruments. The first instrument, Staff Nurse Demographic Questionnaire, was developed specifically for this study to collect demographic data about the participants. The second instrument was CAT-Adm, a 94-item survey developed in 1997, which was designed to measure staff nurses’ perceptions of nurse manager caring behaviors.22,27 The CAT-Adm (1997) was adapted from the Caring Assessment Tool Survey (CAT Survey)© (1994) which was developed to measure patients’ perceptions of nurse caring behaviors. Both CAT-Adm and CAT Survey are similar in that the theoretical framework used in their development was the QCM. 22
The original 94-item CAT-Adm survey was written in English at a fourth-grade reading comprehension level and was administered using paper and pencil. The respondents were asked to circle how often each activity occurred during their work period. Responses were arranged on a 5-point Likert-type scale with anchors 1 (never) to 5 (always) and 20 items were worded negatively to minimize error. 27 The range of scores for the 94 items was 94–470 with lower scores indicating less caring. The original 94-item questionnaire data collected in 1997 were evaluated using exploratory factor analysis (EFA). Three factors were found to explain 63.44% of the variance. Internal consistency and reliability for the total instrument was 0.942 using Cronbach’s alpha. Although the number of participants was satisfactory (N = 1850 RNs) and internal consistency reliability for the total instrument was acceptable, three major concerns impacted the quality of the study; it was conducted in one region of the country using a convenience sample and confirmatory analysis was not completed.
This study was conducted to determine the dimensionality of the concept, nurse manager caring behaviors. Full psychometric testing of the original 94-item CAT-Adm was performed. This included confirmatory factor analysis (CFA) to evaluate whether the eight-factor structure represents the concept nurse manager caring behaviors.
Procedure
Study data were collected and managed using Research Electronic Data Capture (Indiana University (REDCap)) 28 electronic data capture tools hosted at Indiana University. REDCap is a secure, web-based application designed to support data capture for research studies, providing (1) an intuitive interface for validated data entry, (2) audit trails for tracking data manipulation and export procedures, (3) automated export procedures for seamless data downloads to common statistical packages, and (4) procedures for importing data from external sources.
The CAT-Adm was administered through an electronic device within the hospital setting. The CNE from each of the participating hospitals provided one information technology contact person to assist in creating a link to the web-based (REDCap) CAT-Adm survey and provided a connection with the staff nurses’ hospital email addresses. Once the link was operational, an administrative contact person was alerted to begin distribution of Staff Nurse Demographic Questionnaire and CAT-Adm survey to the identified pool of prospective study participants through their hospital-secured email accounts. The hospital-secured email accounts of the study participants provided additional confidentiality protection. The data were collected over a 6-month time period.
Data analysis
The software program for conducting statistical analysis for this study was SAS® (Statistical Analysis Software), version 9.4 of the SAS System. 29 Data from Staff Nurse Demographic Questionnaire and CAT-Adm were exported from REDCap in a form directly usable for SAS.
Data were checked for accuracy prior to conducting statistical analysis. The data cleaning process revealed that the 94-item survey uploaded into REDCap inadvertently deleted three of the survey items. This omission was determined to not pose a significant risk to the integrity of the analysis since the three omitted questions were spread out across the instrument, and an adequate number of items representing the same concept were available for analysis.
Demographics and sample characteristics
There were 1143 staff nurses available for recruitment, and of those, 1080 agreed to participate. Of those who agreed to participate, 995 met inclusion criteria and were enrolled. Of the 995 who were enrolled, 703 completed the questionnaire and comprised the study sample. Sample characteristics are displayed in Table 1.
Of the seven participating hospitals, five were members of two systems. One hospital administrator oversaw three hospitals (system); all the others administered one hospital each. Two hospitals were part of a system, but each had their own hospital administrator.
Demographic data collected from this study are similar to national statistics for age, gender, education, race, and ethnicity.30–32
Most of the participants were white, not Hispanic/Latino, and female and were between the ages of 25 and 64 years. Approximately 50% had a bachelor of science in nursing (BSN) degree and 67.85% were certified in a specialty practice area. The duration of employment on the current unit was >1–3 years (27.74%) followed by those who worked on their units >5–10 years (21.34%). The two largest categories in “years of experience as an RN” were between >5 and 10 years (20.48%) and >25 years (20.77%). Most staff nurse respondents worked with adult patients in either community (44.8%) or academic (53.2%) hospitals. RN certification type was collected; however, small cell sizes precluded use in analyses.
Results
The first aim was to evaluate the construct validity of CAT-Adm survey by testing the factor structure underlying variance in staff nurses’ perceptions of nurse manager caring behaviors. More specifically, the intent was to determine whether the hypothesized factor model fit the data. The null hypothesis was an eight-factor solution based on eight caring factors described in the theoretical framework QCM. To address this aim, CFA was used.
The null hypothesis was rejected since none of the four measures examined for goodness of fit indicated adequate model fit to the data ((chi-square statistic) chi = 14732.7, degree of freedom (df) 3976 (p < 0.0001), standardized root mean square residual (SRMR = 0.047), root mean square of approximation root mean square error approximation (RMSEA = 0.062), and Bentler comparative fit index (0.856)).33,34 Since the model was rejected using CFA, the next logical step was to perform EFA to identify a plausible factor structure given the observed data. 35
EFA included assessment of sampling adequacy, determining the number of factors to extract and retain, estimating factor structure (item loadings) and factor rotation, and interpretation of the factors. Prior to factor analysis, a Kaiser–Meyer–Olkin (KMO) test for sampling adequacy of 0.99 was found, supporting factorability of the correlation matrix. 36
Parallel analysis (PA) was then performed to determine the number of factors to retain. Factor retention was determined by comparing the eigenvalues of the observed data set produced by EFA with random data sets that parallel the observed data. Eigenvalues were extracted from random data sets that parallel the actual data set (an observed data set of this study) in terms of the number of cases and variables. Two factors were retained using this methodology.37,38 The variance that could be explained by each of the factors is displayed in Table 2. As shown, Factor 1 explained 55.67 (61.17%) of the variance and Factor 2 contained 3.89 (4.27%) of the variance. Combined, the two factors explained 59.56 (65.45%) of the total variance.
Factor variance.
Only the first seven components are listed out of 91 variables in this table. Only first two components were retained.
PCA: principal component analysis; PA: parallel analysis.
Orthogonal and oblique factor rotations were then completed to make the loadings more easily interpretable.34,35 Oblique (PROMAX) rotation of the data resulted in an inter-factor correlation of 0.709 indicating that there were substantive correlations between the two factors. Use of oblique rotation (PROMAX) was selected for further interpretation of the data because of high inter-factor correlations.
To assist in factor interpretation, factor loadings were examined using a priori factor loading limits set at 0.60 or higher to designate factor membership (relationship of the variables to the factors). As can be seen in Table 3, Factor 1 had 77 of 91 (84.6%) items loading 0.60 or higher and 73 of 91 (80%) survey items loaded 0.60 or higher on Factor 2. Negatively worded items loaded higher on Factor 2 (12/20, 60%) than on Factor 1 (8/20, 40%). Items loading at minimum 0.60 or higher on both Factor 1 and Factor 2 were 64 of 91 (70%). Factor 2 had 27 of 64 (42%) items loading 0.60 or higher and the factor structure loadings were greater on Factor 2 than on Factor 1. Of those 27 items that had higher correlation with Factor 2 (belonged more to Factor 2, but not exclusively), 16 were negatively worded items. All but one negatively worded item (item number 88) loaded more strongly on Factor 1.
Factor structure correlations.
Selected items for inclusion in the 25-item Caring Assessment Tool—Adm questionnaire are marked with superscript1 and bold italic.
Given (1) the large amount of shared variance (50.3%) between the two factors, (2) the negatively worded items loaded more heavily on Factor 2, and (3) Factor 1 eigenvalue (55.7) was 14 times larger than Factor 2 eigenvalue (3.9), Factor 2 was determined to be redundant with Factor 1 and was eliminated, resulting in a one-factor solution. Only items that loaded both more substantively and stronger on Factor 1 were retained, resulting in a reduction of items on Factor 1 to 59.
After reviewing the items remaining on Factor 1, it was conceptually labeled caring behaviors. These items are congruent with the QCM major tenet of relationship-centered professional encounters, are central to the nurse manager–staff nurse caring relationship, and logically describe the multiple caring behaviors simultaneously displayed by nurse managers during their interactions with staff nurses. For example, a nurse manager’s behavior may be perceived by a staff nurse as “showing concern for me” when the manager was also “checking with me to make sure I understand what is going on in the workplace.” Furthermore, the term caring behaviors may be easier to conceptualize and better recognized by nurses in the work setting.
The second aim was to estimate internal consistency. The Cronbach’s alpha coefficient obtained for the retained Factor 1 was 0.99, which is considered good. 25 According to DeVellis 25 and Streiner, 39 alphas greater than 0.90 suggest the need to shorten the instrument due to redundancy of items.
The third aim, item reduction analysis, was evaluated through a series of procedural steps to reduce the number of items while at the same time maintaining an acceptable Cronbach’s alpha. The goal of reducing the number of survey items (item reduction) was to decrease administrative and participant burden. The procedural steps for item reduction analysis included assessing minimum factor loadings of 0.60 or higher, evaluating survey item-total correlation and alpha.25,34
In the prior steps, 32 survey items were removed, resulting in 59 survey items remaining. Recall from the interpretation of the factors, one-factor was retained and had high internal consistency indicating that all items reflected a relatively homogeneous construct (caring behaviors). In addition, many of the nurses (n = 377, 35%) who agreed to participate in the study failed to complete the entire survey leaving the last items without responses which indicates survey fatigue or time constraints. Therefore, reducing the survey to 25 items was amendable to reduce participant burden in completing the CAT-Adm survey. 25 The decision to reduce the survey instrument was jointly made by the authors, with a target of 25 items for the final scale. 40 To reduce the number of items to 25, 59 items were analyzed using PROC CORR, which generated a randomly chosen subset of items. 41 Randomly selecting items eliminated the potential for investigator bias and ensured that a representative sample of items was obtained. Cronbach’s alpha was then performed on the reduced set of 25 items (α = 0.98), which was determined to be good. Examples of the retained survey items were as follows: keeps me informed, openly shows concern for me, is patient even when I am difficult, and knows what is important to me. See Table 3 for the 25 items that are marked with bold italic and superscript.
Discussion
The purpose of this study was to evaluate the dimensionality of the CAT-Adm survey, estimate internal consistency and reliability, and reduce the number of items on the survey to decrease participant and administrative burden. Using CFA, the hypothesized eight-factor solution was rejected. A follow-up EFA suggested a single factor be retained. Review of the items that loaded ≥0.60 on Factor 1 led to the conceptual label, caring behaviors. These behaviors are congruent with the QCM major tenet of relationship-centered professional encounters and also are central to the nurse manager–staff nurse caring relationship. 22 The one-factor solution representing caring behaviors seemed logical since, in the work environment, nurse managers frequently use multiple behaviors simultaneously. Furthermore, the term caring behaviors may be easier to comprehend and eventually be actualized in the work setting.
Recognizing the absence of a psychometrically sound instrument to measure nurse managers’ caring behaviors and the industry need for such a measure, the CAT-Adm survey needed further development. In 2008, Duffy conducted EFA to begin the validation process of CAT-Adm survey and found a three-factor solution, whereas in this study a one-factor solution was found. Consistent with Duffy’s prior study in 2008 of the 94-item survey, which produced a coefficient alpha of 0.94, the 25-item CAT-Adm survey’s Cronbach’s alpha was 0.98. 27 This new 25-item survey measures nurse manager caring behaviors as perceived by staff nurses and is consistent with the theoretical basis of the QCM and captures the essence of Duffy’s QCM relationship professional encounters. Congruent with Duffy’s QCM, caring behaviors affect relationship-centered professional encounters. This study builds on prior research and provides a valid instrument to advance the research in the field.
Study limitations
One limitation of this study is the extent to which findings may be generalized, since participants were recruited from three states in the Midwestern, Mid-Atlantic, and Southern Regions of the United States. Conducting this study in other regions of the United States would improve generalizability. This is also true with the demographics of this study which were somewhat comparable to the national statistics for RNs overall. Conducting this study in other regions of the United States may increase the number of participants by gender (more males), and diversity related to race and ethnicity other than White. Another limitation of this study was the use of convenience sampling. This sampling method may not yield a representative sample of the population of interest and therefore limits a more general application of the results. Designing a study using random selection would improve the generalizability of the findings.
Other possible limitations are that the participants’ cultural perception of caring, assessing caring behaviors through an electronic format versus face to face, and the influence of factors in the work environment could influence participants’ responses.
Future research/applications
Having a valid and reliable tool (CAT-Adm survey) to measure staff nurses’ perceptions of nurse manager caring behaviors can provide healthcare administrators with important information. For example, findings may point to differences among departments and identify those nurse managers who might benefit from caring behavior educational offerings. In addition, correlation between nurse manager caring behaviors and patient, nurse, and system outcomes could be explicated. Although not evaluated in this study, data were easily collected and analyzed using an electronic platform. Further research evaluating this approach to data capture may lead to less burdensome administration.
Healthcare administrators and nurse leaders who desire to develop a culture of caring within their institutions or who want to evaluate application of a relationship-centered professional practice model may find the reduced 25-item CAT-Adm survey useful in a preliminary examination of relationships between nurse managers and staff nurses before creating or adopting interventions. Designing interventions focused on caring behaviors can then be embedded into on-line learning modules, leadership workshops, simulation laboratory scenarios, and interactive groups. Through these types of activities, nurse managers would be given the opportunity to be engaged and to incorporate their newly learned caring behaviors into their managerial practice. This would ultimately heighten positive work environments that may positively impact nurse retention, engagement, job satisfaction, and patient outcome indicators such as patient experiences. The results can also be evaluated using individual survey items to design specific educational and leadership development programs to create a more caring culture. Together, healthcare administrators and nurse managers can use the results of CAT-Adm to increase awareness of nurse manager caring behaviors and guide leadership development. Awareness of nurse manager caring behaviors may contribute to ongoing self-development of nurse managers and ultimately positively influence staff nurses’ decisions to remain employed and contribute to the quality of the institution.
The 25-item CAT-Adm survey can be used by educators and human resource personnel involved in leadership development programs in the healthcare environment, academic institutions, and proprietary organizations. The goal would be to develop and measure program outcomes when assisting nurse managers with new employee orientation, in improving relationships with employees, and during adoption and implementation of caring-based professional practice models. In addition, adding caring content to nursing administration programs and linking the findings back to initiatives such as healthy work environments would benefit students enrolled in nursing administration programs.
Conclusion
In conclusion, this study was designed to evaluate the psychometric properties of the CAT-Adm. Although the CFA did not demonstrate an eight-factor solution as postulated, the EFA supported a one-factor solution, labeled caring behaviors. Nurse manager caring behaviors provides more clarity for the theoretical concept of relationship-centered professional encounters as documented in the QCM. Based on study findings, there is now evidence supporting acceptable validity and reliability of CAT-Adm. Furthermore, the revised CAT-Adm was reduced to 25 items while maintaining adequate internal consistency, reducing survey fatigue and administrative burden. As a result of this study, healthcare administrators, managers, educators, and researchers now have a shorter, reliable, and valid tool to evaluate nurse manager caring behaviors, an important component of healthy work environments.
Footnotes
Acknowledgements
Special appreciation to Ain Haas, PhD, Professor Emeritus of Sociology, School of Liberal Arts, Indiana University. Dr Haas served on C.L.W.’s dissertation committee and provided valuable insight. The following hospitals participated in this study: Franciscan Health, Indianapolis, IN; University of Florida Health Shands Hospital, Gainesville, FL; University of Florida Health Shands Hospital, Jacksonville, FL; Lakeland Regional Medical Center, Lakeland, FL; and West Virginia University Hospitals, Morgantown, WV.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.