Sage Journals: Discover world-class research

Abstract

In response to the COVID-19 pandemic, specific assessments such as the COVID Stress Scales (CSS) were developed to measure pandemic-specific distress. The present study aimed to further validate the psychometric properties of the CSS and explore relationships between the CSS and measures of well-being. Adults in the U.S. (N = 1,388, 63.3% female, 36.1% male, 58.4% Caucasian) completed the CSS, Satisfaction with Life Scale (SWLS), Patient Health Questionnaire-4 (PHQ-4), Brief Resilient Coping Scale (BRCS), and Cognitive and Affective Mindfulness Scale-Revised (CAMS-R) in June and July of 2020. Confirmatory factor analyses on the CSS were used to examine multiple model solutions. The six-factor solution provided the best fit (CFI = 0.98, TLI = 0.98, RMSEA = 0.06), outperforming the five-factor solution, which showed higher RMSEA (.07) and inadequate item separation according to the Rasch analysis. Rasch analysis found no misfitting items and the data fit the Rasch model. Results of the Differential Item Functioning (DIF) analysis supported gender invariance as only negligible DIF was observed across items. As expected, the CSS was positively correlated with measures of anxiety and depression (PHQ-4) and negatively correlated with life satisfaction (SWLS), resilience (BRCS), and trait mindfulness (CAMS-R). Our results partially replicate the factor structure of the CSS found in different adult samples.

Plain language summary

Examining the Covid Stress Scale’s different dimensions

In this paper, we collected data using the Covid Stress Scale survey. In the literature we can find that there are competing “models” that is, some studies have found the scale has six dimensions or factors and other studies have found five dimensions. We examine these dimensions in our study and we used methods such as confirmatory factor analysis, and Rasch analysis.

Keywords

assessment COVID stress scales depression anxiety clinical practice

Introduction

Major pandemics have been documented throughout history (as early as the Plague of Justinian in 541) and it has been noted that public health measures such as quarantine, isolation, and travel controls have been implemented throughout time to help contain the spread of infectious diseases (Piret & Boivin, 2021). Over time, increased attention has been brought to the adverse mental health impact that pandemics and subsequent public health measures taken to aid in containing outbreaks can have on individuals, such as increased stress, anxiety, and depression symptoms (Soklaridis et al., 2020). The COVID-19 pandemic has been linked to heightened psychological distress worldwide (Pieh et al., 2020; Qiu et al., 2020; Rehman et al., 2021; Rodríguez-Rey et al., 2020; Tsamakis et al., 2020; Vindegaard & Benros, 2020; Wang et al., 2020), and its impact persists in society despite no longer being classified as a public health emergency (Gonzalez, 2023; World Health Organization, 2023).

The COVID-19 pandemic highlighted that mental health professionals in clinical practice need assessment and intervention solutions as they face the unique challenges associated with health-related societal crises such as a pandemic (Elliott et al., 2024; Gruber et al., 2021). Pandemic-specific experiences, such as the fear of becoming infected with the virus, isolation, and being quarantined have been specifically associated with increased psychological distress (Rehman et al., 2021; Taylor et al., 2020; Vindegaard & Benros, 2020). The enduring mental health ramifications from the experiences of the pandemic persist to the present day (Kathirvel, 2020). This calls for continuous evaluation and intervention strategies by mental health professionals to address the evolving needs of affected populations. This persistence is important, given that experiences such as lockdown measures and infection rates, while no longer as prominent, continue to leave lasting impacts on individuals’ well-being (Pennix et al., 2022; Solomou et al., 2024). Measures have been developed to capture the unique impact that these pandemic-specific fears and stressors have on individuals and to aid in assessing those who may need support due to this distress (Ahorsu et al., 2022; McKay et al., 2020; Taylor et al., 2020). In this study, we will examine the psychometric properties of one of the instruments designed to measure the stressors associated with COVID-19.

COVID Stress Scale

As mentioned previously, pandemics have historically led to widespread psychological distress (Piret & Boivin, 2021), yet standardized tools to assess pandemic-specific stress, such as the COVID Stress Scales (CSS), have only recently been developed (Taylor et al., 2020). After reviewing related literature and consulting with health-related anxiety experts, Taylor et al. (2020) identified domains and subsequent scales important for the assessment of pandemic-specific related stress; fears about the dangerousness of the virus (Danger scale), fears about physical sources of virus contamination (Contamination scale), fears related to foreigners being a source of the virus (Xenophobia scale), fears about social and economic consequences of the virus (Socioeconomic Consequences scale), compulsive checking behaviors related to the virus (Compulsive Checking scale), and traumatic stress symptoms related to the pandemic (Traumatic Stress scale). The authors initially developed 58 items that were thought to capture these six domains and after removing five items, were left with 53 initial items. These items were worded in a simplified way that would make adaptation of the items for future pandemics easier (e.g., using the term “the virus” in lieu of COVID-19 specifically).

The psychometric properties of the CSS were originally examined in two large samples of U.S. (N = 3,375) and Canadian (N = 3,479) adults (Taylor et al., 2020). Taylor et al. (2020) used parallel analysis to determine the dimensionality of each scale. For both samples, parallel analysis supported a five rather than a six-factor solution. Each factor corresponded to one of the proposed scales, except for the Danger and Contamination scales, which loaded on a single factor together. Rather than reducing the Danger and Contamination factor to a single six-item scale, all 12 items were retained as two separate scales so that Danger and Contamination could be assessed separately.

Factor Structure

Since its development, the CSS has been adapted and used by researchers to assess for distress related to the COVID-19 pandemic globally (e.g., Asmundson et al., 2020), as well as in Mexico (Delgado-Gallegos et al., 2020), Iran (Khosravani et al., 2021), Egypt and Saudi Arabia (Abbady et al., 2021), Palestine (Mahamid et al., 2022), Germany (Kubb & Foran, 2020), Philippines (Montano & Acebes, 2020) and a total of 24 languages (Rachor et al., 2023). However, the five-factor solution for the CSS initially identified by Taylor et al. (2020) has not always remained consistent among different samples and adaptations of the CSS. For example, Adamczyk et al. (2021) studied the CSS in Polish and Dutch samples. The authors stated that consistent with Taylor et al. (2020) the scale showed a five-factor structure rather than a six-factor solution. However, the authors’ data showed the Traumatic Stress and Checking scales merged into a single factor in the five-factor solution. Historically, in different studies the Danger and Contamination scales have combined to create a single factor yielding a five-factor solution (Mertens et al., 2021; Taylor et al., 2020). Taylor et al.’s (2020) final version recommended a five-factor solution while Mertens et al. (2021) saw a clustering of the items in the Danger and Contamination subscales. Yet, Mahamid et al. (2022) translated the CSS into the Arabic language and reported that a six-factor solution with 31 items among a sample of N = 860 Palestinian adults.

Rasch Analysis

Although prior research has validated the CSS across different populations (Mahamid et al., 2022; Taylor et al., 2020), few studies have examined its item-level psychometric properties using Rasch analysis, limiting our understanding of its internal structure and item difficulty calibration. Rasch modeling is a popular tool in applied health research with the purpose of exploring the psychometric properties of questionnaires and inventories (Dima, 2018). Recently, it was used to assess the psychometric properties of the Fear of COVID (FCV) scale with Chinese and Pakistani populations (Chen et al., 2021; Ullah et al., 2023), and the development of a symptom burden questionnaire for long covid (SBQ-LC; Hughes et al., 2022). Yet, these authors could not find literature on Rasch analysis and the CSS. One advantage that Rasch analysis presents is that the model can place a person’s trait and item difficulty in the same continuum (e.g., logits) which makes it easy for applied researchers to validate the scores of a scale (Boone, 2016; Boone et al., 2013). Research on item selection suggests that Rasch has a slightly higher percentage of variance explained compared to CFA (Chiu et al., 2020). Christensen et al. (2012) recommends the use of both Rasch analysis and CFA to address lack of confidence in the items, and there is interest in the dimensionality of the measure as is the case with the CSS subscales (e.g., Danger and Contamination). Finally, Rasch analysis provides information on item fit based on this information researchers can make decisions regarding revising or removing items which can be useful for the Danger and Contamination subscales.

Differential Item Functioning

Consistently, scale development has involved examining differential item functioning to identify item bias (Desjardins & Bulut, 2018). For example, within the COVID-19 specific scale development area, Sánchez-Teruel and Robles-Bello (2021) and Stănculescu (2022) examined DIF across gender for the Fear of COVID-19 Scale (FCV-19S) in Spanish and Romanian, respectively, as part of their psychometric evaluations of the instrument. When examining the literature for the CSS only Şahin et al. (2022) conducted differential item functioning on the CSS. The authors utilized gender, whether participants had presence or absence of a COVID history, and student status to assess DIF within a sample of Turkish adult volunteers. The authors reported that the DIF analyses did not find evidence of invariance for the CSS items across gender. Although Şahin et al. (2022) did not detect DIF, further research is needed to establish the gender invariance of the CSS across diverse cultural contexts to support valid measurement and meaningful gender comparisons.

Well-Being Measures

Taylor et al. (2020) administered the patient health questionnaire-4 (PHQ-4), short health anxiety inventory (SHAI), the obsessive-compulsive inventory-revised (OCI-R), the xenophobia scale (XS), and the Marlowe Crowne social desirability scale short form (MCSD-SF) to provide convergent and discriminant validity evidence for the CSS. However, they did not include well-being measures. A recent study by Duckering (2022) examined the relationship between the CSS and psychological well-being using psychological inflexibility as a mediator variable and the psychological well-being scale (PWS) as the outcome variable. The single study examined the relationship between CSS and psychological well-being in 152 psychology students that were predominantly Latinx/Hispanic (69%) and reported a weak correlation. Furthermore, Pheko et al. (2023) studied mental well-being shortly after the beginning of the COVID-19 pandemic in the countries of Botswana, Zimbabwe, and Malaysia. The authors found anxiety and loneliness to be moderately and negatively correlated to well-being.

Assessment of well-being and personal strengths alongside measures of psychological distress has been proposed to be helpful and offer more information in clinical practice (e.g., Rashid & Ostermann, 2009). Investigating the relationship between pandemic-specific stress, general distress, and indicators of well-being can better inform our understanding of assessment and treatment efforts during health crises such as a global pandemic (Carver et al., 2021; Gavin et al., 2020; Greenspoon & Saklofske, 2001; Saltzman et al., 2020) as well as inform the development of intervention and prevention targets during present and future pandemics.

Purpose and Rationale

The overall aim of the present study is to examine the psychometric properties of the CSS and examine the relationships between the CSS and measures of well-being (life satisfaction, resilience, and mindfulness), neither of which have been included in the COVID-19 assessment literature. A secondary aim included examining item performance in the CSS by performing Rasch analysis and differential item functioning (DIF) analysis across gender both of which have rarely been performed on the CSS. We predict that the psychometric properties of the CSS, using CFA, would yield similar findings to the original study (Taylor et al., 2020). However, based on our literature review we anticipate examining competitive models based on different number of factors. We also hypothesized that life satisfaction, resilience, and mindfulness would negatively correlate with the CSS (Conversano et al., 2020; Lenzo et al., 2020; Pidgeon & Keye, 2014) and that anxiety and depression would positively correlate with the CSS (Devakumar et al., 2020). While the CSS was developed to assess stress directly related to the pandemic there is a lack of understanding its differential functioning across gender.

Research Questions

How many factors are identified in the CSS?

Using Rasch analysis, how effectively do the items on the CSS Scale measure COVID-19 stress among population in the US?

Does the CSS exhibit measurement invariance, as examined by differential item functioning, across gender?

Method

Ethical Approval

The general protocol and data collection procedures for this study were approved by the University of Texas at Tyler Institutional Review Board (IRB). Prior to beginning the survey, participants provided informed consent. A component of the informed consent was a review of potential risks and benefits of participation. Participants were informed that some of the survey questions may elicit difficult memories or that they may feel distress. As a component of the informed consent, they were reminded that they may stop participation at any time and were provided with contact information of the principal investigator, faculty sponsor, and IRB contact in the event that there were any questions or concerns about the survey. Participants were informed more than once within the informed consent that their participation in this study was entirely voluntary and that they were not required to answer any questions that they did not want to and could stop participating at any time without penalty. To mitigate risks associated with emotional discomfort or distress that participants may have experienced, at the end of the survey each participant was offered a debriefing summary in which resources were offered to them (i.e., mental and behavioral health resources, behavioral support helpline). Participants were informed that study data were kept confidential and personal information was collected and stored separately from survey information. Each participant’s survey data was identifiable by a unique code for each participant.

Overall, the benefits of this research study were viewed by the IRB to outweigh the minimal risks involved with the potential to promote well-validated assessment tools.

Sampling and Data Collection Procedures

Participants completed the CSS in June–July 2020, a period characterized by peak pandemic uncertainty and widespread lockdowns, making this dataset ideal for assessing acute COVID-related distress. Participants were recruited through social media outlets (Twitter, now X, and Facebook), which may have introduced selection bias by over-representing younger, tech-savvy individuals with internet access. After providing informed consent, participants (N = 1,388) who were 18 years of age or older, resided in the U.S., and were able to read and understand the English language began the survey via Qualtrics between June 22 and July 16, 2020. Participants were incentivized to complete the survey by being entered into a drawing for a $25 Amazon gift card. Participant ages ranged from 18 to 83 (M = 30.7, SD = 10.4). In addition to age, participants were asked a series of demographic questions including race and ethnicity, gender, marital status, and highest level of education completed.

Participants

The majority of participants identified as female (63.3%) and the remaining participants identified as male (36.1%) or other/prefer not to answer (.6%). In response to race and ethnicity, over half of the participants identified as White (58.4%) and the remaining participants identified as Hispanic or Latinx (11.9%), Black or African American (10.0%), American Indian or Alaska Native (7.7%), Asian or Asian American (4.8%), Multi-racial (2.7%), Middle Eastern or North African (1.8%), Native Hawaiian or Other Pacific Islander (1.5%) or preferred not to answer. Most of the participants were married (49.1%) or never married (43.6%) and the remaining participants were either divorced, separated, widowed, or preferred not to answer (7.3%). Most participants had completed a level of education ranging from high school to Bachelor’s degree (cumulative 81.5%) and the remaining participants had either not completed high school (2.4%) or completed an education higher than a Bachelor’s degree (16.1%).

In addition to the demographic questions mentioned above, the survey included questions related to the participant’s COVID-19 pandemic experience (see Table 1 for details).

Table 1.

COVID-19 Pandemic Experiences Reported by Participants.

Question	Answer	n (%)
Experienced symptoms of COVID-19	Yes	98 (7.06%)
	No	1,208 (87.03%)
	Prefer not to answer	82 (5.91%)
Tested for COVID-19	Yes	500 (36.02%)
	No	804 (57.93%)
	Prefer not to answer	84 (6.05%)
If tested, tested positive for COVID-19	Yes	40 (8.00%)
	No	439 (87.80%)
	Prefer not to answer	21 (4.20%)
Quarantined due to COVID-19	Yes	345 (24.85%)
	No	961 (69.24%)
	Prefer not to answer	82 (5.91%)
Family member tested positive for COVID-19	Yes	105 (7.56%)
	No	1,191 (85.81%)
	Prefer not to answer	92 (6.63%)
Family member passed away due to COVID-19	Yes	31 (2.23%)
	No	1,265 (91.14%)
	Prefer not to answer	92 (6.63%)

Measures

COVID Stress Scales

The CSS is a 36-item scale designed to assess COVID-specific distress. The CSS consists of 6 scales including Danger (fear of becoming infected), Contamination (fear of coming into contact with contaminated objects or surfaces), Compulsive Checking (checking for and seeking reassurance for pandemic-related threats), Socioeconomic Consequences (fear of the socioeconomic consequences of the pandemic such as job loss), Xenophobia (fear that foreigners are carriers of the virus), and Traumatic Stress (symptoms of traumatic stress related to the pandemic such as intrusive thoughts and nightmares). Items in each scale are rated on a 5-point Likert-type scale, ranging from 0 (not at all or never) to 4 (extremely or almost always) and were summed for each subscale. Higher scores on each scale are indicative of higher distress levels. The scores of the CSS have demonstrated good reliability and validity in previous studies (e.g., Montano & Acebes, 2020; Taylor et al., 2020) and in the present study. Details about reliability for the CSS in this study are reviewed in detail in the results section.

Patient Health Questionnaire-4

The Patient Health Questionnaire (PHQ-4; Kroenke et al., 2009) is a 4-item self-report scale that was created to assess core symptoms of anxiety and depression symptoms that have occurred within the past 2 weeks. Two items screen for depression and 2 items screen for anxiety. Items are rated on a 5-point Likert-type scale ranging from 0 (not at all) to 3 (nearly every day). Anxiety and depression symptoms can be examined together with a total score of all items or can be assessed separately by examining the anxiety and depression subscales. Kroenke et al. (2009) suggested that the following total combined scores are indicative of symptom elevation: 3 to 5 (mild), 6 to 8 (moderate), 9 to 12 (severe). When examining anxiety and depression separately from each other, the authors suggested that scores of ≥3 for both the anxiety and the depression subscales serve as appropriate cut-points to indicate clinical significance. For example, a score of ≥3 on the depression subscale has demonstrated 83% sensitivity and 90% specificity for major depressive disorder and a score of ≥3 on the anxiety subscale has demonstrated 88% sensitivity and 83% specificity for generalized anxiety disorder (Kroenke et al., 2009). The PHQ-4 has demonstrated good reliability in the past (e.g., Löwe et al., 2010) and in the present study. Cronbach’s alpha and McDonald’s omega were α = .80,ω = .80, respectively.

Brief Resilient Coping Scale

The Brief Resilient Coping Scale (BRCS; Sinclair & Wallston, 2004) is a 4-item scale that measures how individuals cope with stress and difficulties. The participants responded to 4 items using a 5-point Likert-type scale ranging from 1 (does not describe me at all) to 5 (describes me very well). Item responses were summed and compared to normative cut-offs: 4 to 13 (low resilient copers), 14 to 16 (medium resilient copers), 17 to 20 (high resilient copers). The BRCS has demonstrated acceptable internal consistency with Cronbach’s alpha between .76 and .78 in previous research (Kocalevent et al., 2017; Sinclair & Wallston, 2004), and fair reliability in the present study (α = .69, ω = .69).

Satisfaction with Life Scale

The Satisfaction with Life Scale (SWLS; Diener et al., 1985) is a 5-item scale designed to assess an individual’s overall satisfaction with life. Each item is rated on a 7-point Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree). The scores were summed and compared to normative cut-offs (Pavot & Diener, 2008): 5 to 9 (extremely dissatisfied), 10 to 14 (dissatisfied), 15 to 19 (slightly dissatisfied), 20 (neutral), 21 to 25 (slightly satisfied), 26 to 30 (satisfied), 31 to 35 (extremely satisfied). The SWLS has demonstrated good reliability and validity in previous studies (Pavot et al., 1991; Yun et al., 2019) and demonstrated good reliability in the present study (α = .86,ω = .86).

The Cognitive and Affective Mindfulness Scale-Revised

The Cognitive and Affective Mindfulness Scale-Revised (CAMS-R; Feldman et al., 2007) is a 12-item measure designed to capture trait mindfulness. Participants rate each item on a Likert-type scale from 1 (rarely/not at all) to 4 (almost always). After appropriate reverse-scoring, participant scores were summed. Higher values reflect greater mindful qualities. The CAMS-R has demonstrated acceptable reliability in previous studies (Feldman et al., 2007) and acceptable reliability in this study (α = .74,ω = .74).

Data Analysis

Data management, internal consistency reliability, and correlations were conducted in IBM SPSS Statistics (Version 24; IBM Corp, 2016). Confirmatory factor analyses (CFA) and differential item functioning (DIF) analysis were performed using the lavaan, nFactors, psych, and lordif packages in R (Choi et al., 2016; Raiche & Magis, 2020; R Core Team, 2022; Revelle, 2022; Rosseel, 2012). Rasch analyses were conducted in Test Analysis Modules (TAM) and the SnowIRT module from jamovi (Jamovi Project, 2021; Robitzsch et al., 2022; Seol, 2023). Analyses were conducted with the completed data for each specific analysis.

Considering that both five- and six-factor model have been proposed among CSS literature (Mahamid et al., 2022; Taylor et al., 2020), we wanted to test both models with our data. The five-factor solutions were formulated from the findings of the original developers of the CSS (Taylor et al., 2020) in which Danger and Contamination items were placed on one factor together, with the other scales being placed on their own respective factor. The six-factor solutions were formulated from the findings of other researchers (e.g., Mahamid et al., 2022) in which the items from each of the six scales were placed in their own respective factor. We performed CFA on the following solutions: five-factor, five-factor (higher order), five-factor (bifactor), six-factor, six-factor (higher order), and six-factor (bifactor). We ran these analyses using the diagonally weighted least squares (DWLS) estimation as it has been shown to offer more accurate parameter estimates and model fit that is more robust to non-normality and variable type (Mîndrilă, 2010).

In the current examination, we tested for differential item functioning across male and female respondents using an iterative approach combining item response theory and ordinal logistic regression methodology. This approach uses item response theory techniques (i.e., grade response model or generalized partial credit model) to estimate each participant’s level of the underlying latent trait assessed by the instrument. Once ability estimates are established, a series of nested logistic regression models (i.e., ability only; ability and group assignment; ability, group assignment, and ability × group assignment) are estimated testing relationships between independent variables and responses for each assessment item. The fit of the nest models for each item is then compared using likelihood ratio tests and significant differences between models are evidence of uniform (i.e., consistent group differences in the probably of selecting a certain response option regardless of the level of the underlying latent trait), non-uniform (i.e., group differences in the probably of selecting a certain response option that differ across different levels of the underlying latent trait) and overall differential item functioning. If DIF is detected in assessment items, the process begins again using modified ability/trait estimates and continues until a stable set of items demonstrating DIF is identified (Choi et al., 2016). This was applied to each of the subscales individually to provide a more accurate assessment of potential item-level bias (De Ayala, 2009; Vallejo-Medina & Sierra, 2015). The magnitude of detected DIF effects was evaluated using ΔR² Nagelkerge and the following criteria: negligible DIF, ΔR² Nagelkerge < .035; moderate DIF, ΔR² Nagelkerge between .035 and .070, Large DIF, ΔR² Nagelkerge > .070 (Jodoin & Gierl, 2001).

Results

Confirmatory Factor Analysis (CFA)

Mardia’s test was conducted to test multivariate normality. The results indicated there were issues with multivariate normality (p < .001), which further supported the use of DWLS estimation when conducting the CFA. Among the models demonstrating acceptable fit, the six-factor model demonstrated the best fit to the data, outperforming the five-factor, five-factor higher-order, and six-factor higher-order models based on model fit indices (see Table 2 for results of each CFA performed).

Table 2.

Results from Confirmatory Factor Analyses by Solution.

Model	CFI	TLI	RMSEA	RMSEA 95% CI	SRMR
Five-factor	0.98	0.98	0.07	[0.065, 0.069]	0.07
Five-factor (higher order)	0.97	0.97	0.07	[0.070, 0.074]	0.08
Five-factor (bifactor)	0.81	0.79	0.20	[0.194, 0.199]	0.19
Six-factor	0.98	0.98	0.06	[0.054, 0.059]	0.06
Six-factor (higher order)	0.97	0.97	0.07	[0.070, 0.074]	0.08
Six-factor (bifactor)	0.80	0.78	0.20	[0.198, 0.202]	0.19

Note. CFI = Comparative Fit Index; TLI = Tucker-Lewis Index; SRMR = standardized root mean square residual; RMSEA = root mean square error of approximation; CI = confidence interval.

Internal Consistency Reliability

Internal consistency reliability was measured by Cronbach’s alpha and McDonald’s omega for each CSS subscale. Following guidelines identified by Streiner (2010), internal consistency for each CSS ranged from acceptable to excellent (George & Mallery, 2019). The following scales demonstrated excellent internal consistency: Socioeconomic Consequences scale, α = .91, ω = .91 and Xenophobia scale, α = .95, ω = .95. The following scales demonstrated good internal consistency: Contamination scale, α = .88, ω = .88; Traumatic Stress scale, α = .88, ω = .88 and Danger scale, α = .87, ω = .87. The Compulsive Checking scale demonstrated acceptable internal consistency (α = .78, ω = .79).

Rasch Analysis

To help us decide between the partial credit model (PCM) and the rating scale model (RSM) prior to conducting Rasch analysis we conducted a model comparison analyses (Table 3; Andrich, 1978; Masters, 1982). The PCM does not assume of fixed item thresholds across in contrast to RSM (Andrich, 1978, 1988; Masters, 1982; Masters & Wright, 1997). The results of the likelihood ratio test of model comparisons supported the use of PCM for our dataset based on lower values of the Deviance, the Akaike’s information criterion (AIC; Akaike, 1987), and the Bayesian information criterion (BIC; Schwarz, 1978).

Table 3.

Rasch Analysis Model Comparison.

CSS subscale	Model	Deviance	AIC	BIC
Danger	PCM	18,859	18,909	19,038
	RSM	19,015	19,035	19,087
Socioeconomic Consequences	PCM	17,899	17,949	18,078
	RSM	18,103	18,123	18,174
Xenophobia	PCM	14,517	14,567	14,696
	RSM	14,630	14,650	14,701
Contamination	PCM	17,986	18,036	18,165
	RSM	18,154	18,174	18,226
Traumatic Stress	PCM	17,117	17,167	17,296
	RSM	17,231	17,251	17,302
Compulsive	PCM	19,851	19,901	20,029 *
Checking	RSM	19,956	19,976	20,027

Except for the Compulsive Checking subscale where two out of the three indices suggest PCM.

Local Independence

To assess local independence for the Rasch analysis, we utilized the effect size of the model fit MAD_aQ3 measure, which represents the average of the absolute values of Q3, for this measure we anticipate values closer to zero which would support local independence (Yen, 1984). The MADaQ3 was ranged from 0.045 to 0.118 for all the subscales (p < .001) indicating local independence.

Item Fit

In Rasch analysis, the purpose of item fit is to identify if the data fit the Rasch model (Linacre, 2002). Linacre (2002) provides guidelines for the item fit measures Infit and Outfit (0.5–1.5). Each of the five scales were examined individually as our previous exploration using CFA indicated the subscales should be examined independently; however, none of the scales had an Infit and Outfit for any item that exceeded the cutoffs provided by Linacre (2002) as can be seen in Table 4. Paying special attention to the scales of Danger and Contamination due to the issues previously discussed in the literature review we found that item fit was not an issue for either of these scales indicating that the items belong to their respective scales. In Table 4 we have also sorted the items within each subscale from the item easiest to endorse to the least difficult to endorse. For example, the highest Infit value was observed in the Danger subscale. The item easiest to endorse (the item that caused the least stress) by participants was Item 6 “I am worried that social distancing is not enough to keep me safe from the virus” at 1.171 and the most difficult to endorse was Item 4 “I am worried that I can’t keep my family safe from the virus” with the lowest Infit 0.877, suggesting minor variation in item response behavior but no significant misfit. Overall, Rasch analysis confirmed model fit, with all items falling within acceptable infit/outfit thresholds.

Table 4.

Item Fit Statistics for All Scales.

CSS subscale	Item	Measure	SE Measure	Infit	Outfit
	6	0.25	0.039	1.171	1.172
	2	0.22	0.039	1.004	0.995
Danger	1	−0.03	0.039	0.877	0.903
	3	−0.18	0.039	1.009	0.982
	5	−0.24	0.039	0.988	0.96
	4	−0.34	0.039	0.991	0.975
Socioeconomic	1	1.28	0.039	0.823	0.799
Consequences	6	1.15	0.039	1.022	0.967
	4	1.13	0.039	1.004	0.912
	3	0.97	0.039	0.997	0.951
	2	0.92	0.039	0.841	0.779
	5	0.04	0.040	1.238	1.266
Xenophobia	4	2.21	0.047	0.974	0.730
	5	2.15	0.047	1.012	0.817
	1	2.08	0.047	0.919	0.862
	3	1.97	0.047	0.891	0.784
	2	1.87	0.047	0.852	0.796
	6	1.83	0.047	0.926	0.908
Contamination	6	1.12	0.041	1.164	1.146
	5	0.61	0.041	0.956	0.923
	4	0.51	0.041	0.858	0.854
	1	0.29	0.041	1.006	1.000
	2	0.26	0.041	0.811	0.802
	3	−0.23	0.041	1.215	1.181
Traumatic	2	1.79	0.042	1.043	0.946
Stress	6	1.71	0.042	0.949	0.888
	4	1.52	0.042	1.058	0.985
	1	1.49	0.042	0.881	0.846
	5	1.33	0.042	0.974	0.907
	3	0.90	0.041	1.173	1.184
Compulsive	6	0.87	0.034	0.977	0.959
Checking	5	0.81	0.033	0.968	0.955
	2	0.80	0.033	0.952	0.94
	3	0.49	0.033	0.878	0.886
	4	0.20	0.034	1.085	1.084
	1	0.07	0.034	1.162	1.165

Person-Item Map

The Person-Item Maps for all the subscales of the CSS can be found in Figure 1. In Figure 1, from left to right we can see the Person-Item Maps for the danger, socioeconomic consequences, and xenophobia are in the first row and Contamination, Traumatic Stress, and Compulsive Checking in the second row. In each Person-Item Map, on the left we have the respondent’s trait and to the right of the graph we have item difficulties for all subscales. The Person-Item Maps indicated that participant trait distribution (pandemic stress levels) exceeded the range of item difficulties. We can particularly see this in the Xenophobia scale which showed a slight right skewness. This suggests that for a number of participants may experience stress beyond what the CSS is able to capture, particularly in socio-political contexts where pandemic-related xenophobia is heightened. The rest of the scales showed an approximately normal distribution of participants traits (e.g., stress depending on the subscale trait).

Figure 1.

Person-Item Maps clockwise for the Danger, Socioeconomic Consequences, Xenophobia, Contamination, Traumatic Stress, and Compulsive Checking.

Differential Item Functioning

Results of a DIF analysis of the CSS across gender classification (female vs. male) indicated that most CSS items demonstrated negligible DIF. However, ΔR² Nagelkerge values revealed evidence of moderate uniform DIF for Compulsive Checking Item 1 (i.e., “checked social media posts concerning COVID-19”). Examination of the item characteristic curve indicated that, after controlling for the underlying latent trait measured by the compulsive checking items, female participants were more likely than male participants to report checking social media for posts related to COVID-19. The results of the DIF analysis are presented in Table 5.

Table 5.

Differential Item Functioning Analysis Results for CSS Subscales.

CSS subscale	Item	Uniform DIF		Total DIF		Non-uniform DIF
	Item	x²p-value	ΔR²	x²p-value	ΔR²	x²p-value	ΔR²
Danger	1	.88	.000	.26	.001	.10	.001
	2	.48	.000	.29	.000	.16	.000
	3	<.001	.004	<.001	.005	.12	.000
	4	.20	.000	.30	.000	.38	.000
	5	<.001	.006	<.001	.008	.002	.002
	6	.60	.000	.81	.000	.69	.000
Socioeconomic Consequences	1	.05	.000	.02	.001	.05	.000
	2	.44	.000	.74	.000	.87	.000
	3	.31	.000	.59	.000	.95	.000
	4	<.001	.004	<.001	.005	.02	.001
	5	<.001	.008	<.001	.009	.09	.001
	6	.007	.001	<.001	.007	<.001	.006
Xenophobia	1	.29	.000	.31	.000	.26	.000
	2	.08	.000	.12	.000	.30	.000
	3	.43	.000	.59	.000	.49	.000
	4	.75	.000	.46	.000	.23	.000
	5	.02	.001	.07	.001	.51	.000
	6	.02	.001	.07	.001	.83	.000
Contamination	1	.36	.000	.58	.000	.60	.000
	2	.76	.000	.90	.000	.73	.000
	3	<.001	.011	<.001	.017	<.001	.005
	4	.89	.000	.24	.000	.09	.000
	5	.11	.000	.17	.000	.32	.000
	6	<.001	.003	.001	.003	.58	.000
Traumatic Stress	1	.88	.000	.98	.000	.97	.000
	2	.35	.000	.14	.001	.08	.000
	3	<.001	.016	<.001	.016	.80	.000
	4	.50	.000	.03	.001	.01	.001
	5	.13	.000	.14	.001	.20	.000
	6	.79	.000	.37	.000	.16	.000
Compulsive Checking	1	<.001	.045	<.001	.047	.09	.001
	2	<.001	.010	<.001	.014	<.001	.003
	3	<.001	.005	<.001	.007	.02	.002
	4	.01	.003	<.001	.014	<.001	.011
	5	<.001	.010	<.001	.012	.004	.002
	6	.02	.001	<.001	.004	.001	.002

Note. Corrected p-value indicating a significant x² test = .01.

Concurrent Validity

To assess concurrent validity, we examined the Pearson Product Moment Correlation coefficients between the PHQ-4 and CSS scales. The CSS demonstrated strong concurrent validity, with significant correlations between CSS and the PHQ-4 subscales (Total: r = .37, p < .001; Depression: r = .30, p < .001; Anxiety: r = .36, p < .001; Table 6 for full details). These findings indicate that higher pandemic-specific stress is associated with greater general psychological distress. We also examined the correlational relationships between well-being measures and the CSS. The CSS scales were correlated negatively with the SWLS, the BRCS, and the CAMS-R, with correlation coefficients ranging from small to moderate. It is important to note that the strongest correlation was observed between the Traumatic Stress subscale and PHQ-4 total score (r = .49, p < .001). This finding could potentially indicate the need for trauma-informed mental health interventions.

Table 6.

Concurrent Validity Pearson Correlation Coefficients.

Subscale	PHQ-4 Depression	PHQ-4 Anxiety	PHQ-4 Total	DDS	SWLS	BRCS	CAMS-R
Danger	0.30*	0.36*	0.37*	0.20*	−0.20*	−0.08*	−0.27*
Socioeconomic Consequences	0.26*	0.23*	0.27*	0.42*	−0.30*	−0.31*	−0.25*
Xenophobia	0.25*	0.15*	0.22*	0.44*	−0.33*	−0.41*	−0.20*
Contamination	0.31*	0.32*	0.35*	0.32*	−0.24*	−0.26*	−0.30*
Traumatic Stress	0.40*	0.48*	0.49*	0.42*	−0.28*	−0.32*	−0.31*
Compulsive Checking	0.28*	0.35*	0.35*	0.26*	−0.20*	−0.10*	−0.23*

Indicates p < .001.

Discussion

It is well established in literature that pandemics are associated with higher levels of psychological distress (e.g., Wang et al., 2020). Indeed, the most recent COVID-19 pandemic has been associated with pandemic-specific stress, anxiety, depression, and traumatic stress worldwide (e.g., Pieh et al., 2020; Qiu et al., 2020; Rehman et al., 2021).

Similarly to previous research utilizing the CSS had encountered issues with the Danger and Contamination subscales along with discrepancies between six or five factor solutions (Mahamid et al., 2022; Mertens et al., 2021). Our results indicated that both a six and a five-factor model fit the data, however, we opted for a six-factor model rather than five-factor model based on interpretability and to align with the original development of the scale. Taylor et al. (2020) in their initial CSS paper, elect to keep the two scales (Danger and Contamination) that loaded on the same factor as separate scales. The authors explained their reasoning arguing they suspected that others may wish to examine Danger and Contamination separately. Despite frequent factor merging in prior studies (Mertens et al., 2021), Rasch analysis in our sample showed distinct item difficulty calibrations between the Contamination and Danger scales, supporting their separation. These discrepancies among findings for the Contamination and Danger scales throughout CSS studies may be explained by data collection occurring at different phases of the pandemic or may be related to the Contamination and Danger scales being closely related, which has been proposed by other researchers (e.g., Mahamid et al., 2022; Mertens et al., 2021). Regardless, the accumulation of research showing the merging of the Danger and Contamination scales perhaps indicates that our interpretation of these subscales should change accordingly. Furthermore, our paper was the first to examine the CSS through a Rasch analysis framework, through the item fit exploration we found no issues with the items misfitting in their respective subscales.

In terms of differential item functioning, our results showed that most CSS items did not show evidence of response bias across male and female participants. The general lack of DIF effects is consistent with the findings of Şahin et al. (2022) which suggested that the CSS instrument demonstrated gender invariance in a sample of Turkish adults. However, it is important to note that our DIF analysis did identify one item that might not be functioning as intended. However, one item from the Compulsive Checking subscale—Item 1, which assesses the tendency to check social media for posts related to COVID-19—demonstrated a moderate level of uniform DIF. Specifically, female participants were more likely than males to endorse this item, even after controlling for the underlying latent trait measured by the subscale. Prior empirical research has provided evidence of gender differences in the tendency to utilize social media to obtain health-related information as well as differences in the type of health-oriented information sought out by males and females on social media platforms (Bidmon & Terlutter, 2015; Hassan & Masoud, 2021; Krasnova et al., 2017). Thus, this item may be capturing general information-seeking tendencies rather than specific compulsive social media checking behaviors related to COVID-19.

As we anticipated, our results indicated that anxiety and depression (PHQ-4) positively correlated with the CSS. As highlighted within the CSS, pandemic-specific distress is often associated with fear and other negative emotions, thus, a positive relationship with anxiety and depression is to be expected. In contrast, well-being indicators such as life satisfaction, resilience, and trait mindfulness were negatively correlated with the CSS in our study. This finding highlights that well-being indicators should be taken into consideration alongside distress to capture the breadth of an individual’s mental health more fully (Carver et al., 2021; Keyes, 2005). For example, an individual who reports high levels of pandemic-related distress may also simultaneously report high levels of resilience and life satisfaction. Intervention needs for someone with high levels of pandemic-related distress and high levels of well-being will be different than someone who has high levels of pandemic-related distress and low levels of well-being. Thus, assessing for indicators of well-being in addition to screening for pandemic-related distress could assist those in clinical practice with formulating individualized intervention targets. Without timely identification and intervention, what may begin as pandemic-specific distress can lead to more severe or chronic symptoms of anxiety and depression (Fofana et al., 2020).

Limitations and Future Directions

There are several limitations to the present study First, the sample of this study was recruited entirely from social media outlets (Twitter, now X, and Facebook), which is not representative of the entire U.S. population, as some individuals do not use social media or have internet access. For example, internet and social media use are impacted by variables such as homelessness and socio-economic status (Guadagno et al., 2013). Second, all data were self-reported, increasing the risk of social desirability bias. Participants may have underreported distress due to stigma surrounding mental health, particularly among male respondents. Future research should conduct longitudinal assessments to determine whether CSS scores remain stable over time, particularly as pandemic stressors shift post-COVID. Longitudinal assessments would also help determine potential effects of long-COVID on responses to the CSS, something that was not identified nor able to be explored in the present study. Third, additional assessments of concurrent validity to the CSS would be helpful to include in future research. For example, PHQ-4 is a strength of the study in terms of its brevity in screening for clinically meaningful anxiety and depression symptoms that was used in the initial validation paper for the CSS (Taylor et al., 2020), other concurrent measures of depression and anxiety would be helpful to include alongside PHQ-4 in future research.

Clinical Implications and Conclusion

The present study contributed to the literature by including measures of well-being along with measures of distress in examining the psychometric properties of the CSS. The data was collected during the height of the pandemic (summer 2020) and the findings have the potential to offer a more holistic view of mental health during the pandemic. The COVID-19 pandemic was just one of the most recent global pandemics; however, in 2024 the World Health Organization (WHO) keeps track of diseases which have the potential to become an epidemic (e.g., acute watery diarrhea [AWD], cholera, malaria among others). Thus, development of instruments to assess pandemic specific stress is useful for public health officials and clinicians. Understanding an individual’s experience of pandemic-related distress in conjunction with other forms of psychological distress that may already be routinely screened for (e.g., anxiety and depression), and in parallel with well-being indicators, can help those in clinical practice guide support and intervention needs. For example, in clinical settings, the CSS can be paired with resilience measures (e.g., BRCS) to tailor interventions. For instance, individuals scoring high on CSS but also demonstrating high resilience may benefit from mindfulness-based interventions, whereas those with low resilience may require cognitive restructuring techniques. The addition of a pandemic-related distress screening such as the CSS and measures of well-being to routine assessment practices could prove beneficial, not only as society continues to recover from the COVID-19 pandemic, but also in the event of future pandemics.

Footnotes

ORCID iDs

Samantha Estrada

Christopher Thomas

Ethical Considerations

This study was reviewed and approved by the University of Texas at Tyler Institutional Review Board.

Consent to Participate

Participants were informed about the purpose of the study, their rights as participants, and the voluntary nature of their participation before beginning the online survey. Survey responses were collected anonymously to protect participants confidentiality.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study. Data is available upon request.

References

Abbady

A. S.

El-Gilany

A. H.

El-Dabee

F. A.

Elsadek

A. M.

ElWasify

Elwasify

(2021). Psychometric characteristics of the of COVID Stress Scales-Arabic version (CSS-Arabic) in Egyptian and Saudi university students. Middle East Current Psychiatry, Ain Shams University, 28(1), 14. https://doi.org/10.1186/s43045-021-00095-8

Adamczyk

Clark

D. A.

Pradelok

(2021). The Polish COVID Stress scales: Considerations of psychometric functioning, measurement invariance, and validity. PLoS One, 16(12), e0260459. https://doi.org/10.1371/journal.pone.0260459

Ahorsu

D. K.

Lin

C. Y.

Imani

Saffari

Griffiths

M. D.

Pakpour

A. H.

(2022). The fear of COVID-19 Scale: Development and initial validation. International Journal of Mental Health and Addiction, 20, 1537–1545. https://doi.org/10.1007/s11469-020-00270-8

Akaike

(1987). Factor analysis and AIC. Psychometrika, 52(3), 317–332. https://doi.org/10.1007/BF02294359

Andrich

(1978). Relationships between the Thurstone and Rasch approaches to item scaling. Applied Psychological Measurement, 2(3), 451–462. https://doi.org/10.1177/014662167800200319

Andrich

(1988). Rasch models for measurement. Sage Publications. https://doi.org/10.4135/9781412985598

Asmundson

G. J. G.

Paluszek

M. M.

Landry

C. A.

Rachor

G. S.

McKay

Taylor

(2020). Do pre-existing anxiety-related and mood disorders differentially impact COVID-19 stress responses and coping? Journal of Anxiety Disorders, 74, 102271. https://doi.org/10.1016/j.janxdis.2020.102271

Bidmon

Terlutter

(2015). Gender differences in searching for health information on the internet and the virtual patient-physician relationship in Germany: Exploratory results on how men and women differ and why. Journal of Medical Internet Research, 17(6), e156. https://doi.org/10.2196/jmir.4127

Boone

W. J.

(2016). Rasch analysis for instrument development: Why, when, and how? CBE Life Sciences Education, Advance online publication. https://doi.org/10.1187/cbe.16-04-0148

10.

Boone

W. J.

Staver

J. R.

Yale

M. S.

(2013). Rasch analysis in the human sciences. Springer Science & Business Media.

11.

Carver

Ismail

Reed

Hayes

Alsaif

Villanueva

Sass

(2021). High levels of anxiety and psychological well-being in college students: A dual factor model of mental health approach. Journal of Positive School Psychology, 5(1), 32–41. https://doi.org/10.47602/jpsp.v5i1.242

12.

Chen

Liang

Yin

Zhou

Gao

(2021). The factor structure and Rasch analysis of the fear of COVID-19 scale (FCV-19S) among Chinese students. Frontiers in Psychology, 12, 678979. https://doi.org/10.3389/fpsyg.2021.678979

13.

Chiu

M. Y. L.

Wong

H. T.

W. W. N.

(2020). A comparative study of confirmatory factor analysis and Rasch analysis as item reduction strategies for SAMHSA recovery inventory for Chinese (SAMHSA-RIC). European Journal of Psychiatry, 34(2), 74–81.

14.

Choi

S. W.

Gibbons

L. E.

Crane

P. K.

(2016). lordif: logistic ordinal regression differential item functioning using IRT (Version 0.3-3) [Computer software]. R Core Team (Version 4.2.0). https://CRAN.R-project.org/package=lordif

15.

Christensen

K. B.

Engelhard

Salzberger

(2012). Ask the Experts: Rasch vs. factor analysis. Rasch Measurement Transactions, 26(3), 1373–1378. https://www.rasch.org/rmt/rmt263a.htm

16.

Conversano

Di Giuseppe

Miccoli

Ciacchini

Gemignani

Orrù

(2020). Mindfulness, age and gender as protective factors against psychological distress during COVID-19 pandemic. Frontiers in Psychology, 11, 1900. https://doi.org/10.3389/fpsyg.2020.01900

17.

De Ayala

R. J.

(2009). The theory and practice of item response theory. Guilford Press.

18.

Delgado-Gallegos

J. L.

Montemayor-Garza

R. J.

Padilla-Rivas

G. R.

Franco-Villareal

Islas

J. F.

(2020). Prevalence of stress in healthcare professionals during the COVID-19 pandemic in Northeast Mexico: A remote, fast survey evaluation, using an adapted COVID-19 stress scales. International Journal of Environmental Research and Public Health, 17, 7624. https://doi.org/10.3390/ijerph17207624

19.

Desjardins

C. D.

Bulut

(2018). Handbook of educational measurement and psychometrics using R. CRC Press.

20.

Devakumar

Shannon

Bhopal

S. S.

Abubakar

(2020). Racism and discrimination in COVID-19 responses. Lancet, 395(10231), 1194. https://doi.org/10.1016/s0140-6736(20)30792-3

21.

Diener

Emmons

R. A.

Larsen

R. J.

Griffin

(1985). The satisfaction with life scale. Journal of Personality Assessment, 49(1), 71–75. https://doi.org/10.1207/s15327752jpa4901_13

22.

Dima

A. L.

(2018). Scale validation in applied health research: Tutorial for a 6-step R-based psychometrics protocol. Health Psychology and Behavioral Medicine, 6(1), 136–161. https://doi.org/10.1080/21642850.2018.1472602

23.

Duckering

(2022). The relationship between COVID-19 stress, psychological inflexibility, and psychological well-being. (Publication No. 1331) [Doctoral dissertation, California State University San Bernardino]. https://scholarworks.lib.csusb.edu/etd/1331

24.

Elliott

K. S.

Nabulsi

E. H.

Sims-Rhodes

Dubre

Barena

Yuen

Morris

Sass

S. M.

Kennedy

Singh

K. P.

(2024). Modality and terminology changes for behavioral health service delivery during the COVID-19 pandemic: A systematic review. Frontiers in Psychiatry, 14, 1265087. https://doi.org/10.3389/fpsyt.2023.1265087

25.

Feldman

Hayes

Kumar

Greeson

Laurenceau

J. P.

(2007). Mindfulness and emotion regulation: The development and initial validation of the cognitive and Affective Mindfulness Scale-Revised (CAMS-R). Journal of Psychopathology and Behavioral Assessment, 29(3), 177–190. https://doi.org/10.1007/s10862-006-9035-8

26.

Fofana

N. K.

Latif

Sarfraz

Bilal

Bashir

M. F.

Komal

(2020). Fear and agony of the pandemic leading to stress and mental illness: An emerging crisis in the novel coronavirus (COVID-19) outbreak. Psychiatry Research, 291, 113230. https://doi.org/10.1016/j.psychres.2020.113230

27.

Gavin

Hayden

Adamis

McNicholas

(2020). Caring for the psychological well-being of healthcare professionals in the COVID-19 pandemic crisis. Irish Medical Journal, 113(4), 51.

28.

George

Mallery

(2019). IBM SPSS statistics 26 step by step: A simple guide and reference. Routledge. https://doi.org/10.4324/9780429056765

29.

Gonzalez

(2023, May 5). WHO: COVID-19 is no longer global health emergency. Axios. https://www.axios.com/2023/05/05/covid-global-health-emergency-end-who

30.

Greenspoon

P. J.

Saklofske

D. H.

(2001). Toward an integration of subjective well-being and psychopathology. Social Indicators Research, 54, 81–108. https://doi.org/10.1023/a:1007219227883

31.

Gruber

Prinstein

M. J.

Clark

L. A.

Rottenberg

Abramowitz

J. S.

Albano

A. M.

Aldao

Borelli

J. L.

Chung

Davila

Forbes

E. E.

Gee

D. G.

Hall

G. C. N.

Hallion

L. S.

Hinshaw

S. P.

Hofmann

S. G.

Hollon

S. D.

Joormann

Kazdin

A. E.

… Weinstock

L. M.

(2021). Mental health and clinical psychological science in the time of COVID-19: Challenges, opportunities, and a call to action. American Psychologist, 76(3), 409–426. https://doi.org/10.1037/amp0000707

32.

Guadagno

R. E.

Muscanell

N. L.

Pollio

D. E.

(2013). The homeless use Facebook?! Similarities of social network use between college students and homeless young adults. Computers in Human Behavior, 29(1), 86–89. https://doi.org/10.1016/j.chb.2012.07.019

33.

Hassan

Masoud

(2021). Online health information seeking and health literacy among non-medical college students: Gender differences. Journal of Public Health, 29, 1267–1273. https://doi.org/10.1007/s10389-020-01243-w

34.

Hughes

S. E.

Haroon

Subramanian

McMullan

Aiyegbusi

O. L.

Turner

G. M.

Jackson

Davies

E. H.

Frost

McNamara

Price

Matthews

Camaradou

Ormerod

Walker

Calvert

M. J.

(2022). Development and validation of the symptom burden questionnaire for long Covid (SBQ-LC): Rasch analysis. BMJ, 377, e070230. https://doi.org/10.1136/bmj-2022-070230

35.

IBM Corp. (2016). SPSS statistics for Windows (Version 24.0) [Computer software]. IBM Corp.

36.

Jamovi Project. (2021). jamovi (Version 1.6) [Computer software]. https://jamovi.org

37.

Jodoin

M. G.

Gierl

M. J.

(2001). Evaluating type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14(4), 329–349. https://doi.org/10.1207/S15324818AME1404_2

38.

Kathirvel

(2020). Post COVID-19 pandemic mental health challenges. Asian Journal of Psychiatry, 53, 102430. https://doi.org/10.1016/j.ajp.2020.102430

39.

Keyes

C. L.

(2005). Mental illness and/or mental health? Investigating axioms of the complete state model of health. Journal of Consulting and Clinical Psychology, 73(3), 539–548. https://doi.org/10.1037/0022-006X.73.3.539

40.

Khosravani

Asmundson

G. J. G.

Taylor

Sharifi Bastan

Samimi Ardestani

S. M.

(2021). The Persian COVID stress scales (Persian-CSS) and COVID-19-related stress reactions in patients with obsessive-compulsive and anxiety disorders. Journal of obsessive-compulsive and related disorders, 28, 100615. https://doi.org/10.1016/j.jocrd.2020.100615

41.

Kocalevent

R. D.

Zenger

Hinz

Klapp

Brähler

(2017). Resilient coping in the general population: Standardization of the brief resilient coping scale (BRCS). Health and Quality of Life Outcomes, 15(1), 251. https://doi.org/10.1186/s12955-017-0822-6

42.

Krasnova

Veltri

N. F.

Eling

Buxmann

(2017). Why men and women continue to use social networking sites: The role of gender differences. Journal of Strategic Information Systems, 26(4), 261–284. https://doi.org/10.1016/j.jsis.2017.01.004

43.

Kroenke

Spitzer

R. L.

Williams

J. B. W.

Löwe

(2009). An ultra-brief screening scale for anxiety and depression: The PHQ–4. Psychosomatics, 50(6), 613–621. https://doi.org/10.1016/s0033-3182(09)70864-3

44.

Kubb

Foran

H. M.

(2020). Measuring COVID-19 related anxiety in parents: Psychometric comparison of four different inventories. JMIR Mental Health, 7(12), e24507. https://doi.org/10.2196/24507

45.

Lenzo

Quattropani

M. C.

Musetti

Zenesini

Freda

M. F.

Lemmo

Vegni

Borghi

Plazzi

Castelnuovo

Cattivelli

Saita

Franceschini

(2020). Resilience contributes to low emotional impact of the COVID-19 outbreak among the general population in Italy. Frontiers in Psychology, 11, 576485. https://doi.org/10.3389/fpsyg.2020.576485

46.

Linacre

J. M.

(2002). What do infit and outfit, mean-square and standardized mean. Rasch Measurement Transactions, 16(2), 878. https://www.rasch.org/rmt/rmt162f.htm

47.

Löwe

Wahl

Rose

Spitzer

Glaesmer

Wingenfeld

Schneider

Brähler

(2010). A 4-item measure of depression and anxiety: Validation and standardization of the Patient Health Questionnaire-4 (PHQ-4) in the general population. Journal of Affective Disorders, 122(1-2), 86–95. https://doi.org/10.1016/j.jad.2009.06.019

48.

Mahamid

F. A.

Veronese

Bdier

Pancake

(2022). Psychometric properties of the COVID Stress Scales (CSS) within Arabic language in a Palestinian context. Current Psychology, 41, 7431–7440. https://doi.org/10.1007/s12144-021-01794-5

49.

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. https://doi.org/10.1007/bf02296272

50.

Masters

G. N.

Wright

B. D.

(1997). The partial credit model. In Linden

W. J.

Hambleton

R. K.

(Eds.), Handbook of Modern Item Response Theory (pp. 101–121). Springer. https://doi.org/10.1007/978-1-4757-2691-6

51.

McKay

Yang

Elhai

Asmundson

G. J. G.

(2020). Anxiety regarding contracting COVID-19 related to interoceptive anxiety sensations: The moderating role of disgust propensity and sensitivity. Journal of Anxiety Disorders, 73, 102233. https://doi.org/10.1016/j.janxdis.2020.102233

52.

Mertens

Duijndam

Smeets

Lodder

(2021). The latent and item structure of COVID-19 fear: A comparison of four COVID-19 fear questionnaires using SEM and network analyses. Journal of Anxiety Disorders, 81, 102415. https://doi.org/10.1016/j.janxdis.2021.102415

53.

Mîndrilă

(2010). Maximum likelihood (ML) and diagonally weighted least squares (DWLS) estimation procedures: A comparison of estimation bias with ordinal and multivariate non-normal data. International Journal for Digital Society, 1(1), 60–66. https://doi.org/10.20533/ijds.2040.2570.2010.0010

54.

Montano

R. L. T.

Acebes

K. M. L.

(2020). Covid stress predicts depression, anxiety and stress symptoms of Filipino respondents. International Journal of Research in Business and Social Science (2147-4478), 9(4), 78–103. https://doi.org/10.20525/ijrbs.v9i4.773

55.

Pavot

Diener

(2008). The satisfaction with life scale and the emerging construct of life satisfaction. Journal of Positive Psychology, 3(2), 137–152. https://doi.org/10.1080/17439760701756946

56.

Pavot

Diener

Colvin

C. R.

Sandvik

(1991). Further validation of the Satisfaction with Life Scale: Evidence for the cross-method convergence of well-being measures. Journal of Personality Assessment, 57(1), 149–161. https://doi.org/10.1207/s15327752jpa5701_17

57.

Pennix

Richardson

Kumar

Bokhari

Aldajani

Amin

Sandhu

Jalaeian

Bhatia

(2022). Abstract No. 70 Radiation exposure reduction with operator experience in prostatic artery embolization: a single-institution study. Journal of Vascular and Interventional Radiology, 33(6), S34. https://doi.org/10.1016/j.jvir.2022.03.151

58.

Pheko

M. M.

Mhaka-Mutepfa

Mfolwe

L. M.

Mathoothe

K. D.

Chua

B. S.

Low

W. Y.

(2023). Psychometrics validation of the WHO-5 Wellbeing Index in Botswana, Zimbabwe, and Malaysia: Coping with COVID-19 pandemic in the Global South Study. Sage Open, 13(4), 21582440231198387. https://doi.org/10.1177/21582440231198387

59.

Pidgeon

A. M.

Keye

(2014). Relationship between resilience, mindfulness, and pyschological well-being in university students. International Journal of Liberal Arts and Social Science, 2(5), 27–32. https://research.bond.edu.au/en/publications/relationship-between-resilience-mindfulness-and-pyschological-wel

60.

Pieh

Budimir

Probst

(2020). The effect of age, gender, income, work, and physical activity on mental health during coronavirus disease (COVID-19) lockdown in Austria. Journal of Psychosomatic Research, 136, 110186. https://doi.org/10.1016/j.jpsychores.2020.110186

61.

Piret

Boivin

(2021). Pandemics throughout history. Frontiers in Microbiology, 11, 631736. https://doi.org/10.3389/fmicb.2020.631736

62.

Qiu

Shen

Zhao

Wang

Xie

(2020). A nationwide survey of psychological distress among Chinese people in the COVID-19 epidemic: Implications and policy recommendations. General Psychiatry, 33(2), e100213. https://doi.org/10.1136/gpsych-2020-100213

63.

Rachor

G. S.

Paluszek

M. M.

Taylor

Asmundson

G. J.

(2023). COVID Stress Scales (CSS). In Hilsenroth

Segal

D. L.

(Eds.), International handbook of behavioral health assessment (pp. 1–47). Springer International Publishing.

64.

Raiche

Magis

(2020). nFactors: Parallel analysis and other non-graphical solutions to the Cattell scree test. research (Version 2.4.1) [Computer software]. R Core Team (Version 4.2.0). https://CRAN.R-project.org/package=nFactors

65.

Rashid

Ostermann

R. F.

(2009). Strength-based assessment in clinical practice. Journal of Clinical Psychology, 65(5), 488–498. https://doi.org/10.1002/jclp.20595

66.

R Core Team. (2022). R: A language and environment for statistical computing (Version 4.2.0) [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/

67.

Rehman

Shahnawaz

M. G.

Khan

N. H.

Kharshiing

K. D.

Khursheed

Gupta

Kashyap

Uniyal

(2021). Depression, anxiety and stress among Indians in times of Covid-19 lockdown. Community Mental Health Journal, 57(1), 42–48. https://doi.org/10.1007/s10597-020-00664-x

68.

Revelle

W. R.

(2022). psych: Procedures for personality and psychological research (Version 2.2.5) [Computer software]. R Core Team (Version 4.2.0). https://CRAN.R-project.org/package=psych.

69.

Robitzsch

Kiefer

(2022). TAM: Test Analysis Modules. R package version 4.1-4, https://CRAN.R-project.org/package=TAM.

70.

Rodríguez-Rey

Garrido-Hernansaiz

Collado

(2020). Psychological impact and associated factors during the initial stage of the coronavirus (COVID-19) pandemic among the general population in Spain. Frontiers in Psychology, 11, 1540. https://doi.org/10.3389/fpsyg.2020.01540

71.

Rosseel

(2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02

72.

Şahin

M. D.

Sen

Güler

(2022). Adaptation and psychometric evaluation of the COVID-19 stress scales in Turkish sample. International Journal of Assessment Tools in Education, 9(3), 593–612. https://doi.org/10.21449/ijate.1067542

73.

Saltzman

L. Y.

Hansel

T. C.

Bordnick

P. S.

(2020). Loneliness, isolation, and social support factors in post-COVID-19 mental health. Psychological Trauma Theory Research Practice and Policy, 12(S1), S55–S57. https://doi.org/10.1037/tra0000703

74.

Sánchez-Teruel

Robles-Bello

M. A.

(2021). The COVID-19 Fear Scale (FCV-19S): Psychometric properties and invariance of the measure in the Spanish version. Actas Espanolas de Psiquiatria, 49(3), 96–105. https://actaspsiquiatria.es/index.php/actas/article/view/320

75.

Schwarz

(1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464. https://doi.org/10.1214/aos/1176344136

76.

Seol

(2023). SnowIRT (Version 4.9.0) [Computer Software]. https://github.com/hyunsooseol/snowIRT/

77.

Sinclair

V. G.

Wallston

K. A.

(2004). The development and psychometric evaluation of the Brief Resilient Coping Scale. Assessment, 11(1), 94–101. https://doi.org/10.1177/1073191103258144

78.

Soklaridis

Lin

Lalani

Rodak

Sockalingam

(2020). Mental health interventions and supports during COVID-19 and other medical pandemics: A rapid systematic review of the evidence. General Hospital Psychiatry, 66, 133–146. https://doi.org/10.1016/j.genhosppsych.2020.08.007

79.

Solomou

Nikolaou

Michaelides

M. P.

Constantinidou

(2024). Long-term psychological impact of the pandemic COVID-19: Identification of high-risk groups and assessment of precautionary measures five months after the first wave of restrictions was lifted. PLOS Global Public Health, 4(2), e0002847. https://doi.org/10.1371/journal.pgph.0002847

80.

Stănculescu

(2022). Fear of COVID-19 in Romania: Validation of the Romanian version of the fear of COVID-19 scale using graded response model analysis. International Journal of Mental Health and Addiction, 20(2), 1094–1109. https://doi.org/10.1007/s11469-020-00428-4

81.

Streiner

D. L.

(2010). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80(1), 99–103. https://doi.org/10.1207/S15327752JPA8001_18

82.

Taylor

Landry

C. A.

Paluszek

M. M.

Fergus

T. A.

McKay

Asmundson

G. J. G.

(2020). Development and initial validation of the COVID Stress Scales. Journal of Anxiety Disorders, 72, 102232. https://doi.org/10.1016/j.janxdis.2020.102232

83.

Tsamakis

Rizos

Manolis

A. J.

Chaidou

Kympouropoulos

Spartalis

Triantafyllis

A. S.

(2020). [Comment] COVID-19 pandemic and its impact on mental health of healthcare professionals. Experimental and Therapeutic Medicine, 19(6), 3451–3453. https://doi.org/10.3892/etm.2020.8646

84.

Ullah

Tahir

M. J.

Ali

Waseem

Griffiths

M. D.

Mamun

M. A.

Lin

C. Y.

Pakpour

A. H.

(2023). COVID-19 fear among Pakistanis: Psychometric evaluation of the fear of COVID-19 scale using Item response theory and confirmatory factor analysis. International Journal of Mental Health and Addiction, 21(2), 1240–1255.

85.

Vallejo-Medina

Sierra

J. C.

(2015). Adaptation and validation of the sexual assertiveness scale (SAS) in a sample of male drug users. Spanish Journal of Psychology, 18, E21. https://doi.org/10.1017/sjp.2015.25

86.

Vindegaard

Benros

M. E.

(2020). COVID-19 pandemic and mental health consequences: Systematic review of the current evidence. Brain, Behavior, and Immunity, 89, 531–542. https://doi.org/10.1016/j.bbi.2020.05.048

87.

Wang

Pan

Wan

Tan

C. S.

R. C.

(2020). Immediate psychological responses and associated factors during the initial stage of the 2019 coronavirus disease (COVID-19) epidemic among the general population in China. International Journal of Environmental Research and Public Health, 17(5), 1729. https://doi.org/10.3390/ijerph17051729

88.

World Health Organization. (2023, May 5). Statement on the fifteenth meeting of the IHR Emergency Committee on the COVID-19 pandemic. https://www.who.int/news/item/05-05-2023-statement-on-the-fifteenth-meeting-of-the-ihr-emergency-committee-on-the-covid-19-pandemic

89.

World Health Organization. (2024). Epidemic and pandemic-prone diseases. https://www.emro.who.int/pandemic-epidemic-diseases/outbreaks/index.html

90.

Yen

W. M.

(1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125–145. https://doi.org/10.1177/014662168400800201

91.

Yun

Y. H.

Rhee

Y. E.

Kang

Sim

J. A.

(2019). The satisfaction with life scale and the subjective well-being inventory in the general Korean population: Psychometric properties and normative data. International Journal of Environmental Research and Public Health, 16(9), 1538. https://doi.org/10.3390/ijerph16091538

Factor Structure and Rasch Analysis of the COVID Stress Scale: Evaluating Competing Factor Models

Abstract

Plain language summary

Keywords

Introduction

COVID Stress Scale

Factor Structure

Rasch Analysis

Differential Item Functioning

Well-Being Measures

Purpose and Rationale

Research Questions

Method

Ethical Approval

Sampling and Data Collection Procedures

Participants

Measures

COVID Stress Scales

Patient Health Questionnaire-4

Brief Resilient Coping Scale

Satisfaction with Life Scale

The Cognitive and Affective Mindfulness Scale-Revised

Data Analysis

Results

Confirmatory Factor Analysis (CFA)

Internal Consistency Reliability

Rasch Analysis

Local Independence

Item Fit

Person-Item Map

Differential Item Functioning

Concurrent Validity

Discussion

Limitations and Future Directions

Clinical Implications and Conclusion

Footnotes

ORCID iDs

Ethical Considerations

Consent to Participate

Funding

Declaration of Conflicting Interests

Data Availability Statement

References