Psychometric Properties and Validation of the Korean Version of Stress Mindset Measure

Abstract

The factor structure of the Stress Mindset Measure (SMM), an 8-item questionnaire, has been explored in several countries, showing inconsistent results. This study aimed to examine the psychometric properties of the SMM with a Korean sample. A total of 912 participants completed the online survey (women = 59.3%) at T1, and a random subsample selected from T1 sample (n = 259, women = 62.9%) participated in the survey again at T2. At both time points, the results of confirmatory factor analyses supported a three-factor hierarchical model with three lower-order factors (i.e., learning and growth, performance and productivity, and health and vitality) and one higher-order general factor with correlated errors among three negatively worded items. In the model, 2 items measuring the “general” domain were deleted based on high correlations with the higher-order factor. Longitudinal factor analysis revealed that the revised factor structure and loadings were invariant over 6-month period. Theoretical validity was supported through significant correlations between the total score of the 6-item SMM and other stress-related measures. Overall, our results indicate that the Korean version of the SMM is a valid instrument and suggest using a total score of 6 items.

Keywords

stress mindset Stress Mindset Measure psychometric properties confirmatory factor analysis

Introduction

Stress is a state of homeostasis being challenged (Lu et al., 2021). It is believed that stress has inherent negative influences on physical and mental health as well as performance (e.g., McEwen & Seeman, 1999; Mroczek & Almeida, 2004). Therefore, stress researchers have focused on effects of stress and how to deal with stress. For instance, Cohen et al. (1983) widely used Perceived Stress Scale implied that stress had negative connotations. The scale asks respondents the degree to which they perceive their lives to be burdensome, out of control, and uncertain (Cohen et al., 1983). A variety of researchers have examined how people cope with stressful situations (e.g., Folkman & Lazarus, 1980). In contrast, several studies have demonstrated that some can experience benefits from stressful experiences (Schneiderman et al., 2005). Specifically, stress was linked to positive outcomes like personal initiative (Fay & Sonnentag, 2002) and health (Quick et al., 2014). Accordingly, a stress belief started to receive attention as a factor that changes the influences of stress.

Crum et al. (2013) adopted the concepts of mindset to stress based on mindsets in a realm of intelligence (Dweck et al., 1995) and aging (Levy et al., 2002). Stress mindset refers to how one evaluates the nature of stress, whether stress has enhancing or debilitating consequences (Crum et al., 2013). Stress-is-enhancing mindset is about the tendency to evaluate stress has enhancing effects while stress-is-debilitating mindset is about the tendency to perceive stress has debilitating effects (Jamieson et al., 2018). In Crum et al., (2013), stress mindset was distinguished from relevant variables including amount of stress, stress appraisals, and coping. Moreover, stress mindset had significant influence on psychological symptoms, work performance, and life satisfaction over and above coping and amount of stress (Crum et al., 2013). In addition, studies suggested that stress mindsets can be changed by rather a simple intervention such as watching video clips and that having a stress-is-enhancing mindset produced positive changes in psychological symptoms, work performance, and growth hormones (Crum et al., 2013, 2017). Huebschmann and Sheets (2020) also demonstrated that having a stress-is-enhancing mindset mitigated the negative effects of perceived stress on depressive symptoms and anxiety. In a study with Navy SEAL candidates (Smith et al., 2020), having an enhancing mindset predicted better performance and evaluations of peers and instructors.

To assess the stress mindset, Crum et al. (2013) developed and validated the Stress Mindset Measure (SMM), a self-report, 8-item instrument. Crum et al. (2013) generated items through a focus group in the four domains (i.e., learning and growth, performance and productivity, health and vitality, and uncertainty and change). Then, the scale was administered to three pilot samples. As the result, 2 items for uncertainty and change domain were deleted for internal consistency, leaving 8 items. The final items measure the consequences of stress in general (2 items) and across three domains (i.e., learning and growth, performance and productivity, and health and vitality; 2 items each for a domain). Stress mindset scores were calculated by reversing the four negative items which represent stress-is-debilitating mindset and taking the mean of 8 items. It implied that the scale is unidimensional having stress-is-debilitating mindset on the opposite of stress-is-enhancing mindset. Confirmatory factor analysis supported this unifactorial structure (Crum et al., 2013).

However, the unidimensional structure of the SMM has not been replicated in other countries. Iwamoto et al. (2020) and Karampas et al. (2020) tested two models, a one-factor model and a two-factor model, with Japanese and Greek samples respectively. A two-factor model, with four reversed items loaded on the stress-is-debilitating factor and four positive items loaded on the stress-is-enhancing factor, showed a better model fit in both studies (Iwamoto et al., 2020; Karampas et al., 2020). Iwamoto et al. (2020) and Karampas et al. (2020) suggested that there are two distinct stress mindsets and that the four negatively worded items to measure the stress-is-debilitating mindset could harm unidimensionality. Vigil-Colet et al. (2020) recommended controlling for response bias when positively and negatively worded items are included in the same measure.

In contrast, a study with a Polish population reported unsatisfactory model fits for both one- and two-factor models (Mierzejewska-Floreani et al., 2022). Mierzejewska-Floreani et al. (2022) further tested four-factor model, bi-factor model, and hierarchical model across two studies. The results supported a four-factor hierarchical model with four lower-order factors (i.e., general, learning and growth, performance and productivity, and health and vitality). The structure was also confirmed in test-retest sample after 10 months. Nevertheless, Mierzejewska-Floreani et al. (2022) questioned the equal property of the general lower-order factor to other three lower-order factors given that the contents of 2 items for the general factor were unspecified (e.g., “The effects of stress are negative and should be avoided.”).

In South Korea, Park (2022) has recently examined the factor structure of the SMM. The study (Park, 2022) reported that the one-factor structure, two-factor structure, and two-factor models considering method effects showed unsatisfactory fits. Instead, Park (2022) found adequate fits for a model with four negatively worded items and a model with four positively worded items. Thus, Park (2022) suggested to use a measure consisting of four negative items which showed superior fits to a model consisting of four positive items. Although it was the first study to investigate the psychometric properties of the SMM in a Korean sample, it was not free from limitations. Park (2022) interpreted the results to reflect the independence between the negative perception of stress and the positive perception of stress. However, in the two factor model, the negative perception construct and the positive perception construct showed a high correlation (r = 0.86, p < .001), questioning the independent nature of the relationships. Assuming the independence between the debilitating and enhancing mindset is conflicting with Crum et al. (2013)’s unidimensional suggestion and other studies’ moderate to high correlations between the factors. The correlation between the two factors were −51 (p < .001) in Karampas et al. (2020) and −41 to −0.60 (p < .01) in Iwamoto et al. (2020). In addition, to suggest that the negative perception and the positive perception are different to be included in one measure, the properties of two constructs need to be explained theoretically and empirically. Yet, the only variable included to test the validity, perceived stress, showed negative correlations with both the negative and the positive perception (Park, 2022), limited to indicate contrasting characteristics of the two. Therefore, more work is still needed to elucidate the psychometric properties of the Korean version of the SMM.

Together, despite the researchers’ efforts to validate the SMM, the results of the factor analysis were divergent. The current study conducted confirmatory factor analysis (CFA) to compare different models supported by prior studies. To date, researchers have investigated the SMM with community samples (e.g., Karampas et al., 2020; Mierzejewska-Floreani et al., 2022). Although Iwamoto et al. (2020) and Park (2022) included undergraduate students in their samples, the current study aimed to investigate emerging adults’ stress belief including undergraduate students but not limited to them. Emerging adulthood is characterized by instability as emerging adults have not yet made long-term decisions in relationships and work (Arnett et al., 2014). While emerging adulthood has been typically defined as the period between 18 and 29 years of age, some developmental researchers postulated that there are variations in the upper age limit of the developmental phases closely related to social role transitions (e.g., adolescence, emerging adulthood) across different countries and cultures because the timing of role transitions has been delayed to varying degrees (e.g., Sawyer et al., 2018). This is also the case for emerging adulthood in South Korea. For instance, South Korean emerging adults who go to 4-year college choose to delay graduation because of high youth unemployment rate (Yi, 2019), and the average age for the first marriage is increasing which reached 33.4 for men and 31.1 for women in 2021 (Statistics Korea, 2022). Such circumstances can prolong instability and pile up stress emerging adults experience. In fact, transitions during emerging adulthood predicted greater stress, physical health symptoms (Barlett et al., 2020), and low life satisfaction (Lane et al., 2017). Stress mindset during the period can be a protective factor against stressors. The present study sought to establish measurement invariance across three groups depending on emerging adults’ status (i.e., students, job seekers, and employees). Therefore, the present study included emerging adults who were engaged in different activities including undergraduate students and graduates who started working full-time and were hunting for jobs. The proposed model was tested again with a random subsample after 6 months in order to test longitudinal stability of the factor structure, because no other study has investigated the factorial stability across time while only Crum et al. (2013) reported the test-retest reliability of the total SMM scores (time interval of 2–3 days). Convergent and discriminant validity were also examined through the associations of the stress mindset with perceived stress, life satisfaction, depressive symptoms, and dispositional emotion regulation.

Method

Participants

A total of 912 South Korean young adults, recruited via online advertisements on social media communities, completed an online questionnaire. Their age ranged from 18 to 32 years (M = 24.2, SD = 2.4), and 541 were women (59.3%). Only those who provided consent for voluntary participation were included in the study. Among the participants, 117 (12.8%) were working currently, 310 (34.0%) were seeking employment, and 483 (53.0%) were studying. A random subsample (N = 300, approximately a third of the original sample) was selected from the initial sample and invited to answer the follow-up questionnaire 6 months after the first measurement. The purpose of collecting the follow-up data from this smaller random subsample was to analyze the factorial stability of SMM. A total of 259 adults (women = 62.9%; employed = 28.2%; seeking employment = 32.8%; students = 39.0%) responded to the invitation and completed the follow-up questionnaire, which comprised the SMM and other stress-related measures. Participants were given gift cards worth USD 4.00 at T1 and USD 8.00 at T2. The study protocol was approved by the Institutional Review Board of the university to which the corresponding author was affiliated.

Measures

Stress mindset

We administered the Korean version of the SMM-General (Crum et al., 2013) used in Kim (2020). Kim (2020) and a native English-speaking graduate student translated and back-translated the original measure into Korean, with an assistance of one of the authors who is fluent in both Korean and English, to establish its cross-cultural semantic equivalence. On a scale from 0 (strongly disagree) to 4 (strongly agree), participants rated the extent to which they agreed with statements about stress (see Table 1), covering four different domains: general, learning and growth, performance and productivity, and health and vitality. Four negatively worded items were reverse-coded, where a higher score represented a stress-is-enhancing mindset. Crum et al. (2013) demonstrated that the measure exhibited good criterion validity and internal consistency (Cronbach’s alpha .86). Kim (2020) also reported good internal consistency of 8 items (Cronbach’s alpha .80). In the current study, Cronbach’s alpha was .84 for the full sample at T1 and .87 for the retest subsample at T2.

Table 1.

Stress Mindset Measure (SMM) Domains and Items.

Domain	Item
General	1. The effects of stress are negative and should be avoided.
Learning and growth	2. Experiencing stress facilitates my learning and growth.
Health and vitality	3. Experiencing stress depletes my health and vitality.
Performance and productivity	4. Experiencing stress enhances my performance and productivity.
Learning and growth	5. Experiencing stress inhibits my learning and growth.
Health and vitality	6. Experiencing stress improves my health and vitality.
Performance and productivity	7. Experiencing stress debilitates my performance and productivity.
General	8. The effects of stress are positive and should be utilized.

Perceived stress

The Korean version of the Perceived Stress Scale (Cohen et al., 1983; Park & Seo, 2010) was used. The participants rated 10 items related to stress (e.g., “How often have you felt nervous and stressed?”) from 0 (not at all) to 4 (very frequently). Park and Seo (2010) reported that the scale is best represented using two subscales—negative perception and lack of positive perception, with Cronbach’s alphas of .76 and .75, respectively. In the current study, the alpha coefficient was .83 for the whole scale, .84 for negative perception, and .82 for lack of positive perception.

Life satisfaction

To measure life satisfaction, we used 5 items from the Korean version of the Satisfaction with Life Scale (Diener et al., 1985; Ryu, 1996). Questions such as “So far I have gotten the important things I want in life” were rated from 1 (strongly disagree) to 7 (strongly agree). The alpha coefficient was .81 in Ryu’s (1996) study and .85 in this study.

Depressive symptoms

Depressive symptoms were measured using the 11-item Korean short version of the Center for Epidemiologic Studies Depression Scale (CES-D; Hoe et al., 2015). Although there is ongoing debate regarding its factor structure, the CES-D short version is thought to measure four symptoms: depressive affect, positive affect, somatic symptoms, and interpersonal difficulties (Shafer, 2006). A recent study (Shin et al., 2017) concluded that the scale is best represented using a bifactor model with a strong general factor, suggesting adequate unidimensionality. In the current study, the reliability coefficient was .91 for all 11 items.

Dispositional emotion regulation

The Korean version of the Emotion Regulation Questionnaire (Shon, 2005) was used to measure reappraisal and suppression. The original scale, developed by Gross and John (2003), includes subscales for reappraisal (6 items; e.g., “When I’m faced with a stressful situation, I make myself think about it in a way that helps me stay calm”) and expressive suppression (4 items; e.g., “I keep my emotions to myself”). Respondents rated the items on a 7-point Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree). Cronbach α was .85 for reappraisal and .73 for suppression in Shon’s (2005) study, and .84 for reappraisal and .75 for suppression in this study.

Data Analysis

Statistical analyses were performed using the R programming language (version 4.1.4), with psych (Revelle, 2022) and lavaan (Rosseel, 2012) packages. Models with different latent factor structures were tested using data collected at T1: a single-factor (Crum et al., 2013), a two-factor (Iwamoto et al., 2020; Karampas et al., 2020), and correlated and hierarchical four-factor models (Mierzejewska-Floreani et al., 2022). Throughout model testing, correlated errors among the negatively worded items were allowed to incorporate the wording effect (Hayduk & Littvay, 2012). The model-testing procedure was repeated with the re-test sample data collected at T2. Next, the cross-sectional factor models for the subsample (collected at both T1 and T2) are combined to establish longitudinal factor analysis model to evaluate the stability of the latent factor structure. To guarantee that the general factor is the same across different time points, we tested the measurement invariance by constraining the factor loadings to be equal (i.e., metric invariance; see Chen et al., 2005). The longitudinal stability of the latent factors was evaluated by the correlation coefficient between the same latent factors across the two time points. After establishing the factorial structure, correlations between SMM and other stress-related variables were examined using T2 data.

Since the analyses primarily involves with fitting and comparing different models using the maximum-likelihood (ML) estimation, model fits were evaluated adopting a series of indices that are known to perform well with ML estimation: The Tucker–Lewis Index (TLI; values ≥ 0.90 for acceptable fit), comparative fit index (CFI; values ≥ 0.90 for acceptable fit), and root mean square error of approximation (RMSEA; values ≤ 0.08 for acceptable fit) (Browne & Cudeck, 1993; Hu & Bentler, 1999).

Results

Preliminary Analysis

Descriptive data analyses were conducted before investigating the factor structure of SMM and presented at Table 2. Items of the SMM showed positive correlation with each other. It was notable that items with different wording (positively- vs. negatively-worded) displayed a somewhat distinctive correlational pattern. To be specific, negatively-worded items (Item 1, 3, 5, 7) showed greater correlations with each other than with positively-worded items (Item 2, 4, 6, 8). This implied that a possible method effect based on the item wording would be inherent in SMM. Items assessing stress-related belief in “learning and growth” domain (Item 2, 5) showed the highest mean scores, while those assessing belief in “health and vitality” domain (Item 3, 6) showed the lowest mean scores. Skewness and Kurtosis statistics suggested that the data distributions were sufficiently normal.

Table 2.

Correlations and Descriptive Statistics of the SMM Items at T1 (N = 912) and T2 (N = 259).

Item Number	1	2	3	4	5	6	7	8
Item 1	—	.50*	.56^*	.42^*	.44^*	.33^*	.55^*	.39^*
Item 2	.29^*	—	.40^*	.59^*	.52^*	.38^*	.57^*	.50^*
Item 3	.46^*	.25^*	—	.34^*	.41^*	.54^*	.51^*	.41^*
Item 4	.29^*	.54^*	.33^*	—	.49^*	.35^*	.65^*	.52^*
Item 5	.41^*	.38^*	.43^*	.40^*	—	.24^*	.63^*	.46^*
Item 6	.31^*	.35^*	.45^*	.45^*	.15^*	—	.37^*	.45^*
Item 7	.41^*	.32^*	.50^*	.48^*	.63^*	.24^*	—	.52^*
Item 8	.35^*	.56^*	.41^*	.59^*	.33^*	.56^*	.39^*	—
Mean	1.72/1.80	2.23/2.28	1.39/1.32	1.97/2.04	2.14/2.15	1.44/1.35	1.85/1.89	1.86/1.88
SD	1.00/1.01	0.92/0.88	0.89/0.86	0.92/0.88	0.96/0.95	0.97/0.89	0.97/0.94	0.94/0.90
Skew	.30/.28	−.35/.64	.52/.31	.06/−.38	−.19/−.38	.49/.35	.09/.03	.02/−.02
Kurtosis	−.56/−.74	−.46/−.12	−.03/−.53	−.63/−.53	−.74/−.44	−.26/−.45	−.75/−.80	−.35/−.34

Note. Correlation coefficients for T1 (total sample) are presented in the lower diagonal and coefficients for T2 (retest subsample) are presented in the upper diagonal. Descriptive statistical values on the left-hand side refer to the values at T1 and on the right-hand side refer to the values at T2.

p < .05.

Confirmatory Factor Analysis

CFA was applied to four models: single-, two-, four-, and hierarchical four-factor models. Both the single-factor and two-factor models with enhancing mindset (four positively worded items) and debilitating mindset (four negatively worded items) showed poor fit indices. The four-factor model, which comprised learning and growth, performance and productivity, health and vitality, and general factors, showed acceptable fit. In this model, the four subscales were highly correlated with each other; hence, the hierarchical model was tested. However, the analysis of the hierarchical model, with one second-order factor (i.e., general stress mindset) and four first-order factors, resulted in inadmissible estimates (Chen et al., 2001). The correlation between the higher-order factor and lower-order “general” factor exceeded 1.0 (r = 1.03); therefore, the residual variance of the “general” lower-order factor was negative (−0.68). This indicated a possibility that the higher-order factor and lower-order “general” factor overlapped conceptually, resulting in model redundancy. As a result, the hierarchical model was revised by deleting 2 items for the “general” lower-order factor and retaining three lower-order factors. The revised hierarchical model showed an acceptable fit (RMSEA = 0.063, CFI = 0.994, TLI = 0.969).

The same procedure was repeated for subsample data at T2. This four-factor hierarchical model analysis also resulted in uninterpretable estimates (i.e., correlations between lower-order “general” factors and higher-order factors exceeding 1.0). The revised hierarchical model met the goodness-of-fit index criteria (RMSEA = 0.049, CFI = 0.997, TLI = 0.985) and produced interpretable estimates. Based on the theoretical proposition that the stress mindset is a unidimensional construct (Crum et al., 2013) and the current statistical findings, the revised hierarchical model containing 6 items—three first-order factors representing specific domains (learning and growth, performance and productivity, and health and vitality) and one overarching second-order factor—was considered the best model representing the factor structure of the SMM (see Figure 1). A summary of the model fit comparisons is presented in Table 3.

Figure 1.

The revised CFA model (total sample at T1, N = 912).

Table 3.

Results of Cross-Sectional CFA for the Korean Version of SMM at T1 and T2.

Model	χ²	df	RMSEA	CFI	TLI
SMM at T1 (N = 912)
Single-factor	156.625	14	0.110	0.919	0.887
Two-factor	292.529	19	0.126	0.900	0.853
Four-factor	27.766	8	0.052	0.993	0.975
Hierarchical four-factor	90.671	10	0.094	0.971	0.918
Hierarchical three-factor (6-item)	13.723	3	0.063	0.994	0.969
SMM at T2 (N = 259)
Single-factor	78.185	14	0.133	0.928	0.856
Two-factor	98.401	19	0.127	0.911	0.868
Four-factor	11.346	8	0.040	0.996	0.987
Hierarchical four-factor	18.62	10	0.058	0.990	0.973
Hierarchical three-factor (6-item)	4.86	3	0.049	0.997	0.985

Note. RMSEA = Root Mean Square Error of Approximation; CFI = Comparative Fit Index; TLI = Tucker–Lewis index.

As a next step, the subsample data collected at both T1 and T2 was combined to establish a longitudinal factor analysis model in order to investigate the factorial stability of SMM. The longitudinal measurement invariance of the best fitting model emerged in cross-sectional CFAs was tested (Table 4). The configural invariance model, in which all parameters were freely estimated except for the minimum constraints for identification, showed an acceptable fit (RMSEA = 0.082, CFI = 0.954, TLI = 0.915). The metric invariance model in which factor loadings were constrained to be equal across the two time points showed no worse fit than the configural invariance model, Δχ²(6) = 5.34, RMSEA = 0.075, CFI = 0.954, TLI = 0.928, suggesting that the latent constructs was measured by the same measurement units across different time (Chen et al., 2005). The stability coefficient between the second-order stress mindset factors was 0.59 (Figure 2).

Table 4.

Results of Longitudinal CFA for the Hierarchical Three-Factor Model (Retest Subsample, N = 259).

Model	χ²	df	RMSEA	CFI	TLI
Configural invariance	98.503	36	0.082	0.954	0.915
Metric invariance	103.799	42	0.075	0.954	0.928

Figure 2.

The longitudinal CFA model (retest subsample, N = 259).

Concurrent and Discriminant Validity

To test the theoretical validity of the Korean version of the SMM, a correlation analysis was carried out with the total score of 6 items of the SMM and other stress-related measures at T2: life satisfaction, perceived stress (total score, negative experience, and lack of positive experience), depressive symptoms, reappraisal, and suppression. SMM was significantly related to stress-related outcome variables. Specifically, as shown in Table 5, SMM exhibited a positive relationship with life satisfaction (r = 0.25), while showing negative relationships with perceived stress (total score: r = −0.29; negative perception: r = −0.25; and lack of positive perception: r = −0.23) and depressive symptoms (r = −0.22). In addition, SMM was significantly related to emotion-regulatory behaviors. SMM was positively related to reappraisal (r = 0.29) and negatively related to suppression (r = −0.11, p < 0.10).

Table 5.

Correlation Coefficients Between Total Score of SMM and Other Related Variables at T2.

Variables	1	2	3	3-1	3-2	4	5	6
1. Stress mindset	1
2. Life satisfaction	.25*	1
3. Perceived stress	−.29*	−.56*	1
3-1. Negative perception	−.25*	−.34*	.85*	1
3-2. Lack of positive perception	−.23*	−.60*	.77*	.33*	1
4. Depressive symptoms	−.22*	−.43*	.67*	.63*	.44*	1
5. Reappraisal	.29*	.24*	−.26*	−.16*	−.27*	−.35*	1
6. Suppression	−.11**	−.21*	.16*	.04	.23*	.13**	.12*	1

p < .05. **p < .10.

Discussion

This study aimed to examine the psychometric properties and factorial structure of the Korean version of the SMM. The original one-factor structure that emerged in a sample of adults in the United States (Crum et al., 2013), the two-factor structure that emerged in a sample with Japanese adults (Iwamoto et al., 2020), and the hierarchical four-factor structure from the study with Polish adults (Mierzejewska-Floreani et al., 2022) were not replicated. Instead, the results suggest that the Korean version of the SMM is best represented by a hierarchical three-factor structure with 6 items representing three domains of stress (i.e., learning and growth, performance and productivity, and health and vitality). Two items from the original measure, representing the “general” stress mindset, were excluded because of multicollinearity. By using the data collected from retest subsample, longitudinal metric invariance for the hierarchical three-factor structure was established across two measurement time points with 6-month intervals, and the higher-order factor showed a reasonable stability over the same time period. The establishment of a higher-order structure serves as the basis for using the total score to represent one’s belief regarding the effects of stress (Sellbom & Tellegen, 2019).

The total SMM score, calculated based on the revised model, showed significant relationships with other stress-related variables. The SMM was positively related to life satisfaction and negatively related to perceived stress and depressive symptoms. Interestingly, the SMM was significantly related to reappraisal, which differed from Crum et al.’s (2013) proposition that the stress mindset is theoretically distinct from stress (re)appraisal, as appraisal of stress is related to the evaluation of a particular stressor (i.e., situation-level), whereas stress mindset is related to the evaluation of the nature of stress itself (i.e., meta-level). Although researchers later acknowledged a common theme underlying the two constructs and proposed an integrated model (Hagger et al., 2020; Jamieson et al., 2018), the possible dynamics of appraisal and stress mindset across situations have not yet been well studied. Further research that captures the dynamic nature of appraisal and stress mindsets across diverse situations and populations is needed.

The findings of this study question the utility of the 2 items evaluating a “general” domain stress mindset (e.g., “The effects of stress are negative and should be avoided”). As Crum et al. (2013) conceptualized, stress mindset refers to “the extent to which one holds the belief that stress has enhancing [or debilitating] consequences for various stress-related outcomes such as performance and productivity, health and wellbeing, and learning and growth” (p. 716). Hence, the mean or average score on the SMM should represent one’s overall belief regarding the effects of stress across different domains. The 2 items are related to the general consequences of stress, while the other 6 items are specifically related to the domains of an individual’s functioning affected by stress. The meaning of the two “general” items would be redundant considering the meaning of the overall stress mindset derived from interrelationships or covariances among the 6 items representing specific domains of functioning. Mierzejewska-Floreani et al. (2022) established a hierarchical four-factor structure; however, their estimated model also produced a correlation close to 1 between the second-order stress mindset factor and first-order “general” factor (.94/.96). They also questioned the meaning of the “general” subscale, but determined that the items should be retained, positing that the subscale may represent an unspecified domain factor rather than an overall belief factor.

Throughout the CFAs, we incorporated shared method variance across negatively worded items (Items 1, 3, 5, and 7). The systematic method variance is beyond what is accounted for by the respective latent factors (Hayduk & Littvay, 2012); therefore, it should be considered when establishing a measurement model. Although Iwamoto et al.’s (2020) solution, which was to construct two separate latent factors with positively and negatively worded items, is one of the ways to integrate the wording effect into the measurement model (e.g., Martín-Albo et al., 2007), it may be merely a nuisance grouping of items rather than a representation of substantive latent constructs, resulting in relatively meaningless factors (McKay et al., 2014; Reise et al., 2010). The CFA results in this study favored the hierarchical model over the two-factor model, supporting the application of correlated errors instead of the introduction of separate factors to incorporate the wording effect into the measurement model. This also supports the conceptualization that the stress mindset is a spectrum between enhancing and debilitating mindsets rather than two opposing beliefs functioning in parallel (Crum et al., 2013; Keech et al., 2021). While Park (2022)’s results suggested that the use of only four negatively- or positively-worded items (i.e., measuring either debilitating or enhancing mindset) would be an alternative of applying the full SMM items, this approach may force researchers to separate one mindset from the other, disregarding the initial concept of stress mindset. Therefore, measuring both types of mindsets with consideration for the wording effect as the current findings suggested, rather than splitting the measure into two, would be recommendable.

The current study has some limitations. First, only the SMM was measured twice, while the other variables were measured once at T2. Therefore, the temporal relationships between the variables were not testable. Second, the current study only included self-rated measures, relying on retrospective recall of stress-related responses and outcomes. Future research could benefit from behavioral or psychophysiological measures to investigate the predictive efficacy of the SMM. Third, this study could not investigate the incremental validity of the SMM over other scales designed to measure stress beliefs, such as the Beliefs About Stress Scale (Laferton et al., 2018) and Stress Control Mindset Measure (Keech et al., 2021). Although other measures also did not address all domains or facets of stress and related beliefs (Kilby et al., 2020, 2021), exploring the relative utility of these scales may inform researchers in this field.

Nevertheless, the current findings suggest that the SMM is a valid measure to capture the stress mindset construct among the early adult population, but may require some revision. In this study, the rationale for removing some items (i.e., items 1 and 8) was that they were redundant both theoretically and psychometrically. To the best of our knowledge, this is the first study to suggest the need for a revision of the SMM. Therefore, researchers should be cautious while adapting our findings to their population of interest and are encouraged to further examine the factor structure of the SMM.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Ministry of Education of the Republic of Korea, the National Research Foundation of Korea (NRF-2020S1A5A2A01043871), and the Korea University Grant.

ORCID iDs

Hyunmo Seong

Sang Min Lee

Data Accessibility statement

Data will be provided upon the request to the corresponding author.

References

Arnett

J. J.

Žukauskienė

Sugimura

(2014). The new life stage of emerging adulthood at ages 18–29 years: Implications for mental health. The Lancet Psychiatry, 1(7), 569–576. https://doi.org/10.1016/S2215-0366(14)00080-7

Barlett

C. P.

Barlett

N. D.

Chalk

H. M.

(2020). Transitioning through emerging adulthood and physical health implications. Emerging Adulthood, 8(4), 297–305. https://doi.org/10.1177/2167696818814642

Browne

M. W.

Cudeck

(1993). Alternative ways of assessing model fit. In Bollen

K. A.

Long

J. S.

(Eds.), Testing structural equation models (pp. 136–162). Sage.

Chen

Bollen

K. A.

Paxton

Curran

P. J.

Kirby

J. B.

(2001). Improper solutions in structural equation models: Causes, consequences, and strategies. Sociological Methods & Research, 29, 468–508. https://doi.org/10.1177/0049124101029004003

Chen

F. F.

Souca

K. H.

West

S. G.

(2005). Testing measurement invariance of second-order factor models. Structural Equation Modeling, 12(3), 471–492. https://doi.org/10.1207/s15328007sem1203_7

Cohen

Kamarck

Mermelstein

(1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24, 385–396. https://doi.org/10.2307/2136404

Crum

A. J.

Akinola

Martin

Fath

(2017). The role of stress mindset in shaping cognitive, emotional, and physiological responses to challenging and threatening stress. Anxiety, Stress, & Coping, 30(4), 379–395. https://doi.org/10.1080/10615806.2016.1275585

Crum

A. J.

Salovey

Achor

(2013). Rethinking stress: The role of mindsets in determining the stress response. Journal of Personality and Social Psychology, 104(4), 716–733. https://doi.org/10.1037/a0031201

Diener

E. D.

Emmons

R. A.

Larsen

R. J.

Griffin

(1985). The satisfaction with life scale. Journal of Personality Assessment, 49, 71–75. https://doi.org/10.1207/s15327752jpa4901_13

10.

Dweck

C. S.

Chiu

Hong

(1995). Implicit theories and their role in judgments and reactions: A world from two perspectives. Psychological Inquiry, 6(4), 267–285. https://doi.org/10.1207/s15327965pli0604_1

11.

Fay

Sonnentag

(2002). Rethinking the effects of stressors: A longitudinal study on personal initiative. Journal of Occupational Health Psychology, 7(3), 221–234. https://doi.org/10.1037/1076-8998.7.3.221

12.

Folkman

Lazarus

R. S.

(1980). An analysis of coping in a middle-aged community sample. Journal of Health and Social Behavior, 21(3), 219–239. https://doi.org/10.2307/2136617

13.

Gross

J. J.

John

O. P.

(2003). Individual differences in two emotion regulation processes: Implications for affect, relationships, and well-being. Journal of Personality and Social Psychology, 85(2), 348–362. https://doi.org/10.1037/0022-3514.85.2.348

14.

Hagger

M. S.

Keech

J. J.

Hamilton

(2020). Managing stress during the coronavirus disease 2019 pandemic and beyond: Reappraisal and mindset approaches. Stress and Health, 36(3), 396–401. https://doi.org/10.1002/smi.2969

15.

Hayduk

L. A.

Littvay

(2012). Should researchers use single indicators, best indicators, or multiple indicators in structural equation models? BMC Medical Research Methodology, 12(1), 159. https://doi.org/10.1186/1471-2288-12-159

16.

Hoe

Park

B. S.

Bae

S. W.

(2015). Testing measurement invariance of the 11-item Korean version CES-D scale. Mental Health & Social Work, 43(2), 313–339.

17.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

18.

Huebschmann

N. A.

Sheets

E. S.

(2020). The right mindset: Stress mindset moderates the association between perceived stress and depressive symptoms. Anxiety, Stress, & Coping, 33(3), 248–255. https://doi.org/10.1080/10615806.2020.1736900

19.

Iwamoto

Takehashi

Taka

(2020). Reliability and validity of a Japanese translation of the Stress Mindset Measure (SMM-J). The Japanese Journal of Psychology, 9(6), 592–602. https://doi.org/10.4992/jjpsy.90.18229

20.

Jamieson

J. P.

Crum

A. J.

Goyer

J. P.

Marotta

M. E.

Akinola

(2018). Optimizing stress responses with reappraisal and mindset interventions: An integrated model. Anxiety, Stress, & Coping, 31(3), 245–261. https://doi.org/10.1080/10615806.2018.1442615

21.

Karampas

Pezirkianidis

Stalikas

(2020). Psychometric properties of the Stress Mindset Measure (SMM) in a Greek sample. Psychology, 11, 1185–1199. https://doi.org/10.4236/psych.2020.118079

22.

Keech

J. J.

Orbell

Hagger

M. S.

O’Callaghan

F. V.

Hamilton

(2021). Psychometric properties of the stress control mindset measure in university students from Australia and the UK. Brain and Behavior, 11(2), e01963. https://doi.org/10.1002/brb3.1963

23.

Kilby

C. J.

Sherman

K. A.

Wuthrich

V. M.

(2020). A scoping review of stress beliefs: Literature integration, measurement issues, and theoretical concerns. Annals of Behavioral Medicine, 54(8), 595–610. https://doi.org/10.1093/abm/kaaa006

24.

Kilby

C. J.

Sherman

K. A.

Wuthrich

V. M.

(2021). How do you think about stress? A qualitative analysis of beliefs about stress. Journal of Health Psychology, 26, 2756–2767. https://doi.org/10.1177/1359105320926543

25.

Kim

(2020). The effect of academic stress mindset on academic engagement and burnout [Unpublished doctoral dissertation]. Korea University.

26.

Laferton

J. A. C.

Stenzel

N. M.

Fischer

(2018). The beliefs about stress scale (BASS): Development, reliability, and validity. International Journal of Stress Management, 25(1), 72–83. https://doi.org/10.1037/str0000047

27.

Lane

J. A.

Leibert

T. W.

Goka-Dubose

(2017). The impact of life transition on emerging adult attachment, social support, and well-being: A multiple-group comparison. Journal of Counseling & Development, 95(4), 378–388. https://doi.org/10.1002/jcad.12153

28.

Levy

B. R.

Slade

M. D.

Kunkel

S. R.

Kasl

S. V.

(2002). Longevity increased by positive self-perceptions of aging. Journal of Personality and Social Psychology, 83(2), 261–270. https://doi.org/10.1037/0022-3514.83.2.261

29.

Wei

(2021). The evolution of the concept of stress and the framework of the stress system. Cell Stress, 5(6), 76–85. https://doi.org/10.15698/cst2021.06.250

30.

Martín-Albo

Núñez

J. L.

Navarro

J. G.

Grijalvo

(2007). The Rosenberg Self-Esteem Scale: Translation and validation in university students. The Spanish Journal of Psychology, 10(2), 458–467. https://doi.org/10.1017/S1138741600006727

31.

McEwen

B. S.

Seeman

(1999). Protective and damaging effects of mediators of stress: Elaborating and testing the concepts of allostasis and allostatic load. Annals of the New York Academy of Sciences, 896(1), 30–47. https://doi.org/10.1111/j.1749-6632.1999.tb08103.x

32.

McKay

M. T.

Boduszek

Harvey

S. A.

(2014). The Rosenberg Self-Esteem Scale: A bifactor answer to a two-factor question? Journal of Personality Assessment, 96(6), 654–660. https://doi.org/10.1080/00223891.2014.923436

33.

Mierzejewska-Floreani

Banaszkiewicz

Gruszczyńska

(2022). Psychometric properties of the Stress Mindset Measure (SMM) in the Polish population. PLoS ONE, 17(3), e0264853. https://doi.org/10.1371/journal.pone.0264853

34.

Mroczek

D. K.

Almeida

D. M.

(2004). The effect of daily stress, personality, and age on daily negative affect. Journal of Personality, 72(2), 355–378. https://doi.org/10.1111/j.0022-3506.2004.00265.x

35.

Park

(2022) Method effects and measurement invariance testing for Korean Stress Mindset Scale. Korean Journal of Industrial and Organizational Psychology, 35(2), 89–111. http://doi.org/10.24230/kjiop.v35i2.89-111

36.

Park

Seo

Y. S.

(2010). Validation of the Perceived Stress Scale (SS) on samples of Korean University students. Korean Journal of Psychology: General, 29(3), 611–629.

37.

Quick

J. C.

Bennett

Hargrove

M. B.

(2014). Stress, health, and wellbeing in practice: Workplace leadership and leveraging stress for positive outcomes. In Chen

Copper

C. L.

(Eds.), Wellbeing: A complete reference guide, Volume III, Work and Wellbeing (pp. 175–204). Wiley-Blackwell.

38.

Reise

S. P.

Moore

T. M.

Haviland

M. G.

(2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92(6), 544–559. https://doi.org/10.1080/00223891.2010.496477

39.

Revelle

W. R.

(2022). Psych: Procedures for personality and psychological research. Software. https://CRAN.R-project.org/package=psych

40.

Rosseel

(2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02

41.

Ryu

(1996). The effect of personality and desire level on life satisfaction [Unpublished master’s thesis]. The Graduate School of Yonsei University.

42.

Sawyer

S. M.

Azzopardi

P. S.

Wickremarathne

Patton

G. C.

(2018). The age of adolescence, The Lancet Child & Adolescent Health, 2(3), 223–228. https://doi.org/10.1016/S2352-4642(18)30022-1

43.

Schneiderman

Ironson

Siegel

S. D.

(2005). Stress and health: Psychological, behavioral, and biological determinants. Annual Review of Clinical Psychology, 1, 607–628. https://doi.org/10.1146/annurev.clinpsy.1.102803.144141

44.

Sellbom

Tellegen

(2019). Factor analysis in psychological assessment research: Common pitfalls and recommendations. Psychological Assessment, 31(12), 1428–1441. https://doi.org/10.1037/pas0000623

45.

Shafer

A. B.

(2006). Meta-analysis of the factor structures of four depression questionnaires: Beck, CES-D, Hamilton, and Zung. Journal of Clinical Psychology, 62(1), 123–146. https://doi.org/10.1002/jclp.20213

46.

Shin

Lee

Yun

S. J.

(2017). A bifactor approach to the factor structure study of the CES-D scale. Korean Journal of Stress Research, 25(4), 272–278. https://doi.org/10.17547/kjsr.2017.25.4.272

47.

Shon

J. M.

(2005). Individual differences in two regulation strategies: Cognitive reappraiser vs. emotion suppressor. [Unpublished master’s thesis]. Seoul National University.

48.

Smith

E. N.

Young

M. D.

Crum

A. J.

(2020). Stress, mindsets, and success in Navy SEALs special warfare training. Frontiers in Psychology, 10, 2962. https://doi.org/10.3389/fpsyg.2019.02962

49.

Statistics Korea. (2022). 2021 marriage and divorce statistics. http://kostat.go.kr/assist/synap/preview/skin/miri.html?fn=e1f84190451908117123911&rs=/assist/synap/preview

50.

Vigil-Colet

Navarro-González

Morales-Vives

(2020). To reverse or to not reverse Likert-type items: That is the question. Psicothema, 32(1), 108–114. https://doi.org/10.7334/psicothema2019.286

51.

P. N.

(2019). The impact of four-year college graduates’ delayed graduation on labor market outcome. The Journal of Economics and Finance of Education, 28(3), 1–27.