Sage Journals: Discover world-class research

Abstract

The Digital Stress Scale (DSS) is used to measure digital stress, which is the perceived stress and anxiety associated with social media use. In this study, the Chinese version of the DSS was validated using a sample of 721 Chinese college students, 321 males and 400 females (KMO = 0.923; Bartlett = 5,058.492, p < .001). Confirmatory factor analysis (CFA) and item response theory (IRT) revealed that the five-factor model fits our data best. And the good criterion validity was shown through the association of the Chinese version of the DSS with anxiety and depression. The results of measurement invariance supported that the scale has measurement equivalence across the region (rural or urban). However, they did not support strict measurement invariance across gender. Overall, our results suggest that the DSS is an ideal tool for measuring digital stress in a Chinese sample. People can benefit from assessing digital stress, which has the potential to moderate the relationship between social media use and mental health. These psychometric investigations of the DSS will allow future research to investigate construct validity better.

Plain language summary

Purpose: We aimed to assess the Psychometric Properties of the Chinese version of the Digital Stress Scale (DSS) among adolescents. Methods: First, we finished the Chinese version of the Digital Stress Scale by back-translation procedure and pre-test. Second, the internal consistency, criterion-related validity, construct validity, measurement equivalence and item analysis of the scales were tested. Conclusions: The DSS is a great instrument for measuring digital stress in a Chinese sample. Confirmatory factor analysis (CFA) and item response theory (IRT) revealed that the five-factor model fits our data well. The results of measurement invariance supported that the scale has measurement equivalence across the region (rural or urban). However, the scale did not show strict measurement invariance across gender. The criterion-related validity and internal consistency were great. Implications: People can benefit from assessing digital stress, which has the potential to moderate the relationship between social media use and mental health. Limitations: First, this study does not provide specific suggestions for future item modifications. Second, research samples are not conducive to the generalization of our findings. Third, this research does not use measures to answer a substantive developmental question using additional variables.

Keywords

digital stress social media use college students confirmatory factor analysis item response theory

Introduction

Social media are immensely popular among Chinese college students, and pervasive in many parts of their lives, from life entertainment to interpersonal interactions. Especially during the global pandemic of COVID-19, all kinds of social media, such as massive open online courses (MOOCs), Rain Classroom, WeChat, QQ, Tencent Meeting, and Ding Talk, have developed and gradually been employed as learning tools in the classroom (Su et al., 2021). However, the increased use of social media by students has led to concerns about possible associations between digital media use and mental health (Twenge & Campbell, 2019). Previous studies have reported that the frequent use of social media may lead to negative emotional stresses, including poor sleep quality, anxiety, depression, lower self-esteem, loneliness, and social stress (Banjanin et al., 2015; Barry et al., 2017; Ehrenreich & Underwood, 2016; Woods & Scott, 2016). Digital stress is a relatively new phenomenon among the negative outcomes of social media use. And it has not been widely concerned in China by researchers. Measuring digital stress is useful for assessing the mental health of Chinese college students. To address this gap in the literature, the present study will adapt the original version of the China College Students’ Digital stress scale (DSS), test its psychometric characteristics, and evaluate its effect in the China sample. It is expected to provide references for future research on the clinical effects of DSS.

Social Media, Mental Health, and Digital Stress

Social media means a variety of Internet-based networks that enable adolescents to create content and interact with others verbally and visually (Carr & Hayes, 2015; Testa et al., 2020). Keles et al. (2020) conducted a systematic narrative review and stated that the relationship between social media use and mental health problems is correlational but not conclusively causative. Odgers and Jensen (2020) summarized the linkages between digital technology usage and adolescent mental health. They documented a small association between social media usage and well-being. Moreover, there are mixed opinions about social media. Some studies concluded that the effects of social media use were positive, while others found that the effects were negative. In a systematic review, Liu et al. (2019) conducted a more focused review of different types of social media use and well-being. They found that the association between digital media use and well-being also depends on how the technology is used. Interaction, self-presentation, and entertainment on social media were positively correlated with well-being, whereas content consumption, such as browsing, was negatively associated with well-being. And above all, most adolescents have experienced both optimism and pessimistic about social media usage. While social media promote interpersonal interactions and allow users to communicate information and express personal experiences, the possible disadvantages of social media should not be underestimated. The emergence and intensive use of social media induce bad behaviors and decrease health both mentally and physically. These negative aspects are the following research priorities. It may help us facilitate individualized prevention and intervention strategies to help individuals maintain or improve their mental health.

Reilly (2020) conducted a survey with adolescents aged 11 to 18 years to discuss the good and bad sides of social media use. On the negative side, they found that the constant use of their devices is associated with sleep quality. One of the narratives in their studies of participants was that always being online makes them wait for responses and not being able to sleep. Moreover, social media usage pressurizes users and holds a potential risk for low mood or anxiety. Adolescents with a strong desire to be accepted tend to excessively compare themselves with others, and focus on the negative aspects of themselves. Taking posting on social media as an example, many adolescents experience high levels of anxiety due to the fear of receiving negative comments online (Calancie et al., 2017). In general, the need to maintain intimacy or close connections with others through social media contributes to rising stress (Weinstein & Selman, 2016). Digital stress is one of the stress responses to social media use that comes with anxious emotions. Hall et al. (2021) defined digital stress as stress and anxiety from notifications or social media usage.

Some people experience digital stress due to social media demands. A study has investigated the psychological health effects of digital stress based on a representative survey of 1,557 German Internet users (Reinecke et al., 2017). They found that the frequent use of social media, including sending, receiving and checking messages, may increase perceived stress. The study by Nick et al. (2022) investigated the self-reported digital stress of students. Some descriptive statistics described those individuals with higher levels of social media usage reported higher levels of digital stress. Therefore, digital stress may be an important intervening factor between digital social media usage and mental health outcomes (Steele et al., 2020). This study considers digital stress one of the most important predictors of negative mental health and may predict the level of mental health related to social media use among adolescents.

The Digital Stress Scale

The preliminary theoretical model of digital stress, as indicated by Steele et al. (2020) via a summary of relevant literature, consists of four components, including availability stress, recognition anxiety, fear of missing out, and communication overload. Hall et al. (2021) validated the latent factor structure of digital stress through exploratory factor analysis (EFA) and Confirmatory factor analysis (CFA) to develop the digital stress scale. It is a five-point Likert-type scale including 24 items and consisting of five dimensions: connection overload, approval anxiety, online vigilance, availability stress, and fear of missing out.

The digital stress scale (DSS) was significantly associated with depressive symptoms, anxiety, and stress, and its reliability and validity were satisfactory (Hall et al., 2021). Connection overload is a phenomenon that involves people being constantly distracted from important life activities by multiple social media platforms, and these distractions are ubiquitous and frequent (LaRose et al., 2014). Likewise, availability stress is the internalization of peer expectations, where individuals are constantly in a state of availability and respond to information through digital means (Hall et al., 2021). Approval anxiety reflects anxiety about others’ comments and reactions to social media posts and images. Online vigilance affects perceived stress levels by promoting and intensifying permanent connected media usage patterns (Freytag et al., 2021). Finally, FoMO has been linked to increased use of social media (such as Facebook) and preoccupation with feeling unpopular or isolated on it (Beyens et al., 2016).

Measurement Invariance of the DSS Across Gender and Region

Measurement invariance can be used to determine whether scale items have the same meaning and function across groups. For example, recent research has examined gender differences on social networking sites (Choi, 2022). The envy generated by passive social media use lowers subjective well-being, and this effect is stronger among women. Adolescent girls spend more time on smartphones than boys, and the associations between heavy digital media use and low psychological well-being are also greater (Twenge & Martin, 2020). However, there is nothing to know whether this difference is brought about by the gender of the sample itself or is induced by the scale. It is valuable to establish that the scales of measured constructs across the gender examined have the same meaning and that there are no biases and differences in the meaning of items. This study will investigate whether DSS is equivalent across gender.

Furthermore, the regional differences in China’s urban-rural gap are large. The previous researchers suggest that there are unavoidable differences between rural and urban areas within China in terms of the living environment, economic development, and education (Lai et al., 2013; Zhang et al., 2018). Comparing the items consistently of DSS between the city and rural students is useful. Therefore, our study also estimates whether DSS is equivalent across regions.

Aims of the Present Study

The current research aimed to validate the Chinese version of the DSS among college students. We paid particular attention to participants’ mental health when using digital social media. Digital media gives the impression that it is a tool to provide convenience services (Davis, 2012). This impression may shift with various psychological vulnerabilities. Digital stress is a form of technology-related psychological stress arising from digital media activities, which is associated with well-being, anxiety, depression, and loneliness in varying degrees (Ahn, 2011; Reinecke et al., 2017; Twenge, 2019). Thus, this study sought to predict the level of mental health associated with adolescent social media use through digital stress. At the same time, the hypotheses related to validating the Chinese version of the DSS among Chinese college students were proposed.

Based on the theoretical model of Hall et al.’s (2021), digital stress should consist of five separate but related components, including connection overload, approval anxiety, online vigilance, availability stress, and fear of missing out. And the evidence suggests that the reliability and validity of the five-factor model is adequate. We expected the five-factor model to fit our data well. Therefore, we hypothesized that the factor structure of DSS in the Chinese sample conformed to the five-factor model (H1). In this regard, Confirmatory factor analysis (CFA) was used to confirm the factor structure of the DSS.

Second, the Criterion-Related validity and internal consistency. DSS would be expected to be positively associated with social anxiety and perceived stress (H2).

Third, the rigorous CFA-based multi-group comparison was used to investigate the scale’s measurement invariance across gender and region. We hypothesized that the Chinese version of the DSS would exhibit rigorous measurement invariance across Chinese college students of different genders and regions (H3).

Finally, IRT analyses were performed to evaluate the psychometric properties using Chinese samples. The present study hypothesized that the Chinese version of the DSS has good psychometric properties (i.e., item parameters, item characteristic curves, and test information) within the IRT framework (H4).

Materials and Methods

Participants and Procedure

The sample consisted of 721 university students (both undergraduates and graduates) recruited from universities or through advertisements on the Internet. All were citizens of China studying at universities in Southeastern China. During the formal test process, participants took a total of 25 min to complete the online questionnaire which included the Chinese versions of the DSS, the Social Interaction Anxiety Scale-6 and the Social Phobia Scale-6 (SIAS-6/SPS-6), and the Perceived Stress Scale-10 (PSS-10). All study procedures involving human participants followed institutional ethical standards, and all participants provided oral informed consent.

The final sample consisted of 721 Chinese students, with 321 males (44.5%) and 400 females (55.5%) ranging in age from 17 to 33 years (M = 19.76, SD = 1.936). The sample came primarily from rural areas (63.5%), while 36.5% were city residents. Only 14.0% indicated they were graduate students, the rest were undergraduates (86%), which included freshmen (31.8%), sophomores (38.1%), juniors (15.0%), and seniors (1.1%).

Measures

The Digital Stress Scale (DSS)

The original version of the Digital Stress Scale (DSS) consists of 24 items (Hall et al., 2021). Five factors are part of DSS: Connection overload (CO, 6 items), Approval anxiety (AA, 6 items), Online vigilance (OV, 4 items), Availability stress (AS, 4 items), and Fear of Missing out (FoMO, 4 items). Each item uses a five-point Likert-type response format ranging from 1 (“never”) to 5 (“always”) to assess subscales. The Cronbach’s $α$ for DSS in total scale, CO, AA, OV, AS, and FoMO is .85, .91, .93, .86, .88, and .87 respectively.

The Social Interaction Anxiety Scale-6 and the Social Phobia Scale-6 (SIAS-6/SPS-6)

The Social Interaction Anxiety Scale (SIAS) and the Social Phobia Scale (SPS) are companion questionnaires developed by Mattick and Clarke (1998). The SIAS-6/SPS-6 are short versions of the SIAS/SPS scales, and are the most commonly used instruments to assess social anxiety. The SIAS-6/SPS-6 has 12 items, and each item is rated on a five-point Likert scale ranging from 1 (“not at all characteristic of me”) to 5 (“extremely characteristic of me”). The Chinese version of the SIAS-6/SPS-6 has been shown to have excellent psychometric properties in Chinese samples (Ouyang et al., 2020). In the current study, Cronbach’s $α$ (.74, .87) were both acceptable for the SIAS-6 and the SPS-6.

The Perceived Stress Scale-10 (PSS-10)

The 10-item version of the Perceived Stress Scale (PSS-10) is a self-report scale used to assess respondents’ self-perceived stress levels (Cohen & Williamson, 1988). Each item is rated on a five-point Likert scale ranging from 1 (“never”) to 5 (“very often”), with items 4, 5, 7, and 8 being reversed scores. Total scores are calculated by adding the response rating of each item, with higher scores representing higher levels of perceived stress. Lu et al. (2017) have reported that the PSS-10 scale shows adequate internal consistency among Chinese undergraduates ( $α$ = .850). The scale has been translated into multiple languages and is available from http://www.psy.cmu.edu/~scohen/scales.html. The Chinese version of the PSS adopted for this study had a Cronbach’s $α$ of .763.

Statistical Analysis

Factor Structure

CFA was conducted to validate the Chinese version of the DSS’s structure in Chinese samples with Mplus7.4 (Muthén & Muthén, 1998–2015). The chi-square (χ²), root mean square error of approximation (RMSEA), standardized root mean square residual (SRMR), comparative fit index (CFI), and Tucker-Lewis index (TLI) were used to evaluate the model fit. The smaller the value of χ², the better the model fit. However, χ² is easily affected by sample size, which increases the probability of model rejection. Therefore, we used the ratio of chi-square to degrees of freedom (χ²/df) instead of the chi-square (χ²). RMSEA and SRMR values less than 0.08, and CFI and TLI values above 0.90 were considered acceptable (Hu & Bentler, 1998; Steiger, 1998).

Internal Consistency and Criterion-Related Validity

Cronbach’s $α$ and were used to examine the internal consistency of the DSS and its subscales. Cronbach’s $α$ is the most widely used measure of scale internal consistency (Streiner, 2003). The estimate of internal consistency was in the range of 0 to 1, and the criteria were as follows: >0.9 = excellent, >0.8 = good, >0.6 = acceptable, <0.6 = insufficient. In addition, the criterion-related validity was assessed using Pearson’s correlation coefficient or Spearman’s correlation to test for correlations between the Chinese version of the DSS and the SIAS-6/SPS-6, the PSS-10.

Measurement Invariance

Measurement invariance shows that a measure is able to provide similarly accurate measurements of the same property under different conditions (Drasgow, 1984; Horn & McArdle, 1992). Multi-group CFA is the most commonly used method to determine measurement invariance in structural equation mode (Van De Schoot et al., 2015). The invariance test is a continuous procedure that gradually constrains the measured parameters (i.e., factor loadings, item thresholds, and residuals) to ensure that they are equal in the sample population at each level of invariance. The model from the previous step is nested in the model from the next stage, with the last model becoming progressively constrained. The exact procedure for assessing invariance is as follows: the Configural Invariance model (no invariance); the Metric Invariance model (factor loading invariant); the Scalar Invariance model (item intercepts invariant as well as the factor loading invariant); and finally, the Strict Invariance model (item uniqueness invariant, factor loading invariant, and item intercept invariant). The model fit indices were used to examine whether there was support for parameter invariance. For example, the constrained model is supported if the decrease in CFI and TLI is less than 0.01 compared to the previous model. When the RMSEA changes are smaller than 0.2, this is also considered to be evidence of a high-quality model. Chi-square difference testing is also performed to compare the less restrictive model with the other more restricted models (Cheung & Rensvold, 2001, 2002). However, significant differences in the chi-square difference test are highly likely to be influenced by the sample size.

IRT Analyses

Given the DSS’s five response options, a graded response model (GRM; Samejima, 1969) was used for the IRT analyses. If we score an item in a graded manner, the amount of information provided by an individual item or a test is greatly increased. The subject’s underlying ability will be denoted by $θ$ , and the probability of each ability level to give a correct answer will be denoted by $P (θ)$ . The GRM assumes that the options of each item are ordered consistently across items, and the GRM estimates the discrimination and threshold parameters for each item. The item discrimination parameter $(a)$ reflects the slope of the item characteristic curve (ICC) at the steepest point. The size of the discrimination parameter determines the magnitude of the difference in the probability of correct responses between adjacent level subjects. Also, $a$ values of less than 0.65 are considered unacceptable, 0.65 to 1.35 are moderate, and more than 1.35 are ideal (Baker, 2001). Threshold parameters $(b_{j})$ , also named difficulty parameters, are estimated for each item step. $b_{j}$ values between −3 and 3 are considered appropriate (Baker, 2001).

PARSCALE 4.1 was used to draw ICCs and test information curves to establish a clear representation of each item’s characteristics. The ICC can be used to describe the relationship between the probability of the correct response to an item and the ability scale. Test information curves are obtained by totaling all item information curves. An item information curve is a graph of the information function that compares the amount of information to a level of ability. The accuracy of the subject’s true ability level can be ensured based on the amount of information available at this ability level. Furthermore, item information is directly connected to measurement standard error (SE). The smaller the standard error of measurement, the more information there is in the test. The item offers the greatest amount of information to subjects at the ability level corresponding to the difficulty parameter.

The Translation Process of the DSS

Stage 1

The current research aimed to follow the rigorous back-translation procedure to create the Chinese version of the DSS (Brislin, 1986). First, the DSS was translated independently by two bilingual researchers (including the first author). During the translation, some disagreements were resolved through discussion. For example, terms that did not match the original intended meaning were revised (“Facebook” in the English version of the measure was replaced with “WeChat” in the Chinese version). Then another bilingual researcher translated the Chinese version back to English without referring to the original version of the DSS. The items in the two English versions were compared word by word. Finally, translators and educational measurement experts made necessary modifications and reached a consensus on the final Chinese version of the DSS.

Stage 2

During the pilot test, the Chinese version of the DSS was then given to 10 Chinese college students who were unrelated to this study. According to participants’ feedback, some items with high repeatability have caused wastefulness and irrationality. For instance, items 7, 9, and 11 which measure approval anxiety, have similar stems (i.e., “I am nervous about how people will respond to my posts and photos”; “I feel nervous after I share a post or a photo to see how others respond to it”; “I feel nervous about how others will respond when I post new updates on social media”). Considering the pilot test and students’ feedback, we found that some items with similar phrasing and conceptual overlap led to duplication and redundancy of items. The items that were removed are as follows: items 1, 9, 11, 12, 15, 18, and 22.

The Final Version

Once the translation and modification process was completed, the final version of the Chinese DSS retained only 17 of the original 24 items still covering the five factors: connection overload (items 2 through 6), approval anxiety (items 7, 8, and 10), online vigilance (items 13, 14, and 16), availability stress (items 17, 19, and 20), and FoMO (items 21, 23, and 24). Each item is rated on a five-point Likert scale.

Results

Factor Structure

The factor structure of the two models is compared in Table 1. Model 1 is the original DSS with 24 items and Model 2 is a short form of the DSS with 17 items. The fit indices of Model 1 were poor, with both CFI and TLI lower than 0.90 (RMSEA = 0.076, SRMR = 0.144, CFI = 0.780, TLI = 0.754). The model fit for Model 2 is sufficiently adequate: χ²/df = 3.63, p < .001, RMSEA = 0.06, SRMR = 0.046, CFI = 0.920, TLI = 0.900. χ²/df doesn’t meet the criterion of <3 due to the sample size. However, this was deemed nonetheless acceptable considering that the other indices had a good fitness. Figure 1 shows the factor loadings of Model 2, and the standardized loadings of all items were significant at p < .001. Also, the vast majority of the items had factor loadings above 0.60. The factor loading of the DSS ranges from 0.571 to 0.830. As a result, the five-factor structure of the DSS fits the Chinese sample well.

Table 1.

Fit Indices for DSS in Confirmatory Factor Analysis

Model	χ² (df)	χ²/df	$RMSEA$	$CFI$	$TLI$	$SRMR$
Model 1	395.533 (109)	3.628	0.060	0.920	0.900	0.046
Model 2	1,514.545 (291)	5.205	0.076	0.780	0.754	0.144

Note. Model 1 = the original DSS with 24 items; Model 2 = the adaption version of DSS with 17 items; $χ^{2}$ = chi-square; $χ^{2} / df$ = the ratio of chi-square to degrees of freedom; RMSEA = root mean square error of approximation; CFI = comparative fit index; TLI = Tucker-Lewis index.

Figure 1.

Five-factor model.

Internal Consistency and Criterion-Related Validity

The internal consistency of the DSS and its five subscales was evaluated by Cronbach’s $α$ . The Cronbach’s $α$ value of the DSS is .917, respectively, both of which are greater than .90 indicating a strong internal consistency. The Cronbach’s $α$ coefficient values for the DSS subscales range from .721 to .838.

According to Pearson’s correlation coefficient analyses, the Chinese version of the DSS demonstrated a good correlation with the SIAS-6/SPS-6. The DSS also correlated well with the PSS-10. The SIAS-6/SPS-6 ( $r$ = 0.560, p < .01) and the PSS-10 ( $r$ = 0.599, p < .01) showed a positive correlation with the DSS. The coefficients of correlations between all DSS subscales and the PSS-10 are positive and significant (0.326 < $r$ < 0.586, p < .01), and every DSS subscale is also positively related to the SIAS-6/SPS-6 (0.387 < $r$ < 0.573, p < .01). Table 2 shows detailed data on internal consistency estimates and the criterion validity of the scales.

Table 2.

Internal Consistency and Criterion-Related Validity.

	M (SD)	Cronbach’s $α$	PSS-10	SIAS-6/SPS-6
1. Total digital stress	45.05 (11.926)	.917	0.599*	0.560*
2. Connection overload	13.60 (4.177)	.838	0.586*	0.448*
3. Approval anxiety	7.85 (2.803)	.775	0.501*	0.467*
4. Online vigilance	9.25 (2.598)	.721	0.368*	0.430*
5. Availability stress	7.20 (2.468)	.764	0.326*	0.387*
6. Fear of missing out	7.15 (2.650)	.775	0.576*	0.573*

Note. Connection overload, approval anxiety, online vigilance, availability stress, and fear of missing out are subscales of the DSS; M = mean; SD = standard deviation.

p < .01.

Measurement Invariance

The results of the measurement invariance according to subjects’ gender are shown in Table 3. The configural invariance model (Model 1) fits the data well (χ² = 552.886, df = 218, RMSEA = 0.065, CFI = 0.923, TLI = 0.904). After constraining the factor loading invariant in Model 2, the results change slightly from Model 1 (RMSEA = 0.001, CFI = 0.001, TLI = 0.004). The chi-square difference test also reveals that the result was not significant. Model 3 has restrictions with item intercepts invariance and factor loading invariance (χ² = 631.202, df = 242, RMSEA = 0.067, CFI = 0.911, TLI = 0.900), and the fit indices change more than in Model 2 (RMSEA = 0.003, CFI = 0.011, TLI = 0.008). Our findings do not support the scalar invariance model since the CFI decreased in comparison to Model 2 to exceed 0.01, and the change in chi-square was also significant. The four-parameter levels of the equivalence test are nested within each other, with the model at the former level nested within the model at the latter level, and only if the equivalence at the former level holds can the equivalence test at the higher level continue (Byrne et al., 1989; Vandenberg & Lance, 2000). The following strict measurement invariance model could not be conducted since the scalar invariance model did not support it.

Table 3.

Measurement Invariance Tests by Gender.

Model	$χ^{2}$ $(df)$	$χ^{2} / df$	$RMSEA$	$CFI$	$TLI$	$Δ χ^{2}$ $(Δ df)$	$RMSEA$	$CFI$	$TLI$
1. Configural invariance	552.886 (218)	2.536	0.065	0.923	0.904
2. Metric invariance	570.55 (230)	2.481	0.064	0.922	0.908	17.664 (12)	0.001	0.001	0.004
3. Scalar invariance	631.202 (242)	2.608	0.067	0.911	0.900	60.652* (12)	0.003	0.011	0.008

Note. $χ^{2}$ = chi-square; $χ^{2} / df$ = the ratio of chi-square to degrees of freedom; RMSEA = root mean square error of approximation; CFI = comparative fit index; TLI = Tucker-Lewis index.

p < .01.

We also assessed measurement invariance according to the participants’ region (Table 4). Similar to gender, the fit indices in Model 1 are high ( $χ^{2}$ = 576.418, $df$ = 218, RMSEA = 0.068, CFI = 0.920, TLI = 0.900). Model 2 shows a small change in comparison to Model 1 (RMSEA = 0.002, CFI = 0, TLI = 0.005). The fit indices (CFI, TLI) are less than 0.01, and RMSEA is less than 0.2, which provides strong support for the metric invariance model. Model 3 is similar to Model 2 (RMSEA = 0.001, CFI = 0.004, TLI = 0.001), indicating that the scalar invariance model is supported. In Model 4 ( $χ^{2}$ = 631.733, $df$ = 259, RMSEA = 0.063, CFI = 0.916, TLI = 0.912), the fit indices are slightly lower than in Model 3 (RMSEA = 0.002, CFI = 0, TLI = 0.006). All results show that factor structure and uniqueness are invariant across the different regions.

Table 4.

Measurement Invariance Tests by Region.

Model	$χ^{2}$ $(df)$	$χ^{2} / df$	$RMSEA$	$CFI$	$TLI$	$Δ χ^{2}$ $(Δ df)$	$RMSEA$	$CFI$	$TLI$
1. Configural invariance	576.418 (218)	2.644	0.068	0.920	0.900
2. Metric invariance	588.376 (230)	2.558	0.066	0.920	0.905	11.958 (12)	0.002	0.000	0.005
3. Scalar invariance	614.806 (242)	2.541	0.065	0.916	0.906	26.43* (12)	0.001	0.004	0.001
4. Strict invariance	631.733 (259)	2.439	0.063	0.916	0.912	16.927 (17)	0.002	0.000	0.006

Note. $χ^{2}$ = chi-square; $χ^{2} / df$ = the ratio of chi-square to degrees of freedom; RMSEA = root mean square error of approximation; CFI = comparative fit index; TLI = Tucker-Lewis index.

p < .01.

IRT Analyses

Item Parameters

The item parameters of the Chinese version of the DSS are presented in Table 5. There are five items designated as connection overload (CO1–CO5), and three items each for approval anxiety (AA1–AA3), online vigilance (OV1–OV3), availability stress (AS1–AS3), and fear of missing out (FoMO1–FoMO3). For all of these 17 items, the discrimination parameters $(a)$ are above the minimum threshold of 0.65, and more than 1. The difficulty parameters $(b_{j})$ of the DSS are qualified with values ranging from −2.571 to 2.644.

Table 5.

Item Parameters.

Item	$a$	b1	b2	b3	b4
CO1	1.389	−1.282	−0.128	1.056	2.129
CO2	1.138	−1.383	−0.229	0.955	2.028
CO3	1.366	−1.16	−0.006	1.178	2.251
CO4	1.092	−1.604	−0.45	0.734	1.807
CO5	1.416	−1.48	−0.326	0.858	1.931
AA1	1.22	−1.579	−0.634	0.376	1.409
AA2	1.932	−0.954	−0.009	1.001	2.034
AA3	1.186	−0.55	0.395	1.405	2.438
OV1	1.071	−2.571	−1.491	−0.365	0.824
OV2	1.512	−1.525	−0.445	0.681	1.87
OV3	1.013	−0.934	−0.445	1.272	2.461
AS1	1.299	−1.15	0.028	1.34	2.394
AS2	1.397	−0.9	0.278	1.59	2.644
AS3	1.197	−1.034	0.144	1.456	2.51
FoMO1	1.377	−0.911	0.158	1.276	2.309
FoMO2	1.197	−0.827	0.242	1.36	2.393
FoMO3	1.522	−0.909	0.16	1.278	2.311

Note. a = discrimination parameters; b (b1–b4) = threshold parameters; connection overload (CO1–CO5); approval anxiety (AA1–AA3); online vigilance (OV1–OV3); availability stress (AS1–AS3); fear of missing out (FoMO1–FoMO3).

Item Characteristic Curves (ICC)

ICC was used to analyze the DSS item characteristics. The vertical axis shows the probability that the subjects answered correctly $(P_{ij}^{*} (θ))$ , and the horizontal axis indicates the subjects’ ability level $(θ_{i})$ . The curve in the coordinate axis is the response category curve, and the number marked on the curve represents the corresponding options (1–5). The options correspond to scores of 0 to 4, respectively, for example, a curve marked with 1 indicates that the subject agreed with the first option of “never” in the item, therefore scoring 0.

Part of the ICCs are shown in Figure 2, and all of them can be found in the Appendix. For instance, Item 1 in the connection overload indicates that the item has five response levels ranging from 1 (“never”) to 5 (“always”). Curve 3 indicates that subjects with a latent $θ$ in the range (0, 1) have the highest probability of scoring 2 (i.e., endorsing the third option) in response to the item. In other words, subjects with a latent ability range of (0, 1) are more likely to choose the third option (“some of the time”). Subjects with a higher ability level are more likely to endorse the fifth option, while the subjects with a lower ability level are more likely to choose the first option. This is to say that, on the same curve level, subjects with different ability levels have different probabilities of scoring along that curve, and the probability of getting a different score level (curve) also differs for subjects of another ability level. Item 1 of connection overload is excellent. Combining the results of the item parameters, however, there are some items that require further analysis. Take the items 1of online vigilance for example. When the subjects’ ability level is negative infinity, their probability of scoring 1 (Grade curve label 2) and 2 (Grade curve label 3) is too high, and the curves labeled 2, 3, and 4 were too closely intertwined. For Item 3 of online vigilance, the curve peaks for each level are not significant. The probability of scoring 0 (Grade curve label 1) is greater than the probability of scoring 1 (Grade curve label 2) for a subject with an ability level of (−1, 0). However, in an ideal case, the probability of scoring 1 should be greater than scoring 0 for a subject with this ability level. Based on the above analysis, then, we can consider that items 1 and 3 of online vigilance might be problematic in terms of their categorization and require further revision. Similar problems are seen with items of approval anxiety.

Figure 2.

Part of the item characteristic curves (ICCs).

Test Information Curves and Standard Error Curves

The information function describes the estimated degree of each ability level. The information value of an item becomes bell-shaped with the level of the subject’s ability trait. As shown in Figure 3, the horizontal axis is the latent ability, the left vertical coordinate is the amount of information in the test, and the right vertical coordinate is the standard error. The dashed line in the graphs indicates the standard error curves, and the solid line indicates the test information curves. The test information curve of connection overload had the highest information with 6.38 at $θ$ = 0. The information reached a maximum of between $θ$ = −0.3 and $θ$ = 0.6, and the SE ranged from 0.07 to 0.09. Marginal reliability is 99.19%. At $θ$ = 0, approval anxiety, availability stress, and FoMO had the most precise information values: 4.37, 3.85, and 4.37, respectively. The marginal reliabilities for all the subscales are greater than 95%. The results show that all subscales provide sufficient information and have good reliability.

Figure 3.

Test information curves and standard error curves for all subscales.

Discussion

Digital stress is prevalent due to the regular use of social media in daily life, and many adolescents experience this stress regardless of gender, race, or ethnicity (Nick et al., 2022). Digital stress is an important indicator of the relationship between moderated social media use and mental health, and people will benefit from the assessment of digital stress. The purpose of this study was to examine the factor structure and psychometric properties of the Chinese version of the DSS in a Chinese sample by assessing the Chinese college students’ level of digital stress. With nearly a quarter of the original DSS items removed, our new brief DSS is still reliable and captures the core features of digital stress. Hence, the following discussion is centered on the hypotheses previously formulated in this study.

According to hypothesis 1 (H1), the five-factor model with 17items is shown to be robust, indicating that the DSS can be used to measure digital stress in the Chinese context. The CFA results demonstrate that the five-factor model of the DSS fits the data well in Chinese samples, and is consistent with the theoretical model of Hall et al. (2021). The index of CFI and TLI are both above 0.90 and the RMSEA is less than 0.80. These findings suggest that the DSS can be generalized to the Chinese population in order to assess digital stress.

Second, consistent with previous research hypotheses (H2), the Chinese version of the DSS shows satisfactory internal consistencies ( $α$ = .85). With the deletion of nearly a quarter of the original DSS items, it also has reliability while still covering the core characteristics of digital stress. Consistent with our expectations, the Chinese version of the DSS also has good internal consistency with Cronbach’s $α$ (.917) when used in a Chinese student sample. In addition, the criterion validity of the DSS is supported by correlations between the DSS and other measures (i.e., the PSS-10 and the SIAS-6/SPS-6). The Perceived Stress Scale (PSS-10) was designed to measure participants’ self-perceived stress levels (Cohen & Williamson, 1988) and the SIAS-6/SPS-6 is a self-report scale used to assess social anxiety (Mattick & Clarke, 1998). FoMO relates positively to anxiety and stress with correlations above .50, while availability stress has the lowest correlation with anxiety and stress among the five dimensions, with correlations only above .30. These findings are consistent with previous literature reviews, mental health correlates of social media use are at risk for anxiety, loneliness, and depressive symptoms (Barry et al., 2017; Rosen et al., 2013; Weinstein et al., 2016). In summary, these correlations in general support the criteria validity of the DSS, with the DSS and its subscales correlating positively and significantly with the PSS-10 and the SIAS-6/SPS-6.

With regard to our third research hypothesis (H3), the measurement invariance tests for gender indicate that the factor structure (factor loading and item intercept) of the DSS is not invariant between males and females. Therefore, subjects of different genders might have biased results when using the DSS for digital stress testing. The effects of social media can be gender-conditional, and different kinds of social media activities are associated with well-being to different degrees (Svensson et al., 2022). Choi (2022) found that women tend to use social media for relationship maintenance and have more social comparison orientation. Females are more likely to use online social media communities than males (Ahn, 2011). Therefore, there is a slight difference between the results obtained using the DSS to measure digital stress levels in men and women, with women being more likely to be exposed to digital stress in the social media environment. Furthermore, the results of the study may be influenced by the male-to-female ratio of the sample. The number of female students in this study surpassed the number of male students by one-fifth. Finally, the factor structure and uniqueness for the DSS also hold for measurement equivalence across different regions, which means that the scale is equally effective for both urban and rural respondents.

Regarding the final research hypothesis of this study (H4), the results of the IRT analyses show that the Chinese version of the DSS has excellent discrimination parameters and threshold parameters. While the ICCs are generally good, a few items do require further adjustment. For example, two items for online vigilance appear to be flawed. The distinction between the scores in Item 1 (“I must have my phone with me to know what is going on”) is not evident, particularly with scores ranging from 0 to 1 (from “never” to “some of the time”). A similar issue occurs with Item 3 (“I feel socially unavailable when I do not have my phone”). Nevertheless, the test information curves show that the Chinese version of the DSS is a credible measure for the vast majority of its intended subjects.

Limitations and Future Direction

There are several limitations to the current study. First, although the study’s findings demonstrate that the subscales are reliable based on the CFA, some of the items will nonetheless need to be modified and enhanced by IRT. And our study does not provide specific suggestions for future item modifications. Second, the current study looked at only undergraduate and graduate students at Chinese universities. Although our sample was well distributed across all study years, it nonetheless comprised primarily freshmen and sophomores (69.9% of the total sample) which is not conducive to the generalization of our findings. Future research should evaluate the model structure and item functioning of the Chinese version of the DSS using other samples. Finally, future researchers can investigate the current status of digital stress levels among Chinese college students and study the relationship between college students’ digital stress levels and academic performance. All offline campus academic activities were suspended during the COVID-19 pandemic, and the virtual community in digital spaces became the only way to participate with others socially. Future research should provide recommendations to help students and instructors overcome the shortcomings of social media.

Conclusion

The current study focused on assessing the psychometric properties adequacy of the Chinese version of the DSS. The excellent fit indices and psychometric properties all prove that the Chinese version of the DSS has adequate reliability and validity in Chinese samples. The Criterion-Related validity also shows that the Chinese version of the DSS has a basic agreement with other indicators of stress. According to its measurement invariance, the Chinese version of the DSS has shown itself to be regionally invariant, but gender differences might influence the effect of the test. Nonetheless, the factor analysis and item function support the Chinese version of the DSS in Chinese college students. The Chinese version of the DSS can serve as a valid and reliable alternative to the original DSS for Chinese college students in nonclinical settings. Additional research would be necessary to determine if the findings generalize to a broader Chinese population. Furthermore, these psychometric investigations of the DSS will allow future research to investigate construct validity better.

Supplemental Material

sj-docx-1-sgo-10.1177_21582440241247406 – Supplemental material for Factor Structure and Psychometric Properties of the Digital Stress Scale in a Chinese College Sample

Supplemental material, sj-docx-1-sgo-10.1177_21582440241247406 for Factor Structure and Psychometric Properties of the Digital Stress Scale in a Chinese College Sample by Chunlei Gao, Mingqing Jian and Ailin Yuan in SAGE Open

Footnotes

Acknowledgements

The author would like to thank the students who kindly gave of their time when participating in this study.

Author’s Contribution

All authors contributed to the study conception and design. Material preparation was performed by Chunlei Gao and Ailin Yuan, data collection and analysis were performed by Chunlei Gao and Mingqing Jian. The first draft of the manuscript was written by Mingqing Jian and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by the National Educational Examination Research Planning 2021 Annual Project (GJK2021025).

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

ORCID iD

Mingqing Jian

Supplemental Material

Supplemental material for this article is available online.

Data Availability Statement

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

Ahn

(2011). Digital divides and social network sites: Which students participate in social media? Journal of Educational Computing Research, 45(2), 147–163. https://doi.org/10.2190/EC.45.2.b

Baker

F. B.

(2001). The basics of item response theory. ERIC Clearinghouse on Assessment and Evaluation.

Banjanin

Dimitrijević

Pantic

(2015). Relationship between internet use and depression: Focus on physiological mood oscillations, social networking and online addictive behavior. Computer in Human Behavior, 43, 308–312. https://doi.org/10.1016/j.chb.2014.11.013

Barry

C. T.

Sidoti

C. L.

Briggs

S. M.

Reiter

S. R.

Lindsey

R. A.

(2017). Adolescent social media use and mental health from adolescent and parent perspectives. Journal of Adolescence, 61, 1–11. https://doi.org/10.1016/j.adolescence.2017.08.005

Beyens

Frison

Eggermont

(2016). I don’t want to miss a thing: Adolescents’ fear of missing out and its relationship to adolescents’ social needs, Facebook use, and Facebook related stress. Computers in Human Behavior, 64, 1–8. https://doi.org/10.1016/j.chb.2016.05.083

Brislin

R. W.

(1986). The wording and translation of research instrument. Sage.

Byrne

B. M.

Shavelson

R. J.

Muthén

(1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105(3), 456–466. https://doi.org/10.1037/0033-2909.105.3.456

Calancie

Ewing

Narducci

L. D.

Horgan

Khalid-Khan

(2017). Exploring how social networking sites impact youth with anxiety: A qualitative study of Facebook stressors among adolescents with an anxiety disorder diagnosis. Cyberpsychology: Journal of Psychosocial Research on Cyberspace, 11(4), Article 2. https://doi.org/10.5817/CP2017-4-2

Carr

C. T.

Hayes

R. A.

(2015). Social media: Defining, developing, and divining. Atlantic Journal of Communication, 23(1), 46–65. https://doi.org/10.1080/15456870.2015.972282

10.

Cheung

G. W.

Rensvold

R. B.

(2001). The effects of model parsimony and sampling error on the fit of structural equation models. Organizational Research Methods, 4(3), 236–264. https://doi.org/10.1177/109442810143004

11.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233–255. https://doi.org/10.1207/S15328007SEM0902_5

12.

Choi

(2022). Do Facebook and Instagram differ in their influence on life satisfaction? A study of college men and women in South Korea. Cyberpsychology: Journal of Psychosocial Research on Cyberspace, 16(1), Article 2. https://doi.org/10.5817/CP2022-1-2

13.

Cohen

Williamson

G. M.

(1988). Perceived stress in a probability sample of United States. In Spacapan

Oskamp

(Eds.), The social psychology of health: Claremont symposium on applied social psychology (pp. 31–67). Sage.

14.

Drasgow

(1984). Scrutinizing psychological tests: Measurement equivalence and equivalent relations with external variables are the central issues. Psychological Bulletin, 95(1), 134–135. https://doi.org/10.1037/0033-2909.95.1.134

15.

Davis

(2012). Friendship 2.0: Adolescents’ experiences of belonging and self-disclosure online. Journal of Adolescence, 35(6), 1527–1536. https://doi.org/10.1016/j.adolescence.2012.02.013

16.

Ehrenreich

S. E.

Underwood

M. K.

(2016). Adolescents’ internalizing symptoms as predictors of the content of their Facebook communication and responses received from peers. Translational Issues in Psychological Science, 2(3), 227–237. https://doi.org/10.1037/tps0000077

17.

Freytag

Knop-Huelss

Meier

Reinecke

Hefner

Klimmt

Vorderer

(2021). Permanently online—Always stressed out? The effects of permanent connectedness on stress experiences. Human Communication Research, 47(2), 132–165. https://doi.org/10.1093/hcr/hqaa014

18.

Hall

J. A.

Steele

R. G.

Christofferson

J. L.

Mihailova

(2021). Development and initial evaluation of a multidimensional digital stress scale. Psychological Assessment, 33(3), 230–242. https://doi.org/10.1037/pas0000979

19.

Horn

J. L.

McArdle

J. J.

(1992). A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research, 18(3–4), 117–144. https://doi.org/10.1080/03610739208253916

20.

L.-T.

Bentler

P. M.

(1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3(4), 424–453. https://doi.org/10.1037/1082-989X.3.4.424

21.

Keles

McCrae

Grealish

(2020). A systematic review: The influence of social media on depression, anxiety and psychological distress in adolescents. International Journal of Adolescence and Youth, 25(1), 79–93. https://doi.org/10.1080/02673843.2019.1590851

22.

Lai

Zhang

Shi

Qiao

Boswell

Rozelle

(2013). Computer assisted learning as extracurricular tutor? Evidence from a randomised experiment in rural boarding schools in Shaanxi. Journal of Development Effectiveness, 5, 208–231. https://doi.org/10.1080/19439342.2013.780089

23.

LaRose

Connolly

Lee

Hales

K. D.

(2014). Connection overload? A cross cultural study of the consequences of social media connection. Information Systems Management, 31(1), 59–73. https://doi.org/10.1080/10580530.2014.854097

24.

Liu

Baumeister

R. F.

Yang

(2019). Digital communication media use and psychological well-being: A meta-analysis. Journal of Computer-Mediated Communication, 24(5), 259–273. https://doi.org/10.1093/jcmc/zmz013

25.

Bian

Wang

Zhao

(2017). Chinese version of the Perceived Stress Scale-10: A psychometric study in Chinese university students. PloS One, 12(12), e0189543. https://doi.org/10.1371/journal.pone.0189543

26.

Mattick

R. P.

Clarke

J. C.

(1998). Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behaviour Research and Therapy, 36(4), 455–470. https://doi.org/10.1016/S0005-7967(97)10031-6

27.

Muthén

L. K.

Muthén

B. O.

(1998–2015). Mplus user’s guide (7th ed.). Muthén & Muthén.

28.

Nick

E. A.

Kilic

Nesi

Telzer

E. H.

Lindquist

K. A.

Prinstein

M. J.

(2022). Adolescent digital stress: Frequencies, correlates, and longitudinal association with depressive symptoms. The Journal of Adolescent Health: Official Publication of the Society for Adolescent Medicine, 70(2), 336–339. https://doi.org/10.1016/j.jadohealth.2021.08.025

29.

Odgers

C. L.

Jensen

M. R.

(2020). Annual research review: Adolescent mental health in the digital age: Facts, fears, and future directions. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 61(3), 336–348. https://doi.org/10.1111/jcpp.13190

30.

Ouyang

Cai

(2020). Psychometric properties of the short forms of the social interaction anxiety scale and the social phobia scale in a Chinese college sample. Frontiers in Psychology, 11, 2214. https://doi.org/10.3389/fpsyg.2020.02214

31.

Reilly

O. M.

(2020). Social media and adolescent mental health: The good, the bad and the ugly. Journal of Mental Health, 29(2), 200–206. https://doi.org/10.1080/09638237.2020.1714007

32.

Reinecke

Aufenanger

Beutel

M. E.

Dreier

Quiring

Stark

Wölfling

Müller

K. W.

(2017). Digital stress over the life span: The effects of communication load and Internet multitasking on perceived stress and psychological health impairments in a German probability sample. Media Psychology, 20, 90–115. https://doi.org/10.1080/15213269.2015.1121832

33.

Rosen

L. D.

Whaling

Rab

Carrier

L. M.

Cheever

N. A.

(2013). Is Facebook creating “iDisorders”? The link between clinical symptoms of psychiatric disorders and technology use, attitudes and anxiety. Computers in Human Behavior, 29(3), 1243–1254. https://doi.org/10.1016/j.chb.2012.11.012

34.

Samejima

(1969). Estimation of latent ability using a response pattern of graded scores (psychometric monograph no. 17). Psychometric Society.

35.

Steele

R. G.

Hall

J. A.

Christofferson

J. L.

(2020). Conceptualizing digital stress in adolescents and young adults: Toward the development of an empirically based model. Clinical Child and Family Psychology Review, 23, 15–26. https://doi.org/10.1007/s10567-019-00300-5

36.

Steiger

J. H.

(1998). A note on multiple sample extensions of the RMSEA fit index. Structural Equation Modeling: A Multidisciplinary Journal, 5(4), 411–419. https://doi.org/10.1080/10705519809540115

37.

Streiner

D. L.

(2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80(1), 99–103. https://doi.org/10.1207/S15327752JPA8001_18

38.

Zhang

Yan

Huang

Cheng

Cai

Cui

(2021). Online medical teaching in China during the COVID-19 pandemic: Tools, modalities, and challenges. Frontiers in Public Health, 9, 797694. https://doi.org/10.3389/fpubh.2021.797694

39.

Svensson

Johnson

Olsson

(2022). Does gender matter? The association between different digital media activities and adolescent well-being. BMC Public Health, 22(1), 273. https://doi.org/10.1186/s12889-022-12670-7

40.

Testa

Massa

Martini

Appio

F. P.

(2020). Social media-based innovation: A review of trends and a research agenda. Information and Management, 57(3), 1–17. https://doi.org/10.1016/j.im.2019.103196

41.

Twenge

J. M.

(2019). More time on technology, less happiness? Associations between digital-media use and psychological well-being. Current Directions in Psychological Science, 28(4), 372–379. https://doi.org/10.1177/0963721419838244

42.

Twenge

J. M.

Campbell

W. K.

(2019). Media use is linked to lower psychological well-being: Evidence from three datasets. The Psychiatric Quarterly, 90(2), 311–331. https://doi.org/10.1007/s11126-019-09630-7

43.

Twenge

J. M.

Martin

G. N.

(2020). Gender differences in associations between digital media use and psychological well-being: Evidence from three large datasets. Journal of Adolescence, 79, 91–102. https://doi.org/10.1016/j.adolescence.2019.12.018

44.

Vandenberg

R. J.

Lance

C. E.

(2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4–70. https://doi.org/10.1177/109442810031002

45.

Van De Schoot

Schmidt

De Beuckelaer

Lek

Zondervan-Zwijnenburg

(2015). Editorial: Measurement invariance. Frontiers in Psychology, 6, 152747, Article 1064. https://doi.org/10.3389/fpsyg.2015.01064

46.

Weinstein

E. C.

Selman

R. L.

(2016). Digital stress: Adolescents’ personal accounts. New Media & Society, 18(3), 391–409. https://doi.org/10.1177/1461444814543989

47.

Woods

H. C.

Scott

(2016). #Sleepyteens: Social media use in adolescence is associated with poor sleep quality, anxiety, depression and low self-esteem. Journal of Adolescence, 51, 41–49. https://doi.org/10.1016/j.adolescence.2016.05.008

48.

Zhang

Jin

Torero

(2018). Teachers and urban-rural gaps in educational outcomes. American Journal of Agricultural Economics, 100, 1207–1223. https://doi.org/10.1093/ajae/aay009

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.20 MB