Sage Journals: Discover world-class research

Abstract

The Multidimensional Psychological Flexibility Inventory (MPFI) is a complex self-report measurement tool designed to assess Acceptance and Commitment Therapy’s (ACT) Hexaflex model components of psychological flexibility and inflexibility. There is a lack of literature on the invariance of the MPFI and its discriminant validity in relationship to second-wave therapy concepts such negative automatic thoughts. In this study, we present a Romanian adaptation and validation of the MPFI performed on a sample of 1109 participants from the general population. The results indicated an adequate fit to the data of the original MPFI factor model, high internal consistency levels, the ability to be invariant across levels of gender and age, and supported the MPFI’s convergent, and discriminant. Some theory incongruent correlations emerged between MPFI dimensions and acceptance and experiential avoidance that suggest the need for further refinement of the MPFI. We discuss this unexpected correlation in relationship to explanatory factors such as the COVID lockdown context, personality, and age. Altogether, our results indicate that the Romanian version of the MPFI is a psychometrically sound instrument that can be used to study the Hexaflex ACT model and to evaluate psychological flexibility and inflexibility and their sub-processes in practice.

Keywords

psychological flexibility MPFI ACT CFA invariance validation

ACT and the Hexaflex Model

Acceptance and Commitment Therapy (ACT) is a third-wave cognitive-behavioral therapy that focuses on fostering psychological flexibility through acceptance and mindfulness techniques combined with commitment and behavioral change strategies (Hayes, 2004). Unlike traditional cognitive-behavioral therapy (CBT), which often emphasizes modifying negative thoughts, ACT encourages individuals to accept their thoughts and emotions rather than fighting against them (Hayes et al., 1999). ACT posits that psychological suffering often arises not from the content of thoughts or feelings themselves but from how individuals respond to these internal experiences. By increasing awareness and fostering an open, non-judgmental stance toward inner experiences, ACT helps people break free from unproductive attempts to control or avoid these experiences (Hayes et al., 2006). Research has shown ACT to be effective in treating a range of psychological disorders, including anxiety, depression, and chronic pain, with meta-analyses indicating that it can be as effective as traditional CBT and sometimes even more so for specific conditions (A-Tjak et al., 2015; Gloster et al., 2020). Its emphasis on values-driven action and meaningful engagement with life makes it attractive to those seeking therapeutic approaches that enhance quality of life, rather than solely aiming to alleviate symptoms (Harris, 2019).

Developed in the 1980s by psychologist Steven Hayes, ACT is grounded in Relational Frame Theory (RFT), which examines how language and cognition affect human behavior. According to RFT, humans are able to use language and cognition to arbitrarily relate events to one another and alter their meaning based on their relations with other things or events (Hayes et al., 2006). In ACT, this ability is called psychological flexibility, while the impairment of this ability is called psychological inflexibility and is considered to be a root cause of psychopathology (Kashdan & Rottenberg, 2010). Psychological flexibility can be operationalized as the tendency to respond to challenging situations in ways that facilitate valued goal pursuit, and is most important in challenging situations that provoke distress (Doorley et al., 2020). Flexible people can deal with difficult situations while being present and fully aware of their thoughts, emotions, and the situation, accepting them without judgment and acting in accordance with their values. An ACT therapist’s primary goal is not the reduction of psychopathological symptoms, but rather the development of psychological flexibility – the ability to relate things to one another and learn from these new relationships.

Psychological flexibility is conceptualized in ACT as consisting of 6 processes: acceptance, present moment awareness, self as context, defusion, values, and committed action. These processes form the ACT hexaflex model of healthy emotional functioning (Stoddard & Afari, 2014). On the other hand, psychological inflexibility consists of six other processes: experiential avoidance, lack of contact with the present moment, self as content, fusion, lack of contact with values, and inaction. These processes form the ACT inflexahex model of psychopathology (Hayes et al., 2006). Traditionally, psychological flexibility and inflexibility and their corresponding processes were viewed as two poles of the same continuum (e.g., low flexibility means high inflexibility, low acceptance means high experiential avoidance, low fusion means high defusion, etc.). However, recently it was proposed that each of the twelve processes is a distinct construct that can be measured by its own indicators, based on recent developments in literature that proposed that flexibility and inflexibility components are distinct processes (Rolffs et al., 2016).

Multidimensional Psychological Flexibility Inventory

The Multidimensional Psychological Flexibility Inventory (MPFI; Rolffs et al., 2016) is a self-report measurement tool based on Acceptance and Commitment Therapy’s (ACT) Hexaflex model (Hayes et al., 2006). MPFI was developed to address the lack of a unitary and exhaustive tool to measure all 12 dimensions of psychological flexibility and inflexibility. At the moment of its conception, available options of ACT measurement tools consisted of either overall assessment of flexibility and inflexibility (Acceptance and Action Questionnaire–II, AAQ-II; Bond et al., 2011), measurements of individual Hexaflex dimensions (Valued Living Questionnaire, VLQ; VanBuskirk et al., 2012), or measurements with close conceptual similarity to psychological flexibility (such as Difficulties in Emotion Regulation scale, DERS; Gratz & Roemer, 2004). Since its development, MPFI has demonstrated psychometric soundness across multiple validation studies (Barr, 2022; Rogge et al., 2019; Seidler et al., 2020; Thomas et al., 2022) and was adapted in numerous languages such as Mandarin Chinese and Japanese (Lin et al., 2020), Italian (Landi et al., 2021), Iranian (Azadfar et al., 2022; Moosivand et al., 2024), Swedish (Sundström et al., 2023; Tabrizi et al., 2023), Spanish (Simkin et al., 2023), Persian (Hekmati et al., 2024). Additionally, a short, 24-item version of the MPFI was developed to provide a quicker assessment tool with psychometric properties that are comparable to the long MPFI (Cunha et al., 2024; Grégoire et al., 2020; Seidler et al., 2020).

To test the structural validity of the MPFI, researchers typically test the foundational ACT theory congruent model in which the 60 items of the MPFI load into 12 factors that represent the elements of the Hexaflex and Inflexahex model—these 12 factors load on two second-order factors – psychological flexibility and psychological inflexibility. The foundational model typically indicates good levels of fit across studies (Azadfar et al., 2022; Hekmati et al., 2024; Landi et al., 2021; Moosivand et al., 2024; Rolffs et al., 2016; Simkin et al., 2023; Sundström et al., 2023; Tabrizi et al., 2023; Thomas et al., 2022). One exception is the study of Seidler et al. (2020) that found that the foundational model is insufficient for a good fit, and had to add extra residual correlations between the fusion and defusion sub-scales and allowed defusion to load on both flexibility and inflexibility to obtain an adequate model fit. Other possible factorial structures, such as a 2-factor model (flexibility and inflexibility), were also previously tested, and varying results were obtained. Thomas et al. (2022) found a good fit of the 2-factor model, while in Rolffs et al. (2016) and Hekmati et al. (2024), this structure showed an unacceptable fit. Additionally, in some studies, authors test as 12-factor solution, without the two second order factor from the foundational model, that on all occasions have shown a good fit (Barr, 2022; Hekmati et al., 2024; Tabrizi et al., 2023) It seems that in the current state of literature, two competing models exist that describe the structure of the MPFI – the foundational model with 12 dimensions and two higher-order factors, and the 12 factors solutions. This indicates that the factorial structure of the MPFI should be further explored.

The literature generally reports MPFI good levels of construct validity (Landi et al., 2021; Rogge et al., 2019; Rolffs et al., 2016; Sundström et al., 2023; Tabrizi et al., 2023), assessed through correlations with conceptually close measures (convergent validity), conceptually distinct measures (discriminant validity), or through examining average explained variance (AVE). However, on some occasions, studies also reported concerns regarding the convergent validity of some of the dimensions of the MPFI. For instance, Rogge et al. (2019) have found low correlations of self-as-content and lack of contact with values dimensions with conceptually similar measures. Additionally, several studies have found convergent validity issues for the acceptance and experiential avoidance subscales (Landi et al., 2021; Sundström et al., 2023; Tabrizi et al., 2023). Tabrizi et al. (2023) reported some discriminant validity concerns for the Values and Committed Action factors. Previous studies also did not explore the ability of the MPFI to discriminate between psychological flexibility and inflexibility and second-wave therapies (Hayes, 2004) constructs, such as negative automatic thoughts (Beck, 2011). ACT, as a third-wave therapy, bears similarities with second and first-wave therapies (Brown et al., 2011). However, ACT has completely re-conceptualised the therapeutic approach, focusing on accepting negative thoughts and feelings as a natural part of life rather than trying to eliminate or modify them, while also developing psychological flexibility and living in accordance with one’s values. Negative automatic thoughts have long been considered to be at the basis of psychopathology (Beck, 2011), similar to psychological inflexibility in the third-wave ACT approach (Kashdan & Rottenberg, 2010). Given the theoretical similarities of these two concepts, it is essential to assess the overlap between MPFI dimensions and measures of negative automatic thoughts.

An underexplored property of the MPFI is measurement invariance - the extent to which the MPFI measures the same construct in the same way across different groups. So far, Lin et al. (2020) have shown cross-cultural invariance of the MPFI across five populations: Caucasian American, Asian American, Chinese, Japanese, Taiwanese, and Landi et al. (2021) have successfully demonstrated the invariance of MPFI for gender (male vs female), age (18–24 years, vs. 25–34 years, vs. 35–44 years, vs. 45–83 years), and in clinical vs non-clinical respondents. Future studies are needed to check if the meaning and interpretation of the test items remain consistent across these different contexts.

Present Study

Given the results of previous studies, we intend to test the following hypotheses.

Hypothesis 1

The foundational model of the MPFI with 12 dimensions and two higher-order factors will show a good fit to the data. We will also test models previously proposed in the literature to compare the foundational model and potentially find if another structural model better fits the data. Specifically, we will test the 2-factor model and the 12-factor model.

Hypothesis 2

The MPFI dimensions will demonstrate convergent validity, assessed through correlations with conceptually similar constructs. We will also examine the average variance explained to establish convergent validity.

Hypothesis 3

The MPFI dimensions will demonstrate discriminant validity and be assessed through correlations with conceptually distinct constructs. To establish discriminant validity, we will also examine the average variance explained, average shared squared variance, and maximum shared square variance.

Hypothesis 4

All MPFI dimensions will be invariant across gender and age.

Method

Sample

We initially collected 1163 responses from Romanian participants recruited among community members and psychology undergraduates, with M_age = 43.39, SD_age = 15.63, of which 46.60% were female. We removed 33 participants who completed the questionnaire in less than 7 minutes, since the questionnaire took about 26.85 minutes on average. An additional 21 participants who failed at least one out of four attention check questions were removed (attention check example: “This is a verification question. Please select option 4 - Frequently.”). One hundred forty participants provided incomplete responses. Out of 140, 64 provided complete responses to the 60 MPFI items (MPFI was the first administered measure). The remaining 76 completed the MPFI and one or more other measures. We included them in the analysis due to a lack of obvious issues with the provided responses, such as incorrect responses to attention check questions, straight liners, short time of completion, and due to the possibility of these responses contributing to the analyses. After participant removal, we were left with 1109 responses included in the analyses, that had a mean age of M = 43.6, SD = 15.7 of which N = 527, 47.5% were female. A more detailed presentation of the demographics is presented in the online supplemental materials (Supplement 1).

Translation and Adaptation Steps

The translation and adaptation of the MPFI followed a forward and back-translation process. We started the process by creating two forward translation versions of the MPFI. The first version was created by author I. C., and the second version was created by author Ș. M. Both authors who participated in the translation are fluent in Romanian and English. We (all the authors) then compared the two versions and created a third final forward translation version. The team involved in the translation process comprises all psychotherapy practitioners (except Ș. M.) with experience with ACT concepts. The third version was back-translated by an English university teacher at West University of Timișoara. The back-translated version was then compared to the original MPFI, and adjustments were made where needed. The back-translated version was also submitted to the original author of the MPFI for feedback. In the final step of the translation, we asked 16 colleagues to rate the clarity of each item on a scale from 1 = “Completely unclear” to 4 = “Completely clear” and write in a textbox their suggestions for better wording and indicate what seems unclear to them. This step helped establish face validity of the Romanian MPFI and adjust the wording of items in places where participants showed a lack of clarity or ambiguity. This step yielded the final version of the Romanian MPFI, after which we proceeded with data collection. We provide the final translated and validated version of the MPFI in an online supplemental material (Supplement 2).

Measures

The Multidimensional Psychological Flexibility Inventory (MPFI; Rolffs et al., 2016) consists of 60 self-report items that measure psychological flexibility and inflexibility. Psychological flexibility is comprised of six sub-dimensions: acceptance, present moment awareness, self as context, defusion, values, committed action; and psychological inflexibility is comprised of other six sub-dimensions: experiential Avoidance, lack of contact with the present moment, self as content, fusion, lack of contact with values, inaction. For each dimension and subdimension, a high score indicates a high level. Answers are collected on a six-point scale where 1 = “never true” and 6 = “always true”.

Acceptance and Action Questionnaire-II (AAQ-II; Bond et al., 2011) is a 10-item measure of psychological inflexibility. Responses to AAQ-II are offered on a 7-point scale where 1 = “never true” and 7 = “always true”. High scores on AAQ-II indicate high levels of psychological inflexibility. In this study, we used the Romanian version of the 10-item version of the AAQ-II (Szabó et al., 2011). However, Bond et al. (2011) have refined the AAQ-II into a shorter 7-item version, proving better psychometric properties than the 10-item version. Because of this, we included both versions of the questionnaire in the analyses.

Satisfaction with Life Scale (SWLS; Diener et al., 1985) is a 5-item self-report measure of satisfaction with life. Items are answered on a 5-point scale from 1 = “Strongly Disagree” to 5 = “Strongly Agree”. A high SWLS score indicates a high level of Satisfaction with life. In this study, we used the Romanian version of the SWLS, created by Dimitrova et al. (2016).

General Anxiety Disorder (GAD-7; Spitzer et al., 2006) is a 7-item self-report measure of anxiety. Answers are recorded on a 4-point scale, where 0 = “Not at all” and 4 = “Almost daily”. A high GAD-7 score indicates high levels of anxiety. We used the Romanian version of GAD-7, which is available at https://www.phqscreeners.com/select-screener.

Patient Health Questionnaire (PHQ-9; Kroenke et al., 2001) is a nine-item depression measure. The answers are recorded on a four-point scale, from 0 = “Not at all” to 4 = “Almost daily.” A high PHQ-9 score indicates high levels of depression. We used the Romanian version of PHQ-9, available at https://www.phqscreeners.com/select-screener.

Comprehensive Assessment of Acceptance and Commitment Therapy Processes (CompACT; Francis et al., 2016) is a 23-item measure of psychological flexibility. It also consists of 3 sub-dimensions: behavioural awareness, valued action, and openness to experience. A high score on CompACT indicates high levels of psychological flexibility. Responses are recorded on a 7-point scale from 1 = “Never true” to 7 = “Always true”. We used the Romanian version of CompACT, adapted by Călinici and Călinici (2021).

Mental Health Continuum Short Form (MHC-SF; Lamers et al., 2011) is a 14-item measure of well-being; a higher score indicates high levels of well-being. MHC-SF has three sub-dimensions of emotional, social, and psychological well-being. Responses are collected on a 6-point scale, where 1 = “Never” and 6 = “Every day.” We used the Romanian adaptation of MHC-SF, created by Żemojtel-Piotrowska et al. (2018).

Cognition Checklist (CCL; Beck et al., 1987) is a 26-item self-report measure of negative automatic thoughts. It has two sub-dimensions: depressive thoughts and anxious thoughts. A high score on the CCL indicates a high-level endorsement of negative thoughts. Responses are recorded on a 5-point scale from 1 = “Never” to 5 = “Always”. No Romanian adaptation study of the CCL is available; however, this scale was previously used by Marian and Sava (2023). In the original development study of the CCL by Beck et al. (1987).

The Automatic Thoughts Questionnaire (ATQ; Hollon & Kendall, 1980) is a 15-item self-report measure to assess negative thinking. A high score on the ATQ indicates a high occurrence of automatic negative thoughts. Responses are collected on a scale of 1 = “Never” to 5 = “Always. In this study, we used the Romanian version of the ATQ created by Hollon and Kendall (2007).

Procedure

The study has received ethical approval from the Scientific Council of University Research and Creation of the West University of Timișoara, ethical approval number 67003/02.12.2021. Data collected for this study and R analysis syntax are available online at https://osf.io/chu6t.

The study survey was constructed using the QuestionPro platform https://www.questionpro.com/ and was administered online. On the first page of the survey, participants were provided with a consent form and were asked to provide informed consent by selecting “I read the informed consent form and I agree with the presented terms”. They were informed about the main aim of the study, how the data they provide will be used, the risks and benefits, and anonymity. They were also informed about the study’s authors and affiliations and provided a contact e-mail. After consenting to participate in the study, they answered the demographic questions and completed the questionnaires in the following order: MPFI, AAQ-II, SWLS, GAD-7, PHQ-9, CompACT, MHC-SF, CCL, ATQ.

To determine the sample size, we used the rule proposed by Bentler and Chou (1987) of 10 participants per estimated parameter rule. In the foundational MPFI model there are a total of 133 parameters to be estimated – 60 factor loadings, 60 residual variances, 12 second-order loadings, and one correlation between factors. This indicates that a sample of around 1330 participants would be necessary for the CFA model. A first set of responses was collected online using a commercial Romania-based survey sampling company, Daedalus (N = 968, 83.23% before removals; N = 933, 84.13% after removals) https://www.daedalusonline.eu/web/. Each respondent received 13.3 RON (∼2.70 €) to complete the survey. Because of funding constraints, the target sample size could not be achieved solely through the survey sampling company, so additional responses were collected from the student population to supplement the dataset. A second set of responses was collected from students enrolled in a psychology program at West University of Timișoara (N=195, 16.77% before removals; N = 176, 15.87% after removals). Anticipating disproportions in terms of age and gender, as psychology undergraduate students are vastly females under 21 years old, we set a sampling target for the data for participants to be over 21 years old, with a 40% female and 60% male gender target proportions. This way we aimed to average out the possible disproportions between the two samples. In exchange for completion of the survey, students received course credits. Only the student sample completed CCL and ATQ, as we wanted to minimize survey length and participant burden in the full sample. By combining these two sources, we obtained a total of 1,163 participants. Although this is below the desired target of 1,330 participants, recruiting additional respondents proved challenging. Prior research indicates that CFA results remain stable with slightly smaller samples, especially given strong factor structure of our model (MacCallum et al., 1999). Therefore, we consider the collected sample adequate for our analyses. The data was collected throughout December 2021 and January 2022.

Data Analytic Strategy

All data analyses were conducted in R version 4.4.1 (R Core Team, 2022). We used the library lavaan version 0.6-18 (Rosseel, 2012) to perform confirmatory factor analysis (CFA). We performed the CFA models using the maximum likelihood estimation with robust standard errors and a Satorra-Bentler scaled test statistic (MLM). We preferred this estimation method as Mardia’s test for multivariate normality suggested significant violations of multivariate normality of the MPFI items in terms of kurtosis z = 204.58, p < .001, and skewness χ²(37820) = 88735.97, p < .001. We used RMSEA, SRMR, CFI, and TLI as fit indices. Hu and Bentler (1999) recommend a combination rule of RMSEA ≤.06 and SRMR ≤.08 to indicate good fit, as these combined criteria yield the smallest likelihood of Type 1 and Type 2 errors. We also considered CFI and TLI ≥.95 values indicative of good fit, and CFI and TLI values ≥.90 as indicative of adequate fit. We also used the lavaan package to verify measurement invariance for age and gender. Notably, we estimated four models: (1) configural – which assumes that the same set of items or observed variables load on the same factors in each group; (2) metric – which assumes equality of loadings; (3) scalar – which assumes equality of intercept; and (4) strict which assumes equality of residuals. To determine whether a model is invariant, we used a threshold of ΔCFI <.01 when comparing the CFI from one step to the other (Cheung & Rensvold, 2002).

To assess convergent and discriminant validities, we used correlation analyses and indices of variance, such as average variance explained (AVE), average shared squared variance (ASV), and maximum shared square variance (MSV). AVE indicates how much variance in observed variables (indicators) is accounted for by their associated latent construct. Convergent validity is established when AVE ≥.50, or in other words, when the latent construct accounts for no less than half of the variance in its indicators (Cheung et al., 2024). ASV is the average squared correlation between all pairs of latent constructs in the model. MSV measures the highest squared correlation between any two constructs in the model. Discriminant validity is achieved when AVE values are greater than both ASV and MSV values (AVE > ASV and AVE > MSV) (Cheung et al., 2024; Fornell & Larcker, 1981). In other words, when a construct explains more variance in its indicators than it shares with any other construct, it has discriminant validity. Furthermore, according to the Fornell–Larcker criterion, discriminant validity is established between two variables if the square root of the AVE for each construct is greater than the correlation between the constructs (Fornell & Larcker, 1981; Rönkkö & Cho, 2020). In the case of CFA models with second-order factors, the Fornell–Larcker criterion is applied to correlations between factors within the same order (Sarstedt et al., 2019).

Another commonly used method for assessing discriminant validity is the Heterotrait–Monotrait (HTMT) ratio, developed by Henseler et al. (2015; Rönkkö & Cho, 2020). The HTMT assesses whether two constructs are empirically distinct. It is calculated as the mean of all correlations between indicators of the two constructs (heterotrait correlations) divided by the square root of the product of the mean correlations of indicators within each construct (monotrait correlations). In other words, it compares correlations between items measuring different constructs with correlations between items measuring the same construct. If the two constructs are empirically distinct, the HTMT ratio should be less than 1, with typical thresholds being HTMT <0.85 for stricter criteria or HTMT <0.90 for more lenient but still acceptable criteria. HTMT ratios in this study were estimated using the htmt() function from the R package semTools (Jorgensen et al., 2012).

Results

The final sample consisted of N = 1109. Total scores included in the correlation analyses were all normally distributed, with raw skewness values between −1.2 and 1.56 and raw kurtosis values between −0.84 and 2.24. Table 2 presents descriptive statistics for all compound scores. To further assess the properties of the sample and assure there is no potential bias induced by different sampling sources, we examined age and gender distributions by sample source (survey panel vs. student sample). In the company sample, 60.9% of participants were male and 39.1% female, whereas the student sample consisted of 7.95% male and 92.05% female participants. Mean age also differed substantially between the groups: 47.9 years in the company sample versus 20.8 years in the student sample. These differences did not require separate control, as all analyses were conducted on the combined full sample. The company sample provided access to a broad adult population, while the student sample included younger participants, many of whom were college students. Despite these demographic differences, combining the two samples produced a dataset with no significant differences in gender distribution, χ²(1) = 2.72, p = .09, V = .05. Age was approximately normally distributed in the combined sample (skewness = −0.14, kurtosis = −1.14).

Confirmatory Factor Analysis

To validate the MPFI measurement model, we performed CFA for three models: (1) a 12-factor model; (2) 2 2-factor model; (3) a second-order model with 12 factors and two second-order factors. The third model was the hypothesized factorial structure of the MPFI. An additional CFA (model 4) on model 3 with one residual correlation was conducted. Model 1 showed a good fit, χ²(1644) = 4231.53, p < .001, CFI = .939, TLI = .934, RMSEA = .038 [.036, .039], SRMR = 0.048. Model 2 indicated poor fit as all examined fit indices were below the set thresholds, χ²(1709) = 16683.13, p < .001, CFI = .646, TLI = .633, RMSEA = .089 [.088, .090], SRMR = .118. Model 3 indicated adequate fit, χ²(1697) = 5301.21, p < .001, CFI = .915, TLI = .911, RMSEA = .044 [.043, .045]; however, the SRMR = .095 was slightly above the set threshold of .08. To further improve the model fit, we included an additional residual correlation between experiential avoidance and psychological inflexibility. This yielded a model with good fit χ²(1696) = 5116.23, p < .001, CFI = .919, TLI = .916, RMSEA = .043 [.041, .044], SRMR = .073, and a correlation of .64, SE = .03, p < .001 between experiential avoidance and psychological inflexibility residual variances. The CFA estimated under model 4 is depicted in Figure 1. The correlation between the two second-order factors, psychological flexibility, and inflexibility, was r = −.51, SE = .03, p < .001. Table 1 depicts unstandardized and standardized factor loadings obtained by estimating model 4. All items and first-order factors revealed strong (above .30) and statistically significant factor loading in relationship to their factors. These findings support the structural validity of the MPFI and replicate the original factor structure of the MPFI proposed by Rolffs et al. (2016).

Figure 1

Second-order CFA of MPFI and unstandardized path coefficients

Table 1

Item Descriptive Statistics, Factor Loadings, and Explained Variance

	M	SD	λ	SE	λ_s	r²
Psychological flexibility
Acceptance	3.66	0.96	0.41	.04	.38	.15
1. I made room to fully experience negative thoughts and emotions, breathing them in rather than pushing them away.	3.86	1.24	0.82	.03	.71	.50
2. When I had an upsetting thought or emotion, I tried to give it space rather than ignoring it.	3.53	1.29	0.85	.04	.71	.51
3. I was receptive to observing unpleasant thoughts and feelings without interfering with them.	3.35	1.25	0.78	.04	.68	.46
4. I tried to make peace with my negative thoughts and feelings rather than resisting them.	3.44	1.40	0.77	.04	.60	.36
5. I opened myself to all my feelings, the good and the bad.	4.12	1.20	0.79	.03	.71	.51
Present moment awareness	4.34	0.94	0.94	.07	.68	.47
6. I paid close attention to what I was thinking and feeling.	4.35	1.16	0.66	.03	.78	.60
7. I was attentive and aware of my emotions.	4.52	1.12	0.68	.03	.84	.70
8. I was in touch with the ebb and flow of my thoughts and feelings.	4.30	1.11	0.67	.03	.83	.69
9. I was in tune with my thoughts and feelings from moment to moment.	4.12	1.10	0.61	.03	.76	.58
10. I strived to remain mindful and aware of my own thoughts and emotions.	4.37	1.12	0.63	.03	.77	.60
Self as context	4.42	0.93	1.66	.12	.86	.73
11. I tried to keep perspective even when life knocked me down.	4.53	1.11	0.45	.03	.78	.61
12. When I was scared or afraid, I still tried to see the larger picture.	4.45	1.06	0.43	.03	.79	.62
13. Even when I felt hurt or upset, I tried to maintain a broader perspective.	4.33	1.06	0.47	.03	.85	.73
14. When something painful happened, I tried to take a balanced view of the situation.	4.37	1.10	0.48	.03	.84	.71
15. I carried myself through tough moments by seeing my life from a larger viewpoint.	4.41	1.11	0.47	.03	.83	.69
Defusion	3.91	1.07	1.36	.07	.81	.65
16. I was able to let negative feelings come and go without getting caught up in them.	4.07	1.26	0.63	.02	.84	.71
17. When I was upset, I was able to let those negative feelings pass through me without clinging to them.	3.94	1.25	0.65	.02	.88	.78
18. When I was scared or afraid, I was able to gently experience those feelings, allowing them to pass.	3.91	1.18	0.61	.02	.88	.77
19. I was able to step back and notice negative thoughts and feelings without reacting to them.	3.72	1.24	0.54	.02	.73	.54
20. In tough situations, I was able to notice my thoughts and feelings without getting overwhelmed by them.	3.93	1.22	0.60	.02	.83	.69
Values	4.63	0.89	1.98	.13	.89	.80
21. I was very in touch with what is important to me and my life.	4.57	1.07	0.41	.02	.84	.71
22. I tried to connect with what is truly important to me on a daily basis.	4.62	1.07	0.41	.02	.86	.73
23. I stuck to my deeper priorities in life.	4.62	1.09	0.40	.02	.82	.67
24. Even when it meant making tough choices, I still tried to prioritize the things that were important to me.	4.59	1.07	0.36	.02	.75	.56
25. My deeper values consistently gave direction to my life.	4.76	1.01	0.32	.02	.71	.51
Committed action	4.68	0.91	1.83	.10	.88	.77
26. Even when I stumbled in my efforts, I didn’t quit working toward what is important.	4.70	1.08	0.41	.02	.80	.64
27. Even when times got tough, I was still able to take steps toward what I value in life.	4.80	1.00	0.41	.02	.87	.75
28. Even when life got stressful and hectic, I still worked toward things that were important to me.	4.81	1.00	0.41	.02	.86	.74
29. I didn’t let setbacks slow me down in taking action toward what I really want in life.	4.62	1.06	0.44	.02	.87	.76
30. I didn’t let my own fears and doubts get in the way of taking action toward my goals.	4.49	1.07	0.41	.02	.80	.64
Psychological inflexibility
Experiential avoidance	4.14	1.07	−0.72	0.04	−.93	.91
31. When I had a bad memory, I tried to distract myself to make it go away.	4.34	1.19	1.29	0.07	.84	.70
32. When unpleasant memories came to me, I tried to put them out of my mind.	4.26	1.22	1.39	0.07	.88	.78
33. I tried to distract myself when I felt unpleasant emotions.	4.13	1.21	1.34	0.07	.85	.73
34. When something upsetting came up, I tried very hard to stop thinking about it.	4.03	1.26	1.33	0.07	.82	.67
35. If there was something I didn’t want to think about, I would try many things to get it out of my mind.	3.97	1.27	1.27	0.07	.77	.60
Lack of contact with the present moment	2.35	1.07	1.08	0.07	.73	.68
36. I did most things on “automatic” with little awareness of what I was doing.	2.73	1.26	0.61	0.03	.71	.51
37. I went through most days on autopilot without paying much attention to what I was thinking or feeling.	2.49	1.28	0.70	0.03	.80	.64
38. I floated through most days without paying much attention.	2.26	1.22	0.75	0.03	.90	.81
39. I did most things mindlessly without paying much attention.	2.16	1.20	0.76	0.03	.93	.86
40. Most of the time, I was just going through the motions without paying much attention.	2.11	1.12	0.65	0.03	.86	.73
Self as content	2.76	1.05	0.96	0.06	.69	.54
41. I thought some of my emotions were bad or inappropriate, and I shouldn’t feel them.	2.65	1.23	0.69	0.03	.78	.61
42. I believed some of my thoughts are abnormal or bad, and I shouldn’t think that way.	2.68	1.30	0.76	0.03	.81	.66
43. I told myself that I shouldn’t be feeling the way I’m feeling.	2.97	1.32	0.72	0.03	.76	.58
44. I told myself I shouldn’t be thinking the way I was thinking.	2.64	1.28	0.68	0.03	.74	.55
45. I criticized myself for having irrational or inappropriate emotions.	2.86	1.32	0.66	0.03	.69	.48
Fusion	2.56	1.15	1.91	0.12	.89	.48
46. Negative thoughts and feelings tended to stick with me for a long time.	2.61	1.35	0.53	0.03	.86	.73
47. Distressing thoughts tended to spin around in my mind like a broken record.	2.41	1.32	0.56	0.03	.91	.82
48. It was very easy to get trapped into unwanted thoughts and feelings.	2.48	1.30	0.54	0.03	.89	.80
49. When I had negative thoughts or feelings, it was very hard to see past them.	2.46	1.23	0.47	0.03	.83	.68
50. When something bad happened, it was hard for me to stop thinking about it.	2.83	1.31	0.47	0.03	.78	.61
Lack of contact with values	2.30	1.00	1.97	0.14	.89	.78
51. My priorities and values often fell by the wayside in my day-to-day life.	2.34	1.16	0.42	0.03	.80	.64
52. The things that I value the most often fell off my priority list completely.	2.34	1.20	0.41	0.03	.75	.56
53. When life got hectic, I often lost touch with the things I value.	2.24	1.17	0.46	0.03	.87	.75
54. I didn’t usually have time to focus on the things that are really important to me.	2.36	1.19	0.43	0.03	.81	.65
55. When times got tough, it was easy to forget about what I truly value.	2.22	1.18	0.44	0.03	.82	.67
Inaction	2.26	1.10	3.14	0.28	.95	.80
56. Negative feelings often trapped me in inaction.	2.23	1.22	0.33	0.03	.90	.81
57. Getting upset left me stuck and inactive.	2.34	1.34	0.36	0.03	.89	.79
58. Negative feelings easily stalled out my plans.	2.34	1.25	0.34	0.03	.90	.80
59. Negative experiences derailed me from what’s really important.	2.19	1.16	0.30	0.03	.86	.73
60. Unpleasant thoughts and feelings easily overwhelmed my efforts to deepen my life.	2.19	1.15	0.29	0.02	.83	.68

All p < .001; n = 1109; M = mean; SD = standard deviation; λ = unstandardized factor loadings; SE = standard error; λ_s = standardised factor loadings; r² = explained variance.

Internal Consistency, Reliability, and Intercorrelations

As presented in Table 2, MPFI dimensions demonstrated good to excellent levels of McDonald’s omega (.80 to .94) and Cronbach’s alpha (.80 to .95) reliability. MPFI dimensions exhibited weak to strong correlations with each other (from r = −.07 to r = .84). One exception was acceptance that did not display any significant correlations with psychological inflexibility processes. Additionally, experiential avoidance resulted as being positively related to general flexibility (r = .37) and flexibility processes (r = .23 – .40), not significantly associated with lack of contact with the present moment, and negatively related with fusion (r = −.09), lack of contact with values (r = −.07), and inaction (r = −.11).

Table 2

MPFI’s AVE, ASV, and MSV Indices, Factor Correlations Extracted From the SEM and HTMT Ratios

	AVE	$\sqrt{A V E}$	ASV	MSV	1	2	3	4	5	6	7	8	9	10	11	12	13	14
1. Psychological flexibility	.59	.77	.26	.26	-							.53
2. Acceptance	.47	.69	.06	.12	.38	-	.60	.31	.24	.27	.25		.1	.09	.04	.11	.07	.13
3. Present moment awareness	.63	.79	.18	.37	.68	.26	-	.61	.5	.66	.59		.25	.28	.07	.16	.21	.22
4. Self as context	.67	.82	.27	.59	.86	.33	.59	-	.8	.73	.72		.4	.3	.16	.4	.33	.41
5. Defusion	.70	.84	.24	.52	.81	.31	.55	.69	-	.66	.65		.41	.24	.24	.52	.35	.46
6. Values	.64	.80	.28	.61	.89	.34	.61	.77	.72	-	.86		.43	.36	.19	.37	.46	.44
7. Committed action	.70	.84	.28	.61	.88	.34	.6	.75	.71	.78	-		.43	.36	.23	.42	.47	.5
8. Psychological inflexibility	.58	.76	.26	.26	−.51	−.20	−.35	−.44	−.41	−.46	−.45	-
9. Experiential avoidance	.70	.84	.08	.18	.48	.18	.33	.41	.38	.42	.42	−.10	-	.06	.11	.08	.07	.08
10. Lack of contact with the p.m.	.71	.84	.19	.49	−.38	−.14	−.26	−.32	−.3	−.34	−.33	.73	−.07	-	.60	.62	.67	.67
11. Self as content	.57	.75	.17	.43	−.36	−.14	−.24	−.3	−.29	−.32	−.31	.69	−.07	.51	-	.70	.58	.63
12. Fusion	.73	.85	.26	.71	−.45	−.17	−.31	−.39	−.37	−.41	−.4	.89	−.09	.65	.61	-	.76	.86
13. Lack of contact with values	.65	.81	.26	.72	−.46	−.18	−.31	−.39	−.37	−.41	−.4	.89	−.09	.65	.62	.79	-	.87
14. Inaction	.76	.87	.29	.72	−.49	−.19	−.33	−.42	−.39	−.44	−.43	.95	−.09	.7	.66	.84	.85	-

Below the diagonal are present factor correlations. Correlations that are problematic according to the Fornell-Larcker criteria are bolded. All between factors correlations are significant at the p < .001 level. Above the diagonal are presented HTMT ratios. Ratios that are above the strict criteria of .85 and indicate problem with discriminant validity are bolded. AVE = average variance explained; ASV = average shared squared variance; MSV = maximum shared square variance; p.m. = present moment.

Convergent Validity

MPFI’s psychological flexibility total score and flexibility processes displayed significant negative correlations with AAQ-II_10-item (r = −.51 – −.08) and AAQ-II_7-item (r = −.47– −.21) and significant positive correlations with CompACT total score and its subscales (r = .05–.62). On exception was acceptance that showed a low significant correlation with AAQ-II_10-item (r = −.08) and a non-significant correlation with AAQ-II_7-item (r = −.00). Psychological inflexibility and inflexibility processes demonstrated positive significant correlations with AAQ-II_10-item (r = .53–.76) and AAQ-II_7-item (r = .53–.78) and significant negative correlations with CompACT and its subscales (r = −.21– −.72). However, experiential avoidance displayed negative correlations with AAQ-II_10-item (r = −.11) and AAQ-II_7-item (r = −.11), positive correlations with CompACT valued actions (r = .23), and non-significant correlations with CompACT total score (r = .00) and CompACT behavioral awareness (r = .06). Correlations between the MPFI and convergent measures are presented in Table 2. We do not include in Table 2 correlations between other measures as they are irrelevant regarding the aim of validating the MPFI.

MPFI dimensions yielded an AVE >.50 (Cheung et al., 2024). Additionally, all standardized factor loadings exceeded 0.4 (Stevens, 2002). Acceptance displayed a slightly concerning situation for convergent validity, as it had an AVE = .47 and a factor loading equal to .38. AVE values can be found in Table 2, and standardized factor loadings are presented in Table 1. Altogether, these results support the convergent validity of the MPFI.

Discriminant Validity

We examined correlations with MHCSF well-being, SWLS satisfaction with life, PHQ-9 depression, GAD-7 anxiety, ATQ, and CCL negative thoughts to assess discriminant validity. In their study, Cheung et al. (2024) recommend several cut-offs for discriminant validity, the strictest being a correlation of .75 between two conceptually different constructs¹. Based on Cheung et al. (2024) threshold, in all the cases, correlations with MPFI’s scores were below .75, indicating no problems of discriminant validity (Cheung et al., 2024). Correlations of MPFI with MHCSF, SWLS, PHQ-9, GAD-7, ATQ, and CCL are presented in Table 3.

Table 3

Descriptive Statistics, Reliability, and Pearson’s Correlations Between MPFI Dimensions and Other Variables

	n	M	SD	ω	α	1	2	3	4	5	6	7	8	9	10	11	12	13	14
1. Psychological flexibility	1109	4.27	0.73	.85	.95
2. Acceptance	1109	3.66	0.96	.81	.81	.56**
3. Present moment awareness	1109	4.34	0.94	.90	.90	.79**	.53**
4. Self as context	1109	4.42	0.93	.91	.91	.84**	.29**	.55**
5. Defusion	1109	3.91	1.07	.92	.92	.80**	.25**	.46**	.74**
6. Values	1109	4.63	0.89	.90	.90	.83**	.27**	.59**	.66**	.60**
7. Committed action	1109	4.68	0.91	.92	.92	.82**	.24**	.54**	.66**	.60**	.78**
8. Psychological inflexibility	1109	2.73	0.77	.80	.95	−.29**	−.01	−.15**	−.26**	−.31**	−.30**	−.34**
9. Experiential avoidance	1109	4.14	1.07	.92	.92	.37**	−.04	.23**	.37**	.38**	.39**	.40**	.20**
10. Lack of contact with the p.m.	1109	2.35	1.07	.92	.92	−.32**	−.05	−.26**	−.27**	−.24**	−.33**	−.34**	.79**	.04
11. Self as content	1109	2.76	1.05	.87	.87	−.18**	.02	−.07*	−.15**	−.23**	−.18**	−.21**	.78**	.12**	.54**
12. Fusion	1109	2.56	1.15	.93	.93	−.37**	.03	−.15**	−.37**	−.48**	−.33**	−.39**	.85**	−.09**	.58**	.63**
13. Lack of contact with values	1109	2.30	1.00	.90	.90	−.37**	−.03	−.20**	−.30**	−.33**	−.42**	−.43**	.82**	−.07*	.62**	.51**	.69**
14. Inaction	1109	2.26	1.10	.94	.94	−.40**	.04	−.20**	−.38**	−.44**	−.41**	−.47**	.86**	−.11**	.63**	.57**	.81**	.80**
AAQ-II_10-item	1045	2.73	1.04	.87	.87	−.50**	−.08**	−.27**	−.47**	−.51**	−.46**	−.51**	.73**	−.11**	.54**	.53**	.72**	.65**	.76**
AAQ-II_7-item	1045	2.56	1.22	.92	.92	−.43**	−.00	−.21**	−.42**	−.47**	−.40**	−.46**	.74**	−.11**	.54**	.53**	.74**	.67**	.78**
SWLS	1044	5.01	1.19	.91	.90	.48**	.15**	.28**	.39**	.41**	.49**	.52**	−.44**	.14**	−.30**	−.30**	−.44**	−.48**	−.47**
GAD-7	1043	1.73	0.74	.93	.93	−.35**	.03	−.14**	−.35**	−.47**	−.31**	−.37**	.60**	−.12**	.41**	.42**	.66**	.52**	.66**
PHQ-9	1042	1.60	0.61	.91	.90	−.36**	.01	−.16**	−.36**	−.42**	−.33**	−.40**	.61**	−.12**	.44**	.43**	.64**	.54**	.64**
CompACT	1036	4.98	0.78	.92	.88	.51**	.17**	.32**	.45**	.44**	.48**	.50**	−.71**	.00	−.59**	−.51**	−.63**	−.63**	−.67**
CompACT-VA	1036	5.61	0.81	.89	.85	.60**	.22**	.41**	.49**	.44**	.62**	.62**	−.39**	.23**	−.38**	−.21**	−.39**	−.47**	−.47**
CompACT-OE	1036	4.30	0.94	.85	.76	.28**	.13**	.12**	.27**	.29**	.22**	.23**	−.57**	−.20**	−.36**	−.49**	−.50**	−.42**	−.47**
CompACT-BA	1036	5.33	1.34	.78	.90	.40**	.05	.30**	.34**	.35**	.39**	.41**	−.72**	.06	−.70**	−.49**	−.62**	−.63**	−.68**
MHCSF	969	4.39	1.04	.96	.94	.54**	.11**	.35**	.44**	.48**	.53**	.56**	−.50**	.21**	−.39**	−.34**	−.55**	−.52**	−.56**
MHCSF-EWB	969	4.74	1.04	.87	.87	.46**	.09**	.28**	.37**	.41**	.46**	.48**	−.50**	.14**	−.40**	−.32**	−.50**	−.52**	−.55**
MHCSF-SWB	969	3.83	1.28	.91	.86	.43**	.08*	.29**	.35**	.41**	.41**	.43**	−.37**	.19**	−.26**	−.27**	−.44**	−.39**	−.42**
MHCSF-PWB	969	4.69	1.11	.87	.92	.55**	.12**	.36**	.45**	.47**	.54**	.58**	−.49**	.21**	−.41**	−.33**	−.53**	−.51**	−.54**
ATQ	181	2.19	1.07	.97	.97	−.47**	−.28**	−.33**	−.40**	−.44**	−.35**	−.43**	.75**	.15*	.61**	.63**	.73**	.66**	.69**
CCL	185	1.96	0.86	.98	.97	−.49**	−.28**	−.35**	−.42**	−.43**	−.40**	−.48**	.72**	.11	.65**	.62**	.70**	.63**	.68**
CCL-depressive thoughts	186	2.03	0.97	.96	.96	−.50**	−.30**	−.37**	−.42**	−.42**	−.42**	−.47**	.73**	.12	.64**	.62**	.69**	.65**	.69**
CCL-anxious thoughts	187	1.90	0.85	.92	.93	−.41**	−.20**	−.28**	−.36**	−.36**	−.32**	−.43**	.62**	.05	.58**	.51**	.62**	.54**	.61**

**p < .01, *p < .05; α = Cronbach’s Alpha; ω – McDonald’s Omega; p.m. = present moment.

We also examined discriminant validity based on AVE and MSV values. AVE was higher than ASV and MSV, indicating that each latent factor’s variance explained by its items is greater than the variance it shares with other constructs from the model. AVE, ASV, and MSV values are presented in Table 2. A minor concern towards discriminant validity regarding the dimension of lack of contact with values displayed an MSV = .72, which was greater than AVE = .65. The analyses based on the Fornell–Larcker criterion and the HTMT ratio suggest a minor concern in the discrimination between lack of contact with values and inaction dimensions (see Table 2). The HTMT ratio also indicates slight issues in the discrimination between values and committed action (HTMT = 0.86) and between inaction and fusion (HTMT = 0.86). However, using the more lenient threshold of 0.90, these results still support discriminant validity for these two cases. No other deviations from discriminant validity were observed. Overall, these findings support the discriminant validity of the MPFI, indicating little to no concern (Table 4).

Table 4

Gender and Age Measurement Invariance

Model	χ2(df)	CFI	RMSEA [90% CI]	SRMR	Model comp	Δχ2(df)	ΔCFI	ΔRMSEA	ΔSRMR	Decision
Gender invariance (male n = 582, female n = 527)
M1: Configural	6900.99*** (3392)	.917	.043 [.042, .044]	.075	-
M2: Metric	6982.96*** (3450)	.916	.043 [.042, .044]	.083	M1	81.97 (58)	−.001	.000	.008	Accept
M3: Scalar	7164.01*** (3496)	.913	.043 [.042, .045]	.084	M2	181.05 (46)	−.003	.001	.001	Accept
M4: Strict	7289.35*** (3556)	.912	.043 [.042, .045]	.084	M3	125.34 (60)	−.001	.000	.000	Accept
Age invariance (18–30 n = 266, 31–44 n = 288, 45–55 n = 257, 56–70 n = 298)
M1: Configural	10815.15*** (6784)	.907	.046 [.045, .048]	.081	-
M2: Metric	11087.96*** (6958)	.904	.046 [.045, .048]	.108	M1	272.81 (174)	−.002	.000	.027	Accept
M3: Scalar	11530.34*** (7096)	.897	.047 [.046, .049]	.108	M2	422.38 (138)	−.007	.001	.000	Accept
M4: Strict	12000.25*** (7276)	.891	.048 [.046, .050]	.109	M3	469.91 (180)	.007	.000	.001	Accept

***p < .001.

Measurement Invariance

Table 4 displays the result of measurement invariance analysis. Based on the recommendations in Putnick and Bornstein (2016), to consider a model to have reached metric invariance, we used the following thresholds for change in fit indices: ΔCFI <.01, ΔRMSEA <.01, ΔSRMR <.03. For scalar and strict invariances, the fit change thresholds were ΔCFI <.01, ΔRMSEA <.01, ΔSRMR <.015. Based on these thresholds, all models demonstrated adequate fit. These results show that the structure of the MPFI, as well as factor loadings, intercepts, and residuals, is invariant across levels of gender (male and female) and age (18–30, 31–44, 45–55, 56–70).

Discussion

In this study, we examined the psychometric properties of the Romanian version of the Multidimensional Psychological Flexibility Inventory (MPFI; Rolffs et al., 2016). Structural validity was demonstrated using confirmatory factor analysis. We tested four models that assumed different factorial structures of the MPFI. The 12-factors model showed the best fit. We also found adequate fit when testing the original MPFI factor structure with 12 first-order factors and two correlated second-order factors proposed by Rolffs et al. (2016). We added one residual correlation between experiential avoidance and psychological inflexibility to the original MPFI factor structure to further perfect the model fit. Another alternative of the factorial structure we tested (second model) implied only two first-order factors (psychological flexibility and inflexibility). This alternative returned a very poor fit, indicating that the variance in MPFI items cannot be accounted for with only two general factors, and a more complex structure is required. These findings replicate the CFA of the original MPFI study (Rolffs et al., 2016) and other validation studies. However, future studies should also consider the 12-factor model as a possible alternative that indicates a better fit.

Internal consistency of the Romanian MPFI was tested using indices of McDonald’s omega and Cronbach’s alpha. All MPFI dimensions and processes showed good to excellent revels of reliability for both McDonald’s omega (.80 to .94) and Cronbach’s alpha (.81 to .95). These levels of reliability are comparable to those reported by Rolffs et al. (2016) in the original article of MPFI development study (Cronbach’s alpha = .87 to .95).

The MPFI’s measurement invariance was tested for gender and age. The results showed that MPFI’s factorial structure, factor loadings, intercepts, and residuals were invariant across the two levels of gender (male and female) and the four levels of age (18–30, 31–44, 45–55, 56–70 years). This indicates that different categories of individuals interpret MPFI’s items similarly and give similar patterns of responses.

Convergent validity was demonstrated using correlations with other measures of psychological inflexibility, such as AAQ-II, and psychological flexibility, such as CompACT. MPFI’s psychological flexibility and its processes displayed negative correlations with AAQ-II and positive correlations with CompACT’s total score and its subscales. Furthermore, MPFI’s psychological inflexibility and its sub-processes demonstrated positive correlations with AAQ-II and negative correlations with CompACT and its subscales. To further test MPFI convergent validity, we examined average explained variance levels (AVE). All MPFI factors displayed acceptable levels of AVE >.50 (Cheung et al., 2024). We also demonstrated MPFI’s discriminant validity by correlating with conceptually distinct constructs such as MHCSF’s well-being, ATQ, and CCL negative thoughts. Also, we demonstrated MPFI’s discriminant validity by examining AVE, ASV, and MSV scores, Fornell–Larcker criterion and HTMT ratios.

We obtained several unexpected results regarding experiential avoidance and acceptance dimensions. Many correlations with experiential avoidance were in the opposite direction as theoretically expected. Namely, experiential avoidance correlated positively with MPFI psychological flexibility and its dimensions, and negatively with MPFI psychological inflexibility dimensions (except self as content and total psychological inflexibility, which was correlated positively). Furthermore, experiential avoidance displayed negative correlations with psychological inflexibility (AAQ-II), anxiety (GAD-7), and depression (PHQ-9); positive correlations with emotional, social, psychological, and total well-being (MHCSF), Compact valued actions, and satisfaction with life (SWLS); and non-significant correlations with Compact total score and compact behavioral awareness.

Previous studies also generally show low and even insignificant correlations of experiential avoidance with flexibility and its dimensions (see Rogge et al., 2019; Rolffs et al., 2016; Seidler et al., 2020; Sundström et al., 2023; Tabrizi et al., 2023; Thomas et al., 2022). These problematic correlations across studies indicate that it is possible that the experiential avoidance items from MPFI do not reflect the construct they intend to measure very well. It’s possible that these items could measure other constructs, such as openness to experience from the Big Five personality traits (McCrae & Costa, 1996). It’s possible that high experiential avoidance in MPFI indicates low openness to experience, which is not necessarily a maladaptive trait and correlates positively with flexibility dimensions. Low openness suggests that an individual values practical, familiar, and more conventional ideas and experiences, and could be as in touch with their values as high openness individuals. Future studies should explore the relationship between MPFI and personality to ensure discriminant validity. Also, it’s unclear to what extent MPFI experiential avoidance items distinguished between the idea of “getting rid of” painful thoughts and emotions and not having them in the first place. It’s possible that these items could be understood in terms of not having painful experiences rather than in terms of burdening oneself with the struggle of getting rid of them.

Another possible explanation could be the cultural context of this validation. Our study is the only one that examined the correlation between flexibility and inflexibility dimensions in a collectivist culture (Hofstede et al., 2010). Experiential avoidance may be understood differently depending on culture. It could be that in individualistic cultures, experiential avoidance is viewed as a way of being disconnected from personal values. In contrast, in collectivistic cultures, this willingness to give up your values is perceived as a way of preserving collective harmony. That is, in collectivistic countries, giving up on personal values is a paradoxical way of actually keeping in touch with group values. The results of Lin et al. (2020) also hint towards cultural differences in psychological flexibility, as in their study, experiential avoidance was related to higher functioning, which they attributed to a particularity of East Asian cultures (which are predominantly collectivist).

Furthermore, Landi et al. (2021) obtained a pattern of results on experiential avoidance strikingly similar to our study: positive correlations with psychological flexibility and some of its scales, and negative correlations with some psychological inflexibility dimensions. In their study, Landi et al. (2021) attributed their unexpected results to the period during which the data was collected, during the COVID-19 lockdown – the same circumstances in which the data for our study was collected. It is possible that the imposed lockdown created a context in which experiential avoidance became necessary, forcing people to adapt by using avoidance strategies that would normally be maladaptive (e.g., avoiding social situations to reduce the risk of contamination, alleviating distress produced by the pandemic by distracting oneself). As is known, psychologically flexible people are able to better adapt to new circumstances due to their ability to better connect with the present moment (Waldeck et al., 2021). Put under the constraint of complying with the lockdown, only the most psychologically flexible would have been able to adapt to the situation and develop experiential avoidance strategies that would help them deal with the associated distress. Therefore, our results showed positive associations of experiential avoidance and psychological flexibility processes and positive associations with well-being and satisfaction with life.

As per the unexpected findings regarding acceptance, in previous studies correlations with acceptance are also generally low (see Rogge et al., 2019; Rolffs et al., 2016; Sundström et al., 2023; Tabrizi et al., 2023) and even occasionally insignificant (Landi et al., 2021; Seidler et al., 2020; Thomas et al., 2022). Given the inability of MPFI acceptance to show strong correlations with the rest of the inventory, this possibly suggests that the items for this dimension need further refinement. Additionally, we observed that correlations with acceptance seemed lower in studies where the sample’s age was higher. Because our study had a sample comprised of somewhat older individuals than in other adaptation studies, we hypothesized that correlations with acceptance depend on age, and this led us to perform an additional exploratory analysis where we correlated acceptance with MPFI dimensions split by three age categories: below μ-σ, between μ-σ and μ+σ, and over μ+σ. These analyses are reported in an online supplemental material in Supplement 3. We found that acceptance correlated significantly in the expected direction in younger participants. In comparison, in older participants, correlations are insignificant, and some are positive (with inflexibility, self as content, and fusion). This indicates that the role and interpretation of acceptance might change with age. A previous study has reported that psychological flexibility and flexibility dimensions start to decrease after a particular moment in life (Slot et al., 2024). Corroborated with our results, acceptance decreases with age, and the role it plays for an individual also changes. Low acceptance in older individuals may not indicate high inflexibility but rather the ability to cope with stressful events in other ways. Future studies should explore developmental aspects of psychological flexibility and inflexibility, as our study indicates that acceptance might function differently at different stages of life.

There are several limitations of our study. First, we could not test measurement invariance in the clinical compared to the non-clinical population. This is a significant limitation, as psychological inflexibility is a major correlate of psychopathology (Fischer et al., 2016) and the MPFI is meant to be used also with clinical populations (Rolffs et al., 2016). Previous studies have demonstrated MPFI measurement invariance in clinical populations in Italian (Landi et al., 2021), Persian (Hekmati et al., 2024), Caucasian American, Asian American, Chinese, Japanese, and Taiwanese samples (Lin et al., 2020). Therefore, future studies should test MPFI invariance in Romanian (and other cultures) clinical vs. non-clinical participants. Additionally, we were unable to verify the test-retest reliability of the MPFI. Test-retest reliability is an essential aspect for the soundness of any questionnaire and should be addressed in future studies. However, this question remains largely unanswered as to our knowledge: only one previous study addressed test-retest reliability of the MPFI, finding good moderate and good intraclass correlations in a Swedish sample (Sundström et al., 2023).

In conclusion, the results of this study support the psychometric soundness and theoretical validity of the Romanian MPFI for the examination of psychological flexibility and inflexibility and their processes. Our results closely replicate the psychometric properties of the MPFI reported in the original development article (Rolffs et al., 2016). The Romanian MPFI is a reliable tool that can be used for both research purposes and to measure ACT processes in clinical practice. However, the usage of acceptance and experiential avoidance items should be approached with care by considering the context, developmental stage, and personality.

Supplemental Material

Supplemental Material - Romanian Adaptation and Validation of the Multidimensional Psychological Flexibility Inventory

Supplemental Material for Romanian Adaptation and Validation of the Multidimensional Psychological Flexibility Inventory by Ștefan Marian, Iulia Crişan, Ștefan Sergiu Varga, Andreea Bogdana Isbăşoiu in Evaluation & the Health Professions

Supplemental Material

Supplemental Material - Romanian Adaptation and Validation of the Multidimensional Psychological Flexibility Inventory

Supplemental Material

Supplemental Material - Romanian Adaptation and Validation of the Multidimensional Psychological Flexibility Inventory

Footnotes

ORCID iDs

Ștefan Marian

Iulia Crişan

Ștefan Sergiu Varga

Andreea Bogdana Isbăşoiu

Ethical Consideration

This study has received ethical approval from the Scientific Council of University Research and Creation of the West University of Timişoara, notice nr. 67003/02.12.2021.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work received funding from Romania’s Ministry of Education research fund Fondul de Dezvoltare Instuțională under the grant agreement No. FDI-2021-0414.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

Data and R script with statistical analyses are available online at .

Supplemental Material

Supplemental material for this article is available online.

Note

References

A-Tjak

J. G.

Davis

M. L.

Morina

Powers

M. B.

Smits

J. A.

Emmelkamp

P. M.

(2015). A meta-analysis of the efficacy of acceptance and commitment therapy for clinically relevant mental and physical health problems. Psychotherapy and Psychosomatics, 84(1), 30–36. https://doi.org/10.1159/000365764

Azadfar

Abdollahi

Patra

Chang

Y. P.

Alghazali

Talib

S. G.

(2022). The Iranian form of psychometric properties of the multidimensional psychological flexibility inventory. Psicologia: Reflexão e Crítica, 35(1), 32. https://doi.org/10.1186/s41155-022-00236-w

Barr

(2022). Validation and exploration of the multidimensional psychological flexibility inventory with a sample of adolescents. Publication Number 30245154. [ProQuest Dissertations & Theses Global]. M.S., Utah State University. https://www.proquest.com/dissertations-theses/validation-exploration-multidimensional/docview/2813466126/se-2?accountid=196263

Beck

A. T.

Brown

Steer

R. A.

Eidelson

J. I.

Riskind

J. H.

(1987). Differentiating anxiety and depression: A test of the cognitive content-specificity hypothesis. Journal of Abnormal Psychology, 96(3), 179–183. https://doi.org/10.1037//0021-843x.96.3.179

Beck

(2011). Cognitive behavior therapy: Basics and beyond. Guilford press.

Bentler

Chou

C. P.

(1987). Practical issues in structural equation modeling. Sociological Methods & Research, 16(1), 78–117. https://doi.org/10.1177/0049124187016001004

Bond

F. W.

Hayes

S. C.

Baer

R. A.

Carpenter

K. M.

Guenole

Orcutt

H. K.

Waltz

Zettle

R. D.

(2011). Preliminary psychometric properties of the Acceptance and Action Questionnaire–II: A revised measure of psychological inflexibility and experiential avoidance. Behavior Therapy, 42(4), 676–688. https://doi.org/10.1016/j.beth.2011.03.007

Brown

L. A.

Gaudiano

B. A.

Miller

I. W.

(2011). Investigating the similarities and differences between practitioners of second- and third-wave cognitive-behavioral therapies. Behavior Modification, 35(2), 187–200. https://doi.org/10.1177/0145445510393730

Călinici

M. S.

Călinici

(2021). Comprehensive assessment of act processes compact. Romanian adaptation and short form validation. Journal of Evidence - Based Psychotherapies, 21(2), 209-226. https://doi.org/10.24193/jebp.2021.2.19

10.

Cheung

G. W.

Cooper-Thomas

H. D.

Lau

R. S.

Wang

L. C.

(2024). Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations. Asia Pacific Journal of Management, 41(2), 745–783. https://doi.org/10.1007/s10490-023-09871-y

11.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233–255. https://doi.org/10.1207/S15328007SEM0902_5

12.

Cunha

Temido

Moniz

Galhardo

(2024). Assessing psychological flexibility and mental health in adults: The Psy-Flex European Portuguese version. Journal of Contextual Behavioral Science, 32(100730), Article 100730. https://doi.org/10.1016/j.jcbs.2024.100730

13.

Diener

Emmons

R. A.

Larsen

R. J.

Griffin

(1985). The satisfaction with life scale. Journal of Personality Assessment, 49(1), 71–75. https://doi.org/10.1207/s15327752jpa4901_13

14.

Dimitrova

Buzea

Abubakar

Stefenel

(2016). Measurement invariance of the satisfaction with life scale in Roma minority and Romanian majority adolescents in Romania. Psihologia Resurselor Umane, 14(2), 111–121. https://www.proquest.com/scholarly-journals/measurement-invariance-satisfaction-with-life/docview/1882017155/se-2?accountid=196263

15.

Doorley

Goodman

Kelso

Kashdan

(2020). Psychological flexibility: What we know, what we do not know, and what we think we know. Social and Personality Psychology Compass, 14(12), 1–11. https://doi.org/10.1111/spc3.12566

16.

Fischer

T. D.

Smout

M. F.

Delfabbro

P. H.

(2016). The relationship between psychological flexibility, early maladaptive schemas, perceived parenting and psychopathology. Journal of Contextual Behavioral Science, 5(3), 169–177. https://doi.org/10.1016/j.jcbs.2016.06.002

17.

Fornell

Larcker

D. F.

(1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50. https://doi.org/10.1177/002224378101800104

18.

Francis

A. W.

Dawson

D. L.

Golijani-Moghaddam

(2016). The development and validation of the Comprehensive Assessment of Acceptance and Commitment Therapy Processes (CompACT). Journal of Contextual Behavioral Science, 5(3), 134–145. https://doi.org/10.1016/j.jcbs.2016.05.003

19.

Gloster

A. T.

Walder

Levin

M. E.

Twohig

M. P.

Karekla

(2020). The empirical status of acceptance and commitment therapy: A review of meta-analyses. Journal of Contextual Behavioral Science, 18, 181–192. https://doi.org/10.1016/j.jcbs.2020.09.009

20.

Gratz

K. L.

Roemer

(2004). Multidimensional assessment of emotion regulation and dysregulation: Development, factor structure, and initial validation of the difficulties in emotion regulation scale. Journal of Psychopathology and Behavioral Assessment, 26(1), 41–54. https://doi.org/10.1023/B:JOBA.0000007455.08539.94

21.

Grégoire

Gagnon

Lachance

Shankland

Dionne

Kotsou

Monestès

J.-L.

Rolffs

J. L.

Rogge

R. D.

(2020). Validation of the English and French versions of the Multidimensional Psychological Flexibility inventory short form (MPFI-24). Journal of Contextual Behavioral Science, 18, 99–110. https://doi.org/10.1016/j.jcbs.2020.06.004

22.

Harris

(2019). ACT made simple: An easy-to-read primer on acceptance and commitment therapy. 2nd ed. New Harbinger Publications.

23.

Hayes

S. C.

(2004). Acceptance and commitment therapy, relational frame theory, and the third wave of behavioral and cognitive therapies. Behavior Therapy, 35(4), 639–665. https://doi.org/10.1016/S0005-7894(04)80013-3

24.

Hayes

S. C.

Luoma

J. B.

Bond

F. W.

Masuda

Lillis

(2006). Acceptance and commitment therapy: Model, processes and outcomes. Behaviour Research and Therapy, 44(1), 1–25. https://doi.org/10.1016/j.brat.2005.06.006

25.

Hayes

S. C.

Strosahl

K. D.

Wilson

K. G.

(1999). Acceptance and commitment therapy: An experiential approach to behavior change. Guilford Press.

26.

Hekmati

Abdollahpour Ranjbar

Ciarrochi

Laghaei

Rezaei Golezani

Eskin

(2024). Psychometric properties and measurement invariance of the Multidimensional Psychological Flexibility Inventory- Persian (MPFI-P): An extensive investigation of long and short versions in community and clinical samples. Journal of Contextual Behavioral Science, 31, Article 100717. https://doi.org/10.1016/j.jcbs.2023.100717

27.

Hofstede

G. H.

Hofstede

G. J.

Minkov

(2010). Cultures and Organizations, Software of the mind. Intercultural Cooperation and Its Importance for survival. In McGraw-Hill eBooks . http://ci.nii.ac.jp/ncid/BB02154491

28.

Hollon

S. D.

Kendall

P. C.

(1980). Cognitive self-statements in depression: Development of an automatic thoughts questionnaire. Cognitive Therapy and Research, 4(4), 383–395. https://doi.org/10.1007/bf01178214

29.

Hollon

Kendall

(2007). Automatic Thoughts Questionnaire (adapted for Romanian language by MoldovanDavid, R.).D. In (coord.), Sistem De Evaluare Clinică. Cluj-Napoca: Editura RTS.

30.

L. t.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

31.

Jorgensen

T. D.

Pornprasertmanit

Schoemann

A. M.

Rosseel

(2012). SEMTools: Useful tools for structural equation modeling [Dataset]. https://doi.org/10.32614/cran.package.semtools

32.

Kashdan

T. B.

Rottenberg

(2010). Psychological flexibility as a fundamental aspect of health. Clinical Psychology Review, 30(7), 865–878. https://doi.org/10.1016/j.cpr.2010.03.001

33.

Kroenke

Spitzer

R. L.

Williams

J. B. W.

(2001). The PHQ-9. Journal of General Internal Medicine, 16(9), 606–613. https://doi.org/10.1046/j.1525-1497.2001.016009606.x

34.

Lamers

S. M. A.

Westerhof

G. J.

Bohlmeijer

E. T.

ten Klooster

P. M.

Keyes

C. L. M.

(2011). Evaluating the psychometric properties of the Mental Health Continuum-Short Form (MHC-SF). Journal of Clinical Psychology, 67(1), 99–110. https://doi.org/10.1002/jclp.20741

35.

Landi

Pakenham

K. I.

Giovannetti

A. M.

Presti

Boccolini

Cola

Grandi

Tossani

(2021). Italian validation of the Italian Multidimensional Psychological Flexibility Inventory (MPFI). Journal of Contextual Behavioral Science, 21, 57–65. https://doi.org/10.1016/j.jcbs.2021.05.007

36.

Lin

Y. Y.

Rogge

R. D.

Swanson

D. P.

(2020). Cross-cultural flexibility: Validation of the traditional Mandarin, simplified Mandarin, and Japanese translations of the multidimensional psychological flexibility inventory. Journal of Contextual Behavioral Science, 15, 73–84. https://doi.org/10.1016/j.jcbs.2019.11.008

37.

MacCallum

R. C.

Widaman

K. F.

Zhang

Hong

(1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99. https://doi.org/10.1037/1082-989x.4.1.84

38.

Marian

Ş.

Sava

F. A.

(2023). Identifying central negative thoughts using experience sampling and network analysis. Cognitive Therapy and Research, 47(5), 707–719. https://doi.org/10.1007/s10608-023-10400-w

39.

McCrae

R. R.

Costa

P. T.

Jr. (1996). Toward a new generation of personality theories: Theoretical contexts for the five-factor model. In Wiggins

J. S.

(Ed.), The five-factor model of personality: Theoretical perspectives (pp. 51–87). Guilford Press.

40.

Moosivand

Kulemarzi

M. J. B.

Shirazi

M. S.

Zaremohzzabieh

(2024). Flexibility and inflexibility in an Iranian sample: Psychometric properties of MPFI based on the Hexaflex model. BMC Psychology, 12(1), 34. https://doi.org/10.1186/s40359-024-01531-0

41.

Putnick

D. L.

Bornstein

M. H.

(2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004

42.

R Core Team . (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

43.

Rogge

R. D.

Daks

J. S.

Dubler

B. A.

Saint

K. J.

(2019). It’s all about the process: Examining the convergent validity, conceptual coverage, unique predictive validity, and clinical utility of ACT process measures. Journal of Contextual Behavioral Science, 14, 90–102. https://doi.org/10.1016/j.jcbs.2019.10.001

44.

Rolffs

J. L.

Rogge

R. D.

Wilson

K. G.

(2016). Disentangling components of flexibility via the Hexaflex model: Development and validation of the Multidimensional Psychological Flexibility Inventory (MPFI). Assessment, 25(4), 458–482. https://doi.org/10.1177/1073191116645905

45.

Rönkkö

Cho

(2020). An updated guideline for assessing discriminant validity. Organizational Research Methods, 25(1), 6–14. https://doi.org/10.1177/1094428120968614

46.

Rosseel

(2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02

47.

Sarstedt

Hair

J. F.

Cheah

Becker

Ringle

C. M.

(2019). How to specify, estimate, and validate higher-order constructs in PLS-SEM. Australasian Marketing Journal, 27(3), 197–211. https://doi.org/10.1016/j.ausmj.2019.05.003

48.

Seidler

Stone

Clark

B. E.

Koran

Drake

C. E.

(2020). Evaluating the factor structure of the Multidimensional Psychological Flexibility Inventory: An independent replication and extension. Journal of Contextual Behavioral Science, 17, 23–31. https://doi.org/10.1016/j.jcbs.2020.04.007

49.

Simkin

Freiberg Hoffmann

Caselli

Prozzillo

Rogge

Wilson

(2023). Adaptation and validation into Spanish of the Multidimensional Psychological Flexibility Inventory (MPFI): Evidence of internal validity and factorial invariance. Revista AJAYU, 21(2), 111–130. https://doi.org/10.35319/ajayu.212188

50.

Slot

Reijnders

Janssens

Simons

Lataster

Jacobs

(2024). Age-related differences in psychological flexibility: A cross-sectional study in a Dutch population sample. International Journal of Applied Positive Psychology, 10(1), 9. https://doi.org/10.1007/s41042-024-00194-x

51.

Spitzer

R. L.

Kroenke

Williams

J. B. W.

Löwe

(2006). A brief measure for assessing generalized anxiety disorder: The GAD-7. Archives of Internal Medicine, 166(10), 1092–1097. https://doi.org/10.1001/archinte.166.10.1092

52.

Stevens

J. P.

(2002). Applied multivariate statistics for the social sciences. 4th ed. Lawrence Erlbaum Associates Publishers.

53.

Stoddard

J. A.

Afari

(2014). The big book of ACT metaphors: A practitioner’s guide to experiential exercises and metaphors in acceptance and commitment therapy. New Harbinger Publications.

54.

Sundström

F. T. A.

Lavefjord

Buhrman

McCracken

L. M.

(2023). Assessing psychological flexibility and inflexibility in chronic pain using the Multidimensional Psychological Flexibility Inventory (MPFI). The Journal of Pain, 24(5), 770–781. https://doi.org/10.1016/j.jpain.2022.11.010

55.

Szabó

K.-G.

Vargha

J.-L.

Balázsi

Bartalus

Bogdan

(2011). Measuring psychological flexibility: Preliminary data on the psychometric properties of the Romanian version of the Acceptance and Action Questionnaire (AAQ-II). Journal of Cognitive and Behavioral Psychotherapies, 11(1), 67–82. https://www.proquest.com/scholarly-journals/measuring-psychological-flexibility-preliminary/docview/868867771/se-2?accountid=196263

56.

Tabrizi

F. F.

Larsson

A. B.

Grönvall

Söderstrand

Hallén

Champoux-Larsson

M.-F.

Lundgren

Sundström

Lavefjord

Buhrman

Sundin

Ö.

McCracken

Åhs

Jansson

(2023). Psychometric evaluation of the Swedish Multidimensional Psychological Flexibility Inventory (MPFI). Cognitive Behaviour Therapy, 52(4), 295–316. https://doi.org/10.1080/16506073.2022.2153077

57.

Thomas

K. N.

Bardeen

J. R.

Witte

T. K.

Rogers

T. A.

Benfer

Clauss

(2022). An examination of the factor structure of the multidimensional psychological flexibility inventory. Assessment, 29(8), 1714–1729. https://doi.org/10.1177/10731911211024353

58.

VanBuskirk

West

Malcarne

, et al. (2012). Confirmatory Factor Analysis of the Valued Living Questionnaire in a Black American Sample: Implications for Cognitive Research and Practice. Cognitive Therapy and Research, 36, 796–805. https://doi.org/10.1007/s10608-011-9405-8

59.

Waldeck

Pancani

Holliman

Karekla

Tyndall

(2021). Adaptability and psychological flexibility: Overlapping constructs? Journal of Contextual Behavioral Science, 19, 72–78. https://doi.org/10.1016/j.jcbs.2021.01.002

60.

Żemojtel-Piotrowska

Piotrowski

J. P.

Osin

E. N.

Cieciuch

Adams

B. G.

Ardi

Bălţătescu

Bogomaz

Bhomi

A. L.

Clinton

de Clunie

G. T.

Czarna

A. Z.

Esteves

Gouveia

Halik

M. H. J.

Hosseini

Khachatryan

Kamble

S. V.

Kawula

Maltby

(2018). The mental health continuum-short form: The structure and application for cross-cultural studies–A 38 nation study. Journal of Clinical Psychology, 74(6), 1034–1052. https://doi.org/10.1002/jclp.22570

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.18 MB

0.50 MB

0.29 MB