A robust effect size measure A w for MANOVA with non-normal and non-homogenous data

Abstract

A common research question in psychology entails examining whether significant group differences (e.g. male and female) can be found in a list of numeric variables that measure the same underlying construct (e.g. intelligence). Researchers often use a multivariate analysis of variance (MANOVA), which is based on conventional null-hypothesis significance testing (NHST). Recently, a number of quantitative researchers have suggested reporting an effect size measure (ES) in this research scenario because of the perceived shortcomings of NHST. Thus, a number of MANOVA ESs have been proposed (e.g. generalized eta squared ${\eta }_{\mathrm{\Lambda }}^2$ , generalized omega squared ${\omega }_{\mathrm{\Lambda }}^2$ ), but they rely on two key assumptions—multivariate normality and homogeneity of covariance matrices—which are frequently violated in psychological research. To solve this problem we propose a non-parametric (or assumptions-free) ES (A_w) for MANOVA. The new ES is developed on the basis of the non-parametric A in ANOVA. To test A_w we conducted a Monte-Carlo simulation. The results showed that A_w was accurate (robust) across different manipulated conditions—including non-normal distributions, unequal covariance matrices between groups, total sample sizes, sample size ratios, true ES values, and numbers of dependent variables—thereby providing empirical evidence supporting the use of A_w, particularly when key assumptions are violated. Implications of the proposed A_w for psychological research and other disciplines are also discussed.

Effect size (ES), a quantity that directly presents or measures the strength of an effect in a study, has received increasing attention. ES is regarded as a supplement to conventional null hypothesis significance testing (NHST) because NHST focuses on making a dichotomized decision to reject or accept a research hypothesis without considering the level or magnitude of the effect observed in a study. In fact, many methodologists, professional associations, and journal editors have suggested that researchers should report ESs in addition to NHST (Murphy, 1997; Thompson, 1994; Trafimow and Marks, 2015). The American Psychological Association (APA) publication manual is more assertive on this matter, stating: “estimates of appropriate effect sizes and confidence intervals are the minimum expectations for all APA journals” (American Psychological Association, 2010: 33). The International Committee of Medical Journal Editors also states: “[a]void relying solely on statistical hypothesis testing, such as P values, which fails to convey important information about effect size.” (Mathews and Mathews, 2007, Section IV.A.6.c).

In light of the call for reporting ESs, different estimates of the true ES in the multivariate analysis of variance (MANOVA) have been developed in the literature. For example, Steyn and Ellis (2009) evaluated the accuracy of conventional MANOVA ESs (e.g. generalized eta squared ${\eta }_{\mathrm{\Lambda }}^2$ , generalized omega squared ${\omega }_{\mathrm{\Lambda }}^2$ , etc.) when the key assumptions—multivariate normality and homogeneity of covariance matrices—are met. Other researchers (e.g. Finch, 2016; Fouladi and Yockey, 2002; Olejnik and Huberty, 2006) also discussed the conceptual and computational details for these MANOVA ESs. However, both of these ESs do not appear to be robust to violations of these assumptions because their calculations are developed based on their corresponding, univariate estimates (i.e. eta squared ( ${\eta }^2$ ) and omega squared ( ${\omega }^2$ )) that were found to be biased ES estimators when these assumptions were violated (e.g. Troncoso Skidmore and Thompson, 2013). Unfortunately, these assumptions are frequently violated in behavioral and social sciences research (e.g. Algina et al., 2005). Hence, there is an urgent need for developing a more appropriate MANOVA ES that does not rely on these assumptions.

This study aims to develop a more robust ES measure (A_w) and compare it with the conventional ESs (i.e. ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ ) for MANOVA when the parametric assumptions (i.e. multivariate normality and homogeneity of covariance matrices) are violated. The proposed A_w is based on Vargha and Delaney’s (2000) development of an ES measure for univariate analysis of variance (i.e. A) with only one dependent variable (DV), which is a more restricted case of MANOVA.

This paper is divided into six sections. The first section explains the importance of the two assumptions in MANOVA ESs. The second section presents the computational details of the conventional (parametric) ESs (i.e. ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ ) that depend upon the two assumptions. The third section discusses the development of the non-parametric A_w and its parametric counterpart CL_w (called common language ES). In the fourth section, methods and design of the Monte-Carlo simulation study are described. The fifth section presents and discusses the simulation findings. In the sixth section, conclusion and implications of the findings are discussed.

Two key assumptions: Multivariate normality and homogeneity of covariance matrices

In most commercial statistical packages (e.g. SPSS), four conventional NHST statistics for MANOVA are typically reported: Pillai’s Trace, Wilks’ Lambda, Hotelling-Lawley Trace, and Roy’s Greatest Root. These statistics are known to be reliable when two key assumptions are met: multivariate normality and homogeneity of covariance matrices. Multivariate normality means that a vector of weighted DV scores is independently and normally distributed for each level of the IV. Homogeneity of covariance matrices requires that the variance and covariance of the DVs are the same for each level of the IV. However, data in behavioral science often deviates from these assumptions (Keselman and Lix, 1997). Researchers in behavioral science often use single- or multi-item measures that employ Likert-scale (e.g. total score of three items each on a 5-point scale), and hence, the boundaries of the total scores are fixed (e.g. 3–15 points) producing a non-normal (i.e. platykurtic) distribution. Data observed in groups (e.g. clinical patients, gifted children) also tend to follow a heavy-tailed (i.e. skewed) distribution. Furthermore, the assumption of homogeneity of covariance matrices is rarely met in behavioral research (Tang and Algina, 1993). For example, data in a clinical group tend to have a smaller variance than in a normal group. A treatment may also be effective in systematically improving the outcome of interest among treated participants, and so they appear to be more homogenous (i.e. ceiling effect) than participants receiving no treatment.

Despite the popularity of the conventional statistics in reporting MANOVA results, previous research has found that they are not robust to violations of these important assumptions. Everitt (1979) found that these statistics tend to have an inflated Type II error (i.e. fail to reject the null hypothesis when it is false) when the degree of skewness in the DVs increases. Algina et al. (1991) found that when data are asymmetrically non-normal, the test statistics lead to inflated Type I error, especially when the covariances are heterogeneous and the sample size is unbalanced between groups. Cole et al. (1994) showed that the performance of these statistics varied. When the off-diagonal elements in the covariance matrices increase in difference across groups, the test statistics become less robust. Hopkins and Clay (1963) showed that when the covariance matrices differ between two groups, the test statistics are unlikely to be robust. Given that the conventional MANOVA ESs are based on Wilk’s Lambda, one of the conventional test statistics for MANOVA, these ESs are expected to be affected by the same assumption violations. However, no study has systematically evaluated the performance of these ES, nor has prior work proposed a non-parametric ES that does not rely on these assumptions. Here we report the first study to do both.

Conventional ESs

In two-independent samples univariate analysis of variance (ANOVA), there are two common families of ESs: the difference (d) family and correlation (r) family. For the d-family, the standardized mean difference between two groups in DV scores can be used to evaluate the strength of the effect. That is, $d=\left(\overline{X_p}-\overline{X_q}\right)/s_{pooled}$ , where the numerator refers to the mean difference between Groups p and q, and $s_{pooled}$ is the pooled standard deviation. For the r-family, ES can be expressed in terms of Pearson’s correlation ratio, which is defined as the proportion of total variability that can be explained by an IV. That is, ${\eta }^2=S{S}_{IV}/S{S}_{Total}$ , where $S{S}_{IV}$ is the sum of squares due to the IV, and $S{S}_{Total}$ is the sum of squares of the DV scores without the IV. ${\eta }^2$ is a measure of model goodness-of-fit and is conceptually equivalent to R² in multiple regression, and hence, it is regarded as a measure of association between the IV and DV. Note that ${\eta }^2$ is mathematically related to d, that is, ${\eta }^2=\frac{d^2}{d^2+\frac{{\left(n_1+n_2\right)}^2}{n_{1\cdot }{n}_2}}$ .

Generalized eta squared ${\eta }_{\mathrm{\Lambda }}^2$

In MANOVA, the generalized eta squared ${\eta }_{\mathrm{\Lambda }}^2$ is defined as the total generalized variance that can be explained by the between-group generalized variance, that is,

$${\eta }_{\mathrm{\Lambda }}^2=1-\mathrm{\Lambda }=1-\left|{\sum }_W\right|/\left|{\sum }_W+{\sum }_B\right|$$

(1)

where $\mathrm{\Lambda }$ is the Wilk’s lambda, which consists of two components. First,

$${\displaystyle \sum }_B=\left(1/N\right){\displaystyle \sum }_{j=1}^J{N}_j\left({\mu }_j-\mu \right){\left({\mu }_j-\mu \right)}^{\prime }$$

is the between-groups covariance matrix, N is the total sample size, N_j is the sample size for j = 1, 2, . . ., J groups, µj is the mean for the jth group, and µ is the grand mean. Second,

$${\displaystyle \sum }_w=\left(1/N\right){\displaystyle \sum }_{j=1}^J{\displaystyle \sum }_{n=1}^{N_j}\left(y_{jn}-{\mu }_j\right){\left(y_{jn}-{\mu }_j\right)}^{\prime }$$

is the within-groups covariance matrix, where $y_{jn}$ is the nth observation for the jth group. According to Cohen (1988), ${\eta }_{\mathrm{\Lambda }}^2$ is similar to R² in multiple regression, which measures the association between the weighted level or set of the DVs and the IV in MANOVA.

Generalized omega squared ${\omega }_{\mathrm{\Lambda }}^2$

The eta squared ( ${\eta }^2$ ) in ANOVA is commonly regarded as a biased estimator for the true ES—it tends to overestimate the true ES (Mordkoff, 2019). Researchers have proposed and developed another ES, called omega squared ( ${\omega }^2$ ), for ANOVA, which can adjust for the bias found in ${\eta }^2$ . The multivariate generalization of ${\omega }^2$ is known as the generalized omega squared,

$${\omega }_{\mathrm{\Lambda }}^2=1-\left(N\mathrm{\Lambda }\right)/\left(N-J+\mathrm{\Lambda }\right)$$

(2)

where J is the number of groups in the IV, and N and $\mathrm{\Lambda }$ are defined in equation (1). ¹

Recently developed ESs

Parametric common language ES (CL) in ANOVA

The idea of common language ES (CL) can be found in statistical studies published almost 80 years ago when Wilcoxon (1945) proposed a rank-order comparison for scores observed between two treatments. Mann and Whitney (1947) extended Wilcoxon’s method, which calculated the rank numbers in which the scores in treatment A are larger than the scores in treatment B, by further defining the statistical properties (e.g. probability distribution) of Wilcoxon’s statistical measure, known as the U statistic. Govindarajulu (1967) was one of the early studies that formally defined the meaning of a probability estimate, P(X < Y), where X refer to continuous scores in a random sample A (e.g. treatment), and Y refer to continuous scores in a random sample B (e.g. control). This measure quantifies the probability that a score in one sample is stochastically smaller than a score in another sample. Govindarazulu focused on deriving an analytic method that constructs the confidence intervals surrounding the measure of P(X < Y). Wolfe and Hogg (1971) published a tutorial paper that encourages the use of P(X < Y) among applied statisticians and researchers.

Based on these early papers published in statistics journals, McGraw and Wong (1992) was one of the pioneer studies in psychology that proposed the use of a statistic that measures “how often a score sampled from one distribution will be greater than a score sampled from another distribution” (p. 361). They labeled this statistic a common language ES (CL) and proposed its use as a type of probability of superiority statistic for univariate ANOVA. CL estimates the parameter, P(Y_p > Y_q), which measures the probability that a score in group p is higher than a score in group q. For example, if a researcher is comparing the effect of an intervention group relative to a control group, the researcher can present the CL that estimates the probability (e.g. 80% chance) that the observations (e.g. self-esteem) would be better for a randomly selected member of the intervention group than for a randomly selected member of the control group. Hsu (2004) regarded this as a more intuitive way to conceptualize ES, as it is easy for researchers and practitioners to understand even without formal training in statistics. According to Ruscio (2008), when data meet the parametric assumptions (i.e. normality and homogeneity of variances), CL can be expressed as $CL=\mathrm{\Phi }\left[\left(\overline{X_p}-\overline{X_q}\right)/s_{pooled}\right]$ , where $\mathrm{\Phi }$ is the normal cumulative distribution function, $\overline{X_p}$ and $\overline{X_q}$ are the means of Groups p and q, respectively, and $s_{pooled}=\sqrt{\left[\left(N_p-1\right)s_p^2+\left(N_p-1\right)s_q^2\right]/\left(N_p+N_q-2\right)}$ is the pooled SD of the DV scores for the two groups.

Non-parametric A in ANOVA

Vargha and Delaney (2000) criticized McGraw and Wong’s (1992) CL on the basis that it assumes the scores follow a normal distribution. To overcome this problem they proposed a non-parametric estimator, known as the measure of stochastic superiority (i.e. $A,$ equation (2) in Vargha and Delaney, 2000), for use in ANOVA with two independent samples. $A$ measures effect size based on the probability that a score in group p is higher than a score in group q, that is,

$$A=\left[\#(p>q)+.5\#(p=q)\right]/n_p{n}_q,$$

(3)

where # is the count function, p and q are vectors of scores for the two samples, and $n_j$ is the sample size in group j = p, q. Assume p = {5, 7, 6, 5} and q  = {3, 5, 5, 1}, the count function (i.e. $\#([p=5]>[q=3,5,5,1])+.5\#([p=5]=[q=3,5,5,1])$ ) yields a total count of 3. Repeat this process for the remaining elements in p , A = (3 + 4 + 4 + 3)/16 = 0.875, meaning that there is a 87.5% chance that the observation would be higher for a randomly selected member of group p than for a randomly selected member of group q.

Parametric $C{L}_w$ and non-parametric A_w in MANOVA

To generalize $A$ in two independent-groups ANOVA to the multivariate complement (i.e. A_w) in MANOVA, we can substitute the w_p and w_q vectors of scores—a linear composite of two or more DVs for each participant in group p and q, respectively—into the p and q vectors of scores in equation (3). Assume $X^{(j)}$ is the n (number of participants) by p (number of DVs) data matrix for group j = 1 and 2. $X^{(j)}=X_1^{(j)},X_2^{(j)},\dots ,X_v^{(j)}$ , where $X_v^{(j)}$ is the vector of scores of the vth DV in group j. The linear composite becomes, $w_j=a_1^{(j)}\cdot X_1^{\left(j\right)}+a_2^{\left(j\right)}\cdot X_2^{\left(j\right)}+\dots +a_v^{(j)}\cdot X_v^{\left(j\right)}$ , where, $a_1^{(j)},a_2^{(j)},...,a_v^{(j)}$ are the weights (or eigenvectors) for score vectors $X_1^{(j)},X_2^{(j)},...,X_v^{(j)}$ for group j = 1 and 2 based on discriminant function analysis. The vector of a weights can be estimated by

$$a={s_{pooled}}^{-1}(\overline{X_p}-\overline{X_q}),$$

(4)

where $\overline{X_p}$ and $\overline{X_q}$ are the v by 1 vectors that contains the means of the DVs for group j = p and q, respectively, and ${s_{pooled}}^{-1}=\left(SSC{P}_p+SSC{P}_q\right)/\left(N_p+N_q-2\right)$ with $SSC{P}_p$ and $SSC{P}_q$ are the sum of squares and cross products matrices for group j = p and q, respectively. These weights are selected to produce the maximum possible ( w_p - w_q ) difference, which can be obtained in a statistical package (e.g. SPSS). Hence, A_w can be expressed as

$$A_w=\left[\#(w_p>w_q)+.5\#(w_p=w_q)\right]/n_p{n}_q,$$

(5)

which expresses the probability (e.g. 90%) that the linear composite $w_j$ would be higher for a randomly selected member of group p than for a randomly selection member of group q. The parametric estimator for the probability of superiority in MANOVA ( $C{L}_w$ ) can be expressed as

$$C{L}_w=\mathrm{\Phi }\left[\left(\overline{w_p}-\overline{w_q}\right)/s_w\right],$$

(6)

where $\overline{w_p}$ and $\overline{w_q}$ are the means of the weighted composite scores for groups p and q, respectively, and $s_w$ is the pooled SD of the weighted composite scores from the two groups.

This study evaluates the performance of two conventional parametric ESs— ${\eta }_{\mathrm{\Lambda }}^2$ (equation (1)) and ${\omega }_{\mathrm{\Lambda }}^2$ (equation (2))—and two probability-based ESs—non-parametric $A_w$ (equation (5)) and parametric $C{L}_w$ (equation (6)).

Monte-Carlo study

Design

Seven factors that would affect the performance of ${\eta }_{\mathrm{\Lambda }}^2$ , ${\omega }_{\mathrm{\Lambda }}^2$ , $C{L}_w$ , and $A_w$ were evaluated.

Factor 1: Standardized mean vector difference (δ; four levels)

This parameter reflects the level of standardized mean difference between the weighted DV scores in two groups, which is similar to Cohen’s standardized mean difference d in ANOVA. According to Cohen (1988), in social science research a d of 0.20, 0.50, and 0.80 reflects a small, moderate and large ES respectively. In addition, the value of 1.50 was included to examine the impact of an extremely strong δ on the observed ESs. The corresponding values for ${\eta }_{\mathrm{\Lambda }}^2$ are 0.0099, 0.0588, 0.1379, and 0.3600, and for $C{L}_w$ and $A_w$ are 0.5793, 0.6915, 0.7881, and 0.9332. The corresponding true values for ${\omega }_{\mathrm{\Lambda }}^2$ are similar to but slightly deviated from the true values for ${\eta }_{\mathrm{\Lambda }}^2$ . The adjustment depends on two factors: the sample size (N) and number of groups (J). That is, combining equations (1) and (2), ${\omega }_{\mathrm{\Lambda }}^2$ is adjusted by: ${\omega }_{\mathrm{\Lambda }}^2=1-\left[N(1-{\eta }_{\mathrm{\Lambda }}^2)\right]/\left[N-J+(1-{\eta }_{\mathrm{\Lambda }}^2)\right]$ . Hence, taking a simulated condition with the total sample size of 150 (i.e. N = 150) and the number of groups equals 2 (i.e. J = 2) as an example, when the value for ${\eta }_{\mathrm{\Lambda }}^2$ was 0.0099, the adjusted value for ${\omega }_{\mathrm{\Lambda }}^2$ was estimated to be 0.0032 (i.e. $1-\left[150(1-.0099)\right]/\left[150-2+(1-.0099)\right]$  = 0.0032).

Factor 2: Distribution (Θ; six levels)

In addition to the normal distribution (N (0, 1) with skewness (ϒ₁) = 0 and kurtosis (ϒ₂) = 0], five non-normal (i.e. two peaked, two skewed, and one mixed normal) distributions were generated based on Algina et al. (2005). The peaked distribution is characterized by a long (or short) tail with few (or most) scores clustered around the center of the distribution. The skewed distribution is characterized by unequal-length tails along a distribution. The mixed normal distribution appears to be a normal distribution but with longer tails on both ends, mimicking a distribution with outliers on both ends. Following Algina et al., for the peaked and skewed distributions, the normal scores were multiplied by specific g and h values so that the transformed scores followed the target non-normal distributions. Specifically, when g and h were nonzero,

$$Y=exp\left(h{Z}^2/2\right)\cdot \left[exp(gZ)-1\right]/g$$

(7)

where Y is the transformed score and Z is the original normal score. When g was zero,

$$Y=Z\cdot exp\left(h{Z}^2/2\right)$$

(8)

According to Algina et al., the target peaked distributions were manipulated at (1) ϒ₁ = 0 and ϒ₂ = 6 (i.e. g = 0 and h = 0.142) and (2) ϒ₁ = 0 and ϒ₂ = 154.84 (i.e. g = 0 and h = 0.225), and the target skewed distributions were fixed at (3) ϒ₁ = 2 and ϒ₂ = 6 (i.e. g = 0.76 and h = −0.098; an exponential distribution) and (4) ϒ₁ = 4.90 and ϒ₂ = 4,673.80 (i.e. g = 0.225 and h = 0.225), which are common in social sciences research. Note that positively (or negatively) skewed distributions often have ϒ_1 > 0 (or ϒ_1 < 0), and shorted-tailed (or long-tailed; e.g. t distribution) distributions often have ϒ_2 < 0 (or ϒ_2 > 0). For the mixed normal distribution, only 90% of the observations come from the normal distribution with mean 0 and SD 1 and 10% come from the normal distribution with mean 0 and SD 10. This distribution has ϒ₁ = 0 and ϒ₂ = 24.95.

Factor 3: Number of DVs (ν; three levels)

Three numbers, 2, 5, and 8, were evaluated, which cover a range of values that seem to be practical in real-world research.

Factor 4: Variance ratio (π ; three levels)

Variance ratio is defined as the ratio of the variance in Group 1 to the variance in Group 2 (Ruscio and Mullen, 2012). The ratio was fixed at 1, 4, and 0.25. The value of 1 means that the variances are homogenous for the two groups, and the values of 4 and 0.25 indicate violations of the homogeneity of covariance matrices assumption, which are commonly found in social sciences research.

Factor 5: Correlations between DVs (R; $2\times 2$ levels)

The DVs were expected to measure the same construct in MANOVA, and hence, they were manipulated to be correlated with one another in each group. Two levels of correlations, 0.50 and 0.80, were evaluated for the two groups, respectively. The value of 0.50 followed the design in Fouladi and Yockey (2002), which mimicked a moderate-to-large association between items. The value of 0.80 was included to examine the impact of extremely strong relationship. The manipulated levels in Factors 4 and 5 mimic the data conditions that meet or violate the assumption of homogeneity of covariance matrices.

Factor 6: Total sample size (N; two levels)

Two levels, 50 and 150, were simulated, thereby representing a small and moderate-large sample typically found in behavioral research.

Factor 7: Base rate (b; three levels)

In Ruscio and Mullen (2012), base rate is defined as the ratio of the sample size in Group 1 to the sample size in Group 2. Base rate was evaluated at the levels 0.25, 0.50, and 0.75. Thus, the samples sizes could be equal for the two groups, or it was three times larger in one sample than the other.

In sum, the factors were factorially combined to produce a design with $4\times 6\times 3\times 3\times 2\times 2\times 2\times 3=5,184$ conditions. Each condition was replicated 10,000 times. The simulation was conducted in the R Project programming environment (R Core Team, 2014), and the code can be found in Supplemental Material.

Evaluation criteria

Two evaluation criteria were used. First, for each of the 5184 simulation conditions, percentage bias (bias) was computed to evaluate the average performance of an ES relative to its true value, that is, $\mathrm{bias}=\left[\left(\overline{ES}-\phi \right)/\phi \right]\cdot 100\%$ , where $\overline{ES}$ is the mean of 10,000 simulated ESs, and $\phi$ is the true criterion ES value presented in standardized mean difference, eta squared, omega squared, or common-language metric. That is, as noted above, the true criterion values for δ are 0.20, 0.50 0.80, and 1.50, ${\eta }_{\mathrm{\Lambda }}^2$ are 0.0099, 0.0588, 0.1379, and 0.3600, and $C{L}_w$ and $A_w$ are 0.5793, 0.6915, 0.7881, and 0.9332. According to Li et al. (2011), a bias is considered reasonable if it is within $\pm 10\%$ . Second, to summarize the overall performance of an ES across 5184 conditions, mean absolute percentage error (MAPE) was computed, that is, MAPE =  ${\displaystyle \sum }_{i=1}^{5,184}\left|bias\left(i\right)\right|/5,184$ . According to Brockwell and Davis (2002), A MAPE within 10% is regarded as a desirable fit.

Results

Overall performance

Comparing the four ESs, A_w performed the best. As shown in Figure 1, the biases ranged from −31.57% to 52.35%, with a mean of −4.16%, which is within the criterion for a good fit ( $\pm 10\%)$ . Of the 5184 conditions, 3548 (or 68.44%) resulted in a bias within $\pm 10\%$ of the true ES. To summarize the overall performance across 5184 conditions, the MAPE was 8.20%, which is regarded as a desirable fit. The second reasonable ES in this study is CL_w. The biases ranged from −42.19% to 65.58%, with a mean of −7.67%, which is still within the criterion of $\pm 10\%$ . Of the 5184 conditions, 2666 (or 51.43%) produced a bias within $\pm 10\%$ . Regarding the overall performance, the MAPE was 12.02%, which is slightly beyond the criterion of $\pm 10\%$ .

Figure 1.

Percentage biases of the generalized eta squared ( ${\eta }_{\mathrm{\Lambda }}^2$ ), generalized omega squared ( ${\omega }_{\mathrm{\Lambda }}^2$ ), non-parametric probability of superiority effect size ( $A_w$ ), and parametric probability of superiority effect size ( $C{L}_w$ ) across 5184 simulation conditions.

By contrast, the conventional parametric ESs ( ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ ) were less than optimal. The biases of ${\eta }_{\mathrm{\Lambda }}^2$ ranged from −94.13% to 4884.73%, with a mean of 224.68%. Of the 5184 conditions, only 561 (or 10.82%) produced a bias within $\pm 10\%$ . The MAPE was 251.49%, indicating unsatisfactory overall performance. Moreover, the biases of ${\omega }_{\mathrm{\Lambda }}^2$ ranged from −4652.83% to 5322.91%, with a mean of 38.42%. Of the 5184 conditions, only 501 (or 9.66%) resulted in an acceptable bias. The MAPE was 358.10%, showing undesirable overall performance.

Note that the reason for a wide range of biases is in part due to the metrics of the true ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ in some conditions. For example, when δ = 0.20, Θ = normal, ν = 2, $\pi =$ 0.25, N = 150, b = 0.25, and R = (0.50, 0.50), the expected true ${\eta }_{\mathrm{\Lambda }}^2$ is equal to 0.0099 and the expected true ${\omega }_{\mathrm{\Lambda }}^2$ is equal to 0.0032. The means of the 10,000 simulated ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ values were found to be 0.0483 and 0.0417, respectively. Hence, the biases were very large (i.e. 388.20% and 1205.79%) because the simulated means were substantially larger than their true values. Despite the impacts of the metrics of the true ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ on the observed biases, ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ are still regarded as less accurate than the other ESs (i.e. A_w and CL_w). The following sections discuss the effects of the manipulated factors on the ESs.

Effects of manipulated factors

Given that the correlations between DVs (R) did not show any effects on all the ESs, their impact is not discussed in this section. As shown in Table 1, first, when the total sample size (N) increased from 50 to 150, the accuracy of ${\eta }_{\mathrm{\Lambda }}^2$ , A_w, and CL_w generally increased (biases decreased) because more data points were available for estimating the strength of the effect. The accuracy of ${\omega }_{\mathrm{\Lambda }}^2$ only improved with a larger sample size when the true standardized difference (δ) was larger than or equal to 0.50. When δ was 0.20, the biases changed from negative to positive with an increasing sample size. This was probably due to the effect of a small ES, which was less likely to be accurately detected or estimated. When N was small, ${\omega }_{\mathrm{\Lambda }}^2$ tended to underestimate the true value; when N was large, ${\omega }_{\mathrm{\Lambda }}^2$ tended to overestimate the true value.

Table 1.

Percentage biases of ${\eta }_{\mathrm{\Lambda }}^2$ , ${\omega }_{\mathrm{\Lambda }}^2$ , $A_w$ , and $C{L}_w$ when the correlations between items were equal to (0.50, 0.50) for two groups.

ν = 2, δ = 0.2
π	b	N	Θ = 1				Θ = 2				Θ = 3				Θ = 4				Θ = 5				Θ = 6
			${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw
0.25	0.25	50	11.32	−10.71	0.13	0.14	8.09	−7.64	0.08	0.11	10.73	−10.15	0.12	0.13	10.30	−9.75	0.11	0.13	10.34	−9.78	0.09	0.14	10.00	−9.46	0.11	0.13
		150	3.88	12.06	0.03	0.01	2.33	7.22	−0.03	0.00	3.37	10.46	0.02	0.00	3.10	9.62	0.01	0.00	2.96	9.20	−0.07	0.04	2.52	7.83	0.00	0.00
	0.50	50	4.84	−4.57	0.07	0.04	3.55	−3.35	0.01	0.03	4.34	−4.10	0.06	0.04	4.11	−3.88	0.05	0.04	3.59	−3.39	−0.01	0.06	3.69	−3.49	0.04	0.03
		150	1.93	5.99	0.01	−0.05	0.58	1.82	−0.04	−0.04	1.38	4.29	0.00	−0.05	1.13	3.51	0.00	−0.05	0.77	2.39	−0.13	0.03	0.54	1.67	−0.03	−0.05
	0.75	50	1.72	−1.62	0.06	0.00	1.11	−1.05	0.00	0.00	1.38	−1.30	0.05	0.00	1.25	−1.18	0.05	0.00	0.78	−0.73	−0.03	0.02	0.95	−0.89	0.03	−0.01
		150	0.62	1.93	0.02	−0.07	−0.28	−0.88	−0.03	−0.06	0.24	0.73	0.01	−0.07	0.06	0.20	0.00	−0.07	−0.31	−0.95	−0.15	0.02	−0.33	−1.03	−0.02	−0.08
1	0.25	50	4.02	−3.80	0.09	0.05	3.35	−3.16	0.03	0.03	3.71	−3.51	0.08	0.05	3.58	−3.38	0.07	0.04	3.82	−3.60	0.09	0.02	3.57	−3.37	0.07	0.04
		150	1.31	4.07	0.02	−0.04	0.47	1.45	−0.04	−0.05	0.96	2.98	0.00	−0.04	0.80	2.48	0.00	−0.05	1.03	3.19	0.03	−0.11	0.78	2.41	−0.01	−0.06
	0.50	50	4.44	−4.19	0.07	0.01	3.32	−3.13	0.01	−0.01	4.02	−3.80	0.06	0.00	3.83	−3.61	0.05	0.00	4.09	−3.86	0.06	−0.03	3.79	−3.58	0.05	−0.01
		150	1.65	5.13	0.01	−0.07	0.51	1.57	−0.05	−0.07	1.18	3.65	0.00	−0.07	0.96	2.97	−0.01	−0.07	1.26	3.90	0.01	−0.14	0.91	2.83	−0.02	−0.09
	0.75	50	3.94	−3.72	0.09	0.01	3.32	−3.14	0.03	0.00	3.63	−3.43	0.08	0.01	3.50	−3.31	0.07	0.00	3.70	−3.50	0.07	−0.01	3.48	−3.29	0.06	0.00
		150	1.34	4.16	0.02	−0.06	0.51	1.57	−0.04	−0.08	0.98	3.05	0.01	−0.07	0.82	2.54	0.00	−0.07	1.03	3.21	0.02	−0.13	0.78	2.43	−0.01	−0.09
4	0.25	50	0.88	−0.83	0.03	−0.06	0.90	−0.85	−0.01	−0.06	0.82	−0.77	0.02	−0.06	0.81	−0.77	0.01	−0.06	2.90	−2.74	−0.02	−0.03	1.09	−1.02	0.01	−0.06
		150	−0.19	−0.58	−0.03	−0.10	−0.41	−1.27	−0.07	−0.11	−0.28	−0.87	−0.04	−0.11	−0.32	−1.00	−0.04	−0.11	2.31	7.16	−0.05	−0.07	0.07	0.20	−0.05	−0.11
	0.50	50	3.69	−3.48	0.05	−0.04	3.32	−3.14	0.00	−0.04	3.56	−3.36	0.04	−0.04	3.50	−3.31	0.03	−0.04	6.81	−6.43	0.00	0.00	3.89	−3.67	0.03	−0.04
		150	0.80	2.49	−0.02	−0.10	0.42	1.29	−0.07	−0.10	0.66	2.05	−0.03	−0.10	0.59	1.84	−0.03	−0.10	4.95	15.38	−0.05	−0.04	1.20	3.71	−0.04	−0.10
	0.75	50	10.35	−9.79	0.12	0.10	7.80	−7.37	0.08	0.06	9.95	−9.41	0.11	0.10	9.62	−9.10	0.10	0.09	12.13	−11.48	0.08	0.14	9.70	−9.17	0.10	0.09
		150	2.87	8.92	0.01	−0.06	2.18	6.77	−0.04	−0.07	2.71	8.43	0.00	−0.06	2.61	8.10	−0.01	−0.06	7.28	22.62	−0.04	0.02	3.15	9.79	−0.01	−0.06
ν = 2, δ = 0.5
π	b	N	Θ = 1				Θ = 2				Θ = 3				Θ = 4				Θ = 5				Θ = 6
			${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw
0.25	0.25	50	2.06	3.18	0.01	0.02	0.85	1.31	−0.05	−0.05	1.70	2.61	0.00	0.00	1.50	2.31	−0.02	−0.01	1.28	1.97	−0.02	−0.02	1.36	2.09	−0.02	−0.02
		150	0.94	1.06	−0.05	−0.08	−0.23	−0.26	−0.12	−0.15	0.50	0.56	−0.06	−0.12	0.28	0.32	−0.07	−0.13	−0.29	−0.32	−0.11	−0.18	−0.01	−0.02	−0.09	−0.16
	0.50	50	1.17	1.81	0.00	−0.06	0.13	0.21	−0.08	−0.12	0.75	1.15	−0.02	−0.09	0.55	0.84	−0.03	−0.10	0.13	0.19	−0.06	−0.15	0.34	0.52	−0.04	−0.12
		150	0.74	0.84	−0.02	−0.10	−0.54	−0.61	−0.10	−0.20	0.24	0.27	−0.04	−0.15	0.00	−0.01	−0.05	−0.17	−0.59	−0.67	−0.08	−0.27	−0.32	−0.37	−0.06	−0.22
	0.75	50	0.34	0.52	0.02	−0.07	−0.43	−0.65	−0.07	−0.15	0.00	0.01	0.00	−0.11	−0.15	−0.22	−0.01	−0.13	−0.46	−0.70	−0.03	−0.22	−0.31	−0.47	−0.02	−0.17
		150	0.18	0.20	0.01	−0.09	−0.76	−0.86	−0.07	−0.22	−0.20	−0.22	−0.01	−0.15	−0.38	−0.42	−0.02	−0.18	−0.76	−0.85	−0.03	−0.33	−0.59	−0.67	−0.03	−0.24
1	0.25	50	0.63	0.96	0.00	−0.05	−0.11	−0.17	−0.09	−0.13	0.33	0.51	−0.03	−0.08	0.20	0.30	−0.04	−0.10	0.41	0.63	0.02	−0.09	0.18	0.27	−0.04	−0.10
		150	0.22	0.24	−0.04	−0.12	−0.65	−0.73	−0.12	−0.20	−0.13	−0.15	−0.06	−0.16	−0.30	−0.34	−0.07	−0.17	−0.08	−0.09	−0.01	−0.20	−0.33	−0.37	−0.07	−0.20
	0.50	50	0.93	1.43	−0.02	−0.08	−0.05	−0.07	−0.10	−0.15	0.54	0.83	−0.04	−0.11	0.36	0.55	−0.05	−0.13	0.63	0.97	0.00	−0.14	0.32	0.50	−0.05	−0.14
		150	0.50	0.57	−0.04	−0.12	−0.62	−0.70	−0.12	−0.22	0.06	0.06	−0.06	−0.17	−0.16	−0.18	−0.08	−0.19	0.12	0.14	−0.02	−0.22	−0.20	−0.23	−0.08	−0.21
	0.75	50	0.61	0.94	0.00	−0.06	−0.12	−0.18	−0.09	−0.14	0.32	0.49	−0.03	−0.09	0.18	0.27	−0.04	−0.11	0.38	0.59	0.00	−0.13	0.15	0.23	−0.04	−0.13
		150	0.21	0.24	−0.04	−0.11	−0.65	−0.74	−0.12	−0.22	−0.14	−0.15	−0.06	−0.16	−0.30	−0.34	−0.07	−0.18	−0.09	−0.10	−0.02	−0.22	−0.33	−0.38	−0.08	−0.21
4	0.25	50	−0.44	−0.68	−0.09	−0.18	−0.63	−0.96	−0.15	−0.21	−0.53	−0.81	−0.10	−0.19	−0.57	−0.87	−0.11	−0.20	0.06	0.10	−0.10	−0.12	−0.48	−0.73	−0.11	−0.19
		150	−0.62	−0.70	−0.12	−0.22	−0.87	−0.98	−0.18	−0.25	−0.72	−0.82	−0.13	−0.23	−0.77	−0.87	−0.14	−0.24	−0.03	−0.03	−0.11	−0.14	−0.65	−0.73	−0.14	−0.23
	0.50	50	0.12	0.19	−0.09	−0.16	−0.18	−0.28	−0.14	−0.19	0.00	0.01	−0.10	−0.17	−0.05	−0.08	−0.11	−0.18	0.87	1.34	−0.11	−0.10	0.06	0.10	−0.11	−0.17
		150	−0.35	−0.40	−0.13	−0.21	−0.71	−0.80	−0.19	−0.24	−0.50	−0.56	−0.14	−0.23	−0.56	−0.64	−0.15	−0.23	0.61	0.69	−0.13	−0.13	−0.38	−0.43	−0.15	−0.22
	0.75	50	1.18	1.81	−0.05	−0.05	0.55	0.84	−0.09	−0.10	1.04	1.60	−0.05	−0.06	0.95	1.47	−0.06	−0.07	1.69	2.61	−0.08	0.01	1.00	1.54	−0.07	−0.06
		150	−0.06	−0.07	−0.13	−0.18	−0.42	−0.47	−0.18	−0.22	−0.19	−0.21	−0.13	−0.19	−0.25	−0.28	−0.14	−0.20	0.97	1.10	−0.14	−0.08	−0.08	−0.09	−0.14	−0.19
ν = 2, δ = 1.5
π	b	N	Θ = 1				Θ = 2				Θ = 3				Θ = 4				Θ = 5				Θ = 6
			${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw
0.25	0.25	50	0.38	0.40	−0.06	0.03	−0.32	−0.33	−0.14	−0.14	0.16	0.17	−0.08	−0.01	0.03	0.04	−0.10	−0.05	0.02	0.03	−0.11	−0.07	−0.03	−0.03	−0.11	−0.07
		150	0.30	0.31	−0.07	0.05	−0.61	−0.62	−0.15	−0.27	0.03	0.03	−0.10	−0.01	−0.13	−0.14	−0.11	−0.07	−0.22	−0.23	−0.13	−0.10	−0.26	−0.26	−0.13	−0.12
	0.50	50	0.38	0.40	−0.02	0.05	−0.42	−0.45	−0.11	−0.20	0.12	0.13	−0.05	0.00	−0.02	−0.02	−0.07	−0.04	−0.05	−0.05	−0.10	−0.07	−0.11	−0.12	−0.09	−0.08
		150	0.33	0.34	−0.02	0.06	−0.67	−0.69	−0.10	−0.31	0.03	0.04	−0.06	0.00	−0.14	−0.15	−0.08	−0.06	−0.20	−0.20	−0.10	−0.08	−0.27	−0.27	−0.10	−0.11
	0.75	50	0.08	0.09	0.01	0.06	−0.63	−0.67	−0.08	−0.23	−0.16	−0.17	−0.03	0.02	−0.29	−0.31	−0.05	−0.03	−0.29	−0.31	−0.07	−0.05	−0.37	−0.39	−0.07	−0.07
		150	0.07	0.07	0.00	0.07	−0.79	−0.81	−0.07	−0.32	−0.21	−0.21	−0.03	0.02	−0.37	−0.38	−0.05	−0.05	−0.39	−0.40	−0.08	−0.06	−0.47	−0.48	−0.07	−0.10
1	0.25	50	0.05	0.05	−0.04	0.03	−0.64	−0.68	−0.14	−0.25	−0.19	−0.20	−0.08	−0.04	−0.32	−0.34	−0.10	−0.09	−0.13	−0.14	−0.05	−0.01	−0.34	−0.36	−0.10	−0.09
		150	0.03	0.03	−0.04	0.05	−0.80	−0.81	−0.12	−0.34	−0.24	−0.25	−0.09	−0.03	−0.40	−0.40	−0.11	−0.11	−0.20	−0.20	−0.06	−0.01	−0.43	−0.44	−0.11	−0.12
	0.50	50	0.26	0.27	−0.04	0.04	−0.58	−0.61	−0.14	−0.27	−0.01	−0.01	−0.08	−0.03	−0.16	−0.17	−0.10	−0.08	0.05	0.06	−0.06	−0.01	−0.18	−0.19	−0.10	−0.10
		150	0.21	0.22	−0.04	0.05	−0.75	−0.76	−0.12	−0.35	−0.09	−0.09	−0.09	−0.03	−0.26	−0.27	−0.11	−0.11	−0.04	−0.04	−0.06	0.00	−0.30	−0.31	−0.11	−0.12
	0.75	50	0.05	0.06	−0.04	0.03	−0.64	−0.68	−0.14	−0.26	−0.18	−0.20	−0.08	−0.03	−0.31	−0.33	−0.10	−0.08	−0.13	−0.13	−0.06	−0.01	−0.33	−0.36	−0.10	−0.10
		150	0.03	0.03	−0.04	0.05	−0.80	−0.81	−0.12	−0.34	−0.24	−0.25	−0.09	−0.03	−0.40	−0.40	−0.11	−0.10	−0.20	−0.20	−0.06	−0.01	−0.43	−0.44	−0.11	−0.12
4	0.25	50	−0.59	−0.63	−0.16	−0.17	−0.86	−0.91	−0.24	−0.36	−0.70	−0.74	−0.18	−0.25	−0.75	−0.80	−0.19	−0.29	−0.49	−0.52	−0.08	−0.11	−0.72	−0.76	−0.18	−0.27
		150	−0.62	−0.63	−0.17	−0.19	−0.93	−0.95	−0.23	−0.42	−0.74	−0.75	−0.19	−0.28	−0.80	−0.81	−0.20	−0.32	−0.52	−0.53	−0.08	−0.12	−0.76	−0.78	−0.19	−0.30
	0.50	50	−0.39	−0.41	−0.18	−0.18	−0.74	−0.79	−0.27	−0.34	−0.52	−0.56	−0.20	−0.24	−0.59	−0.63	−0.22	−0.27	−0.24	−0.25	−0.13	−0.10	−0.55	−0.59	−0.21	−0.25
		150	−0.46	−0.47	−0.19	−0.20	−0.88	−0.90	−0.27	−0.41	−0.62	−0.63	−0.22	−0.28	−0.69	−0.71	−0.23	−0.32	−0.30	−0.30	−0.13	−0.11	−0.64	−0.65	−0.22	−0.29
	0.75	50	−0.30	−0.32	−0.20	−0.15	−0.63	−0.67	−0.26	−0.28	−0.41	−0.43	−0.22	−0.19	−0.46	−0.49	−0.23	−0.21	−0.17	−0.18	−0.18	−0.08	−0.45	−0.47	−0.23	−0.20
		150	−0.45	−0.46	−0.23	−0.20	−0.83	−0.84	−0.30	−0.38	−0.59	−0.60	−0.24	−0.26	−0.66	−0.67	−0.25	−0.30	−0.25	−0.25	−0.19	−0.09	−0.60	−0.62	−0.25	−0.27
ν = 8, δ = 0.2
π	b	N	Θ = 1				Θ = 2				Θ = 3				Θ = 4				Θ = 5				Θ = 6
			${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw
0.25	0.25	50	42.97	−40.88	0.04	−0.03	32.93	−31.27	0.04	−0.04	41.47	−39.43	0.03	−0.04	40.14	−38.16	0.03	−0.05	40.95	−38.93	0.07	0.02	40.10	−38.12	0.05	−0.02
		150	14.58	45.31	0.00	−0.08	10.26	31.88	−0.04	−0.10	13.89	43.16	−0.01	−0.10	13.39	41.60	−0.02	−0.10	12.89	40.06	−0.06	−0.23	12.93	40.17	−0.02	−0.13
	0.50	50	19.75	−18.70	0.05	−0.05	17.58	−16.64	0.04	−0.04	19.18	−18.15	0.04	−0.06	18.83	−17.82	0.04	−0.07	19.08	−18.06	0.02	−0.12	18.97	−17.95	0.04	−0.07
		150	6.72	20.87	0.02	−0.09	5.09	15.82	−0.03	−0.11	6.08	18.88	0.00	−0.10	5.78	17.97	0.00	−0.10	4.98	15.45	−0.05	−0.27	5.21	16.19	−0.01	−0.15
	0.75	50	7.77	−7.33	0.08	−0.04	8.83	−8.34	0.05	−0.02	7.46	−7.04	0.07	−0.06	7.39	−6.98	0.07	−0.06	7.02	−6.62	0.04	−0.15	7.40	−6.99	0.07	−0.07
		150	2.74	8.52	0.04	−0.08	1.87	5.80	−0.01	−0.11	2.27	7.05	0.03	−0.09	2.08	6.46	0.02	−0.10	1.39	4.30	−0.02	−0.28	1.64	5.09	0.02	−0.15
1	0.25	50	16.50	−15.61	0.13	0.02	15.90	−15.04	0.10	0.01	16.20	−15.32	0.12	0.01	16.06	−15.19	0.11	0.01	16.45	−15.55	0.11	0.04	16.06	−15.19	0.11	0.01
		150	5.67	17.60	0.05	−0.06	4.67	14.50	−0.01	−0.09	5.22	16.22	0.03	−0.08	5.03	15.61	0.02	−0.08	5.35	16.62	0.03	−0.08	4.99	15.51	0.02	−0.10
	0.50	50	16.94	−16.03	0.17	0.09	16.16	−15.29	0.13	0.06	16.58	−15.68	0.16	0.07	16.41	−15.52	0.16	0.07	16.76	−15.86	0.14	0.10	16.41	−15.52	0.15	0.07
		150	6.07	18.86	0.07	−0.05	4.71	14.63	0.01	−0.08	5.52	17.13	0.05	−0.06	5.27	16.37	0.04	−0.06	5.66	17.57	0.05	−0.06	5.23	16.25	0.04	−0.08
	0.75	50	16.57	−15.68	0.28	0.25	15.98	−15.12	0.21	0.19	16.28	−15.40	0.27	0.24	16.15	−15.28	0.26	0.23	16.41	−15.53	0.22	0.24	16.14	−15.27	0.25	0.22
		150	5.65	17.54	0.11	0.01	4.66	14.46	0.03	−0.04	5.21	16.20	0.10	−0.01	5.03	15.61	0.09	−0.01	5.32	16.54	0.08	0.00	4.99	15.50	0.09	−0.02
4	0.25	50	6.68	−6.30	0.19	0.07	8.26	−7.80	0.15	0.05	6.65	−6.27	0.18	0.07	6.72	−6.34	0.18	0.06	8.56	−8.08	0.16	0.14	6.92	−6.53	0.17	0.07
		150	1.61	4.99	0.06	−0.07	1.64	5.10	0.00	−0.08	1.49	4.63	0.05	−0.07	1.46	4.54	0.04	−0.07	4.62	14.35	0.06	0.04	1.87	5.81	0.03	−0.07
	0.50	50	18.19	−17.22	0.27	0.25	16.86	−15.95	0.23	0.19	17.95	−16.98	0.26	0.24	17.77	−16.81	0.26	0.24	19.66	−18.60	0.24	0.30	17.73	−16.77	0.25	0.24
		150	5.16	16.03	0.09	−0.02	4.69	14.55	0.04	−0.04	5.00	15.54	0.08	−0.02	4.93	15.30	0.08	−0.02	9.56	29.71	0.07	0.10	5.43	16.87	0.07	−0.01
	0.75	50	42.32	−40.25	0.49	0.63	32.23	−30.59	0.39	0.51	41.03	−39.01	0.48	0.62	39.79	−37.83	0.47	0.62	39.29	−37.35	0.44	0.62	39.10	−37.16	0.46	0.61
		150	13.40	41.63	0.18	0.17	9.84	30.57	0.13	0.09	13.07	40.61	0.18	0.16	12.72	39.53	0.17	0.16	16.42	51.02	0.14	0.27	12.84	39.90	0.16	0.16
ν = 8, δ = 0.5
π	b	N	Θ = 1				Θ = 2				Θ = 3				Θ = 4				Θ = 5				Θ = 6
			${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw
0.25	0.25	50	7.24	11.20	0.04	0.08	5.21	8.05	−0.02	−0.05	6.86	10.62	0.01	0.02	6.58	10.17	0.00	−0.01	6.99	10.81	0.06	0.17	6.64	10.27	0.02	0.03
		150	2.90	3.28	−0.02	−0.04	1.31	1.48	−0.09	−0.19	2.44	2.76	−0.05	−0.10	2.19	2.48	−0.06	−0.13	1.76	1.98	−0.09	−0.12	2.01	2.27	−0.07	−0.15
	0.50	50	3.90	6.02	0.05	0.02	2.94	4.53	−0.01	−0.07	3.51	5.41	0.02	−0.04	3.32	5.11	0.00	−0.06	3.43	5.28	0.00	0.01	3.34	5.14	0.00	−0.05
		150	1.87	2.11	0.02	−0.04	0.46	0.52	−0.08	−0.20	1.30	1.47	−0.01	−0.11	1.04	1.18	−0.03	−0.14	0.39	0.44	−0.06	−0.19	0.75	0.85	−0.05	−0.17
	0.75	50	1.69	2.60	0.08	0.03	1.27	1.95	0.00	−0.07	1.34	2.06	0.05	−0.04	1.19	1.83	0.03	−0.07	1.08	1.66	0.01	−0.05	1.15	1.76	0.03	−0.08
		150	0.81	0.92	0.05	−0.03	−0.26	−0.29	−0.05	−0.21	0.37	0.41	0.02	−0.11	0.17	0.19	0.00	−0.14	−0.30	−0.34	−0.03	−0.20	−0.06	−0.07	−0.02	−0.18
1	0.25	50	2.85	4.38	0.05	0.02	2.32	3.56	0.00	−0.07	2.56	3.95	0.03	−0.03	2.44	3.75	0.01	−0.05	2.75	4.23	0.05	0.09	2.44	3.75	0.02	−0.04
		150	1.13	1.28	0.01	−0.06	0.15	0.17	−0.08	−0.21	0.74	0.83	−0.03	−0.12	0.56	0.63	−0.04	−0.15	0.85	0.97	0.00	−0.01	0.53	0.60	−0.05	−0.15
	0.50	50	3.20	4.92	0.08	0.06	2.47	3.80	0.01	−0.03	2.82	4.35	0.05	0.01	2.66	4.09	0.04	−0.01	3.03	4.67	0.06	0.12	2.64	4.07	0.04	0.00
		150	1.50	1.70	0.02	−0.05	0.22	0.25	−0.08	−0.20	1.00	1.13	−0.02	−0.11	0.76	0.86	−0.03	−0.14	1.14	1.29	0.01	0.00	0.73	0.82	−0.04	−0.13
	0.75	50	2.83	4.36	0.14	0.17	2.30	3.53	0.06	0.06	2.54	3.91	0.12	0.13	2.41	3.71	0.10	0.10	2.69	4.14	0.09	0.18	2.40	3.69	0.09	0.10
		150	1.14	1.29	0.04	−0.01	0.15	0.17	−0.06	−0.17	0.74	0.84	0.01	−0.07	0.56	0.63	−0.01	−0.10	0.85	0.96	0.02	0.02	0.53	0.60	−0.01	−0.10
4	0.25	50	0.63	0.96	0.04	−0.05	0.74	1.13	−0.01	−0.10	0.54	0.82	0.02	−0.06	0.51	0.78	0.02	−0.07	10.11	10.70	0.04	0.05	0.55	0.85	0.01	−0.07
		150	−0.23	−0.26	−0.06	−0.17	−0.49	−0.55	−0.13	−0.22	−0.36	−0.40	−0.07	−0.19	−0.41	−0.46	−0.08	−0.20	0.50	0.57	−0.02	−0.02	−0.28	−0.32	−0.08	−0.18
	0.50	50	2.68	4.12	0.09	0.09	2.24	3.44	0.05	0.02	2.54	3.92	0.08	0.08	2.47	3.80	0.08	0.07	3.04	4.68	0.07	0.16	2.45	3.76	0.07	0.07
		150	0.49	0.55	−0.04	−0.13	0.07	0.08	−0.10	−0.18	0.32	0.37	−0.06	−0.15	0.25	0.28	−0.07	−0.16	1.54	1.75	−0.03	0.03	0.41	0.46	−0.07	−0.14
	0.75	50	6.50	10.05	0.26	0.37	4.73	7.30	0.17	0.28	6.23	9.63	0.25	0.36	6.00	9.28	0.24	0.36	5.94	9.18	0.21	0.37	5.84	9.02	0.23	0.35
		150	1.77	2.00	0.01	0.02	0.94	1.07	−0.04	−0.07	1.61	1.82	0.00	0.00	1.51	1.70	0.00	−0.01	2.56	2.90	0.00	0.15	1.57	1.77	−0.01	0.00
ν = 8, δ = 1.5
π	b	N	Θ = 1				Θ = 2				Θ = 3				Θ = 4				Θ = 5				Θ = 6
			${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw	${\eta }_{\mathrm{\Lambda }}^2$	${\omega }_{\mathrm{\Lambda }}^2$	Aw	CLw
0.25	0.25	50	0.99	1.06	0.02	0.07	0.55	0.59	−0.05	0.03	0.87	0.93	0.00	0.07	0.79	0.85	−0.02	0.06	0.94	1.02	0.02	0.07	0.80	0.86	−0.01	0.07
		150	0.64	0.65	−0.02	0.07	−0.13	−0.13	−0.12	−0.09	0.42	0.43	−0.05	0.06	0.29	0.30	−0.08	0.04	0.31	0.31	−0.09	0.07	0.23	0.23	−0.09	0.04
	0.50	50	0.81	0.87	0.03	0.07	0.39	0.42	−0.04	0.02	0.64	0.69	0.00	0.07	0.55	0.59	−0.01	0.06	0.68	0.73	0.00	0.07	0.54	0.58	−0.02	0.06
		150	0.60	0.61	0.01	0.07	−0.26	−0.27	−0.10	−0.15	0.34	0.35	−0.02	0.06	0.19	0.20	−0.05	0.04	0.20	0.20	−0.07	0.06	0.11	0.11	−0.07	0.03
	0.75	50	0.43	0.46	0.04	0.07	0.01	0.01	−0.03	0.00	0.24	0.25	0.02	0.07	0.13	0.14	0.00	0.06	0.24	0.25	0.00	0.07	0.11	0.11	−0.01	0.06
		150	0.31	0.32	0.03	0.07	−0.53	−0.54	−0.07	−0.18	0.04	0.04	0.00	0.06	−0.11	−0.11	−0.02	0.04	−0.09	−0.10	−0.05	0.07	−0.19	−0.19	−0.05	0.03
1	0.25	50	0.43	0.46	0.01	0.07	0.02	0.02	−0.08	−0.03	0.25	0.27	−0.03	0.05	0.15	0.16	−0.05	0.03	0.35	0.38	−0.01	0.07	0.14	0.15	−0.05	0.04
		150	0.28	0.29	−0.01	0.07	−0.55	−0.56	−0.11	−0.24	0.02	0.02	−0.05	0.04	−0.12	−0.13	−0.07	0.00	0.10	0.10	−0.03	0.07	−0.15	−0.15	−0.08	0.01
	0.50	50	0.62	0.66	0.01	0.07	0.15	0.16	−0.07	−0.03	0.42	0.45	−0.02	0.05	0.31	0.33	−0.04	0.04	0.53	0.57	0.00	0.07	0.30	0.32	−0.04	0.04
		150	0.46	0.47	0.00	0.07	−0.47	−0.48	−0.11	−0.23	0.19	0.20	−0.05	0.04	0.03	0.03	−0.07	0.00	0.28	0.28	−0.03	0.07	0.01	0.01	−0.08	0.01
	0.75	50	0.43	0.46	0.02	0.07	0.02	0.02	−0.06	−0.01	0.25	0.26	−0.01	0.06	0.15	0.16	−0.02	0.04	0.35	0.37	0.00	0.07	0.14	0.15	−0.03	0.04
		150	0.28	0.29	0.00	0.07	−0.55	−0.56	−0.11	−0.21	0.02	0.02	−0.04	0.04	−0.12	−0.13	−0.07	0.01	0.10	0.10	−0.03	0.07	−0.15	−0.15	−0.07	0.01
4	0.25	50	−0.32	−0.34	−0.08	−0.04	−0.49	−0.52	−0.16	−0.17	−0.42	−0.45	−0.11	−0.09	−0.47	−0.50	−0.12	−0.12	−0.21	−0.22	−0.04	0.03	−0.46	−0.49	−0.12	−0.11
		150	−0.46	−0.47	−0.12	−0.09	−0.82	−0.83	−0.22	−0.35	−0.60	−0.61	−0.15	−0.18	−0.67	−0.68	−0.17	−0.23	−0.33	−0.33	−0.04	0.00	−0.63	−0.64	−0.15	−0.19
	0.50	50	0.08	0.08	−0.08	−0.01	−0.18	−0.19	−0.14	−0.11	−0.03	−0.03	−0.10	−0.04	−0.09	−0.09	−0.11	−0.06	0.17	0.18	−0.06	0.04	−0.10	−0.10	−0.11	−0.06
		150	−0.23	−0.24	−0.14	−0.09	−0.66	−0.68	−0.23	−0.31	−0.39	−0.40	−0.17	−0.17	−0.47	−0.48	−0.18	−0.21	−0.04	−0.05	−0.08	0.01	−0.43	−0.43	−0.17	−0.17
	0.75	50	0.49	0.53	−0.02	0.05	0.13	0.14	−0.09	0.00	0.41	0.44	−0.03	0.04	0.35	0.38	−0.04	0.04	0.42	0.45	−0.05	0.06	0.31	0.33	−0.05	0.04
		150	−0.10	−0.10	−0.14	−0.05	−0.53	−0.54	−0.21	−0.22	−0.23	−0.24	−0.16	−0.10	−0.31	−0.31	−0.18	−0.12	0.06	0.06	−0.11	0.04	−0.29	−0.29	−0.17	−0.10

π is the variance ratio, b is the base rate, N is the total sample size, ν is the number of dependent variables, δ is the true effect size expressed as the standardized mean difference, $\Theta$ indicates six distributions: 1 = normal, 2 = mixed normal, 3 = peaked (ϒ₁ = 0 and ϒ₂ = 6), 4 = peaked (ϒ₁ = 0 and ϒ₂ = 154.84), 5 = skewed (ϒ₁ = 2 and ϒ₂ = 6), and 6 = skewed (4.90 and ϒ₂ = 4,673.80 ), where ϒ₁ refers to skewness and ϒ₂ refers to kurtosis, ${\eta }_{\mathrm{\Lambda }}^2$ the generalized eta squared, ${\omega }_{\mathrm{\Lambda }}^2$ is the generalized omega squared, $A_w$ is the non-parametric probability of superiority effect size, and $C{L}_w$ is the parametric probability of superiority effect size.

Second, the variance ratio (π; i.e. variance ratio of the treatment group) did not show obvious impact on A_w and CL_w. For ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ , the biases were smaller when $\pi =1$ and $\delta \ge .50$ . When $\pi \ne 1$ , the biases depended on the level of $\delta$ . When $\delta \ge .50$ and π changed from 0.25 to 4.00, the biases changed from positive to negative. This was because the stronger group had a smaller (or larger) variance, and hence, the estimated sum of squares became over-precise (or under-precise) so that a stronger ES resulted for $\pi =.25$ (or $\pi =4$ ). When $\delta \le .20$ , there was no obvious pattern of relationships between π and biases.

Third, the base rate (b; i.e. proportion of the sample size of the control group) did not show a clear pattern of relationship with ${\omega }_{\mathrm{\Lambda }}^2$ , A_w and CL_w. For ${\eta }_{\mathrm{\Lambda }}^2$ , the effects were more complicated. When $\delta \le .50$ and $\pi =1$ , b did not show much impact on ${\eta }_{\mathrm{\Lambda }}^2$ . When b changed from 0.25 to 0.75 and $\pi =.25$ biases generally decreased; but when $\pi =4$ , biases generally increased. There was no clear explanation for this pattern, but we are sure that the accuracy of ${\eta }_{\mathrm{\Lambda }}^2$ fluctuated substantially with varying levels of the base rate and variance ratio, and thus, it was not robust to these factors.

Fourth, when the true ES (δ) increased, the biases of ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ generally decreased due to the magnitude of the effect being more accurately detected when it is large. The other ESs (A_w and CL_w) were generally accurate regardless of the magnitude of δ.

Fifth, when the number of DVs (ν) increased, the accuracy of ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ generally decreased. This was probably due to the more complicated mathematical procedure required to obtain the a weights (equation (4)) that maximized the weighted difference, given that other factors were held constant. On the other hand, A_w and CL_w were generally robust to the increased number of DVs.

Sixth, it came as a surprise that the performance of ${\eta }_{\mathrm{\Lambda }}^2$ , ${\omega }_{\mathrm{\Lambda }}^2$ , and CL_w did not differ across the six distributions. For ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ , this was because the sum-of-squares based Wilks Lambda ( $\mathrm{\Lambda }$ ; equation (1)) appeared to be robust to any shape of the distribution, if both groups shared the same distribution. That is, the sum of squares measures the degree to which the scores surround the mean. When the two groups shared the same distribution, the sum of squares would measure the variability of the scores on the same metric, and hence, the Wilks Lambda as well as ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ were unaffected by non-normal distributions. On the other hand, if the distributions are different for the two groups (e.g. normal and skewed), ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ may become inaccurate, but the present study did not examine this factor. CL_w was also found to be robust to any shape of distribution. It appeared that the parametric mean difference estimator (i.e, $\left(\overline{w_p}-\overline{w_q}\right)/s_w$ ; equation (6)) was also robust to non-normal distributions, when the two groups shared the same non-normal distributions. For the last ES (A_w), its mathematical equation did not rely on any parametric assumption, and hence, it was robust to any of the six distributions as expected.

Conclusion and discussion

This study proposes and develops a non-parametric ES (A_w) for MANOVA. The results of a Monte-Carlo simulation showed that A_w was accurate across the simulated conditions and robust to violations of the two key assumptions (multivariate normality and homogeneity of covariance matrices). It also outperformed the two conventional parametric ESs (i.e. ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ ) and its parametric counterpart CL_w. Hence, researchers and practitioners are encouraged to report A_w for ES evaluation in a MANOVA scenario, especially when the key assumptions are violated.

The proposed A_w is important for researchers in behavioral and social sciences research because evaluating the conventional ESs ( ${\eta }_{\mathrm{\Lambda }}^2$ and ${\omega }_{\mathrm{\Lambda }}^2$ ) could be misleading in the existing literature. Our findings showed that ${\eta }_{\mathrm{\Lambda }}^2$ could be 4297% larger than its true ES when the homogeneity of variance assumption was violated (i.e. variance ratio = 0.25), the sample size ratio of the control group to the treatment group was 0.25, the number of DVs was 8, the sample size was 50, and the true ES was small (0.20). By the same token, the conventional ${\omega }_{\mathrm{\Lambda }}^2$ could be 4088% below its true value under the same conditions. These findings align with a recent claim by Hoekstra et al. (2012) who strongly encouraged researchers to check the assumptions for conventional analyses. The authors found that only 33.3% and 25% of researchers in psychology have correctly checked the assumptions of homogeneity of variances and normality, respectively, for conventional and widely-used analyses such as t-test, ANOVA, and regression. The majority of the participants in Hoekstra et al.’s study stated that they were either unfamiliar with these assumptions or did not understand how to check these assumptions. Moreover, Ruscio and Roche (2012) reviewed 455 studies published in leading psychology journals and found that the reported variance ratios often fail to meet the requirement, thereby posing a threat or risk of inflated Type 1 error rates when using conventional, parametric statistical methods. We hope that as researchers in psychology start to embrace the recommended practice of reporting ESs, they will be more aware of the importance of these assumptions and will consider adopting non-parametric alternatives where appropriate. Considering the current lack of awareness about the importance of parametric assumptions for conventional ESs, and the prevalence of research situations in which the assumptions are violated, this study offers a non-parametric alternative (A_w) for MANOVA that does not depend on these assumptions.

The implications of this study can be generalized to researchers and practitioners in a wide range of other disciplines, both in social and natural sciences, who often use MANOVA. For example, clinical trials researchers often examine the difference between a treatment group and a placebo group in a number of health-related criterion measures (e.g. body mass index, blood pressure). On some occasions, their data may violate the assumptions for the traditional ESs, and hence, the proposed A_w can provide a more trustworthy measure for evaluating the difference between the two groups. Biological researchers are often interested in comparing the difference between an experimental group and a control group in a lab setting (e.g. effects of room temperature and absolute zero degree on cellular motility and signaling), and they could also report A_w for this scenario.

Limitations and directions for future research

A first area of ongoing research lies in examining the effects of different distributions for the two groups of observations. In this study, the two groups of scores were assumed to follow the same (either normal or non-normal) distributions. Future research should include a simulation study to examine the effects of unbalanced distributions (e.g. normal vs skewed) on the ESs in MANOVA.

A second area of research involves generalization of the proposed A_w to the one-way MANOVA with more than two independent samples as well as to the multi-way MANOVA that involves multiple IVs (e.g. factorial and mixed designs). These more general or complicated types of MANOVA are also popular in psychology research. This study lays foundation for the non-parametric ES in a simpler MANOVA. Ruscio and Gera (2013) have recently provided the extensions of A_w to one-way ANOVA. Additional research can derive mathematical equations for one-way MANOVA based on Ruscio and Gera’s study and provide empirical evidence for this statistic based on a simulation study.

In addition to the reporting of ES, the new statistical practices suggest that researchers report the CIs surrounding a reported ES. Therefore, a third area for future research is examination of the sampling distribution or confidence intervals (CIs) surrounding the proposed A_w. Ruscio and Mullen (2012) found that the bootstrap CIs constructed for the non-parametric A in ANOVA were accurate. Further research can also examine the use of the bootstrap procedure for the CIs surrounding the proposed A_w in MANOVA.

Supplemental Material

sj-pdf-1-mio-10.1177_20597991211055949 – Supplemental material for A robust effect size measure A_w for MANOVA with non-normal and non-homogenous data

Supplemental material, sj-pdf-1-mio-10.1177_20597991211055949 for A robust effect size measure A_w for MANOVA with non-normal and non-homogenous data by Johnson Ching-Hong Li, Marcello Nesca, Rory Michael Waisman, Yongtian Cheng and Virginia Man Chung Tze in Methodological Innovations