Sage Journals: Discover world-class research

Abstract

Testing the equivalence rather than the difference of two-group means might be of interest. Researchers may verify randomization validity by testing the equivalence of baseline means in randomized controlled trials and the equivalence of post-treatment outcomes when they expect equal means, while other dimensions (e.g., costs) differ. However, the analytical and design strategies for equivalence tests are limited. For example, the analytical and design methods usually do not include covariate adjustments, and the power formula development has not used the correct product function. The present study develops and validates the statistical power formula for randomized controlled trials to detect statistical equivalence with and without covariate adjustment. It further presents an optimal design framework for identifying the sample allocation that maximizes statistical power under a fixed budget. The proposed methods have been implemented in the R package anomo. Illustration examples are also provided.

Keywords

equivalence test statistical power randomized controlled trials optimal sample allocation

Get full access to this article

View all access options for this article.

References

Anderson

S. F.

Kelley

(2024). Sample size planning for replication studies: The devil is in the design. Psychological Methods, 29(5), 844–867. https://doi.org/10.1037/met0000520

Bonett

D. G.

(2021). Design and analysis of replication studies. Organizational Research Methods, 24(3), 513–529. https://doi.org/10.1177/1094428120911088

Campbell

(2024). Equivalence testing for linear regression. Psychological Methods, 29(1), 88–98. https://doi.org/10.1037/met0000596

Campbell

Pierce

S. J.

Sharma

D. B.

Feeney

Fehler-Cabral

(2016). Should rape kit testing be prioritized by victim-offender relationship? Empirical comparison of forensic testing outcomes for stranger and non-stranger sexual assaults. Criminology and Public Policy, 15(2), 555–583. https://doi.org/10.1111/1745-9133.12205

Cribbie

Alter

Beribisky

Chalmers

Counsell

Farmus

Gutierrez

(2024). negligible: A collection of functions for negligible effect/equivalence testing. Retrieved from https://cran.r-project.org/package=negligible.

Dong

Lipsey

M. W.

(2018). Can propensity score analysis approximate randomized experiments using pretest and demographic information in pre-K intervention research? Evaluation Review, 42(1), 34–70. https://doi.org/10.1177/0193841X17749824

Guo

J. H.

Chen

H. J.

Luh

W. M.

(2011). Sample size planning with the cost constraint for testing superiority and equivalence of two independent groups. British Journal of Mathematical and Statistical Psychology, 64(3), 439–461. https://doi.org/10.1348/000711010X512408

Guo

J. H.

Chen

H. J.

Luh

W. M.

(2019). Optimal sample sizes for testing the equivalence of two means. Methodology, 15(3), 128–136. https://doi.org/10.1027/1614-2241/a000171

Hayes

A. F.

Scharkow

(2013). The relative trustworthiness of inferential tests of the indirect effect in statistical mediation analysis: Does method really matter? Psychological Science, 24(10), 1918–1927. https://doi.org/10.1177%2F0956797613480187

10.

Hedges

L. V.

Hedberg

E. C.

(2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. https://doi.org/10.3102/0162373707299706

11.

Hoogland

J. J.

Boomsma

(1998). Robustness studies in covariance structure modeling: An overview and a meta-analysis. Sociological Methods & Research, 26(3), 329–367. https://doi.org/10.1177/0049124198026003003

12.

Imai

Tingley

Yamamoto

(2013). Experimental designs for identifying causal mechanisms. Journal of the Royal Statistical Society Series A: Statistics in Society, 176(1), 5–51. https://doi.org/10.1111/j.1467-985X.2012.01032.x

13.

Jan

S. L.

Shieh

(2017). Optimal sample size determinations for the heteroscedastic two one-sided tests of mean equivalence: Design schemes and software implementations. Journal of Educational and Behavioral Statistics, 42(2), 145–165. https://doi.org/10.3102/1076998616671974

14.

Korendijk

E. J.

Moerbeek

Maas

C. J.

(2010). The robustness of designs for trials with nested data against incorrect initial intracluster correlation coefficient estimates. Journal of Educational and Behavioral Statistics, 35(5), 566–585. https://doi.org/10.3102/1076998609360774

15.

Kraft

M. A.

(2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241–253. https://doi.org/10.3102/0013189X20912798

16.

Lai

M. H.

Kwok

O. M.

(2015). Examining the rule of thumb of not using multilevel modeling: The “design effect smaller than two” rule. The Journal of Experimental Education, 83(3), 423–438. https://doi.org/10.1080/00220973.2014.907229

17.

Lakens

(2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–362. https://doi.org/10.1177/1948550617697177

18.

Lakens

Scheel

A. M.

Isager

P. M.

(2018). Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science, 1(2), 259–269. https://doi.org/10.1177/2515245918770963

19.

Liu

(2003). Statistical power and optimum sample allocation ratio for treatment and control having unequal costs per unit of randomization. Journal of Educational and Behavioral Statistics, 28(3), 231–248. https://doi.org/10.3102/10769986028003231

20.

MacKinnon

D. P.

Lockwood

C. M.

Williams

(2004). Confidence limits for the indirect effect: Distribution of the product and resampling methods. Multivariate Behavioral Research, 39(1), 99–128. https://doi.org/10.1207/s15327906mbr3901_4

21.

Mascha

E. J.

Sessler

D. I.

(2011). Equivalence and noninferiority testing in regression models and repeated measures designs. Anesthesia & Analgesia, 112(3), 678–687. https://doi.org/10.1213/ane.0b013e318206f872

22.

Nahum-Shani

Qian

Almirall

Pelham

W. E.

Gnagy

Fabiano

G. A.

Murphy

S. A.

(2012). Experimental design and primary data analysis methods for comparing adaptive interventions. Psychological Methods, 17(4), 457–477. https://doi.org/10.1037/a0029372

23.

Nam

J. M.

(1973). Optimum sample sizes for the comparison of the control and treatment. Biometrics, 29(1), 101–108. https://doi.org/10.2307/2529679

24.

Preacher

K. J.

Selig

J. P.

(2012). Advantages of Monte Carlo confidence intervals for indirect effects. Communication Methods and Measures, 6(2), 77–98. https://doi.org/10.1080/19312458.2012.679848

25.

Raudenbush

S. W.

(1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2(2), 173–185. https://doi.org/10.1037/1082-989X.2.2.173

26.

R Core Team. (2025). R: A language and environment for statistical computing (Version 4.5.2) [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org .

27.

Schuirmann

D. J.

(1987). A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Biopharmaceutics, 15(6), 657–680. https://doi.org/10.1007/BF01068419

28.

Seaman

M. A.

Serlin

R. C.

(1998). Equivalence confidence intervals for two-group comparisons of means. Psychological Methods, 3(4), 403–411. https://doi.org/10.1037/1082-989X.3.4.403

29.

Shadish

Cook

T. D.

Campbell

(2002). Experimental and quasi-experimental designs for generalized causal inference (2nd ed.). Houghton Mifflin.

30.

Shen

(2026). anomo: Analysis of moderation, statistical power, and optimal design for studies detecting difference and equivalence (Version 1.3.2) [Computer software]. https://cran.r-project.org/package=anomo.

31.

Shen

Kelcey

(2020). Optimal sample allocation under unequal costs in cluster-randomized trials. Journal of Educational and Behavioral Statistics, 45(4), 446–474. https://doi.org/10.3102/1076998620912418

32.

Shen

Kelcey

(2022a). Optimal sample allocation in multisite randomized trials. The Journal of Experimental Education, 90(3), 693–711. https://doi.org/10.1080/00220973.2020.1830361

33.

Shen

Kelcey

(2022b). Optimal sample allocation for three-level multisite cluster-randomized trials. Journal of Research on Educational Effectiveness, 15(1), 130–150. https://doi.org/10.1080/19345747.2021.1953200

34.

Shen

Leite

Zhang

Quan

Kuang

(2025). Using ant colony optimization to identify optimal sample allocations in cluster-randomized trials. The Journal of Experimental Education, 93(1), 167–185. https://doi.org/10.1080/00220973.2024.2306392

35.

Shen

Leite

(2024). Statistical power and optimal design for randomized controlled trials investigating mediation effects. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000698

36.

Shieh

(2016). Exact power and sample size calculations for the two one-sided tests of equivalence. PLoS ONE, 11(9), Article e0162093. https://doi.org/10.1371/journal.pone.0162093

37.

Steiner

P. M.

Sheehan

Wong

V. C.

(2025). Correspondence measures for assessing replication success. Psychological Methods, 30(4), 793–814. https://doi.org/10.1037/met0000597

38.

Tang

(2018a). Exact and approximate power and sample size calculations for analysis of covariance in randomized clinical trials with or without stratification. Statistics in Biopharmaceutical Research, 10(4), 274–286. https://doi.org/10.1080/19466315.2018.1459312

39.

Tang

(2018b). A noniterative sample size procedure for tests based on t distributions. Statistics in Medicine, 37(22), 3197–3213. https://doi.org/10.1002/sim.7807

40.

Tryon

W. W.

Lewis

(2009). Evaluating independent proportions for statistical difference, equivalence, indeterminacy, and trivial difference using inferential confidence intervals. Journal of Educational and Behavioral Statistics, 34(2), 171–189. https://doi.org/10.3102/1076998609332753

41.

Tunes da Silva

Logan

B. R.

Klein

J. P.

(2009). Methods for equivalence and noninferiority testing. Biology of Blood and Marrow Transplantation, 15(1, Suppl.), 120–127. https://doi.org/10.1016/j.bbmt.2008.10.004

42.

Westlake

W. J.

(1976). Symmetrical confidence intervals for bioequivalence trials. Biometrics, 32(4), 741–744. https://www.jstor.org/stable/2529259. https://doi.org/10.2307/2529259

43.

What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance (NCEE). This report is available on the What Works Clearinghouse website at https://ies.ed.gov/ncee/wwc/Handbooks.

44.

Zhang

(2003). A simple formula for sample size calculation in equivalence studies. Journal of Biopharmaceutical Statistics, 13(3), 529–538. https://doi.org/10.1081/BIP-120022772

Optimal Design and Analysis for Equivalence Tests in Randomized Controlled Trials

Abstract

Keywords

Get full access to this article

References