Our paper provides a conceptualization of magnitude-based hypotheses (MBHs). We define an MBH as a specific type of hypothesis that tests for relative differences in the independent impact (i.e., effect size difference) of at least two explanatory variables on a given outcome. We reviewed 1,715 articles across eight leading management journals and found that nearly 10% (165) of articles feature an MBH, employing 41 distinct methodological approaches to test them. However, approximately 40% of these papers show missteps in the post-estimation process required to evaluate MBHs. To address this issue, we offer a conceptual framework, an empirical illustration using Bayesian analysis and frequentist statistics, and a decision-tree guideline that outlines key steps for evaluating MBHs. Overall, we contribute a framework for applying MBHs, demonstrating how they can shift theoretical inquiry from binary questions of whether an effect exists, to more comparative questions about how much a construct matters,compared to what, and under which conditions.
American Statistician. (2019). Statistical inference in the 21st century: A world beyond p<.05.
2.
AndersonA. A. (2019). Assessing statistical results: Magnitude, precision, and model uncertainty. American Statistician, 73(suppl. 1), 118–121. https://doi.org/10.1080/00031305.2018.1537889
3.
BeliaS.FidlerF.WilliamsJ.CummingG. (2005). Researchers misunderstand confidence intervals and standard error bars. Psychological Methods, 10(4), 389. https://doi.org/10.1037/1082-989X.10.4.389
4.
BerghD. D.SharpB. M.AguinisH.LiM. (2017). Is there a credibility crisis in strategic management research? Evidence on the reproducibility of study findings. Strategic Organization, 15(3), 423–436. https://doi.org/10.1177/1476127017701076
5.
BettisR. A. (2012). The search for asterisks: Compromised statistical tests and flawed theories. Strategic Management Journal, 33(1), 108–113. https://doi.org/10.1002/smj.975
6.
BettisR. A.HelfatC. E.ShaverJ. M. (2016). The necessity, logic, and forms of replication. Strategic Management Journal, 37(11), 2193–2203. https://doi.org/10.1002/smj.2580
7.
CalabròA.MinichilliA.AmoreM. D.BrogiM. (2018). The courage to choose! primogeniture and leadership succession in family firms. Strategic Management Journal, 39(7), 2014–2035. https://doi.org/10.1002/smj.2760
8.
CastroS. L. (2002). Data analytic methods for the analysis of multilevel questions: A comparison of intraclass correlation coefficients, rwg (j), hierarchical linear modeling, within-and between-analysis, and random group resampling. Leadership Quarterly, 13(1), 69–93. https://doi.org/10.1016/S1048-9843(01)00105-9
9.
CertoS. T.AlbaderL. A.RaneyK. E.BusenbarkJ. R. (2024). A Bayesian approach to nested data analysis: A primer for strategic management research. Strategic Organization, 22(2), 241–268. https://doi.org/10.1177/14761270211072248
10.
CohenJ. (2013). Statistical power analysis for the behavioral sciences. Routledge.
11.
CummingG. (2009). Inference by eye: Reading the overlap of independent confidence intervals. Statistics in Medicine, 28(2), 205–220. https://doi.org/10.1002/sim.3471
12.
DiMaggioP. J.PowellW. W. (1983). The iron cage revisited: Institutional isomorphism and collective rationality in organizational fields. American Sociological Review, 48(2), 147–160. https://doi.org/10.2307/2095101
13.
DurandR.PaugamL.StolowyH. (2019). Do investors actually value sustainability indices? Replication, development, and new evidence on CSR visibility. Strategic Management Journal, 40(9), 1471–1490. https://doi.org/10.1002/smj.3035
14.
EdwardsJ. R.BerryJ. W. (2010). The presence of something or the absence of nothing: Increasing theoretical precision in management research. Organizational Research Methods, 13(4), 668–689. https://doi.org/10.1177/1094428110380467
15.
FunderD. C.OzerD. J. (2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2(2), 156–168. https://doi.org/10.1177/2515245919847202
16.
GrantA. M.PollockT. G. (2011). Publishing in AMJ—Part 3: Setting the hook. Academy of Management Journal, 54(5), 873–879. https://doi.org/10.5465/amj.2011.4000
17.
GreenlandS.SennS. J.RothmanK. J.CarlinJ. B.PooleC.GoodmanS. N.AltmanD. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. https://doi.org/10.1007/s10654-016-0149-3
18.
HoekstraR.MoreyR. D.RouderJ. N.WagenmakersE. J. (2014). Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review, 21(5), 1157–1164. https://doi.org/10.3758/s13423-013-0572-3
19.
HoijtinkH.MulderJ.van LissaC.GuX. (2019). A tutorial on testing hypotheses using the Bayes factor. Psychological Methods, 24(5), 539–556. https://doi.org/10.1037/met0000201
20.
HüffmeierJ.ZerresA.FreundP. A.BackhausK.TrötschelR.HertelG. (2019). Strong or weak synergy? Revising the assumption of team-related advantages in integrative negotiations. Journal of Management, 45(7), 2721–2750. https://doi.org/10.1177/0149206318770245
21.
JebbA. T.WooS. E. (2015). A Bayesian primer for the organizational sciences: The “two sources” and an introduction to BugsXLA. Organizational Research Methods, 18(1), 92–132. https://doi.org/10.1177/1094428114553060
22.
KassR. E.RafteryA. E. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795.
KhalilzadehJ.TasciA. D. (2017). Large sample size, significance level, and the effect size: Solutions to perils of using big data for academic research. Tourism Management, 62, 89–96. https://doi.org/10.1016/j.tourman.2017.03.026
KrauseR.WuZ.BrutonG. D.CarterS. M. (2019). The coercive isomorphism ripple effect: An investigation of nonprofit interlocks on corporate boards. Academy of Management Journal, 62(1), 283–308. https://doi.org/10.5465/amj.2017.0064
27.
KruschkeJ. K.AguinisH.JooH. (2012). The time has come: Bayesian methods for data analysis in the organizational sciences. Organizational Research Methods, 15(4), 722–752. https://doi.org/10.1177/1094428112457829
28.
KruschkeJ. K.LiddellT. M. (2018). The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review, 25, 178–206. https://doi.org/10.3758/s13423-016-1221-4
29.
MellJ. N.DeChurchL. A.LeendersR. T. A.ContractorN. (2020). Identity asymmetries: An experimental investigation of social identity and information exchange in multiteam systems. Academy of Management Journal, 63(5), 1561–1590. https://doi.org/10.5465/amj.2018.0325
30.
MittalN.BhandariM.KumbhareD. (2019). A tale of confusion from overlapping confidence intervals. American Journal of Physical Medicine & Rehabilitation, 98(1), 81–83. https://doi.org/10.1097/PHM.0000000000001016
31.
PaternosterR.BrameR.MazerolleP.PiqueroA. (1998). Using the correct statistical test for the equality of regression coefficients. Criminology; An Interdisciplinary Journal, 36(4), 859–866. https://doi.org/10.1111/j.1745-9125.1998.tb01268.x
32.
ReillyG.NybergA. J.MaltarichM.WellerI. (2014). Human capital flows: Using context-emergent turnover (CET) theory to explore the process by which turnover, hiring, and job demands affect patient satisfaction. Academy of Management Journal, 57(3), 766–790. https://doi.org/10.5465/amj.2012.0132
33.
RomerD. (2020). In praise of confidence intervals. In AEA Papers and Proceedings (Vol. 110, pp. 55–60).
34.
RubensteinA. L.Kammeyer-MuellerJ. D.ThundiyilT. G. (2020). The comparative effects of supervisor helping motives on newcomer adjustment and socialization outcomes. Journal of Applied Psychology, 105(12), 1466–1489. https://doi.org/10.1037/apl0000492
35.
SchwabA.AbrahamsonE.StarbuckW. H.FidlerF. (2011). Perspective—Researchers should make thoughtful assessments instead of null-hypothesis significance tests. Organization Science, 22(4), 1105–1120. https://doi.org/10.1287/orsc.1100.0557
36.
SchwabA.StarbuckW. H. (2017). A call for openness in research reporting: How to turn covert practices into helpful tools. Academy of Management Learning & Education, 16(1), 125–141. https://doi.org/10.5465/amle.2016.0039
37.
SemadeniM.WithersM. C.CertoS. T. (2014). The perils of endogeneity and instrumental variables in strategy research: Understanding through simulations. Strategic Management Journal, 35(7), 1070–1079. https://doi.org/10.1002/smj.2136
38.
ShroutP. E.RodgersJ. L. (2018). Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis. Annual Review of Psychology, 69, 487–510. https://doi.org/10.1146/annurev-psych-122216-011845
39.
SimsekZ.VaaraE.ParuchuriS.NadkarniS.ShawJ. D. (2019). New ways of seeing big data. Academy of Management Journal, 62(4), 971–978. https://doi.org/10.5465/amj.2019.4004
40.
TendeiroJ. N.KiersH. A.HoekstraR.WongT. K.MoreyR. D. (2024). Diagnosing the misuse of the Bayes factor in applied research. Advances in Methods and Practices in Psychological Science, 7(1), 1–19. https://doi.org/10.1177/25152459231213371
41.
TongC. (2019). Statistical inference enables bad science; statistical thinking enables good science. The American Statistician, 73(suppl. 1), 246–261. https://doi.org/10.1080/00031305.2018.1518264
WilmotM. P.OnesD. S. (2021). Occupational characteristics moderate personality–performance relations in major occupational groups. Journal of Vocational Behavior, 131, Article 103655. https://doi.org/10.1016/j.jvb.2021.103655
44.
ZyphurM. J.OswaldF. L. (2015). Bayesian Estimation and inference: A user’s guide. Journal of Management, 41(2), 390–420. https://doi.org/10.1177/0149206313501200
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
