Sage Journals: Discover world-class research

Abstract

Methodology serves an essential role in advancing psychological science. However, meta-science research points to a leaky translational pipeline in which substantive research often fails to utilize recommended methodological practices. Various explanations for this problem include valuing the development of methods over methodology (making tools over using tools), incentives for methodological research, incentives for promoting pedagogy in methodology, an insufficient number of quantitative methodologists in the discipline, and scarcity of resources for substantive researchers seeking more advanced methodological training. Policy makers might consider several recommendations that could mitigate extant leaks in the translational pipeline.

Keywords

quantitative methodology methods training pedagogy

Get full access to this article

View all access options for this article.

References

Abelson

R. P.

(1995). Statistics as principled argument. Taylor & Francis.

Aiken

L. S.

West

S. G.

Millsap

R. E.

(2008). Doctoral training in statistics, measurement, and methodology in psychology: Replication and extension of Aiken, West, Sechrest, and Reno’s (1990) survey of PhD programs in North America. American Psychologist, 63(1), 32–50. https://doi.org/10.1037/0003-066X.63.1.32

Aiken

L. S.

West

S. G.

Sechrest

Reno

R. R.

Roediger

H. L.

III Scarr

Kazdin

A. E.

Sherman

S. J.

(1990). Graduate training in statistics, methodology, and measurement in psychology: A survey of PhD programs in North America. American Psychologist, 45(6), 721–734. https://doi.org/10.1037/0003-066X.45.6.721

American Psychological Association (2009). Report of the task force for increasing the number of quantitative psychologists. https://www.apa.org/science/leadership/bsa/quantitative

Anderson

J. C.

Gerbing

D. W.

(1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103, 411–423. https://doi.org/10.1037/0033-2909.103.3.411

Bakk

Kuha

(2021). Relating latent class membership to external variables: An overview. British Journal of Mathematical and Statistical Psychology, 74(2), 340–362. https://doi.org/10.1111/bmsp.12227

Baron

R. M.

Kenny

D. A.

(1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173–1182. https://doi.org/10.1037/0022-3514.51.6.1173

Bauer

D. J.

(2017). A more general model for testing measurement invariance and differential item functioning. Psychological Methods, 22(3), 507–526. https://doi.org/10.1037/met0000077

Bauer

D. J.

Hussong

A. M.

(2009). Psychometric approaches for developing commensurate measures across independent studies: Traditional and new models. Psychological Methods, 14(2), 101–125. https://doi.org/10.1037/a0015583

10.

Beaujean

A. A.

(2021, October). Message from the new president. The Score, the Division 5 (Quantitative and Qualitative Methods) Newsletter of the American Psychological Association. https://www.apadivisions.org/division-5/publications/score/2021/10/message-president

11.

Benson

(1998). Developing a strong program of construct validation: A test anxiety example. Educational Measurement: Issues and Practice, 17(1), 10–17. https://doi.org/10.1111/j.1745-3992.1998.tb00616.x

12.

Chalmers

R. P.

(2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06

13.

Clay

(2005). Too few in quantitative psychology. Monitor on Psychology, 36(8), 26–28.

14.

Cohen

(1988). Statistical power analysis for the behavioral sciences. Academic Press.

15.

Cohen

(1992). Statistical power analysis. Current Directions in Psychological Science, 1(3), 98–101. https://doi.org/10.1111/1467-8721.ep10768783

16.

Cohen

West

S. G.

Aiken

L. S.

(2003). Applied multiple regression/correlation analysis for the behavioral sciences. Lawrence Erlbaum.

17.

Collins

Watt

(2021). Using and understanding power in psychological research: A survey study. Collabra: Psychology, 7(1), Article 1. https://doi.org/10.1525/collabra.28250

18.

Cronbach

L. J.

(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555

19.

Cronbach

L. J.

Rajaratnam

Gleser

G. C.

(1963). Theory of generalizability: A liberation of reliability theory. British Journal of Statistical Psychology, 16(2), 137–163. https://doi.org/10.1111/j.2044-8317.1963.tb00206.x

20.

Curran

P. J.

Hussong

A. M.

(2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14(2), 81–100. https://doi.org/10.1037/a0015914

21.

Flake

J. K.

Davidson,

I. J.

Pek

(2020). Is methods moving into practice? The critical role of quantitative training. In Rodgers

J. E.

(Ed.), Teaching statistics and quantitative methods into the 21st century (pp. 70–86). Taylor & Francis.

22.

Flake

J. K.

Davidson

I. J.

Wong

Pek

(2022). Construct validity and the validity of replication studies: A systematic review. American Psychologist, 77, 576–588. https://doi.org/10.1037/amp0001006

23.

Flake

J. K.

Pek

Hehman

(2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8(4), 370–378. https://doi.org/10.1177/1948550617693063

24.

Fried

E. I.

van Borkulo

C. D.

Cramer

A. O.

Boschloo

Schoevers

R. A.

Borsboom

(2017). Mental disorders as networks of problems: A review of recent insights. Social Psychiatry and Psychiatric Epidemiology, 52(1), 1–10. https://doi.org/10.1007/s00127-016-1319-z

25.

GAISE College Report American Statistical Association Revision Committee (2016). Guidelines for assessment and instruction in statistics education college report 2016. http://www.amstat.org/education/gaise

26.

Gelfand

L. A.

Mensinger

J. L.

Tenhave

(2009). Mediation analysis: A retrospective snapshot of practice and more recent directions. The Journal of General Psychology, 136(2), 153–178. https://doi.org/10.3200/GENP.136.2.153-178

27.

Graham

J. W.

(2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60(1), 549–576. https://doi.org/10.1146/annurev.psych.58.110405.085530

28.

Green

(2016, May 12). What are the most-cited publications in the social sciences (according to Google Scholar)? LSE Impact Blog. https://blogs.lse.ac.uk/impactofsocialsciences/2016/05/12/what-are-the-most-cited-publications-in-the-social-sciences-according-to-google-scholar/#author

29.

Greenland

(2012). Nonsignificance plus high power does not imply support for the null over the alternative. Annals of Epidemiology, 22(5), 364–368. https://doi.org/10.1016/j.annepidem.2012.02.007

30.

Hayes

A. F.

(2022). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach (2nd ed.). Guilford Press.

31.

Hoenig

J. M.

Heisey

D. M.

(2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1), 19–24. https://doi.org/10.1198/000313001300339897

32.

Hoisington-Shaw

K. J.

Pek

(2021). Using dynamic graphics to teach the sampling distribution with active learning. The Quantitative Methods for Psychology, 17(2), v1–v9. https://doi.org/10.20982/tqmp.17.2.v001

33.

Hoisington-Shaw

K. J.

Wong

Matthews

Wegener

D. T.

Pek

(under review). The practice of power analysis and its implications: A meta-science review .

34.

Jones

L. V.

Thissen

D. T.

(2007). A history and overview of psychometrics. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics: Psychometrics (vol. 26, pp. 1–27). Elsevier.

35.

Kaiser

H. F.

(1974). An index of factorial simplicity. Psychometrika, 39(1), 31–36. https://doi.org/10.1007/BF02291575

36.

Kraemer

H. C.

Blasey

(2015). How many subjects? Statistical power analysis in research (2nd ed.). Sage Publications.

37.

Lord

F. M.

Novick

M. R.

(1968). Statistical theories of mental test scores. Information Age Publishing.

38.

MacKinnon

D. P.

Fairchild

A. J.

Fritz

M. S.

(2007). Mediation analysis. Annual Review of Psychology, 58(1), 593–614. https://doi.org/10.1146/annurev.psych.58.110405.085542

39.

Maxwell

S. E.

Cole

D. A.

(2007). Bias in cross-sectional analyses of longitudinal mediation. Psychological Methods, 12(1), 23–44. https://doi.org/10.1037/1082-989X.12.1.23

40.

Maxwell

S. E.

Delaney

H. D.

Kelley

(2017). Designing experiments and analyzing data: A model comparison perspective. Routledge.

41.

Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science (New York, N.Y.), 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

42.

Pek

Losardo

Bauer

D. J.

(2011). Confidence intervals for a semiparametric approach to modeling nonlinear relations among latent variables. Structural Equation Modeling: A Multidisciplinary Journal, 18(4), 537–553. https://doi.org/10.1080/10705511.2011.607072

43.

Pek

(2022, October). Uncertainty quantification in power analysis. The Score, the Division 5 (Quantitative and Qualitative Methods) Newsletter of the American Psychological Association, 44(4). https://www.apadivisions.org/division-5/publications/score/2022/10/power-analysis

44.

Podsakoff

P. M.

MacKenzie

S. B.

Podsakoff

N. P.

(2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63(1), 539–569. https://doi.org/10.1146/annurev-psych-120710-100452

45.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods. Sage.

46.

Rodgers

J. L.

(2010). The epistemology of mathematical and statistical modeling: A quiet methodological revolution. American Psychologist, 65, 1–12. https://doi.org/10.1037/a0018326

47.

Schoenfeld

A. H.

(1998). Making mathematics and making pasta: From cookbook procedures to really cooking. In Greeno

J. G.

Goldman

S. V.

(Eds.), Thinking practices in mathematics and science learning (pp. 299–319). Lawrence Erlbaum.

48.

Shaw

Flake

J. K.

(2022). Introduction to multilevel modeling. https://www.learn-mlms.com/

49.

Shrout

P. E.

Fleiss

J. L.

(1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420

50.

Shrout

P. E.

Rodgers

J. L.

(2018). Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis. Annual Review of Psychology, 69(1), 487–510. https://doi.org/10.1146/annurev-psych-122216-011845

51.

Spearman

(1904). 'General intelligence,' objectively determined and measured. The American Journal of Psychology, 15(2), 201–293. https://doi.org/10.2307/1412107

52.

Thurstone

L. L.

(1934). The vectors of mind. Psychological Review, 41(1), 1–32. https://doi.org/10.1037/h0075959

53.

Thurstone

L. L.

(1937). Psychology as a quantitative rational science. Science, 85, 227–232. https://doi.org/10.1126/science.85.2201.227

54.

Tukey

J. W.

(1969). Analyzing data: Sanctification or detective work? American Psychologist, 24(2), 83–91. https://doi.org/10.1037/h0027108

55.

Wild

C. J.

Pfannkuch

(1999). Statistical thinking in empirical enquiry. International Statistical Review, 67(3), 223–248. https://doi.org/10.1111/j.1751-5823.1999.tb00442.x

56.

Wilkinson

, & The Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54(8), 594–604. https://doi.org/10.1037/0003-066X.54.8.594

How Can We Move Advanced Methodology into Practice More Effectively?

Abstract

Keywords

Get full access to this article

References