Introducing Advances in Methods and Practices in Psychological Science

These are exciting times for psychology. The past 7 years have seen a dramatic and fieldwide transformation, with more and more people becoming interested in evaluating and improving their own research practices and those of the field as a whole. Discussions of research practices have gone mainstream, and changes to research and publishing practices are happening faster now than at any point in our field’s recent history. The primary mission of Advances in Methods and Practices in Psychological Science (AMPPS) is to foster such discussions of and advances in practices, research design, and statistical methods.

For decades, experts such as Cohen (1962), Cronbach and Meehl (1955; Meehl, 1967), de Groot (1956/2014), Loevinger (1957), and many others repeatedly raised concerns about small-sample studies, questionable research practices, poor design, noisy measures, violated statistical assumptions, flawed inferences, a lack of direct replication, and publication bias. Although these problems linger, I am more optimistic about the state of our field now than at any earlier point in my career.

Fewer than 10 years ago, nobody had heard the terms p-hacking and researcher degrees of freedom (Simmons, Nelson, & Simonsohn, 2011), and few knew the problems with “HARKing” (Kerr, 1998). ¹ Preregistration was rare outside of clinical trials, stand-alone direct replications were barely publishable, and multilab collaborations were uncommon. Badges and incentives for open practices were nonexistent. Facebook groups were not actively discussing research methods and practices. The Transparency and Openness Promotion (TOP) Guidelines for publishing, spearheaded by the Center for Open Science (n.d.-b) and now adopted by more than 5,000 journals and organizations (including APS), had not yet been conceived. Few journals, funders, or societies had policies that promoted data sharing. Novel article formats such as Registered Reports—in which reviewers evaluate a study’s rigor and design before data collection (Chambers, 2013; for more information, see Center for Open Science, n.d.-a)—were not yet among our publishing options.

In many ways, the Association for Psychological Science (APS) has been a leader in supporting improved research and reporting practices. With Bobbie Spellman as Editor, Perspectives on Psychological Science published a series of groundbreaking articles on research practices, and Associate Editor Alison Ledgerwood organized several special sections on research methods and metascience. Perspectives also launched Registered Replication Reports as a new way to evaluate the strength of evidence for important effects (Simons, Holcombe, & Spellman, 2014; AMPPS will be the new home for these reports). At Psychological Science, Eric Eich implemented changes in reporting practices to allow more comprehensive Method and Results sections and more transparent and complete reporting, and he incentivized authors to be transparent by awarding badges for open data, open materials, and preregistration. His successor, Steve Lindsay, has continued that tradition by adding consulting statisticians to the journal’s editing team, asking authors to make their data and materials accessible to the editors and reviewers, and requesting that authors report on their use (or nonuse) of open-science practices (https://www.psychologicalscience.org/publications/psychological_science/ps-submissions#OPS). Steve Lindsay also adopted a variant of the Pottery Barn rule (Srivastava, 2012) by creating an article format for replications of studies published in Psychological Science (Lindsay, 2017). As editor of Clinical Psychological Science, Scott Lilienfeld also adopted badges and reporting standards that provide incentives for best practices.

The APS Observer magazine publishes a yearly methods issue along with articles and tutorials on a wide range of methodological and statistical topics (e.g., Bayesian analysis, sample-size planning, the “new statistics,” R programming, and preregistration). And the annual APS convention includes a methodology track featuring presentations about research practices and practical, hands-on workshops intended to help psychological scientists improve their research. Those sessions have consistently drawn large crowds, especially early-career researchers.

In launching AMPPS, APS hopes to reach a broad audience, consolidating in a single outlet a range of novel approaches to experimentation (e.g., the Registered Replication Reports), articles on metascience and best practices, and tutorials on research methods and practices. As do all the APS journals, AMPPS emphasizes both innovation and accessible communication, and has a mandate to help researchers from across psychological science to improve the quality of their research and the rigor of our discipline.

The Audience for AMPPS

Improved research practices require clear channels of communication between statisticians and methodologists, on the one hand, and psychology researchers, on the other (Sharpe, 2013). Reaching the broad audience of researchers who want to improve their methods and skills is core to the mission of AMPPS.

Although AMPPS has “methods” in its title, it is not a traditional methods or statistics journal. Several excellent journals focusing on methods and statistics in psychology regularly publish state-of-the-art developments, but most target a readership of expert methodologists and statisticians; they speak to methodologists interested in research, not researchers interested in methods, or researchers interested in research. In recent years, some have pushed for improved accessibility in order to reach a broader audience (Harlow, 2017). AMPPS makes broad access core to its mission. The primary audience for AMPPS is the broad spectrum of psychological scientists who are interested in learning more about methods and practices but who do not regularly read methods journals. Unlike other methods-focused journals, AMPPS will not publish articles written exclusively for methods experts. Articles in AMPPS will convey important advances, but will be written for research producers and consumers; it is a place to communicate innovative methods and to discuss practices in a way that is broadly understandable.

To ensure accessibility of the prose, authors should write the main text of submitted manuscripts in plain English, with all terms defined and explained. The prose should draw in researchers, helping them to understand core issues of relevance to them. AMPPS balances this need for accessibility with the importance of precision by encouraging the use of in-detail boxes where authors can convey the more technical content and equations necessary for a full understanding. These boxes are ideal for content that is not strictly necessary to understand the conceptual point of an article but that adds to a deeper understanding (e.g., glossaries of technical terms, worked case examples, derivations, proofs). Readers who choose to skip the in-detail boxes should be able to understand the main ideas in any article in AMPPS. The main text of each article should be a gateway to greater understanding—it should get a broad audience hooked and encourage them to learn more.

Types of Articles

The submission guidelines for AMPPS (https://www.psychologicalscience.org/publications/ampps/ampps-submission-guidelines) include details about the types of articles and their required formatting. As of its launch, AMPPS accepts three main article types: general articles on research practices, empirical articles featuring innovative research methods and practices, and tutorials describing the how-tos of a research method or practice. On occasion, it will also feature special collections of invited articles that discuss and debate issues of broad interest in the field. For example, the first issue includes a collection of articles on making data as available as possible, focusing especially on cases in which making data publicly available is challenging for practical or ethical reasons. The second issue will contain a forum with practical and philosophical guidance on how to provide evidence against the presence of a meaningful effect.

General, nonempirical articles in AMPPS can address a wide variety of topics, including research practices and metascience; they can present simulation studies, reinterpret earlier findings using new analytic approaches, evaluate and compare different practices, and so on. All should consider the practical importance of the issues for the practices of researchers across psychology. General articles may also form structured debates or collections of articles on a theme, or provide other more interactive content intended to convey different perspectives on a problem.

Empirical articles in AMPPS differ in scope and structure from those appearing in Psychological Science and Clinical Psychological Science. AMPPS will not publish single-lab empirical reports that have a natural home at other APS journals (except, perhaps, when the focus is entirely on a methodological issue that is of broad interest). Empirical articles appropriate for AMPPS should report on studies that adopted novel approaches to research, often large-scale, multilab collaborations: consortium studies, adversarial collaborations, ManyLabs projects, Registered Replication Reports, and so on.

Empirical research published in AMPPS typically will have been preregistered. Note that preregistration does not preclude a complete and careful evaluation of the data and evidence; exploration is the engine of discovery and the source of new hypotheses even if it does not support confirmatory hypothesis tests (see Lindsay, Simons, & Lilienfeld, 2016). Authors of empirical articles should make all materials, code, and de-identified data as publicly available as possible. Some multilab empirical projects will follow the Registered Report process, being reviewed prior to data collection and provisionally accepted in advance of the outcome being known (https://www.psychologicalscience.org/publications/ampps/ampps-submission-guidelines#RR).

Tutorials are the most practical of the articles appearing in AMPPS. Some will provide an introductory overview of an important concept, and others will introduce new tools and techniques. Tutorials will provide concrete guidance to researchers, allowing them to acquire new skills and better use existing ones. Like the other articles in AMPPS, tutorials need not focus exclusively on statistics and methods; they can also discuss broader issues such as lab management and other practical issues that affect the field. Tutorials on practical techniques should be written with an eye toward adoption in research-methods and statistics courses, and they should indicate any prerequisite skills or knowledge necessary to make use of them. They must cover topics that would be useful in many areas of psychology and not only to specialists within a particular subfield.

Standards for the Peer-Review Process

The review process at AMPPS is modeled after the process used at Psychological Science. Each article is initially reviewed by the Editor in Chief and one or more Associate Editors to evaluate whether it is a fit for AMPPS on the basis of adherence to four core principles:

• Accessibility: Articles should be accessible to and understandable by nonexperts. Authors should aim to make their articles understandable to a first-year graduate student in psychology who has taken one or two introductory statistics courses.

• Relevance: Articles should convey why the contents are important to the field as a whole and not just to a small subset of the field. A core goal of AMPPS is to bridge subfields of psychology by communicating useful approaches developed in one area to the field as a whole. The ideal article will address both principles and practices using concrete examples that will be interesting to psychologists in any subfield.

• Rigor: Articles in AMPPS should adhere to and document their use of best practices in research methodology, statistics, and reporting.

• Transparency: Articles should adhere to principles of open science and transparency, both illustrating best practices and informing readers about them.

Articles that clear this editorial review stage will be sent for external review, and those that do not will be declined (i.e., “desk rejected”). In some cases, when the editors feel that a submitted manuscript could be revised to meet these core principles (e.g., if it could be rewritten to be more accessible to the journal’s audience), they may encourage a revision prior to external review. Once a manuscript proceeds to external review, the process is similar to that of other journals.

Although AMPPS does not have strict page limits for articles, the submission guidelines indicate the suggested length for each article type, and authors should contact me prior to submitting a manuscript with a word count exceeding those guidelines. Authors should keep introductory material focused on the specific issue addressed in the article, homing in on the key point quickly and concisely. For example, unless an article is about the reproducibility crisis or is a historical review of closely related issues, it should not cover the crisis as background or motivation.

Concluding Thoughts

Twenty-five years ago, in an introductory graduate statistics course he cotaught with Don Rubin, Bob Rosenthal spoke of the importance of thinking in terms of real-world consequences and effect sizes rather than p values. He highlighted the dangers of treating p < .05 as a magic threshold, the need for quantitative synthesis, and the ways that practices such as optional stopping undermine inference. His admonitions about questionable practices and recommendations for improved ones made a lasting impression on me, but one bit of advice stuck with me more than any other: He told us that, as researchers familiar with such best practices, we would occasionally have to educate journal editors who might have misconceptions.

The field of psychology is catching up to Bob and the many other luminaries who have promoted improved practices over the past 60 years. As the field debates best practices and develops new tools to test our intuitions and to improve research methods and statistics, I hope that AMPPS will help researchers across psychology better their own methods and research skills. I look forward to learning from the many authors and reviewers who will contribute to AMPPS.

—Daniel J. Simons