Sage Journals: Discover world-class research

Abstract

Research synthesis is based on the assumption that when the same association between constructs is observed repeatedly in a field, the relationship is probably real, even if its exact magnitude can be debated. Yet the probability that the relationship is real is a function not only of recurring results, but also of the quality and consistency of the empirical procedures that produced those results and that any meta-analysis necessarily inherits. Standardized protocols in data collection, analysis, and interpretation are foundations of empiricism and a healthy sign of a discipline’s maturity. I propose that meta-analysis as typically applied in psychology will benefit from complementing aggregation of observed effect sizes with systematic examination of the standardization of the methodology that deterministically produced them. I describe potential units of analysis and offer two examples illustrating the benefits of such efforts. Ideally, this synergetic approach will advance theory by improving the quality of meta-analytic inferences.

Keywords

meta-analysis metascience methodology standardization synthesis open data

For the past 8 years, psychologists have undergone a phase of intense self-examination regarding standards and practices in the production, dissemination, and evaluation of knowledge. This critical scrutiny has covered integral aspects of psychological science’s entire enterprise, including the maintenance of psychological theories (Ferguson & Heene, 2012); the procedures by which given hypotheses are confirmed or disconfirmed and by which new research questions are generated and explored (Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012); prevailing norms in the collection (Lakens & Evers, 2014), analysis (Wagenmakers, Wetzels, Borsboom, & van der Maas, 2011), reporting (Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, 2016), and sharing (Wicherts, Bakker, & Molenaar, 2011) of data; the consistency and accuracy of interpretations of observed effect sizes in various subfields (e.g., industrial-organizational psychology and management—Wiernik, Kostal, Wilmot, Dilchert, & Ones, 2017; personality and individual differences—Gignac & Szodorai, 2016; language learning—Plonsky & Oswald, 2014; cognitive neuroscience—Szucs & Ioannidis, 2017); publication practices (Nosek & Bar-Anan, 2012), peer review, and quality management (Wicherts, Kievit, Bakker, & Borsboom, 2012); the reproducibility and replicability of research findings (Open Science Collaboration, 2015); the communication of these findings to the public (Sumner et al., 2014); and the contingent incentive structures under which academics operate (Nosek, Spies, & Motyl, 2012).

Psychological scientists have admitted that they frequently use questionable research practices (John, Loewenstein, & Prelec, 2012) that improve their odds of publishing their research given the discipline’s rigid preference for statistically significant results (Kühberger, Fritz, & Scherndl, 2014). This admission and an increasing number of failures to replicate empirical findings (Hagger et al., 2016; Open Science Collaboration, 2015; Wagenmakers et al., 2016) have worked in concert to erode trust in psychological science. Introductory textbooks, meta-analyses, literature reviews, and policy statements are all built on decades of (purportedly) cumulative research, but how reliable is this knowledge?

Methodological Flexibility and Meta-Analyses: Known Knowns and Known Unknowns

Doubts about the reliability of psychological research have emerged, in particular, with regard to opaque reporting of data-collection rules, flexible use of methods and measurement, and arbitrary analytic and computational procedures. This class of behaviors, which collectively facilitate inflated probabilities of obtaining and subsequently publishing statistically significant results, in concert with the omission of analyses yielding (potentially conflicting) nonsignificant results, is colloquially known as p-hacking (Simonsohn, Nelson, & Simmons, 2014a). We know that these strategies can be quite effective, alone or in conjunction (see, e.g., Schönbrodt, 2016), and we know that psychologists use these strategies frequently (John et al., 2012; LeBel et al., 2013).

Given that p-hacking, by definition, has resulted in relevant information being omitted from the public record, identifying whether questionable research practices have shaped the results in published research reports (and if so, which practices have been involved) is an intricate process. Outcome switching, optional stopping of data collection, and the use of multiple computation techniques for the same target variable are all known types of methodological flexibility, yet which of these behaviors and procedures were appropriated to produce inflated proportions of statistically significant results in any particular case is usually unknown. This problem is exacerbated when individual studies are combined in an attempt to synthesize a body of research (LeBel, McCarthy, Earp, Elson, & Vanpaemel, 2018).

Effective meta-analyses in psychology inform researchers about (a) the mean and variance of underlying population effects, (b) variability in effects across studies, and (c) potential moderator variables (Field & Gillett, 2010). They do so by identifying a relevant population of research reports and selecting those meeting predefined inclusion criteria, such as adherence to methodological gold standards.

As p-hacking and the file-drawer problem (i.e., greater difficulty publishing statistically nonsignificant results than publishing statistically significant results) typically work in conjunction with each other¹, any attempt at (quantitative) research synthesis becomes quite cumbersome. Certainly, there is an abundance of techniques for estimating and correcting bias (Schmidt & Hunter, 2015), such as the test of insufficient variance (Schimmack, 2014), the p-uniform method (van Assen, van Aert, & Wicherts, 2015), the p-curve method (Simonsohn et al., 2014a, 2014b), trim-and-fill-based and funnel-plot-based methods (Duval & Tweedie, 2000), the precision-effect test and the use of precision-effect estimates with standard error (known as PET-PEESE; Stanley & Doucouliagos, 2014), and various selection models (McShane, Böckenholt, & Hansen, 2016). These techniques may reveal that some unknown bias is present in a population of statistical results. Yet when a body of empirical research is meta-analyzed, researchers might have trouble determining the extent to which an obtained effect-size estimate is affected specifically by publication bias, methodological flexibility, or an interaction of the two. Furthermore, although such statistical estimation techniques are helpful in demonstrating that a body of research exhibits signs of bias or questionable research practices, they might be less effective in identifying the specific type of bias or practice a literature is affected by. Two recent simulation studies (Carter, Schönbrodt, Gervais, & Hilgard, 2019; Renkewitz & Keiner, 2018) compared bias-estimation techniques (in several models of publication bias and p-hacking) and found that (a) no method consistently outperformed the others, (b) the methods’ performance was poor under specific assumptions (that may be applicable to psychological research domains), and (c) researchers usually do not have the information required for selecting the best technique in a given case.

Generally, research synthesis (meta-analytic or otherwise) is based on the assumption that when the same association between constructs is observed repeatedly in a field, the relationship is probably real, even if its exact magnitude can be debated (e.g., Ellis, 2010). Yet the probability that the relationship is real is a function not only of recurring patterns of results on the construct level, but also of the quality and consistency of data collection, measurement, transformation of the data, computation, and reporting. Any meta-analysis necessarily inherits the properties of those empirical practices. With increasing diversity, the certainty that methods used in studies purportedly assessing the same constructs actually provide evidence about the same relationship decreases. Without a gold standard of assessment agreed upon by a scientific community, it can be quite difficult to estimate the consequences of inconsistent methodological approaches. And the more the consistency of research outcomes in a given field depends on flexibility in methods and measures, the greater the probability that the real association approaches zero.

Meta-Method Analysis

Meta-analyses might therefore benefit if meta-analysts complement the aggregation of observed effect sizes with systematic examination of the standardization of the methodologies that produced them. Standardized protocols in data collection, analysis, and interpretation are foundations of empirical research and a healthy sign of a discipline’s maturity. Further, agreement on the implementation of scientific procedures (i.e., standardization) and documentation of those procedures in instruction manuals and handbooks (so how is prescribed and the influence of who, where, and when on data quality is reduced) are fundamental to the objectivity of research because they guide scientists and facilitate the independence of psychological experiments or tests from basic boundary conditions.² The extent to which researchers in a given field have been able to agree (either implicitly or explicitly) on methodological prescriptions is key to the degree to which meaningful inferences can be extracted from meta-analytic effect-size estimates.

In this article, I propose a set of steps for evaluating the standardization of methodology in order to facilitate research synthesis. These steps do not constitute a wholly new meta-analytic procedure. Rather, they should be considered a specific type of literature review procedure that is typically overlooked in psychological meta-analyses, which are largely concerned with estimating the true distribution of effect sizes observed in primary research, and less concerned with systematically identifying, coding, and evaluating study methodology. Guidelines for systematic reviews in medicine (Higgins & Green, 2011) and social sciences (The Campbell Collaboration, 2019) require giving substantial attention to critically evaluating the methodologies used in a literature and the limits these place on conclusions, recognizing that searching for and evaluating studies is separate from meta-analytic modeling of study outcomes. Reporting guidelines for systematic reviews, such as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Moher, Liberati, Tetzlaff, & Altman, 2009) and psychology’s Journal Article Reporting Standards (JARS; Appelbaum et al., 2018), demand that meta-analysts describe how they assessed bias in both the design and the outcomes of individual studies, and how this assessment informed the synthesis itself. Therefore, the present article should be understood as an attempt at reintroducing these critical steps into the process of synthesizing psychological research.

Does this mean that the suggestions made here are simply rephrasings of the purpose and procedures of the “systematic review” part of meta-analyses? Although there certainly might be some redundancy when these steps are incorporated into the meta-analysis routine, particularly with regard to the systematic search for and screening of primary studies, it may be valuable to use the term meta-method analysis to refer specifically to the critical evaluation of the designs and analytic choices in a literature. Further, to echo Puljak (2019), methodological studies evaluating evidence are not necessarily systematic reviews. Whereas the purpose of the latter is to identify and pool all available empirical evidence (whose relevance is determined through predefined criteria) to answer a substantive research question, the aim of the former is to explore research practices and answer methodological questions without necessarily collating all empirical studies.

Aims and purpose

Examining the degree of standardization is valuable for contextualizing and evaluating observed meta-analytic effect sizes. In fact, standardization is, to some degree, a prerequisite of any meaningful synthesis of a research literature. It informs researchers about the psychometric objectivity of results, the agreed definitions of constructs and how they should be measured, the existence or lack of prevailing norms regarding what should be considered robust evidence, and the latent meaning of previous observations. Whereas meta-analyses in psychology are typically concerned with the synthesis of outcomes of studies (e.g., effect sizes), meta-method analyses dissect procedural decisions that deterministically lead to study outcomes (see Fig. 1). A side-by-side comparison of procedural characteristics or analytic strategies of studies on the same research question, ideally using the same empirical paradigm, may reveal idiosyncratic flexibility and peculiar deviations of individual studies in areas with consistent disciplinary norms (for an example, see Simonsohn, 2013) or endemic flexibility in areas where there is no standardization at all.

Fig. 1.

The empirical cycle involving iteratively generating or modifying theory, deriving hypotheses, conducting studies, and interpreting their outcomes. Traditionally, meta-analysis of psychological studies has involved synthesizing effect sizes from a set of systematically selected studies. Meta-method analysis informs this synthesis with an examination of the study corpus’s operationalization of constructs, observation and measurement choices, and analytic methodologies.

Units of analysis

Any procedural routine, such as operationalization, data collection, computation, analysis, or interpretation, may be the subject of a systematic meta-method analysis. The extent to which a comprehensive meta-method analysis indicates the robustness and replicability of the body of research and informs the results of a meta-analysis naturally depends on the research question. Whereas, for example, diverse samples and sampling methods might be considered healthy in the literature on predictors of psychological well-being in the general population, substantial constraints on sampling methods would be expected in studies on the effectiveness of psychotherapy. The consistent application of narrowly defined inclusion criteria could even increase trust in the robustness of the psychotherapy literature.

Consequently, and ironically, there may not be a standardized way of conducting a meta-method analysis, as procedures and paradigms differ substantially between research domains, and a given unit of analysis may be relevant to some meta-analytic research questions but not others. Table 1 lists potential units of analysis, citing example publications (when available) reporting procedural examinations that may inform meta-analyses. In the next two sections, I provide detailed examples illustrating the usefulness of such efforts.

Table 1.

A Nonexhaustive List of Potential Units of Analysis for Meta-Method Analyses

• Stimuli
○ Presentation (Wessel, 2018; also see Example 2 in the text)
○ Sequence
• Sampling
○ Statistical power (Schimmack & Chen, 2017)
○ Recruitment
○ Sample characteristics
○ Population characteristics
• Operationalization
○ Selection of items, trials, or subscales (Fiedler, Messner, & Bluemke, 2006; Weidman, Steckler, & Tracy, 2017)
○ Number of items or trials (Credé, Harms, Niehorster, & Gaye-Valentine, 2012)
○ Item formulation (Weinstein, 2018)
○ Computational strategies (Elson, Mohseni, Breuer, Scharkow, & Quandt, 2014; also see Example 1 in the text)
• Selection and filtering of data points
○ Regions of interest
○ Cutoffs and outliers
• Analysis
○ Analytic procedures (Carp, 2012a)
○ Critical parameters
○ Moderator variables (Harris, Chabot, & Mickes, 2013)
• Interpretation
○ Thresholds (Vul, Harris, Winkielman, & Pashler, 2009)
○ Categorization

Note: In addition to systematically assessing these variables in psychological research, it can be illuminating to consider the presence or absence of reporting standards (see, e.g., Carp, 2012b).

Disclosures

All data from Example 1 are available at the Open Science Framework, at https://osf.io/4pdvg. Jan Wessel has given permission to make all data used in Example 2 available at the Open Science Framework, at https://osf.io/xytch.

Example 1: Aggressive Behavior

The causes of aggressive behavior hypothesized by psychologists range from drugs, alcohol, and hot temperatures to ostracism, frustration, and violent media (Sturmey, 2017). One of the most commonly used laboratory measures of aggressive behavior is the competitive reaction time task (CRTT), sometimes also called the Taylor aggression paradigm. In the CRTT, participants play a computerized reaction time game against another participant (actually a preprogrammed algorithm). At the beginning of each round, both players set the intensity (by setting the volume, the duration, or both) of a noise blast, after which they are prompted to react to a stimulus as quickly as possible by pressing a button. The slower player is then punished with a noise blast at the intensity set by the other player at the beginning of the round. The intensity settings are used as the measure of aggressive behavior.

Flexibility of the CRTT

The CRTT ostensibly measures how much unpleasant, or even harmful, noise a participant is willing to administer to a (nonexistent) opponent, but an examination of published studies in which the CRTT was used reveals that researchers have followed many different strategies for extracting aggression scores from the raw data. Using a strict definition of what constituted a distinct strategy, I identified more than 150 quantification strategies³ in 130 studies (Elson, 2016; see also Elson, Mohseni, Breuer, Scharkow, & Quandt, 2014).

Not all of these quantification strategies were substantially different (e.g., average volume over 25 trials vs. average volume over 30 trials), and thus, choosing one over the other would not be expected to change a study’s results dramatically. In other cases, however, the differences were more substantial (e.g., log of the product of volume and duration on the first trial vs. highest volume across 30 trials), and only occasionally did the researchers justify their specific choice in their publication.

Most of the quantification strategies (more than 100) were used just once, in a single research report, and never reoccurred in the literature; 3 strategies were used in more than 10 publications. Similarly, many studies reportedly relied on 1 or 2 quantification strategies, although others included up to 10. One potential explanation of these differences across studies is that because there is no documentation of a standardized CRTT procedure, researchers run the task with their own idiosyncratic choices. However, variability was also observed within—not just between—individuals and labs. In fact, a relatively small number of researchers generated the majority of the quantification strategies over time. A further cause for concern is that the number of CRTT studies and the number of different quantification strategies appear to be increasing almost linearly with time (see Fig. 2).

Fig. 2.

Accumulated number of published studies using the competitive reaction time task and of strategies for calculating aggression scores on this task (Elson 2016). Studies were identified through a Google Scholar search for the term “competitive reaction time task” and snowball sampling from reference lists.

Although arguably relevant, these considerations have hardly informed the meta-analyses that have included laboratory aggression studies. For example, in their meta-analysis on the link between playing violent video games and aggression, Anderson et al. (2010) highlighted the CRTT’s prominence and validity as a behavioral measure. However, only a few, if any, of the CRTT studies Anderson et al. included in their analyses were identical with regard to their reported computational operationalization of aggression (see Table 2).

Table 2.

Summary of the Implementation of the Competitive Reaction Time Task in the English-Language Studies Included in Anderson et al.’s (2010) Meta-Analysis

Study	Volume settings^a	Duration settings^a	Number of quantification strategies	Reported quantification strategies
Anderson and Carnagey (2009), Experiment 3	✓		2	Mean volume setting, number of high-volume settings^b
Anderson and Dill (2000), Experiment 2	✓	✓	4	Mean log-transformed duration setting in trials after winning and trials after losing, mean volume setting in trials after winning and trials after losing
Anderson, Gentile, and Buckley (2007), Experiment 1	✓		1	$\sqrt{Number of high - volume {settings}^{b}}$
Anderson and Murphy (2003)	✓		1	Number of high-volume settings^b
Anderson et al. (2004), Experiment 2	✓		4	Volume setting in Trial 1, mean volume setting in 3 trial blocks (Trials 2–9, 10–17, and 18–25)
Anderson et al. (2004), Experiment 3	✓		1	Mean volume setting
Arriaga, Esteves, Caneiro, and Monteiro (2008)	✓	✓	1	Mean of (volume setting + duration setting)/2
Bartholow and Anderson (2002), Experiment 1	✓		2	Mean volume setting, number of high-volume settings^b
Bartholow, Sestir, and Davis (2005), Experiment 2	✓	✓	2	Mean Volume Setting × Mean Duration Setting, Volume Setting × Duration Setting in Trial 1
Carnagey and Anderson (2005), Experiment 3	✓	✓	1	Mean across trials of Volume Setting × $\sqrt{Duration Setting}$
Ferguson, Rueda, Cruz, Ferguson, and Fritz (2008), Experiment 1	✓		1	Mean volume setting
Konijn, Nije Bijvank, and Bushman (2007)	✓		2	Volume setting in Trial 1, volume setting in Trials 2–24

Note: I was unable to identify the quantification strategies reported in the Japanese-language publications included in Anderson et al.’s (2010) analysis because of my lack of proficiency in Japanese.

Checkmarks in these columns indicate that the volume or duration of noise was reportedly set by participants. ^bSettings of 8, 9, or 10 on a 10-point scale were considered to be high-volume settings.

Implications for meta-analyses

Given that little is known about the validity of any reported CRTT quantification strategy, it is debatable whether a meaningful meta-analytic synthesis of this literature can actually be achieved. There might simply be too much uncertainty regarding both the convergence of the computational strategies on the underlying construct and the amount of bias introduced by this apparent flexibility. This observation, of course, has substantial implications for the informational value of meta-analyses in the domain of aggression research, where versions of the CRTT are routinely used (e.g., Anderson et al., 2010; Denson, O’Dean, Blake, & Beames, 2018).

Arguably, the magnitude of the problem corresponds to the degree to which the multiple CRTT quantification strategies fail to converge. Switching between or omitting quantification strategies would not affect the outcomes of statistical analyses (or inferences from them) if the strategies were all perfectly correlated. However, if that were the case, it would be even more puzzling why so many different computational strategies have been documented in the literature. Although publication bias typically occurs through selective reporting of studies with statistically significant results, it is also possible that it results from selective reporting (and omission) of quantification strategies, particularly if different quantification strategies applied to the same data do not produce converging results. In a recent study, Hyatt, Chester, Zeichner, and Miller (2019) examined bivariate correlations of various personality traits with 10 different quantifications of the same CRTT data and found directionally consistent relations and moderate heterogeneity in effect sizes. Further validation studies will be required to enable aggression researchers to make sense of the available literature.

Sharing of trial-level CRTT data may ensure that studies gain new utility when the research community finally agrees on some standardized approach. I recommend that researchers archive their CRTT data on public repositories such as the Open Science Framework (www.osf.io) so that meta-analysts can perform checks on the data to identify the most appropriate meta-analytic modeling independently. Furthermore, the extracted computational operationalizations may inspire multiverse analyses (Steegen, Tuerlinckx, Gelman, & Vanpaemel, 2016) of CRTT studies.⁴

Example 2: Inhibitory Control

Whereas Example 1 focused on a lack of standardization in analytic strategies, this second example focuses on procedural differences in data collection that pose a potential concern for measurement validity.

Inhibitory control enables individuals to rapidly cancel motor activity even after its initiation. It is a cognitive process and an important human capacity that permits an individual to inhibit impulses or dominant behavioral responses to stimuli in order to select other (ideally more appropriate) responses. Inhibitory control is involved in everyday decision making, such as stopping while crossing the street when a car approaches or suppressing the urge to eat a high-calorie snack when one is trying to lose weight.

One of the most commonly used laboratory paradigms for studying inhibitory control is the go/no-go task. This task consists of a number of trials on which the stimulus presented indicates either “go” (i.e., the participant should respond, e.g., by pressing a button) or “no go” (i.e., the participant should do nothing). Response accuracy on no-go trials is used as a measure of inhibitory control. The most critical component in designing this task is to ensure that prepotent motor activity is initially elicited on each trial, so that no-go trials truly test inhibitory control. To achieve this, two parameters of the experimental design are important: First, no-go trials should be less frequent than go trials, so that it is strategically beneficial to initiate a go response on every trial. Second, trials should be presented at a rapid pace, so that responses need to be made quickly.

Flexibility of the go/no-go task

Wessel (2018) conducted a meta-method analysis of these two well-known requirements in 241 published studies using the go/no-go task and found that the percentage of no-go trials ranged from 6 to 60% and the maximum trial duration ranged from 800 to 17,500 ms (see Fig. 3). Go and no-go trials were equiprobable in about 40% of the go/no-go experiments. Additionally, although stimuli were presented at a relatively rapid pace in most of the studies, almost 20% used maximum trial durations greater than 4,000 ms, which allowed a deliberate response to the stimulus instead of a prepotent response or its immediate inhibition. Most of the studies did not meet the empirically validated recommendations for (a) fewer than 20% no-go trials and (b) varying trial durations with a maximum of less than 1,500 ms (Elson, 2017; Wessel, 2018).

Fig. 3.

Scatterplot showing the percentage of no-go trials as a function of the maximum trial duration in the 241 go/no-go studies identified by Wessel (2018). Each circle represents one study. Slight jitter has been added to increase visibility.

Implications for meta-analyses

Studies using the go/no-go task need to be carefully designed to ensure that inhibitory control is actually required to perform correctly on no-go trials. Otherwise, any differential effects of no-go trials cannot be interpreted as effects of inhibitory-control activity. It is unclear whether slow-paced or equiprobable go/no-go tasks elicit prepotent motor activity—and hence, whether they are valid measures of inhibitory control. Consequently, go/no-go studies with widely different parameters may not assess a consistent, underlying effect or test the same theoretical prediction coherently, as only a subset of the studies may have been designed appropriately to capture inhibitory control. If a meta-analysis were conducted without using a meta-method analysis to select studies for inclusion, the best-case scenario would be that the precision of the meta-analytic effect-size estimates would be reduced to some extent; in the worst case, the estimates would not be meaningfully interpretable at all because the included studies actually operationalized different constructs.

Conducting a Meta-Method Analysis

Psychologists seeking to inform a meta-analysis by means of a meta-method analysis must ask themselves which of the potentially infinite number of units of analysis their meta-method analysis should focus on. The principled, yet somewhat unsatisfying, answer is that this focus should be consequentially determined by the meta-analytic research question. Several taxonomies could be employed to guide this decision making, and one useful approach involves framing the meta-method analysis on the continuum from confirmation to exploration.

Planning a meta-method analysis

The choice of the unit of analysis is perhaps most obvious when the meta-analysis concerns a body of work in which empirical procedures are prescribed, either through highly documented assessment batteries (e.g., intelligence tests) or, ideally, through methodological specifications derived from formalized theories (which are, admittedly, currently quite rare in psychological science). In such a case, researchers may conduct a meta-method analysis to verify the (reported) conformity to those prescriptions and to determine whether any deviations occurred systematically.

In the absence of such clear norms, researchers may adopt a more pragmatic approach and conduct a meta-method analysis with two questions in mind. First, which empirical procedures should be standardized across studies in order to allow a meaningful meta-analytic inference? If the target literature shows a high degree of standardization of those procedures, this would indicate agreement on best practices, whereas a low degree of standardization might raise doubts about the replicability, generalizability, and synthesizability of observations. A meta-method analysis based on this question would provide researchers with information concerning whether a meta-analysis could yield meaningful conclusions and would also lay the foundation for establishing gold standards. Second, which empirical procedures might be prone to abuse of degrees of freedom to produce consistent results? Researchers may already suspect that flexibility is more common in some research practices than in others (e.g., because of the community’s perception of what is acceptable) and may systematically search for confirmatory evidence of that flexibility. A meta-method analysis may instead uncover a high degree of consistency in those procedures, which would be a sign of transparency and reliability, or it may reveal the expected low degree of consistency, which could indicate that the body of literature is affected by questionable research practices.

Finally, researchers may conduct a meta-method analysis on specific procedures out of genuine curiosity. The goal may not be to confirm that studies conform to prescriptions, but rather to assess the degree to which implicit agreement or field-specific traditions have already been established and shape studies’ methodologies.

General implications for meta-analyses

Whether meta-method analysis is used as a quantitative or qualitative technique for research synthesis depends on the meta-analytic research question it is applied to. However, there are important considerations to be taken into account if the plan is to incorporate meta-method-analytic results into meta-analytic statistics (e.g., in a moderator analysis): First, the greater the observed procedural differences across individual studies, the fewer studies per procedure are available, and consequently the lower the statistical power of a moderator test. Second, studies that have used different procedures may not assess the same variables and relationships between them, and therefore cannot be meaningfully included in the same analysis, even when the intent is to quantify the effect of methodological flexibility.⁵

Conversely, a meta-method analysis may reveal psychometric artifacts that can be controlled for statistically with appropriate meta-analytic correction techniques. For example, range restriction (Sackett & Yang, 2000) and dichotomization of continuous variables (Hunter & Schmidt, 1990) have predictable effects on study results, and a psychometric meta-analysis can adjust for the variability they introduce in meta-analytic effect-size estimates (Schmidt & Hunter, 2015). Thus, depending on the specific findings of a meta-method analysis and the availability of appropriate adjustment techniques, it may be neither efficient nor appropriate to exclude studies from a meta-analysis, to use ratings of studies’ “methodological rigor” as a moderator in a meta-analysis, or to conclude that a meta-analysis cannot be conducted at all because of inconsistent methodologies. Researchers need to distinguish methodological artifacts that are correctable (accounting for all assumptions and limitations of the correction techniques themselves) and those that may be disqualifying factors.

The most important implication of meta-method analyses is that they provide context required to meaningfully interpret observed meta-analytic effect-size estimates. In psychology, meta-analysis is occasionally treated rather uncritically as the ultimate tool to produce final answers to a research question (Schmidt & Oh, 2016). For example, Huesmann (2010) argued that Anderson et al.’s (2010) meta-analysis (see Example 1, earlier in this article), although based partially on the documented, inconsistent use of a primary measure, “[nailed] the coffin shut on doubts that violent video games stimulate aggression” (p. 179). A meta-method analysis could inform that interpretation of the consistency of study outcomes and—given the context of unexplained flexibility in data collection—might suggest that the coffin lid is actually still somewhat ajar.

Meta-method analyses could also improve the usefulness of meta-analyses in nascent fields where the number of studies is probably too small for a meaningful collation of the available evidence. Instead of attempting to (prematurely) obtain conclusive answers to research questions, researchers could systematically examine methodological standards and norms so as to cautiously “take stock” of the available knowledge and its ability to provide meaningful conclusions. Such analyses could be done on a small number of pioneer studies, steering fields in productive directions before deficient empirical practices proliferate.

Beyond standardization

I have framed meta-method analysis somewhat narrowly as an approach to document and evaluate standardization, which is critical for maximizing psychometric objectivity, across studies. However, it is entirely conceivable that a measure is used consistently in a literature even though there is little available evidence for its construct or criterion validity, or that common design choices (e.g., a relative overabundance of simple cross-sectional studies) limit the meta-analytic conclusions that can be drawn about causality and the robustness of the relationship between variables. Certainly, in such cases, a meta-method analysis concerned solely with issues of standardization would yield different—that is to say, possibly less relevant—conclusions than would one with broader considerations.

There is no reason in principle why design and analytic choices in a literature cannot also be systematically, critically assessed with regard to their implications for psychometric reliability or validity (and for the validity of meta-analytic conclusions). The scope of this article is not meant to discourage researchers from extending the general concept of meta-method analysis to higher-level psychometric properties beyond consistency. Rather, my intent is to emphasize the importance of psychometric objectivity as a criterion of quality and a deterministic prerequisite to reliability and validity within classical test theory. Thus, this article is meant to (re)introduce the basic concept of meta-method analysis to psychological research synthesis while reaffirming the foundational value of psychometric objectivity specifically, and the importance of standardized protocols for meaningful cumulative research more generally.

General Implications for Primary Research

Both the conceptualization and the outcomes of meta-method analyses have potential benefits for empirical studies. A relatively obvious one is that they can offer systematic documentation of proven and tested empirical procedures and instructions for implementation in future studies (e.g., stimulus materials to use and to avoid, a decision tree for analytic strategies). This documentation could, in turn, cumulatively increase the informational value of research synthesis.

As much as meta-method analyses benefit from clear, detailed descriptions of empirical studies, they may also shape standards for transparent reporting as researchers seek to facilitate their studies’ accessibility. Current reporting standards, such as the American Psychological Association’s JARS (Appelbaum et al., 2018), already demand that procedures be described in great detail. However, researchers (e.g., those outside the United States) may not be aware of these guidelines, or if they are, they may find it difficult to apply broad recommendations to their specific research areas. Surely, no researchers believe they are doing a bad job describing their study protocols, but they may disagree about which information is relevant or should be made available, and where emphasis should be put. Although JARS specifically mentions the importance of transparent reporting so that scientific claims can be understood and evaluated, it may not be obvious which information is critical to meaningful research synthesis specifically.

Further, journals do not routinely enforce compliance with these guidelines, nor is compliance systematically assessed. Thus, meta-method analyses may, at least in part, document the extent to which reporting practices in a given field already correspond to these standards, and point researchers to areas where they might be improved.

Conclusion

Meta-analysis is an accessible, systematic way of synthesizing and quantifying a body of research reports to answer overarching research questions. The effectiveness of meta-analyses is constrained by a range of challenges, such as biases (e.g., publication bias), methodological issues (e.g., appropriateness of search and selection procedures), and metamethodological concerns (e.g., discipline-specific norms for empirical work). As meta-analyses necessarily inherit the quality of the included evidence, studies must meet a number of basic requirements in order to allow meaningful meta-analytic inferences. Empirical research benefits not only from agreement on terminology and fundamental theoretical constructs, but also from rigorous best practices and gold standards. Fields in which researchers have not yet been able to agree on and develop useful procedural protocols might simply not be mature enough for a sophisticated synthesis. Although tradition might be a weak form of protocol, it can still offer guidance. When there is no protocol for the procedures that many people are using, though, to what extent can individual studies be compared, combined, synthesized, and meta-analyzed? Researchers forming a community of practice need to figure out such protocols collectively before a meta-analysis is potentially used to present research results as more conclusive than the underlying research standards could possibly allow and to perpetuate empirical practice that may be detrimental to meaningful research synthesis and truth seeking more generally.

Conversely, the robustness and trustworthiness of meta-analyses in research domains in which emphasis is put on standardization and documentation would be further corroborated by meta-method analyses. Evidence of rigor across studies may increase confidence in the replicability of individual studies and the precision of meta-analytic effect-size estimates, offer guidance to researchers entering a field as they look for tried-and-tested procedures, and inspire systematic methodological research.

This article has laid out an approach to research synthesis that can contextualize meta-analytic effect-size estimates conventionally reported in psychological research syntheses: systematic examination of reported procedures that deterministically generate empirical outcomes. This approach is conservative in nature because it relies on what has been reported and does not involve inferring hidden causes (e.g., an inaccessible file drawer of studies) or biases occurring at a conceptual stage (Fiedler, 2011), which may not even be considered researcher degrees of freedom. This synergetic approach emphasizes the role of methods in advancing theory (Greenwald, 2012) by improving the quality of meta-analytic inferences. And as the quality of meta-analyses increases through these techniques, so will empirical standards in individual studies.

Supplemental Material

Elson_Rev_Open_Practices_Disclosure – Supplemental material for Examining Psychological Science Through Systematic Meta-Method Analysis: A Call for Research

Supplemental material, Elson_Rev_Open_Practices_Disclosure for Examining Psychological Science Through Systematic Meta-Method Analysis: A Call for Research by Malte Elson in Advances in Methods and Practices in Psychological Science

Footnotes

Acknowledgements

I sincerely thank Brenton Wiernik for his very thoughtful signed review of this manuscript, which substantially improved it. I also thank Nick Brown, Julia Eberle, James Ivory, Richard Morey, Julia Rohrer, and Tim van der Zee for comments on earlier versions of this manuscript.

Action Editor

Frederick L. Oswald served as action editor for this article.

Author Contributions

M. Elson is the sole author of this article and is responsible for its content.

ORCID iD

Malte Elson

Declaration of Conflicting Interests

The author(s) declared that there were no conflicts of interest with respect to the authorship or the publication of this article.

Funding

This research is supported by the Digital Society research program funded by the Ministry of Culture and Science of North Rhine-Westphalia, Germany.

Open Practices

Open Data: https://osf.io/4pdvg,

Open Materials: not applicable

Preregistration: not applicable

All data have been made publicly available via the Open Science Framework and can be accessed at https://osf.io/4pdvg (Example 1) and https://osf.io/xytch (Example 2). The complete Open Practices Disclosure for this article can be found at http://journals.sagepub.com/doi/suppl/10.1177/2515245919863296. This article has received the badge for Open Data. More information about the Open Practices badges can be found at .

Notes

References

Anderson

C. A.

Carnagey

N. L.

(2009). Causal effects of violent sports video games on aggression: Is it competitiveness or violent content? Journal of Experimental Social Psychology, 45, 731–739. doi:10.1016/j.jesp.2009.04.019

Anderson

C. A.

Carnagey

N. L.

Flanagan

Benjamin

A. J.

Jr. Eubanks

Valentine

J. C.

(2004). Violent video games: Specific effects of violent content on aggressive thoughts and behavior. In Zanna

M. P.

(Ed.), Advances in experimental social psychology (Vol. 36, pp. 199–249). New York, NY: Elsevier. doi:10.1016/S0065-2601(04)36004-1

Anderson

C. A.

Dill

K. E.

(2000). Video games and aggressive thoughts, feelings, and behavior in the laboratory and in life. Journal of Personality and Social Psychology, 78, 772–790. doi:10.1037/0022-3514.78.4.772

Anderson

C. A.

Gentile

D. A.

Buckley

K. E.

(2007). Violent video game effects on children and adolescents: Theory, research, and public policy. New York, NY: Oxford University Press.

Anderson

C. A.

Murphy

C. R.

(2003). Violent video games and aggressive behavior in young women. Aggressive Behavior, 29, 423–429. doi:10.1002/ab.10042

Anderson

C. A.

Shibuya

Ihori

Swing

E. L.

Bushman

B. J.

Sakamoto

. . . Saleem

(2010). Violent video game effects on aggression, empathy, and prosocial behavior in Eastern and Western countries: A meta-analytic review. Psychological Bulletin, 136, 151–173. doi:10.1037/a0018251

Appelbaum

Cooper

Kline

R. B.

Mayo-Wilson

Nezu

A. M.

Rao

S. M.

(2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73, 3–25. doi:10.1037/amp0000191

Arriaga

Esteves

Caneiro

Monteiro

M. B.

(2008). Are the effects of Unreal violent video games pronounced when playing with a virtual reality system? Aggressive Behavior, 34, 521–538. doi:10.1002/ab.20272

Bartholow

B. D.

Anderson

C. A.

(2002). Effects of violent video games on aggressive behavior: Potential sex differences. Journal of Experimental Social Psychology, 38, 283–290. doi:10.1006/jesp.2001.1502

10.

Bartholow

B. D.

Sestir

M. A.

Davis

E. B.

(2005). Correlates and consequences of exposure to video game violence: Hostile personality, empathy, and aggressive behavior. Personality and Social Psychology Bulletin, 31, 1573–1586. doi:10.1177/0146167205277205

11.

The Campbell Collaboration. (2019). Campbell systematic reviews: Policies and guidelines. Oslo, Norway. doi:10.4073/cpg.2016.1

12.

Carnagey

N. L.

Anderson

C. A.

(2005). The effects of reward and punishment in violent video games on aggressive affect, cognition, and behavior. Psychological Science, 16, 882–889. doi:10.1111/j.1467-9280.2005.01632.x

13.

Carp

(2012a). On the plurality of (methodological) worlds: Estimating the analytic flexibility of fMRI experiments. Frontiers in Neuroscience, 6, Article 149. doi:10.3389/fnins.2012.00149

14.

Carp

(2012b). The secret lives of experiments: Methods reporting in the fMRI literature. NeuroImage, 63, 289–300. doi:10.1016/j.neuroimage.2012.07.004

15.

Carter

E. C.

Schönbrodt

F. D.

Gervais

W. M.

Hilgard

(2019). Correcting for bias in psychology: A comparison of meta-analytic methods. Advances in Methods and Practices in Psychological Science, 2, 115–144.

16.

Credé

Harms

Niehorster

Gaye-Valentine

(2012). An evaluation of the consequences of using short measures of the Big Five personality traits. Journal of Personality and Social Psychology, 102, 874–888. doi:10.1037/a0027403

17.

Denson

T. F.

O’Dean

S. M.

Blake

K. R.

Beames

J. R.

(2018). Aggression in women: Behavior, brain and hormones. Frontiers in Behavioral Neuroscience, 12, Article 81. doi:10.3389/fnbeh.2018.00081

18.

Duval

Tweedie

(2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463. doi:10.1111/j.0006-341X.2000.00455.x

19.

Ellis

P. D.

(2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge, England: Cambridge University Press.

20.

Elson

(2016). Competitive Reaction Time Task. Retrieved from the FlexibleMeasures.com website: http://www.flexiblemeasures.com/crtt/

21.

Elson

(2017). Go/No-Go task. Retrieved from the FlexibleMeasures.com website: http://www.flexiblemeasures.com/nogo/

22.

Elson

Mohseni

M. R.

Breuer

Scharkow

Quandt

(2014). Press CRTT to measure aggressive behavior: The unstandardized use of the Competitive Reaction Time Task in aggression research. Psychological Assessment, 26, 419–432. doi:10.1037/a0035569

23.

Ferguson

C. J.

Heene

(2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7, 555–561. doi:10.1177/1745691612459059

24.

Ferguson

C. J.

Rueda

S. M.

Cruz

A. M.

Ferguson

D. E.

Fritz

(2008). Violent video games and aggression: Causal relationship or byproduct of family violence and intrinsic violence motivation? Criminal Justice and Behavior, 35, 311–332. doi:10.1177/0093854807311719

25.

Fiedler

(2011). Voodoo correlations are everywhere—Not only in neuroscience. Perspectives on Psychological Science, 6, 163–171. doi:10.1177/1745691611400237

26.

Fiedler

Messner

Bluemke

(2006). Unresolved problems with the “I”, the “A”, and the “T”: A logical and psychometric critique of the Implicit Association Test (IAT). European Review of Social Psychology, 17, 74–147. doi:10.1080/10463280600681248

27.

Field

Gillett

(2010). How to do a meta-analysis. The British Journal of Mathematical and Statistical Psychology, 63, 665–694. doi:10.1348/000711010X502733

28.

Gignac

G. E.

Szodorai

E. T.

(2016). Effect size guidelines for individual differences researchers. Personality and Individual Differences, 102, 74–78. doi:10.1016/j.paid.2016.06.069

29.

Greenwald

A. G.

(2012). There is nothing so theoretical as a good method. Perspectives on Psychological Science, 7, 99–108. doi:10.1177/1745691611434210

30.

Hagger

M. S.

Chatzisarantis

N. L. D.

Alberts

Anggono

C. O.

Batailler

Birt

A. R.

. . . Zwienenberg

(2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11, 546–573. doi:10.1177/1745691616652873

31.

Harris

C. R.

Chabot

Mickes

(2013). Shifts in methodology and theory in menstrual cycle research on attraction. Sex Roles, 69, 525–535. doi:10.1007/s11199-013-0302-3

32.

Higgins

J. P. T.

Green

(Eds.). (2011). Cochrane handbook for systematic reviews of interventions (Ver-sion 5.1.0). Retrieved from www.handbook-5-1.cochrane.org

33.

Huesmann

L. R.

(2010). Nailing the coffin shut on doubts that violent video games stimulate aggression: Comment on Anderson et al. (2010). Psychological Bulletin, 136, 179–181. doi:10.1037/a0018567

34.

Hunter

J. E.

Schmidt

F. L.

(1990). Dichotomization of continuous variables: The implications for meta-analysis. Journal of Applied Psychology, 75, 334–349. doi:10.1037/0021-9010.75.3.334

35.

Hyatt

C. S.

Chester

D. S.

Zeichner

Miller

J. D.

(2019). Analytic flexibility in laboratory aggression paradigms: Relations with personality traits vary (slightly) by operationalization of aggression. Aggressive Behavior, 45, 377–388. doi:10.1002/ab.21830

36.

John

L. K.

Loewenstein

Prelec

(2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532. doi:10.1177/0956797611430953

37.

Konijn

E. A.

Nije Bijvank

Bushman

B. J.

(2007). I wish I were a warrior: The role of wishful identification in the effects of violent video games on aggression in adolescent boys. Developmental Psychology, 43, 1038–1044. doi:10.1037/0012-1649.43.4.1038

38.

Kühberger

Fritz

Scherndl

(2014). Publication bias in psychology: A diagnosis based on the correlation between effect size and sample size. PLOS ONE, 9(9), Article e105825. doi:10.1371/journal.pone.0105825

39.

Lakens

Evers

E. R. K.

(2014). Sailing from the seas of chaos into the corridor of stability: Practical recommendations to increase the informational value of studies. Perspectives on Psychological Science, 9, 278–292. doi:10.1177/1745691614528520

40.

LeBel

E. P.

Borsboom

Giner-Sorolla

Hasselman

Peters

K. R.

Ratliff

K. A.

Smith

C. T.

(2013). PsychDisclosure.org: Grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8, 424–432. doi:10.1177/1745691613491437

41.

LeBel

E. P.

McCarthy

R. J.

Earp

B. D.

Elson

Vanpaemel

(2018). A unified framework to quantify the credibility of scientific findings. Advances in Methods and Practices in Psychological Science, 1, 389–402. doi:10.1177/2515245918787489

42.

McShane

B. B.

Böckenholt

Hansen

K. T.

(2016). Adjusting for publication bias in meta-analysis. Perspectives on Psychological Science, 11, 730–749. doi:10.1177/1745691616662243

43.

Moher

Liberati

Tetzlaff

Altman

D. G.

(2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. BMJ, 339, Article b2535. doi:10.1136/bmj.b2535

44.

Nelson

L. D.

Simmons

Simonsohn

(2018). Psychology’s renaissance. Annual Review of Psychology, 69, 511–534. doi:10.1146/annurev-psych-122216-011836

45.

Nosek

B. A.

Bar-Anan

(2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23, 217–243. doi:10.1080/1047840X.2012.692215

46.

Nosek

B. A.

Spies

J. R.

Motyl

(2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631. doi:10.1177/1745691612459058

47.

Nuijten

M. B.

Hartgerink

C. H. J.

van Assen

M. A. L. M.

Epskamp

Wicherts

J. M.

(2016). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 48, 1205–1226. doi:10.3758/s13428-015-0664-2

48.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, Article aac4716. doi:10.1126/science.aac4716

49.

Plonsky

Oswald

F. L.

(2014). How big is “big”? Interpreting effect sizes in L2 research. Language Learning, 64, 878–912. doi:10.1111/lang.12079

50.

Puljak

(2019). Methodological studies evaluating evidence are not systematic reviews. Journal of Clinical Epidemiology, 110, 98–99. doi:10.1016/j.jclinepi.2019.02.002

51.

Renkewitz

Keiner

(2018). How to detect publication bias in psychological research? A comparative evaluation of six statistical methods. PsyArXiv. doi:10.31234/osf.io/w94ep

52.

Sackett

P. R.

Yang

(2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85, 112–118. doi:10.1037/0021-9010.85.1.112

53.

Schimmack

(2014, December 30). The test of insufficient variance (TIVA): A new tool for the detection of questionable research practices [Blog post]. Retrieved from https://replicationindex.com/2014/12/30/the-test-of-insufficient-variance-tiva-a-new-tool-for-the-detection-of-questionable-research-practices/

54.

Schimmack

Chen

(2017, September 4). The power of the pen paradigm: A replicability analysis [Blog post]. Retrieved from https://replicationindex.com/2017/09/04/the-power-of-the-pen-paradigm-a-replicability-analysis/

55.

Schmidt

F. L.

Hunter

J. E.

(2015). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). London, England: Sage. doi:10.4135/9781483398105

56.

Schmidt

F. L.

I.-S.

(2016). The crisis of confidence in research findings in psychology: Is lack of replication the real problem? Or is it something else? Archives of Scientific Psychology, 4(1), 32–37. doi:10.1037/arc0000029

57.

Schönbrodt

F. D.

(2016). p-hacker: Train your p-hacking skills! Retrieved from http://shinyapps.org/apps/p-hacker/

58.

Simonsohn

(2013). Just post it: The lesson from two cases of fabricated data detected by statistics alone. Psychological Science, 24, 1875–1888. doi:10.1177/0956797613480366

59.

Simonsohn

Nelson

L. D.

Simmons

J. P.

(2014a). p-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534–547. doi:10.1037/a0033242

60.

Simonsohn

Nelson

L. D.

Simmons

J. P.

(2014b). p -curve and effect size: Correcting for publication bias using only significant results. Perspectives on Psychological Science, 9, 666–681. doi:10.1177/1745691614553988

61.

Stanley

T. D.

Doucouliagos

(2014). Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods, 5, 60–78. doi:10.1002/jrsm.1095

62.

Steegen

Tuerlinckx

Gelman

Vanpaemel

(2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11, 702–712. doi:10.1177/1745691616658637

63.

Sturmey

(Ed.). (2017). The Wiley handbook of violence and aggression. Hoboken, NJ: Wiley.

64.

Sumner

Vivian-Griffiths

Boivin

Williams

Venetis

C. A.

Davies

. . . Chambers

C. D.

(2014). The association between exaggeration in health related science news and academic press releases: Retrospective observational study. BMJ, 349, Article g7015. doi:10.1136/bmj.g7015

65.

Szucs

Ioannidis

J. P. A.

(2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLOS Biology, 15(3), Article e2000797. doi:10.1371/journal.pbio.2000797

66.

van Assen

M. A. L. M.

van Aert

R. C. M.

Wicherts

J. M

. (2015). Meta-analysis using effect size distributions of only statistically significant studies. Psychological Methods, 20, 293–309. doi:10.1037/met0000025

67.

Vul

Harris

Winkielman

Pashler

(2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science, 4, 274–290. doi:10.1111/j.1745-6924.2009.01125.x

68.

Wagenmakers

E.-J.

Beek

Dijkhoff

Gronau

Q. F.

Acosta

Adams

R. B.

Jr. . . . Zwaan

R. A.

(2016). Registered Replication Report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11, 917–928. doi:10.1177/1745691616674458

69.

Wagenmakers

E.-J.

Wetzels

Borsboom

van der Maas

H. L. J.

(2011). Why psychologists must change the way they analyze their data: The case of Psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426–432. doi:10.1037/a0022790

70.

Wagenmakers

E.-J.

Wetzels

Borsboom

van der Maas

H. L. J.

Kievit

R. A.

(2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. doi:10.1177/1745691612463078

71.

Weidman

A. C.

Steckler

C. M.

Tracy

J. L.

(2017). The jingle and jangle of emotion assessment: Imprecise measurement, casual scale usage, and conceptual fuzziness in emotion research. Emotion, 17, 267–295. doi:10.1037/emo0000226

72.

Weinstein

(2018). Mind-wandering, how do I measure thee with probes? Let me count the ways. Behavior Research Methods, 50, 642–661. doi:10.3758/s13428-017-0891-9

73.

Wessel

J. R.

(2018). Prepotent motor activity and inhibitory control demands in different variants of the go/no-go paradigm. Psychophysiology, 55(3), Article e12871. doi:10.1111/psyp.12871

74.

Wicherts

J. M.

Bakker

Molenaar

(2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLOS ONE, 6(11), Article e26828. doi:10.1371/journal.pone.0026828

75.

Wicherts

J. M.

Kievit

R. A.

Bakker

Borsboom

(2012). Letting the daylight in: Reviewing the reviewers and other ways to maximize transparency in science. Frontiers in Computational Neuroscience, 6, Article 20. doi:10.3389/fncom.2012.00020

76.

Wiernik

B. M.

Kostal

J. W.

Wilmot

M. P.

Dilchert

Ones

D. S.

(2017). Empirical benchmarks for interpreting effect size variability in meta-analysis. Industrial and Organiza-tional Psychology, 10, 472–479. doi:10.1017/iop.2017.44

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.64 MB