Estimating Evidential Value From Analysis of Variance Summaries: A Comment on Ly et al. (2018)

The past decade has witnessed a tremendous increase in the number of tools that enable social-science researchers to perform Bayesian inference. Ly et al. (2018) recently described one such tool: the Summary Stats module included as part of the open-source software package JASP (JASP Team, 2018). With this tool, researchers can input a minimal set of summary statistics (either from their own previously run analysis or from the published results of other researchers) and obtain a Bayesian reevaluation of the results. In particular, the Summary Stats module reports the Bayes factor (Kass & Raftery, 1995), a continuous index of the extent to which observed data are more likely under one hypothesis than under another, competing hypothesis. For example, the Bayes factor BF₀₁ describes the factor by which one’s prior belief about the relative likelihood of the null hypothesis H₀ over the alternative hypothesis H₁ should be updated after one observes data. This characterization makes the Bayes factor a useful measure of the evidential value of data (Etz & Vandekerckhove, 2017).

As Ly et al. (2018) described so well, the JASP Summary Stats module is a powerful tool that gives the user access to some of the core elements of Bayesian inference without need for accompanying raw data. This provides users with flexibility; researchers may wish to assess the evidential value of their own data to sensibly ground their interpretations, and reviewers or editors may wish to do the same for reported data to help with their own interpretations and decisions regarding publication. However, at present, the Summary Stats module does not include an option for directly reanalyzing the results of an analysis of variance (ANOVA). ANOVA is often characterized as the “workhorse” of experimental psychology (Rouder, Engelhardt, McCabe, & Morey, 2016, p. 1779). Thus, it would be quite helpful to the academic consumer to be able to compute Bayes factors from minimal ANOVA summaries.

To this end, I wrote an interactive Web application for calculating Bayes factors from minimal single-factor ANOVA summaries (see Fig. 1). The application, which can accessed by any Web browser at http://tomfaulkenberry.shinyapps.io/anovaBFcalc, performs calculations that are based on the Bayesian information criterion (BIC; Faulkenberry, 2018, 2019; Masson, 2011; Nathoo & Masson, 2016; Wagenmakers, 2007). It requires minimal input; the user need only specify the F statistic and the degrees of freedom for the ANOVA. Additionally, the user may specify the design (between subjects or repeated measures) and a prior probability for H₀ (default = .5). In return, the application provides the user with estimates of BF₀₁ and the reciprocal BF₁₀ (i.e., 1/BF₀₁) a sentence interpreting what the estimate of the Bayes factor for the “winning” model means, a graphical display of the strength of evidence indicated by the Bayes factor (i.e., a pizza plot; ¹ Wagenmakers et al., 2017), and an estimate of the posterior probability of each hypothesis.

Fig. 1.

Screenshot of the output of the interactive Bayes factor (BF) calculator that can be accessed at http://tomfaulkenberry.shinyapps.io/anovaBFcalc. When provided minimal summary statistics from an analysis of variance (ANOVA; e.g., the F statistic and the degrees of freedom), the calculator displays Bayes factors and posterior probabilities for the null hypothesis (H₀) and the alternative hypothesis (H₁), as well as a pizza plot (Wagenmakers et al., 2017) showing the relative extent to which H₀ and H₁ predict the observed data. Note that the pizza plot reflects the Bayes factor, not the posterior probabilities of H₀ and H₁. This screenshot summarizes a test for a between-groups difference in Rovenpor et al.’s (2019) Study 4.

Disclosures

The source code for the interactive Bayes factor calculator described in this article can be accessed at https://github.com/tomfaulkenberry/anovaBFcalc.

Example

For an illustration of the use of the online calculator, consider the following results from Rovenpor et al. (2019). In several experiments, Rovenpor et al. investigated whether violent conflict provides people with an enduring sense of meaning, thus perpetuating further intergroup conflict. In one study (Study 4), they measured perceptions of the scope of world conflict as a function of whether or not subjects first wrote about it. Rovenpor et al. reported no significant difference between groups, F(1, 226) = 2.17, p = 0.143. Though they interpreted this as a null effect, a strictly frequentist framework gives no indication of the degree of support for this hypothesis. As Figure 1 displays, the Bayes factor calculator gives additional information that may be helpful in assessing the evidential value of this result. First, it shows that the value of BF₀₁ is 5.08; that is, the observed data are approximately 5.08 times more likely under H₀ than H₁. Further, the calculator shows that if H₀ has a prior probability of .5, its posterior probability is .8355. That is, observing these data increases the plausibility of H₀ from 50% (prior) to 83.55% (posterior). The user can specify other values for the prior probability of H₀. For example, someone skeptical of the manipulation’s effect might assign H₀ a high prior probability of, say, .80. The resulting posterior probability for H₀ is .9531; observing these data should increase this user’s belief in H₀. Alternatively, someone who believes the manipulation is effective might assign H₀ a prior probability of .20. In this case, the posterior probability for H₀ is .5595; this user should also shift toward greater belief in H₀. Of course, this is a natural consequence of the Bayes factor, which by definition is the factor by which prior odds are multiplied after data are observed.

Sometimes, Bayesian reanalyses can reveal less evidence than might be originally thought. In Study 5 of Rovenpor et al. (2019), a large sample of subjects (N = 352) were randomly assigned to one of three conditions; in two of the noncontrol conditions, they watched a video about a terrorist attack, one of which framed the attack as “meaningful.” Each subject was then given a series of questions assessing perceptions of meaning in conflict. The authors reported a significant difference among the three conditions in perceived meaning of conflict, F(2, 349) = 6.21, p = 0.002. Traditionally, this would be viewed as support for the alternative over the null hypothesis (i.e., support for H₁ over H₀). However, the interactive Bayes factor calculator gives BF₁₀ a value of 1.34, indicating that the observed data are only 1.34 times more likely under H₁ than under H₀. The plausibility of H₁ is increased only from 50% (prior) to 57.22% (posterior). The Bayesian reanalysis indicates that the observed data are not very evidential; that is, they do not sway belief toward either model very strongly.

Discussion

In summary, the interactive Bayes factor calculator provides a useful supplement to the tools described by Ly et al. (2018) for assessing evidential value from studies. As does JASP’s Summary Stats module, the calculator requires only minimal input, so it is easy for the user to obtain a measure of evidential value from minimal ANOVA summaries. In addition, the calculator provides interpretation of the Bayes factors that are output, which may be useful to users who are new to Bayesian inference (see also van Doorn et al., 2019).

One potential downside to the calculator is that the computations are based on a very specific choice of prior (the unit information prior). This prior is distributed over a large range of possible effects; on one hand, such a prior lets the data “speak for” themselves (Gelman et al., 2013, p. 51), but on the other hand, the resulting Bayes factors are often inflated for H₀ and undersized for H₁. Because of this, one may wonder how the BIC prior performs compared with other default priors, such as the JZS prior (Rouder, Morey, Speckman, & Province, 2012). It turns out that the BIC and JZS priors are quite comparable, as I have demonstrated with a wide range of simulated data sets (Faulkenberry, 2018).

There are a few advantages that the calculator (and the BIC method more generally) provides over some other options. One is that the calculator works for repeated measures designs. Though R users can use the oneWayAOV.Fstat function from the BayesFactor package (Morey & Rouder, 2018) to calculate Bayes factors from summary statistics, this function works only for balanced, between-subjects designs. Note that although the calculator works with unbalanced single-factor designs (in single-factor designs, the sum-of-squares calculations work out the same whether the design is balanced or unbalanced), researchers are currently actively investigating how these closed-form Bayes factor computations extend to more general designs (unbalanced designs with multiple factors, etc.). In summary, I think that the calculator will be useful to many people, especially as a means of following emerging recommendations on conducting and reporting Bayesian analyses.

Supplemental Material

Faulkenberry_AMPPSOpenPracticesDisclosure-v1.0-signedTJF – Supplemental material for Estimating Evidential Value From Analysis of Variance Summaries: A Comment on Ly et al. (2018)

Supplemental material, Faulkenberry_AMPPSOpenPracticesDisclosure-v1.0-signedTJF for Estimating Evidential Value From Analysis of Variance Summaries: A Comment on Ly et al. (2018) by Thomas J. Faulkenberry in Advances in Methods and Practices in Psychological Science