The Fallacy of Replacing an a Priori Significance Level With an a Posteriori Significance Level

Abstract

In an effort to "improve" experimental results, many researchers may be tempted to replace an a priori significance level with an a posteriori significance level. There are, however, at least four reasons why such a replacement is incorrect procedure. First, the probability under the null hypothesis that the observed probability level is less than the a posteriori significance level is not, as is thought by many researchers, generally equal to the a posteriori significance level. Second, this probability cannot be interpreted as the probability of a Type I error. Third, an a posteriori significance level cannot be interpreted as an a priori significance level. And fourth, replacing an a priori significance level with an a posteriori significance level is unethical procedure. Some comments are made on current APA reporting procedures for significance levels.