Monte Carlo simulations were used to examine three different cases of using analysis of variance sequentially. The first case dealt with adding additional data to original data that were not significant and then reanalyzing. Alpha levels reached a high of .086 in this case. In the second case, data were added only when the F-ratio was not significant but exceeded 1.00; in this case, the alpha level reached a high of .16. The third case added data only when the p value exceeded 0.05 but was less than a predetermined limit (between .06 and .10); alpha levels rose to as much as .39 in this case. The message is clear: ANOVA was not intended as a sequential test and should not be used as such.
Get full access to this article
View all access options for this article.
References
1.
American Psychological Association (1987). Summary report of journal operations: 1986. American Psychologist, 42, 744.
2.
Barnard, G. A. (1946). Sequential tests in industrial statistics. Supplement to the Journal of the Royal Statistical Society, 8, 1-26.
3.
Centra, J. A. (1979). Deternining faculty effectiveness. San Francisco: Jossey-Bass.
4.
Gans, D. J. (1984). The search for significance: Different tests on the same data. Journal of Statistical Computation and Simulation, 19, 1-21.
5.
Marsaglia, G. , Ananthanarayanan, K., and Paul, N. J. (1976). Improvements on fast methods for generating normal random variables. Information Processing Letters, 5, 27-30.
6.
Quereshi, M. Y. , Buckley, J. M., and Fadden, T. F. (1981). Some determinants of psychologists employability in academic set- tings. Personnel Psychology, 34, 301-308.
7.
Ryan, T. A. (1985). "Ensemble-adjusted p values": How are they to be weighted?, Psychological Bulletin, 97, 521-526.
8.
Seldin, P. (1984). Changing practices in faculty evaluation: A critical assessment and recommendations for improvement. San Francisco: Jossey-Bass.
9.
Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance-or vice versa. Journal of the American Statistical Association, 54, 30-34.
10.
Tullock, G. (1959). Publications decisions and tests of significance: A comment. Journal of the American Statistical Association, 54, 593.
