Monte Carlo resampling methods to obtain probability values for chi-squared and likelihood-ratio test statistics for multiway contingency tables are presented. A resampling algorithm provides random arrangements of cell frequencies in a multiway contingency table, given fixed marginal frequency totals. Probability values are obtained from the proportion of resampled test statistic values equal to or greater than the observed test statistic value.
Get full access to this article
View all access options for this article.
References
1.
AgrestiA. (1990) Categorical data analysis. New York: Wiley.
2.
AgrestiA. (2002) Categorical data analysis. (2nd ed.) New York: Wiley.
3.
AgrestiA., & FinlayB. (2009) Statistical methods for the social sciences. (4th ed.) Upper Saddle River, NJ: Prentice Hall.
4.
BakemanR.RobinsonB. F., & QueraV. (1996) Testing sequential association: estimating exact p values using sampled permutations. Psychological Methods, 1, 4–15.
5.
BergsonJ. (1978a) Do the marginal totals of the 2×2 table contain relevant information concerning the table proportions?Journal of Statistical Planning and Inference, 2, 43–44.
6.
BergsonJ. (1978b) In dispraise of the exact test. Journal of Statistical Planning and Inference, 2, 27–42.
7.
BerryK. J.MielkeP. W.Jr., & MielkeH. W. (2002) The Fisher-Pitman permutation test: an attractive alternative to the F test. Psychological Reports, 90, 495–502.
8.
BerryK. J.MielkeP. W.Jr., & ZeltermanD. (1994) Fisher's exact test of mutual independence for 2×2×2 cross-classification tables. Educational and Psychological Measurement, 54, 110–114.
9.
BishopY. M.FienbergS. E., & HollandP. W. (1975) Discrete multivariate analysis: theory and practice. Cambridge, MA: MIT Press.
10.
BoxG. E. P., & AndersonS. L. (1955) Permutation theory in the derivation of robust criteria and the study of departures from assumption. Journal of the Royal Statistical Society, Series B, 17, 1–34.
11.
BrownL.SherbenouR. J., & JohnsenS. K. (1997) Manual of Test of Nonverbal Intelligence. (3rd ed.) Austin, TX: PRO-ED.
12.
EdenT., & YatesF. (1933) On the validity of Fisher's z test when applied to an actual example of non-normal data. The Journal of Agricultural Science, 23, 6–17.
FeinsteinA. R. (1993) Permutation tests and “statistical significance.”M. D. Computing: Computers in Medical Practice, 10, 28–41.
15.
FisherR. A. (1935) The design of experiments. Edinburgh, Scotland: Oliver & Boyd.
16.
GilbertN. (1993) Analyzing tabular data: loglinear and logistic models for social researchers. London: UCL Press.
17.
GoodmanL. A. (1970) The multivariate analysis of qualitative data: interactions among multiple classifications. Journal of the American Statistical Association, 65, 226–256.
18.
GoodmanL. A. (1971) The analysis of multidimensional contingency tables: stepwise procedures and direct estimation methods for building models for multiple classifications. Technometrics, 13, 33–61.
19.
GroveD. M. (1984) Discussion of “Tests of significance for 2×2 tables.”Journal of the Royal Statistical Society, Series A, 147, 454–455.
20.
HaberM. (1990) Comments on “The test of homogeneity for 2×2 contingency tables: a review of and some personal opinions on the controversy” by G. Camilli. Psychological Bulletin, 108, 146–149.
21.
HabermanS. J. (1978) Analysis of qualitative data. Vol. 1. Introductory topics. New York: Academic Press.
22.
HabermanS. J. (1979) Analysis of qualitative data. Vol. 2. New developments. New York: Academic Press.
23.
HoeffdingW. (1952) The large sample power of tests based on permutations of observations. The Annals of Mathematical Statistics, 23, 169–192.
24.
HunterM. A., & MayR. B. (1993) Some myths concerning parametric and nonparametric tests. Canadian Psychology, 34, 384–389.
25.
JohnstonJ. E.BerryK. J., & MielkeP. W.Jr. (2007) Permutation tests: precision in estimating probability values. Perceptual and Motor Skills, 105, 915–920.
26.
KempthorneO. (1955) The randomization theory of statistical inference. Journal of the American Statistical Association, 50, 946–967.
27.
KempthorneO. (1978) In dispraise of the exact test: reactions. Journal of Statistical Planning and Inference, 2, 199–213.
28.
KennedyJ. J. (1983) Analyzing qualitative data: introductory log-linear analysis for behavioral research. New York: Praeger.
29.
KennedyP. E. (1995) Randomization tests in econometrics. Journal of Business & Economic Statistics, 13, 85–94.
30.
KroonenbergP. M. (2008) Applied multiway data analysis. Hoboken, NJ: Wiley.
31.
LloydC. J. (1988) Some issues arising from the analysis of 2 × 2 contingency tables. Australian Journal of Statistics, 30, 35–46.
32.
LudbrookJ. L. (2008) Analysis of 2 × 2 tables of frequencies: matching test to experimental design. International Journal of Epidemiology, 37, 1430–1435.
33.
LudbrookJ. L., & DudleyH. (1998) Why permutation tests are superior to t and F tests in biomedical research. The American Statistician, 52, 127–132.
34.
MayR. B., & HunterM. A. (1993) Some advantages of permutation tests. Canadian Psychology, 34, 401–407.
35.
MielkeP. W.Jr., & BerryK. J. (1988) Cumulant methods for analyzing independence of r-way contingency tables and goodness-of-fit frequency data. Biometrika, 75, 790–793.
36.
MielkeP. W.Jr., & BerryK. J. (2007) Permutation methods: a distance function approach. (2nd ed.) New York: Springer-Verlag.
37.
MielkeP. W.Jr.BerryK. J., & JohnstonJ. E. (2007) Resampling programs for multiway contingency tables with fixed marginal frequency totals. Psychological Reports, 101, 18–24.
38.
OlzakL. A., & WickensT. D. (1983) The interpretation of detection data through direct multivariate frequency analysis. Psychological Bulletin, 93, 574–585.
39.
PearsonK. (1900/1948) On a criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philosophical Magazine, 50, 157–175. Reprinted in E. S. Pearson (Ed.), Karl Pearson's early statistical papers. Cambridge, UK: Cambridge Univer. Press. Pp. 339–357.
40.
PearsonK. (1904/1948) Mathematical contributions to the theory of evolution: XIII. On the theory of contingency and its relation to association and normal correlation. Drapers' Company Research Memoirs, Biometric Series, 1, 1–35. Reprinted in E. S. Pearson (Ed.), Karl Pearson's early statistical papers. Cambridge, UK: Cambridge Univer. Press. Pp. 443–475.
41.
PitmanE. J. G. (1937a) Significance tests which may be applied to samples from any populations. Supplement to the Journal of the Royal Statistical Society, 4, 119–130.
42.
PitmanE. J. G. (1937b) Significance tests which may be applied to samples from any populations: II. The correlation coefficient test. Supplement to the Journal of the Royal Statistical Society, 4, 225–232.
43.
PitmanE. J. G. (1938) Significance tests which may be applied to samples from any populations: III. The analysis of variance test. Biometrika, 29, 322–335.
44.
RaoJ. N. K., & ScottA. J. (1984) On chi-squared tests for multiway contingency tables with cell proportions estimated from survey data. The Annals of Statistics, 12, 46–60.
45.
ReadT. R. C., & CressieN. A. C. (1988) Goodness-of-fit for discrete multivariate data. New York: Springer-Verlag.
46.
VokeyJ. R. (2003) Multiway frequency analysis for experimental psychologists. Canadian Journal of Experimental Psychology, 57, 257–264.
47.
WaldA., & WolfowitzJ. (1944) Statistical tests based on permutations of the observations. The Annals of Mathematical Statistics, 15, 358–372.
48.
WelchB. L. (1937) On the z test in randomized blocks and Latin squares. Biometrika, 29, 21–52.
49.
WickensT. D. (1989) Multiway contingency tables analysis for the social sciences. Hillsdale, NJ: Erlbaum.
50.
WickensT. D. (1993) Analysis of contingency tables with between-subjects variability. Psychological Bulletin, 113, 191–204.
51.
WilksS. S. (1935) The likelihood test of independence in contingency tables. The Annals of Mathematical Statistics, 6, 190–196.
52.
WilksS. S. (1938) The large-sample distribution of the likelihood ratio for testing composite hypotheses. The Annals of Mathematical Statistics, 9, 60–62.
53.
YatesF., with Discussants. (1984) Tests of significance for 2×2 contingency tables. Journal of the Royal Statistical Society, Series A, 147, 426–463.
