Sage Journals: Discover world-class research

Abstract

Sampling fluctuation is an inherent characteristic of data. We cannot pretend that it does not exist, nor can we abandon significance testing without a viable replacement. It remains a concern as long as we wish to draw research conclusions within a positivistic paradigm. It is there regardless of whether we conduct a large-group or a single-subject study. It is also there whether we conduct a simple norm-referenced assessment, a complex judgment-based assessment, or any other assessment. The widespread misuse of significance testing suggests that sampling fluctuation has been an overbearing guest who has become the inappropriate center of attention at the party. Instead of wishing this guest away, we should put sampling fluctuation in its proper place; namely, as but one of the numerous epistemological concerns in any positivistic research.

Get full access to this article

View all access options for this article.

References

Baer, D.M. (1977). Perhaps it would be better not to know everything. Journal of Applied Behavior Analysis, 10, 167–172.

Barber, T. (1973). Pitfalls in research: Nine investigator and experimenter effects. In R.M.W. Travers (Ed.), Second handbook of research on teaching (pp. 382–404). Chicago: Rand McNally.

Box, G.E.P. , & Jenkins, G.M. (1976). Time series analysis: Forecasting and control (rev. ed.). San Francisco: Holden-Day.

Brennan, R.L. (1983). Elements of generalizability theory. Iowa City, IA: ACT.

Campbell, D.T. , & Stanley, J.C. (1966). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.

Carver, R. (1978). The case against statistical significance testing. Harvard Educational Review, 48(3), 378–399.

Cipani, E. , & Waite, V.A. (1980). Experimenter bias effects: A direct replication. Perceptual and Motor Skills, 51, 129–130.

Cronbach, L.J. , Gleser, G.C. , Nanda, H. , & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.

Edgington, E.S. (1967). Statistical inference from N = 1 experiments. Journal of Psychology, 65, 195–199.

10.

Edgington, E.S. (1975). Randomization tests for one-subject operant experiments. Journal of Psychology, 90, 57–68.

11.

Edgington, E.S. (1982). Nonparametric tests for single-subject multiple schedule experiments. Behavioral Assessment, 4, 83–91.

12.

Glass, G.V. , McGaw, B. , & Smith, M.L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.

13.

Gottman, J.M. (1981). Time-series analysis: A comprehensive introduction for social scientists. Cambridge, England: Cambridge University Press.

14.

Hartmann, D.P. , Gottman, J.M. , Jones, R.R. , Gardner, W. , Kazdin, A.E. , & Vaught, R.S. (1980). Interrupted time-series analysis and its application to behavioral data. Journal of Applied Behavior Analysis, 13, 543–559.

15.

Kleijnen, J. , Knipschild, P. , ter Riet, G. (1991). Clinical trials of homeopathy. British Medical Journal, 302, 316–323.

16.

Michael, J. (1974). Statistical inference for individual organism research: Mixed blessing or curse? Journal of Applied Behavior Analysis, 7, 647–653.

17.

Reid, J.B. (1982). Observer training in naturalistic research. In D.P. Hartmann (Ed.), Using observers to study behavior (pp. 37–50). San Francisco: Jossey-Bass.

18.

Roethlisberger, F.J. , & Dickson, W.J. (1939). Management and the worker. Cambridge, MA: Harvard University Press.

19.

Rosenthal, R. , & Jacobson, L. (1968). Pygmalion in the classroom. New York: Holt, Rinehart & Winston.

20.

Sackett, G.P. (1974). A nonparametric lag sequential analysis for studying dependency among responses in observational scoring systems. Unpublished manuscript, University of Washington, Seattle.

21.

Saretsky, G. (1975, April). The John Henry effect: Potential confounder of experimental vs. control group approaches to the evaluation of educational innovations. Paper presented at the annual meeting of the American Educational Research Association, Washington, DC. (Eric Document Reproduction Service No. ED 106 309)

Significance Testing

Abstract

Get full access to this article

References