The use of direct observation methods to collect data relevant to research and practice in special education is widespread. Although the reliability of such data has often been addressed, far less attention has focused on the accuracy of these data. The purposes of this article are (a) to review research on factors that adversely affect the accuracy of observers, and (b) to provide recommendations for reducing their possible influence. The areas discussed include reactivity, observer drift; the recording procedure; location of the observation; reliability; expectancy and feedback; and the characteristics of subjects, observers, and settings.
Get full access to this article
View all access options for this article.
References
1.
BarkleyR. A. (1981). Hyperactive children. New York: Guilford Press.
2.
BoydR. D., & deVaultM. V. (1966). The observation and recording of behavior. Review of Educational Research, 36, 529–551.
3.
BoykinR. A., & NelsonR. O. (1981). The effects of instructions and calculation procedures on observers' accuracy, agreement, and calculation correctness. Journal of Applied Behavior Analysis, 14, 479–489.
4.
BrooksP. H., & BaumeisterA. A. (1977). A plea for consideration of ecological validity in the experimental psychology of mental retardation: A guest editorial. American Journal of Mental Deficiency, 81, 407–416.
5.
BrulleA. R., & BartonL. E. (1980). The accuracy of momentary time sampling procedures when used in applied setting. Paper presented at the Annual Meeting of the Association for Behavior Analysis, Dearborn, Michigan.
6.
BrulleA. R., & ReppA. C. (1984). An investigation of the accuracy of momentary time sampling procedures with time series data. British Journal of Psychology, 75, 481–485.
7.
CampbellD. T., & StanleyJ. C. (1963). Experimental and quasi-experimental designs for research. Boston: Houghton Mifflin Company.
8.
ChristensenA., & HazzardA. (1983). Reactive effects during naturalistic observation of families. Behavioral Assessment, 5, 349–362.
9.
CunninghamT. R., & TharpR. G. (1981). The influence of settings on accuracy and reliability of behavioral observation. Behavioral Assessment, 3, 67–68.
10.
DeMasterB.ReidJ. B. (1973). Effects of feedback procedures in maintaining observer reliability. Eugene, OR: Oregon Research Institute (cited in JohnsonS. M. & BolstadO. D.Methodological issues in naturalistic observation: Some problems and solutions for field research. In HamerlynckL. A.HardyL. C., & MashE. J. (Eds.), Behavior Change: Methodology, concepts and practice. Champaign, IL: Research Press).
11.
DeMasterB.ReidJ., & TwentymanC. (1977). The effects of different amounts of feedback on observer's reliability. Behavior Therapy, 8, 317–329.
12.
DunbarR. I. M. (1976). Some aspects of research design and their implications in the observational study of behaviour. Behaviour, 58, 78–98.
13.
FlandersN. A. (1965). Teacher influences, pupil attitudes, and achievement. U.S. Department of Health, Education, and Welfare, Office of Education, Cooperative Research Monograph No. 12. Washington, DC: Government Printing Office.
14.
FultonW. R., & RupiperO. J. (1962). Observation of teaching: Direct vs. vicarious experiences. Journal of Teacher Education, 13, 157–164.
15.
GreenS. B., & AlversonL. G. (1978). A comparison of indirect measures for long-duration behaviors. Journal of Applied Behavior Analysis, 11, 530.
16.
GurwitzS. B., & DodgeK. A. (1975). Adults' evaluation of a child as a function of sex of adult and sex of child. Journal of Personality and Social Psychology, 32, 822–828.
17.
HarrisF. C., & CimineroA. R. (1978). The effect of witnessing consequences on the behavioral recordings of experimental observers. Journal of Applied Behavior Analysis, 11, 513–521.
18.
HartmannD. P. (1977). Considerations in the choice of interobserver reliability estimates. Journal of Applied Behavior Analysis, 10, 103–116.
19.
HartmannD. P., & WoodD. D. (1982). Observational methods. In BellackA.HersenM., & KazdinA. E. (Eds.), International handbook of behavior modification and therapy. New York: Plenum.
20.
HawkinsR. P. (1979). The functions of assessment: Implications for selection and development of devices for assessing repertoires in clinical, educational, and other settings. Journal of Applied Behavior Analysis, 12, 501–516.
21.
HayL. R.NelsonR. O., & HayW. M. (1977). Some methodological problems in the use of teachers as observers. Journal of Applied Behavior Analysis, 10, 345–348.
22.
HayL. R.NelsonR. O., & HayW. M. (1980). Methodological problems in the use of participant observers. Journal of Applied Behavior Analysis, 13, 501–504.
23.
HaynesS. N., & HornW. F. (1982). Reactivity in behavioral observation: A review. Behavioral Assessment, 4, 369–385.
24.
HersenM., & BarlowD. H. (1976). Single-case experimental designs: Strategies for studying behavior change. New York: Pergamon Press.
25.
HopkinsB. L. (1979). Proposed conventions for evaluating observer reliability: A commentary on two articles by Birkimer and Brown. Journal of Applied Behavior Analysis, 12, 561–564.
26.
HornW. F., & HaynesS. N. (1981). An investigation of sex bias in behavioral observations and ratings. Behavioral Assessment, 3, 173–183.
27.
JohnsonS. M., & BolstadO. D. (1973). Methodological issues in naturalistic observation: Some problems and solutions for field research. In HamerlynckL. A.HardyL. C., & MashE. J. (Eds.), Behavior change: Methodology, concepts, and practice. Champaign, IL: Research Press.
28.
JonesR. R.ReidJ. B., & PattersonG. R. (1975). Naturalistic observation in clinical assessment. In McReynoldsP. (Ed.), Advances in psychological assessment (Vol. 3). San Francisco: Jossey-Bass.
29.
KauffmanJ. M. (1981). Characteristics of children's behavior disorders, second edition. Columbus, OH: Charles E. Merrill Publishing.
30.
KazdinA. E. (1977). Artifact, bias, and complexity of assessment: The ABCs of reliability. Journal of Applied Behavior Analysis, 10, 141–150.
31.
KazdinA. E. (1979). Unobtrusive measures in behavioral assessment. Journal of Applied Behavior Analysis, 12, 713–724.
32.
KazdinA. E. (1980). Research design in clinical psychology. New York: Harper & Row.
33.
KentR. N., & FosterS. L. (1977). Direct observational procedures: Methodological issues in naturalistic settings. In CimineroA. R.CalhounK. S., & AdamsH. E. (Eds.), Handbook of behavioral assessment. New York: John Wiley & Sons.
34.
KentR. N.O'LearyK. D.DiamentC., & DietzA. (1974). Expectation biases in observational evaluation of therapeutic change. Journal of Consulting and Clinical Psychology, 42, 774–780.
35.
KentR. N.O'LearyK. D.DietzA. & DiamentC., (1979). Comparison of observational recordings in vivo, via mirror, and via television. Journal of Applied Behavior Analysis, 12, 517–522.
36.
KratochwillT. R. (1979). Just because it's reliable doesn't mean it's believable: A commentary on two articles by Birkimer and Brown. Journal of Applied Behavior Analysis, 12, 553–558.
37.
LaGrowS. L., & ReppA. C. (1984). Stereotypic responding: A review of intervention research. American Journal of Mental Deficiency, 88, 595–609.
38.
Landesman-DwyerS. (1981). Living in the community. American Journal of Mental Deficiency, 86, 223–234.
39.
LipinskiD., & NelsonR. (1974). Problems in the use of naturalistic observation as a means of behavioral assessment. Behavior Therapy, 5, 341–357.
40.
MashE. J., & McElweeJ. D. (1974). Situational effects on observer accuracy: Behavior predictability, prior experience, and complexity of coding categories. Child Development, 45, 367–377.
41.
MossH. A., & JonesS. J. (1977). Relations between maternal attitudes and maternal behavior as a function of social class. In LeidermanP. H. & TulkinS. R. (Eds.), Cultural and social influences on behavior in infancy and early childhood. New York: Academic Press.
42.
NelsonR. O. (1983). Behavioral assessment: Past, present, and future. Behavioral Assessment, 5, 195–206.
43.
O'LearyK. D., & KentR. N. (1977). Sources of bias in observational recording. In EtzelB. C.LeBlancJ. M., & BaerD. M., New developments in behavioral research, theory, method, and application. Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers.
44.
O'LearyK. D., & KentR. N., & KanowitzJ. (1975). Shaping data collection congruent with experimental hypotheses. Journal of Applied Behavior Analysis, 8, 43–51.
45.
PowellJ.MartindaleA., & KulpS. (1975). An evaluation of time-sample measures of behavior. Journal of Applied Behavior Analysis, 8, 463–469.
46.
PowellJ.MartindaleB.KulpS.MartindaleA., & BaumanR. (1977). Taking a closer look: Time sampling and measurement error. Journal of Applied Behavior Analysis, 10, 325–332.
47.
RedfieldJ., & PaulG. L. (1976). Bias in behavioral observation as a function of observer familiarity with subjects and typicality of behavior. Journal of Consulting and Clinical Psychology, 44, 156.
48.
ReidJ. B. (1970). Reliability assessment of observation data: A possible methodological problem. Child Developmental, 41, 1143–1150.
49.
ReidJ. B. (1973a). The relationship between complexity of observer protocol and interobserver agreement for twenty-five reliability assessment sessions: A technical note. Unpublished manuscript, University of Oregon.
50.
ReidJ. B. (1973b). Differences in the complexity of reliability assessment vs. adjacent non-reliability assessment observation sessions: A technical note. Unpublished manuscript, University of Oregon.
51.
ReppA. C. (1983). Teaching the mentally retarded. Englewood Cliffs, NJ: Prentice Hall.
52.
ReppA. C.DeitzD. E. D.BolesS. M.DeitzS. M., & ReppC. F. (1976). Differences among common methods for calculating interobserver agreement. Journal of Applied Behavior Analysis, 9, 109–113.
53.
ReppA. C.HarmonM. L., & FelceD. (1984). A real-time, parallel entry, portable computer system for observational research. Paper presented at the annual meeting of the Association for Behavior Analysis.
54.
ReppA. C.RobertsD. M.SlackD. J.ReppC. F., & BerklerM. S. (1976). A comparison of frequency, interval, and time-sampling methods of data collection. Journal of Applied Behavior Analysis, 9, 501–508.
55.
RojahnJ., & SchroederS. R. (1983). Behavioral Assessment. In MatsonJ. L. & MulickJ. A. (Eds.), Handbook of mental retardation (pp. 227–243). New York: Pergamon Press.
56.
RomanczykR. G.KentR. N.DiamentC., & O'LearyK. D. (1973). Measuring the reliability of observational data: A reactive process. Journal of Applied Behavior Analysis, 6, 175–184.
57.
RumfordH. P. (1962). An experiment in teaching elementary school methods via closed circuit television. Journal of Educational Research, 56, 139–143.
58.
SackettG. P. (Ed.) (1978a). Observing behavior, Vol. I: Theory and application in mental retardation. Baltimore, MD: University Park Press.
59.
SackettG. P. (Ed.) (1978b). Observing behavior, Vol. II: Data collection and analysis methods. Baltimore, MD: University Park Press.
60.
SalviaJ., & YsseldykeJ. E. (1981). Assessment in special and remedial education, second edition. Boston: Houghton-Mifflin Co.
61.
Sanson-FisherR. W.PooleA. D., & DunnJ. (1980). An empirical method for determining an appropriate interval length for recording behavior. Journal of Applied Behavior Analysis, 13, 493–500.
62.
SchoggenP. (1964). Mechanical aids for making specimen records of behavior. Child Development, 35, 985–988.
63.
SchuelerH., & GoldM. H. (1964). Video records of student teachers: A report of the Hunter College research project evaluating the use of kinescopes in preparing student teachers. Journal of Teacher Education, 15, 358–364.
64.
SchwartzM. S., & SchwartzC. G. (1955). Problems in participant observation. American Journal of Sociology, 60, 343–353.
65.
SkindrudK. D. (1973). An evaluation of observer bias in experimental-field studies of social interaction. Dissertation Abstract International, 334, 4989.
66.
StonemanZ.BrodyG. H., & AbbottD. (1983). In-home observations of young Down Syndrome children with their mothers and fathers. American Journal of Mental Deficiency, 87, 591–600.
67.
StrainP. S.LambertD. L.KerrM. M.StaggV., & LenknerD. A. (1983). Naturalistic assessment of children's compliance to teachers' requests and consequences for compliance. Journal of Applied Behavior Analysis, 16, 243–249.
68.
Sulzer-AzaroffB., & MayerG. R. (1977). Applying behavior-analysis procedures with children and youth. New York: Holt, Rinehart & Winston.
69.
TaplinP. S., & ReidJ. B. (1973). Effects of instructional set and experimental influences on observer reliability. Child Development, 44, 547–554.
70.
ThomasD. S.LoomisA. M., & ArringtonR. E. (1933). Observational studies of social behavior. New Haven: Institute of Human Relations, Yale University.
71.
ThomasJ. D.PreslandI. E.GrantM. D., & GlynnT. L. (1978). Natural rates of teacher approval and disapproval in grade-7 classrooms. Journal of Applied Behavior Analysis, 11, 91–94.
72.
WasikB. H., & LovenM. O. (1980). Classroom observational data: Sources of inaccuracy and proposed solutions. Behavioral Assessment, 2, 211–227.
73.
WhiteM. A. (1975). Natural rates of teacher approval and disapproval in the classroom. Journal of Applied Behavior Analysis, 8, 367–372.
74.
WildmanB. G.EricksonM. T., & KentR. N. (1975). The effect of two training procedures on observer agreement and variability of behavior ratings. Child Development, 46, 520–524.
75.
WildmanB. G., & EricksonM. T. (1977). Methodological problems in behavioral observation. In ConeJ. D. & HawkinsR. P. (Eds), Behavioral assessment: New directions in clinical psychology. New York: Brunner/Mazel.
76.
YarrowM. R., & WaxierC. Z. (1979). Observing interactions: A confrontation with methodology. In CairnsR. B. (Ed), The analysis of social interactions: Methods, issues, and illustrations. Hillsdale, NJ: Erlbaum.
