The present study compared the accuracy of observers employing momentary time sampling vs. continuous data collection systems when scoring behaviors that differed along the dimensions of frequency and predictability. Twelve trained observers were randomly assigned to one of the two methods of data collection when scoring a series of four videotapes of teacher-student interaction. The results indicated that there was no overall difference in observer accuracy between the two methods of data collection. Interesting disordinal interactions, however, were identified. Continuous data collection significantly enhanced observer accuracy when scoring high frequency behaviors and behaviors with high transitional probabilities to other behaviors. Continuous data collection, however, impeded accuracy when behaviors displayed low levels of predictability (low transitional probabilities). Momentary time sampling produced the most stable levels of observer accuracy. Some implications of these findings for the selection of recording procedures and observer training are presented.
Get full access to this article
View all access options for this article.
References
1.
AltmannJ. (1974). Observational study of behavior: Sampling methods. Behavior, 49, 227–267.
2.
ArickJ.R.KrugD.A.RathkeyJ.FlaxS. (1978). Collecting trial-by-trial in severely handicapped children with the aid of a microprocessor. Journal of Special Education Technology, 2, 22–27.
3.
AryD.E. (1984). Mathematical prediction of sampling component of behavior observational measurement error. Behavioral Assessment, 6, 221–228.
4.
AryD.E.SuenH.K. (1983). The use of momentary time sampling to assess both frequency and duration of behavior. Journal of Behavioral Assessment, 5, 143–150.
5.
AryD.E.Van AckerR.KarshK.L. (1986, April). Comparing behavior duration reliabilities of momentary time-sampling and continuous electronic data gathering. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
6.
BakemanR.GottmanJ.M. (1986). Observing interaction: An introduction to sequential analysis. Cambridge: Cambridge University Press.
7.
BassR.F.AserlindL. (1984). Interval and time-sample data collection procedures: Methodological issues. In GadowK. (Ed.), Advances in Learning and Behavioral Disabilities: Vol. 3 (pp. 1–39). Greenwich, CT: JAI.
8.
BaumW.M. (1975). Time Allocation in human vigilance. Journal of the Experimental Analysis of Behavior, 23, 45–53.
9.
BobbittR.A.GordonB.N.JensenG.D. (1966). Development and application of an observational method: Continuing reliability testing. Journal of Psychology, 63, 83–88.
10.
CohenJ. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.
11.
CohenJ. (1968), Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213–220.
12.
CronbachL.J.GlaserG.C.NandaH.RajaratnamN. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
13.
CrossmanE.K.WilliamsJ.C.ChambersJ.H. (1978). Using the PET microcomputer for collecting and analyzing observational data in the classroom. Behavior Research Methods & Instrumentation, 10 (4), 563–566.
14.
CunninghamT.R.TharpR.G. (1981). The influence of settings on accuracy and reliability of behavioral observation. Behavioral Assessment, 3, 67–78.
15.
DorseyB.L.NelsonR.O.HayesS.C. (1986). The effects of code complexity and of behavioral frequency on observer accuracy and interobserver agreement. Behavioral Assessment, 8, 349–363.
16.
FagotB.HaganR. (1988). Is what we see what we get? Comparisons of taped and live observations. Behavioral Assessment, 10, 367–374.
17.
FleissJ.L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378–382.
18.
GreenS.B.AlversonL.G. (1978). A comparison of indirect measures of long-duration behaviors. Journal of Applied Behavior Analysis, 19, 73–77.
19.
HanleyJ.A. (1987). Standard error of the Kappa statistic. Psychological Bulletin, 102, 315–321.
20.
HarropA.DanielsM. (1986). Methods of time sampling: A reappraisal of momentary time sampling and partial interval recording. Journal of Applied Behavior Analysis, 19, 73–77.
21.
HaynesS.N.HornW.F. (1982). Reactivity in behavioral observation: A review. Behavioral Assessment, 4, 369–385.
22.
HollandJ.G. (1963). Human vigilance. In BucknerD.N.McGrathJ.J. (Eds.). Vigilance: A symposium. New York: McGraw-Hill.
23.
HornW.R.HaynesS.N. (1981). An investigation of sex bias in behavioral observations and ratings. Behavioral Assessment, 3, 173–183.
24.
JohnsonJ.M.PennypackerH.S. (1980). Strategies and tactics of human behavioral research. Hillsdale, NJ: Laurence Erlbaum.
25.
KapustJ.A.NelsonR.O. (1984). Effects of the rate and spatial separation of target behaviors on observer accuracy and observer agreement. Behavioral Assessment, 6, 253–262.
26.
KazdinA.E. (1977). Artifact, bias, and complexity of assessment: The ABC's of reliability. Journal of Applied Behavior Analysis. 10, 141–150.
27.
KellyM.B. (1977). A review of the observational data collection and reliability procedures reported in theJournal of Applied Behavior Analysis. Journal of Applied Behavior Analysis, 10, 97–101.
28.
KentR.N.FosterS.L. (1977). Direct observational procedures: Methodological issues in naturalistic settings. In CimineroA.R.CalhounK.S.AdamsH.E. (Eds.), Handbook of behavioral assessment. New York: Wiley.
29.
KentR.N.O'LearyD.DietzA.DiamentC. (1979). Comparison of observational recordings in vivo, via mirror, and via television. Journal of Applied Behavior Analysis, 12, 517–522.
30.
KirkR.E. (1982). Experimental design: Procedures for the behavioral sciences (2nd ed.). Belmont, CA: Brooks/Cole.
31.
LovaasO.I.KoegelR.SimmonsJ.Q.LongJ.S. (1973). Some generalization and follow up measures on autistic children in behavior therapy. Journal of Applied Behavioral Analysis, 6, 131–166.
32.
MacLeanW.E.TappJ.T.JohnsonW.L. (1985). Alternative methods and software for calculating interobserver agreement for continuous observation data. Journal of Psychopathology and Behavioral Assessment, 7, 75–73.
33.
MadsenC.H.Jr.MadsenC.K. (1974). Teaching/discipline: A positive approach for educational development (2nd ed.). Boston: Allyn & Bacon.
34.
MashE.J.McElweeJ. (1974). Situational effects on observer accuracy: Behavioral predictability, prior experience, and complexity of coding categories. Child Development, 45, 367–377.
35.
MilarC.HawkinsR. (1976). Distorted results from the use of internal recording techniques. In BringhamT.HawkinsR.ScottJ.LaughlinT. (Eds.), Behavior analysis in education: Self-control and reading. Dubuque, IA: Kendall/Hunt.
36.
O'LearyK.D.KentR.N. (1973). Behavior modification for social action: Research tactics and problems. In HamerlynckL.A.HandyL.C.MashE.J. (Eds.), Behavior change: Methodology, concepts, and practice. Champaign, IL: Research Press.
37.
PowellJ.MartindaleA.KulpS. (1975). An evaluation of time-sample measures of behavior. Journal of Applied Behavior Analysis, 8, 463–469.
38.
PowellJ.MartindaleB.KulpS.MartindaleA.BaumanR. (1977). Taking a closer look: Time-sampling and measurement error. Journal of Applied Behavior Analysis, 10, 325–332.
39.
ReidJ.B. (1970). Reliability assessment of observation data: A possible methodological problem. Child Development, 41, 1143–1150.
ReppA.C.KarshK.L.Van AckerR.HarmanM.L.FelceD. (1989). A computer-based system for collecting and analyzing observational data. Journal of Special Education Technology, 9(4), 207–217.
42.
RiethH.HausG.J.BahrC.M. (1989). The use of portable microcomputers to collect student and teacher behavior data. Journal of Special Education Technology, 9, 190–199.
43.
Sanson-FisherR.W.PooleA.D.DunnJ. (1980). An empirical method for determining an appropriate interval length for recording behavior. Journal of Applied Behavior Analysis, 13, 493–500.
44.
SaudargasR.A.BunnR.D. (1989). A handheld computer system for classroom observation. Journal of Special Education Technology, 9, 200–206.
45.
SkindrudK. (1973), Field evaluation of observer bias under overt and covert monitoring. In HamerlynckL.A.HandyL.C.MashE.J. (Eds.), Behavior change: Methodology, concepts, and practice (pp. 97–117). Champaign, IL: Research Press.
46.
SmithJ.B.MadsenC.H.CipaniE.C. (1981). The effects of observational session length, method of recording, and frequency of teacher behavior on reliability and accuracy of observational data. Behavior Therapy, 12, 565–569.
47.
ThorneR.M.SchlottmannR.S.SeayB. (1969). Single animal observation versus pair observation inMacaca vius. Journal of Genetic Psychology, 115, 17–32.
48.
Van AckerR.BushJ.R.GettyJ.E.GrantS.H. (1991). A software package for the intermittent time sampling of behavior. Manuscript submitted for publication.
49.
WasikB.H.LovenM.D. (1980). Classroom observational data: Sources of inaccuracy and proposed solutions. Behavioral Assessment, 2, 211–227.
50.
WatzlawickP.Helmick-BeavinJ.JacksonD.D. (1967). Pragmatics of human communication: A study of interactional patterns, pathologies, and paradoxes. New York; W.W. Norton & Co.
51.
WeinrottM.JonesR.R. (1985). Overt versus covert assessment of observer reliability. Child Development, 55, 1125–1137.
52.
WildmanB.G.EricksonM.T. (1977). Methodological problems in behavioral observation. In ConeJ.D.HawkinsR.P. (Eds.), Behavioral assessment: New directions in clinical psychology. New York: Brunner/Mazel.
53.
WilsonC.C.RobertsonS.J.HerlongL.H.HaynesS. (1979). Vicarious effects of time out in the modification of aggression in the classroom. Behavior Modification, 3, 97–111.
54.
YarrowM.R.WaxlerC.Z. (1979). Observing interaction: A confrontation with methodology. In CravinsR.B. (Ed.). The analysis of social interactions, methods, issues, and illustrations. Hillside, NJ: Erlbaum.
