Time-sampling techniques are popular among school practitioners and researchers alike when conducting observations of student behavior. Despite their popularity, the psychometric properties of such observation data have not received much attention, likely given that both procedures and targets vary widely across individual coding systems. A systematic literature review was therefore conducted to provide insight into how various procedural aspects may influence the psychometric properties of time-sampling data. Specifically, the influences of the number of target behaviors, time-sampling method, interval length, observation length, and number of observations on the psychometric properties of time-sampling data are explored. Both implications for practice and future research are discussed.
BrieschA. M.ChafouleasS. M.Riley-TillmanT. C. (2010). Generalizability and dependability of behavior assessment methods to estimate academic engagement: A comparison of systematic direct observation and Direct Behavior Rating. School Psychology Review, 39, 408–421.
2.
BrieschA. M.VolpeR. J.FergusonT. D. (2013). The influence of student characteristics on the dependability of behavioral observation data. School Psychology Quarterly, 29, 171–181. doi:10.1037/spq0000042
3.
BrulleA. R.ReppA. C. (1984). An investigation of the accuracy of momentary time sampling procedures with time series data. British Journal of Psychology, 75, 481–485.
4.
ConeJ. (1977). The relevance of reliability and validity for behavioral assessment. Behavior Therapy, 8, 411–426.
5.
CrockerL.AlginaJ. (2006). Introduction to classical and modern test theory. Mason, OH: Cengage Learning.
6.
CronbachL. J. (1955). Processes affecting scores on “understanding others” and “assumed similarity.”Psychological Bulletin, 52, 177–193.
7.
DollB.ElliottS. (1994). Research methods representativeness of observed preschool social behaviors: How many data are enough?Journal of Early Intervention, 18, 227–238.
8.
DorseyB. L.NelsonR. O.HayesS. C. (1986). The effects of code complexity and of behavioral frequency on observer accuracy and interobserver agreement. Behavioral Assessment, 8, 349–363.
9.
FergusonT.BrieschA.VolpeR.DanielsB. (2012). The influence of observation length on the dependability of data. School Psychology Quarterly, 27, 187–197.
10.
GreenwoodC. (1996). The case for performance-based instructional models. School Psychology Quarterly, 11, 283–296.
11.
GunterP. L.VennM. L.PatrickJ.MillerK. A.KellyL. (2003). Efficacy of using momentary time samples to determine on-task behavior of students with emotional/behavioral disorders. Education and Treatment of Children, 26, 400–412.
12.
HintzeJ. M.MatthewsW. J. (2004). The generalizability of systematic direct observations across time and setting: A preliminary investigation of the psychometrics of behavioral observation. School Psychology Review, 33, 258–270.
13.
HintzeJ. M.VolpeR.ShapiroE. (2008). Best practices in systematic direct observation of student behavior. In ThomasA.GrimesJ. (Eds.), Best practices in school psychology V (pp. 993–1006). Bethesda, MD: National Association of School Psychologists.
14.
HuttS. J.HuttC. (1970). Direct observation and measurement of behavior. Springfield, IL: Charles C. Thomas.
15.
JohnsonA. H.ChafouleasS. M.BrieschA. M. (2017). Dependability of data derived from time sampling methods with multiple observation targets. School Psychology Quarterly, 32, 22–34.
16.
KarweitN. L.SlavinR. E. (1982). Time-on-task: Issues of timing, sampling, and definition. Journal of Educational Psychology, 74, 844–851.
17.
LeffS. S.LakinR. (2005). Playground-based observational systems: A review and implications for practitioners and researchers. School Psychology Review, 34, 475–489.
18.
McConaughyS.AchenbachT. M. (2009). Manual for the ASEBA Direct Observation Form. Burlington, Ontario, Canada: Research Center for Children, Youth, & Families, University of Vermont.
19.
McConaughyS.RitterD. (2008). Best practices in multimethod assessment of emotional and behavioral disorders. In ThomasA.GrimesJ. (Eds.), Best practices in school psychology V (pp. 697–715). Bethesda, MD: National Association of School Psychologists.
20.
McWilliamR. A.WareW. B. (1994). The reliability of observations of young children’s engagement: An application of generalizability theory. Journal of Early Intervention, 18, 34–47.
21.
MurphyG.GoodallE. (1980). Measurement error in direct observation: A comparison of common recording methods. Behaviour Research and Therapy, 18, 147–150.
22.
OlswangL. B.SvenssonL.CogginsT. E.BeilinsonJ. S.DonaldsonA. L. (2006). Reliability issues and solutions for coding social communication performance in classroom settings. Journal of Speech, Language, and Hearing Research, 49, 1058–1071.
23.
PowellJ.MartindaleA.KulpS. (1975). An evaluation of time-sample measures of behavior. Journal of Applied Behavior Analysis, 8, 463–469.
24.
PowellJ.MartindaleA.KulpS.MartindaleA.BaumanR. (1977). Taking a close look: Time sampling and measurement error. Journal of Applied Behavior Analysis, 10, 325–332.
25.
ReynoldsC.KamphausR. W. (2015). Behavior Assessment System for Children, Third Edition. San Antonio, TX: Pearson.
26.
SaudargasR.FellersG. (1986). State-event classroom observation system (SECOS). Unpublished manuscript, University of Tennessee, Knoxville.
27.
ShapiroE. S. (1996). Academic skills problems workbook. New York, NY: Guilford Press.
28.
ShapiroE. S.HeickP. F. (2004). School psychologist assessment practices in the evaluation of students referred for social/behavioral/emotional problems. Psychology in the Schools, 41, 551–561.
29.
ShavelsonR. J.WebbN. M. (1991). Generalizability theory. Newbury Park, CA: SAGE.
30.
SilvaF. (1993). Psychometric foundations of behavioral assessment. Newbury Park, CA: SAGE.
31.
SteegeM.WatsonT. (2008). Best practices in functional behavioral assessment. In ThomasA.GrimesJ. (Eds.), Best practices in school psychology V (pp. 337–348). Bethesda, MD: National Association of School Psychologists.
32.
VolpeR. J.DiPernaJ.HintzeJ.ShapiroE. (2005). Observing students in classroom settings: A review of seven coding schemes. School Psychology Review, 34, 454–474.
33.
VolpeR. J.McConaughyS. H.HintzeJ. M. (2009). Generalizability of classroom behavior problem and on-task scores from the Direct Observation Form. School Psychology Review, 38, 382–401.
34.
WhitcombS. A.MerrellK. (2012). Behavioral, social, and emotional assessment of children and adolescents (4th ed.). New York, NY: Routledge.
