Reliability of direct observation outcomes ensures the results are consistent, dependable, and trustworthy. Typically, reliability of direct observation measurement approaches is assessed using interobserver agreement (IOA) and the calculation of observer agreement (e.g., percentage of agreement). However, IOA does not address intraobserver agreement, or the withinrater agreement, analogues to internal consistency. One approach to assess both IOA and intraobserver agreement is generalizability theory. This paper describes generalizability theory and provides a case study example of using generalizability theory with direct observation measures in emotional and/or behavioral disorders research.
Get full access to this article
View all access options for this article.
References
1.
AdamsonR., & WachsmithS. (2014). A review of direct observation reliability within the past decade for students with emotional and behavioral disorders.Behavioral Disorders, 39, 181–189.
2.
AlldayR. A., Hinkson-LeeK., HudsonT., Neilsen-GattiS., KleinkeA., & RusselC. S. (2012). Training general educators to increase behavior-specific praise: Effects on students with EBD.Behavioral Disorders, 37, 87–98.
3.
BaerD. M., HarrisonR., FradenburgL., PetersenD., & MillaS. (2005). Some pragmatics in the valid and reliable recording of directly observed behavior.Research on Social Work Practice, 15, 440–451. doi:10.1177/1049731505279127
4.
BaerD. M., WolfM. M., & RisleyT. R. (1968). Some current dimensions of applied behavior analysis.Journal of Applied Behavior Analysis, 1, 91–97. doi:10.1901/jaba.1968.1-91
5.
BaughF. (2002). Correcting effect sizes for score reliability: A reminder that measurement and substantive issues are linked inextricably.Educational and Psychological Measurement, 62, 254–263. doi:10.1177/0013164402062002004
6.
BennerG.J., NelsonJ. R.SandersE. A., & RalstonN. C. (2012). Behavior interventions for students with externalizing behavior problems: Primary-level standard protocol.Exceptional Children, 78, 181–198.
7.
BradleyR., DoolittleJ., & BartolottaR. (2008). Building on the data and adding to the discussion: The experiences and outcomes of students with emotional disturbance.Journal of Behavioral Education, 17(1), 4–23.
8.
BradleyR., HendersonK., & MonforeD. A. (2004). A national perspective on children with emotional disorders.Behavioral Disorders, 29(3), 211–223.
9.
BrennanR. L. (2001). Generalizability theory.New York: Springer-Verlag.
10.
BrieschA. M., ChafouleasS. M., & Riley-TillmanT. C. (2010). Generalizability and dependability of behavior assessment methods to estimate academic engagement: A comparison of systematic direct observation and Direct Behavior Rating.School Psychology Review, 39, 408–421.
11.
BrucknerC. T., YoderP. J., & McWilliamR. A. (2006). Generalizability and decision studies: An example using conversational language samples.Journal of Early Intervention, 28, 139–153.
12.
ChristT. J., & HintzeJ. M. (2007). Psychometric considerations when evaluating response to intervention. In JimersonS. R., BurnsM. K., & BanDerHeydenA. M. (Eds.), Handbook of response to intervention: The science and practice of assessment and intervention (pp. 93–105). New York: Springer.
13.
ConeJ. D. (1977). The relevance of reliability and validity for behavioral assessment.Behavior Therapy, 8, 411–426.
14.
ConroyM. A., SutherlandK. S., AlginaJ. J., WilsonR. E., MartinezJ. R., & WhalonK. J. (2014). Measuring teacher implementation of the BEST in CLASS intervention program and corollary child outcomes.Journal of Emotional and Behavioral Disorders. Advanced online publication. doi:10.1177/1063426614532949
15.
CooperJ. O., HeronT. E., & HewardW. L. (2007). Applied behavior analysis (2nd ed.). Upper Saddle River, NJ: Pearson.
16.
CrockerL., & AlginaJ. (2008). Introduction to classical and modern test theory.Independence, KY: Cengage Learning.
17.
CronbachL. J., GleserG. C., NandaH., & RajaratnamN. R. (1972). The dependability of behavioral measurements: Theory of generalizability of scores and profiles.New York: John Wiley.
18.
CronbachL. J., RajaratnamN. R., & GleserG. C. (1963). Theory of generalizability: A liberalization of reliability theory.The British Journal of Statistical Psychology, 16, 137–163.
19.
EasonS. (1991). Why generalizability theory yields better results than classical test theory: A primer with concrete examples. In ThompsonB. (Ed.), Advances in educational research: Substantive findings, methodological developments, (Vol. 1, pp. 83–98). Greenwich, CT: JAI Press.
20.
EndlerN. S. (1966). Estimating variance components from mean squares for random and mixed effects analysis of variance models.Perceptual and Motor Skills, 22, 559–570.
21.
FergusonT. D., BrieschA. M., VolpeR. J., & DanielsB. (2012). The influence of observation length on the dependability of data.School Psychology Quarterly, 27, 187–197. doi:10.1037/spq0000005
22.
GerstenR., BakerS. K., HaagerD., & GravesA. W. (2005). Exploring the role of teacher quality in predicting outcomes for first-grade English learners: An observational study.Remedial and Special Education, 26, 197–206.
23.
HartmannD. P. (1977). Consideration in the choice of interobserver reliability estimates.Journal of Applied Behavior Analysis, 10, 103–116.
24.
HaydonT., ConroyM. A., ScottT. M., SindelarP. T., BarberB. R., & OrlandoA. M. (2009). A comparison of three types of opportunities to respond on student academic and social behaviors.Journal of Emotional and Behavioral Disorders, 18, 27–40.
25.
HendersonK., KleinS., GonzalezP., & BradleyR. (2005). Teachers of children with emotional disturbance: A national look at preparation, teaching conditions, and practices.Behavioral Disorders, 31(1), 6–17.
26.
HillH. C., CharalambosY. C., & KraftM. A. (2012). When rater reliability is not enough: Teacher observation systems and a case or the generalizability study.Educational Researcher, 41, 56–64. doi:10.3102/0013189X12437203
27.
HintzeJ. M. (2005). Psychometrics of direct observation.School Psychology Review, 34, 507–519.
28.
HintzeJ. M., & MathewsW. J. (2004). The generalizability of systematic direct observations across time and setting: A preliminary investigation of the psychometrics of behavioral observations.School Psychology Review, 33, 258–270.
29.
IovannoneR., GreenbaumP. E., WangW., KincaidD., DunlapG., & StrainP. (2009). Randomized controlled trial of the Prevent-Teach-Reinforce (PTR) tertiary intervention for students with problem behaviors: Preliminary outcomes.Journal of Emotional and Behavioral Disorders, 17, 213–225. doi:10.1177/1063426609337389
30.
KazdinA. E. (2011). Single-case research designs: Methods for clinical and applied settings (2nd ed.). New York: Oxford University Press.
31.
LewisT. J., Lewis-PalmerT., StichterJ., & NewcomerL. L. (2004). Applied behavior analysis and the education and treatment of students with emotional and behavioral disorders. In RutherfordR., QuinnM. M., & MathurS. (Eds.), Handbook of research in behavioral disorders (pp. 523–545). New York: Guilford.
32.
MacSuga-GageA. S., & SimonsenB. (in press). A systematic review of teacher-directed opportunities to respond.Education and Treatment of Children.
33.
MashburnA. J., DownerJ. T., RiversS. E., BrackettM. A., & MartinezA. (2014). Improving the power of an efficacy study of a social and emotional learning program: Application of generalizability theory to the measurement of classroom-level outcomes.Prevention Science, 15, 146–155. doi:10.1007/s11121-012-0357-3
34.
McLeskeyJ., LandersE., WilliamsonP., & HoppeyD. (2012). Are we moving toward educating students with disabilities in less restrictive environments.Journal of Special Education, 46, 131–140. doi:10.1177/0022466910376670
35.
McWilliamR. A., & WareW. B. (1994). The reliability of observations of young children's engagement: An application of generalizability theory.Journal of Early Intervention, 18, 34–47.
36.
PiantaR. C., La ParoK., & HamreB. K. (2008). Classroom assessment scoring system (CLASS).Baltimore, MD: Paul H. Brookes.
37.
RaudenbushS. W., & SadoffS. (2008). Statistical inference when classroom quality is measured with error.Journal of Research on Educational Effectiveness, 1, 138–154. doi:10.1080/19345740801982104
38.
RowleyG. (1978). The relationship of reliability in classroom research to the amount of observation: An extension of the Spearman-Brown formula.Journal of Educational Measurement, 15, 165–180.
39.
ScottT. M., AlterP. J., & HirnR. G. (2011). An examination of typical classroom context and instruction for students with and without behavioral disorders.Education and Treatment of Children, 34, 619–641.
40.
ShavelsonR. J., & WebbN. M. (1991). Generalizability theory: A primer.Newbury Park, CA: Sage.
41.
SimonsenB., FairbanksS., BrieschA. M., MeyersD., & SugaiG. (2008). Evidence-based practices in classroom management: Considerations for research to practices.Education and Treatment of Children, 31(3), 351–380.
42.
SnyderP. A., HemmeterM. L., FoxL., BishopC. C., & MillerM. D. (2013). Developing and gathering psychometric evidence for a fidelity instrument: The Teaching Pyramid Observation Tool-pilot version.Journal of Early Intervention, 35, 150–172. doi:10.1177/1053815113516794
43.
SuenH. K., & AryD. (1989). Analyzing quantitative behavioral observation data.Hillsdale, NJ: Lawrence Erlbaum Associates.
44.
SugaiG., SpragueJ. R., HornerR. H., & WalkerH. M. (2000). Preventing school violence: The use of office discipline referrals to assess and monitor school-wide discipline interventions.Journal of Emotional and Behavioral Disorders, 8, 94–102.
45.
TappJ. (1995). Multiple options for observation in experimental studies.Nashville, TN: Vanderbilt Kennedy Center.
46.
ThompsonB., & Vacha-HaaseT. (2000). Psychometrics is datametrics: The test is not reliable.Educational and Psychological Measurement, 60, 174–195. doi:10.1177/0013164400602002
47.
VolpeR. J., McConaughyS. H., & HintzeJ. M. (2009). Generalizability of classroom behavior problem and on-task scores from the Direct Observation Form.School Psychology Review, 38, 382–401.
48.
WagnerM., KutashK., DuchnowskiA. J., EpsteinM. H., & SumiW. C. (2005). The children and youth we serve: A national picture of the characteristics of students with emotional disturbances receiving special education.Journal of Emotional and Behavioral Disorders, 13(2), 79–96.
