The Reliability of Observations of Young Children's Engagement

Abstract

The reliability of measurement of how young children spend their time has traditionally been computed in terms of interobserver exact agreement This study sought to apply generalizability theory to the measurement of engagement Forty-seven young children, 15 of whom had disabilities, were observed four times in their child care setting. Types and levels of engagement were coded by three raters. Using ANOVA procedures for determining the relative contribution of different sources of error, a fully crossed (with subjects) two-facet (sessions, raters) generalizability (G) study was employed. The nine outcome measures were four types and five levels of engagement Results showed that raters accounted for less than 2% of the variance in the error of the scores, while sessions accounted for most of the variance other than between subject variance. The outcome measures proved to vary in their reliability robustness. The G study was followed by a decision (D) study to determine the levels of the facets that would be required to achieve a generalizability coefficient of .80. The number of sessions could be realistically increased to achieve this aim, but the increase in “dependability” achieved with the addition of each rater was minimal. Conclusions are drawn about the importance of assessing more than one source of error, raters, in observational research Overall, molecular methods were determined to be relatively unstable for measuring the molar construct of engagement, but certain engagement outcomes were stable across sessions and raters.

Get full access to this article

View all access options for this article.

References

Algina, J. (1989). Elements of classical reliability theory and generalizability theory. In B. Thompson (Ed.), Advances in social science methodology: A research annual (Vol. 1). Greenwich, CT: JAI Press Inc.

Barker, R.G. , & Wright, H. (1955). Midwest and its children. New York: Harper & Row.

Berk, R.A. (1979). Generalizability of behavioral observations: A clarification of interobserver agreement and interobserver reliability. American Journal of Mental Deficiency, 83, 460–472.

Burt, C. (1955). Test reliability estimated by analysis of variance. British Journal of Statistics, 8, 103–118.

Cronbach, L.J. , Gleser, G.C. , Nanda, H. , & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of gener-alizability for scores and profiles. New York: Wiley.

Dunst, C.J. , & McWilliam, R.A. (1988). Cognitive assessment of multiply handicapped young children. In T.D. Wachs & R. Sheehan (Eds.), Assessment of young developmentally disabled children (pp. 213–238). New York: Plenum Press.

Evertson, C.M. , & Green, J.L. (1986). Observation as inquiry and method. In M. C. Wittrock (Ed.), Handbook of research on teaching (3rd Ed.) (pp. 162–213). New York: Macmillan.

Goodwin, L.D. , & Goodwin, W.L. (1991). Using generalizability theory in early childhood special education. Journal of Early Intervention, 15, 193–204.

Herbert, J. , & Attridge, C. (1975). A guide for developers and users of observational systems and manuals. American Educational Research Journal, 12, 1–20.

10.

Hoge, R.D. (1985). The validity of direct observation measures of pupil classroom behavior. Review of Educational Research, 55, 469–483.

11.

Lindquist, E.F. (1953). Design and analysis of experiments in psychology and education. Boston: Houghton-Mifflin.

12.

McWilliam, R.A. , Trivette, C.M. , & Dunst, C.J. (1985). Behavior engagement as a measure of the efficacy of early intervention. Analysis and Intervention in Developmental Disabilities, 5, 33–45.

13.

Medley, D.M. , & Mitzel, H.E. (1963). Measuring classroom behavior by systematic observation. In L. Gage (Ed.), Handbook of research on teaching (pp. 247–328). Chicago: Rand McNally.

14.

Mitchell, S.K. (1979). Interobserver agreement, reliability, and generalizability of data collected in observational studies. Psychological Bulletin, 86, 376–390.

15.

Mudford, O.C. , Beale, I.L. , & Singh, N.N. (1990). The representativeness of observational samples of different durations. Journal of Applied Behavior Analysis, 23, 323–331.

16.

Newborg, J. , Stock, J.R. , Wnek, L. , Guidubaldi, J. , & Svinicki, J. (1984). Battelle developmental inventory. Allen, TX: DLM Teaching Resources.

17.

Rich, H.L. , & Ross, S.M. (1989). Students' time on learning tasks in special education. Exceptional Children, 55, 508–515.

18.

Shavelson, R.J. , & Webb, N.M. (1991). Generalizability theory: A primer. Newbury Park, CA: SAGE.

19.

Sparling, J. , & Lewis, I. (1979). Learningames for the first three years. New York: Berkley Books.

20.

Sparling, J. , & Lewis, I. (1984). Learningames for threes and fours. New York: Berkley Books.

21.

Stodolsky, S. (1984). How children find something to do in preschools. Genetic Psychology Monographs, 90, 245–303.

22.

Thorndike, R.L. (1947). Research problems and techniques. Report No. 3, AAF Aviation Psychology Program Research Reports. Washington, DC: U.S. Government Printing Office.

23.

Webber, L. , Thompson, B. , & Berenson, G. (1986). Estimating the reliability of dynamic variables requiring rater judgment: A generalizability paradigm. ERIC Document #ED286902. Paper presented at the Mid-South Educational Research Association meeting, Memphis, TN, November 19, 1986.

24.

Wolery, M. (1991). Instruction in early childhood special education: “Seeing through a glass darkly … Knowing in part” Exceptional Children, 58, 127–135.

25.

Wolery, M. , Strain, P.S. , & Bailey, D.B. (in press). Applying the framework of developmentally appropriate practice to children with special needs. In S. Bredekamp & T. Rosegrant (Eds.), Reaching potentials: Curriculum and assessment for 3 to 8 year olds. Washington, DC: NAEYC.