This article offers an alternative methodology for practitioners and researchers to use in establishing interrater reliability for testing purposes. The majority of studies on interrater reliability use a traditional methodology where by two raters are compared using a Pearson product-moment correlation. This traditional method of estimating interrater reliability uses a correlation of the two raters' scores for the same subject. This study used an observer-rater paradigm in which 20 observers' scores were compared to the scores of an expert rater. Results bring into question the common practice of examining interrater reliability. The correlation of two raters may not be a reliable way of determining if every evaluator using the instrument will obtain scores that accurately report the developmental status of young children.
Get full access to this article
View all access options for this article.
References
1.
Aycock, T. (1993). Is it correct to say, "The test is reliable?" Review of literature and implications for research practice. ERIC Document Reproduction Service No. ED 355 275. Retrieved March 3, 2002 , from www.ericae.net/scripts/
2.
Bagnato, S.J., Neisworth, J.T., & Munson, S.M. (1989). Linking assessment and early intervention: An authentic curriculum-based approach. Baltimore, MD: Paul H. Brookes Publishing Company.
3.
Baltimore County Infants and Toddlers Task Force on Assessment. (1997). Best practices in early childhood evaluations. Towson, MD: Author .
4.
Barnes, K.E. (1982). Preschool screening: The measurement and prediction of children at-risk. Springfield, IL: Thomas.
5.
Bayley, N. (1993). Bayley scales of infant development manual (2nd ed.). San Antonio, TX: The Psychological Corporation.
6.
Bellack, A.S., & Hersen, M. (1988). Behavioral assessment: A practical handbook (3rd ed.). New York: Pergamon Press.
7.
Brannigan, G.C., & Brunner, N.A. (1989). The modified version of the Bender-Gestalt Test for preschool and primary school children. Brandon, VT : Clinical Psychology Publishing.
8.
Bricker, D.D., Bailey, E.J., & Slentz, K. (1990). Reliability, validity, and utility of the Evaluation and Programming System for Infants and Young Children (EPS-I). Journal of Early Intervention, 14(2), 147-158.
9.
Bricker, D., Gentry, D., & Bailey, E. (1985). The evaluation and programming system for infants and young children: Assessment level I: Developmentally 1 month to 3 years. (EPS-I). Eugene: University of Oregon, Center on Human Development.
Chattin, S.H., & Bracken, B. (1989). School psychologists' evaluation of the K-ABC, McCarthy Scales, Stanford-Binet IV, and the WISC-R. Journal of Psychoeducational Assessment, 7, 112-130.
12.
Cook, M.J., Holder-Brown, L., Johnson, L.J., & Kilgo, J.L. (1989). An examination of the stability of the Bayley Scales of Infant Development with high-risk infants. Journal of Early Intervention, 13(1), 45-49.
13.
Falk, R., & Well, A.D. (1997). Many faces of the correlation coefficient. Journal of Statistics Education, 5(3), 219-238.
14.
Fan, F., & Chen, M. (1999). When inter-rater reliability is obtained from only part of a sample. ERIC Document Reproduction Service No. ED 430049. Retrieved March 3, 2002, from www.Ericae.net/scripts/
Fuller, G.B., & Vance, R. (1995). Interscorer reliability of the Modified Version of the Bender-Gestalt Test for Preschool and Primary school. Psychology in the Schools, 32, 264-266.
17.
Gay, L.R. (1992). Education research competencies for analysis and approach (4th ed.). New York : Merrill.
18.
Gibbs, E. D., & Teti, D. M. (Eds.). (1990). Interdisciplinary assessment of infants: A guide for early intervention professionals. Baltimore, MD: Paul H. Brookes Publishing Company.
19.
Glascoe, E.P., & Byme, K.E. (1993). The accuracy of three developmental screening tests. Journal of Early Intervention, 17(4), 368-379.
20.
Glover, M.E., Preminger, J.L., & Sanford, A.R. (1995). The Early Learning Accomplishment Profile for Developmentally Young Children: Birth to 36 months manual (rev. ed.). Lewisville, NC: Kaplan Press.
21.
Goldstein, G., & Hersen, M. (Eds.). (1990). Handbook for psychological assessment (2nd ed.). New York : Pergamon Press.
22.
Hardin, B.J., & Peisner-Feinberg, E.S. (2001). The Early Learning Accomplishment Profile (Early LAP) examiner's manual and reliability and validity technical report. Chapel Hill, NC: Chapel Hill Training Outreach Project, Inc., Kaplan Press.
23.
Horner, T.M. (1980). Test-retest and home-clinic characteristics of the Bayley Scales of Infant Development in nine- and fifteen-month old infants . Child Development, 51, 761-758.
24.
McLean, M., Bailey, D.B., & Wolery, M. (1996). Assessing infants and preschoolers with special needs (2nd ed.). Englewood Cliffs, NJ: Merrill.
25.
McLean, M., McCormick, K., & Baird, S. (1991). Concurrent validity of the Griffiths' Mental Development Scales with a population of children under 24 months. Journal of Early Intervention, 15(3), 338-344.
26.
McLean, M., McCormick, K., Bruder, M.B., & Burdg, N. (1987). An investigation of the validity and reliability of the Battelle Developmental Inventory with a population of children younger than 30 months with identified handicapping conditions. Journal of the Division of Early Childhood, 11, 238-246.
27.
Meisels, S.J., Henderson, L.W., Liau, F., Browning, K., & Have, T.T. (1993). New evidence for the effectiveness of the Early Screening Inventory. Early Childhood Research Quarterly, 8, 327-346.
28.
Meisels, S.J., & Wiske, M.S. (1983). The Early Screening Inventory. New York: Teachers College Press.
29.
Meisels, S.J., Wiske, M.S., & Tivan, T. (1984). Predicting school performance with the Early Screening Inventory. Psychology in the Schools, 21, 25-33.
30.
Miller, L.J. (1988). Miller Assessment for Preschoolers. San Antonio, TX: The Psychological Corporation.
31.
Minium, E.W., King, B.M., & Bear, G. (1993). Statistical reasoning in psychology and education . New York: John Wiley & Sons, Inc.
32.
Muris, P., Steememan, P., & Ratering, E. (1997). Brief report: Interrater reliability of the Psychoeducational Profile (PEP). Journal of Autism and Developmental Disorders , 27(5), 621-626.
33.
Newborg, J., Stock, J.R., Wnek, L., Guidubaldi, J., & Svinicki, J. (1984). The Battelle Developmental Inventory. Itasca, IL: The Riverside Publishing Co .
34.
Pulakos, E.D. (1986). The development of training programs to increase accuracy on different rating forms. Organizational Behavior and Human Decision Processes, 38, 76-91.
35.
Rogers, S., & D'Eugenio, D.B. (1996). Developmental programming for infants and young children: Assessment and application. Ann Arbor: The University of Michigan Press.
36.
Rogers, S.J., D'Eugenio, D.B., Brown, S.L., Donovan, C.M., & Lynch, E.W. (1981). Early Intervention Developmental Profile (EIDP) . Ann Arbor: The University of Michigan Press.
37.
Rudner, L.M. (1992). Reducing errors due to the use of judges. Practical Assessment, Research & Evaluation, 3(3). Retrieved March 2, 2002, from www.ericae.net/scripts/
38.
Sailor, W., & Mix, B.J. (1975). The TARC assessment system. Austin, TX: Pro-Ed Publishers.
39.
Sattler, J.M. (1988). Assessment of children (3rd ed.). San Diego, CA: Author.
40.
Sax, G. (1980) Principles of educational and psychological measurement and evaluation (2nd ed.). Belmont, CA: Wadsworth.
41.
Schopler, E., & Reichler, R.J. (1979). Psychoeducational Profile . Baltimore, MD: University Park Press.
42.
Schouten, P.G. (1992). Test review: Miller Assessment for Preschoolers . Diagnostique, 17(2), 145-153.
43.
Schouten, P.G., & Kirkpatrick, L.A. (1993). Artifact in the total score of the Miller Assessment for Preschoolers. Journal of Psychoeducational Assessment, 11, 67-71.
44.
Sheehan, R., & Snyder, S. (1990). Battelle Developmental Inventory and the Battelle Developmental Inventory Screening Test. Diagnostique, 15(2), 16-30.
45.
Simeonsson, R.J. (1981). Carolina Record of Individual Behavior. Chapel Hill, NC: Frank Porter Graham Child Development Center.
46.
Snyder, P., Lawson, S., Thompson, G., Stricklin, S., & Sexton, D. (1993). Evaluating the psychometric integrity of instruments used in early intervention research: The Battelle Developmental Inventory . Topics in Early Childhood Special Education, 13(2), 216-232.
47.
Ubersax, J.S. (2000). Statistical methods for rater agreement. Retrieved March 3, 2002, from www.ericae.net/scripts/
48.
Uzgiris, I.C., & Hunt, J.M. (1978). Ordinal Scales of Infant Psychological Development . Urbana: University of Illinois Press.
49.
Werner, E.E., & Bayley, N. (1966). The reliability of Bayley's revised Scale of Mental and Motor Development during the first years of life. Child Development, 37, 39-50.
50.
Ysseldyke, J.E., Thurlow, M.L., O'Sullivan, P., & Bursaw, R.A. (1986). Current screening and diagnostic practices in a state offering free preschool screening since 1977: Implications for the field. Journal of Psychoeducational Assessment, 4, 191-201.
51.
Zwick, R. (1988). Another look at interrater agreement. Psychological Bulletin,103, 374-378.
