The reliabilities and errors of judgmental-based assessment outcomes derived through use of the System to Plan Early Childhood Services (SPECS; Bagnato & Neisworth, 1990) were investigated within the framework of the generalizability theory of measurement (Cronbach, Gleser, Nanda, & Rajaratnam, 1972). The SPECS was designed explicitly to produce collaborative/team decisions with regard to child development status and need and routinely involves more than one rater. Overall (with the exception of the unusual situation in which only one rater was employed), scores from SPECS generally had adequate levels of reliability under the interpretation scenarios investigated.
Get full access to this article
View all access options for this article.
References
1.
Bagnato, S. J., & Neisworth, J. T. (1990). SPECS: System to Plan Early Childhood Services. Circle Pines, MN: American Guidance Service.
2.
Bagnato, S. J., & Neisworth, J. T. (1991). Assessment for early intervention: Best practices for professionals. New York: Guilford Press.
3.
Bagnato, S. J., & Neisworth, J. T. (1992). Pennsylvania preschool integration initiative (PAPII). Harrisburg, PA: Department of Education, Commonwealth of Pennsylvania.
4.
Brennan, R. L. (1983). Elements of generalizability theory. Iowa City, IA: ACT Publications.
5.
Brennan, R. L. (1984). Estimating the dependability of the scores. In R. A. Berk (Ed.), A guide to criterion-referenced test construction. Baltimore: Johns Hopkins University Press.
6.
Crick, J. E., & Brennan, R. L. (1983). GENOVA: A generalized analysis of variance system. Iowa City, IA: ACT Publications.
7.
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: John Wiley.
8.
Gillmore, G. M., Kane, M. T., & Naccarato, R. W. (1978). The generalizability of student ratings of instruction: Estimation of the teacher and course components. Journal of Educational Measurement, 15, 1-13.
9.
Goodwin, L. D., & Goodwin, W. L. (1991). Using generalizability theory in early childhood special education. Journal of Early Intervention, 15, 193-204.
10.
Goodwin, L. D., Sands, D. J., & Kazleski, E. B. (1991). Estimating interviewer reliability of interview schedules used in special education research. Journal of Special Education, 25, 73-89.
11.
Harmann, D. P. (1982). Assessing the dependability of obervational data. In D. P. Hartmann (Ed.), Using observers to study behavior (pp. 51-66). San Francisco: Jossey-Bass.
12.
Hayes, S. C., Nelson, R. O., & Jarrett, R. B. (1987). The treatment utility of assessment: A functional approach to evaluating assessment quality. American Psychologist, 42, 963-974.
13.
Rosenfield, S., & Reynolds, M. C. (1990). Mainstreaming school psychology: A proposal to develop and evaluate alternative assessment methods and intervention strategies. School Psychology Quarterly, 5, 55-65.
14.
Shavelson, R. J., Webb, N. M., & Burstein, L. (1985). The measurement of teaching. In M. C. Wittrock (Ed.), Handbook for research on teaching (3rd ed., pp. 50-91). New York: Macmillan.
15.
Shavelson, R. J., Webb, N. M., & Rowley, G. L. (1989). Generalizability theory. American Psychologist, 44, 922-932.
16.
Suen, H. K. (1988). Agreement, reliability, accuracy, and validity: Toward a clarification. Behavioral Assessment, 10, 1-22.
17.
Suen, H. K., & Ary, D. (1989). Analyzing quantitative behavioral observation data. Hillsdale, NJ: Erlbaum.
