Observational data collected in ecologically valid measurement contexts are likely to be influenced by contextual factors irrelevant to the research question. Using multiple sessions and raters often improves the stability of scores for variables from such contexts. Generalizability (G) theory can be used to give researchers important information about sources and amounts of error in their observed scores. G theory also can be used to estimate the number of raters and sessions necessary to obtain reliable scores by conducting a decision (D) study. A general overview of G theory is provided in the present paper and its potential application for one type of a two-dimensional, fully crossed observational design is illustrated, using conversational language samples obtained from preschoolers with grammatical and phonological impairments. The paper includes the Internet link to an Excel ™ spreadsheet, which calculates values necessary for the type of G and D studies exemplified here.
Get full access to this article
View all access options for this article.
References
1.
Brennan, R.L. (2001). Generalizability theory. New York: Springer.
2.
Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory . Belmont, CA: Wadsworth Group.
3.
Cohen, J., Cohen, P., West, S.G., & Aiken, L.S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
4.
Cronbach, L.J., Gleser, G.C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability of scores and profiles. New York: John Wiley.
5.
Cronbach, L.J., Rajaratnam, N.R., & Gleser, G.C. (1963). Theory of generalizability: A liberalization of reliability theory. The British Journal of Statistical Psychology , 16, 137-163.
6.
Cronbach, L.J., & Shavelson, R.J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64, 391-418.
7.
Eason, S. (1991). Why generalizability theory yields better results than classical test theory: A primer with concrete examples. In B. Thompson (Ed.), Advances in educational research: Substantive findings, methodological developments (Vol. 1, pp. 83-98). Greenwich, CT: JAI Press.
8.
Evans, J.L. (2001). An emergent account of language impairments in children with SLI: Implications for assessment and intervention. Journal of Communication Disorders, 34, 39-54.
9.
Feldt, L.S., & Brennan, R.L. (1989). Reliability. In R. L. Linn (Ed.), Educational measurement (3rd. ed., pp. 105-147). New York: Macmillan .
10.
Fudala, J., & Reynolds, W. (1986). Arizona Articulation Proficiency Scale (2nd ed.). Los Angeles: Western Psychological Services.
11.
Kent, R. (1993). Speech-intelligibility and communication competence in children. In A. Kaiser & D. Gray (Eds.), Enhancing children's communication: Research foundations for intervention (pp. 223-242). Baltimore: Brookes.
12.
Knowles, E.S., & Condon, C.A. (2000). Does the rose still smell as sweet? Item variability across test forms and revisions. Psychological Assessment , 12, 245-252.
13.
Lord, F.M. (1955). Estimating test reliability. Educational and Psychological Measurement, 15, 325-336.
14.
Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
15.
McWilliam, R.A., & Ware, W.B. (1994). The reliability of observations of young children's engagement: An application of generalizability theory. Journal of Early Intervention, 18, 34-47.
16.
Miller, J. (1981). Assessing language production in children: Experimental procedures. Boston: Allyn and Bacon .
17.
Miller, J., & Chapman, R. (1993). SALT: Systematic analysis of language transcripts . Madison, WI: University of Wisconsin .
18.
O'Brian, N., O'Brian, S., Packman, A., & Onslow, M. (2003). Generalizability theory: Assessing reliability of observational data in the communication sciences. Journal of Speech Language and Hearing Research, 46, 711-7.
19.
Odom, S.L., Favzza, P.C., Brown, W.H., & Horn, E.M. (2000). Approaches to understanding the ecology of early childhood environments for children with disabilities. In T. Thompson, D. Felce , & F. J. Symons (Eds.), Behavioral observation: Technology and application in developmental disabilities (pp. 193-214). Baltimore: Brookes .
20.
Parkes, J. (2000). The relationship between the reliability and cost of performance assessments. Educational Policy Analysis Archives , 8, 16, Retrieved December 2, 2005, from http://epaa.asu.edu/epaa/v8n16/
21.
Rogosa, D., & Ghandour, G. (1991). Statistical models for behavioral observations . Journal of Educational Statistics, 16, 157-252.
22.
Scarsellone, J.M. (1998). Analysis of observational data in speech language research using Generalizability theory. Journal of Speech Language and Hearing Research, 41, 1341-1347.
23.
Shavelson, R.J., & Webb, N.M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
24.
Stanley, J.C. (1971). Reliability. In R. L. Thorndike (Ed.), Educational measurement (pp. 356-442). Washington, D. C.: American Council on Education.
25.
Thompson, B. (1991). Review of Generalizability theory: A primer by R. J. Shavelson & N. M. Webb. Education and Psychological Measurement, 51, 1069-1075.
26.
Thompson, B. (Ed.). (2003). Score reliability: Contemporary thinking on reliability issues. Newbury Park, CA: Sage.
27.
Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement , 60, 174-195.
28.
Vacha-Haase, T., Henson, R.K., & Caruso, J.C. (2002). Reliability generalization: Moving towards improved understanding and use of score reliability. Educational and Psychological Measurement, 62, 562-569.
29.
Vygotsky, L.S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.
30.
Webb, N.M., Rowley, G.L., & Shavelson, R.J. (1988). Using generalizability theory in counseling and development. Measurement and Evaluation in Counseling and Development, 21, 81-90.
31.
Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
32.
Yoder, P.J., Gardner, E., & Camarata, S. (2005). Treatment effect on and predictors of speech intelligibility and length of utterance in children with specific language and intelligibility impairments. Journal of Early Intervention, 28, 34-49.
33.
Zimmerman, I., Steiner, B., & Pond, R. (1992). Preschool Language Scale (3rd ed.). San Antonio: Psychological Corporation.
