The use of standards-based classroom assessments to test English learners' language proficiency is increasingly prevalent in the United States and many other countries. In a large urban school district in California, for example, a classroom assessment is used to make high-stakes decisions about English learners' progress from one level to the next, and as one of the criteria for reclassifying students as Fluent English Proficient. Yet many researchers have questioned the validity of using classroom assessments for making high-stakes decisions about students (Brindley, 1998; 2001; Rea-Dickins and Gardner, 2000). One way to investigate the validity of the inferences drawn from these assessments is to examine them in relation to other measures of the same ability. In this study, a multivariate analytic approach was used to examine the extent to which the English Language Development (ELD) Classroom Assessment measures the same constructs as the CELDT (California English Language Development Test), the statewide standardized test of English proficiency. Using confirmatory factor analysis of multitrait-multimethod data, this study investigates the construct validity of these measures by focusing on evidence of convergence, discrimination, and method effects longitudinally over three years. The study concludes that the evidence gathered via the ELD Classroom Assessment is consistent with that provided by the CELDT, the standardized measure.
Get full access to this article
View all access options for this article.
References
1.
Arkoudis, S. and O'Loughlin, K.2004: Tensions between validity and outcomes: Teacher assessment of written work of recently arrived immigrant ESL students. Language Testing21: 284โ304.
2.
Bachman, L.F., Davidson, F., Ryan, K. and Choi, I-C.1995: An investigation into the comparability of two tests of English as a foreign language: The CambridgeโTOEFL comparability study . Studies in Language Testing, 1. Cambridge: Cambridge University Press.
3.
Bachman, L.F. and Palmer, A.1981: The construct validation of the FSI oral interview. Language Learning31: 67โ86.
4.
โโโ 1982: The construct validation of some components of communicative proficiency. TESOL Quarterly16: 449โ65.
5.
Bae, J. and Bachman, L.F.1998: A latent variable approach to listening and reading: Testing factorial invariance across two groups of children in the Korean/English two-way immersion program. Language Testing15: 380โ414.
6.
Bentler, P.M.1985โ2005: EQS for Windows 6.1. Encino, CA: Multivariate Software.
7.
โโโโ 1990: Comparative fit indexes in structural models. Psychological Bulletin107: 238โ46.
8.
Bentler, P.M. and Bonnet, D.G.1980: Significance tests and goodness-of-fit in the analysis of covariance structures. Psychological Bulletin88: 588โ606.
9.
Bentler, P.M. and Dijkstra, T.1985: Efficient estimation via linearization in structural models . In Krishnaiah, P.R., editor, Multivariate analysis VI. Amsterdam: North-Holland, 9โ42.
10.
Breen, M.P., Barrat-Pugh, C., Derewianka, B., House, H., Hudson, C., Lumley, T. and Rohl, M.1997: Profiling ESL Children: How teachers interpret and use national and state assessment frameworks. Canberra: Department of Employment, Education, Training, and Youth Affairs, Commonwealth of Australia.
11.
Brindley, G.1998: Outcomes-based assessment and reporting in language learning programmes: A review of the issues. Language Testing15: 45โ85.
12.
โโโโ 2001: Outcomes-based assessment in practice: Some examples and emerging insights. Language Testing18: 393โ407.
13.
Brown, A.1995: The effect of rater variables on the development of an occupation-specific language performance test. Language Testing12: 1โ15.
14.
Byrne, B.M.2006: Structural equation modeling with EQS: Basic concepts, applications, and programming. Mahwah, NJ: Lawrence Erlbaum.
15.
Campbell, D.T. and Fiske, D.W.1959: Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin56: 81โ105.
16.
Chalhoub-Deville, M.1995: Deriving oral assessment scales across different tests and rater groups. Language Testing12: 16โ33.
17.
Conway, J.M., Scullen, S.E., Lievens, F. and Lance, C.E.2004: Bias in the correlated uniqueness model for MTMM data . Structural Equation Modeling: A Multidisciplinary Journal11: 535โ59.
18.
Cumming, A., Grant, L., Mulcahy-Ernt, P. and Powers, D.E.2004: A teacher-verification study of speaking and writing prototype tasks for a new TOEFL. Language Testing21: 107โ45.
19.
Davison, C.2004: The contradictory culture of teacher-based assessment: ESL teacher assessment practices in Australian and Hong Kong secondary schools . Language Testing21: 305โ34.
20.
Elder, C.1993: How do subject specialists construe classroom language proficiency?Language Testing10: 233โ54.
21.
Epp, L. and Stawychny, M.2001: Using the Canadian Language Benchmarks (CLB) to benchmark college programs/courses and language proficiency tests. TESL Canada Journal18: 32โ47.
22.
Gattullo, F.2000: Formative assessment in ELT primary (elementary) classrooms: An Italian case study. Language Testing17: 278โ88.
23.
Gipps, C.1994: Beyond testing: Towards a theory of educational assessment . London: Falmer Press.
24.
Grant, L.1997: Testing the language proficiency of bilingual teachers: Arizona's Spanish proficiency test. Language Testing14: 23โ46.
25.
Hoge, R. and Coldarci, T.1989: Teacher-based judgments of academic achievement: A review of literature. Review of Educational Research59: 297โ313.
26.
Hu, L.T. and Bentler, P.M.1999. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal6: 1โ55.
27.
Kenny, D.A.1976: Am empirical application of confirmatory factor analysis to the multitrait-multimethod matrix. Journal of Experimental Social Psychology12: 247โ52.
28.
โโโโ 1979: Correlation and causality. New York : Wiley.
29.
Kenny, D.A. and Kashy, D.A.1992: Analysis of the multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin112: 165โ72.
30.
Kunnan, A.J.1995: Test taker characteristics and test performance: A structural modeling approach. Cambridge: Cambridge University Press.
31.
Lance, C.E., Noble, C.L. and Scullen, S.E.2002: A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. Psychological Methods7: 228โ44.
32.
Leung, C. and Mohan, B.2004: Teacher formative assessment and talk in classroom contexts: Assessments as discourse and assessment of discourse. Language Testing21: 335โ59.
33.
Linquanti, R.2001: The redesignation dilemma: Challenges and choices in fostering meaningful accountability for English learners. Policy Report 2001โ1 . Santa Barbara, CA: University of California Linguistic Minority Research Institute, University of California , Santa Barbara.
34.
Llosa, L.2005a: Assessing English learners' language proficiency: A qualitative investigation of teachers' interpretations of the California ELD Standards . The CATESOL Journal17: 7โ18.
35.
โโโโ 2005b: Building and supporting a validity argument for a standards-based classroom assessment of English proficiency. Unpublished PhD dissertation, University of California, Los Angeles.
36.
Lumley, T.J.N. and McNamara, T.F.1995: Rater characteristics and rater bias: Implications for training. Language Testing12: 54โ71.
37.
Marsh, H.W.1989: Confirmatory factor analyses of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement15: 47โ70.
38.
Marsh, H.W., Byrne, B.M. and Craven, R.1992: Overcoming problems in confirmatory factor analyses of MTMM data: The correlated uniqueness model and factorial invariance. Multivariate Behavioral Research27: 489โ507.
39.
Marsh, H.W. and Grayson, D.1995: Latent variable models of multitrait-multimethod data . In Hoyle, R.H., editor, Structural equation modeling: Concepts, issues, and applications. Thousand Oaks, CA: Sage, 177โ98.
40.
McNamara, T.2001: Language assessment as social practice: Challenges for research. Language Testing18: 333โ49.
41.
Meisels, S.J., Bickel, D.D., Nicholson, J., Xue, Y. and Atkins-Burnett, S.2001: Trusting teacher judgments: A validity study of a curriculum-embedded performance assessment in kindergarten to grade 3. American Educational Research Journal38: 73โ95.
42.
Muthรฉn, B.O. and Kaplan, D.1985: A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology38: 171โ89.
43.
North, B.1993: The development of descriptors on scales of language proficiency . College Park, MD: The National Foreign Language Center.
44.
โโโโ 1995: The development of a common framework scale of descriptors of language proficiency based on a theory of measurement. System23: 445โ65.
45.
โโโโ 2000: The development of a common framework scale of language proficiency. New York: Peter Lang .
46.
O'Sullivan, B., Weir, C. and Saville, N.2002: Using observation checklists to validate speaking-test tasks. Language Testing19: 33โ56.
47.
Popham, W.J.2003: The trouble with testing: Why standards-based assessment doesn't measure up. American School Board Journal190: 14โ17.
48.
Rea-Dickins, P.2001: Mirror, mirror on the wall: Identifying processes of classroom assessment. Language Testing18: 429โ62.
49.
Rea-Dickins, P. and Gardner, S.2000: Snares and silver bullets: Disentangling the construct of formative assessment. Language Testing17: 215โ43.
50.
Rindskopf, D. and Rose, T.1988: Some theory and applications of confirmatory second-order factor analysis. Multivariate Behavioral Research23: 51โ67.
51.
Satorra, A. and Bentler, P.M.1988: Scaling corrections for chi-square statistics in covariance structure analysis. Proceedings of the American Statistical Association, 308โ13.
52.
Sawaki, Y.2007: Construct validation of analytic rating scales in a speaking assessment: Reporting a score profile and a composite. Language Testing24(3), 355โ90.
53.
Shepard, L.A.2000: The role of classroom assessment in teaching and learning. CSE Technical Report 517. National Center for Research on Evaluation, Standards, and Student Testing (CRESST). University of California , Los Angeles. Retrieved on 10 April 2005 from: www.cse.ucla.edu/CRESST/Reports/TECH517.pdf
54.
Shin, S.K.2005: Did they take the same test? Examinee language proficiency and the structure of language tests. Language Testing22: 31โ57.
55.
SPSS for Windows, Rel. 14.0.0. 2005. Chicago : SPSS.
56.
Stansfield, C. and Kenyon, D.1996: Comparing the scaling of speaking tasks by language teachers and by the ACTFL guidelines. In Cumming, A. and Berwick, R., editors, Validation in language testing. Clevedon, UK: Multilingual Matters, 124โ53.
57.
Stevens, J.J. and Clauser, P.1995: Multitrait-multimethod comparisons of a writing portfolio and the ITBS. Paper presented at the annual meeting of the National Council for Measurement in Education, San Francisco, CA.
58.
Taylor, C.S.2002: Incorporating classroom-based assessments into large-scale assessment programs. In Tindal, G. and Haladyna, T.M., editors, Large-scale assessment: Programs for all students. Mahwah, NJ: Lawrence Erlbaum, 233โ59.
