This study explores the utility of analytic scoring for TAST in providing useful and reliable diagnostic information for operational use in three aspects of candidates' performance: delivery, language use and topic development. One hundred and forty examinees' responses to six TAST tasks were scored analytically on these three aspects of speech. G studies were used to investigate the dependability of the analytic scores, the distinctness of the analytic dimensions, and the variability of analytic score profiles. Raters' perceptions of dimension separability were obtained using a questionnaire. It was found that the dependability of analytic scores averaged across six tasks and double ratings was acceptable for both operational and practice settings. However, scores averaged across two tasks and double ratings were not reliable enough for operational use. Correlations among the analytic scores by task were high but those between delivery and topic development were lower. These results were corroborated by raters' perceptions. When averaged across tasks or task types, correlations among the analytic scores were very high, and the profiles of scores were flat. The utility of analytic scoring is discussed, considering both score dependability and whether analytic scores provide diagnostic information beyond that provided by holistic scores.
Get full access to this article
View all access options for this article.
References
1.
Bachman, L.F.1988: Problems in examining the validity of the ACTFL oral proficiency interview. Studies in Second Language Acquisition10(2): 149—64.
2.
Bachman, L.F. and Savignon, S.J.1986: The evaluation of communicative language proficiency: A critique of the ACTFL Oral Interview. The Modern Language Journal70(4): 380—90.
3.
Bachman, L.F., Lynch, B.K., and Mason, M.1995: Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Language Testing12: 238—57.
4.
Bolus, R.E., Hinofotis, F.B., and Bailey, K.M.1982: An introduction to generalizability theory in second language research. Language Learning32: 245—58.
5.
Brennan, R.L.1992: Elements of generalizability theory, revised edition. Iowa City, IA: American College Testing.
6.
—— 2000: (Mis)conceptions about generalizability theory. Educational Measurement: Issues and Practice19(1): 5—10.
—— 2002: Performance assessments from the perspective of generalizability theory. Applied Psychological Measurement24(4): 339—53.
9.
Brennan, R.L., Gao, X., and Colton, D.A.1995: Generalizability analyses of Work Keys Listening and Writing tests. Educational and Psychological Measurement55: 157—76.
10.
Brown, A.1995: The effect of rater variables in the development of an occupation-specific language performance test. Language Testing12: 1—15.
11.
Brown, J.D.1984: A norm-referenced engineering reading test. In Pugh, A.K. and J.M. Ulijn , editors, Reading for professional purposes: Studies and practices in native and foreign language. London: Heinemann Educational Books.
12.
—— 1990: Short-cut estimators of criterion-referenced test consistency . Language Testing7: 77—97.
13.
—— 1993: A comprehensive criterion-referenced testing project. In Douglas, D. and C. ChapelleA new decade of language testing research. Alexandra, VA: TESOL
14.
—— 1999: The relative importance of persons, items, subtests and languages to TOEFL test variance. Language Testing16: 217—38.
15.
Brown, J.D. and Bailey, K.M.1984: A categorical instrument for scoring second language writing skills. Language Learning34: 21—42.
16.
Brown, J.D. and Hudson, T.2002: Criterion-referenced language testing. Cambridge: Cambridge University Press.
17.
Brown, J.D. and Ross, J.A.1996: Decision dependability of item types, sections, tests, and the overall TOEFL test battery. In Milanovic, M. and Saville, N., editors, Performance testing, cognition and assessment. Cambridge: Cambridge University Press.
18.
Butler, F.A., Eignor, D., Jones, S., McNamara, T., and Suomi, B.K.2000: TOEFL 2000 speaking framework: A working paper (TOEFL Monograph Series No. 20). Princeton, NJ: Educational Testing Service.
19.
Chalhoub-Deville, M.2003: Second language interaction: Current perspectives and future trends. Language Testing20: 369—83.
20.
Chiu, C.W.T.1999: Scoring performance assessments based on judgments: Utilizing meta-analysis to estimate variance components in generalizability theory for unbalanced situations. Unpublished doctoral dissertation, Michigan State University.
21.
—— 2000, April: A subdividing method for generalizability theory: Precision of measurement errors and patterns of missing data. Paper presented at the annual meeting of the American Educational Research Association (AERA), New Orleans, LA.
22.
Chiu, C.W.T. and Wolfe, E.W.2002: A method for analyzing sparse data matrices in the generalizability theory framework. Applied Psychological Measurement26: 321—38.
23.
Cronbach, L.J., Gleser, G.C., Nanda, H., and Rajaratnam, N.1972: The dependability of behavioral measurements: Theory of generalizability for scores and profilesNew York: Wiley.
24.
Derwing, T.M. and Munro, M.J.1997: Accent, intelligibility, and comprehensibility: Evidence from four L1s. Studies in Second Language Acquisition19: 1—16.
25.
Douglas, D.1997: Testing speaking ability in academic contexts: Theoretical considerations (TOEFL Monograph Series No. 8). Princeton, NJ: Educational Testing Service.
26.
Douglas, D. and Smith, J.1997: Theoretical underpinnings of the Test of Spoken English revision project (TOEFL Monograph Series No. 9). Princeton, NJ: Educational Testing Service.
27.
Fulcher, G.1997: The testing of speaking in a second language. In Clapham, C. and D. Corson, editors, Encyclopedia of Language and Education: Vol 7. Language Testing and Assessment. Dordrecht, the Netherlands: Kluwer, 75—85.
28.
Gass, S. and Varonis, E.M.1984: The effect of familiarity on the comprehensibility of non-native speech. Language Learning34: 65—89.
29.
Ingram, D. and Wylie, E.1993: Assessing speaking proficiency in the International English Language Testing System. In Douglas, D. and C. Chapelle, editors, A new decade of language testing research: Selected papers from the 1990 Language Testing Research Colloquium. Alexandria, VA: TESOL.
30.
Joe, G.W. and Woodward, J.A.1976: Some developments in multivariate generalizability. Psychometrica41: 205—17.
31.
Kunnan, A.J.1992: An investigation of a criterion-referenced test using G-theory, and factor analysis. Language Testing9: 30—49.
32.
Lee, Y-W.2005: Dependability of scores for a new ESL speaking test: Evaluating prototype tasks (TOEFL Research Monograph No. 28). Princeton, NJ: Educational Testing Service.
33.
Lee, Y-W. and Kantor, R.2005: Dependability of new ESL writing test scores: Evaluating prototype tasks and alternative rating schemes (TOEFL Research Monograph No. 31). Princeton, NJ: Educational Testing Service.
34.
Linn, R.L. and Burton, E.1994: Performance-based assessment: Implications of task specificity . Educational Measurement: Issues and Practice13: 5—15. Marcoulides, G.A.1994: Selecting weighting schemes in multivariate generalizability studies. Educational and Psychological Measurement54(91): 3—7.
35.
Rosenfeld, M., Leung, S., and Oltman, P.K.2001: The reading, writing, speaking, and listening tasks important for academic success at the undergraduate and graduate levels (TOEFL Monograph Series No. 21). Princeton, NJ: Educational Testing Service.
36.
Sawaki, Y.2002, December: Multivariate analytic approaches to validation of a speaking test. Paper presented at the Language Testing Research Colloquium, Hong Kong.
37.
—— 2003: A comparison of summarization and free recall as reading comprehension tasks in Web-based assessment of Japanese as a foreign language . Unpublished doctoral dissertation, University of California , Los Angeles.
38.
Shavelson, R.J. and Webb, N.M.1981: Generalizability theory: 1973—1980. British Journal of Mathematical and Statistical Psychology34: 133—66.
39.
Smith, P.L.1980, April: Some approaches to determining the stability of estimated variance components. Paper presented at the Annual Meeting of the American Educational Research Association (AERA) , Boston, MA.
40.
Stansfield, C.W. and Kenyon, D.M.1992: Research of the compatibility of the oral proficiency interview and the simulated oral proficiency interview. System20: 347—64.
41.
Underhill, N.1987: Testing spoken English. Cambridge: Cambridge University Press.
42.
Waters, A.1996: A review of research into needs in English for academic purposes of relevance to the North American higher education context (TOEFL Monograph Series No. 6). Princeton, NJ: Educational Testing Service.
43.
Weir, C.J.1990: Communicative language testing. Englewood Cliffs, NJ: Prentice-Hall.
44.
Webb, N.M. and Shavelson, R.J.1981: Multivariate generalizability of general educational development ratings. Journal of Educational Measurement18: 13—22.
45.
Webb, N.M., Shavelson, R.J. and Maddahian, E.1983: Multivariate generalizability theory. In Fyans, L.J., Jr., editor, Generalizability theory: Inferences and practical applications: No. 18. New directions for testing and measurement, San Francisco: Jossey-Bass.
46.
Wiley, D.E.1992: Pooling variance component estimates from random effects analyses of variance used to evaluate the generalizability of test tasks and scores. Studies of the California Assessment Program Spring 1992 Field Trials .
47.
Xi, X.2003: Investigating language performance on the graph description task in a semi-direct oral test. Unpublished doctoral dissertation, University of California, Los Angeles.
48.
Xi, X. and Mollaun, P.2006: Investigating the utility of analytic scoring for TOEFL Academic Speaking Test (TAST) (TOEFL iBT Research Report No. 1). Princeton, NJ: Educational Testing Service.
