For Chinese as a second language (L2 Chinese), there has been little research into distinguishing features (Fulcher, 1996; Iwashita et al., 2008) used in scoring L2 Chinese speaking performance. The study reported here investigates the relationship between the distinguishing features of L2 Chinese spoken performances and the scores awarded by raters to the performances using holistic scoring. Seven distinguishing features – representing four major categories of Pronunciation, Fluency, Vocabulary and Grammar in the L2 Chinese speaking construct – were employed. An L2 Chinese speaking test was developed to assess the overall communicative ability in L2 spoken Chinese within an academic context. Speech samples of 66 candidates on the speaking test (i.e. 9 minutes’ speech length for each candidate) were analyzed in terms of the seven distinguishing features, with correlations and standard multiple regression being employed. Results showed that, first, each of the seven distinguishing features was significantly correlated to the scores, producing large or medium effect sizes; second, 79% and 77% of the variance in the scores could be explained by the distinguishing features (incorporating word tokens and word types respectively) in two regression analyses. The current study has established a link between distinguishing features and scores, contributing empirical evidence of candidate performance to the validation of assessing speaking proficiency in the L2 Chinese context.
AldersonJ. C. (1991). Bands and scores. In AldersonJ. C.NorthB. (Eds.), Language testing in the 1990s: The communicative legacy (pp. 71–86). London: Macmillan.
2.
BachmanL. F.PalmerA. S. (2010). Language assessment in practice. Oxford: Oxford University Press.
3.
BachmanL. F.SavignonS. J. (1986). The evaluation of communicative language proficiency: A critique of the ACTFL Oral Interview. Modern Language Journal, 70(4), 380–390.
4.
BrownA. (2000). An investigation of the rating process in the IELTS oral interview. In TullohR. (Ed.), IELTS research reports 2000 (pp. 49–84.). Canberra: IELTS Australia.
5.
BrownA. (2006a). An examination of the rating process in the revised IELTS Speaking Test. In McGovernP.WalshS. (Eds.), IELTS research reports 2006 (pp. 41–70). Canberra & Manchester: IELTS Australia and British Council.
6.
BrownA. (2006b). Candidate discourse in the revised IELTS Speaking Test. In McGovernP.WalshS. (Eds.), IELTS research reports 2006 (pp. 71–89). Canberra & Manchester: IELTS Australia and British Council.
7.
BrownA.IwashitaN.McNamaraT. (2005). An examination of rater orientations and test taker performance on English-for-Academic-Purposes speaking tasks (TOEFL Monograph No. 29). Princeton, NJ: Educational Testing Service.
8.
BrownA.TaylorL. (2006). A world survey of examiners’ views and experience of the revised IELTS Speaking Test. Cambridge ESOL: Research Notes, 26, 14–18.
9.
CareyM. D.MannellR. H.DunnP. K. (2011). Does a rater’s familiarity with a candidate’s pronunciation affect the rating in oral proficiency interviews?Language Testing, 28(2), 201–219.
10.
ChapelleC. A.EnrightM. K.JamiesonJ. M. (Eds.). (2008). Building a validity argument for the Test of English as a Foreign Language. New York: Routledge.
11.
ChengJ. (2009). The study of educational structure for international students from1950 to 2007 in China. Beijing: China Doctoral Dissertations Full-text Database.
12.
China National Office for Teaching Chinese as a Foreign Language. (1996). Chinese proficiency scales and syllabus of graded grammar. Beijing: Higher Education Press.
13.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
14.
CohenJ.CohenP.WestS. G.AikenL. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
15.
DaviesA. (1991). The native speaker in applied linguistics. Edinburgh: Edinburgh University Press.
16.
DaviesA. (2003). The native speaker: Myth and reality. Clevedon, UK: Multilingual Matters.
17.
DaviesA. (2008). Assessing academic English: Testing English proficiency, 1950–1989: The IELTS solution. Cambridge: Cambridge University Press.
18.
DouglasD. (1994). Quantity and quality in speaking test performance. Language Testing, 11(2), 125–144.
19.
DouglasD.SelinkerL. (1992). Analyzing oral proficiency test performance in general and specific purpose contexts. System, 20(3), 317–328.
20.
DouglasD.SelinkerL. (1993). Performance on a general versus a field-specific test of speaking proficiency by international teaching assistants. In DouglasD.ChapelleC. (Eds.), A new decade of language testing research: Selected papers from the 1990 Language Testing Research Colloquium (pp. 235–256). Alexandria, VA: TESOL.
21.
Educational Testing Service. (2005). TOEFL iBT tips: How to prepare for the next generation TOEFL test and communicate with confidence. Princeton, NJ: Educational Testing Service.
22.
EllisR.BarkhuizenG. (2005). Analysing learner language. Oxford: Oxford University Press.
FulcherG. (1996). Does thick description lead to smart tests? A data-based approach to rating scale construction. Language Testing, 13(2), 208–238.
25.
FulcherG. (2003). Testing second language speaking. London: Pearson Education.
26.
FulcherG. (2008). Criteria for evaluating language quality. In ShohamyE.HornbergerN. H. (Eds.), Encyclopedia of language and education (2nd ed.), Vol. 7: Language testing and assessment (pp. 157–176). New York: Springer.
27.
FulcherG. (2010). Practical language testing. London: Hodder Education.
28.
FulcherG.DavidsonF. (2007). Language testing and assessment: An advanced resource book. London and New York: Routledge.
29.
GuoS. (2011). The construction and application of Chinese corpora. Shanghai: Shanghai Foreign Language Education Press.
30.
GuoX. (2007). Quantifiable measures of speaking fluency in Chinese as a second language. Journal of Xiangtan Normal University (Social Science edition), 29(4), 91–94.
31.
Hanban/Confucius Institute Headquarters. (2009). Test syllabus for HSK–Advanced level. Beijing: The Commercial Press.
32.
HuntK. W. (1970). Syntactic maturity in school children and adults. Monographs of the Society for Research in Child Development, 35(1), (1, Serial No. 134).
IngramD. (1985). Assessing proficiency: An overview of some aspects of testing. In HyltenstamK.PienemannM. (Eds.), Modelling and assessing second language development (pp. 215–276). Clevedon, UK: Multilingual Matters.
35.
IngramD. E.WylieE. (1993). Assessing speaking proficiency in the International English Language Testing System. In DouglasD.ChapelleC. (Eds.), A new decade of language testing research: Selected papers from the 1990 Language Testing Research Colloquium (pp. 220–234). Alexandria, VA: TESOL.
36.
IwashitaN.BrownA.McNamaraT.O’HaganS. (2008). Assessed levels of second language speaking proficiency: How distinct?Applied Linguistics, 29(1), 24–49.
37.
JinT.MakB.ZhouP. (2012). Confidence scoring of speaking performance: How does fuzziness become exact?Language Testing, 29(1), 43–65.
38.
KangO.RubinD.PickeringL. (2010). Suprasegmental measures of accentedness and judgments of language learner proficiency in oral English. Modern Language Journal, 94(4), 554–566.
39.
KimY.-H. (2009). An investigation into native and non-native teachers’ judgments of oral English performance: A mixed methods approach. Language Testing, 26(2), 187–217.
40.
KuoC.-C. (2007). Phonetic and phonological background of Chinese spoken languages. In LeeC.-H.LiH.LeeL.-S.WangR.-H.HuoQ. (Eds.), Advances in Chinese spoken language processing (pp. 33–55). Singapore: World Scientific Publishing.
41.
LantolfJ. P.FrawleyW. (1985). Oral-proficiency testing: A critical analysis. Modern Language Journal, 69(4), 337–345.
42.
Larson-HallJ. (2010). A guide to doing statistics in second language research using SPSS. New York and London: Routledge.
43.
LazaratonA. (1996). Interlocutor support in oral proficiency interviews: The case of CASE. Language Testing, 13(2), 151–172.
44.
LazaratonA. (2002). A qualitative approach to the validation of oral language tests. Cambridge: Cambridge University Press.
45.
LennonP. (1991). Error: Some problems of definition, identification, and distinction. Applied Linguistics, 12(2), 180–196.
46.
LiA.ZuY. (2007). Corpus design and annotation for speech synthesis and recognition. In LeeC.-H.LiH.LeeL.-S.WangR.-H.HuoQ. (Eds.), Advances in Chinese spoken language processing (pp. 243–268). Singapore: World Scientific Publishing.
47.
LiQ. (Ed.). (2006). Curriculums, syllabi and teaching models for teaching Chinese as a second language. Beijing: The Commercial Press.
48.
Liskin-GasparroJ. E. (1984). The ACTFL proficiency guideline: Gateway to testing and curriculum. Foreign Language Annals, 17(5), 475–489.
49.
LiuL. (1997). An exploration into testing speaking in HSK (advanced). Chinese Teaching in the World, 2, 48–53.
50.
LiuY.Hakkani-TürD. (2011). Speech summarization. In TurG.De MoriR. (Eds.), Spoken language understanding: Systems for extracting semantic information from speech (pp. 357–396). Chichester: John Wiley & Sons.
51.
LiuY.TanQ.ShenX. (1994). Segmentation specifications and automatic segmentation methods for contemporary Chinese language words. Beijing: Tsinghua University Press; Nanning: Guangxi Science and Technology Press.
52.
LoweP. (1985). The ILR proficiency scale as a synthesizing research principle: The view from the mountain. In JamesC. J. (Ed.), Foreign language proficiency in the classroom and beyond (pp. 9–53). Lincolnwood, IL: National Textbook.
53.
LuJ. (1994). The analysis of grammatical errors in foreigners’ learning of Chinese as a second language. Language Teaching and Linguistic Studies1, 49–64.
54.
MaoS.YeJ. (Eds.). (2002). Testing pronunciation in teaching Chinese as a second language. Beijing: China Social Sciences Press.
55.
MatthewsM. (1990). The measurement of productive skills: Doubts concerning the assessment criteria of certain public examinations. ELT Journal, 44(2), 117–121.
56.
MehnertU. (1998). The effects of different lengths of time for planning on second language performance. Studies in Second Language Acquisition, 20(1), 83–108.
57.
Ministry of Education & State Language Commission, the People’s Republic of China. (2011). Spoken Chinese proficiency grading standards and testing guidelines. Beijing: Beijing Language and Culture University Press.
58.
Ministry of Education & State Language Commission, the People’s Republic of China. (2010). The graded Chinese syllables, characters and words for the application of teaching Chinese to the speakers of other languages. Beijing: Language and Literature Press.
59.
NormanJ. (1988). Chinese. Cambridge: Cambridge University Press.
60.
NorthB. (2000). The development of a common framework scale of language proficiency. New York: Peter Lang.
61.
O’LoughlinK. (1995). Lexical density in candidate output on direct and semi-direct versions of an oral proficiency test. Language Testing, 12(2), 217–237.
62.
O’LoughlinK. (2001). The equivalence of direct and semi-direct speaking tests. Cambridge: Cambridge University Press.
63.
PollittA.MurrayN. L. (1996). What raters really pay attention to. In MilanovicM.SavilleN. (Eds.), Performance testing, cognition and assessment: Selected papers from the 15th Language Testing Research Colloquium (LTRC), Cambridge and Arnhem (pp. 74–91). Cambridge: Cambridge University Press.
64.
RossS.BerwickR. (1992). The discourse of accommodation in oral proficiency interviews. Studies in Second Language Acquisition, 14(2), 159–176.
65.
ShiJ. (2002). A case study of the acquisition of Chinese sentence patterns by Korean learners. Chinese Teaching in the World, 4, 34–42.
66.
ShohamyE. (1994). The validity of direct versus semi-direct oral tests. Language Testing, 11(2), 99–123.
67.
StraussA.CorbinJ. (1994). Grounded theory methodology: An overview. In DenzinN.LincolnY. (Eds.), Handbook of qualitative research (pp. 273–285). Thousand Oaks, CA: SAGE Publications.
68.
SunC. (2006). Chinese: A linguistic introduction. Cambridge: Cambridge University Press.
69.
TabachnickB. G.FidellL. S. (2007). Using multivariate statistics (5th ed.). Boston: Pearson Education.
70.
TangS.-W. (2010). Formal Chinese syntax. Shanghai: Shanghai Educational Publishing House.
71.
TaylorL.FalveyP. (Eds.). (2007). IELTS collected papers: Research in speaking and writing assessment. Cambridge: Cambridge University Press.
72.
Test Center in National Chinese Proficiency Test Committee Office. (2001). Syllabus of graded words and characters for Chinese proficiency. Beijing: Economic Science Press.
73.
The Office of Chinese Language Council International. (2007). Chinese language proficiency scales for speakers of other languages. Beijing: Foreign Language Teaching and Research Press.
74.
The Office of Chinese Language Council International. (2008). International curriculum for Chinese language education. Beijing: Foreign Language Teaching and Research Press.
75.
UpshurJ.TurnerC. E. (1995). Constructing rating scales for second language tests. ELT Journal, 49(1), 3–12.
76.
van LierL. (1989). Reeling, writhing, drawling, stretching, and fainting in coils: Oral proficiency interviews as conversation. TESOL Quarterly, 23(3), 489–508.
77.
WangJ. (2002). A study of the scoring of three types of oral test items (in Chinese). Chinese Teaching in the World, 4, 63–77.
78.
WangJ. (2011). Feasibility analysis of computer aided Chinese grammar error identification. Applied Linguistics (Yuyan Wenzi Yingyong), 1, 135–142.
79.
WildsC. (1975). The oral interview test. In JonesR. L.SpolskyB. (Eds.), Testing language proficiency. Arlington, VA: Center for Applied Linguistics.
80.
XiX. (2007). Evaluating analytic scoring for the TOEFL® Academic Speaking Test (TAST) for operational use. Language Testing, 24(2), 251–286.
81.
YoungR. (1995a). Conversational styles in language proficiency interviews. Language Learning, 45(1), 3–42.
82.
YoungR. (1995b). Discontinuous interlanguage development and its implications for oral proficiency rating scales. Applied Language Learning, 6(1&2), 13–26.
83.
YoungR.HeA. W. (Eds.). (1998). Talking and testing: Discourse approaches to the assessment of oral proficiency. Amsterdam: John Benjamins.
84.
YoungR.MilanovicM. (1992). Discourse variation in oral proficiency interviews. Studies in Second Language Acquisition, 14(4), 403–424.
85.
YuS.DuanH.ZhuX.SunB. (2002). The basic processing of Contemporary Chinese Corpus at Peking University: Specification. Journal of Chinese Information Processing, 16(5), 49–64; 16(6), 58–64.
86.
ZhaiY. (2011). An objective study on the subjective criteria of speaking fluency [in Chinese as a second language]. Language Teaching and Linguistic Studies, 5, 79–86.
87.
ZhangX. (Ed.). (2009). A history of Chinese language education in the world. Beijing: The Commercial Press.
88.
ZhaoJ. (2002). The rank of ill-formed sentences made by CFL-learners based on error analysis. Language Teaching and Linguistic Studies, 2, 1–9.
ZhuS. (2009). A study on the dynamic oral text of Korean learners of Chinese (in Chinese). Beijing: China Master Theses Full-text Database.
91.
ZongC.RenF. (2003). Chinese utterance segmentation in spoken language translation. In GelbukhA. (Ed.), Lecture Notes in Computer Science: Vol. 2588. Computational Linguistics and Intelligent Text Processing (pp. 19–29). Berlin/Heidelberg: Springer.
