Rating scale design and development for testing speaking is generally conducted using one of two approaches: the measurement-driven approach or the performance data-driven approach. The measurement-driven approach prioritizes the ordering of descriptors onto a single scale. Meaning is derived from the scaling methodology and the agreement of trained judges as to the place of any descriptor on the scale. The performance data-driven approach, on the other hand, places primary value upon observations of language performance, and attempts to describe performance in sufficient detail to generate descriptors that bear a direct relationship with the original observations of language use. Meaning is derived from the link between performance and description. We argue that measurement-driven approaches generate impoverished descriptions of communication, while performance data-driven approaches have the potential to provide richer descriptions that offer sounder inferences from score meaning to performance in specified domains. With reference to original data and the literature on travel service encounters, we devise a new scoring instrument, a Performance Decision Tree (PDT). This instrument prioritizes what we term ‘performance effect’ by explicitly valuing and incorporating performance data from a specific communicative context. We argue that this avoids the reification of ordered scale descriptors which we find in measurement-driven scale construction for speaking tests.
Alderson JC ( 1991). Bands and scores. In Alderson JC, North B (Eds.), Language testing in the 1990s. London: Modern English Publications and the British Council.
2.
Bachman LF ( 1990). Fundamental considerations in language testing . Oxford: Oxford University Press .
3.
Baker R. ( 1997). Classical test theory and item response theory in test analysis. Lancaster: Department of Linguistics and Modern English Language.
4.
Bastos LC ( 1996). Power, solidarity and the construction of requests in service encounters. ESPecialist, 17(2), 151-174.
5.
Biber D., Conrad S., Reppen R., Byrd P., and Helt M. ( 2002). Speaking and writing in the university. A multidimensional comparison. TESOL Quarterly, 36(1), 9-48.
6.
Brooks L. ( 2009). Interacting in pairs in a test of oral proficiency: Co-constructing a better performance. Language Testing, 26(3), 341-366.
7.
Canale M. ( 1983a). From communicative competence to communicative language pedagogy. In Richards C, Schmidt RW (Eds.) Language and communication (pp. 2-27). London: Longman.
8.
Canale M. ( 1983b). On some dimensions of language proficiency. In Oller JW (Ed.), Issues in language testing research (pp. 333-342). Rowley, MA: Newbury House.
9.
Canale M., Swain M. ( 1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics , 1(1), 1-47.
10.
Chalhoub-Deville M. ( 2003). Second language interaction: Current perspectives and future trends. Language Testing, 20(4), 369-383.
11.
Chandon JL, Piere-Yves L., and Philippe J. ( 1997). Service encounter dimensions - a dyadic perspective: Measuring the dimensions of service encounters as perceived by customers and personnel. International Journal of Service Industry Management , 8(1), 65-86.
12.
Council of Europe (2001). Common European Framework of Reference for Languages: Learning, teaching and assessment. Cambridge: Cambridge University Press. Available online at: www.coe.int/t/dg4/linguistic/Source/Framework_EN.pdf .
13.
Coupland N. ( 1983). Patterns of encounter management: Further arguments for discourse variables. Language in Society, 12, 459-476.
14.
Coupland N., Ylänne-McEwen V. ( 2000). Talk about the weather: Small talk, leisure talk, and the travel industry. In Coupland J (Ed.), Small talk (pp. 163-182). London: Longman .
15.
Davidson F., Fulcher G. ( 2007). The Common European Framework of Reference (CEFR) and the design of language tests: A matter of effect. Language Teaching, 40(3), 231-241.
16.
Fairclough N. ( 1995). Critical discourse analysis: The critical study of language . London: Longman.
17.
Fulcher G. ( 1987). Tests of oral performance: The need for data-based criteria . English Language Teaching Journal, 41(4), 287-291.
18.
Fulcher G. ( 1993). The construction and validation of rating scales for oral tests in English as a Foreign Language. Unpublished PhD thesis, University of Lancaster, UK.
19.
Fulcher G. ( 1996). Does thick description lead to smart tests? A data-based approach to rating scale construction. Language Testing, 13(2), 208-238.
20.
Fulcher G. ( 2003). Testing second language speaking. London: Longman/Pearson.
21.
Fulcher G. ( 2008). Criteria for evaluating language quality. In Shohamy E (Ed.), Language testing and assessment. Encyclopedia of Language and Education, Vol. 7 (pp. 157-176). Amsterdam: Springer .
22.
Fulcher G., Davidson F. ( 2007). Language testing and assessment. London and New York: Routledge .
23.
Fulcher G., Davidson F. ( 2009). Test architecture, test retrofit. Language Testing, 26(1), 123-144.
24.
Gabbott M., Hogg G. ( 2000). An empirical investigation of the impact of non-verbal communication on service evaluation. European Journal of Marketing , 34(3/4), 384-398.
25.
Gremler DD, Gwinner KP ( 2000). Customer-employee rapport in service relationships . Journal of Service Research, 3(1), 82-104.
26.
Grönroos C. ( 1993). Toward a third phase in service quality research: Challenges and future directions. In Swartz AT, Bowen DE, and Brown SW (Eds.), Advances in service marketing management, Vol. II (pp. 49-64). Greenwich CT: JAI Press.
27.
Halliday Mak ( 1985). Context of situation. In Halliday MAK, Hasan R, Language, context, and text: Aspects of language in a social-semiotic perspective (pp. 3-14). Victoria, Australia: Deakin University Press.
28.
Hamp-Lyons L. ( 1991). Scoring procedures for ESL contexts. In Hamp-Lyons, L. (Ed.), Assessing second language writing in academic contexts (241-276). Norwood, NJ: Ablex.
29.
Hasan R. ( 1985). The structure of a text. In Halliday MAK, Hasan R (1985 ). Language, context, and text: Aspects of language in a social-semiotic perspective (pp. 52-69). Victoria , Australia: Deakin University Press.
30.
Hulstijn JA ( 2007). The shaky ground beneath the CEFR: Quantitative and qualitative dimensions of language proficiency. The Modern Language Journal , 91(4), 663-667.
31.
Jacoby S., McNamara, T. ( 1999). Locating competence. English for Specific Purposes, 18(3), 213-241.
32.
Jacoby S., Ochs E. ( 1995). Co-construction: An introduction. Research on Language and Social Interaction, 28(3), 171-183.
33.
Keaveney SM ( 1995). Customer switching behavior in service industries: An exploratory study. Journal of Marketing, 59(2), 71-82.
34.
Kramsch C. ( 1986). From language proficiency to interactional competence . The Modern Language Journal, 70(4), 366-372.
35.
Lowe P. ( 1986). Proficiency: Panacea, framework, process? A reply to Kramsch, Schulz, and particularly Bachman and Savignon. Modern Language Journal, 70(4), 391-397.
36.
Lumley T., Lynch B., and McNamara T. ( 1994). A new approach to standard-setting in language assessment . Melbourne Papers in Language Testing, 3(2), 19-40.
37.
May LA ( 2007). Interaction in a paired speaking test: The rater’s perspective. Unpublished PhD thesis, School of Languages and Linguistics, University of Melbourne, Australia.
38.
McCarthy M. ( 2000). Captive audiences: The discourse of close contact service encounters. In Coupland J (Ed.), Small talk (pp. 84-109). London: Longman .
39.
McCarthy M. ( 2003). Talking back: ‘Small’ interactional response tokens in everyday conversation. Research on Language and Social Interaction, 36(1), 33-63.
40.
McCarthy M., Carter R. ( 1994). Language as discourse: Perspectives for language teaching . London: Longman.
41.
Meisel JM ( 1980). Linguistic simplification. In Felix S (Ed.), Second language development: Trends and issues (pp. 13-40). Tübingen: Gunter Narr.
42.
Mills S. ( 2009). An evaluation of a paired format oral test for Korean learners of English for Hotel and Tourism. University of Leicester: Unpublished MA dissertation.
43.
Mitchell TF ( 1957/1978). The language of buying and selling in Cyrenaica: A situational statement. HespėrisXLIV: 31-71. Reprinted in TF Mitchell (1978). Principles of Firthian linguistics (pp. 167-200). London: Longman .
44.
North B. ( 1993). Scales of language proficiency: A survey of some existing systems. Strasbourg: Council of Europe, Council for Cultural Co-operation, CC-LAND 94 24.
45.
North B. ( 1995). The development of a common framework scale of descriptors of language proficiency based on a theory of measurement. System, 23(4), 445-465.
46.
North B. ( 1996). The development of a common framework scale of descriptors of language proficiency based on a theory of measurement. In Huhta A, Kohonen V, Kurki-Suonio L, and Luoma S (Eds.), Current developments and alternatives in language assessment. Proceedings of the LTRC1996 (pp. 423-447). Jyvaskyla: University of Jyvaskyla Press.
47.
North B. ( 2000). The development of a common framework scale of language proficiency. New York, Peter Lang .
48.
North B., Schneider G. ( 1998). Scaling descriptors for language proficiency scales . Language Testing, 15(2), 217-263.
49.
Parasuraman A., Zeithaml AV, and Berry LL ( 1988). SERVQUAL: A multiple-item scale for measuring customer perceptions of service quality, Journal of Retailing , 64, 12-40.
50.
Ryoo H.-K (2005). Achieving friendly interactions: a study of service encounters between Korean shopkeepers and African-American customers. Discourse and Society, 16(1), 79-105.
51.
Shiel C., Bowen DE, and Pearson CM ( 1992). Service encounters as rites of integration: An information processing model. Organizational Science, 3(4), 537-555.
52.
Spencer WJ ( 1997). ‘We don’t pay for bus tickets, but we can help you find work’: The micropolitics of trouble in human service encounters . The Sociological Quarterly, 38(1), 185-203.
53.
Stern BB, Thompson CJ, and Arnould EJ ( 1998). Narrative analysis of a marketing relationship: The consumer’s perspective. Psychology and Marketing , 15(3), 195-214.
54.
Storbacka K., Standvik T., and Grönroos, C. (1994). Managing customer relationships for profit: The dynamics of relationship quality. International Journal of Service Industry Management, 5(5), 21-38.
55.
Suprenant CF, Solomon MR ( 1987). Predictability and personalization in the service encounter . Journal of Marketing, 51(2), 86-96.
56.
Swain M. ( 2001). Examining dialogue: Another approach to content specification and to validating inferences drawn from test scores. Language Testing, 18(3), 275-302.
57.
Turner CE ( 2000). Listening to the voices of rating scale developers: Identifying salient features for second language performance assessment. Canadian Modern Language Review, 56(4), 555-584.
58.
Turner CE, Upshur J. ( 2002). Rating scales derived from student samples: Effects of the scale marker and the student sample on scale content and student scores . TESOL Quarterly, 36(1), 49-70.
59.
Upshur J., Turner CE ( 1995). Constructing rating scales for second language tests . English Language Teaching Journal, 49(1), 3-12.
60.
Upshur J., Turner CE ( 1999). Systematic effects in the rating of second-language speaking ability: Test method and learner discourse.’ Language Testing, 16(1), 82-111.
61.
van Lier L. ( 1989). Reeling, writhing, drawling, stretching, and fainting in coils: Oral Proficiency Interviews as conversation. TESOL Quarterly, 23(3), 489-508.
62.
Ventola E. ( 1987). The structure of social interaction. London: Francis Pinter.
63.
Ventola E. ( 2005). Revisiting service encounter genre: Some reflections . Folia Linguistica, 39(1-2), 19-43.
64.
Westhoff G. ( 2007). Challenges and opportunities of the CEFR for reimagining foreign language pedagogy. The Modern Language Journal, 91(4), 676-679.
65.
Wilds C. ( 1975). The oral interview test. In Jones RL, Spolsky B (Eds.), Testing language proficiency (pp. 29-44). Arlington, VA : Center for Applied Linguistics.
66.
Ylänne-McEwen V. ( 2004). Shifting alignment and negotiating sociality in travel agency discourse. Discourse Studies, 6(4), 517-536.
67.
Zeithaml VA ( 2000). Service quality, profitability, and the economic worth of customers: What we know and what we need to learn. Journal of the Academy of Marketing Science, 28(1), 67-85.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
