Despite the growing interest of the language testing community in standard setting, primarily due to the use of the Common European Framework of Reference (CEFR-Council of Europe, 2001), the participants’ decision-making process in the CEFR standard setting context remains unexplored. This study attempts to fill in this gap by analyzing these participants’ group discussions during a CEFR standard setting research project. Using an inductively and deductively-built analytical framework, it was found that decision-making was affected by factors that were irrelevant to the judgment task and that setting CEFR cut scores was not without problems for the participants. Given that examination results are nowadays reported with specific reference to the language ability levels presented in the CEFR, these results have implications for examination providers and score users.
Alderson, J.C. ( 2005). Editorial. Language Testing, 22(3), 257-260.
2.
Alderson, J.C., Figueras, N., Kuijper, H., Nold, G., Takala, S., & Tardieu, C. ( 2006). Analysing tests of reading and listening in relation to the Common European Framework of Reference: The experience of the Dutch CEFR Construct Project. Language Assessment Quarterly, 3(1), 3-30.
3.
Bachman, L.F. ( 2004). Statistical analyses for language assessment. Cambridge: Cambridge University Press.
4.
Banerjee, J. ( 2004). Qualitative analysis methods. Section D of the Reference Supplement to the preliminary version of the Manual for relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment. Strasbourg: Council of Europe.
5.
Brindley, G. ( 1998). Describing language development? Rating scales and second language acquisition. In L. F. Bachman & A. D. Cohen (Eds.), Interfaces between second language acquisition and language testing research (pp. 112-140). Cambridge: Cambridge University Press.
6.
Buckendahl, C.W. ( 2005). Guest editor’s introduction: Qualitative inquiries of participants’ experiences with standard setting. Applied Measurement in Education, 18(3), 219-221.
7.
Cizek, G. J. (Ed.). (2001). Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum.
8.
Cizek, G.J., & Bunch, M. ( 2007). Standard setting: A guide to establishing and evaluating performance standards on tests. London: Sage Publications.
9.
Cohen, L., Manion, L., & Morrison, K. ( 2000). Research methods in education (5th ed.). London: Routledge.
10.
Council of Europe. (2001). Common European Framework of Reference for Languages: Learning, teaching, assessment . Cambridge: Cambridge University Press.
11.
Council of Europe. (2003). Manual for relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment. Preliminary pilot version. Strasbourg: Council of Europe.
12.
Council of Europe. (2009). Relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment. A Manual. Retrieved 15 February 2009, from http://www.coe.int/T/DG4/Portfolio/documents/Manual%20Revision%20-%20proofread%20-%20FINAL.pdf
13.
Creswell, J.W. ( 2005). Educational research: Planning, conducting and evaluating quantitative and qualitative research (2nd ed.). Upper Saddle River, NJ: Pearson Merrill Prentice Hall.
14.
Ferdous, A.A., & Plake, B.S. ( 2005). Understanding the factors that influence decisions of panelists in a standard setting study. Applied Measurement in Education, 18(3), 257-267.
15.
Figueras, N., North, B., Takala, S., Verhelst, N., & Van Avermaet, P. (2005). Relating examinations to the Common European Framework: A manual. Language Testing , 22(3), 261-279.
16.
Fulcher, G. ( 2004). Deluded by artifices? The Common European Framework and harmonization. Language Assessment Quarterly, 1(4), 253-266.
17.
Giraud, G., & Impara, J.C. ( 2005). Making the cut: The cutscore setting process in a public school district. Applied Measurement in Education, 18(3), 289-312.
18.
Giraud, G., Impara, J.C., & Plake, B.S. ( 2005). Teachers’ conceptions of the target examinee in Angoff standard setting. Applied Measurement in Education , 18(3), 223-232.
19.
Glass, G.V. ( 1978). Standards and criteria. Journal of Educational Measurement, 15(4), 237-261.
20.
Hambleton, R.K. ( 2001). Setting performance standards on educational assessments and criteria for evaluating the process. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 89-116). Mahwah, NJ: Lawrence Erlbaum.
21.
Hasselgreen, A. ( 2005). Assessing the language of young learners. Language Testing, 22(3), 337-354.
22.
Holstein, J.A., & Gubrium, J.F. ( 2004). The active interview. In D. Silverman (Ed.), Qualitative research: Theory, method and practice (pp. 140-161). London: Sage Publications .
23.
Joffe, H., & Yardley, L. ( 2004). Content and thematic analysis. In D. F. Marks & L. Yardley (Eds.), Research methods for clinical and health psychology. London : Sage Publications.
24.
Kaftandjieva, F. ( 2004). Standard setting. Section B of the Reference Supplement to the preliminary version of the Manual for relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment. Strasbourg: Council of Europe.
25.
Kaftandjieva, F. ( 2009). Basket procedure: The breadbasket or the basket case of standard setting methods? In N. Figueras & J. Noijons (Eds.), Linking to the CEFR levels: Research perspectives (pp. 21-34). Arnhem : Cito.
26.
Kaftandjieva, F., & Takala, S. ( 2002). Council of Europe scales of language proficiency: A validation study. In J. C. Alderson (Ed.), Common European Framework of Reference for Languages: Learning, teaching, assessment. Case studies (pp. 106-129). Strasbourg: Council of Europe.
27.
Kane, M. ( 2001). So much remains the same: conception and status of validation in setting standards. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 53-88). Mahwah, NJ: Lawrence Erlbaum.
28.
Kane, M., Crooks, T., & Cohen, A.S. ( 1999). Validating measures of performance. Educational Measurement: Issues and Practice, 18(2), 5-17.
29.
Kecker, G., & Eckes, T. ( 2007). Linking the TestDaF to the CEFR: The case of writing proficiency. Paper presented at the Fourth Annual Conference of EALTA, Sitges, Spain. Retrieved 4 September 2007, from http://www.ealta.eu.org/conference/2007/docs/pres_sunday/Kecker&Eckes.pdf
30.
Keddle, J.S. ( 2004). The CEF and the secondary school syllabus. In K. Morrow (Ed.), Insights from the Common European Framework (pp. 43-54). Oxford: Oxford University Press.
31.
Lumley, T. ( 2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19(3), 246-276.
32.
Martyniuk, W. ( 2006). Relating language examinations to the Common European Framework of Reference for Languages (CEFR). Paper presented at the Into Europe Conference, Budapest, Hungary. Retrieved 20 May 2009, from http://www.examsreform.hu/Media/Relatinglanguage_exam.ppt
33.
McGinty, D. ( 2005). Illuminating the "Black Box" of standard setting: An exploratory qualitative study. Applied Measurement in Education , 18(3), 269-287.
34.
Morgan, D.L. ( 1988). Focus groups as qualitative research. London: Sage Publications.
35.
Muhr, T. ( 2004). Atlas.ti (Version 5.2). Berlin: ATLAS.ti Scientific Software Development GmbH.
36.
North, B. ( 2000). The development of a common framework scale of language proficiency. New York: Peter Lang .
37.
North, B. ( 2004). Europe’s framework promotes language discussion, not directives. Retrieved 31 January 2007, from http://education.guardian.co.uk/tefl/story/0,,1191130,00.html
38.
North, B., & Hughes, G. ( 2003). CEF performance samples for relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment. Retrieved 20 January 2006, from http://www.coe.int/T/DG4/Portfolio/documents/videoperform.pdf
39.
Popham, W.J. ( 1978). As always, provocative. Journal of Educational Measurement, 15(4), 297-300.
40.
Raymond, M.R., & Reid, J.B. ( 2001). Who made thee a judge? Selecting and training participants for standard setting. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 119-158). Mahwah, NJ: Lawrence Erlbaum.
41.
Skorupski, W.P., & Hambleton, R.K. (2005). What are panelists thinking when they participate in standard setting studies? Applied Measurement in Education, 18(3), 233-256.
42.
Takala, S. (Ed.). (2004). Reference Supplement to the preliminary version of the Manual for relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching and assessment. Strasbourg: Council of Europe.
43.
Tanko, G. ( 2004). Into Europe: The Writing handbook. Budapest: British Council.
44.
Tannenbaum, R.J., & Wylie, E.C. ( 2005). Mapping English language proficiency test scores onto the Common European Framework. Princeton, NJ: Educational Testing Service. Retrieved 14 May 2007, from http://www.ets.org/Media/Research/pdf/RR-05-18.pdf.
45.
Taylor, L., & Jones, N. ( 2006). Cambridge ESOL exams and the Common European Framework . Cambridge ESOL Research Notes, May 2006(24), 2-5. Retrieved 04/09/2007, from http://www.cambridgeesol.org/rs_notes/rs_nts2024.pdf
46.
Trinity College London. (2005a). Graded Examinations in Spoken English 2004-2007. (2nd impression). LondonTrinity CollegeLondon.
47.
Trinity College London. (2005b). Integrated Skills in English Examinations 0, I, II, III (4th ed.). London : Trinity CollegeLondon.
48.
Trinity College London. (2007). Relating the Trinity College London GESE and ISE exams to the Common European Framework of Reference: Piloting of the Council of Europe draft Manual. (Final project report). London: Trinity College London. Retrieved 20 May 2009, from http://www.trinitycollege.co.uk/resource/?id=2261
49.
Weir, C.J. ( 2005). Limitations of the Common European Framework of Reference for Languages (CEFR) for developing comparable examinations and tests. Language Testing, 22(3), 281-300.
50.
Wu, J., & Wu, R. ( 2007). Using the CEFR in Taiwan: The perspective of a local examination board. Paper presented at the Fourth Annual Conference of EALTA, Sitges, Spain. Retrieved 4 September 2007, from http://www.ealta.eu.org/conference/2007/docs/pres_sunday/Wu&Wu.pdf
51.
Zieky, M.J., & Perie, M. ( 2006). A primer on setting cutscores on tests of educational achievement. Retrieved 25 January 2007, from http://www.ets.org/Media/Research/pdf/Cut_Scores_Primer.pdf
52.
Zieky, M.J., Perie, M., & Livingston, S.A. (2008). Cutscores: A manual for setting standards of performance on educational and occupational tests. Princeton, NJ: Educational Testing Service.
