The paper examines the results of the CEFR alignment project for the Slovenian national examinations in English. The authors aim to validate externally the standard-setting procedures by adopting a socio-cognitive model of validation (Khalifa & Weir, 2009; Weir, 2005) to analyse the scoring, context and cognitive validity of three reading subtests: the Slovenian B2 national examination and the international examinations FCE and CAE, aligned with B2 and C1 respectively. The relative comparability between the three subtests is determined by analysing the results of tests that have been administered to a group of 80 test-takers (expected CEFR level: B2). The placement of the test-takers also reveals to what extent the judgements of the Slovenian panellists about CEFR levels coincide with those reported for FCE and CAE. The study thus also explores whether the high degree of agreement between the judges on the alignment panel can be solely attributed to their adequate and precise understanding of CEFR descriptors – which is directly mirrored in their setting of the cut scores and relating the examination to relevant CEFR levels – or whether it can also be ascribed to their shared educational, national and cultural background. The answers to these questions are paramount because they reveal the descriptive adequacy of CEFR descriptors and because different interpretations of CEFR levels can significantly affect national testing policies and, consequently, language teaching and testing.
AldersonJ. C. (2000). Assessing reading. Cambridge: Cambridge University Press.
2.
AldersonJ. C. (2007). The CEFR and the need for more research. The Modern Language Journal, 91(4), 659–663.
3.
AldersonJ. C.ClaphamC.WallD. (1995). Language test instruction and evaluation. Cambridge: Cambridge University Press.
4.
AldersonJ. C.FiguerasN.KuijperH.NoldG.TakalaS.TardieuC. (2004). The development of specifications for item development and classification within the Common European Framework of Reference for Languages: Learning, teaching, assessment. Reading and listening. Final report of The Dutch CEF Construct Project.Working paper. Retrieved February26, 2014, from Lancaster University Publications & Outputs: http://eprints.lancs.ac.uk/44/1/final_report.pdf
5.
BachmanL. F. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.
6.
BachmanL. F.PalmerA. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press.
7.
Bitenc PeharcSTratnikA. (2014). Umestitev nacionalnih izpitov iz angleščine v skupni evropski okvir. Zaključno poročilo o izvedbi projekta. Ljubljana: Državni izpitni center.
CarverR. P. (1990). Reading rate: A review of research and theory. New York: Academic Press.
13.
CizekG. J. (Ed.). (2001). Setting perfomance standards, concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum.
14.
CizekG. J.BB. M. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: SAGE Publications.
15.
CobbT. (n.d.). Web Vocabprofile. Retreived August5, 2014, from www.lextutor.ca/vp/ (an adaptation of Heatley, Nation, & Coxhead’s (2002) Range).
16.
Council of Europe. (2001). Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge: Cambridge University Press.
17.
Council of Europe. (2004). A reference supplement. Strasbourg: Language Policy Division.
18.
Council of Europe. (2009). Relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment (CEFR). A manual. Strasbourg: Language Policy Division.
EALTA. (2006, May20). EALTA Guidelines for Good Practice in Language Testing and Assessment. Retrieved February14, 2014, from EALTA, European Association for Language Testing and Assessment: www.ealta.eu.org/documents/archive/guidelines/English.pdf
21.
FiguerasN. (2012). The impact of the CEFR. ELT Journal, 66(4), 477–485.
22.
FiguerasN.NorthB.TakalaS.VerhelstN.Van AvermaetP. (2005). Relating examinations to the Common European Framework: A manual. Language Testing, 22, 262–279.
23.
FulcherG. (2004). Deluded by artifices? The Common European Framework and harmonization. Language Assessment Quarterly, 1(4), 253–266.
HughesA. (2002). Testing for language teachers. Cambridge: Cambridge University Press.
26.
JonesN. (2009). A comparative approach to constructing a multilingual proficiency framework: Constraining the role of standard-setting. In FiguerasN.NoijonsJ. (Eds.), Linking to the CEFR levels: Research perspectives (pp. 35–43). Arnhem: Cito, EALTA. Retrieved February14, 2014, from www.coe.int/t/dg4/linguistic/Proceedings_CITO_EN.pdf
27.
KaftandijevaF. (2010). Methods for setting cut scores in criterion-referenced achievement tests. A comparative analysis of six recent methods with an application to tests of reading in EFL. Cito, Arnhem: EALTA. Retrieved February5, 2014, from www.ealta.eu.org/documents/resources/FK_second_doctorate.pdf
28.
KeckerG.EckesT. (2010). Putting the Manual to the test: The TestDaF – CEFR linking project. In MartyniukW. (Ed.), Aligning tests with the CEFR. Reflections on using the Council of Europe’s draft Manual (pp. 50–79). Cambridge: Cambridge University Press.
29.
KhalifaH.WeirC. J. (2009). Examining Reading: Research and Practice in Assessing Second Language Reading. UK: Cambridge University Press.
30.
KodaK. (2005). Insights into second language reading. A cross-linguistic approach. Cambridge: Cambridge University Press.
31.
LittleD. (2007). The Common European Framework of Reference for Languages: Perspectives on the making of supranational language education policy. The Modern Language Journal, 91(4), 645–655.
32.
MadsenH. S. (1983). Techniques in Testing. Oxford: Oxford University Press.
33.
McNamaraT. (2000). Language Testing. Oxford: Oxford University Press.
34.
McNamaraD. S.GraesserA.CaiZ.DaiJ. (2013, January1). Coh-Metrix Common Core T.E.R.A. version 1.0. Retrieved August5, 2014, from http://coh-metrix.commoncoretera.com.
35.
McNamaraD. S.LouwerseM. M.CaiZ.GraesserA. (2005, January1). Coh-Metrix version 1.4. Retrieved August5, 2014, from http://cohmetrix.memphis.edu.
36.
MoeE. (2009). Jack of more trades? Could standard setting serve several functions? In FiguerasN.NoijonsJ. (Eds.), Linking to the CEFR levels: Research perspectives (pp. 131–138). Arnhem: Cito, EALTA. Retrieved February14, 2014, from www.coe.int/t/dg4/linguistic/Proceedings_CITO_EN.pdf
37.
NorthB. (2007). The CEFR Illustrative Descriptor Scales. The Modern Language Journal, 91(4), 656–659.
38.
NorthB.JonesN. (2009). Relating language examinations to the Common European Framework of Reference for Languages … further material on maintaining standards across languages, contexts and administrations by exploiting teacher judgment and IRT scaling. Strasbourg: Language Policy Division. Retrieved January11, 2014, from www.coe.int/t/dg4/linguistic/Manual%20-%20Extra%20Material%20-%20proofread%20-%20FINAL.pdf
39.
PapageorgiouS. (2010). Investigating the decision-making process of standard setting participants. Language Testing, 27(2), 261–282.
40.
PižornK. (2009). Designing proficiency levels for English for primary and secondary school students and the impact of the CEFR. In FiguerasN.NoijonsJ. (Eds.), Linking to the CEFR levels: Research perspectives (pp. 87–102). Arnhem: Cito, EALTA. Retrieved February14, 2014, from www.ealta.eu.org/documents/resources/Research_Colloquium_report.pdf
41.
ReckaseM. D. (2009). Standard setting theory and practice: Issues and difficulties. In FiguerasN.NoijonsJ. (Eds.), Linking to the CEFR levels: Research perspectives (pp. 13–20). Cito, EALTA: Arnhem. Retrieved February14, 2014, from www.ealta.eu.org/documents/resources/Research_Colloquium_report.pdf
42.
UrquhartA. H.WeirC. (1998). Reading in a second language: Process, product and practice. London: Longman.
43.
WeirC. J. (1990). Communicative language testing. New York: Prentice Hall.
44.
WeirC. J. (2005). Limitations of the Common European Framework for developing comparable examinations and tests. Language Testing, 22(3), 281–300.
