Assessing the content of learners’ compositions is a common practice in second language (L2) writing assessment. However, the construct definition of content in L2 writing assessment potentially underrepresents the target competence in content and language integrated learning (CLIL), which aims to foster not only L2 proficiency but also critical thinking skills and subject knowledge. This study aims to conceptualize the construct of content in CLIL by exploring subject specialists’ perspectives on essays’ content quality in a CLIL context. Eleven researchers of English as a lingua franca (ELF) rated the content quality of research-based argumentative essays on ELF submitted in a CLIL course and produced think-aloud protocols. This study explored some essay features that have not been considered relevant in language assessment but are essential in the CLIL context, including the accuracy of the content, presence and quality of research, and presence of elements required in academic essays. Furthermore, the findings of this study confirmed that the components of content often addressed in language assessment (e.g., elaboration and logicality) are pertinent to writing assessment in CLIL. The manner in which subject specialists construe the content quality of essays on their specialized discipline can deepen the current understanding of content in CLIL.
BallP.KellyK.CleggJ. (2015). Putting CLIL into practice. Oxford University Press.
4.
BarkaouiK. (2011). Think-aloud protocols in research on essay rating: An empirical study of their veridicality and reactivity. Language Testing, 28(1), 51–75. https://doi.org/10.1177/0265532210376379
5.
BarkaouiK. (2020). Deepening and broadening our understanding of second language writing ability. In RiaziA. M.ShiL.BarkaouiK. (Eds.), Studies and essays on learning, teaching and assessing L2 writing in honour of Alister Cumming (pp. 296–308). Cambridge Scholars Publishing.
6.
BeanJ. C.MelzerD. (2021). Engaging ideas: The professor’s guide to integrating writing, critical thinking, and active learning in the classroom (3rd ed.). Jossey-Bass.
7.
BraunV.ClarkeV. (2022). Thematic analysis: A practical guide. Sage.
8.
BukhariN.JamalJ.IsmailA.ShamsuddinJ. (2021). Assessment rubric for research report writing: A tool for supervision. Malaysian Journal of Learning and Instruction, 18(2), 1–43. https://doi.org/10.32890/mjli2021.18.2.1
9.
CandlinC. N.CromptonP.HatimB. (2016). Academic writing step by step: A research-based approach. Equinox.
10.
CarrN. T. (2011). Designing and analyzing language tests. Oxford University Press.
11.
CloudN.GeneseeF.HamayanE. (2000). Dual language instruction: A handbook for enriched education. Heinle & Heinle.
12.
Council of Europe. (2001). Common European framework of reference for languages: Learning, teaching, assessment. Cambridge University Press.
13.
CoyleD.HoodP.MarshD. (2010). CLIL: Content and language integrated learning. Cambridge University Press.
14.
CummingA. H.KantorR.PowersD. E. (2001). Scoring TOEFL essays and TOEFL 2000 prototype writing tasks: An investigation into raters’ decision making and development of a preliminary analytic framework (TOEFL Monograph No. MS-22). Educational Testing Service.
15.
DaleL.van der EsW.TannerR. (2011). CLIL skills. European Platform.
16.
DriskoJ. W.MaschiT. (2016). Content analysis. Oxford University Press.
EricssonK. A.SimonH. A. (1993). Protocol analysis: Verbal reports as data. MIT Press.
19.
GallowayN.RoseH. (2015). Introducing global Englishes. Routledge.
20.
GaoY.DribanA.McManusB. X.MusiE.DaviesP. M.MuresanS.PassonneauR. J. (2019). Rubric reliability and annotation of content and argument in source-based argument essays. In YannakoudakisH.KochmarE.LeacockC.MadnaniN.PilánI.ZeschT. (Eds.), Proceedings of the fourteenth workshop on innovative use of NLP for building educational applications (pp. 507–518). Association for Computational Linguistics.
21.
GarzaN. F.Finkenstaedt-QuinnS. A.WilhelmC. A.KoutmousK. S.ShultzG. V. (2021). Reporting biochemistry to the general public through a science communication writing assessment. Journal of Chemical Education, 98(3), 930–934. https://doi.org/10.1021/acs.jchemed.0c01176
22.
GebrilA.PlakansL. (2014). Assembling validity evidence for assessing academic writing: Rater reactions to integrated tasks. Assessing Writing, 21, 56–73. https://doi.org/10.1016/j.asw.2014.03.002
23.
GreenA. (1998). Verbal protocol analysis in language testing research. Cambridge University Press.
24.
HirvelaA. (2017). Argumentation and second language writing: Are we missing the boat?Journal of Second Language Writing, 36, 69–74. https://doi.org/10.1016/j.jslw.2017.05.002
25.
HylandK. (2019). Second language writing (2nd ed.). Cambridge University Press.
26.
JacobsH.ZinkgrafS.WormuthD.HartfielV.HugheyJ. (1981). Testing ESL composition: A practical approach. Newbury House.
27.
JenkinsJ. (2002). A sociolinguistically based, empirically researched pronunciation syllabus for English as an International Language. Applied Linguistics, 23(1), 83–103. https://doi.org/10.1093/applin/23.1.83
28.
JenkinsJ. (2015). Global Englishes (3rd ed.). Routledge.
KuikenF.VedderI. (2017). Functional adequacy in L2 writing: Towards a new rating scale. Language Testing, 34(3), 321–336. https://doi.org/10.1177/0265532216663991
31.
LeeY.-W.GentileC.KantorR. (2008). Analytic scoring of TOEFL® CBT essays: Scores from humans and e-rater® (TOEFL Research Report No. RR-81). Educational Testing Service.
32.
LiH.HeL. (2015). A comparison of EFL raters’ essay-rating processes across two types of rating scales. Language Assessment Quarterly, 12(2), 178–212. https://doi.org/10.1080/15434303.2015.1011738
33.
LlosaL. (2017). Assessing students’ content knowledge and language proficiency. In ShohamyE.OrI. G.MayS (Eds.), Language testing and assessment (3rd ed., pp. 3–14). Springer.
34.
MacaroE. (2018). English medium instruction. Oxford University Press.
35.
MehistoP.TingT. (2017). CLIL essentials for secondary school teachers. Cambridge University Press.
36.
NosichG. (2022). Critical writing: A guide to writing a paper using the concepts and processes of critical thinking. Rowman & Littlefield.
37.
O’HaganS. (2014). Variability in assessor responses to undergraduate essays: An issue for assessment quality in higher education. Peter Lang.
38.
PaulR.ElderL. (2021). Critical thinking: Tools for taking charge of your learning and your life (4th ed.). Rowman & Littlefield.
39.
PetelinR. (2022). How writing works: A field guide to effective writing (2nd ed.). Routledge.
40.
PolioC.LimJ. (2020). Revising a writing rubric based on raters’ comments: Does it result in more valid and reliable assessment? In RiaziA. M.ShiL.BarkaouiK. (Eds.), Studies and essays on learning, teaching and assessing L2 writing in honour of Alister Cumming (pp. 336–364). Cambridge Scholars Publishing.
41.
PurpuraJ. E. (2017). Assessing meaning. In ShohamyE.OrI. G.MayS. (Eds.), Language testing and assessment (3rd ed., pp. 33–61). Springer.
42.
SatoT. (2023). Assessment in CLIL. In BanegasD. L.Zappa-HollmanS. (Eds.), The Routledge handbook of content and language integrated learning (pp. 355–370). Routledge.
43.
SatoT.McNamaraT. (2019). What counts in second language oral communication ability? The perspective of linguistic laypersons. Applied Linguistics, 40(6), 894–916. https://doi.org/10.1093/applin/amy032
44.
SatoT.YokomotoK.MackenzieG. (2021). Current practice and challenges of assessment in CLIL in a Japanese university context. In HemmiC.BanegasD. L. (Eds.), International perspectives on CLIL (pp. 63–83). Palgrave Macmillan.
45.
Schmidt-UnterbergerB. (2018). The English-medium paradigm: A conceptualisation of English-medium teaching in higher education. International Journal of Bilingual Education and Bilingualism, 21(5), 527–539. https://doi.org/10.1080/13670050.2018.1491949
46.
SeidlhoferB. (2011). Understanding English as a lingua franca. Oxford University Press.
47.
SpandelV.StigginsR. J. (1997). Creating writers: Linking writing assessment and instruction (2nd ed.). Longman.
48.
University of Cambridge Local Examinations Syndicate. (2020). Assessing writing for Cambridge English qualifications: A guide for teachers. Cambridge English Language Assessment.
49.
WalvoordB. E.AndersonV. J. (2010). Effective grading: A tool for learning and assessment in college (2nd ed.). Jossey-Bass.
50.
WeigleS. C. (2002). Assessing writing. Cambridge University Press.
51.
WeigleS. C.MonteeM. (2012). Raters’ perceptions of textual borrowing in integrated writing tasks. In Van SteendamE.TillemaM.RijlaarsdamG.Van Den BerghH. (Eds.), Measuring writing: Recent insights into theory, methodology and practices (pp. 117–151). Brill.
52.
WetteR. (2021). Writing using sources for academic purposes: Theory, research and practice. Routledge.
53.
ZhangY.ElderC. (2011). Judgments of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs?Language Testing, 28(1), 31–50. https://doi.org/10.1177/0265532209360671
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
