This study examined the writing score and writing feature changes of 562 repeat test takers who took the Canadian English Language Proficiency Index Program–General (CELPIP–General) test at least three times, with a short (30–40 day) interval between the first and second attempts and a longer (90–180 day) interval between the first and third attempts. Analysis was conducted to uncover whether changes occurred at different testing durations (short vs. long) and whether the observed changes varied across repeater’s initial proficiency groups (low, mid, high). The writing scores measured by CELPIP bands showed great stability over the 6-month period, but the trends of development differed by proficiency group. Low proficiency test takers were more likely to have faster observable score gains, compared to the medium proficiency group, whereas high proficiency repeaters may not maintain their score levels at later attempts. Writing quality was analyzed using natural language processing (NLP) tools. Results suggested that for all proficiency groups, lexical features were more likely to improve over the 6-month period, with some measures showing improvement at 1 month; features in cohesion and syntactic sophistication, however, did not change significantly.
AndrulisR. S.StarrL. M.FurstL. M. (1978).
The effects of repeaters on test equating. Educational and Psychological Measurement,
38(2), 341–349. https://doi.org/10.1177/001316447803800215
2.
ArnaudP. (1984). The lexical richness of L2 written productions and the validity of vocabulary tests. In CulhaneT.Klein-BradleyC.StevensonD. K. (Eds.), Practice and problems in language testing: Papers from the International Symposium on Language Testing (pp. 14–28).
Colchester: University of Essex.
3.
BarkaouiK. (2016). What changes and what doesn’t? An examination of changes in the linguistic characteristics of IELTS repeaters’ Writing Task 2 scripts. (IELTS Research Report Series, No. 3.) https://www.ielts.org/teaching-and-research/research-reports
4.
BarkaouiK. (2017).
Examining repeaters’ performance on second language proficiency tests: A review and a call for research. Language Assessment Quarterly,
14(4), 420–431. https://doi.org/10.1080/15434303.2017.1347790
5.
BarkaouiK. (2019).
Examining sources of variability in repeaters’ L2 writing scores: The case of the PTE Academic writing section. Language Testing,
36(1), 3–25. https://doi.org/10.1177/0265532217750692
6.
BatesD.MächlerM.BolkerB.WalkerS. (2015).
Fitting linear mixed-effects models using lme4. Journal of Statistical Software,
67(1), 1–48. https://doi.org/10.18637/jss.v067.i01
7.
BiberD.GrayB.PoonponK. (2011).
Should we use characteristics of conversation to measure grammatical complexity in L2 writing development?TESOL Quarterly,
45(1), 5–35. https://doi.org/10.5054/tq.2011.244483
BurchB.EgbertJ.BiberD. (2017).
Measuring and interpreting lexical dispersion in corpus linguistics. Journal of Research Design and Statistics in Linguistics and Communication Science,
3(2), 189–216. https://doi.org/10.1558/jrds.33066
ColtheartM. (1981).
The MRC psycholinguistic database. Quarterly Journal of Experimental Psychology A: Human Experimental Psychology,
33A(4), 497–505. https://doi.org/10.1080/14640748108400805
14.
ConnorU. (1984).
A study of cohesion and coherence in English as a second language students’ writing. Papers in Linguistics,
17(3), 307–316. https://doi.org/10.1080/08351818409389208
15.
Council of Europe. (2001). Common European framework of reference for languages: Learning, teaching, assessment.
Cambridge University Press. https://rm.coe.int/1680459f97
16.
CovingtonM. A.McFallJ. D. (2010).
Cutting the Gordian Knot: The Moving-Average Type–Token Ratio (MATTR).Journal of Quantitative Linguistics,
17(2), 94–100. https://doi.org/10.1080/09296171003643098
17.
CrossleyS. A.FengS.CaiZ.McNamaraD. S. (2013). Computer simulations of MRC Psycholinguistic Database word properties: Concreteness, familiarity, and imageability. In JarvisS.DallerH. (Eds.), Vocabulary knowledge: Human ratings and automated measures (pp. 135–156).
John Benjamins. https://doi.org/10.1075/sibil.47.07ch5
18.
CrossleyS. A.KyleK.McNamaraD. S. (2016).
The Tool for the Automatic Analysis of Text Cohesion (TAACO): Automatic assessment of local, global, and text cohesion. Behavior Research Methods,
48(4), 1227–1237. https://doi.org/10.3758/s13428-015-0651-7
19.
CrossleyS. A.SalsburyT.McNamaraD. (2010).
The development of polysemy and frequency use in English second language speakers. Language Learning,
60(3), 573–605. https://doi.org/10.1111/j.1467-9922.2010.00568.x
20.
CrossleyS. A.SubtireluN.SalsburyT. (2013).
Frequency effects or context effects in second language word learning: What predicts early lexical production?Studies in Second Language Acquisition,
35(4), 727–755. https://doi.org/10.1017/S0272263113000375
21.
CrossleyS. A.WestonJ. W.McLain SullivanS. T.McNamaraD. S. (2011).
The development of writing proficiency as a function of grade level: A linguistic analysis. Written Communication,
28(3), 282–311. https://doi.org/10.1177/0741088311410188
22.
CummingA.KantorR.BabaK.ErdosyU.EouanzouiK.JamesM. (2005).
Differences in written discourse in independent and integrated prototype tasks for next generation TOEFL. Assessing Writing,
10(1), 5–43. https://doi.org/10.1016/j.asw.2005.02.001
DaviesM. (2009).
The 385+ million word Corpus of Contemporary American English (1990–2008+): Design, architecture, and linguistic insights. International Journal of Corpus Linguistics,
14(2), 159–190. https://doi.org/10.1075/ijcl.14.2.02dav
25.
EllisN. C. (2002).
Frequency effects in language processing: A review with implications for theories of implicit and explicit language acquisition. Studies in Second Language Acquisition,
24(2), 143–188. https://doi.org/10.1017/S0272263102002024
26.
EngberC. A. (1995).
The relationship of lexical proficiency to the quality of ESL compositions. Journal of Second Language Writing,
4(2), 139–155. https://doi.org/10.1016/1060-3743(95)90004-7
FergadiotisG.WrightH. H.WestT. M. (2013).
Measuring lexical diversity in narrative discourse of people with aphasia. American Journal of Speech-Language Pathology,
22(2). https://doi.org/10.1044/1058-0360(2013/12-0083)
29.
Goldberg, A. E. (1995). Constructions: A Construction Grammar approach to argument structure. Chicago: University of Chicago Press.
30.
GraesserA. C.McNamaraD. S.KulikowichJ. M. (2011).
Coh-Metrix: Providing multilevel analyses of text characteristics. Educational Researcher,
40(5), 223–234. https://doi.org/10.3102/0013189X11413260
31.
GreenA. (2005).
EAP study recommendations and score gains on the IELTS Academic writing test. Assessing Writing,
10(1), 44–60. https://doi.org/10.1016/j.asw.2005.02.002
32.
GriesS. T. (2008).
Dispersions and adjusted frequencies in corpora. International Journal of Corpus Linguistics,
13(4), 403–437. https://doi.org/10.1075/ijcl.13.4.02gri
33.
GriesS. T. (2010). Dispersions and adjusted frequencies in corpora: Further explorations. In GriesS. T.WulffS.DaviesM. (Eds.), Corpus linguistic applications: Current studies, new directions (pp. 197–212).
Rodopi. https://doi.org/10.1163/9789042028012_014
34.
HallidayM. A. K.HasanR. (1976). Cohesion in English.
Longman.
KemmerS.BarlowM. (Eds.). (2000). Usage-based models of language.
Center for the Study of Language and Information.
38.
KowalI. (2014).
Fluency in second language writing: A developmental perspective. Studia Linguistica Universitatis Iagellonicae Cracoviensis,
131(2014), 229–246. https://doi.org/10.4467/20834624SL.14.013.2321
39.
KyleK. (2016). Measuring syntactic development in L2 writing: Fine grained indices of syntactic complexity and usage-based indices of syntactic sophistication (doctoral dissertation). http://scholarworks.gsu.edu/alesl_diss/35
40.
KyleK.CrossleyS. A. (2015).
Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Quarterly,
49(4), 757–786. https://doi.org/10.1002/tesq.194
41.
KyleK.CrossleyS. A. (2017).
Assessing syntactic sophistication in L2 writing: A usage-based approach. Language Testing,
34(4), 513–535. https://doi.org/10.1177/0265532217712554
42.
LangackerR. W. (1987). Foundations of cognitive grammar, vol. II: Theoretical prerequisites.
Stanford University Press.
LuX. (2010).
Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics,
15(4), 474–496. https://doi.org/10.1075/ijcl.15.4.02lu
48.
MalvernD.RichardsB.ChipereN.DuránP. (2004). Lexical diversity and language development: Quantification and assessment.
Palgrave Macmillan. https://doi.org/10.1057/9780230511804
49.
McCarthyP. M.JarvisS. (2010).
MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods,
42(2), 381–392. https://doi.org/10.3758/BRM.42.2.381
50.
McNamaraD. S.CrossleyS. A.McCarthyP. M. (2010).
Linguistic features of writing quality. Written Communication,
27(1), 57–86. https://doi.org/10.1177/0741088309351547
51.
MearaP. (1996). The dimensions of lexical competence. In BrownG.MalmkjaerK.WilliamsJ. (Eds.), Performance and competence in second language acquisition (pp. 35–53).
Cambridge University Press.
52.
OrtegaL. (2003).
Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing. Applied Linguistics,
24(4), 492–518. https://doi.org/10.1093/applin/24.4.492
53.
PalviainenÅ.KalajaP.MäntyläK. (2012).
Development of L2 writing: Fluency and proficiency. AFinLA-e: Soveltavan Kielitieteen Tutkimuksia,
4, 47–59.
54.
Paragon Testing Enterprises. (2015). CELPIP study guide: Reading and writing.
Paragon Testing Enterprises.
55.
RaymondM. R.KahramanN.SwygertK. A.BalogK. P. (2011).
Evaluating construct equivalence and criterion-related validity for repeat examinees on a standardized patient examination. Academic Medicine,
86(10), 1253–1259. https://doi.org/10.1097/ACM.0b013e31822bc0a4
56.
RaymondM. R.NeustelS.AndersonD. (2009).
Same-form retest effects on credentialing examinations. Educational Measurement: Issues and Practice,
28(2), 19–27. https://doi.org/10.1111/j.1745-3992.2009.00144.x
57.
RaymondM. R.SwygertK. A.KahramanN. (2012).
Psychometric equivalence of ratings for repeat examinees on a performance assessment for physician licensure. Journal of Educational Measurement,
49(4), 339–361. https://doi.org/10.1111/j.1745-3984.2012.00180.x
58.
ReadJ. (1998). Validating a test to measure depth of vocabulary knowledge. In KunnanA. (Ed.), Validation in language assessment (pp. 41–60).
Lawrence Erlbaum.
SalsburyT.CrossleyS. A.McNamaraD. S. (2011).
Psycholinguistic word information in second language oral discourse. Second Language Research,
27(3), 343–360. https://doi.org/10.1177/0267658310395851
61.
StaplesS.EgbertJ.BiberD.GrayB. (2016).
Academic writing development at the university level: Phrasal and clausal complexity across level of study, discipline, and genre. Written Communication,
33(2), 149–183. https://doi.org/10.1177/0741088316631527
62.
Tomasello, M. (2003). Constructing a language: A usage-based theory of language acquisition. Harvard University Press.
63.
ToddR. W.KhongputS.DarasawangP. (2007).
Coherence, cohesion and comments on students’ academic essays. Assessing Writing,
12(1), 10–25. https://doi.org/10.1016/j.asw.2007.02.002
64.
Van Waes, L., & Leijten, M. (2015). Fluency in writing: A multidimensional perspective on writing fluency applied to L1 and L2. Computers and Composition, 38, 79–95. https://doi.org/10.1016/j.compcom.2015.09.012
65.
WilsonK. M. (1987).
Patterns of test taking and score change for examinees who repeat the Test of English as a Foreign Language. ETS Research Report Series,
1987(1), 1–68. https://doi.org/10.1002/j.2330-8516.1987.tb00207.x
66.
Wolfe-QuinteroK.InagakiS.KimH.-Y. (1998). Second language development in writing: Measures of fluency, accuracy and complexity. (Technical Report 17.) University of Hawaii Press.
67.
YuG. (2010).
Lexical diversity in writing and speaking task performances. Applied Linguistics,
31(2), 236–259. https://doi.org/10.1093/applin/amp024
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
