Given the lack of computational tools available for assessing second language (L2) production in Korean, this study introduces a novel automated tool called the Korean Syntactic Complexity Analyzer (KOSCA) for measuring syntactic complexity in L2 Korean production. As an open-source graphic user interface (GUI) developed in Python, KOSCA provides seven indices of syntactic complexity, including traditional and Korean-specific ones. Its validity was tested by investigating whether the syntactic complexity indices measured by it in L2 written and spoken production could explain the variability of L2 Korean learners’ proficiency. The results of mixed-effects regression analyses showed that all seven indices significantly accounted for learner proficiency in Korean. Subsequent stepwise multiple regression analyses revealed that the syntactic complexity indices explained 56.0% of the total variance in proficiency for the written data and 54.4% for the spoken data. These findings underscore the validity of the syntactic complexity indices measured by KOSCA as reliable indicators of L2 Korean proficiency, which can serve as a valuable resource for researchers and educators in the field of L2 Korean learning and assessment.
AiH.LuX. (2013). A corpus-based comparison of syntactic complexity in NNS and NS university students’ writing. In Díaz-NegrilloA.BallierN.ThompsonP. (Eds.), Automatic treatment and analysis of learner corpus data (pp. 249–264). John Benjamins. http://doi.org/10.1075/scl.59.15ai
3.
AkaikeH. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716–723. http://doi.org/10.1109/TAC.1974.1100705
4.
AlexopoulouT.MichelM.MurakamiA.MeurersD. (2017). Task effects on linguistic complexity and accuracy: A large-scale learner corpus analysis employing natural language processing techniques. Language Learning, 67, 180–208. https://doi.org/10.1111/lang.12232
5.
Bardovi-HarligK. (1992). A second look at T-unit analysis: Reconsidering the sentence. TESOL Quarterly, 26, 390–395. http://doi.org/10.2307/3587016
6.
BarrD. J.LevyR.ScheepersC.TilyH. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68, 255–278. http://doi.org/10.1016/j.jml.2012.11.001
7.
BatesD.MaechlerM.BolkerB.WalkerS. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48. http://doi.org/10.18637/jss.v067.i01
8.
BiberD.GrayB.PoonponK. (2011). Should we use characteristics of conversation to measure grammatical complexity in L2 writing development?TESOL Quarterly, 45, 5–35. http://doi.org/10.5054/tq.2011.244483
9.
BiberD.GrayB.StaplesS.EgbertJ. (2020). Investigating grammatical complexity in l2 English writing research: Linguistic description versus predictive measurement. Journal of English for Academic Purposes, 46, 100869. https://doi.org/10.1016/j.jeap.2020.100869
10.
BrownR. (1973). A first language: The early stages. Harvard University Press. http://doi.org/10.4159/harvard.9780674732469
CrossleyS. A. (2020). Linguistic features in writing quality and development: An overview. Journal of Writing Research, 11, 415–443. https://doi.org/10.17239/jowr-2020.11.03.01
13.
FoxJ. (1991). Regression diagnostics. Sage.
14.
FrançoisT.FaironC. (2012, July). An “AI readability” formula for French as a foreign language. In TsujiiJ.HendersonJ.PaşcaM. (Eds.), Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (pp. 466–477). Association for Computational Linguistics. https://aclanthology.org/D12-1043
15.
GaillatT.SimpkinA.BallierN.StearnsB.SousaA.BouyéM.ZarroukM. (2022). Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach. ReCALL, 34, 130–146. https://doi.org/10.1017/S095834402100029X
16.
GeorgilaK.LemonO.HendersonJ.MooreJ. D. (2009). Automatic annotation of context and speech acts for dialogue corpora. Natural Language Engineering, 15, 315–353. http://doi.org/10.1017/S1351324909005105
17.
GriesS. T. (2021). (Generalized linear) mixed-effects modeling: A learner corpus example. Language Learning, 71, 757–798. http://doi.org/10.1111/lang.12448
18.
HallidayM. A. K. (1989). Spoken and written language. Oxford University Press.
19.
HawkinsJ. A.ButteryP. (2010). Criterial features in learner corpora: Theory and illustrations. English Profile Journal, 1, Article e5. https://doi.org/10.1017/S2041536210000103
20.
HousenA.KuikenF. (2009). Complexity, accuracy, and fluency in second language acquisition. Applied Linguistics, 30, 461–473. https://doi.org/10.1093/applin/amp048
HwangH. (2023b, November21). Korean Syntactic Complexity Analyzer (KOSCA): An NLP application for the analysis of syntactic complexity in second language production. https://doi.org/10.17605/OSF.IO/YJCZE.
23.
HwangH.JungH.KimH. (2020). Effects of written versus spoken production modalities on syntactic complexity measures in beginning-level child EFL learners. The Modern Language Journal, 104, 267–283. http://doi.org/10.1111/modl.12626
24.
HwangH.KimH. (2024). An automated tool for assessing sentence complexity in second language Korean production. OASIS Summary of Hwang, H., & Kim, H (2024) in Language Testing. https://oasis-database.org/
25.
IwashitaN.BrownA.McNamaraT.O’HaganS. (2008). Assessed levels of second language speaking proficiency: How distinct?Applied Linguistics, 29, 24–49. http://doi.org/10.1093/applin/amm017
JiangJ.BiP.LiuH. (2019). Syntactic complexity development in the writings of EFL learners: Insights from a dependency syntactically-annotated corpus. Journal of Second Language Writing, 46, 100666. https://doi.org/10.1016/j.jslw.2019.100666
28.
KelloggR. T. (1996). A model of working memory in writing. In LevyC. M.RansdellS. (Eds.), The science of writing: Theories, methods, individual differences and applications (pp. 57–71). Lawrence Erlbaum Associates.
29.
KimH.HwangH. (2022). Assessing verb-construction integration in young learners of English as a foreign language. Language Learning, 72, 497–533. http://doi.org/10.1111/lang.12480
30.
KimY.NamJ.LeeS.-Y. (2016). Correlation of proficiency with complexity, accuracy, and fluency in spoken and written production: Evidence from L2 Korean. Journal of the National Council of Less Commonly Taught Languages, 19, 147–181. https://doaj.org/article/5b33f84d53354f8d8ac35353aa2c107e
31.
KisselevO.KlimovA.KopotevM. (2022). Syntactic complexity measures as linguistic correlates of proficiency level in learner Russian. In Leńko-SzymańskaA.GötzS. (Eds.), Complexity, accuracy and fluency in learner corpus research (pp. 51–80). John Benjamins. https://doi.org/10.1075/scl.104.03kis
32.
KisselevO.SoyanR.PastushenkovD.MerrillJ. (2022). Measuring writing development and proficiency gains using indices of lexical and syntactic complexity: Evidence from longitudinal Russian learner corpus data. The Modern Language Journal, 106, 798–817. https://doi.org/10.1111/modl.12808
33.
Kivy. (2023). Kivy: The open source Python app development framework. https://kivy.org/
34.
KyleK. (2021). Natural language processing for learner corpus research. International Journal of Learner Corpus Research, 7, 1–16. http://doi.org/10.1075/ijlcr.00019.int
35.
KyleK.CrossleyS. A. (2017). Assessing syntactic sophistication in L2 writing: A usage-based approach. Language Testing, 34, 513–535. http://doi.org/10.1177/0265532217712554
36.
KyleK.CrossleyS. A.BergerC. (2018). The tool for the automatic analysis of lexical sophistication (TAALES): Version 2.0. Behavior Research Methods, 50, 1030–1046. http://doi.org/10.3758/s13428-017-0924-4
37.
LambertC.NakamuraS. (2019). Proficiency-related variation in syntactic complexity: A study of English L1 and L2 oral descriptive discourse. International Journal of Applied Linguistics, 29, 248–264. http://doi.org/10.1111/ijal.12224
38.
LeeD. J.YeonJ. H.HwangI. B.LeeS. G. (2010). KKMA: a tool for utilizing Sejong corpus based on relational database. The KIISE Transactions on Computing Practice, 16, 1046–10. http://kkma.snu.ac.kr/
39.
LeeG. G.ChaJ.LeeJ. H. (2002). Syllable-pattern-based unknown-morpheme segmentation and estimation for hybrid part-of-speech tagging of Korean. Computational Linguistics, 28, 53–70. http://doi.org/10.1162/089120102317341774
40.
LeeM. (2019). Effects of case-marking on the anticipatory processing of Korean Sentences. Journal of Cognitive Science, 20, 339–364. https://doi.org/10.17791/jcs.2019.20.3.339
41.
LeveltW. (1989). Speaking: From intention to articulation. MIT Press.
42.
LuX. (2011). A corpus-based evaluation of syntactic complexity measures as indices of college-level ESL writers’ language development. TESOL Quarterly, 45, 36–62. https://doi.org/10.5054/tq.2011.240859
43.
McNamaraD. S.LouwerseM. M.McCarthyP. M.GraesserA. C. (2010). Coh-Metrix: Capturing linguistic features of cohesion. Discourse Processes, 47, 292–330. http://doi.org/10.1080/01638530902959943
44.
MinM.LeeJ. J.LeeK. (2022). Detecting illegal online gambling (IOG) services in the mobile environment. Security and Communication Networks, 2022, 3286623. https://doi.org/10.1155/2022/3286623
45.
National Institute for International Education. (2023). Test of Proficiency in Korean. https://www.topik.go.kr/
NoG. (2012). Acquisition of case markers and grammatical functions. In LeeC. (Ed.), The handbook of East Asian psycholinguistics (pp. 50–62). Cambridge University Press.
48.
NorrisJ. M.OrtegaL. (2009). Towards an organic approach to investigating CAF in instructed SLA: The case of complexity. Applied Linguistics, 30, 555–578. https://doi.org/10.1093/applin/amp044
49.
O’GradyW.LeeM.ChooM. (2003). A subject-object asymmetry in the acquisition of relative clauses in Korean as a second language. Studies in Second Language Acquisition, 25, 433–448. http://doi.org/10.1017/S0272263103000172
50.
OrtegaL. (2003). Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing. Applied Linguistics, 24, 492–518. http://doi.org/10.1093/applin/24.4.492
51.
OrtegaL. (2015). Syntactic complexity in L2 writing: Progress and expansion. Journal of Second Language Writing, 29, 82–94. http://doi.org/10.1016/j.jslw.2015.06.008
52.
ParkE. (2014). KoNLPy: Korean natural language processing in Python. In ParkE. L.ChoS. (Eds.), Proceedings of the 26th Annual Conference on Human & Cognitive Language Technology (pp. 133–136). Korean Institute of Information Scientists and Engineers, The Korean Society for Cognitive Science.
53.
PinkerS. (1989). Learnability and cognition: The acquisition of argument structure. MIT Press. http://doi.org/10.7551/mitpress/9700.001.0001
54.
RobitzschA. (2020). Why ordinal variables can (almost) always be treated as continuous variables: Clarifying assumptions of robust continuous and ordinal factor analysis estimation methods. Frontiers in Education, 5, Article 589965. https://doi.org/10.3389/feduc.2020.589965
55.
SeoS. J. (2009). Study on the interlanguage development of Korean language learners by syntactic proficiency assessment: Based on analyzing syntactic features of writings [Master’s thesis, Yonsei University]. Yonsei University Library. https://library.yonsei.ac.kr/search/detail/CATTOT000000732514
56.
ShinS. C. (2016). English L1-Korean L2 learners’ cognitive knowledge and difficulty of grammatical error items. Language Facts and Perspectives, 37, 173–201.
VuD. T.YuG.LeeC.KimJ. (2022). Text data augmentation for the Korean language. Applied Sciences, 12(7), 3425. https://doi.org/10.3390/app12073425
59.
WeissZ.ChenX.MeurersD. (2021). Using broad linguistic complexity modeling for crosslingual readability assessment. In AlfterD.VolodinaE.PilanI.GraënJ.BorinL. (Eds.), Proceedings of the 10th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2021) (pp. 38–54). LiU Electronic Press. https://aclanthology.org/2021.nlp4call-1.4
60.
Wolfe-QuinteroK.InagakiS.KimH.-Y. (1998). Second language development in writing: Measures of fluency, accuracy & complexity. University of Hawaii Press.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
