In this work, we explore the use of digital technologies and statistical analysis to monitor how Italian secondary school students’ writing changes over time and how comparisons can be made across different high school types. We analyzed more than 2,000 exam essays written by Italian high school students over 13 years and in five different school types. Four indicators of writing characteristics were considered—text length, text complexity, and two indicators of source use, all extracted using natural language processing tools—which provided insights into students’ citation practices over time and in different school contexts. In particular, we measured the portion of students’ essays that included text from source material as well as the amount of copied text that was not properly referenced. We found that student essays became shorter in length over time while also getting more complex. We also found that the tendency to copy uncited text in the essay decreased. High school curricula predict different writing strategies: essays written by students attending scientific and humanistic high schools are longer and less subject to incorrect citations. We argue that such text analysis enables the study of writing features in high school classes and supports the evaluation of curricula.
AbelA.GlaznieksA.NicolasL.StemleE. (2016). An extended version of the koko German l1 learner corpus [Conference session]. Proceedings of the Third Italian Conference on Computational Linguistics CLiC-it 2016, Bari, Italy.
2.
Angelil-CarterS. (2000). Understanding plagiarism differently. In LeibowitzB.MohamedY. (Eds.), Routes to writing in Southern Africa (pp. 154–177). Silk Road International Publishers.
3.
AugstG.DisselhoffK.HenrichA.PohlT.VölzingP. L. (2007). Text-Sorten-Kompetenz. Eine echte Longitudinalstudie zur Ent wick lung der Textkompetenz im Grundschulalter. Lang.
4.
BaileyA. (2007). The language demands of school: Putting academic English to the test. Yale University Press.
5.
BarbagliA.LucisanoP.Dell’OrlettaF.MontemagniS.VenturiG. (2015). CItA: un Corpus di Produzioni Scritte di Apprendenti l’Italiano L1. Annotato con Errori. 31–35. https://doi.org/10.4000/books.aaccademia.1298
6.
Becker-MrotzekM.BöttcherI.DreherJ. (2012). Schreibkompetenz entwickeln und beurteilen. Cornelsen Verlag.
7.
BerklingK.FayJ.GhayoomiM.HeinK.LavalleyR.LinhuberL.StükerS. (2014). A database of freely written texts of German school students for the purpose of automatic spelling error classification [Conference session]. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Granada, Spain.
8.
BoscoloP. (2014). Come scrivono gli adolescenti: un’indagine sulla scrittura scolastica e sulla didattica della scrittura. Il Mulino.
9.
BrunatoD.CiminoA.Dell’OrlettaF.VenturiG.MontemagniS. (2020). Profiling-UD: A tool for linguistic profiling of texts [Conference session]. Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France. https://aclanthology.org/2020.lrec-1.883
10.
BrunatoD.Dell’OrlettaF. (2016). ISACCO: A corpus for investigating spoken and written language development in Italian school–age children. Italian Journal of Computational Linguistics, 2(1), 63-76.
11.
CeskaZ.FoxC. (2009). The influence of text pre-processing on plagiarism detection [Conference session]. Proceedings of the International Conference RANLP-2009. Association for Computational Linguistics, Borovets, Bulgaria.
12.
ChongM.SpeciaL.MitkovR. (2010). Using natural language processing for automatic detection of plagiarism [Conference session]. Proceedings of the 4th International Plagiarism Conference (IPC-2010), Uppsala, Sweden.
13.
ChristenburyL.BomerR.SmagorinskyP. (2011). Handbook of adolescent literacy research. Guilford Press.
14.
CisottoL.NovelloN. (2012). La scrittura di sintesi di studenti del primo anno di scienze della formazione primaria. Italian Journal of Educational Research, 8, 41-57.
15.
ClaeskensG.HjortN. (2008). Model selection and model averaging. Cambridge University Press.
16.
CloughP.StevensonM. (2011). Developing a corpus of plagiarised short answers. Language Resources and Evaluation, 45(1), 5-24.
17.
CrossleyS. A. (2020). Linguistic features in writing quality and development: An overview. Journal of Writing Research, 11(3), 415-443.
18.
CrossleyS. A.KimM. (2022). Linguistic features of writing quality and development: A longitudinal approach. Journal of Writing Analytics, 6(1), 59-93.
19.
CrossleyS. A.KyleK.DascaluM. (2019). The tool for the automatic analysis of cohesion 2.0: Integrating semantic similarity and text overlap. Behavior Research Methods, 51, 14-27.
20.
CrossleyS. A.KyleK.DavenportJ.McNamaraD. S. (2016). Automatic assessment of constructed response data in a chemistry tutor [Conference session]. Proceedings of the 9th International Educational Data Mining. EDM Society, Raleigh, North Carolina, USA.
21.
CrossleyS. A.McNamaraD. (2011). Understanding expert ratings of essay quality: Cohmetrix analyses of first and second language writing. International Journal of Continuing Engineering Education and Life Long Learning, 21(2-3), 170-191.
22.
CrossleyS. A.VarnerL. K.McNamaraD. (2013). Cohesion-based prompt effects in argumentative writing [Conference session]. Proceedings of the 26th International Florida Artificial Intelligence Research Society (FLAIRS) Conference. The AAAI Press, Menlo Park, CA.
23.
CrossleyS. A.WestonJ.McLain SullivanS.McNamaraD. (2011). The development of writing proficiency as a function of grade level: A linguistic analysis. Written Communication, 28(3), 282-311.
24.
DascaluM.Gutu-RobuG.RusetiS.ParaschivI.DessusP.McNamaraD.Trausan-MatuS. (2017). ReaderBench: A multi-lingual framework for analyzing text complexity [Conference session]. 12th European Conference on Technology Enhanced Learning. Tallinn, Estonia.
25.
De SantisC.GattaF. (2013). Notizie dalla scuola. le competenze grammaticali e testuali degli studenti madrelingua all’uscita dalla scuola secondaria. risultati di un’indagine. Studi di grammatica italiana, 31-32, 411-441.
26.
Dell’OrlettaF.MontemagniS.VenturiG. (2011). READ–IT: Assessing readability of Italian texts with a view to text simplification [Conference session]. Second Workshop on Speech and Language Processing for Assistive Technologies. Association for Computational Linguistics, Edinburgh, Scotland, UK.
27.
DunnP.SmythG. (2018). Models for proportions: Binomial GLMs. In DunnP.SmythG. (Eds.), Generalized linear models with examples in R (pp. 333-369). Springer.
28.
FornaraS.CignettiL.DemartiniS. (2016). Come tiscrivo? la scrittura a scuola tra teoria e didattica. Aracne.
29.
FoxJ. (2015). Applied regression analysis and generalized linear models. Sage.
30.
GraesserA.McNamaraD.LouwerseM.CaiZ. (2004). Coh-metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36, 193-202.
31.
HallidayM.MartinJ. (2015). Writing science. Literacy and discursive power. Routledge.
32.
HardinJ.HilbeJ. (2007). Generalized linear models and extensions. Stata Press.
33.
HastieT.PregibonD. (1992). Generalized linear models. In ChambersJ. M.HastieT. J. (Eds.), Statistical models in S, chapter generalized linear model (pp. 197-250). Wadsworth & Brooks/Cole.
34.
HinkelE. (2002). Second language writers’ text. Routledge.
35.
HowardR. M. (1999). Standing in the shadow of giants: Plagiarists, authors, collaborators. Greenwood Publishing Group.
36.
HowardR. M.ServissT.RodrigueT. K. (2010). Writing from sources, writing from sentences. Writing and Pedagogy, 177(2), 177-192.
37.
HylandK. (2002). Activity and evaluation: Reporting practices in academic writing. In FlowerdewJ. (Ed.), Academic discourse (pp. 115-130). Routledge.
38.
JettonT.ShanahanC. (2012). Adolescent literacy in the academic disciplines: General principles and practical strategies. Guilford Press.
39.
LaiA.TetreaultJ. (2018). Discourse coherence in the wild: A dataset, evaluation and methods. arXiv preprint arXiv:1805.04993.
LucisanoP.PiemonteseM. (1988). Gulpease: una formula per la predizione della difficolt`a dei testi in lingua italiana. Scuola e città, 3, 110-124.
42.
ManningC. D.SurdeanuM. B. J.FinkelJ. R.BethardS.McCloskyD. (2014). The Stanford CoreNLP natural language processing toolkit [Conference session]. Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Baltimore, Maryland, USA.
43.
McCutchenD.TeskeP.BankstonC. (2008). Writing and cognition: Implications of the cognitive architecture for learning to write and writing to learn. In BazermanC. (Ed.), Handbook of research on writing: History, society, school, individual (pp. 451-470). Taylor & Francis Group.
44.
McNamaraD. S.CrossleyS. A.McCarthyP. (2010). Linguistic features of writing quality. Written Communication, 27, 57-86.
45.
McNamaraD. S.CrossleyS. A.RoscoeR. D.AllenL. K.DaiJ. (2015). A hierarchical classification approach to automated essay scoring. Assessing Writing, 23, 35-59.
46.
MiaschiA.DavidsonS.BrunatoD.Dell’OrlettaF.SagaeK.Sanchez-GutierrezC. H.VenturiG. (2020). Tracking the evolution of written language competence in l2 Spanish learners [Conference session]. Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, Seattle, WA, USA.
47.
Mott-SmithJ. A. (2013). Viewing student behavior through the lenses of culture and globalization: Two narratives from a US college writing class. Teaching in Higher Education, 18(3), 249-259.
48.
MotterC. (2020). Il percorso di ricerca. Come cambia la scrittura a Scuola. Rapporto di ricerca, IPRASE, 2, 81-111.
49.
OliverE. (1995). The writing quality of seventh, ninth, and eleventh graders, and college freshmen: Does rhetorical specification in writing prompts make a difference?Research in the Teaching of English, 29(4), 422-450.
50.
Palmero AprosioA. (2021). Tint, the Swiss-Army tool for natural language processing in Italian [Conference session]. Proceedings of the Fifth Workshop on Natural Language for Artificial Intelligence (NL4AI 2021) co-located with 20th International Conference of the Italian Association for Artificial Intelligence (AI*IA 2021), online.
51.
ParrJ. (2010). A dual purpose data base for research and diagnostic assessment of student writing. Journal of Writing Research, 2(2), 129-150.
52.
PecorariD. (2008). Academic writing and plagiarism: A linguistic analysis. Bloomsbury Publishing.
53.
R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing.
54.
RobackP.LeglerJ. (2021). Beyond multiple linear regression: Applied generalized linear models and multilevel models in R. CRC Press.
55.
RoigM. (2001). Plagiarism and paraphrasing criteria of college and university professors. Ethics & Behavior, 11(3), 307-323.
56.
RuggianoF. (2013). L’italiano scritto a scuola: fenomeni di lingua in elaborati di studenti di scuola secondaria dal primo al terzo anno. Aracne.
57.
SchleppegrellM. J. (2001). Linguistic features of the language of schooling. Linguistics and Education, 12(4), 431-459.
58.
SchleppegrellM. J. (2013). The role of metalanguage in supporting academic language development. Language learning, 63(Suppl. 1), 153-170.
59.
SnowC. (2010). Academic language and the challenge of reading for learning about science. Science, 328(5977), 450-452.
60.
SprugnoliR.TonelliS.Palmero AprosioA.MorettiG. (2018). Analysing the evolution of students’ writing skills and the impact of neo-standard italian with the help of computational linguistics [Conference session]. Fifth Italian Conference on Computational Linguistics (CLiC-it 2018), Turin, Italy.
61.
TonelliS.Tran ManhK.PiantaE. (2012). Making readability indices readable [Conference session]. First Workshop on Predicting and Improving Text Readability for target reader populations. Association for Computational Linguistics, Montréal, Canada.
62.
vor der BrückT.HartrumpfS.HelbigH. (2008). A readability checker with supervised learning using deep indicators. Informatica, 32(4), 429-435.
63.
VriezeS. (2012). Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychological Methods, 17(2), 228–243. https://doi.org/10.1037/a0027127
64.
Weston-SementelliJ.AllenL. K.McNamaraD. (2016). Comprehension and writing strategy training improves performance on content-specific source-based writing tasks. International Journal of Artificial Intelligence in Education, 28, 106–137.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
