A Multi-Strategy Computer-Assisted EFL Writing Learning System With Deep Learning Incorporated and Its Effects on Learning: A Writing Feedback Perspective

Abstract

Language learning has increasingly benefited from Computer-Assisted Language Learning (CALL) technologies, especially with Artificial Intelligence involved in recent years. CALL in writing learning acknowledged as the core of language learning is being realized by technologies like Automated Writing Evaluation (AWE), and Automated Essay Scoring (AES), which have developed considerably in both computer and language education fields. AWE has effectively enhanced EFL students’ writing performance to some extent, but such technology can only provide an evaluation in the form of scores, the majority of which are based on holistic scoring, resulting in the inability to provide comprehensive and detailed content-based feedback. In order to provide not only the writing multiple trait-specific evaluation scores, but also detailed writing feedback, we proposed a computer-assisted EFL writing learning system incorporating the neural network models and a couple of semantic-based NLP techniques, MsCAEWL, which fully meets the requirements of writing feedback theory, i.e., multiple, continuous, timely, clear, and multi-aspect guidance interactive feedback. The results of comparison experiments with the AWE baseline models and human raters demonstrated the superiority and the high correlation contained by the proposed system. The independent-sample t-test and paired-sample t-test results of the experiments on MsCAEWL effect validation suggested the significant impact of our proposed system in enhancing students’ EFL writing proficiency.

Introduction

Computer-Assisted Language Learning (CALL) technologies have profited greatly from the quick development of Artificial Intelligence and have been increasingly involved and developed in language learning. Writing to learn and writing to learn is identified as essential component in ESL/EFL (Zhang, 2013), the Computer-assisted writing evaluation technology thereby has drawn adequate attention both in computer science and language education.

Writing learning has traditionally been viewed as crucially dependent on writing evaluation or feedback (Lloyd-Jones, 1977). Writing feedback helps improve learning outcomes, draws L2 learners’ attention to the gap between the target language and the interlanguage, and stimulates L2 learners to internalize L2 knowledge. At the same time, teachers can also adjust the teaching method through feedback (Zamel, 1982). Effective feedback is supposed to include multiple modifications and reciprocating processes and to indicate revision instructions covering dimensions such as organization, content, and mechanics (Cohen & Cavalcanti, 1990; Ferris, 1995; Hedgcock & Lefkowitz, 1994).

For many non-native English-speaking countries, EFL/ESL writing is the least gratifying task for both teachers and students, especially in China (Mo, 2012). EFL teachers in Chinese high schools or colleges have to face 40-80 students in one class, which is the only one of 2–5 classes he or she teaches in a semester. Thus, the workload of the TEFL is abnormally high. Moreover, in China, the largest English learning group is non-English majors, and there is no exclusive course for writing in high schools and universities for this group. The large class size makes it almost impossible for teachers to assign enough writing tasks, not to mention providing detailed evaluations or individual instructions. Most of the writing feedback students receive is vague, overall, or inconsistent (Beach & Friedrich, 2006).

Recently the TEFL, especially writing teaching, has remarkably benefited from the rapid development of CALL technologies. Computer-assisted writing evaluation technology, known as Automated Writing Evaluation (AWE), Automated Essay Scoring (AES), and Computer essay grading, has emerged and flourished since the 1960s. In addition to scoring compositions, computer-assisted writing evaluation technology can also provide diagnostic feedback concerning content, logic, vocabulary, grammar, spelling, etc., which is personalized, timely, objective, and constant (Li et al., 2015; Polio, 2012). With the AWE or AES system, students can receive timely and detailed writing feedback and freely modify articles at unlimited times, so as to achieve the purpose of internalizing knowledge and improving cognitive and speech ability. The existing AWE technologies or products can provide effective vocabulary and simple syntax modification suggestions. Their feedback on complex syntax errors is, however, less satisfactory (Fang, 2010).

The existing AWE/AES technologies or applications provide feedback that is capable of increasing the composition mechanics, such as accuracy, complexity of words, and average length of sentences (Li et al., 2017). Nevertheless, there is little help they can offer in feedback on content, organization, and unity, which does not accord with the requirements of writing feedback of second language acquisition theory. Furthermore, neural network-based AWE models rely too much on labeled training data. At present, more than 90% of AWE models use scored corpora from Kaggle ASAP (2012), which may lead to less objective scoring results. In addition, lacking the ability to evaluate the content relevance, the AWE models are vulnerable to spoofing or attacks of adversarial texts (Higgins & Heilman, 2014; Kabra et al., 2021; Parekh et al., 2020).

In this study, following the guidance of writing feedback theory and adopting the analytical scoring method, we proposed a Multi-strategy Computer-assisted EFL Writing Learning System (MsCAEWL) incorporating a variety of semantic-based NLP technologies and models, including deep learning in writing evaluation in order to provide not only the writing evaluation score but also the detailed writing feedback. Its introduction is able to overcome the limitation of previous AWE/AWE technologies which are powerless to evaluate texts from content-based dimensions, such as assessment of content and organization. Moreover, it addresses the problem that the improvement of writing ability benefited from AWE/AES models is limited due to their common implementation of holistic scoring. The proposed system is capable of providing evaluation sub and total scores from multiple traits while outputting the improving and correcting writing instructions for students writing learning.

Our contributions are summarized as follows:

1) Following the writing feedback theory, our model provides timely, detailed, multiple trait-specific, accurate analytic evaluation feedback on compositions utilizing multiple indicators, rather than the commonly used holistic evaluation in prior AWE models.

2) A package strategy of the Natural Language Processing technologies including neural networks is introduced and modified to yield multiple trait-specific evaluations of an essay based on the sentiment, such as thesis, fluency, and content.

3) Grammatical Error Correction (GEC) and Grammatical Error Diagnosis (GED) are introduced to provide revision and evaluation feedback.

4) A novel method is introduced to assess the competence of complexity of word and syntax usage and provide detailed lexical and syntactic usage suggestions in accordance with the level of the writer’s language abilities.

Literature Review

On top of scoring compositions, Automated Writing Evaluation (AWE), Automated Essay Scoring (AES), Computer essay grading, Computer-Assisted Writing Evaluation, and other computer-assisted writing evaluation technologies can also provide diagnostic feedback on content, logic, vocabulary, grammar, and spelling. Furthermore, AWE/AES technologies are distinguished by their uniqueness, timeliness, impartiality, and consistency (Li et al., 2015; Polio, 2012). With the AWE or AES system, students can receive timely and detailed writing feedback and freely modify essays with no limit of rounds, thus realizing the requirements and goals of the process-oriented writing approach and achieving the purpose of internalizing knowledge and improving cognitive and speech abilities. Students can use automated feedback to assist them improve their linguistic correctness by employing automated writing evaluation (AWE) technology (Saricaoglu & Bilki, 2021). AWE/AES technologies and their effects have elicited vehement debates. The findings of extensive research and quantitative analyses concerning the effects of AWE/AES improvements in various aspects of writing have demonstrated that there have been noteworthy long- and short-term improvements in accuracy, learn autonomy, and interaction (Link et al., 2022; Wang et al., 2013). Especially in grammatical error correction, the errors of certain categories were demonstrated to be greatly decreased when AWE was used (Saricaoglu & Bilki, 2021). Note that improvements like these may occur during the later stages of a period of sustained use of AWE for writing assistance (Liao, 2016). The utilization of AWE systems has been shown to generate not only positive effects on students’ writing performance, but also to significantly enhance the quality of teacher feedback in several dimensions, including mode, amount, types, and levels of feedback (Jiang et al., 2020). Thus, teachers’ attitudes and methods of use of AWE systems are positively associated with the performance of such systems in the context of writing learning., underscoring the importance of teachers’ roles in successful integration of these systems (Li, 2021), besides, different combinations of multi-approach writing teaching integrating AWE yield varying effects on writing performance (Tang & Rich, 2017).

In the industry, a large number of commercial AWE and AES products have emerged, including E-rater, Project Essay Grade (PEG), Writing Roadmap, My Access, Intelligent Essay Grade (IEA), Criterion, Write to Learn and Summary, pigai.org, Youdao Writing, I-write, Bingo English, etc. The early stage of AES/AWE development is dominated by the traditional machine learning technologies based on Bayes’ theorem (Rudner & Liang, 2002), linear regression (Phandi et al., 2015), rank preference learning (Chen & He, 2013; Yannakoudakis et al., 2011), reinforcement learning (Wang et al., 2015), etc. In recent years, the accelerated development of deep learning, especially neural network technology, has also greatly benefited the field of writing evaluation, such as Recurrent Neural Networks (Cai, 2019), Long Short-term Memory (Alikaniotis et al., 2016; Jin et al., 2018; Liu et al., 2019; Taghipour & Ng, 2016), Convolutional Neural Networks (CNN) (Dong & Zhang, 2016; Dong et al., 2017; Farag et al., 2017), BERT-based approach (Sharma et al., 2021). The introduction of attention mechanisms also significantly improved AES SOTA performance (Dong et al., 2017). Automated Writing Evaluation also benefited from pre-training strategies implemented at the forefront of natural language processing (Cai, 2019; Farag et al., 2017; Mim et al., 2021). Many studies adopted a variety of solutions based on neural networks, such as generating adversarial samples for learning (Farag et al., 2018; Liu et al., 2019), Multitask Learning (Cummins & Rei, 2018), and Self-Supervised Learning (Cao et al., 2020), Graph Algorithms (Jiang et al., 2021). Some studies proposed different approaches to address the problem of scarce training data for writing evaluation (Li et al., 2020; Ran et al., 2018). There are also scoring models that integrate multiple neural network models and schemes (Beseiso et al., 2021).

Grammatical Error Correction (GEC) or Grammatical Error Detection (GED) is an independent track of NLP missions, which is capable of automatically correcting or detecting the mechanical error of composition. The task content of GEC/GED overlaps with the parts of AES/AWE. AWE tasks are always accompanied by a diagnosis of grammatical errors, most of which need to be pointed out in their correct form. However, the AWE studies pay little attention to the GEC approaches, and only a few studies introduced the early GEC model into the AWE works (Liu et al., 2019). The initial research on GEC and GED mostly adopts N-gram (Xie et al., 2015; Zhang & Wang, 2014), Confusion Set (Lin & Chu, 2015; Rozovskaya & Roth, 2010), and language model (Brockett et al., 2006) including BERT (Devlin et al., 2019; Hong et al., 2019; Zhang et al., 2020) to diagnose and correct grammatical errors. However, the above solutions hardly solve problems such as disorder and component omission. Encoder-decoder architecture (Ge et al., 2018; Sutskever et al., 2014) has achieved similar success in GEC by end-to-end addressing the issues that other models fail to address at one time. It is worth noting that most recent advanced GEC architectures exploit the advantages of multiple models to solve different kinds of problems (Chollampatt & Ng, 2018; Rozovskaya & Roth, 2010; Zhang & Wang, 2014), including the Transformer-based approaches (Lichtarge et al., 2019; Zhao et al., 2019).

Writing Feedback and ESL riting

Zamel defines feedback as teaching information that helps the writing revision process and improves language learning (Zamel, 1982). Writing feedback is regarded as input information about second language writing provided to authors to improve their accuracy of expression (Arndt, 1992), and also as information provided to authors to modify their interlanguage (Keh, 1990). The function of feedback is generally considered to be to confirm, understand and clarify requirements (Ellis, 1994). Modification serves as its theoretical core. Learners use modification feedback as a clue to conduct exploratory and open learning, and gradually form scaffolding instruction.

Good and effective writing feedback not only needs to point out whether it is correct or not but also provides suggestions for modifying or improving performance (Zamel, 1982). Based on the practice and verification of generations of pedagogics and linguists, writing feedback is essential to the success of writing instruction, and a few key requirements should be met in order to ensure its effectiveness (B. Chen & J. Zhang, 2022). Clarity is the first, and it should be ensured in order for students to make the appropriate changes (Conrad & Goldstein, 1999; Ferris & Roberts, 2001; Freedman, 1984). Feedback must be clear and unambiguous to be beneficial for writing performance (Polio, 2012). Secondly, feedback should also occur in multiple rounds to guide the repeated composition modifications during instructional interactions (Zamel, 1982). Through the coordination of multiple modifications and feedback, student’s cognition, thinking, and writing skills will be improved (Onozawa, 2010), which is the goal of process-oriented writing. Furthermore, feedback should be timely. Students can benefit from timely feedback-modification interactions and see a marked improvement in their writing performance (Beach & Friedrich, 2006; Ferris, 1995; Freedman, 1984; Leki, 1990; Zamel, 1982). Studies show that if formative feedback is given and executed quickly during L2 writing instruction, there will be an immediate improvement in students’ performance (Beach & Friedrich, 2006).

To be more effective, writing feedback should cover all aspects of writing, including content, organization, grammar, and mechanics (Cohen & Cavalcanti, 1990; Ferris, 1995; Hedgcock & Lefkowitz, 1994). Holistic scoring, in which an overall score is set to reflect the writing performance, is traditional and widely adapted by AWE models or teachers. Under this circumstance, single-trait holistic scoring cannot provide diagnostic information (Cohen, 1994) and is not adequate as ideal writing feedback. Comparatively, analytic scoring assigns separate sore from different dimensions as mentioned above to assess writing performance, thus can provide adequate diagnostic information of splendor or insufficiency in particular performance and detailed guidance for the development of students’ writing (Bauer, 1981; Klein et al., 1998; Weir, 1988).

Theoretically, multiple, continuous, timely, clear, and multi-aspect guidance interactive feedback is an optimal way to improve writing ability. In order to maximize the feedback effect, it is necessary to put forward specific suggestions for improvement.

However, in ESL teaching, the writing feedback students receive is mostly vague, global, or inconsistent (Beach & Friedrich, 2006), and in most cases delayed. Especially in China, it is difficult for teachers to follow writing feedback in the teaching process. In the context of poor educational resources, it is impossible to require teachers to provide personalized, continuous, and detailed feedback for each student each time they write or modify an essay.

In accordance with the requirements of writing feedback theory for writing evaluation, this study, therefore, seeks to propose an easy-to-operate automatic machine writing evaluation method, which aims to enable users to obtain immediate responses and use them to revise their compositions repeatedly, and the response are supposed to contain the specific suggestions of organization, use of grammar and vocabulary, cohesion, etc.

Methodology

Grading Criteria

IELTS and TOEFL tests although adapt different writing scoring rubrics, they take a similar view on the properties of well-written essays. By and large, coherence, cohesion, vocabulary usage, grammar usage, etc. are the major concerns in evaluating an IELTS or TOEFL essay. Nevertheless, in the routine teaching and evaluation of EFL writing, teachers pay more attention to the development of students’ writing ability in mechanics, grammar, organization, style, unity, vocabulary, content, etc. (Pishghadam et al., 2014). Constrained by the limitations of the prior art in computer-assisted writing evaluation, the style-related essay assessment remains elusive. To this end, the evaluation categories adopted in this study involve unity, organization, fluency, content, vocabulary, grammar, and mechanics (see Table 1).

Table 1.

Definitions of Evaluation Categories.

UNITY/COHERENCE	Relevance of Paragraphs to a Single clear central idea (Prompt), relevance of Paragraphs to each other, transitions
ORGANIZATION/COHESION	Paragraph development, paragraph structure, essay development, essay structure
FLUENCY	The clear, smooth use of language
CONTENT	Clear central idea, development of an idea
VOCABULARY	Proper word usage and word choice, use of different and complex vocabulary
GRAMMAR	Correctness in tense, sentence structure, number, agreement, article, preposition, pronoun, etc.; complexity in structure, phrasal structure
MECHANICS	Punctuation, spelling, capitalization, face, paragraphing, handwriting, space

According to the table above, in the proposed computer-assisted writing learning, we set five modules to execute the above five evaluation requirements for writing: THESIS, FLUENCY, CONTENT, COMPLEXITY, and CORRECTNESS. In particular, “Thesis” evaluates the unity and organization, and sees the degree how the paragraphs serve the topic; “Complexity” consists of grammar and vocabulary assessments, it is a check of the authors’ mastery of morphology and syntax listed in the Table 1, meanwhile, “Correctness” checks the use of vocabulary, mechanics, and grammar conforming to the standard English. The schema of MsCAEWL’s evaluation criteria is shown in Figure 1.

Figure 1.

The mapping between evaluation modules of MsCAEWL and the evaluation categories of composition properties. The proposed computer-assisted EFL writing learning system consists of five main modules which function following the requirements of the general EFL writing evaluation categories, that is, unity, organization, content, fluency, vocabulary, grammar, and mechanics.

Task Definition

In accordance with the requirements of writing feedback on the EFL writing teaching, this paper proposes a novel multi-strategy automatic computer-assisted EFL writing learning system MsCAEWL. The flow chart of the proposed system is shown in Figure 2.

Figure 2.

The flow chart of MsCAEWL. Essays and prompts are combined as inputs into MsCAEWL. The Theory, Fluency and Content modules generate the corresponding scores respectively. In the complexity module, sub-scores will be provided for each of the sentence and vocabulary levels, with the final completion score being driven by a combination of the two sub-scores. In addition to detecting and scoring syntactic and lexical errors, the Correction module also provides suggestions for error correction.

The proposed system aims to solve the defects of the prior studies and proposes a series of solutions involving deep learning methods to give users detailed vocabulary and syntax modification or improvement suggestions and gives itemized and overall feedback and evaluations on compositions based on big data so that teachers and students can be fully aware of the writing process. MsCAEWL evaluates compositions from five indicators of thesis (unity and organization), fluency, content, complexity (lexical and syntactic complexity), correctness (vocabulary, mechanics, and grammar), and incorporates the corresponding modules to conduct evaluations. Moreover, it diagnoses the errors of words, phrases, sentences, and improper use in essays, puts forward suggestions for correction, points out where words and sentences can be improved, and gives feedback. We proposed a model to describe the calculation of the evaluation process of MsCAEWL in equations (1)–(4), the notations are listed in Table 2.

$FinalScore={10\alpha }_t·S_t+{10\alpha }_f·S_f+10{\alpha }_{con}·S_{con}+10{\alpha }_l·S_l+{10{\alpha }_s·S}_s+10{\alpha }_c·S_c$

(1)

$\left[\begin{array}{c}{S}_t\\ S_f\\ S_{con}\\ S_s\end{array}\right]=\delta \left(\left[\begin{array}{c}{I}_t\\ I_f\\ I_{con}\\ I_s\end{array}\right]\right)$

(2)

$\{\begin{array}{c}\delta \left(\left[\begin{array}{c}{I}_t\\ I_f\\ I_{con}\\ I_s\end{array}\right]\right)=5,if\hspace{0.17em}\frac{4}{5}\hspace{0.17em}\sigma \left[\begin{array}{c}{S}_t^{\prime }\\ S_f^{\prime }\\ S_{con}^{\prime }\\ S_s^{\prime }\end{array}\right]\le \left[\begin{array}{c}{I}_t\\ I_f\\ I_{con}\\ I_s\end{array}\right]\le \sigma \hspace{0.17em}\left[\begin{array}{c}{S}_t^{\prime }\\ S_f^{\prime }\\ S_{con}^{\prime }\\ S_s^{\prime }\end{array}\right]\\ \delta \left(\left[\begin{array}{c}{I}_t\\ I_f\\ I_{con}\\ I_s\end{array}\right]\right)=4,if\hspace{0.17em}\frac{3}{5}\hspace{0.17em}\sigma \left[\begin{array}{c}{S}_t^{\prime }\\ S_f^{\prime }\\ S_{con}^{\prime }\\ S_s^{\prime }\end{array}\right]\le \left[\begin{array}{c}{I}_t\\ I_f\\ I_{con}\\ I_s\end{array}\right]\le \frac{4}{5}\sigma \hspace{0.17em}\left[\begin{array}{c}{S}_t^{\prime }\\ S_f^{\prime }\\ S_{con}^{\prime }\\ S_s^{\prime }\end{array}\right]\\ \delta \left(\left[\begin{array}{c}{I}_t\\ I_f\\ I_{con}\\ I_s\end{array}\right]\right)=3,if\hspace{0.17em}\frac{2}{5}\hspace{0.17em}\sigma \left[\begin{array}{c}{S}_t^{\prime }\\ S_f^{\prime }\\ S_{con}^{\prime }\\ S_s^{\prime }\end{array}\right]\le \left[\begin{array}{c}{I}_t\\ I_f\\ I_{con}\\ I_s\end{array}\right]\le \frac{3}{5}\sigma \hspace{0.17em}\left[\begin{array}{c}{S}_t^{\prime }\\ S_f^{\prime }\\ S_{con}^{\prime }\\ S_s^{\prime }\end{array}\right]\\ \delta \left(\left[\begin{array}{c}{I}_t\\ I_f\\ I_{con}\\ I_s\end{array}\right]\right)=2,if\hspace{0.17em}\frac{1}{5}\hspace{0.17em}\sigma \left[\begin{array}{c}{S}_t^{\prime }\\ S_f^{\prime }\\ S_{con}^{\prime }\\ S_s^{\prime }\end{array}\right]\le \left[\begin{array}{c}{I}_t\\ I_f\\ I_{con}\\ I_s\end{array}\right]\le \frac{2}{5}\sigma \hspace{0.17em}\left[\begin{array}{c}{S}_t^{\prime }\\ S_f^{\prime }\\ S_{con}^{\prime }\\ S_s^{\prime }\end{array}\right]\\ \delta \left(\left[\begin{array}{c}{I}_t\\ I_f\\ I_{con}\\ I_s\end{array}\right]\right)=1,if\hspace{0.17em}\left[\begin{array}{c}{I}_t\\ I_f\\ I_{con}\\ I_s\end{array}\right]\le \frac{1}{5}\sigma \hspace{0.17em}\left[\begin{array}{c}{S}_t^{\prime }\\ S_f^{\prime }\\ S_{con}^{\prime }\\ S_s^{\prime }\end{array}\right]\end{array}$

(3)

$\sigma \hspace{0.17em}\left(x\right)=\frac{1}{1+e^{-x}}$

(4)

Table 2.

Notations.

$S_t$	Score of the Thesis evaluation
$S_f$	Score of the fluency evaluation
$S_{con}$	Score of the content evaluation
$S_l$	Score of the lexical complexity evaluation
$S_s$	Score of the syntactic complexity evaluation
$S_c$	Score of the correctness evaluation
${\alpha }_t$	Coefficient of the thesis evaluation
${\alpha }_f$	Coefficient of the fluency evaluation
${\alpha }_{con}$	Coefficient of the content evaluation
${\alpha }_l$	Coefficient of the lexical complexity evaluation
${\alpha }_s$	Coefficient of the syntactic complexity evaluation
${\alpha }_c$	Coefficient of the correctness evaluation
$\sigma \left(x\right)$	Logistic sigmoid function
$\delta \left(x\right)$	The normalizing and scaling function
$I_t$ , $I_f$ , $I_{con}{I}_s$	The intermediate values to scale values of $S_t^{\prime }$ , $S_f^{\prime }$ , $S_{con}^{\prime }{S}_s^{\prime }$

In the proposed model, each evaluation module possesses different approaches to modeling and calculating in scoring compositions, the ranges of scores of the Thesis, Fluency, Content, and Syntactic Complexity modules vary, thus the functions are implemented to normalize and scale the above sub-evaluation sores. The logistic sigmoid function $\sigma \left(x\right)$ is introduced to normalize the scores into [0, 1], then $\delta \left(x\right)$ divides the sigmoid result of scores of the Thesis, Fluency, Content, Syntactic Complexity modules into five levels based on the mean and assigns {1, 2, 3, 4, 5} credits respectively to each level.

Considering the value ranges of the scoring modules still differ (see Table 3), coefficients are introduced to regulate the individual score of each module and the total score. ${\alpha }_t$ , ${\alpha }_f$ , ${\alpha }_{con}$ , ${\alpha }_l$ , ${\alpha }_s$ , ${\alpha }_c$ are leveraged as the coefficients adjustable according to the upper limit of the total score or the focus of the writing evaluation. In our study, we set a scenario where the total score of composition is 100 credits, the thesis, fluency, and content aspects of writing are paid more attention in evaluation and scoring. Thus, the score assignment and coefficients are set as shown in Table 4. Follow the settings, the total score turns out 100 credits calculated as equation (1). All the above settings and the upper limit of the total score can be adjusted according to requirements, for instance, the maximum total score could be 10 credits along with the corresponding parameter settings.

Table 3.

Value Range of the Scoring Modules or Functions.

Function or score	Range
$\left[\begin{array}{c}{S}_t\\ S_f\\ S_{con}\\ S_s\end{array}\right]=\delta \left(\left[\begin{array}{c}{I}_t\\ I_f\\ I_{con}\\ I_s\end{array}\right]\right)$	{1, 2, 3, 4, 5}
$S_l$	[0, 2]
$S_c$	[0, 1]
$\sigma \left(I_t\right)$ , $\sigma \left(I_f\right)$ , $\sigma \left(I_{con}\right)\sigma \left(I_s\right)$	(0, 1)

Table 4.

The Score Assignment and Coefficient Setting.

	Thesis	Fluency	Content	Complexity		Correctness
				Lexical Complexity	Syntactic Complexity
Score	20	15	30	10	10	15
Coefficient ( $\alpha$ )	0.4	0.3	0.6	0.5	0.2	1.5

The Thesis Module

The majority of EFL writing exams contain opinion/discussion writing tasks essentially with prompts in them. The writing prompt students must respond to in the writing task of EFL is a compact set of writing instructions that assist students to focus on a certain topic, task, or goal. Writing prompt is the key to achieving Unity.

Quality of Organization is related to whether ideas are arranged logically and accurately. Each part of the essay is supposed to be clearly organized and flow smoothly. An essay with good organization indicates that paragraphs should support the thesis of the paper.

However, the EFL students often fail to grasp the topic or have problems in developing content with regard to the prompt, thus possibly resulting in digression or failure to arrange the ideas or statements to support the thesis. Due to incapacity in identifying the content of texts, the prior AWE models do not attend to this issue and are inadequate to address it. Much worse, students tend to cheat models with fancy expressions and words while leveraging automatic writing evaluation means, which is inevitable given the current state of the art.

This module is incorporated to check the accordance between the prompt and the main body, and the unity of each paragraph, i.e., the Thesis module merges the Unity and Organization evaluation. A traditional wide-applied Natural Language Processing method is implemented to accomplish this end. TF–IDF, short for Term Frequency–Inverse Document Frequency, is a numerical statistic to qualify how important or relevant a word/words is to a set of a document amongst a collection of documents. It is often used as a weighting factor in searches for information retrieval, text mining, and user modeling. TF looks at the frequency of a particular term (word) in a document, while IDF - the commonness of the term in a corpus. In this case, All The News corpus from Kaggle is incorporated (see section “Datasets”). TF-IDF seeks the importance of the term inversely related to its frequency in a corpus and is calculated as follows.

$TF-IDF\left(word,document,corpus\right)=TF\left(word,document\right)·IDF\left(word,corpus\right)$

(5)

where

$TF\left(word,document\right)=\frac{number\hspace{0.17em}of\hspace{0.17em}occurrences\hspace{0.17em}of\hspace{0.17em}the\hspace{0.17em}word\hspace{0.17em}in\hspace{0.17em}the\hspace{0.17em}document}{total\hspace{0.17em}number\hspace{0.17em}of\hspace{0.17em}words\hspace{0.17em}in\hspace{0.17em}the\hspace{0.17em}document}$

(6)

$IDF\left(word,corpus\right)=\mathit{log}\left(\frac{total\hspace{0.17em}number\hspace{0.17em}of\hspace{0.17em}documents}{number\hspace{0.17em}of\hspace{0.17em}documents\hspace{0.17em}with\hspace{0.17em}the\hspace{0.17em}word\hspace{0.17em}in\hspace{0.17em}it}\right)$

(7)

In the Thesis evaluation task, two word vectors are calculated by TF-IDF utilizing two word sets containing the keywords and excluding the Stop words respectively from the prompt paragraphs and the composition, only the synonyms of the prompt word set are added after selection. The cosine similarity of these two vectors which are the semantic representations of the prompt, and the composition is gained to measure the degree of the Thesis between the prompt and the essay. The larger the value, the more relevant the two are, and the higher the Thesis degree is. The calculation is shown in equation (8).

$S_t^{\prime }=Sim\hspace{0.17em}\left(V_p,V_{com}\right)=\mathit{cos}\hspace{0.17em}\left(\theta \right)=\frac{V_p·V_{com}}{\Vert V_p\Vert \times \Vert V_{com}\Vert }$

(8)

where, $S_t^{\prime }$ is the score of the Thesis module, $Sim\hspace{0.17em}\left(\right)$ is the cosine similarity function, $V_p\hspace{0.17em}is\hspace{0.17em}prompt\hspace{0.17em}vector=IDF\hspace{0.17em}\left(prompt\right),$ $V_{com}\hspace{0.17em}is\hspace{0.17em}compositon\hspace{0.17em}vector=IDF\hspace{0.17em}\left(composition\right)$ .

The Fluency Module

Writing Fluency refers to a student’s ability to write with a natural flow and rhythm, utilizing appropriate word patterns, vocabulary, and content. Syntactic fluency refers to the extent to which a writer constructs a sentence containing linguistically complex structures (Shapiro, 1999). More specifically, fluent composition flows smoothly from word to word, phrase to phrase, and sentence to sentence. In Natural Language Processing the N-gram-based metrics carry a similar function achieving the above requirement for Writing Fluency.

As an automatic evaluation metric of text generation or machine translation BLEU (Bilingual Evaluation Understudy) measures the closeness by comparing the machine output against the human reference and is well-known for its strong correlation with human evaluation in adequacy and fluency. Its calculation is as follows:

$BLEU=BP·\exp \left({\displaystyle \sum }_{n=1}^N{w}_n\mathit{log}{p}_n\right)$

(9)

$BP=\{\begin{array}{c}1\\ e^{\left(1-\frac{r}{c}\right)}\end{array}\hspace{0.17em}\begin{array}{l}if\hspace{0.17em}c>r\\ if\hspace{0.17em}c\le r\end{array}$

(10)

where $p_n$ is the precision of $n-gram$ , $c$ is the length of the system output, and $r$ is the length of the reference corpus, $BP$ indicates the brevity penalty.

BLEU compares the n-gram of the candidate with the reference corpus to count the number of overlaps that are independent of the word positions. This n-gram precision scoring captures two aspects of translation: adequacy and fluency. The longer n-gram overlaps between the candidate and reference account for fluency, which means the generated sentences are well-formed and mature in length and structure compared with the reference.

In this study, BLEU with 4-g is utilized for writing fluency evaluation matric. We take compositions as system output candidates in machine translation, and a standard corpus with sufficient capacity is used as a reference. Considering that EFL writing requires writing with standard English, we choose a news set as the reference corpus. The Fluency score is the average BLEU score of each sentence from the essay against the news corpus (All The News, see section “Datasets”). The calculation is shown in equation (11).

$S_f^{\prime }=\frac{{\displaystyle \sum }_{i=1}^n{Bleu}_i}{n}$

(11)

where, $S_f^{\prime }$ is the score of the Fluency module, ${Bleu}_i$ indicates the BLEU score of the $i-th$ sentence, $n$ is the total number of the sentences of the corresponding essay.

The Content Module

In essence, grading the content of writing is a comprehensive evaluation of whether the thesis and purpose are clear, whether expectations and development are in a consistently excellent manner according to the viewpoint, whether the structure and word choice and use are fluent and graceful, and so on. Content evaluation is the core of the human judgment of an essay; thus, it regularly presents as holistic scoring.

The AWE/AES models assign holistic scores based on the writing content. More than 90% of AES models are trained and tested leveraging the ASAP (The Automated Student Assessment Prize) (see section “Comparison experiments”) corpus and its scoring labels. Based on the ASAP corpus as well, this article established an LSTM-based model to assess the writing content. Recurrent Neural Network (RNN) has been widely applied in various NLP tasks for their outstanding time series feature extraction capability. Long-short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) (the formula is as follows) is an improvement to solve the problems such as the disappearance of the RNN gradient. LSTM is introduced in this section to extract sequential features from essays with different scores in the training set so as to predict the holistic score of the essays to be tested.

In an LSTM cell, the hidden state is split into two vectors: $h_t$ and $c_t$ , which respectively represent the short-term state and the long-term state. The current input vector $x_t$ and the previous short-term state $h_{t-1}$ are fed to four different fully connected layers. One layer outputs $\widehat{c_t}$ analyzing the current inputs $x_t$ and the previous (short-term) state $h_{t-1}$ . The other three are gate controllers. They control information by activation functions, whose range is from 0 to 1. The forget gate $f_t$ determines what proportion of the previous long-term state $c_{t-1}$ should be erased. The input gate $i_t$ governs to what extent $\widehat{c_t}$ should be added to the current long-term state $c_t$ . Finally, the output gate $o_t$ addresses what should be output to $h_t$ from the long-term state at this time step. The score of the Content module $S_{con}^{\prime }$ is the output $Y_{pred}$ of LSTM networks, that is the rating of a whole essay. The processing progress is depicted in equations (12) and (13).

$\{\begin{array}{c}\left[\begin{array}{c}\begin{array}{c}{i}_t\\ f_t\end{array}\\ \begin{array}{c}{o}_t\\ {\widehat{c}}_t\end{array}\end{array}\right]=\left[\begin{array}{c}\begin{array}{c}\mathit{\sigma }\\ \sigma \end{array}\\ \begin{array}{c}\sigma \\ \mathit{tanh}\end{array}\end{array}\right]W·\left[h_{t-1},x_t\right]\\ c_t=f_t\odot {\mathrm{c}}_{t-1}+i_t\odot \widehat{c_t}\\ h_t=o_t\odot \mathit{tanh}\left(c_t\right)\end{array}$

(12)

$S_{con}^{\prime }=Y_{pred}$

(13)

The Complexity Module

As stated in section “Grading Criteria.” Complexity checks the lexical and structural diversity and complexity, i.e., the proficiency in vocabulary and grammar of writers. Two sub-modules are incorporated to achieve this end: the Lexical Complexity and the Syntactic Complexity.

The Lexical Complexity

The existing AWE research and products give improving feedback on vocabulary choice or usage by listing the possible synonyms of words or phrases in the essays, merely based on dictionary-matching technology. The listed words or phrases will not be changed according to the learners’ level. Most students can’t benefit from feedback and learn new-to-them vocabulary and usage. Thus, the word list scales based on the EFL learners’ lexical knowledge are introduced to address this problem. The New JACET 8000 (Committee, 2016) is the scaled list of basic words established by the Japan Association of College English Teachers (JACET). Eight levels are divided among 8000 words conforming to the lexical profile of Japanese EFL learners, and each level contains 1000 words. On the basis of the author’s vocabulary level being positioned leveraging the word scales, the gap between the author’s level and the writing requirements is judged and corresponding evaluation and suggestions on vocabulary use are given accordingly.

According to the learners’ level, a higher vocabulary set and a lower vocabulary set are established. The words or phrases in the composition are replaced by the synonyms from the higher and lower vocabulary sets respectively using the thesaurus. If the replacement occurs in the higher vocabulary set, it indicates that the author still has room for improvement in the use of this replaceable word or phrase. The system will output the words in the higher vocabulary set as improving suggestions, and at the same time reduce the score of Lexical Complexity. On the contrary, the replacement can only be found in the lower vocabulary set, the system will output no suggestions, and the score is increased.

A novel model for scoring lexical complexity is proposed in this study. We use the ratio of the total number of edits that can be replaced to the weighted total number of words after deduplication as the basis for the vocabulary complexity score. The calculation is as follows:

$S_l=\left(1+\frac{f\hspace{0.17em}\left(R\right)}{C\times a\%}\right)$

(14)

$\{\begin{array}{c}f\hspace{0.17em}\left(R\right)=C\times a\%,if\hspace{0.17em}R\ge C\times a\%\\ f\hspace{0.17em}\left(R\right)=R,if-C\times a\%\le R\le C\times a\%\\ f\hspace{0.17em}\left(R\right)=-C\times a\%,if\hspace{0.17em}R\le -C\times a\%\end{array}$

(15)

$R=P+N$

(16)

where $S_l$ is the final score of the Lexical Complexity, $P$ is a positive value, representing the number of bonus credits, $N$ is a negative value and represents the number of penalty credits, $R$ is the total number of the replaceable edits, i.e., the sum of $P$ and $N$ , $C$ denotes the count of the words after deduplication. Note that $a$ functions as an adjustable parameter and represents the word number. The weight $a\%$ specifies the limit that the system allows for rewards or penalties for lexical complexity. We take $a=30$ as the threshold, that is, the bonus credits greater than or equal to 30% of the composition words will reach the upper limit of the reward, and the penalty credits less than or equal to 30% of the composition words will reach the upper limit of the penalty. The range of $S_l$ is from 0 to 2. When $R$ exceeds the reward upper limit, $\frac{f\hspace{0.17em}\left(R\right)}{C\times a\%}=1,$ i.e., $S_l$ equals 2, which means the proficiency in vocabulary mastery of this composition is excellent. When $R$ exceeds the penalty upper limit, $\frac{f\hspace{0.17em}\left(R\right)}{C\times a\%}=-1$ , then, $S_l$ equals 0, which means the vocabulary mastery of this composition does not meet the requirements.

The syntactic complexity

The Syntactic Complexity mainly depends on the diversity of structural patterns and the flexible application of the grammar, such as the diversity of active and passive sentences, clauses, participles, etc. From the linguistic point of view, these syntactic phenomena have their own characteristic function words. These function words are selected into a set, such as: which, when, copular verbs, adverbs, etc. TextRank in NLP is incorporated to calculate the representation of function words in the set based on lexical relations, i.e., the appropriateness of the syntax use represented by the function-word-centered lexical relationship. The TextRank values based on function words in the news corpus (All The News, corpus from Kaggle in section “Datasets”) and each composition are calculated and compared respectively. The higher the similarity between the two, the better the Syntactic Complexity, and the higher the score. The Syntactic Complexity is calculated based on a real-world corpus, there are thus no worries about being cheated by adversarial samples.

TextRank, as an application of graph algorithm, takes words of text as graph nodes, constructs a graph model, and then calculates the importance of each word based on the weights between words. That is, the ratio of the weight $w_{\left\{ji\right\}}$ of the $\left(j,i\right)$ edge formed by word $i$ between its frontal each point (word) $j$ to the sum of the weights of the point $j$ to the other edges is the TextRank value $S\left(v_i\right)$ of the word, and the calculation is shown in equation (17).

$S\left(v_i\right)=\left(1-d\right)+d{\displaystyle \sum }_{\left(j,i\right)\in \epsilon }\frac{w_{\left\{ji\right\}}}{{\sum }_{v_k\in out\left(v_j\right)}{w}_{jk}}S\left(v_j\right)$

(17)

The Syntactic Complexity score is the average TextRank score of each sentence from the essay. The calculation is shown in equation (18).

$S_s^{\prime }=\frac{{\displaystyle \sum }_{i=1}^n{Tr}_i}{n}$

(18)

where, $S_s^{\prime }$ is the score of the Syntactic Complexity module, ${Tr}_i$ denotes the TextRank sore of the $i-th$ sentence, $n$ is the total number of the sentences of the corresponding essay.

The Correctness Module

As stated in section “Grading Criteria,” Correctness checks the proper use of vocabulary, mechanics, and grammar, i.e., whether the compositions are written in correct standard English. Two sub-modules are incorporated to achieve this goal: the GED module locates the lexical or syntactic errors, and the GEC module corrects them and outputs the correction suggestions.

The Grammatical Error Diagnosis module

Grammar error detection (GED) can be analyzed as a Sequence Tagging task, that is, the task of predicting labels based on sequential features. Although BERT and Transformer techniques are popular in all aspects of NLP, the RNN family is more dominant in the extraction of sequential features. Therefore, this study adopts Bi-directional Long-short term memory neural networks (Bi-LSTM) to detect grammatical errors.

A large number of experimental studies and applications have proved that the BERT model pre-trained based on a huge multilingual corpus can obviously improve the effect of various downstream tasks because of its excellent semantic/syntactic representation ability (Devlin et al., 2019; Herzig et al., 2020; Sun et al., 2019; Zhu et al., 2019). In this study, BERT’s pre-training strategy is implemented to Fine-tune the training data, and then the task of grammar error detection is carried out.

After the text enters the BERT model for fine-tuning, a new semantic vector representation is obtained, which is then processed by Bi-LSTM and conditional random field (CRF) to finally output the error type tags. The calculation process is shown in Figure 3:

Figure 3.

The diagram of the Grammatical Error Diagnosis model. This vector representation is used to make predictions regarding the type of errors within the text, utilizing both Bi-LSTM and CRF for the task. The BERT model is fine-tuned with the text, resulting in a new semantic vector representation. This vector is then processed with Bi-LSTM and CRF, ultimately outputting the error type tags.

The Grammatical Error Correction module

Grammatical Error Correction models with SOTA scores mostly regard Grammatical Error Correction as a machine translation task and often adopt an Encoder-decoder structure. In this study, a novel Grammatical Error Correction architecture is inspired by the previous study (B. Chen & J. Zhang, 2022). As the neural networks in the Grammatical Error Diagnosis module, the training text obtains a new semantic representation after pre-training via BERT, and then enters the Encoder-decoder model with Global attention, where the Encoder involves a uni-direction LSTM, and the Decoder takes Bi-LSTM as the core. The specific architecture is depicted in Figure 4.

Figure 4.

The Architecture of Grammatical Error Correction Model. In the Grammatical Error Diagnosis module, the training text is pre-trained using BERT to obtain a new semantic representation, which is then fed into the Encoder-Decoder model with Global Attention. For the Encoder, a uni-directional LSTM is employed. The Decoder, on the other hand, utilizes a Bi-LSTM as its central component.

The scoring method of the correctness evaluation

The Grammatical Error Diagnosis module charges for identifying and showing the errors to the users, meanwhile, the Grammatical Error Correction module is responsible for correcting the located errors. The errors located and corrected are supposed to be marked as the candidate items of losing credits. The mechanic of penalty credits for grammatical errors is depicted in equations (19) and (20).

$S_c=\frac{\gamma \left(a\%\times C-e\right)}{a\%\times C}$

(19)

$\gamma \left(x\right)=ReLU\left(x\right)=\mathit{max}\hspace{0.17em}\left(0,x\right)$

(20)

where, $S_c$ is the score of the Correctness module, $e$ denotes the total number of errors, $C$ denotes the word count of the essay after deduplication. $ReLU$ , the Rectified linear unit function, is a piecewise linear function that outputs any input value to zero if it is negative, otherwise, outputs input to itself. $a$ is the adjustable threshold that controls the maximum number of errors that can be tolerated. $a=20$ is adopted in our study, which means $S_c$ turns out 0 credits if the error count exceeds 20% of the whole composition under the control of $ReLU$ function. The range of $S_c$ is from 0 to 1, where $S_c$ equals one when $e$ equals 0, that is, there are no errors in the essay.

Experiments

Datasets

a) Reference corpus in the model

In sections of the Thesis, Fluency module, and the Syntactic Complexity of the Complexity module, the role of the reference corpus is essential. The quantity and quality of the corpus will lead to a change in scores, making the scores unreliable, which means a massive and error-free corpus is required. Most EFL writing tasks focus on elaborating or discussing opinions and examining the author’s proficiency in standard English. Therefore, the writing style requires written language rather than spoken language. In view of this, the news texts with rigorous terms are adopted as the basic dataset.

The All The News ¹ from Kaggle consists of 143,000 news articles covering a wide range of topics from 15 American publications including the New York Times, Breitbart, CNN, Business Insider, etc. This reference corpus derives from the news from 2016 to 2017, with a total size of about 1.2 Gb.

b) The scored essay corpus

The ASAP corpus is the dataset incorporated in the Kaggle (2012) AES competition, the Automated Student Assessment Prize. The corpus has a total of 12,976 compositions under eight prompts, the distribution of the essays under prompts and the score range from human raters are shown in Table 5. The score range varies. Furthermore, it is worth noting that there are one or more human raters on individual composition, and the final scores are determined or calculated in different ways according to the corresponding prompt. The comparison experiments of most AES/AWE studies are conducted based on the final resolved score, and so do the experiments in this study.

Table 5.

The Statistics of the ASAP Corpus.

	Essay Type	Essays	Score Range	Avg words	Avg sentences
Prompt 1	Argumentative	1783	2–12	350	23
Prompt 2	Argumentative	1800	1–6	350	20
Prompt 3	Source-dependent	1726	0–3	150	6
Prompt 4	Source-dependent	1772	0–3	150	4
Prompt 5	Source-dependent	1805	0–4	150	7
Prompt 6	Source-dependent	1800	0–4	150	8
Prompt 7	Narrative	1569	0–30	250	12
Prompt 8	Narrative	732	0–60	650	35

Comparison Experiments

Evaluation metric

Quadratic Weighted Kappa (QWK) which is the official metric of the Automated Student Assessment Prize and regarded as the major metric of the performance of AWE models is introduced as the evaluation metric in this section. It measures the agreement between the prediction score set and the label score set. The QWK value ranges from 0 to 1, which indicates the agreement arises from arbitrariness to completeness. In this sense, the human annotator rating set is considered as the labels, and the system outputs present as the prediction scores.

An N-by-N quadratic weight matrix $W$ is first computed to encode the rating information.

$W_{i,j}=\frac{{\left(i-j\right)}^2}{{\left(N-1\right)}^2}$

(21)

where $N$ is the number of possible ratings. An N-by-N matrix $A$ is calculated such that $A_{i,j}$ corresponds to the number of essays that receive a score $i$ by the human rater, and a score $j$ by the scoring system. Another N-by-N matrix $B$ is constructed as the outer product of the histogram vectors of the two ratings. $A$ and $B$ are then normalized such that they have the same sum. Finally, from the three matrices, the quadratic weighted kappa is calculated as follows:

$k=1-\frac{{\sum }_{i,j}{W}_{i,j}{A}_{i,j}}{{\sum }_{i,j}{W}_{\left(i,j\right)}{B}_{i,j}}$

(22)

Comparison based on holistic scoring

Most AWE models adopt a holistic scoring method, i.e., an AWE model outputs an overall score of the corresponding essay, which is also an essential part of the scoring system of our proposed model. In order to validate the performance on holistic scoring of the proposed model, frontier models in AES/AWE (Table 6) are introduced as baseline models compared with MsCAEWL.

Table 6.

The Descriptions of the Baseline Models.

Baseline models	Sources	Backbone
S1	RL1 (Wang et al., 2018)	BERT, LSTM, boosting tree
S2	TSLF (Liu et al., 2019)	Dilated LSTM
S3	BLRR (Phandi et al., 2015)	EASE, BLRR, SVM
S4	LSTM-CNN-att (Dong et al., 2017)	LSTM, CNN, attention
S5	SKIPFLOW LSTM (Tay et al., 2018)	LSTM
S6	BERT + Essay-level features (Uto et al., 2020)	BERT

ASAP corpus with holistic scores as training labels serves as experiment data set in this section. Although more than 10 years have passed since the Kaggle AES competition, the labeled data set continue to benefit the AWE/AES field. Remarkable models have been emerging, especially for ones with neural networks incorporated, long after the competition event.

However, there are two issues that do not fit the requirements of our experiment and need to be modified in the pre-processing.

First, the number of human annotators in each essay is not consistent. Most essays contain two rater scores while some of them have three rater scores. More, given that one set of essays adopts trait scoring evaluation, i.e., the analytic scoring, while the others incorporate the holistic scoring method, the resolved score is obtained inconsistently: either the higher rater score is directly elected, or the composite score is calculated by relevant equations. The score range of each prompt thus differs. In this section, we take the resolved scores as baseline scores or labels regardless of the acquiring method. Before experiments, the score range should be normalized into (0, 1).

Second, out of privacy protection, the competition organizer has removed any personal information and replaced the relevant entities with marks, such as @ORGANIZATION1, @PERSON1, etc. It is worth noting that the replacements often act as vital sentence constituents like subjects or objects. For people and machines, this is obviously a huge obstacle to reading or understanding, especially for machine learning methods with semantic-based models as their backbones. Since all the modules of the proposed model are highly semantic-dependent, these marks will infect the performance to a different extent. Hence, we randomly choose words under the marked categories to restore the sentence, such as replacing @ORGANIZATION1 with “bank” and @PERSON1 with “Jackson.”

In the comparison experiments, the resolved scores of human raters are utilized to calculate the QWK values of all the baseline models and MsCAEWL under each prompt, and the average QWK value of all prompts from the corresponding model is presented as well. All results are shown in Table 7, the value in the bolt indicates the optimal performance in the corresponding comparison.

Table 7.

The Comparison with the Baseline Models on Holistic Scoring.

	S1	S2	S3	S4	S5	S6	MsCAEWL
Prompt 1	0.766	0.852	0.761	0.822	0.832	0.852	0.889
Prompt 2	0.659	0.736	0.606	0.682	0.684	0.651	0.781
Prompt 3	0.688	0.731	0.621	0.672	0.695	0.804	0.815
Prompt 4	0.778	0.801	0.742	0.814	0.788	0.888	0.831
Prompt 5	0.805	0.823	0.784	0.803	0.815	0.885	0.873
Prompt 6	0.791	0.792	0.775	0.811	0.810	0.817	0.821
Prompt 7	0.76	0.762	0.730	0.801	0.800	0.864	0.834
Prompt 8	0.545	0.684	0.617	0.705	0.697	0.645	0.713
Avg	0.724	0.773	0.705	0.764	0.764	0.801	0.820

The comparison results show that MsCAEWL gains the five top accomplishments out of eight prompts and the optimal performance in the track of average QWK score. On the whole, S6, S2, and S4 have achieved relatively satisfactory in part of the tracks. From their experimental results and the model architecture, it can be concluded that the sentiment-oriented structures based on the cutting-edge language models greatly boost the experimental results, such as LSTM, BERT, and attention mechanism. Though the S6 obtains astonishing results in most prompts, the two obvious weakness items occur in prompts 2 and 8 which contain the biggest average length of the essay, 350 words. However, in these two tracks, the top models including ours are based on RNN serial models, namely, LSTM. That is, the RNN architecture remains the superior ability to capture sequential semantic features, especially in a long text, despite the fact that S6 particularly adapts the BERT to fit the large-scale text mission.

Owning to the multiple strategies incorporated in NN, the proposed system can benefit from the advantages of various technologies so as to meet different challenges. There is an obvious superiority in tackling long text, the proposed model dominates in almost all essay sets with more than 150 words size, which mainly leads to the superiority in average score.

Comparison based on the analytic scoring

In this section, we aim to verify the analytic scoring of the proposed system. Hence the dataset with trait-specific scores as labels is primary to the validation experiments. Though the ASAP corpus provides the trait-specific scores on part of sets, there are two obstacles making the dataset unsuitable for the experiments. Only two prompts are presenting the analytic scores, prompts 7 and 8. However, the trait categories differ from each other. The scoring of prompt 7 evaluates Ideas, Organization, Style, and Conventions, while Ideas and Content, Organization, Voice, Word Choice, Sentence Fluency, and Conventions involves in the assessment of prompt 8. That means, the evaluation rubrics of each prompt are different, and the focus on parts of writing ability from the examiners are different. Moreover, the compositing methods of total scores are distinct. The final total score on each essay is simply obtained by adding 2 raters’ sum scores of all traits in prompt 7, rather than the weighted sum of scores on each trait by 2 taters in prompt 8. Since the traits and compositing methods of MsCAEWL are not consistent with any prompts above, it reveals unlikely to be utilized as the basis of the analytic scoring comparison.

By all accounts, a new validation experiment is set as follows. A group of college students is asked to write an essay according to a prompt. The essays then are scored by human raters and MsAWE as well following the detailed analytic scoring instructions concretely categorized as Thesis, Fluency, Content, Complexity, and Correctness. Accordingly, the trait-specific and total scores between human and machine raters are obtained eventually to be leveraged to reveal the analytic scoring ability of the proposed writing computer-assisted writing learning by statistical analysis.

In this experiment, 80 sophomores majoring in English are invited as subjects, i.e., 80 compositions will be produced. Three experienced college teachers of English language and literature review and score each essay so that every essay contains three score sets with six scores. The writing prompt (see Table 8) is selected from the Tests for English Majors Grade Four (TEM 4) which is specialized for evaluating the comprehensive EFL ability of English majors in China.

Table 8.

Writing Prompt From TEM 4.

Prompt	Source
Read carefully the following excerpt on term-time holiday arguments in the UK, and then write your response in NO LESS THAN 200 words, in which you should	TEM-4 of 2016
1) Summarize the main message of the excerpt, and then
2) Comment on whether parents should take children out of school for a holiday during term time in order to save money
You should support yourself with information from the excerpt
Marks will be awarded for content relevance, content sufficiency, organization, and language quality. Failure to follow the above instructions may result in a loss of marks
*Specific source reading material is omitted here

The scoring rubrics, coefficient settings, score scope, and the score compositing equations in this experiment are shown in section “Task Definition,” which are mainly derived from the grading criteria of TEMS. The statistical results of trait and total scores from human and machine raters are shown in Table 9.

Table 9.

The Statistical Results of Trait and Total Scores From Human Raters and MsCAEWL.

	Thesis		Fluency		Content		Complexity		Correctness		Total
	Mean	Std	Mean	Std	Mean	Std	Mean	Std	Mean	Std	Mean	Std
R1	14.10	1.29	10.43	0.84	20.25	1.54	13.89	1.17	11.23	0.85	69.90	5.71
R2	13.74	1.39	9.43	0.81	21.84	1.62	12.10	1.23	10.93	0.81	68.02	5.85
R3	14.47	1.18	9.93	0.80	21.05	1.61	12.98	1.09	11.02	0.76	69.45	5.43
Avg	14.10	1.33	9.93	0.91	21.04	1.72	12.99	1.37	11.05	0.82	69.16	5.35
MsCAEWL	13.84	0.86	10.21	0.70	20.40	1.18	13.42	0.75	11.37	0.53	69.25	4.02

The box-plot diagrams in Figure 5 depict the distribution of rating scores of MsCAEWL and human annotators. It can be seen that, overall, the machine rating keeps constituent with the average rating of three human raters, which is also consistent with the QWK results in section “Comparison based on holistic scoring.” That is, MsCAEWL is reliable in scoring in comparison with the overall human level.

Figure 5.

Box-plot diagram of score distribution by MsCAEWL and human raters. The box-plot diagrams show the distribution of rating scores of MsCAEWL and human annotators. Comparison of the diagrams indicates that the machine rating is consistent with the average rating of three human raters. This implies that MsCAEWL is reliable for scoring in relation to the overall human level.

It should be noted that the data distribution of MsCAEWL ratings is significantly less volatile than that of the human ratings. More important, human ratings vary widely from one another, and it is difficult to maintain consistency under each category. The opinions of human raters keep the relative agreement in thesis and correctness. It shows that human raters are easier to reach an agreement at judging digression and quality of organization. Furthermore, the review of grammar, spelling, mechanics, and other errors is more objective, and human judges are less likely to disagree on such issues. However, for evaluations that need multi-dimensional consideration, such as fluency, content, and complexity, human ratings are prone to diverge. To recap, the scoring of MsCAEWL is more robust.

Validation of effects on writing learning assistance of MsCAEWL

The aforementioned experiments can only compare model performance based on their scores. Our model may offer not only scores, but also feedback such as improvements in vocabulary and grammar, error correction, and so on. Scores are solely used as a reference for validation of improvements in writing ability, the major priority of the proposed system is to enhance students’ writing skills. By that means, in this part, we devised experiments to verify if MsCAEWL can increase students’ writing abilities.

Method

In this section, 70 high school sophomores with comparable scores in a high school in Guizhou province were selected as subjects for a 60-day writing learning experiment. Among them, 35 were randomly assigned to the experimental group and 35 to the control group. There was no significant difference in writing proficiency between the two groups. The experimental group and the control group received ordinary face-to-face writing instructions from the same teacher during the experiment. Differently, in the experimental group, the MsCAEWL is adopted to assist writing learning. According to the writing feedback by MsCAEWL, the experimental group students revise their essays by themselves. A pre-test was performed before the experiment, and a post-test was performed after the experiment for the experimental and the control group. The performances of the two groups are analyzed by independent-sample t-test and paired-sample t-test with SPSS 26.0 to verify the effects of MsCAEWL in improving students’ English writing proficiency. All compositions in the experimental and control group were scored by MsCAEWL.

Participants

Participants $\left(n=70\right)$ were high school sophomores from the same school in Guizhou province, China, aged 16–17 years. All participants take eight English lessons per week, each lesson lasts 45 minutes according to their school schedule. There were 35 participants in the experimental group $\left(male=20,female=15\right)$ , and 35 participants in the control group $\left(male=16,female=19\right)$ . The two groups took a writing test before the writing instruction experiment. There was no significant difference in writing proficiency between the two groups. The average scores of participants between the control and experimental group in the pre-test are comparable: 62.32 (SD 6.99) and 61.68 (SD 5.85), respectively.

Materials

In the writing learning experiments, several essay prompts from the former National Matriculation English Tests are adopted as the writing learning and teaching materials. Given the short period of the verification experiment, the familiar materials are more beneficial for students to achieve better writing performance. Moreover, the prompt design of the National Matriculation English Tests is more rigorous, and the difficulty of each prompt varies slightly, so it is more suitable for the experiment of this study. The prompts are selected randomly as follows:

Procedure

Before the experiment, the experimental and the control group were pre-tested with the first writing prompt to determine whether there was a difference in writing proficiency between them. At the end of the experiment, the two groups were post-tested using the fourth prompt. During the experiment, two groups of students received four writing instructions from the same English teacher, with an interval of 20 days. The wring prompts are shown in Table 10. Teachers’ guidance, explanation of relevant vocabulary and sentence patterns, and thesis discussion are arranged in each instruction. In most scenarios of EFL lessons in China, there is merely no time for exclusive writing teaching and learning due to the large number of students and the heavy workload. Consequently, writing teaching could only be an adjunct to the explanation of the text and the test questions. The abundant essays after assignments make teachers have little time to review and issue detailed feedback. Commonly, a holistic score with a short comment is attached to each essay after the short-term review.

Table 10.

Prompts Adopted in Writing Learning.

Round	Prompt	Sours
First: Pre-test	Assuming you are Li Hua, and you would like to invite Henry, a foreign teacher, to visit the Chinese paper-cutting art exhibition. Please write him a letter, which includes: (1) Exhibition time and place; (2) Exhibition content	National test II of 2017
Second	Assuming that you are Li Hua, your New Zealand friend terry will visit a Chinese friend’s home and send an email to ask you about the customs. Please reply to the email, which includes: (1) arrival time; (2) appropriate gifts; (3) table manners	National test I of 2018
Third	Last weekend, you and your classmates took part in a picking activity. Please write a short essay for the class English corner to introduce this activity, including (1) farm activities; (2) the picking process; (3) personal feelings	National test II of 2020
Fourth: Post-test	Assuming you are Li Hua, and your English friend peter wrote to ask about the students’ sports in your school. Please write back to him, including (1) sports venues in the school; (2) main kinds of sports; (3). Your favorite sports	National test III of 2018

Thus, in order to simulate the real writing teaching and learning scene, each instruction is within 15 minutes. For the control group, the students will receive a holistic score, marks for grammatical and lexical errors, and a general comment just like the daily teaching. With regard to the evaluation in the control group, the raters are also asked to finish the review of each composition within 10 minutes. In this way, the marks of errors and the comment may be rough, inadequate, or even faulty. Note that only the holistic score by the teacher will be presented to the writer. Even though MsCAEWL will review each essay and output the sub and overall scores as experimental data which are completely blind to the control group.

As for the experimental group, besides the ordinary writing instructions, their essays are completely evaluated by the system rather than the teacher. After finishing the writing and evaluation, the students may revise each composition according to MsCAEWL’s multiple trait-specific scores and writing feedback. The revision effect can be checked because the revised essay can be scored again immediately. This computer-assisted English writing learning round is without limitation of times. However, in this experiment, the revision shall not stop unless at least three rounds are completed, or the score stops rising. All writing tests are carried out according to the requirements of The National Matriculation English Test, with a full score scope of 100 credits.

Results and discussions

a) The Effects of Writing Learning Assistance of MsCAEWL on Overall Performance

In order to verify if a difference exists in the means scores between the control and experimental group on the pre-test, an independent-sample t-test was conducted to compare the English writing competence between the control group and the experimental group (see Tables 11–14).

Table 11.

Group Statistics of Pre-Test Scores in Experimental Group and Control Group.

Group	N	Mean	Std. Deviation	Std. Error Mean
Control group	35	62.32	6.99	1.18
Experimental group	35	61.68	5.85	0.99

Table 12.

Independent-Sample t-Test of Pre-Test Scores of Experimental Group and Control Group.

			Equal variances assumed	Equal variances not assumed
Levene’s test for Equality of variances	F		0.692
	P		0.408
MsCAEWL	t		0.413	0.413
	df		68	65.972
	Sig.(2-tailed)		0.681	0.681
	Mean difference		0.637	0.637
	Std.Error difference		1.542	1.542
	95%Confidence interval of the difference	Lower	−2.441	−2.442
		Upper	3.714	3.717

Table 13.

Group Statistics of Post-Test Scores of Control and Experimental Group.

Group	N	Mean	Std. Deviation	Std. Error Mean
Control group	35	63.39	6.89	1.16
Experimental group	35	69.33	6.44	1.08

Table 14.

Independent-sample t-Test of Post-Test Scores of Experimental Group and Control Group.

			Equal variances assumed	Equal variances not assumed
Levene’s test for Equality of variances	F		0.016
	P		0.899
t-test for Equality of means	t		−3.725	−3.725
	df		68	67.682
	Sig.(2-tailed)		0.000	0.000
	Mean difference		−5.942	−5.942
	Std.Error difference		1.595	1.595
	95%Confidence interval of the difference	Lower	−9.126	−9.126
		Upper	−2.759	−2.759

The results of the independent-sample t-test for the pre-test in Tables 11 and 12 showed that there was no statistically significant difference in the scores for the control group ( $M=62.32,SD=6.99$ ) and the experimental group ( $M=61.68,SD=5.85$ ); $t\left(68\right)=0.413,P=0.408,Sig.\left(2-tailed\right)=0.637$ (see Tables 11 and 12), which suggests that the writing proficiency did not differ between the two groups before the experiments.

In order to examine if the effects of writing learning assistance of MsCAEWL took place at the end of wringing instructions in the experimental group and the control group, after the experiment, the post-test scores of the two groups were investigated by independent-sample t-test.

The results in Tables 13 and 14 revealed a statistically significant mean difference between the control ( $M=63.39,SD=6.89$ ) and experimental group ( $M=69.33,SD=6.44$ ) after the experiments in terms of the writing proficiency; $t\left(68\right)=-3.725,P=0.899,Sig.\left(2-tailed\right)=0.000$ . The mean score of the experimental group is greater than that of the control group. The experimental results show that there is a statistically significant difference between the two groups in writing proficiency, and the writing competence of the experimental group is higher. In other words, the computer-assisted EFL Writing Learning System, MsCAEWL, based on writing feedback theory plays an essential role in improving students’ English writing proficiency.

b) The Effects of Writing Learning Assistance of MsCAEWL on Trait-specific Performance

On the basis that MsCAEWL can effectively assist students in learning English writing, in order to have a deeper understanding of which aspects of writing learning and to what extent our system has produced positive effects, paired-sample t-tests were conducted to compare pre-test and post-test scores in each trait of writing, i.e., Thesis, Fluency, Content, Complexity, Correctness, and total score.

For reference, the paired-sample t-test results in Tables 15 and 16 indicate that there is no statistically significant difference between the pre-test and post-test in each track in the control group. Regardless of the increase in mean scores for Thesis, Fluency, Content, Complexity, and Correctness, the improvement was insufficient, not to mention the decrease in Fluency. According to the findings, it is difficult for students to enhance their writing proficiency with only limited unilateral instructions and feedback from teachers. In addition, simply doubling the number of writing assignments cannot considerably boost students’ writing learning.

Table 15.

Statistics of Control Group.

		N	Mean	Std. Deviation	Std. Error Mean
Thesis	Pre-test	35	12.306	1.357	0.229
	Post-test	35	12.557	1.654	0.279
Fluency	Pre-test	35	9.280	1.007	0.170
	Post-test	35	9.149	1.235	0.208
Content	Pre-test	35	18.931	2.018	0.341
	Post-test	35	19.171	2.476	0.418
Complexity	Pre-test	35	12.566	1.466	0.247
	Post-test	35	12.846	1.534	0.259
Correctness	Pre-test	35	9.237	1.147	0.194
	Post-test	35	9.669	1.106	0.187
Total	Pre-test	35	62.319	6.994	1.182
	Post-test	35	63.391	6.898	1.166

Table 16.

Paired-Sample t-Test of Control Group.

			Thesis	Fluency	Content	Complexity	Correctness	Total
Paired differences	Mean		−0.251	0.131	−0.240	−0.280	−0.431	−1.071
	Std. Deviation		2.298	1.429	3.406	1.902	1.708	9.980
	Std. Error mean		0.388	0.241	0.575	0.321	0.288	1.687
	95% confidence interval of the difference	Lower	−1.040	−0.359	−1.410	−0.934	−1.018	−4.499
		Upper	0.538	0.622	0.930	0.374	0.155	2.357
t			−0.647	0.544	−0.417	−0.871	−1.494	−0.635
df			34	34	34	34	34	34
Sig. (2-Tailed)			0.522	0.590	0.679	0.390	0.144	0.530

In order to whether and on which trait MsCAEWL yields writing learning assistance, a paired-sample t-test was conducted to compare the writing scores between the pre-test and the post-test of the experimental group.

According to Tables 17 and 18, a statistically significant difference can be observed between the pre-test and post-test in every trait and the total score. That indicates that our proposed system is capable of effectively assisting EFL writing learning. It is worth noting that the scores of the experimental group in Complexity and Correctness improved comparatively greater. The mean scores of Complexity have significantly increased by 10% from pre-test to post-test: an increase of $Mean=-2.017$ out of the full score of 20 credits, $t\left(34\right)=-5.371,p=0.000$ , and the mean scores of Correctness have significantly increased by 11%: an increase of $Mean=-1.622$ out of the full score of 15 credits, $t\left(34\right)=-24.982,p=0.000$ . The findings suggest that even though MsCAEWL can improve students’ writing ability from all-around aspects, students can obtain better benefits in morphology and syntax than in mastering unity and organization. Judging from the results, students can more easily get promotion from the corrections of lexical and syntactic errors, and more suggestions on the use of words and sentences, which is also the most significant advantage of computer-assisted writing learning.

Table 17.

Statistics of Experimental Group.

		N	Mean	Std. Deviation	Std. Error Mean
Thesis	Pre-test	35	12.240	1.268	0.214
	Post-test	35	13.680	1.699	0.287
Fluency	Pre-test	35	9.360	0.940	0.158
	Post-test	35	10.237	1.381	0.233
Content	Pre-test	35	18.626	1.887	0.319
	Post-test	35	20.320	2.213	0.374
Complexity	Pre-test	35	12.251	1.740	0.294
	Post-test	35	14.269	1.601	0.270
Correctness	Pre-test	35	9.206	1.016	0.171
	Post-test	35	10.829	1.364	0.230
Total	Pre-test	35	61.682	5.859	0.990
	Post-test	35	69.334	6.441	1.088

Table 18.

Paired-Sample t-Test of Experimental Group.

			Thesis	Fluency	Content	Complexity	Correctness	Total
Paired differences	Mean		−1.440	−.877	−1.694	−2.017	−1.622	−7.651
	Std. Deviation		0.451	1.566	0.352	2.221	0.384	3.728
	Std. Error mean		0.076	0.264	0.059	0.375	0.065	0.630
	95% confidence interval of the difference	Lower	−1.595	−1.415	−1.815	−2.780	−1.754	−8.932
		Upper	−1.284	−.338	−1.573	−1.253	−1.490	−6.370
t			−18.851	−3.312	−28.455	−5.371	−24.982	−12.140
df			34	34	34	34	34	34
Sig. (2-Tailed)			0.000	0.002	0.000	0.000	0.000	0.000

Conclusion and Future Works

How MsCAEWL Can Benefit Writing Learning Under the Guidance of Writing Feedback

In this study, we proposed a computer-aided writing learning system that integrates the neural network models and a couple of semantic-based NLP techniques. Different from the previous AWE or AES models, which mostly adopt the holistic scoring method, the concept of our proposed system follows the writing feedback theory. Combining the advantages of the AWE or AES model, neural networks, and multiple NLP technologies, our proposed system can provide a package of writing feedback that fully meets the requirements of writing feedback theory, namely, multiple, continuous, timely, clear, and multi-aspect guidance interactive feedback.

More specifically, as with the other AWE models, MsCAEWL is designed for easy operation, allowing students and teachers to use it multiple times without burden and obtain feedback immediately. The analytic scoring method, multiple trait-specific evaluations, is adopted in MsCAEWL in order to provide the multi-aspect guidance feedback. Thank to this, a temporal matrix of writing assessment can be easily formed based on feedback on each trait-specific evaluation after each revision of a composition. Students and teachers can intuitively observe the continuous changes in the composition and provide dynamic feedback.

Users can also use the scores of trait-specific evaluations in this system (rather than all of them) to compose different assessment matrices, thereby achieving different teaching or learning goals. It is worth noting that the utilization of writing feedback theory or AWE models is predicated upon a fundamental principle, namely that learners possess sufficient self-regulation to execute or teachers can oversee multiple careful revisions of a writing task. In our observation, even in instances where AWE or teachers proffer detailed and constructive feedback, the benefits derived from such feedback are exceedingly circumscribed if the author only revises once without attending to the feedback thereafter.

Conclusion

In this article, following the guidance of writing feedback theory, we designed a Multi-strategy Computer-assisted EFL Writing Learning System (MsCAEWL) in order to provide the writing multiple trait-specific evaluation scores and the detailed writing instructions.

In the comparison experiment of holistic scoring, MsCAEWL achieved the best performance on multiple prompts and the final average score. Due to the introduction of various semantic-based NLP technologies, our proposed system outperformed other outstanding baseline models incorporating the single model in their backbone structure.

The experiment of analytic scoring proves that in multiple trait-specific scoring, our system is more reliable and more robust than human raters. In other words, MsCAEWL is more suitable as a reference standard for writing verification experiments. It can not only provide more comprehensive multi-aspect assessments but also avoid individual differences in human scoring and inconsistent rating due to fatigue.

However, scoring compositions are only one of MsCAEWL’s capabilities. The most valuable task is to improve the author’s writing ability through multiple rounds of writing revision based on the interactive and all-around feedback provided to the author. In the validation experiment of assistance effect in writing learning, learners may utilize MsCAEWL’s writing feedback to edit their compositions without the restriction of use times, and they can immediately acquire the score changes after one round of modification to lead the next round of modification. With the assistance of our system, learners’ writing skills may be greatly enhanced over time, indicating that the writing feedback theory assisted by the computer is more effective.

Future Works

In the future, we will focus on text evaluation based on paragraph and text intention extraction, as well as evaluating the continuity between paragraphs and support for the topic based on paragraph intention. We will strive to bring the model closer to human thought in works of writing evaluation and feedback. Simultaneously, we will consult with second language acquisition specialists on the weight allocation and rationality of the traits in writing assessment.