Sage Journals: Discover world-class research

Abstract

In recent years, automated feedback (AF) has greatly increased its applicability as a mean to support development of English as a foreign language (EFL) learners’ writing competence in the context of face-to-face education. This study examines contribution of AF to the development of writing competence of EFL learners in open and distance learning context. Anadolu University, Open Education Faculty learners received regular feedback on their writing tasks by an AF tool, Write & Improve. The contribution of AF to academic writing success was observed by statistically comparing the grades of the learners for their writing tasks at the beginning and at the end of the process and also by correlation analysis between AF frequency and achievement scores. In the second stage of the study, detailed data about learners’ experiences were obtained by interviewing volunteer participants. The quantitative and qualitative findings of the study showed that AF significantly contributed to the development of writing competence of EFL learners at a distance. Interviews with learners revealed themes such as effective learning process, contribution to foreign language development, motivation increase, effective and fast feedback, user-friendly learning environment, support for individual learning, and contribution to distance learning as the strengths of AF. The interviews also included participants’ suggestions on how to use these tools more effectively. Based on the findings, pedagogical implications are discussed regarding the incorporation of AF in distance EFL learning.

Keywords

Automated feedback distance learning online learning automated writing evaluation

Introduction

It is very important to provide continuous feedback using different methods in order to promote foreign language competence. A more efficient process can be obtained when foreign language learning processes are supported with formative feedback rather than summative feedback (Alvarez et al., 2014). Formative feedback can be defined as any kind of indirect, direct or semi-corrective remedial written comment, sign or indicator that the learner can confirm, add on, and restructure his/her current knowledge whenever he/she needs during the learning process. Feedback, especially applied in writing activities, with multiple drafts in the form of review, is seen as an important developmental tool for learners to express themselves effectively, increase their motivation and reinforce what they have learned (Hyland and Hyland, 2006). With increasing number of learners in open and distance learning context, standard approaches in providing individualized feedback for tens of thousands of learners do not seem to be applicable (D’antoni et al., 2015). Assessment and feedback are regarded as an area of deficiency by learners due to the difficulties faced in many higher education institutions in open and distance learning (Marriott and Teoh, 2014). Need for continuous formative feedback in the processes of English as a foreign language (EFL) learning, and the inability to meet this need in an effective, efficient and attractive way in the face of the increasing number of learners in the context of open and distance learning poses a problem. As a solution to this problem, the study investigates the contribution of an artificial intelligence-based Automated Feedback (AF) tool, Write & Improve, to the development of writing competence in distance EFL learning.

Interaction and formative feedback in foreign language learning

According to the Interaction Hypothesis in the field of foreign language acquisition, structures such as input (Krashen, 1985; Long, 1996), interaction (Pica, 1994), feedback and output (Swain, 1995) play important roles in the learning process. According to this hypothesis, interaction is the context in which language is used and learners can receive feedback that confirms their communicative success or points to failure in their production and thus helps them to recognize problematic language use. This feedback encourages learners to create hypotheses about the nature of language problems by reviewing what they produce and experimenting with new language forms and structures (Swain, 1995). Learners engage in dialogue within themselves by making reflections on what they produce, develop their own understanding, and ensure their language development by internalizing correct forms (Swain, 1995). Similarly, Ellis (2000) defines one of the assumptions of the Interaction Hypothesis as the internal reasoning of the learners between the correct structures and the structures they produce in the process of foreign language learning. Therefore, language learners need tools such as repetitions, affirmations, revisions, and controls to reinforce the meaning (Pica, 1994). Taken in the context of the Interaction Hypothesis, AF can point to a problematic area and make the learner think about it and try again. The learners can test their hypothesis by becoming aware of their mistakes thanks to the inputs provided by the immediate feedback, which in turn can ensure language success (Long, 1983). Considering that interaction takes place not only in communication environments with other people but also with computers (Chapelle, 2003), this interaction can also promote language development (Chapelle, 1998) by means of using AF, where learners’ output can be marked to alert them to errors in their output. Such feedback can focus learners’ attention on language form, enable them to recognize the discrepancy between their forms and target forms and correct their errors, and eventually improve the language learning process (Robinson et al., 2012).

Automated feedback can be employed as a solution to the increasing workforce need that arises with the increasing number of learners in open and distance EFL learning environment. After each attempt on these tools, learners can realize their problems or subjects they need to improve related to the target language, as well as the structures they know wrong thanks to feedback they receive on the texts they produce. This awareness can be achieved through explicit or implicit feedback provided by AF tools.

Interaction and formative feedback in open and distance learning

Interaction is seen as the most important component of open and distance learning (Anderson, 2003; Moore, 1989). Anderson (2003) claims that independent work and interaction can be combined with the development of new and cost-effective technologies that support learning in all directions. As in all communication processes, feedback plays an important role in the efficiency of the interaction in open and distance learning. Feedback is an effective factor in increasing learners’ belonging to the program, increasing teacher-learner interaction, and participation in activities (Moore and Kearsley, 1996). As learners become aware of their progress with the continuous formative feedback they receive, their motivation and their interaction with the program they study will increase (Moore and Anderson, 2007). In addition, feedback is effective in increasing learners’ academic achievement by means of acquiring target skills and increasing their motivation to be a part of the community (Boling et al., 2012; Bonnel, 2008; Hattie and Timperley, 2007). Although it has been suggested that feedback in open and distance learning reduces the sense of isolation of the learner by enabling more effective communication (Alger and Kopcha, 2009; Simpson, 2006), it is often not possible to provide formative feedback to learners due to the rapidly increasing number of learners (Simonson et al., 2000). At this point, AF can be a solution to improve the process.

Automated feedback (AF) in foreign language writing

Artificial intelligence, machine learning, and natural language processing technologies have contributed to the development of automated writing evaluation systems that provide AF and these AF tools aim to process and evaluate learner input to provide error detection, error source and instant individualized feedback (Al Emran and Shaalan, 2014). In open and distance learning context where there are large classes and limited resources, it is not always possible for teachers to provide individualized and timely help for written works of the learners. For this reason, many learners can receive very little guidance for the drafts they produce to improve their writing competence (Caccamise et al., 2007). Effective writing training includes providing feedback to each draft assignment of each learner (Warschauer and Grimes, 2008). In this context, AF provides great convenience in the learning experience, especially in terms of the effective and fast formative feedback it provides in language learning (Boud and Molloy, 2013; Nicol, 2010).

Some widely used AF tools, such as Criterion by Educational Testing Service, WriteToLearn by Pearson, MyAccess by Vantage Learning, draw attention as they are as accurate as human raters in evaluating and responding to learner writing (Attali and Burstein, 2006). Some AF tools (e.g. Criterion and MyAccess) are basically based on statistical modelling in which learner experiments are analysed according to linguistic features designed on algorithms (Roscoe et al., 2014). Other tools like Writelab, Summary Street® and Write and Improve can also provide graphical feedback showing how well the learner is writing (Caccamise et al., 2007). By using natural language processing technology, Sentence Fairy helps learners evaluate and improve the grammatical quality of their work (Harbusch et al., 2008). Grammarly and LanguageTool are AF tools that provide iterative error checking with immediate feedback to support writing competence. With the studies conducted on these programs, it has been determined that grammatical and spelling errors correction and frequent error patterns are successfully handled by these tools (Rudzewitz et al., 2018).

The contribution of AF to the development of writing competence in a foreign language has been examined in some studies (Wilson et al., 2014). Some research results show positive correlations between AF and the improvement of learners’ academic performance both in writing and in foreign language learning in general (Shermis et al., 2008). Some findings show that AF helps learners reduce the number of errors in grammar, mechanics, and writing style (Kellogg et al., 2010). It has been observed that using these tools increases the content and overall quality of the compositions (Wade-Stein and Kintsch, 2004). Studies show that learners who receive feedback through such tools regularly tend to write more, produce higher quality drafts and make fewer mistakes (Shermis et al., 2008).

In the interviews conducted in the study of Sherafati, Largani, and Amini (2020), learners emphasized that although they found the feedback they received from the computer useful, AF could be valuable when used in the presence of a teacher as a support to teacher feedback. In the study of Link et al. (2014), teachers whose opinions were taken on AF stated that these tools can provide instant feedback to learners and take on the role of an assistant, thus reducing their workload and shortening the learner’s grading time for their papers. The authors concluded that automatic feedback, which assumed the role of an assistant, helped learners gain autonomy and motivation (Link et al., 2014). Another study aimed to investigate how novice research writers interact with Research Writing Tutor (RWT), which is an automated rhetorical feedback platform developed for graduate student writers to complement formal academic writing instruction. Results showed that students’ interactions with the platform varied, which means that the platform supports learners to individualize their own paths (Cotos et al., 2020).

It is seen that some experimental studies are success-oriented studies in writing skills with pre-test post-tests analyses, usually with the experimental group using only the AF tool and the control group gets feedback from the teacher or their peers, and the practices in these studies are carried out only with the guidance of the teacher (e.g. Cheng, 2017; Huang and Renandya, 2020). While there are few studies in the literature in which different automatic feedback tools are employed (Cheng et al., 2016), studies on AF tool Write and Improve, which is the tool used in this study, are quite limited (e.g. Curry and Riordan, 2021; Huijser and Wali, 2018; Karpova, 2020). The study conducted by Curry and Riordan (2021) revealed the contribution of Write and Improve to learner engagement, motivation, improvements in writing practices and written language proficiency. Researchers point out that when Write and Improve deals with spelling and syntactic accuracy, it helps teachers focus on more complex aspects of learner texts. Similarly, in the study conducted by Karpova (2020) learners found Write and Improve to be highly effective both in teaching and learning processes and in assessment. Kostikova and Miasoiedova (2019) conducted an experimental study in which they presented Write and Improve as an additional extra-class tool for post graduate students. Findings revealed significant difference between experimental and control group in terms of improvement of academic writing skills. In the study, language development was investigated based on different criteria which were content, communicative achievement, organisation and language. Most significant progress was observed in language (vocabulary and grammar).

Write and improve

Developed with the collaboration of Cambridge English an AF tool Write and Improve stands out as a tool that provides different types of AF for written tasks in different English proficiency levels. Write and Improve can determine the language level of students who are compared according to the common European Framework of Reference (CEFR). This tool provides summative feedback, indirect and formative feedback, and students’ progress feedback on writing. By this way, technology aims to guide students to recognize and address language mistakes while facilitating students’ autonomy and participation (Curry and Riordan, 2021). Providing different workbooks for writing practice, this platform offers a pedagogical practice environment rather than acting as a text editing tool (Cambridge English, 2021). The platform lets learners to choose a topic from the workbooks provided and get immediate feedback on grammar, spelling, vocabulary, style and register (Kostikova and Miasoiedova, 2019). Write and Improve may have potential to offer an environment for distance language learners where they can practice completely individually.

When learners complete their texts and click the control button, the tool provides four types of feedback - summative, formative (indirect semi-corrective), overall performance and progress chart. As summative feedback, it provides an overall assessment of writing competence and indicates the level of English proficiency from A1 to C2. As formative feedback, the tool highlights the problematic parts of the text with certain icons pointing to errors at the sentence or word level by colouring the relevant areas after each attempt. Overall performance feedback is done in a similar way to the teacher encouraging the learner and it enables the learner to obtain information about their current performance. It allows the learner to measure his own learning by comparing the learner’s latest performance with his previous performances. Finally, individual progress chart helps learners visualize their progress in an activity. This graph emphasizes that progress usually takes place in small, steady steps and it gives the learner awareness of their progress. Figure 1 shows a sample feedback display provided by Write and Improve.

Figure 1.

A sample feedback provided by write and improve.

Write and Improve as an AF tool was preferred in this study for some reasons. First of all, the tool has features that cannot only help native speakers in their academic writing development, but also provide support to EFL learners with different language proficiency levels. Also, the tool, which is available for free, differs from other AF tools developed entirely for commercial purposes. Learners can make countless experiments with just three sentences with the help of this feedback tool. This tool was found to be usable regardless of the length of the texts and the level of language proficiency. Also, the features of other tools that are widely used around the world do not match the target audience and purpose of this study. For example, Criterion stands out as a tool developed to give feedback to essay texts consisting of five paragraphs (Chukharev-Hudilainen and Saricaoglu, 2016). Other tools such as MyAcsess and Writetolearn are mostly developed for academic essay writing and have a scoring component according to certain criteria (Warschauer and Grimes, 2008). It is also known that these two tools are mostly preferred for classroom use (Ware and Warschauer, 2006) and do not make process-oriented evaluations (Calvo et al., 2010). As another tool, W-Pal was not found suitable for the study, since it was designed to improve the composition skills of native speakers (Roscoe et al., 2014). Considering that the target audience of this study is distance learners who are educated within the Open Education Faculty, the general English proficiency is at A2 level with certain exceptions, and these learners have not received a formal training in academic writing activities, the tools developed for article evaluation and scoring would not coincide with the levels of the learners.

The aim of this study was to examine whether using an AF tool, Write and Improve, for providing formative feedback to the writing tasks of open and distance EFL learners contributes to their writing competence. Based on the quantitative and qualitative data obtained at the end of the process, it was aimed to observe whether continuous formative AF and its frequency contribute to the writing achievement. Due to the limited empirical research investigating the contribution of AF to EFL learners’ writing achievement in open and distance learning context, and limited research on Write and Improve as an AF tool, this study set out to investigate the following research questions:

RQ1. Does getting formative AF in the context of open and distance learning contribute to the development of English writing?

RQ2. What are the opinions of distance EFL learners regarding the AF they received for their writing tasks?

Method

In order to serve the purpose of the research, mixed methods sequential explanatory design was used in the study. Mixed method research refers to the researcher’s collecting data by combining qualitative and quantitative methods, approaches and concepts in a study (Creswell, 2014). Mixed method studies can capture a larger picture of the phenomenon under investigation. In sequential explanatory design, quantitative data are collected and analysed first. Qualitative data are then collected to explain, elaborate and strengthen the quantitative results obtained (Creswell, 2014). Mixed method research makes it possible to look for answers to both “what” and “why” questions at the same time and it is powerful because it has a ‘triangulation’ feature (Flick, 2018). In this context, the quantitative dimension of the study involved calculation of correlation coefficient of the students’ scores for the tasks with the number of attempts they made. Also, the pre-test/post-test achievement scores were compared. The qualitative dimension of the research was completed with the content analysis of the interviews conducted with the participants at the end of the process.

Participants

The sample of this research consisted of volunteer distance EFL learners who study at different undergraduate programs of Anadolu University Open Education Faculty. The participants were determined through appropriate sampling. They all successfully completed their distance English Language Course based on Touchstone Self-Study Edition coursebook and were at A2 proficiency level according to CEFR. Seventy-nine adult learners responded positively to the invitation to participate in the study, and enrolled in Write and Improve AF platform. The age range of the participants was 19–53. Participants were expected to complete five different writing tasks, and the number of participants who completed all activities and formed the sample of the study was 29. Other participants were not included in the study because they either did not do any activities or completed less than five activities. The high drop-out rate of the participants might be justified with the fact that this activity was a completely extracurricular activity that had no effect on overall academic achievement and the participants were all adult learners who could not spare time due to different responsibilities including family and work.

Participants were divided into groups according to the attempt frequency and task achievement scores factors. An invitation was sent to all of these groups and interviews were conducted with nine participants who responded positively. By this way, interviews were conducted with a heterogeneous group of participants with different AF attempt frequency and success scores.

Procedure

The participants were enrolled in an online class on Write and Improve. The researcher was able to follow all the activities completed by each learner and all the attempts for the activities, the feedback given, the frequency of attempts, and the progress graphs both individually and as a group throughout the process. Before the activity orientation videos on how to register and use the tool were prepared by the researcher shared with the participants. Writing tasks on five different topics were carried out for 5 weeks. The tasks were based on the themes chosen from Touchstone Self-Study Edition coursebook, and all of them were designed as descriptive tasks in order to eliminate the impact of genre difference on writing achievement. Each task required participants to write 30–60 words. Anadolu University School of Foreign Languages writing evaluation criteria, which had components as content, organisation, grammatical competence, lexical competence was used to evaluate the tasks. First and last drafts produced by the learners for each task were scored by two independent raters. By this way, participants’ writing achievement could be observed throughout the process. The averages of the scores given by both raters for each task were calculated. Inter-rater reliability for the given scores for the first and final drafts was calculated, and correlation was found to be 0.889 and 0.875 respectively. Considering Intraclass Correlation Coefficient (ICC) of 0.80 or higher indicates high reliability (Prince, Makrides, & Richman, 1980), it can be concluded that there was high level of inter-rater reliability between the two raters of the study. In addition, the average of the number of attempts made in all activities for each learner was calculated. For the second research question, interviews with nine volunteers were conducted at the end of the process. The interviews were conducted in participants’ mother tongue in order to get more insight into the phenomenon.

Data analysis

In order to observe the contribution of the AF to writing development, the scores given to the first draft in the first task and the last draft in the fifth task, namely the pre-test and the post-test scores, were compared by paired sample t-test analysis. Following the normality tests, the t-test analysis was carried out to find out whether the success scores of the learners before receiving and after receiving AF showed a statistically significant difference.

Following the normality tests, correlation analysis was conducted to determine whether there was a relationship between the number of attempts and achievement scores. For this, the average of the number of attempts of each learner for all activities and the average of the attempt scores of each learner for all activities were taken.

Within the scope of qualitative data, audio recordings of interviews with learners were transcribed and content analysis was carried out. The content analysis of the data was carried out by two independent researchers using the Continuous Comparison Method. In this method, analysis is done by comparing all semantic units obtained by inductive category coding simultaneously (Glaser and Strauss, 1967). In the process of content analysis, two independent researchers identified the expressions that they thought were directly related to the phenomenon in order to reveal the essence of the experience and formed independent meaningful units (codes) by grouping these expressions with colour codes. The same procedure was repeated for each participant and the texts were constantly read again throughout the process. The researchers came together and compared all the codes they obtained and formed the themes thought to describe the phenomenon.

Findings

In this section, quantitative findings on whether the artificial intelligence-based AF contributes to the development of EFL writing and comments regarding these findings are included. In addition, the findings of the content analysis conducted for the interviews with the volunteer participants were compiled and interpreted.

Contribution of AF to writing achievement

A paired-samples t-test was conducted to compare pre-test (no AF condition) and post-test scores (AF condition). There was a significant difference in the no AF (M = 9.86, SD = 2.65) and AF (M = 15, SD = 2.73) conditions; t(28) = −9.9, p = .000 Table 1.

Table 1.

Paired samples t-test results of pre-test and post-test.

		Mean	N	Std. Deviation		Std. Error Mean
Pair 1	Pre-test (no AF)	9.862	29	2.6555	0.4931
Pair 1	Post-test (AF)	15.00	29	2.739	0.509

Paired samples correlations
	Mean	Std. Deviation	Std. Error Mean	95% confidence interval of the difference		T	df	Sig.(2-tailed)
	Mean	Std. Deviation	Std. Error Mean	Lower	Upper	T	df	Sig.(2-tailed)
Pre-test (no AF) post-test (AF)	−5.13	2.78	0.51	−6.19	−4.07	−9.9	28	0.000

Correlation analysis was conducted to determine whether there was a relationship between the frequency of attempts the learners made on the artificial intelligence-based AF tool and their achievement scores. Although average scores for achievement showed a normal distribution, the frequency of the attempts did not show a normal distribution. For this reason, the nonparametric Spearman Rank correlation coefficient analysis, which is used in cases where the variables do not have a normal distribution, was preferred. The correlation coefficient (r = 0.437) seen in Table 2 shows that there is a moderate positive relationship between the number of trials and success levels. In addition, this relationship was found to be statistically significant (p < .01).

Table 2.

Correlation between attempt frequency and achievement scores.

		Attempt frequency	Achievement scores
Spearman’s rho Attempt frequency	Correlation coefficient	1.000	0.437**
	Sig. (1-Tailed)	.	0.009
	N	29	29
Achievement scores	Correlation coefficient	0.437**	1.000
	Sig. (1-Tailed)	0.009	.
	N	29	29

**Correlation is significant at the 0.01 level (1-tailed).

Qualitative findings

Content analysis of the online interviews revealed eight themes (Figure 2). The fact that most of the emerging themes have positive meanings shows that the learners generally think positively about AF. The participants also expressed the problems they experienced and suggested solutions.

Figure 2.

Themes derived from content analysis.

The participants evaluated the AF process positively, and they mentioned how much they benefited from the process of receiving feedback from this tool and how valuable they found this process. Most of the data obtained from the interviews with the learners were on the contribution of the AF tool to foreign language skills. According to learners, the AF process has created awareness to grammar, repetitive mistakes, and the importance of writing competence. The participants also accepted that they had important gains especially in terms of vocabulary, and the process made a significant contribution to vocabulary use, spelling, sentence formation and new vocabulary learning. Positive effects such as personal development, self-confidence and learning to learn were also mentioned in their expressions. Some of the participants stated that the process contributed not only to writing competence in a foreign language, but also to language competence in general.

Participants highlighted that they can also get unlimited AF and this process contributes to learning by making mistakes, learning through trial and error. They stated that by constantly trying and making mistakes in their drafts, their writing competence improved, and they were satisfied with this awareness. Some of the sample expressions are as follows:

Onder: “You have a chance to try unlimited times. This affects the language learning process very, very positively.”

Huner: “I see my mistake instantly, so I am learning by making mistakes… it is really great, like a private lesson, not just individual study, I can learn English just with its support.”

Content analysis revealed AF advantages such as receiving timely, detailed and fast feedback, receiving feedback that encourages individual learning and raising language awareness. In addition, the participants emphasized that the AF process contributed to their individual learning. They stated that this contribution was in the form of encouraging them to correct the error by inferring with indirect feedback, reflecting by following their progress and achievements, and encouraging them to strive through trial and error. In the light of these findings, it can be thought that AF can contribute to self-directed learning, gaining autonomy and individual learning especially in the context of distance learning. Some expressions of learners on this subject are as follows:

Huner: “We can see our mistakes right away and fix them instantly. AF tool rates us, and this is very nice. So where am I right now? What stage am I at? It shows my progress.”

Erdem: “For example, those colours guide us very well. The colours indicate that the sentence is completely correct or there are few mistakes or there are many mistakes… it guides well in different signs.

Metin: “You see the mistakes and try again. That is, it gives 2–3 levels of feedback; There is a word error, then a 1st level warning, a 2nd level warning, so we can understand exactly how many mistakes we have.”

Emir: “The tool doesn’t reveal our mistakes directly, and wants us to deal with our mistakes. This is also a good thing, I have to work again to correct that sentence; If it told me my mistakes the first time, it wouldn’t be so effective.”

The learners stated that they were able to realize their foreign language proficiency levels thanks to the formative feedback provided by the tool, and they could realize how much they improved thanks to the progress graph. They emphasized that the progress graph showing their language progress in particular encourages them to try more, to develop their language skills more, and to increase their level of proficiency. Also, the participants highlighted that their motivation, self-confidence and self-efficacy increased by receiving unlimited feedback any time they needed. Some of the original statements of learners on this subject are as follows:

Aysen: “When I saw the graph got higher, I said I am getting better.”

Emir: “There is a graph showing your level starting from A and progressing to C. On that graph, you can already see that you are getting better as the errors are getting fewer.”

Erdem: “I was obsessed with that graph. I go back to work and my mind is there… Colour change helped me a lot. Progress graph and colouring are great, they really encouraged me”

The learners emphasized that they can receive feedback from the AF tool whenever and wherever they want in the distance learning process they are in, and this situation is similar to the in-class applications and the support of the teacher. They stated that the negative situations they described, such as being away from any classroom environment and not being able to access a teacher at any time, disappeared thanks to this tool. They stated that it is a privilege in the context of distance learning to receive feedback without wasting time, without waiting, without hesitation, and without the need for anyone else, and this situation positively contributes to their individual development. They mentioned that they can access feedback exactly when they need it and receive detailed feedback from the tool as if they were receiving feedback from a teacher. Considering that the participants are distance learners, it can be deduced from the statements that timely, rapid, and limitless feedback plays an important role in the learning process and that it is possible to reduce the need for teachers in the context of distance learning to some extent with such tools. Some of their original statements are as follows:

Derya: “I do not have a teacher opportunity. I am trying to learn on my own and the only thing that can help me in this environment is the internet and it was a great privilege for me to have such an application on the Internet.”

Aysen: “I get feedback at any time any place. I recommend such a system to all distance learners.”

Metin: “While the teacher can give feedback to a person, the software can give thousands of feedbacks to many; It is absolutely beautiful, at home you can log in and try it alone, you can see your own level, you can add something on it yourself, it is definitely a very nice application without the need for a teacher.”

Participants also mentioned the difficulties they encountered while completing their tasks on the AF tool. They stated that their English proficiency levels were low in understanding the feedback they received from the AF tool, following the instructions and correcting the errors detected. Participants who had difficulties in understanding exactly what their errors were, how they could be corrected, what the guidance and suggestion windows and icons meant by the tool, stated that they sometimes tried to correct the wrong part repeatedly, and sometimes gave up trying. Some of the original expressions are as follows:

Derya: “I could not understand what those yellow colours mean, it was complicated.”

Emir: “Some mistakes are stated very implicitly, after a while I said enough is enough and left it because I cannot find what my mistake is, and there is no Turkish content at all.”

Erdem: “I did not find the icons, triangles, pop ups, the question marks efficient enough. Do I need to put something in there? Should I exclude a part? I don’t understand.”

The participants made various suggestions regarding the problems. The most emphasized suggestions for the improvement of the tool used are about providing a more detailed explanation and a clearer orientation, providing concrete suggestions, and guidance in mother tongue. As can be understood from the suggestions of the learners, the low level of foreign language proficiency prevents them from understanding and using implicit feedback effectively. According to the learners these problems can be solved with clearer, direct and explanatory feedback. Some of the original statements of the learners on the subject are as follows:

Derya: “It would be better for me to show our mistakes in more detail, it would be better if there was a more detailed explanation.”

Metin: “It would be nice if it showed us what the mistakes are driectly after trying something like 10 times.”

Nalan: “You made mistakes right here, yes, you should use that word, you didn’t use that structure or you forgot these punctuations, I would like it to tell me my mistakes directly. I wish it led in my mother tongue.”

Discussion and conclusion

According to the result obtained, the writing competence of the distance EFL learners improved from the first attempt to the last attempt they performed on the AF tool. The findings showed that formative feedback provided by AF tool significantly contributed to the development of EFL writing. As an important part of learner success in achieving target skills, feedback plays an important role in foreign language learning regardless of its source (Boling et al., 2012; Bonnel, 2008; Hattie and Timperley, 2007). With regard to contribution of formative feedback to EFL writing, this study showed parallel findings with the foreign language learning literature (Alharbi, 2016; Purnawarman, 2011). Similarly, automated formative feedback’s potential has been emphasized and proven by many studies in the literature (Curry and Riordan, 2021; Huijser and Wali, 2018; Hyland and Hyland, 2006; Karpova, 2020). As one of the language learning theorists Swain (1995) argued in the Output Hypothesis, learners can develop their foreign language skills only when they can transform their inputs into meaningful outputs and test the accuracy of what they produce by receiving feedback on these outputs. AF used in this study seemed to be a convenient platform where learners could transform their input into meaningful outputs and test their hypotheses.

The findings also showed that there was a moderate positive relationship between the number of attempts made by the participants, namely frequency of the attempts, on AF tool and their achievement levels. It is seen that regularly receiving formative AF in online learning environments and doing exercises frequently contribute to success in EFL writing at a distance. The correlation between the frequency of attempts and writing scores shows that the learners improve their writing competence as they try. These results show similarities with many studies in the literature (Black and William, 2002; Cotos, 2011; Huijser and Wali, 2018). Although it is not possible to deduce cause-effect relationship with correlation analysis, seeing a positive relationship between these two variables suggests that they tend to act together. With this finding, it is possible to infer that learners improve their writing competence as they experiment by making use of AF. That foreign language learners need tools such as repetitions, affirmations, corrections, and checks to reinforce meaning is supported by studies in the literature (Harrison, 2017; Hyland and Hyland, 2006; Long, 1996; Pica, 1994; Swain, 1995). A moderate positive relationship between the two variables can be considered to be an expected finding. However, what makes this situation special is that learners have benefited from an artificial intelligence-based AF tool, not a teacher, while making all those confirmations, checks and corrections. Findings show that regularly receiving formative feedback from an AF tool in online learning environments and doing exercises frequently contribute to distance EFL writing achievement.

The themes obtained as a result of the analysis of the interviews indicate that distance learners have mostly positive opinions about the AF tool. Participants focused on different themes such as effective learning process, contribution to foreign language development, motivation, effective and fast feedback, user-friendly learning environment and contribution to distance learning as the strengths of AF. Learners stated that they were satisfied with the writing activities they carried out on AF tool and this process contributed to the development of their foreign language writing competence. It is clear that AF enables distance learners to achieve success in their learning process, which is a similar finding in AF literature (Huijser and Wali, 2018; Fang, 2010; Roscoe et al., 2014). The majority of the learners who participated in Fang’s (2010) study emphasized that they benefited from AF in developing their writing competence and recommended the use AF in the future. Likewise, 85% of the participants in the study of Elliot and Mikulua (2004) believed the AF tool made accurate evaluations, and it was useful not only for improving writing competence but also other language skills. The participants in this study also stated that the tool helped them learn vocabulary, comprehend grammar structures, and become better readers. Similarly, in the study by Cotos (2011), the participants suggested that there was an improvement in the quality of the discourse as they made changes on the content, vocabulary, grammar, structure and mechanics of the article.

The views of the participants who experienced Write and Improve in the study of Wali and Huijser (2018), are largely overlapping with those of in this study. The majority of the learners found AF very useful and emphasized that it contributed to their development not only in writing but also in grammar and vocabulary. Learners emphasized that they strive to reduce their mistakes more and more every time with the quick feedback they receive from the tool. This motivation can be attributed to the attractiveness of AF. The fact that learners can consult to AF whenever they want and get a fast response make such tools attractive in the learning process. Wang and Wang’s (2012) study revealed similar results and suggested that learners can be encouraged to practice writing continuously and learners can recognize their strengths and weaknesses more easily thanks to the instant feedback provided with AF.

Participants claimed that AF contributed to open and distance learning process by emphasizing that need for a classroom environment and a teacher disappeared as they could receive feedback without time, place, and amount limitations. According to Moore and Kearsley (1996), in an open and distance learning context, most students seek immediate feedback, and few are satisfied with one-way communication going on without feedback. In this context, the AF used in the study has shown that it can meet the urgent feedback needs of distance English learners. Participants also stated that they constantly made attempts in order to advance on the individual progress graph provided by the tool. This proves that not only feedback for foreign language forms, mechanical errors, spelling errors, but also feedback for progress increases motivation. As participants had the opportunity to see their previous mistakes to correct them, and as the errors in their texts decreased, they felt more willing to experiment. It is anticipated that the foreign language learning process can be supported positively with the formative AF in an open and distance learning environment. Thanks to the interconnected nature of learning, feedback loops with repetitive trials increased the performance level of learners. AF not only encouraged writing more, but also enabled learners to focus on improving certain aspects of their performance, such as vocabulary, punctuation, and discourse. With this study, it was revealed that when distance learners benefit from AF for a certain period of time to revise their drafts, this eventually contributes to development of writing competence, and there is an increase in writing achievement, a finding which is in line with the literature (Attali and Burstein, 2006; Fang, 2010; Lee et al., 2011). Especially in the context of open and distance learning, considering that adult learners have a wide variety of learner profiles and need study environments that allow self-directed learning by planning their own learning, this study has proven the potential of AF for distance language learning.

The majority of the problems encountered were that the foreign language proficiency levels of the participants did not match the language level of the instructions, contents and feedback provided by the tool. Participants highlighted that they could not benefit from the AF tool because it had English content and it did not provide native language support, and their proficiency levels were not high enough to interpret the comments. Participants emphasized that they especially did not understand indirect feedback, and they could not receive adequate guidance because they did not understand the content in the guidance presented in suggestion boxes. Participants admitted that due to these problems, after a while, they felt despaired and gave up trying. It can be concluded that in the context of open and distance learning, especially learners with low proficiency levels may need teacher guidance in the use of such tools. Similarly, in the interviews conducted in the study of Sherafati, Largani, and Amini (2020), learners emphasized that although the feedback they received from the computer was useful, AF could be valuable when used in the presence of the teacher as a support of the teacher feedback. Another solution suggested by learners of this study is that interface and feedback content could be provided in the native language as an option.

Implications and recommendations

AF tools can provide flexibility to meet individual learning needs in heterogeneous distance foreign language classes with level, learning speed, time and space flexibility, which can respond to a wide variety of learner profiles. Mistakes frequently repeated by the learner can be detected by the tool, and additional exercises can be provided for areas of weakness. AF tools can also be presented with diagnostic tests. Learners can start receiving feedback suggestions by performing writing assignments according to their levels. In order to enable learners with lower proficiency level to catch up with the level of their peers, AF tools can be offered as individual study tools in the form of additional exercises and assignments. Features like e-portfolio, progress chart, gradebook in AF tools can be used for progress tracking in the context of open and distance learning. With these tools, an individualized general performance evaluation can be presented to the learner, which includes a description associated with learner’s previous performance.

Limitations

This study is limited to the data obtained from 29 learners who are continuing or graduated from various programs in an Open Education Faculty and volunteered to participate in the study. This research is focused on individual academic development. Therefore, no group success comparison was made. Class or community success rates are beyond the scope of this study. Therefore, generalizations on the findings cannot be made.

Footnotes

Author Note

This study was based on a Ph.D. thesis titled “Investigation of the contribution of automated feedback to the improvement of distance foreign language learners’ writing activities” at Graduate School of Social Sciences, Anadolu University, Eskisehir, Turkey in 2021.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethical statement

All learners participated in the study voluntarily and a consent form was shared with them. There is no conflict of interest about this study. A permission was taken from the Anadolu University Ethical Committee (Protocol Number:3852, Date, 29.01.2020)

ORCID iD

Ayse Taskiran

Author Biographies

Ayse Taskiran is a lecturer at College of Liberal Arts School of English Studies, Wenzhou-Kean University. Taskiran gained her BA in English Language Teaching (ELT) in 1999 at Middle East Technical University. In 2010, she completed her MA in ELT at the Institute of Educational Sciences, Anadolu University. Taskiran completed her Ph.D. in Distance Education Department at the Institute of Social Sciences at Anadolu University. Her academic interests include automated feedback systems for foreign language teaching and learning, artificial intelligence, e-learning, educational networks, online-immersive learning, mobile learning, achievement motivation, augmented reality, gamification, interactive course materials, and instructional technologies.

Mujgan Yazici works as a full Professor at the Open Education Faculty, Anadolu University, Turkey. Her academic interest areas are effective communication, communication theories, new technologies in an open and distance learning, e learning, life-long learning, adult learning, material design for individual learners. She has been teaching mass communication theories and material design courses at the Ph.D levels. She has been involved both as coordinator and researcher in several national and international projects. She has several journal articles published in international indexes and international book chapters, as well as some national and international articles submitted to international meetings.

Irem Erdem Aydin, completed her doctoral degree in Communication Sciences Program of Anadolu University after earing her masters’ degree on distance education. She is a researcher and faculty member in the Department of Distance Education, Open Education Faculty at Anadolu University, Turkey. She worked as a instructional designer in Open Education Faculty of Anadolu University for more than fifteen years, responsible for design and production of textbooks for distance learners. She is also served as the Vice Director of the Distance Education Department and the Assistant-Coordinator of the Instructional Design Team in Open Education Faculty. Her primary research interests include distance learning and teaching, online learning, instructional design/technology, teaching and learning in online Technologies.

References

Alger

Kopcha

(2009) eSupervision: a technology framework for the 21st century field experience in teacher education. Issues in Teacher Education 18(2): 31–46.

Alharbi

(2016) Effect of teachers’ written corrective feedback on Saudi EFL university students’ writing achievements. International Journal of Linguistics, 8(5): 15–29.

Alvarez

Ananda

Walqui

, et al. (2014) Focusing Formative Assessment on the Needs of English Language Learners. San Francisco, CA: WestEd.

Al Emran

Shaalan

(2014) A survey of intelligent language tutoring systems. In: 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Delhi, India, 24–27 September 2014. IEEE, pp. 393–399.

Anderson

(2003) Getting the mix right again: an updated and theoretical rationale for interaction. IRRODL 4(2): 9–14.

Attali

Burstein

(2006) Automated essay scoring with e-rater® V. 2. The Journal of Technology, Learning and Assessment 4(3): 1–30.

Black

William

(2002) Improved Standards Achieved by Transforming Assessment for Learning. Kings College London: News Archive.

Boling

Hough

Krinsky

, et al. (2012) Cutting the distance in distance education: perspectives on what promotes positive, online learning experiences. The Internet and Higher Education 15(2): 118–126.

Bonnel

(2008) Improving feedback to students in online courses. Nursing Education Perspectives 29(5): 290–294.

10.

Boud

Molloy

(2013) Rethinking models of feedback for learning: the challenge of design. Assessment & Evaluation in Higher Education 38(6): 698–712.

11.

Caccamise

Franzke

Eckhoff

, et al. (2007) Guided practice in technology-based summary writing. In: Reading Comprehension Strategies: Theories, Interventions, and Technologies. McNamara

(ed) Mahwah, NJ: Erlbaum, pp. 375–396.

12.

Calvo

O’Rourke

Jones

, et al. (2010) Collaborative writing support tools on the cloud. IEEE Transactions on Learning Technologies 4(1): 88–97.

13.

Cambridge English (2021). https://www.cambridgeenglish.org/learning-english/free-resources/write-and-improve/.

14.

Cheng

Law

Wong

(2016, December) Investigating effects of automated feedback on EFL students' reflective learning skills. In 2016 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE) (pp. 226–230). IEEE.

15.

Cheng

(2017) The impact of online automated feedback on students' reflective journal writing in an EFL course. The Internet and Higher Education, 34: 18–27.

16.

Chapelle

(1998) Multimedia CALL: lessons to be learned from research on instructed SLA. Language Learning &Technology 2(1): 21–39.

17.

Chapelle

(2003) English Language Learning and Technology: Lectures on Applied Linguistics in the Age of Information and Communication Technology. Philadelphia: John Benjamins. DOI: 10.1075/lllt.7.

18.

Chukharev-Hudilainen

Saricaoglu

(2016) Causal discourse analyzer: improving automated feedback on academic ESL writing. Computer Assisted Language Learning 29(3): 494–516.

19.

Cotos

(2011) Potential of automated writing evaluation feedback. Calico Journal 28(2): 420–459.

(2020) Understanding graduate writers’ interaction with and impact of the research writing tutor during revision. Journal of Writing Research 12(1): 187–232.

21.

Creswell

(2014) A Concise Introduction to Mixed Methods Research. Thousand Oaks, CA: Sage Publications.

22.

Curry

Riordan

(2021) Intelligent CALL systems for writing development: investigating the use of write & improve for developing written language and writing skills. In: CALL Theory Applications for Online TESOL Education. IGI Global, pp. 252–273.

, et al. (2015) How can automatic feedback help students construct automata? ACM Transactions on Computer-Human Interaction (TOCHI) 22(2): 9–24.

24.

Elliot

Mikulua

(2004) The impact of MyAccess! Use on student writing performance: a technology overview and four studies. In: Annual Meeting of the American Educational Research Association. San Diego, CA.

25.

Ellis

(2000) Learning a Second Language through Interaction. Amsterdam: John Benjamins.

26.

Fang

(2010) Perceptions of the computer-assisted writing program among EFL college learners. Educational Technology & Society 13(3): 246–256.

27.

Flick

(2018) Why triangulation and mixed methods in qualitative research? In: Qualitative Research Kit: Doing Triangulation and Mixed Methods. Flick

(ed) London, UK: SAGE Publications Ltd, pp. 1–10.

28.

Glaser

Strauss

(1967) The Discovery of Grounded Theory: Strategies for Qualitative Research. New York: Aldine.

, et al. (2008) The sentence fairy: a natural-language generation system to support children's essay writing. Computer Assisted Language Learning 21(4): 339–352.

30.

Harrison

(2017) Developing an ELT product based on machine learning: Write & Improve. Available at: https://eltjam.com/developing-an-elt-product-based-on-machine-learning-write-improve/

31.

Hattie

Timperley

(2007) The power of feedback. Review of Educational Research 77(1): 81–112.

32.

Huang

Renandya

(2020) Exploring the integration of automated feedback among lower-proficiency EFL learners. Innovation in Language Learning and Teaching, 14(1): 15–26.

33.

Huijser

Wali

(2018) Write to improve: exploring the impact of an online feedback tool on Bahraini learners of English. Learning and Teaching in Higher Education: Gulf Perspectives 15(1): 1–21.

34.

Hyland

(2006) Feedback on second language students’ writing. Language Teaching 39(2): 83–101.

35.

Karpova

(2020) Integration of “Write and Improve” awe tool into EFL at higher educational establishment: case study. Celtic: A Journal of Culture, English Language Teaching, Literature and Linguistics 7(2): 137–150.

36.

Kellogg

Whiteford

Quinlan

(2010) Does automated feedback help students to write? Journal of Educational Computing Research 42: 173–196.

37.

Kostikova

Miasoiedova

(2019) Supporting post-graduate students writing skills development with the online machine learning tool: write & improve. Information Technologies and Learning Tools 74(6): 238–249.

38.

Krashen

(1985) The Input Hypothesis: Issues and Implications. New York: Addison-Wesley Longman Ltd.

39.

Lee

Noh

Lee

, et al. (2011) On the effectiveness of robot-assisted language learning. ReCALL 23(1): 25–58.

40.

Link

Dursun

Karakaya

, et al. (2014) Towards better ESL practices for implementing automated writing evaluation. Calico Journal 31(3): 323–344.

41.

Long

(1983) Native speaker/non-native speaker conversation and the negotiation of comprehensible input. Applied Linguistics 4: 126–141.

42.

Long

(1996) The role of the linguistic environment in second language acquisition. In: Ritchie

Bhatia

(eds) Handbook of Language Acquisition: Vol. 2. Second language acquisition (pp. 413–468). San Diego, CA: Academic Press.

43.

Marriott

Teoh

(2014) Computer-based Assessment and Feedback: Best Practice Guidelines. York: Higher Education Academy UK.

44.

Moore

(1989) Three types of interaction. The American Journal of Distance Education 3(2): 1–6.

45.

Moore

Anderson

(2007) Handbook of Distance Education. Mahwah, NJ: L.

46.

Moore

Kearsley

(1996) Distance Education: A Systems Review. Belmont: Wadsworth Publishing Company.

47.

Nicol

(2010) From monologue to dialogue: improving written feedback processes in mass higher education. Assessment & Evaluation in Higher Education 35(5): 501–517.

48.

Pica

(1994) Research on negotiation: what does it reveal about second-language learning conditions, processes, and outcomes? Language Learning 44(3): 493–527.

49.

Prince

Makrides

Richman

(1980) Research methodology and applied statistics. Part 2: the literature search. Physiotherapy Canada. Physiotherapie Canada, 32(4): 201–206.

50.

Purnawarman

(2011) Impacts of different types of teacher corrective feedback in reducing grammatical errors on ESL/EFL students' writing (Doctoral dissertation, Virginia Tech).

51.

Robinson

Mackey

Gass

, et al. (2012) Attention and awareness in second language acquisition. In: The Routledge Handbook of Second Language Acquisition. Gass

Mackey

(eds) New York: Routledge, pp. 247–267.

52.

Roscoe

Allen

Weston

, et al. (2014) The Writing Pal intelligent tutoring system: usability testing and development. Computers and Composition 34: 39–59.

53.

Rudzewitz

Ziai

De Kuthy

, et al. (2018) Generating feedback for english foreign language exercises. In: Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, New Orleans, LA: Association for Computational Linguistics, pp. 127–136.

54.

Sherafati

Largani

Amini

(2020) Exploring the effect of computer-mediated teacher feedback on the writing achievement of Iranian EFL learners: Does motivation count? Education and Information Technologies, 25(5), 4591–4613.

55.

Shermis

Garvan

Diao

(2008) The impact of automated essay scoring on writing outcomes. In: The Annual Meetings of the National Council on Measurement in Education, New York, NY, 25–27 March, 2008.

56.

Simonson

Smaldino

Albright

, et al. (2000) Assessment for distance education (ch 11). In: Teaching and Learning at a Distance: Foundations of Distance Education. Upper Saddle River, NJ: Prentice-Hall.

57.

Simpson

(2006) Predicting student success in open and distance learning. Open Learning: The Journal of Open, Distance and E-Learning 21(2): 125–138.

58.

Swain

(1995) Three functions of output in second language learning. In: Principle and Practice in Applied Linguistics. Cook

Seidlhofer

(eds) Oxford: Oxford University Press, pp. 125–144.

59.

Wade-Stein

Kintsch

(2004) Summary street: interactive computer support for writ- ing. Cognition and Instruction 22: 333–362.

60.

Ware

Warschauer

(2006) Electronic feedback and second language writing. In: Feedback in Second Language Writing: Contexts and Issues. Cambridge: Cambridge University Press, pp. 105–122.

61.

Warschauer

Grimes

(2008) Automated writing assessment in the classroom. Pedagogies: An International Journal 3(1): 22–36.

62.

Wilson

Olinghouse

Andrada

(2014) Does automated feedback improve writing quality? Learning Disabilities: A Contemporary Journal 12(1): 93–118.

63.

Wang

(2012) A comparative study on the influence of automated evaluation system and teacher grading on students’ English writing. Procedia Engineering 29: 993–997.

Contribution of automated feedback to the English writing competence of distance foreign language learners

Abstract

Keywords

Introduction

Interaction and formative feedback in foreign language learning

Interaction and formative feedback in open and distance learning

Automated feedback (AF) in foreign language writing

Write and improve

Method

Participants

Procedure

Data analysis

Findings

Contribution of AF to writing achievement

Qualitative findings

Discussion and conclusion

Implications and recommendations

Limitations

Footnotes

Author Note

Declaration of Conflicting Interests

Funding

Ethical statement

ORCID iD

Author Biographies

References