Abstract
This study examined Chinese EFL researchers’ English abstract writing in language education. Using open-ended questionnaires, it first investigated 24 Chinese EFL researchers’ perceptions of their challenges in writing English abstracts. Using generalizability theory and follow-up interviews, it then invited 16 experienced English journal reviewers to assess 27 published English abstracts written by Chinese EFL researchers to identify common errors and suggest improvement tips. Finally, it examined eight selected Chinese EFL researchers’ critical reflections on the English journal reviewers’ assessment of the 27 English abstracts. The results indicated that Chinese EFL researchers experienced several challenges in writing English abstracts. Further, the English reviewers’ assessment of the 27 abstracts was consistent and reliable. The common errors were associated with the accuracy, non-evaluative nature, coherence and readability, and conciseness of an English abstract. The English reviewers provided tips for improvement, and Chinese EFL researchers expressed their critical reflections on their assessment. Educational implications are discussed.
Keywords
Introduction
An abstract is a brief and comprehensive summary of a proposal or a completed article (American Psychological Association [APA], 2020; Fowler, 2011; Goldborl, 2002; Heseltine, 2012; Price, 2014; Swales & Feak, 2009). It provides the reviewers with a summary of the proposal or the article to make acceptance decisions (Happell, 2007). It gives “synthesized, focused, and succinctly written information” about proposed and completed work (Pearce & Ferguson, 2017, p. 453). Although their depth and breadth vary among conferences, grant organizations, and scholarly journals, they are essential for conference presentations, grant proposals, and journal article submissions (Happell, 2007; Pearce & Ferguson, 2017; Price, 2014; Swales & Feak, 2009). More specifically, a published journal article abstract communicates the significant components of the article (APA, 2020; Swales & Feak, 2009). It is “an advertisement” for the article (Sheldon & Jackson, 1999, p. 78). Moreover, it provides the readers with their first exposure to the research article (Hyland, 2004; Swales & Feak, 2009). It is “often the point at which they decide whether to continue and give the accompanying article further attention or to ignore it” (Hyland, 2004, p. 63). The abstract is also a “window to the world” on the article (Heseltine, 2012, p. 204). It helps the authors maximize the exposure of their research through journal article publications (Hyland, 2004; Swales & Feak, 2009).
It is customary that articles published in non-English journals require accompanying English abstracts to establish the international visibility of the research, the journals, as well as the authors (Burgoine et al., 2011; Cilveti & Pérez, 2006; Klimova, 2013; Linder, 2014; Lorés-Sanz, 2016). Undoubtedly, these English abstracts play a significant role in establishing the international visibility (Burgoine et al., 2011; Friginal & Mustafa, 2017; Hosseingholipour et al., 2021; Hyland, 2002; Linder, 2014). Nowadays, more and more Chinese EFL (English as a foreign language) researchers publish research articles in English journals hosted in English speaking countries such as the United States and Great Britain (Huang et al., 2021; Li & Flowerdew, 2007). Due to their non-English background, they face challenges composing high-quality English abstracts, which will eventually prevent them from publishing in English journals and establishing the international visibility (Hyland, 2004; Hyland & Shaw, 2016; Hu & Cao, 2011; Li & Flowerdew, 2007; Lu & Deng, 2019; Ruan, 2018; Ye & Wang, 2013).
However, there is little research assessing the quality of English abstracts written by Chinese EFL researchers within the frameworks of both modern assessment theories, for example, the generalizability (G-) theory, and discourse analysis, for example, the four-move introduction-methods-results-discussion (IMRD) model. This study aimed to bridge this gap. The results will have important implications for Chinese researchers and English journal editors and reviewers in the international academia.
A Brief Narrative Literature Review
Informative and structured abstracts are the two major types of journal article abstracts (Cals & Kotz, 2013; Hartley, 2003; Mosteller et al., 2004). Informative abstracts are very common in library journals, generally one to two paragraphs (between 100 to 200 words) in length; whereas structured abstracts are very common in science, technology, and medical journals but less common in library journals (Atanassova et al., 2016; Pearce & Ferguson, 2017). Structured abstracts typically include research purpose, research design or methodology, results and findings, limitations, practical implications, and originality/value (Pearce & Ferguson, 2017; Price, 2014).
This brief narrative literature review section (Feak & Swales, 2009) reviewed relevant past research in humanities and social sciences in the following three categories and synthesized it into a coherent discussion: a) studies that investigated the quality of abstracts and identified common errors committed by EFL researchers in abstract writing, b) studies that examined differences in genre of abstracts across English and other languages including Chinese, and c) studies that offered tips for teaching writing a good abstract.
Common Errors Committed by EFL Researchers in Abstract Writing
Several studies examined the quality of abstracts and identified the common errors committed by EFL researchers in humanities and social sciences (Hosseingholipour et al., 2021; Klimova, 2013; Linder, 2014; Lorés-Sanz, 2016). For example, Klimova (2013) examined the quality of 66 English abstracts to identify the common errors committed by the undergraduate EFL students majoring in management of tourism at a university in the Czech Republic. The results indicated that these abstracts were poor in quality; students simply translated their abstracts from their native language into English. The identified errors in their English abstracts included a) structures that were not logical and clear; b) abstracts exceeded the word limit; and c) linguistic-stylistic errors such as word order, articles, subject and predicate agreement, prepositions, spelling, punctuation, and capitalization errors. However, these 66 abstracts were written by undergraduate EFL students and they were not officially published in scholarly journals, which may limit the generalization of the findings to EFL researchers in the field.
Unlike Klimova (2013), Linder (2014) investigated the quality of 197 published English abstracts selected from Spain’s ten most prestigious open-access translation studies journals. Although the overall quality of these abstracts was much better than what Klimova (2013) had reported, the results indicated that 73 English abstracts contained 128 grammatical, vocabulary-related, and typographical errors.
Unlike Klimova (2013) and Linder (2014) who investigated the quality of actual abstracts written by EFL researchers, Hosseingholipour et al. (2021) examined the perceptions of nine physical education professors and four Ph.D. students at Iranian universities about the factors affecting the quality of Iranian researchers’ English abstracts as well as their common errors in writing English abstracts. The results indicated that problems with the English language negatively affect the quality of Iranian researchers’ English abstracts; and English language errors were commonly committed by them. Their errors included incorrect word choices and spellings, ungrammatical sentences, wrong rhetorical moves, and poor abstract organization.
Different from Klimova (2013), Linder (2014), and Hosseingholipour et al. (2021), who examined the quality of English abstracts written by EFL researchers only from a single non-English speaking country, Lorés-Sanz (2016) examined the quality of 66 English abstracts written by EFL researchers from 17 non-English speaking countries (e.g., Netherlands, Germany, Israel, Turkey, and Korea). Furthermore, these abstracts were published in the 2012, 2013, and 2014 volumes of Social Science Research, an American-based and one of the most prestigious journals in the field of sociology. Specifically, the researcher analyzed the rhetorical moves in these abstracts and found that they did not strictly follow the conventional four-move introduction-methods-results-discussion (IMRD) rhetorical structure typical of the English academic world. The number of moves was reduced in their English abstracts. Furthermore, their abstracts showed “an inclination towards rhetorical simplification, while maintaining the convention of the discipline to include aims, methods, and results” (pp. 67-68). However, the simplification of the conventional rhetorical structure, as argued by the research, shows a higher degree of textual complexity.
These studies suggest that there is much room for EFL researchers, student researchers in particular, to improve their English abstracts. Their common errors may prevent readers from understanding the research problems under investigation and the significant research findings. As a result, these errors would negatively impact the international dissemination and visibility of their research studies (Hosseingholipour et al., 2021; Klimova, 2013; Linder, 2014; Lorés-Sanz, 2016).
Differences in Genre of Abstracts Across English and Other Languages Including Chinese
A few studies compared genre of abstracts in humanities and social sciences and identified differences across English and other languages including Chinese (Behnam & Golpour, 2014; Friginal & Mustafa, 2017; Hu & Cao, 2011; Ozdemira & Longo, 2014; Ruan, 2018). For example, Behnam and Golpour (2014) examined the move structure differences of published research article abstracts written by native English and Iranian researchers in applied linguistics. They reported that English abstracts included all conventional moves, whereas Persian abstracts had more move omissions. Later, Friginal and Mustafa (2017) found differences in terms of how information is formed and shared and the way directness and argumentation authors articulate by examining the linguistic characteristics of English research article abstracts in linguistics, applied linguistics, and English education, which were published in the United States and Iraq and written by Iraqi EFL researchers.
Unlike Behnam and Golpour (2014) and Friginal and Mustafa (2017) who examined research article abstracts differences between native English and Iranian and Iraqi EFL researchers, Ozdemira and Longo (2014) examined graduate thesis abstracts differences in metadiscourse features between American and Turkish students. They reported that American students used more “evidential, endophorics, code glosses, boosters, attitude markers, and self-mentions” (p. 59) than Turkish students; however, American students used less metadiscourse transitions, frame markers, and hedges than Turkish students in their thesis abstracts, suggesting that Turkish EFL students may have challenges in writing good thesis abstracts which can follow the logic of arguments.
A couple of studies examined the English abstract differences in metadiscourse and linguistic features between native English and Chinese EFL researchers in applied linguistics (Hu & Cao, 2011; Ruan, 2018). Both studies reported significant differences across English and Chinese. For example, Hu and Cao (2011) analyzed 649 abstracts selected from eight journals in applied linguistics to examine whether hedging and boosting strategies differ between Chinese and English medium journals. The results indicated that abstracts published in English medium journals used more hedges than abstracts published in Chinese medium journals. Interestingly, by analyzing a corpus of 200 abstracts in four applied linguistic journals, Ruan (2018) reported that native English researchers used more simple noun phrases, whereas Chinese EFL researchers used more complex noun phrases in research article abstracts.
Tips for Teaching Writing a Good Abstract
Several researchers in the field of humanities and social sciences offered tips for teaching writing a good abstract (Friginal & Mustafa, 2017; Hyland, 2003, 2007; Klimova, 2013; Ruan, 2018; Stotesbury, 2003; Swales & Feak, 2009).
Hyland (2003, 2007) suggested genre-based pedagogies for EFL academic writing teachers to assist their students to produce effective texts and abstracts. Swales and Feak (2009) defined genre as “a name for a type of text or discourse designed to achieve a set of communicative purposes” (p. 1). Following this definition, the research article abstract is a genre (Swales, 2004). Genre-based pedagogies require the teacher to create a contextual framework of writing the abstract, help students understand the structure of an abstract, and then assist them toward a command of writing a good abstract (Hyland, 2003, 2007).
Moreover, Stotesbury (2003) suggested that EFL academic writing teachers teach students the conventions of abstract writing in their own fields. Specifically, they should be analyzing abstracts in scholarly journals of their own disciplines. Students should have concrete examples to learn to write good abstracts. These suggestions were further explained by Friginal and Mustafa (2017) who argued that successful academic writing and production of high-quality research article abstracts “can be addressed by exploring data and developing teaching materials from data-driven measures. Cross-cultural and cross-disciplinary comparisons using a specialized corpus appear to be an effective starting point in understanding existing discourse patterns” in teaching writing a good English abstract (p. 55).
In addition, Swales and Feak (2009) suggested a five rhetorical moves model be followed in teaching writing a good abstract, that is, Move 1—background/introduction/situation, Move 2—present research/purpose, Move 3—methods/materials/subjects/procedures, Move 4—results/findings, and Move 5—discussion/conclusions/implications/recommendations. It was identified by most researchers in various fields and languages and found effective in teaching writing a good abstract.
Finally, since it is common that EFL student researchers write their abstracts in their first language and then translate them into English, Klimova (2013) suggested that translation be avoided in composing abstracts due to linguistic differences between English and other languages. EFL academic writing teachers should encourage and assist their students to write abstracts in English (Klimova, 2013). They should also be taught to use compact grammatical features but still write with clarity of meaning to achieve their rhetorical and pragmatic goals in writing good English abstracts (Ruan, 2018).
To conclude, although several studies investigated the common errors committed by EFL researchers in humanities and social sciences and further identified the differences in genre of abstracts across English and other languages including Chinese, the assessment of the quality of English abstracts written by Chinese EFL researchers in humanities within the G-theory and IMRD theoretical frameworks is still under-researched. Therefore, this research aimed to a) examine Chinese EFL researchers’ challenges in writing journal article abstracts in English; b) investigate English journal reviewers’ assessment and evaluation of the quality of published English abstracts written by Chinese EFL researchers; c) provide Chinese EFL researchers with improvement tips; and d) explore Chinese EFL researchers’ critical reflections on English journal reviewers’ assessment and evaluation.
Theoretical Frameworks
G-theory (Cronbach, Gleser, Nanda, & Rajaratnam, 1972) was used as a theoretical framework guiding the assessment of the quality of English abstracts written by Chinese EFL researchers. It is a modern test theory and powerful in identifying the sources of assessment score variance and error and then estimating the impact of these variance components on the score reliability (Brennan, 2001; Shavelson et al., 1993; Shavelson & Webb, 1991). It is commonly used by educational researchers to examine assessment variability and reliability (Huang et al., 2021; Li & Huang, 2022).
In addition, the four-move IMRD model in discourse analysis was adopted as the theoretical framework for the development of research instruments for data collection and data analysis of this study (Bhatia, 2002; Swales, 1990; Swales & Feak, 2009). A research article abstract usually contains the following four moves: Move 1—introduction (e.g., why is the research problem or topic important? what is the current study about?); Move 2—methods (e.g., how was the study conducted?); Move 3—results (e.g., what was discovered?); and Move 4—discussion (e.g., what are the conclusions? What are the implications?) (Bhatia, 2002; Swales & Feak, 2009).
Finally, the APA (2010) abstract standards were also used to guide the development of research instruments for data collection and the analysis of collected data of this study. According to the APA (2010) Publication Manual Sixth Edition, a good abstract is
It is important to mention that the data collection of this study had been completed before the APA (2020) Publication Manual Seventh Edition was released. The qualities of a good abstract in the seventh edition are still the same, that is, being
Research Questions
The following five research questions guided this study:
a) What are Chinese EFL researchers’ major challenges in writing journal article abstracts in English?
b) What is the quality of English journal reviewers’ quantitative assessment of the published English abstracts written by Chinese EFL researchers?
c) What are the common errors in Chinese EFL researchers’ English abstracts identified by the English journal reviewers?
d) What are the English journal reviewers’ tips for Chinese EFL researchers to improve their abstract writing in English? And
e) What are Chinese EFL researchers’ critical reflections on the English journal reviewers’ quantitative and qualitative assessments of the published English abstracts written by Chinese EFL researchers?
Methods
The Research Design
This cross-sectional study was conducted in four phases. Phase One used open-ended questionnaires to investigate Chinese EFL researchers’ challenges in writing journal article abstracts in English.
Phase Two involved experienced English journal reviewers’ quantitative and qualitative assessments of Chinese EFL researchers’ English abstracts published in top-tier Chinese journals in the field of language education. For the quantitative assessment proportion, G-theory (Cronbach et al., 1972) was adopted as a methodological framework because it can analyze more than one measurement facet simultaneously in investigations of score variability and dependability assigned by assessors and evaluators (i.e., the experienced English journal reviewers in this study) (Brennan, 2001; Huang, 2012; Liu & Huang, 2020; Zhao & Huang, 2020). For the qualitative assessment proportion, English journal reviewers’ written comments on various aspects of each abstract were examined so that the common errors that occurred in these abstracts could be identified.
Phase Three included follow-up semi-structured interviews with selected English journal reviewers. These interviews aimed to elicit their suggestions for Chinese EFL researchers to improve abstract writing in English.
Phase Four included follow-up semi-structured interviews with selected Chinese EFL researchers who participated in Phase One of the study. These interviews aimed to obtain Chinese EFL researchers’ critical reflections on the English journal reviewers’ quantitative and qualitative assessments of the published English abstracts written by Chinese EFL researchers.
The Participants
The purposive sampling method was adopted and purposive samples of 24, 16, 8, and 8 were selected for Phases One, Two, Three, and Four of this study, respectively (Creswell, 2014). Phrase One participants included 24 EFL researchers from different 4-year universities across China. The criteria for selecting Phase One participants were a) they must hold a master’s degree; b) they must have more than 3 years of research experience; and c) they must have published at least three research articles in either English or non-English journals.
Among the 24 participants from China, 12 were male and 12 were female researchers; 13 obtained a doctoral degree, and 11 had a master’s degree. Eight of the 11 participants with a master’s degree were currently studying for their doctoral degrees in educational fields at English-medium universities. It was important to note that all 24 participants had more than 3 years of research experience in education, and 14 of them had published in English journals.
Phrase Two participants included 16 experienced English journal reviewers. The criteria for selecting these reviewer participants were a) they must hold a doctoral degree; b) they must have served as reviewers of English journals for at least 3 years; and c) they must have published at least five articles in English journals. Among the 16 reviewers, 7 were male, and 9 were female; and all of them have published more than eight articles in English journals. Furthermore, 11 of them were from the United States, 3 from Canada, 1 from Commonwealth of the Bahamas, and 1 from Hong Kong. Their areas of expertise were humanities and education. The corresponding author of this article was responsible for recruiting them because he had worked as an education professor at an American university and served as a reviewer of several English journals before he relocated to China.
Phrase Three participants included eight selected English journal reviewers who participated in Phase Two of the study. After performing quantitative and qualitative assessments of the published English abstracts written by Chinese EFL researchers, they were further purposefully selected for follow-up interviews about their tips for Chinese EFL researchers to improve their abstract writing in English. Among the eight interviewees, four were male and four were female; and all of them had served as English journal reviewers for over 6 years.
Phrase Four participants included eight selected EFL researchers who had participated in Phase One of the study. They were first presented the findings of Phases Two and Three results and then interviewed about their critical reflections on these results. Among the eight interviewees, four were male and four were female; five had obtained a doctoral degree, and three were studying for their doctoral degrees at English-medium universities.
The Selection of English Abstracts Published in Chinese Journals
A total of 27 English informative abstracts published in three top-tier language education journals (i.e., A, B, and C) in China across 3 years (i.e., 2014, 2015, and 2016) were selected for Phases Two and Three of this study. The three top-tier language education journals were selected by citations. Three abstracts in each year of each journal were selected through random purposive sampling for inclusion in this study. Journals A, B, and C are top-tier peer-reviewed journals that mainly publish research articles in Chinese in the areas of language research, linguistics, applied linguistics, foreign language education, and translation studies. The 27 abstracts were selected from the two areas of applied linguistics and foreign language education. Regardless of the language in which the article is written, an English abstract is required and the author(s) must be responsible for it. The purpose of the English abstract is for the journal and author(s) to establish the international visibility. For example, among Journals A, B, and C, the article titles and abstracts published in Journal B have been included in the database of the Modern Language Association of America.
The Instruments
The open-ended questionnaire questions (see Appendix A) for Phase One of this study focused on participants’ significant challenges in writing journal article abstracts in English. The instrument for Phase Two was constructed in three steps: a) the researchers of this study consulted with three English journal editors in the field of language education about their advice on the assessment criteria for English abstracts; b) they further studied the abstract criteria outlined by APA (2010) and then constructed a Journal Article Abstracts Assessment Form with a 6-point holistic scoring rubric (see Appendix B); and c) the instrument was finally reviewed and slightly altered by the three English journal editors before the data collection. These three steps ensured that the final instrument for Phase Two was accurate and valid. Furthermore, it has also been reliable, with a Cronbach’s alpha reliability coefficient of .88, that is, the internal consistency reliability coefficient, calculated using SPSS and Phase Two holistic assessment scores of the 27 abstracts assigned by the 16 English journal reviewers.
In Phase Two, the 16 English journal reviewers were invited to a) use the provided scoring rubric to assign a holistic score to each of the 27 English abstracts published in Journals A, B, and C on a six-point scale, and b) to provide written comments on various aspects of each abstract (see Appendix B). Follow-up interview questions (see Appendix C) for Phase Three of this study focused on the selected English journal reviewers’ suggestions for Chinese EFL researchers to improve their abstract writing in English. Follow-up interview questions (see Appendix D) for Phase Four of this study focused on the selected Chinese EFL researchers’ critical reflections on the significant findings of Phases Two and Three of the study.
Data Collection Procedures
Phase One data were collected through emails. Email invitations were sent out to invite participants. The researchers provided the invited participants with information about the study and they all understood that their participation was voluntary. The completed responses to the open-ended questionnaire questions were sent to the researchers through emails for data analysis. It is important to note that the open-ended questions were asked in English, and the participants also answered these questions in English.
Phase Two data were collected through emails with the 16 experienced English journal reviewers. After they had expressed their willingness to participate in the assessments of the 27 English abstracts, the Journal Article Abstracts Assessment Form was emailed to them and their completed rating forms were emailed back to the researchers.
Due to time and location inconveniences, Phases Three and Four data were collected through follow-up online interviews with eight purposefully selected Phase Two English journal reviewers and Phase One Chinese EFL researchers, respectively. The follow-up online interviews were conducted in English and between the researchers and the invited interviewees. Again, in Phases Two through Four, participants were provided with information about the study and their participation was voluntary.
Data Analysis
All qualitative data collected in four phases were analyzed as follows. The researchers first entered qualitative data into Excel spreadsheets to ensure data consistency and integrity. They then color-coded the responses under each open-ended interview question. Following that, they began to sort them into different categories and subcategories individually, then organized them collaboratively by content, and finally, discussed conceptually similar responses, grouped them together, and then categorized them by the recurring themes. This process aimed to ensure inter-coder reliability of the qualitative data analysis. Also, direct quotes from the participants were incorporated to enhance the validity of the results (Creswell, 2014). It is important to note that both researchers have rich qualitative data analysis experience.
Phase Two quantitative data were analyzed at three levels: a) descriptive statistical analysis to obtain the means and standard deviations of the holistic assessment scores of the 27 abstracts assigned by the 16 English journal reviewers; b) a person nested within journal-by-reviewer (p:j) × r mixed effects G-study; c) a person-by-reviewer (p × r) random effects G-study; and d) a random effects person-by-reviewer (p × R) D-study. The results obtained from these G- and D-studies were used to examine the quality of English journal reviewers’ quantitative evaluation, that is, their assessment variability and reliability, of Chinese EFL researchers’ English abstracts published in the three top-tier Chinese journals. The computer programs Microsoft Excel and GENOVA (Crick & Brennan, 1983) were used for performing these data analyses.
Results
Phase One: Chinese EFL Researchers’ Major Challenges
Phase One open-ended questions asked about the 24 EFL researchers’ major challenges in writing abstracts in English. They all reported that they had experienced challenges using appropriate and idiomatic expressions and coherent and logical structures in English abstracts. Further, they found it challenging to decide what should be included and how to organize such information transparently within the word limit. “I often struggle with writing up all the aspects necessary for a solid abstract in less than 200 words”; “my biggest challenge is to how to use English to describe all the information clearly, concisely, and idiomatically”; “there is so much information in a research article; I do not know how to decide between the important and the unimportant information for an abstract”; “my abstracts are either too simple or too detailed”; and “how much detailed information I should provide in the abstract” were their common responses.
Furthermore, 5 out of 24 participants expressed difficulties in writing good English abstracts because they simply translated the abstracts from Chinese into English. For example, one participant who had successfully published two research articles in English commented that “I usually wrote my abstracts in Chinese and then translated them directly into English; however, due to the language and format differences in scholarly writing, I had experienced tremendous difficulty in producing good English abstracts; and I had to revise them many times before they became acceptable.”
Phase Two: English Journal Reviewers’ Quantitative Assessment of the 27 Published Abstracts
Descriptive statistical analysis was performed prior to the G-theory analyses. The purpose of descriptive statistics was to obtain the means and standard deviations of the holistic assessment scores assigned to the 27 English abstracts by the 16 English journal reviewers. The results are presented in Table 1.
Descriptive Statistical Results of Holistic Assessment Scores by Reviewers.
As shown in Table 1, the mean assessment scores for the nine English abstracts published in Journal A were between 2.19 and 5.38 out of a total score of 6; similarly, the score ranges for the nine English abstracts published in Journals B and C were between 2.25 and 5.12, and 1.5 and 4.69, respectively. Further, only 7 of the 27 abstracts (25.9%) received a score over 4, indicating that the quality of 20 of these English abstracts (74.1%) was inadequate and unacceptable as evaluated by the 16 journal reviewers according to the holistic scoring rubric (see Appendix B).
In addition, also as shown in Table 2, the standard deviations of the holistic scores for all 27 English subtracts were minimal (between .40 and .72), indicating that the assessment score variability of these 27 English abstracts was small. In other words, the 16 English journal reviewers assessed these abstracts written by Chinese EFL researchers considerably consistently.
Results of Variance Components for Mixed Effects (p:j) × r G-Study .
Following the descriptive statistics, a series of G- and D-studies were conducted in Phase Two to further investigate the assessment score variability and reliability of the 27 English abstracts published in three Chinese journals and assigned by the 16 English journal reviewers. The results are displayed in Tables 2 to 4.
Table 2 reports the person nested within journal-by-reviewer (p:j) × r mixed effects G-study results. As shown in Table 2, the person nested within the journal (p:j) was the largest variance component (75.22% of the total variance), indicating that the abstract assessment scores received by Chinese EFL researchers within each journal were the single largest source of variance. In other words, within each journal, Chinese EFL researchers’ abstract assessment scores differed substantially. The residual yielded the second largest variance component (19.28% of the total variance). The residual contains the variability due to the interaction between reviewers, abstracts, persons within journals, and other unexplained systematic and unsystematic sources of error. Reviewer (r) yielded the third largest variance component (5.22% of the total variance), suggesting that these 16 journal reviewers differed slightly from one another in terms of the leniency of rating the 27 English abstracts. The remaining facets in the design were 0 or close to 0. It is important to note that the variance component for journals explained 0% of the total variance, suggesting that there was no difference in abstract writing performances that could be attributed to the three journals. Since the journal facet explained 0% of the total variance, it was not considered as a facet in the following person-by-reviewer (p × r) random effects G-study design whose results are presented in Table 3.
Results of Variance Components for Random Effects p × r G-Study.
As shown in Table 3, the object of measurement, person (p), explained the largest score variance (75.08% of the total variance), suggesting that the 27 Chinese EFL researchers differed considerably in their abstract writing skills. The residual yielded the second largest variance component (19.66% of the total variance). The residual contains the variability due to the interaction between reviewers and persons, and other unexplained systematic and unsystematic sources of error. The reviewer (r) facet was the third largest variance component (5.26% of the total variance), suggesting that the journal reviewers rated these 27 English abstracts slightly differently.
Table 4 reports the random effects person-by-reviewer (p × R) D-study results. As shown in Table 4, the G-coefficient for the norm-referenced and Phi-coefficient for criterion-referenced interpretations for just one reviewer were .79 and .75, respectively, indicating fairly high reliability coefficients. The results of this study suggested that the G- and Phi-coefficients would increase to .88 and .86 for two reviewers, and .92 and .90 for three reviewers, respectively. Two or three independent reviewers are usually invited to review each scholarly journal article submission. The results showed that these experienced English journal reviewers consistently reviewed the submissions.
A Summary of G- and Phi-coefficients.
English Journal Reviewers’ Identified Common Errors in the 27 Published Abstracts
In addition to assigning a holistic assessment score to each of the 27 English abstracts, the 16 English journal reviewers also provided written comments on each abstract to identify the common errors in Chinese EFL researchers’ English abstracts. Table 5 is a summary of Phase Two qualitative findings.
A Summary of Common Errors Identified in the English Abstracts.
As shown in Table 5, several errors were identified in the following four aspects of an abstract: a) its accuracy, b) its non-evaluative nature, c) its coherence and readability, and d) its conciseness. First, the missing information about the problem under investigation, the participants, the study method, the findings, the conclusion, and the implications leads to an abstract’s inaccuracy. Among the 27 English abstracts, 20 (74.1%) were found to have such errors. Further, some abstracts did not include statistical significance levels for significant quantitative findings. For example, “As for the gender differences, females’ writing, reading, translation and total scores are significantly higher than males’, … Besides, females use each strategy significantly more frequently than males” (An excerpt from Abstract #12 published in Journal B).
Second, 13 out of 27 (48.1%) abstracts were evaluative. The authors evaluated rather than reported in these abstracts. Abstract # 24 published in Journal C is an excellent example of such problematic abstracts (see Appendix E). Further, these abstracts read like personal opinions or subjective judgments which are not based on research findings. For example, “In linguistic studies we need to change our way of thinking, start from Chinese practice and study tradition with western languages and linguistic tradition as a reference, and seek for the true characteristics of the Chinese language for the establishment of Chinese linguistics as a contribution to the study of general linguistics” (An excerpt from Abstract #11 published in Journal B).
Third, 23 out of 27 (85.2%) abstracts were found to have coherence and readability problems associated with their English use. Specifically, these errors included misspellings and grammatical errors, misuse of verbs tenses (e.g., it uses the past tense to describe conclusions and the present tense to describe how the research was conducted) and voices (e.g., it uses the passive voice rather than the active voice), and the ELF English use. These errors impact their coherence and readability. For example, “Their researches have drawn the conclusion that the mechanisms are themes, new information, temporality, semantic gravity and semantic density” (An excerpt from Abstract #23 published in Journal C); and “Stage 3 extends from about 2010 till many years from now… It is necessary for English departments of different universities to tailor their teaching programme to their current status and local and national needs so that high-level English majors can be trained” (An excerpt from Abstract #7 published in Journal A).
Finally, 18 out of 27 (66.7%) abstracts were found to have problems with their conciseness. Specifically, abstracts were not brief, and sentences were not maximally informative; some abstracts included too many important concepts, findings, or implications. Abstract #24 published in Journal C (see Appendix E) is an excellent example of that category of common errors.
As shown in Table 6, the four reviewers assessed this abstract very similarly regarding the assigned quantitative assessment scores and provided qualitative assessment comments. These results suggested that this abstract failed to meet the common criteria for a good abstract and therefore, further revision and editing are needed.
Assessment Summary of Abstract #24 by Four Selected English Journal Reviewers.
Note. The four aforementioned common errors identified in these English abstracts written by Chinese EFL researchers would lower their readability and international visibility. To help demonstrate some of these errors, the holistic scores and written comments by four selected experienced reviewers (A, B, C, and D) for Abstract # 24 published in Journal C (see Appendix E) are included in Table 6.
Phase Three: English Journal Reviewers’ Tips for Chinese EFL Researchers to Improve Their Abstracts
Phase Three of this study involved online semi-structured interviews with eight purposefully selected English journal reviewers from those who participated in Phase Two of this study. The main themes of the findings are as follows.
Seven out of eight reviewers interviewed agreed that the overall quality of these 27 English abstracts was poor. The common errors were reported in Phase Two qualitative results section (see above). They offered improvement tips for both these journals and the Chinese EFL researchers. As shown in Table 7, the two crucial improvement tips for these journals were to a) establish clear abstract instructions and guidelines for authors to follow in writing the English abstracts; and b) implement strict evaluation procedures for English abstracts.
A Summary of Phase Three Results.
Further, their improvement tips for Chinese EFL researchers are summarized. First, all reviewers suggested that Chinese EFL researchers familiarize themselves with English abstracts published in English journals. They can select several published English articles related to their research interests to read first, then learn the basic moves, and finally get familiar with the standards for English abstracts. As commented by one reviewer, this process could help Chinese EFL researchers “learn the basic moves in the abstract,” and, by another reviewer, “get familiar with the common criteria for acceptable English abstract.”
Second, six out of eight reviewers recommended that Chinese EFL researchers ensure that a good English abstract contains all necessary information. It must reflect the problem under investigation and include the participants, research procedures, results, conclusions, and the implications for practice and policymaking. “These essential elements of an English abstract guarantee that the abstract is complete and informative,” as commented by one reviewer.
Third, five out of eight reviewers mentioned that acceptable English abstracts must be grammatically correct and structurally coherent. Although it is not easy for Chinese EFL researchers to write high-quality English abstracts that meet these criteria, they need to keep this rule in mind while writing English abstracts. One suggested, “they [Chinese EFL researchers] could self-evaluate the grammatical accuracy and structural coherence of their completed abstracts before submission.”
Finally, three reviewers’ specific tips for a successful English abstract writing procedure can be summarized as “learning by reading, learning by doing, and learning by reflection” strategies. One reviewer described this writing procedure as follows: “they [Chinese EFL researchers] need to read the author instructions and guidelines established by the [English] journal; they must also practice and reflect since English is not their native language.” One senior reviewer further explained that through extensive reading, “the scheme of writing a good abstract may be developed and improved …”“having more practice on writing various abstracts on different topics, then, the candidates [Chinese EFL researchers] will become more competent and skillful …” and they will be able to write good English abstracts by reflecting on the criteria for assessing the quality of abstracts.
Phase Four: Chinese EFL Researchers’ Critical Reflections on English Journal Reviewers’ Assessments
Phase Four of this study involved online semi-structured interviews with eight EFL researchers purposefully selected from those who participated in Phase One of this study about their critical reflections on Phases Two and Three results. The major findings are summarized in the following section.
Overall, all eight participants were satisfied with the reliability of the English journal reviewers’ quantitative assessment of the 27 published English abstracts written by Chinese EFL researchers. They expressed their agreement with the reviewers regarding the common errors identified in these published English abstracts; they also found the reviewers’ tips for writing high-quality English abstracts constructive and valuable
More importantly, they offered the following three critical comments on Phases Two and Three results. First, six out of eight Chinese EFL researchers stated that the 27 English abstracts were published in the three top-tier language education journals in China; the generally low quality of these abstracts could have prevented the researchers from communicating their research findings within the international academia. “Who should be responsible for the quality of a published article abstract, the author, or the journal editor, or both?” one participant raised such a question.
Second, five out of eight Chinese EFL researchers suggested that international English journal reviewers support EFL researchers across the globe. When the reviewers are reading their submitted abstracts, they may consider their EFL background and unfamiliarity with the international standards for good abstracts. One EFL researcher commented that “… even an abstract plays an important role, the study design, data collection and analysis, and reporting of the findings should all be considered when the reviewers are making their acceptance decision.” One senior EFL researcher further explained that “English journal reviewers may be tolerate with a Chinese EFL researcher’s unacceptable English abstract and give him or her another chance to revise, edit, or even rewrite the abstract if the article is generally acceptable.”
Finally, seven out of eight Chinese EFL researchers suggested that they must equip themselves with the skills to write internationally acceptable article abstracts if they want their research articles published in English journals. “Each English journal has specific guidelines, and they [Chinese EFL researchers] must closely follow the guidelines while composing an abstract,” and “they [Chinese EFL researchers] should also read the instructors for authors when they are preparing their abstracts for submission” as commented by two Chinese EFL researchers, respectively.
Discussion and Conclusions
Phase One of this study investigated Chinese EFL researchers’ reported challenges in writing English abstracts. They experienced challenges in using effective linguistic features and correct rhetorical moves; they also felt difficult to organize their English abstracts in a logic and meaningful way. These challenges were similar to those reported in the literature (Hosseingholipour et al., 2021; Klimova, 2013; Linder, 2014; Lorés-Sanz, 2016). In addition, direct translation from Chinese into English caused them challenges in writing English abstracts. These two languages are different in genre of abstracts (Hu & Cao, 2011; Ruan, 2018). Therefore, it is not wise for Chinese researchers to use translation in writing English abstracts.
Phases Two and Three of this study invited 16 English journal reviewers to assess the 27 published English abstracts written by Chinese EFL researchers both quantitatively and qualitatively. The results suggested that the 27 English abstracts were generally poor in quality across the three journals; the reviewers’ holistic scores assigned to these abstracts were fairly consistent resulting in acceptable reliability coefficients. Although few studies employed G-theory to examine journal reviewers’ assessment reliability of the quality of English abstracts written by EFL authors, the G-theory results of this study could be compared with such results reported in EFL writing assessment studies (Huang & Foote, 2010; Liu & Huang, 2020; Zhao & Huang, 2020). Unlike the assessment of EFL essays, the quality assessment of these journal article abstracts was much more consistent and reliable. It is believed that these experienced English journal reviewers followed the scoring criteria closely while evaluating these abstracts (Li & Huang, 2022).
The investigation of the common errors and the improvement tips became the foci of Phases Two and Three of this study. Chinese EFL researchers face difficulties in four aspects of an abstract including its accuracy, non-evaluative nature, coherence and readability, and conciseness. These errors were similar to what other researchers had reported (Hosseingholipour et al., 2021; Klimova, 2013; Linder, 2014; Lorés-Sanz, 2016). It is essential to mention that these errors are associated with the language and format differences between abstracts published in English versus those published in Chinese journals (Hu & Cao, 2011; Ruan, 2018; Ye & Wang, 2013). Many abstracts were directly translated from the Chinese versions without considering the criteria for a good English abstract. These errors would surely lead to lower readability and visibility of the abstracts in international academia (Friginal & Mustafa, 2017; Hosseingholipour et al., 2021; Hyland, 2002; Lorés-Sanz, 2016).
Furthermore, the English reviewers’ tips for improvement were highly constructive and valuable for the Chinese EFL researchers. For example, they should familiarize themselves with the criteria of a good English abstract; they also need to learn the conventional moves in writing a good abstract; and they can learn to write good English abstracts by reading, doing, and reflection (APA, 2020; Feak & Swales, 2011; Swales, 2004; Swales & Feak, 2004). These tips were similar to what researchers in humanities and social sciences offered for teaching writing a good abstract (Friginal & Mustafa, 2017; Hyland, 2003, 2007; Klimova, 2013; Ruan, 2018; Stotesbury, 2003; Swales & Feak, 2009).
Finally, Chinese EFL researchers critically reflected on the English journal reviewers’ assessment outcomes. These reflections were helpful for both Chinese EFL researchers and English journal reviewers. On the one hand, Chinese EFL researchers are expected to follow the standards established by the English journals if they want to have their research published in them; on the other hand, the English reviewers are suggested to support Chinese EFL researchers and help to include their research articles in the English journals.
The present study was limited in the following three ways. First, the data collection of this study had been completed before the APA (2020) Publication Manual Seventh Edition was released; the assessment criteria for the 27 abstracts were adopted from the APA (2010) Publication Manual Sixth Edition, which were slightly different from the Seventh Edition (APA, 2020), which may have limited the interpretation of the results. Second, this study only examined 24 Chinese EFL researchers’ challenges in writing English abstracts, which may limit the generalization of the findings to other EFL researchers in China. Finally, this study involved only three journals in one discipline of foreign language education in China, which may limit the generalization of the findings to other journals in different disciplines. Therefore, it is suggested that the results of this study be interpreted with caution.
In light of these limitations, the following four conclusions were reached. First, the English journal reviewers’ assessment of the quality of 27 English abstracts written by Chinese EFL researchers was consistent and reliable; and these researchers’ English abstracts were generally poor. Given the critical role an English abstract plays in the international dissemination of the research study, Chinese researchers should be aware that their English abstracts need to maintain high levels of accuracy, clarity, conciseness, coherence, and readability (APA, 2010).
Second, in the international dissemination of research, a journal provides clear abstract instructions and guidelines for its authors. Without such guiding rules, the English abstracts written by Chinese researchers will result in low international visibility (Hosseingholipour et al., 2021; Hyland, 2002; Klimova, 2013; Linder, 2014). The errors made by the Chinese EFL researchers in their English abstract writing are commonly found in the EFL field (Friginal & Mustafa, 2017). These errors are caused by different language or format differences between English and Chinese journals (Hu & Cao, 2011; Ruan, 2018; Ye & Wang, 2013). Third, there are solutions to the identified problems. The English journal reviewers’ tips for improvement work for Chinese EFL researchers (Ruan, 2018). They could follow the suggestions and make every effort to improve their abstract writing in English (Hosseingholipour et al., 2021; Hyland, 2002; Linder, 2014).
Finally, it is believed that the English journal reviewers’ awareness and understanding of Chinese EFL researchers’ challenges in writing English abstracts and their kind support and suggestions will make Chinese EFL researchers more and more proficient in writing English abstracts (Huang & Foote, 2010). Chinese EFL researchers belong to the international research community, and the global research community also needs Chinese EFL researchers (Huang et al., 2021).
The results of this study provide important implications for Chinese EFL researchers, including graduate students. To compose high-quality English abstracts, they are encouraged to follow the English journals’ guidelines closely, and the strategies recommended by the English reviewers so that their English abstracts are of high quality and their research studies become more internationally visible.
Footnotes
Appendix A: Phase One Open-ended Questions
Appendix B: Journal Article Abstracts Assessment Form
Please use the following rubric to score each journal article abstract holistically.
The Holistic Rating Rubric (6-point scale)
Please also provide your written comments on the following four aspects of each abstract.
Appendix C: Phase Three Follow-Up Interview Questions
Appendix D: Phase Four Follow-up Interview Questions
Acknowledgements
The researchers of this study would like to express their sincere thanks to a) Mrs. Manhong Li for selecting the journal abstracts; b) the Chinese EFL researchers for their participation in Phases One and Four of this study; and c) the English journal reviewers for providing their holistic scores and valuable comments and responses during Phases Two and Three of this study.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by a) the teaching research project “A study on the improvement of ideological and political teaching literacy and ability of college English teachers in the context of new liberal arts” granted by Wuhan Institute of Technology (Project number: X2022044); b) the ideological and political teaching demonstration course “College English A2” granted by Wuhan Institute of Technology; and c) a first-class undergraduate course titled “Chinese excellent traditional culture: Contemporary Inheritance and Glory” granted by Wuhan Institute of Technology.
Ethical Statement
The treatment of human subjects was in accord with ethical standards and other requirements in China.
