Purposeful turns for more equitable and transparent publishing in language testing and assessment

This Editorial comes at a time when the after-effects of the acute phase of the COVID-19 pandemic are still being felt but when, in most countries around the world, there has been some easing of restrictions and a return to (quasi-)normalcy. In the language testing and assessment community, many colleagues relished the opportunity to meet and participate in events at the 44th annual Language Testing Research Colloquium (LTRC) in New York in July 2023. This was after 4 years of LTRC exclusively being held online due to public health concerns, restrictions on movement, and other policy-related and logistical matters. In the context of this Editorial, which comes out annually, we find it liberating to be able to focus on matters that are non-pandemic related. In terms of the day-to-day business of managing the journal, we have moved beyond a time of crisis, as reflected in the removal of a note about pandemic effects in our Author and Reviewer e-mail invitation templates.

In this annual address, we note a change of the guard in the editorial team that will have come into effect by the time this Editorial is published and some elements of continuity. We also reflect on developments over the past year while briefly touching on what lies ahead.

“To everything turn, turn, turn; There is a season turn, turn, turn; And a time to every purpose . . . ”

The above lyrics written by Peter Seeger in 1954 and popularized by the American rock band, the Byrds, signifies that in a dynamic, everchanging world, there is a time and place for everything (Hartel, 2019). The time has come to address some of the journal-related matters discussed below, and the winds continue to blow with changes to personnel.

Paula Winke stepped down from her position as Co-Editor of Language Testing at the end of December 2023 after a five-year tenure. The mantle has been taken up by Xun Yan, who served on the Editorial Board for the past eight years and who will work closely with ongoing Co-Editor, Talia Isaacs, to oversee the strategic direction and management of Language Testing, steering the journal into the next chapter. As of January 2024, Associate Editor Ruslan Suvorov is assuming an expanded role, assisting with the internal review process in addition to managing Brief Reports and producing content for the journal’s YouTube channel and other fora. We are grateful to Ute Knoch, our Test Review Editor, and Benjamin Kremmel, our Book Review Editor, for their continued service and support.

Paula mentored several Editorial Assistants at Michigan State University, whose time and assistance were generously subsidized by the university—notably Dylan Burton, who served in the role for four years, preceded by Caitlin Cornell and followed by April Day. Sofya Styrina has now taken over the Editorial Assistant role, supported through funding from Xun’s university, the University of Illinois at Urbana-Champaign. We are pleased that Paula and Dylan are serving as Editorial Board Members from January 2024 to provide continuity through this period of change and beyond. We also welcome Franz Holzknecht to the Board and warmly thank Ofra Inbar, Spiros Papageorgiou, and Jonathan Schmidgall, who rotated off the Board at the end of December 2023, for their valuable service.

During Paula’s term as Co-Editor, she passionately championed open science principles, enacting her vision and putting the journal on a positive, agenda-setting trajectory to keep pace with and even be at the cutting edge of developments designed to increase transparency in research conduct and dissemination (e.g., Winke, in press). A major plank of the changes she enacted was the introduction of six new journal article types: Meta-Analyses, Systematic Reviews, Registered Reports, Brief Reports, Obituaries, and Viewpoint articles (Harding & Winke, 2022). Seeing further uptake of these new submission categories in the coming years will be part of her legacy and is already driving growth for the journal in terms of the number of submissions, which is on an upward trajectory.

We are delighted to be publishing our very first Viewpoint article in this issue. This contribution by Jennifer Randall and her co-authors (2024), entitled, “Our validity looks like justice. Does yours?” is based on Randall’s Samuel J. Messick Memorial Lecture presented at LTRC in New York in June 2023. The authors’ reconceptualization of validity and emphasis on a justice-oriented approach has the potential to be paradigm-shifting and to inspire further considered thinking about validity in language testing and assessment. As part of the broader backdrop of timely fieldwide and societal discussions and action on equality, diversity, inclusion, and social justice (e.g., Shohamy, 2022; Taylor & Banerjee, 2023), the questions that the authors raise could (and arguably should) be embedded into routine thinking about tests and constructs. We also underscore here the utility of the Viewpoint forum, which features positions papers or authors’ perspectives, including in response to previously published content, to encourage innovative thinking and ignite impassioned and respectful discussion and debate.

In reference to open science, we particularly want to highlight the new Registered Report article type (see Marsden et al., 2018). Registered reports are primarily suitable for, although not exclusive to confirmatory, quantitative studies. Authors publish a research blueprint or plan for their paper and, following review, receive an in-principal acceptance to publish their full study if they follow the plan or can offer robust justification for minor deviations. Prospective authors who have been awarded a research grant, for example, could consider submitting a planned paper centering on one of the work packages as a Registered report. One benefit of publishing a Registered Report is that authors receive an extra citable publication and an invitation to publish their full-length article in Language Testing. Another advantage is that authors may struggle to publish statistically non-significant results due to a well-documented phenomenon known as publication bias (Franco et al., 2014). But withholding the publication of null findings or prioritizing the reporting of statistically significant results over null results can distort the picture, both for the individual study in question, and for any secondary analyses that includes that study (e.g., systematic reviews, meta-analyses). This can result in misleading, irreplicable, or undisseminated research (Bradley et al., 2020). Pre-specifying a priori the analyses to be undertaken and variables of interest in a Registered Report can help editors and reviewers hold authors to account by discouraging “cherry picking” (i.e., selective reporting) including reporting significant over null results (Isaacs & Chalmers, in press). This is just one example of an innovation made during Paula’s term as Co-Editor that has the potential to advance openness and transparency in our field. We look forward to future discussions about open science and to new developments, including in the 2024 Special Issue on “Open science practices in language testing and assessment” Guest Edited by Dan Isbell and Benjamin Kremmel and due to be published later this year.

Another development that is already revolutionizing academic publishing, the assessment enterprise, and much else and that is rapidly evolving is the use of large language models—a key component of natural language processing—and the growing availability and use of open-source artificial intelligence (AI) tools, including but not limited to ChatGPT (see, for example, Liu et al., 2023). In addition to short-term uptake, the widespread use of these transformative technologies also has long-term repercussions on the publishing world and dissemation of assessment-related work that are difficult to fully anticipate. It follows that the Editorial team and SAGE, our publisher, are keeping a close eye on the developments, challenges, and responses from academic bodies, professional associations, and other journals (e.g., Levene, 2023; Russell Group, 2023). We anticipate discussing Language Testing’s approach on ethical uses of AI tools in our next annual Editorial (January 2025).

What is clear—and the use of newly available technologies is a case in point—is that as circumstances change, the journal needs to be responsive to those changes and proactive in adapting. This means that guidelines for authors and reviewers, and, in some cases, processes need to evolve, including based on new trends or behaviors that we Editors and Editorial Board Members observe. For example, we now provide authors with explicit information on required components of cover letters and guidance for authors with multiple institutional affiliations. We have also updated our e-mail templates to encourage authors whose papers are formally accepted and who do not have access to “gold” open access funding to consider making author-final versions of their manuscripts available on a public digital repository—ideally, one such as Open Science Framework that provides authors with a digital object identifier (DOI) for their deposit and is indexed to SHARE systems (e.g., Web of Science, Scopus, arXiv, GoogleScholar, etc.; see Winke, in press).

In the next two paragraphs, we describe two changes to Language Testing processes and guidance implemented over the past year. First, in academic disciplines such as science and medicine, author contribution statements for articles, which report on the role that all named authors have played in contributing to research and dissemination, are common and even normative. Our field does not have this tradition of documenting, in a short statement, how each author has contributed to a manuscript. Language Testing is one of SAGE’s first social science/education journals to participate in the Contributor Roles Taxonomy (CRediT) pilot scheme, which requires reporting, in a succinct, standardized way, the role(s) that each author has played in conducting and disseminating the research (Allen et al., 2014). This is done through the submitting author’s selection of each named author’s contribution from 14 pre-specified roles in a dropdown menu (e.g., Conceptualization, Formal Analysis, Funding acquisition, Methodology, Project administration, Supervision, Validation, Writing—original draft, Writing—review & editing, etc.; National Information Standards Organization [NISO], 2023). Co-authors should review and approve these role allocations prior to article submission. In addition to increasing transparency by offering insight into and acknowledging authors’ roles and responsibilities, contribution statements could demonstrate individual author contributions for tenure files. They could also be useful in helping resolve authorship disputes and discouraging unethical behavior (e.g., gift authorship; Royal Society, 2023). We should note that author contribution statements are not made available to reviewers during the peer-review process. As we underscore in our YouTube video introducing authors to CRediT (Language Testing, 2023), some of the category labels may not be intuitive for language testers (e.g., data curation). However, our participation in SAGE’s pilot scheme provides us with the opportunity to feed back on the system’s usefulness, suitability, and functionality.

Second, there has recently been a spotlight on conflict-of-interest declarations in our field as part of wider conversations on research ethics in applied linguistics. In Isbell and Kim’s (2023) systematic review on conflict-of-interest reporting in high-stakes language proficiency testing research, a third of the included articles were published in Language Testing—the highest proportion of the five journals they examined in their search. The authors revealed that just 4% to 7% of included articles made appropriate conflict-of-interest disclosures, highlighting the need for substantial improvement. In response to these findings and to the authors’ recommendations to journal editors on combating the under-reporting or distortion of conflicts of interest, we now require authors to disclose any financial or non-financial conflicts-of-interest to the Editors during the peer-review process (this information is not available to reviewers). We are working to destigmatize and normalize conflict-of-interest reporting. The reality is that most established language testers do have conflicts of interest due to the nature of our field, funding, and employment or partnerships with assessment bodies and other organizations who fund research on the development, validation, and use of language assessments. Openly declaring conflicts of interest can enable readers to make up their own minds about any potential bias and should be encouraged. Withholding them undermines transparency and can be unethical.

As part our endeavor to mirror practice in some of the top scientific journals, we have published 2022–2023 conflict-of-interest statements for all five Language Testing Editors who oversee and manage peer review in the “Conflicting interests” section of the journal’s author submission guidelines. We pledge to release updated lists of the Editorial team’s disclosures on an annual basis. We also require conflict-of-interest declarations from authors and outline our requirements, drawing on guidance from the Committee on Publication Ethics (2023) and precedents from other fields (e.g., International Committee of Medical Journal Editors [ICJME], 2023). We believe that these efforts, which implement Isbell and Kim’s (2023) recommendations, are long overdue and necessary to pave the way forward. We also understand that an overhaul of the International Language Testing Association’s (ILTA) Code of Ethics will be conducted in the coming months and that managing and disclosing conflicts of interests is likely to be an area of focus (Deygers & Malone, 2023). Language Testing is, thus, leading the way in an effort to drive fieldwide change, and we hope that other language and educational assessment journals will follow suit. We appreciate authors’ cooperation in making open and transparent conflict-of-interest declarations. We are also grateful for reviewers’ continued judgments about their own conflicts of interest when considering peer-review invitations in the context of a small, well-networked field.

Finally, as the journal’s 40th anniversary has come and gone, we have included a list of the 16 current and former Editors of Language Testing on the website. We are conscious of how much the journal has evolved, and, as we approach 25 years after the start of the new millennium, eagerly await what comes next.