Abstract
Qualitative research plays an important role in helping us describe, interpret and generate theories about complex phenomena in healthcare. Complete and transparent reporting of research informs readers about the significance and rigor of the work. The aim of this scientometric study was to determine the quality of reporting of qualitative research in nursing social science. Studies were identified by manually searching the table of contents for qualitative papers published in the June (or closest subsequent) 2018 issue of 115 nursing journals. Adherence with the 32-item Consolidated Criteria for REporting Qualitative (COREQ) research was determined for each study by two researchers. Additional information about the study (e.g., sample size, field of nursing) and the publishing journal (e.g., endorsement of COREQ) were also extracted. Using established criteria, COREQ compliance was coded either good (≥ 25 items), moderate (17 to 24), poor (9 to 16), very poor (≤ 8) based on the number of items addressed in each study. One hundred and ninety-seven manuscripts were included. The quality of reporting was generally rated as either moderate (57%) or poor (38%). Journal endorsement of qualitative reporting guidelines was associated with better reporting. The reporting of qualitative research in nursing social science journals is suboptimal. Researchers, authors, reviewers and journal editors need to ensure their papers comprehensively address the requirements of COREQ to ensure comprehensive and transparent reporting of their research.
Keywords
Introduction
Qualitative research focuses on developing a deeper understanding of complex phenomena that can be difficult to measure empirically (Johnson & Waterfield, 2004). There remains considerable scepticism, about the value of qualitative research in informing evidence-based practice (Panter et al., 2016). Qualitative studies are sometimes perceived as being subjective, difficult to understand, and prone to bias (Mackieson et al., 2019). A review of studies published in general medical journals reported that whilst the proportion of qualitative papers was increasing, they still made up a tiny fraction (4% in 2007) of published research (Shuval et al., 2011). The editors of the British Medical Journal have publicly stated that qualitative research is better suited to specialist journals because this type of research is usually exploratory and does not generate observations that can be generalized (Loder et al., 2016). Since this contentious statement (Greenhalgh et al., 2016), other leading journals have gone on to publish high quality qualitative analysis (Braun & Clarke, 2019) potentially reflecting an increasing acknowledgment of the value of qualitative methods within medical research (Livingston et al., 2019). Given nursing social science’s tradition of utilizing qualitative methodologies, ensuring that qualitative research is reported clearly and transparently may go some way to offsetting the negative perceptions of qualitative research. It will further increase the value of such research in informing policy and practice by providing high-quality evidence that can be evaluated based on shared criteria.
Debate over whether qualitative research should be critically appraised and systematically reported continues (Garside, 2014), particularly within nursing (Sandelowski, 2015). Some authors have argued that they are an attempt to reduce qualitative research to a list of procedures that results in the “tail wagging the dog” (Peditto, 2018; Smith et al., 2018). A recent mapping exercise of existing tools used to appraise methodological strengths and limitations of qualitative research identified over 100 critical appraisal tools (Munthe-Kaas et al., 2019). The authors concluded that few tools explicitly included explanations about how they were developed, or why a new tool was required. The most commonly used reporting guidelines for qualitative studies are COREQ (Tong et al., 2007) and the Standard for Reporting Qualitative Research (SRQR, O’Brien et al., 2014). COREQ guidelines were introduced in 2007 and are considered to be the most comprehensive (Smith et al., 2018) and most widely cited (Al-Moghrabi et al., 2019). Developed by Tong et al. (2007) the COREQ guideline was produced based on a comprehensive review of 22 checklists for qualitative studies. The authors initially identified 76 candidate items that were grouped into three domains: research team and reflexivity, study design and data analysis, and reporting. The current version of the COREQ guidelines applies to interview and focus group research and has 32 items grouped under these three domains. Notable nursing journals have strongly endorsed COREQ reporting guidelines for qualitative research (Smith et al., 2018).
Previous Studies of COREQ Compliance
There has been one previous study that has examined COREQ compliance in qualitative dental studies (Al-Moghrabi et al., 2019). The authors reviewed 100 studies with a median sample size of 20. Just over half (17, 53%) of the 32 COREQ items were adequately addressed in included studies, the reporting of most were categorized as moderate (17–24 COREQ items checked) (51%) or poor (9–16 COREQ items checked) (34%). There have been no previous studies that have checked COREQ compliance in nursing social science.
Objectives
The primary objective of this scientometric study was to determine the quality of reporting of qualitative studies involving interviews or focus groups published in nursing social science research journals.
Additionally, we planned to determine if the quality of reporting differed between different fields of nursing (adult, child, learning disability, mental health, midwifery) and if markers of journal quality (i.e., impact factor) were correlated with reporting quality. Finally, we planned two sub-group analyses to determine if 1. journal endorsement and 2. the inclusion of an author statement that reporting complied with COREQ, was associated with better reporting.
Method
A list of the 2017 JCR social science journals was downloaded from the Clarivate website. Three researchers manually screened the titles and abstracts of articles published in the June 2018 issue of each of the 115 included journals. If the journal did not publish an issue in June the subsequent issue was selected. Researchers checked the manuscript title, abstract and keywords for the terms: “qualitative,” “thematic analysis,” “interview,” “focus group,” “grounded theory,” “ethnography,” or “case study.”
Full texts were extracted and included in the study if they reported primary qualitative research, using either focus groups or interviews as a method of data collection and publish in English. Studies where qualitative data were collected using survey (e.g., questionnaire) or mixed methods, were excluded, given that COREQ guidelines are applicable to interview and focus group data collection methods only. Systematic reviews or meta-synthesis of qualitative research were also excluded.
Data extraction was undertaken by eight researchers using a template devised for this study. The following information was extracted from included studies: journal, the country where fieldwork was conducted, the field of nursing, ethics committee approval, author statement that their reporting adhered to COREQ guidelines, reporting of study limitations, total number of participants, method of data collection (interview, focus group), participants, duration of interviews, approach to data analysis (e.g., thematic, grounded theory). COREQ compliance for each included study was checked against each of the 32 criteria (rated yes/no) by two researchers. Data were checked and compiled by a third researcher. Discrepancies in study inclusion or item ratings were resolved with one other researcher.
We extracted the following journal level information the (2016) JCR impact factor, the number of papers (excluding educational content such as clinical updates) published in the included issue, and whether the journal stated that it endorsed qualitative reporting guidelines (defined as an editorial statement, requirement or recommendation in the journal author guidelines; or requirement to submit a reporting checklist as part of the submission process).
Analysis
The number and proportion of studies that checked each COREQ item were calculated. Study quality was categorized using the (Al-Moghrabi et al., 2019) criteria: good (≥ 25 items), moderate (17 to 24 items), poor (9 to 16 items), very poor (≤ 8 items). Pearson correlation was used to test the relationship between study quality and journal impact factor. The association between study quality score and field of nursing, whether the study author stated they adhered to COREQ and Journal endorsement of reporting guidelines was testing the using the chi-square statistics.
A ranking table of journals was generated by calculating a mean percentage of COREQ score for each journal. Journals were ranked low to high based on this score.
Results
Figure 1 shows the flow of papers through the study. Over the study period, nursing social science journals published 1,742 papers. From title, abstract and keyword screening we identified 229 potentially relevant studies. At full-text screening an additional 32 papers were excluded. The final sample for this review comprised 197 studies published across 71 journals (11% of all published papers). The mean number of papers per journal was 2.87 (
PRISMA flow diagram.
Characteristics of Included Studies
Table 1 shows the characteristics of included studies. Fieldwork for most studies was undertaken in Europe or North America. Two thirds of the included studies were in the field of adult nursing, none were in learning disabilities. Ten authors (5%) stated in their manuscript that they followed COREQ reporting guidelines when drafting their manuscript. Nearly all of the included studies (194, 98%) reported the number of participants, with the mean and modal sample size being 21.55 (
Characteristics of Included Studies.
COREQ Compliance
Table 2 shows the study compliance against the 32 individual COREQ criteria. The mean score per paper was 16.91/32 (
Number and Percentage of Studies Reporting Against Each COREQ Item.
Ladder of Included Journals
a Average score indicates the proportion of COREQ items reported per paper.
Characteristics of Included Journals
The mean impact factor of the 71 included journals was 1.40 (
Analysis
Post-hoc, the field of nursing was recoded into two groups (adult, other) because of the small number of children, mental health, learning disability and midwifery studies. Because of small cell sizes the quality of reporting was also recoded into two groups (good/moderate and very poor/poor) post hoc. No association between the field of nursing and COREQ compliance was observed (
Ranking of Journals
The 71 journals that published at least one included study were ranked (Table 3). The top three journals were the Journal of Psychiatric and Mental Health Nursing, Journal of the Association of Nurses in AIDS Care and Rehabilitation Nursing.
Discussion
The aim of this study was to determine the quality of reporting of interview or focus group based qualitative studies in a representative sample of papers published in nursing social science journals in 2018. Using 32 COREQ criteria, we reviewed 197 studies, overall the quality of reporting was suboptimal. Only five included studies were judged to meet criteria for good reporting (Al-Moghrabi et al., 2019; Dunt & McKenzie, 2012; Munthe-Kaas et al., 2019; O’Brien et al., 2014). This is the first study to examine adherence to COREQ reporting guidelines in nursing social science. Our observations were broadly consistent with similar research in a health related disciplines (Al-Moghrabi et al., 2019).
There are a number of potential explanations about the apparent poor reporting of qualitative research that need careful consideration. The most obvious explanation for the poor COREQ compliance we observed is that qualitative researchers are not aware of, or do not see the value of reporting guidelines. That so few authors included a statement in their manuscript that their reporting complied with COREQ guidelines may be consistent with this argument. The relevance of all COREQ items may also impact how guidelines are perceived. In the guidelines all COREQ items are given equal weighting. Some authors might argue that some COREQ items are irrelevant in some studies. For example, most (99%) studies reported the number of participants in the study (item 12), an aspect of reporting that might be considered essential. Conversely, only 8% of studies reported the gender of the researcher that it might be argued is not routinely required (especially in a female dominated profession such as nursing).
The majority of journals included in this review did not endorse COREQ, despite a number of nursing editors asserting the importance of guidelines for improving the reporting of research (Gray, 2018; Smith et al., 2018). Journal endorsement was associated statistically with better reporting. Enhanced reporting of clinical trials, observational studies and systematic reviews has been shown to improve if journals endorse reporting guidelines (Samaan et al., 2013). The findings from our study strengthens the case that all nursing Journals should formally endorse COREQ reporting guidelines.
It is plausible that authors, when initially submitting their manuscript, addressed COREQ items more comprehensively but were subsequently required by editors or reviewer to remove methodological detail considered superfluous. This could be due to space considerations; typically, qualitative studies have substantially larger word counts than observational or experimental studies. Testing this idea would require follow-up interviews with authors about their views of reporting guidelines.
It is concerning that there was ambiguity about the ethical standing of the research in 16 of the studies we reviewed. We acknowledge that some service evaluations do not require ethical approval, however, it is important this is made explicit to the reader. We have written previously that within nursing, authors often fail to provide adequate consideration of the ethical considerations of their research (McKenna & Gray, 2018).
Improving Reporting Practice
There is a consensus that the trustworthiness, relevance and transferability of qualitative research will be enhanced by better, more comprehensive reporting (O’Brien et al., 2014). Researchers should make use of the guidelines when both planning projects and reporting findings. Journals could adopt a mandatory requirement for completed COREQ checklist to be submitted with the papers? As part of the peer-review process manuscripts should be appraised against reporting guidelines and authors should be required to address omissions.
Limitations
There are several limitations in this study that are important to acknowledge and discuss. We selected studies published in a single issue. A number of journals published no studies in the included issue and consequently are not represented in this study, this included the
Researchers reported that some COREQ criteria/items were challenging to rate because they required a subjective judgment to be made, this issue particularly related to items related to the reporting of results (items 29 through 32). For example, items 30 and 31 ask if there is consistency between the data and the findings and if major themes were clearly presented; both require the rater to make an interpretation, the rating of the item is not a clear yes or no. Difficulties in applying reporting criteria because of the undefined and fluctuating nature of qualitative research have been acknowledged by other authors (e.g., Peditto, 2018). Researchers were advised to give the “benefit of the doubt” when they were unsure about how to rate. Consequently, scores for these items may be inflated.
Authors have observed there are important omissions in the COREQ checklist (O’Brien et al., 2014) that include ethical issues pertaining to human subject research and consideration of study limitations. We did address this, in part, by adding additional items (ethical considerations, study limitations) to our data extraction form. However, there were still possible omissions, for example, O’Brien et al., (2014) suggests that a problem formulation, details about the purpose of research and a statement about conflicts of interest are required. We did not measure these in our review, and this is consequently an acknowledged limitation.
We only recorded if the authors reported their research according to COREQ guidelines; we did not check if they followed other checklists such as the SRQR (Standard for Reporting Qualitative Research, O’Brien et al., 2014). This was an omission in our methodology, and we suggest future research should report if authors followed any recognized reporting guideline.
Our study only included qualitative research where data were collected by interview or focus group, given this is the emphasis of the COREQ guidelines. Consequently, we have not appraised the quality of reporting where data were collected using textual, media, virtual or other data collection techniques (Braun & Clarke, 2019). The SRQR is a more recent measure of quality that—the authors argue—can be applied to all qualitative research (O’Brien et al., 2014). Whilst, not as extensively used as COREQ it may be that the SRQR is a more comprehensive measure of the quality of research reporting. Readers should note that whilst both COREQ and SRQR were developed based on a review of the literature (Peditto, 2018) the psychometric properties of COREQ (and the SRQR for that matter) have not been formally investigated and this is work needs to be done.
Finally, it is important to remember that the quality of reporting and the rigor and importance of research are not synonymous. A study may check all COREQ items but still be poor quality research with high risk of bias, this is a point to note when interpreting our observations.
Conclusion
This is the first study that has systematically appraised the quality of reporting of qualitative research in nursing disciplines. Our observations suggest that reporting of qualitative research in nursing social science is suboptimal and there is a need for a concerted effort to enhance reporting. Authors need to pay close attention to the requirements of reporting guidelines when planning and reporting their research. Journal editors needs to ensure that the qualitative papers they publish have fully complied with COREQ or similar reporting guidelines. Authors of qualitative research may argue that requiring all researchers to adhere to one standard is restrictive for such a broad field of enquiry. We reject this argument. All research needs to be clearly and transparently reported so that the reader can make informed judgments about the merit of the research to health policy and practice.
Footnotes
Author Contributions
RG, SW and MJ conceived and designed the study. SW, MJ, DB, LM, EB, ST, MS, and RG extracted and checked data from included papers. SW completed data checking and cleaning and tabulated results. RG drafted the first version of the manuscript. All authors have contributed to revising the paper and have approved the final version of the manuscript.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: MJ is an editor of the
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
