Abstract
Heterogeneity in outcomes measured in trials limits accurate comparison of bronchiectasis studies. A core outcome set (COS) is an agreed, standardized set of outcomes that should be measured in trials for specific clinical areas. A COS for bronchiectasis could encourage consistency in future studies. An overview of systematic reviews and qualitative study on outcome selection in bronchiectasis informed an initial list of outcomes. A Delphi panel (n = 86) rated the importance of each outcome from 1 to 9 in 3 sequential questionnaires, as a means to achieve consensus: 1–3 = ‘of limited importance’; 4–6 = ‘important, but not critical’; and 7–9 = ‘critical’. Outcomes rated ‘critical’ by ≥70% of the panel were added to the COS. Eighty-two participants responded to the first questionnaire. Attrition between each questionnaire was 5%. After 3 rounds of questioning, 18 outcomes exceeded the threshold for consensus and were included in the COS. This study has achieved consensus on 18 outcomes that should be measured in trials of interventions for bronchiectasis. Selection of the highest ranked outcomes may represent a pragmatic means for comparison. Further research is required to condense the number of outcomes selected and to determine its relevance to interventions.
Background
Recent developments in bronchiectasis research have been attributable to an increased interest and investment into the condition. 1 Although there are currently no licensed therapies for bronchiectasis, randomized controlled trials (RCTs) investigating the management of the condition are under way. 1 Systematic review and meta-analysis to synthesize the results of emerging RCT data will follow. However, meta-analysis between similar interventions is often not possible or limited because there is marked variability in the outcomes that are measured and reported by researchers. For example, meta-analysis was limited in a 2015 Cochrane systematic review of prolonged antibiotics for bronchiectasis because of the diversity of outcomes reported in the included trials. 2 Heterogeneity between studies makes it difficult to compare the effectiveness of two interventions when they have been investigated in different ways. 3 It also limits the extent to which the data can be used to inform clinical guidelines. 4
A new approach in research methodology that is gaining recognition in the scientific literature is the development and implementation of a core outcome set (COS). 3 This approach seeks to overcome the challenges presented by heterogeneity between studies. A COS is defined as an agreed, standardized set of outcomes that should be measured and reported as a minimum in all clinical trials for a specific clinical area. 3 The aim in the development of a COS is to reach consensus on ‘what’ outcomes should be measured and reported, rather than ‘how’ or ‘when’ such outcomes should be measured. 3 The Core Outcome Measures in Effectiveness Trials (COMET) Handbook was published in June 2017 (i.e. after the present study had been undertaken) and recommends a four-step process to COS development (Figure 1). 5
Four-step process to the development of a COS (adapted from the COMET Handbook (version 1.0). 5 COS: core outcome set; COMET: Core Outcome Measures in Effectiveness Trials; COS-STAR: Core Outcome Set-STAndards for reporting.
A COS for bronchiectasis could ensure consistency in future clinical trials designed to test the effectiveness of new and existing management strategies for the condition. It would allow for better comparison between studies and ensure only the most effective, rigorously tested strategies are recommended for patients. 6 The study described in this article aimed to develop a COS for the long-term management of bronchiectasis in adults.
Methods
Definitive guidance on COS development was unavailable at the time this study was conducted. Therefore, an approach detailed in a discussion paper by Williamson et al., 3 which preceded the COMET Handbook (version 1.0), was followed. 5 The method used closely aligns with the four steps illustrated in Figure 1 and differs only by the omission of a face-to-face meeting to discuss the final COS, which had been previously recommended to be an optional step in COS development. 3
The scope of the intended COS was defined to include all interventions for the long-term management of bronchiectasis in adults. Both pharmacological and non-pharmacological interventions were included in the scope to provide a COS that, if implemented in all trials, could help clinicians directly compare a wide range of strategies and inform recommendations about those most suitable for patients with different preferences and needs.
The COMET database was searched using the keywords ‘Bronchiectasis’ and ‘Lungs & airways’ in February 2016 and did not identify COS studies of relevance. For the purposes of this study, the term ‘outcome’ was defined as a consequence of bronchiectasis that is experienced directly by the patient. It is distinct from the terms ‘marker’, ‘measure’ or ‘outcome measure’, which refer to the measurements or instruments known to be associated with outcomes. 7,8
The study was prospectively registered on the COMET database (registration number 936; http://www.comet-initiative.org/studies/details/936). A protocol was developed that established a plan for identifying the outcomes that should be included in a COS for bronchiectasis. This involved (1) a review of existing knowledge on outcomes selection in bronchiectasis 9 and (2) a consensus study to gather opinions of key stakeholders in bronchiectasis. 10 These publications were used as they represented the most recent publications at the time this study was conducted in terms of synthesis of evidence 9 and consultation with key stakeholders, including health care professionals and patients.
Review of existing knowledge
A review of outcomes selection for bronchiectasis trials was conducted as part of an overview of interventions included in Cochrane systematic reviews for the treatment of bronchiectasis. 9 Outcomes that had been measured in previous trials were summarized to ‘inform a set of standard outcomes for future research studies’. 9
In 2014, a consultation with key stakeholders in the management of bronchiectasis was conducted as part of a study to develop an adherence intervention for bronchiectasis. 10 Researchers in Belfast, Northern Ireland, facilitated a series of three expert panels attended by patients (n = 11), health care professionals and academics (n = 9) from different backgrounds (bronchiectasis management, intervention development, commissioning of services and adherence research). 10 Panel members were asked, among other topics, to discuss their views on how the intervention could be evaluated, specifically what outcomes the study should measure. 10
Together these studies provided a convenient starting point for the development of a COS for bronchiectasis. Outcomes within the scope of the COS were extracted from both studies and compiled into a longlist of potential outcomes for consideration for inclusion in the COS (see Online Supplemental Material). Items that were outside the scope of the COS and those considered to be measures or markers of an outcome were excluded. Duplicates were removed and terminology standardized.
Consensus exercise
The Delphi technique has been proposed as the suitable method of reaching consensus when developing a COS and was the approach taken for this study. 5,11 Participants for the Delphi panel were recruited from the European Multicentre Bronchiectasis Audit and Research Collaboration (EMBARC) network and from a patient advisory group coordinated by the European Lung Foundation. A pragmatic target sample size of at least 100 participants consisting of at least 40 secondary care physicians, 12 physiotherapists, 12 nurses and 35 patient representatives was set. A higher proportion of physicians to others was chosen to reflect the higher proportion of physicians who are involved with bronchiectasis research. With an anticipated response rate of 70–75%, a minimum of 140 (at least 55 physicians, 15 physiotherapists, 15 nurses and 50 patient representatives) invitees were required. This response rate was based on a modified Delphi study, which achieved a 72% response rate to questionnaires sent to 138 EMBARC members. 12
In response to slow recruitment of nurses and physiotherapists, the protocol was amended to invite contacts known to the research team and to ask recruited participants to refer nurse and physiotherapists who would be interested in taking part. Those who had proven direct involvement with the care of people with bronchiectasis, or in bronchiectasis research, were invited to take part.
The views of participants on outcomes that should be included in the COS were gathered using a series of three consecutive Delphi questionnaires distributed by email, with a Web-based survey tool (SurveyGizmo®). Participants were asked to complete the questionnaire within 3 weeks; non-respondents were sent a reminder email 2 weeks later. Only participants who responded to the first questionnaire were invited to participate in the second round of questioning, and likewise, only participants who responded to the second questionnaire were invited to complete the third questionnaire. Questionnaires were piloted by a COS developer at Queen’s University Belfast who was not connected with the current study to ensure that there were no technical difficulties and that questions were in a logical sequence.
In the first questionnaire, participants were presented with an initial list of outcomes (Table 1), derived from the review of existing knowledge. 9,10 These were accompanied by plain language explanations where required. Participants were instructed to rate the relative importance of each outcome. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group scale was used to help participants consider the relative importance of each outcome in the list and was used to determine when consensus had been achieved (see Table 2). 3,13 A screenshot of part of the online questionnaire is provided in Figure 2.
Outcomes presented in first questionnaire.
GRADE: Grading of Recommendations Assessment, Development and Evaluation; COS: core outcome set.
Screenshot of part of questionnaire 1.
In the first questionnaire, participants also had the opportunity to suggest outcomes they believed should be included. The appropriateness of outcomes proposed by participants was reviewed and discussed by the study group. An outcome was defined as ‘appropriate’ if it represented a new and distinct outcome and was not considered to be a subset of any existing outcome. Outcomes that had been suggested by participants that were deemed appropriate by the study group were included in the second questionnaire and are shown in Figure 3.3
Outcome selection and progression through three Delphi questionnaires.
In the second questionnaire, participants were presented with a revised list of outcomes comprising the original list and any new outcomes yielded from the first questionnaire. Feedback about how the panel had collectively rated each outcome in the first questionnaire was displayed under each outcome. Participants were also sent a summary about how they had personally rated each outcome. This allowed participants to reflect on their own responses in relation to the collective response of the panel before rating the outcome again.
In the third questionnaire, participants were presented with a list of outcomes for which consensus had not been reached and asked to rate them again using the same scoring system as before. Similar to the second round, participants received individual and group feedback on how each outcome was scored. Suggestions for additional outcomes were not collected in the second or third questionnaires.
After each round of questioning, the percentage of participants who rated each outcome in terms of importance was determined. Consensus ‘in’, that is, the outcome should be included in the COS, was confirmed, if after the second or third round of questioning, 70% or more of participants rated an outcome 7 to 9, that is ‘Critical’ and fewer than 15% of participants rated it 1 to 3, that is ‘Of limited importance’. 13 The same thresholds have been used in the development of COS in other clinical conditions. 14,15
The Core Outcome Set-STAndards for Reporting (COS-STAR) guidelines have been used to report the findings of this study, and a completed checklist is provided in the Online Supplemental Material. 16
Ethics approval and consent to participate
A proportionate review subcommittee of the London – Camden & King’s Cross Research Ethics Committee gave the study a favourable ethical opinion on 13 September 2016 regarding the involvement of patient representatives from the United Kingdom (UK; reference number 16/LO/1712). The School of Pharmacy Ethics Committee at Queen’s University Belfast (QUB) also reviewed the study with regard to the involvement of health professionals and non-UK patient representatives and granted its approval on 5 September 2016 (reference number 018PMY2016). Permission to implement an amendment was granted by the QUB School of Pharmacy Ethics Committee on 3 November 2016.
Results
An initial list of 20 outcomes was compiled from those specified by Welsh et al. 9 and suggested by the bronchiectasis stakeholder panels (see Table 1 and Figure 3). 10 All outcomes suggested by Welsh et al. were included (n = 15). 9 Outcomes that had been listed separately according to how they were measured were included as one outcome. For example, ‘exacerbations per year’ and ‘time to first exacerbation’ were listed singularly as ‘Exacerbations’. Nine outcomes were suggested in the McCullough study, four overlapped with those identified by Welsh et al., yielding an additional five outcomes for consideration by the Delphi panel, resulting in a final list of 20 outcomes. 9,10
A total of 180 individuals were invited to participate in the Delphi panel: 70 doctors, 60 allied health professionals (AHPs), including nurses and physiotherapists and 50 patient representatives. Eighty-six participants were successfully recruited to the study (recruitment rate 48%). The respective recruitment rates for doctors, AHPs and patient representatives were 64% (n = 45), 30% (n = 18) and 46% (n = 23), respectively. Twenty-two European countries were represented. Participants covered 22 European countries; most participants (n = 42, 51%) lived or worked in the United Kingdom, with the second largest cohort living or working in Spain (n = 7, 9%). Four participants did not respond to the first questionnaire (response rate = 95%). There was a 5% attrition rate between each subsequent round of questioning. The numbers and characteristics of participants who responded to the questionnaires at each of the three rounds of questioning are provided in Table 3.
Characteristics of respondents to Delphi questionnaires.
As planned and regardless of how they were rated, all 20 outcomes proceeded to the second round of questioning. An additional 12 outcomes that had been suggested by participants and deemed appropriate by the study team (see Figure 3) were included in the second questionnaire. As such, 32 outcomes were rated in the second questionnaire. Fifteen of these outcomes met the criteria for consensus ‘in’ and were added to the COS. In the third questionnaire, participants rated the remaining 17 outcomes and three more outcomes (‘Haemoptysis’, ‘Shortness of breath’ and ‘Activities of daily living’) met the criteria for consensus ‘in’ and were added to the COS. At the end of three questionnaires, consensus was reached regarding the inclusion of 18 outcomes in the final COS. The selection and progression of outcomes through the Delphi exercise is presented in Figure 3. Tables outlining how each outcome was rated by participants in the three questionnaires are provided in Online Supplemental Material.
The final 18 outcomes selected for inclusion in the COS were ranked in order starting with the highest proportion of participants rating the outcome as ‘Critical’ to the lowest proportion as shown in Table 4.
Final COS ranked in order from the highest to the lowest proportions of participants rating the outcome as ‘Critical’.a
COS: core outcome set.
a The five highest-ranking outcomes are listed in boldface.
b Added to COS after the third questionnaire.
Discussion
This study followed a recommended process for the development of a COS using a similar approach to the latest published guidelines. 3,5 We avoided the unnecessary duplication of research by building upon the findings of two published studies on outcomes selection for bronchiectasis trials. 9,10 The outcomes identified by Welsh and colleagues were extracted from nine high-quality Cochrane systematic reviews and provided an accurate summary of what previous researchers measured when investigating interventions for bronchiectasis. 5,9 The findings from a consultation with key bronchiectasis stakeholders led to the inclusion of outcomes in the COS that had been suggested by participants in the stakeholder panels. 10
The Delphi technique is an established method for reaching consensus in situations where there is some degree of uncertainty, controversy or incomplete evidence. 17 It is currently the method of choice in most COS development studies. 5 This study sought to involve a range of different stakeholders. Broad consensus is important to COS development to encourage wide acceptance and implementation. 18 Representation from patients, doctors, nurses and physiotherapists was achieved.
Eighteen outcomes were selected to be included in the COS by the Delphi panel. This was higher than recently published COSs for other clinical conditions. 19 –21 There is no recommended maximum number of outcomes that should be included in a COS. To collect the amount of data required to measure 18 outcomes would neither be acceptable nor be feasible in a clinical trial setting. There are different aspects to the long-term management of bronchiectasis, and some of the outcomes will be more appropriate than others for different studies. Investigators may need to choose outcomes that are of greatest relevance to the intervention under investigation. However, to preserve a certain degree of homogeneity, selection of the highest ranking outcomes in the COS, for example, outcomes achieving greater than 90% agreement (see Table 3), would be recommended.
Further attempts to reduce the number of outcomes in a COS are required. One option would be to conduct a face-to-face consensus meeting to allow stakeholders to deliberate on the suitability of selected outcomes. The COMET Handbook (version 1.0), 5 which was published after this study had been completed, recommends that stakeholders are given the opportunity to discuss findings of a Delphi study before a final COS is agreed. It was not possible to conduct a face-to-face meeting for this study for financial reasons, but the planning of such a meeting would be the next logical step in the development of the first COS for bronchiectasis. Conducting a ‘virtual’ meeting may also be possible, provided that there is appropriate technical support and participants are able to access and feel comfortable using the infrastructure for such a meeting.
Particular strengths of this study that support the generalizability of the developed COS were the size and composition of the Delphi panel. However, this study did not attempt to recruit participants from the pharmaceutical industry, who could have provided a different perspective on the importance of certain outcomes. The study was limited to European patient representatives and health professionals, the majority of whom were based in the United Kingdom, which may limit the wider generalizability of the results.
A major limitation to this study that restricts the immediate implementation of the developed COS is the large number of outcomes yielded from the consensus process. Although the decision to target a broad scope was deliberate, it is possible that considering the importance of both pharmacological and non-pharmacological interventions made it more difficult for participants to prioritize their ratings, resulting in a large number of outcomes reaching consensus. Limiting the scope of the COS to one type of intervention may have reduced the number of outcomes selected. An alternative approach to reducing the number of outcomes in the COS could have been to increase the threshold for ‘consensus in’ to limit the number of outcomes selected for the COS. However, changing the criteria for consensus after the results have been analysed would potentially introduce bias to the process, reducing the reliability of the study. 5
The high retention of participants throughout each stage of the study (approximately 95%) means the reliability of the results was not substantially compromised by participants dropping out because their views were not the same of other members. 11 The online distribution of Delphi questionnaires made it possible to involve a wide variety of participants across a large geographical area. Use of free survey software also allowed costs to be minimized. There was a low time commitment required from participants, each questionnaire taking around 10 minutes to complete. A limitation to the remote completion of questionnaires was that it was difficult to assess the participants’ comprehension of the task of rating the importance of outcomes, and therefore, it cannot be confirmed that the views of the participants were reliably collected. However, attempts to optimize participant understanding were made by providing explanations where needed.
A COS specifies what outcomes should be measured, rather than how they should be measured. Yet to fully specify an outcome, information about the measurement tool and metric used to characterize results are also required to fulfil the purpose of a COS. 5 The development work described in this article can be viewed as the first step towards the creation of an internationally agreed COS and core measurement set for bronchiectasis. In its current format, it provides valuable insight into the outcomes of bronchiectasis that are considered of critical importance to both the health professionals who are investigating and caring for people with bronchiectasis and those living with the condition. Periodic review of the COS and validation studies will also be required to ensure that the COS remains relevant and up to date. 3
Conclusion
This study has achieved broad consensus from a varied group of stakeholders on the inclusion of 18 outcomes in a COS for the long-term management of bronchiectasis. Consideration of at least the five highest ranking outcomes could provide the homogeneity required to better inform clinicians to make evidence-based decisions. Further research is needed to condense this list of outcomes and to reach consensus on standardized methods of measuring each outcome in the agreed COS. This study has provided key bronchiectasis stakeholders across Europe with the opportunity to help set the direction for bronchiectasis research and clinical care.
Supplemental Material
Supplemental_Material-CRD - Development of a core outcome set for trials investigating the long-term management of bronchiectasis
Supplemental_Material-CRD for Development of a core outcome set for trials investigating the long-term management of bronchiectasis by Maureen Spargo, Cristín Ryan, Damian Downey, and Carmel Hughes in Chronic Respiratory Disease
Footnotes
Acknowledgements
The authors would like to acknowledge the support of the European Bronchiectasis Network (EMBARC), the European Lung Foundation and their members for their support in this study.
Author contributions
All authors contributed to the study design and interpretation of data. MS collected and analysed data. All authors were involved in writing and revising the manuscript before submission.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded internally by Queen’s University Belfast. The funder had no role in the study design, data analysis, data interpretation or writing of the manuscript.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
