Development of minimum reporting sets of patient characteristics in epidemiological research: A methodological systematic review

Abstract

Background

Core patient characteristic sets (CPCSs) are increasingly developed to identify variables that should be reported to describe the target population of epidemiological studies in the same medical area, while keeping the additional burden on the data collection acceptable.

Methods

We conduct a systematic review of primary studies and protocols that aim to develop a CPCS, using the PubMed database. We extract information on the study design and the characteristics of the proposed CPCS. The quality of Delphi studies is assessed by a tool proposed in the literature. All results are reported descriptively.

Results

Among 23 eligible studies, Delphi survey is the most frequently used technique to obtain consensus in CPCS development (69.6%, n = 16). Most studies do not include patients as stakeholders. The final CPCS rarely includes socioeconomic factors (26.1%, n = 6). Besides, 60.9% (n = 14) and 26.1% (n = 6) of the studies provide definitions and measurement methods for items in the CPCS, respectively.

Conclusion

This review identifies considerable variation and suboptimality in many methodological aspects of CPCS studies. To improve these shortcomings, guidance on the conduct and reporting of CPCS studies should be established in the future.

Keywords

Systematic review patient characteristics data collection Delphi survey consensus

Introduction

In epidemiological research, collecting and reporting patient characteristics are of great importance. These data allow to assess the generalizability (or external validity) of research findings to settings that are diffrent from those originally examined.¹ When comprehensive patient characteristic data are available, the difference between a study sample and a clinically relevant patient population can even be statistically accounted for.² Besides, patient characteristic data are also crucial for improving internal validity. For instance, by assessing the balance of important outcome prognostic factors across different treatment groups in a randomized controlled trial, one can assess whether there might be imperfect randomization. This aspect is pivotal when trials are with specific design (such as clusters randomized) or of small sample size (such as in cancerology, where complex algorithms are often used to determine the treatment assignment for each patient based on their characteristics).^3,4 In pragmatic trials, detailed patient characteristic data are also strongly needed to account for adherence and drop-out, especially when the aim is to estimate per-protocol treatment effects or to handle missing data.⁵ Likewise, in observational studies, assessing the balance of exposure and non-exposure groups after propensity score-based stratification or matching, for instance, requires extensive data on patient characteristics.⁶ In systematic reviews and evidence synthesis, when the eligible studies collect and report data on a common set of patient characteristics, the assessment of the target population (factor P in the PICO criteria) across studies is facilitated. A more insightful evaluation of the heterogeneity observed among trial results is also possible.^7,8 Recently, novel methods for causally interpretable meta-analysis have been proposed.^8–11 These frameworks also rely on having a rich set of (prognostic) patient characteristics collected across individual studies.

Despite its importance in practice, the collection and reporting of patient characteristic data remain inconsistent and suboptimal. Cahan et al (2017) recently showed that among 186,941 trials on ClinicalTrials.gov, only 8.9% reported baseline participant measures, and up to 85% of those measures were reported only once in the entire registry.¹² The lack of adequate reporting of important prognostic factors was also highlighted by Wertli et al. (2013), when they assessed 84 low back pain trials and found that almost half of them incompletely reported variables that are of prognostic importance, even with easily obtainable variables such as age or comorbidities.¹³ Similar issues are also prevalent in many other medical fields, including asthma, diabetes, hypertension, or colorectal cancer.^14–18

In these recent years, significant efforts have been made to standardize the collection and reporting of patient characteristics in epidemiological research. Across many therapeutic areas, a so-called core patient characteristic set (CPCS) is specifically developed to identify all key prognostic factors that should be commonly collected and reported (among studies and databases evaluating a target medical condition), while keeping the additional burden on the implementation acceptable (Figure 1). Beyond the variables proposed in the core set, researchers are free to measure and report additional patient characteristics that are of relevance to their topic. This CPCS concept is inspired by (and hence closely related to) the concept of core outcome set (COS) proposed in clinical research.¹⁹ However, while the methodology for COS development is increasingly enriched in the literature, little attention has been given so far to CPCS and how to develop it in practice.

Figure 1.

Core patient characteristic set (CPCS) in epidemiological research. The arrow from one box to the next reflects the generation and synthesis of clinical evidence in research practice. A CPCS could be particularly helpful in multiple steps.

In this paper, we aim to describe the methodology of studies establishing a core set of patient characteristics that should be commonly measured and reported in epidemiological studies and/or in large medical cohorts. By shedding light on current practice and challenges in CPCS development, this review could pave the way for future recommendations and guidelines on methodological standards of CPCS, thus enhancing the adoption of this concept in epidemiological research.

Methods

Study design

We conduct a methodological systematic review conforming to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 statement.²⁰

Eligibility criteria

We include primary studies or study protocols aiming to establish a core set of patient characteristics that should be commonly measured and reported in epidemiological studies and/or databases of a pre-specified medical condition, published between 01/01/2001 and 11/08/2022. We exclude studies that establish patient characteristics sets for other purposes such as to guide therapeutic decision-making in clinical practice. Conference abstracts, editorials, commentaries, letters to the editor, non-English publications and articles without full-text accessibility are also excluded from our review.

Search strategy

A structured search in the PubMed database is undertaken by P.H.T.T on 12/08/2022. The full search strategy is available in Appendix S1. This search strategy is first developed by two reviewers (P.H.T.T and K.L.D), then further optimized by a senior researcher (T.T.V) and a librarian specialized in epidemiological systematic reviews. We also manually screen the reference lists of the eligible articles to identify additional eligible studies.

Study selection

The search results are downloaded into Endnote and imported into Rayyan web-based software.²¹ Duplicates are removed by the duplicate search function in Endnote and by manually reviewing the records list. Three reviewers (P.H.T.T, K.L.D, M.L.V) and one research assistant independently screen titles and abstracts of retrieved records to select eligible papers based on the inclusion criteria. Each researcher screens 25% of the total number of records and double-checks 20% of the work of another researcher. Disagreements are resolved by discussion among the four researchers, and a senior researcher (T.T.V) is consulted if needed.

Data extraction and assessment

The data extraction form is constructed by M.L.V and P.H.T.T, pilot-tested and refined by K.L.D and T.T.V (Appendix S2). Our data extraction form is inspired by another form previously developed by Boulkedid et al (2011), who described the reporting of Delphi consensus method in developing healthcare quality indicators.²² We adapt this form by following recent methodological recommendations for Delphi studies.^22,23 Besides, we develop addtional items to extract data of other non-Delphi studies. Data extraction is performed by M.L.V, P.H.T.T and K.L.D. Each reviewer extracts 33% and double-checks 33% of the total number of records. Any discrepancy is resolved by discussing among the three reviewers.

The following information is collected from the eligible studies: (1) publication year, (2) target medical conditions, (3) purposes of the developed CPCS (to use in epidemiological studies or in registry settings), (4) study design (consensus-reaching or non-consensus methods), and (5) geographical scope of the study (international or national-wide).

We then evaluate the methodological and reporting quality of eligible studies in detail. For Delphi studies, the following characteristics are additionally collected: (1) study participants (number, response rate, types, selection criteria of participants, and whether authors report how representativeness of participants is ensured); (2) type of Delphi, i.e. traditional Delphi (which only involves asking questions to experts via questionnaires or interviews), or modified Delphi (which includes extra meeting rounds for experts’ interaction)^24,25; (3) method to establish the primary list of items before Delphi rounds; (4.1) questionnaire round characteristics: number of rounds, purpose of each round, questions formulation (rating scale or open questions), whether the rating scale (if used) is well-defined (i.e. the number and the meaning of levels in the scales are specified), whether the questionnaire’s content is publicly available and is piloted in advance, summary information sent to respondents after each round, and methods used to encourage participants to complete the questionnaires; (4.2) characteristics of in-person meetings or teleconferences (for modified Delphi studies): number of meetings and purposes, form of rating scale (if used) and whether the rating scale is well-defined, whether participants from questionnaire rounds are all invited to the meetings or only selectively, the timing of meetings and whether new items are allowed to be added between questionnaire and/or meeting rounds; and (5) how consensus is defined and attained, and how the Delphi process is terminated. Due to the lack of a standardized, validated quality assessment tool for Delphi studies, we evaluate these studies by using the checklist proposed by Diamond et al (2014).²³ Although this tool has not been validated, it serves as a reasonable initial approach for examining the conduct and reporting of Delphi studies in the absence of other metrics.²³ Four items in the tool include (i) the reproducibility of criteria for participant selection and whether (ii) the number of Delphi rounds, (iii) the criteria for dropping items at each round and (iv) the criteria to stop the Delphi process are stated and prespecified. The number of items satisfied in each Delphi study is then reported as a quality score. Three reviewers (M.L.V, P.H.T.T and K.L.D) independently assess the quality of all Delphi studies by this tool and reach final consensus.

When methods other than Delphi are used to establish the CPCS (i.e., the so-called non-Delphi methods), we describe the study design and extract detailed information on the number and type of experts participating in the CPCS development, method to establish the primary and final list of items, and how to attain consensus between participants when it is required.

Finally, we extract details of the obtained CPCS. These include (1) whether a description of item flow is reported, (2) whether only the final set or also intermediate results are reported, (3) whether the items in the final set are ranked and how, (4) the number of items in the final set, (5) whether the definition and measurement methods of included items are provided, and (6) the domains of items in the CPCS (demographic, clinical, patient history, socioeconomic, or healthcare setting factors).

Data synthesis

Continuous variables are presented as median and interquartile range. Categorical variables are summarized with frequencies and percentages. To investigate the content pattern of the final lists of items across eligible studies, we perform a hierarchical, complete-linkage clustering analysis.²⁶ For this, we first calculate the percentage of the five domains in each CPCS. The domain profiles are then used to calculate the matrix of between-study Euclidean distances. The final result is visualized by a tree-structure graphic.

Data analysis is performed using Microsoft Excel 365 and R version 4.1.1.

Results

Study selection

The PRISMA flow diagram summarizing the screening process is presented in Figure 2. Of all 5819 references identified, 23 articles meet the inclusion criteria. All 23 articles are primary studies and none is study protocol.

Figure 2.

Study selection PRISMA flowchart. PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

General characteristics

The general characteristics of included studies^27–45 are provided in Table 1. Among 23 eligible studies, 73.9% (n = 17) develop a CPCS specifically for epidemiological studies, 21.7% (n = 5) develop a CPCS for healthcare registries, and one study (4.3%) develops a CPCS that could be used for both registries and epidemiological studies. About 91% of the studies (n = 21) are published after 2011, and 78.3% (n = 18) have an international scope. Regarding the methodology, 87.0% of the studies (n = 20) consider a consensus reaching method to develop the core set, with Delphi being the most frequently used technique (69.6%, n = 16). Methods that are not based on consensus reaching include systematic review (8.7%, n = 2) and conceptual analysis (4.3%, n = 1).

Table 1.

General characteristics of eligible studies (N = 23).

Study characteristics	N	%
Publication year
- Before 2011	2	8.7
- 2011-2015	5	21.7
- 2016-2022	16	69.6
Target medical conditions
- Circulatory system	5	21.7
- Oncology	4	17.4
- Pediatrics	3	13.0
- Musculoskeletal system and connective tissue	2	8.7
- General symptoms and signs	2	8.7
- Medical intervention: Laparoscopic hysterectomy	2	8.7
- Others^a	5	21.7
Purposes: Improving patient characteristics reporting in
- Epidemiological studies	17	73.9
• Clinical trials	8	34.8
• Both clinical trials and observational studies	9	39.1
- Registries or healthcare programs	5	21.7
- Both registry and epidemiological studies	1	4.3
Study designs
- Consensus-reaching methods	20	87.0
• Traditional Delphi	8	34.8
• Modified Delphi	8	34.8
• Non-Delphi	4	17.4
- Non-consensus-reaching methods	3	13.0
• Systematic review	2	8.7
• Conceptual analysis	1	4.3
Geographic scopes of study participants
- International	18	78.3
- National-wide	1	4.3
- Not reported	4	17.4

^aIncluding burns, chronic fatigue syndrome, rehabilitation, hemophilia, and substance use disorder.

Methodological characteristics of Delphi studies

The methodological characteristics of 16 eligible Delphi studies are provided in Table 2 and Appendix S3. Remarkably, almost all studies involve healthcare professionals (93.8%, n = 15) or researchers (81.3%, n = 13), whereas only one study (6.3%) involves patients or patient representatives. The criteria for selecting participants are quite diverse across studies, but most commonly based on scientific renown and/or expertise level (56.3%, n = 9). Although the acceptance rate in these eligible studies is relatively low (median of 30 participants versus 62 invitations), only 37.5% of the studies (n = 6) report how they ensure the representativeness of participants.

Table 2.

Methodological characteristics of studies using Delphi consensus approaches (N = 16).

Methodological characteristics	N	%
1. Participants
Type of stakeholders^a
• Healthcare professionals	15	93.8
• Researchers, including clinical trialists, epidemiologists, statisticians, and/or public health experts	13	81.3
• Data or registry/program managers	3	18.8
• Policy makers	2	12.5
• Insurance experts	2	12.5
• Patient representatives	1	6.3
The study reports how representativeness of participants is ensured	6	37.5
2. Delphi rounds
2.1. Questionnaire rounds (in all 16 Delphi studies)
Number of rounds, median [IQR]	2.5 [2–4]
Question formulation
• Rating scale for each item	16	100.0
• Binary scale	3	18.8
• Likert scale	13	81.3
• Unclear scale format	1	6.3
• Open questions in addition to rating scale	10	62.5
Rating scale/score well defined	14	87.5
2.2. In-person/teleconference meeting rounds (in 8 modified Delphi studies)
Number of rounds, median [IQR]	2 [1–3]
Rating scale formulation	7	43.8
• Binary scale	5	31.3
• Six-point Likert scale	1	6.3
• Unclear scale format	1	6.3
• Rating scale/score well defined	4	25.0
3. Consensus definition and attainment
Criteria for selecting/dropping items at each round based on
• (i) Pre-defined cut-off(s) of % of participants voting certain rating level(s)	9	56.3
• (ii) Pre-defined cut-off(s) of a median score on a rating scale	1	6.3
• Both (i) and (ii)	4	25.0
• Unclear/not reported	2	12.5
Reason to terminate the Delphi process
• After completing the number of rounds prespecified	14	87.5
• When consensus is reached	1	6.3
• Unclear	1	6.3

^aEach study may be classified in more than one category; IQR: interquartile range.

Across all studies, rating scales are used to judge the importance of items during the questionnaire rounds (100%, n = 16). These scales range from two-point to ten-point, with five-point scales being the most commonly used (31.3%, n = 5). The scale is deemed as well-defined in 87.5% of the studies (n = 14). Apart from item rating, open-ended questions are also included in 62.5% of the studies (n = 10), mostly to collect qualitative feedback from participants (62.5%, n = 10). Besides, 43.8% of the studies (n = 7) report the use of a specific method to encourage participants to complete the questionnaires (e.g., by sending them reminders or vouchers).

In eight modified Delphi studies, in-person meetings or teleconferences are additionally organized. The median number of meetings is two (IQR 1–3). The aims of these meetings are (i) to have further discussions among participants before rerating the existing items (37.5%, n = 6), (ii) to add new items (6.3%, n = 1), and (iii) to review the final results (6.3%, n = 1). The rating scales used in these meeting rounds are mainly binary scales (31.3%, n = 5), and are well-defined in four studies (25.0%). Meetings are scheduled at different timepoints, either before (6.3%, n = 1), in between (18.8%, n = 3), or after the questionnaire rounds (25%, n = 4).

Finally, 12.5% of the studies (n = 2) do not report the criteria for selecting or dropping an item (Appendix S3). In 87.5% of the studies (n = 14), the Delphi process is terminated when the preplanned rounds are completed, regardless of the stability of responses or whether consensus has been obtained for all items. In one study, the reason for termination is unclear. As stopping the Delphi not based on response stability or consensus is deemed as suboptimal,²³ all studies are penalized for this in the subsequent quality assessment. More precisely, 56.3% (n = 9) of the studies have a quality score of three, and 43.8% (n = 7) have a quality score of one or two, on the four-point quality assessment tool proposed by Diamond et al. (2014)²³ (Appendix S3).

Methodological characteristics of non-Delphi studies

The methodological characteristics of seven non-Delphi studies are provided in Table 3. In general, only one study (14.3%) reports the types of stakeholders participating in the construction of the CPCS, and no studies report the number nor the proportion of different types of stakeholders. Similarly, no studies report the criteria for selecting/dropping each item, nor how consensus is reached after each round and at the end.

Table 3.

Non-Delphi methods to construct core patient characteristic sets (CPCSs).

No	Study	Study design	Settings	Participant characteristics	Methodology
Consensus-reaching studies
1	Jones 2018	NDC	Core data elements to promote interoperability across registries, clinical care and trials of medical devices for peripheral vascular intervention (PVI)	Types of stakeholders- Healthcare professionals- Researchers- Policy makers- Health information technology vendors- Study device manufacturersNumber and proportion of each type of stakeholders- Not reported	Method to establish the primary list of items- A review of existing registry data forms and of case report forms from medical device supplying companies is conducted by a working group, to identify a draft list of itemsMethod to reach consensus- Interactive web conferences and face-to-face meetings are organized for the stakeholders to reach consensus on each item- The included items are ranked for priority based on their availability in existing data sources and on their applicability in PVI clinical studiesCriteria for selecting/dropping items and consensus attainment- Not reported
2	Storrow 2012	NDC	Reporting guideline for studies evaluating acute heart failure syndromes	Types of stakeholdersNumber and proportion of each type of stakeholders:- Not reported	Method to establish the primary list of items- A literature review of existing clinical guidelines is conducted by a working group, to identify a draft list of itemsMethod to reach consensus- At in-person meetings, each item in the draft list is discussed among the stakeholders to be incoporated as a core measure, supplemental measure, or to be dropped without further consideration
					- External experts from two specialized organizations are invited to review the draft guideline
					- The working group finalizes the guideline before publication
					Criteria for selecting/dropping items and consensus attainment
					- Not reported
3	Kwakkel 2017	NDC	Core measurement set in stroke rehabilitation and recovery trials	Types of stakeholders- Stakeholders are international multidisciplinary stroke experts from a medical association	Method to establish primary list of items- A literature review on clinical predictors of stroke recovery is conducted. The predictors identified from this review are discussed in the next step
				- However, no further details of these stakeholders are providedNumber and proportion of each type of stakeholders- 60 experts	Method to reach consensus- Data are presented to the working group at a meeting. Participants then provide their feedback and discussions after the meeting- No formal consensus approach is employed due to the shortness of time
				- Proportion of stakeholder types: not reported	Criteria for selecting/dropping items and consensus attainment- Not reported
4	Jason 2012	NDC	Minimum data elements in chronic fatigue syndrome research reports	Types of stakeholders- Unclear	Not reported
				Number and proportion of each type of stakeholders
				- Not reported
Non consensus studies
5	Driessen 2016	SR	Minimum set of potential case-mix variables for studies in laparoscopic hysterectomy	Not applicable	Method to establish the CPCS- A systematic review is conducted to identify patient characteristics that influence the surgical outcomes of laparoscopic hysterectomy
					- These characteristics are then classified into three groups based on the strength of the evidence (assessed by the Newcastle-Ottawa Quality Assessment Scale)
6	Osooli 2015	SR	Minimum set of patient characteristics that predicts the development of inhibitory antibodies of blood clotting factors in hemophilia in hemophilia registries	Not applicable	Method to establish the CPCS- The authors conduct a systematic review of registry-based studies to identify potential predictors of the clinical outcomes of interest- The authors then recommend a minimum list of variables that should be collected in future hemophilia registries. The exact steps to establish this list are unclear
7	Meyer 2020	CA	Reporting standards of patients’ characteristics in rehabilitation trials	Types of stakeholders- Authors and participants of a conference in rehabilitation methodology	Method to establish the CPCS- The main author conducts a conceptual analysis to identify a draft list of core patient characteristics that should be reported in a rehabilitation trial. The exact steps in this analysis are unclear
				Number and proportion of stakeholders- Not reported	- Results of this analysis are then revised multiple times, following the feedback of other co-authors and of participants of a conference in rehabilitation methodology

NDC: non-Delphi consensus; SR: systematic review; CA: conceptual analysis.

Characteristics of the final lists of patient characteristics

The reporting of results and characteristics of the final CPCSs are provided in Table 4 and Figure 3. Almost all studies (91.3%, n = 21) report the final CPCS. A CPCS developed for registries often have more items than a CPCS developed for epidemiological studies (26 [10-31] vs 17 [10-23]) (Table 4). Most CPCSs contain demographic factors (e.g., age, gender, race), clinical factors (e.g., disease severity, presence of a symptom, laboratory test), and patient history factors (e.g., lifestyle, comorbidities, family history). In contrast, socioeconomic factors (e.g., level of education) and healthcare settings factors (e.g., standard inpatient care, ambulatory or intensive care) are often absent in most final lists (Figure 3).

Table 4.

The reporting of results among all eligible studies (N = 23).

Characteristics	Target of the CPCS, N (%)		All studies N (%)
Characteristics	Epidemiological studies (N = 17)	Registries (N = 6)	(N = 23)
The final CPCS reported	16 (94.1)	5 (83.3)	21 (91.3)
Number of items in the CPCS, median [IQR]	17 [10–23]	26 [10–31]	17 [10–25]
Priority of items in the CPCS is dermined
• Based on level of consensus	3 (17.6)	1 (16.7)	4 (17.4)
• Based on level of detail/complexity	3 (17.6)	1 (16.7)	4 (17.4)
• Based on level of evidence	1 (5.9)	—	1 (4.3)
• Unclear	2 (11.8)	—	2 (8.7)
• Not considered	8 (47.1)	4 (66.7)	12 (52.2)
Items in the CPCS defined	10 (58.8)	4 (66.7)	14 (60.9)
Measurement methods of complex items (e.g. lab tests, quality of life, etc.) specified	2 (11.8)	4 (66.7)	6 (26.1)

IQR: interquartile range; N: number of studies.

Figure 3.

Hierarchical clustering of 21/23 CPCS based on five variable domains, namely [1] Demographic factors (age, gender, race), [2] Clinical factors (e.g., disease severity, signs and symptoms, laboratory test), [3] Patient history factors (e.g., lifestyle factors, comorbidities, family history), [4] Socioeconomic factors (e.g., level of education, income, occupation), [5] Healthcare setting factors (e.g., standard inpatient care, ambulatory or intensive care). Each slice of the chart represents one CPCS. The sectors in each chart indicate what type of variables are included in each CPCS, with the area of each sector corresponds to the proportion of each variable type within one CPCS. For instance, the CPCS developed by Khalil et al. (2019) consists of two variable domains: demographic factors and patient history factors, which make up 25% and 75% of the CPCS, respectively. The blue lines starting from the center of the chart define how the tools are divided into the six clusters. Clusters #3 and #4, and #5 and #6 are grouped as sub-nodes of two major nodes, meaning that the tools in these sub-nodes have more similar domain profile compared to the tools in other clusters.

Items included are defined in 60.9% of the CPCSs (n = 14). Besides, 26.1% (n = 6) of the CPCSs have specific recommendations on the measurement of complex items, which are variables that must be measured by a subjective or complex tool such as quality of life, lab tests, etc. (Table 4).

Discussion

The call for better patient characteristics collection and reporting in epidemiological research is not new. The Consolidated Standards of Reporting Trials (CONSORT) 2010 statement is one of the first initiatives aiming to improve the reporting of trials, including the selection criteria (item 4a) and the description of the resulting samples (item 15).⁴⁶ A table showing baseline demographic and clinical characteristics for each treatment group, including the baseline measurement of the outcome, is required.⁴⁷ However, the CONSORT statement provides no further indication of which patient characteristics to report. Extensions of the CONSORT statement specify that information on socioeconomic variables should be added, and that all relevant prognostic variables should be reported, but only one CONSORT extension explicitly asks to include comorbidity.^48,49 Another initiative is the Food and Drug Administration Amendments Act (FDAAA) mandates, which require all covered studies to report results (including participants’ age, gender, race or ethnicity, and the baseline measures of the primary outcome) within 1 year of completion.⁵⁰

Constructing core patient characteristics sets is increasingly considered as a new method to further improve the collection and reporting of patient characteristics. Most CPCSs are developed within the last 10 years, not only to improve internal and external validity of epidemiological studies, but also to increase the quality of patient characteristics data in registries.^44,45 This is essential because registries are becoming important data sources for recent epidemiological research.

In this review, we identify many different methods to construct a CPCS. Among these methods, consensus-reaching techniques such as Delphi survey are the most frequently used. Indeed, Delphi is one of the ideal methods to collect expert-based judgements when the available knowledge is incomplete, which is often the case in CPCS or core outcome set development.⁵¹

Most Delphi studies in our review do not include patients as stakeholders. This is probably because CPCS development requires specialized knowledge on prognostic factors of a certain disease. Therefore, involving patients would bring little benefit to the process. However, embracing patients’ perspective on certain variables in the final set could be helpful, especially when these variables are private information of patients such as socioeconomic status, income, family history, etc. Methods for patient engagement have recently been proposed for core outcome sets, which could be further adjusted for the development of CPCSs.^52,53 Besides, many CPCS studies do not report how the representativeness of participants is ensured. Such information is important to determine the quality of the obtained CPCS and its uptake, hence should be better reported in future practice.

Our review has identified a wide range of consensus definitions employed by Delphi studies, with the most common definitions based on the pre-defined cut-offs of percentage of participants voting certain rating levels. This is in line with findings from previous reviews.^23,54 Earlier studies also acknowledged the difficulty of ascertaining the validity of consensus definitions, and there has been no specific guidance on methods to define consensus, which could explain the observed variability in our study.²³ However, the minimum standard is to report carefully how consensus is defined and achieved throughout the process. This is not satisfied by one-sixth of eligible studies, which renders these studies susceptible to bias and arbitrariness during data collection, analysis, and interpretation.⁵⁴

Most of the studies stop the Delphi process after completing a pre-specified number of rounds, regardless of the consensus attainment status. Considering the scarcity and/or divergence of evidence for each item, perfect consensus for 100% of items may not be achievable. Indeed, it has been shown that the evidence of many prognostic variables greatly suffers from a high risk of publication bias, selective reporting biases, poor statistical analyses, and so forth⁵⁵ To compromise on this issue, many CPCS studies group items into different sets with different priorities (based on level of evidence and/or consensus), so that researchers will also be informed about the quality of the variables in the final set. On the other hand, it is important to update the CPCS over time when further evidence for new (and current) prognostic factors are available in the literature.

Regarding non-Delphi studies, the reporting quality is relatively weak. Many important factors such as characteristics of study participants, method to establish the final list and consensus attainment are often not reported. This raises concerns about the rigor of the CPCSs obtained from these studies.

Our review also provides many important remarks on the final core sets across studies. First, while demographic, clinical and patient history factors are dominant in all final sets, socioeconomic and healthcare setting factors are often overlooked. This is suboptimal. Indeed, the socioeconomic gradient in health is ubiquitous and has been described across pathologies, in life expectancy, and mortality.^56–58 Meanwhile, describing the healthcare setting is important to assess the applicability of any epidemiological findings in practice. Thus, these factors are as important as other clinical factors often included in the CPCSs.

Second, the number of (final) items in CPCSs for registries is often higher than that of in CPCSs for epidemiological studies. This could be because registries are of large scale and have more (financial and human) resources for data collection than in traditional epidemiological research.⁴⁴ The disparity between CPCSs for registries and for epidemiological studies, however, could pose a challenge to the interoperability between these two settings, and to the adoption of a CPCS developed in one setting in the other within one medical field.

Finally, apart from a list of important patient characteristics to collect and report, many CPCSs also provide recommendations on the measurement methods for complex, or subjective items. Doing so could further reduce the heterogeneity and inconsistency in the data collection. However, when the recommended measurement method is uncommon or costly, the applicability of such method in practice could be undermined. These practical concerns should be considered when making recommendations on the measuremennt of items in the CPCS.

It is important to acknowledge some limitations of our study. First, given the already large number of records that we identify from Pubmed, we decide not to search a second database. Besides, we limit the eligibility criteria to articles published in English. Therefore, appropriate studies that are not indexed on Pubmed or not published in English might be missed. Second, the great difference between the number of records identified from the literature and the number of eligible studies may arise from the fact that the specificity and coverage of our search strategy are not optimal. Such a challenge stems from the fact that there is no standardized terminology for CPCS, as opposed to core outcome set. We mitigate the above-mentionned issues by consulting a librarian specialized in epidemiological systematic reviews to optimize the search strategy, and by manually searching for additional eligible studies from the reference list of identified eligible studies. Finally, we are not able to conduct a formal quality assessment for Delphi studies or for CPCS studies in general, because specific tools for this purpose are not yet available in the literature.

Conclusion

The methodological systematic review has identified deficiencies in the implementation and reporting of CPCS studies. A conducting and reporting guideline for CPCS studies is thus neccessary to further enhance the quality of CPCSs, and to promote the adoption of this concept in epidemiological research.

Supplemental Material

Supplemental Material - Development of minimum reporting sets of patient characteristics in epidemiological research: A methodological systematic review

Supplemental Material for Development of minimum reporting sets of patient characteristics in epidemiological research: A methodological systematic review by My Luong Vuong, Pham Hien Trang Tu, Khanh Linh Duong and Tat-Thang Vo in Research Methods in Medicine & Health Sciences

Supplemental Material

Supplemental Material - Development of minimum reporting sets of patient characteristics in epidemiological research: A methodological systematic review

Supplemental Material

Supplemental Material - Development of minimum reporting sets of patient characteristics in epidemiological research: A methodological systematic review

Supplemental Material

Supplemental Material - Development of minimum reporting sets of patient characteristics in epidemiological research: A methodological systematic review

Supplemental Material

Supplemental Material - Development of minimum reporting sets of patient characteristics in epidemiological research: A methodological systematic review

Footnotes

Acknowledgments

We would like to thank Dr Sonja Swanson, University of Pittsburgh for her valuable feedback on the design of this review. We also thank Dr Sherry Morgan, Holman Biotech Commons library, University of Pennsylvania for her great support in optimizing the search strategy of this review. We also greatly appreciate the help of Thi Huyen Trang Nguyen, Hanoi University of Pharmacy, as a research assistant in the stage of article screening of this review.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethics approval

Our study results are based on previously published data, none of which is stated as research involving human or animal subjects. Ethical approval is thus not required.

Registration and protocol

This review includes no clinical studies and therefore no protocol is pre-registered.

ORCID iD

My Luong Vuong

Data availability Statement

All data extracted for the systematic review and R code are made available as supplementary information (Appendix S4 & S5).

Supplemental Material

Supplemental material for this article is available online.

References

Dekkers

Algra

, et al. How to assess the external validity of therapeutic trials: a conceptual approach. Int J Epidemiol 2009; 39(1): 89–94.

Lesko

Ackerman

Webster-Clark

, et al. Target validity: bringing treatment of external validity in line with internal validity. Current Epidemiology Reports 2020; 7(3): 117–124.

Dickinson

Hosokawa

Waxmonsky

, et al. The problem of imbalance in cluster randomized trials and the benefits of covariate constrained randomization. Fam Pract 2021; 38(3): 368–371.

Altman

Bland

. Treatment allocation by minimisation. BMJ 2005; 330(7495): 843.

Murray

Swanson

Young

, et al. Guidelines for estimating causal effects in pragmatic randomized trials. 2019. arXiv preprint arXiv:1911.06030.

Hartz

Marsh

. Methodologic Issues in Observational Studies 2003; 413: 33–42.

Debray

Moons

van Valkenhoef

, et al. Get real in individual participant data (IPD) meta-analysis: a review of the methodology. Res Synth Methods 2015; 6(4): 293–309.

T-T

Porcher

Chaimani

, et al. A novel approach for identifying and addressing case-mix heterogeneity in individual participant data meta-analysis. Res Synth Methods 2019; 10(4): 582–596.

Dahabreh

Petito

Robertson

, et al. Toward Causally Interpretable Meta-analysis: Transporting Inferences from Multiple Randomized Trials to a New Target Population. Epidemiology 2020; 31(3): 334–344.

10.

Manski

. Toward credible patient-centered meta-analysis. Epidemiology 2020; 31(3): 345–352.

11.

Sobel

Madigan

Wang

. Causal inference for meta-analysis and multi-level data structures, with application to randomized studies of vioxx. Psychometrika 2017; 82(2): 459–474.

12.

Cahan

Anand

. Second thoughts on the final rule: An analysis of baseline participant characteristics reports on ClinicalTrials.gov. PLoS One 2017; 12: e0185886.

13.

Wertli

Schöb

Brunner

, et al. Incomplete reporting of baseline characteristics in clinical trials: an analysis of randomized controlled trials and systematic reviews involving patients with chronic low back pain. PLoS One 2013; 8(3): e58512.

14.

Hemmingsen

Lund

Gluud

, et al. Intensive glycaemic control for patients with type 2 diabetes: systematic review with meta-analysis and trial sequential analysis of randomised clinical trials. BMJ 2011; 343: d6898.

15.

Sorbye

Köhne

Sargent

, et al. Patient characteristics and stratification in medical treatment studies for metastatic colorectal cancer: a proposal for standardization of patient characteristic reporting and stratification. Ann Oncol 2007; 18(10): 1666–1672.

16.

van Boven

de Jong

Braunstahl

G-J

, et al. A meta-analysis of baseline characteristics in trials on mite allergen avoidance in asthmatics: room for improvement. Clin Transl Allergy 2020; 10(1): 2.

17.

van de Laar

Akkermans

van Binsbergen

. Limited evidence for effects of diet for type 2 diabetes from systematic reviews. Eur J Clin Nutr 2007; 61(8): 929–937.

18.

Zhang

Kang

, et al. The external validity of randomized controlled trials of hypertension within China: from the perspective of sample representation. PLoS One 2013; 8(12): e82324.

19.

Williamson

Altman

Bagley

, et al. The COMET Handbook: version 1.0. Trials 2017; 18(Suppl 3): 280.

20.

Page

McKenzie

Bossuyt

, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021; 372: n71.

21.

Ouzzani

Hammady

Fedorowicz

, et al. Rayyan-a web and mobile app for systematic reviews. Syst Rev 2016; 5(1): 210.

22.

Boulkedid

Abdoul

Loustau

, et al. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One 2011; 6(6): e20476.

23.

Diamond

Grant

Feldman

, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol 2014; 67(4): 401–409.

24.

Dalkey

Helmer

. An experimental application of the DELPHI method to the use of experts. Manag Sci 1963; 9(3): 458–467.

25.

Eubank

Mohtadi

Lafave

, et al. Using the modified Delphi method to establish clinical consensus for the diagnosis and treatment of patients with rotator cuff pathology. BMC Med Res Methodol 2016; 16(1): 56.

26.

Superchi

González

Solà

, et al. Tools used to assess the quality of peer review reports: a methodological systematic review. BMC Med Res Methodol 2019; 19(1): 48.

27.

Ahmadi

Alipour

Mohammadi

, et al. Development a minimum data set of the information management system for burns. Burns 2015; 41(5): 1092–1099.

28.

Damhuis

Bloomfield

Khalil

, et al. A core outcome set and minimum reporting set for intervention studies in growth restriction in the NEwbOrN: the COSNEON study. Pediatr Res 2021; 89(6): 1380–1385.

29.

Driessen

Sandberg

la Chapelle

, et al. Case-mix variables and predictors for outcomes of laparoscopic hysterectomy: a systematic review. J Minim Invasive Gynecol 2016; 23(3): 317–330.

30.

Goey

KKH

Sørbye

Glimelius

, et al. Consensus statement on essential patient characteristics in systemic treatment trials for metastatic colorectal cancer: Supported by the ARCAD Group. Eur J Cancer 2018; 100: 35–45.

31.

Jason

Unger

Dimitrakoff

, et al. Minimum data elements for research reports on CFS. Brain Behav Immun 2012; 26(3): 401–406.

32.

Jones

Krucoff

Morales

, et al. Registry assessment of peripheral interventional devices (RAPID): Registry assessment of peripheral interventional devices core data elements. J Vasc Surg 2018; 67(2): 637–44.e30.

33.

Khalil

Gordijn

Beune

, et al. Essential variables for reporting research studies on fetal growth restriction: a Delphi consensus. Ultrasound Obstet Gynecol 2019; 53(5): 609–614.

34.

Klimstra

Modlin

Adsay

, et al. Pathology reporting of neuroendocrine tumors: application of the Delphic consensus process to the development of a minimum pathology data set. Am J Surg Pathol 2010; 34(3): 300–313.

35.

Kwakkel

Lannin

Borschmann

, et al. Standardized measurement of sensorimotor recovery in stroke trials: Consensus-based core recommendations from the Stroke Recovery and Rehabilitation Roundtable. Int J Stroke 2017; 12(5): 451–461.

36.

Lux

Osborne

. A proposal for case definitions and outcome measures in studies of infantile spasms and west syndrome: consensus statement of the west delphi group. Epilepsia 2004; 45(11): 1416–1428.

37.

Meyer

Selb

Kiekens

, et al. Toward better reporting standards of patients' characteristics in rehabilitation trials: applying a new conceptual framework to current standards. Am J Phys Med Rehabil 2020; 99(3): 216–223.

38.

Osooli

Berntorp

. Inhibitors in haemophilia: what have we learned from registries? A systematic review. J Intern Med 2015; 277(1): 1–15.

39.

Rio

Mc Auliffe

Kuipers

, et al. ICON PART-T 2019-International Scientific Tendinopathy Symposium Consensus: recommended standards for reporting participant characteristics in tendinopathy research (PART-T). Br J Sports Med 2020; 54(11): 627–630.

40.

Spronk

PER

Begum

Vishwanath

, et al. Toward international harmonization of breast implant registries: international collaboration of breast registry activities global common data set. Plast Reconstr Surg 2020; 146(2): 255–267.

41.

Storrow

Lindsell

Collins

, et al. Standardized reporting criteria for studies evaluating suspected acute heart failure syndromes in the emergency department. J Am Coll Cardiol 2012; 60(9): 822–832.

42.

Ter Veer

van Rijssen

Besselink

, et al. Consensus statement on mandatory measurements in pancreatic cancer trials (COMM-PACT) for systemic treatment of unresectable disease. Lancet Oncol 2018; 19(3): e151–e160.

43.

Wildi

Hensel

Wertli

, et al. Relevant baseline characteristics for describing patients with knee osteoarthritis: results from a Delphi survey. BMC Muscoskel Disord 2013; 14: 369.

44.

Behrendt

Bertges

Eldrup

, et al. International Consortium of Vascular Registries Consensus Recommendations for Peripheral Revascularisation Registry Data Collection. Eur J Vasc Endovasc Surg 2018; 56(2): 217–237.

45.

Cadilhac

Bagot

Demaerschalk

, et al. Establishment of an internationally agreed minimum data set for acute telestroke. J Telemed Telecare 2021; 27(9): 582–589.

46.

CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med. 2010;152(11):726-732.

47.

Schulz

Altman

Moher

. CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomised trials. J Clin Epidemiol 2010; 63(8): 834–840.

48.

Zhai

, et al. CONSORT extension for reporting N-of-1 trials for traditional Chinese medicine (CENT for TCM) : Recommendations, explanation and elaboration. Compl Ther Med 2019; 46: 180–188.

49.

Improving the Reporting of Clinical Trials of Infertility Treatments (IMPRINT): modifying the CONSORT statement. Fertil Steril. 2014;102(4):952-959.e15.

50.

Liberopoulos

Trikalinos

Ioannidis

. The elderly were under-represented in osteoarthritis clinical trials. J Clin Epidemiol 2009; 62(11): 1218–1223.

51.

Niederberger

Spranger

. Delphi technique in health sciences: a map. Front Public Health 2020; 8.

52.

Beyer

MacLennan

Moris

, et al. The key role of patient involvement in the development of core outcome sets in prostate cancer. Eur Urol Focus 2021; 7(5): 943–946.

53.

Vanderhout

Smith

Pallone

, et al. Patient and family engagement in the development of core outcome sets for two rare chronic diseases in children. Research Involvement and Engagement 2021; 7(1): 66.

54.

Jünger

Payne

Brine

, et al. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med 2017; 31(8): 684–706.

55.

Riley

Hayden

Steyerberg

, et al. Prognosis research strategy (PROGRESS) 2: prognostic factor research. PLoS Med 2013; 10(2): e1001380.

56.

Braveman

Cubbin

Egerter

, et al. Socioeconomic status in health research: one size does not fit all. JAMA 2005; 294(22): 2879–2888.

57.

Kaplan

Keil

. Socioeconomic factors and cardiovascular disease: a review of the literature. Circulation 1993; 88(4 Pt 1): 1973–1998.

58.

Khalatbari-Soltani

Cumming

Delpierre

, et al. Importance of collecting data on socioeconomic determinants from the early stage of the COVID-19 outbreak onwards. J Epidemiol Community Health 2020; 74(8): 620–623.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.07 MB

0.19 MB

0.04 MB

0.16 MB