Poor reporting quality of observational studies in children with non-syndromic cleft palate makes evidence synthesis difficult

Abstract

Objective

To assess the reporting quality of observational studies included in a systematic review of the management of chronic otitis media with effusion in children with non-syndromic cleft lip and palate using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist.

Methods

Medline, CINAHL, Scopus and Embase, were searched for studies comparing the use of ventilation tubes to surveillance. Two reviewers screened potential eligible articles, extracted data independently and assessed reporting quality using the STROBE checklist.

Results

The median compliance rate with individual STROBE items was low at 25% (range:0-100%) with 11 of the 22 items not reported in any of the studies. Items reported inconsistently pertained to potential sources of bias, sample size calculations, how loss to follow-up was addressed and management of missing data.

Conclusion

The development of this systematic review highlights the inadequate reporting standards in this field. Differences in the way the outcomes are defined, reported, and measured leads to variability in the observed intervention effects and difficulty in interpreting the true effect size. Future researchers are encouraged to use STROBE guidelines for the design and reporting of observational studies in this field.

Introduction

Cleft lip and palate are common congenital malformations that result in variable defects of the lip, soft and hard palate. ¹ Due to the abnormal insertion of the muscles of the soft palate, there is impaired formation and function of the eustachian tube which normally helps to ventilate the middle ear and drain any fluid. ² Thus, there is a strong association between cleft lip and palate and chronic otitis media with effusion (COME). This chronic inflammation of the middle ear develops within the first year of life which is the primary developmental period for speech and language.^3,4 COME can persist even after surgical repair of the lip and palate defect and, if not appropriately managed, can eventually lead to deafness. It is important that this condition is managed in a timely manner to ensure that the developmental milestones of these children are at par with their peers, as well as mitigating the associated costs to the public health system. ⁵

A recent systematic review conducted to determine the best management for chronic otitis media with effusion in children with non-syndromic cleft palate could not derive concrete conclusions regarding the most effective management. The review highlights the difficulties in evidence synthesis that are brought about by inconsistent and incomplete reporting of outcomes in observational studies, especially when reporting guidelines are not utilised.

Clinicians and policy makers rely on published evidence to help guide decision-making. ⁶ It is well documented in medical literature that vital information about how outcomes were measured and analysed is often missing or poorly reported. A lack of transparency in reporting study outcomes limits limits the synthesis and generalisability of results, and overall contributes to reporting biases. This has the potential to bias not only the conclusions of meta-analyses and systematic reviews, but also has far-reaching impact in the development of inaccurate clinical protocols. To minimise this, various reporting guidelines have been developed for different types of studies to facilitate consistent and reproducible methodology. One such guideline is the STRengthening the Reporting of OBservational studies in Epidemiology (STROBE) statement which was developed and published in 2015. ⁷ It comprises of a checklist of 22 items which relate to the title, abstract, introduction, methods, results and discussion sections of articles. The STROBE statement provides guidance to authors to facilitate the transparency and completeness of reporting in observational studies. Medical research is largely observational, and many clinical guidelines are based on evidence gleaned from observation studies ⁸ especially when randomised controlled trials cannot be conducted due to ethical or logistical considerations. The STROBE checklist is a useful tool to evaluate the reporting quality of the included studies in order to highlight the gaps and to facilitate more robust research in this field in the future.

Methods

The review was conducted in accordance to JBI methodology for systematic reviews of effectiveness ⁹ and the protocol was registered with PROSPERO (CRD42021255861). ⁵ The outcomes of interest in the review were hearing thresholds, speech and language development and otological complications and these were informed by a study that integrated the views of clinical and non-clinical stakeholders in developing core outcomes sets for children with otitis media with effusion and cleft palate. ¹⁰

Quality assessment of included studies

The studies were compared to the 22 item STROBE checklist by two independent reviewers. Each item was assessed with a response option of ‘present’ or ‘absent’ depending on whether the study included that checklist item, and any disagreements were reconciled by re-reading the study. Data processing was completed using Microsoft Excel 2016 (Microsoft Corporation, Redmond, WA, USA).

Results

Four articles were identified that fit the review’s inclusion criteria. All four included studies were cohort studies,^11–14 three of them retrospective and one prospective. The characteristics of the studies are summarised in Table 1.

Table 1.

Characteristics of included studies.

Author (year)	Country	Study design	Total participants n, intervention n, control	Age range	Cleft type classification	Intervention	Follow-up period	Outcomes measures
Robson (1992) 11	UK	Retrospective cohort	70 participants	6 months–12 years	Complete versus incomplete	Ventilation tubes observation	5 years	- Hearing
			38					- Speech/language
			32					- Retympanostomy rate
								- Presence of effusion
								- Otological complications
Ŝubarević (2015) 14	Serbia	Prospective cohort	90 participants	6 months–5 years	Not specified	Ventilation tubes observation	5 years	- Hearing
			45					- Speech/language
			45					- Tympanogram type
								- Presence of effusion
								- Otological complications
Hubbard (1985) 12	USA	Retrospective cohort	48 participants	5–11 years	Complete versus incomplete	Early ventilation tubes observation/Delayed ventilation tubes at >36 months	Not specified	- Hearing
			24					- Speech/language
			24					- Tympanogram type
								- Presence of effusion
								- Otological complications
Shaw (2003) 13	UK	Retrospective audit	72 participants	Not specified	LAHSHSAL 1	Ventilation tubes observation	Not specified	- Speech/language
			20
			52

Table 2 shows the proportion of studies that satisfied each check-list item on the STROBE checklist and the inter-observer agreement for each item. The inter-observer agreement (Cohen’s kappa) between the two reviewers varied between for 0.5 and 1.0 which suggests moderate to complete agreement.^7,17 The median compliance rate with individual STROBE items was very low at 25% (range: 0–100%). Eleven of the 22 items addressed in STROBE were not reported in any of the studies, which means no study achieved a high reporting quality label (greater than 15 items reported). These items pertained to reporting potential sources of bias, sample size calculations, sensitivity analyses, how loss to follow-up was addressed, management of missing data, discussion regarding external validity and explicit statements regarding sources of funding. The study by Hubbard et al. ¹² had the highest score with 12/22 STROBE items reported and Shaw et al. ¹³ had the lowest with 7/22 items reported.

Table 2.

Overall reporting quality of included studies using STROBE checklist (n = 4).

Item		Criteria to score as adequately reported	Proportion (%)	Kappa
Title and abstract
1(a)		Indicate the study’s design with a commonly used term in the title or the abstract	100	1.0
1(b)		Provide in the abstract an informative and balanced summary of what was done and what was found	100	1.0
Introduction
2	Background/rationale	Explain the scientific background and rationale for the investigation being reported	100	1.0
3	Objectives	State specific objectives, including any prespecified hypotheses	25	0.5
Methods
4	Study design	Present key elements of study design early in the paper	100	1.0
5	Setting	Describe the setting, locations, and relevant dates, including periods of recruitment, exposure, follow-up, and data collection	50	0.9
6(a)	Participants	Cohort study—Give the eligibility criteria, and the sources and methods of selection of participants. Describe methods of follow-up	75	0.7
6(b)		Cohort study—For matched studies, give matching criteria and number of exposed and unexposed	25	0.9
7	Variables	Clearly define all outcomes, exposures, predictors, potential confounders, and effect modifiers. Give diagnostic criteria, if applicable	25	0.8
8*	Data sources	For each variable of interest, give sources of data and details of methods of assessment (measurement)	100	1.0
9	Bias	Describe any efforts to address potential sources of bias	0	1.0
10	Study size	Explain how the study size was arrived at	0	1.0
11	Quantitative variables	Explain how quantitative variables were handled in the analyses. If applicable, describe which groupings were chosen, and why	50	0.9
12(a)	Statistical methods	Describe all statistical methods, including those used to control for confounding	75	0.5
12(b)		Describe any methods used to examine subgroups and interactions	0	1.0
12(c)		Explain how missing data were addressed	25	—
12 (d)		Cohort study—If applicable, explain how loss to follow-up was addressed	0	0.9
12(e)		Describe any sensitivity analyses	0	0.9
Results
13*(a)	Participants	Report the numbers of individuals at each stage of the study—e.g., numbers potentially eligible, examined for eligibility, confirmed eligible, included in the study, completing follow-up, and analysed	100	0.7
13*(b)		Give reasons for non-participation at each stage	25	1.0
13*(c)		Consider use of a flow diagram	0	1.0
14*(a)	Descriptive data	Give characteristics of study participants (e.g., demographic, clinical, social) and information on exposures and potential confounders	75	1.0
14*(b)		Indicate the number of participants with missing data for each variable of interest	25	0.8
14*(c)		Cohort study—summarise follow-up time (e.g., average and total amount)	50	1.0
15*	Outcome data	Cohort study—Report numbers of outcome events or summary measures over time	25	0.8
16(a)	Main results	Give unadjusted estimates and, if applicable, confounder-adjusted estimates and their precision (e.g., 95% confidence interval). Make clear which confounders were adjusted for and why they were included	0	1.0
16(b)		Report category boundaries when continuous variables were categorized	0	1.0
16(c)		If relevant, consider translating estimates of relative risk into absolute risk for a meaningful time period	0	0.8
17	Other analyses	Report other analyses done—e.g., analyses of subgroups and interactions, and sensitivity analyses	0	1.0
Discussion
18	Key results	Summarise key results with reference to study objectives	100	1.0
19	Limitations	Discuss limitations of the study, taking into account sources of potential bias or imprecision. Discuss both direction and magnitude of any potential bias	25	0.7
20	Interpretation	Give a cautious overall interpretation of results considering objectives, limitations, multiplicity of analyses, results from similar studies, and other relevant evidence	100	0.9
21	Generalisability	Discuss the generalisability (external validity) of the study results	0	1.0
Other information
22	Fundings	Give the source of funding and the role of the funders for the present study and, if applicable, for the original study on which the present article is based	0	1.0

More specifically, reporting in the title and abstract, and introduction section was completed satisfactorily in all studies with rates of 81% on average. Reporting within the methods and results sections were often unclear resulting in an increased risk of bias. The average reporting rate for the study methods section was 38% with minimal documentation on how missing data was addressed, reasons for non-participation or clear definitions of outcomes and potential confounders. Reporting of results was slightly better at 56%, but discussion of study limitations was notably lacking.

The handling of quantitative variables and use of appropriate statistics was only appropriately reported in two studies which significantly affected capacity for pooled analysis for the review, particularly for the three main outcomes of interest: hearing thresholds, speech and language development and otological complications.

The results of the evaluation against the STROBE checklist are similar to the results of critical appraisal using standardized instruments from JBI for observational studies whichw as included in teh review (see Table 3). ⁹ Hubbard et al. also had the highest score, satisfying 8 out of 11 questions and Shaw et al. had the lowest score at 2 out of 11 when using the JBI checklist.

Table 3.

Critical Appraisal of eligible cohort studies.

Study	Q1	Q2	Q3	Q4	Q5	Q6	Q7	Q8	Q9	Q10	Q11
Šubarević et al.	Y	Y	Y	U	U	N	Y	Y	N	N	Y
Shaw R et al.	Y	Y	N	N	N	U	U	U	N	N	U
Robson et al.	Y	Y	Y	U	U	Y	Y	Y	U	U	Y
Hubbard et al.	Y	Y	Y	Y	Y	N	Y	Y	N	N	Y

Y, yes; U, unclear; JBI appraisal checklist for cohort studies.

Q1 = Were the two groups similar and recruited from the same population?

Q2 = Were the exposures measured similarly to assign people to both exposed and unexposed groups?

Q3 = Was the exposure measured in a valid and reliable way?

Q4 = Were confounding factors identified?

Q5 = Were strategies to deal with confounding factors stated?

Q6 = Were the groups/participants free of the outcome at the start of the study (or at the moment of exposure)?

Q7 = Were the outcomes measured in a valid and reliable way?

Q8 = Was the follow up time reported and sufficient to be long enough for outcomes to occur?

Q9 = Was follow up complete, and if not, were the reasons to loss to follow up described and explored?

Q10 = Were strategies to address incomplete follow up utilized?

Q11 = Was appropriate statistical analysis used?

Discussion

Comparison against the STROBE checklist highlights that the reporting quality of the studies included in the review was insufficient in key methodological areas, including reporting potential sources of bias, reporting of outcome data and discussion of limitations. Furthermore, risk of bias assessment using GRADE (Grading of Recommendations, Assessment, Development, and Evaluation) found that the quality of the studies was considered low with a moderate to high risk of bias. ¹⁶ Thishad significant impact on the conclusions that could be derived for the three main outcomes of interest. Ultimately, there is a lack of good evidence to support the insertion of short-term ventilation tubes as the mainstay treatment or close surveillance as an alternative. Each of outcomes considered had several limitations that hindered evidence synthesis.

Hearing thresholds

Hearing thresholds are normally measured with an audiometer to determine the volume of sound that can be perceived by an individual in decibels over different frequencies. ¹⁷ It is commonly accepted that a measurement greater than 21 dB at any frequency is considered a hearing deficit. However, this ranged from 16-25 dB in different studies resulting instudy participants being considered as having hearing loss in some studies but normal hearing in others.

Furthermore, each study that was included in the systematic review reported their data in various formats. Most studies gave a mean with either a range, confidence interval or standard deviations. The calculations required to transform these values into uniform data for pooled analysis creates opportunities for errors that can be inaccurately interpreted as statistically significant. Unfortunately, none of the authors replied to requests for raw data, and this was likely exacerbated by the fact that some of the studies were over 30 years old and the data may no longer be available.^11,12 As a core outcome measure for this population group, consensus on what is considered hearing loss is vital. In addition, documentation of mean hearing thresholds as unadjusted estimates and with consistent measure of statistical precision i.e., confidence interval needs to be established to allow for accurate aggregation of data for meta-analysis.

Speech and language development

Speech and language development is another core outcome that had significant variability between studies. Each of the four included studies measured speech and language development differently. Two studies noted only the proportion of participants that needed speech pathologist input and how many sessions were required with no descriptions regarding the pathologies that were being assessed.^11,14 The other two studies considered nasal resonance and consonant articulation but each study used a different scale.^12,13 The lack of uniformity in the measurement of speech and language outcomes as well as the lack of quantitative data limited the systematic review to descriptive analysis of this outcome.

A validated tool for assessing cleft speech was developed in 2006 for use in inter-centre audit studies. It was noted to have moderate to high intra- and inter-examiner reliability and has been recommend for use in the United Kingdom and Ireland with potential extension to other English-speaking countries. ¹⁸ This tool is particularly useful as it also gives an indication of treatment needs and continuing burden of care from a health economics perspective. The use of a standardised and validated tool to measure speech and language development would not only be useful when assessing outcomes between intervention groups but also between international cohorts, allowing the results to be translatable across various populations.

Otological complications

Complications arising from each intervention are important for clinicians when recommending any treatment to a patient. The complication rates were reported in all four included studies. One study reported the proportion of the participants that had any complication between the intervention and comparator groups but did not specify what they considered as complications. ¹⁴ The remaining three studies gave a breakdown of each complication and the proportion of participants affected ^11,12,15 but there was no clear documentation about when these complications were noted within the study period. This could have significant impact on the results as some of these complications can self-resolve if given enough time.

Several common complications were noted amongst all studies that either contribute to permanent hearing loss like perforation and cholesteatoma, or can affect a child’s quality of life like persisting ear discharge. While it would be difficult to pick which of these complications is the most important or contributes the most significantly, agreement on how these complications are reported would allow for better evaluation of the true complication rate of each intervention and appropriate counselling for parents and caregivers.

Inconsistencies in reporting also extend to reporting requirements by journals. When considering the overall use of reporting guidelines in otolaryngology journals, uptake is less than 50%.

Conclusion

Although research quality is not directly related to the reporting quality of a study, poor reporting impedes on accurate assessment of the research quality. The development of this systematic review highlights the inadequate reporting standards for studies in this area. The differences in the way the outcomes are defined, reported, and measured between studies could lead to variability in the observed intervention effects, and difficulty in interpreting the true effect size.^7,19 This impacts on the capacity to aggregate and synthesize these studies and limits their overall generalisability. While the difficulties associated with translating research to clinical practice are multifactorial, poorly reported study outcomes are a significant contributing factor.^6–8

Given the constraints that can be associated with conducting randomised trials, observational studies offer useful insights in medical research and the development of clinical guidelines, but this can only be achieved when studies employ rigorous methodology in their design and are transparent in reporting of outcomes. One way to achieve this is through the use of reporting guidelines.