Abstract
Loss to follow-up or patient attrition is common in longitudinal studies of traumatic brain injury (TBI). Lack of understanding exists between the relation of study design and patient attrition. This review aimed to identify features of study design that are associated with attrition. We extended the analysis of a previous systematic review on missing data in 195 TBI studies using the Glasgow Outcome Scale Extended (GOSE) as an outcome measure. Studies that did not report attrition or had heterogeneous methodology were excluded, leaving 148 studies. Logistic regression found seven of the 14 design features studied to be associated with patient attrition. Four features were associated with an increase in attrition: greater follow-up frequency (odds ratio [OR]: 1.2, 95% confidence interval [CI]: 1.0–1.3), single rather than multi-center design (OR: 1.6, 95% CI: 1.2–2.2), enrollment of exclusively mild TBI patients (OR: 2.8, 95% CI: 1.6–4.9), and collection of the GOS by post or telephone without face-to-face contact (OR: 1.6, 95% CI:1.1–2.4). Conversely, two features were associated with a reduction in attrition: recruitment in an acute care setting defined as the ward or intensive care unit (OR: 0.58, 95% CI: 0.47–0.72) and a greater duration of time between injury and follow-up (OR: 0.93, 95% CI: 0.88–0.99). This review highlights design features that are associated with attrition and could be considered when planning for patient retention. Further work is needed to establish the mechanisms between the observed associations and potential remedies.
Introduction
Longitudinal studies commonly experience patient attrition, defined as the loss of patients to follow-up and therefore the absence of outcome data. Patient attrition reduces power and can potentially introduce bias. 1 –3 Traumatic brain injury (TBI) is recognized as a chronic and, at times, progressive disease and relies on long-term follow-up to monitor outcomes, 4 such as those described by the Glasgow Outcome Score Extended (GOSE). 5
Missing outcome data can arise because of multiple reasons, such as the patient lacking motivation to participate in follow-up or being unable to complete questionnaires because of a language barrier. Most commonly, however, missing outcome data occur because of patient loss to follow-up whereby it is impossible to contact the patient and determine his or her outcome. Despite the best efforts of researchers, a proportion of patients are almost always lost to follow-up. 6 –9 Various techniques are available to mitigate the effects of missing outcome data. We previously published guidance on their use in longitudinal studies of TBI. 8 Prevention of missing data, however, will always remain superior to compensatory statistical techniques.
Attrition rates can be influenced by both patient specific and study design factors. Previous studies have explored patient factors associated with loss to follow-up in TBI and identified low socioeconomic and ethnic minority status, education, injury severity, and substance abuse as risk factors. 6,7,9 –11 Because these patient groups are prevalent within the TBI population, excluding them to maximize retention would limit the generalizability of the findings from such studies.
Careful study design therefore remains one of the main and few options available to TBI investigators to potentially modify and minimize attrition. Study design factors that may influence attrition include the extent of patient involvement and methods of follow-up. A well designed and executed study method can reduce loss to follow-up; however, the association between study design factors and attrition has not yet been explored specifically in TBI research. 12,13 This review sought to identify study design factors associated with attrition in patients with TBI.
Methods
Study selection and data extraction
This review builds on the analysis presented in a previous review on missing data.
8
Both reviews were conducted and reported in line with the Cochrane guidelines for methodological reviews and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement for systematic reviews.
14,15
The protocol for the initial systematic review was registered on PROSPERO and is accessible online (
The methods for literature search and data extraction were presented previously. 8 In short, we included English language studies on TBI published between January 1, 2012 and October 27, 2017, using either the GOS or GOSE as an outcome measure, and measured at least three months after injury. Studies with predominantly adult populations (defined as less than 25% of patients under 16 years of age), recruited within two months of initial injury, of at least 100 patients, and of purely traumatic etiology were included. Case-control studies, case reports, and case series were excluded, because by definition, these study designs do not have missing outcome data. In addition, incomplete publications were excluded because insufficient data were available for extraction.
The previous review examined the reporting and handling of missing outcome data in 195 studies and found that 46 studies did not report whether any patients were lost to follow-up. Studies were excluded when it was unclear whether loss to follow-up occurred or that combined multiple heterogeneous study designs. Data were extracted independently by two reviewers on piloted forms, with disagreements being resolved through discussion and consensus with a third reviewer.
The following variables were extracted (1–3) or derived (4,5) and subsequently used to calculate attrition. Variables 2–5 were extracted at the final follow-up time point of each study.
Number of patients initially enrolled in the study. Two patterns of reporting were observed. Reporting data on all patients eligible for inclusion regardless of whether follow-up was achieved were described as the inclusive approach. Alternatively, reporting only on patients who were successfully followed up was described as the exclusive approach. We regarded patients excluded because of loss to follow-up as missing and added these patients to the total number of patients initially enrolled.
Number of patients successfully followed up at the end of the study. This number includes patients who are known to have died before follow-up, because their GOSE is available (i.e., GOSE = 1).
Number of patients lost to follow-up. This was calculated as the difference between the number of patients initially enrolled and those successfully followed up.
The percentage of patient attrition. Attrition (%) = (number of patients lost to follow-up)/(number of patients initially recruited) * 100.
The odds of attrition. Odds of attrition = (number of patients lost to follow-up)/(number of patients successfully followed up). In the event of multiple follow-up time points, the final time point was used to calculate attrition.
We extracted features of the study design that were frequently and consistently described in most studies, considered to potentially influence follow-up, and were therefore modifiable by the investigators.
Interventional versus observational study design. Interventional studies, or clinical trials, assigned an intervention not part of standard care to a treatment group and compared this with a control. Observational studies included retrospective or prospective cohort studies, observed the effect of a naturally occurring exposure, and compared this with a non-exposed control.
Percentage of male patients.
Age. Age in each study was recorded as the mean (or alternatively median) age.
Study duration in months. Duration was defined as the time between initial injury and the last follow-up time point in the study.
Number of discrete follow-up time points after the initial recruitment.
Initial injury severity as per the Glasgow Coma Scale (GCS) score. This was classified as “mild” when the study included only patients with GCS score ≥13, “moderate” if GCS score 8–12, “severe” if GCS score <8, “moderate to severe” if GCS score 3–12, and “mixed” if all GCS categories were included.
Military population. Studies were considered military if they were conducted on military personnel, both active and veterans, or in a military-run institution.
Single versus multi-center design. This was based on the number of participating institutions.
Development status of recruiting country, as measured by the Human Development Index (HDI) for the year 2017. 17 This composite index takes into consideration the life expectancy, education level, and gross national income of a country and is used by the United Nations to measure a country's development. 18 When multiple international settings were used, the mean HDI was calculated.
Recruitment setting. The setting or place of recruitment was classified as “hyperacute” if patients were recruited in the pre-hospital setting or emergency department, “acute” if patients were recruited as hospital inpatients on wards or the intensive care unit, and “subacute” if patients were recruited in outpatient clinics or rehabilitation facilities.
Non-routine clinical engagement. Depending on the study objective, some studies perform investigations and measures outside standard clinical practice. Given the additional effort or engagement required of patients, we looked at its influence on follow-up. Non-routine clinical engagement was defined as non-routine steps taken at the time of recruitment or follow-up and performed for the purposes of the study alone. Non-routine clinical engagement steps were classified as interventional, patient tissue sampling, radiological investigations, neurocognitive tests, and questionnaires (such as patient surveys).
Interventions were defined as per interventional studies and included all clinical trials. Patient tissue sample included blood, cerebrospinal fluid, and saliva taken for non-clinical purposes such as novel biomarker measurements and genotyping. Radiological investigations were considered non-routine when they would not be indicated on clinical grounds alone and included magnetic resonance imaging (MRI) and positron emission tomography (PET) scans. Neurocognitive tests included psychiatric evaluations and functional neurological assessments above standard assessments and outcome measures. Questionnaires included surveys completed by the patients at home or over the telephone.
The use of GOSE as primary or secondary outcome measure. We recorded this when it was stated explicitly. In other studies, where a variety of outcome variables were collected, the variable of most importance to the analysis in the manuscript was determined as the primary outcome measure, with all others viewed as secondary. In the event of a composite outcome, this was treated as a secondary outcome measure unless stated by the authors as a primary outcome.
Retrospective or prospective collection of outcome data. This describes the timing of outcome data collection relative to the time of study conduct. Retrospective studies identified eligible patients and began data collection after the outcome had already occurred and used pre-existing data databases or patient notes. Attrition occurred where no outcome data were reported on a patient included in the data source. Data were considered prospectively collected when stated so by authors.
The use of a time period or window in which collected patient outcomes were compiled. This was compared with follow-up at a pre-specified or discrete time point.
Method of GOSE assessment that included in person, by telephone, mail, or a combination. Follow-up was defined as “exclusively in person” if follow-up occurred face-to-face in all patients. If outcomes were acquired in person for at least some of the patients, this was classified as “partially in person.” The use of only telephone or mail was defined as “not in person.”
Whether a proxy such as a relative or carer was allowed to report on outcome, if the patient was unable or unavailable.
Statistical analysis
Modeling attrition
Data were analyzed in R version 3.6.1. 19 The outcome was modeled as “logit of patients lost to follow-up” (i.e., the log odds of patient attrition by the end of the study) using a logistic regression model. The rationale for modeling proportions in that manner has been described elsewhere. 20 Notably, it accounts for the fact that some studies are larger than others, and the number of patients lost can be no less than 0 and no more than the number of patients recruited initially. The model included all design factors listed above, except for military population (because only two of 148 studies were military) and proxy report of outcome (which was undocumented in 73% of studies).
Using the generalized variance-inflation factor corrected by the number of degrees of freedom with a cut-off of two (analogous to a conventional variance-inflation factor of four), 21 none of the variables raised concern for multi-collinearity. Missing values were imputed using multiple imputation (see below) to allow inclusion of all 148 studies. Model fit was assessed using the McFadden Pseudo-R-squared on each of the 20 imputed datasets. Statistical significance was determined by 95% confidence intervals (CIs). As a sensitivity analysis, the model was repeated using only the 104 studies with complete reporting.
Multiple imputation
The majority of studies had some missing data because of incomplete reporting. Of 148 studies, data were complete for 33 studies. All other studies were missing at least one value (maximum five values missing per study) with a total of 191 missing values (see Results). We handled missing data as per a previously published framework. 8 A comparison of studies with and without complete reporting suggested no evidence against the hypothesis that data on design factors were missing completely at random (Little MCAR test p = 0.74, package BaylorEdPsych 0.5). There were no red flags suggesting data were missing not at random.
We used multiple imputation to create and analyze 20 imputed datasets (package mice 3.1.0). Incomplete variables were imputed using the package default settings, with the exception of continuous variables for which we used predictive mean matching with distance aided selection of donors. Adequacy of imputations was confirmed by the presence of convergence and plausibility on strip and density plots.
Results
A total of 148 studies from the original 195 TBI studies were included (Fig. 1). Studies followed patients for a median duration of six months from injury (range 3–120 months) and scheduled a median of one follow-up time point (range 1–5 time points). The median patient attrition by the end of the study was 9% but varied widely from 0 to 88%. A summary of included studies is shown in Table 1.
PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart of studies included in this review. GOS, Glasgow Outcome Scale; TBI, traumatic brain injury.
Characteristics of Included Studies
IQR, interquartile range; GOSE, Glasgow Outcome Scale Extended.
The percentages describe the proportion of studies within a category when the variable was reported on. IQR = Interquartile range. Design factors which were not reported were subsequently imputed for the purposes of the model.
The association between patient attrition and study design was modeled using logistic regression. The model fit the data well according to the McFadden Pseudo-R-squared (median 0.52, interquartile range 0.48–0.50 across 20 imputed datasets). Note that values of pseudo-R2 tend to be considerably lower than those of ordinary R-squared, such that values of 0.20–0.40 already “represent an excellent fit.” 22
Results of the regression pooled from 20 imputed datasets are summarized in Table 2 and illustrated in Figures 2 and 3. Results of the sensitivity analysis using only the 104 studies with complete reporting are similar to those of the primary analysis, bar smaller CIs, and therefore more results that reached statistical significance (Supplementary Table S1).
Illustration of the association between study design factors and patient attrition, based on 148 studies of traumatic brain injury using a logistic regression model. Top row: Attrition based on recruitment setting and injury severity. The three panels represent the three settings in which patients were recruited: in the pre-hospital setting or emergency department (Hyperacute), as hospital inpatients on wards or intensive care units (Acute), in outpatient clinics or rehabilitation facilities (Subacute). The lines represent studies recruiting patients of different injury severities. Note that the lines for “moderate & severe” patients and “severe only” patients overlap. Bottom row: Attrition based on recruiting centers and method of Glasgow Outcome Scale Extended (GOSE) collection. The three panels represent the three methods of collection for the GOSE): only face-to-face (Exclusively in person), some patients being interviewed face-to-face with others being contacted by telephone or mail (Partially in person), use of only telephone and/or mail (Not in person).
Illustration of the association between study design factors and patient attrition, based on 148 studies of traumatic brain injury using a logistic regression model. The y-axis shows the odds of attrition based on time since injury and total number of follow-up time points. Time since injury refers to the total study duration—i.e., the time between injury and the final follow-up. Thus, two studies may run for the same duration (x-axis), but one study may follow patients up more frequently than the other (lines and symbols).
The Effect of Study Design Factors on the Odds of Patient Attrition by the End of the Study Based on 148 Studies of Traumatic Brain Injury
GOS, Glasgow Outcome Scale.
Attrition was modeled using logistic regression and results were pooled across 20 multiply imputed datasets. The column “Significant” highlights where the confidence intervals do not include 1.0.
Four factors significantly increased the odds of attrition: a greater follow-up frequency (odds ratio [OR] 1.16, 95% CI 1.04–1.29), a single-center as opposed to multi-center design (OR 1.61, CI 1.18–2.2), studying exclusively mild TBI patients compared with all severities (OR 2.78, CI 1.58–4.91), and collecting GOSE data by mail or telephone but not in person, compared with methods that included a face-to-face assessment (OR 1.61, CI 1.1–2.38).
Two factors significantly decreased the odds of attrition: a greater study duration between injury and final follow-up time point (OR 0.93 per six-months interval, CI 0.88–0.99) and recruiting among hospital inpatients rather than in the pre-hospital setting or emergency department (OR 0.58, CI 0.47–0.72).
Discussion
Building on a previously published dataset of longitudinal TBI studies, 8 this review explored which study design factors are associated with attrition in longitudinal studies of patients after TBI. The features investigated were based on frequently reported and modifiable variables that were hypothesized to influence attrition. Patient related factors affecting follow-up have been explored previously in other chronic disease areas including infectious diseases, 23 genetic counseling, 24 and psychiatric conditions 25 where loss to follow-up is common. For example, study site, treatment setting, living situation, and employment status are just some factors associated with follow-up in psychiatric research. 25
Unsurprisingly, increasing frequency of follow-up within a study was associated with greater odds of attrition or overall dropout. Asking patients to participate in multiple follow-up visits was associated with an increased likelihood of their missing at least one. The reasons for dropping out or missing a follow-up event were not explored but may be related to patients tiring of the repeated demands on their time.
The use of multiple follow-up points, however, is not uncommon in TBI research because of the chronicity of the disease. Outcomes can change for both better or worse, sometimes years after the initial injury, and there is no clear consensus on the optimal timing of outcome assessment. 26 In studies with multiple follow-up points, the missing data handling technique of last outcome carried forward is used frequently to minimize missing data and therefore optimize total outcome data. The limitations of this technique have been debated previously because it assumes the absent data have remained unchanged with time. 27
In the case of studies with multiple time points, the final time point was used to calculate overall attrition for that study, with attrition often being highest at that time. Therefore, one would assume, the longer the duration of a study, the higher the dropout rate. Within our dataset, however, the opposite was found. Across all studies, the longer the duration, the lower the odds of attrition. To clarify this relation further, imagine two studies that followed patients for the same period—for example, one year. It is the study with the lower follow-up frequency during that year that experienced lower patient attrition.
In contrast, take two studies with the same number of follow-up time points. It is the study with the greater duration between the injury and final follow-up time point that showed better patient retention. These findings suggest investigators may yield more complete outcome data by focusing on a single and later follow-up time point. A limitation of this analysis is that few studies considered follow-up beyond two years, and hence attrition at truly long follow-up time points could not be analyzed.
This seemingly paradoxical relation may be explained by a number of putative reasons— for example, the increasing preponderance of patients who have died during the study period and are therefore known to be GOS 1. Patients with TBI have chronically increased death from all causes compared with non-TBI trauma matched controls, and it is often easier to establish the death of a patient from death registries than the functional status of those who are still alive. 28
Alternatively, early follow-up runs the risk of clashing with clinical commitments from other injuries or patients may be unable to attend early on because of on-going TBI rehabilitation. Similarly, most patients improve functionally with time and have a greater capacity to engage with follow-up after an increased time after injury. Further, patients are often advised that symptoms will continue to change and often improve for at least one year after injury. Persistence of symptoms beyond this period may prompt them to engage with follow-up more than previously.
The majority of the reasons described, however, assume a relatively high degree of disease severity and on-going symptom burden. A more plausible option is likely to be the presence of an unmeasured confounder such as greater funding and therefore greater efforts for follow-up in studies of longer duration. Regardless of the cause, the effect size is small and unlikely to change patient retention substantially in planning of future studies.
Mild TBI was associated with poorer follow-up relative to all severities of TBI, consistent with previous studies in TBI. 7 This may, at least in part, be secondary to asymptomatic patients not perceiving any relevance of attending follow-up, and the nature of their brief encounter in the emergency department means they do not have the same investment in returning. The lack of representation of patients with mild TBI will undoubtedly bias outcome data, and while high attrition in this patient group is likely to correlate with good outcomes, 29 it should not be assumed that patients who do not attend follow-up are therefore doing well.
It might be expected that single centers would be better at retaining patients at follow-up. Our results, however, suggest studies involving single centers have poorer patient retention compared with studies involving multiple centers. This again may reflect confounders involved in multi-center studies such as increased funding and level of planning invested to perform such studies. Multi-center studies are also more likely to include randomized controlled trials or large-scale observational studies with increased research resources and personnel dedicated to perform follow-up.
Unsurprisingly, the method of GOSE collection was found to influence attrition. Studies that did not review patients in person at follow-up had higher attrition rates compared with studies that involved at least some face-to-face assessment. These studies relied on the techniques of telephone assessment, mail questionnaires, or a combination to assess outcome. While the overall effort required for both patient and researcher is less for these techniques, the patient is likely to gain less from the experience compared with an in-person assessment and therefore may be less inclined to engage.
Again, an unmeasured confounder such as study funding may also influence attrition rate in this context. Questionnaires are perceived as easy for researchers and less expensive than reviewing a patient in person. Unfortunately, techniques related to outcome assessment after TBI are highly variable in both the assessment tool used and also the method of assessment, 30 and consensus is required to minimize the potential bias that varying techniques could introduce.
We also observed better follow-up in studies recruiting in the acute setting (hospital wards and intensive care units) compared with those recruiting in the hyperacute setting (in the pre-hospital setting or emergency department), especially among observational studies. This may be the result of different techniques of consent employed in these two settings. In the hyperacute setting, patients may be too unstable or unconscious to give consent and relatives may not yet be around to provide proxy consent. In addition, time is tight, and research is frequently given lower priority in such settings.
In contrast, in the acute setting, such as the hospital ward, the patient may have regained capacity to consent or may have relatives available who are aware of the patient's attitudes toward research and provide proxy consent. Studies recruiting in the acute setting and beyond may inadvertently pre-select a cohort of patients who are more research interested. Further, the majority of patients with TBI recruited in the emergency department (hyperacute setting) would not be admitted to hospital. In contrast, admitted patients (acute setting) will be more likely to have persistent symptoms, receive care they may remember, and had time to build a relationship with the research and clinical team and are thus likely more invested into the research project.
Our result thus likely reflects an underlying selection bias. This highlights the need for keeping a recruitment log or registry of all patients approached for the study, including those who declined. Comparing patients who do return to follow-up with those who were recruited but dropped out as well as those who were never recruited, could then provide an excellent guide on the generalizability of study results.
This has both strengths and limitations. Strengths include the sample size which, with 148 studies and more than 70,000 patients, is comparatively large, as well as the careful handling of missing data. This review was based on an earlier review that used exclusively GOSE as an outcome measure for TBI, but multiple outcome scales for TBI exist, which themselves could influence attrition rate differently from GOSE. This is true for mild TBI where GOSE may lack the sensitivity to detect subtle symptomatology and may also explain the relatively low representation of mild TBI in our cohort despite this being the more prevalent severity subtype.
Further, the goal of this analysis was not to predict patient attrition, but to understand which study design factors are associated with it. Our model therefore serves to highlight which design factors should be considered when planning a TBI study, rather than a tool to calculate anticipated follow-up rates. Importantly, as with all observations, association does not equate to causation, and adopting a specific study design feature does not guarantee patient retention. In addition, there are aspects of the study design (such as site experience and patient incentives) that may contribute to successful follow-up, but that could not be extracted retrospectively from published manuscripts. These would be important to include in future prospective studies of patient attrition.
Finally, as well as the features described here, there are factors beyond the study design (thus beyond our model) that influence the success of follow-up. In our experience, a common but often unmeasurable cause of loss to follow-up and inability to perform specific outcome measures are extremes of functional outcome. Patients with good outcomes who recover completely may lose motivation to continue in research while those with very poor outcomes may be unable to return because of dependency on carers to attend or complete follow-up assessments. Efforts should be made to obtain follow-up data from all patients and in the event of its absence, determine the reasons for loss to follow-up.
Conclusions
We highlight study design features associated with loss to follow-up in longitudinal TBI studies. When planning a TBI study, we suggest considering how the study design may relate to patient attrition. Particular thought should be given to how often and when patients are asked to return, the method of outcome collection in the case of GOSE, and timing of patient recruitment. A balance between focusing all resources onto follow-up and answering the specific study question has to be managed carefully, however. This may require a registry of non-recruited eligible patients to understand how well the study population represents the general TBI population.
Footnotes
Funding Information
The work of HFL, AIRM, DN, EWS, DKM and LW was supported in the context of CENTER-TBI (
Author Disclosure Statement
Lindsay Wilson has provided advice to industry on outcomes assessment and received funds for this work administered by the University of Stirling. For the remaining authors, no competing financial interests exist.
Supplementary Material
Supplementary Table S1
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
