Sage Journals: Discover world-class research

Abstract

Introduction:

The trend toward requiring explicit consent from patients participating in observational research increases time and resources required to perform such research. Informed consent introduces the potential for “consent bias”—either through selection bias or through the “Hawthorne effect,” where patients may alter responses based upon the awareness of participation in a study, thus potentially limiting its applicability to a generalized orthopedic practice. We hypothesized that administering Quick Disabilities of the Arm, Shoulder, and Hand Questionnaire (QuickDASH) to patients on the day of surgery with informed consent would alter responses in a statistically and clinically meaningful way compared to patients who complete QuickDASH as a quality control measure.

Methods:

We previously instituted the QuickDASH questionnaire as the standard new patient intake and postoperative questionnaire for quality assurance purposes. We retrospectively reviewed data on a cohort of patients who underwent isolated carpal tunnel release (CTR) who had completed preoperative and postoperative QuickDASH forms without providing consent for study participation. Next, a cohort of patients scheduled to undergo isolated CTR who completed the intake questionnaire was approached on the day of surgery for consent to participate in the study. After obtaining consent but prior to surgery, these patients completed a second questionnaire and then completed a postoperative questionnaire on follow-up at a mean of 8 weeks postoperatively.

Results:

Thirty-nine patients and 35 patients were included in the retrospective and prospective cohorts, respectively. No significant differences were observed in age, gender, symptom duration, nerve conduction study/electromyography results, or disease severity between the two groups. We identified no statistically significant difference in preoperative or postoperative QuickDASH score between the retrospective and prospective cohorts (39.8 ± 22.7 vs. 39.7 ± 19.1 preoperatively; 27.3 ± 24.7 vs. 18.7 ± 13.3 postoperatively) or within the prospective cohort before and after obtaining informed consent.

Conclusion:

Informed consent did not significantly alter patient responses to the QuickDASH questionnaire. These results suggest that both “opt-in” and “opt-out” approaches to observational research in hand surgery provide results that may be applicable to a generalized orthopedic practice.

Clinical Relevance:

This study provides evidence that will inform the interpretation of observational research findings in hand surgery.

Keywords

elbow and shoulder general hand trauma

Introduction

Requiring explicit consent from patients participating in observational research increases time and resources required to perform such research. The requirement for explicit consent may introduce the potential for selection bias or “consent bias,” which is traditionally defined as differences in patient characteristics between consenters and non-consenters.¹ Several authors demonstrate that the informed consent process introduces a systemic bias where non-consenters and consenters have significant differences in the estimation of disease prevalence, disease severity, clinical parameters, and socioeconomic status.^2

–5 These authors argue that using an “opt-in” approach, where a patient must actively consent to participation in a study, versus opt-out approach, where patient data may be included in a study (i.e. database or registry) without explicit consent limits the applicability of a research result to a generalized population.

The informed consent process may also introduce another form of bias—one where the behavior of study participants changes based upon the awareness of participation in a study—this is known as the “Hawthorne effect.”⁶ To the best of our knowledge, no study in the hand surgery literature addresses bias introduced through the informed consent process, either as selection bias or as the Hawthorne effect. This study attempts to define the effect of informed consent on patient responses to a standard questionnaire in an orthopedic hand surgery practice.

A paucity of medical literature addresses the Hawthorne effect, and controversy exists as to whether the effect actually exists.⁷ McCarney et al. evaluated a cohort of patients with mild-to-moderate dementia enrolled in a placebo-controlled trial of ginkgo biloba.⁸ The investigators found a significantly improved measure of cognitive function at 6 months in patients who received intensive follow-up (baseline and 2-, 4-, and 6-month follow-up) compared to patients who received minimal follow-up (baseline and 6-month follow-up only). Conversely, Fernald et al. found no evidence of a Hawthorne effect in a study of patients with skin and soft tissue infections.⁹

We hypothesized that awareness of participation of study subjects in an observational study by informed consent will alter questionnaire responses compared to patients completing the questionnaire as a quality control measure. A difference in outcome would support the argument for use of an “opt-out” approach, whereas negative result would support the validity of either opt-in or opt-out approaches to observational research in hand surgery.

Methods

The University of Rochester’s Institutional Review Board reviewed and approved the research design.

Patient selection

Patients 18 years of age and older with a diagnosis of carpal tunnel syndrome (CTS) who underwent primary carpal tunnel release (CTR) were eligible for inclusion in the study. For inclusion, patients must have undergone electromyography/nerve conduction study (EMG/NCS) with findings consistent with isolated CTS. Patients undergoing revision surgery were excluded, as were patients with any other comorbid condition affecting the upper extremity. Data collected via chart review included age, gender, occupation, symptom duration, and electrodiagnostic findings. Patients with electrodiagnostic findings other than CTS were excluded.

For the retrospective cohort, we identified 39 patients who underwent isolated CTR between January and August 2011. All patients included in this study completed an intake and postoperative Quick Disabilities of the Arm, Shoulder, and Hand Questionnaire (QuickDASH) questionnaire. Twenty-five patients completed preoperative questionnaires on the day of surgery, and 14 patients completed the questionnaire at the initial visit (mean: 103 days preoperatively; range 10–366 days).

For the prospective cohort, 35 patients enrolled in the study on the day of surgery. We obtained informed consent using a consent form approved by the Institutional Review Board, which specified that the purpose of the study was to ascertain differences in questionnaire responses when made aware that they were participating in a study. None of the patients refused to participate and consented to this study. All patients had previously completed the intake form. Twenty-seven patients completed preoperative questionnaires on the day of surgery, and eight completed the questionnaire at the initial visit (mean: 97 days preoperatively; range 15–372 days). Immediately following the consent process, patients completed a second preoperative QuickDASH form, approximately 1 h after completing the initial questionnaire for those completing the intake questionnaire on the day of surgery. These patients subsequently completed a postoperative questionnaire either at the time of clinic follow-up or via telephone if the patient did not follow-up at a sufficient length of time. All follow-up data were completed at an average of 8–10 weeks postoperatively. Three patients were lost to follow-up. Figure 1 shows the schematic depiction of patient selection.

Figure 1.

Selection process.

Nerve conduction study/electromyography

Nerve conduction study results were available for all patients. All patients had isolated CTS. Six reports from outside institutions did not include the motor and sensory latency data in the retrospective cohort, and one patient in the prospective cohort did not have a report available as they worked as an EMG technician and was tested without a formal report. Exclusion of this patient’s data did not alter our results; therefore, we included this patient’s data in the final analysis.

CTS severity was graded based upon EMG/NCS findings as previously described.¹⁰ Patients with slowing of conduction (either sensory or motor) were grade 1, whereas slowing of both fiber types was assigned grade 2. Findings of thenar muscle denervation in the presence of slowed conduction represented grade 3. Evidence of diffuse but incomplete denervation was assigned grade 4, and complete absence of response was given grade 5.¹⁰

Outcomes questionnaire

The DASH questionnaire is a validated outcome measure used to compare pre and postsurgical level of function in patients with conditions affecting the upper extremity including CTS.^11

–16 The minimal clinically important difference (MCID) for the DASH questionnaire is previously established to be 10 points on a 0–100 scale, with 100 being the worst possible score.¹⁵ The QuickDASH questionnaire is a shortened version (11 questions) of the DASH questionnaire (30 questions) survey used to assess severity of symptoms and difficulty carrying out specific tasks with the upper extremity.¹⁶ The reliability, validity, and responsiveness of QuickDASH are comparable to that of the original DASH questionnaire and are commonly used due to relative ease of completion by patients.^17,18

We previously instituted the QuickDASH form as the standardized new patient intake questionnaire for all new and postoperative patients in the upper extremity practices of two surgeons prior to the initiation of this study without explicitly consenting to study participation. Due to variation in the protocol between surgeons, most patients from one practice received the QuickDASH at the time of the initial visit, and patients in another practice received the QuickDASH in the waiting room on the day of their surgery.

We made two primary comparisons. First, we compared the retrospective cohort intake (preoperative) and postoperative questionnaire responses to the post-consenting preoperative and postoperative questionnaire responses in the prospectively enrolled group. Second, we compared the pre-consent questionnaire responses to the questionnaire responses obtained immediately after obtaining informed consent to serve as an internal control study arm.

Statistical analysis

The primary outcome measure was different in QuickDASH score. Based upon the established MCID, power analysis determined that 32 patients per group were necessary to detect a 10-point difference in QuickDASH score, given alpha error = 0.05 and beta error = 0.8.

Statistical analysis was carried out using standard functions in Microsoft Excel (Redmond, Washington, USA). Categorical data were analyzed using the Fisher’s exact test. For continuous variables, samples were checked for normalcy of distribution. For this study, samples were not normally distributed; therefore, we used the Mann–Whitney test for statistical analysis of continuous variables. A biostatistician reviewed the experimental design, results, and analysis.

Results

Thirty-nine patients were included in the retrospective cohort, and 35 patients were prospectively enrolled. Table 1 provides a summary of demographic characteristics. No statistically significant differences were observed in the EMG/NCS findings or carpal tunnel severity score between the two cohorts.

Table 1.

Demographics and carpal tunnel characterization.

	Retrospective cohort (n = 39)	Prospective cohort (n = 35)	p Value^a
Age (SD)	55.5 (13.2)	56.5 (14.9)	0.76
Gender
Male	13 (33.3%)	14 (40.0%)	0.63
Insurance
Privately insured	25	21	0.82
Medicare	5	10	0.15
Medicaid	5	1	0.09
Workers compensation	4	3	0.70
Duration of symptoms (months)	31.3 (37.4)	32.1 (34.6)	0.49
Nerve conduction test results (SD)	n = 39	n = 34
Motor latency above normal (ms)	1.5 (2.0)	1.5 (1.9)	0.82
No motor response detected	1	0	>0.99
Sensory latency above normal (ms)	0.9 (1.3)	0.5(1.4)	0.15
No sensory response detected	9	10	0.61
EMG abnormal	11	11	0.80

Carpal tunnel severity^1 –5	2.5 (0.7)	2.5 (0.9)	0.92

EMG: electromyography; SD: standard deviation.

^a p Values were determined by Fisher’s exact test for categorical variables and Mann–Whitney test for continuous variables.

We compared the preoperative and postoperative QuickDASH scores of the retrospective (no consent) and prospective (consented) cohorts (Table 2). Postoperatively, patients in the retrospective cohort completed the questionnaire at 9.9 ± 4.0 weeks (range 4–17 weeks), and the prospective cohort completed the questionnaire at 8.0 ± 2.0 weeks (range 4–14 weeks; p = 0.06). Preoperatively, the retrospective and prospective groups had similar QuickDASH scores (p = 0.86). Postoperatively, mean QuickDASH differences were not significantly different between groups (p = 0.41).

Table 2.

Retrospective versus prospective (consented) cohorts’ QuickDASH response.

	Retrospective cohort	Prospective cohort (consented)	p Value^a
Preoperative QuickDASH	39.8 (22.7)^b	39.7 (19.1)	0.86
Postoperative QuickDASH	27.3 (24.7)	18.7 (13.3)	0.41

QuickDASH: Quick Disabilities of the Arm, Shoulder, and Hand Questionnaire.

^a p Values were determined by Mann–Whitney test.

^b Standard deviations are in parentheses.

Next, we compared the pre-consent and post-consent QuickDASH results within the prospective cohort (Table 3). Study subjects did not alter their questionnaire responses after being informed of and consenting to participation in a study (p = 0.9766).

Table 3.

Prospective cohort internal control.

Preoperative (pre-consent)	Preoperative (consented)	p Value^a
39.0 (17.8)^b	39.7 (19.1)	0.9766

^a p Values were determined by Mann–Whitney test.

^b Standard deviations are in parentheses.

Discussion

Our results do not indicate that the informed consent process introduces a systemic bias. The demographic characteristics, EMG/NCS findings, carpal tunnel severity, and symptom duration were all similar between the retrospective and prospective cohorts. QuickDASH responses did not vary between the retrospective and prospective cohort or within the prospective cohort using their intake questionnaire responses as an internal control. While there was a numerical difference of 8.6 points between the retrospective and the prospective groups postoperatively, this finding was neither statistically significant (p = 0.41) nor clinically significant based on the MCID of 10 points. It is possible, however, that although we powered the study to identify a 10-point difference in QuickDASH score, the study is underpowered to identify small differences and would, therefore, identify a statistically significant small difference if the cohort was larger. Regardless of statistical significance, a difference of this magnitude would not demonstrate clinically significant bias.

Our review of the literature revealed no comparable studies in the orthopedic or hand surgery literature. Our results are most comparable to Fernald et al.⁹ who did not identify a Hawthorne effect in their study of patients with skin and soft tissue infections. McCarney et al. did find that patients in a dementia study with more intensive follow-up had significantly higher cognition scores than those with minimal follow-up, suggesting that awareness of participation in a study may overstate treatment benefit.⁸ Our findings, although not statistically significant, may also represent an overestimation of treatment effect in the consented cohort compared to the retrospective (non-consented) cohort.

This study has several strengths. We performed a power analysis based upon the MCID for QuickDASH and exceeded the number needed to identify a clinically meaningful difference in questionnaire responses. Additionally, we were able to answer our question in two different ways. First, we compared a retrospective (un-consented) cohort to a prospectively consented cohort. Second, we used the prospective cohort as an internal control by comparing their intake questionnaire responses with responses after obtaining informed consent. We feel that if a clinically meaningful consent bias was present, this study would have identified such bias.

Although the aim of our study was not specifically to determine outcomes of patients after carpal tunnel surgery, the improvement in score reflects that of other CTS outcome studies.^12,19
–21

We chose the QuickDASH form as the standard patient intake questionnaire in our practice, because it is validated for a variety of conditions in the upper extremity, including CTS. While other questionnaires are more responsive to detecting changes in CTS (Brigham & Women’s Carpal Tunnel Questionnaire, Michigan Hand Questionnaire, and Carpal Tunnel Questionnaire),^11
–13 we feel that the QuickDASH score is an appropriate outcome questionnaire for a general upper extremity practice. Given the study design, using a different questionnaire for the study would be impractical.

We chose CTS as the model disease for our study because it is common; several validated outcome measures are reported in the literature including DASH, and a standard objective measure of the disease is available in the form of electrodiagnostic studies. We chose our minimum follow-up duration to be 6 weeks, because most patients who undergo uncomplicated CTR in our practice do not return for further follow-up. Using this time point ensures optimal participation in the study.

The study does have several weaknesses. The ideal study design to answer our question would involve a randomized controlled trial where an un-consented cohort is prospectively studied. However, institutional review boards are unlikely to approve such a study design. Next, although we desired follow-up at a single time point, for practical reasons, the follow-up duration was variable due to variation in patient follow-up preferences and compliance with completing the follow-up questionnaire. Second, the variability in timing of intake questionnaire completion (either at first visit or on the day of surgery) introduces the possibility that patients who completed both preoperative questionnaires on the same day could recall their responses to the initial questionnaire. However, we feel that if recall bias had a significant influence on our findings, then the first comparison (retrospective vs. prospective) would have shown a difference in parameters, and the internal control comparison would have been without any difference. Both these analyses failed to support the idea that recall bias was a significant factor in our study. Moreover, we had a 100% consent rate for our study. Therefore, our study cannot address the question of whether any identifiable patient factors vary between consenters and non-consenters (i.e. selection bias) alters questionnaire responses. Third, a possible limitation of the study could be the subtle but real potential that a patient’s willingness to participate in the study on the day of their surgery could be in fear of not upsetting or angering their surgeon. Finally, it is important to note that our results may not be generalized to other types of study enrollments using the QuickDASH. For example, in the setting of recent trauma, administering the QuickDASH at the time (or day) of surgery may (or may not) alter its validity.

One valid concern is that no specific comparison between our cohort sets was undertaken. We chose not to do mixed comparisons because retrospectively compared patients were fundamentally different than prospectively evaluated patients. One group allowed comparisons of patient responses to their own responses previously. That group might have had a recall bias. In the other group, patients were all naive, allowing comparisons with only the day of inquiry differing. However, there may be subtle differences in the extent of disease between the day of surgery group and the initial consultation group, not large enough to manifest in electrodiagnostics, which might have introduced a subtle bias of its own. It is for this reason that both experiments (retrospective and prospective) were implemented, because neither experiment was sufficient in its own. Comparing the results from both of these approaches did not yield any difference between cohorts within each experiment. Moreover, we informally lumped all patients together (the day of surgery prospectively and retrospectively vs. the day of consultation prospectively and retrospectively) and still found no difference. Perhaps this is not surprising, given the fact that neither experiment yielded a significant difference alone. We, therefore, do not believe there to be a very significant role for the timing of this outcomes measurement.

In summary, we did not identify a systematic bias introduced through the informed consent process that alters response to a standard outcome questionnaire (QuickDASH) in our hand surgery practice. Our findings provide support for comparable generalizability of studies using both an “opt-in” or “opt-out” approach to study patient recruitment, as long as the consent rate is high when an “opt-in” approach is used.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Junghans

Jones

. Consent bias in research: how to avoid it. Heart 2007; 93: 1024–1025.

Buckley

Murphy

Byrne

. Selection bias resulting from the requirement for prior consent in observational research: a community cohort of people with ischaemic heart disease. Heart 2007; 93: 1116–1120.

Jackson

Crossland

Dennis

. Assessing the impact of the requirement for explicit consent in a hospital-based stroke study. QJM 2008; 101: 281–289.

Junghans

Feder

Hemingway

. Recruiting patients to medical research: double blind randomised trial of “opt-in” versus “opt-out” strategies. BMJ 2005; 331: 940.

Mitchell

Kline

. Systematic bias introduced by the informed consent process in a diagnostic research study. Acad Emerg Med 2008; 15: 225–230.

Berthelot

Le Goff

Maugars

. The Hawthorne effect: stronger than the placebo effect? Joint Bone Spine 2011; 78: 335–336.

Kompier

. The “Hawthorne effect” is a myth, but what keeps the story going? Scand J Work Environ Health 2006; 32: 402–412.

McCarney

Warner

Iliffe

. The Hawthorne effect: a randomised, controlled trial. BMC Med Res Methodol 2007; 7: 30.

Fernald

Coombs

DeAlleaume

. An assessment of the Hawthorne effect in practice-based research. J Am Board Fam Med 2012; 25: 83–86.

10.

Elfar

Yaseen

Stern

. Individual finger sensibility in carpal tunnel syndrome. J Hand Surg 2010; 35: 1807–1812.

11.

Changulani

Okonkwo

Keswani

. Outcome evaluation measures for wrist and hand: which one to choose? Int Orthop 2008; 32: 1–6.

12.

Gay

Amadio

Johnson

. Comparative responsiveness of the disabilities of the arm, shoulder, and hand, the carpal tunnel questionnaire, and the SF-36 to clinical change after carpal tunnel release. J Hand Surg 2003; 28: 250–254.

13.

McMillan

Binhammer

. Which outcome measure is the best? evaluating responsiveness of the Disabilities of the Arm, Shoulder, and Hand Questionnaire, the Michigan Hand Questionnaire and the Patient-Specific Functional Scale following hand and wrist surgery. Hand (NY) 2009; 4: 311–318.

14.

SooHoo

McDonald

Seiler

JG 3 rd

. Evaluation of the construct validity of the DASH questionnaire by correlation to the SF-36. J Hand Surg Am 2002; 27: 537–541.

15.

Gummesson

Atroshi

Ekdahl

. The disabilities of the arm, shoulder and hand (DASH) outcome questionnaire: longitudinal construct validity and measuring self-rated health change after surgery. BMC Musculoskelet Disord 2003; 4: 11.

16.

Hudak

Amadio

Bombardier

. Development of an upper extremity outcome measure: the DASH (disabilities of the arm, shoulder and hand) [corrected]. The upper extremity collaborative group (UECG). Am J Ind Med 1996; 29: 602–608.

17.

Beaton

Wright

Katz

. Development of the QuickDASH: comparison of three item-reduction approaches. J Bone Joint Surg 2005; 87: 1038–1046.

18.

Gummesson

Ward

Atroshi

. The shortened disabilities of the arm, shoulder and hand questionnaire (QuickDASH): validity and reliability based on responses within the full-length DASH. BMC Musculoskelet Disord 2006; 7: 44.

19.

Hobby

Watts

Elliot

. Validity and responsiveness of the patient evaluation measure as an outcome measure for carpal tunnel syndrome. J Hand Surg Br 2005; 30: 350–354.

20.

Jenkins