Sage Journals: Discover world-class research

Abstract

Background:

Lateral elbow tendinopathy, also known as “tennis elbow” or “lateral epicondylitis,” is a common disease leading to pain in the lateral side of the elbow and disability during hand gripping. A counterforce brace is one of the most conventional treatments. However, its effects on outcomes remain inconclusive.

Objectives:

To investigate the effects of counterforce braces on pain in subjects with lateral elbow tendinopathy. Grip strength was reviewed as a secondary outcome.

Study design:

Systematic review and meta-analysis of randomized controlled trials.

Methods:

PubMed, Embase, Scopus, Web of Science, CENTRAL, PEDro, ProQuest, RECAL, and RehabData were searched from January 1, 1995, through June 15, 2019.

Results:

Seventeen studies were included with a total of 1145 participants. A small improvement in pain over the short term (standardized mean difference −0.02; 95% confidence interval: −0.85 to 0.80) and a moderate-to-large improvement in pain in subjects 45 years or younger (standardized mean difference −0.86; 95% confidence interval: −2.45 to 0.72) in favor of the brace versus physiotherapy interventions were found. In contrast, over the long-term physiotherapy interventions (standardized mean difference 1.17; 95% confidence interval: −0.00 to 2.34), wrist splint (standardized mean difference 0.35; 95% confidence interval: −0.07 to 0.76), and laser therapy (standardized mean difference 0.58; 95% confidence interval: −0.44 to 1.59) had better effects on pain improvement versus the brace.

Conclusion:

The results indicated that physiotherapy interventions compared to counterforce braces have better effects, especially over the long-term. However, counterforce braces may have better effects on pain in younger people (<45 years old) over the short term (<6 weeks).

Clinical relevance

The results suggest that counterforce bracing is a reasonable strategy to alleviate pain over the short term. However, the subgroup analysis suggests that factors such as age may have a role in their effectiveness.

Keywords

Counterforce brace lateral elbow tendinopathy elbow pain systematic review

Background

Lateral elbow tendinopathy (LET), also known as “tennis elbow” or “lateral epicondylitis” is a common disease leading to pain in the lateral side of the elbow and decrease in hand grip strength.¹ It is estimated that the prevalence of LET is 1.3% and 1.1% among men and women, respectively,² and 5%–10% among tennis players.³ LET also reduces productivity, suggesting that it also has an economic impact.⁴ Overload and repetitive excursion of the wrist extensor muscles, specifically the extensor carpi radialis brevis (ECRB) and the extensor digitorum communis (EDC), which originate on the lateral epicondyle of the humerus, are generally considered as the pathophysiology of LET.^5,6

Although LET is commonly self-limiting and a wait-and-see policy is frequently adopted,⁷ many treatment strategies, including physiotherapy exercises, acupuncture, extracorporeal shock-wave therapy (ECSW), laser therapy, dry needling, orthoses, injections, and surgery, have been used to relieve pain, alleviate inflammation, and improve grip strength.^6,8,9 Conservative treatments result in clinical improvements in 75%–90% of subjects with LET.¹⁰ A counterforce brace, which is also known as a “proximal forearm band” or “epicondylitis bandage,” is one conventional treatment.^11,12 However, its effects on relevant outcomes remain inconclusive.

Two systematic reviews^13,14 have been conducted with regards to the effects of orthoses in subjects with LET. Published in 2002 and 2004, these studies reviewed the available evidence qualitatively and reported inconclusive results based on the limited number of studies with heterogeneous methodologies. Since then, a number of randomized controlled trials (RCTs) have been published on this topic with more consistent methodologies.

To date, no systematic review and meta-analysis has been conducted on RCTs evaluating the effects of counterforce brace on pain in subjects with LET. The primary aim of this investigation was to determine the effects of counterforce braces on pain in subjects with LET. A secondary aim was to examine the effects of counterforce braces on grip strength.

Methods

This systematic review and meta-analysis was reported according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement.¹⁵ Furthermore, the AMSTAR 2 appraisal tool¹⁶ was used to promote the quality of reporting. The protocol for this review was registered in the PROSPERO database (registration number: CRD42019133024).

Search strategy

PubMed, Embase, Scopus, Web of Science (WoS), Cochrane Central Register of Controlled Trials (CENTRAL), Physiotherapy Evidence Database (PEDro library), ProQuest, RECAL database, and Rehab Data were searched from January 1, 1995, until June 15, 2019. Based on our preliminary findings, there are no relevant RCTs before 1995; therefore, we used 1995 as the search start date. The search strategy was developed using the population and intervention components of the PICO (Population, Intervention, Comparison, and Outcome) method. Medical subject headings (MeSH), free-text words, and the Emtree thesaurus were used to select the relevant terms. Search strings were first developed for PubMed (Supplemental Table 1) and then adjusted for use in the other databases. The search process was conducted by one reviewer (S.SH) with no language restriction considered. In addition, Google Scholar, OpenGrey, F1000Research, and registry databases such as World Health Organization (WHO) International Clinical Trials Registry Platform (ICTRP), ClinicalTrials.Gov, and ISRCTN.com were searched to find gray literature and unpublished data. Manual searching of key journals (Journal of Hand Therapy, Journal of Orthopaedic & Sports Physical Therapy, Journal of Shoulder and Elbow Surgery, Prosthetics and Orthotics International, and The American Journal of Sports Medicine) and the reference lists of relevant reviews and included studies was also conducted.

Eligibility criteria

The following criteria were used to determine study eligibility: (1) RCTs (either crossover or parallel designs); (2) adult subjects (greater than 18 years old); (3) reported pain scores in both intervention and comparison groups; and (4) studies in which subjects treated with a counterforce brace were compared to other interventions (physiotherapy interventions, other orthoses, laser therapy, or sham). Exclusion criteria were as follows: (1) clinical trials without a control group; (2) other study designs (observational studies, case series, case reports, trial protocols, abstracts, or theses with an insufficient description of findings); (3) in vitro studies; (4) evaluation of the effects of a counterforce brace on healthy subjects; (5) subjects who suffered from chronic, rheumatologic, or psychiatric diseases; (6) subjects with fracture or dislocation in the affected upper limb; and (7) subjects with a history of elbow surgery.

All of the search results were imported into EndNote X7 software (Thomson Reuters, New York, NY, USA). After removing duplicate references, two reviewers (S.SH and M.J.) independently screened the remaining studies based on titles and abstracts. After identifying potentially suitable studies, two reviewers (S.SH and M.J.) initiated the process of selecting eligible studies based on the predefined criteria. Any disagreement was resolved through discussion between the two reviewers (S.SH and M.J.) and consultation with the third reviewer (S.T.H.).

Data extraction

Data were independently extracted by two reviewers (S.SH and M.J.). Any disagreement was resolved through discussion and consultation with the third reviewer (S.T.H.). Data extracted from eligible studies included: (1) first author’s name; (2) country; (3) publication year; (4) study design; (5) mean age of each group; (6) gender of subjects; (7) sample size; (8) mean and standard deviation (SD) of outcomes of interest for each group; (9) disease duration; (10) type of intervention; (11) type of comparison; (12) follow-up duration; (13) outcomes and measurement tools, and (14) funding source. When the primary outcome data were not complete, the corresponding authors of those specific studies were contacted by e-mail to request the unpublished information. Furthermore, when two studies published information from the same population, we only considered the study with the data in which we were interested. If the data were expressed as figures or plots, WebPlotDigitizer ( https://automeris.io/WebPlotDigitizer/ ) was used to extract the outcome data.

Risk of bias assessment

Two reviewers (S.SH and M.J.) independently evaluated the risk of bias of the included studies using the Cochrane Risk of Bias Tool.¹⁷ As with other steps, disagreements were resolved through discussion with a third reviewer (S.T.H.) if necessary. Risk of bias assessment was conducted based on six of the seven domains, specifically: random sequence generation; allocation concealment; blinding of outcome assessment; selective reporting; incomplete outcome data; and other bias. In orthotic interventions, it is typically difficult to blind the participants and personnel; thus, this domain of the tool was not considered relevant to risk of bias assessment. Moreover, the second question of the Verhagen quality assessment tool,¹⁸ that is, “were the groups similar at baseline regarding the most important prognostic indicators?” was used instead of the “other bias” domain.¹⁹ In each domain, the risk of bias was evaluated based on the criteria indicated in the Cochrane handbook,¹⁷ and the response for every domain given as “high risk,” “low risk,” and “unclear risk” of bias. The overall methodological status of each study was as follows: “good,” one of the criteria was unclear; “fair,” one or two criteria were high risk; and “poor,” more than two criteria were unclear or high risk.

Statistical analysis

For quantitative analysis, the overall effect was calculated as the standardized mean difference (SMD) and 95% confidence interval (CI) between the counterforce brace and comparison intervention using a random-effects model with the DerSimonian and Laird method.²⁰ Due to the sample size in some studies being less than 20 in each group, SMD (Glass’s Δ) was used.²¹ When the outcomes of interest were reported as a median and interquartile range (IQR), the mean and SD were calculated using the method proposed by Hozo et al.²² To facilitate the interpretation of pooled estimates and to determine the minimal clinically important difference (MCID), effect sizes were described as small (up to 0.2), moderate (0.2–0.5), large (0.5–0.8), and very large (greater than 0.8).²³ Cochran’s Q-test and I² statistics were used to assess statistical heterogeneity across studies, with values characterized as low (0%–25%), moderate (25%–50%), and high (50%–100%).²⁴ Subgroup analysis was used to explore the source of heterogeneity based on potential available variables such as: (1) age (⩽45 and >45); (2) random sequence generation (“low risk” and “high risk”); (3) allocation concealment (“low risk” and “high risk”); and (4) blinding of outcome assessment (“low risk” and “high risk”). Sensitivity analyses were used to evaluate the influence of each individual study on the pooled SMDs using the jackknife method (leave-one-out method).²⁵ Publication bias was investigated by the Begg and Mazumdar rank correlation test and Egger’s regression method.²⁶ In addition, the Duval and Tweedie “trim and fill” analysis was used to detect the potential influence of publication bias.²⁷ The funnel plot could not be used because the number of included studies was limited for each comparison (<10). All analyses were conducted using STATA 13.0 (StataCorp LP, College Station, TX, USA).

The GRADE approach

Overall evaluation of the evidence relating to the primary and secondary outcomes was conducted using the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach.²⁸ The following six criteria were evaluated: publication bias, effect size, risk of bias, imprecision, indirectness, and inconsistency. GRADE ranking is categorized as very low, low, moderate, and high.

Results

Study selection

The PRISMA flowchart (Figure 1) summarizes the results of the search strategy and screening. The initial search strategy resulted in a total of 1224 studies; 429 studies remained for screening after removing duplicates. After screening based on title and abstract, 79 studies were selected for full-text evaluation. Finally, 17 RCTs^3,29–44 were considered in the final synthesis. However, one study³ was removed from the meta-analysis due to categorical data.

Figure 1.

PRISMA flowchart.

Supplemental Table 2 lists the excluded studies alongside the reason for exclusion.

Study characteristics

Characteristics of the included studies are outlined in Table 1. A total of 1145 subjects participated in the included studies, and 1086 (94.8%) subjects completed the trials. All of the included trials were parallel-design, and four trials^31,36,38,43 had three groups.

Table 1.

Characteristics of included studies.

Author (year) Country	BMI (I:C) Mean age (I:C)	N (M:F) dropouts		Disease duration (weeks)	Comparison group(s)	Follow-up period (weeks)	Outcomes	Quality status	Funding source
Author (year) Country	BMI (I:C) Mean age (I:C)	Intervention group	Comparison group(s)	Disease duration (weeks)	Comparison group(s)	Follow-up period (weeks)	Outcomes	Quality status	Funding source
Akkurt et al.²⁹ (2018) Turkey	(28.97:27.86) (45.15:44.22)	41 (27:14) 2	45 (36:9) 2	9.5	Wrist splint	6	Pain (VAS), grip strength (Jamar Dynamometer), disability (DASH), health-related quality of life (SF-36)	Poor	JPJS KAKENHI (grant/award number: 26350630)
Altan and Kanat³⁰ (2008) Turkey	N/A (50.36:50.36)	25 (4:21) 0	25 (3:22) 1	6	Wrist splint	2 and 6	Pain (VAS), grip strength (Jamar Dynamometer), sensitivity (Force Dial FDK 60)), Response to treatment (patients’ views)	Fair	Without any funding source
Dundar et al.³¹ (2015) Turkey	(27.9:27.1,26.9)^a (33.6:32.6,33.4)	30 (15:15) 1	30 (17:13) 1 31 (17:14) 0	12	HILT Sham therapy	4 and 12	Pain (VAS), grip strength (Dynamometer), disability (PRTEE), quality of life (SF-36)	Good	Without any funding source
Forogh et al.³² (2012) Iran	N/A (46.5:46.5)	12 (4:8) 0	12 (3:9) 0	N/A	New orthosis	4	Pain (PRTEE), function (PRTEE), grip strength (Jamar Dynamometer), pain threshold (algometer)	Fair	N/A
Garg et al.³³ (2010) USA	N/A (53.1:51.8)	18 (11:7) 0	24 (9:15) 0	N/A	Wrist splint	6	Pain (ASES), function (MEP)	Poor	Without any funding source
Kachanathu et al.³⁴ (2013) India	N/A 30	15 (N/A) 0	15 (N/A) 0	N/A	Elbow taping, Physiotherapy	2 and 4	Pain (PRFEQ), disability (PRFEQ), grip strength (Dynamometer)	Poor	Prof. Sami S AlAbdulwahab
Kakati and Dutta³⁵ (2015) India	N/A (36.8:37.3)	25 (N/A) 0	25 (N/A) 0	N/A	Mulligan’s MWM technique	Immediate	Pain (VAS), grip strength (Jamar Dynamometer)	Poor	N/A
Kroslak et al.³ (2019) Australia	N/A (51:51)	22 (14:8) 5	23 (15:8) 9	16	Sham brace	2, 6, 12, 26	Pain (mLSRQ), grip strength (Dynamometer), tenderness, function	Good	Without any funding source
Luginbühl et al.³⁶ (2008) Switzerland	N/A 47	10 (N/A) 0	9 (N/A) 0 10(N/A) 0	42	Strengthening exercises, Combined intervention	6, 12, 52	Pain (mNPPS), grip strength (Jamar Dynamometer), compliance	Poor	N/A
Nishizuka et al.³⁷ (2017) Japan	N/A (52.6:54.6)	55 (33:22) 4	55 (34:21) 4	9	Exercise therapy	4, 12, 25, 52	Upper extremity disorder (hand10 score), pain (pain score), satisfaction, tenderness, Thomsen test, compliance	Fair	N/A
Oken et al.³⁸ (2007) Turkey	N/A (44.5:46.5,45.1)^a	20 (2:18) 0	18 (4:14) 1 20 (3:17) 0	17	Ultrasound, laser therapy	0, 2, 6	Pain (VAS), grip strength (Jamar Dynamometer), global assessment of improvement (Six-point scale)	Poor	N/A
Ökmen et al.³⁹ (2017) Turkey	N/A (45.63:43.90)	30 (14:16) 2	31 (10:21) 2	21	Counterforce brace Plus Peloid therapy	0, 4	Pain (PRTEE), function (PRTEE)	Good	Without any funding source
Salli et al.⁴⁰ (2016) Turkey	(29.2:29.4) (45.6:47.5)	34 (10:24) 0	31 (8:23) 0	34	Laser therapy	6	Pain (VAS), grip strength (Jamar Dynamometer), disability (DASH), quality of life (SF-36)	Poor	Without any funding source
Soderberg et al.⁴¹ (2012) Sweden	N/A (50:48)	22 (7:15) 3	20 (11:9) 2	>1 year	Counterforce brace plus exercise	0, 3, 6	Pain (VAS), grip strength (Dynamometer)	Fair	Rehband, Sweden, sponsored the study through providing Epi Forsa forearm bands
Streek et al.⁴² (2004) The Netherlands	N/A (43.5:42.30)	20 (6:14) N/A	23 (7:16) N/A	N/A	Wrist splint	6	Pain (PRFEQ), grip strength (Jamar Dynamometer), function (PRFEQ)	Poor	N/A
Struijs et al.⁴³ (2004) The Netherlands	N/A (46:43,47)	63 (27:36) 10	53 (36:17) 6 54 (28:26) 4	20	Physical therapy. Combined intervention	6, 26, 52	Pain (Numeric rating scale and PFFQ), grip strength (Jamar Dynamometer), disability (PFFQ), pressure pain (PTM), satisfaction	Good	Financed by Bauerfeind, manufacturer of orthotic devices
Uygur et al.⁴⁴ (2017) Turkey	N/A (47.7:48)	41 (N/A) N/A	51 (N/A) N/A	N/A	Dry needling	3, 26	Pain (PRTEE), function (PRTEE)	Fair	Without any funding source

Three groups

ASES: American Shoulder and Elbow Society; BMI: body mass index; C: comparison group; DASH: Disabilities of the Arm, Shoulder, and Hand questionnaire; F: female; HILT: high-intensity laser therapy; I: intervention group; M: male; MEP: Mayo Elbow Performance; mLSRQ: modified L’Insalata Shoulder Rating Questionnaire; mNPPS: modified Nirschl/Pettrone Pain Score; MWM: movement with mobilization; N/A: not applicable or mentioned; PFFQ: Pain-Free Function Questionnaire; PRFEQ: Patient-Rated Forearm Evaluation Questionnaire; PRTEE: patient-rated tennis elbow evaluation; PTM: pressure threshold meter; RCT: randomized controlled trial; VAS: visual analog scale.

Risk of bias assessment

The risk of bias assessment is presented in Supplemental Figure 1. Most of the included studies considered randomization criteria, but only some of them^{3,31,34,37,39–41,44} described clearly the random sequence generation process. Furthermore, concealment of allocation was reported in only some trials.^{3,32,37,39,41–44} Disregarding these items can lead to selection biases and influence the internal and external validity of findings.⁴⁵ With respect to the third item, blinding of the outcome assessment, eight studies^{3,30–32,38,39,43,44} were judged as having a low risk of bias. Only six studies^{30,31,37,39,41,43} had a low risk of bias for the incomplete outcome data item. Ignoring this item results in attrition bias. According to the selective reporting of outcomes, all included studies had a low risk of bias. Finally, almost all of the studies had a low or unclear risk of bias regarding whether the groups were similar at baseline for the most important prognostic indicators, only one RCT was assessed as having a high risk of bias.⁴⁴ Overall, the quality of four studies^3,31,39,43 was considered “good,” five studies^{30,32,37,41,44} were considered “fair,” and eight studies^{29,33–36,38,40,42} were considered “poor.” Supplemental Figure 2 presents the percentage of reviewers’ judgments for each risk of bias item. Information on funding source was identified where possible for included studies (see Table 1) to address agenda bias; however, this information was not reported in all studies.

Meta-analysis results of primary outcome: pain

Counterforce brace versus physiotherapy interventions; short-term follow-up (0–6 weeks)

The pooled estimate for six studies^{34,35,37,43,44,46} showed that a counterforce brace did not significantly relieve elbow pain versus physiotherapy interventions in subjects with LET (SMD −0.02 (95% CI −0.85 to 0.80) (Figure 2(a)). Statistical heterogeneity was high among the six studies (Heterogeneity χ² = 69.36 (p < 0.001) I² = 92.8%). Regarding the included trials in this category, the major limitation was small sample size leading to under-powering of the results.

Figure 2.

Forest plot comparing counterforce brace to physiotherapy interventions: (a) short-term follow-up and (b) long-term follow-up, primary outcome of pain.

Supplemental Figure 3 shows the forest plots of the subgroup analysis of this category based on participants’ age (Supplemental Figure 3(a)), random sequence generation (Supplemental Figure 3(b)), allocation concealment (Supplemental Figure 3(c)), and blinding of the outcome assessment (Supplemental Figure 3(d)). Although the statistical heterogeneity results were not substantially different from the overall findings, pooled estimates were slightly different (see Table 2). Age subgroup analysis showed superior clinical effects of a counterforce brace on pain in subjects 45 years or younger (SMD −0.86; 95% CI −2.45 to 0.72). However, based on other factors, only high-risk bias trials showed better effects of a counterforce brace, which may reflect the impact of selection and detection bias.

Table 2.

Subgroup analysis between potential factors for primary outcome of pain.

Potential		SMD (CI 95%)	No. of studies	Heterogeneity χ²	p value	I² (%)
Counterforce brace vs. physiotherapy interventions (short-term follow-up)
All studies		−0.02 (−0.85 to 0.80)	6	69.36	<0001	92.8
Age subgroup	⩽45 years>45 years	−0.86 (−2.45 to 0.72)0.64 (−0.45 to 1.74)	33	24.9839.07	<0.001<0.001	92.094.9
Random sequence generation	Low riskHigh risk	−0.24 (−2.27 to 1.77)0.04 (−0.47 to 0.57)	33	60.725.96	<0.0010.051	96.766.4
Allocation concealment	Low riskHigh risk	0.64 (−0.45 to 1.74)−0.86 (−2.45 to 0.72)	33	39.0724.98	<0.001<0.001	94.992.0
Blinding of outcome assessment	Low riskHigh risk	0.50 (−0.86 to 1.88)−0.63 ( −1.91 to 0.65)	33	41.7023.56	<0.001<0.001	95.291.5
Counterforce brace vs. physiotherapy interventions (long-term follow-up)
All studies		1.17 (−0.00 to 2.34)	4	53.66	<0.001	94.4
Age subgroup	⩽50 years>50 years	1.09 (−0.64 to 2.83)1.40 (0.92 to 1.87)	31	46.40–	<0.001–	95.7–
Random sequence generation	Low riskHigh risk	1.98 (0.79 to 3.16)0.19 (−0.41 to 0.81)	22	8.561.74	0.0030.187	88.342.5
Allocation concealment	Low riskHigh risk	1.31 (−0.13 to 2.76)0.69 (−0.27 to 1.65)	31	53.55–	<0.001–	96.3–
Blinding of outcome assessment	Low riskHigh risk	1.29 (−1.26 to 3.84)1.17 (0.52 to 1.82)	22	46.381.67	<0.0010.196	97.840.1
Counterforce brace vs. wrist splint
All studies		0.35 (−0.07 to 0.76)	5	9.08	0.059	55.9
Age subgroup	⩽45 years>45 years	−−0.04 (−0.40 to 0.31)0.65 (0.16 to 1.14)	23	0.042.96	0.8350.228	0.0032.4
Random sequence generation	Low riskHigh risk	0.44 (−0.20 to 1.09)0.27 (−0.44 to 0.98)	32	4.843.86	0.0890.050	58.774.1
Allocation concealment	Low riskHigh risk	0.60 (−0.65 to 1.86)0.26 (−0.18 to 0.70)	23	4.793.95	0.0290.139	79.149.4
Blinding of outcome assessment	Low riskHigh risk	0.86 (0.28 to 1.43)0.04 (−0.26 to 0.35)	23	1.200.93	0.2730.628	16.70.0
Counterforce brace vs. laser therapy
All studies		0.58 (−0.44 to 1.59)	3	17.36	<0.001	88.5
Age subgroup	⩽45 years>45 years	0.92 (−1.12 to 2.96)0.04 (−0.44 to 0.52)	21	16.07–	<0.001–	93.8–
Random sequence generation	Low riskHigh risk	−0.02 (0.37 to 0.32)2.00 (1.11 to 2.88)	21	0.14−	0.712−	0.0−
Allocation concealment	Low riskHigh risk	−0.09 (−0.59 to 0.41)0.98 (−0.93 to 2.90)	12	−14.37	−< 0.001	−93.0
Blinding of outcome assessment	Low riskHigh risk	0.92 (−1.12 to 2.96)0.04 (−0.44 to 0.52)	21	16.07−	<0.001−	93.8−

Counterforce brace versus physiotherapy interventions; long-term follow-up (12–52 weeks)

The pooled estimate from four included studies^36,37,43,44 revealed that other physiotherapy interventions in comparison with a counterforce brace decreased pain scores clinically (SMD 1.17; 95% CI −0.00 to 2.34), with significant statistical heterogeneity among the four trials (heterogeneity χ² = 53.66 (p < 0.001) I² = 94.4%). Body weight is shown by the forest plot in Figure 2(b). Although three studies^36,37,44 showed that physiotherapy interventions had considerable effects on pain reduction, Struijs et al.⁴³ concluded that there was no difference between the two interventions, counterforce brace versus ultrasound, on pain at long-term follow-up (SMD 0.00; 95% CI −0.37 to 0.37). However, this comparison had limited studies with small samples resulting in inconclusive findings.

In the majority of subgroup analyses, there was no significant difference between the subsets of eligible studies when considering participants’ age, random sequence generation, allocation concealment, or blinding of outcome assessment (Table 2). Supplemental Figure 4 shows forest plots for these subgroup analyses.

Counterforce brace versus wrist splint

Five trials^{29,30,32,33,42} were included in this comparison. The meta-analysis (Figure 3(a)) showed a small-to-moderate improvement in favor of the wrist splint group on pain scores (SMD 0.35; 95% CI −0.07 to 0.76) with moderate statistical heterogeneity (heterogeneity χ² = 9.08 (p = 0.059) I² = 55.9%). However, only Akkurt et al.²⁹ showed a small improvement in favor of a counterforce brace group at six weeks (SMD −0.07; 95% CI −0.50 to 0.36). The trial by Forogh et al.³² used a new orthosis with a covered forearm in the comparison group. Although the sample size in each group was very small (n = 10), the SMD was clinically and statistically significant in favor of the new orthosis group (SMD 1.30; 95% CI: 0.33 to 2.27). Based on available data, the subgroup analysis was used to find the source of heterogeneity (Supplemental Figure 5). The subgroup analysis by participants’ mean age (Supplemental Figure 5(a)) showed that the pooled effect size was in favor of the counterforce brace group in subjects 45 years or younger (SMD −0.04; 95% CI −0.40 to 0.31) with low statistical heterogeneity (heterogeneity χ² = 0.04 (p = 0.835) I² = 0.00%). Subgroup analysis among subjects >45 years was in favor of the wrist splint group (SMD 0.65; 95% CI 0.16 to 1.14) with moderate statistical heterogeneity (heterogeneity χ² = 2.96 (p = 0.228) I² = 32.4%). Table 2 outlines other subgroup analyses for this comparison.

Figure 3.

Forest plot comparing (a) counterforce brace to wrist splint and (b) counterforce brace to laser therapy, primary outcome of pain.

Counterforce brace versus laser therapy

Three studies^31,38,40 showed a moderate improvement in favor of the laser therapy group (SMD 0.58; 95% CI −0.44 to 1.59), although between-study heterogeneity was significant (heterogeneity χ² = 17.36 (p < 0.001) I² = 88.5%) (Figure 3(b)). To investigate potential factors for heterogeneity, subgroup analyses were conducted (Table 2 and Supplemental Figure 6). Altogether, limited studies with small samples led to unreliable findings for this comparison.

Meta-analysis results of secondary outcome: grip strength

Counterforce brace versus physiotherapy interventions

Grip strength was reported in five trials.^{34–36,38,43} The pooled SMD showed a small improvement in favor of the physiotherapy group (SMD −0.12; 95% CI −0.43 to 0.20) with a low level of heterogeneity (heterogeneity χ² = 5.51 (p = 0.239) I² = 27.4%). Although three studies^35,36,38 revealed moderate effectiveness in favor of physiotherapy, one study⁴³ showed a superior effect for the counterforce brace (SMD 0.21; 95% CI −0.15 to 0.58). Supplemental Figure 7(a) shows that all of the included studies have wide CIs reflective of their small sample size and insufficient statistical power. Supplemental Table 3 outlines the results of subgroup analyses, and Supplemental Figure 8 shows the body weights by forest plots.

Counterforce brace versus wrist splint

A comparison of grip strength between a counterforce brace and wrist splint was conducted in four studies^29,30,32,42 (Supplemental Figure 7(b)). The pooled SMD demonstrated a small improvement in favor of the counterforce brace (SMD 0.37; 95% CI −0.12 to 0.86) but with large heterogeneity (heterogeneity χ² = 7.74 (p = 0.052) I² = 61.3%). All of the included studies favored the counterforce brace except one³² that compared a new orthosis to a counterforce brace without covered forearm (SMD −0.53, 95% CI −1.36 to 0.30). Limited studies with low statistical power in this comparison resulted in inconclusive findings. The results of subgroup analyses are shown in Supplemental Figure 9.

Counterforce brace versus laser therapy

Three studies^31,38,40 compared the effect of a counterforce brace to laser therapy on grip strength. The pooled estimation showed a moderate improvement in favor of laser therapy (SMD −0.48; 95% CI −1.46 to 0.50), although the between-study heterogeneity was considerable (heterogeneity χ² = 16.69 (p < 0.001) I² = 88.0%). For this comparison, a large and significant improvement in favor of laser therapy was shown by one of the studies³⁸ (SMD −1.79; 95% CI −2.64 to −0.95), while the other studies had contrary results (Supplemental Figure 10).

Sensitivity analysis

The results of sensitivity analysis based on the leave-one-out method showed that certain trials influenced the overall SMDs of the primary outcome (pain) over the short term, from −0.34 (95% CI −0.96 to 0.28) after removing the study by Uygur et al.⁴⁴ to 0.40 (95% CI −0.33 to 1.14) after removing the study by Kachanathu et al.³⁴ Furthermore, for the primary outcome over the long term, sensitivity analysis showed that certain trials also had a considerable effect on the pooled estimate. Supplemental Figures 11 and 12 show the results of sensitivity analysis for the primary and secondary outcome (grip strength), respectively.

Publication bias assessment

No potential publication bias was found for pain over short-term follow-up using Egger’s linear regression method (intercept = −2.716, standard error = 4.838, 95% CI −16.14 to 10.71, p = 0.604) and Egger’s graph (Figure 4(a)). The “trim and fill” method also did not find any missing studies. However, Egger’s linear regression (intercept = 6.305, standard error = 6.612, 95% CI −22.14 to 34.75, p = 0.441) showed that there is considerable publication bias for pain over long-term follow-up (Figure 4(b)). Using the “trim and fill” method, two potentially missing studies were identified in this meta-analysis. Inclusion for missing publications led to an adjusted SMD that was unchanged (Table 3). Egger’s linear regression also detected a publication bias in the meta-analysis assessing the effect of counterforce brace versus wrist splint on pain (intercept = 4.656, standard error = 1.854, 95% CI −1.24 to 10.55, p = 0.087) (Figure 5(a)). The “trim and fill” method identified a missing study and, therefore, changed the pooled estimate (SMD 0.20; 95% CI −0.253 to 0.667). Although Egger’s linear regression showed a suspected publication bias for the fourth comparison, counterforce brace versus laser therapy (intercept = 4.656, standard error = 1.854, 95% CI −1.24 to 10.55, p = 0.087), the “trim and fill” method did not identify any missing study for this comparison with limited trials (Figure 5(b)). Table 3 and Supplemental Figure 13 show the results of the publication bias assessment for the secondary outcome.

Figure 4.

Egger’s graphs for primary outcome of pain: (a) counterforce brace vs. physiotherapy interventions, short-term follow-up and (b) counterforce brace vs. physiotherapy interventions, long-term follow-up.

Table 3.

Publication bias assessment.

Analysis	No. of studies	Publication bias
Analysis	No. of studies	Begg’s test:z-testp value	Egger’s test:Interceptt-test p value	Trim and fill (95% CI) (no. of studies trimmed)
Primary outcome: pain
Counterforce brace vs. physiotherapy interventions (short-term)	6	−1.320.188	−2.716−0.560.604	Unchanged
Counterforce brace vs. physiotherapy interventions (long-term)	4	0.680.497	6.3050.950.441	1.18(0.321 to 4.358)(2)
Counterforce brace vs. wrist splint	5	0.980.327	4.652.510.08	0.20(−0.253 to 0.667)(1)
Counterforce brace vs. laser therapy	3	0.520.602	106.300.1	Unchanged
Secondary outcome: grip strength
Counterforce brace vs. physiotherapy interventions	5	−0.980.327	−2.69−2.360.09	Unchanged
Counterforce brace vs. wrist splint	4	−1.360.174	−4.22−1.220.34	Unchanged
Counterforce brace vs. laser therapy	3	−1.570.117	−10.58−7.990.07	Unchanged

Figure 5.

Egger’s graphs for primary outcome of pain: (a) counterforce brace vs. wrist splint and (b) counterforce brace vs. laser therapy.

“Trim and fill” plots for pain and grip strength are shown in Supplemental Figures 14 and 15, respectively.

Quality of evidence

Table 4 shows the GRADE evidence profile. The quality of evidence was evaluated as low and very low in both of the outcomes because of limited sample size, high level of statistical heterogeneity, publication bias, and small effect sizes.

Table 4.

GRADE evidence profile.

Absolute effectSMD(95% CI)	No. of studies	Study design	Risk of bias	Inconsistency	Indirectness	Imprecision	Publication bias	Effect size	GRADE quality
Primary outcome: pain
Counterforce brace vs. physiotherapy interventions (short-term follow-up)
−0.02(−0.85, 0.80)(favors intervention)	6	RCT	−1^a	−1^b	0	0	0	0	+ +− − (low)
Counterforce brace vs. physiotherapy interventions (long-term follow-up)
1.17(0.00, 2.34)(favors comparison)	4	RCT	−1	−1	0	−1^c	−1^d	+2^e	+ +− − (low)
Counterforce brace vs. wrist splint
0.35(−0.07, 0.76)(favors comparison)	5	RCT	−1	0	0	0	−1	0	+ +− − (low)
Counterforce brace vs. laser therapy
0.58(−0.44, 1.59)(favors comparison)	3	RCT	−1	−1	0	−1	−1	+ 1^f	(very low)
Secondary outcome: grip strength
Counterforce brace vs. physiotherapy interventions
−0.12(−0.43, 0.20)(favors intervention)	5	RCT	−1	0	0	0	−1	0	+ +− − (low)
Counterforce brace vs. wrist splint
0.37(−0.12, 0.86)(favors comparison)	4	RCT	−1	−1	0	−1	−1	0	+ +− − (very low)
Counterforce brace vs. laser therapy
−0.48(−1.46, 0.50)	3	RCT	−1	−1	0	−1	−1	+ 1	+ +− − (very low)

The symbols + +− − shows the quality of evidence.

CI: confidence interval; GRADE: Grades of Recommendation, Assessment, Development and Evaluation; RCT: randomized controlled trial; SMD: standardized mean difference.

Down-graded one level as the moderate risk of bias.

Down-graded one level as the I² was > 50%.

Down-graded one level as the number of studies was < 5.

Down-graded one level due to potential publication bias.

Up-graded two levels due to large effect size (SMD > 1).

Up-graded one level due to moderate effect size (SMD > 0.5).

Discussion

To our knowledge, this is the first, comprehensive, systematic review and meta-analysis about the effects of a counterforce brace on pain and grip strength in subjects with LET. The results of our review suggest that use of a counterforce bracing compared to physiotherapy interventions did not demonstrate any superior effect on pain scores over the short term. However, physiotherapy interventions compared to counterforce bracing showed improvements in both pain and grip strength over the long term. A comparison of a wrist splint to a counterforce brace showed that improvements in pain favored the wrist splint and improvements in grip strength favored the counterforce brace. Our review also indicated that the quality of evidence was low and very low for most of the comparisons, due to high risk of bias, statistical heterogeneity, and publication bias.

Effects on pain

Counterforce braces may have mechanical⁴⁷ and neuromuscular effects⁴⁸ on elbow proprioception. Since motor control is reduced in older adults,⁴⁹ it is not surprising that our review suggested that the effects of the brace decline with age. While this may be a notable finding, due to the limited number of trials in each subgroup, it should be considered with caution. Counterforce braces prevent extensor muscle expansion, creating a new extensor origin and giving the soft tissue time to heal.^8,50 It is possible that the better effect of the counterforce brace on pain in younger adults be the result of better remodeling capability of their musculoskeletal system in comparison to older adults. However, the effect of potential confounding variables like chronicity, the extent of intrasubstance damage, the severity of pain, and the extent of pressure on the tendon from the counterforce cuff should not be underestimated. For example, although theoretical, for a counterforce brace to work based on the new origin theory,⁵¹ a snug enough strap must be provided, which was not measured or reported in the reviewed studies. A previous study reported a positive correlation between strap tension of the counterforce brace and pain.⁵² By comparison, physiotherapy modalities theoretically work based on mechanisms of action like increased blood flow, softening scar tissue, and maintaining mobility within soft tissue.⁵³ Therefore, physiotherapy modalities may be a more effective mechanism for healing compared to counterforce braces. It is possible that the combined effect of physiotherapy and a counterforce brace may be worth consideration. Regardless, according to GRADE findings, quality of evidence was low to very low; therefore, conducting high-quality studies is necessary to arrive at more conclusive results. In addition, economic evaluation studies (cost-effectiveness, cost–utility, and cost–benefit analyses) should be undertaken to help health insurers and health policy-makers to determine the most cost-effective intervention.⁵³

Effects on grip strength

This review included trials evaluating the effects of counterforce braces compared to physiotherapy interventions on grip strength as a secondary outcome. The pooled estimate showed a small and nonsignificant improvement in grip strength favoring physiotherapy interventions. Although the between-study heterogeneity was low, wide CIs around the point estimates did not engender confidence in the robustness of the findings. GRADE assessments were low and very low for this outcome. Thus, future trials should consider the MCID criteria to calculate a sufficient sample size to arrive at more conclusive results.

Study limitations

This review had a few limitations. First, some of the comparisons and subgroups included a limited number of studies with high levels of statistical heterogeneity. Second, most of the included studies had low methodological quality due to the inability to blind completely. Third, due to a lack of data in most of the included studies, we could not consider other potential confounding factors such as body mass index (BMI), duration of disease, and physical activities.

Conclusion

Despite the limitations of this systematic review and meta-analysis, the results indicated that physiotherapy interventions compared to counterforce braces may have better effects, especially over the long term. However, counterforce braces may have better effects on pain in younger people (<45 years old) over the short term (<6 weeks). Furthermore, counterforce braces were slightly superior to wrist splints in grip strength improvement.

Supplemental Material

10.1177_0309364620930618_Supplemental_material – Supplemental material for The effects of counterforce brace on pain in subjects with lateral elbow tendinopathy: A systematic review and meta-analysis of randomized controlled trials

Supplemental material, 10.1177_0309364620930618_Supplemental_material for The effects of counterforce brace on pain in subjects with lateral elbow tendinopathy: A systematic review and meta-analysis of randomized controlled trials by Saeed Shahabi, Kamran Bagheri Lankarani, Seyed Taghi Heydari, Maryam Jalali, Sulmaz Ghahramani, Mojtaba Kamyab, Reza Tabrizi and Mostafa Hosseinabadi in Prosthetics and Orthotics International

Footnotes

Associate Editor: Andrew Sawers

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Saeed Shahabi

Supplemental material

Supplemental material for this article is available online.

Trial registration

The protocol of this study has been registered in the PROSPERO database (registration number: CRD42019133024).

References

Renstrom

Ackermann

PW.

Pathophysiology of tendinopathy: implications for tennis elbow. In: Di Giacomo

Ellenbecker

T-S

Kibler

(eds) Tennis Medicine. Cham: Springer 2018, pp. 263–275.

Faro

Wolf

JM.

Lateral epicondylitis: review and current concepts. J Hand Surg 2007; 32: 1271–1279.

Kroslak

Pirapakaran

Murrell

GAC

. Counterforce bracing of lateral epicondylitis: a prospective, randomized, double-blinded, placebo-controlled clinical trial. J Shoulder Elbow Surg 2019; 28: 288–295.

Krey

Borchers

McCamey

Tendon needling for treatment of tendinopathy: a systematic review. Phys Sportsmed 2015; 43: 80–86.

Whaley

Baker

CL.

Lateral epicondylitis. Clin Sports Med 2004; 23: 677–691.

Bateman

Titchener

Clark

, et al. Management of tennis elbow: a survey of UK clinical practice. Shoulder Elbow 2019; 11: 233–238.

Boden

Scott

Dalwadi

, et al. Platelet-rich plasma versus Tenex in the treatment of medial and lateral epicondylitis. J Shoulder Elbow Surg 2019; 28: 112–119.

Bateman

Whitby

Kacha

, et al. Current physiotherapy practice in the management of tennis elbow: a service evaluation. Musculoskeletal care 2018; 16: 322–326.

J-Y

Wang

F-S.

Extracorporeal shockwave therapy for tendinopathy. In: Wang

C-J

Schaden

J-Y

(eds) Shockwave medicine. Basel: Karger Publishers 2018, pp. 27–41.

10.

Vaquero-Picado

Barco

Antuña

SA.

Lateral epicondylitis of the elbow. EFORT Open Rev 2016; 1: 391–397.

11.

Tarpada

Morris

Lian

, et al. Current advances in the treatment of medial and lateral epicondylitis. J Orthop 2018; 15: 107–110.

12.

Hijmans

Postema

Geertzen

JH.

Elbow orthoses: a review of literature. Prosthet Orthot Int 2004; 28: 263–272.

13.

Struijs

PAA

Smidt

Arola

, et al. Orthotic devices for the treatment of tennis elbow. Cochrane Database Syst Rev 2002; CD001821.

14.

Borkholder

Hill

Fess

EE.

The efficacy of splinting for lateral epicondylitis: a systematic review. J Hand Ther 2004; 17: 181–199.

15.

Moher

Liberati

Tetzlaff

, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009; 151: 264–269.

16.

Shea

Reeves

Wells

, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017; 358: j4008.

17.

Higgins

Altman

Gøtzsche

, et al. The cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011; 343: d5928.

18.

Verhagen

de Vet

de Bie

, et al. The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol 1998; 51: 1235–1241.

19.

Shahabi

Shabaninejad

Kamali

, et al. The effects of ankle-foot orthoses on walking speed in patients with stroke: a systematic review and meta-analysis of randomized controlled trials. Clin Rehabil 2019; 34: 145–159.

20.

DerSimonian

Laird

Meta-analysis in clinical trials. Control Clin Trials 1986; 7: 177–188.

21.

Beretvas

SN.

Meta-analysis. The reviewer’s guide to quantitative methods in the social sciences. Abingdon: Routledge 2018, pp. 260–268.

22.

Hozo

Djulbegovic

Hozo

Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol 2005; 5: 13.

23.

Cohen

A power primer. Psychol Bull 1992; 112: 155–159.

24.

Huedo-Medina

Sánchez-Meca

Marín-Martínez

, et al. Assessing heterogeneity in meta-analysis: Q statistic or I² index? Psychol Methods 2006; 11: 193–206.

25.

Iyengar

Greenhouse

Sensitivity analysis and diagnostics. In: Cooper

Hedges

(eds) Handbook of research synthesis and meta-analysis. New York: Russell Sage Foundation 2009, pp. 417–433.

26.

Rothstein

Sutton

Borenstein

(eds) Publication bias in meta-analysis: prevention, assessment and adjustments. Publication bias in meta-analysis. Hoboken: Wiley 2005, pp. 1–7.

27.

Duval

Tweedie

Trim and fill: a simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics 2000; 56: 455–463.

28.

Guyatt

Oxman

Vist

, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008; 336: 924–926.

29.

Akkurt

Kocabas

Yilmaz

, et al. Comparison of an epicondylitis bandage with a wrist orthosis in patients with lateral epicondylitis. Prosthet Orthot Int 2018; 42: 599–605.

30.

Altan

Kanat

Conservative treatment of lateral epicondylitis: comparison of two different orthotic devices. Clin Rheumatol 2008; 27: 1015–1019.

31.

Dundar

Turkmen

Toktas

, et al. Effectiveness of high-intensity laser therapy and splinting in lateral epicondylitis; a prospective, randomized, controlled study. Lasers Med Sci 2015; 30: 1097–1107.

32.

Forogh

Khalighi

Javanshir

, et al. The effects of a new designed forearm orthosis in treatment of lateral epicondylitis. Disabil Rehabil Assist Technol 2012; 7: 336–339.

33.

Garg

Adamson

Dawson

, et al. A prospective randomized study comparing a forearm strap brace versus a wrist splint for the treatment of lateral epicondylitis. J Shoulder Elbow Surg 2010; 19: 508–512.

34.

Kachanathu

Miglani

Grover

, et al. Forearm band versus elbow taping: as a management of lateral epicondylitis. J Musculoskelet Res 2013; 16: 1–9.

35.

Kakati

Dutta

A comparative study to find out immediate effectiveness of movement with mobilization versus elbow orthosis on pain and grip strength in lateral epicondylitis in housewives. Int J Physiother 2015; 2: 1085–1090.

36.

Luginbuhl

Brunner

Schneeberger

AG.

No effect of forearm band and extensor strengthening exercises for the treatment of tennis elbow: a prospective randomised study. Chir Organi Mov 2008; 91: 35–40.

37.

Nishizuka

Iwatsuki

Kurimoto

, et al. Efficacy of a forearm band in addition to exercises compared with exercises alone for lateral epicondylitis: a multicenter, randomized, controlled trial. J Orthop Sci 2017; 22: 289–294.

38.

Oken

Kahraman

Ayhan

, et al. Laser therapy in lateral epicondylitis: comparison with brace and ultrasound treatment. Ann Rheum Dis 2007; 66: 241.

39.

Okmen

Eroksuz

Altan

, et al. Efficacy of peloid therapy in patients with chronic lateral epicondylitis: a randomized, controlled, single blind study. Int J Biometeorol 2017; 61: 1965–1972.

40.

Salli

Akkurt

Izki

, et al. Comparison of high intensity laser and epicondylitis bandage in the treatment of lateral epicondylitis. Arch Rheumatol 2016; 31: 234–238.

41.

Soderberg

Grooten

Ang

BO.

Effects of eccentric training on hand strength in subjects with lateral epicondylalgia: a randomized-controlled trial. Scand J Med Sci Sports 2012; 22: 797–803.

42.

Van De Streek

Van Der Schans

De Greef

, et al. The effect of a forearm/hand splint compared with an elbow band as a treatment for lateral epicondylitis. Prosthet Orthot Int 2004; 28: 183–189.

43.

Struijs

Kerkhoffs

Assendelft

, et al. Conservative treatment of lateral epicondylitis: brace versus physical therapy or a combination of both-a randomized clinical trial. Am J Sports Med 2004; 32: 462–469.

44.

Uygur

Aktas

Ozkut

, et al. Dry needling in lateral epicondylitis: a prospective controlled study. Int Orthop 2017; 41: 2321–2325.

45.

Jüni

Altman

Egger

Assessing the quality of controlled clinical trials. BMJ 2001; 323: 42–46.

46.

Oken

Kahraman

Ayhan

, et al. The short-term efficacy of laser, brace, and ultrasound treatment in lateral epicondylitis: a prospective, randomized, controlled trial. J Hand Ther 2008; 21: 63–67.

47.

Takasaki

Aoki

Oshiro

, et al. Strain reduction of the extensor carpi radialis brevis tendon proximal origin following the application of a forearm support band. J Orthop Sports Phys Ther 2008; 38: 257–261.

48.

Kroslak

Murrell

GAC

. Tennis elbow counterforce bracing. Tech Shoulder Elb Surg 2007; 8: 75–79.

49.

Seidler

Bernard

Burutolu

, et al. Motor control and aging: links to age-related brain structural, functional, and biochemical effects. Neurosci Biobehav Rev 2010; 34: 721–733.

50.

Vellilappilly

Rai

Varghese

, et al. Counterforce orthosis in the management Of lateral epicondylitis. J Ayub Med Coll Abbottabad 2017; 29: 328–334.

51.

Meyer

Pennington

Haines

, et al. The effect of the forearm support band on forces at the origin of the extensor carpi radialis brevis: a cadaveric study and review of literature. J Hand Ther 2002; 15: 179–184.

52.

GYF

Chan

. The immediate effects of tension of counterforce forearm brace on neuromuscular performance of wrist extensor muscles in subjects with lateral humeral epicondylosis. J Orthop Sports Phys Ther 2004; 34: 72–78.

53.

Shahabi

Rezapour

Arabloo

Economic evaluations of physical rehabilitation interventions in older adults with hip and/or knee osteoarthritis: a systematic review. Eur J Physiother 2019; 1–11.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

3.14 MB