Sage Journals: Discover world-class research

Abstract

Background

The integrative medicine (IM) clinic is an innovative care model that may increase access to guideline-concordant nonpharmacologic treatment use in healthcare delivery systems for prevalent conditions such as low back pain (LBP).

Objective

To describe the use and effectiveness of IM services for LBP in IM clinics.

Research Design

Prospective cohort study.

Subjects

Adult patients with LBP enrolled at seventeen IM clinics.

Measures

Patterns of IM service use were assessed over 12 months. Changes in clinical outcomes were assessed between index visit and 12-month follow-up using linear mixed-effects models. Primary (pain interference, physical function) and secondary (pain intensity, anxiety, depression, fatigue, sleep disturbance, social participation) outcomes were obtained from the PROMIS-29 instrument.

Results

We identified 660 participants with LBP (mean age = 51.6 years, 75% female). Over the 12-month study period, common IM services were IM consults (56%), acupuncture (44%), chiropractic care (24%), physical therapy (19%), and massage (17%). Over two-thirds (70%) of participants received at least one guideline-concordant nonpharmacologic treatment. Participants with follow-up outcome data (n = 443, 67%) reported a modest reduction in pain interference with life activities in the short- and long-term (2-month mean difference [MD] = −1.47, 95%CI = −2.98, −0.64; 12-month MD = −1.98, 95%CI = −3.12, −0.88). By contrast, improvements in physical function were not statistically or clinically significant (2-month MD = 0.37, 95%CI = −0.28, 1.01; 12-month MD = 0.69, 95%CI = −0.31, 1.69). At 12 months, small improvements were observed on all secondary outcomes (pain intensity, anxiety, depression, and social participation) except fatigue and sleep disturbance.

Conclusions

Most patients with LBP receiving care at IM clinics received at least one guideline-recommended nonpharmacologic treatment. However, improvements on clinical outcomes were relatively small. Additional multi-site studies are needed to explore the optimal implementation approach.

Keywords

low back pain practice-based research implementation science integrative medicine acupuncture chiropractic care massage therapy physical therapy

Introduction

Low back pain (LBP) leads all causes of disability and healthcare spending in the United States.^1-3 Current clinical practice guidelines (e.g., American College of Physicians [ACP]) state patients with acute or chronic LBP should initially receive nonpharmacologic treatment.^4,5 Implementing these guidelines requires a shift from over-the-counter medications (e.g., non-steroidal anti-inflammatory drugs or NSAIDs) to nonpharmacologic treatments as first-line choices.^4,5 This shift is aligned with current evidence suggesting nonpharmacologic treatments are similarly effective to NSAIDs, opioids, and other analgesics, with fewer adverse effects.^4-6 Nonpharmacologic treatments include rehabilitative (e.g., physical therapy, chiropractic care), behavioral (e.g., cognitive behavioral therapy), and integrative approaches (e.g., acupuncture, massage, yoga). Although nonpharmacologic treatments are commonly provided for LBP in community-based private clinics, they are infrequently provided in healthcare systems as part of ‘usual medical care’.^7-9 In addition, while nonpharmacologic treatments have demonstrated effectiveness for LBP in the resource-rich context of randomized controlled trials (RCTs),^10,11 their effectiveness as part of usual care in large healthcare systems is relatively unexplored.

Integrative medicine (IM) combines evidence-based conventional, traditional, and complementary approaches in a coordinated way.^12,13 The ‘IM clinic’ is a unique care model where patients may receive multiple services, including both targeted education around self-management and nonpharmacologic treatments, to address health conditions and improve overall wellbeing.^12-14 Care is often coordinated by a conventional healthcare provider (e.g., primary care provider with IM training) through ‘IM consults’ which are typically longer than a standard primary care visit (60 minutes vs 20 minutes). In turn, a longer visit allows IM providers the time needed to incorporate education around the self-management of pain and other conditions (to teach acupressure, nutrition, mindfulness, or other mind-body approaches) and, in some cases, the delivery of guideline-recommended nonpharmacologic treatments such as acupuncture or biofeedback.^12-15 IM consults may also involve a referral to guideline-recommended nonpharmacologic treatments (e.g., acupuncture, chiropractic care, massage, physical therapy) provided within the same IM clinic or the community. While guideline-recommended nonpharmacologic treatments remain uncommon in usual care for LBP in primary care settings,^7-9 they are increasingly used by Americans to manage pain conditions and are gradually being incorporated within diverse healthcare delivery systems.^13,16 However, RCTs have typically tested the efficacy of a single nonpharmacologic treatment in a controlled setting with a specified dose of the intervention. Individuals with LBP in IM clinics likely utilize untested combinations and variable doses of multiple treatments at the same time.¹⁷

The development of IM clinics or multidisciplinary teams to manage LBP has been described in previous cross-sectional studies.^18-21 While prospective observational studies have described the use of varied nonpharmacologic approaches for LBP in multidisciplinary settings, few existing models have published data on their clinical outcomes.^22,23 A single-site study of 309 participants with 12-month of follow-up found IM services provided at an IM clinic performed at least as well as usual medical care.²² Another single-site primary care-based observational study reported improvements in several outcomes among 177 patients receiving IM services after 1 year of follow-up, but did not report results specific to LBP due to an insufficient sample size.²⁴ Additional studies that include multiple sites and longer follow-up periods are needed to understand the use and effectiveness of LBP treatments at IM clinics.

To address these gaps in knowledge, we performed an analysis of patients with LBP in the Patients Receiving Integrative Medicine Interventions Effectiveness Registry (PRIMIER).^25,26 PRIMIER enrolled patients from 17 sites in the BraveNet practice-based research network (PBRN). PBRNs can evaluate the use and effectiveness of real-world clinical care across multiple healthcare systems.²⁷ Our study had two aims: (1) Describe patterns of IM service use for LBP over one year of follow-up; (2) Report whether pain interference and physical function (primary outcomes) or secondary patient-reported outcomes improved in the short- or long-term (2 months and 1 year, respectively).

Methods

Design

PRIMIER is a prospective longitudinal observational cohort study of patients receiving IM services in the BraveNet PBRN. The PRIMIER study protocol and patient reported outcomes from the entire PRIMIER cohort are described in detail elsewhere.^25,26 The Institutional Review Board of 17 participating sites and the data coordinating center at Albert Einstein College of Medicine approved the protocol. While nearly all participating sites were IM clinics within a single department at an academic medical center or large healthcare system, one site was an IM clinical program with independent IM services embedded in multiple departments and another was a private IM clinic (Appendix Table 1). Representatives from each site were asked to indicate which IM services they offered during the PRIMIER enrollment period that were relevant for patients with LBP (Appendix Table 2), which was from August 2013 to November 2018. Eligible adults were consented and completed a baseline survey at a PRIMIER index visit. Additional surveys capturing patient reported outcomes were administered at 2, 4, 6, and 12 months. Healthcare utilization (i.e., IM service use) was evaluated using medical CPT codes from the electronic health record (EHR).

Participants

Patients were eligible for PRIMIER if they were adults (age ≥18 years old) currently receiving IM services at a participating clinic, and willing to complete follow-up surveys over a 1-year period. For this analysis, we restricted the PRIMIER cohort to 660 participants with LBP based on ICD-diagnosis codes extracted from their EHR and/or self-report of LBP on the baseline survey. We assessed outcomes in 443 participants with LBP who completed at least one follow-up survey.

Measurement

Healthcare Utilization

Participants in PRIMIER could either access IM services directly, through referrals following an IM consult or through referrals from other healthcare providers. We extracted CPT codes from each participant’s EHR to capture the frequency and type of IM services utilized by participants with a diagnosis of LBP. Potential IM services included but were not limited to acupuncture, biofeedback, chiropractic care, energy therapies, IM consults, massage therapy, mindfulness approaches such as mindfulness-based stress reduction [MBSR], naturopathic medicine, physical therapy, tai chi, and yoga. The total number of IM services used by participants, and the number of clinic visits reported for each type, were summarized for the overall 12-month study period and for 62-month assessment periods (0–2, 2–4, 4–6, 6–8, 8–10, and 10–12 months).

Patient Reported Outcomes

Our primary outcomes, pain interference and physical function, were measured using the 29-item Patient Reported Outcomes Measurement Information System (PROMIS-29 profile v 2.1) completed on paper or electronic forms.²⁸ Use of PROMIS measures is recommended by the NIH Task Force on Research Standards for Chronic Low Back Pain.²⁹ PROMIS physical function is comprised of four items that have similar internal consistency and validity when compared to longer established measures of back-related physical function, e.g., the Roland-Morris Disability Questionnaire.³⁰

Secondary outcomes were pain intensity, anxiety, depression, fatigue, sleep disturbance, and social participation. Pain intensity was measured on an 11-point (0 to 10) numerical rating scale with higher scores indicating more severe pain. The remaining outcomes were PROMIS-29 domains comprised of four items rated on a five-point Likert scale over the past seven days. We used the recommended T-score method for interpretation of scores on all PROMIS-29 domains.²⁸ This method results in a score that ranges from 20 to 80, where a score of 50 represents the reference population average and +/− 10 points corresponds to one standard deviation away from the average. High T-scores (55-60 mild; 60-70 moderate; 70 and above severe) indicate clinically relevant pain interference, anxiety, depressive symptoms, fatigue, or sleep disturbance. Low physical function and social participation are indicated by below average T-scores (45-40 mild; 40-30 moderate; below 30 severe).²⁸

Patient Characteristics

Sociodemographic characteristics included age, sex, race/ethnicity, highest education attainment, employment status, and household income. Using body mass index (BMI, kg/m²), we defined underweight (<18.5), normal weight (18.5–25), overweight (25-30), and obese (>30). We used the patient activation measure (PAM), a brief 13-item validated instrument assessing knowledge, skills, and confidence in managing one’s health.³¹ Respondents with LBP who selected “yes” to the question (“Have you experienced any pain rated as 4 or greater on a scale of 0 to 10 for a duration of 3 months or longer?”) were considered to have chronic back pain. Other participants were classified as having acute back pain. We assessed by self-report whether the participant had received an opioid prescription. We identified participants who had an index visit on the same day as their first visit to the IM clinic and who had no self-reported use of IM approaches in the previous 6 months.

Statistical Analysis

Demographic and clinical characteristics are presented using descriptive statistics. All analyses were performed by using SAS 9.4 (SAS Institute Inc., Cary NC, USA) or R statistical software (R Core Team, 2021). This is not a powered study. However, to aid interpretation of findings, we have included a 95% confidence interval for all estimates which is based on a p < .05 threshold for statistical significance.

Healthcare Utilization

Use of individual IM services and the number of IM services over the 1-year follow-up period were presented using descriptive statistics. We also counted the number of guideline-recommended nonpharmacologic treatments utilized by participants with LBP using ACP guidelines, including acupuncture, biofeedback, chiropractic care, massage, physical therapy, tai chi, and yoga.⁴ To explore complex patterns of IM service use we performed a cluster analysis allowing us to create a heatmap and dendrogram to visualize patterns of IM service use at the level of the individual participant.³² This involved hierarchical clustering by type of IM service and the number of visits across the seven 2-month windows between 2 months prior and 12 months after the index visit. Complete linkage was used as the dissimilarity metric after log2(1 + x) transformation of the number of visits.³²

Patient Reported Outcomes

The primary clinical outcomes of interest were changes in PROMIS pain interference and physical function scores. To account for longitudinal changes in outcomes that may depend on the outcome level at index visit (i.e., pain interference or physical functioning at index), we fit a linear mixed effects model using the patient-reported outcome at each visit as the dependent variable. An interaction term between time (2, 4, 6 or 12-month) and the outcome level at the index visit (categorized by tertile) were the predictors of interest. We included the main effects of outcome level at index visit, as well as age and sex as covariates. We accounted for correlation within repeated measures from the same subjects and same site using random intercepts. We also measured overall change in primary outcomes (not stratified by outcome level at index visit) with the time indicators as the predictors of interest and the intercept, age, and sex as covariates. We repeated an analysis of overall change for secondary outcomes (PROMIS anxiety, depression, fatigue, sleep disturbance, social participation, and pain intensity).

Sensitivity Analyses

Potential selection bias due to missing follow-up data on outcomes was addressed by applying stabilized inverse probability weights (IPW) to the multivariable model for the pain interference outcome.³³ The use of this method in the PRIMIER cohort is described in detail elsewhere.²⁶

Exploratory Subgroup Analyses

We explored post hoc whether characteristics at index visit were predictors of short-term (2-month) improvements in PROMIS pain interference and physical function. This included sex- and age-stratified results to estimate changes for men, women, and participants <65 and ≥65 years old. We estimated changes among patients identified as naïve to IM approaches (defined as index visit is first visit to IM clinic; no IM service use in the previous 6 months). We also stratified on pain-related characteristics (acute/chronic pain; current use of opioids). Lastly, we stratified on the four levels of the Patient Activation Measure (level 1: starting to take a role; level 2: building knowledge and confidence; level 3: taking action; level 4: maintaining behaviors). We used a t test or ANOVA to evaluate between group differences.

Results

From August 2013 to November 2018, 4883 adults were enrolled in PRIMIER. Of 3658 participants with a complete baseline survey, available medical record data, and ≥1 IM service documented in their participation year, 660 (18%) had LBP (Figure 1). All 17 clinical sites contributed participants with LBP to the analysis (median = 34 individuals, range 1-105, Appendix Table 1). Participants with LBP (mean age = 51.6) were mostly female (75%) and White (85%) and 66% were characterized as having chronic LBP (Table 1). Most had private insurance (70.2%) and planned to bill their insurance for care (69.7%). Over a third of patients reported that their index visit co-occurred with their first visit to the IM clinic (37%), although nearly all (89%) reported use of IM services or practices in the past 6 months. On PROMIS measures, participants had scores indicating mild to moderate pain interference (mean = 59.9, SD = 9.3), physical function limitations (mean = 43.1, SD = 9.1) and pain intensity (mean = 4.9, SD = 2.4). Outcome data were available for 443 patients (67% of patients with LBP) who had at least one completed follow-up survey. Participants with outcome data were generally similar to overall sample (Table 1).

Figure 1.

Flow diagram.

Table 1.

Characteristics of Sample of PRIMIER Participants With Low Back Pain.

Characteristics^a	Utilization cohort (n = 660)	Outcomes cohort (n = 443)
Age, mean (SD)	51.6 (13.9)	52.3 (13.8)
Female	489/653 (74.9)	338/437 (77.3)
Race
White	537/635 (84.6)	368/428 (86.0)
Black	46/635 (7.2)	29/428 (6.8)
Other	52/635 (8.2)	31/428 (7.2)
Ethnicity, Hispanic or Latino	53/631 (8.4)	31/425 (7.3)
Education, college degree or higher	476/655 (72.7)	335/438 (76.5)
Working full-time	278/650 (42.8)	192/435 (44.1)
Income
< $20,000	91/612 (14.9)	54/412 (13.1)
$20,000-50,000	123/612 (20.1)	82/412 (19.9)
$50,001-$100,000	186/612 (30.4)	125/412 (30.3)
>$100,000	212/612 (34.6)	145/412 (35.2)
Primary health insurance
Private	457/651 (70.2)	311/435 (71.5)
Medicare	150/651 (23.0)	95/435 (21.8)
Medicaid	37/651 (5.7)	24/435 (5.5)
None	7/651 (1.1)	5/435 (1.1)
Plan to bill insurance, yes	384/551 (69.7)	278/400 (69.5)
Index visit is first visit to IM clinic	183/494 (37.0)	106/331 (32.0)
IM service use in prior 6 months^b	582/654 (89.0)	403/439 (91.8)
BMI, mean (SD)	27.6 (6.6)	27.2 (6.5)
Patient activation measure categories
Starting to take a role	82/655 (12.5)	38/441 (8.6)
Building knowledge and confidence	87/655 (13.3)	56/441 (12.7)
Taking action	178/655 (27.2)	122/441 (27.7)
Maintaining behaviors	308/655 (47.0)	225/441 (51.0)
Chronic back pain	437 (66.2)	283 (63.9)
Pain intensity (0-10), mean (SD)	4.9 (2.4)	4.7 (2.3)
PROMIS-29, mean (SD)^c
Pain interference	59.9 (9.31)	58.9 (9.1)
Physical function	43.1 (9.11)	44.0 (8.8)
Anxiety	55.3 (9.46)	54.6 (9.2)
Depression	52.7 (9.53)	51.8 (9.0)
Fatigue	56.3 (10.21)	55.5 (9.6)
Sleep disturbance	53.0 (8.25)	52.8 (8.0)
Social participation	46.7 (8.94)	47.4 (8.6)
Opioid prescription, any	95/463 (20.5)	55/318 (17.3)

IM, Integrative Medicine; PAM; Patient Activation Measure.

^aValues are n (%) unless otherwise noted.

^bSelf-report of any integrative medicine services either at institutions or outside the healthcare system.

^cHigher scores indicate worse pain interference, fatigue, sleep disturbance, anxiety and depression; and better physical and social participation with a score of 50 as the average score for the general US population.

Availability of Clinical Services

We have summarized the general availability of IM services, including ACP guideline-recommended treatments for LBP,⁴ as reported by site representatives in Appendix Table 2. IM consults and acupuncture were available at all sites; several services were reported at most sites (massage, MBSR, tai chi, yoga). A minority of clinics reported offering chiropractic care (41%) or physical therapy (24%).

Healthcare Utilization

Table 2 shows the use of IM services among the overall sample of 660 participants with LBP. Over the 12-month period, the most commonly utilized services were IM consults (56%), acupuncture (44%), chiropractic care (24%), physical therapy (19.5%), and massage (17%). Over two-thirds (70%) of participants received at least one ACP guideline-concordant nonpharmacologic treatment. The median number and range of visits is also presented in Table 2 for each IM service. For example, among those who received an IM consult, the median number of IM consults among participants was 3 and the range of ‘IM consult’ visits ranged from 1 to 37.

Table 2.

Summary of IM Service Use Over 12-Month Period.

Services	Utilization Cohort (n = 660)	Outcomes Cohort (n = 443)
Integrative medicine consult
Any, No. (%)	372 (56.4)	250 (56.4)
Number of visits, median (range)	3 (1, 37)	3 (1, 37)
Acupuncture
Any, No. (%)	288 (43.6)	195 (44.0)
Number of visits, median (range)	6 (1,43)	7 (1, 43)
Chiropractic care
Any, No. (%)	160 (24.2)	126 (28.4)
Number of visits, median (range)	5.5 (1, 45)	6 (1, 45)
Massage
Any, No. (%)	112 (17)	85 (19.2)
Number of visits, median (range)	3 (1, 56)	3 (1, 56)
Physical therapy
Any, No. (%)	129 (19.5)	94 (21.2)
Number of visits, median (range)	3 (1, 65)	3.5 (1, 64)
Other IM services^a
Any, No. (%)	105 (15.9)	67 (15.1)
Number of visits, median (range)	2 (1, 18)	2 (1, 18)
Number of IM services^b, No. (%)
One	335 (51)	213 (48)
Two	204 (31)	138 (31)
Three	72 (11)	49 (11)
Four or more	49 (7)	43 (10)
Number of ACP guideline recommended nonpharmacologic treatments^c, No. (%)
None	198 (30)	120 (27.1)
One	298 (45.2)	199 (44.9)
Two	107 (16.2)	76 (17.2)
Three	45 (6.8)	39 (8.8)
Four or more	12 (1.8)	9 (2)

ACP: American College of Physicians.

^aOther Integrative medicine encounters observed included prolotherapy, herb, homeopathy, allergy, infusion, energy therapy (including reiki), coaching consult (e.g., nutrition, general health and wellness), mind and body (e.g., biofeedback, hypnotherapy, tai chi, yoga), other manual therapy (e.g., trigger point, frequency specific microcurrent, rolfing).

^bNumber of unique IM services participants accessed at least one time for the six main categories summarized in this table and in cluster analysis shown in Figure 1 (i.e., integrative medicine consult, acupuncture, chiropractic care, massage, physical therapy, and other).

^cNumber of unique guideline-concordant nonpharmacologic treatments participants accessed at least one time. Guideline-concordant based on the 2017 guideline from the American College of Physicians and included acupuncture, biofeedback, chiropractic care, massage, physical therapy, tai chi, and yoga.

IM services were often combined. Half of the participants (49%) reported engagement with multiple IM services during the follow-up period, i.e., 31%, 11%, and 7% reported use of two, three, and four or more IM services, respectively (Table 2). Some of the most common combinations of IM services included: IM consult with acupuncture (18.5%), chiropractic care with physical therapy (13.5%), and IM consult with chiropractic care (12.0%). Patterns of use were similar for participants with and without follow-up outcome data.

Results from cluster analyses are presented in a heat map illustrating patterns of IM service use over one year displayed in 2-month increments (Figure 2). Many participants had low use of IM services (e.g., participants with a single IM consult), while other participants had multiple visits for LBP including use of a range of IM services. Figure 2 displays the heterogeneity of participant engagement in IM services.

Figure 2.

Heat map of IM service use among 660 adults with low back pain. Note: Each of the 660 participants with LBP are represented as a row in the heat map with patients receiving similar IM services clustered together. Use of IM services is represented in six main columns (i.e., IM consult, acupuncture [acu.], chiropractic care [chiro.], massage, physical therapy [PT], and other). The following ‘other’ Integrative medicine encounters were observed: prolotherapy, herb, homeopathy, allergy, infusion, energy therapy (including reiki), coaching consult (e.g., nutrition, general health and wellness), mind and body (e.g., biofeedback, hypnotherapy, tai chi, yoga), other manual therapy (e.g., trigger point, frequency specific microcurrent, rolfing). Within each major column there are smaller bands with the color of the band corresponding to the number of clinical visits per each 2-month assessment period. Light grey corresponds to no use of the service and shades of color correspond to the number of visits for a particular service in the 2-month period as indicated in the legend, e.g., dark blue corresponds to use of a service once in a 2-month period. To the left of the columns is a dendrogram, which shows the hierarchical relationship between individual rows representing the use of IM services by individual participants as described above. The leftmost side of the dendrogram shows the first split of the entire sample into two clusters, with subsequent splits from left to right further dividing the sample into additional clusters.

Patient Reported Outcomes

Changes in PROMIS-29 pain interference and physical function outcomes adjusting for age and sex are shown in Table 3. The results are consistent with the unadjusted analysis shown in Appendix Figure 1. A modest reduction in pain interference was observed in the short- and long-term (2-month MD = −1.26, 95%CI = −2.01,-0.50; and 12-month MD = −1.98, 95%CI = −2.85,-1.11). By contrast, improvements in physical function were small and not statistically significant (2-month MD = 0.29, 95%CI = −0.34,0.92; and 12-month MD = 0.69, 95%CI = −0.03,1.41). After adjusting for the pain level at the index visit using tertiles, a larger reduction in pain interference was observed in the short- and long-term for the medium and high pain tertile groups, but not for the low tertile group. For physical function, a clinically significant improvement was observed in the lowest physical function groups (low and medium tertiles). In comparison, there was a small worsening of function in those who started with the highest tertile of function at index. Secondary outcomes are also shown in Table 3. At twelve months, small improvements were observed on all secondary outcomes (pain intensity, anxiety, depression, and social participation) except fatigue and sleep disturbance.

Table 3.

Patient Reported Outcomes Among 443 Adults With Low Back Pain.

	Index visit Mean (SD)	Adjusted^a 2-month Mean change (95% CI)	Adjusted^a 12-month Mean change (95% CI)
Primary outcomes
PROMIS pain interference
Overall	58.9 (9.1)	−1.26 (−2.01, −0.50)	−1.98 (−2.85, −1.11)
Low tertile	49.5 (5.9)	0.92 (−0.27, 2.11)	0.17 (−1.17, 1.50)
Middle tertile	60.0 (2.0)	−2.04 (−3.40, −0.68)	−2.06 (−3.68, −0.44)
High tertile	68.3 (4.0)	−3.10 (−4.42, −1.79)	−4.35 (−5.86, −2.85)
PROMIS physical function
Overall	44.0 (8.8)	0.29 (−0.34, 0.92)	0.69 (−0.03, 1.41)
Low tertile	35.1 (3.9)	1.22 (0.16, 2.29)	2.57 (1.33, 3.81)
Middle tertile	44.0 (2.9)	0.97 (0.05, 1.89)	1.60 (0.55, 2.66)
High tertile	56.6 (3.9)	−2.34 (−3.56, −1.11)	−3.31 (−4.68, −1.94)
Secondary outcomes
Global pain (0-10)	4.7 (2.3)	−0.56 (−0.75, −0.37)	−0.64 (−0.85, −0.42)
PROMIS anxiety	54.6 (9.2)	−0.56 (−1.35, 0.22)	−1.45 (−2.35, −0.56)
PROMIS depression	51.8 (9.1)	−0.57 (−1.31, 0.17)	−1.62 (−2.46, −0.78)
PROMIS fatigue scale	55.5 (9.6)	−0.62 (−1.45, 0.21)	−0.05 (−1.00, 0.89)
PROMIS sleep disturbance	52.8 (8.0)	−0.75 (−1.49, −0.02)	−0.23 (−1.07, 0.61)
PROMIS social participation	47.4 (8.6)	0.32 (−0.44, 1.08)	2.23 (1.37, 3.10)

^aEstimates for mean change for primary and secondary outcomes are adjusted for age and gender. Mean changes of primary outcomes are presented overall and for each of pain interference and physical function tertiles. Improvements are positive scores for physical function (increasing function) and negative cores for pain interference (reduced pain interference).

Sensitivity Analyses

A sensitivity analysis applying stabilized inverse probability weights to address missing outcome data found similar results in changes in observed pain interference at 12 weeks (Appendix Table 3).

Subgroup Analyses

Exploratory subgroup analyses are shown in Appendix Table 4. Improvements were larger among individuals who reported that their index visit was their first visit to the IM clinic compared to participants who reported prior visits to the IM clinic for pain (2-month MD = −1.14; 95%CI = −3.22, 0.94) and physical function (2-month MD = 1.48; 95%CI = −0.12, 3.08), although differences were not statistically significant. Larger improvements in pain and physical function were also observed for participants under the age of 65 compared to adults aged 65 or older for pain (2-month MD = −2.10, 95%CI = −4.29, 0.08) and physical function (2-month MD = 1.89, 95%CI = 0.13, 3.65). While no other differences between groups were statistically significant, additional subgroups with larger improvements in pain and physical function were men, those with chronic back pain, and those characterized as level 3 on PAM: “taking action on their health”.

Discussion

In this analysis of a national prospective cohort of patients with LBP receiving clinical services at 17 IM clinics, two-thirds of the patients received at least one ACP guideline-concordant nonpharmacologic treatment during the 12 months of their study participation. Patients with LBP receiving care at IM clinics had modest improvements in PROMIS-29 pain interference and most relevant secondary outcomes, but not in physical function.

We identified a set of common IM services that may represent ‘usual care’ for LBP in the IM clinic context. This included several evidence-based guideline-recommended nonpharmacologic treatments.^4,5 This contrasts with ‘usual care’ for LBP observed in the primary care context where few patients receive nonpharmacologic treatment (apart from physical therapy) and most receive pain medications.^7-9 For example, while nearly half of PRIMIER participants had documented acupuncture use, the estimated prevalence of acupuncture use among Americans, or Americans with LBP, is much lower (1%–3%).^16,34,35 Our findings are more consistent with estimated acupuncture use among Americans who report IM use for pain conditions, which was over 70% in 2022.¹⁶ Our findings are also consistent with a previous cluster analysis IM service use for LBP in a single IM clinic,¹⁷ suggesting that IM clinics improve access to care that is consistent with current clinical practice guidelines.^4,5

Half of the participants received an ‘IM consult’. Participants receiving an IM consult had a median of three visits, with a range of one to thirty-seven IM consults observed for a single participant during the follow-up period. This suggests IM consults can involve a single visit, potentially with some follow-up, involving education and coordination of services. However, it appears that IM consults can also involve a longitudinal sequence of visits, monthly or more frequently. While we anticipate more frequent visits involve the teaching of self-care techniques (e.g., mindfulness, acupressure) or delivery of services (e.g., acupuncture, biofeedback), the content of these visits could not easily be captured using the EHR. This is in large part because billing for an IM consult typically involves the use of evaluation and management billing codes (e.g., 99204/99214) without the use of procedure codes (e.g., 97810 for acupuncture).^12-15 Future research is needed to explore and evaluate the content of IM consult visits for LBP and other common health conditions.

We observed modest improvements in pain interference among individuals receiving IM services. While this result is consistent with systematic reviews and meta-analyses supporting use of nonpharmacologic treatments for LBP,^10,11 changes in pain interference were smaller than expected and physical function did not improve. One potential explanation for the modest changes is that we included individuals with a range of mild, moderate, and severe LBP at their index visit. The changes we observed for those who began with moderate and severe pain are robust and more similar to what might be observed in an RCT where individuals with mild symptoms are typically excluded. Alternatively, our findings may reflect how nonpharmacologic treatments are currently delivered in real-world clinical settings in terms of variability of dose, frequency, and individual components. For example, while a previous RCT determined 12 visits to be the most effective dose of spinal manipulative therapy for chronic LBP,³⁶ the median number of visits to a chiropractor in our study was lower, i.e., 6 visits. Quality improvement efforts to facilitate fidelity to a known optimal dose may improve real-world outcomes. With PRIMIER, we also see that patients utilize novel combinations of IM services frequently; these are combinations that have not been evaluated in RCTs. While some combinations are starting to be evaluated,^37-39 it seems unlikely to efficiently assess all possible IM service combinations and doses in the traditional RCT. However, this may be feasible in a larger prospective cohort using modern causal inference methods, e.g., target trial emulation.⁴⁰

The study of IM service implementation in large healthcare delivery systems has predominantly occurred in the Veterans Health Administration (VHA). In the VHA context, IM services are associated with positive impacts on mental health outcomes, with some heterogeneity on pain outcomes.^41-44 While the engagement in IM services for pain within the VHA has been high, implementation challenges remain and include a high demand for services⁴⁵ that may outpace availability, space, and staffing.⁴⁶ In addition, the VHA has noted differences in IM use by race and ethnicity explained by geographic access issues.⁴⁷ Our study sample was majority white, female, higher education level, and higher income. This is consistent with prior evidence that suggests socioeconomic disparities in nonpharmacologic treatment use among civilian and veteran populations with LBP.^16,48 IM clinics may be a novel strategy to deliver nonpharmacologic treatments, particularly in civilian healthcare settings where implementation has been limited. Limited insurance coverage and financial barriers are often perceived as a significant barrier to nonpharmacologic treatment use.^49,50 However, it is important to note that several guideline-recommended nonpharmacologic treatments that are reimbursable remain uncommon in civilian healthcare systems (e.g., acupuncture, chiropractic care).⁵¹ Thus, further efforts are needed to identify and address barriers to ensure equitable implementation.^49,50

Our study has several limitations. We relied on EHR data to identify receipt of IM services within IM clinics, which may underestimate the overall use as it would exclude community-based IM services or those practiced at home. In addition, not all IM services would be captured by billing codes utilized in this cohort. For example, an IM consult may involve multiple services while utilizing a single traditional evaluation and management billing code. Similarly, some general categories of care were generated by profession (e.g., chiropractic care, physical therapy) with common reimbursable codes including spinal manipulation (i.e., CPT 98940-98942) and therapeutic exercise (i.e., CPT 97110) although these professionals often take a multicomponent approach including non-billable components.^52,53 The frequency and type of IM services participants use (or the components professionals deliver) may reflect cost and insurance factors that we were not able to capture in this data set. Relying on EHR data may also underestimate services that are rarely or never documented in the EHR. For example, most PRIMIER sites indicated yoga, tai chi and MBSR classes were offered at their IM clinic. However, these services were only rarely documented in the EHR which may imply that sites are unaware of potentially reimbursable treatment codes (e.g., CPT 97150) or that these treatment codes are not being regularly reimbursed despite yoga, tai chi and MBSR being recommended by clinical practice guidelines.⁴ Not entering referrals or treatment codes into the EHR makes it difficult for researchers, healthcare systems, and referring clinicians, to understand the use of these IM approaches. Future research should evaluate the extent to which patients of IM clinics utilize IM services both within and outside of their healthcare system.

Additional limitations may impact the interpretation of changes in patient reported outcomes. First, the index visit was not the first IM clinic visit for most participants. Thus, benefits from IM services may have occurred before enrollment in PRIMIER, which would reduce the magnitude of overall clinical change that we observed. This hypothesis is supported by our exploratory subgroup analysis showing larger improvements in pain and physical function among participants whose index visit coincided with their first clinic visit compared to those who reported prior visits to the IM clinic, although the differences were not statistically significant. Second, back-related IM services varied in terms of type, number, and combinations of IM services received. Thus, it is difficult to attribute changes to a particular IM service, IM service dose, or combination of IM services. Our findings may better reflect the effectiveness of the overall approach of care delivery (e.g., the IM clinic). Third, this is a descriptive study without a control group. It is possible that changes are explained by the natural history of LBP which is generally thought to be favorable, particularly for acute LBP.⁵⁴ However, it is important to note that we observed larger changes in patients reporting chronic pain at their index visit, where the natural history of LBP is less favorable. In addition, we did not have access to data about what pharmacologic treatments participants may also have been using. Participants may have received pharmacologic treatments as part of IM consults or through engagement in usual primary care. Lastly, missing follow-up data on clinical outcomes from a proportion of the sample may have introduced bias. We believe the risk of bias is low given our sensitivity analyses which yielded similar results. Nonetheless, routine capture of patient reported outcomes in the EHR, rather than paper or electronic surveys, may improve yield in future research.²⁶

Conclusions

Most patients with LBP receiving care at IM clinics received at least one guideline-recommended nonpharmacologic treatment. However, improvements in clinical outcomes were relatively small. Additional multi-site studies are needed to explore the optimal implementation approach for nonpharmacologic treatments in large healthcare delivery systems.

Supplemental Material

Supplemental Material - Nonpharmacologic Back Pain Treatment Use and Associated Patient Reported Outcomes in US-Based Integrative Medicine Clinics

Supplemental Material for Nonpharmacologic Back Pain Treatment Use and Associated Patient Reported Outcomes in US-Based Integrative Medicine Clinics by Eric J. Roseen, Emily E. Hurstak, Ryung S. Kim, Qi Gao, Carol M. Greco, David R. Vago, Robert B. Saper, Benjamin Kligler, M. Diane McKee, Jeffery A. Dusek, and For the PRIMIER Study in Global Advances in Integrative Medicine and Health.

Footnotes

Acknowledgement

We are indebted to our participants for their invaluable contributions to this project. We are also grateful to our site collaborators who contributed enormously to the success of the PRIMIER: Alliance Institute for Integrative Medicine: Steve Amoils MD, Sandy Amoils MD, Nancy Merrel, Alex Crumley; Boston Medical Center at Boston University: Paula Gardiner MD, Chelsey Lemaster MD, MPH, Iniya Rajendran, MD, MPH; Center for Integrative Medicine at the University of Maryland School of Medicine: Brian Berman MD, William Rollow MD, Chris D’Adamo PhD, Mary Bahr, Mei Zheng; Duke Integrative Medicine at Duke University: Adam Perlman MD, Elizabeth Matteson-Bechard, Teresa Keever; Integrative Medicine at the University of Colorado Denver: Lisa Corbin MD, Jacinda Niklas MD, Kayla Mieczkowski; Jefferson Myrna Brind Center for Integrative Medicine at Thomas Jefferson University: Andrew Newberg MD, Nancy Wintering MSW; Mt. Sinai Mount Sinai Beth Israel Medical Center for Health and Healing: Woodson Merrell MD, Jenna Cantor, Elidania Perez; Osher Center for Integrative Medicine at Northwestern University; Melinda Ring MD, David Victorson PhD, Natalie Pace; Osher Center for Integrative Medicine at the University of California- San Francisco: Margaret Chesney PhD, Kevin Barrows MD, Vierka Goldman; Osher Center for Integrative Medicine at the Vanderbilt University: Roy Elam MD, Gurjeet Birdee MD, Isabella Ahrens; Penny George Institute for Health and Healing: Courtney Baechler MD, Stephanie Wallerius; Scripps Center for Integrative Medicine: Chris Suhar MD, Eva Stuart, RN, Tiffany Bryant; Simms/Mann Health and Wellness Center, Program in Integrative Medicine at Venice Family Clinic: Myles Spar MD, Nancy Rodriguez, Ryan Woodson, Cristina Perez and the University of Pittsburgh Center for Integrative Medicine: Neil Ryan MD, Carol Greco PhD, Christine McFarland. We also thank our colleagues at the US Department of Veteran’s Affairs Office of Patient-Centered Care and Cultural Transformation: Tracy Gaudet MD; US Department of Veteran’s Affairs Quality Enhancement Research Initiative: Barbara Bokour PhD, Rani Elway PhD, and Dorothy Plumb. Special gratitude to Claudia Lechuga, Albert Einstein College of Medicine of Yeshiva University (current BraveNet Data and Statistical Coordinating Center) and to Ron Roddy, Duke Clinical Research Institute (prior BraveNet Data and Statistical Coordinating Center), for successfully managing this multicenter project.

Author Contributions

Authors: Roseen, Kim, Gao, Dusek had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Roseen, Hurstak, Kim, Gao, Greco, Vago, Saper, Kligler, McKee, Dusek. Acquisition, analysis, or interpretation of data: Roseen, Hurstak, Kim, Gao, Greco, Vago, Saper, Kligler, McKee, Dusek. Critical revision of the manuscript for important intellectual content: Roseen, Hurstak, Kim, Gao, Greco, Vago, Saper, Kligler, McKee, Dusek. Statistical analysis: Kim, Gao, Obtained funding: Roseen, Dusek. Administrative, Technical, or Material Support: Roseen. Study supervision: Roseen, Dusek.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding to support the PBRN and PREMIER study was generously provided by The Bravewell Collaborative. Dr. Roseen is the recipient of a career development award from the National Center for Complementary and Integrative Health (NCCIH, K23-AT010487), which supported his work on this manuscript. The content is solely the responsibility of the authors and does not necessarily represent the official views of NCCIH. The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. The contents of this manuscript are solely the responsibility of the authors and do not necessarily represent the official views of the Bravewell Collaborative or NCCIH.

Clinical Trial

ClinicalTrials.gov NCT01754038

ORCID iDs

Eric J. Roseen

Carol M. Greco

Benjamin Kligler

Jeffery A. Dusek

Supplemental Material

Supplemental material for this article is available online.

References

Dieleman

Cao

Chapin

, et al. US Health care spending by Payer and health condition, 1996-2016. JAMA. 2020;323(9):863-884.

Collaborators

GLBP

. Global, regional, and national burden of low back pain, 1990–2020, its attributable risk factors, and projections to 2050: a systematic analysis of the Global Burden of Disease Study 2021. Lancet Rheumatol. 2023;5(6):E316-E329.

USBoD Collaborators Mokdad

Ballestros

, et al. The state of US health, 1990-2016: Burden of diseases, injuries, and risk factors among US States. JAMA. 2018;319(14):1444-1472.

Qaseem

Wilt

McLean

, et al. Clinical guidelines Committee of the American College of P. Noninvasive treatments for acute, subacute, and chronic low back pain: A clinical practice guideline from the American College of Physicians. Ann Intern Med. 2017;166(7):514-530.

WHO Guideline for Non-surgical Management of Chronic Primary Low Back Pain in Adults in Primary and Community Care Settings. Geneva: World Health Organization; 2023. Licence: CC BY-NC-SA 3.0 IGO.

Skelly

Chou

Dettori

, et al. Noninvasive Nonpharmacological Treatment for Chronic Pain: A Systematic Review Update. Rockville (MD); 2020.

Roseen

Conyers

Atlas

Mehta

. Initial management of acute and chronic low back pain: Responses from brief interviews of primary care providers. J Altern Complement Med. 2021;27(S1):S106-S114.

Kamper

Logan

Copsey

, et al. What is usual care for low back pain? A systematic review of health care provided to patients with low back pain in family practice and emergency departments. Pain. 2020;161(4):694-702.

Joyce

Roseen

Smith

, et al. A cluster analysis of initial primary care orders for patients with acute low back pain. JABFM. 2023;1-16. (Ahead of print).

10.

Chou

Deyo

Friedly

, et al. Noninvasive Treatments for Low Back Pain. Rockville, MD: Agency for Healthcare Research and Quality (US); 2016.

11.

Skelly

Chou

Dettori

, et al. Noninvasive Nonpharmacological Treatment for Chronic Pain: A Systematic Review. Rockville, MD: Agency for Healthcare Research and Quality (US); 2018.

12.

NCCIH . Complementary, alternative, or integrative health: what’s in a name? 2021. https://www.nccih.nih.gov/health/complementary-alternative-or-integrative-health-whats-in-a-name. Accessed 5 March 2024.

13.

Jonas

Mehta

. Integrative care interest and offerings grow. NEJM Catalyst Innovations in Care Delivery. 2023;4(9):357.

14.

Langevin

. Moving the complementary and integrative health research field toward whole person health. J Altern Complement Med. 2021;27(8):623-626.

15.

Frenkel

Cohen

Peterson

Palmer

Swint

Bruera

. Integrative medicine consultation service in a comprehensive cancer center: findings and outcomes. Integr Cancer Ther. 2010;9(3):276-283.

16.

Nahin

Rhee

Stussman

. Use of complementary health approaches overall and for pain management by US adults. JAMA. 2024;9:16.

17.

Aickin

McCaffery

Pugh

, et al. Description of a clinical stream of back-pain patients based on electronic medical records. Complement Ther Clin Pract. 2013;19(3):158-176.

18.

Maiers

Westrom

Legendre

Bronfort

. Integrative care for the management of low back pain: use of a clinical care pathway. BMC Health Serv Res. 2010;10:298.

19.

Eisenberg

Kaptchuk

Post

, et al. Establishing an integrative medicine program within an academic health center: essential considerations. Acad Med. 2016;91(9):1223-1230.

20.

Taylor

Bolton

Huynh

, et al. What should health care systems consider when implementing complementary and integrative health: Lessons from veterans health administration. J Altern Complement Med. 2019;25(S1):S52-S60.

21.

Herman

Dodds

Logue

, et al. IMPACT--Integrative Medicine PrimAry Care Trial: protocol for a comparative effectiveness study of the clinical and cost outcomes of an integrative primary care clinic model. BMC Complement Altern Med. 2014;14:132.

22.

Wayne

Eisenberg

Osypiuk

, et al. A multidisciplinary integrative medicine team in the treatment of chronic low-back pain: An observational comparative effectiveness study. J Altern Complement Med. 2018;24(8):781-791.

23.

Sundberg

Petzold

Wandell

Ryden

Falkenberg

. Exploring integrative medicine for back and neck pain - a pragmatic randomised clinical pilot trial. BMC Complement Altern Med. 2009;9:33.

24.

Crocker

Hurwitz

Grizzle

, et al. Real-world evidence from the integrative medicine primary care trial (IMPACT): Assessing patient-reported outcomes at baseline and 12-month follow-up. Evid Based Complement Alternat Med. 2019;2019:8595409.

25.

Dusek

Abrams

Roberts

, et al. Patients receiving integrative medicine effectiveness Registry (PRIMIER) of the BraveNet practice-based research network: study protocol. BMC Complement Altern Med. 2016;16:53.

26.

Dusek

Gao

Kim

, et al. Patients receiving integrative medicine effectiveness Registry (PRIMIER) of the BraveNet practice-based research network: outcomes of the PRIMIER cohort. Complement Ther Med. 2022;71:102904.

27.

Mold

Peterson

. Primary care practice-based research networks: working at the interface between research and quality improvement. Ann Fam Med. 2005;3(Suppl 1):S12-20.

28.

Craig

Reeve

Brown

, et al. US valuation of health outcomes measured using the PROMIS-29. Value Health. 2014;17(8):846-853.

29.

Deyo

Dworkin

Amtmann

, et al. Report of the NIH Task Force on research standards for chronic low back pain. J Pain. 2014;15(6):569-585.

30.

Deyo

Katrina

Buckley

, et al. Performance of a patient reported outcomes measurement information system (PROMIS) short form in older adults with chronic musculoskeletal pain. Pain Med. 2016;17(2):314-324.

31.

Hibbard

Stockard

Mahoney

Tusler

. Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res. 2004;39(4 Pt 1):1005-1026.

32.

Hartigan

. Clustering Algorithms. New York, NY: Wiley; 1975.

33.

Seaman

White

. Review of inverse probability weighting for dealing with missing data. Stat Methods Med Res. 2013;22(3):278-295.

34.

Ghildayal

Johnson

Evans

Kreitzer

. Complementary and alternative medicine use in the US adult low back pain population. Glob Adv Health Med. 2016;5(1):69-78.

35.

Candon

Nielsen

Dusek

, et al. Utilization of reimbursed acupuncture therapy for low back pain. JAMA Netw Open. 2024;7(8):e2430906.

36.

Haas

Vavrek

Peterson

Polissar

Neradilek

. Dose-response and efficacy of spinal manipulation for care of chronic low back pain: a randomized controlled trial. Spine J. 2014;14(7):1106-1116.

37.

Teets

Nielsen

Moonaz

, et al. Group acupuncture therapy with yoga therapy for chronic neck, low back, and osteoarthritis pain in safety net settings for an underserved population: a feasibility pilot study. Glob Adv Integr Med Health. 2023;12:27536130231202515.

38.

Zeliadt

Coggeshall

Gelman

, et al. Assessing the relative effectiveness of combining self-care with practitioner-delivered complementary and integrative health therapies to improve pain in a pragmatic trial. Pain Med. 2020;21(Suppl 2):S100-S109.

39.

Burton

Wayne

Litrownik

, et al. Integrating chiropractic care and tai chi training for the treatment of chronic nonspecific neck pain in nurses: a single-arm mixed-methods pilot trial. J Integr Complement Med. 2024;17.

40.

Hernan

Wang

Leaf

. Target trial emulation: a framework for causal inference from observational data. JAMA. 2022;328(24):2446-2447.

41.

Elwy

Taylor

Zhao

, et al. Participating in complementary and integrative health approaches is associated with veterans' patient-reported outcomes over time. Med Care. 2020;58(Suppl 2 9S):S125-S132.

42.

Bokhour

DeFaccio

Gaj

, et al. Changes in patient-reported outcomes associated with receiving whole health in the veteran health administration (VHA)'s national demonstration project. J Gen Intern Med. 2024;39(1):84-94.

43.

Smeeding

Bradshaw

Kumpfer

Trevithick

Stoddard

. Outcome evaluation of the veterans Affairs Salt Lake City integrative health clinic for chronic nonmalignant pain. Clin J Pain. 2011;27(2):146-155.

44.

Dyer

Surdam

Dusek

. A systematic review of practiced-based research of complementary and integrative health therapies as provided for pain management in clinical settings: Recommendations for the future and a call to action. Pain Med. 2022;23(1):189-210.

45.

Taylor

Gelman

DeFaccio

, et al. We built it, but did they come: Veterans' use of VA healthcare system-provided complementary and integrative health approaches. J Gen Intern Med. 2022;38:1-8.

46.

Farmer

McGowan

Yuan

Whitehead

Osawe

Taylor

. Complementary and integrative health approaches offered in the veterans health administration: results of a national Organizational survey. J Altern Complement Med. 2021;27(S1):S124-S130.

47.

Tobin

Zeliadt

Upchurch

, et al. Racial and ethnic variation in complementary and integrative health therapy use among US veterans. JAMA Netw Open. 2023;6(6):e2318020.

48.

Roseen

Patel

Ward

, et al. Trends in chiropractic care and physical rehabilitation use among adults with low back pain in the United States, 2002 to 2018. J Gen Intern Med. 2024;39(4):578-586.

49.

Roseen

Joyce

Winbush

, et al. Primary care barriers and facilitators to nonpharmacologic treatments for low back pain: a qualitative pilot study. Pharm Manag PM R. 2024;187.

50.

Miller

Roseen

Stone

JAM

, et al. Incorporating acupuncture into American healthcare: Initiating a discussion on implementation science, the status of the field, and stakeholder considerations. Glob Adv Health Med. 2021;10:21649561211042574.

51.

Heyward

Jones

Compton

, et al. Coverage of nonpharmacologic treatments for low back pain among US public and private insurers. JAMA Netw Open. 2018;1(6):e183044.

52.

Jenkins

Downie

Wong

, et al. Patient and provider characteristics associated with therapeutic intervention selection in a chiropractic clinical encounter: a cross-sectional analysis of the COAST and O-COAST study data. Chiropr Man Therap. 2023;31(1):39.

53.

Beliveau

PJH

Wong

Sutton

, et al. The chiropractic profession: a scoping review of utilization rates, reasons for seeking care, patient profiles, and care provided. Chiropr Man Therap. 2017;25:35.

54.

Downie

Hancock

Rzewuska

Williams

Lin

Maher

. Trajectories of acute low back pain: a latent class growth analysis. Pain. 2016;157(1):225-234.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.24 MB