Bridging the gap: Efficacy of combined therapies for cognitive,behavioral,and functional outcomes in Alzheimer's disease – results from a systematic review and network meta-analysis

Abstract

Background

Evidence regarding efficacy of combined pharmaco- and non-pharmacotherapies and their comparative effectiveness for Alzheimer's disease (AD) is limited.

Objective

To estimate the comparative efficacy of pharmacotherapies, non-pharmacotherapies, and combined therapies for improving cognitive, behavioral, and functional outcomes in patients with AD.

Methods

Relevant studies were identified from Medline via PubMed and Scopus databases (March 2021–December 2022). Randomized-controlled trials were eligible if they assessed the efficacy of pharmacotherapies, non-pharmacotherapies, or combined therapies in patients aged 60 years or older, and measuring cognitive, behavioral, or functional outcomes. A network meta-analysis was conducted to estimate relative treatment effects, and interventions were ranked using surface under the cumulative ranking (SUCRA) curve. Confidence in the findings was evaluated using the Confidence in Network Meta-Analysis (CINeMA) framework.

Results

A total of 153 randomized-controlled trials were analyzed. Compared to placebo/usual care, donepezil plus cognitive therapy and rivastigmine plus cognitive rehabilitation significantly improved Mini-Mental State Examination scores. Behavioral outcomes were improved by rivastigmine plus cognitive stimulation, brain stimulation plus exercise, and occupational therapy. Functional status improved significantly with rivastigmine plus cognitive stimulation and exercise. Based on SUCRA ranking, rivastigmine plus cognitive rehabilitation ranked highest for cognitive improvement (92.8%), brain stimulation plus exercise ranked highest for the behavioral outcome (93.1%), and rivastigmine plus cognitive stimulation ranked highest for functional improvement (94.1%).

Conclusions

Donepezil plus cognitive therapy and rivastigmine plus cognitive rehabilitation were the most effective treatments for improving cognitive outcomes. Rivastigmine plus cognitive stimulation ranked highest for both behavioral and functional outcomes, while exercise remains an important strategy for supporting daily functioning in patients with AD.

Keywords

Alzheimer's disease network meta-analysis non-pharmacotherapy pharmacotherapy systematic review

Introduction

Dementia is a significant cause of disability worldwide, with the most common form being Alzheimer's disease (AD).¹ The global Burden of Disease Study 2019 forecasted the number of people with dementia would increase from 57.4 million in 2019 to 152.8 million by 2050, with the highest numbers in North Africa, the Middle East, and eastern sub-Saharan Africa.² The relentless progression of AD impacts patients, families, and society, costing approximately US$1 trillion annually.² The World Health Organization (WHO) published a Blueprint for Dementia Research,³ which identified research gaps and milestone targets. The successful and timely implementation of research outcomes, especially in low-income and middle-income countries (LMICs), is one of the primary goals, given that approximately 60% of people with dementia reside in LMICs.⁴

According to current guidelines,^5,6 the pharmacotherapies recommended for improving cognitive symptoms are acetylcholinesterase inhibitors (AChEIs) including donepezil, rivastigmine, galantamine, and non-competitive antagonists of N-methyl-D-aspartate (NMDA) glutamate receptors (i.e., memantine). Non-pharmacological therapies are also recommended to target cognitive and non-cognitive outcomes, especially for improving behavioral symptoms. Results from several systematic reviews (SR)^7–12 comparing the efficacy among pharmacotherapies (i.e., AChEIs and memantine) suggest that these drugs as monotherapy can reduce the rate of cognitive decline in mild to moderate AD, while combination of AChEIs and memantine is recommended for slowing the cognitive decline in patients with moderate to severe AD. However, these combinations are associated with multiple adverse events (e.g., gastrointestinal pain, weight loss, nausea, and chronic dizziness) leading to high discontinuation rate (>50%).¹³ Currently, there is growing interest in complementary therapies (e.g., huperzine A,¹⁴ ginkgo biloba leaf (EGb761),¹⁵ and traditional Chinese medicines)¹⁶ as well as non-pharmacotherapies (e.g., cognitive stimulation (CS), cognitive rehabilitation (CR), cognitive training (CT), music therapy, exercise, and non-invasive brain stimulation) for slowing dementia progression.^17–20 A recent network meta-analysis (NMA)²¹ comparing efficacy of combined pharmacotherapies and pharmacotherapies combined with non-pharmacotherapies focused solely on depressive symptoms in AD patients, and the results suggested that CS plus AChEIs, exercise, and social interaction were more efficacious than pharmacotherapies alone. However, there is no systematic review and NMA that compares the efficacy of pharmacotherapies, non-pharmacotherapies, and combined pharmacotherapies with non-pharmacotherapies in improving cognitive, behavioral, and functional outcomes in AD patients. Network meta-analysis allows for the comparison of all available treatments within a single analysis by integrating both direct (head-to-head) and indirect evidence across a network of studies. Additionally, NMA can rank treatments based on their efficacy using the Surface Under the Cumulative Ranking curve (SUCRA), which supports clinical decision-making and the development of treatment guidelines.²²

We therefore performed a SR-NMA to compare the efficacy of pharmacotherapies, non-pharmacotherapies, and combined pharmacotherapies with non-pharmacotherapies in improving cognitive, behavioral, and functional outcomes in AD patients. Additionally, treatment ranking was performed to identify which treatments were likely to be most efficacious for each specific outcome.

Methods

This SR-NMA was conducted and reported complying with the Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) for NMA.²³ This study is a part of the full review protocol²⁴ (PROSPERO CRD42021228245), which focused on cognitive, behavioral, and functional outcomes.

Study selection

We identified studies from Medline via PubMed and Scopus databases from inception to December 2022. Search terms and search strategies for each database are presented in the eAppendix A.

Two reviewers (CB, TA) independently selected studies initially based on titles and abstracts. Full texts were reviewed where decisions could not be made. Discrepancies in study selection were resolved by a third party (AT). Study selection was performed as per the following steps: First, SRs of randomized-controlled trials (RCTs) were eligible if they studied AD patients aged 60 years or older, pooled relative treatment effects for any pair of pharmacological therapies, non-pharmacological therapies, or their combination, and measured cognitive, behavioral, or functional outcomes. Second, individual RCTs included in the eligible SRs or additionally identified were selected with the same inclusion criteria as used in the identification of the SRs. Studies were excluded if they had insufficient data for pooling or compared similar medications with different doses.

Additionally, studies were excluded if they involved interventions outside the predefined scope. Excluded pharmacological interventions included agents not commonly used for AD treatment or lacking clinical relevance, such as atypical antipsychotics, statins, selective serotonin reuptake inhibitor, carbamazepine, trazodone, tolterodine, citicoline, tacrine, hormone replacement therapy, velnacrine, cyproterone, latrepiridine, PPAR-γ agonists, phytostigmines, metrifonate, rosiglitazone, pioglitazone, sirtuins, aminopyridines, and anticonvulsants. These were excluded due to limited evidence or investigational status.

Non-pharmacotherapies excluded electromagnetic field therapies, nutritional counseling, dietary interventions, vitamin B supplementation, general supplements, and vegan diets—primarily due to limited clinical applicability, heterogeneity, or insufficient replication in high-quality trials. Studies with inadequate data for pooling or those comparing only different doses of the same medication were also excluded.

Interventions of interest

Interventions included pharmacotherapies, non-pharmacotherapies, and combined pharmaco- and non-pharmacotherapies (see details in Supplemental Table 1). Briefly, pharmacotherapies comprised of (1) AChEIs (i.e., donepezil, galantamine, and rivastigmine), and (2) NMDA antagonists, i.e., memantine, and (3) herbal remedies including Egb761, huperzine A, yokukansan, and cooked or Chinese herbs (e.g., saffron, panax ginseng and Korean ginseng). Non-pharmacotherapies included CT, CS, CR, occupational therapy, alternative therapy, pet therapy, exercise, music therapy, brain-stimulation, psychosocial intervention, multicomponent interventions, caregiver education, and multisensory behavior therapy (MSBT). Eligible comparators included usual/standard care or placebo.

Outcomes of interest

The outcomes of interest in this study encompassed cognitive, behavioral, and functional performance in AD patients. Cognitive outcomes were measured using the Mini-Mental State Examination (MMSE), with a minimal clinically important difference (MCID) identified as an increase of 3.50 to 3.95 points.²⁵ Behavioral outcomes were assessed using the Neuropsychiatric Inventory (NPI), with an MCID marked by an 8-point reduction.²⁶ Functional outcomes were evaluated using various tools, including the Alzheimer Disease Cooperative Study Activity of Daily Living (ADCS-ADL), Basic Activity of Daily Living (BADL), and the Barthel or modified Barthel index. Further details on these measurement tools are provided in Supplemental Table 1.

Data extraction

Characteristics of participants (i.e., mean age, sex, and dementia severity), interventions (i.e., types, dose, duration), and outcomes (i.e., type, period, number of patients, mean, and standard deviation (SD) for each outcome) were independently extracted by pairs of reviewers (CB, US, MT, WT, WW, TA, NP). Discrepancies in data extraction were resolved by consensus with adjudicators (AT, SR).

Risk of bias assessment

Risk of bias was independently assessed by the same reviewers above using a revised Cochrane risk-of-bias tool (RoB 2).²⁷ Five domains were assessed including randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported results. Risk of bias for each domain was judged as low risk of bias, high risk of bias, or some concerns. The results of risk of bias assessment are presented in the Supplemental Table 3.

Statistical analysis

A pairwise meta-analysis (MA) was performed if there were at least 3 studies with the same treatment comparisons and outcomes. Unstandardized mean differences (MDs) were used to estimate relative treatment effects when outcomes were measured using similar tools. Standardized mean differences (SMDs) were applied when outcomes were assessed using different tools across studies. SMD/MDs were pooled using the random effects model (i.e., DerSimonian Laird method). Heterogeneity was assessed using τ², heterogeneity (Q) test and I² statistic.

Regarding the NMA, relative treatment effects by MDs for cognitive and behavioral outcomes, and SMDs for function outcome, were pooled across studies using a network meta-analytic framework with a consistency model.^28,29 The relative treatment effects of all treatment comparisons are presented in a network league table. The SUCRA was applied to estimate the probability of each treatment being the most effective. The SUCRA values for all interventions are presented using rankograms and summary plots, providing a visual representation of the ranking probabilities across treatments. The consistency assumption of the NMA was assessed using a design-by-treatment interaction model with a global Chi-square test, as well as a local loop-specific approach.^30,31 In the loops with significant incosistency factors (IFs) (p < 0.1), we further explored the distribution of patient and study characteristics (e.g., mean age, percent female, study setting, and AD severity) to identify potential sources of inconsistency. A sensitivity analysis was performed by excluding studies with markedly different characteristics from those loops.²² Publication bias was assessed by a comparison-adjusted funnel plot and Egger test.³² All statistical analyses were performed using STATA version 18.

The certainty of evidence from the NMA was assessed using the Confidence in Network Meta-Analysis (CINeMA) tool,³³ which evaluated six domains: (1) within-study bias, (2) reporting bias, (3) indirectness, (4) imprecision, (5) heterogeneity, and (6) incoherence. For within-study bias, the overall risk of bias was scored for each study as 1 for low risk, 2 for some concern, and 3 for high risk of bias. Clinically meaningful thresholds were defined as MD of 3 points for MMSE, 8 points for NPI, and 0.5 for SMD of functional outcome. Indirectness or transitivity was assessed by exploring distributions of possible effect modifiers (e.g., age, sex, and severity of AD at baseline).

Each domain was rated as having no concerns, some concerns, or major concerns. The overall confidence in each treatment comparison was initially graded as high confidence and was downgraded by one level for each domain with some concerns, and by two levels for each domain with major concerns. The confidence of the domains of indirectness and incoherence, and the domains of heterogeneity, imprecision, reporting, and within-study bias were also related with each other and then were considered together rather than separately.

Patient and public involvement

Patients or the public were not involved in the design, conduct, reporting, or dissemination plans of our research.

Results

The study selection flow is presented in Figure 1(a) and (b). First, of the 2131 SRs identified, 228 SRs (101, 124, and 3 for pharmacotherapies, non-pharmacotherapies, and combined pharmacotherapies and non-pharmacotherapies) with a total of 1239 individual RCTs were eligible. Second, of the 1239 RCTs screened, only 153 RCTse1-e153 met our inclusion criteria and were finally included in analyses.

Figure 1.

Flow diagram of study selection process.

Study characteristics are described in Supplemental Table 2. Most studies (73%) were conducted in a single-center, study duration varied from 1 week to 2 years. Mean age and percent female ranged from 64 to 86.52 years, and 9% to 100%, respectively. AD severity was reported in 138 studies, most of them were mild (N = 32, 23%) and mild to moderate (N = 81, 59%).

Cognitive outcome

Pairwise meta-analysis

MMSE data for treatment comparisons from 110 RCTs (n = 12,326) are described in eAppendix B-Table 1. Only donepezil, brain stimulation, CS, CT, and exercise significantly improved MMSE scores relative to placebo/usual care with MDs (95% CI) of 1.13 (0.86, 1.41), 2.41 (0.80, 4.02), 1.87 (0.39, 3.35), 1.67 (0.59, 2.75), and 2.02 (0.62, 3.41), respectively (see eAppendix B-Figure 1A-I). However, heterogeneity varied across these pooled outcomes with I² values between 0% and 88.44% and τ² values between 0 and 4.932.

Network meta-analysis

A total of 110 RCTs were included in an NMA of MMSE scores consisting of 6 combined pharmaco- and non-pharmacotherapies, 4 dual non-pharmacotherapies, 13 mono-non-pharmacotherapies, 5 dual-pharmacotherapies, and 8 mono-pharmacotherapies (see Figure 2(a)) Relative treatment effects for all treatment comparisons were estimated and presented in eAppendix B-Table 2. The results suggested that combined pharmaco- and non-pharmacotherapies (i.e., donepezil plus CT, rivastigmine plus CR), mono- non-pharmacotherapies (i.e., exercise, psychological interventions, CT, CS, and brain-stimulation), combined pharmacotherapies (i.e., huperzine + memantine), and mono-pharmacotherapies (i.e., huperzine, and donepezil) significantly improved MMSE scores relative to placebo/usual care. However, inconsistency was observed from the global test (Chi-square = 54.12 p = 0.001). The IFs revealed significant differences between direct and indirect comparisons including CS versus placebo/usual care, CT versus placebo/usual care, donepezil plus CS versus donepezil plus alternative therapy, and donepezil plus CS versus placebo/usual care, and CT versus CS (see eAppendix B-Table 3). Further investigation identified six studies with distinct patient characteristics (i.e., higher mean age^{e66, e67}, lower percent female^{e68, e142}, and different study setting (i.e., ambulatory^{e64, e65}).

Figure 2.

Network map of treatment comparisons for cognitive, behavioral and functional outcomes. The size of each node represents the number of participants randomly assigned to each treatment. The width of each line reflects the number of studies comparing the connected treatments, with the exact number of studies indicated above each line. Alter: alternative therapies; Brainstim: brain stimulation; CR: cognitive rehabilitation; CS: cognitive stimulation; CT: cognitive therapy; CE: focused caregiver education; Donep: donepezil; Galan: galantamine; Huper: huperzine; Meman: memantine; MS: multisensory behavior therapy; MT: music therapy; Multi: multicomponent intervention; OT: occupational therapy; PE: increased physical activity and exercise; PE + CT: increased physical activity plus cognitive therapy; Pet: pet therapy; Psycho: psychological intervention; RivasSD: rivastigmine standard dose; RivasHD: rivastigmine high dose; UC: placebo/usual care; Yoku: yokukansan.

A sensitivity analysis excluding these six studies was conducted. The NMA met the consistency assumption (Chi-square = 7.04, p = 0.900) and the relative treatment effects showed similar trend as in the overall pooling, see eAppendix B-Table 4. This finding suggested that combined pharmaco- and non-pharmacotherapies (i.e., donepezil plus CT, rivastigmine plus CR), mono-non-pharmacotherapies (i.e., exercise, CT, CS, and brain-stimulation), and mono-pharmacotherapies (i.e., huperzine, and donepezil) significantly improved MMSE scores relative to placebo/usual care (Figure 3(a)). SUCRA values indicated that the top three treatments associated with improved MMSE scores were rivastigmine plus CR (92.8%), donepezil plus CT (90.3%), and huperzine (88.8%) (Figure 4 and eAppendix B-Figure 2).

Figure 3.

Relative treatment effects of active treatments compared to placebo/usual care from network meta-analysis. 95% Prl indicates the 95% confidence interval estimated from the predictive interval plot. Brain-stim: brain stimulation; CT: cognitive training; CR: cognitive rehabilitation; CS: cognitive stimulation; Egb 761: Extract of Ginkgo biloba 761; CE: focused caregiver education; Multicomponent: multicomponent intervention; MS: multisensory behavior therapy; PE: increased physical activity and exercise; Rivastigmine-HD: rivastigmine high dose; Rivastigmine-SD: rivastigmine standard dose.

Figure 4.

Rank-heat plot comparing the SUCRA values of cognitive, behavioral, and functional outcomes among all treatments. The number in each box represents the SUCRA value of the treatment. Brain-stim: brain stimulation; CT: cognitive training; CR: cognitive rehabilitation; CS: cognitive stimulation; Egb 761: Extract of Ginkgo biloba 761; Multicomponent: multicomponent intervention; MSBT: multisensory behavior therapy; Rivastigmine-HD: rivastigmine high dose; Rivastigmine-SD: rivastigmine standard dose.

Results from the subgroup analysis, which included only studies involving patients with mild to moderate AD (87 studies), were consistent with those of the full analysis. Specifically, donepezil plus CT, rivastigmine plus CR, exercise, psychological interventions, CT, CR, brain stimulation, huperzine, and donepezil were all associated with significant improvements in MMSE scores compared to placebo or usual care (see eAppendix B–Table 5). The SUCRA rankings are similar to those of the full analysis, with rivastigmine plus CR, donepezil plus CT, and huperzine emerging as the top three treatments for improving MMSE scores (see eAppendix B-Figure 3). For moderate to severe AD, only 11 studies with 7 interventions (i.e., donepezil, memantine, donepezil + memantine, MSBT, music therapy, exercise, and pet therapy) reported cognitive outcome in this group of patients. The results found that only exercise and donepezil significantly improved MMSE score, when compared to placebo/usual care (see eAppendix B-Table 6).

Egger's test showed no significant small study effect (p = 0.733). However, the comparison-adjusted funnel plot exhibited slight asymmetry, suggesting potential publication bias (eAppendix B-Figure 4).

The certainty of evidence from the NMA is presented in eAppendix B–Table 7. In the sensitivity analysis, the characteristics of studies included in the NMA were similar across most studies supporting the assumption of transitivity (see eAppendix B-Table 8); accordingly, indirectness was rated as having no concerns. Overall, 64% of treatment comparisons were rated as having moderate confidence, 35% as low confidence, and only 1% as having very low confidence, primarily involving comparisons of huperzine. In contrast, the majority of comparisons involving rivastigmine plus CR and donepezil plus CT were rated as having moderate confidence.

Behavioral outcome

Pairwise meta-analysis

Sixty-two RCTs measured NPI scores (N = 10,336). MA was performed for 8 treatment comparisons indicating that only mono-non-pharmacotherapies of CT, multicomponent interventions, and exercise significantly lowered NPI scores relative to usual care with MDs (95% CI) of −3.31 (−6.57, −0.04), −4.18 (−4.81, −3.56), and −4.55 (−8.25, −0.86), respectively (eAppendix C-Figure 1A-H). However, heterogeneity was found for CT versus usual care (τ² = 7.003; I² = 67.29%) and PE versus usual care (τ² = 11.524; I² = 76.68%).

Network meta-analysis

Data from 62 RCTs (n = 10,336) consisting of 2 combined pharmaco- and non-pharmacotherapies, 13 non-pharmacotherapies, and 11 pharmacotherapies were included in a NMA of behavioral outcomes, see Figure 2(b) and eAppendix C-Table 1. The global test indicated consistent relative-treatment effects (Chi-square = 7.22, p = 0.7044). All relative treatment comparisons are presented in eAppendix C-Figure 2.

When compared to placebo/usual care, rivastigmine plus CS, brain stimulation plus exercise, exercise, occupational therapy, multicomponent interventions, donepezil plus memantine, and Egb761 significantly improved NPI score with MDs (95% CI) of −11.72 (−19.23, −4.21), −12.02 (−20.89, −3.14), −4.13 (−7.29, −0.98), −10.58 (−17.65, −3.51), −3.85 (−7.49, −0.22), and −5.33 (−10.08, −0.59), respectively (see Figure 3(b)).

The comparative efficacy of combined pharmaco- and non-pharmacotherapies, mono-non-pharmacotherapies, and mono-pharmacotherapies in improving behavioral symptoms are summarized in eAppendix C-Table 3. Results from SUCRA suggested that brain stimulation plus exercise (93.1%), rivastigmine plus CS (93.0%), and occupational therapy (90.5%) were the top three ranked treatments for improving NPI score (see Figure 4 and eAppendix C-Figure 2). A comparison adjusted funnel plot and results from Egger test (p = 0.410) indicated no small study effect (see eAppendix C-Figure 3).

The certainty of evidence from the NMA is presented in eAppendix C–Table 4. The characteristics of studies included in the NMA of behavioral outcomes were similar across most studies; therefore, transitivity or indirectness was rated as having no concerns (see eAppendix C-Table 5). Most treatment comparisons (82%) were rated as having moderate confidence, followed by 17% with low confidence. Only 1 treatment comparison (CR versus CT) was rated as having very low confidence. In contrast, treatment comparisons involving occupational therapy were rated as having high (2 in 26 comparisons) and moderate (22 in 26 comparisons) confidence. Additionally, treatment comparisons involving brain stimulation plus exercise (24 in 26 comparisons), and rivastigmine plus CS (24 in 26 comparisons) were graded as moderate confidence.

Functional performance of ADL

Pairwise meta-analysis

Forty-three RCTs reported functional outcomes, measured by ADL in 10 studies, ADL-IDAL in 5 studies, ADCS-ADL in 15 studies, BADLs in 9 studies, and Barthel in 1 study (eAppendix D-Table 1). All scores have the same direction, i.e., that a higher score represents better functional outcome. SMD was applied for pooling the data. Of 4 treatment comparisons (eAppendix D-Figure 1A-D), MAs suggested that only exercise significantly improved functional ADLs, compared to usual care with SMD of 0.82 (0.34, 1.30). However, results of τ² (2.165) and I² (89.25%) suggest the presence of heterogeneity between studies.

Network meta-analysis

A total of 43 studies with 4 combined pharmaco- and non-pharmacotherapies, 13 non-pharmacotherapies, and 11 pharmacotherapies were included in the NMA of functional performance (see Figure 2(c)). Treatment comparison data are presented in eAppendix D-Table 2. The NMA results indicated that only exercise significantly improved functional outcome compared to usual care/placebo with SMD of 0.83 (0.27,1.39), see eAppendix D-Table 2. However, the global test indicated significant inconsistency across the network (Chi-square = 22.59, p = 0.007). The IFs were estimated indicating significant difference between direct and indirect estimates for comparisons of psychological interventions versus placebo/usual care and CR versus psychological interventions (see eAppendix D-Table 3). Upon further investigation, three studies were found to differ from others, primarily due to lower percentage of female^e100,e147, and higher mean age of participants^e102. A sensitivity analysis excluding these three studies was performed, resulting in a consistent NMA (Chi-square = 0.95, p = 0.996).

Updated relative treatment effects are presented in eAppendix D-Table 4. The results showed that rivastigmine plus CS and exercise significantly improved functional outcome compared to usual care/placebo, with SMD of 0.92 (0.25, 1.58) and 0.41 (0.06,0.76), respectively (see Figure 3(c)). Furthermore, exercise significantly improved functional scores when compared to other mono-non-pharmacotherapies including psychological interventions (SMD = 0.66; 95% CI: 0.13, 1.19), alternative therapy (SMD = 0.54; 95% CI: 0.06, 1.01), and pharmacotherapies including memantine + huperzine, memantine + galantamine, huperzine, galantamine, and donepezil. Rivastigmine plus CS also significantly improved functional score, when compared to non-pharmacotherapies (i.e., psychological interventions, music therapy, CT, alternative therapy), and pharmacotherapies (i.e., memantine + huperzine, memantine + galantamine, donepezil + memantine, huperzine, memantine, donepezil, galantamine, and rivastigmine), see eAppendix D-Table 4. SUCRA values suggested that rivastigmine plus CS (94.1%), donepezil plus CT (80.9%), brain stimulation (79.3%), and exercise (78.8%) were the top ranked treatments associated with higher functional status (see Figure 4 and eAppendix D-Figure 2). p-value from Egger test was 0.451 but a comparison adjusted funnel plot was asymmetric, indicating the presence of small study effect (see eAppendix D-Figure 3).

The certainty of evidence from the NMA is presented in eAppendix D–Table 5. The characteristics of studies included in the sensitivity analysis were similar across most studies; therefore, transitivity or indirectness was rated as having no concerns (see eAppendix D-Table 6). Most treatment comparisons were rated as having moderate (41%) or low (49%) confidence, while 10% were rated as having very low confidence. Of the comparisons involving rivastigmine plus CS, 21 out of 26 were graded as having moderate certainty. In contrast, 7 out of 26 and 2 out of 26 comparisons involving exercise were rated as having very low certainty. Additionally, 15 out of 26 comparisons involving donepezil plus CT were graded as having low certainty.

Discussion

Principal findings

Our SR-NMA suggests that combined pharmaco- and non-pharmacotherapies provide the most efficacious treatments for AD patients. With respect to cognitive outcome, rivastigmine plus CR and donepezil plus CT were the most efficacious treatments, while rivastigmine plus CS was identified as the second-best intervention for reducing behavioral problems and the first ranked intervention for improving functional status. For non-pharmacotherapies, brain-stimulation plus exercise, occupational therapy, and exercise were also ranked as the top-three best treatments for both behavioral and functional outcomes, while huperzine was the only pharmacotherapy that was ranked in the top-three best treatments for cognitive outcomes.

Previous SR-NMAs that focused exclusively on pharmacotherapies^10,12,34 suggested a combination of donepezil and memantine offered the optimal therapy for improved cognitive function. However, our findings suggested that combinations of pharmaco- and non-pharmacotherapies, donepezil plus CT and rivastigmine plus CR offer the most effective treatments for improved MMSE scores compared to placebo/ usual care and meet the criteria for MCID. Furthermore, our NMA ranked both interventions as the most beneficial for improving cognitive outcomes. Although our findings indicate improved cognitive outcome for donepezil relative to placebo/usual care, its mean difference in MMSE failed to reach the MCID threshold. As such, the combination therapy of AChEIs with non-pharmacotherapies may potentially provide improved cognitive outcomes in AD patients compared to the use of AChEIs or non-pharmacotherapies individually. Furthermore, the efficacy of combination therapy approaches that include pharmacotherapies and non-pharmacotherapies have been proposed in a previous systematic review and NMA²¹ that compared the efficacy of pharmacotherapies and non-pharmacotherapies in the mitigation of depressive symptoms in individuals with dementia. Their findings concluded that the combination of AChEIs with cognitive stimulation represented the most effective intervention for alleviating depressive symptoms in individuals with dementia. Nevertheless, caution should be exercised in the interpretation of our findings as the evidence supporting the benefits of AChEIs with non-pharmacotherapy combination therapies is based on the limited number of studies included.

In terms of behavioral outcomes, our study highlighted the importance of non-pharmacotherapies, specifically the combination of brain stimulation with exercise, and occupational therapy. These interventions could be highly effective in reducing behavioral problems in individuals with AD, surpassing the MCID threshold for the NPI. These findings not only reinforce the value of non-pharmacotherapies for individuals with AD but also underscore their clinically significant benefits. Furthermore, it is worth noting that rivastigmine in combination with cognitive stimulation demonstrated a significant advantage over placebo/usual care in relation to NPI scores, resulting in an impressive 11-point reduction of NPI score. Our findings were inconsistent with previous SR-NMA findings that focused solely on non-pharmacotherapies.³⁵ This SR indicated that photobiomodulation, exercise, and cognitive stimulation ranked highest as non-pharmacotherapies for improving cognitive outcomes in AD patients. To clarify, photobiomodulation³⁶ is a form of light therapy utilizing infrared light to stimulate biological or cellular processes in the brain, whereas non-invasive brain stimulation techniques such as transcranial magnetic stimulation³⁷ and direct current stimulation³⁸ target the dorsolateral prefrontal cortex and may impact neurotransmitter function and neuroplasticity. Notably, our NMA also demonstrated that non-invasive brain stimulation (rTMS and tDCS) proved more effective than usual care in enhancing cognitive function and outranked exercise and cognitive stimulation. The previous systematic reviews predominantly considered pharmacotherapies and non-pharmacotherapies as separate entities. In contrast, our study offers a more comprehensive evaluation by simultaneously assessing the efficacy of all pharmacotherapies, non-pharmacotherapies, and combination therapies together. This approach provides a more robust interpretation of the treatment landscape for AD patients.

Another noteworthy finding from our study highlights the beneficial effects from increased physical activity and exercise in enhancing cognitive, behavioral, and functional status among individuals with AD. Our NMA suggests that exercise intervention incorporating aerobic, strength, and balance exercises are more efficacious compared to usual care across all evaluated outcomes.

For herbal remedies, our findings suggest that huperzine and Egb761 were more efficacious than AChEIs (specifically donepezil, galantamine, and rivastigmine) and memantine in terms of improved cognitive function and reduced behavioral problems, respectively. However, it is crucial to consider these findings with caution due to the limited number of studies available for huperzine (only 1 study) and Egb761 (just 3 studies). Additionally, of note the studies involving huperzine and Egb761 exhibited a high and concerning risk of bias. Consequently, the level of confidence in this evidence should be considered relatively low.

Clinical and policy implications

Our findings align partially with the National Institute for Health and Care Excellence dementia care guidelines,³⁹ which recommend cognitive stimulation therapy to improve cognitive function, independence, and wellbeing. However, our NMA emphasizes the clinical advantages of combining pharmacotherapies and non-pharmacotherapies for AD patients. The NMA also highlights the efficacy of various non-pharmacotherapies, such as cognitive therapy, cognitive stimulation, occupational therapy, and physical exercise, underscoring the need for greater accessibility in clinical and community settings. Collaboration among clinicians, policy makers, and health services researchers is crucial to ensure equitable access to these interventions. For example, cognitive stimulation is integrated into new care models in the UK, such as small group homes for dementia which promote mental stimulation, reminiscence, social interaction. Physical exercise was identified as the intervention maintaining functional outcomes, highlighting its importance in policy implementation. Moderate to high intensity exercise programs including aerobic, balance and strengthening, should be widely promoted across various settings including community, clinics, long term care facilities and nursing home settings.^40–44

Strength and limitations

Our study has several significant strengths. It is the first SR-NMA to comprehensively compare the efficacy of pharmacotherapies, non-pharmacotherapies, and their combination in improving cognitive, behavioral, and functional outcomes in AD patients. The study provides an extensive evaluation of various interventions and outcomes, offering a holistic view of AD treatment. Commonly used tools such as the MMSE scores for cognitive function and the NPI for behavioral problems, were employed to ensure the relevance and applicability of the findings. In addition to achieving statistical significance, several interventions demonstrated improvements likely to be clinically meaningful. For cognitive outcomes, donepezil combined with cognitive therapy and rivastigmine combined with cognitive rehabilitation achieved mean improvements in MMSE scores that approach or exceed the reported MCID of 3.5 to 3.95 points. Similarly, behavioral outcomes improved with rivastigmine plus cognitive stimulation and brain stimulation combined with exercise, reaching reductions in NPI scores consistent with the clinically important threshold of an 8-point reduction.

Moreover, rivastigmine plus cognitive stimulation and physical exercise significantly improved or maintained functional status. Preservation of activities of daily living (ADLs) is highly meaningful in the management of Alzheimer's disease, as it can delay nursing home admission, lessen caregiver burden, and enhance patients’ quality of life. Therefore, beyond statistical significance, the magnitude and direction of effects observed in this study suggest that several interventions, particularly combined pharmacological and non-pharmacological therapies, may offer real-world benefits for patients and healthcare systems.

However, our study has several limitations. The evidence supporting the benefits of combined pharmacotherapies-non-pharmacotherapies is limited due to quantity and quality of available studies. Further RCTs directly comparing combined pharmacotherapies-non-pharmacotherapies with pharmacotherapies or non-pharmacotherapies alone, are needed to validate these findings. Many included studies were classified as having some concern or high risk of bias, potentially affecting the reported outcomes, especially for interventions, such as herbal remedies which may have overestimated effects. Additionally, inconsistency was detected in the NMAs of both cognitive and functional outcomes, indicating disagreement between direct and indirect evidence. This inconsistency may impact the validity and interpretability of the overall findings, potentially reflecting differences in the characteristics of the included studies, such as variations in baseline populations or study design. However, sensitivity analyses that excluded studies with different characteristics yielded consistent networks and provided robust findings that were comparable to the full analyses. Moreover, asymmetrical comparison- adjusted funnel plots for cognitive and functional outcomes suggest the presence of heterogeneity or publication bias or both; for the latter, smaller studies with positive results are more likely to be published, potentially skewing the overall findings.

Further study

Our SR-NMA did not encompass drugs targeting amyloid-β, like lecanemab, donanemab, and gantenerumab. This omission stems from the ongoing debate regarding whether these drugs yield clinically significant effects.⁴⁵ As a result, it is imperative to gather more evidence regarding the effectiveness of this class of drugs to conduct a comprehensive comparison of their efficacy in enhancing cognitive outcomes for AD patients in the future.

Conclusion

Our SR-NMA indicated that combining pharmacotherapy and non-pharmacotherapy is likely the most effective approach for treating AD. Donepezil combined with cognitive therapy, and rivastigmine with cognitive rehabilitation were highly ranked for improving cognitive outcomes. While rivastigmine plus cognitive stimulation ranked highest for behavioral and functional outcomes, physical exercise remains an important strategy to support daily functioning in patients with AD. This comprehensive analysis provides valuable evidence to guide therapeutic decisions and inform policymaking for AD treatment.

Supplemental Material

sj-docx-1-alz-10.1177_13872877251378354 - Supplemental material for Bridging the gap: Efficacy of combined therapies for cognitive, behavioral, and functional outcomes in Alzheimer's disease – results from a systematic review and network meta-analysis

Supplemental material, sj-docx-1-alz-10.1177_13872877251378354 for Bridging the gap: Efficacy of combined therapies for cognitive, behavioral, and functional outcomes in Alzheimer's disease – results from a systematic review and network meta-analysis by Chitima Boongird, Thunyarat Anothaisintawee, Worapong Tearneukit, Wannisa Wongpipathpong, Unyaporn Suthutvoravut, Masatha Thongpan, Natthakorn Pongsettakul, John Attia, Gareth J McKay, Sasivimol Rattanasiri and Ammarin Thakkinstian in Journal of Alzheimer's Disease

Supplemental Material

sj-docx-2-alz-10.1177_13872877251378354 - Supplemental material for Bridging the gap: Efficacy of combined therapies for cognitive, behavioral, and functional outcomes in Alzheimer's disease – results from a systematic review and network meta-analysis

Supplemental material, sj-docx-2-alz-10.1177_13872877251378354 for Bridging the gap: Efficacy of combined therapies for cognitive, behavioral, and functional outcomes in Alzheimer's disease – results from a systematic review and network meta-analysis by Chitima Boongird, Thunyarat Anothaisintawee, Worapong Tearneukit, Wannisa Wongpipathpong, Unyaporn Suthutvoravut, Masatha Thongpan, Natthakorn Pongsettakul, John Attia, Gareth J McKay, Sasivimol Rattanasiri and Ammarin Thakkinstian in Journal of Alzheimer's Disease

Supplemental Material

sj-docx-3-alz-10.1177_13872877251378354 - Supplemental material for Bridging the gap: Efficacy of combined therapies for cognitive, behavioral, and functional outcomes in Alzheimer's disease – results from a systematic review and network meta-analysis

Supplemental material, sj-docx-3-alz-10.1177_13872877251378354 for Bridging the gap: Efficacy of combined therapies for cognitive, behavioral, and functional outcomes in Alzheimer's disease – results from a systematic review and network meta-analysis by Chitima Boongird, Thunyarat Anothaisintawee, Worapong Tearneukit, Wannisa Wongpipathpong, Unyaporn Suthutvoravut, Masatha Thongpan, Natthakorn Pongsettakul, John Attia, Gareth J McKay, Sasivimol Rattanasiri and Ammarin Thakkinstian in Journal of Alzheimer's Disease

Supplemental Material

sj-docx-4-alz-10.1177_13872877251378354 - Supplemental material for Bridging the gap: Efficacy of combined therapies for cognitive, behavioral, and functional outcomes in Alzheimer's disease – results from a systematic review and network meta-analysis

Supplemental material, sj-docx-4-alz-10.1177_13872877251378354 for Bridging the gap: Efficacy of combined therapies for cognitive, behavioral, and functional outcomes in Alzheimer's disease – results from a systematic review and network meta-analysis by Chitima Boongird, Thunyarat Anothaisintawee, Worapong Tearneukit, Wannisa Wongpipathpong, Unyaporn Suthutvoravut, Masatha Thongpan, Natthakorn Pongsettakul, John Attia, Gareth J McKay, Sasivimol Rattanasiri and Ammarin Thakkinstian in Journal of Alzheimer's Disease

Supplemental Material

sj-docx-5-alz-10.1177_13872877251378354 - Supplemental material for Bridging the gap: Efficacy of combined therapies for cognitive, behavioral, and functional outcomes in Alzheimer's disease – results from a systematic review and network meta-analysis

Supplemental material, sj-docx-5-alz-10.1177_13872877251378354 for Bridging the gap: Efficacy of combined therapies for cognitive, behavioral, and functional outcomes in Alzheimer's disease – results from a systematic review and network meta-analysis by Chitima Boongird, Thunyarat Anothaisintawee, Worapong Tearneukit, Wannisa Wongpipathpong, Unyaporn Suthutvoravut, Masatha Thongpan, Natthakorn Pongsettakul, John Attia, Gareth J McKay, Sasivimol Rattanasiri and Ammarin Thakkinstian in Journal of Alzheimer's Disease

Footnotes

Acknowledgements

We would like to thank Miss Paichit Inpanya, Mrs. Wannaporn Bunrung for coordinating the project management.

ORCID iDs

Thunyarat Anothaisintawee

John Attia

Ammarin Thakkinstian

Ethical considerations

Ethical approval was not required for this study as it was based solely on published literature that met accepted ethical standards, and there were no concerns regarding privacy or confidentiality. The evidence obtained will be disseminated through peer-reviewed publication to inform clinical practice with rigorous scientific evidence.

Consent to participate

Not applicable.

Consent for publication

Not applicable

Author contribution(s)

Chitima Boongird: Conceptualization; Data curation; Funding acquisition; Investigation; Methodology; Project administration; Validation; Writing – original draft; Writing – review & editing.

Thunyarat Anothaisintawee: Conceptualization; Data curation; Formal analysis; Investigation; Methodology; Writing – original draft; Writing – review & editing.

Worapong Tearneukit: Data curation; Investigation; Validation; Writing – review & editing.

Wannisa Wongpipathpong: Methodology; Data curation; Investigation; Validation; Writing – review & editing.

Unyaporn Suthutvoravut: Data curation; Investigation; Methodology; Validation; Writing – review & editing.

Masatha Thongpan: Data curation; Investigation; Validation; Writing – review & editing.

Natthakorn Pongsettakul: Data curation; Investigation; Validation; Writing – review & editing.

John Attia: Formal analysis; Methodology; Supervision; Validation; Writing – review & editing.

Gareth J McKay: Formal analysis; Methodology; Supervision; Validation; Writing – review & editing.

Sasivimol Rattanasiri: Formal analysis; Investigation; Methodology; Software; Validation; Writing – review & editing.

Ammarin Thakkinstian: Conceptualization; Formal analysis; Investigation; Methodology; Supervision; Validation; Writing – review & editing.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research received a grant from Health Systems Research Institute, Thailand, grant number (65-008).

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability statement

The data supporting the findings of this study are available within the article and/or its supplemental material.

Supplemental material

Supplemental material for this article is available online.

References

World Health Organization . Global action plan on the public health response to dementia 2017–2025. Geneva, Switzerland: World Health Organization, 2017.

Nichols

Steinmetz

Vollset

, et al. Estimation of the global prevalence of dementia in 2019 and forecasted prevalence in 2050: an analysis for the global burden of disease study 2019. Lancet Public Health 2022; 7: e105–e125.

World Health Organization . A blueprint for dementia research. Geneva, Switzerland: World Health Organization, 2022.

Alzheimer's Disease Facts & Figures . WHO report, https://www.who.int/news-room/fact-sheets/detail/dementia (2018, accessed 22 February 2021).

Fazio

Pace

Maslow

, et al. Alzheimer’s association dementia care practice recommendations. Gerontologist 2018; 58: S1–S9.

Shaji

Sivakumar

Rao

, et al. Clinical practice guidelines for management of dementia. Indian J Psychiatry 2018; 60: S312–s328.

Di Santo

Prinelli

Adorni

, et al. A meta-analysis of the efficacy of donepezil, rivastigmine, galantamine, and memantine in relation to severity of Alzheimer's disease. J Alzheimers Dis 2013; 35: 349–361.

Kobayashi

Ohnishi

Nakagawa

, et al. The comparative efficacy and safety of cholinesterase inhibitors in patients with mild-to-moderate Alzheimer's disease: a Bayesian network meta-analysis. Int J Geriatr Psychiatry 2016; 31: 892–904.

Dou

Tan

, et al. Comparative safety and effectiveness of cholinesterase inhibitors and memantine for Alzheimer's disease: a network meta-analysis of 41 randomized controlled trials. Alzheimers Res Ther 2018; 10: 126.

10.

Tricco

Ashoor

Soobiah

, et al. Comparative effectiveness and safety of cognitive enhancers for treating Alzheimer's disease: systematic review and network metaanalysis. J Am Geriatr Soc 2018; 66: 170–178.

11.

Tsoi

Chan

, et al. Monotherapy is good enough for patients with mild-to-moderate Alzheimer's disease: a network meta-analysis of 76 randomized controlled trials. Clin Pharmacol Ther 2019; 105: 121–130.

12.

Guo

Wang

Liu

, et al. Memantine, donepezil, or combination therapy-what is the best therapy for Alzheimer's disease? A network meta-analysis. Brain Behav 2020; 10: e01831.

13.

Campbell

Perkins

Gao

, et al. Adherence and tolerability of Alzheimer's disease medications: a pragmatic randomized trial. J Am Geriatr Soc 2017; 65: 1497–1504.

14.

Liang

Jia

, et al. Identification of the optimal cognitive drugs among Alzheimer's disease: a Bayesian meta-analytic review. Clin Interv Aging 2018; 13: 2061–2073.

15.

Thancharoen

Limwattananon

Waleekhachonloet

, et al. Ginkgo biloba extract (EGb761), cholinesterase inhibitors, and memantine for the treatment of mild-to-moderate Alzheimer's disease: a network meta-analysis. Drugs Aging 2019; 36: 435–452.

16.

May

Feng

Hyde

, et al. Comparisons between traditional medicines and pharmacotherapies for Alzheimer disease: a systematic review and meta-analysis of cognitive outcomes. Int J Geriatr Psychiatry 2018; 33: 449–458.

17.

Groot

Hooghiemstra

Raijmakers

, et al. The effect of physical activity on cognitive function in patients with dementia: a meta-analysis of randomized control trials. Ageing Res Rev 2016; 25: 13–23.

18.

Kim

Han

, et al. Cognitive stimulation as a therapeutic modality for dementia: a meta-analysis. Psychiatry Investig 2017; 14: 626–639.

19.

Liang

Lin

Wang

, et al. Non-pharmacological therapeutic strategy options for patients with dementia based on cognitive function-A Bayesian network meta-analysis of randomized controlled trials. Ageing Res Rev 2019; 56: 100965.

20.

Chou

Ton That

Sundman

. A systematic review and meta-analysis of rTMS effects on cognitive enhancement in mild cognitive impairment and Alzheimer's disease. Neurobiol Aging 2020; 86: 1–10.

21.

Watt

Goodarzi

Veroniki

, et al. Comparative efficacy of interventions for reducing symptoms of depression in people with dementia: systematic review and network meta-analysis. Br Med J 2021; 372: n532.

22.

Salanti

. Indirect and mixed-treatment comparison, network, or multiple-treatments meta-analysis: many names, many benefits, many concerns for the next generation evidence synthesis tool. Res Synth Methods 2012; 3: 80–97.

23.

Hutton

Salanti

Caldwell

, et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med 2015; 162: 777–784.

24.

Boongird

Tearneukit

Wongpipathpong

, et al. Efficacy of pharmacotherapy and non-pharmacotherapy of Alzheimer dementia: a protocol for systematic review and meta-analysis. Medicine (Baltimore) 2022; 101: e32382.

25.

Burback

Molnar

St John

, et al. Key methodological features of randomized controlled trials of Alzheimer's disease therapy. Minimal clinically important difference, sample size and trial duration. Dement Geriatr Cogn Disord 1999; 10: 534–540.

26.

Howard

Phillips

Johnson

, et al. Determining the minimum clinically important differences for outcomes in the DOMINO trial. Int J Geriatr Psychiatry 2011; 26: 812–817.

27.

Whiting

Savović

Higgins

, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 2016; 69: 225–234.

28.

White

. Multivariate random-effects meta-analysis. Stata J 2009; 9: 40–56.

29.

White

. Network meta-analysis. Stata J 2015; 15: 951–985.

30.

Higgins

Jackson

Barrett

, et al. Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies. Res Synth Methods 2012; 3: 98–110.

31.

Jackson

Boddington

White

. The design-by-treatment interaction model: a unifying framework for modelling loop inconsistency in network meta-analysis. Res Synth Methods 2016; 7: 329–332.

32.

Chaimani

Higgins

Mavridis

, et al. Graphical tools for network meta-analysis in STATA. PLoS One 2013; 8: e76654.

33.

Nikolakopoulou

Higgins

JPT

Papakonstantinou

, et al. CINeMA: an approach for assessing confidence in the results of a network meta-analysis. PLoS Med 2020; 17: e1003082.

34.

Veroniki

Ashoor

Rios

, et al. Comparative safety and efficacy of cognitive enhancers for Alzheimer's dementia: a systematic review with individual patient data network meta-analysis. BMJ Open 2022; 12: e053012.

35.

Luo

Zhang

Song

, et al. Effectiveness of non-pharmacological therapies on cognitive function in patients with dementia-A network meta-analysis of randomized controlled trials. Front Aging Neurosci 2023; 15: 1131744.

36.

Salehpour

Khademi

Hamblin

. Photobiomodulation therapy for dementia: a systematic review of pre-clinical and clinical studies. J Alzheimers Dis 2021; 83: 1431–1452.

37.

Koch

Bonnì

Pellicciari

, et al. Transcranial magnetic stimulation of the precuneus enhances memory and neural activity in prodromal Alzheimer's disease. Neuroimage 2018; 169: 302–311.

38.

Pellicciari

Miniussi

. Transcranial direct current stimulation in neurodegenerative disorders. J ECT 2018; 34: 193–202.

39.

NICE . Dementia: assessment, management, and support for people living with dementia and their carers, http://www.nice.org.uk/guidance/ng97 (2018).

40.

Lamb

Sheehan

Atherton

, et al. Dementia and physical activity (DAPA) trial of moderate to high intensity exercise training for people with dementia: randomised controlled trial. Br Med J 2018; 361: k1675.

41.

Vreugdenhil

Cannell

Davies

, et al. A community-based exercise programme to improve functional ability in people with Alzheimer's disease: a randomized controlled trial. Scand J Caring Sci 2012; 26: 12–19.

42.

Rolland

Pillard

Klapouszczak

, et al. Exercise program for nursing home residents with Alzheimer's disease: a 1-year randomized, controlled trial. J Am Geriatr Soc 2007; 55: 158–165.

43.

Venturelli

Scarsini

Schena

. Six-month walking program changes cognitive and ADL performance in patients with Alzheimer. Am J Alzheimers Dis Other Demen 2011; 26: 381–388.

44.

Santana-Sosa

Barriopedro

López-Mojares

, et al. Exercise training is beneficial for Alzheimer's patients. Int J Sports Med 2008; 29: 845–850.

45.

Prillaman

. Alzheimer's drug slows mental decline in trial – but is it a breakthrough? Nature 2022; 610: 15–16.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.10 MB

0.01 MB

0.71 MB

0.39 MB

0.37 MB