Sage Journals: Discover world-class research

Abstract

Objective

During the Coronavirus Disease 2019 (COVID-19) pandemic, digital health technologies (DHTs) became increasingly important, especially for older adults. The objective of this systematic review was to synthesize evidence on the rapid implementation and use of DHTs among older adults during the COVID-19 pandemic.

Methods

A structured, electronic search was conducted on 9 November 2021, and updated on 5 January 2023, among five databases to select DHT interventional studies conducted among older adults during the pandemic. The bias of studies was assessed using Version 2 of the Cochrane Risk-of-Bias Tool for randomized trials (RoB 2) and Risk of Bias in Non-randomized Studies of Interventions (ROBINS-I).

Results

Among 20 articles included in the review, 14 (70%) focused on older adults with chronic diseases or symptoms, such as dementia or cognitive impairment, type 2 diabetes, and obesity. DHTs included traditional telehealth interventions via telephone, video, and social media, as well as emerging technologies such as Humanoid Robot and Laser acupuncture teletherapy. Using RoB 2 and ROBINS-I, four studies (20%) were evaluated as high or serious overall risk of bias. DHTs have shown to be effective, feasible, acceptable, and satisfactory for older adults during the COVID-19 pandemic compared to usual care. In addition, some studies also highlighted challenges with technology, hearing difficulties, and communication barriers within the vulnerable population.

Conclusions

During the COVID-19 pandemic, DHTs had the potential to improve various health outcomes and showed benefits for older adults’ access to health care services.

Keywords

COVID-19 pandemic digital health older adults systematic review telehealth

Introduction

Healthcare systems around the world experienced challenges due to the Coronavirus Disease 2019 (COVID-19) pandemic. Therefore, routine healthcare services were severely restricted in many parts of the world, especially for vulnerable populations such as older adults. During the COVID-19 outbreak, many community services, which are deemed “nonessential,” were suspended, including senior citizen centers, dementia care centers, geriatric clinics, and long-term care facilities.¹ According to findings from the National Poll on Healthy Aging, 29% of participants indicated a disruption of primary care appointments and 28% indicated a disruption of procedures, tests, or operations in 2021.² A specialized dental clinic for older adults in Switzerland reported a potential 81% decrease in clinical activity by December 2020 compared to 2019 (p < 0.0001).³ The pandemic and lockdowns also brought significant challenges to the care for older adults with cancer^4–6 and mental health.^7–9 Additionally, older adults living in rural areas faced additional risks and challenges related to COVID-19 as rural hospitals and clinics were vulnerable to closure during this pandemic, due to limited elective and routine procedures.^10,11 Other social and economic infrastructure collapse in rural communities also had negative long-term impact on rural older adults beyond COVID-19.^10,12

According to the U.S. Food and Drug Administration (FDA), digital health technologies (DHTs) are electronic tools, systems, software, or devices that collect, share, and analyze health information for purposes of improving health outcomes and healthcare delivery.^13–15 During the pandemic, DHTs played essential roles in aiding diagnosing, monitoring, screening, and prognosticating the infection.^16,17 As a result, DHT usage for older adults surged in 2020. In the US, the growth of telemedicine offset about two-thirds of the decrease in in-person clinical visits during the COVID-19 pandemic.¹⁸ Data from Singapore indicated that a larger proportion of older adults accepted teleconsultations compared with younger patients seen by the general medicine department.¹⁹ A study conducted in the UK assessed consultations between February and May 2020 for older adult patients and found the rate of telephone and electronic/video consultations more than doubled across the study period (106% and 103%, respectively), while face-to-face consultations fell by 65% and home visits decreased by 63%.²⁰

There was increasing evidence in challenges of implementation and utilization of DHTs for older adults during unique circumstances, such as the COVID-19 pandemic, when DHTs were the only available options for healthcare.²¹ The sudden surge in demand and limited transition time placed immense pressure on digital health systems, leading to technical difficulties and service delays.²² Also, the pandemic has highlighted disparities in access to digital health tools particularly among vulnerable populations who may not have access to the necessary technology or internet connectivity, such as older adults.^23–25 An umbrella review of systematic reviews evaluated publications from January 2000 to October 2019, indicating the possibility for digital interventions to reduce demands on hospitals.¹⁴ Scoping reviews reported that mental health and DHTs were major factors influencing older people's health during the pandemic.^26–28 A systematic review found that telemedicine was a feasible approach to assist older patients with dementia.²⁹ A comprehensive understanding of what happened in the wake of the rapid implementation and use of DHTs be vital for communities, health professionals and stakeholders. Lessons learned during pandemic can be used beyond the pandemic. Therefore, the aim of this systematic review is to answer the research question: what is the current evidence on the overall implementation and use of DHTs among older adults during the COVID-19 pandemic?

Methods

Search strategy

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines provided the framework for this systematic review. Databases included in the search were PsycINFO, Ovid National Library of Medicine's bibliographic database/article index (MEDLINE), Cumulative Index to Nursing and Allied Health Literature (CINAHL), International Pharmaceutical Abstracts (IPA), and Web of Science. The search dates included 1 January 2020, the beginning of the pandemic, to 9 November 2021. The search was then updated on 5 January 2023, to include newer studies published after 9 November 2021.³⁰ To identify relevant articles, we also hand searched the reference lists of included articles and review papers and those articles identified via this method were also evaluated for eligibility.³¹ An extensive search strategy was developed with the guidance of a Health Sciences Librarian at the primary author's university (Supplementary Appendix 1). To detect duplicates and screen articles in accordance with the PRISMA guidelines, all study citations were exported into the Endnote X9 (Philadelphia, PA, USA) reference management software. Participant consent was not required for this study because all data were from previously published articles. Additionally, there was no direct involvement with patients in the current study.

Eligibility criteria

To identify eligible studies, CZ and AH independently screened these studies based on their titles and abstracts, and all duplicates were removed. Full text articles meeting the following criteria were retained and reviewed by CZ and AH: (a) peer-reviewed studies written in English and with completed results; conference abstracts, letters to editors, and research proposals were excluded; (b) applied experimental study design instead of observational studies; (c) the targeted interventions were related to any kind of DHTs; (d) the target population included older adults (aged 65+) only with samples specifically identified (i.e. aged 60+ or aged 55+, vs “older adults” or “elder population”); (e) the interventions were conducted after the start of COVID-19; (f) the study was focused on clinical or humanistic outcomes (e.g. acceptability, feasibility, effectiveness, adherence, and satisfaction) of DHTs. Studies focused economic outcomes,³² were excluded. All included and excluded items were reviewed by CZ and AH to determine whether they met the criteria, and conflicts were solved by discourse and consensus among the research team. This review followed the FDA definition of digital health, which includes “mobile health (mHealth), health information technology (IT), wearable devices, telehealth and telemedicine, and personalized medicine.”¹⁵ Artificial intelligence (such as robotic devices) was also included as a digital health intervention.³³

Data extraction

Data extraction was performed by CZ and verified by AH, JQ, and BIF. The following data were extracted from the selected full-text studies: (a) authors; (b) study design (i.e. quasi-experimental study, randomized controlled trial (RCT); non-RCT); (c) DHT interventions; (d) comparations; (e) sample size; (f) samples; (g) settings; (h) outcomes; (i) results; and (j) location.

This study followed the Synthesis without meta-analysis in systematic reviews guidance.³⁴ Because of the heterogeneity in settings, outcome measures, and study designs, conducting a meta-analysis was not feasible for this study.³⁵ A narrative synthesis was performed by grouping DHT outcomes: effectiveness, feasibility, acceptability, and satisfaction. Effectiveness was measured by the P-values for the clinical outcomes from the individual study,³⁴ while feasibility was assessed based on the recruitment and retention.³⁶ Acceptability and satisfaction were evaluated through humanistic outcomes.³⁷

Quality assessment

The bias of RCT studies was assessed using Version 2 of the Cochrane Risk-of-Bias Tool for randomized trials (RoB 2).³⁸ This tool includes five domains of bias: randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported results. All domains of RoB 2 and the overall risks of bias were judged as high, low, or some concerns.³⁹ The bias of other studies (including both quasi-experimental studies and non-randomized clinical trials) was assessed using Risk of Bias in Non-randomized Studies of Interventions (ROBINS-I).⁴⁰ This tool includes seven domains of bias: confounding, selection of participants into the study, classification of interventions, deviation from intended interventions, missing data, measurement of outcomes, and selection of reported result. Domains for ROBINS-I were judged as low, moderate, serious, critical, no information, and overall risks of bias were judged as low, moderate, serious, and critical.⁴¹

Results

A total of 569 records were identified through database searching and hand-searching, and 52 articles were excluded after removing duplications. After screening titles and abstracts, we included 42 articles for full-text review. Finally, 15 full-text articles met all inclusion and exclusion criteria and were selected for this systematic review based on the original search, and additional five articles were included from the updated search conducted on 5 January 2023 (Figure 1).

Figure 1.

Study selection flowchart.

Characteristics of the studies

The overall characteristics of the 20 included studies are described in Table 1. Most (12, 60%) were published in 2021. Five (25%) studies were conducted in Asia, five (25%) in Europe, and three (15%) in the Middle East. There were six (30%) RCTs, while the rest 14 were quasi-experimental studies (one study with a separate comparison group and thirteen studies with one-group pretest–posttest design). The total number of participants of RCTs and non-RCTs was 542, accounting for 47.70% of the total number of participants of 20 included studies. The longest duration of interventions was 6 months,⁴² while most studies lasted for 1 to 3 months. Hsieh CH et al. explored immersive garden experience using virtual reality devices in nursing homes, so their intervention lasted 6 min.⁴³

Table 1.

Characteristics of included studies.

Author	Study design	DHT interventions	Comparisons	Sample size	Participants	Settings	Outcomes	Results	Location
Plumb Vilardaga et al.	RCT	Two telephone-delivered treatment sessions with a fitness tracker, and 6 weeks follow-up	Usual care plus a fitness tracker	40	Adults (>65 yrs) with OA pain in the knee or hip	Individual	Feasibility and acceptability	Significant improvements in arthritis pain severity (M_diff = 1.68, p = 0.044) and self-reported activity (M_diff= 0.875, p = 0.038) for intervention group. 5% attrition, 100% completion, and 89% satisfaction.	USA
Adly AS et al.	RCT	Laser acupuncture (6 sessions per week for 4 weeks), methotrexate and a tele-rehabilitation program	Methotrexate and a tele-rehabilitation program	60	Adults (65–75 yrs) with rheumatoid arthritis	Individual	Effectiveness and satisfaction	Statistically significant differences (p < 0.05) in CRP, RAQoL, IL-6 and MDA for intervention group. Satisfaction scored 4.7/5.0.	Cairo
Wong AKC et al.	RCT	Weekly case management via telephone call and video messages for 3 months	Monthly social telephone calls	68	Homebound adults (>60 yrs)	Individual	Effectiveness	Significant improvement in group interactions, time's effects on medication adherence (β = −8.30; p = .001), and QoL (β = 4.99; p = .04) for intervention group.	China
Goodman-Casanova JM et al.	RCT	TV-based Assistive Integrated Service for 12 months	No intervention	93	Community-dwelling adults (>60 yrs) with mild cognitive impairment or dementia	Community	Effectiveness	Respondents with TV-AssistDem performed more memory exercises (52% vs 17.4%; p < .001).	Spain and Romania
Park EA, Jung AR, & Lee KA	RCT	Robot-assisted cognitive training program, twice a week for 6 weeks	No intervention	135	Community-dwelling adults (>60 yrs) with mild cognitive impairment	Community	Effectiveness	Significant improvement in in cognitive function (t = 4.707, p < 0.001), memory (t = −2.282, p = 0.007), executive function (t = 4.610, p < 0.001), and depression (t = −3.307, p = 0.004) for intervention group.	Korea
Shapira S et al.	RCT	Seven twice-weekly online online guided sessions via Zoom	No intervention	86	Community-dwelling adults (>65 yrs)	Community	Effectiveness	Significant improvement in loneliness (d = 0.58) and depressive symptoms (d = 0.43) for intervention group.	Israel
Lai FH-Y et al.	Non-randomized controlled trial	Weekly health services are delivered through video communication apps and weekly care service via telephone	Weekly care service via telephone	60	Adults (65–80 yrs with NCD) and caregivers	Community	Effectiveness	Supplementary telemedicine had averted the deterioration cognitive assessment (η_p² = 0.50) and QoL-AD (η_p² = 0.23).	China
Jiwani R et al.	Pre-Post	Self-monitoring Fitbit device for 6 months	N/A	20	Community-dwelling overweight or obese adults (>65 yrs) with T2D	Community	Acceptability	High acceptability of the program. Participants reported favorable experiences and high QoL.	USA
Brandao L et al.	Pre-Post	Playful living program, video calls, and WhatsApp groups for 3 months	N/A	24	Adults (>60 yrs) with low socioeconomic conditions and medical history of stroke or diagnosis of dementia	Individual	Feasibility	83.7% of adherence and sustainability.	Brazil
Cooper C et al.	Pre-Post	Weekly group-based video-call for 10 weeks	N/A	17	Adults (>60 yrs) with memory concerns but without dementia	Individual	Acceptability and feasibility	83.3% sessions were attended. Social connections supported group attendance and implementation of lifestyle changes.	UK
Schwartz H et al.	Pre-Post	Online PA protocol via Zoom and WhatsApp for 8 weeks	N/A	30	Community-dwelling adults (>60 yrs)	Community	Feasibility and satisfaction	90% Adherence. Satisfaction 6/7.	Israel
van Dijk SDM et al.	Pre-Post	Online therapy program for 20 weeks	N/A	4	Adults (>60 yrs) with chronic affective disorders and personality problems	Individual	Feasibility	100% Adherence. High Satisfaction.	Netherlands
Gareri P et al.	Pre-Post	Remote care management via telephone/video call for 90 days	N/A	90	Older adults with cognitive disorders	Ambulatory outpatients and home care patients	Effectiveness	Significant decrease in ADL (p = 0.001), MMSE (p < 0.001), and EuroQoL (p = 0.005).	Italy
Hsieh CH et al.	Pre-Post	6 min garden video experience	N/A	14	Adults (>50 yrs) with mild-to-moderate cognitive impairment living in nursing homes	Long-term care facilities	Effectiveness and satisfaction	Significant improvement in SDNN (Z = 2.229, p = 0.026). Participants expressed their satisfaction.	Taiwan
Prieto-Moreno et al.	Pre-Post	ActiveHip + mHealth mobile application for 12 weeks	N/A	69	Adults (>60 yrs) had undergone hip surgery and their caregivers	Individual	Feasibility	Positive effects on functional status (68.30/105.73 vs. 109.20/121.21), physical performance (3.14/7.22 vs. 3.36/6.85) and pain (5.28/2.14) vs. 5.21/3.06).	Spain and Belgium
Appel et al.	Pre-Post	VR films and weekly online sessions for 4 weeks	N/A	10	Grandchildren and grandparents (66–84 yrs)	Community	Effectiveness	Feel less isolated, more connected with other people and more confident in learning technology	Canada
Sekhon et al.	Pre-Post	Weekly phone calls from trained volunteers for 8 weeks	N/A	112	Adults (>60 yrs). Exclude individuals had psychotic symptoms, severe hearing impairment, or active suicidal ideation.	Community	Effectiveness	Significant improvements in depression (mean change score = −2.27, p = 0.003) for participants with higher baseline depression scores.	Canada
Johnson N et al.	Pre-Post	Online intervention contents via Google Classroom for 6 weeks	N/A	13	Cognitively intact adults (>65 yrs) with BMI ≥ 30 kg/m²	Individual	Effectiveness	Moderate-to-vigorous PA increased by 2 min/day (CI: −21, 26) and 12 min/week (CI: −154, 180).	USA
Kim A et al.	Pre-Post	Home-based telehealth exercise program for 8 weeks	N/A	21	Adults (>62 yrs) with parkinsonism	Outpatient	Effectiveness	Significant increase in the total amount of exercise (p < .001), PDQ-39 (p = 0.02), and GDS (p = 0.04).	Korea
Kirwan et al.	Pre-Post	Twice-weekly supervised group exercise and education program via videoconference for 8 weeks	N/A	171	Adults with T2D	Community	Effectiveness	Significant improvements in waist circumference, aerobic capacity, muscular strength, flexibility, and balance (all p < 0.001).	Australia

Note: BMI: body mass index; CRP: C-reactive protein; GDS: geriatric depression scale; IL-6: interleukin-6 inflammatory markers; MDA: plasma malondialdehyde oxidative marker; NCD: neurocognitive disorder; OA: osteoarthritis; PA: physical activity; PDQ-39: Parkinson's disease questionnaire-39; Pre-Post: one-group pretest-posttest design; QoL: quality of life; QoL-AD: quality of life in Alzheimer's disease; RAQoL: rheumatoid arthritis quality of life; RCT: randomized control trails; SDNN: standard deviation of NN intervals (interval between two heartbeats); T2D: type 2 diabetes.

Features for participants and interventions

Most (14, 70%) studies focused on older adults with chronic diseases or symptoms, and eight studies targeted older adults with mental health issues, including cognitive impairment or dementia,^43–47 memory concerns,⁴⁸ neurocognitive disorder,⁴⁹ and affective disorders.⁵⁰ Other chronic diseases or symptoms included pain,⁵¹ type 2 diabetes,^42,52 Parkinsonism,⁵³ rheumatoid arthritis,⁵⁴ and obesity.^42,55 It is worth mentioning that two studies targeted both older adults and caregivers,^49,56 and one study also included grandparents and grandchildren.⁵⁷

DHTs encompassed various distinct applications, such as health care services delivered via television,⁴⁷ telephone,^51,58,59 video meeting,^46,48,49,60 social media,^45,61 mobile applications,^53,56 and website-based telehealth.^50,52,55 In addition to “traditional” DHTs, several new approaches were implemented in health care sectors. Adly AS et al. evaluated a novel teletherapy approach (i.e. laser acupuncture) for older adults with rheumatoid arthritis.⁵⁴ Park EA, Jung AR, and Lee KA conducted an RCT and indicated that robot-assisted, multi-domain cognitive training can promote cognitive function and depression for older adults with mild cognitive impairment.⁴⁴ Some studies investigated the utilization of wearable devices, such as virtual reality devices,^43,57 and self-monitoring Fitbit devices.⁴²

Effectiveness

These twenty studies used various measures and outcomes to assess the effectiveness of DHTs, including mental wellbeing, physical activity, quality of life, and other clinical outcomes. Studies reported various mental health-related outcomes.^{44,47,49,51,53,59,60} Park EA, Jung AR, and Lee KA found that participants receiving robot-assisted cognitive training had significant improvement in cognitive function (t = 4.707, p < 0.001), memory (t = −2.282, p = 0.007), executive function (t = 4.610, p < 0.001), and depression (t = −3.307, p = 0.004). Shapira S et al. found a medium ameliorative effect on loneliness (d = 0.58), and marginally significant improvement on depressive symptoms (d = 0.43) for the intervention group, which completed online, guided sessions via the Zoom videoconferencing platform.⁶⁰

DHT interventions also focused on sedentary behavior and physical activity among older adults.^52,53,55 During the COVID-19 pandemic, an in-person, community-based exercise program for adults with type 2 diabetes was transited to a web-based group program, and significant improvements were found in five physical fitness outcomes.⁵² Kim A et al. used International Physical Activity Questionnaire (IPAQ), Parkinson's Disease Questionnaire-39 (PDQ-39), and Geriatric Depression Scale (GDS) to measure physical activity levels for older adults with Parkinsonism.⁵³ After an 8-week home-based exercise program, participants reported a significant increase in the total amount of exercise (p < .001) and significant improvements in the IPAQ (p = 0.006), PDQ-39 (p = 0.02), and GDS (p = 0.04) scores.

Feasibility

Several studies included in this review reported the high feasibility of DHTs for older adults. Brandao L et al. found that the group-based remote activities were feasible and well-received among older adults living with cognitive impairments.⁴⁵ Additionally, a physical activity protocol that involved training at home was shown to be feasible for community-dwelling older adults,⁶¹ and sessions were also found to be “relaxing and fun” for patients with affective disorders.⁵⁰ Despite the promising feasibility of DHTs for older adults, Prieto-Moreno et al. emphasized the limitations in preliminary results and called for future RCTs to better measure the clinical outcomes of DHTs.⁵⁶

Acceptability and satisfaction

Studies focusing on participants’ acceptability and experience applied qualitative methods, such as focus groups and interviews. Included studies reported high acceptability of DHTs, and participants reported favorable experiences.^{42,43,45,48,50,51,54,61} Cooper C et al. identified that the overarching theme during the APPLE-Tree program was social connectedness, as well as three sub-themes: retaining independence and social connectedness, adapting social connectedness in the pandemic, and managing social connections within and through the intervention.⁴⁸

Challenge and difficulties for older adults to adopt DHTs

Studies also highlighted challenges within this population. Jiwani R et al. reported that challenges during the Fitbit Program included “physical impediments”, “resistance to lifestyle changes”, and “operating a new piece of technology”.⁴² Schwartz H et al. found that some participants reported technology issues, such as login difficulties, unstable internet connection, and inability to turn on the camera. Other comments are about the only exercise protocol that some instructions were difficult and unclear.⁶¹

Quality of included studies

The results of quality assessment of RCT studies using RoB 2 are shown in Table 2. Because of missing information about allocation concealment, blinding, or random sequence generation, five studies have some concerns of the overall risk of bias, and one was judged as serious overall risk. The results of quality assessment of the other studies using ROBINS-I are shown in Table 3. There were three studies judged as serious risk of bias in at least one domain, nine were judged as moderate overall risk of bias, and two were judged as low risk of bias at all seven domains. No study is judged to be at critical risk of bias.

Table 2.

Quality assessment of five RCT studies using RoB 2.

Author	Domain 1	Domain 2	Domain 3	Domain 4	Domain 5	Overall
Adly AS et al.	Some concerns	Some concerns	Low risk	Some concerns	Low risk	Some concerns
Goodman-Casanova JM et al.	Some concerns	Some concerns	Some concerns	Some concerns	Some concerns	Some concerns
Park EA, Jung AR, & Lee KA	Some concerns	Low risk	Low risk	Low risk	Low risk	Some concerns
Plumb Vilardaga et al.	High risk	Low risk	Low risk	Some concerns	Low risk	High
Shapira S et al.	Some concerns	Some concerns	Some concerns	Low risk	Low risk	Some concerns
Wong AKC et al.	Some concerns	Some concerns	Low risk	Low risk	Low risk	Some concerns

Note: Domain 1: risk of bias arising from the randomization process; Domain 2: risk of bias due to deviations from the intended interventions; Domain 3: missing outcome data; Domain 4: risk of bias in measurement of the outcome; Domain 5: risk of bias in selection of the reported result.

Table 3.

Quality assessment of the other 10 studies using ROBINS-I.

Author	Domain 1	Domain 2	Domain 3	Domain 4	Domain 5	Domain 6	Domain 7	Overall
Appel et al.	Moderate	Moderate	Low	Low	Low	Moderate	Low	Moderate
Brandao L et al.	Moderate	Low	Low	Moderate	Low	Moderate	Low	Moderate
Cooper C et al.	Serious	Serious	Low	Moderate	Moderate	Low	Low	Serious
Gareri P et al.	Moderate	Low	Serious	Serious	Low	Low	Moderate	Serious
Hsieh CH et al.	Moderate	Low	Low	Moderate	Low	Low	Low	Moderate
Jiwani R et al.	Moderate	Moderate	Moderate	Moderate	Low	Moderate	Low	Moderate
Johnson N et al.	Moderate	Moderate	Low	Moderate	Low	Low	Low	Moderate
Kim A et al.	Low	Low	Low	Low	Low	Low	Low	Low
Kirwan et al.	Low	Low	Low	Low	Low	Low	Low	Low
Lai FH-Y et al.	Moderate	Low	Moderate	Moderate	Low	Low	Low	Moderate
Prieto-Moreno et al.	Moderate	Low	Low	Low	Low	Moderate	Low	Moderate
Sekhon et al.	Low	Low	Low	Low	Low	Moderate	Low	Moderate
Schwartz H et al.	Moderate	Low	Moderate	Moderate	Low	Low	Low	Moderate
van Dijk SDM et al.	Moderate	Serious	Serious	Serious	Low	Serious	Low	Serious

Note: Domain 1: confounding; Domain 2: selection of participants; Domain 3: classification of interventions; Domain 4: deviations from intended interventions; Domain 5: missing data; Domain 6: measurement of outcomes; Domain 7: selection of reported results.

Discussion

This systematic review synthesized evidence in the implementation and use of DHTs among older adults during the COVID-19 pandemic. In this systematic review, 20 articles implemented various forms of DHT programs in different settings, highlighting the breakthrough of DHTs in healthcare, mental health, community, and long-term care among older patients and caregivers. During the COVID-19 pandemic, DHTs met the current health care needs of older adults while demonstrating the potential to improve various health outcomes.

Current evidence supported the importance of DHTs among the older adult population. This systematic review found that DHTs had been implemented or incorporated into other programs to address the challenges and difficulties associated with healthcare for older adults. During the COVID-19 pandemic, DHTs were used as a feasible alternative and adjunct to conventional usual care delivery to improve clinical outcomes, promote mental well-being, and provide educational and communicational support. Banskota S et al. evaluated 15 smartphone applications for older adults during the COVID-19 pandemic lockdown, indicating the potential of mobile technologies to improve older adults’ well-being and quality of life.⁶² Recent systematic reviews of telehealth have demonstrated its capability to improve health service delivery, reduce medical costs, and decrease the risk of morbidity and mortality among older adults.^63,64 Consistent with these studies, the current systematic review found DHTs to be effective among the elder population, including participants with chronic diseases or mental health problems. Results of this study are consistent with a previous meta-analysis focusing on telemedicine in patients with chronic disease.⁶⁵ Authors reported that telemedicine had a positive effect on the management of several chronic diseases during the COVID-19, including diabetes, hypertension, and rheumatoid arthritis. Findings of this systematic review are consistent with the previous review conducted by Rodrigues NG et al. that summarized the benefits of online interventions to combat social isolation among older adults, revealing that remote interventions could be the “new normal in the COVID era.”⁶⁶

Implications

The COVID-19 pandemic accelerated the adoption of DHTs, and this transit has important implications for older adults and health providers. Despite the importance and popularity of DHTs, health providers should be aware that not all older patients may be amenable to DHTs, and the implementation of DHTs may not be appropriate in all settings.⁶⁷ Studies included in this systematic review revealed barriers and challenges, such as patient resistance of change and technology issues.^42,61 Older adults may benefit from health services via DHTs, while Haimi, M and Gesser-Edelsburg, also reported insufficient services were developed and implemented to fully satisfy the needs of this population.⁶⁴ Hawley CE et al. identified patient-reported challenges to integrating home telehealth visits, including interest, access to technology, and confidence.⁶⁸ Additionally, DHTs cannot substitute completely for in-person visits in some specific settings. Research conducted in Spain found that the reduction in face-to-face visits may negatively impact follow-up and blood sugar control among patients with type 2 diabetes mellitus. Regulation issues, such as privacy disclosures and reimbursement policies, are also vital factors during the implementation of DHTs. In the U.S., the government issued an emergency waiver suspending the requirement to comply with the Health Insurance Portability and Accountability Act (HIPAA), meaning that video platforms like Apple FaceTime and Facebook Messenger video chat—which are not HIPAA compliant—may be used.⁶⁹ The U.S. Center for Medicaid and Medicare Services also updated their guidelines to allow patients to use remote counseling for Medicare telehealth encounters. As flexible interventions, DHTs may contribute to various programs for older adults, but further research is needed to optimize DHTs across settings.

While the use of DHTs among older adults has been shown to be important, external factors such as socioeconomic status, geographic location, and digital literacy can pose barriers to older adults seeking to adopt DHTs, which could increase the health disparities.^70–72 To address these issues, efforts can be made to provide training and resources to increase digital literacy among underserved populations.⁷³ Additionally, expanding access to affordable DHTs, such as through subsidized programs or partnerships with healthcare providers, could help to increase the adoption rates and feasibility of DHTs.^74,75 It is also crucial to consider diversity in study populations and ensure that DHT interventions are accessible, affordable, user-friendly, and tailored to meet the unique needs of all older adults, including those from diverse backgrounds.^76,77 By addressing these challenges and promoting equitable access to DHTs, older adults could benefit from the full potential of these technologies to improve their health outcomes and quality of life.

The diverse characteristics of DHTs studies, such as design methodologies, target populations, and intervention and delivery types, may impact the outcomes of these studies. It is possible that studies with different designs (e.g. RCTs, quasi-experimental studies, and qualitative studies) yield different results, with RCTs generally considered more robust than quasi-experimental studies.⁷⁸ Also, the inclusion of diverse populations could lead to various experiences of DHTs. For example, older adults with cognitive impairments may face additional challenges in using DHTs, such as difficulties with technology and communication and need more personalized and accommodated support.^79–81 Additionally, preferences of patients and health providers regarding the intervention and delivery types of DHTs may vary. Lai R et al. reported that older adults with cognitive impairment were more receptive to the reminder function, emergency feature, and wearable form of the telehealth than carer monitoring.⁷⁹ Future study is needed to understand to whom, when, and how clinicians and institutions should deploy larger-scale efforts to integrate DHTs into the care of older adults, with a larger sample size and longer follow up.

Limitations

This study includes articles published from the beginning of the pandemic (1 March 2020) until data extraction (5 January 2023), for a total of approximately 22 months. Relevant articles were identified through a comprehensive literature search using well-defined inclusion criteria. However, there are several limitations and challenges in this systematic review. First, our search might have missed studies published after the date of data extraction, conducted during the early months of the pandemic that are not yet published, missing from the search-engines or databases, or published in a language other than English.⁶⁶ Second, since we are limited to interventional studies and original research papers, considering the complexity and time requirement to conduct randomized or controlled experiments, it is difficult to report new insights in a short period of time. Most experimental studies applied non-randomized or no controlled study design and for RCTs, the sample size is relatively small with short follow-up. Typically, studies with small numbers of observations and less robust designs may result in unreliable or inconclusive results.^82,83 Another limitation is that a variety of sampling, designs, qualities, and settings in the included articles may affect synthesis. Rodrigues NG et al. also indicated that it could result in bias to combine community-based studies and research conducted in long-term care facilities in a single review.⁶⁶ In addition, a meta-analysis could not be conducted in this study, which limits the generalizability and broader implications of findings.

Conclusion

This systematic review found that DHTs have been shown to be effective, feasible, acceptable, and satisfactory in delivering care for older adults during the COVID-19 pandemic. The significance of this study lies in its contribution to the understanding of the challenges and opportunities—broadly—of implementing DHTs in older adult populations. Patients in the older adult population need more resources to improve their awareness of and technical skill with mobile devices. One example could be creating user-friendly digital health tools that are specifically tailored to older adults’ needs and preferences, such as larger text, simple navigation, and voice commands.^84,85 However, due to the heterogeneous methodologies and varying quality of the studies included in this review, it was challenging to draw definitive conclusions on certain aspects, such as calculating the exact efficacy size of DHTs. For future studies, RCTs with larger sample sizes, blinding techniques, and longer follow-up periods are expected to minimize the risk of bias.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076231191050 - Supplemental material for A systematic review of digital health technologies for the care of older adults during COVID-19 pandemic

Supplemental material, sj-docx-1-dhj-10.1177_20552076231191050 for A systematic review of digital health technologies for the care of older adults during COVID-19 pandemic by Chenyu Zou, Abbey Harvard, Jingjing Qian and Brent I Fox in DIGITAL HEALTH

Footnotes

Acknowledgements

The authors would like to thank Mrs. Adelia Grabowsky for her guidance in developing the literature search strategy.

Contributorship

CZ: conceptualization, article screening, data extraction, quality assessment, interpretation of the findings, and writing (original draft). AH: article screening, data extraction, quality assessment, and writing (review and editing). JQ: conceptualization, writing (review and editing), and interpretation of the findings. BIF: conceptualization, writing (review and editing), interpretation of the findings, and supervision. All authors contributed to the article and approved the submitted version.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

This article does not contain any studies with patient-level human data performed by any of the authors.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Guarantor

BIF.

ORCID iD

Chenyu Zou

Supplemental Material

Supplemental material for this article is available online.

References

Lim

W-S

Liang

C-K

Assantachai

, et al. COVID-19 and older people in Asia: Asian Working Group for Sarcopenia calls to action. Geriatr Gerontol Int 2020; 20: 547–558.

Jeffrey Kullgren

. Pandemic disruptions mean many older adults still haven’t gotten needed care, (2022, accessed 15 November 2022).

Lundberg

Hillebrecht

McKenna

, et al. COVID-19: impacts on oral healthcare delivery in dependent older adults. Gerodontology 2021; 38: 174–178.

Krok-Schoen

Pisegna

BrintzenhofeSzoc

, et al. Experiences of healthcare providers of older adults with cancer during the COVID-19 pandemic. J Geriatr Oncol 2021; 12: 190–195.

Okechukwu

Deb

, et al. Precautionary measures before tailoring and commencing a tele-supervised home-based exercise oncology program for older patients with cancer and post-treatment cancer survivors in the COVID-19 era. J Geriatr Oncol 2022; 13: 241–244.

Mourey

Falandry

de Decker

, et al. Taking care of older patients with cancer in the context of COVID-19 pandemic. Lancet Oncol 2020; 21: e236.

Yang

Wang

Tian

, et al. The experiences of community-dwelling older adults during the COVID-19 lockdown in Wuhan: a qualitative study. J Adv Nurs 2021; 77: 4805–4814.

Shteinlukht

Patel

Asghar-Ali

, et al. Addressing behavioral health of the diverse older adults in the various health care setting during COVID pandemia. Am J Geriatr Psychiatry 2022; 30: S20–S21.

Van der Roest

Prins

van der Velden

, et al. The impact of COVID-19 measures on well-being of older long-term care facility residents in The Netherlands. J Am Med Dir Assoc 2020; 21: 1569–1570.

10.

Henning-Smith

. The unique impact of COVID-19 on older adults in rural areas. J Aging Soc Policy 2020; 32: 396–402.

11.

Hagens

. Socioeconomic disparities in the demand for and use of virtual visits among senior adults during the COVID-19 pandemic: cross-sectional study. JMIR Aging 2022; 5: e35221.

12.

Hale

Meit

Pettyjohn

, et al. The implications of long COVID for rural communities. J Rural Health 2022; 38: 945–947.

13.

Sharma

Harrington

McClellan

, et al. Using digital health technology to better generate evidence and deliver evidence-based care. J Am Coll Cardiol 2018; 71: 2680–2690.

14.

Kunonga

Spiers

Beyer

, et al. Effects of digital technologies on older people's access to health and social care: umbrella review. J Med Internet Res 2021; 23: e25887.

15.

Food and Drug Administration. What is Digital Health? (2020, accessed 15 September 2022).

16.

Tilahun

Gashu

Mekonnen

, et al. Mapping the role of digital health technologies in prevention and control of COVID-19 pandemic: review of the literature. Yearb Med Inform 2021; 30: 26–37.

17.

Wosik

Fudim

Cameron

, et al. Telehealth transformation: COVID-19 and the rise of virtual care. J Am Med Inform Assoc 2020; 27: 957–962.

18.

Patel

Mehrotra

Huskamp

, et al. Trends in outpatient care delivery and telemedicine during the COVID-19 pandemic in the US. JAMA Intern Med 2021; 181: 388–391.

19.

Tan

Ho Wen Teng

Seetharaman

, et al. Facilitating telehealth for older adults during the COVID-19 pandemic and beyond: strategies from a Singapore geriatric center. Geriatr Gerontol Int 2020; 20: 993–995.

20.

Joy

McGagh

Jones

, et al. Reorganisation of primary care for older adults during COVID-19: a cross-sectional database study in the UK. Br J Gen Pract 2020; 70: e540.

21.

Shen

Chen

Yue

, et al. Digital technology-based telemedicine for the COVID-19 pandemic. Front Med (Lausanne) 2021; 8: 646506.

22.

Perlman

Vodonos Zilberg

Bak

, et al. Characteristics and symptoms of app users seeking COVID-19—related digital health information and remote services: retrospective cohort study. J Med Internet Res 2020; 22: e23197.

23.

Mace

Mattos

Vranceanu

. Older adults can use technology: why healthcare professionals must overcome ageism in digital health. Transl Behav Med 2022; 12: 1102–1105.

24.

Kichloo

Albosta

Dettloff

, et al. Telemedicine, the current COVID-19 pandemic and the future: a narrative review and perspectives moving forward in the USA. Fam Med Community Health 2020; 8: e000530.

25.

Cheshmehzangi

Zou

Zhang

, et al. Commentary: reflecting on the neglected digital divide barriers of telemedicine during COVID-19. Front Public Health 2022; 10: 915401.

26.

Doraiswamy

Jithesh

Mamtani

, et al. Telehealth use in geriatrics care during the COVID-19 pandemic—a scoping review and evidence synthesis. Int J Environ Res Public Health 2021; 18: 1755.

27.

Bastani

Mohammadpour

Samadbeik

, et al. Factors influencing access and utilization of health services among older people during the COVID-19 pandemic: a scoping review. Arch Public Health 2021; 79: 190.

28.

Kruse

Fohn

Wilson

, et al. Utilization barriers and medical outcomes commensurate with the use of telehealth among older adults: systematic review. JMIR Med Inform 2020; 8: e20359.

29.

Elbaz

Cinalioglu

Sekhon

, et al.

A Systematic review of telemedicine for older adults with dementia during COVID-19: an alternative to in-person health services?

Front Neurol 2021; 12. DOI: 10.3389/fneur.2021.761965.

30.

Bramer

Bain

. Updating search strategies for systematic reviews using EndNote. J Med Libr Assoc 2017; 105: 285–289.

31.

Nwanosike

Conway

Merchant

, et al. Potential applications and performance of machine learning techniques and algorithms in clinical practice: a systematic review. Int J Med Inf 2022; 159: 104679.

32.

Middleton

Simpson

Bettger

, et al. COVID-19 pandemic and beyond: considerations and costs of telehealth exercise programs for older adults with functional impairments living at home-lessons learned from a pilot case study. Phys Ther 2020; 100: 1278–1288.

33.

Mathews

McShea

Hanley

, et al. Digital health: a path to validation. npj Digit Med 2019; 2: 38.

34.

Campbell

McKenzie

Sowden

, et al. Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. Br Med J 2020; 368: l6890.

35.

Ray

Sharon

Zoe

, et al. Impact of COVID-19 pandemic on utilisation of healthcare services: a systematic review. BMJ Open 2021; 11: e045343.

36.

King

Doherty

Corcos

, et al. Examining recruitment feasibility and related outcomes in adults post-stroke. Pilot Feasibility Stud 2020; 6: 160.

37.

Sekhon

Cartwright

Francis

. Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res 2017; 17: 88.

38.

Sterne

JAC

Savović

Page

, et al. Rob 2: a revised tool for assessing risk of bias in randomised trials. Br Med J 2019; 366: l4898.

39.

Roman

Burela

Pasupuleti

, et al. Ivermectin for the treatment of COVID-19: a systematic review and meta-analysis of randomized controlled trials. Clin Infect Dis 2021; 74: 1022–1029.

40.

Schünemann

Cuello

Akl

, et al. GRADE Guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. J Clin Epidemiol 2019; 111: 105–114.

41.

Thomson

Craig

Hilton-Boon

, et al. Applying the ROBINS-I tool to natural experiments: an example from public health. Syst Rev 2018; 7: 15–15.

42.

Jiwani

Dennis

Bess

, et al. Assessing acceptability and patient experience of a behavioral lifestyle intervention using fitbit technology in older adults to manage type 2 diabetes amid COVID-19 pandemic: a focus group study. Geriatr Nurs 2021; 42: 57–64.

43.

Hsieh

Chen

Yang

, et al. The effects of immersive garden experience on the health care to elderly residents with mild-to-moderate cognitive impairment living in nursing homes after the COVID-19 pandemic. Landsc Ecol Eng 2021; 18: 45–56.

44.

Park

Jung

Lee

. The humanoid robot Sil-Bot in a cognitive training program for community-dwelling elderly people with mild cognitive impairment during the COVID-19 pandemic: a randomized controlled trial. Int J Environ Res Public Health 2021; 18: 8198.

45.

Brandao

Bauer

Haas

, et al. Playing remotely in times of crisis: a program to overcome social isolation. Int J Geriatr Psychiatry 2021; 37. DOI: 10.1002/gps.5638.

46.

Gareri

Fumagalli

Malara

, et al. Management of older outpatients during the COVID-19 pandemic: the GeroCovid ambulatory study. GERONTOLOGY 2021; 68: 412–417.

47.

Goodman-Casanova

Dura-Perez

Guzman-Parra

, et al. Telehealth home support during COVID-19 confinement for community-dwelling older adults with mild cognitive impairment or mild dementia: survey study. J Med Internet Res 2020; 22: e19434.

48.

Cooper

Mansour

Carter

, et al. Social connectedness and dementia prevention: pilot of the APPLE-Tree video-call intervention during the COVID-19 pandemic. Dementia (London) 2021; 20: 2779–2801.

49.

Lai

FH-Y

Yan

EW-H

KK-Y

, et al. The protective impact of telemedicine on persons with dementia and their caregivers during the COVID-19 pandemic. Am J Geriatr Psychiatry 2020; 28: 1175–1184.

50.

van Dijk

SDM

Bouman

Folmer

, et al. (Vi)-rushed into online group schema therapy based day-treatment for older adults by the COVID-19 outbreak in The Netherlands. Am J Geriatr Psychiatry 2020; 28: 983–988.

51.

Plumb Vilardaga

Kelleher

Diachina

, et al. Linking physical activity to personal values: feasibility and acceptability randomized pilot of a behavioral intervention for older adults with osteoarthritis pain. Pilot Feasibility Stud 2022; 8: 164.

52.

Kirwan

Chiu

Laing

, et al. A web-delivered, clinician-led group exercise intervention for older adults with type 2 diabetes: single-arm pre-post intervention. J Med Internet Res 2022; 24: e39800.

53.

Kim

Yun

Sung

, et al. Exercise management using a mobile app in patients with parkinsonism: prospective, open-label, single-arm pilot study. JMIR Mhealth Uhealth 2021; 9: e27662.

54.

Adly

, et al. A novel approach utilizing laser acupuncture teletherapy for management of elderly-onset rheumatoid arthritis: a randomized clinical trial. J Telemed Telecare 2021; 27: 298–306.

55.

Johnson

Bradley

Klawitter

, et al. The impact of a telehealth intervention on activity profiles in older adults during the COVID-19 pandemic: a pilot study. Geriatrics (Basel) 2021; 6: 68.

56.

Prieto-Moreno

Estévez-López

Molina-Garcia

, et al. Activehip+: a feasible mHealth system for the recovery of older adults after hip surgery during the COVID-19 pandemic. Digit Health 2022; 8: 20552076221139694.

57.

Appel

Lewis

Kisonas

, et al. VRCHIVE: experiences conducting an online workshop teaching intergenerational participants to create virtual reality films about their lives during the COVID pandemic. Educ Gerontol 2022; 48: 305–330.

58.

Wong

AKC

Wong

FKY

Chow

KKS

, et al. Effect of a telecare case management program for older adults who are homebound during the COVID-19 pandemic: a pilot randomized clinical trial. JAMA Netw Open 2021; 4: e2123453.

59.

Sekhon

Lavin

Vacaflor

, et al. Isolating together during COVID-19: results from the Telehealth Intervention Program for older adults. Front Med (Lausanne) 2022; 9. DOI: 10.3389/fmed.2022.948506.

60.

Shapira

Yeshua-Katz

Cohn-Schwartz

, et al. A pilot randomized controlled trial of a group intervention via Zoom to relieve loneliness and depressive symptoms among older persons during the COVID-19 outbreak. Internet Interv 2021; 24: 100368.

61.

Schwartz

Har-Nir

Wenhoda

, et al. Staying physically active during the COVID-19 quarantine: exploring the feasibility of live, online, group training sessions among older adults. Transl Behav Med 2021; 11: 314–322.

62.

Banskota

Healy

Goldberg

. 15 Smartphone apps for older adults to use while in isolation during the COVID-19 pandemic. West J Emerg Med 2020; 21: 514–525.

63.

Abbaspur-Behbahani

Monaghesh

Hajizadeh

, et al. Application of mobile health to support the elderly during the COVID-19 outbreak: a systematic review. Health Policy Technol 2022; 11: 100595.

64.

Haimi

Gesser-Edelsburg

. Application and implementation of telehealth services designed for the elderly population during the COVID-19 pandemic: a systematic review. Health Informatics J 2022; 28: 14604582221075561.

65.

Zhao

, et al. Telemedicine application in patients with chronic disease: a systematic review and meta-analysis. BMC Med Inform Decis Mak 2022; 22: 105.

66.

Rodrigues

Han

CQY

, et al. Psychological impacts and online interventions of social isolation amongst older adults during COVID-19 pandemic: a scoping review. J Adv Nurs 2022; 78: 609–644.

67.

Schrack

Wanigatunga

Juraschek

. After the COVID-19 pandemic: the next wave of health challenges for older adults. J Gerontol Ser A 2020; 75: e121–e122.

68.

Hawley

Genovese

Owsiany

, et al.

Rapid integration of home telehealth visits amidst COVID-19: what do older adults need to succeed?

J Am Geriatr Soc 2020; 68: 2431–2439.

69.

Padala

Wilson

Gauss

, et al. VA Video connect for clinical care in older adults in a rural state during the COVID-19 pandemic: cross-sectional study. J Med Internet Res 2020; 22: e21561.

70.

Wegermann

Wilder

Parish

, et al. Racial and socioeconomic disparities in utilization of telehealth in patients with liver disease during COVID-19. Dig Dis Sci 2022; 67: 93–99.

71.

Rodriguez

Betancourt

Sequist

, et al. Differences in the use of telephone and video telemedicine visits during the COVID-19 pandemic. Am J Manag Care 2021; 27: 21–26.

72.

Albon

Van Citters

Ong

, et al. Telehealth use in cystic fibrosis during COVID-19: association with race, ethnicity, and socioeconomic factors. J Cyst Fibros 2021; 20: 49–54.

73.

Blandford

Wesson

Amalberti

, et al. Opportunities and challenges for telehealth within, and beyond, a pandemic. Lancet Glob Health 2020; 8: e1364–e1365.

74.

VanderWerf

Bernard

Barta

, et al. Pandemic action plan policy and regulatory summary telehealth policy and regulatory considerations during a pandemic. Telemed e-Health 2021; 28: 457–466.

75.

Lindeman

. Lessons from lighthouse: operationalizing technology to support older adults in affordable housing communities. Innov Aging 2021; 5: 465.

76.

Barwise

Huschka

Woo

, et al. Perceptions and use of telehealth among diverse communities: multisite community-engaged mixed methods study. J Med Internet Res 2023; 25: e44242.

77.

Cardona

Fien

Myooran

, et al. Clinical and cost-effectiveness of telehealth for Indigenous and culturally and linguistically diverse (CALD) people: a scoping review. Ethn Health 2023; 28: 114–135.

78.

Miller

Smith

Pugatch

. Experimental and quasi-experimental designs in implementation research. Psychiatry Res 2020; 283: 112452.

79.

Lai

Tensil

Kurz

, et al. Perceived need and acceptability of an app to support activities of daily living in people with cognitive impairment and their carers: pilot survey study. JMIR Mhealth Uhealth 2020; 8: e16928.

80.

Hedman

Lindqvist

Nygård

. How older adults with mild cognitive impairment relate to technology as part of present and future everyday life: a qualitative study. BMC Geriatr 2016; 16: 73.

81.

Thordardottir

Malmgren Fänge

Lethin

, et al. Acceptance and use of innovative assistive technologies among people with cognitive impairment and their caregivers: a systematic review. Biomed Res Int 2019; 2019: 9196729.

82.

Fogel

. Factors associated with clinical trials that fail and opportunities for improving the likelihood of success: a review. Contemp Clin Trials Commun 2018; 11: 156–164.

83.

Bangdiwala

Bhargava

O'Connor

, et al. Statistical methodologies to pool across multiple intervention studies. Transl Behav Med 2016; 6: 228–235.

84.

Wildenbos

Jaspers

MWM

Schijven

, et al. Mobile health for older adult patients: using an aging barriers framework to classify usability problems. Int J Med Inf 2019; 124: 68–77.

85.

Rodriguez

Clark

Bates

. Digital health equity as a necessity in the 21st century cures act era. JAMA 2020; 323: 2381–2382.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.04 MB