Test–retest reliability of the computer-based cognitive assessment tool for community-dwelling older adults in Japan: The Otassha study

Abstract

Aim

Early detection of dementia is crucial for effective intervention. The computer-based cognitive assessment tool (CompBased-CAT) was designed to assess cognitive function using a tablet computer. While its predictive validity for mild cognitive impairment (MCI) is established, its test–retest reliability remains unclear. This study aimed to evaluate the test–retest reliability of CompBased-CAT among older Japanese adults in a community setting.

Methods

This is a methodological study that examined the test–retest reliability of CompBased-CAT. Community-dwelling older adults aged 65 years or older who participated in both the pre-test and post-test were included. Pre-test assessments were conducted during the 2023 Otassha study from 28 September to 8 October 2023, with the post-test administered 62 days (±14 days) later. Subtest scores were normalized to z-scores, and the total CompBased-CAT score was computed by summing these z-scores. Test–retest reliability was assessed using paired t-tests and intraclass correlation coefficient (ICC) two-way random-effects models, with analyses stratified by age, sex, and MCI status.

Results

A total of 80 participants (mean age: 76.8 years; 27.5% male) were analyzed. Paired t-tests showed no significant difference in total scores between the pre-test (mean = −0.75; standard deviation [SD] = 3.73) and the post-test (mean = −0.42; SD = 4.45). The total score exhibited an ICC of 0.64 (95% confidence interval [CI] = 0.49–0.75) for all participants, increasing to 0.74 (95% CI = 0.48–0.88) among those with MCI.

Conclusion

CompBased-CAT demonstrated good test–retest reliability, with an ICC of 0.64 among all participants, which increased to 0.74 among participants with MCI over a 2-month period, indicating its potential for monitoring cognitive function through repeated assessments.

Keywords

Cognitive function dementia community-dwelling older adults screening

Introduction

Dementia is characterized by a progressive decline in cognitive function beyond normal aging.¹ Japan, a super-aging society, has one of the highest global prevalence rates of dementia.² A 2021 study projected that without medical advancements to prevent, slow, or cure dementia, over 25% of the Japanese population could develop the condition.³ Currently, no curative treatments exist, and management strategies focus primarily on prevention.⁴ Mild cognitive impairment (MCI) is an early stage of dementia, with approximately 15% of individuals with MCI progressing to dementia annually.⁵ Studies have shown that effective interventions can slow this progression,^6,7 highlighting the importance of early detection and intervention for MCI.

The Mini-Mental State Examination (MMSE) and the Montreal Cognitive Assessment (MoCA) are widely used cognitive assessment tools with validated predictive validity for MCI.^8,9 However, these tools use fixed items, leading to potential learning effects with repeated use and require administration by trained professionals in specific settings.¹⁰ To address these limitations, the computer-based cognitive assessment tool (CompBased-CAT) was developed. It comprises six subtests—digit span forward, digit span backward, memory of item names, memory recall of item names, Stroop test, and figure recognition—and can be administered using a tablet computer without trained personnel.¹¹ CompBased-CAT has shown moderate correlation with MMSE scores, high sensitivity and specificity in detecting cognitive impairment, and promising predictive capabilities for identifying new MCI cases.^11,12

Although the predictive ability of CompBased-CAT for MCI has been established, its test–retest reliability remains unverified, as the six subtests are designed to minimize learning effects. Older adults often experience rapid changes in cognitive function, physical abilities, and psychological well-being, which present substantial challenges for research in this population. Developing a reliable tool that provides consistent measurements is crucial for the early detection of longitudinal cognitive changes among the older adults. We hypothesized that the CompBased-CAT would demonstrate moderate to high reliability among community-dwelling older adults. Therefore, this study aimed to evaluate the test–retest reliability of CompBased-CAT. Given the challenges older adults and patients with MCI face in adopting new technologies¹³ and potential variations in learning effects based on age, sex, and cognitive impairment, participants were stratified and analyzed accordingly.

Methods

Study design and participants

This study is a methodological study that examines the test–retest reliability of CompBased-CAT. Participants were recruited from the 2023 Otassha study, an ongoing cohort study initiated in 2011 and conducted annually in September and October at the Tokyo Metropolitan Institute for Geriatrics and Gerontology (TMIG). The Otassha study used census sampling, targeting older adults aged 65 and above residing in nine districts of Itabashi Ward, Japan, through mail. This project examined the relationships between physical, mental, social, oral, and cognitive functions and health outcomes.¹⁴

To evaluate the test–retest reliability of CompBased-CAT scores, the pre-test was conducted as part of the 2023 Otassha study from 28 September to 8 October 2023. The post-test was performed during the “Measurement of Body Movement in Community-Dwelling Older Adults” survey from 20 November to 23 December 2023, targeting older adults diagnosed with sarcopenia or knee extensor strength <3.0 N/kg to design supportive products for older adults. The cognitive functions of the participants were confirmed to be representative of a community-based older population.

CompBased-CAT measurement

The CompBased-CAT is a multidomain cognitive assessment tool comprising six subtests: (1) digit span forward (attention and concentration); (2) digit span backward (attention and concentration); (3) memory of item names (immediate memory); (4) memory recall of item names (remote memory); (5) Stroop test (executive function and selective attention); and (6) recognition of figures (spatial perception).¹¹ During the study, participants received audio instructions via a tablet computer (ASUS TransBook T100HA, ASUS, Taipei, Taiwan; OS: Windows 10) and wore noise-canceling headphones for clarity. Each subset began with a practice trial, and for subtest 3, two trials were conducted, with the second attempt used for analysis. The entire test took approximately 30 min and took place at the multipurpose room in the TMIG. Collected data is storage and analyzed in the research lab at TMIG. Subtest scores were converted to z-scores based on the pre-test population's mean and standard deviation (SD), and total CompBased-CAT scores were calculated by summing these z-scores.

Other measurements

Participant characteristics (age, sex, comorbidities) and cognitive function were collected during the pre-test. Comorbidities assessed included hypertension, diabetes mellitus, stroke, and heart disease, obtained through face-to-face nurse interviews. Cognitive function was assessed using the MMSE, which demonstrates good validity and reliability for evaluating cognitive function.¹⁵ The MMSE was conducted by trained staff through face-to-face interviews. Subsequently, participants were categorized into two groups based on their MMSE scores: MMSE ≤ 27, indicating MCI, and MMSE > 27, indicating robust cognitive function.¹⁶

Statistical analysis

Baseline characteristics were described using frequencies and percentages for categorical variables and mean with SD for continuous variables. Paired t-tests were used to compare the mean scores of each subtest and the total CompBased-CAT score between pre-test and post-test assessments. The intraclass correlation coefficient (ICC) two-way random-effects model was used to examine the test–retest reliability, with ICC ranges interpreted as follows: 0.81–1.00 (excellent reliability), 0.61–0.80 (good reliability), 0.41–0.60 (moderate reliability), and <0.40 (poor reliability).¹⁷ Stratified analyses were performed by age, sex, and baseline MMSE score. All statistical analyses were conducted using IBM SPSS software (version 29.0; IBM Japan, Tokyo, Japan), with a significance level of 0.05 and a 95% confidence interval (95% CI).

Ethical approval

All participants were provided detailed information regarding the study's objectives and procedures, and written consent was obtained. The study protocol was approved by the Ethics Committee of the TMIG (approval number: R23-047).

Results

Figure 1 illustrates the study flowchart. Of the 587 participants in the 2023 Otassha study, 574 completed the CompBased-CAT pre-test. Among these, 151 qualified for the second survey, and 84 agreed to participate. Four participants had missing data, resulting in 80 participants being included in the final analysis, with an average interval of 62 days (SD = 14 days) between pre- and post-tests. Notably, the MMSE distribution between pre- and post-test participants showed no significant differences (Supplementary Table 1).

Figure 1.

Study flow chart.

Table 1 shows the characteristics of the study participants. The average age was 76.7 years (SD = 6.4 years), and 27.5% were male. Among the participants, 41.3% had hypertension, 13.8% had diabetes, and 18.8% had a history of heart disease. The average MMSE score was 27.7 (SD = 2.7), indicating robust cognitive function, with 28.7% classified as having MCI.

Table 1.

Baseline characteristics of study participants (n = 80).

	Mean (SD), n (%)
Age, years (SD)	76.7 (6.4)
Males, n (%)	22 (27.5)
Comorbidities
Hypertension, n (%)	33 (41.3)
Diabetes mellitus, n (%)	11 (13.8)
Stroke, n (%)	3 (3.8)
Heart disease, n (%)	15 (18.8)
MMSE	27.7 (2.7)
>27, n (%)	57 (71.3)
≤27, n (%)	23 (28.7)

MMSE: Mini-Mental State Examination; SD: standard deviation

Table 2 displays the mean and SD values of the pre- and post-test CompBased-CAT scores. Paired t-test results revealed no significant changes in mean scores for each section between the pre- and post-tests, except for the “digit span backward” section, which showed a higher mean score in the post-test than pre-test.

Table 2.

Comparison of CompBased-CAT scores between pre-test and post-test.

		Pre-test		Post-test
	Scale range	Mean	SD	Mean	SD	p
Digit span forward	0–8	5.41	1.28	5.58	1.42	0.33
		−0.19	1.07	−0.05	1.18
Digit span backward	0–8	4.35	1.33	4.79	1.24	<0.01
		−0.43	1.05	−0.09	0.98
Memory of item names	0–10	8.76	0.97	8.82	1.05	0.57
		−0.08	0.96	−0.02	1.04
Memory recall of item names	0–10	8.44	1.15	8.36	1.42	0.60
		−0.01	0.93	−0.07	1.15
Stroop	0–20	17.48	4.55	17.48	4.06	1.00
		−0.12	1.17	−0.12	1.05
Recognition of figures	0–6	4.93	0.99	4.74	1.23	0.25
		0.09	0.81	−0.07	1.01
Total CompBased-CAT score	NA	NA	NA	NA	NA
		−0.75	3.73	−0.42	4.45	0.41

The top row shows the original scores, and the bottom row shows the corresponding z-scores. SD: standard deviation; NA: not available.

Table 3 presents the test–retest reliability of the CompBased-CAT for all participants. The total CompBased z-score demonstrated an ICC coefficient (95% CI) of 0.64 (0.49–0.75) between the pre- and post-tests.

Table 3.

Test–retest reliability of the CompBased-CAT assessment.

	ICC (2,1)	95% CI	p
Digit span forward	0.41	0.21 to 0.58	<0.01
Digit span backward	0.46	0.26 to 0.62	<0.01
Memory of item names	0.53	0.35 to 0.67	<0.01
Memory recall of item names	0.52	0.34 to 0.67	<0.01
Stroop	0.39	0.19 to 0.56	<0.01
Recognition of figures	0.15	−0.07 to 0.35	0.10
Total CompBased z-score	0.64	0.49 to 0.75	<0.01

ICC: intraclass correlation coefficient; CI: confidence interval.

Table 4 shows stratification based on baseline cognitive function, age (65–74 and 75 + years) and sex. The reliability of the total CompBased-CAT z-score was 0.74 (0.48–0.88) among participants with MCI (MMSE ≤ 27) and 0.54 (0.34–0.70) among participants with robust cognitive function (MMSE > 27). Supplementary Figure 1 provides scatter plots of the pre- and post-total CompBased-CAT z-scores for overall and cognitive function stratification. The pre- and post-test scores of MCI participants were more concentrated around the line y = x, indicating a higher level of reliability among MCI participants. Participants aged ≥75 years showed slightly higher reliability in total CompBased-CAT z-score than those aged 65–74 (ICC = 0.61, 95% CI: 0.39–0.77 for 75 + years; ICC = 0.50, 95% CI: 0.20–0.71 for 65–74 years). Both males and females exhibited good reliability (ICC = 0.63, 95% CI: 0.29–0.83 for males; ICC = 0.64, 95% CI: 0.46–0.77 for females).

Table 4.

Test–retest reliability of the CompBased-CAT assessment, stratified by cognitive function, age, and sex.

	MMSE ≤ 27 (n = 23)	MMSE > 27 (n = 57)	65–74 years old (n = 36)	≥75 years (n = 44)	Males (n = 22)	Females (n = 58)
	ICC (2,1) [95% CI]	ICC (2,1) [95% CI]	ICC (2,1) [95% CI]	ICC (2,1) [95% CI]	ICC (2,1) [95% CI]	ICC (2,1) [95% CI]
Digit span forward	0.59 [0.23, 0.80]***	0.35 [0.10, 0.55]***	0.15 [−0.19, 0.46]	0.59 [0.36, 0.75]***	0.52 [0.13, 0.77]***	0.36 [0.11, 0.56]***
Digit span backward	0.80 [0.58, 0.91]***	0.31 [0.07, 0.52]**	0.49 [0.20, 0.70]***	0.39 [0.11, 0.62]***	0.29 [−0.15, 0.63]*	0.52 [0.28, 0.69]***
Memory of item names	0.74 [0.48, 0.88]***	0.37 [0.12, 0.57]***	0.66 [0.43, 0.81]***	0.42 [0.14, 0.64]***	0.43 [0.01, 0.72]*	0.55 [0.34, 0.71]***
Memory recall of item names	0.43 [0.01, 0.71]**	0.57 [0.36, 0.72]***	0.52 [0.24, 0.73]***	0.49 [0.23, 0.69]***	0.48 [0.08, 0.75]*	0.53 [0.32, 0.69]***
Stroop	0.49 [0.12, 0.75]**	0.26 [0.01, 0.49]**	0.02 [−0.32, 0.35]	0.40 [0.11, 0.62]***	0.53 [0.16, 0.77]***	0.33 [0.08, 0.54]***
Recognition of figures	0.15 [−0.29, 0.53]	0.14 [−0.12, 0.39]	0.28 [−0.06, 0.55]	−0.01 [−0.30, 0.28]	−0.02 [−0.46, 0.41]	0.18 [−0.08, 0.42]
Total CompBased z-score	0.74 [0.48, 0.88]***	0.54 [0.34, 0.70]***	0.50 [0.20, 0.71]***	0.61 [0.39, 0.77]***	0.63 [0.29, 0.83]***	0.64 [0.46, 0.77]***

ICC: intraclass correlation coefficient; CI: confidence interval.

*p < 0.5, **p < 0.1, ***p < 0.01.

Discussion

This study investigated the test–retest reliability of the CompBased-CAT among community-dwelling older adults, revealing good reliability across all participants, with particularly higher reliability among those with MCI.

The pre- and post-tests, conducted within a 2-month interval, showed no significant differences in means and an ICC of 0.64, indicating good reliability. Consistent with our findings, previous studies on tablet-based cognitive assessment tools reported ICCs ranging from 0.54 to 0.79.^18,19

Subtest analysis revealed that compared to the other subtests, higher ICC values were found for “Memory of item names” and “Memory recall of item names,” reflecting immediate and remote memories, respectively. These results align with previous findings, suggesting that memory-related questions may be easier for older adults to understand.²⁰ Conversely, the “Recognition of Figures” section, which evaluates spatial cognition, showed the lowest reliability. Generally, spatial cognition tests are inherently complex, requiring high-level cognitive processing.²¹ Notably, the “Recognition of Figures” showed extremely low reliability among participants aged 75 and older in this study. Given reports that individuals with Alzheimer's disease often experience declines in spatial cognition,²² assessment of spatial cognition is crucial. However, because such tasks may be more difficult in older adults, it is important to consider changes in overall scores, including spatial cognition, when providing feedback on cognitive assessment.

Stratification by age, sex, and MCI status was conducted to account for potential variations due to unfamiliarity with smartphones or tablets among older adults and potential learning effects in younger or cognitively robust populations. While consistent reliability was observed across sexes, differences in ICC values were noted between age groups and MCI status. Specifically, participants aged ≥75 years and those with MCI had higher ICC values than the 65–74 age group and non-MCI participants, respectively. These results are consistent with previous studies utilizing the MMSE and MoCA tools,^15,23 indicating that CompBased-CAT is a reliable assessment tool for older adults and those with MCI. We posit that the lower ICC in participants aged 65–74 and those with normal cognitive function may be attributed to learning effects, as these groups showed slightly higher post-test scores than pre-test scores, although not statistically significant (ages 65–74: pre-test mean = 1.09, SD = 3.09; post-test mean = 1.27, SD = 3.21; p = 0.742; normal cognitive function: pre-test mean = −0.01, SD = 1.03; post-test mean = 0.18, SD = 0.92; p = 0.140). This finding suggests a better understanding of the assessment process over time, potentially leading to higher post-test scores and lower ICCs. However, the differences in total CompBased-CAT z-scores between pre- and post-tests in these groups were not statistically significant, indicating the absence of a definitive learning effect. Therefore, although the CompBased-CAT was reliable across all stratification groups in this study, it is crucial to ensure sufficient intervals between repeated assessments in early and cognitively normal older adults to achieve more accurate results.

With advancements in information and communication technology, tablet and smartphone use among older adults has significantly increased.²⁴ As a result, tablet-based cognitive assessments are easy and can be conveniently performed for older adults in community settings without professional supervision. This implies that early detection of MCI and appropriate interventions can be widely implemented. Particularly, the CompBased-CAT validated as a good predictor for identifying MCI¹² and was confirmed to have test–retest reliability in this study, accommodating changes in cognitive function due to repeated measurements. This tool can facilitate the development of appropriate community-based interventions for cognitive decline.

Despite its strengths, this study had some limitations. First, the interval between the two tests was 2 months, whereas previous studies examining the reliability of the MMSE and MoCA have generally used periods of 2 weeks to 1 month.^15,25 Although no significant change in physical condition was observed among the participants during the 2-month period, the possibility that cognitive decline over time may have influenced the results cannot be ruled out. Second, the pre- and post-tests were conducted in a laboratory setting. As CompBased-CAT can be administered at home, potential bias may arise depending on the environment.

In conclusion, our findings revealed a good test–retest reliability associated with CompBased-CAT. Notably, higher reliability was observed specifically among participants aged ≥75 years and those with MCI, suggesting that it is a useful tool in populations at a higher risk of cognitive decline. Further studies are required to verify the impact of environmental factors. Alternatively, researchers using the CompBased-CAT should carefully consider the testing environment.

Conclusion

Supplemental Material

sj-docx-1-dhj-10.1177_20552076251317627 - Supplemental material for Test–retest reliability of the computer-based cognitive assessment tool for community-dwelling older adults in Japan: The Otassha study

Supplemental material, sj-docx-1-dhj-10.1177_20552076251317627 for Test–retest reliability of the computer-based cognitive assessment tool for community-dwelling older adults in Japan: The Otassha study by Jou-Yin Chen, Hisashi Kawai, Junta Takahashi, Manami Ejiri, Keigo Imamura and Shuichi P Obuchi in DIGITAL HEALTH

Footnotes

Acknowledgments

The authors appreciate the staff who assisted with the survey and the participants who took part in this study.

Contributorship

SPO and HK conceived and designed the study; SPO, HK, JT, ME, KI, and J-YC conducted the survey; HK and J-YC performed the data analysis; J-YC drafted the manuscript with revisions provided by KI and HK.

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

The study protocol was approved by the Ethics Committee of the Tokyo Metropolitan Institute of Gerontology (approval number: R23-047).

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the team expenses of Digital Transformation for Aging Society, Tokyo Metropolitan Institute for Geriatrics and Gerontology, JSPS KAKENHI [grant number 23K11959], and research funding from the National Institute of Advanced Industrial Science and Technology.

Informed consent

All participants were provided detailed information regarding the study's objectives and procedures, and written consent was obtained.

ORCID iD

Jou-Yin Chen

Hisashi Kawai

Junta Takahashi

Manami Ejiri

Keigo Imamura

Shuichi P Obuchi

Supplemental material

Supplemental material for this article is available online.

References

Livingston

Sommerlad

Orgeta

, et al. Dementia prevention, intervention, and care. Lancet 2017; 390: 2673–2734.

Kinoshita

Kishimoto

. Dementia in Japan: a societal focus. Lancet Neurol 2023; 22: 1101–1102.

Nakahori

Sekine

Yamada

, et al. Future projections of the prevalence of dementia in Japan: results from the Toyama Dementia Survey. BMC Geriatr 2021; 21: 602.

Passeri

Elkhoury

Morsink

, et al. Alzheimer’s disease: treatment strategies and their limitations. Int J Mol Sci 2022; 23: 13954.

Petersen

Lopez

Armstrong

, et al. Practice guideline update summary: mild cognitive impairment: report of the Guideline Development, Dissemination, and Implementation Subcommittee of the American Academy of Neurology. Neurology 2018; 90: 126–135.

van der Flier

de Vugt

Smets

EMA

, et al. Towards a future where Alzheimer’s disease pathology is stopped before the onset of dementia. Nat Aging 2023; 3: 494–505.

Lissek

Suchan

. Preventing dementia? Interventional approaches in mild cognitive impairment. Neurosci Biobehav Rev 2021; 122: 143–164.

Duffner

Deckers

Cadar

, et al. The role of cognitive and social leisure activities in dementia risk: assessing longitudinal associations of modifiable and non-modifiable risk factors. Epidemiol Psychiatr Sci 2022; 31: e5.

Dong

Sun

, et al. Cognitive function assessed by Mini-Mental State Examination and risk of all-cause mortality: a community-based prospective cohort study. BMC Geriatr 2021; 21: 524.

10.

Nieuwenhuis-Mark

. The death knoll for the MMSE: has it outlived its purpose? J Geriatr Psychiatry Neurol 2010; 23: 151–157.

11.

Takahashi

Kawai

Suzuki

, et al. Development and validity of the computer-based cognitive assessment tool for intervention in community-dwelling older individuals. Geriatr Gerontol Int 2020; 20: 171–175.

12.

Takahashi

Kawai

Ejiri

, et al. Predicting the incidence of mild cognitive impairment with a computer-based cognitive assessment tool in community-dwelling older adults: the Otassha study. PLoS One 2024; 19: e0297433.

13.

Pan

Jordan-Marsh

. Internet use intention and adoption among Chinese older adults: from the expanded technology acceptance model perspective. Comput Hum Behav 2010; 26: 1111–1119.

14.

Ejiri

Kawai

Fujiwara

, et al. Social participation reduces isolation among Japanese older people in urban area: a 3-year longitudinal study. PLoS One 2019; 14: e0222887.

15.

Baek

Kim

Park

, et al. The validity and reliability of the mini-mental state examination-2 for detecting mild cognitive impairment and Alzheimer’s disease in a Korean population. PLoS One 2016; 11: e0163792.

16.

Kaufer

Williams

Braaten

, et al. Cognitive screening for dementia and mild cognitive impairment in assisted living: comparison of 3 tests. J Am Med Dir Assoc 2008; 9: 586–593.

17.

Bushnell

Johnston

Goldstein

. Retrospective assessment of initial stroke severity: comparison of the NIH Stroke Scale and the Canadian Neurological Scale. Stroke 2001; 32: 656–660.

18.

McMurray

Levy

Pang

, et al. Psychometric evaluation of a tablet-based tool to detect mild cognitive impairment in older adults: mixed methods study. J Med Internet Res 2024; 26: e56883.

19.

Makizako

Shimada

Park

, et al. Evaluation of multidimensional neurocognitive function using a tablet personal computer: test-retest reliability and validity in community-dwelling older adults. Geriatr Gerontol Int 2013; 13: 860–866.

20.

Manami

, et al. Validity and reliability of a computerized cognitive assessment tool ‘higher brain functional balancer’ for healthy elderly people. Jpn J Cogn Neurosci 2010; 12: 191–197.

21.

Hegarty

Waller

. Individual differences in spatial abilities. In: Shah

Miyake

(eds) The Cambridge handbook of visuospatial thinking. Cambridge: Cambridge University Press, 2005, pp.121–169.

22.

Pal

Biswas

Pandit

, et al. Study of visuospatial skill in patients with dementia. Ann Indian Acad Neurol 2016; 19: 83–88.

23.

Ratcliffe

Hale

McDonald

, et al. The Montreal Cognitive Assessment: norms and reliable change indices for standard and MoCA-22 administrations. Arch Clin Neuropsychol 2024; 39: 747–765.

24.

Ministry of Internal Affairs and Communications, Japan. Results of the Communication Usage Trend Survey of 2022, 2023. https://www.soumu.go.jp/main_sosiki/joho_tsusin/eng/pressrelease/2023/5/29_1.html .

25.

Nasreddine

Phillips

Bédirian

, et al. The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc 2005; 53: 695–699.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.08 MB