Usability and user experience impressions of older adults with cognitive impairment and people with schizophrenia towards GRADIOR,a cognitive rehabilitation program: A cross-sectional study

Abstract

Objective: The aim of this study was to evaluate and compare the impressions of older adults with mild dementia/MCI (mild cognitive impairment) and people with schizophrenia towards the usability of GRADIOR (version 4.5) and their user experience (UX) with this computerized cognitive rehabilitation program. Methods: The impressions towards the usability of GRADIOR and the UX of 41 older adults with mild dementia/MCI and 41 people with schizophrenia were obtained using the User Experience Questionnaire. Results: Older adults with dementia/MCI had more positive impressions than people with schizophrenia. Both agreed that its quality was lower in Dependability. Conclusion: GRADIOR meets users’ needs and preferences but needs improvements to ensure they feel more in control when interacting with it. For people with schizophrenia, other aspects of usability and UX need improvement. Usability and UX evaluation allow the verification of technological acceptability and functionality, and to identifying specific improvements for each user group.

Keywords

mild cognitive impairment dementia schizophrenia usability cognitive rehabilitation software

Introduction

Worldwide, dementia is one of the most important causes of disability and dependence among older people.¹ Dementia has psychological, physical, economic, and social consequences not only for those who are affected by it but also for their families, caregivers, and society in general.¹ Non-pharmacological interventions can delay or prevent the conversion from mild cognitive impairment (MCI) to dementia, helping at the same time to reduce the consumption and cost of medicine and institutionalization.^2,3

Schizophrenia is another serious mental disorder that places a burden on public health.⁴ Cognitive deficits in schizophrenia are one of the main causes of disabilities in everyday functioning,⁵ which makes them a major target for treatment; however, the effects of medication appear to be limited.⁶ Non-pharmacological interventions are gaining increasing interest and importance⁷ because it has been shown that they can also benefit people with schizophrenia.⁸

Non-pharmacological interventions include cognitive, physical, and psychosocial activities, electrical stimulation, a combination of the above, etc.^9,10 This article focuses on cognitive rehabilitation, which has proven its effectiveness in slowing the progression of dementia^11,12 and has promising positive effects on MCI,¹³ also promoting cognition in people with schizophrenia.⁸

While cognitive interventions typically have a traditional pencil-and-paper format and require face-to-face contact,¹⁴ technological innovation has driven them to take the form of computer-based programs, which are more cost-effective and can reach homebound people as well as those living in centers or far away.^13,14 Computer-based programs also make it possible to create personalized and flexible treatments that can be adjusted to individual conditions.^15,16 In addition, they provide real-time feedback and allow a more accurate recording of patients’ performance in each session.¹⁷ They also present tasks in a more attractive and fun way, increasing people’s motivation.¹⁶

Computers are one of the most widely used hardware devices for cognitive interventions¹⁸ because they facilitate user interaction.¹⁹ In terms of effectiveness, better outcomes in cognition, and even depression, have been obtained with computer-based cognitive interventions compared to noncomputerized interventions in people with dementia.²⁰ In older adults with MCI, there is evidence that computerized cognitive interventions improve cognitive functions such as learning and short-term memory, as well as behavioral symptoms.^21,22 Moreover, in people with schizophrenia, computer-based cognitive interventions have led to improvements in neuropsychological performance, quality of life, and self-esteem.²³ Hence, treating the cognitive area can bring benefits in other areas as well.

Regarding the sustainability of the effects of computerized cognitive interventions, significant small to moderate near-transfer effects have been found for older people with MCI,^22,24,25 but the far-transfer effects have been small or null.^24,25 In people with dementia, significant moderate effects were demonstrated in cognitive areas, but no significant effects were found in relation to daily living activities.²⁰ In people with schizophrenia, significant near-transfer effects were demonstrated²⁶; but in terms of far transfer, while gains were reported in certain tasks, others showed inconsistent effects.²⁷ Due to the methodological differences in the studies, several authors recommend further research to examine the retention of treatment effects,^20,22,23 and to redesign cognitive interventions to improve the skills needed to carry out daily activities.²⁴

GRADIOR is a computer-based cognitive assessment and rehabilitation program aimed at training and recovering cognitive functions in people with cognitive deficits/impairment (e.g., dementia, mental illness, etc.), as well as for cognitive stimulation in healthy subjects. One of the pioneer programs in Spain,¹⁷ its first version was released in 1997 and has continued to improve since then.²⁸ GRADIOR contains dynamic exercises to stimulate and rehabilitate different cognitive functions (perception, attention, orientation, etc.), and also has tools for neuropsychological assessment. In GRADIOR the user interacts with the touch screen or with the mouse following the instructions of the exercises. Each cognitive function represents a module (e.g., executive function module) that has sub-modalities (e.g., visual inhibition sub-modality within the executive function module) with different levels of difficulty. The exercises are adjusted to the needs of each user in terms of difficulty level, duration, and frequency. In addition, a randomization system avoids repetition of exercises.^17,28

GRADIOR version 4.5 includes the following improvements: a more intuitive user interface supported by a touch screen computer and written and audio instructions, new exercises with real images, and a renewed logo.^28,29 The next planned improvement is to adapt the program for use on tablets and smartphones.

Therapists commonly help users at the beginning until they become familiar with GRADIOR.²⁹ Advanced age, added to older people’s lack of skills/experience with technology, makes such help essential.³⁰ It is also considered important to help people with schizophrenia through the first sessions because they do not usually handle modern technology¹⁹ due to their economic situation, cognitive difficulties, and symptoms.³¹ In this regard, apathy is one of the main symptoms that frequently affects them.³² Therefore, an environment that supports users in learning to use technological tools is necessary.³³

It is also crucial that technologies are designed according to the needs, abilities, and preferences of users³⁴ so that they can use them easily, and autonomously and achieve the therapeutic objectives.¹⁸ In addition to usability, being able to provide a pleasant user experience (UX)¹⁸ is also essential for the acceptability of technological tools and to encourage users’ motivation and interest in using them over time.³⁵ That the technology is perceived as useful and can meet users’ expectations is another decisive aspect to ensure a positive UX.³⁵

Therefore, users’ feedback and perceptions towards technologies are essential for their proper development and improvement.³⁶ Although there are program designs that are based only on the opinion of physicians and researchers, which may differ from the opinion of the intended users of the program,³⁷ it is essential to involve end-users, despite the challenges posed by difficulties in their cognitive ability.³¹

In the case of GRADIOR, there are studies where its usability and the level of satisfaction in users with mental illness,¹⁹ elderly with cognitive impairment, and health professionals²⁹ have been assessed. GRADIOR’s usability has also been examined in subjects with and without cognitive impairment using the Long Lasting Memories program, which includes physical activities.³⁸ However, this study is the first to compare UX and GRADIOR’s usability in two groups of people with dependence and cognitive impairment of different etiology: elderly population with mild dementia/MCI and people with schizophrenia. Both make a large group of users whose impressions are crucial to improving the program in terms of usability and UX. As mentioned above, these two aspects are critical to the success and effectiveness of a program.^39,40

The objective of this study was to evaluate and compare the impressions of older adults with MCI/mild dementia and people with schizophrenia towards the usability of GRADIOR (version 4.5) and their UX with it. The findings will be useful to identify strengths and areas for improvement in the program to better adapt it according to users’ impressions. The results can also contribute to research and innovation in the design, development, and application of appropriately adapted technological cognitive interventions to improve healthcare for people with cognitive impairment and dependence.

Methods

Design

This is a cross-sectional study focused on evaluating and comparing the UX and GRADIOR’s usability in two groups of individuals affected by dependence and cognitive impairment of different etiology.

The present study, which is focused on usability and UX, was carried out in 2020 as part of a clinical trial [ISRCTN 15742788] that evaluated the efficacy of GRADIOR for cognitive rehabilitation in older subjects with MCI/mild dementia.⁴¹ An extension of the clinical trial was conducted in 2022 to evaluate the UX and usability of GRADIOR in people with schizophrenia (who were already using GRADIOR for therapeutic purposes) because they are a large group of users whose impressions are also important to improve the program.

Participants

Of the experimental group of the clinical trial recruited (43 older adults with MCI/mild dementia), two individuals could not participate in this study due to travel or medical intervention. The entire group of people with schizophrenia recruited (n = 41), who were already regular GRADIOR users, participated in the present study. Therefore, a total of 82 subjects participated. Recruitment, inclusion/exclusion criteria, and sociodemographic data (Table 1) are detailed below.

Table 1.

Participants’ sociodemographic characteristics.

Participants		Sex	Age (SD)	Educational level^a
Older adults (n = 41)	MCI	Men 10 (37%)	71.52	Read and write 2 (7.4%)
	MMSE = 25.22	Women 17 (63%)	±6.33	Primary school 18 (66.7%)
	SD±1.83	Women 17 (63%)	±6.33	Secondary school 7 (25.9%)
	MD	Men 5 (35.7%)	75.79	Read and write 1 (7.1%)
	MMSE = 22.93	Women 9 (64.3%)	±5.56	Primary school 12 (85.7%)
	SD±1.73	Women 9 (64.3%)	±5.56	Bachelor’s degree 1 (7.1%)
People with schizophrenia (n = 41)		Men 19 (46.3%)	51.63	Read and write 3 (7.3%)
		Women 22 (53.7%)	±11.66	Primary school 13 (31.7%)
				Secondary school 14 (34.1%)
				Certificate Program/Associate’s degree 6 (14.6%)
				Bachelor’s degree 5 (12.2%)

MCI = Mild Cognitive Impairment; MD = Mild Dementia; MMSE = Mini-Mental State Examination; n = Sample; SD = Standard Deviation.

^aDefinitions of each level of education (minimum to maximum).^42,43

Read and write = Have acquired and developed reading and writing skills in a selftaught manner, without the formal instruction of an educational system.

Primary School = Educational stage consisting of 6 academic years, typically completed between the ages of 6 and 12.

Secondary School = Educational stage consisting of 4 academic years, typically completed between the ages of 12 and 16 years old.

Certificate Program = Theoretical specialized course in a field. Its duration is between 6 months and 1 year.

Associate’s Degree = Theoretical and practical training in a professional field. Its duration is 2 academic years.

Bachelor’s Degree = University studies. It typically takes 4 academic years, although the duration may vary depending on the field.

People with MCI

This group was recruited from the Memory Workshops and the Memory Clinic of INTRAS Foundation (located in Zamora and Valladolid, Spain), and from Rio Hortega University Hospital (Valladolid, Spain).

Inclusion criteria were: a) being over 60 years old; b) having a diagnosis of MCI according to Petersen’s criteria,^44,45 which was verified by a psychiatrist of the research team; c) having a score between 26 and 27 points on the Mini-Mental State Examination (MMSE),⁴⁶ which was adjusted according to the participant’s age and educational level⁴⁷; d) having a score of ≤5 on the Geriatric Depression Scale (GDS)^48,49; e) willingness to voluntarily participate in the study; f) having a reference person/caregiver; and g) being able to read and write in Spanish.

Participants were also required to have been using GRADIOR (version 4.5) for 4 months, attending two or three 30-min sessions per week,. The mean of sessions attended over the 4 months was: 34.15 sessions (minimum = 18; maximum = 45; SD ± 8.01; 25^th percentile = 29; 75^th percentile = 41.5). Most of the participants were women with primary education level studies (Table 1).

Exclusion criteria were: a) severe auditory or visual alterations; b) severe physical comorbidity; c) meeting the Diagnostic and Statistical Manual for Mental Disorders (DSM-5) criteria for anxiety or depressive disorder; d) neurological disorders (Huntington’s disease, Parkinson’s disease, Lewy Body Dementia, stroke); e) substance dependence.

People with mild dementia

This group was recruited from the Memory Workshops and the Memory Clinic of INTRAS Foundation (located in Zamora and Valladolid, Spain), and from Rio Hortega University Hospital (Valladolid, Spain).

Inclusion criteria were: a) being over 60 years old; b) meeting the DSM-5 criteria for major neurocognitive disorder (which is the current term for dementia).⁵⁰ Their diagnoses were confirmed by a psychiatrist of the research team; c) having a score between 20 and 25 points on the MMSE,⁴⁶ which was adjusted according to the participant’s age and educational level⁴⁷; d) having a score of ≤5 on the GDS^48,49; e) willingness to voluntarily participate in the study; f) having a reference person/caregiver; and g) being able to read and write in Spanish.

Participants were also required to have 4 months of experience using GRADIOR version 4.5, attending two or three weekly 30-min sessions,. The mean of sessions attended over the 4 months was: 29.93 sessions (minimum = 24; maximum = 39; SD ± 5.37; 25^th percentile = 26.25; 75^th percentile = 31.75). Most of the sample were women with primary education level studies (Table 1).

People with schizophrenia

This group was recruited from the INTRAS Foundation Residence (Toro, Spain), and from the Psychosocial Rehabilitation Center (Zamora, Spain), which is also part of the INTRAS Foundation. Participants had already been diagnosed by a psychiatrist and met the DSM-5 criteria for schizophrenia.⁵⁰ In addition, they had to be 18 years of age or older and be able to read and write. It was also required that they be mentally stable during the study (i.e. not in the severe stages of schizophrenia), and they had to already be GRADIOR users with at least 4 months of experience using version 4.5.

This group of participants already used GRADIOR in their centers as part of their therapeutic activities. They had to attend two, three, or five sessions per week or as many as they wanted, each lasting 30 min. The mean of sessions attended in the 4 months prior to the study was: 36.92 sessions (minimum = 1; maximum = 89; SD ± 28.03; 25^th percentile = 14; 75^th percentile = 44.50). As previously mentioned, they could attend as many sessions as they wanted. Only sessions attended in the last 4 months prior to the study were collected. Most of the sample were women with secondary school education (Table 1).

Materials

The informed consent and the sociodemographic data form corresponding to the 4-months study protocol were used⁴¹ alongside the User Experience Questionnaire (UEQ), which allows a quick assessment of the usability (pragmatic) and UX (hedonic) aspects of interactive products,⁵¹ as well as of their users’ general impression (Attractiveness), which is influenced by the mentioned aspects.⁵² The validated Spanish pencil-and-paper version was used, which consists of 26 items grouped into six scales⁵¹ (Table 2). It can be downloaded free of charge from the official UEQ website: https://www.ueq-online.org/

Table 2.

UEQ scales and items.

Aspect	Scale	Item	Item Nº
General impression	Attractiveness	Annoying/enjoyable	1
		Good/bad	12
		Unlikable/pleasing	14
		Unpleasant/pleasant	16
		Attractive/unattractive	24
		Friendly/unfriendly	25
Usability	Efficiency	Fast/slow	9
		Inefficient/efficient	20
		Impractical/practical	22
		Organized/cluttered	23
	Perspicuity	Not understandable/ understandable	2
		Easy to learn/difficult to learn	4
		Complicated/easy	13
		Clear/confusing	21
	Dependability	Unpredictable/predictable	8
		Obstructive/supportive	11
		Secure/not secure	17
		Meets expectations/does not meet expectations	19
User experience	Stimulation	Valuable/inferior	5
		Boring/exciting	6
		Not interesactting/interesting	7
		Motivating/demotivating	18
	Novelty	Creative/dull	3
		Inventive/conventional	10
		Usual/leading edge	15
		Conservative/innovative	26

In a random order, half of the UEQ items start with a positive term and the other half with a negative one. Participants state their level of agreement with the positive or negative term on a seven-point Likert scale.⁵² The UEQ items are scaled from −3 (most negative answer) to +3 (most positive answer), with 0 representing a neutral answer.⁵¹ Values above 0 imply a positive user impression and values below 0 a negative one.⁵³ The most extreme values are infrequently observed, so values close to +2 represent a very positive, close to optimal, impression.⁵¹

The UEQ data analysis tool version 10 has a benchmark data set, which contains 21,175 responses from 468 studies that have used the UEQ to evaluate different products. From the responses of those studies, the creators of the UEQ obtained the mean for each UEQ scale and established intervals in the benchmark data set, corresponding to the following five classification categories⁵³:

• Excellent: the evaluated product is in the range of the 10% best results in the benchmark.

• Good: 10% of the results in the benchmark are better than the result for the evaluated product, while 75% of the benchmark results are worse.

• Above average: 25% of the results in the benchmark are better than result for the evaluated product, and 50% of the results are worse.

• Below average: 50% of the results in the benchmark are better than the result for the evaluated product, while 25% of the results are worse.

• Bad: the evaluated product is in the range of the 25% worst results in the benchmark.

In the UEQ data analysis tool, participants’ answers are entered, and the tool automatically calculates the mean values for each scale of the evaluated product, and compares them to the existing values from the benchmark data set. This makes it possible to further compare the quality of the evaluated product, as well as to identify its strengths and areas for improvement.⁵³

Procedure

The ethical approval for this study was obtained from the Provincial Hospital of Zamora (Number: 387- E.C.). According to Rauschenberger et al.,⁵¹ 30 participants are sufficient to obtain a valid impression on the UEQ; in other words, 30 subjects are the minimum sample size needed for this purpose. Hence, a sample of ≥30 participants per group was collected.

Participants were informed about the purpose of the study, the confidentiality of the data, and the voluntary nature of their participation. Once they had agreed to sign the written informed consent form, their sociodemographic data were obtained and, finally, the UEQ was applied. A member of the research team clarified the participants’ doubts where necessary and ensured that they completed the documents so that there were no missing data. All documents were completed individually in an accessible room. The material was kept in the centers, in secure filing cabinets that only members of the research team had access to.

The participants’ sociodemographic data and the UEQ data were analyzed as described in the following section.

Statistical analysis

All participants completed a sociodemographic data form and the UEQ. Details on the data analysis are provided below.

The SPSS statistical program (version 25) was used to analyze participants’ sociodemographic data: sample size, percentages, means, standard deviations, maximum, minimum, and percentiles.

The UEQ Excel tool for data analysis (version 10) and for product comparison (version 4) were used to analyze the UEQ data (scale means, standard deviations, confidence intervals,t test) and compare the impressions of both groups of participants. The UEQ Excel tools were also used to compare the impressions of both groups of participants with the Bechmark data set contained in these tools.

Results

People with MCI and those with mild dementia were in their late adulthood^54,55 (mean age 71.52 and 75.79, respectively), while subjects with schizophrenia were in their middle adulthood^54,55 (mean age 51.63), as shown in Table 1. The similarity in age of the MCI and the mild dementia individuals led to the decision to consider them as a single group (older adults), while people with schizophrenia formed another group.

A member of the research team verified that participants fully completed every document so that there were no missing data.

UEQ

Overall results

Figure 1 shows the impressions of both groups of participants towards GRADIOR on each of the UEQ scales. In the t test, an Alpha level of 0.05 was used.

Figure 1.

Participants’ impressions of GRADIOR on the UEQ scales.

On the Attractiveness scale, older adults with cognitive impairment had a very positive impression of GRADIOR (2.56), while people with schizophrenia had a positive impression of it (1.43). The t test yielded a significant difference (p = .0000), which means that, as regards the Attractiveness scale, what older people with MCI/mild dementia liked the most was the program’s enjoyable and pleasing items.

On the Perspicuity scale, the impression of older adults with cognitive impairment towards the program was also very positive (2.09), while the impressions of the schizophrenia group were just positive (1.32). The significant difference yielded by the t test (p = .0001) means that older adults with MCI/mild dementia found it easier to become familiar with the program, understand it, and learn how to use it.

Regarding the Efficiency scale, once again, older adults with cognitive impairment had a very positive impression of GRADIOR (2.00), while the impression of people with schizophrenia was just positive (1.00). The t test revealed a significant difference (p = .0000), showing that older people with MCI/mild dementia considered GRADIOR’s interface more organized, faster, and appropriate for them to do the exercises without unnecessary effort.

On the Dependability scale, although the positive value attributed to the program by older adults with cognitive impairment was lower (1.63), it was still a higher positive score than the one given by participants with schizophrenia (0.90). The t test yielded a significant difference (p = .0002), which implies that older adults with MCI/mild dementia felt more confident and in control when interacting with the program.

Regarding the Stimulation scale, older adults with cognitive impairment showed a very positive impression of GRADIOR (2.46), while the impression of people with schizophrenia was just positive (1.13). The significant difference found in the t test (p = .0000) means that people with MCI/mild dementia felt more motivated, and excited, also showing a greater interest in using GRADIOR.

On the Novelty scale, the positive value attributed to the program by older adults with cognitive impairment was lower (1.73), although it was still a higher positive score than the one given by participants with schizophrenia (0.79). A significant difference was found in the t test (p = .0001), showing that older adults with MCI/mild dementia perceived the program’s design as more creative, innovative, and better able to capture their attention.

The above meanings of the scales were obtained from Rauschenberger et al.,⁵¹ and Schrepp et al.⁵²

Results per item

Figure 2 shows the mean of each of the scales’ items. This detailed analysis reveals which specific aspects of each scale are a strength, which can be improved, and which items might be difficult for participants to interpret. GRADIOR obtained positive values from both groups of participants (older adults with cognitive impairment between 1.3 and 2.7 and people with schizophrenia between 0.2 and 2), except in item 8, which achieved negative values from both groups.

Figure 2.

Participants’ impressions of GRADIOR on each UEQ item.

On the Attractiveness scale, the item Good achieved the highest mean in both groups of participants. Older adults with MCI/mild dementia gave the lowest positive score to the Pleasing and Friendly items, while people with schizophrenia did so to the Pleasing and Attractive items.

On the Perspicuity scale, the item that older adults with cognitive impairment rated the highest was Clear, while participants with schizophrenia gave the highest score to Understandable. Both groups agreed that the item with the lowest positive score was Easy to use.

On the Efficiency scale, the items most highly rated by the elderly with MCI/mild dementia were Efficient and Organized, and people with schizophrenia gave the highest score to Efficient. Both groups of participants rated the item Fast with the lowest positive score.

On the Dependability scale, older adults with cognitive impairment placed the highest value on the Secure and Meets Expectations items, while participants with schizophrenia did so on Supportive. Item 8 (Predictable) was the only one with a negative mean on this scale in both groups of participants, which suggests that they could have misinterpreted it, reading it out of context if they could predict the program’s behavior.

On the Stimulation scale, the item that achieved the highest scores from older adults with MCI/mild dementia was Interesting, and the lowest positive item was Exciting. People with schizophrenia gave the highest positive score to Interesting, and the lowest positive score to Motivating.

On the Novelty scale, the Creative item achieved the highest mean whereas Inventive showed the lowest positive mean in both groups.

Data benchmarking

In most of the scales, older adults with MCI/mild dementia categorized GRADIOR as excellent in the following order: Attractiveness, Stimulation, Perspicuity, Efficiency, and Novelty (Figure 3). This means that GRADIOR is in the range of the 10% best results in the benchmark in terms of being enjoyable, interesting, easy to learn and use, as well as in its ability to engage users’ attention. However, on the Dependability scale, older people categorized GRADIOR as good (Figure 3), which means that 10% of the results in the benchmark are better than the results of GRADIOR in terms of the confidence and control perceived by its users when interacting with it.

Figure 3.

Older adults with cognitive impairment’s impressions of GRADIOR with the benchmark. Legend: The black line represents the results of GRADIOR 4.5. The colored bars represent the different ranges of the mean values of each scale contained in the Benchmark.

On the other hand, the group of people with schizophrenia categorized GRADIOR as above average in most of the scales, in the following order: Attractiveness, Perspicuity, Stimulation, and Novelty (Figure 4). This means that 25% of the benchmark results are above the results for GRADIOR in terms of being enjoyable, easy to learn, interesting, and capturing users’ attention. Conversely, they considered that GRADIOR was below average in Efficiency and Dependability (Figure 4), which means that 50% of the benchmark results are better than the results for GRADIOR in terms of organization, speed of the program, confidence, and control perceived by users when interacting with it.

Figure 4.

People with schizophrenia’s impressions of GRADIOR with the benchmark. Legend: The black line represents the results of GRADIOR 4.5. The colored bars represent the different ranges of the mean values of each scale contained in the Benchmark.

Scales with inconsistent answers

In the UEQ, inconsistent answers are defined as those where there is a difference >3 between the best and worst items on a scale. If this difference appears in two or more scales, it indicates that participants have probably given random/not serious answers and/or misinterpreted items. This occurred in the present study as detailed below.

In the group of older adults with MCI/mild dementia, 12 (29.3%) participants showed inconsistencies in their responses on 2, 3, and/or four scales: 10 of them answered inconsistently on the Novelty scale, six on the Dependability scale, five on the Perspicuity scale, five on the Efficiency scale, and one on the Stimulation scale.

On the other hand, 14 (34.2%) participants from the group of people with schizophrenia presented inconsistencies in their responses on 2, 3, 4, and/or five scales: 11 of them answered inconsistently on the Perspicuity scale, nine on the Dependability scale, nine on the Novelty scale, eight on the Attractiveness scale, seven on the Efficiency scale, and five on the Stimulation scale.

Discussion

The purpose of this study was to assess and compare the impressions of older adults with MCI/mild dementia and people with schizophrenia regarding the usability of GRADIOR (version 4.5) and their UX using the UEQ. The results provide important cues not only to improve GRADIOR but also to develop and improve other computer-based cognitive rehabilitation programs so that they can be appropriately adapted to these populations.

In this study, the impressions towards GRADIOR were very positive and its quality was categorized as excellent in Attractiveness by older adults with cognitive impairment. Another study reports that older people with cognitive impairment found GRADIOR attractive and enjoyable.²⁹ Besides, according to a further research project, the pleasure perceived by the elderly when using technology is a determining factor in their intention to use it,⁵⁶ which is also reflected in the present study, as most of the older adults with cognitive impairment completed the 4-month period of using the program. Thus, it seems that finding the software attractive counterbalances the digital gap and older people’s fear of not being able to cope with technology.³⁰

As for the participants with schizophrenia, they showed positive impressions towards GRADIOR in Attractiveness, considering its quality as above average. Nonetheless, their impressions indicate that the program can be improved. Indeed, in another study, people with mental illness suggested making GRADIOR more enjoyable.¹⁹ Van Der Krieke et al.⁵⁷ found that many people with schizophrenia believed that technological tools could be made more attractive and pleasing by using more colors, images, and multimedia elements, while others preferred simplicity. This led to the conclusion that the interface should be customizable. In any case, it should be mentioned that negative symptoms such as apathy are a core and frequent component of schizophrenia, and prominent negative symptoms might affect up to 60% of patients. In fact, some of them experience a considerable number of negative symptoms, despite also having positive symptoms.^32,58,59 Therefore, apathy could explain the lower acceptance of GRADIOR among this group. Thus, it could be useful to spend more time explaining the purpose of the software to people with schizophrenia and, even, to develop special features for them, considering that the cause of their cognitive impairment is different from that of the other group.

Concerning Perspicuity, the positive outcomes found in this study are supported by other research in both older adults with MCI/mild dementia^29,38 and people with schizophrenia.¹⁹ This means that GRADIOR looks easy to learn and use, and that its instructions are clear enough and understandable. However, the fact that, in this study, participants with schizophrenia produced lower positive impressions than older adults with cognitive impairment indicates that the system needs to be improved to guide their learning and use to a greater extent. Nevertheless, therapists often help users at the beginning to become familiar with GRADIOR.²⁹ Indeed, providing an environment that helps users to learn and use technology is important,³³ because it makes them feel comfortable with technological tools that they have little experience in using.³⁷

Regarding Efficiency, according to the impressions of older adults with cognitive impairment, GRADIOR worked properly and its structure was not cluttered. The simplicity of GRADIOR’s interface is not only supported by Irazoki et al.,²⁹ but another study also agrees that simplicity helps older people to stay focused, minimizing the risk of mistakes, and not burdening their memory.⁶⁰ However, Irazoki et al.²⁹ suggested that the sensitivity of the touch screen should be improved to quickly capture the responses of GRADIOR users. This recommendation is in line with the present study because, in this group, the Fast item received the lowest positive score on the Efficiency scale. Although the touch screen has proven to be an easy input device, its surface must be sufficiently touch-sensitive to support usability, quality of use, and user productivity.⁶¹ Therefore, the sensitivity of the GRADIOR touch screen should be improved.

On the other hand, the impression towards GRADIOR of participants with schizophrenia was positive on the Efficiency scale. This finding is consistent with another study in which users with schizophrenia assessed the functionality and efficiency of GRADIOR as good, although they felt that there was too much content in the exercises.¹⁹ This observation may shed light on what could have led the participants with schizophrenia in this study to rate GRADIOR as below average on this scale, and why the organization of the exercise’s contents should be improved for this population. Another aspect that should be addressed is interface response speed because this group gave the Fast item the lowest positive score.

In terms of Stimulation, the impressions towards GRADIOR of older adults with MCI/mild dementia are very positive and they considered its quality to be excellent. Other studies report that older adults with cognitive impairment thought that GRADIOR was fun, motivating, and entertaining.²⁹ Participants with schizophrenia in the present study also perceived the program as motivating, which is consistent with the findings stated in another research project.¹⁹ However, in the present study, people with schizophrenia considered the quality of GRADIOR to be above average on the Stimulation scale, which indicates that it can be improved. According to Lumsden et al.,⁶² incorporating elements of gamification into cognitive exercises can improve the engagement of people with psychiatric disorders. Such elements could be an avatar, interactive characters, performance bonuses, scenario changes in response to their actions, etc. These elements could be explored for GRADIOR and developed in the near future.

In Novelty, impressions towards GRADIOR were positive in both groups of participants. This finding is consistent with other investigations on this program^19,29 that highlight the importance of colors, images, and visual elements to capture users’ attention. Other studies claim that colors⁶³ and familiar images of the environment⁶⁴ contribute to user satisfaction. In this regard, real images were precisely one of the improvements introduced in GRADIOR 4.5. However, the fact that older adults with cognitive impairment gave this category a lower positive score while participants with schizophrenia considered it to be above average indicates that it can be further enhanced. Inventiveness could be an area for improvement, as this item received the lowest positive score on this scale in both groups. Although GRADIOR 4.5 had also been improved through the inclusion of new exercises, the above finding suggests that the novelty effect could diminish after continued use of the program. Maggio & Calabrò⁶⁵ suggest that virtual reality is an innovative method for cognitive rehabilitation due to its multisensory interfaces. Thus, using virtual reality for GRADIOR’s exercises might improve the mentioned scores.

Regarding Dependability, both groups of participants produced positive impressions towards GRADIOR, which is in line with another investigation in which the program met the expectations of users with schizophrenia.¹⁹ According to a different study, a platform that included GRADIOR and physical exercises met the expectations of older adults with cognitive impairment and made them feel comfortable using it.¹⁶ However, GRADIOR obtained lower positive scores from both groups of participants in this study on the quality scale than on the others, which suggests the need for improvement. Indeed, Irazoki et al.²⁹ concluded that older adults should use GRADIOR software and hardware more autonomously. In this sense, a very intuitive design is essential for them to control technological features.⁶⁶ In the case of people with schizophrenia, an investigation proved that some users had difficulty interacting with different elements of GRADIOR at the same time.¹⁹ Therefore, a simpler design could be a targeted improvement for this population. In fact, it has been found that including elements, even if they are motivational, can cause stress or trigger new cognitive demands when interacting with a program.⁶⁷

On the other hand, in this study, there were scales where inconsistent responses were given. This was expected, as participants suffer from cognitive difficulties that affect their functioning and accuracy.^68,69 Even so, such inconsistencies represented a small percentage of the sample in both groups. Another finding was difficulty in understanding the Predictable item in the Dependability context. Nonetheless, even information technology students might misunderstand or misinterpret certain UEQ items.⁵¹

Although sample size could be regarded as one of the limitations of the study, 30 participants are considered sufficient to obtain a valid impression and approach on the UEQ.⁵¹ Secondly, this is a cross-sectional study and, consequently, participants had not used GRADIOR for the same length of time; for instance, schizophrenic patients had been using it for 4 months or more. Nonetheless, participants’ results were consistently positive, which demonstrates a trend in their impressions of the program.

Another limitation is that certain participants may have felt committed to respond favorably on the UEQ because they were in the INTRAS Foundation centers, GRADIOR’s main developer and promoter. Efforts were made to prevent this bias in the present study through the UEQ instructions, which explained to participants that they should select responses that best reflected their impressions of GRADIOR, emphasizing that there were no right or wrong answers, and that only their personal opinions mattered. Additionally, the informed consent form clarified that participation in the study was voluntary and that they were free to withdraw at any time without any consequences. In addition, other studies support the results obtained.

Another potential bias in the results could have been that it only assessed participants that had engaged with the program for 4 months or more prior to providing their feedback. However, in the design of the study it was determined to obtain feedback after that time range so that the participants could have a real and daily interaction with the program on which to base their feedback, identifying problems which could be unnoticed or be inaccurate in a demonstration mode or from initial use. Nevertheless, studies have found that pragmatic (usability) attributes change as the technology is used, while hedonic (UX) attributes appear to be more stable over time.⁷⁰ Therefore, further studies to explore these aspects are needed.

For future progress, more studies on usability and UX in older adults with cognitive impairment and people with schizophrenia are needed, not only using quantitative but also qualitative approaches, to continue improving cognitive rehabilitation technologies according to their impressions. Furthermore, comparing their impressions as users with those of professionals, caregivers, and other user groups would help to identify other potential barriers and propose solutions.

Conclusions

This study is the first to compare the UX and usability of GRADIOR in two groups with dependence and cognitive impairment of different etiology: older adults with MCI/mild dementia and people with schizophrenia. The positive impressions collected from the participants suggest that GRADIOR is suitably aimed at satisfying their needs and preferences, especially in the case of older adults with MCI/mild dementia, who gave higher positive scores. However, GRADIOR needs to be improved to make users feel more in control when interacting with it. Other aspects of usability and UX also need to be improved for people with schizophrenia. Formally, GRADIOR was designed for cognitive impairment, regardless of its cause. This study shows the need to develop special features for different pathologies. Although cognitive impairment may seem similar, the approach required for each group of patients is different, revealing the need to develop customizable features according to each pathology.

Due to their cognitive impairment/deficits, obtaining the impressions of technological tools of these population groups can be challenging but it is, indeed, necessary. Currently, a suitable approach would be to apply simple questionnaires such as the UEQ. Involving users in the process of improving programs is essential to provide them with better experiences, encouraging the use and adherence to cognitive technology interventions. The findings of this study can help in the design and development of customizable, acceptable, and functional technological cognitive intervention programs aimed at improving healthcare for people with cognitive impairment and dependence.

Footnotes

Author Notes

The following author moved to a new institution since completing the research:

• Susana Gil-Martínez: Educational Guidance Counselor at the Junta de Castilla y León, Zamora, Spain.

The following authors have added a new affiliation since completing the research:

• José Miguel Toribio-Guzmán: Institute for Biomedical Research of Salamanca (IBSAL), University of Salamanca, Salamanca, Spain.

• Esther Parra-Vidales: QUARERE, S.L. Valladolid, Spain.

The following authors no longer have the following affiliations since completing the research:

• José Miguel Toribio-Guzmán: Department of Research and Development, Iberian Institute of Research in Psycho-Sciences, INTRAS Foundation, Zamora, Spain.

• Leslie María Contreras-Somoza: Psycho-Sciences Research Group, Institute for Biomedical Research of Salamanca (IBSAL), University of Salamanca, Salamanca, Spain.

Acknowledgements

Thanks to the INTRAS Foundation for its collaboration in the recruitment of the sample.

Author contributions

Conceptualization, M.Á.F.-M.; Methodology, L.M.C.-S., E.I. and J.M.T.-G.; Formal analysis, L.M.C.-S. and J.M.T.-G.; Investigation, L.M.C.-S., E.I. and M.J.V.-R.; Resources, J.M.T.-G, M.J.V.-R., S.G.-M., M.C.-A., E.L.C. and E.P.-V.; Writing original draft, L.M.C.-S.; Writing review and editing, L.M.C.-S., M.Á.F.-M., J.M.T.-G. and M.V.P.-B.; Visualization, L.M.C.-S. and J.M.T.-G.; Supervision, M.Á.F.-M. and J.M.T.-G.; Project administration, M.Á.F.-M. and J.M.T.-G. All authors have read and agreed to the final version of the manuscript.

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The authors J.M.T.-G., M.J.V.-R., S.G.-M., M.C.-A., E.L.C. and M.Á.F.-M. were/are linked to INTRAS Foundation, a non-profit entity, which is the main developer and promoter of GRADIOR, a cognitive rehabilitation software.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethical statement

ORCID iDs

Leslie María Contreras-Somoza

José Miguel Toribio-Guzmán

María Victoria Perea-Bartolomé

Data Availability Statement

The data that support the findings of this study are available upon reasonable request from the corresponding author.

References

World Health Organization [WHO] . Dementia 2022. https://www.who.int/news-room/fact-sheets/detail/dementia (accessed 15 October 2022).

Steinbeisser

Schwarzkopf

Graessel

, et al. Cost-effectiveness of a non-pharmacological treatment vs. “care as usual” in day care centers for community-dwelling older people with cognitive impairment: results from the German randomized controlled DeTaMAKS-trial. Eur J Heal Econ 2020; 21: 825–844.

Shao

Zhang

, et al. Dose-response relationship in non-pharmacological interventions for individuals with mild cognitive impairment: a systematic review and meta-analysis of randomised controlled trials. J Clin Nurs 2022; 1: 3390–3401.

Patel

. Why mental health matters to global health. Transcult Psychiatry 2014; 51: 777–789.

Harvey

Strassnig

Silberstein

. Prediction of disability in schizophrenia: symptoms, cognition, and self-assessment. J Exp Psychopathol 2019; 10: :73–79. DOI: 10.1177/2043808719865693.

Tripathi

Kar

Shukla

. Cognitive deficits in schizophrenia: understanding the biological correlates and remediation strategies. Clin Psychopharmacol Neurosci 2018; 16: 7–17.

Stevović

Repišti

Radojičić

, et al. Non-pharmacological treatments for schizophrenia in Southeast Europe: an expert survey. Int J Soc Psychiatry 2022; 68: 1141–1150.

Samochowiec

Szulc

Bieńkowski

, et al. Polish Psychiatric Association consensus statement on non-pharmacological methods in the treatment of negative symptoms of schizophrenia. Psychiatr Pol 2021; 55: 719–742.

Lissek

Suchan

. Preventing dementia? Interventional approaches in mild cognitive impairment. Neurosci Biobehav Rev 2021; 122: 143–164.

10.

Hegde

Kelly

Guimond

, et al. Impact of non-pharmacological interventions on brain structure and function in schizophrenia. In: Neuroimaging in schizophrenia. Berlin, Germany: Springer, 2020: 385–409.

11.

Amieva

Robert

Grandoulier

, et al. Group and individual cognitive therapies in Alzheimer’s disease: the ETNA3 randomized trial. Int Psychogeriatr 2016; 28: 707–717.

12.

Diaz Baquero

Perea Bartolomé

Toribio-Guzmán

, et al. Determinants of adherence to a “GRADIOR” computer-based cognitive training program in people with mild cognitive impairment (MCI) and mild dementia. J Clin Med 2022; 11: 1714–1718.

13.

Zhu

, et al. Technology-based cognitive training and rehabilitation interventions for individuals with mild cognitive impairment: a systematic review. BMC Geriatr 2018; 18: 213. DOI: 10.1186/s12877-018-0893-1.

14.

Kueider

Parisi

Gross

, et al. Computerized cognitive training with older adults: a systematic review. PLoS One 2012; 7: e40588.

15.

Irazoki

Contreras-Somoza

Toribio-Guzmán

, et al. Technologies for cognitive training and cognitive rehabilitation for people with mild cognitive impairment and dementia. A systematic review. Front Psychol 2020; 11: 648–715.

16.

González-Palau

Franco

Toribio

, et al. Designing a computer-based rehabilitation solution for older adults: the importance of testing usability. PsychNology J 2013; 11: 119–136.

17.

Toribio-Guzmán

Parra Vidales

Viñas Rodríguez

, et al. Rehabilitación Cognitiva por Ordenador en Personas Mayores: Programa GRADIOR. Aula 2018; 24: 61–75.

18.

Contreras-Somoza

Irazoki

Toribio-Guzmán

, et al. Usability and user experience (UX) of cognitive intervention technologies for elderly people with mild cognitive impairment (MCI) or dementia. A systematic review. Front Psychol 2021; 12: 1–15.

19.

Góngora Alonso

Fumero Vargas

Morón Nozaleda

, et al. Usability analysis of a system for cognitive rehabilitation, ‘Gradior’, in a Spanish Region. Telemed e-Health 2020; 26: 671–682.

20.

García-Casal

Loizeau

Csipke

, et al. Computer-based cognitive interventions for people living with dementia: a systematic literature review and meta-analysis. Aging Ment Heal 2017; 21: 454–467.

21.

Klimova

Maresova

. Computer-based training programs for older people with mild cognitive impairment and/or dementia. Front Hum Neurosci 2017; 11: 262. DOI: 10.3389/fnhum.2017.00262.

22.

Zhang

Huntley

Bhome

, et al. Effect of computerised cognitive training on cognitive outcomes in mild cognitive impairment: a systematic review and meta-analysis. BMJ Open 2019; 9: e027062. DOI: 10.1136/bmjopen-2018-027062.

23.

Garrido

Barrios

Penadés

, et al. Computer-assisted cognitive remediation therapy: cognition, self-esteem and quality of life in schizophrenia. Schizophr Res 2013; 150: 563–569.

24.

Nguyen

Murphy

Andrews

. A game a day keeps cognitive decline away? A systematic review and meta-analysis of commercially-available brain training programs in healthy and cognitively impaired older adults. Neuropsychol Rev 2022; 32: 601–630.

25.

Sala

Aksayli

Tatlidil

, et al. Near and far transfer in cognitive training: a second-order meta-analysis. Collabra Psychol 2019; 5: 1–22.

26.

Zhu

Fan

Zou

, et al. Computerized or manual? Long term effects of cognitive remediation on schizophrenia. Schizophr Res 2022; 239: 47–54.

27.

Cassetta

Tomfohr-Madsen

Goghari

. A randomized controlled trial of working memory and processing speed training in schizophrenia. Psychol Med 2019; 49: 2009–2019.

28.

Franco-Martín

MÁ

Diaz-Baquero

Bueno-Aguado

, et al. Computer-based cognitive rehabilitation program GRADIOR for mild dementia and mild cognitive impairment: new features. BMC Med Inform Decis Mak 2020; 20: 274–315.

29.

Irazoki

Sánchez-Gómez

Contreras-Somoza

, et al. A qualitative study of the cognitive rehabilitation program GRADIOR for people with cognitive impairment: outcomes of the focus group methodology. J Clin Med 2021; 10: 859.

30.

Pirhonen

Lolich

Tuominen

, et al. “These devices have not been made for older people’s needs” – older adults’ perceptions of digital technologies in Finland and Ireland. Technol Soc 2020; 62: 101287.

31.

Lindberg

Jormfeldt

Bergquist

. Unlocking design potential: design with people diagnosed with schizophrenia. Inform Health Soc Care 2019; 44: 31–47.

32.

Raffard

Bortolon

Yazbek

, et al. The cognitive, affective motivational and clinical longitudinal determinants of apathy in schizophrenia. Eur Arch Psychiatry Clin Neurosci 2019; 269: 911–920.

33.

Chiu

Lin

, et al. The attitudes, impact, and learning needs of older adults using apps on touchscreen mobile devices: results from a pilot study. Comput Human Behav 2016; 63: 189–197.

34.

Contreras-Somoza

Irazoki

Castilla

, et al. Study on the acceptability of an ICT platform for older adults with mild cognitive impairment. J Med Syst 2020; 44: 120–212.

35.

Biduski

Bellei

Rodriguez

JPM

, et al. Assessing long-term user experience on a mobile health application through an in-app embedded conversation-based questionnaire. Comput Human Behav 2020; 104: 106169. DOI: 10.1016/j.chb.2019.106169.

36.

Contreras-Somoza

Toribio-Guzmán

Sánchez-Gómez

, et al. Perceptions of older people with cognitive impairment , caregivers , and professionals about ehcoBUTLER (tablet health care platform): a qualitative focus group study. Int J Environ Res Public Health 2022; 19: 6761–6823.

37.

Reeder

Pile

Crawford

, et al. The feasibility and acceptability to service users of CIRCuiTS, a computerized cognitive remediation therapy programme for schizophrenia. Behav Cogn Psychother 2016; 44: 288–305.

38.

Góngora Alonso

Toribio Guzmán

Sainz de Abajo

, et al. Usability evaluation of the eHealth Long Lasting Memories program in Spanish elderly people. Health Informatics J 2020; 26: 1728–1741.

39.

Brade

Lorenz

Busch

, et al. Being there again – presence in real and virtual environments and its relation to usability and user experience using a mobile navigation task. Int J Hum Comput Stud 2017; 101: 76–87.

40.

Lau

Bartle-Clar

Bliss

, et al. Improving usability, safety and patient outcomes with health information technology. In: From research to practice. Amsterdam: IOS Press, 2019.

41.

Vanova

Irazoki

García-Casal

, et al. The effectiveness of ICT-based neurocognitive and psychosocial rehabilitation programmes in people with mild dementia and mild cognitive impairment using GRADIOR and ehcoBUTLER: study protocol for a randomised controlled trial. Trials 2018; 19: 100–115.

42.

Ministerio de Educación Formación Profesión y Deportes. Enseñanzas. Educagob 2024, https://educagob.educacionfpydeportes.gob.es/ensenanzas.html (accessed 5 July 2024).

43.

Gobierno de España . Organización del sistema educativo. Administracion.gob.es 2024. https://administracion.gob.es/pag_Home/Tu-espacio-europeo/derechos-obligaciones/ciudadanos/educacion/sistema-educativo/organizacion.html (accessed 5 July 2024).

44.

Petersen

Caracciolo

Brayne

, et al. Mild cognitive impairment: a concept in evolution. J Intern Med 2014; 275: 214–228.

45.

Petersen

Smith

Waring

, et al. Mild cognitive impairment: clinical characterization and outcome. Arch Neurol 1999; 56: 303–308.

46.

Arevalo-Rodriguez

Smailagic

Roqué

, et al. Mini-Mental State Examination (MMSE) for the detection of Alzheimer’s disease and other dementias in people with mild cognitive impairment (MCI). Cochrane Database Syst Rev 2015; 2015: CD010783. DOI: 10.1002/14651858.CD010783.pub2.

47.

Blesa

Pujol

Aguilar

, et al. Clinical validity of the ‘mini-mental state’ for Spanish speaking communities. Neuropsychologia 2001; 39: 1150–1157.

48.

Martínez de la Iglesia

Onís Vilches

Dueñas Herrero

, et al. Versión española del cuestionario de Yesavage abreviado (GDS) para el despistaje de depresión en mayores de 65 años: adaptación y validación. Medifam 2002; 12: 620–630.

49.

Yesavage

Sheikh

. Geriatric Depression Scale (GDS): recent evidence and development of a shorter version. Clin Gerontol 1986; 5: 165–173.

50.

American Psychiatric Association . Diagnostic and statistical manual of mental disorders. DSM-55th ed. Arlington, VA: American Psychiatric Association, 2013.

51.

Rauschenberger

Schrepp

Perez-Cota

, et al. Efficient measurement of the user experience of interactive products. How to use the user experience questionnaire (UEQ).Example: Spanish language version. Int J Interact Multimed Artif Intell 2013; 2: 39–45.

52.

Schrepp

Hinderks

Thomaschewski

. Design and evaluation of a short version of the user experience questionnaire (UEQ-S). Int J Interact Multimed Artif Intell 2017; 4: 103–108.

53.

Schrepp

Hinderks

Thomaschewski

. Construction of a benchmark for the user experience questionnaire (UEQ). Int J Interact Multimed Artif Intell 2017; 4: 40–44.

54.

Beyer

Lazzara

. Psychology through the lifespan. Canvas Commons, 2020.

55.

Medley

. Life satisfaction across four stages of adult life. Int J Aging Hum Dev 1980; 11: 193–209.

56.

Macedo

. Predicting the acceptance and use of information and communication technology by older adults: an empirical examination of the revised UTAUT2. Comput Human Behav 2017; 75: 935–948.

57.

Van Der Krieke

Emerencia

Aiello

, et al. Usability evaluation of a web-based support system for people with a schizophrenia diagnosis. J Med Internet Res 2012; 14: e24. DOI: 10.2196/jmir.1921.

58.

Correll

Schooler

. Negative symptoms in schizophrenia: a review and clinical guide for recognition, assessment, and treatment. Neuropsychiatr Dis Treat 2020; 16: 519–534.

59.

Krause

Zhu

Huhn

, et al. Antipsychotic drugs for patients with schizophrenia and predominant or prominent negative symptoms: a systematic review and meta-analysis. Eur Arch Psychiatry Clin Neurosci 2018; 268: 625–639.

60.

Johnson

Finn

. Designing user interfaces for an aging population: towards universal design. Cambridge: Morgan Kaufmann, 2017.

61.

Chang

Tsai

Chang

, et al. Touch panel usability of elderly and children. Comput Human Behav 2014; 37: 258–269.

62.

Lumsden

Edwards

Lawrence

, et al. Gamification of cognitive assessment and cognitive training: a systematic review of applications and efficacy. JMIR Serious Games 2016; 4: e11. DOI: 10.2196/games.5888.

63.

Punchoojit

Hongwarittorrn

. Usability studies on mobile user interface design patterns: a systematic literature review. Adv Human-Computer Interact 2017; 2017: 1–22.

64.

Mol

Duque

Fonseca

, et al. Desirable characteristics of games for older adults. Entertain Comput 2021; 39: 100440.

65.

Maggio

Calabrò

. Virtual reality and cognitive rehabilitation after traumatic brain injury. Amsterdam: Elsevier Inc, 2022. DOI: 10.1016/b978-0-12-823347-4.00014-2.

66.

Lee

Coughlin

. PERSPECTIVE: older adults’ adoption of technology: an integrated approach to identifying determinants and barriers. J Prod Innov Manag 2015; 32: 747–759.

67.

Katz

Jaeggi

Buschkuehl

, et al. Differential effect of motivational features on training improvements in school-based cognitive training. Front Hum Neurosci 2014; 8: 242–310.

68.

Schmitter-Edgecombe

Parsey

. Assessment of functional change and cognitive correlates in the progression from healthy cognitive aging to dementia. Neuropsychology 2014; 28: 881–893.

69.

Keefe

RSE

Harvey

. Cognitive impairment in schizophrenia. Handb Exp Pharmacol 2012; 213: 11–37.

70.

Hornbæk

Hertzum

. Technology acceptance and user experience: a review of the experiential component in HCI. ACM Trans Comput Interact 2017; 24: 1–30. DOI: 10.1145/3127358.