Older users,multimodal reminders and assisted living technology

Abstract

The primary users of assisted living technology are older people who are likely to have one or more sensory impairments. Multimodal technology allows users to interact via non-impaired senses and provides alternative ways to interact if primary interaction methods fail. An empirical user study was carried out with older participants which evaluated the performance, disruptiveness and subjective workload of visual, audio, tactile and olfactory notifications then compared the results with earlier findings in younger participants. It was found that disruption and subjective workload were not affected by modality, although some modalities were more effective at delivering information accurately. It is concluded that although further studies need to be carried out in a real-world settings, the findings support the argument for multiple modalities in assisted living technology.

Introduction

A recent report has claimed that the quality of home care in the UK is so poor that it contravenes the human rights of patients. ¹ The report highlights a number of reasons for low quality: services are not customised to account for sensory impairment or disability, services are not flexible in changing circumstances, patients have little-to-no control over how their care is delivered, and care workers have difficulty finding enough time for all their patients.

Assisted living technology (ALT) can address a number of these issues by helping the patient to manage their own care. This reduces the pressure on care workers and gives care recipients more control over the type of care they receive and how it is delivered. A home reminder system is one type of ALT which delivers messages and notifications to the user to help them manage their conditions, environment and lifestyle. Such notifications might remind users about appointments, meals and medicine times, running taps, unlocked windows and grocery needs; they could even help to promote healthy social behaviour such as phoning friends and family members.

Sensory impairments are common amongst care recipients ¹ and must be considered in providing effective care. For home reminder ALTs, multimodal interaction can be an effective solution. Multimodal interaction considers communication in multiple sensory channels and can allow users with impairments to interact using non-impaired channels (e.g. replacing an alarm with a flashing light). For a home reminder system this means delivering notifications in a range of different modalities.

When designing multimodal reminder systems it is important to consider both the effectiveness of the reminder modality (can it successfully deliver the intended message and produce the desired effect) and how acceptable it is to the user (is the delivery method appropriate for the message, situation and user). This article presents a user study that examined how effective different types of unimodal reminders were at delivering information to both older and younger users.

Related work

Many researchers have called for investigations into a wider range of interaction modalities,^2–5 particularly into systems equipped with a larger number of modalities that are able to switch between them to suit the situation or environment. When designing such systems it is crucial to understand how acceptable and effective different interaction modalities are. Unfortunately, there are few studies that evaluate a wide range of notification modalities with older users.

Warnock et al. ³ carried out the most comprehensive evaluation of multimodal reminder techniques when they compared the performance of visual, auditory, tactile and olfactory notifications. They concluded that differences between modalities were a result of their inherent properties, e.g. that tactile notifications were attention-grabbing. The authors suggested that some of the notifications would be more suited to particular situations than the others and that they form a collective ‘toolkit’ to aid communication in a complex environment such as the home, a point which was also argued by Perry et al. ⁵

Vastenburg et al. ⁶ argued that low-priority notifications should be delayed and/or delivered using less salient methods to increase user acceptability. Arroyo et al. ⁷ have also shown that managing how and when notifications are delivered can be a successful strategy. However, these projects do not consider different modalities. Perry et al. ⁵ considered multimodal interaction for their part in the Millennium Home project, arguing that modalities should be selected based on the user and their activities, as different modalities would provide a range of functionality.

An ALT equipped with multiple modalities should be able to use a notification modality that will minimise interference to the user and their current activity while remaining effective in delivering the intended message. Additionally, multiple modalities provide other opportunities to improve user acceptance, such as using discrete or abstract modalities to deliver potentially embarrassing reminders.

Care is needed when developing dynamic technology for the home. McBryan and Gray ² described a scenario where notifications are delivered to a hypothetical user, Fred, via mobile phone. Phone-based messages irritate Fred when he is at home, so he turns off his phone to force further messages through his television. Disabling his phone means his wife cannot call him and without the phone the system has fewer options for message delivery.

Berg et al. ⁸ found that most falls amongst the elderly at home were preventable as they were caused by haste or a lack of attention. A reminder system must be careful not to annoy, distract or stress the user; however avoiding this requires a deeper understanding of how and when to deliver reminders in the home, and a deeper understanding of individual notification modalities.

Although the principal recipients of home care are over 65 years of age, earlier work by Warnock et al.^3,9 was based around younger users. The findings of Warnock et al. cannot be transferred to older users. Park ¹⁰ presents a number of reasons for this: reduced speed of information processing, reduced working memory capacity and a reduced ability to ignore distractions. In order to understand more about how the modality of the reminder might influence its acceptability and appropriateness for older users, more experiments need to be carried out with older participants, evaluating different notification modalities both subjectively and objectively in terms of disruption, annoyance, workload and response accuracy.

This article presents a study that evaluated how older users performed when faced with notifications delivered in modalities that have previously been found effective for younger participants. ³ A core part of this study was the evaluation of subjective workload, which revealed crucial information about how older people felt about the different modalities.

Multimodal reminder study

Study design

A between-groups study was carried out to compare older participants with younger ones. The first experiment used younger participants (12 males, 8 females) aged 18–30 years and had already been completed and published.^{3, 9} This article presents a new experiment carried out following the same design using older participants (6 males, 10 females) aged 50–90 years. The experiment consisted of a primary task (playing a card-matching game on a computer) and a secondary task (acknowledging or ignoring notifications delivered at random points).

The independent variables in the experiments were the modality of the notification and the age group of the participants. There were three dependent variables: response accuracy, cards matched and subjective workload. Response accuracy measured the percentage of correct responses given to the notifications. Cards matched measured the number of card pairs that the participant could match within the time limit (ranging from 0 to 12). Subjective workload was measured by the NASA Task Load Index (NASA-TLX) subjective workload assessment. ¹¹ NASA-TLX measures subjective workload on six 21-point Likert scales representing Mental Demand, Physical Demand, Temporal Demand, Performance, Effort and Frustration. These are then summed to produce an overall workload score.

The hypotheses for the study were as follows:

• H1: older participants will report higher subjective workload (NASA-TLX) ratings;

• H2: modality will not affect subjective workload;

• H3: older participants will have poorer primary task performance;

• H4: modality will not affect primary task performance, and no interaction effects exist between modality and age;

• H5: older participants will have lower notification response accuracy;

• H6: olfactory and tactile modalities will have lower notification response accuracy scores than the other modalities, but no interaction effects will be found between modality and age.

Primary task

The primary task was a card-matching game called ‘Concentration’, as used by Warnock et al. ³ Pairs of cards were presented face-down to the player who then turned over two cards per turn to find pairs and remove them from the game. The cards showed alphabet caricatures and each game comprised 24 cards with a 60-second time limit. Although not a typical home care task, the game allowed us to measure the impact the reminder delivery had on a user when performing a mentally demanding primary task. Concentration is a simple leisure activity that reflects the type of task we expect users to carry out at home; other tasks such as walking or reading could also be used but are harder to administer and measure.

Notifications

Eight modalities were used in the study: text, pictograms, abstract visual messages, speech, earcons, auditory icons, tactons and olfactory notifications. There were three messages simulating home management reminders for each modality: ‘lights’, ‘heating’ and ‘telephone’. These were configured in the same way as in Warnock et al.’s ³ experiment with younger participants, as shown in Figure 1.

Figure 1.

Experiment configuration

Text and pictogram notifications were delivered directly into the game window to the top of the play area. The abstract visual display was created by positioning a short-throw projector to shine a coloured square against the adjacent wall, as shown in Figure 2. All auditory notifications were delivered through a pair of Sennheiser HD 25-1 II closed-back headphones (Sennheiser, Hanover, Lower Saxony, Germany), which helped to prevent interference from background noise. Tactile notifications were delivered via an Engineering Acoustics Inc. (Casselberry, Florida, USA) C2 vibrotactile actuator, secured to the top of the wrist on the participant’s non-dominant hand with a stretchable bandage as shown in Figure 3. Olfactory notifications were delivered using a Dale Air Vortex Active smell device (Dale Air, Whitworth, Rochdale, UK). Smells are stored on one-inch discs which are blown by a fan to distribute the smell. The device was placed directly in front of the participant, as shown in Figure 4.

Figure 2.

Abstract visual

Figure 3.

Tactile

Figure 4.

Olfactory

Method

Participants were asked to respond to notifications by pressing a button while playing the card-matching game. There were eight experimental conditions (one for each modality) and one control condition (where no reminders were delivered). Participants were allowed to practice before the experiment started.

In each experimental condition participants were asked to acknowledge one ‘target’ notification and ignore two ‘distractor’ notifications of the same modality. To acknowledge the target notification participants pressed a large yellow button on the desk. A small training segment preceded each condition, which began by introducing all three notifications before randomly picking one to be the target notification, which was then presented to the participant. Participants were presented all three notifications again and were expected to correctly acknowledge the target notification and ignore the distractor notifications to complete the training.

There were four trials in each condition, each consisting of one game and three notifications. Target and distractor notifications were balanced over the conditions to achieve a 1:1 ratio. At the end of each condition, participants filled in a NASA-TLX form. Participants were paid on completion. The experiment lasted 60–90 minutes.

Results

As shown in Figure 5, older participants reported much higher subjective workload ratings than younger participants; age had a significant effect on overall workload (U = 19,324; p <0.001), mental demand (U = 16,138; p <0.001), physical demand (U = 17,671; p <0.001), performance (U = 20,909; p <0.001) and frustration (U = 20,021; p <0.001). With NASA-TLX, a lower performance rating means higher success. This evidence strongly supports hypothesis H1; the only workload components that were not statistically significant were temporal demand and effort.

Figure 5.

Subjective workload by age group

For older users, no effect of modality was found on the TLX workload scores. With younger users a main effect of modality was found on both the overall workload rating (x²(8) = 20.35, p <0.01) and frustration (x²(8) = 16.33, p <0.05). Post-hoc pair-wise comparisons corrected with the Bonferroni method found significant differences between the control and olfactory conditions for both overall workload (p <0.01) and frustration (p <0.05). The findings support hypothesis H2; there were no statistically significant differences between the TLX workload ratings of any of the modalities.

A mixed-design ANOVA evaluated the effect of modality (as the within-groups variable) and age (as the between-groups variable) on the average matches per game. Modality had a main effect on average matches per game (F (8,272) = 7.45, p <0.001) and older users matched fewer cards in the game (F(1,34) = 85.9, p <0.001), but no interaction effects were found. Post-hoc pairwise comparisons with Sidak corrections found that significant differences only existed between the control and experimental conditions, as shown in Figure 6. The findings strongly support both hypotheses H3 and H4; younger participants out-performed older participants, modality did not affect primary task performance and there was no interaction between modality and age.

Figure 6.

Average matches per game by age group

A mixed-design ANOVA was used to evaluate response accuracy with modality as the within-groups variable and age as the between-groups variable. The results, shown in Figure 7, violated Mauchly’s test of sphericity (x²(27) = 97.8, p <0.01) so were corrected using the Greenhouse-Geisser method (ϵ = 0.36). Modality had a main effect on response accuracy (F(4.4,150.8) = 13.5, p <0.001) and age had a between-groups effect (F(1,34) = 9.2, p <0.01), but no interaction effects were found. Post-hoc pairwise comparisons with Sidak corrections found that the visual and audio modalities were all significantly different from the olfactory condition. The tactile condition was not significantly different from the olfactory, text and earcon conditions, but was significantly different from the other audio and visual modalities. The findings support both hypotheses H5 and H6.

Figure 7.

Graph of correct response by age group

Discussion

Older participants had more trouble with the tasks; they reported higher subjective workloads, matched fewer cards and correctly responded to fewer notifications. Park ¹⁰ suggested that the most likely reason for this difference is a reduction in the speed of working memory, which could also explain why older participants matched fewer cards in the control condition (without notifications).

Older participants were less proficient at correctly responding to notifications compared to the younger participants, but showed similar results. Older users had trouble with tactile and olfactory notifications, although response accuracy levels of 75–80% demonstrated that those modalities were still capable information-delivery mechanisms. The findings suggest that tactile and olfactory notifications should not be used to deliver important or urgent home care reminders unless there are special circumstances, for example, significant visual and audio impairment.

The subjective workload scores revealed that older participants, as with younger participants, ⁹ did not find any particular modality to have a greater workload. Modality was not found to effect primary task performance, again in line with younger participants. ³ These results have identified six modalities (text, pictogram, abstract visual, speech, earcon and auditory icon) that are capable of being used as primary interaction methods and two modalities (tactile and olfactory) that would be suitable for secondary interaction methods. Differences within the visual and audio methods, such as salience and discretion, can be used to deliver notifications in a more intelligent way by selecting the modality ‘on the fly’ to suit various situations and message types (e.g. delivering a sensitive reminder in an abstract method).

Perry et al. ⁵ argued in favour of dynamic multimodal ALTs for home care but also noted that dynamic technology such as this can introduce a number of other issues, for example, confusing the user when switching modes and inherent differences between modalities when constructing dialogues. Further work is needed to investigate this and to evaluate the real-world benefits that could be provided by such multimodal ALTs.

Conclusion

The results of this study confirm that modality does not play a major role in determining the subjective workload or primary task performance in older users, confirming that earlier findings with younger participants are still valid for older people. Older people found all of the tasks more difficult and exhibited poorer performance, although these differences were not so severe as to eliminate the possibility of using some modalities in the home for reminder delivery.

In conclusion, to ensure that multimodal notifications in the home are both acceptable and effective, different reminder modalities can be used depending on the message, user and environment. Research is needed into dynamic reminder delivery systems able to switch modality to suit the situation. This technology also needs to be evaluated in real-world settings through home trials with end users.