Measurement of older adults’ performance in digital technology-mediated occupations and management of digital technology

Abstract

Introduction

Supporting older adults’ digital engagement requires an understanding of how occupational performance and technology use are related, as well as having a range of methods that can assist occupational therapists while observing occupational performance and management of technology. The study objectives were to investigate how older adults’ ability to perform digital technology-mediated occupations and ability to manage digital technology could be measured and to examine the association between these two abilities.

Method

Twenty-five older adults were observed performing digital technology-mediated occupations and managing digital technologies, and were scored on two instruments: the Assessment of Computer-Related Skills and the Management of Everyday Technology Assessment. FACETS was used to generate respective multifaceted Rasch measurement models for scores on the instruments. The Spearman correlation test was used to investigate correlation between person ability measures from respective Rasch models of the instruments.

Results

The results include item, occupation, and technology difficulty estimates, as well as person ability measures that could illustrate older adults’ ability to perform occupations and to manage technology. There is also a strong positive correlation between these abilities.

Conclusion

Insight into an older person’s ability to manage technology can provide information about his or her ability to perform digital technology-mediated occupations and vice versa.

Keywords

Assessment digital engagement occupational therapy occupational therapy informatics observation technology use

Introduction and literature review

As the digitalization of societies persists, occupational injustices emerge from digital inequalities brought about by uneven access to and use of digital technologies (Fischl et al., 2020b; Kottorp et al., 2016; Nilsson and Townsend, 2010). Older adults, who represent the majority of infrequent or non-users of the internet in Sweden (Internetstiftelsen, 2019), become vulnerable to occupational injustices. In particular, they become at risk of being excluded from occupations that require the use of digital technologies, such as electronic citizen identification and e-Health applications (Fischl et al., 2020b; Kottorp et al., 2016). In order to reduce older adults’ risk of experiencing occupational injustices, it becomes pertinent for occupational therapists to support their engagement in digital technology-mediated occupations in particular contexts.

Occupations can be viewed as a transaction between the person and his or her context (Dickie et al., 2006). A transactional perspective implies that though the person and the environment can be seen as distinct entities, they are interdependent and a part of each other (Dickie et al., 2006; Law et al., 1996). Occupation serves as a bridge that allows a person to act on the environment through performance of or engagement in occupation (Townsend and Polatajko, 2013). Furthermore, technology can also be seen as an inherent part of occupation (Smith, 2017). Digital technology, for instance, has influenced the development of contemporary occupations and the transformation of more traditional occupations (Agger, 2011; Fischl et al., 2017). From this perspective, it becomes relevant to understand the relation between occupational performance and technology use so that occupational therapists can better support older persons’ engagement in occupations mediated by digital technologies.

There are studies that have implemented individualized interventions intended to enhance performance in digital technology-mediated occupations based on older adults’ preferences and occupation-focused goals (Fischl et al., 2020a; Larsson et al., 2016). Additionally, occupational therapists have introduced digital devices and provided training to older adults in order to improve their digital engagement (Kaldenberg and Smallfield, 2017; Swan et al., 2018). It has also been reported that older adults are more likely to use technology when it matches their needs and desires and provides them with satisfactory experiences (Fischl et al., 2020b; Ryd et al., 2018).

The core domain of occupational therapy is occupation and its relation to health and wellbeing (Townsend and Polatajko, 2013). Yet, recent studies in occupational therapy have focused more on improving technology use intended to support engagement rather than on enhancing engagement in occupations supported by technology (see for example Patomella et al., 2018; Swan et al., 2018). It has been argued that in order to clarify the profession’s occupation-centred perspective to clients and professionals in other disciplines, occupational therapists should use occupation-based and occupation-focused assessments and interventions (Fisher, 2013). It would therefore be pertinent to investigate the association between the observed ability to perform occupations mediated by digital technology and the observed ability to manage technology in order to understand how older adults’ occupational engagement could be supported.

Thus, the objectives of this study were to investigate how older adults’ ability to perform digital technology-mediated occupations and ability to manage digital technology could be measured and to examine the association between these two abilities. With regard to association, are older adults who easily perform digital technology-mediated occupations more likely to easily manage technology? And are older adults who have difficulties performing digital technology-mediated occupations more likely to have difficulties managing technology?

Method

The study used cross-sectional data from observations of older adults performing occupations and using technologies of their choice. Data was collected through the utilization of the Assessment of Computer-Related Skills (ACRS) (Fischl and Fisher, 2007) and the Management of Everyday Technology Assessment (META) (Malinowsky et al., 2011). The ACRS and the META were chosen from occupational therapy instruments intended to assess various aspects of occupational performance or digital technology use (Dumont and Mazer, 2013; Fischl and Fisher, 2007; Lane and Ziviani, 2003; Laver et al., 2012; Malinowsky et al., 2011; Rosenberg et al., 2009). The ACRS is intended to measure the ability to perform occupations using digital technology, while the META is intended to measure the ability to manage everyday technologies, including digital technology. Both instruments are observation-based and have been previously used in studies with older adults (Larsson et al., 2016; Malinowsky et al., 2018). The first author had the main responsibility for participant recruitment and data collection. All authors took part in the study design, data analysis, and reporting of results.

Participants

A municipality in Northern Sweden was chosen because of its well-developed municipal broadband infrastructure, and community-dwelling older residents of that municipality were invited to join the study. Invitations to an information meeting were posted on the physical bulletin boards of privately owned retirement communities and on the web-based calendar of a municipality-arranged activity centre. During the meeting, the researcher presented information about the study. Persons who filled in a Request More Information form were then contacted, informed individually about the study, and contacted again after an agreed period of time to consider participation. To be included in the study, participants had to be at least 65 years old, have access to digital technology at home, and be able to sustain a dialogue about their occupations involving technologies. A sample size ranging from 16 to 36 participants can contribute to the instruments’ item and person estimate stability at ±1 logit with a 95% confidence interval (Linacre, 1994). Most of the participants were selected through purposive sampling. A few participants were additionally recruited through snowball sampling. Written informed consent was given by all participants.

Ten men and 15 women, aged 71–93 years (mean =82.4 years) participated in the study (Table 1). All participants lived in rented apartments in attached multiple-unit housing, of whom 22 participants lived in residences that were part of retirement communities. Twenty-four persons reported having access to the internet at home, of whom 18 reported using the internet daily. One participant who reported not having internet access was observed to have a smart television connected to a digital box at home (Table 2). All communicated in Swedish.

Table 1.

Participant description (n = 25).

Demographic information	n
Age (mean = 82.36)
71–79	7
80–89	17
≥90	1
Gender
Male	10
Female	15
Civil status
Single	1
Married	6
Co-habiting	1
Widowed/widowered	15
Divorced	2
Children
No	2
Yes	23
Highest educational attainment
Primary education	4
Secondary education	8
Higher education	13
Occupational field prior to retirement^a
Managers	5
Occupations requiring advanced level of higher education	11
Occupations requiring higher education qualifications or equivalent	2
Administration and customer service clerks	3
Service, care, and shop sales workers	1
Agricultural, horticultural, forestry, and fishery workers	1
Building and manufacturing workers	1
Mechanical manufacturing and transport workers, etc.	1

^aOccupational fields based on the Swedish Standard Classification of Occupations (Statistics Sweden, 2012).

Table 2.

Participants’ reported home access to and use of digital technologies (n = 25).

Digital technology	Access at home	Frequency of use
Digital technology	Access at home	daily	sometimes	rarely	never
Stationary computer	4		2	2
Laptop computer	17	8	6	3	1
Tablet	18	10	3	5	1
Mobile phone	12	10	2
Smartphone	14	12	1		1
Printer	12	1	8	4	1
Internet	24	18	5	1

Note that responses relating to frequency of use may not match access at home since access to and use of digital technologies outside of one’s home was not examined.

Data collection

Instruments and tools

A questionnaire on demographic information and participants’ access to and use of digital technologies was used to collect information such as date of birth, sex, civil status, highest educational attainment, and living arrangements (Table 1). The questionnaire also included questions about access to common digital technologies at home and frequency of use for each technology to which they had identified having access (Table 2). It also included two questions to assist identification of occupations and technology that could be observed; in particular, which digital technologies does the participant consider important and which digital technologies would the participant like to try or be better at using. The questionnaire was paper-based and administered as an interview guide.

Participants were observed while performing occupations and using technologies of their choice. All observations were video filmed using one or two cameras. One camera was usually placed slightly behind the participant and focused towards the participant’s hands and input devices. The other camera was aimed towards the visual display. In instances where the interface for inputting information was near the visual display (for example a tablet or smartphone), only one camera was used. The films provided opportunities to view the performances repeatedly for separate scoring on the ACRS and the META and to review details to confirm scores made.

Assessment of Computer-Related Skills

The ACRS was developed to assess the quality of a person’s performance of computer-related occupations. It includes 37 skills related to a person’s interactions with the physical environment, interactions with the virtual environment, adaptation, temporal organization, and task completion (Appendix 1). The assessment procedure starts with a dialogue between the rater and client, wherein they discuss occupations involving computers or other digital devices which the client usually does, has difficulty with, or would like to do. They agree on at least two occupations for the client to perform, which the rater observes. The occupations should be considered by the client as relevant yet challenging to do. Novel activities can also be chosen. The rater then observes the client’s performance of chosen occupations and scores each skill item on a four-point scale, with 4 as competent, 3 as questionable, 2 as inefficient, and 1 as deficient. Each occupational performance is scored separately. Besides being used in a study with older adults (Larsson et al., 2016), the ACRS is validated on small samples of healthy people of working age (Fischl and Fisher, 2007) and adults with rheumatoid arthritis (Fischl et al., 2010).

Management of Everyday Technology Assessment

The META was developed to assess a person’s ability to manage everyday technology (Malinowsky et al., 2011). Everyday technology (ET) consists of ‘technological, electrical, and mechanical artefacts and services used in everyday life’. ET includes ‘newly-developed technology, e.g., telephone services and Internet-based services’ (Malinowsky et al., 2018: 99). In an assessment using META, a rater starts by interviewing a client about what he or she usually does and further explores ETs that the client describes as relevant and preferably somewhat challenging. The choice of two to four ETs for observation is based on their relevance and the challenge they present for the client. The rater then observes the client using the ET and scores each observation on items related to performance skills, intrapersonal capacities, environmental characteristics, and safety and importance (Appendix 2). META uses a four-point scale, with 4 as competent handling/management, 3 as deficits in this skill occasionally or slightly disturb the person’s use of the technology, 2 as deficits in this skill obviously disturb the person’s use of the technology, and 1 as deficits in this skill hinder the person’s use of the technology and/or the person is in need of assistance to perform the skill completely. Separate scoring is done for each ET used. Scores on 11 items, which comprise Part A: Performance Skills, were included in this study. The META has been tested for validity and reliability in samples with varying age groups, diagnosis, and cognitive functions (see for example Malinowsky and Larsson-Lund, 2016; Malinowsky et al., 2018).

Procedures

Data collection took place in the participants’ homes and took from one to four occasions. Each occasion lasted approximately 1 hour, and the number of occasions depended on the person’s length of performance of chosen occupations and energy. Data collection started with the questionnaire and a discussion of possible occupations that could be performed or digital technologies that could be used. The occupations that were agreed upon were activities that the participant performed regularly, was obliged to do, or identified as challenging or novel. The technologies that were chosen were existing technologies in the participants’ homes that participants considered relevant yet challenging to use. The chosen occupations and chosen technologies overlapped, so the occasions which were scored with the two instruments were the same.

While the participant performed chosen occupations and used chosen technologies, video recordings were made. Afterwards, the films were viewed and scored on the ACRS and then viewed again and scored on the META by the first author, who was trained in the use of the instruments. Observations that were challenging to score were discussed with the second author. Observations were scored on separate forms for each respective occupational performance or technology use. The scores were then compiled in separate Microsoft Excel worksheets in preparation for data analysis. Participants chose between two and eight occupations or technologies for observation, which resulted in a total of 97 linked sets of scores for the ACRS and 97 linked sets of scores for the META.

Data analysis

There were two stages to the data analysis in order to address the objectives of the study. The first stage involved Rasch analyses using FACETS version 3.83.0 in order to generate two separate multifaceted Rasch measurement models that show measures of persons’ ability to perform occupations and ability to manage technology. The Rasch models were generated from the data collected in this study and were not anchored to previously collected data. The second stage involved Spearman correlation test using IBM SPSS Statistics version 25 to investigate association between these two abilities.

Exploring how abilities are measured

Rasch analysis was chosen as it has been previously used in the construction and validation of the ACRS and the META. Additionally, a multifaceted Rasch measurement (MFRM) model could illustrate a linear relationship between the instrument items (facet 1) in relation to the person (facet 2) as well as occupations performed or technologies used (facet 3). Thus, a person based on his or her ability measure, expressed in log-odds units or logits, could be located on the linear scale in relation to the difficulty estimates of items and occupations or technologies, which are also in logits. Higher logits for persons mean that they are more able to perform occupations or manage technologies. Items, occupations, or technologies with higher logits would be considered as more difficult, while items and occupations or technologies with lower logits would be considered as easier.

In each generated model, the goodness-of-fit statistics were first examined to determine how well the skills, occupations, and persons fit the MFRM model of the ACRS and respectively how well the skills, technologies, and persons fit the MFRM model of the META. The fit of items to their respective models was in focus in this examination as items that were found difficult in this sample should be expected to be just as difficult in observations of other people (Wright and Linacre, 1994). Therefore, items had to demonstrate goodness-of-fit before other facets were examined. On the other hand, person ability as well as occupations and technologies could change over time. Person ability as well as occupation or technology difficulty estimates that did not exhibit goodness-of-fit were noted in the results. To ensure that there is goodness-of-fit in items, both the mean square fit statistics (MnSq) should be less than or equal to 1.4 (Linacre, 2002; Wright and Linacre, 1994) and the standardized z score should be less than 2.0 (Linacre, 2002) in either infit or outfit. Mean square indicates the randomness of the data and corresponds to chi-square statistics divided by their degrees of freedom, while standardized fit statistics indicate the predictability of the data and correspond to t-tests of the hypothesis, Do the data fit the model? Infit reflects the pattern of scores on items for each person, and outfit reflects the scores on items with difficulty far from the person due to guessing or mistakes (Linacre, 2002). The generation of the MFRM models was done iteratively, removing the items that did not fit one step at a time, until goodness-of-fit was demonstrated. For ACRS, even items that did not demonstrate fit in previous validity studies (Fischl and Fisher, 2007; Fischl et al., 2010) were removed. When the item difficulty estimates demonstrated fit the MFRM models of the ACRS and the META, respectively, then the person ability measures were compiled in preparation for the next stage of analysis.

Standard error of measurement (SE) and separation ratio were also examined to provide indications of reliability (Fisher, 1992). SE values are estimates of how close each person, item, occupation, or technology estimate is to its true measure. An SE that is less than 2 logits indicates a precise measure (Linacre, 2013). Considering the small sample size, an SE value equal to or less than 0.30 was set as acceptable. Separation ratios are estimates of the true distribution of the measures considering the SE. Separation ratios (G) for items, occupations, and technologies signified how many distinct levels of difficulty were observed, while a separation ratio for persons showed how many distinct levels of ability participants demonstrated. The distinct levels can be computed with the formula: (4G+1)/3 (Fisher, 1992).

Investigating the association between abilities

The Spearman correlation test, chosen because of the small sample size, was administered to investigate association between person ability measures on the ACRS and the META. The resulting Spearman correlation coefficient (r) and significance level (p) were noted. This study adopted a conventional interpretation of the correlation coefficient (0.00–0.10 = negligible correlation, 0.10–0.39 = weak correlation, 0.40–0.69 = moderate correlation, 0.70–0.89 = strong correlation, 0.90–1.00 = very strong correlation) (Schober et al., 2018).

Ethical considerations

Participants were informed about the study objectives and procedures, and assured that their personal information would be kept confidential. For the data collection, it was made clear to participants that occupations relating to personal economy and finances were to be excluded from observations and tailoring. It was also clarified to participants that when they keyed in personal identity numbers or passwords, the researcher would look away and cover the lens of the camera. They were also reminded not to verbalize this information while typing it in. Though providing specific information was reflected in skill items in both instruments, the scoring was based on the sounds made while keying information (for example slow typing, participant sighing, sound notification from device) and observation of whether the participant could move on to the next step after login. Furthermore, the actual content of messages, social media posts, and search results were not the focus of the observations, but the spelling of common words, placement of punctuation marks, order of characters, and specificity of search terms were observed and scored on skill items in both instruments.

Data collection sessions were concluded with a summary of what was observed, and participants were given an opportunity to discuss issues about their performances. All participants were reassured that the observations were intended to illuminate challenges encountered by many older adults, often because of the complexity of digital technologies, and that they have contributed to the efforts for making improvements. This was particularly stressed to participants who expressed any form of dissatisfaction over their performance or articulated their inability as an explanation for not completing a chosen occupation as expected.

Results

ACRS items

Half of the ACRS items were removed from the analysis, based on the examination of goodness-of-fit statistics in the MFRM model iterations for the ACRS scores. The final item difficulty estimates ranged from 0.91 to –1.68 logits, with SE lower than 0.30 for all estimates (Table 3). The separation ratio for items was 4.47, which indicates six levels of item difficulty.

Table 3.

Final estimates for items that demonstrated goodness-of-fit in the Assessment of Computer-Related Skills, arranged by difficulty.

					Infit		Outfit
	Item		Measure (logits)	SE (logits)	MnSq	z	MnSq	z
More difficult	AD-4	Adapts performance	0.91	0.15	1.23	1.4	1.04	0.2
↑	TO-4	Terminates actions	0.73	0.13	0.98	0.0	0.86	−0.6
	VE-2	Locates virtual object	0.54	0.14	1.13	0.9	1.21	0.9
	VE-3	Selects appropriate virtual object	0.54	0.13	0.81	−1.3	0.79	−0.9
	VE-8	Seeks information	0.52	0.14	0.85	−1.0	0.71	−1.3
	TC-3	Sustains effort	0.42	0.13	1.01	0.1	0.97	0.0
	AD-2	Modifies behaviour	0.39	0.13	0.79	−1.4	0.79	0.0
	TO-2	Continues actions	0.20	0.13	0.70	−2.2	0.76	−1.0
	AD-1	Responds to cues	0.11	0.13	0.70	−2.2	0.79	−0.8
	TC-2	Maintains focus	0.11	0.13	0.80	−1.4	0.77	−0.9
	VE-7	Provides information	0.09	0.19	1.38	1.9	1.46	1.3
	TO-5	Paces actions	−0.03	0.13	0.68	−2.4	0.96	0.0
	TO-3	Sequences actions	−0.08	0.14	1.04	0.2	0.91	−0.2
	VE-4	Uses virtual object	−0.14	0.13	0.92	−0.5	0.72	−1.0
	VE-6	Inserts virtual object	−0.71	0.17	1.10	0.6	1.20	0.6
	PE-10	Executes coordinated movements	−0.81	0.16	1.16	0.9	1.71	1.7
↓Less difficult	TO-1	Initiates actions	−1.10	0.17	1.25	1.3	1.48	1.1
↓Less difficult	PE-9	Moves tools and materials	−1.68	0.20	0.87	−0.5	2.02	1.5

META items

Ten items in the META demonstrated goodness-of-fit to the MFRM model. Item A11 was not scored at all in any of the observations because of the participants’ choices of technology use; thus, it was not shown in the results. All 10 items displayed goodness-of fit to the model in the second iteration. The final item difficulty estimates ranged from 1.20 to –2.16 logits, with three SE values higher than 0.30 (Table 4). The separation ratio for items was 3.59, indicating five levels of item difficulty.

Table 4.

Item estimates for the Management of Everyday Technology Assessment, arranged by difficulty.

	Item		Measure (logits)	SE (logits)^a	Infit		Outfit
	Item		Measure (logits)	SE (logits)^a	MnSq	z	MnSq	z
More difficult	A10	Notice information and respond adequately	1.20	0.12	0.87	−0.9	0.81	−1.2
↑	A5	Choose correct button or command	0.86	0.12	0.82	−1.3	0.78	−1.4
	A7	Use appropriate force, tempo, and precision	0.62	0.12	0.98	−0.1	1.06	0.4
	A2	Identify and select services and functions within a technology	0.45	0.14	1.20	1.4	1.18	1.0
	A3	Perform steps and actions in logical sequence	0.37	0.13	0.85	−0.9	0.85	−0.8
	A8	Coordinate different parts of the technology	0.35	0.35	0.40	−2.0	0.37	−1.5
	A4	Manage a series of numbers/letters	0.30	0.18	1.23	1.2	1.38	1.6
	A9	Coordinate the technology with another technology without physical contact	−0.22	0.20	0.85	−0.6	1.07	0.3
↓	A6	Turn a button or knob in correct direction and position	−1.78	0.71	0.27	−0.9	0.17	−0.7
Less difficult	A1	Identify and select/separate technologies	−2.16	0.31	1.05	0.2	1.81	1.4

^aSE>0.30 in italics.

ACRS occupations

Participants were observed to perform 45 different occupations. The occupation difficulty estimates ranged from 3.10 to –4.95, with 15 estimates with SE higher than 0.30 (Table 5). Reading an e-book and following the news on text TV did not demonstrate goodness-of-fit to the MFRM model of the ACRS, having MnSq>1.4 and z≥2. The separation ratio for occupations, without extremes, was 4.24, which indicates six levels of occupation difficulty.

Table 5.

Final estimates for occupation difficulty in the Assessment of Computer-Related Skills, arranged by difficulty.

	Occupation	Measure (logits)	SE (logits)^b
More difficult	Save or send a webpage	3.10	0.27
↑	Search for information on a predetermined topic and save	3.01	0.26
	Upload a photo to the web	2.84	0.59
	Make an appointment	2.23	0.30
	Copy/move photo to another device	1.84	0.16
	Save a photo attached in an email	1.78	0.32
	Edit and save photo	1.76	1.01
	Send an existing photo to contact	1.50	0.18
	Take and send a photo	1.35	0.14
	Show photos	1.09	0.51
	Create a document with photos and text	1.00	0.19
	Play app-based word or card games	0.96	0.29
	Search for and play music	0.67	0.26
	Search for government information	0.63	0.27
	Search for information about weather and send to contact	0.62	0.23
	Search for products and save in shopping cart	0.32	0.39
	Print photo	0.22	0.27
	Search for community events and activities	0.22	0.19
	Print document	0.08	0.27
	Organize and delete file	0.03	0.38
	Take a photo and check if it has been saved	−0.01	0.21
	Search for specific multimedia and check information	−0.03	0.10
	Write email with text	−0.37	0.12
	Add appointment on digital calendar	−0.40	0.32
	Play a particular TV programme	−0.57	0.18
	Read latest news	−0.61	0.12
	Search for and play TV programme on play-on-demand services	−0.71	0.21
	Read TextTV^a	−0.72	0.19
	Write message with text	−0.73	0.39
	Check message and replies	−0.76	0.28
	Check posts/newsfeed on social networking site	−0.77	0.15
	Search for information on an old acquaintance	−0.91	0.27
	Check winning teams in sports	−0.92	0.28
	Create a document or presentation with text	−0.97	0.45
	Read an e-book^a	−0.98	0.42
	Play music	−1.34	0.20
	Play web-based games	−1.77	0.71
	Search for a reference (synonym) and read or use information	−1.82	0.26
	Check the weather/pollen conditions	−1.97	0.33
	Search for information on a predetermined topic and read/check information	−2.24	0.29
	Play TV programme (with written instructions)	−2.51	0.28
	Call a contact	−4.15	.071
	Search for a product or service and read/check information	(−3.88)	(1.83) ^c
↓Less difficult	Save e-book in bookshelf	(−4.16)	(1.84)
↓Less difficult	Read/check email	(−4.95)	(1.82)

^aOccupations that did not demonstrate goodness-of-fit having MnSq>1.4 and z≥2.

^bSE>0.30 in italics.

^cEstimates and SE in parentheses were observed only in one occasion and considered ‘extreme’.

META technologies

Participants were observed using 37 combinations of technologies using the META. The estimates for technology difficulty ranged from 1.65 to –4.42 logits (Table 6). Twenty-six technology difficulty estimates were associated with SE higher than 0.30. The use of an email app on a tablet did not demonstrate goodness-of-fit to the MFRM model of the META, having MnSq>1.4 and z≥2. The separation ratio for technologies, without extremes, was 1.91, suggesting that there are at least two levels of technology difficulty.

Table 6.

Final estimates for technology difficulty in the Management of Everyday Technology Assessment, arranged by difficulty.

	Technology	Measure (logits)	SE (logits)^b
More difficult↑	Tablet, TV play-on-demand services	1.65	0.48
More difficult↑	Tablet, camera	1.53	0.31
	Stationary computer, photo app	1.27	0.54
	Laptop computer, USB memory stick	0.94	0.42
	Tablet, e-book	0.82	0.65
	Stationary computer, maintenance	0.82	0.59
	Smartphone, photo app	0.76	0.29
	Tablet, gaming app	0.66	0.45
	Smartphone, laptop computer	0.56	0.29
	Tablet, news app	0.54	0.45
	Laptop computer, email	0.48	0.21
	Laptop computer, printer	0.48	0.28
	Smartphone, camera	0.43	0.20
	Smartphone, web browser	0.32	0.24
	Laptop computer, calendar	0.24	0.47
	Tablet, web browser	0.17	0.19
	Smartphone, TextTV	0.00	0.58
	Tablet, message app	−0.05	0.54
	Smart TV, Text TV	−0.05	0.34
	Laptop computer, word or presentation app	−0.10	0.27
	Smartphone, message app	−0.12	0.44
	Laptop computer, web browser	−0.18	0.10
	Smart TV, TV play-on-demand services	−0.21	0.26
	Tablet, email^a	−0.32	0.29
	Smartphone, music player	−0.49	0.31
	Stationary computer, music player	−0.52	0.39
	Tablet, social networking site	−0.55	0.52
	Smart TV, TV play-on-demand services, written cues	−0.74	0.47
	Smart TV, regular TV programming	−1.14	0.70
	Smartphone, social networking site	−1.18	0.53
	Smartphone, news app	−1.68	0.41
	Smartphone, pollen control app	−1.68	0.57
	Mobile phone, call function	−2.68	0.70
	Tablet, music player	(−2.65)	(1.82)
	Smartphone, email	(−3.95)	(1.76)
↓Less difficult	Smartphone, voice-controlled assistant	(−3.95)	(1.76)
↓Less difficult	Smartphone, e-book	(−4.42)	(1.83)

^aTechnology that did not demonstrate goodness-of-fit having MnSq>1.4 and z≥2.

^bSE>0.30 in italics.

^cEstimates and SE in parentheses were observed only in one occasion and considered ‘extreme’.

Person ability measures for the ACRS and the META

Person ability measures in the ACRS ranged from 5.41 to –2.05 logits, with only one SE value greater than 0.30. On the other hand, person ability measures in the META ranged from 3.09 to –0.75 logits with nine measures associated with high SE values. (Table 7). Goodness-of-fit was demonstrated, with the exception of three persons in the ACRS model and one person in the META model. The person that did not demonstrate fit in both models was a woman aged 93 years. The separation ratio for person ability measures in the ACRS was 8.30 (11 levels of person ability), while the separation ratio for person ability measures in the META was 3.37 (4 levels of person ability).

Table 7.

Person ability measures in the Assessment of Computer-Related Skills and Management of Everyday Technology Assessment, arranged by ability.

ACRS						META
ID	Gender	Age	Measure (logits)	SE(logits)^b		ID	Gender	Age	Measure (logits)	SE (logits)^b
1	F	75	5.41	.50	More able	1	F	75	3.09	.44
3	F	77	3.30	.18	↑	21	F	84	2.83	.34
21	F	84	2.92	.25		10^a	F	93	2.65	.35
10^a	F	93	2.47	.29		15	F	72	2.48	.32
15^a	F	72	2.18	.25		14	F	84	1.98	.38
24	F	78	1.47	.15		9	M	86	1.74	.31
9	M	86	1.19	.17		3	F	77	1.71	.31
12	F	77	1.07	.15		12	F	77	1.43	.23
14^a	F	84	1.05	.30		13	M	84	1.37	.23
16	M	82	.99	.16		2	M	86	1.23	.26
17	F	82	.97	.20		11	M	87	1.15	.20
11	M	87	.68	.14		23	M	81	1.08	.22
18	F	88	.66	.18		17	F	82	1.06	.30
8	F	87	.64	.14		20	F	86	.82	.24
13	M	84	.05	.14		24	F	78	.76	.20
23	M	81	−0.02	.14		4	M	81	.74	.23
4	M	81	−0.07	.16		16	M	82	.66	.23
2	M	86	−0.16	.15		18	F	88	.42	.47
25	M	71	−0.17	.12		8	F	87	.33	.23
22	M	86	−1.00	.17		5	F	88	.32	.25
19	F	86	−1.30	.13		22	M	86	.23	.22
20	F	86	−1.31	.17		25	M	71	.07	.22
6	F	74	−1.41	.20		6	F	74	−0.35	.24
7	M	84	−1.84	.22	↓Less able	19	F	86	−0.37	.22
5	F	88	−2.05	.19	↓Less able	7	M	84	−0.75	.34

^aPerson ability measures that did not demonstrate goodness-of-fit having MnSq>1.4 and z≥2.

^bSE>0.30 in italics.

Correlation between abilities

In the examination for correlation, Spearman’s rho was equal to 0.825 (p<0.01). It indicated a significant and strong positive correlation between the person ability measures in the ACRS and in the META. Figure 1 shows a representation of this correlation.

Figure 1.

Scatterplot of Assessment of Computer-Related Skills (ACRS) and Management of Everyday Technology Assessment (META) person ability measures.

Discussion and implications

The first objective of this study was to explore how older adults’ ability to perform digital technology-mediated occupations and ability to manage digital technology could be measured. The resulting goodness-of-fit of items and MFRM models for the ACRS and the META demonstrated the instruments’ potential in the measurement of older adults’ ability to perform digital technology-mediated occupations and ability to manage digital technology. Both MFRM models for the instruments provided estimates and hierarchies of difficulty for items, occupations, and technologies. These hierarchies give information about which items, occupations, and digital technologies could be expected to be easier to perform or use and which could be expected to be more difficult to perform or use. Items in the ACRS that did not demonstrate goodness-of-fit in this study, which could have been influenced by technological changes, suggest further examination of the instrument’s construct validity. Person ability measures for both instruments also provided insight about how well older adults could perform digital technology-mediated occupations and manage digital technology.

With regard to the second objective – to explore the association between older people’s ability to perform occupations mediated by digital technology and ability to manage digital technology – there was a significant and strong positive correlation between these two abilities. This result means that older adults who easily perform digital technology-mediated occupations are more likely to easily manage technology and vice versa. In contrast, older adults who have difficulties performing digital technology-mediated occupations are more likely to have difficulties managing technology and vice versa. This result also indicates preliminary evidence of concurrent validity between the ACRS and the META. Using one of the instruments would provide an insight into both abilities, but complementary use of the two instruments would help occupational therapists identify which aspects of occupational performance and technology management need to be put in focus in planning intervention. Additionally, the two instruments would add to the inventory of tools that could be used to address needs in various research and practice contexts.

How are the ability to perform occupations mediated by digital technology and the ability to manage technology related? Since causality is not examined in this paper, it cannot be concluded that improving one ability will help improve the other ability. However, with knowledge of the linear hierarchies of the META and the ACRS in this study, it can be implied that awareness about an older client’s ability to manage technology can provide information about his or her ability to perform digital technology-mediated occupations and vice versa. Thus, assessment instruments that intend to measure either technology use or occupational performance can help establish baseline information for identifying occupation-focused goals and planning interventions. Assessing either occupational performance or the use of technology, being an inseparable part of occupation (Smith, 2017), fits into the transactional perspective that the focus of assessment should be the transaction between the person and the environment (Law et al., 1996). This suggests that a focus on technology use is an occupation-based approach. In addition, knowledge about what to focus on during assessment contributes towards addressing the need for assessment methods in occupational therapy (Larsson-Lund, 2018).

As an interesting result, the strong positive correlation between the person estimates brings forth the question of whether the ACRS and Part A of the META, which focuses on performance skills, measure the same latent trait. This relationship should be investigated in consideration of the theoretical backgrounds of the instruments in order to further understand the construct of occupational performance.

As observations were based on doing occupations or using technology that participants considered relevant to themselves, it is important to highlight the role of relevance. The relevance of the occupation or technology to the older adult has been considered essential in order to support his or her digital engagement (Fischl et al., 2020a; Ryd et al., 2018). Relevance pertains to the alignment of the person’s needs, goals, values, and beliefs with the perceived utility of the occupation or technology (Amini et al., 2014). From an occupational justice perspective, digitalization and rapid technological developments affect the relevance of occupations or technologies, subsequently impacting older adults’ participation in society (Kottorp et al., 2016). There is a risk that occupations may become more difficult to engage in and lose relevance if digitalization forces abrupt changes to the requirements to perform occupations (for example social activities only announced or arranged online). There is also the risk that a technology an older person currently owns may lose its relevance if rapid technological developments cause digital technologies to become obsolete in a short time because of lack of technical support and maintenance. Technologies can also become irrelevant due to increasing complexity during developments causing further difficulties in their use (Okonji et al., 2015). As digitalization is inevitable, it would be worthwhile investigating whether there is a causal relationship between relevance, ability to perform digital technology-mediated occupations, and ability to manage digital technology.

Methodological considerations

Internal validity was evaluated through goodness-of-fit of items in both MFRM models to examine for internal validity. In the ACRS, 18 out of 37 items demonstrated goodness-of-fit. Items that did not fit were related to interaction with the physical environment, as well as organizing and restoring the virtual environment; a few of these items were also reported not to fit in previous validation studies (Fischl and Fisher, 2007; Fischl et al., 2010). On other hand, 10 out of 11 META items exhibited goodness-of-fit to its model. The ACRS items that misfit could have been affected by developments in interfaces, while the META item that misfit could have been due to the choice of relevant technologies to use. This suggests that the instruments can be vulnerable to technology changes. Further development of these instruments should also include focusing on skill items that are not dependent on technology, so that skill items remain stable even if occupations and technology change (Wright and Linacre, 1994). The remaining items demonstrated goodness-of-fit in the MFRM model of the ACRS and the META, indicating preliminary evidence of internal validity. A bigger sample could contribute to a stronger evidence of internal validity.

In the ACRS, there is a degree of logic in the occupation hierarchy as occupations placed higher in the hierarchy require more steps (for example search and send) and more complex information processing (for example remember keywords, evaluate content, and share), while occupations lower in the hierarchy have fewer steps (for example read) and less complex information processing (for example receive information). In the META, it appears logical that smartphones are ranked as being easier, as they are reported to be used daily by almost half of the participants (Table 4). This technology has also been reported as easier to use in other studies (Malinowsky and Larsson-Lund, 2016; Malinowsky et al., 2011). Data collection on a larger sample can contribute to the hierarchies of difficulty estimates.

With regard to reliability, observing each participant perform at least two occupations and use at least two technologies increased the observed count for each item on the instruments in order to enhance reliability of person estimates (Kruyen, 2012; Linacre, 1994). However, among estimates of occupation difficulty in the ACRS, one third were associated with high SE. This could be due to a low number of observations for these occupations, as the choice of occupations was based on what participants considered as relevant to them. Among estimates of technology difficulty in the META, 70% were associated with higher SE values, which can be explained by the low number of observations due to the choice of relevant technologies and by the low observed count based on 10 META items. These results reinforce the need for testing on larger samples performing occupations and using technologies, thereby contributing to reliability.

It could be surmised that the low participation rate in this study could be attributed to the perceived (ir)relevance of digital technologies in older adults’ lives. However, validity and reliability could be improved with more participants performing similar occupations and using similar technologies as well as more raters who could score observations on both the ACRS and the META. Further examination of the goodness-of-fit of items in the ACRS can also be done in studies with bigger samples. No statistical analysis was done to investigate differences in participants’ abilities based on sociodemographic variables and frequency of use. Such investigation could be the focus of a future study.

Conclusion

With further studies to examine the instruments’ psychometric properties, the ACRS and the META can become useful tools for occupational therapists to measure older adults’ ability to perform digital technology-mediated occupations and ability to manage digital technologies. The significant and strong positive correlation between these abilities implies that insight about an older person’s ability to manage technology can provide information about his or her ability to perform digital technology-mediated occupations and vice versa. This knowledge of areas to focus on during assessment could contribute to occupational therapists’ potential to support older adults’ digital engagement.

Key findings

The ACRS and the META have potential in measuring older adults’ digital engagement.

Technology management is positively correlated to performance of digital technology-mediated occupations.

What the study has added

Knowledge about the measurement of the ability to manage technology and the ability to perform occupations could help support older persons’ digital engagement.

Footnotes

Research ethics

Ethical approval was granted in 2014 by the Regional ethical review board in Umeå, Sweden (Dnr 2013-418-31Ö). Written informed consent was given by all participants.

Declaration of conflicting interests

The authors declare no potential conflicts of interest with respect to the authorship, research or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This work was financially supported by the Swedish Research Council for Health, Working Life and Welfare (Forskningsrådet för hälsa, arbetsliv och välfärd) under Grant number 2013-2143 and the European Regional Development Fund through the Botnia-Atlantica 2014-2020 project @geing Online.

Contributorship

Caroline Fischl had the main responsibility for participant recruitment and data collection. All authors took part in the study design, data analysis, and reporting of results.

ORCID iD

Caroline Fischl

Note

Training in the administration of the ACRS and the META are the responsibility of the respective instrument developers.

Appendix 1. Skill items in the Assessment of Computer-Related Skills

Appendix 2. Items in the Management of Everyday Technology Assessment

References

Agger

(2011) iTime: Labor and life in a smartphone era. Time & Society 20(1): 119–136.

Amini

Kannenberg

Bodison

, et al. (2014) Occupational therapy practice framework: Domain & process 3rd edition. American Journal of Occupational Therapy 68(Supplement 1): S1–S48.

Dickie

Cutchin

Humphry

(2006) Occupation as transactional experience: A critique of individualism in occupational science. Journal of Occupational Science 13(1): 83–93.

Dumont

Mazer

(2013) Assessment of Computer Task Performance for children: Standardization, validation and reliability. Technology and Disability 25(1): 27–35.

Fischl

Fisher

(2007) Development and Rasch analysis of the Assessment of Computer-Related Skills. Scandinavian Journal of Occupational Therapy 14(2): 126–135.

Fischl

Asaba

Nilsson

(2017) Exploring potential in participation mediated by digital technology among older adults. Journal of Occupational Science 24(3): 314–326.

Fischl

Blusi

Lindgren

, et al. (2020a) Tailoring to support digital technology-mediated occupational engagement for older adults – A multiple case study. Scandinavian Journal of Occupational Therapy. Available at: https://www.tandfonline.com/doi/full/10.1080/11038128.2020.1760347 (accessed 12 May 2020).

Fischl

Hammar

Johansson

, et al. (2010) Validity and rater reliability of the Assessment of Computer-Related Skills for persons with rheumatic diseases. Philippine Journal of Occupational Therapy 4(1): 4–11.

Fischl

Lindelöf

Lindgren

, et al. (2020b) Older adults’ perceptions of contexts surrounding their social participation in a digitalized society—An exploration in rural communities in Northern Sweden. European Journal of Ageing. Available at: https://doi.org/10.1007/s10433-020-00558-7 (accessed 11 February 2020).

10.

Fisher

(2013) Occupation-centred, occupation-based, occupation-focused: Same, same or different? Scandinavian Journal of Occupational Therapy 20(3): 162–173.

11.

Fisher

Jr (1992) Reliability, separation, strata statistics. Rasch Measurement Transactions 6(3): 238.

12.

Internetstiftelsen [The Swedish Internet Foundation] (2019) Svenskarna och internet 2019 [The Swedes and the Internet 2019]. Available at: https://svenskarnaochinternet.se/rapporter/svenskarna-och-internet-2019/ (accessed 17 February 2020).

13.

Kaldenberg

Smallfield

(2017) Training older adults with low vision to use a computer tablet: A feasibility study. British Journal of Occupational Therapy 80(2): 117–122.

14.

Kottorp

Nygård

Hedman

, et al. (2016) Access to and use of everyday technology among older people: An occupational justice issue – but for whom? Journal of Occupational Science 23(3): 382–388.

15.

Kruyen

(2012) Using Short Tests and Questionnaires for Making Decisions about Individuals: When is Short too Short. Ridderkerk: Ridderprint.

16.

Lane

Ziviani

(2003) Assessing children’s competence in computer interactions: Preliminary reliability and validity of the Test of Mouse Proficiency. OTJR–Occupation Participation and Health 23(1): 18–26.

17.

Larsson

Padyab

Larsson-Lund

, et al. (2016) Effects of a social internet-based intervention programme for older adults: An explorative randomised crossover study. British Journal of Occupational Therapy 79(10): 629–636.

18.

Larsson-Lund

(2018) The digital society: Occupational therapists need to act proactively to meet the growing demands of digital competence. British Journal of Occupational Therapy 81(12): 733–735.

19.

Laver

George

Ratcliffe

, et al. (2012) Measuring technology self efficacy: Reliability and construct validity of a modified computer self efficacy scale in a clinical rehabilitation setting. Disability and Rehabilitation 34(3): 220–227.

20.

Law

Cooper

Strong

, et al. (1996) The Person-Environment-Occupation Model: A transactive approach to occupational performance. Canadian Journal of Occupational Therapy 63(1): 9–23.

21.

Linacre

(1994) Sample size and item calibration [or person measure] stability. Rasch Measurement Transactions 7(4): 328.

22.

Linacre

(2002) What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions 16(2): 878.

23.

Linacre

(2013) Rasch Forum 2013. Available at: www.rasch.org/forum2013b.htm (accessed 28 February 2020).

24.

Malinowsky

Larsson-Lund

(2016) The match between everyday technology in public space and the ability of working-age people with acquired brain injury to use it. British Journal of Occupational Therapy 79(1): 26–34.

25.

Malinowsky

Fallahpour

Lund

, et al. (2018) Skill clusters of ability to manage everyday technology among people with and without cognitive impairment, dementia and acquired brain injury. Scandinavian Journal of Occupational Therapy 25(2): 99–107.

26.

Malinowsky

Nygård

Kottorp

(2011) Psychometric evaluation of a new assessment of the ability to manage technology in everyday life. Scandinavian Journal of Occupational Therapy 18(1): 26–35.

27.

Nilsson

Townsend

(2010) Occupational justice: Bridging theory and practice. Scandinavian Journal of Occupational Therapy 17(1): 57–63.

28.

Okonji

Lhussier

Bailey

, et al. (2015) Internet use: Perceptions and experiences of visually impaired older adults. Journal of Social Inclusion 6(1): 120–145.

29.

Patomella

Lovarini

Lindqvist

, et al. (2018) Technology use to improve everyday occupations in older persons with mild dementia or mild cognitive impairment: A scoping review. British Journal of Occupational Therapy 81(10): 555–565.

30.

Rosenberg

Nygård

Kottorp

(2009) Everyday Technology Use Questionnaire: Psychometric evaluation of a new assessment of competence in technology use. OTJR–Occupation, Participation and Health 29(2): 52–62.

31.

Ryd

Malinowsky

Öhman

, et al. (2018) Older adults’ experiences of daily life occupations as everyday technology changes. British Journal of Occupational Therapy 81(10): 601–608.

32.

Schober

Boer

Schwarte

(2018) Correlation coefficients: Appropriate use and interpretation. Anesthesia & Analgesia 126(5): 1763–1768.

33.

Smith

(2017) Technology and occupation: Past, present, and the next 100 years of theory and practice (2017 Eleanor Clarke Slagle Lecture). American Journal of Occupational Therapy 71(6): 7106150010p1–7106150010p14.

34.

Statistics Sweden (2012) Swedish standard classification of occupations. Available at: www.scb.se/en/finding-statistics/statistics-by-subject-area/other/other/other-publications-non-statistical/pong/publications/swedish-standard-classification-of-occupations-2012/ (accessed 29 January 2020).

35.

Swan

Hitch

Pattison

, et al. (2018) Meaningful occupation with iPads: Experiences of residents and staff in an older person’s mental health setting. British Journal of Occupational Therapy 81(11): 649–656.

36.

Townsend

Polatajko

(2013) Enabling Occupation II: Advancing an Occupational Therapy Vision for Health, Well-being & Justice through Occupation: 9th Canadian Occupational Therapy Guidelines. Ottawa: CAOT Publications ACE.

37.

Wright

Linacre

(1994) Reasonable mean-square fit values. Rasch Measurement Transactions 8(3): 370.