The lights are on,but no one’s home: A performance test to measure digital skills to use IoT home automation

Abstract

As the Internet of Things (IoT) is making its entrance in people’s homes, differences in the skills to operate smart home devices need to be considered. This study examined (1) the levels of digital skills to use IoT home automation among Dutch adult citizens and (2) differences of these skills over gender, age, and education. Therefore, a performance test with actual real-life tasks was conducted among a representative sample (N = 99) of the Dutch adult population to measure digital skill levels. The participants performed tasks while using interconnected smart home devices in a virtual test environment. The results revealed that the Dutch adult population possesses insufficient data and strategic skills to use smart home devices to its full potential. Even less likely to benefit are the elderly and less educated; they showed the lowest levels of data and strategic skills. In addition, the elderly lack operational skills to use IoT home automation beneficially.

Keywords

Digital divide digital inequality digital skills home automation Internet of Things IoT performance test smart home

Introduction

The Internet of Things (IoT) is a network of interconnected ubiquitous everyday objects that gather and analyze data about their environment, communicate these data with other users and objects, and make autonomous data-based decisions (Van Deursen and Mossberger, 2018). One of the most common application domains of the IoT is home automation, wherein smart (home) devices such as thermostats, lighting, energy meters, and plugs are connected to the Internet (Tabassum et al., 2019). The possibility to remotely control and automate smart home devices holds the potential to increase energy efficiency and cost reduction, comfort and support, and security (Zimmermann et al., 2018). Users of IoT home automation could, for example, save energy and money by configuring their smart thermostat and lights to automatically turn off when leaving the house. To increase comfort, they could automate smart lighting to change light intensity and color based on the time of day. In terms of security, smart lights could, for example, be automated to follow a complex lighting schedule when on holiday. Aside from the potential of the smart home from an individual user perspective, it could also serve societal purposes because widespread adoption could, for example, contribute to reduced energy consumption (e.g. Ponce et al., 2016) or independent living for the aging population (e.g. Carnemolla, 2018).

The autonomous character of IoT (home automation) makes it seem that digital skills have become subordinate; “smart” devices work in the background and make decisions for us (Van Deursen and Mossberger, 2018). However, paradoxically, without the digital skills to operate smart home devices, to retrieve and interpret collected data, and to use it strategically, the potential benefits remain techno-utopian (De Boer et al., 2019; Van Deursen and Mossberger, 2018). To date, the promises of IoT home automation developers, that these interconnected devices will simplify and improve users’ daily lives and lead to many economic and societal benefits, have not been met because users experience difficulties and frustrations when using smart home devices (Lupton, 2020). Users tend to stick with simple uses of these devices (Hargreaves et al., 2018) relying on basic operational skills (e.g. configuring the smart thermostat manually using an app) because they lack the data skills to familiarize themselves with the continuously gathered data and the strategic skills to adapt to the decisions that are autonomously made by the IoT (Van Deursen et al., 2021). Users need to manage a growing amount of data that are collected without their interference and make sense of these data, for example, by determining what device collected certain data and what event(s) these data reflect. In addition, they must have an understanding of how they (and potential third parties involved) can use all these available data (Pins et al., 2021). Due to the smart devices’ autonomously made decisions based on data and algorithms, users are less actively involved in the decision-making process. However, for the IoT-system to make beneficial decisions, users should be able to configure automations, reflect on their consequences in light of their goals, and, if necessary, interrupt unwanted automated actions (Van Deursen and Mossberger, 2018). Some (less skilled) users are left out of this decision-making process entirely because the household member with the most proficient skills is often responsible for the setup of the smart devices. Therefore, the decisions made do not always fit (all) users’ needs (Geeng et al., 2019; Van der Zeeuw et al., 2020; Zeng et al., 2017).

Because digital skills are not distributed equally, some users are better able to benefit from using IoT home automation than others, contributing to existing digital inequalities. Highly skilled users are better equipped to retrieve, combine and use data to fine-tune the setup and reduce unnecessary (energy) use of the IoT. Furthermore, the better their smart devices are automated and working together, the more benefits users will experience from these configurations (Van der Zeeuw, 2021). For example, skilled users could make their smart thermostat and lighting respond to their location through geo-fencing or motion sensors. Across web developments, from a medium restricted to reading content (web 1.0), to a medium where users can create, store and share content themselves (web 2.0 and 3.0) and, eventually, introduce devices to the network (web 4.0) (Noh, 2015), users from a lower socioeconomic status and older users are likely to be disadvantaged as they possess the least developed digital skills (Van Deursen and Mossberger, 2018). Thus, users that could potentially benefit most from developments such as the IoT, are least likely to fully exploit it (De Boer et al., 2020). In effect, existing digital inequalities are expected to be exacerbated because the IoT’s continuous data collection and autonomous character put more stress on users’ digital skills compared with previous web developments (Van Deursen and Mossberger, 2018). As digital skills already play a major role in using the Internet, we expect that gender, age, and education—the most common and persistent determinants of digital skills (Scheerder et al., 2017)—remain relevant determinants of digital inequality in the IoT-context (De Boer et al., 2019). Thus, in this study, we aim to answer the following questions:

(1) What are the levels of digital skills to use IoT home automation among the Dutch adult population?

(2) How do gender, age, and education contribute to the level of these skills?

To answer these questions, we conducted a skill-based performance test in which participants were asked to perform tasks while using interconnected smart home devices in a virtual test environment (living room) specifically designed for this study.

Theoretical background

Skills to use IoT home automation

As argued in the introduction, the characteristic of the IoT to act autonomously might create the impression that digital skills are less relevant when using smart home devices. However, to manage the ambiguity of these devices’ (automated) actions and the vast amount of data collected, skills are imperative (Van Deursen and Mossberger, 2018). Users should be able to control smart home devices through an interface; they must understand the data provided to make informed decisions, and the automation of the smart home devices must be configured to simplify its use (Balta-Ozkan et al., 2013). Correspondingly, users of a smart home IoT-system need operational, data, and strategic skills.

Operational skills

Operational skills refer to a set of technical competencies. Without sufficient levels of operational skills, users are not able to perform actions on the Internet that involve skills as retrieving and interpreting information (and data) or using collected information strategically. Therefore, operational skills can be considered the starting point for using Internet technologies (Van Deursen et al., 2021; Van Dijk and Van Deursen, 2014). At first sight, operational skills appear to be less important in the IoT context due to its autonomous character (Van Deursen and Mossberger, 2018). Despite the need for operational skills being less frequent and less visible, they are vital to use the IoT beneficially because they are critical for the initial setup of smart devices and the control of settings (Van Deursen and Mossberger, 2018). Users must install and connect smart devices to a separate app or central platform, navigate through the app or platform’s structure, change settings, and construct algorithms (Van Deursen et al., 2021). By interconnecting devices to a central platform—a central hub (e.g. Amazon Alexa, Google Assistant, Apple HomeKit, and Athom Homey) or cloud-based integration (e.g. IFTTT) (Zeng et al., 2017)—more complex commands and configurations are enabled because devices can work together autonomously through rules (simplified algorithms) programmed by the user (e.g. Lin and Parkin, 2020; Van der Zeeuw et al., 2020; Zeng et al., 2017). Thus, those who possess the skills to interconnect devices have more opportunities to benefit from the IoT’s potential (Noura et al., 2019). However, these opportunities can only be seized upon with the operational skills to recognize and use buttons, change settings, and configure algorithms. Without these skills, users cannot retrieve data gathered by smart devices and use their functions and collected data strategically (Van der Zeeuw et al., 2020; Van Deursen et al., 2021).

Data skills

With the introduction of smart devices to the home network, data about users’ living conditions are gathered autonomously and continuously (Van Deursen and Mossberger, 2018). The complexity of these (big) data causes a shift from the traditional information Internet skills—involving searching, selecting, and evaluating information (Van Dijk and Van Deursen, 2014)—toward data skills focusing on retrieving and interpreting data (Prado and Marzal, 2013). Users of IoT home automation need to know what platform—app, website, or a central platform—can provide them with access to the data they are searching for. To identify and select these data in the IoT’s infinitive dataset, they require a clear idea of which data are relevant to their envisioned goal (Atzori et al., 2010; De Boer et al., 2020). Once the relevant data have been identified, users must be able to determine what format is the most suitable to present the data in an understandable manner. Although the raw data are often cleaned by the IoT-system itself, users must still choose a visualization that fits their goal (e.g. textually or graphically).

Subsequently, users must extract meaning from this data visualization and recognize the context in which the data were gathered (Barnaghi et al., 2012). For example, when users of smart home devices want to know how much energy the smart lights in the living room consume when turned on, the most suitable way to access the data is using a central platform to which all lights are connected. In this platform, they must select a relevant data visualization that only provides an overview of the energy consumption of (all) the smart lights placed in the living room instead of, for example, choosing a general overview all of the lights installed in their home or all the smart devices placed in the living room. From these data visualizations, users should be able to extract meaning and evaluate the quality of the data in terms of accuracy and completeness to ascertain the relevance of the conclusions drawn (Pins et al., 2021). They can do this by comparing the data with previously gathered measurements or to personal experiences (Van Deursen and Mossberger, 2018), for example, by observing the lights that turn on when being present in the living room and comparing this with the lights included in the data visualization of that instance.

Strategic skills

Strategic skills are necessary to make informed decisions and gain benefits from using smart home devices. With regard to previous web developments, strategic skills can be divided into four distinct steps of decision-making: orienting goals, taking required actions, making decisions, and implementing decisions (Van Deursen et al., 2021). To fit the context of IoT wherein the devices play a larger role in the decision-making process, De Boer et al. (2020) translated these steps to (1) orientating goals, (2) combining data with other information sources, (3) making data-driven decisions and assessing automated (or proposed) actions of the IoT, and (4) taking actions and reflecting upon the actions of the IoT.

In the first step regarding goal orientation, users determine what data and functions of the smart home can help to save energy, increase comfort, or increase safety. Then, they must translate these data and functions to specific goals concerning the potential benefits (Li et al., 2010). Users of smart home devices could, for example, set a goal to save energy by familiarizing themselves with the data and functions of their smart thermostat. In the second step, users combine data with prior knowledge, experience, and other information sources to observe the feasibility of the goal and what progress can be made toward it (De Boer et al., 2020; Filonik et al., 2013). When using a smart thermostat to save energy, users could, for example, notice that their heating schedule overlaps with moments wherein (structurally) nobody is home or awake. The third step involves using the observations to make substantiated data- and action-based decisions toward the goal (De Boer et al., 2020). When the house is still being heated, while the residents are away or to bed, they could, depending on their digital skills, choose to turn down the heat manually through the IoT platform, alter the heating schedule, or configure the thermostat to turn down the heat automatically when nobody is home (e.g. through geo-fencing) or when no movement is detected for a certain amount of time (e.g. through a motion sensor). Last, in the fourth step, users implement these decisions by taking actions toward the goal and reflecting on the actions of the smart home itself (Van Deursen and Mossberger, 2018). The users from the example must ask themselves whether the alterations made to the smart thermostat’s settings resulted in energy savings. If not, or when new insights are gained due to the continuous data stream, users can choose to repeat the steps of decision-making because it is an iterative process (Holmes et al., 2015).

Predictors of digital skills for IoT

Digital skills are a vital asset to gain benefits from using the Internet (of Things). However, previous research on digital inequality has shown that the possession of such skills differs among users (e.g. De Boer et al., 2020; Van Deursen and Van Dijk, 2011). To study these skill differences when using IoT home automation, we focus on three of the most commonly investigated determinants of digital skills: gender, age, and educational attainment level (Scheerder et al., 2017). To our knowledge, variables have not been studied through performance testing in the context of digital skills required for the smart home.

Findings regarding gender and digital skills have been inconsistent through web developments. Self-evaluations have found that men possess more skills required to use both the Internet and the IoT (Van Deursen et al., 2019; Van Deursen and Van Dijk, 2015; Wasserman and Richmond-Abbott, 2005). However, these gender differences might be the result of the fact that women are more prone to underestimate their digital skill levels than men (Hargittai and Shafer, 2006). In regard to digital housekeeping (e.g. setup, usage, and maintenance of the smart home technology), women tend to downplay their technical abilities, thereby implicitly conforming to the gender stereotype of using technology being masculine (Rode and Poole, 2018). Previous studies measuring digital skills through performance tests have found no gender differences regarding Internet skills (Van Deursen and Van Dijk, 2011) or the skills to use IoT in the form of an activity tracker (De Boer et al., 2020). Therefore, we expect to find no gender differences in skills to use IoT home automation either. This resulted in the following hypothesis:

H1. There are no differences in operational, data, and strategic skills to use IoT home automation between men and women.

Previous digital inequality research showed that older people experience more problems when using Internet technologies than younger people (Hunsaker and Hargittai, 2018) because they do not have the opportunity to familiarize themselves with these technologies from an early age (Blažič and Blažič, 2020; Van Dijk and Van Deursen, 2014). Young people’s lives are immersed with the use of technologies such as smartphones and tablets, while the elderly often stick to desktop computers, although tablets and smartphone use are being used more and more commonly (Wilson et al., 2022). This adoption is necessary because these technologies are the primary operation platforms of IoT home automation (Balta-Ozkan et al., 2013; Karadal and Abubakar, 2021). In addition, elderly Internet-technology users have less access to support and may be impaired by their mental and physical conditions (Brown et al., 2013; Rasheva-Yordanova et al., 2021). These findings led to the following hypothesis:

H2. Age contributes negatively to operational, data, and strategic skills to use IoT home automation.

A consistent determinant that is expected to exacerbate digital inequality using IoT home automation is the level of educational attainment. Shin et al. (2018) found that higher educated users of IoT home automation attach more importance to interconnecting smart devices, which allows more comprehensive uses and, therefore, opportunities to develop skills and benefit from the IoT’s potential (Lin and Parkin, 2020; Noura et al., 2019). Also, they have better (educational and financial) opportunities to keep up with technological advancements and access to modular devices compared with those who attained lower education (Balta-Ozkan et al., 2013). Thus, adopters of IoT home automation are generally higher educated (Shin et al., 2018), which provides them with early opportunities to develop digital skills to make use of the potential of IoT home automation (Rasheva-Yordanova et al., 2021; Van der Zeeuw et al., 2020). Higher educated users of Internet technologies are known to possess more advanced digital skills (Scheerder et al., 2017). We expect that this holds true for IoT home automation because extracting meaning from data and comprehending the context in which they were gathered cannot be mimicked by technology (Flores et al., 2020). Thus, we hypothesize the following:

H3. Education contributes positively to operational, data, and strategic skills to use IoT home automation.

Methods

Sample

In this study, we sought 100 people who were willing to participate in a performance test measuring skills required for IoT home automation. We recruited participants by distributing an online registration questionnaire using PanelClix, a Dutch professional market research organization, and via public posts on social media. After registration, potential participants were selected through quota sampling for gender, age (18+ years), and educational attainment (low-middle-high). Further inclusion criteria were that they had access to a laptop or PC with a working Internet connection and that they had no prior experience with the Homey app, the IoT platform used in this study. In total, 243 individuals completed the online registration (183 via PanelClix; 60 via social media post). From these registrations, 101 individuals were invited via phone to participate, and a time for the research session was scheduled. These participants received a confirmation email including the agreed-upon time and a link to the video conversation. A reminder email was sent 2 or 3 days prior to the session. When completing the 1-hour research session, participants were rewarded with 50 euros.

Due to technical problems during data collection, two participants were excluded from the data analysis. Thus, this study analyzed the data of 99 participants. An overview of the distribution of participants by gender, age, and education is shown in Table 1.

Table 1.

Distribution of participants by gender, age, and education (n = 99).

Demographic	Value, n
Demographic	Operational skills	Data skills	Strategic skills
Gender
Male	48	48	47
Female	51	51	51
Age
18–29	24	24	24
30–39	22	22	22
40–54	27	27	26
55–80	26	26	26
Education
Low	32	32	31
Middle	33	33	33
High	34	34	34
Missing	0	0	1

Measures

Gender, age, and education

All participants completed an online registration to participate in the performance test. During registration, questions regarding gender, age (in years), and level of educational attainment were recorded. All participants in this study identified themselves as either male or female. To conduct the analyses regarding the contribution of age to digital skills, we categorized the participants into age groups of 18–29, 30–39, 40–55, and 55–80 years old. With regard to the participants’ level of educational attainment, we made a distinction between lower, middle, and higher education following the Dutch school system. Seven of the answer options were categorized as lower education, four as middle education and three as higher education.

Virtual test environment

The performance tests used a virtual test environment using InVision (brand) that was created for this study. This test consisted of a simplified version of the Homey app, a central platform developed by Athom, on the left side of the participants’ screen. In this platform, the participants could add smart home devices and manually operate them; retrieve and interpret data gathered by these devices; and set them up to operate autonomously. We chose the Homey app as a platform because of its ability to directly interconnect smart devices of different brands and wireless protocols (e.g. Wi-Fi, Bluetooth, Zigbee, and Z-wave), whereas more conventional platforms from, for example, Google and Apple can only connect with devices of specific brands running on specific protocols (Noura et al., 2019). On the right side of the screen, the room where the devices were located was visualized. These smart home devices included a motion sensor, a smart thermostat, smart plugs (2), smart lighting (2), and a robot vacuum cleaner. They were chosen based on their popularity. Specifically, devices regarding energy management (15%) and comfort and lighting (14%) are the most popular devices among Dutch households (Statista Market Forecast, 2021). Whenever user actions were initiated by the participants, feedback was provided by the devices in the visual representation of the room. A screenshot of the virtual test environment (see Figure 1) shows how the floor lamp in the room is lit after turning it on in the platform by clicking the “Dimmable Floor Lamp” button.

Figure 1.

Screenshot of the virtual test environment after turning on the floor lamp in the platform by clicking the “Dimmable Floor Lamp” button.

Tasks

The tasks that the participants had to complete were designed to measure the different facets of operational, data, and strategic skills by using the functions, data, and automations of the smart home devices in the virtual IoT test environment. The tasks reflect situations of everyday use of common IoT applications among the Dutch adult population, namely the use of the smart thermostat and smart lighting (Van Deursen et al., 2019). Tasks featuring the smart thermostat, for example, revolve around changing the heating program and extracting meaning from the data the thermostat generates. Examples of everyday uses of smart lighting are manually operating specific lights (e.g. turning it on and dimming it), retrieving their energy consumption, and configuring them to operate autonomously (e.g. following a time schedule). All tasks had a specific correct action or answer that was administered by recording screen actions and/or filling out the answer in a separate tab in the virtual test environment. The participants completed a total of 16 tasks, which can be found in Appendix 1. One task exclusively measured operational skills, and seven tasks primarily focused on data skills. Three tasks consisted of subtasks measuring operational and data skills independently. Therefore, data skill subtasks could still be completed when unsuccessful in completing operational skill subtasks. Also, there were five tasks that measured strategic skills.

In the operational skill tasks, participants were asked to, for example, add a specific type of light bulb to the platform (Task 1). Data skill tasks involved retrieving and interpreting data visualizations of devices’ energy consumption, including the visualization of lighting over a prolonged period of time (Tasks 8, 9, and 10). In the tasks that combined operational and data skills, participants had to, for example, turn on a light as an operational skill subtask before retrieving and interpreting the data of its current energy consumption for the data skill subtask (Task 3). Finally, in the strategic skill tasks, participants were asked to describe settings that were set up to reach the goal(s) of increasing sustainability comfort and/or safety autonomously (e.g. describe the preset rule configurations in Figure 2 that will automatically turn on the dimmable floor lamp in the living room before sunrise and movement is detected; Task 7) and to verbalize how they would set up smart home devices to make them operate autonomously to match a predetermined goal (e.g. what rules they should configure to automatically turn down/up the thermostat when leaving/entering the house; Tasks 14 and 15). All tasks were pilot tested with seven participants of different ages and educational levels to ensure comprehensibility and applicability.

Figure 2.

One of the preset rule configurations explained by the participants in Task 7.

Procedure

Performance tests were conducted from 1 April through 23 June 2021. Due to the COVID-19 pandemic, these tests were conducted online through a video conversation using Microsoft Teams. Prior to the performance test, all participants completed the online registration questionnaire.

To participate in the performance test, the participants joined the video conversation using their laptop or PC. After welcoming them into the conversation, they were asked to share their screen so the experiment leader could follow their progress when executing the tasks in the virtual test environment. Once the participants shared their screen, they received a link via the video conversation’s chat function leading to the virtual test environment. The participants opened the link in their preferred browser.

To become acquainted with the virtual test environment, the participants were led through the functions of the central platform in the test environment before they could start with the tasks of the performance test. When there were no questions about the virtual test environment, they continued to the tasks. The participants themselves decided when they had finished a task or when they wanted to give up. However, participants were gently asked to move on to the next task after a time limit, which was determined from the pilot tests, had passed. The order of completing the tasks was the same for all participants. Appendix 1 shows the order and the time limits of the tasks.

Data analysis

The levels of the participants’ skills to use IoT home automation were determined by focusing on successful task completion. We scored successful task completion by assigning one point for each successfully completed task. The sum scores determined the participants’ skill levels. We made a distinction between the successful completion of operational, data, and strategic skill tasks. Appendix 2 shows the distribution of tasks across these skills.

We conducted one-way analyses of variance (ANOVAs) with the task completion score as the dependent variable to identify factors contributing to the level of operational, data, and strategic skills. The factors considered in this study were gender, age, and educational attainment level. In addition, a Bonferroni post hoc test was conducted to determine whether and how digital skills to use IoT home automation differed across genders, age groups and educational attainment levels.

Results

Levels of digital skills for IoT home automation

As shown in Table 2, on average, participants completed 78% of the operational skill tasks. Most participants could add a dimmable light bulb to the central platform (Task 1). However, they then struggled with the operational skill task to dim it (Task 4.1). Regarding the data skill tasks, participants were less successful. On average, 67% of these tasks were completed successfully. Participants were rather proficient at reading the real-time energy consumption of the light that they were operating at that moment (Tasks 2, 3.2, and 5). However, they experienced problems during the execution of two tasks where they had to find a deviation in a graph visualizing the energy consumption over a time period of a week (Task 11.1) and come up with a possible reason for the deviation in that graph (Task 11.2). For the strategic skill tasks, 37% were completed correctly. Many participants could propose two triggers that could be used to strategically automate the dimmable lamp through separate rules—simplified algorithms—(e.g. When it is dark outside [trigger], then the lamp turns on [automation]) (Task 6). The strategic skill task that participants struggled with the most was the task where they had to propose four rules that would turn on and off a light according to the graph that visualized the use of that light over a time period of a week (Task 12). Appendix 3 provides an overview of the percentages of successful task completion per operational, data, and strategic task.

Table 2.

Overview of successful task completion.

Digital skill	Successful task completion		%
Digital skill	M (SD)	Min–max	%
Operational skills	3.12 (1.15)	0–4	78
Data skills	7.38 (2.42)	1–11	67
Strategic skills	2.93 (2.13)	0–8	37

SD: standard deviation.

Skill differences for gender, age, and education

Table 3 shows the operational, data, and strategic skills task completion scores distributed by gender, age, and education. Regarding operational skills to use smart home devices, there was a significant difference between the age groups, F(3, 95) = 7.42, p < .001. A Bonferroni post hoc test showed that the eldest participants (55–80) scored significantly lower than the two youngest age groups (18–29 and 30–39). Appendix 3 shows that the task they struggled with the most was dimming the light in the living room (Task 4.1). No significant differences in task completion were found when comparing the age group of 40–54 years with the other age groups. An overview of the Bonferroni post hoc tests is shown in Table 3. Also, no significant differences were found for gender, F(1, 97) = 0.81, p = .37, or educational attainment levels, F(2, 96) = 1.14, p = .32.

Table 3.

Post hoc tests (Bonferroni with 5% significance level) for operational, data, and strategic skills completion scores over gender, age, and education (M[SD]).

	Gender		Age				Education
	Male	Female	18–29	30–39	40–54	55–80	Low	Middle	High
Operational skills	3.23 (1.06)^a	3.02 (1.24)^a	3.71 (.62)^a	3.50 (.80)^a	2.96 (1.06)^{a, b}	2.42 (1.47)^b	2.94 (1.22)^a	3.06 (1.17)^a	3.35 (1.07)^a
Data skills	7.52 (2.21)^a	7.25 (2.61)^a	8.63 (1.74)^a	8.09 (2.09)^a	7.11 (2.10)^{a, b}	5.92 (2.76)^b	6.53 (2.55)^a	7.48 (2.24)^{a, b}	8.09 (2.26)^b
Strategic skills	3.00 (2.24)^a	2.86 (2.04)^a	4.13 (2.03)^a	3.45 (2.09)^a	2.73 (2.05)^{a, b}	1.58 (1.53)^b	2.06 (1.59)^a	2.94 (2.19)^{a, b}	3.71 (2.24)^b

SD: standard deviation.

For each skill, within each row, means with noncommon superscripts are significantly different.

For data skill tasks, significant differences in successful completion were found between the age groups, F(3, 95) = 7.17, p < .001, and educational attainment levels, F(2, 96) = 3.66, p = .03. Again, a Bonferroni post hoc test showed that the eldest participants (55–80) scored significantly lower than the two youngest age groups (18–29 and 30–39 years). No significant differences in task completion were found when comparing those aged 40–44 years with the other age groups. Regarding educational attainment level, less educated participants scored significantly lower compared with higher educated participants. As shown in Appendix 3, most of the elderly and less educated participants could not detect the deviation in the pattern of the energy consumption of a light over a time period of a week that was visualized in a graph (Task 11). Table 3 shows an overview of the Bonferroni post hoc tests. No significant differences were found between genders, F(1, 97) = 0.30, p = .59.

Regarding the strategic skill tasks, significant differences in successful task completion were found between the age groups, F(3, 94) = 7.97, p < .001, and educational attainment levels, F(2, 95) = 5.26, p < .01. By conducting a Bonferroni post hoc test, we found that the eldest participants scored significantly lower than the participants from the two youngest age groups. Participants belonging to the 40- to 55-year-old age group did not score significantly differently on the skill tasks compared with the participants from the other age groups. Regarding educational attainment, less educated participants scored significantly lower than higher educated participants. Appendix 3 shows that both the elderly and the less educated participants experienced the most problems constructing rules (simplified algorithms) when managing smart home devices (Tasks 12 and 14). An overview of the Bonferroni post hoc test is shown in Table 3. No significant gender differences were found in the completion scores of the strategic skill tasks, F(1, 96) = 0.10, p = .75.

Hypotheses

Hypothesis 1—that there are no differences in operational, data, and strategic skills to use IoT home automation between men and women—is supported. Gender did not contribute significantly to any of these digital skills.

Hypothesis 2—that age contributes negatively to operational, data, and strategic skills to use IoT home automation—is partly supported. The participants of the oldest age group were outperformed by the participants from the two youngest age groups. The two youngest groups did not differ significantly.

Hypothesis 3—that education contributes positively to operational, data, and strategic skills to use IoT home automation—is partly supported. The less educated participants performed poorly compared with the higher educated participants with regard to the data and strategic skill tasks.

Discussion

Main findings

The benefits that can be obtained from IoT home automation are promising and are expected to become more pronounced as smart home technologies continue to develop. In addition to the individual benefits of saving energy and costs, and increasing comfort, support, and security (Zimmermann et al., 2018), these technologies could serve societal purposes such as overcoming the problem of overloading the power grid (e.g. Ponce et al., 2016) and supporting the aging population to live independently (e.g. Carnemolla, 2018). However, to benefit from IoT home automation, users require the operational, data, and strategic skills to control and automate smart devices, retrieve and understand collected data, and make informed decisions (Balta-Ozkan et al., 2013).

In light of the first research question, the results of our performance tests suggest that the Dutch adult population is not sufficiently skilled to use the IoT home automation to its full potential. Overall, they possess sufficient operational tasks to set up and operate smart home devices. However, to a large extent, they lack data skills to retrieve and interpret the data gathered by the devices. They struggled even more when having to use the IoT settings and data in a goal-oriented manner because only approximately one-third of the strategic skill tasks were completed successfully. This result might explain why users’ interest in IoT home automation rarely translates into pro-environmental behavior change (Snow et al., 2013). Users of smart thermostats, for example, do not fully use the thermostats’ potential because they are unable to set up and use advanced functions such as the automatic adjustment of energy consumption patterns (e.g. through GPS or motion sensors) (Ponce et al., 2019). In contrast, underdeveloped skills could lead to increased energy consumption due to inappropriate application of prewarming rooms or simply due to the inefficient adoption of energy-consuming smart home devices (Hargreaves et al., 2018). Therefore, these skill-related difficulties could possibly even have negative implications (e.g. unnecessarily energy costs).

In light of the second research question, results indicate that older individuals possess the least developed digital skills to use IoT home automation, comparable to their struggles with previous web developments (Hunsaker and Hargittai, 2018). Therefore, the promise that IoT home automation will simplify and improve users’ daily lives does not hold true, at least for the time being. This result is unfortunate because it could potentially enable older individuals to live independently at home as they progress into old age. Daily tasks and home maintenance could be automated using smart home devices, and data gathered by the IoT could be used to detect possible emergencies (Carnemolla, 2018). Interestingly, when considering 40- to 55-year-olds, no significant difference from the youngest age group was observed, which implies that, even though they did not have the opportunity to familiarize themselves with Internet technologies from an early age, digital skills to use IoT home automation can be acquired later in life as well.

Furthermore, our study shows that less educated people are less skilled than higher educated people with regard to data-related and strategic uses of IoT home automation. This suggests that operating smart home devices does not depend on level of educational attainment. However, retrieving and interpreting IoT data are more challenging for less educated users. Although IoT home automation can be used for simple purposes by all users, less educated users do not fully benefit due to deficient data and strategic skills. Thus, less educated users use smart home devices in a less sustainable way, which leads to lower energy savings and thus suboptimal financial benefits compared with their higher educated peers (Van Deursen and Van Dijk, 2014). The lack of data and strategic skills could also affect them financially because they have more difficulty reading the data and understanding the consequences involved when choosing between energy providers, prices, and new smart home devices to adopt (Bray et al., 2022).

The general conclusion is that the digital skills of the Dutch adult population to use IoT home automation need improvement to benefit from its full potential. In agreement with the proposed hypotheses, this conclusion applies even more to the older and less educated population. Subsequently, differences in digital skill levels could potentially exacerbate digital inequality as IoT home automation is expected to become integrated into our daily lives more widely (Statista Market Forecast, 2021). In addition, the older and less educated populations cannot optimally benefit from these integrations because they are disproportionately affected by the fact that their limited IoT use causes them to be underrepresented in the data exploited by third parties (e.g. to determine energy prices or insurance costs) (Lupton, 2020).

Policy implications

Policies should address the lack of digital skills from both supply and demand perspectives. Suppliers should consider designing their smart devices and IoT platforms in an understandable way that fits the needs of the individual user, for example, through an adaptable design wherein functions and data visualizations can be adapted to individual needs and desires (e.g. changing interface appearance by ordering functions according to the frequency they’re used, removing the clutter of unused data, or simply changing font size) (Filonik et al., 2013) or through explainable artificial intelligence (AI) so that users gain insights into the IoT’s decision-making process (e.g. by visualizing the algorithms involved through a decision tree) (Weitz et al., 2021). From the demand perspective, formal and informal support should be facilitated considering that digital skills can be developed at a later age (Van Dijk and Van Deursen, 2014) and that users of IoT home automation can be supported by co-users’ skills (Van der Zeeuw et al., 2020). Education should account for digital skills by raising awareness of the data gathered by the IoT and its possible uses by users and third parties. Furthermore, attention should be paid to the retrieval, interpretation, and strategic use of the collected data. Besides formal education, public support could be provided through informal initiatives such as help desks or peer support at community centers that users can consult when requiring assistance to use smart devices and collected data beneficially. This support is particularly important to the populations that are disadvantaged the most because they are the ones with the least supportive environments (Van Deursen and Mossberger, 2018).

Limitations and future research

This study provides new insights into digital skills that are required to use IoT home automation. Due to the COVID-19 pandemic, performance tests were conducted in a virtual test environment. Thus, the participants required access to a laptop or PC and a certain level of digital skills to participate. This fact could imply that the people who are having the most trouble accessing and using Internet technologies were excluded from this study, and digital inequality to use IoT home automation could be even larger than shown. In addition, because of using a virtual test environment, physical operational skills such as installing or replacing hardware (e.g. by following the app’s instruction to manually turn the smart light on and off repeatedly when adding it to the app) and resetting it (e.g. by pushing a needle in the reset hole or by unplugging the device) could not be tested. In future research, we therefore recommend performing performance tests in a setting wherein participants use smart home devices in real life.

To investigate the digital skills of smart home users, we used tasks involving devices primarily focusing on energy management and lighting. We have no reason to assume that the distribution of digital skills would differ when using other types of smart devices since we’ve designed the tasks around the different facets of operational, data, and strategic skills, rather than device-specific functionalities. Recent findings on health-related IoT support this assumption (De Boer et al., 2020, 2022). Furthermore, we used a hub as the central platform, considering that the interconnectivity of different devices is a key aspect of the IoT. However, most users still depend on separate apps to operate their smart home devices (Balta-Ozkan et al., 2013; Noura et al., 2019). This fact makes goal-related use of smart home devices even more complex in real life because managing devices and retrieving data occurs in multiple datasets. Also, there are fewer means to act toward a goal because devices cannot cooperate (Balta-Ozkan et al., 2013; Van der Zeeuw et al., 2020). The complexity of strategically using smart home devices separately, together with the notion of interconnecting smart devices being more important to higher educated users compared with their less educated peers (Shin et al., 2018), could imply that less educated IoT users have fewer opportunities to benefit from IoT home automation. Also, the challenge of limited standardized compatibility functions of smart home devices from different brands has proven particularly difficult for elderly users as they struggle to adapt to the disparate—occasionally contrasting—propositions and actions of separate devices (Ma et al., 2022). In the realm of digital inequality, it could therefore be of added value to consider studying the skills necessary to integrate smart devices and to use them simultaneously.

This study provides a general overview of the operational, data, and strategic skill levels to use IoT home automation among Dutch citizens. However, further research is required to identify the different facets of these digital skills that users of IoT home automation flounder. Also, skills other than operational, data, and strategic skills should be considered. Skills that could be worth investigating are collaboration skills. In this study, we focused on one user using smart home devices. However, when a household consists of multiple members, users must work together with the devices and collaborate with other household members to make their smart home work for all of them. They must consider that different household members have different attitudes, skills, and preferences (Kraemer et al., 2019). These differences influence their ways of using smart home devices, which could cause power imbalances between household members (Geeng et al., 2019; Van der Zeeuw et al., 2020; Zeng et al., 2017). To study these dynamics between household members, it could furthermore be of interest to study the digital skills of children and adolescents as they often are not considered enough as individuals with unique skills and needs (Sun et al., 2021).

Other skills that could be investigated are privacy skills. Smart home devices are privacy-sensitive because they continuously gather data regarding users and their home environment. They can, for example, register information about household members’ whereabouts and activities. Therefore, users of IoT home automation should have an idea of what data are available to who to make well informed decisions regarding their interpersonal privacy (e.g. whether to invite their housekeeper or guests to the IoT’s operating platform) (Geeng et al., 2019). Also, they should manage passwords securely and keep the IoT system updated to secure their IoT system from potential breaches and protect their privacy (Lee, 2020). Last, they must have an understanding of the implications of sharing data with third parties (e.g. energy companies having access to the data gathered by their smart thermostat) (Pins et al., 2021). Insufficient privacy skills to protect personal data or mitigate risks make users of IoT home automation susceptible to people with malicious intentions and financial exploitation by third parties (Van Deursen and Mossberger, 2018).

This study focused on the second-level digital divide, measuring differences in users’ digital skills. However, unlike for general Internet access, access to IoT home automation is still far from saturated in Western countries such as the Netherlands (Lutz, 2019). Therefore, we suggest future studies investigating digital inequalities regarding IoT (home) automation to consider the first-level digital divide covering motivational and material access as well.

Footnotes

Appendix 1

Appendix 2

Appendix 3 Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Dutch research council (grant number: 452-17-001).

ORCID iDs

Pia S de Boer

Alexander JAM van Deursen

Author biographies

Pia S de Boer is a PhD candidate at the University of Twente at the Department of Communication Science. She is currently involved in a project on Internet-of-Things skills. She measures these skills in performance tests in which people actually use IoT devices.

Alexander JAM van Deursen is Professor and Chair of the Department of Communication Science at the University of Twente in the Netherlands. Most of his research focuses on three lines of research with the overarching theme of digital inequality. He maps barriers of online engagement and explains differences in outcomes from Internet (of Things) use. Research projects Alexander leads are Digital Inequality in the Netherlands, twenty-first-century digital skills in the creative industry, inequality in Internet of Things skills and studying digital inequality in the social context of the home. Alexander holds Visiting Scholar positions at the London School of Economic and Political Science and Arizona State University.

Thomas JL van Rompay is an associate professor at the Department of Communication Science of the University of Twente and a fellow at the UT’s DesignLab. He has a background in cognitive psychology. He studies design experience from an embodied cognition perspective, investigating how design communicates meaning and affect. His current research projects take place on the threshold of design and psychology.

References

Atzori

Iera

Morabito

(2010) The Internet of Things: a survey. Computer Networks 54(15): 2787–2805.

Balta-Ozkan

Davidson

Bicket

, et al. (2013) Social barriers to the adoption of smart homes. Energy Policy 63: 363–374.

Barnaghi

Wang

Henson

, et al. (2012) Semantics for the internet of things: early progress and back to the future. International Journal on Semantic Web and Information Systems 8(1): 1–21.

Blažič

(2020) Overcoming the digital divide with a modern approach to learning digital skills for the elderly adults. Education and Information Technologies 25(1): 259–279.

Bray

Mejía Montero

Ford

(2022) Skills deployment for a “just” net zero energy transition. Environmental Innovation and Societal Transitions 42: 395–410.

Brown

Coughlan

Lawson

, et al. (2013) Exploring interpretations of data from the Internet of Things in the Home. Interacting with Computers 25(3): 204–217.

Carnemolla

(2018) Ageing in place and the internet of things—how smart home technologies, the built environment and caregiving intersect. Visualization in Engineering 6(1): 1–16.

De Boer

Van Deursen

AJAM

Van Rompay

TJL

(2019) Accepting the Internet-of-Things in our homes: the role of user skills. Telematics and Informatics 36: 147–156.

De Boer

Van Deursen

AJAM

Van Rompay

TJL

(2020) Internet-of-Things skills among the general population: task-based performance test using activity trackers. JMIR Human Factors 7(4): e22532.

10.

De Boer

Van Deursen

AJAM

Van Rompay

TJL

(2022) Health on the move—can we keep up? Activity tracker performance test to measure data and strategic skills. International Journal of Human-Computer Interaction. Epub ahead of print 27 July. DOI: 10.1080/10447318.2022.2097367.

11.

Filonik

Medland

Foth

, et al. (2013) A customisable dashboard display for environmental performance visualisations. In: Berkovsky

Freyne

(eds) Persuasive Technology. PERSUASIVE 2013 (Lecture Notes in Computer Science). Berlin: Springer, pp. 51–62.

12.

Flores

(2020) Human cyber-physical systems: a skill-based correlation between humans and machines. In: 16th IEEE international conference on automation science and engineering (CASE), Hong Kong, China, 20–21 August, pp. 1313–1318. New York: IEEE.

13.

Geeng

Roesner

Allen

(2019) Who’s in control? Interactions in multi-user smart homes. In: Proceedings of the 2019 CHI conference on human factors in computing systems, Glasgow, 4–9 May, pp. 1–13. New York: ACM.

14.

Hargittai

Shafer

(2006) Differences in actual and perceived online skills: the role of gender. Social Science Quarterly 87(2): 432–448.

15.

Hargreaves

Wilson

Hauxwell-Baldwin

(2018) Learning to live in a smart home. Building Research and Information 46(1): 127–139.

16.

Holmes

Wieman

Bonn

(2015) Teaching critical thinking. Proceedings of the National Academy of Sciences of the United States of America 112(36): 11199–11204.

17.

Hunsaker

Hargittai

(2018) A review of Internet use among older adults. New Media & Society 20(10): 3937–3954.

18.

Karadal

Abubakar

(2021) Internet of things skills and needs satisfaction: do generational cohorts’ variations matter? Online Information Review 45(5): 898–911.

19.

Kraemer

Flechais

Webb

(2019) Exploring communal technology use in the home. In: Proceedings of the halfway to the future symposium (HTTF), Nottingham, 19–20 November, pp. 1–8. New York: ACM.

20.

Lee

(2020) Home IoT resistance: extended privacy and vulnerability perspective. Telematics and Informatics 49: 1–12.

21.

Dey

Forlizzi

(2010) A stage-based model of personal informatics systems. In: SIGCHI conference on human factors in computing systems, Atlanta, GA, 10–15 April, pp. 557–566. New York: ACM.

22.

Lin

Parkin

(2020) Transferability of privacy-related behaviours to shared smart home assistant devices. In: 7th international conference on Internet of Things: systems, management and security (IOTSMS 2020), Paris, 14–16 December, pp. 1–8. New York: IEEE.

23.

Lupton

(2020) The Internet of Things: social dimensions. Sociology Compass 14(4): e12770.

24.

Lutz

(2019) Digital inequalities in the age of artificial intelligence and big data. Human Behavior and Emerging Technologies 1(2): 141–148.

25.

Guerra-Santin

Mohammadi

(2022) Smart home modification design strategies for ageing in place: a systematic review. Journal of Housing and the Built Environment 37: 625–651.

26.

Noh

(2015) Imagining Library 4.0: creating a model for future libraries. The Journal of Academic Librarianship 41(6): 786–797.

27.

Noura

Atiquzzaman

Gaedke

(2019) Interoperability in Internet of Things: taxonomies and open challenges. Mobile Networks and Applications 24: 796–809.

28.

Pins

Jakobi

Boden

, et al. (2021) Alexa, we need to talk: a data literacy approach on voice assistants. In: Designing interactive systems conference 2021 (DIS ’21), Virtual, 28 June–2 July, pp. 495–507. New York: ACM.

29.

Ponce

Peffer

Molina

(2019) Usability perceptions and beliefs about smart thermostats by chi-square test, signal detection theory, and fuzzy detection theory in regions of Mexico. Frontiers in Energy 13(3): 522–538.

30.

Ponce

Polasko

Molina

(2016) End user perceptions toward smart grid technology: acceptance, adoption, risks, and trust. Renewable and Sustainable Energy Reviews 60: 587–598.

31.

Prado

Marzal

MÁ

(2013) Incorporating data literacy into information literacy programs: core competencies and contents. Libri 63(2): 123–134.

32.

Rasheva-Yordanova

Kostadinova

Toleva-Stoimenova

(2021) IoT: a new dimension of digital divide. In: 14th annual international conference of education, research and innovation, Virtual, 8–9 November, pp. 1811–1816. Valencia: IATED.

33.

Rode

Poole

(2018) Putting the gender back in digital housekeeping. In: Proceedings of the 4th conference on gender & IT, Heilbronn, 14–15 May, pp. 79–90. New York: ACM.

34.

Scheerder

Van Deursen

AJAM

Van Dijk

JAGM

(2017) Determinants of Internet skills, uses and outcomes. A systematic review of the second- and third-level digital divide. Telematics and Informatics 34(8): 1607–1624.

35.

Shin

Park

Lee

(2018) Who will be smart home users? An analysis of adoption and diffusion of smart homes. Technological Forecasting and Social Change 134: 246–253.

36.

Snow

Buys

Roe

, et al. (2013) Curiosity to cupboard: self reported disengagement with energy use feedback over time. In: Proceedings of the 25th Australian computer-human interaction conference: augmentation, application, innovation, collaboration, OzCHI 2013, Adelaide, SA, Australia, 25–29 November, pp. 245–254. New York: ACM.

37.

Statista Market Forecast (2021) Smart Home—Netherlands. Available at: https://www.statista.com/outlook/dmo/smart-home/netherlands (accessed 4 January 2022).

38.

Sun

Zou

Radesky

, et al. (2021) Child safety in the smart home: parents' perceptions, needs, and mitigation strategies. Proceedings of the ACM on Human-Computer Interaction 5(CSCW2): 1–41.

39.

Tabassum

Kosinski

Lipford

(2019) ‘I don’t own the data’: end user perceptions of smart home device data practices and risks. In: Fifteenth symposium on usable privacy and security (SOUPS 2019), Santa Clara, CA, 12–13 August, pp. 435–450. Berkeley, CA: USENIX Association.

40.

van der Zeeuw

(2021) IoT as simple as Do Re Mi: A micro-figurational approach to the social context of Internet of Things skills and digital inequalities. University of Twente. Available at: https://doi.org/10.3990/1.9789036552844

41.

Van der Zeeuw

Van Deursen

AJAM

Jansen

(2020) How to apply IoT skills at home: inequalities in cultural repertoires and its interdependency chains. Poetics 83: 101486.

42.

Van Deursen

AJAM

Mossberger

(2018) Any thing for anyone? A new digital divide in Internet-of-Things skills. Policy & Internet 10(2): 122–140.

43.

Van Deursen

AJAM

Van Dijk

JAGM

(2011) Internet skills and the digital divide. New Media & Society 13(6): 893–911.

44.

Van Deursen

AJAM

Van Dijk

JAGM

(2014) The digital divide shifts to differences in usage. New Media & Society 16(3): 507–526.

45.

Van Deursen

AJAM

Van Dijk

JAGM

(2015) Toward a multifaceted model of internet access for understanding digital divides: an empirical investigation. The Information Society 31(5): 379–391.

46.

Van Deursen

AJAM

Van der Zeeuw

De Boer

, et al. (2019) Digital inequalities in the Internet of Things: differences in attitudes, material access, skills, and usage. Information, Communication & Society. Epub ahead of print 27 July. DOI: 10.1080/1369118X.2019.1646777.

47.

Van Deursen

AJAM

Van Der Zeeuw

De Boer

, et al. (2021) Development and validation of the Internet of Things Skills Scale (IoTSS). Information, Communication & Society. Epub ahead of print 25 March. DOI: 10.1080/1369118X.2021.1900320.

48.

Van Dijk

JAGM

Van Deursen

AJAM

(2014) Digital Skills, Unlocking the Information Society. New York: Palgrave Macmillan.

49.

Wasserman

Richmond-Abbott

(2005) Gender and the Internet: causes of variation in access, level, and scope of use. Social Science Quarterly 86(1): 252–270.

50.

Weitz

Schiller

Schlagowski

, et al. (2021) “Let me explain!”: exploring the potential of virtual agents in explainable AI interaction design. Journal on Multimodal User Interfaces 15(2): 87–98.

51.

Wilson

Byrne

Rodgers

, et al. (2022) A systematic review of smartphone and tablet use by older adults with and without cognitive impairment. Innovation in Aging 6(2): 1–19.

52.

Zeng

Mare

Roesner

, et al. (2017) End user security and privacy concerns with smart homes end user security & privacy concerns with smart homes. In: Thirteenth symposium on usable privacy and security (SOUPS 2017), Santa Clara, CA, 12–14 July, pp. 65–80. Berkeley, CA: USENIX Association.

53.

Zimmermann

Bennighof

Edel

, et al. (2018) ‘Home, smart home’—exploring end users’ mental models of smart homes. In: Mensch und Computer—Workshopband, Dresden, 2–5 September, pp. 407–417. Bonn: Gesellschaft für Informatik e.V.