Sage Journals: Discover world-class research

Abstract

Objective

To examine changes in the uptake of the National Health Service (NHS) COVID-19 proximity (contact) tracing application (‘app’) over one year, amongst smartphone users in England and Wales.

Methods

We conducted a longitudinal survey between October 2020 and September 2021, amongst an online panel representative of smartphone users aged 18–79 and a purposeful sample from six of the largest minority ethnic groups. We fitted pooled logistic regression models to examine factors associated with app installation and a longitudinal logistic regression model to estimate factors associated with installing/uninstalling the app over time.

Results

Around 50% of respondents had the app installed at each time point. The majority of installations took place soon after its launch. The key reason for installing at launch was ‘civic, public or social responsibility’. Amongst those who installed the app later, it was ‘needed to scan NHS QR code’. Uptake was higher amongst individuals who considered themselves vulnerable to COVID-19 or were concerned about the risk COVID-19 posed, were more highly educated, of White ethnicity, and who reported higher levels of trust in government information. Factors associated with installing the app over time included becoming more concerned about the risk COVID-19 poses to the country, or perceiving that the crisis in their local area had worsened.

Conclusions

Despite changes in pandemic response and case numbers, app installation in England and Wales remained relatively stable after launch. If governments wish to increase app installation and use rates in future pandemics, they need to highlight those app features that encourage engagement, and take related action to allay privacy concerns and improve trust in government information sharing.

Keywords

Contact tracing COVID-19 mHealth public health

Introduction

Traditionally, contact tracing involves public health authorities manually identifying individuals who have come into contact with a source of infection and alerting them to the risk of infection so appropriate action to stem the spread of infection (such as testing, quarantine, and treatment) can be taken. Implemented effectively, contact tracing reduces infections in the population. Limitations of manual contact tracing include reliance on cases to recall close contacts, limited scalability, and the need for effective public health departments at the local level.¹ Digital proximity (more commonly referred to as ‘contact’) tracing applications (‘apps’) have been proposed as an adjunct to manual contact tracing,² and were recommended by the European Centre for Disease Prevention and Control in response to the SARS-CoV-2 (Coronavirus) pandemic.³ Several health authorities around the world developed apps as part of their public health response.

Most apps utilise Bluetooth technology on smartphones to estimate the distance and duration of contact between two users, and use this information to determine exposure risk.³ Depending on where users’ information is stored, apps are either: centralised, with contact data processed centrally, usually by a health authority, or decentralised, with information kept on individual devices. Public health authorities in England and Wales launched the decentralised National Health Service (NHS) COVID-19 application (hereafter ‘the app’) in the autumn of 2020 (see Box 1). Box 1.

Overview of the NHS COVID-19 app's development and main function.

Although evidence on the impact of apps on controlling disease transmission was scarce,⁴ modelling in spring 2020 suggested that effectiveness would be proportional to the percentage uptake by the population.² Success of app-based contact tracing is therefore dependent on individuals’ willingness to install an app, with early evidence amongst smartphone users suggesting widespread support across countries and different sub-populations.⁵

In this panel study, we draw on seven waves of longitudinal survey data collected over the first year of the NHS COVID-19 app's implementation in England and Wales to examine changes in the uptake of the app amongst a representative sample of smartphone users aged 18 to 79. Second, given emerging evidence of COVID-19's disproportionate impact on minority ethnic groups,⁶ we included a boost sample of key minority ethnic groups to explore differences in uptake between minority ethnic groups.

Methods

Study design and participants

We recruited two samples of smartphone users aged 18 to 79 in England and Wales through YouGov's volunteer online panel (https://yougov.co.uk/about/our-panel/). First, a representative sample of smartphone users (hereafter ‘general population’ sample), with quotas set on age, gender, social grade, and region. We recruited 2032 individuals. Second, we purposively sampled at least 100 participants from six of the largest minority ethnic groups (Bangladeshi, Black African, Black Caribbean, Chinese, Indian, and Pakistani). It was not possible to set quotas for these groups as national data on smartphone users by ethnic group are not available.

Participants consented before starting the first survey and were free to withdraw at any time. Ethics approval was obtained from the London School of Hygiene & Tropical Medicine ethics committee (reference number 22483).

Data collection

We undertook seven online surveys between 14th October 2020 and 13th September 2021. Surveys were conducted roughly every six to eight weeks; timing varied as the pandemic and response changed, and in response to policy makers’ requirements for insight into changes in app use. The questionnaire consisted of both new questions developed by the researchers as well as questions adapted from previous surveys.^7,8 Questions covered attitudes to the app, whether the app was installed and, if so, use of the app's features,⁹ as well as questions on general attitudes toward COVID-19, protective behaviours adopted, perceptions of the Governments’ responses in England and Wales, and demographic information. The baseline questionnaire is provided in the online Supplementary Material (S3); the core content of the questionnaire remained the same across all waves, but some questions were added or removed in response to the evolving pandemic. Participants who completed the baseline survey were invited to participate in each subsequent wave.

Outcome and explanatory variables

We defined three time-varying binary outcome variables: (1) ‘app installation status’ – defined as having the app installed or not at each wave; (2) ‘installed’ – defined as having installed or reinstalled the app since the last wave versus not having the app installed at the current wave; and (3) ‘uninstalled’ – defined as having uninstalled the app since the last wave versus having the app installed at the current wave.

We examined socio-demographic factors linked to poorer COVID-19 outcomes and time-varying factors likely to affect the uptake of the app.⁶ We dropped all variables with more than 5% missing data.

Analysis

The main analysis was restricted to participants who responded to all seven waves of the survey and the two sample populations were analysed separately. All statistical analyses were conducted using Stata 17 (StataCorp, 2021, College Station, TX).

For the general population sample, the achieved sample at wave 1 was weighted to be representative of the population of smartphone users in England and Wales in terms of age within gender, social grade, and region using data from the 2019 Ofcom Technology Tracker Survey.¹⁰ A longitudinal weight was generated to account for loss to follow-up, see Supplementary Material S1.1 for more details. Analysis of the minority ethnic group sample was unweighted.

For both samples, we fitted a pooled logistic regression model to examine factors associated with app installation status. Additionally, for the general population sample, we fitted a longitudinal logistic regression to estimate factors associated with installing or uninstalling the app over time.¹¹ For all models, we included variables significantly associated with the outcome in univariate analysis. To finalise the variable selection, we used backward stepwise regression and tested for multicollinearity. Models were fitted using normal standard errors derived from heteroscedasticity tests and we used the random effects approach for longitudinal models based on the Hausman test.¹² For the general population sample, we included population weights calculated at the final survey wave.¹³ See Supplementary Material S1.3 for more details of the modelling approach.

We performed sensitivity analyses including running the models with all participants who had responded to two or more waves of the survey. The results showed broadly consistent findings (Supplementary Material S2).

Role of the funding source

The funder was given the opportunity to comment on the content and the timing of the questionnaires, but had no role in the overall study design, wording of questions, data collection, analysis, or interpretation of findings.

Results

Of the 2023 respondents in the general population sample, 1067 (53%) completed all seven waves; for the minority ethnic group sample, the figures were 684 and 321 (47%; Table S2.1). Baseline characteristics of participants in all seven waves for the two samples are presented in Table 1.

Table 1.

Overview of characteristics for participants in all waves (column percentages).

	General populationWeighted % (n = 1067)	Minority ethnicUnweighted % (n = 321)
Gender
Male	48.9	39.9
Female	51.1	60.1
Age group
18–29	14.8	16.2
30–39	16.5	27.4
40–49	23.5	26.8
50–59	20.3	19.0
60–69	15.1	6.9
70–79	9.8	3.7
Ethnicity
Bangladeshi	0.1	14.0
Black African	1.6	14.6
Black Caribbean	1.0	15.3
Chinese	0.7	15.6
Indian	3.5	19.3
Pakistani	1.1	21.2
White^a	88.1	n/a
Other^a	3.9	n/a
Region
North East	4.1	2.2
North West	13.1	10.9
Yorkshire and the Humber	9.2	7.5
East Midlands	7.2	7.2
West Midlands	10.9	11.5
East of England	11.4	9.0
London	12.3	31.2
South East	17.3	10.6
South West	8.5	7.2
Wales	6.1	2.8
Highest level of education attainment
No formal qualifications	4.2	1.3
GCSE or equivalent	12.9	7.8
A-level or equivalent	15.5	10.6
Higher education	58.0	73.2
Other	7.2	3.1
Prefer not to say/did not answer	2.3	4.1
Employment status
Currently working	54.7	67.3
Not currently working	4.0	3.7
Unpaid or voluntary work	0.2	0.3
Looking after home or family	5.4	6.5
Unemployed	5.7	7.2
Permanently sick or disabled	6.1	1.6
Education	2.9	2.8
Retired	19.5	8.4
Other	1.6	2.2
Key worker
Health worker	4.0	9.4
Care worker	1.6	2.2
Other key worker	16.6	17.5
Not a keyworker	77.8	71.0
Household income
Under £14,999	13.0	10.9
£15,000–£24,999	15.1	12.2
£25,000–£34,999	14.1	13.4
£35,000–£60,000	19.8	21.2
Over £60,000	16.0	19.3
Prefer not to say/did not answer	22.0	23.1
Personal income
Under £14,999	26.8	18.4
£15,000–£24,999	19.1	16.8
£25,000–£34,999	15.6	15.9
£35,000–£60,000	12.2	18.1
Over £60,000	6.1	5.9
Prefer not to say/did not answer	20.1	24.9
IMD
1 – Most deprived	17.5	24.9
2	18.1	24.6
3	21.6	15.6
4	20.5	14.6
5 – Least deprived	22.4	20.3
Relationship status
Married/civil partnership/live with partner	59.9	61.2
Divorced or separated	9.3	4.4
Widowed	2.1	0.6
Not living with partner	4.7	3.1
Single	23.1	27.4
Other	0.1	0.3
Prefer not to say/did not answer	1.0	2.5
Living arrangements
Lives alone	21.4	12.2
Other adult(s), no children	54.2	48.6
Children, no other adults	2.6	2.2
Other adult(s) and children	21.8	37.1
Household ownership
Own	62.7	58.3
Rent, private landlord	15.9	14.3
Rent, local authority, or housing association	10.5	10.9
Live with friends or family	9.0	15.0
Other	1.8	1.6
Self-reported health status
Very good	25.9	30.8
Good	47.8	46.1
Fair	19.1	16.5
Bad	5.3	2.2
Very bad	1.1	1.3
Not answered	1.0	3.1
Health or disability issue in the last 12 months
Limited a lot	9.2	8.4
Limited a little	16.4	11.5
No	72.9	77.0
Not answered	1.5	3.1
Consider themselves vulnerable to COVID-19
Due to age	12.8	11.8
Multiple reasons	14.9	13.4
Pre-existing medical condition	12.3	10.9
Pregnancy	1.0	0.3
Other	3.4	5.0
Not vulnerable	54.5	55.5
Did not answer	1.4	3.2
Vote in the last election
Brexit party	1.9	1.6
Conservative	34.5	22.5
Green	3.5	0.9
Labour	36.1	48.8
Liberal Democrat	11.7	11.6
Plaid Cymru	0.2	0.3
Other	2.3	1.2
Do not know	9.7	13.1
Vote in the 2016 European Union (EU) referendum
Remain	51.6	62.3
Leave	38.9	21.5
Did not vote	8.4	14.0
Cannot remember/not answered	1.0	2.2

Notes: The minority ethnic group sample did not include White or any other ethnic group.

Installation of the app

As demonstrated in Figure 1, the overall percentage of respondents with the app installed in the general population sample remained relatively stable over the first year of implementation; it was highest at wave 2 conducted during the second national lockdown (12–23 November 2020) and lowest at wave 5 as restrictions started to be eased (15–31 March 2021); 52.3% (95% CI: 48.8–55.8) and 46.7% (95% CI: 43.2–50.3), respectively. Amongst the minority ethnic group sample, installation rates were highest amongst Indian respondents and lowest amongst Pakistani respondents throughout.

Figure 1.

Percentage of respondents with the app installed by survey wave, mapped against the number of new COVID-19 cases in England & Wales and key changes in lockdown restrictions.

The majority of respondents who installed the app, did so for the first time before wave 1 (14–22 October 2020). The most frequently selected reasons for installing it at the start of the study was ‘civic, public or social responsibility to download the app to help stop the spread of COVID-19 and return to normal’ (58.7%; 95% CI: 54.6–62.8) and that it would ‘let me know when I've been at risk’ (56.5%; 95% CI: 52.3–60.5), see Figure 2 and Table S2.3 in Supplementary Material. Amongst those who installed the app for the first time from wave 2 onward, the most frequently selected reason was ‘needed to scan the NHS QR code to enter a venue’ (35.3%; 95% CI: 26.8–44.7). The most frequently cited reason for not downloading the app at the start of the study was being ‘worried about privacy’ (40.0%; 95% CI: 35.5–44.7) although this declined over time to 24.3% (95% CI: 19.7–29.7) by wave 7, see Figure 3 and Table S2.4. From wave 3 (28 December 2020–6 January 2021) onward, not wanting to use the app became the most frequently selected response. Not wanting to be told to self-isolate, as a reason for not downloading the app increased from 6.5% at wave 1 to 12.9% at wave 7. Of those who had downloaded the app before wave 1, 33.5% (95% CI: 29.5–37.6) were completely comfortable storing data in the app compared to 16.3% (9.8–25.8) of those who downloaded the app after wave 1.

Figure 2.

Most frequently selected reasons for downloading the app among the general population sample.

Figure 3.

Most frequently selected reasons for not trying to download the app among the general population sample.

Table 2 shows the results of the pooled logistic regression for the general population and minority ethnic group samples. For the general population sample, installation of the app was lower amongst individuals living in the North East of England (aOR: 0.53; 95% CI: 0.38–0.73), North West of England (aOR: 0.72; 95% CI: 0.59–0.89), and Wales (aOR: 0.49; 95% CI: 0.37–0.67), ethnic groups other than White (aOR: 0.59; 95% CI: 0.50–0.71), lower education levels (aOR: 0.86; 95% CI: 0.74–0.99) and those who were looking after home/family or involved in voluntary work (aOR: 0.61; 95% CI: 0.47–0.78) or unemployed, permanently sick, or disabled (aOR: 0.65; 95% CI: 0.54–0.79). Installation was higher amongst those who considered themselves as having an underlying condition that made them vulnerable to COVID-19 (aOR: 1.38; 95% CI: 1.21–1.58), and those who had received a COVID-19 vaccine, whilst those who were not at all concerned that COVID-19 posed a risk to the country (aOR: 0.66; 95% CI: 0.45–0.98) or themselves (aOR: 0.58; 95% CI: 0.44–0.75), were less likely to install. Individuals who reported they had voted for the United Kingdom to leave the European Union in the 2016 referendum, and who were less trustful of information provided by the government, were also less likely to install ((aOR: 0.77; 95% CI: 0.68–0.88); (aOR: 0.38; 95% CI: 0.32–0.46), respectively).

Table 2.

Pooled multivariate logistic regression to explore the association between app installation and explanatory variables.

	General population sample (n = 1024 individuals)			Minority ethnic group sample (n = 289 individuals)
	aOR	95% CI	p-value	aOR	95% CI	p-value
Age categories (Ref: 50–59)
18–29	1.02	0.83, 1.26	.85	1.35	0.89, 2.04	.16
30–39	1.19	0.99, 1.44	.06	0.91	0.63, 1.31	.62
40–49	0.99	0.83, 1.17	.88	1.40	1.00, 1.97	.05
60–69	1.16	0.91, 1.47	.23	0.75	0.44, 1.29	.3
70–79	1.10	0.80, 1.52	.55	0.56	0.26, 1.23	.15
Ethnicity (Ref: White)
All other ethnic groups	0.59***	0.50, 0.71	<.001	n/a	n/a	n/a
Ethnicity (Ref: Indian)
Bangladeshi	n/a	n/a	n/a	0.49***	0.33, 0.75	<.001
Black African	n/a	n/a	n/a	0.55***	0.38, 0.80	<.001
Black Caribbean	n/a	n/a	n/a	0.85	0.58, 1.25	.41
Chinese	n/a	n/a	n/a	0.73	0.50, 1.06	.10
Pakistani	n/a	n/a	n/a	0.24***	0.16, 0.35	<.001
Region of the United Kingdom (Ref: London)
North East	0.53***	0.38, 0.73	<.001	0.26*	0.09, 0.79	.02
North West	0.72***	0.59, 0.89	<.001	0.88	0.58, 1.34	.56
Yorkshire and the Humber	0.75*	0.59, 0.94	.01	1.64*	1.06, 2.54	.03
East Midlands	0.78*	0.61, 0.99	.04	0.87	0.56, 1.33	.51
West Midlands	0.81	0.65, 1.01	.06	1.16	0.79, 1.72	.45
East of England	1.09	0.87, 1.36	.46	1.1	0.73, 1.66	.66
South East	0.88	0.72, 1.07	.21	1.75***	1.20, 2.55	<.001
South West	0.78*	0.62, 0.98	.03	1.47	0.92, 2.35	.11
Wales	0.49***	0.37, 0.67	<.001	0.63	0.34, 1.16	.14
IMD quintiles (Ref: 5 – least deprived)
1 – Most deprived	0.97	0.81, 1.15	.70	1.00	0.68, 1.46	.99
2	1.17	0.98, 1.38	.08	0.58***	0.41, 0.82	<.001
3	1.05	0.89, 1.23	.57	0.62*	0.43, 0.89	.01
4	1.12	0.95, 1.32	.17	0.67*	0.46, 0.98	0.04
Education ^b (Ref: Higher education)
Lower than A-level	0.86*	0.74, 0.99	.04	0.75	0.52, 1.08	.12
A-level or equivalent	1.04	0.89, 1.21	.62	0.60**	0.41, 0.88	.01
Employment status (Ref: Currently working)
Not currently working	0.99	0.76, 1.30	.97	1.02	0.57, 1.83	.94
Voluntary/look after home or family	0.61***	0.47, 0.78	<.001	0.79	0.48, 1.29	.34
Unemployed/perm sick or disabled	0.65***	0.54, 0.79	<.001	1.1	0.70, 1.71	.69
Education	0.95	0.74, 1.23	.72	2.02*	1.17, 3.49	.01
Retired	0.78	0.60, 1.01	.06	1.71	0.97, 3.02	.06
Housing tenure (Ref: Own)
Rent	0.96	0.84, 1.10	.54	1.11	0.83, 1.49	.48
Other, including living with parents/family	1.28**	1.06, 1.54	.01	1.02	0.72, 1.46	.90
Vote in 2016 EU referendum (Ref: Remain)
Leave	0.77***	0.68, 0.88	<.001	0.94	0.71, 1.26	.70
Did not vote/cannot remember	0.63***	0.52, 0.78	<.001	0.66*	0.46, 0.96	.03
Self-reported health ^a (Ref: Good)
Bad/very bad	0.87	0.67, 1.11	.26	0.92	0.50, 1.69	.79
Fair	1.01	0.88, 1.16	.90	0.84	0.62, 1.14	.26
Very good	1.08	0.94, 1.24	.30	0.83	0.64, 1.09	.19
Any underlying condition making them vulnerable to COVID-19 (Ref: No)
Yes	1.38***	1.21, 1.58	<.001	1.53***	1.19, 1.96	<.001
Concern risk of COVID-19 poses to the country ^b (Ref: Fairly concerned)
Not at all concerned	0.66*	0.45, 0.98	.04	0.11**	0.02, 0.55	.01
Not very concerned	0.72***	0.58, 0.88	<.001	1.2	0.77, 1.87	.41
Very concerned	1.14	0.99, 1.30	.06	1.18	0.92, 1.52	.20
Concern risk of COVID-19 poses to self ^b (Ref: Fairly concerned)
Not at all concerned	0.55***	0.43, 0.72	<.001	0.40***	0.22, 0.74	<.001
Not very concerned	0.95	0.83, 1.08	.42	1.23	0.92, 1.66	.17
Very concerned	1.01	0.85, 1.20	.91	0.70*	0.53, 0.94	.02
How COVID-19 crisis has changed in local area ^b (Ref: Baseline/not changed)
Improved a lot	1.41***	1.13, 1.77	<.001	1.53	0.97, 2.41	.07
Improved a little	1.06	0.90,1.25	.46	1.13	0.83, 1.54	.44
Worsened a little	1.20*	1.02,1.42	.03	1.18	0.85, 1.64	.33
Worsened a lot	1.38***	1.14,1.67	<.001	1.63*	1.11, 2.39	.01
Had or currently have COVID-19 ^b (Ref: Definitely not had it)
Definitely/probably had it	1.05	0.86, 1.29	.62	1.32	0.94, 1.86	.11
Do not know whether had it or not	0.67***	0.57, 0.80	<.001	1.62***	1.17, 2.23	<.001
Probably not had it	0.93	0.82, 1.05	.24	1.15	0.88, 1.50	.31
Had COVID-19 vaccine ^b (Ref: No, neither dose)
Yes, one dose	1.57***	1.23, 2.00	<.001	1.38	0.87, 2.18	.17
Yes, two doses	1.87***	1.38, 2.53	<.001	1.47	0.86, 2.48	.16
Perception of Govt· handling of the COVID-19 pandemic ^b (Ref: Very badly)
Fairly badly	0.90	0.78, 1.04	.16	0.53***	0.40, 0.71	<.001
Fairly well	0.87	0.72, 1.04	.12	0.69*	0.49, 0.97	.03
Very well	0.94	0.69, 1.28	.68	0.78	0.43, 1.41	.42
Extent trust information provided by Govt· on the COVID-19 pandemic ^b (Ref: A fair amount)
Not at all	0.38***	0.32, 0.46	<.001	0.29***	0.20, 0.43	<.001
Not very much	0.60***	0.53, 0.69	<.001	0.65***	0.50, 0.85	<.001
A great deal	1.19	0.97, 1.47	.10	1.69**	1.15, 2.47	.01
No. days went shopping in the last seven days (Ref: 2–3 days)
Never	0.90	0.77, 1.05	.19	1.24	0.89, 1.73	.20
1 Day only	0.89	0.79, 1.01	.06	1.20	0.94, 1.53	.15
4–6 Days	0.78	0.61, 1.02	.07	1.45	0.87, 2.42	.16
Everyday	0.47***	0.30, 0.74	<.001	0.81	0.38, 1.72	.58
No. days met family/friends in last seven days (Ref: Never)
1 Day only	1.08	0.95, 1.23	.23	1.32*	1.01, 1.71	.04
2–3 Days	1.03	0.87, 1.22	.73	1.58*	1.10, 2.27	.01
4–6 Days	0.98	0.69, 1.40	.91	1.33	0.61, 2.88	.47
Everyday	1.94*	1.03, 3.64	.04	3.82**	1.48, 9.85	.01
No. days went out to walk/exercise in last seven days (Ref: 2–3 days)
Never	0.79***	0.68, 0.93	<.001	0.82	0.61, 1.10	.18
1 Day only	0.99	0.83, 1.18	.94	1.00	0.73, 1.36	.98
4–6 Days	1.03	0.87, 1.21	.73	0.95	0.67, 1.34	.77
Everyday	0.96	0.82, 1.12	.58	1.09	0.74, 1.61	.66
Wave-fixed effect (Ref: w1)
w2	0.77*	0.62, 0.96	.02	1.01	0.66, 1.54	.95
w3	0.67***	0.53, 0.84	<.001	0.74	0.47, 1.16	.19
w4	0.67***	0.54, 0.83	<.001	0.93	0.61, 1.43	.76
w5	0.51***	0.40, 0.66	<.001	0.66	0.40, 1.07	.09
w6	0.50***	0.36, 0.69	<.001	0.78	0.43, 1.41	.41
w7	0.41***	0.28, 0.58	<.001	0.69	0.37, 1.29	.25
Constant term	2.93***	2.07, 4.16	<.001	1.41	0.73, 2.75	.31

CI: confidence interval; aOR: adjusted odds ratio; Ref: reference group (comprised of an average individual within the sampled participants).

Comprised five groups, there was no significant difference so we grouped into three categories.

Time-varying; respondents were asked questions each time surveyed – the question asked ‘Since you completed the last survey …’.

Robust standard errors and population-adjusted weights were used.

*p ≤ .05, **p ≤ .01, ***p ≤ .001.

The pooled logistic regression of the minority ethnic group sample highlights significant differences between ethnic groups, with lower uptake amongst Pakistani (aOR: 0.24; 95% CI: 0.16–0.35), Bangladeshi (aOR: 0.49; 95% CI: 0.33–0.75) and Black African (aOR: 0.55; 95% CI: 0.38–0.80) respondents (Table 2).

Change in installation status

Almost a third of respondents had the app installed at all survey waves (30.7%; 95% CI: 27.9–33.7) and just over a third never installed the app (37.2%; 95% CI: 34.2–40.3). The remaining respondents changed app installation once on average (IQR: 1–2). The percentage of respondents reporting a change was highest at wave 6 and lowest at wave 5, see Figure 4. Factors associated with either installing or uninstalling the app are presented in Table 3. Factors associated with installing the app include becoming more concerned about the risk COVID-19 poses to the country (aOR: 3.27; 95% CI: 1.28–8.31), considering that the crisis has worsened in their local area (aOR: 3.45; 95% CI: 1.35–8.82) and going shopping less often in the last seven days compared to the previous survey (aOR: 4.50; 95% CI: 2.03–9.98). Having had COVID-19 or receiving a COVID-19 vaccine since the previous survey was associated with both installing and uninstalling the app.

Figure 4.

Changes in installation status from questionnaire wave two onwards.

Table 3.

Multivariate longitudinal logistic regression to explore what explanatory variables caused individuals to install or uninstall the app over time amongst the general population sample.

	Installing (n = 1020 individuals)			Uninstalling (n = 627 individuals)
	aOR	95% CI	p-value	aOR	95% CI	p-value
Age categories (Ref: 50–59)
18–29	2.94	0.55, 15.63	.21	5.55***	1.75, 17.63	<.001
30–39	2.16	0.50, 9.43	.30	2.08	0.73, 5.91	.17
40–49	1.88	0.50, 7.17	.35	1.43	0.53, 3.83	.48
60–69	1.14	0.17, 7.61	.89	1.28	0.33, 4.94	.72
70–79	0.89	0.09, 9.24	.92	1.02	0.17, 6.11	.98
Ethnicity (Ref: White)
All other ethnic groups	1.22	0.28, 5.42	.79	4.40**	1.69, 11.48	<.001
Region of the UK (Ref: London)
North East	0.47	0.04, 5.93	.56	4.62	0.66, 32.23	.12
North West	1.47	0.27, 8.01	.66	3.40*	1.06, 10.92	.04
Yorkshire and the Humber	1.05	0.15, 7.17	.96	4.48*	1.27, 15.84	.02
East Midlands	1.02	0.13, 7.98	.99	4.43*	1.11, 17.75	.04
West Midlands	1.14	0.19, 6.70	.88	2.12	0.62, 7.23	.23
East of England	5.47	0.84, 35.72	.08	4.96**	1.51, 16.34	.01
South East	1.81	0.36, 9.19	.47	2.65	0.90, 7.76	.08
South West	2.29	0.39, 13.38	.36	2.41	0.66, 8.72	.18
Wales	1.24	0.09, 17.38	.87	4.96	0.98, 25.21	.05
IMD quintiles (Ref: 5 – least deprived)
1 – Most deprived	1.72	0.42, 7.13	.45	2.02	0.75, 5.45	.16
2	1.9	0.49, 7.41	.35	2.18	0.88, 5.38	.09
3	2.46	0.72, 8.46	.15	3.29**	1.40, 7.73	.01
4	3.07	0.87, 10.77	.08	1.84	0.77, 4.37	.17
Education (Ref: Higher education)
Lower than A-level	0.95	0.27, 3.30	.93	0.93	0.40, 2.16	.87
A-level or equivalent	1.49	0.42, 5.27	.53	1.12	0.49, 2.55	.79
Housing tenure (Ref: Own)
Rent	1.09	0.35, 3.39	.88	0.99	0.47, 2.07	.98
Other, including living with parents/family	1.73	0.37, 8.06	.49	1.15	0.44, 2.99	.78
Employment status (Ref: Currently working)
Not currently working	1.48	0.16, 14.07	.73	0.19*	0.05, 0.81	.03
Voluntary/look after home or family	0.17	0.02, 1.61	.12	3.90*	1.13, 13.52	.03
Unemployed/perm sick or disabled	0.82	0.22, 3.05	.77	0.91	0.35, 2.34	.84
Education	4.09	0.64, 26.01	.14	0.09***	0.02, 0.37	<.001
Retired	2.43	0.41, 14.52	.33	0.75	0.20, 2.78	.67
Vote in 2016 EU referendum (Ref: Remain)
Leave	1.08	0.41, 2.86	.88	2.54**	1.26, 5.10	.01
Did not vote/cannot remember	0.00***	0.00, 0.00	<.001	4.03*	1.29, 12.58	.02
Self-reported health ^a (Ref: Good)
Bad/very bad	1.37	0.28, 6.73	.70	2.63	0.82, 8.49	.11
Fair	1.92	0.77, 4.81	.16	1.91*	1.05, 3.47	.03
Very good	1.94	0.87, 4.32	.11	1.14	0.63, 2.08	.66
Any underlying condition making them vulnerable to COVID-19 (Ref: No)
Yes	3.16*	1.10, 9.11	.03	0.59	0.30, 1.17	.13
Concern risk of COVID-19 poses to country ^b (Ref: Baseline/not changed)
Less concerned	1.85	0.80, 4.26	.15	1.18	0.72, 1.92	.50
More concerned	3.27*	1.28, 8.31	.01	1.72	0.96, 3.08	.07
Concern risk of COVID-19 poses to self ^b (Ref: Baseline/not changed)
Less concerned	0.54	0.18, 1.56	.25	0.97	0.58, 1.63	.91
More concerned	1.26	0.47, 3.38	.65	1.17	0.67, 2.06	.58
How COVID-19 crisis has changed in local area ^b (Ref: Baseline/not changed)
Worsened a lot	3.45**	1.35, 8.82	.01	0.77	0.42, 1.41	.40
Worsened a little	2.05	0.81, 5.15	.13	0.63	0.35, 1.13	.12
Improved a little	1.01	0.40, 2.56	.99	1.24	0.74, 2.09	.41
Improved a lot	1.15	0.32, 4.06	.83	0.63	0.29, 1.34	.23
Had or currently have COVID-19 ^b (Ref: Definitely not had it)
Definitely/probably had it	8.55***	2.79, 26.22	<.001	2.87**	1.36, 6.07	.01
Do not know whether had it or not	1.40	0.54, 3.63	.50	1.41	0.71, 2.77	.32
Probably not had it	1.57	0.75, 3.27	.23	0.65	0.40, 1.06	.08
Had COVID-19 vaccine ^b (Ref: Not had vaccine/Vaccine status not changed)
Received first dose	28.57***	11.20, 72.91	<.001	1.75	0.99, 3.10	.05
Received second dose	31.72***	13.55, 74.22	<.001	2.80***	1.67, 4.67	<.001
Perception of Govt. handling of the COVID-19 pandemic ^b (Ref: Baseline/not changed)
Got worse	1.91	0.73, 4.95	.19	0.78	0.43, 1.42	.42
Got better	1.07	0.44, 2.63	.88	0.89	0.53, 1.48	.65
Extent trust information provided by Govt. on the COVID-19 pandemic^b (Ref: Baseline/not changed)
Less trusting	0.28*	0.09, 0.85	.03	1.15	0.69, 1.92	.60
More trusting	0.83	0.36, 1.90	.65	0.81	0.48, 1.38	.44
How often went shopping in last seven days (Ref: Baseline/not changed)
Less often	4.50***	2.03, 9.98	<.001	1.07	0.66, 1.75	.78
More often	1.61	0.71, 3.65	.25	0.85	0.53, 1.34	.48
How often met family/friends in last seven days (Ref: Baseline/not changed)
Less often	1.10	0.43, 2.84	.84	0.79	0.47, 1.34	.38
More often	2.55*	1.20, 5.42	.02	1.33	0.86, 2.06	.20
How often went for walk/exercise in last seven days (Ref: Baseline/not changed)
Less often	0.59	0.25, 1.37	.22	0.86	0.54, 1.37	.54
More often	0.74	0.33, 1.65	.47	0.87	0.55, 1.39	.57
Constant term	13.89*	1.81, 106.30	.01	0.00***	0.00, 0.01	<.001

CI: confidence interval; aOR: adjusted odds ratio; Ref: reference group (comprised of an average individual within the sampled participants).

Comprised five groups, there was no significant difference so we grouped into three categories.

Time-varying; respondents were asked questions each time surveyed – the question asked ‘Since you completed the last survey …’.

Robust standard errors and population-adjusted weights were used.

*p ≤ .05, **p ≤ .01, ***p ≤ .001.

Amongst all respondents who uninstalled the app, the key reason for doing so was ‘not finding the app useful’ (18.4%; 95% CI: 13.9–24.1). At the final survey wave the key reason was ‘having no confidence in the app e.g., sending false alerts or not sending alerts when should have done’, see Table S2.5. Amongst respondents who reinstalled the app the key reason for doing so was ‘needing to scan a QR code at a venue wished to visit’, ranging from 61.7% (95% CI: 43.9–76.8) at wave 6 to 3.8% (95% CI: 0.5–23.5) at wave 4, see Table S2.6.

Discussion

The contribution of contact tracing apps to support efforts to control COVID-19 and similar infectious diseases is dependent on their uptake by the general population.² In this panel study of smartphone users in England and Wales, we examined the uptake of the NHS COVID-19 contact tracing app over its first year of implementation. We found that despite changes in policy and case numbers, installation of the app remained relatively stable, although we observed significant differences between minority ethnic groups. The majority of respondents who had ever installed the app did so at the time of the app's launch in September 2020.

It is likely that at the rate of uptake observed, the app had the potential to meaningfully reduce COVID-19, given evidence that tangible benefits are achieved at even modest levels of uptake.¹⁴ However, early modelling suggested that suppressing COVID-19 would require higher uptake than found here.^2,15 Uptake of the app was also lower than might have been anticipated given evidence prior to the app's launch indicated widespread support for app-based contact tracing and that 62% of the British public would be willing to download an app.^5,8 Low uptake has been documented in England¹⁶ and elsewhere,¹⁷ indicating a potential ‘intention action gap’.

Novelty and a perceived sense of urgency potentially drove uptake at the initial stages. Amongst respondents who installed the app at the start of the study, the key reason was a sense of civic duty, and those who perceived COVID-19 posed a greater risk were more likely to have the app installed. A perceived change in the severity of the crisis was also a trigger for installing the app. Higher levels of uptake amongst those who perceive COVID-19 as a threat to themselves and others have been reported elsewhere.¹⁷ For these individuals, the perceived benefit of having the app might have outweighed other concerns. The observation that contracting COVID-19 or receiving the vaccine was a trigger for installing and uninstalling the app likely reflects differences in individuals’ perception of risk; while it may have heightened awareness of COVID-19, prompting some individuals to install the app, others might no longer have perceived the need for the app.

The app's launch was highly promoted initially. In the absence of sustained promotion, features which encourage active participation (such as QR codes) might act as a reminder to individuals of the personal benefit or need for the app. However, there was only a limited window in which QR codes could be used, given the significant periods of lockdowns and social contact restrictions in the first year of implementation. By 19th July 2021, most social contact requirements in England, including the need to ‘check into’ venues using the app, had been dropped. Other studies have found differences in uptake in England by the level of restrictions in place.¹⁶ This potentially explains the lower uptake observed in Northern regions, which were placed in higher tiers of restriction when the English Government introduced the three-tier system shortly after the app was launched. In Wales, where QR codes were never mandatory, we also observed lower uptake. Interestingly, we did not observe the large drop in installation following so-called ‘freedom day’ (19th July 2021) that has been reported elsewhere.¹⁸ This suggests possible inertia with those who had the app on their phones leaving it there without using it.

Our findings also indicate a lack of trust in information provided by government and privacy concerns amongst those who did not install the app. Privacy concerns shaped many governments’ approach to the development of apps, including in England, where the government switched from a centralised to a decentralised model.¹⁹ This highly publicised ‘U-turn’ along with delays to roll-out, potentially undermined trust in the technology. Further, media discourse characterising apps as controversial has been reported as a challenge across countries.²⁰ In the United Kingdom, significant media coverage was given to the role of the private sector in developing the NHS Test and Trace system.²¹ Further, the English Government did little to counter false claims around the ‘Pingdemic’, reported as a defect of the app's alert system rather than a result of increasing community infections due to the Delta variant.²² Amongst those who uninstalled the app in this period, we found an increase in the perception that the app was sending false alerts. Concerns from users about the accuracy of the app have been reported elsewhere.^23,24

Similar to other studies, we found socio-demographic differences in uptake, with those with the lowest educational attainment, unemployed or permanently sick, or from minority ethnic groups less likely to have installed the app.¹⁶ Findings from our minority ethnic group sample point to considerable differences in uptake between minority ethnic groups, with those identifying as Pakistani and Bangladeshi least likely to install the app. This finding is of concern given evidence that COVID-19 has disproportionately affected minority ethnic groups.²⁵

Strengths and weaknesses of this study and analysis

The strengths of this study were its panel structure with seven waves of data collection over a year, plus a minority ethnic group sample that enabled the identification of differences between ethnic minorities. The substantial heterogeneity between groups provided further evidence of why minority ethnic groups should not be treated as a collective.

The general population sample was representative of smartphone users, but it was recruited from a voluntary online panel. Online panels tend to have higher levels of educational attainment than the general population,²⁶ which may make it difficult to provide robust population estimates. The finding of a divide by educational attainment and employment is likely to be more pronounced in the general public, suggesting our findings may overestimate app uptake. This, however, was not the main aim of the study, which was more interested in overall trends and changes over time.

The rate of attrition was relatively low with 65% of respondents participating in the final wave; although only 53% completed all seven waves. Those under 40, in full-time education, living with other adults and children, and who reported their health to be good or very good health with no disabilities, were less likely to participate in all seven waves, while those currently working or retired, living alone and who owned their house were more likely to respond to all waves. We found limited evidence of these variables being associated with uptake.

Ethnic minority groups were purposively sampled to allow us to examine differences between groups. Obtaining a representative sample of ethnic minority groups is challenging using online panels. For example, since only English speakers are recruited, we are potentially missing an important subset of individuals within some of these minority groups. A particular challenge for our study is that no data are available for smartphone ownership within minority ethnic groups so it is difficult to judge how representative our minority ethnic group samples are, and as a result, we could not weight the data.

We were able to draw on substantial demographic data on the panel members previously collected by YouGov, which reduced the amount of missing data for these variables. However, we excluded income and how respondents had voted in the previous General Election, because of high levels of missing data or respondents reporting ‘don’t know’; both have been found to be associated with uptake of the app elsewhere.^27,28

Installation of the app is a proxy for use of contact tracing. The overall effectiveness of the app will also depend on having Bluetooth enabled and compliance with exposure notifications.¹⁵ While we do not have evidence of compliance, our results suggest that not wanting to self-isolate was not a key reason for not installing the app or uninstalling the app.

The generalisability of these findings to other populations will be limited by the availability of compatible smartphones in different jurisdictions where the technology is introduced; in 2021 it was estimated that less than half of the world's population were smartphone users.²⁹ It will also be of less relevance to settings where the use of the technology was mandated by authorities.

Policy and practice implications for the current pandemic and in the longer term for future pandemic management

Contact tracing apps are a relatively unproven technology first developed in response to the SARS outbreak in South East Asia. Their use during the COVID-19 pandemic was novel in most health systems. The uptake of most apps has been lower than hoped for, with our findings suggesting that uptake of the NHS COVID-19 contact tracing app stagnated after the launch. Funding for the app was extended to the end of 2022, long after the removal of all pandemic restrictions, and it may play a role in future pandemics. It is therefore important to explore ways to maximise uptake.

Our results indicate that appealing to people's desire to contribute to the common good and emphasising the benefits of the app may increase uptake in some individuals. In addition, policy requirements that encourage use, such as scanning QR codes to enter venues, could also increase uptake. Other studies have demonstrated the benefits of pro-social media campaigns.³⁰ However, these also document a limit to the effectiveness of such approaches; suggesting the need for further strategies to encourage uptake. ‘Nudge’ initiatives and even monetary incentives could be explored,¹⁷ though the latter would require additional measures in the case of an anonymous app, to prevent the incentives system from being abused.

The use of contact tracing apps also depends on the wider social context, in particular, the degree to which the public trusts the government's response to any pandemic and its ability to keep personal data secure. However, acting to allay privacy concerns is complex and can have perverse consequences; for example, evidence from Mexico suggests that focusing on efforts to address privacy concerns can reduce willingness to download a contact tracing app, possibly by triggering apprehensions about data privacy.³¹ Given these technologies were relatively untested, greater evidence of the efficacy of these apps, may also serve to increase trust, and demonstrate the tangible benefits of their use.

Lastly, future efforts to promote app use should seek to target groups with lower uptake, given that disparities in uptake have the potential unintended effect of exacerbating pre-existing COVID-19 inequalities, in particular disparities between ethnic groups. This warrants further attention in an appropriately powered study.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076231159449 - Supplemental material for One year of digital contact tracing: Who was more likely to install the NHS COVID-19 app? Results from a tracker survey in England and Wales

Supplemental material, sj-docx-1-dhj-10.1177_20552076231159449 for One year of digital contact tracing: Who was more likely to install the NHS COVID-19 app? Results from a tracker survey in England and Wales by Mustafa Al-Haboubi, Josephine Exley, Kasim Allel, Bob Erens and Nicholas Mays in Digital Health

Footnotes

Acknowledgements

The authors would like to thank the survey respondents for their time and effort. We are grateful for the flexibility provided by Gavin Ellison at YouGov in facilitating each wave of the research at short notice, and for the statistical input provided by Paul Boadu. Lastly, we would like to thank the organisations within the DHSC that provided us with information on the app's features upon request.

Author Contributions

MA-H, BE, and NM conceptualised and designed the study, JE and KA undertook the analysis, JE and MA-H drafted the manuscript. All authors reviewed, commented on, edited and approved the final version of the manuscript.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the NIHR Policy Research Programme through the Policy Innovation and Evaluation Research Unit (PR-PRU-1217-20602). The views expressed are those of the authors and are not necessarily those of the NIHR or the Department of Health and Social Care.

Guarantor

MA-H.

ORCID iDs

Mustafa Al-Haboubi

Kasim Allel

Supplemental Material

Supplemental material for this article is available online.

References

Dominik

Felix

. Fostering participation in digital public health interventions: The case of digital contact tracing . 2020.

Ferretti

Wymant

Kendall

, et al. Quantifying SARS-CoV-2 transmission suggests epidemic control with digital contact tracing. Science (New York, NY) 2020; 368. 2020/04/03. DOI: 10.1126/science.abb6936.

European Centre for Disease Prevention and Control. Mobile applications in support of contact tracing for COVID-19 – A guidance for EU/EEA Member States. Stockholm: ECDC, 2020.

Braithwaite

Callender

Bullock

, et al. Automated and partly automated contact tracing: a systematic review to inform the control of COVID-19. Lancet Digit Health 2020; 2: e607–e621. 2020/08/26.

Altmann

Milsom

Zillessen

, et al. Acceptability of app-based contact tracing for COVID-19: cross-country survey study. JMIR Mhealth Uhealth 2020; 8: e19857. 2020/08/08.

Public Health England. Disparities in the risk and outcomes of COVID-19. 2020.

Ipsos MORI. Ipsos MORI Coronavirus Polling, https://www.ipsos.com/en-uk/coronavirus-data-trends-britons-concern-country-continues-decline (2020, accessed 12/06/2020).

The Health Foundation. Survey on COVID-19 track and trace smartphone app for The Health Foundation, https://www.ipsos.com/en-uk/survey-covid-19-track-and-trace-smartphone-app-health-foundation (2020, accessed 04/06/2020).

NHS Test and Trace. NHS COVID-19 app support: what the app does, https://www.covid19.nhs.uk/what-the-app-does.html (2022, accessed 07/03/2022).

10.

Ofcom. Ofcom's Technology Tracker 2019 Technical Report, https://www.ofcom.org.uk/__data/assets/pdf_file/0016/144025/technology-tracker-2019-technical-report.pdf (2019).

11.

Liu

Engel

. Predicting longitudinal trajectories of health probabilities with random-effects multinomial logit regression. Stat Med 2012; 31: 4087–4101. 20120724.

12.

Hausman

. Specification tests in econometrics. Econometrica 1978; 46: 1251–1271.

13.

Kaminska

Lynn

. Selecting the correct weight for your analysis . 2019.

14.

Wymant

Ferretti

Tsallis

, et al. The epidemiological impact of the NHS COVID-19 app. Nature 2021; 594: 408–412.

15.

Hinch

Probert

Nurtay

, et al. Effective Configurations of a Digital Contact Tracing App: A report to NHSX . 2020.

16.

Horvath

Banducci

Blamire

, et al. Adoption and continued use of mobile contact tracing technology: multilevel explanations from a three-wave panel survey and linked data. BMJ Open 2022; 12: e053327.

17.

Munzert

Selb

Gohdes

, et al. Tracking and promoting the usage of a COVID-19 contact tracing app. Nature Human Behaviour 2021; 5: 247–255.

18.

JL Partners. Daily Mail: Freedom Day Polling, https://www.jlpartners.co.uk/s/JL-Partners-The-Daily-Mail-Freedom-Day-Poll.xlsx (2021, accessed 06/05/2022).

19.

Lucivero

Hallowell

Johnson

, et al. COVID-19 and contact tracing apps: ethical challenges for a social experiment on a global scale. J Bioeth Inq 2020; 17: 835–839. 2020/08/25.

20.

Amann

Sleigh

Vayena

Digital contact-tracing during the COVID-19 pandemic: an analysis of newspaper coverage in Germany, Austria, and Switzerland. PLoS One 2021; 16: e0246524. 20210203.

21.

Mahase

. COVID-19: NHS test and trace failed despite “eye watering” budget, MPs conclude. Br Med J 2021; 375: n2606.

22.

NHS Test and Trace. NHS COVID-19 app statistics, https://stats.app.covid19.nhs.uk/ (2022, accessed 09/08/2022).

23.

Garousi

Cutting

Felderer

Mining user reviews of COVID contact-tracing apps: an exploratory analysis of nine European apps. J Syst Softw 2022; 184: 111136. 20211104.

24.

O’Callaghan

Abbas

Buckley

, et al. Public opinion of the Irish “COVID tracker” digital contact tracing app: a national survey. DIGITAL HEALTH 2022; 8: 20552076221085065.

25.

Mathur

Rentsch

Morton

, et al. Ethnic differences in SARS-CoV-2 infection and COVID-19-related hospitalisation, intensive care unit admission, and death in 17 million adults in England: an observational cohort study using the OpenSAFELY platform. Lancet 2021; 397: 1711–1724. 20210430.

26.

Erens

Burkill

Couper

, et al. Nonprobability web surveys to measure sexual behaviors and attitudes in the general population: a comparison with a probability sample interview survey. J Med Internet Res 2014; 16: e276. 20141208.

27.

von Wyl

Höglinger

Sieber

, et al. Drivers of acceptance of COVID-19 proximity tracing apps in Switzerland: panel survey analysis. JMIR Public Health Surveill 2021; 7: e25701. 20210106.

28.

Bradshaw

Ryan

Noetel

, et al. Information safety assurances increase intentions to use COVID-19 contact tracing applications, regardless of autonomy-supportive or controlling message framing. Front Psychol 2021; 11. Original Research. DOI: 10.3389/fpsyg.2020.591638.

29.

Newzoo. Global Mobile Market Report, https://newzoo.com/insights/trend-reports/newzoo-global-mobile-market-report-2021-free-version (2021, accessed 21/10/2022).

30.

Jordan

Yoeli

Rand

. Don't get it or don't spread it: comparing self-interested versus prosocial motivations for COVID-19 prevention behaviors. Sci Rep 2021; 11: 20222. 20211012.

31.

Martínez

Parilli

Rojas

, et al. Do you have COVID-19? How to increase the use of diagnostic and contact tracing apps. PloS One 2021; 16: e0253490–e0253490.

32.

Downey

NHS Covid-19 contact-tracing app launched in England and Wales, https://www.digitalhealth.net/2020/09/nhs-covid-19-contact-tracing-app-launched-england-wales/ (2020, accessed 21/04/2022).

33.

Cellan-Jones

. Disappearing Covid-19 app alerts cause alarm, https://www.bbc.co.uk/news/technology-54389083 (2020, accessed 08/07/2022).

34.

Muir

Test and trace: 10 per cent of Nissan workers told to self isolate, https://www.cityam.com/test-and-trace-10-per-cent-of-nissan-workers-told-to-self-isolate/ (2021, accessed 11/05/2022).

35.

Stewart

Keir Starmer: tweaking NHS Covid app ‘like taking batteries out of smoke alarm’, https://www.theguardian.com/world/2021/jul/09/keir-starmer-tweaking-nhs-covid-app-taking-batteries-smoke-alarm (2021, accessed 19/10/2022).

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.64 MB