Abstract
Objectives:
Toward common methods for system monitoring and evaluation, we proposed a key performance indicator framework and discussed lessons learned while implementing a statewide exposure notification (EN) system in California during the COVID-19 epidemic.
Materials and Methods:
California deployed the Google Apple Exposure Notification framework, branded CA Notify, on December 10, 2020, to supplement traditional COVID-19 contact tracing programs. For system evaluation, we defined 6 key performance indicators: adoption, retention, sharing of unique codes, identification of potential contacts, behavior change, and impact. We aggregated and analyzed data from December 10, 2020, to July 1, 2021, in compliance with the CA Notify privacy policy.
Results:
We estimated CA Notify adoption at nearly 11 million smartphone activations during the study period. Among 1 654 201 CA Notify users who received a positive test result for SARS-CoV-2, 446 634 (27%) shared their unique code, leading to ENs for other CA Notify users who were in close proximity to the SARS-CoV-2–positive individual. We identified at least 122 970 CA Notify users as contacts through this process. Contact identification occurred a median of 4 days after symptom onset or specimen collection date of the user who received a positive test result for SARS-CoV-2.
Practice Implications:
Smartphone-based EN systems are promising new tools to supplement traditional contact tracing and public health interventions, particularly when efficient scaling is not feasible for other approaches. Methods to collect and interpret appropriate measures of system performance must be refined while maintaining trust and privacy.
Keywords
Digital exposure notification (EN) systems use smartphones to anonymously identify and alert individuals who have been in proximity to a person who has received a positive test result for SARS-CoV-2. Such systems can automate the identification and notification of exposed contacts, supplementing traditional contact tracing programs and serving as a nonpharmaceutical intervention aimed at curbing community viral transmission. Numerous EN systems have been designed1,2 and deployed globally. In April 2020, Google and Apple released the Google Apple Exposure Notification (GAEN) 3 framework for Android and iOS devices that enables an anonymous exchange of keys among smartphone users when they are in close proximity, estimated by the strength of the exchanged low-energy Bluetooth transmissions. As of May 2022, this framework has been adopted and fully operationalized by 42 countries, excluding the United States. Within the United States, 24 states, Guam, and the District of Columbia have adopted GAEN. 4
Evidence on the performance of EN systems has been accumulating through controlled experiments 5 and observational studies.6,7 Furthermore, several reports and modeling studies have demonstrated the potential public health benefits of EN systems.8-14 However, any potential public health impact stems from user adoption and engagement, which seem to be influenced by the design and implementation of the EN systems.15-17 As such, designing for real-world use and evaluating key performance indicators (KPIs)18,19 of the EN system is critical to assessing its impact.
California was an early adopter of Exposure Notification Express, a streamlined version of the GAEN framework. In September 2020, the state initiated a pilot of Exposure Notification Express (branded CA Notify) at the University of California San Diego and University of California San Francisco. The successful pilot prompted a statewide launch on December 10, 2020, at the peak of the winter COVID-19 surge in California. 20 During this time, daily case counts in California exceeded the capacity of traditional contact tracing efforts, which impeded timely identification and notification of all contacts.
In this study, we present a proposed KPI framework for CA Notify along with lessons learned and future directions for the implementation of GAEN-based systems.
Materials and Methods
Design Process
Deploying CA Notify for people residing in California involved a human-centered design approach of iterative development and codesign with broad, diverse engagement from the community and continuous vendor feedback. We deployed and tested prototypes (touchpoints, communications language, visuals, workflow) of increasing sophistication during the pilot phase, which informed the CA Notify system configuration and final service for statewide launch. The resulting service model and current communications and designs can be found at canotify.ca.gov. Data, which contained no personally identifiable information, were shared between the University of California San Diego and California Department of Public Health by a data use agreement. The CA Notify design process did not involve research with human subjects as per 45 CFR 46.
User Journey
Users of CA Notify who receive a positive test result for SARS-CoV-2 in California can voluntarily use a unique code, which results in the anonymous alerting of other CA Notify users who came in close contact with them (Figure 1). These codes are generated automatically for >90% of all electronic laboratory records with polymerase chain reaction–positive test results in California and distributed to users via text message. When a code is sent to a CA Notify user, the user may choose to use the code by first clicking on a unique URL link that expires in 24 hours (code claimed) and then authorizing the CA Notify system to share the code with the national key server hosted by the Association of Public Health Laboratories (code used). Authorizing use of the code then leads to the release of EN alerts for other CA Notify users who have been in close proximity to the infected individual. Users who receive an EN alert are directed to a CA Notify webpage and provided public health information and directions on current guidance, such as recommendations for testing, wearing a face mask, and, if applicable, quarantining.
CA Notify service model and user journey. CA Notify is a digital exposure notification (EN) system in California that uses smartphones to anonymously identify and alert individuals who have been in proximity to a person who has received a positive test result for SARS-CoV-2. The system functions as follows: (1) a user activates CA Notify on one’s smartphone; (2) activated smartphones exchange keys when they are close for a defined period; (3) users who receive a positive test result for SARS-CoV-2 then receive a code that can be entered into the system; (4) inputting codes leads to notifications to users who have been in contact with a person who received a positive test result for SARS-CoV-2; (5) if a person who received an EN gets tested and has a positive test result, the user is encouraged to isolate and is sent, and can input, a code to the system, continuing the alerting cycle, or (6) a vaccinated user may choose only to monitor for symptoms and isolate if symptoms occur. The overall process leads to (7) protection of others in the community. The process depicted in the figure reflects the CA Notify functionality as well as isolation and quarantine guidance at the time of manuscript submission. The CA Notify service model and user journey are licensed under a Creative Commons Attribution–Non-commercial 4.0 International License (2021). Published with permission of CA Notify.
Key Performance Indicators
We defined 6 KPIs to track, along with the associated data sources, metrics, methods, and limitations for calculation or estimation. Given data constraints imposed by privacy policies, several KPIs are an estimation through indirect methods (Table 1).
Key performance indicator metrics, definitions, data sources, and limitations of CA Notify a
Abbreviations: —, does not apply; EN, exposure notification.
CA Notify is a digital EN system in California that uses smartphones to anonymously identify and alert individuals who have been in proximity to a person who has received a positive test result for SARS-CoV-2.
This study included system activity data generated by CA Notify from December 10, 2020, to July 1, 2021. We aggregated these data and reported them as KPIs, which were estimated to the extent possible, given privacy constraints. The data were then made accessible to public health authorities through a dashboard, updated daily, that included additional contextual information such as case rates and demographic information.
We modeled impact, cases, and deaths averted by using the method from Wymant et al 8 : specifically, their result of a 0.79% decrease in forecasted cases for every 1% increase in adoption of the EN system. We used the model assumption from Wymant et al of a minimum 15% of the population needing to adopt the EN system for it to have an effect, and we assumed that the secondary attack rate, level of quarantine adherence, and transmission dynamics, as well as other background interventions, policies, and protective behaviors, in California are similar to those in the United Kingdom. We estimated the number of predicted deaths averted by calculating a state death rate from the total number of COVID-19–related deaths and the total number of cases and then multiplying that value by the predicted number of averted cases. We calculated the predictions using data from December 11, 2020, to July 1, 2021 (unpublished data, California Department of Public Health).
Results
A total of 10 910 971 smartphone activations of CA Notify (Table 1) occurred during the study period. A total of 1 654 201 codes were generated for individuals with a polymerase chain reaction–positive test result for SARS-CoV-2 and sent to their registered cell phone numbers, without knowledge of their CA Notify user status. For system use, 446 634 (27%) codes were claimed by CA Notify users, and 68 107 codes were used (15% of 446 634 codes claimed) during the study period.
During the study period, 110 497 CA Notify users were identified as contacts by measuring website visit activity. For each person who received a positive test result for SARS-CoV-2 and used the code, the system generated an average of 2 exposure alerts (contacts notified) during the study period (ENs acknowledged/codes used; Table 1).
The mean (SD) number of days from symptom onset/positive test specimen collection date to code use (4.7 [3.4] days) (Figure 2) was empirically less than the mean number of days required for contact notification via traditional contact tracing methods used by California’s contact tracing program (7 days).
Summary of key data available for monitoring and evaluating the CA Notify system. A, Adoption (estimated by total activations). B, Number of EN page visits (note brief outage in measurement but not the system) and codes used. C, Number of days from symptom onset to code use. D, Number of cumulative codes distributed by call center by language. E, Percentage of EN users who used a code. F, Percentage of EN users initiating code use. G, Number of COVID-19 cases in California during the study period. H, Number of deaths in California during the study period. CA Notify data sources are listed in Table 2. California cases and deaths were obtained from the California Department of Public Health website. 20 CA Notify is a digital EN system in California that uses smartphones to anonymously identify and alert individuals who have been in proximity to a person who has received a positive test result for SARS-CoV-2. Abbreviation: EN, exposure notification.
Discussion
While developing, deploying, and monitoring the CA Notify system, we operationalized our lessons learned into a continuous improvement framework, which we applied to our KPIs. We present public health implications and actions to mitigate challenges specific to our KPIs (Table 2). We identified data requirements for what type of data must be gathered, during what period, by what means, and with what frequency.
CA Notify key performance indicators: observations and implications, December 12, 2020–July 1, 2021 a
Abbreviations: EN, exposure notification; SMS, short message service.
CA Notify is a digital EN system in California that uses smartphones to anonymously identify and alert individuals who have been in proximity to a person who has received a positive test result for SARS-CoV-2.
Use of the Exposure Notification Express version of GAEN led to challenges in estimating KPIs. These challenges are not unique to CA Notify and stem from the intrinsic privacy-preserving nature of the platform. EN systems in countries such as the United Kingdom enable the direct calculation of critical metrics, such as the secondary attack rate. However, in California, these approaches remain to be implemented, and future consideration will need to balance privacy and public acceptability with the need for system monitoring, security, and impact measurement, thereby establishing benefits over harms to inform policy decisions.
Despite innate limitations introduced by a privacy-preserving policy of minimal data collection and by intrinsic privacy measures, we attempted to provide a structured KPI framework and the context needed to interpret the value and limitations of the data. For example, using activations as a surrogate for adoption presents an upper bound only and is itself potentially biased by out-of-state adopters, those who uninstall and reactivate the app, or those who upgrade to a new cell phone. Related to this limitation, estimation of retention, or the number of active daily users, should be achievable with opt-in privacy-preserving analytics; however, the system developed for Exposure Notification Express was not immediately available at the time of launch and infuses privacy-preserving noise (random numbers) into the system to further anonymize data. To date, this injection of noise has made estimating active daily users difficult and detection of SARS-CoV-2 among EN users not feasible.
Measuring the influence of CA Notify on public health outcomes (cases, hospitalizations, and deaths) through the estimated KPIs and mathematical models is a topic of ongoing research. The final KPI, an estimate of impact, should be considered more preliminary than the first 5 KPIs and is the subject of ongoing investigation. The uncertainty, in addition to that which is inherent to the mathematical models, is due to a lack of accurate estimates for key model input metrics, including the true number of active daily users, the percentage of those alerted who are infected with SARS-CoV-2, and whether they follow the protective behaviors (eg, testing, self-quarantine) required for impact. In the absence of such metrics and models for California to date, if we (1) apply the model developed by Wymant et al 8 to CA Notify adoption data and (2) make a series of assumptions that secondary attack rate, quarantine adherence, and transmission dynamics, in addition to other background interventions, policies, and protective behaviors, in California are similar to those in the United Kingdom, then we would have averted >31 000 cases and >600 deaths in the state from December 11, 2020, to July 1, 2021. While estimates are speculative, they are encouraging evidence to support extending and expanding our evaluation efforts. The applicability of external models to other programs (eg, European models for US EN programs) remains controversial because of differences in system design, implementation characteristics, and data availability. Therefore, we are developing an agent-based model and aggregating survey and contact tracing data to estimate the impact of CA Notify on cases and deaths.
Practice Implications
In practice, we must be careful not to overinterpret our estimated KPIs considering assumptions required due to limited data, as previously discussed. Our study had important limitations pertaining to data access and reporting for CA Notify. While activations reached substantial levels (up to 40% of eligible users), these estimates likely reflect overcounting. Higher EN adoption and mitigation of factors leading to barriers to access and disparities in availability and use must be pursued. These concerns are the topic of ongoing investigation. Similarly, more granular estimates of active daily users are lacking, and for these users, it is difficult to directly ascertain the type of behavior changes (eg, testing, wearing face masks) taken by individuals who receive a notification alert. The impact of CA Notify is predicated on such changes in behavior, and we are currently seeking to better assess these changes by using voluntary prospective and retrospective surveys. Finally, our estimate of impact was limited to the use of a published model that has not been validated outside its intended context and, thus, should be considered preliminary. Future models that are bespoke to EN system implementation in California are in development to address these shortcomings.
Several initiatives are being undertaken to address data availability due to the privacy-preserving features of CA Notify and our estimation limitations, including conducting multiple surveys of California residents, opt-in prospective repeated measures surveying CA Notify users, and targeted retrospective analyses. Furthermore, to the best of our ability, adoption and use barriers have been minimized by applying a participatory design approach through each phase of our operations. Notably, this human-centered design approach brings together developers, users, public health, and other stakeholders in the community to negotiate complex issues of communication, accessibility, value proposition, and policies for implementation. While we will continue to refine our approach, we encourage others to consider these methods in their COVID-19 public health programs. In our case, engagement of diverse groups, in multiple languages, drawn from state government and university communities and through established community-based organizations provided important insights to reduce barriers and foster adoption. Still, formative work did not include a sufficiently representative sample of residents of California; as such, it will be important to observe trends in adoption as CA Notify is promoted across the state. Moreover, we are uncertain what motivates an individual to claim and then use a code (behavior), which will notify others who may have been exposed. Measures to protect privacy and anonymity were prioritized, which limited data utility. To determine whether prioritizing privacy is a public preference (by using Bluetooth rather than global positioning system location coordinates and avoiding collecting data identifiers), we will systematically survey residents of California to gather behavioral data and privacy preferences.
During the study period, up to 30% of California’s population (10.9 million activations; state population, 39.4 million) and about 40% (assuming 33 million) of all smartphone users 21 activated the CA Notify system, with EN alerts typically occurring faster than traditional contact tracing and notification efforts and reaching a greater number of contacts per index case. We believe that this level of adoption was facilitated by ease of activation (particularly for iOS instances) and availability alerts and, in part, because of the process of engagement and stakeholder input into the system design. The disproportionately higher adoption by iOS users as compared with Android users is notable and has been observed in other settings. 8 These data provide some evidence that a system-integrated solution, rather than a stand-alone app, may more effectively promote technology adoption.
Finally, there was a close and collaborative feedback loop throughout planning and deployment of CA Notify among academic implementing partners, public health officials, Google, Apple, and others. This partnership allowed design ideas and communications to flow from formative research by University of California investigators to industry partners, while focus groups tested new functionality and public health officials provided requirements and direction in real time. This close collaboration was a key to project success and produced lessons learned that can benefit others who are developing digital EN programs.
Our experience suggests that digital EN systems offer a synergistic addition to nonpharmaceutical interventions available to public health authorities, particularly extending the reach of traditional contact tracing methods during surges of infection. A growing corpus of literature,9-11 notably focused outside the United States, and our preliminary investigations of impact show the potential for such programs. As others have noted, global smartphone adoption rates are high, even in the most remote and resource-constrained settings (where novel pathogens often emerge), providing a potential global sensing and action network capable of functioning within and beyond the scope of public health authorities. Future work and coalitions are needed to create readily deployable, transparent, community-supported instances, as well as standardized methods to govern, evaluate, and evolve the system, considering the costs, benefits, and ethical design with privacy-preserving protocols. EN is a compelling tool during surges of COVID-19 and SARS-CoV-2 variants and may help to prevent or mitigate future pandemics.
Footnotes
Acknowledgements
The authors acknowledge Google and Apple for developing GAEN as well as their continued close collaboration with partners. We thank Florgin Dela Luna, MHA, Zachary Herman, MBA, Geoffery Smith, MS, and Marc Sylwestrzak, BS, at the University of California San Diego for program management and web development; the Healthcare Human Factors team, including Jennifer Rosart, MBA, Pia Nyakairu, BS, and Joe Cafazzo, PhD, for human factors research, design, and development; colleagues at the California Department of Health, including Brian Melville, MBA, Tina Piniro, MS, Wei Garland, BS, Ali Bay, BS, Evelyn Ho, BS, Will Wheeler, MBA, and Julie Rooney, MS, for programmatic support and contributions; Kenneth Cutler, MD, for his instrumental leadership and scientific contribution; Amy Tong, MBA, Carolyn Nordstrom, BS, and the California Department of Technology for technical implementation and project leadership; Bonny John, PhD, and Fahd Arshad, PhD, from the COVID Alert NY team and Devon Mann, MD, Stephen Robinson, PhD, and Veronica Alfaro, PhD, from New York University for valuable design insights; and the many individuals using EN systems, as well as the advisory boards and focus group participants who have given invaluable feedback to ensure that these systems work in their communities.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: the authors received funding from the California Department of Public Health, agreement 20-10777.
