Identification of Core Outcome Domains and Design of a Survey Questionnaire to Evaluate Impacts of Digital Health Solutions That Matter to People With Diabetes

Abstract

Background:

Digital health solutions (DHS) are increasingly used to support people with diabetes (PwD) to help manage their diabetes and to gather and manage health and treatment data. There is a need for scientifically reliable and valid methods to measure the value and impact of DHS on outcomes that matter to PwD. Here, we describe the development of a survey questionnaire designed to assess the perceptions of PwD toward DHS and their prioritized outcomes for DHS evaluation.

Method:

We applied a structured process for engagement of a total of nine PwD and representatives of diabetes advocacy organizations. Questionnaire development consisted of a scoping literature review, individual interviews, workshops, asynchronous virtual collaboration, and cognitive debriefing interviews.

Results:

We identified three overarching categories of DHS, which were meaningful to PwD and crucial for the identification of relevant outcomes: (1) online/digital tools for information, education, support, motivation; (2) personal health monitoring to support self-management; (3) digital and telehealth solutions for engaging with health professionals. Overall outcome domains identified to be important were diabetes-related quality of life, distress, treatment burden, and confidence in self-management. Additional positive and negative outcomes specific to DHS were identified and corresponding questions were incorporated into the survey questionnaire.

Conclusion:

We identified the need for self-reporting of quality of life, diabetes distress, treatment burden, and confidence in self-management, as well as specific positive and negative impacts of DHS. We designed a survey questionnaire to further assess the perceptions and perspectives of people with type 1 and 2 diabetes on outcomes relevant for DHS evaluations.

Introduction

Digital health solutions (DHS) are increasingly used as additional health care service offerings to support people with diabetes (PwD) in self-managing their condition and receiving diabetes care. ¹ Digital health solutions have a broad scope that includes categories such as mobile health, health information technology, clinical decision support, wearable devices, telemedicine, and personalized medicine. ² Digital health solutions cover a range of support features, including provision of information, sleep and distress, education and engagement, monitoring of food intake, physical activity, menstrual tracking, tracking of blood glucose and medication data, and enabling improved communication and data-sharing with health care professionals. ³ Evaluative DHS studies in diabetes today deploy a wide variety of outcome measurement approaches that make it impossible to combine or compare findings. Furthermore, DHS studies in diabetes do not apply a consistent approach to measuring what matters to PwD. While DHS have been shown to improve weight management and treatment engagement, ⁴ as well as psychosocial ⁵ and clinical outcomes such as HbA1c,^{6 -8} the lack of use of patient-informed outcome measures that are proven to matter to PwD in relation to DHS is most likely a contributing factor to the limited evidence regarding benefits of DHS.^3,9 There is a need to establish an evaluation framework with standardized tools for the evaluation of diabetes DHS, which reflects the priorities of PwD and ensures that each outcome measure is fit for purpose. Standardized DHS outcomes evaluation is of value to PwD, health care professionals,decision makers and digital technology developers. ¹⁰ Moreover, the harmonization of methods will facilitate the real-world evidence generation required for approval by authorities and payers to ensure the timely and optimized access of quality DHS to a wider population.^10,11

Health authorities largely focus their evaluation of DHS on functional and technical aspects such as data security and functionality,^12,13 while clinical research mainly concentrates on clinical endpoints such as HbA1c and time in range. ¹⁴ However, evaluation of outcomes that matter to PwD require the additional use of patient-reported outcomes (PROs).^15,16 Patient-reported outcome measures (PROMs) have been increasingly applied to evaluate self-reported effects of DHS on diabetes-related distress, depression, self-efficacy, self-management, lifestyle behaviors, knowledge, treatment and therapy satisfaction. ¹⁷ Despite increasing research on PROs relevant for the evaluation of diabetes-specific DHS, there is, however, presently no consensus on what outcome domains to assess and what PROMs to use to appropriately evaluate the impact of DHS. ¹⁸ As a result, DHS studies measure different concepts and often use different PROMs, which may not cover all the issues that matter to PwD. There is a need for building consensus across PwD, health care professionals, researcher and other stakeholders on what PROMs to use to evaluate effects of DHS.^15,19

In this article, we report the initial identification of core outcome concepts that are hypothesized to be generally relevant and important for PwD to evaluate DHS. We then describe the codesign of a survey questionnaire for completion by PwD to examine the perceptions around these impacts in a larger representative sample of people with type 1 and 2 diabetes.

Methods

Design

We applied a three-step process to identify relevant person-centered outcome constructs for the evaluation of DHS in diabetes and develop a survey to explore these in a population sample (Figure 1). First, an initial literature review along with a supportive desk research was carried out to assess DHS definitions, taxonomies, standards, and methods for evaluation of DHS according to the perspective of PwD. Second, PwD and representatives of diabetes advocacy organizations were engaged through individual meetings, workshops, and virtual collaboration to evaluate and prioritize outcome constructs specifically relevant for DHS and to cocreate the survey questionnaire. Finally, cognitive debriefing interviews were undertaken with a separate group of PwD to test the comprehension, relevance, feasibility, and acceptability of the survey questionnaire before deployment.

Figure 1.

Three-step process to identify types of DHS and person-centered outcome constructs to draft and finalize the survey questionnaire.

Literature Search and Supportive Desk Research

A PubMed and Embase search strategy with supplementation of hand-search of gray literature included the following search terms: [digital health (mobile health, mhealth, apps, tele-medicine, online, health IT, data monitoring)], [intervention], [patient involvement (patient participation, co-creation, user involvement)], [diabetes (type 1, type 2)], patient reported outcomes (self-report questionnaires, health-related quality of life, quality of life, psychosocial, self-management, self-efficacy, PRO, PROM, patient outcomes)], [qualitative research], [evaluation (core outcomes set)], [review]. Focus was on publications from 2017 to 2022, but pivotal earlier studies and manual search were also applied if deemed relevant.

Engagement of Representatives of PwD and Diabetes Advocacy Organizations

Participants

A group of PwD and representatives of a diabetes advocacy organization were engaged (1) to align the study design to PwD priorities, (2) to identify positive and negative DHS impacts that PwD consider as important outcomes, and (3) to develop content and format of the survey questionnaire. Engagement methodology was based on the patient engagement quality guidance of the global Patient Focused Medicines Development initiative. ²⁰ The guidance identified shared purpose, respect and accessibility, representativeness of stakeholders, roles and responsibilities, capacity and capability for engagement, transparency in communication and documentation, and continuity and sustainability as quality criteria to guide the planning of engagement of people with lived experience.

Five persons between the ages of 30 and 65 years (2 male and 3 female) from the United Kingdom, Australia, Sweden, Italy, and Finland were recruited, each with multiple years of experience living with and advocating for themselves and other PwD.

Workshop

In preparation of the first workshop (December 7, 2021), individual meetings were held to (1) provide information enabling participants to make an informed decision on participation and (2) introduce and initially discuss key questions with each individual. During the first workshop, the following subjects were discussed: How to categorize DHS in a meaningful way from the perspective of a PwD, what expectations PwD have toward benefits and drawbacks of DHS, and what outcomes need to be considered for the evaluation of DHS benefits and impacts. Key themes for discussion were self-efficacy and empowerment, diabetes-related quality of life, and access to care and support. Participants further discussed the rationale of and the content requirements for a survey questionnaire to be used to evaluate the experiences of PwD in a broader population. Finally, participants took part in defining ways to maximize the use of the survey study results to accelerate person-centered DHS development and evaluation.

Virtual online asynchronous engagement with PwD

Following the workshop, over a period of four weeks, participants evaluated the core outcome domains in greater detail and reviewed a draft survey questionnaire using a virtual online asynchronous engagement platform by answering open and closed questions and providing free-text comments. The aim of this process was to validate content and optimize language, to assess the relevance of proposed outcome domains and questions, and to collect suggestions regarding the comprehensiveness of questionnaire items.

Interviews

To complement the initial input from experienced diabetes advocates, semi-structured interviews were undertaken with additional four PwD to obtain extra input on core outcomes and impacts of DHS. Interviews covered the experience of PwD regarding diabetes’ impact on daily lives, the DHS used for diabetes management, DHS’ benefits and drawbacks, their perceptions regarding monitoring of the condition, the support they receive, and general questions about digital innovations and tools in diabetes. PwD (male/female 50:50) included in this study were from France, were 40 to 68 years of age, and had been diagnosed with diabetes between 10 and 34 years ago (type 1/type 2, 50:50). Interviews were conducted in November 2021 and lasted 90 minutes each.

Cognitive debriefing of final survey questionnaire draft

Following the process of optimization of the survey questionnaire, five lay PwD were interviewed for 60 minutes each in March 2022 using cognitive debriefing methodology (four adult PwD from France, type 1/type 2 50:50; one adult PwD from the United Kingdom with type 2 diabetes). Interviews were conducted by trained researchers using a semi-structured interview guide to debrief the final questionnaire draft. The interview guide was designed to collect data regarding participants’ understanding of individual items and the survey as a whole to ensure adequate acceptability and completion rates. This included PwD providing feedback concerning survey length, comprehension, and complexity.

Results

Scoping Review

Previous research identified outcomes and PRO constructs considered relevant and important to PwD in general. These included general and mental health, quality of life and well-being, diabetes-related quality of life, emotional and somatic symptom distress, treatment burden, hypoglycemia and hyperglycemia burden, self-efficacy, experience of person-centered care and support beyond the medical setting.^15,19 Our literature search did not identify previous studies that specifically identified core outcome or PROM standard sets for DHS evaluation in diabetes, endorsed or verified as representing the priority of PwD. However, some studies defined core outcomes for very specific digital interventions, highlighting the variability of outcomes being considered as relevant for the assessment of different DHS impacts. PROs applied in DHS evaluation studies covered health status, ²¹ depression, ²¹ quality of life, ²² health literacy, knowledge, disease burden, diabetes distress,^21,23 self-efficacy, locus of control, disease acceptance, self-management, ²⁴ and access to care, education, and information. Three overall outcome concepts were identified, which were frequently applied across studies to evaluate impacts of DHS and found to be relevant for the evaluation of DHS. These key concepts were self-efficacy for diabetes self-management, including autonomy and sense of empowerment, diabetes-related quality of life and distress, and perceptions of access to quality and person-centered diabetes care.

Establishment of DHS Categories

Our literature search identified a lack of consistency of DHS categorizations and a lack of categories that are understandable and meaningful for PwD. We analyzed the main frameworks for defining DHS (World Health Organization, National Institute for Health and Care Excellence, Medical Device Regulation, International Medical Device Regulators Forum) and found that categories were rather defined according to technical features than the perspective of PwD as DHS end-users. Initial overall categories of DHS reflecting the end-user perspective were drafted for further exploration and development with PwD.

Engagement of PwD and Representatives of Diabetes Advocacy Organizations

The participants agreed on the need to define DHS types that are comprehensible and meaningful for end-users and agreed in a codevelopmental process on three overarching DHS categories (Figure 2). Exploring the most important benefits of different DHS, participants highlighted the ease of self-management, the saving of time required for glucose monitoring and diabetes treatment, and the reduction of condition-associated fear and burden, depending on individual treatment modalities and individual circumstances. As potential DHS drawbacks, participants highlighted limited DHS accessibility, challenges related to health and information technology literacy, complexity of use of applications, and information overload. It was also noted that excessive focus on continuous self-improvement (eg, by sharing and benchmarking health data) could cause feelings of stress.

Figure 2.

Categories of digital health solutions were defined on the basis of literature research and then confirmed by representatives of diabetes advocacy organizations.

Through a structured process, participants identified key outcome constructs that were believed to be relevant for PwD for each DHS category and would need to be evaluated in the planned survey research to confirm their relevance in a broader diverse population. . The impact of DHS on physical health, burden of daily treatment, confidence in the ability to manage diabetesand access to care were considered very important across DHS solutions. Figure 3 shows the key topics identified as relevant for each and all the categories. . Negative impacts of DHS and glucose regulation were considered important. The impact on family involvement was seen as moderately important. Burden and management of diabetes-related complications were suggested as additional areas for DHS evaluation.

Figure 3.

Overview of content and structure of the final survey questionnaire for online completion by people with diabetes.

Participants additionally identified the following factors as being important: impact of DHS on peer support, burden of hypoglycemia, importance of psychosocial outcomes as a complement to clinical outcomes. The challenge for PwD in navigating among an abundance of DHS was highlighted as a problem, supporting the importance of standardizing PwD-important evaluation methods. Participants highlighted the importance of examining the potential challenges of health equity when implementing DHS in diabetes and ensuring coverage of the variable individual needs across the wider diabetes population and across the stages of living with the condition. Participants highlighted the outcomes being most important for PwD change over time.

Furthermore, participants provided detailed input on requirements to terminology and style of the questionnaire to ensure a respectful and empowering language. While participants agreed that the items were generally clear, they suggested to remove wording that could be misleading or misinterpreted, provide concrete examples to increase clarity, and adjust terminology (eg, change “blood sugar” to “blood glucose”; change “taking responsibility of [for]” to “managing my diabetes in a way that is right for me”; change “medicine” to “medications”). The asynchronous collaboration phase using a virtual online cocreation platform was used to support the confirmation of both core themes to be included in the survey and specific wordings to capture relevant nuances.

PwD highlighted that digital tools and services should provide them with more autonomy and ability to do things themselves to alleviate the mental burden related to diabetes management and give them an improved sense of control of diabetes. PwD stated that DHS can improve the self-management ability and sense of control, but they also expressed some concerns. They reported anxiety when using social media and networking platforms related to diabetes, raised concerns about the reliability of information found on the internet, and reported concerns and limited progress regarding remote communication with health care teams.

In the cognitive debriefing, PwD suggested to shorten the questionnaire and to modify Likert-scale response formats into dichotomous items to minimize complexity, to lower the cognitive burden, and to reduce the risk of questionnaire fatigue. They also suggested to delete items regarding quality of life and to simplify impact items that do not constitute core constructs.

Questionnaire

The content of the codesigned survey questionnaire to evaluate perceptions and priorities of PwD related to outcomes of DHS is depicted in Figure 3. The questionnaire was designed for use in a large diabetes population to evaluate experiences and expectations toward positive and negative impacts of each of three categories of DHS and to assess the relevance and importance of outcomes for DHS evaluation in diabetes.

The questionnaire consists of brief, short, multiple-choice questions and a few open-ended questions for online self-completion by adults with type 1 or type 2 diabetes. The topics covered and the order in which they are presented are shown in Figure 3. They were developed together with the PwD to ensure that the survey would generate meaningful insights from PwD. Sections A, B, and C include brief questions about the respondent’s profile, care situation, and general diabetes well-being, as these were identified as important by PwD to interpret the respondent’s experiences of DHS. The questionnaire includes few brief standardized PROMs (Perceived Competence for Diabetes Scale, ²⁵ the Problem Areas In Diabetes Scale 5, ²⁶ Diabetes Distress Scale 2 ²⁷ ) since diabetes self-efficacy and emotional distress were identified as potential perception modifiers of DHS impact.

Section D of the questionnaire asked respondents about their concrete experience with using each of the three categories of DHS. Depending on personal experience, respondents answered questions about their perceived or assumed impact of each DHS category on four main outcome domains identified from the insights obtained from engagement with PwD: perceived impact of DHS on physical health, mental health, diabetes self-management, as well as access to care and support. Respondents were also asked to rate the extent of experienced or assumed negative impact related to six key potential drawbacks of DHS for PwD: feeling overwhelmed by too much information, feeling overwhelmed by the demands of diabetes, feeling they are failing their diabetes management, feeling that diabetes takes up too much space in their daily lives, feeling stressed because of not knowing which digital health information to trust, and feeling anxious due to alarming information about risk of diabetes-related complications. Section E of the questionnaire asked questions about perceptions of the impact of DHS as a whole on (1) general and diabetes-specific quality of life, (2) self-care and lifestyle issues specific to diabetes, and (3) experience of autonomy, empowerment, and ability to play an active role. A final question in this section asked respondents to identify up to five most important outcomes that should be prioritized when evaluating DHS. A final open-ended question invited respondents to provide their own narrative about how DHS have affected them and their life with diabetes, either positively or negatively. This offers an opportunity for PwD to highlight additional issues of importance that may not have been captured by closed-ended questions.

Discussion

In this article, we describe (1) the definition of three new DHS diabetes categories comprehensible for PwD and suitable for creating differentiated outcome evaluation strategies, (2) the identification of core positive and negative outcome domains of particular importance to PwD, and (3) the development of a survey questionnaire to collect empirical data regarding the perceptions of the impact on the identified outcome domains and the specific priorities among PwD for DHS evaluation. The population survey research using the questionnaire in a representative population is currently ongoing, and the results will provide a broader evidence base for identifying relevant impacts of DHS on PwD.

In line with recent models for patient-centered impact assessments, ²⁸ we followed a developmental process with the involvement of PwD to identify diabetes outcomes that are specifically relevant for DHS evaluations according to them. Not surprisingly, the majority of these outcome constructs map onto previously established outcomes measured using PRO, including general health and well-being and diabetes-specific quality of life, diabetes-related distress and symptom distress related to complications, treatment and therapy burden and experience, confidence in self-management, and access to continuous person-centered care and support. Past research on PwD-relevant outcome measures either covered technical^12,13 and clinical aspects ¹⁴ or focused on specific DHS purposes such as monitoring, ¹⁵ self-management, ²⁹ outcome measures for clinical trials, ³⁰ and automated insulin delivery technology. ²²

Our findings concur with those of a previous study that identified mental well-being, diabetes distress, and depression as important domains. ¹⁵ Important new constructs specific to DHS, however, involved negative DHS impacts such as being overwhelmed by either information and distrust in information or the use of digital health information.

We recognize three limitations of our study. First, the number of PwD involved was limited, but data from upcoming survey research are aimed at obtaining data from a larger diabetes population. Second, the initial working group comprised PwD and representatives of diabetes advocacy organizations related to type 1 diabetes exclusively. Participants noted the importance to strengthen the type 2 diabetes relevance because several DHS aspects might be specific for either type of diabetes. However, participants had significant experience in advocating for people with different types of diabetes, and people with type 2 diabetes will be prioritized for inclusion in upcoming surveys. Third, the role of DHS at different stages of the condition (eg, newly diagnosed vs long-standing diabetes) is not fully considered in the current questionnaire. However, any extension of the questionnaire would need to be balanced against the questionnaires’ length to prevent questionnaire fatigue.

To the best of our knowledge, this is the first study that seeks to identify specific outcome constructs that PwD find important to assess the impacts of different types of DHS. Our research identified outcomes that can be measured by PROMsand may be further developed into components of a future diabetes DHS outcome set through a systematic instrument design. To further explore, evaluate, and validate the hypothesized relevance of the outcome concepts, we developed a survey questionnaire for use in larger populations with type 1, type 2, and other types of diabetes. The survey study will provide important insights into the heterogeneity and nature of DHS impacts as perceived by PwD across a population with diversity in terms of age, gender, sociodemographics, state of the condition, duration, treatment, and type of diabetes. The growth in innovative DHS holds significant opportunities for reducing the daily burden of diabetes and improving treatment outcomes. Considering outcomes that are important to PwD will be critical to accelerate DHS development, reimbursement, and implementation in real-world settings to ensure equitable access to quality DHS for all PwD. ¹¹ We hope that our work will support future efforts to facilitate multistakeholder agreement on how to scientifically measure DHS outcomes that matter to PwD.

Conclusion

We defined three DHS categories that are meaningful to PwD and can be used to evaluate differential perceived impacts of DHS.Moreover, we identified priority outcome domains related to disease and self-care burden suitable for evaluating benefits and drawbacks of DHS that are important to PwD. We designed a survey questionnaire for the use in real-world studies to further characterize perceptions and priorities of impacts of DHS among PwD.