Abstract
Background:
Clinical trials often face challenges in efficient data collection and participant monitoring. To address these issues, we developed the IMPACT platform, comprising a real-time mobile application for data collection and a web-based dashboard for remote monitoring and management.
Methods:
This article presents the design, development, and usability assessment of the IMPACT platform customized for patients with post-bariatric surgery hypoglycemia (PBH). We focus on adapting key IMPACT components, including continuous glucose monitoring (CGM), symptom tracking, and meal logging, as crucial elements for user-friendly and efficient PBH monitoring.
Results:
The adapted IMPACT platform demonstrated effectiveness in data collection and remote participant monitoring. The mobile application allowed patients to easily track their data, while the clinician dashboard provided a comprehensive overview of enrolled patients, featuring filtering options and alert mechanisms for identifying data collection issues. The platform incorporated various visual representations, including time plots and category-based visualizations, which greatly facilitated data interpretation and analysis. The System Usability Scale questionnaire results indicated a high level of usability for the web dashboard, with an average score of 86.3 out of 100. The active involvement of clinicians throughout the development process ensured that the platform allowed for the collection and visualization of clinically meaningful data.
Conclusions:
By leveraging IMPACT’s existing features and infrastructure, the adapted version streamlined data collection, analysis, and trial customization for PBH research. The platform’s high usability underscores its alignment with the requirements for conducting research using continuous real-world data in PBH patients and other populations of interest.
Keywords
Introduction
The collection of real-world data during clinical trials imposes high demands on researchers due to the need for data integration, structuring, access-protected viewing, exploration, and analysis, especially if derived from various sources (eg, wearables, questionnaires, and symptom/event logs). 1 Nowadays, the data collection process involves the use of digital infrastructures, for example, mobile platforms, that are designed and tuned to address specific trial requirements and researcher expectations. 2 The success of a platform for data collection is based on facilitators, that is, features that reduce the burden of data acquisition tasks by participants, good compatibility with clinical workflows, ease of use for participant data monitoring and data management, as well as the possibility of easily implement modifications to the platform to account for new requirements. On the contrary, the failure of a platform is usually based on the inability of overcoming several barriers, such as the high complexity, issues with interoperability, and lack of technical robustness. 3 Additional challenges for data collection platforms are present if the real-time monitoring of study participants is required, as there is the necessity to promptly identify lack of data availability (eg, nonadherence or technical problems), potential adverse events, and logistical planning. Furthermore, the necessity of manual input is also a very critical, as the complexity of the acquisition of these inputs and user compliance may jeopardize the final success of the platform. 4 Finally, when designing and implementing a novel platform, exploring the experiences of researchers is crucial for the successful uptake.
To address the challenges of real-time and real-world data acquisition in people with diabetes, we developed the mobile platform IMPACT. 5 Real-world data collection in this population is increasingly involving wearable devices, such as continuous glucose monitors (CGMs) and physical activity trackers, whose data automatic acquisition and alignment with manually acquired information is far from trivial. The IMPACT platform is composed of three main components: (1) a mobile app allowing participants acquire data automatically from multiple wearable devices, (2) a web interface for clinical researchers to remotely monitor participants and make dynamic adjustments to study-related device settings, and (3) a custom backend that exposes ad-hoc RESTful APIs to store, ingest, and transfer data on a cloud server to guarantee their secure storage and preservation. Finally, the platform is easily customizable to suit different needs and uses advanced features to minimize the amount of data that patients need to enter manually.
This last key feature made IMPACT a suitable solution for CGM, wearable, and manual input data collection in a different population, that is, in individuals who have undergone bariatric surgery and have postprandial hypoglycemia, a metabolic complication of bariatric surgery. To date, dietary measures represent the first-line treatment to prevent the hypoglycemic events. 6 However, dietary management without guidance of glucose monitoring may not sufficiently alleviate the burden of hypoglycemia. To address these limitations, the use of continuous glucose monitoring (CGM) systems, which measure BG levels in real time every 1 to 5 minutes for several consecutive days, can drastically improve PBH management. 7 In fact, CGM systems, combined with a dedicated mobile application, can help patients tracking their BG levels to promptly identify hypoglycemic events, but also monitor the impact of diet and exercise on the metabolism and receive personalized recommendations. For this reason, recently IMPACT has been redesigned and adapted to meet the needs of the PBH forecast study (ie, Post-Bariatric Hypoglycemia forecasting) (NCT05212207 8 ), which aims to develop a decision support tool to prevent hypoglycemic events in the PBH population. In the specific, IMPACT has been employed in a trial that involves a cohort of 50 participants with PBH after Roux-en-Y gastric bypass. This population used IMPACT for 50 consecutive days, collecting longitudinal data from CGM devices, smartwatches, and electronic manual input of events (meals, symptoms, activity). From the technical point of view, the redesign and adaptation of IMPACT involved the following areas: the mobile application got enhanced by allowing manual sensor calibrations at specific times as required by the study protocol, the web dashboard was enhanced to allow better overall patient analysis and monitoring, as well as more refined remote adaptations of device settings and the platform cloud server were improved to allow backup and better data management features.
The aim of this article is to outline the design process used in the development and tailoring of IMPACT’s core components for the PBH study. We will delve into our nested design approach, with a strong emphasis on the user-centered principles that guided the process and the inherent modularity of the platform. In addition, we will evaluate the effectiveness of the IMPACT adaptation through a thorough usability assessment, including the administration of the System Usability Scale (SUS) questionnaire and the collection of demographic data from the clinical team actively participating in the trial.
Methods
The adaptation process of the IMPACT platform followed a nested design approach. 9 Such design model, depicted in Figure 1, consists of five sequential steps, which consider both upstream and downstream validation to deliver a suitable prototype and ensure effective performance during the clinical trial.
The nested design model, as described in the work by Weijers et al. 9 Each box represents a relevant stage of development and analysis, while the bottom arrows represent the different direction of validation (in blue the validation path of the various steps; in green the sequence of different design stages).
During the process, the requirements from the clinical researchers and participants were collected to suit the needs of the PBH study.
In the next sections, details on the five design steps are reported highlighting the main aspects that have been implemented to maximize the platform usability.
Step 1: Assessing Intended Users and Important Tasks
The first step is the definition of the target user of the IMPACT platform and which are the tasks the user is expect to perform.
The assessment started with a comprehensive literature review focused on understanding the PBH population characteristics, the different types of tasks necessary to monitor the relevant clinical variables,6,10,11 and dedicated interviews with the clinical research team to gather info on their past experience on this specific population. These investigations revealed that previous studies for data collection in PBH individuals mainly relied on paper-based non-structured methods, without any integration with the automatic acquisition of glucose data performed with the CGM sensor. Limitations of paper-based methods are not only a high burden on the user side (both participants and researcher) but also the suboptimal foundation for data processing and analysis.
One of the key requirements that the clinical research team stressed to be useful in a platform for real-time data collection is the dashboard, which should allow monitoring the data acquisition progress, identifying potential failures or malfunctioning of wearable sensors, calling for intervention to recover the situation, and, finally, improving data visualization and management. In fact, efficient trial management requires the most complete overview of all enrolled participants with the ability to filter participants and/or variables according to specific criteria. A further requirement is a marking feature, which has been integrated to easily identify participants experiencing device-related issues or health conditions requiring intervention. One example of this last feature is the notification system implemented to promptly alert clinicians if there is a lack of data synchronization for more than 24 hours, which is designed to prevent data loss and safeguard participants’ safety. The clinical research team also recognized as key to incorporate a feature enabling remote unblinding or blinding of CGM data visualization is an important requirement which should be at the discretion of the clinical investigator. Finally, it has been evidenced as essential the possibility of de-activating on the CGM sensor the urgent low alert, which is a visual/acoustic alarm specifically designed for people with diabetes triggered when a specific very low glucose concentration is predicted, and thus not applicable to the same extent in the PBH population. Strictly connected to the previous point, alert settings have been fully customized according to individual needs and purposes of the study.
In addition to study participants and medical staff taking care of patients, needs of researchers in charge of processing and analyzing data are also important to consider. Their primary requirement is the ability to easily and consistently download the data for further analysis.
Adherence to established standards, most notably the Good Clinical Practice (GCP) guidelines, 12 constitutes an essential imperative for any digital health platform employed in clinical investigations. These guidelines serve as a foundational framework to uphold rigorous quality standards in trial design and documentation by delineating specific prerequisites for all facets involved in clinical trials, encompassing digital platforms. Particularly, throughout the iterative phases of designing and development of IMPACT, we committed to the ALCOA+ principles by ensuring that each data point is Attributable (all data are identified by the patient id to whom it relates), Legible (the data are stored in raw format, without any preprocessing), Contemporaneous (the data are logged in real time, all signals in sync), Original (the database where the data are stored is backed up regularly), and Accurate (the data are collected and stored as per the trial protocol).
Throughout the iterative phases of designing and development of IMPACT, we committed to these guidelines. This helped ensuring that the IMPACT platform could achieve a superior level of usability.
This assessment laid the foundation for the subsequent design levels, ensuring a shared vision and a user-centered approach to platform development.
Step 2: Domain and Data Characterization
The second step of the design process involved the analysis and comprehension of the diverse data types to be collected and visualized. From the literature review of the previous step, 11 it has emerged that blood glucose sampling, symptom tracking, and meal recordings were the primary data that patients need to collect to observe and mitigate PBH events shortcoming. Particularly, fostered by a recent work by Schönenberger et al, 7 the IMPACT platform was designed to minimize the number of manual tasks required of participants by automating data collection wherever possible. Indeed, one key feature of the platform is its ability to collect data with commercially available continuous glucose monitoring sensor, which automatically collects blood glucose data every five minutes and requires only sparse sensor calibrations.
To ensure the high quality of the collected data, it was imperative to define the characteristics, structure, and source of each data type, that is,
Glucose: timestamp, value (numeric, mmol/L) (collected by the mobile application from CGM sensors);
Symptom: timestamp, value (categorical), type (categorical) (collected by the mobile application as manual input from the patient);
Meal intake: timestamp, quantity/type (categorical) (collected by the mobile application as manual input from the patient);
Activity: timestamp, type (categorical), intensity (categorical) (collected by the mobile application from physical activity trackers);
Sleep: timestamp, type (categorical) (collected by the mobile application from physical activity trackers);
Steps: timestamp, value (numeric, step/min) (collected by the mobile application from physical activity trackers);
Glucose threshold to raise a “low glucose alarm”: value (set manually in the web application by clinicians);
Patient weight: value (set manually in the web application by clinicians);
Patient height: value (set manually in the web application by clinicians);
Blind mode: boolean (set manually in the web application by clinicians).
These definitions enabled a seamless and integrated dataflow encompassing data collection, analysis, and visualization, forming the cornerstone of the IMPACT platform’s adaptation for the PBH study, while always prioritizing user-friendliness to maximize their effectiveness in clinical practice.
Step 3: Visual Encoding and Interaction Design
The visual encoding and interaction design of the platform interfaces have been informed by multiple inputs, including the results of the previous tasks, user feedback from platform testing, and iterative validations. High-level mockups were used in the early stages to determine the optimal setup and layout for the different parts of the interface together with clinicians.
The clinician interface received the most attention since the mobile app had been developed based on suggestions of the clinical research team to collect data necessary that may be considered for the development and deployment of a PBH forecasting and potentially decision support team. Therefore, the goal was not to interfere with or specify the behavior of participants to minimize potential confounders. Once the platform is used in a therapeutic intention (eg, decision support to inform about timing and nature of hypoglycemia corrective measures), it will be crucial to perform usability assessments in the target patient population. The main dashboard page has been designed with a grid view to provide clinicians with an overview of all enrolled participants while also allowing them to filter the results. Feedback from clinicians during the validation of this mockup suggested that notifications and the current state of the patient should also be integrated into the view. The final version of the dashboard page and its early-stage mockup are depicted in Figure 2. The design has focused on ease of visualization of the relevant information. Positioned at the top of the page, a search bar allows for efficient filtering and selection of specific participants from the entire enrolled cohort. Adjacent to the search bar, there is a button for enrolling new participants in the study, which has been relocated to the bottom right corner for enhanced accessibility.
Mockup (top) and final version (bottom) of the dashboard page on the website. Here, the clinical researcher can oversee the recruitment status and the proportion of ongoing and completed participants. In red cards, the clinical researcher can easily see which participants require a direct action due to some alarm.
Below these elements, a grid view presents an overview of all subjects participating in the trial. Notably, in Figure 2, participants requiring additional scrutiny in the data collection process are highlighted with a red border, drawing attention to the need for closer examination. Conversely, participants who have successfully completed the study are denoted by a grayed-out card. By tapping on a card, the user can explore the details of the selected patient, navigating to the patient detail view.
The summary and settings section of the participant-facing component has been revised many times during the design phase to condense as much meaningful information in one view while keeping it easily accessible. This section (shown if Figure 3) provides an overview of the patient in the trial, with statistics over the last relevant period and alerts for the specific participant, such as blinding mode and/or threshold customization. The section is divided into two parts, with the top containing all the relevant trial information and possible alerts, and the bottom containing all the statistics to monitor the overall glycemic control of the patient, that is, average glucose, percentage of days with active monitoring in the last period, glucose variability, and Glucose Management Indicator. 13 The design of this section was inspired by the Ambulatory Glucose Profile, which represents the standardized way to report glucose management statistics in people with type 1 diabetes. 13
Patient detail view—summary page first mockup (top) and final deployed version (bottom). The view is divided in two main parts, the top one with all the participant and trial information, the bottom one with relevant statistics and analysis results over the last data period.
The second section of the patient detail page contains a plot that visually represents all characteristics of the different data types. From a visualization standpoint, the most suitable approach for presenting data to users is through time plots, as they facilitate the evaluation of concurrent relationships among different data points. Numeric data types, such as glucose, are straightforward to encode and plot on the platform’s interface, thus the primary focus has been on addressing the visualization of categorical data. For instance, symptom data are categorized based on a limited number of macro-descriptors and the intensity level of the symptom. This classification renders symptom data a hybrid type that necessitates appropriate visualization on the interface. Similarly, meal intakes, physical activity, and sleep are categorized by the patients within the application. The activity and sleep data, which are time periods rather than single data points, are displayed with an area overlay that increases in color saturation to differentiate levels of activity intensity and sleep phase. Steps are encoded with stem type plotting to improve the visualization of active time during the day.
Tooltips, accessible by pressing on the data point, were designed to contain all relevant information about the specific data. The plot was refined during the design process, and the final version is depicted in Figure 4.
Final version of the plot showing the collected data for the selected patient for a specific day. The time plot condenses all the relevant data in a single view, allowing to identify concurrent events and patterns.
Overall, multiple iterations of feedback collection on proposed mockups led to the final design of the platform interface, which has been later prototyped and tested.
Step 4: Prototype Development
The development of the prototype was facilitated by the successful completion of the previous stages.
The whole platform is developed in Flutter, a multiplatform framework which allows to ensure consistency in the visual design and functionalities among all components across different operating systems and eases the development by reducing the required iterations to ensure the same functionality in all platforms. 14 This multiplatform approach allowed us to integrate the design choices from the previous and develop a consistent design among both the mobile application and the clinical dashboard. To optimize usability, availability, and integration of the tool into real-life scenarios, the dashboard prototype was developed as a web application. Throughout the development process, multiple update meetings were conducted with the clinical team to showcase progress and enhance the visualization of data in different sections.
Step 5: First Testing and Validation
The prototype underwent multiple iterations of testing and validation with users, whose feedbacks on feature requests and general improvements were collected. The primary platform testing session lasted two weeks, during which clinicians acted as patients and used the mobile application to gather data, both manually and automatically. In parallel, they reviewed the IMPACT system features (mobile application and web dashboard) and user experience. This real-world simulation provided valuable feedback on the usability of the platform, allowing for the identification and resolution of minor issues with the implemented data logging and alerting mechanism. The feedback also led to the development of additional features, such as the ability to export participant raw data for research purposes on demand and to export a pdf report with the clinical team in charge of participants’ usual care. Overall, the testing and validation process ensured that the platform was ready for deployment in the actual clinical trial.
Assessment of Usability
After completing all the design and development steps, the platform was deployed in the PBH study. During the study period, which started in January 2022 and will be completed by end of September 2023, further feedback was collected, and minor changes were made to the platform, including the ability to restore a previous backup of data for a specific patient upon app reinstallation.
At the end of the 50 days data collection period, usability feedback was collected from the clinical team regarding the usage of the web-based dashboard. To this aim, usability has been measured using the validated 10-item SUS questionnaire. 15 Briefly, SUS is a flexible questionnaire designed to assess the usability of any system interacting with human users. The SUS is relatively quick and easy to complete. The questionnaire consists of ten alternating positive and negative statements that are scored on a five-point scale of strength of agreement, with final scores ranging from 0 to 100 (with 100 indicating the highest usability result). Generally, a system with a score above 70 indicates acceptable usability; a lower score means that the system needs more scrutiny and further improvement. 16
Along with the SUS, demographics and self-reported levels of experience with digital health tools on a one to five scale were also collected from the clinical team. These data have been collected to further investigate possible factors that influence usability and type of experience with the dashboard.
All data were collected through an ad hoc website accessible only to the clinical team involved in the trial.
Results
Platform Adaptation
The multistep design and validation which consisted in multiple testing iterations, allowed to build a solution tailored to the PBH needs. The adapted IMPACT platform has been tested by the clinical team for two weeks to ensure all features were implemented as intended and has been then deployed in the prospective observational study involving 50 participants for 50 days of monitoring.
Usability Assessment
Nine members of the clinical team (average age 29 ± five years) have filled the questionnaire. Among them, four clinicians also contributed to the early and late design phases and testing of the prototype. Table 1 shows the results of the SUS among all members.
Results of the Usability Assessment.
The table shows the results for the specific questions of the SUS with collected details of the subject (sex, age, experience with digital tools). each score ranges from 1 to 5, with 5 being the maximum for the odd questions, and 1 being the maximum for the even ones.
On average, the self-reported level of experience with telemedicine and digital health tools was 4.25, which denotes that the team was already well-experienced in the use of digital tools in the clinical practice. The mean SUS score for the assessed platform is 86.3, indicating good usability according to the threshold proposed in the work by Bangor et al 16 of a score above 70. In addition, the standard deviation of the scores is relatively low, at 10.8, suggesting that there is a general agreement among the participants on the usability of the platform.
Looking at the individual question scores, the questions with the highest scores were Q2 (ease of use), which scored 1 (on the reverse scale where 1 is the best score), and Q5 (integrated well). The questions with the lowest scores were Q6 (confidence in using) and Q9 (need for technical support), which scored 2 and 3, respectively.
During the collection of the SUS questionnaire and during the prospective study for longitudinal data collection, we have also received various direct feedbacks from the clinical research team. Researchers appreciated the features of platform, in particular, its flexibility which allows to implement study-specific and population-specific requirements. The user experiences were, however, negatively influenced by rare events of limited technical robustness which resulted in frequent connectivity issues. Resulting data gaps not only affects data quality but also results in frequent interactions with study participants thereby increasing the burden of study participation. These issues have been investigated and mitigated during the trial period with subsequent updates of the mobile application.
Discussion
Overview
This article reports the results of the stepwise development and usability evaluation of the IMPACT mobile platform for easing the conductance of clinical research in the PBH population with continuous multi-level data collected in real-life settings. According to the obtained results, the design and development approach used in this work has proven to improve the quality of the final application, ensuring that both features and their usability are always developed with the same level of detail and focus. Moreover, this work showed that the collaboration between clinical and technical teams during the development of new solutions leads to (1) an optimal developing ground for realizing meaningful visualization of complex health information, (2) the maximization of the platform adoption rate, and (3) improving CT conduction. Unfortunately, this development approach is not yet common within health care. 4
Usability Results Analysis
The SUS individual questions scores can give some further insights when deeper analyzed. In particular, Q2 and Q5 obtained the highest score, which suggests that the platform was perceived as easy to use and well-integrated into the workflow of the clinicians. On the contrary, Q6 to Q9 received relatively high scores, but the lowest among all collected. This indicate that the users had some concerns regarding their confidence in using the platform and the need for technical support.
Literature suggests that age and previous experience with technology are two factors that may impact the perceived usability of digital health tools.17,18 Therefore, the high SUS scores in this study could be partly explained by the demographics of the sample, which consisted of young adults with experience with digital health tools. This factor considered, clinical practice has high digital tool utilization nowadays, so it is crucial to continue researching and focusing on usability of these tools.
One of the main concerns in the usability has been the increased burden on the clinical and technical team to troubleshoot the connectivity issues that arose in the early stages of the trial period. These troubleshooting iterations had a negative impact on the usability of the platform, also meaning numerous interactions with patients were required, which led to lower adherence to the trial protocol in some cases. The connectivity issues have been identified in many cases to be primarily attributed to human factors. This challenge could potentially be addressed in future works by enhancing and promoting user education regarding the correct utilization of the platform, along with readily available in-app tutorials aimed at enhancing user understanding and proficiency.
Overall, the results suggest that the platform developed for the clinical trial has good usability and is well-integrated into the workflow of the clinical team. The feedback regarding the need for technical support and confidence in using the platform can be used to identify areas for improvement in future iterations of the platform. However, it is important to note that usability is just one aspect of technology acceptance and adoption, and future research should investigate other factors such as usefulness, user satisfaction, and adoption intentions.
Limitations
The platform has been adapted for the specific needs of the PBH study. While showing high modularity and customizability, IMPACT could have been enhanced with other useful features. For example, an unmet need concerns the nonintrusive collection of meal data beyond a semi-quantitative approach as implemented in the current work.
Moreover, patient could have been kept in the loop during the development phases to further improve their experience. This has not been done because of the strict protocol requirements of the study, but it will be done in possible future iterations of the platform. Of course, we plan to perform usability assessment of the mobile application at the completion of the study also on a sample of patients.
The developed platform underwent several iterations of design and development, which ensured that IMPACT met the needs of its intended users. However, there were some limitations to the development process that may have impacted it, such as not having performed dedicated interviews with patients on the desired features and requirements for the mobile app.
Finally, the usability assessment involved a small sample size, which may have limited the platform’s ability to identify all potential usability issues. In this sense, while the platform was designed with input from clinicians, it is possible that certain features or design elements may not have been fully tested or validated by users with diverse backgrounds and experiences.
Conclusions
The adaptation of the IMPACT platform to the needs of the PBH study has shown promising results in terms of usability and acceptance among clinical researchers. The involvement of clinicians in the development phase was fundamental to design a platform that could effectively address study-specific needs and workflows. This multi-step design and validation model has proven its effectiveness in the field, improving the quality of the developed solution. The collection of usability data through the SUS questionnaire provided valuable insights into the strengths and weaknesses of the platform, which will guide future improvements. The high SUS scores obtained, together with positive feedback from clinicians, support the potential for this platform to become an effective tool for conduction of pilot CTs.
Footnotes
Abbreviations
CGM, continuous glucose monitoring; PBH, post-bariatric hypoglycemia; SUS, system usability scale; CT, clinical trial.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work has been partially funded by DVTDSS project (“SID-Networking Project 2021” initiative, Dpt. of Information Engineering, University of Padova, Italy).
