Abstract
Objective: Family health history (FHx) is an important tool in assessing one’s risk towards specific health conditions. However, user experience of FHx collection tools is rarely studied. ItRunsInMyFamily.com (ItRuns) was developed to assess FHx and hereditary cancer risk. This study reports a quantitative user experience analysis of ItRuns. Methods: We conducted a public health campaign in November 2019 to promote FHx collection using ItRuns. We used software telemetry to quantify abandonment and time spent on ItRuns to identify user behaviors and potential areas of improvement. Results: Of 11,065 users who started the ItRuns assessment, 4305 (38.91%) reached the final step to receive recommendations about hereditary cancer risk. Highest abandonment rates were during Introduction (32.82%), Invite Friends (29.03%), and Family Cancer History (12.03%) subflows. Median time to complete the assessment was 636 s. Users spent the highest median time on Proband Cancer History (124.00 s) and Family Cancer History (119.00 s) subflows. Search list questions took the longest to complete (median 19.50 s), followed by free text email input (15.00 s). Conclusion: Knowledge of objective user behaviors at a large scale and factors impacting optimal user experience will help enhance the ItRuns workflow and improve future FHx collection.
Keywords
Introduction
Family health history (FHx) is an important tool for assessing a patient’s risks of specific health conditions such as cancer. Knowledge of FHx could help healthcare professionals determine the potential risks for an individual and recommend targeted intervention and preventive measures. Web-based, patient-facing tools have been developed to systematically collect and organize FHx outside of the provider’s office, improving efficiency of clinical visits while increasing completeness and accuracy.1–4 These tools allow users to generate pedigree, collaborate with family, provide decision support, and support health information technology (HIT) integration.
Automated conversational agents (chatbots) have emerged as a promising approach for health data collection. Chatbots promote a friendly, intuitive, and engaging user experience.5,6 One such effort is ItRunsInMyFamily.com (ItRuns) - a web-based, simple, and scalable tool that engages users in a conversation using a chat interface to collect FHx and provide recommendations regarding hereditary cancer risks. 3 Research has demonstrated data collection chatbots (such as ItRuns) reduce the workload on users and are perceived as more usable and satisfactory compared to standard web-based forms, even when collecting granular FHx. 6
Collection of FHx can involve detailed questions and significant time investment, making it imperative to provide users with an optimal and engaging experience in using FHx tools. Very few studies have explored the user experience of these tools. 1 Research has found that FHx collection tools can produce poor user satisfaction with respect to assessment format, data-input method, and complex clinical jargon. 2 Patients’ incomplete knowledge of family members’ health history and confidentiality concerns remain as barriers to FHx tool adoption.7,8 In addition, factors such as time taken to collect comprehensive FHx and types and amount of data inputs could also lead to under-utilization and poor user experience.
Research has focused on understanding user experience and preferences employing qualitative and experimental approaches such as surveys, interviews, comparative assessments, and randomized control trials.2–4,6,9 The purpose of this study is to explore quantitative software telemetry (collection of software usage information) to assess granular, objective user behaviors, abandonment (where and when a user leaves the FHx tool) trends, and time to complete various steps to identify potential pitfalls and areas of user experience improvement. 10 To the best of our knowledge, no studies have explored user behaviors of completing FHx assessments using telemetry data. In this paper, we explore the software telemetry to assess user behaviors at a large scale and identify potential factors impacting user experience. The insights gathered from this quantitative evaluation will allow us to enhance the overall user experience of ItRuns assessment.
Methods
ItRunsInMyFamily.com and November 2019 10K campaign
ItRuns is a web based FHx collection and hereditary cancer risk assessment tool developed by researchers at the Medical University of South Carolina and ItRunsInMyFamily.com LLC. ItRuns is a free, secure, browser-based, and mobile-first application that does not require the user to download software or to create an account. 3 ItRuns uses a dialogue-based text chat interface to mimic human-to-human conversation providing its users a natural and engaging experience analogous to talking with providers. The current version of ItRuns uses an innovative chatbot interface that simulates a natural conversational dialogue to collect FHx from users. The chatbot engages users in a structured and intuitive way, rather than multiple pages of online forms, tables, or a series of complex questions compared to other tools. Upon completion of the ItRuns assessments, a personalized risk assessment report (section 3.2.1) is sent to the user’s email address. 3
This paper reports on data collected from an online FHx collection campaign conducted using ItRuns. 11 This campaign was conducted between November 1, 2019 and November 30, 2019 in conjunction with Family Health History Month. 11 The ItRuns team aimed to recruit at least 10,000 English-speaking users via paid and unpaid marketing campaigns through social media promotion, contacting previous users of ItRuns, using promotional banners on the ItRuns website, and direct contact with cancer support groups. Interested users completed the assessment using the ItRuns assessment link on mobile devices or computers. Users consented to take the ItRuns assessment by clicking on consent checkboxes within the assessment. This secondary analysis study was deemed not human subject research by Medical University of South Carolina Institutional Review Board (Pro00094990).
ItRunsInMyFamily workflow
This section provides a step-by-step description of the ItRuns assessment workflow divided into 13 subflows (SF) as explained below (See Figure 1). ItRuns assessment workflow.
SF 1 | Introduction: The assessment begins with an introduction to Dokbot - the chatbot persona users engage with during the assessment. 12 Dokbot introduces the user to the purpose of ItRuns and obtains informed consent.
SF 2 | Basic Demographics: Upon consent, Dokbot collects the user’s name and gender. Dokbot also asks if the user is 18 years or older. If yes, Dokbot continues with the assessment. If not, the assessment ends with a message explaining that users have to be at least 18 years of age to take the assessment.
SF 3 | Patient Cancer History: This step collects users’ cancer history. If the user reports a history, Dokbot collects information related to the type of cancer and age of diagnosis. Some cancers include specific follow up questions. For example: for breast cancer, questions such as if the cancer was in one or both breasts.
SF 4 | Genetic Testing: Following the personal cancer questions, Dokbot asks the user if they have ever been tested for a genetic mutation in a cancer-causing gene. If answered yes, Dokbot asks if the result was positive and, if so, for which gene.
SF 5 | Family Cancer History: Dokbot then asks if anyone in the family has had cancer. If the user reports yes, Dokbot asks which relative, their name, cancer, age of diagnosis, as well as cancer-specific follow up questions, if the relative had any other cancers, if they are still alive, their current age or age when they died, and whether the family member has ever been tested for a genetic mutation in a cancer-causing gene. This sequence is repeated for each relative the user reports as having been diagnosed with cancer.
SF 6 | Pedigree: Dokbot asks the total number of daughters, sons, sisters, brothers, maternal aunts, maternal uncles, paternal aunts, paternal uncles in their family. Numbers of nieces and nephews were not assessed due to the complexity of the data and its limited utility. For female users, age when they had their first child is assessed during this sequence. In this manner, a 3-generation pedigree is collected.
SF 7 | Physical Traits: Dokbot collects users’ age, weight, and ancestral origin. If users indicate that they are of European descent, Dokbot asks whether they have Ashkenazi Jewish ancestry as these individuals are at a higher risk of BRCA gene mutation. 13
SF 8 | Female Details: If the user is a female (indicated in SF 2), they are asked the age at menarche; if they have had their breasts, ovaries, or uterus removed, if they have gone through menopause and at what age; if they take hormone replacement therapy if they had a mammogram. If they have had a mammogram, Dokbot asks the date of the last mammogram and the history of dense breasts. They are also asked if they’ve had a breast biopsy, if yes, number of biopsies, any abnormal findings and type of abnormality.
SF 9 | Colorectal Cancer Screen: Dokbot asks all users about the history of colonoscopy. If yes, Dokbot further asks if polyps were found and, if so, the number and type.
SF 10 | Lifestyle: Dokbot collects a user’s zip code, smoking history, tobacco use and alcohol use. If the user reports having smoked cigarettes, Dokbot asks for their age when they started smoking and the age when they stopped; if they report they are no longer a current smoker, what number of cigarettes per day when they did smoke. Dokbot also collects ibuprofen and aspirin use. SF 11 | Thank you and email: Dokbot thanks the user for participation and collects the user’s email to send the ItRuns risk analysis report.
SF 12 | Recommendations and Report: Dokbot provides high-level risk recommendation based on family health history and guidelines for hereditary cancer risks. Dokbot also informs the user that a detailed report (section 2.4.1) will be shared via email.
SF 13 | Invite Friends: Users have an option to share ItRuns assessment with friends and relatives on Facebook by posting on their public profile.
ItRuns risk assessment report
ItRuns uses ontologies, ontological reasoners, clinical practice guidelines, and web services to provide evidence-based recommendations to users based on their FHx. Owlready2 ontology module and Protégé ontology platform were used to develop a lightweight, patient-centric clinical practice guideline domain ontology using hereditary cancer criteria from the American College of Medical Genetics and Genomics and the National Cancer Comprehensive Network. The development of this ontology-driven clinical practice guideline criteria risk assessment is published in a separate publication. 14 A risk analysis PDF report is emailed to participants who completed the ItRuns assessment. The report contains an Executive Summary, followed by a Guidelines section consisting of the hereditary cancer predisposition criteria the user meets and the published recommendations for cancer predisposition assessment they should follow. Next, the Health History section includes information about the user and their health history (including a family pedigree), a breakdown of relatives with cancer, relatives who are also at risk, and family cancer statistics for the user’s family. Finally, the Recommendations section includes additional details about the hereditary cancer syndrome the user for which the user might be at risk, available genetic counseling resources, and additional genetic testing information. The About section includes content about the product, contact information, support, references, and a legal disclaimer.
Study participants
Potential users were invited via paid and unpaid campaigns as described in Section 2.2. They provided consent to participate in the study at the beginning of the assessment. Adult users (≥18 years) were allowed to continue with the assessment after providing consent. Users were asked for their email address to send their Cancer Risk Assessment report at the end of the assessment. No other personal identifying information was collected.
Telemetry data
During the assessment, the Dokbot software collected and stored in a relational database telemetry data detailing the user’s behavior interacting with the software, including timestamps of all actions, transitions between steps, geographical location, referral URL, browser information, and supported languages.
Data analysis
Once the campaign was completed, telemetry data was extracted from the Dokbot database. A script was run to remove small amounts of unusable data, including records where a user loaded the ItRuns workflow but did not submit any single step as well as researcher/developer test records. Then, we extracted elements such as unique random user ID, step description, step start and end timestamps, etc. to assess the experience.
Using an analysis script, we quantified the telemetry data to calculate the usage, abandonment rate, and time per step. We tabulated and graphically represented this data using Microsoft Excel software. Descriptive measures were used to obtain the frequencies, percentages, mean, and median related to completion, abandonment rate, and time taken for each subflow and related steps.
Results
Summary of completion rate, departures per step, abandonment, and average time taken for key assessment steps. The time spent on the Goodbye step was not calculated as the users may or may not have closed the browser window after completing the assessment, leading to inaccurate time.
Overall abandonment by steps
High abandonments were seen in the Introduction subflow (32.81%) and in Invite Friends (23.90%) subflow where users were asked to invite friends and family members to complete the ItRuns assessment via Facebook. Collecting Family Cancer History subflow had the third largest drop of 12.03%. See Figure 2. Percentage of retained users by subflows.
All users (11,065; 100.0%) first experienced the Introduction subflow with the highest abandonment rate (32.81%). Within this subflow, 22.03% of users left at the first introduction step (Figure 3), followed by a 5.71% drop when users were asked to agree to the consent statements of the assessment. Stepwise abandonment in the Introduction subflow.
A high drop off from 11,065 to 3841 showed 34.7% of all users continued to Invite Friends. However, of the 3841 there were 2645 (68.9%) who left the step. Therefore, 1196 out of 11,065 (10.8%) completed Invite Friends subflow.
The third largest overall drop was seen in the Family Cancer History subflow (Figure 4). Users may have to enter cancer history details of one or more family members (e.g., mother, father, etc.), so the user may complete all or some of the steps in this subflow multiple times. In this study, we did not determine when the user abandoned the assessment if they completed the subflow multiple times. The highest (5.26%, 43.73% of the subflow) abandonment was seen during the step when users were asked about which of their family members had cancer followed by relative’s names (2.89%) and types of cancer they had (2.08%). Stepwise abandonment in the Family Cancer History subflow.
Abandonment by steps
Top abandonment by step.
Time per step
The median time to complete the assessment was 636 s or 10.60 min. Table 1 demonstrates the total median time spent on each subflow. Users spent the highest median time to complete the Proband Cancer History (124.00 s) followed by Family Cancer History (119.00 s), Female Details (67.00), Pedigree (55.00 s), and Lifestyle (53.00 s). Users spent the least amount of time in Basic Demographics (7.00 s), Lifestyle (8.00 s), and Invite Friends (9.00 s) subflows.
Top steps with highest time per step.
Question types
Abandonment and time taken per question type.
Discussion
We sought to understand the user experience of ItRuns to enhance FHx collection. This paper quantitatively explores software telemetry data to identify actual user behaviors and actions using software telemetry. Our findings suggest that factors such as lack of knowledge or interest in FHx, trust in FHx tools, data sharing and confidentiality concerns about sharing personal and family health information, lack of collaboration with family members, and increased time spent on conversational text-based workflow can contribute to suboptimal user experience. We discuss potential considerations for improving user experience of FHx tools and implications for ItRuns.
Main findings
About one in three users (32.81%) abandoned the assessment in the initial Introduction subflow with a 22.03% drop after the greeting statement. We hypothesize that potential reasons for this drop could be the lack of knowledge or interest in cancer risk assessment and distrust or privacy concerns in using a casual, informal, chat-based agent to share personal and family health information. In a previous study, users expressed concerns regarding confidentiality of sharing FHx using digital tools. 7 In healthcare chatbot applications, lower reading level and informal language is sometimes perceived to be less trustworthy. 15 We observed abandonment when asked to provide their consent to begin assessment. While ItRuns ensures users that their data is protected with compliance regulations (including HIPAA, GDPR, and CCPA), concerns over sharing personal health information with a third-party solution (not directly recommended by their provider or as a part of a marketing campaign) could restrict users from moving forward. Research shows that while patients are comfortable with sharing personal health information with their own providers, they often restrict sharing information outside of their care network. 16
Over 12% of user drop was seen in the Family Cancer History subflow. Commonly, users abandoned the assessment when asked about which relatives had cancer, their names, and types of cancer they had. As seen in previous research, users may have abandoned at this stage of the assessment if they did not have the correct or complete information about family cancer history.4,17 As ItRuns currently does not allow live collaboration with family members or return to complete assessment at a later time, users could abandon the assessment considering the potential lack of specific family health information. Also, the distrust of the technology and skepticism to share personal health information of family members with ItRuns (a third-party solution) could also play a role. In addition, the looping nature of this workflow required users to add one or more family members’ information one-at-a-time and with one or more occurrences of adding cancer and genetic testing information. About 7000 users completed some steps of these workflows over 22,000 times (average of three repetitions per user). This repetitive nature of flow took an average of 119 s to complete with the top steps averaging 53.00 s.
A high (23.90%) drop was seen at the end of the workflow when users were asked if they would like to share ItRuns assessment with family members and friends via Facebook. While waiting for their risk assessment results, the assessment asked, “While you are waiting on your report, will you invite your friends and family to use ItRunsInMyFamily so they can learn their family cancer risk as well?”. Research suggests that users may not be interested in sharing health information online but rather personally via direct communication. 18 People tend to be cautious of sharing health information online, as few formal policies exist to ensure protection of users’ information and the ease of which aggregated health information can be re-identified.19–21 In future, development of approaches such as sharing with family and friends personally via emails or text message could help better promote ItRuns and spread awareness about FHx collection.
Users took about 636 s (10.60 min) to complete the assessment. This time is lower compared to reported times for FHx collection in-person or using digital health tools (average 15 to 60 min).1,7,8,22 In a previous study, the previous version of ItRuns took a higher time compared to standard FHx web-based form. 6 Our initial hypothesis was that ItRuns will require a high completion time resulting in increased abandonment. As ItRuns uses a conversational text message-like interface, users completed certain questions (types of cancer and genetic tests, family member details) multiple times, not aware of these repetitions at the beginning, able to see all information at once, or recollect previously added information without scrolling up the chat history. Also, in the version, users were unable to change any incorrect responses. In contrast to our hypothesis, ItRuns took less time to collect FHx. This could point to the potential benefit of ItRuns in reducing the FHx collection time, while employing an intuitive and friendly interface. Previous research suggests that users perceive chatbots as intuitive, engaging, and friendly for FHx and other health data collection. 6
Users spent the longest (19.50 s) in responding to search list questions. The search lists included cancer types (∼100 options) and genetic test types (∼160 options). From the cancer list, ‘Other Cancer’ was the 4th most common option users chose. 23 As previously reported, the position of the ‘Other Cancer’ option and lack of cancer synonyms (uterine cancer instead of endometrial cancer) could have led to selection of this choice. 23 Also, if users misspelled a cancer, they might not see the correct option due to the filters. These factors could have led to higher time spent on these subflows. Users also spent about 15 s on the question asking for their email address. It is possible that users may have taken time to decide which email address to provide and type it in the text field, which is time consuming. Although, ItRun’s privacy statement ensures users that their personal information will not be shared, an initial hesitation to submit personal email address in a marketing campaign could have resulted in a longer response time. We saw a total of 6.69% abandonment in two text input questions asking for user’s name or family member names, which may likely be due to hesitation to share their own family members’ personal information.
Limitations
The majority of participants were women between the ages 40 and 60 years. The findings may not be representative of the general population. Although, research shows no noticeable age and gender group differences in technology literacy compared to our population. 24
We aimed to recruit a large sample (over 10,000 individuals) employing a wide cast marketing strategy. This could have resulted in individuals completing the assessment inaccurately or without genuine interest and providing fake data. It is possible that individuals may have completed the assessment in a hurry or taken additional time to complete the assessment at their own convenience. In future research, we aim to recruit a representative clinical population to validate these findings.
The lack of collaboration capabilities for adding personal or family health history within ItRuns could have led to increased times for completing the assessments. Allowing ItRuns users to collaborate with family members and the ability to interact with personal or electronic health records could promote better experience and engagement.
ItRuns collects information in a conversational chat-based approach potentially resulting in a higher time to complete the assessment. However, research shows that users prefer the chat-based experience compared to standard, web-based data collection approaches. 6
The analyses presented in this study were conducted with data extracted from a large-scale software database. The validity and quality of large data sets can vary substantially. 25 It is particularly important to validate minimum database quality in healthcare contexts. 26 Our team completed error checks and validations to ensure data integrity.
Future work and implications
This quantitative exploration has future research implications. To the best of our knowledge, we are one of the first teams to conduct a FHx public health campaign to recruit a large-scale sample of over 10,000 individuals and contribute to user experience assessment by analyzing software telemetry data.
Based on this research, we recommend the following considerations to enhance the user experience of end user facing FHx tools and describe how we will improve ItRuns in future.
In addition, future research could also address the following: This study did not directly study the user’s preferences, but rather via their software usage and actions. Future studies could supplement such analyses with user interviews or focus groups to better understand their experience, perceptions towards ItRuns, and test the hypotheses for assessment abandonment at various stages. This study did not aim to assess the quality of data gathered during this study. Future researchers should also consider assessing the accuracy and completeness of the data collected using the chatbot interface.
Lastly, our approach could be readily applied to other healthcare assessments to evaluate user experience and engagement. As a part of this research, we have developed a generalizable program script which could be used to extract and summarize telemetry data of assessments developed using chatbot application- Dokbot. 12 We aim to continue studying and improving on other custom and standardized Dokbot assessments to enhance patient and provider experience of healthcare data collection.
Conclusion
This paper quantitatively explores the user experience of ItRunsInMyFamily.com FHx assessment. We used a unique approach by employing software telemetry to assess actual user behaviors related to abandonments and time spent on various ItRuns Steps. The findings provide insights on factors impacting user experience and assessment completions. Key factors include the potential lack of knowledge or interest in FHx, lack of trust in FHx tools, confidentiality concerns about sharing personal and family health information, lack of collaboration with family members, and increased time due to conversational text-based workflow. We discuss considerations for improving FHx collection experience and how our findings will help in the development of the future ItRuns versions for better user experience and engagement.
Footnotes
Acknowledgements
We thank Kurt Hills (senior technical writer, Doxy.me Inc) for providing his time and expertise in reviewing this manuscript.
Declaration of conflicting interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Dr Welch is a shareholder of Doxy.me Inc, Dokbot LLC, 
, and Adhere.ly LLC, all of which integrate Dokbot into their software solutions. Heath Morrison is a shareholder of Doxy.me Inc, Dokbot LLC and ItRunsInMyFamily.com. Dr Schiffman is an employee of PEEL Therapeutics, has ownership interests in ItRunsInMyFamily.com and PEEL Therapeutics and serves in a consulting role in N-of-One and Fabric Genomics. Dr Bunnell holds a leadership position in Doxy.me Inc and is a shareholder of Adhere.ly LLC. Authors Hiral Soni, Dinko Vasilev, Triton Ong, Hattie Wilczewski, Alexa Narma, Julia Ivanova are employees of Doxy.me, Inc, a commercial telemedicine company and declare no conflict of interest. Other authors report no other conflicts of interest.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Cancer Institute (NCI) grant Identifying hereditary cancer patients in low-resource community cancer settings using an innovative informatics solution (1R41CA239867-01A1). The content is solely a responsibility of the authors and does not necessarily represent the official views of the NCI. Dr Welch was funded by the National Cancer Institute (Grant Number K07CA211786 and 1R41LM013419-01). Dr Bunnell was funded by the National Institute of Mental Health (Grant Numbers K23MH118482 and R41MH126734). Dr Allen was funded by the National Cancer Institute (Grant Number 1F99CA253576-01).