Abstract
Objectives
Health risk assessment (HRA) questionnaires have become a popular tool to help quantify health issues within populations. Over the last decade HRAs have increasingly been delivered in the online environment. The objective of this study was to create and validate an HRA that is optimized for delivery via the Internet.
Design
After an iterative process of user testing and interface design the RedBrick Health Assessment (RBHA) was validated against known domain specific questionnaires with 464 working Americans, and with medical claims data from over 25,000 employees.
Setting
All consumer testing, data capture and analysis occurred at the offices of RedBrick Health Corporation, Minneapolis, USA and via a secure online portal.
Participants
Individuals in full-time employment in the USA, who were between 18 and 65 years of age at the time inquiry.
Main outcome measures
Correlation of the included RBHA domains with the output from known gold standard health question sets for each assessed health domain.
Results
The iterative development process employed in creating the RBHA produced a tool that had a high degree of user acceptability. The domains demonstrated good correlations with relevant gold standard questionnaire measures, good internal consistency, and acceptable sensitivity and specificity when compared to gold standard risk stratification and high-risk classification (specificity of domains ranged from 76–94%). A test–retest correlation co-efficient of 0.7, or greater, was achieved 8 weeks after initial completion.
Conclusions
The RBHA is a new breed of HRA that has been specifically developed for capturing health status information in an online environment. At its heart is user centricity and this focus has enabled the creation of a tool that is not only highly engaging but also captures accurate and robust health status information.
Background
Over the last decade health risk assessment (HRA) questionnaires have become increasingly popular tools used by organizations and health insurers to help quantify the health issues of their employee or member base. 1 A recent survey found that 74% of US and 46% of UK companies offer an annual HRA to their employees as part of their employee health management initiatives. The popularity of this method of data capture is likely to increase in the coming years as more organizations accept that maintaining a healthy workforce can deliver real productivity and performance benefits. 2
DECLARATIONS
PRM consults for organizations in the UK and US, through his company Glasslyn Health Solutions, on the development of technology-focused health management solutions; WSM, CMB, JRWB and EMZ are all employed by, and hold stock in, RedBrick Health Corporation; PRM, WSM, CMB and JRWB are all listed as inventors on the RBHA patent application
Funding for this research project was provided by RedBrick Health Corporation
Formal ethical approval was not sought for this study as it was conducted outside of a hospital or academic institution. However, all participants were required to accept privacy and security terms that were specific about how their data were to be used, who had access to the data, and where the data would be stored. All data were stored and protected in accordance with the Health Insurance Portability and Accountability Act (HIPPA) 1996. Individuals were explicitly informed that they were free to cease participation in the study at any stage during its course and have their data deleted from our database
PRM
PRM, WSM, CMB, MEH and JRWB developed and designed the RBHA; PRM and WSM designed the investigative protocol; WSM and MEH performed user testing; PRM analysed the data; CMB coordinated data collection and extraction; EMZ coordinated the employee participation groups; PRM, WSM and EMZ wrote the manuscript
Kaveh Sanati
HRAs typically capture responses on a wide range of medical, lifestyle and behavioral topics. In addition some also inquire about biometric and quality of life parameters in order to create a broad overview of an individual's health status. A review of the historical background, scientific basis and uses of HRAs is beyond the scope of this paper, however the reader is referred to two good overview chapters in the most recent edition of the American College of Sports Medicine's Worksite Health Handbook for context.3,4
HRA data are valuable in a number of distinct ways. In their amalgamated form they can help organizations and healthcare providers target appropriate resources for individual and group-wide interventions. 5 In addition, together with historical medical and pharmacy cost data, they can also help predict future cost liabilities, both in terms of medical care requirements, but also in relation to work performance and productivity.2,6–8 From the individual perspective they can help respondents better understand their own health status, how they compare to others like them, as well as the specific areas that they should focus on improving. It is unlikely, however, that this knowledge alone will effect significant population-wide behavior change. Rather HRA completion needs to be augmented with appropriate consumer-focused health improvement programs to achieve health benefits. 3
The last 10 to 15 years has seen a wealth of research on predicting costs of medical care, absenteeism, productivity and disability.9–15 In the majority of these studies data from an HRA were an essential first step in defining the ‘at-risk’ population and observing how specific health risks relate to the different cost parameters.
With the increasing penetration of the Internet in both business and society as a whole, HRAs have shifted from being a predominantly paper-based solution to being almost exclusively delivered via the world wide web. 16 The advantages of this approach are significant; web delivery allows for a lower cost of implementation as well as almost limitless scalability compared to paper implementation. These practical considerations, together with the interactivity and ability of the web to serve up ‘real-time’ information and recommendations to consumers, make it the main delivery channel for the majority of vendors and purchasers of HRA services.
Over 75% of the US population has access to the Internet, with the vast majority of these individuals having broadband access. 17 Outside of the US a number of countries in Europe and Asia have even higher population penetration of Internet services. With this almost ubiquitous Internet presence in our lives it is perhaps surprising that there has been so little published work examining how to optimize the HRA experience for online delivery. Many commercially available HRA products started out as paper tools that subsequently got placed into online environments, with little or no modifications to layout or emphasis. In addition, despite their extensive use there are very few that have gone through a rigorous questionnaire validation process. 5
Although evidence does suggest that responses are not significantly different when comparing completers of paper and online versions of the same questionnaires there would appear to be some utility in creating a better, more robust and validated online HRA that makes use of the unique characteristics of the 21st century world wide web. 18
Recently the Patient Protection and Affordable Care Act in the United States has mandated that an HRA needs to be part of the annual wellness visit Medicare beneficiaries are entitled to. Similarly, healthcare reform in the UK is placing much greater emphasis on preventive health strategies as a way to mitigate increasing healthcare delivery costs.
In this article we report on the design, development and validation of a next generation HRA. The RedBrick Health Assessment (RBHA) has been created specifically for the online environment so as to maximize engagement, understanding and accuracy of completion as well as minimize respondent time commitment necessary to achieve this.
Methods
The development of the RBHA involved four separate phases: (1) design of the graphical user interface; (2) development of the question set; (3) user testing and face validation; and (4) validation of the included domains against domain specific ‘gold standard’ measures.
Design of the graphical user interface
The graphical user interface (GUI) design process started with a review of existing, easily accessible, online HRAs and the approaches taken to ask questions and collect information. This review demonstrated an almost exclusive use of text in question stems and answer options. The use of scripting to surface relevant information or to skip or re-word irrelevant questioning, based upon previous answers, was used sparingly and by a minority (less than 40%) of the HRAs that we reviewed. In general it was concluded that the commonly available HRAs were text heavy, lacking in graphical components that could aid the users’ understanding, and took a significant amount of time to complete.
Based upon this initial market scan we developed a number of templates that could be populated with specific health-related questions. To aid this process we reviewed some of the most popular, non-health-related websites, including Amazon, eBay and Expedia to observe how they presented information and asked users questions. The templates incorporated placeholders for images as well as interactive response options. Figure 1 gives examples of some of these templates.
Examples of interactive response options utilized in the RBHA
Prior to populating the templates with specific health-related questions they were all tested on a group of six volunteers to ensure usability and ease of understanding. Dummy questions were used while volunteers were video-taped completing them on screen. Software that tracked eye movements as well as mouse movement and clicks (Silverback 2.0, Brighton, UK) was also employed.
The output from this initial user testing resulted in minor changes to the GUI prior to population with the health-specific questions.
Development of the question set
Prior to the development of the question set we reviewed the scientific literature to establish which health-related domains to create questions for. Based on the published evidence for specific health, wellness and lifestyle areas having an impact upon medical and pharmacy claims costs, absenteeism, productivity, workers compensation or disability claims, and also the previous research that one of the authors (PRM) has done in the area, we developed questions for 15 separate domains in the RBHA. 5
Using the templates developed in phase 1 an iterative process of question development and user testing was employed until we were satisfied that each question was easily understandable by users and collected the necessary information required to stratify respondents into risk groups for each of the domains.
User testing and face validity
Once the whole question set had been developed, end-to-end testing was undertaken with a cohort of 1200 employees from a national employer. Particular attention was paid to the length of time it took individuals to complete the RBHA as well as the proportion of the population classified as high-risk in each of the domains. In addition each respondent was asked to provide feedback on ease of use as well as areas that could be improved.
In addition to the end-user testing of the RBHA we also engaged a number of industry experts to provide their opinions and recommendations for improvements. Three individuals, each with at least a decade of experience in research and development of population health management solutions, including the development of HRAs for health plans and commercial health management organizations, were provided with online access to the HRA and asked for their general comments as well as responses to a short structured questionnaire.
This phase of testing resulted in changes to templates, layouts and question wording in order to improve accuracy and timeliness of completion.
Validation of the RBHA
The final phase of the development of the RBHA was validation against relevant ‘gold standard’ measures for the included domains. Although there is no universally recognized ‘gold standard’ HRA there are a number of well validated questionnaires that assess many of the domains included within the RBHA.7,19–23 The purpose of this part of the development process was to ensure that the questions that required non-binary responses correlated well with the chosen gold standard measure of that domain. A number of appropriate gold standard question sets were identified and incorporated into our online environment for the purposes of this part of development process. The gold standard question set was randomly positioned before or after the RBHA so as to minimize any ‘training’ impact one question set may have on the other based upon which was answered first. Table 1 provides further details on the gold standard measures used.
Gold standard questionnaires used in the validation phase of the RBHA development
In addition to correlating responses to domain questions between the RBHA and the gold standard questions sets, those domains that were composed of multiple items were also examined for internal consistency by calculating the Cronbach-α and inter-item correlation values.
Volunteers were recruited by an online survey organization (Market Tools Inc, San Francisco, CA, USA) to complete both the RBHA and the appended gold standard questions in one sitting. Individuals were incentivized to complete the question set by the survey organization with a $25 voucher redeemable at a number of online outlets. Eligibility requirements were: (1) being in full-time employment within the United States of America; (2) being 18 to 65 years of age at the time of survey completion; and (3) having access to the Internet. All respondents were asked to complete the full question set within a three-week period during July 2010. A random subgroup of 100 respondents were asked to complete the RBHA eight weeks after their initial completion date in order to assess test–retest validity of the questionnaire.
The final part of this phase of the investigation validated respondents’ answers to the medical condition checklist with historical medical and pharmacy claims data. Data were merged from a cohort of 27,215 individuals from a variety of different employers who completed the RBHA in the latter half of 2010 and who also had claims data for the period 2009–2010. Using medical and pharmacy claims as the ‘gold standard’ (Symmetry Episode Treatment Groups, Ingenix, USA) we analysed the sensitivity and specificity of the self-reported medical conditions question of the RBHA for correctly identifying common conditions.
All data analysis was carried out using Statistica 9, a statistical software package distributed by Statsoft, Inc (Tulsa, OK, USA; http://www.statsoft.com).
Results
The user-centric development process employed to create the RBHA enabled us to develop a new level of insight into how individuals interact with onscreen HRA questions, as well as how to present questions and answers so that responses are both intuitive and accurate.
Figure 2 shows an example of the GUI iterations that were developed as a result of user feedback and the way individuals responded to the questions in the RBHA. At each step in the development process user feedback allowed us to hone the graphics, wording and layout of the questions so that when presented on screen the respondent quickly understood what was being asked of them, as well as how to answer appropriately.
Four screen shots showing evolution of the physical activity question as a result of iterative user feedback
Face-validity testing with industry experts allowed us to get an unbiased overview of how the RBHA compared to other tools in the market place. All three experts agreed that the RBHA was a significant improvement on existing HRAs as it provided the respondent with a more intuitive and easy to understand interface than currently exists. In addition, all three experts strongly agreed that the included domains and the way the questions within the domains were presented were appropriate and valid.
Table 2 provides further detail on the RBHA domains and the types of question included within each domain.
Details of the 15 domains in the RBHA and an overview of how the responses to the questions within the domains were used to attribute risk status
Six hundred individuals were invited to participate in the external validation phase of the RBHA, of these 464 (77%) completed both question sets within the three-week timeframe. The mean age of respondents was 43.3 years (SD 14.4), 54% were women and 86% classified themselves as Caucasian.
The mean time respondents took to complete the RBHA plus the gold standard question set was 35.8 min (median 32 min). The mean time to complete the RBHA alone was 29.5 min (median 18.2 min).
Table 3 shows how eight of the RBHA domains correlated with their associated gold standard measure. For each domain a Pearson correlation co-efficient (r value) between the responses to the relevant RBHA questions and those of the gold standard was established. In addition, for each gold standard questionnaire a cut-off value commonly used to indicate ‘high-risk’ or ‘at-risk’ status was used to calculate the sensitivity, specificity, and positive and negative predictive values (PPV and NPV) of the RBH domain.
Table showing the Pearson correlation co-efficients (r values) for the relationship between the measured RBHA domain and the associated gold standard question set. For each gold standard measure a commonly used score or response ‘cut-off’ value was used to classify respondents as high-risk. These cut-off values were used to assess the sensitivity, specificity, negative predictive value (NPV) and positive predicative value (PPV) of the RBHA domain question(s) as compared to the gold standard questionnaire
P < 0.001
GHQ-12 = General Health Questionnaire 12; REAP-S = Short Form Rapid Eating and Activity Assessment; PSQI = Pittsburgh Sleep Quality Index; IPAQ = International Physical Activity Questionnaire; SF-36 = Short Form 36 Questionnaire; WHO-HPQ = World Health Organization Health and Work Productivity Questionnaire
As well as looking at the correlation between overall nutritional balance by comparing the computed score from the six nutritionally-focused questions in the RBHA with the REAP-S questionnaire, we also examined the observed correlations between the major constituent food groups. Statistically significant (P <0.001 for all) correlations were shown between the RBHA measures of fat intake, fibre intake, salt intake, and fruit and vegetable consumption with the appropriate question sets within the REAP-S.
Examination of the internal consistency of the six items within the nutrition domain of the RBHA revealed a Cronbach-α value of 0.74 and an inter-item correlation of 0.37.
One hundred randomly selected individuals were invited to retake the RBHA eight weeks after they had originally taken the questionnaire. Eighty-two responded, the average time between original completion and repeat completion was 61.4 days (range 50.3–67.6 days). For all of the domains the Pearson correlation co-efficient was 0.7 or higher (P < 0.001 for all).
The sensitivity and specificity of the RBHA medical checklist items in correctly identifying medical conditions flagged by a medical and pharmacy claims analysis program ranged from 65–75% and 72–94%, respectively.
Discussion
HRAs, and the data they generate, have become an integral component of population health management strategy for many employers, health insurers and healthcare providers. 1 The last decade has seen HRA deployment shift from being predominantly paper-based to almost exclusive delivery via the Internet. With this shift in delivery channel comes a need to ensure the tools that we are using, together with how they are deployed, are valid and optimized in order to provide the best possible foundation for subsequent population health interventions.
The characteristics of the online environment are markedly different from print media, and although a web page can function as a digital representation of a piece of paper there is now the opportunity to use modern browser functionality to enhance user experience and accuracy of data collection. The RBHA represents such a new breed of HRA in that it has been developed specifically for Internet delivery and undergone extensive user testing and validation analyses.
In developing the RBHA we looked outside the traditional health and medical paradigms that we normally operate within and from the outset employed recognized principles of good website development and design that are common in the retail and marketing sectors. Although the process added a significant amount of time to the HRA development it did enable us to create a tool that has user-centricity at its core and that we believe is unique in the market place.
One of the central tenets of the development process for the RBHA was to keep completion time as short as possible. Our own experience of implementing HRAs has shown significant drop-off in completion after 20 minutes of engagement, we therefore worked to making sure completion time did not exceed this. To this end the majority of the domains that make up the RBHA consist of single or double items, with only nutrition having a greater number of questions. This approach kept the necessary completion time to a minimum, but as we have shown, still enabled the collection of reliable health data.
At the outset it should be recognized that an HRA is not a ‘diagnostic’ tool, rather it is an instrument that can assist in categorizing individuals into risk groups based upon their responses, much like triage in the medical world. In essence an HRA is triaging respondents so that those individuals considered to be at increased risk in specific areas can receive further assessment and intervention as necessary.
As well as streamlining the user experience we also wanted to make sure that the data we were collecting were robust and ‘measured what we thought we were measuring’. There are very few formal HRA validation studies published in the scientific literature, which in itself begs the question as to whether some instruments are accurately collecting health risk data. By comparing the main RBHA domains with simultaneously collected responses from gold standard domain specific questionnaires we have demonstrated statistically significant correlations in all areas. This suggests that at a minimum the RBHA domains are tapping into the same constructs as the gold standard question sets. In addition, we chose commonly used cut-off values for each of the gold standard questionnaires to represent the boundary between high-risk status and normality in each of the investigated domains. Clearly using such categorization is not diagnostic, but in the absence of formal physician review of each respondent it gave us an indication of how the RBHA might perform in such situations.
As is the case with many questionnaires, especially those that use a few items in each domain, the sensitivity of the RBHA questions in each domain was modest; however for the main part the specificities and negative predictive values (NPV) were high. Having a high value for specificity and NPV is an important attribute of a screening/triage questionnaire as one can be reasonably confident that if an individual is classified as not being at risk this will indeed turn out to be the case. As an HRA should be used as a triage instrument those individuals who are classified as high risk will inevitably go through further inquiry as to whether they meet risk criteria or not. The only area where the RBHA specificity fell below 80% was in the physical activity domain. This may be due to the fact that the IPAQ categorized more than 60% of the population as not meeting minimum levels of regular activity; a figure that is significantly higher than published epidemiological data for the United States. 24
Similarly, medical and pharmacy claims data are not always accurately coded, however, despite this potential limitation respondents answers’ to the question enquiring about current medical diagnoses showed a high degree of correlation with diagnoses derived from claims.
Not all of the domains in the RBHA were included in the gold standard validation process reported in this paper. There were a number of reasons for this; for some domains (cardiovascular risk, body weight, preventive care) standard algorithms or consensus recommendations were used, and unchanged, to classify risk status. For other areas (smoking, alcohol consumption) respondents were simply asked to quantify their usage status, much as a physician would enquire and record during a medical history. In addition, for some areas we could not find an easy-to-administer questionnaire that covered the area of the domain in question (risk behavior).
The presented research is only the start of ongoing validation work for the RBHA. Further research will look specifically at the ability of the RBHA to prospectively identify individuals who will have high healthcare utilization and high cost interactions with the healthcare system in the short to medium term.
Conclusions
In summary, we present the process by which we have created a new health risk assessment instrument. With the increasing digitization of almost all aspects of our lives, we believe it is imperative that the creation of consumer-focused health-related tools and applications are grounded in consumer usability and user centricity as well as scientific validity.
Footnotes
Acknowledgements
The authors thank Joseph Leutzinger, Thomas Golaszewski and Robin Rager for reviewing the HRA during its development, CJ Segal Isaacson for assistance in interpreting the REAP questionnaire, and Karin Bultman for review and recommendations during manuscript preparation. For a full demonstration of the patent pending RedBrick Health Assessment please contact Wendy Masloski (