Abstract
Health care providers (HCPs) serving HIV-infected patients, especially in rural and underserved areas, have limited access to continuing medical education.
Objective:
To identify subgroups of HCPs who might benefit from a telehealth training program focusing on HIV/AIDS care based on HCPs’ objective for attending the training.
Methods:
Hierarchical cluster analysis combined with thematic analysis identified the subgroups.
Results:
A total of 56 HCPs attended between 1 and 9 of the 12 sessions conducted (mean ≈ 2, standard deviation = 1.6). Subgroups identified included knowledge (HCPs interested in gaining, increasing, or updating their knowledge in HIV care), learn–expert (HCPs interested in learning, obtaining, or providing expert opinion), and observe–apply–network (HCPs interested in observing the training, applying knowledge gained to practice, and networking). No group differences were found in the participants’ reaction to the session, change in knowledge following the training, and other important characteristics.
Conclusion:
Methodological contributions of the study are discussed.
Introduction
Health care providers (HCPs) serving persons at risk of and infected with HIV, especially in rural and underserved areas, face numerous challenges in providing care.1,2 They have limited access to continuing medical education and encounter professional or geographic isolation from tertiary care or teaching hospitals, where much of the information is likely to be obtained.3,4 A needs assessment survey conducted by the Florida/Caribbean AIDS Education Training Center (F/C AETC) with clinicians in Florida (N = 252), Puerto Rico (N = 122), and the US Virgin Islands (USVI; N = 140) revealed that these HCPs need training in antiretroviral (ARV) treatment, ARV resistance, treatment adherence, perinatal transmission, substance abuse, mental health, hepatitis B and C, and management of adolescent population. In Florida, live HIV-related training opportunities currently include HIV education at regional conferences, education provided by pharmaceutical companies, and hands-on training via the F/C AETC preceptorship program. However, most HCPs in rural communities seldom attend off-site trainings due to restrictions on travel, inadequate funding, and inability to be away from clinical practice due to lack of coverage. 3 Thus, most HCPs opt to obtain continuing education via online programs. However, online programs offer limited interactions with the trainer, yet trainees often need clarification of the information presented.
The F/C AETC Project Extension for Community Healthcare Outcomes Program
To increase the capacity of HCPs in rural and underserved areas of Florida, Puerto Rico, and the USVI to manage HIV-infected patients, the F/C AETC is modeling the Project Extension for Community Healthcare Outcomes (Project ECHO) program. In ECHO, Adobe Connect Pro, a live audio–video-based Internet-conferencing technology, and a telephone conference line are used to connect HCPs and specialists. This platform provides the opportunity for interaction in that participants can hear and/or see one another and ask or answer questions during the session.
A session begins by a 5-minute introduction of participants by a faculty facilitator followed by a 5-minute overview of important points and a 15-minute brief didactic presentation by a faculty member on a current HIV treatment issue. Next is an interactive discussion of cases addressing questions related to treatment, adherence, social issues, challenges encountered, and other concerns that would benefit from consultation with experts in the field of HIV. This segment that lasts about 1 hour also includes question-and-answer sessions. After each session, the participants complete an online evaluation survey. Continuing education credits are available to interested HCPs.
By attending ECHO sessions, it is envisaged that HCPs acquire knowledge and skills to deliver specialty HIV/AIDS care in areas where such best practice care would be inaccessible; HCPs’ capacity to safely and effectively manage patients with HIV/AIDS within their own clinics increases; fewer patients are referred out for care; most patients are retained in their respective clinics; and feelings of isolation among HCPs are reduced. This model of telehealth care education was first developed in 2003 by the University of New Mexico Health Sciences Center and was found to be successful in the management of hepatitis C virus infection.5,6
The impact of a telehealth educational intervention such as ECHO partly depends on its participation rate. Research shows that continuing medical education intervention including multiple exposures as opposed to single exposures improves practice performance.7,8 Consistent with this line of research, the F/C AETC has conducted several ECHO training sessions for eligible HCPs. However, attaining high participation rates, defined here as attendance of multiple sessions, continues to be a challenge in the ECHO program.
Purpose of the Present Study and Research Questions
In this study, the primary purpose was to determine whether clusters (subgroups) of HCPs exist that might benefit from attending ECHO training. If clusters existed, we determined whether they can predict attendance of ECHO trainings. The immediate task was thus to determine clusters of HCPs whose members demonstrate similar response patterns on evaluation surveys. Efforts were made to ensure clusters generated are reasonable in number (neither too numerous so it consists of few HCPs each, nor so few as to be meaningless) and medically meaningful (a typical HCP should be able to relate to the cluster’s description).
In this study, cluster analysis, a technique that groups entities (individuals or variables) based on similarities in measurements, was used. Briefly, cluster analysis is a mathematical method that creates distinct clusters of entities based on similarities among measured features. 9 It allows us to address 2 research questions (1) What types of domain-specific profiles exist among HCPs attending ECHO trainings? (2) To what extent do these profiles differ with HCPs’ reaction to the training, change in knowledge due to training, frequency of attending ECHO trainings, and other important characteristics? We hypothesized that HCPs with different participation rates will differ on how they react to the session, how their knowledge changes following the training, and on other characteristics.
Methods
Data Source
The first author is engaged in evaluation of all F/C AETC programs. Thus, data for this study were collected as part of a larger evaluation project. Data were based on HCPs who attended at least 1 ECHO training between November 2011 and May 2012. Target audience included HCPs from rural and underserved areas of Florida, Caribbean, and USVI. Participants were required to be licensed practitioners, have access to a phone and/or a computer with Internet connection preferably with a webcam, and be willing to attend the session at a specific day and time. One of the items in the survey required HCPs to state the objective for attending the training. Other items were based on the first 2 of Kirkpatrick’s 4 levels of program evaluation, namely, reaction (how participants react to the session) and learning (extent to which HCPs knowledge and skills changed as a result of the training).10,11 A panel of experts including medical doctors, a pharmacist, a measurement and evaluation specialist, and a statistical data analyst reviewed survey items for appropriateness and relevance to the target audience.
Data Analysis Process
Following data collection, the determination of clusters involved making critical decisions at 5 stages of the analysis process. These are discussed next.
Choice of domain
Given our interest in participation, we needed to choose a domain constituting the basis of classification that was associated with attending ECHO training. We focused on participants’ self-stated objective for attending ECHO training, postulating that, controlling other factors, an HCP is more likely to attend multiple sessions if the training is consistent with the HCP’s objectives. Participant’s significant statements of objectives were coded “in vivo” to obtain different objectives types for attending ECHO training. A significant statement contains words or a phrase construed to have meaningful response to the question asked. In vivo coding refers to the assignment of a label to a section of data (statement) using a word or short phrase taken from that section. It ensures that concepts stay as close as possible to participants’ own words or use their own terms to capture key elements of the construct being described. To avoid redundancy, each code was constantly compared with the preceding ones. For each HCP, the presence or absence of an objective type was represented with a 1 or 0, respectively. Thus, a significant statement was scored 1 if it belonged to an objective type, otherwise it was scored 0. The emergent objective types became the variables for cluster analysis.
Choice of similarity/proximity measure
Distance measures in which larger values indicate greater dissimilarity between pairs of HCPs were used to quantify degree of similarity or closeness between HCPs. Specifically, the Euclidean distance (d), defined as the square root of the squared discrepancies between 2 HCPs summed over objective types, was used to compare HCPs.
Choice of clustering method
Hierarchical agglomerative clustering method was used to group HCPs such that HCPs who are most similar or who are the least far apart were grouped together to form a cluster. 9 Using this method, each HCP starts as its own cluster. Next, the 2 most similar clusters are merged, and the new cluster replaces the 2 former clusters. The process continues until there is only 1 cluster containing all HCPs.
Choice of clustering algorithm
We used Ward’s sum of squares method 12 in which every possible cluster combination is considered at each step of agglomerative hierarchical clustering, and the combination resulting in the smallest increase in the overall sum of squares within-cluster distance is selected. This method minimizes the within-cluster variance and tends to produce homogenous clusters of about similar sizes and maximal distances among clusters. 9 Because results of cluster analysis differ by method, 13 Ward’s results were compared with results based on average linkagemethod in which similarity (distance) between any 2 clusters is computed as the average distance between pairs of HCPs, 1 in each cluster. It begins by identifying 2 closest HCPs and merging these into a cluster. Next, 2 closest points (HCP or a cluster of HCPs) are merged, and the averaging process continues over all pairs of HCPs until all clusters are fused together. Conceptually, both methods build an inverted tree diagram (dendrogram) structure from branches to roots and are useful when the optimal number of clusters is not known a priori, 14 which is the case in the present investigation.
Cluster interpretation and validation
Determination of possible clustering solutions was achieved using 2 methods. First, we examined pseudo F, pseudo t, and R2 statistics for different numbers of clusters formed using Ward’s method. We desired to obtain the number of clusters in which the proportion of variance explained was at least 50%. Next, by comparing cluster membership between Ward’s and the average linkage method, similarity of cluster membership in the 2 methods reinforced stability of the clustering process. Once the most parsimonious solution providing clinically relevant clusters was determined, we labeled the clusters based on objective type with the highest proportion in each cluster. Clusters were compared based on HCPs’ reaction to training, amount of learning, HCP characteristics, and frequency of attending training. For all analyses, SAS version 9.2 (SAS Institute, Cary, North Carolina) was used. The PROC CLUSTER procedure that displays a history of the clustering process and statistics useful for estimating the number of clusters in the population from which the data were sampled was used. The procedure also creates an output data set that was used by the TREE procedure to draw dendrogram of cluster membership at each level.
Results
Between November 3, 2011 and May 22, 2012, a total of 12 ECHO trainings were conducted in which 56 HCPs attended. These HCPs shared important characteristics including being predominantly white female nurses or physicians working in an academic health center or state/local health department, with the primary functional role of a care provider or clinician (Table 1). The number of sessions attended ranged from 1 to 9 (mean = 1.9, standard deviation =1.6; median = mode = 1).
Characteristics of Care Providers Who Attended ECHO Trainings (N = 56).
Abbreviations: ECHO, Extension for Community Healthcare Outcomes; ER, emergency room.
Objective Types
Each HCP stated at least 1 objective for attending ECHO training, and 98% of them responded that they achieved their stated objective. A total of 121 significant statements were made, an average of 2.2 statements per HCP. The following 7 objective types emerged from in vivo coding: (1) knowledge: statements coded under this objective type contained the phrase, “To increase knowledge,” “To gain knowledge," or “To update knowledge”; (2) expert, which included statements with the phrase, “To obtain expert opinion” or “To provide expert opinion”; (3) learn, with statements containing the word “learn” or “education”; (4) discuss, with statements containing the word “discuss”; (5) network, with statements containing the word “network”; (6) observe, with statements containing the word “observe”; and (7) application, with statements suggesting application of knowledge to improve patient care (Table 2). For 19 (34%) HCPs who attended multiple sessions, their stated objectives stayed the same 70% of the time.
Objective Types and Corresponding Sample Significant Statements (Verbatim).
Abbreviations: ECHO, Extension for Community Healthcare Outcomes; F/C AETC, the Florida/Caribbean AIDS Education Training Center.
Domain-Specific Cluster Profiles of HCP Attending ECHO Training
Using Ward’s method as the primary clustering algorithm, the 7 objective types formed 3 clusters consisting of 21 (38%), 18 (32%), and 17 (30%) HCPs, respectively. Fifty percent of the variance was accounted for by the 3 clusters (R2), cubic clustering criterion (CCC) was 4.63, pseudo t2 = 44.5 was largest at 3 clusters, and there were no ties for these 3 clusters. These statistics indicate that the data can be clustered into 3 clusters (Figure 1).
Horizontal dendrogram displaying 3 distinct clusters generated using Ward’s method applied to squared Euclidean distances (clusters are based on similarities on objective types).
Using average linkage (dendrogram not presented to preserve space), we determined that a 4-cluster solution, with sample sizes 23, 21, 10, and 2 HCPs, described the data. For the 4 clusters, R2 was 59%, CCC was 5.9, pseudo t2 = 53.7 was largest at 3 clusters, and there were no ties. Clusters obtained from Ward’s method were comparable with those of average linkage (Table 3). Based on this comparison, Ward’s 3-cluster solution appeared to be the most parsimonious and provided the most clinically meaningful description of the data. Next, we describe the 3 clusters.
Comparison of Cluster Solutions and Means on the 7 Objective Types.
Abbreviations: SD, standard deviation; Sig, statistically significant mean difference; NS, not statistically significant mean difference.
a ☑ denotes objective type belongs to similar cluster across the two methods; × denotes objective type belongs to different cluster across the 2 methods; number following symbols denotes the cluster number.
The first cluster, labeled knowledge, included only 1 of the 7 objective types, “knowledge” across the 2 analytic methods. The HCPs in this cluster are interested in gaining, increasing, or updating their knowledge. The second cluster, learn–expert, comprised 2 objective types, “learn” and “expert” across the 2 methods. The HCPs in this cluster are interested in “learning,” obtaining “expert” opinion, or providing expert opinion. The third cluster, observe–apply–network, consisted of objective types “observe” and “apply,” which were similarly clustered across the 2 analytic methods. In addition, it included “network” (belonging to cluster 2 in average linkage) and “discuss” (belonging to cluster 4 in average linkage). The HCPs in this cluster are interested in observing the training, applying knowledge gained to practice, networking with others, or discussing HIV/AIDS management issues. In terms of the proportion of the variance explained (R2) by each objective type, knowledge (86%) had the highest value followed by learn (51%), with discuss (8%) having the least value.
Cluster Differences in Reaction to ECHO Training
Based on the 11 items assessing HCPs’ reactions to ECHO training, there were no statistically significant cluster differences in the response to questions regarding reaction to ECHO training (Table 4). The majority of the participants either agreed or strongly agreed with each of the statements, mean score of 4.4, 4.5, and 4.4, respectively, for clusters 1, 2, and 3.
Reaction to ECHO Training.
Abbreviation: ECHO, Extension for Community Healthcare Outcomes.
Cluster Differences in Change in Knowledge due to ECHO Training
Participants were asked to rate their level of knowledge of the content covered in ECHO training using a 5-point scale (1 = Novice, 2 = Beginner, 3 = Competent, 4 = Proficient, 5 = Expert) before and after the session (Table 5). Clusters 2 and 3 had larger drops in the proportion of “novices” (18% and 17%) compared with cluster 1 (10%). The proportion of “beginners” did not change in cluster 1 (0%), decreased by 6% in cluster 2, but increased by 5% in cluster 3. The proportion of “competent” and “proficient” increased in all clusters except in cluster 3, which had no change in proportion of “proficient.”
Rating of Change in Knowledge due to ECHO Training.a
Abbreviation: ECHO, Extension for Community Healthcare Outcomes.
a Δ indicates (after–before), the change in knowledge level after training; ↓, a drop; ↑,a rise; !, unexpected result
Cluster Differences in Participation and Other HCP Characteristics
Participation
Clusters 1 and 3 had low frequency of attendance, an average of less than 2 sessions attended, whereas cluster 2 had high-frequency attendance, an average of at least 2 sessions attended (Table 6). Differences in mean number of sessions attended, however, were not statistically significant among clusters (F 2,53 = 1.60, P= .21).
Comparison of Clusters by Participation and Other Characteristics.
Abbreviations: ANP, advanced nurse practitioners; SD, standard deviation.
Health care practitioner characteristics
Clusters 1 and 2 had higher proportion of HCPs providing services directly to clients/patients compared with cluster 3. Cluster 1 had an equal mix of nurses, physicians, and pharmacists; cluster 2 was predominantly nurses and physicians; and cluster 3 was predominantly nurses/advanced nurse practitioners (ANPs). About half of HCPs in each cluster had a care provider or clinician role. The primary employment setting in clusters 1 and 3 was state health department or academic health center; whereas in cluster 2, it was community health centers. Clusters 1 and 2 had a lower white to black ratio (1.4:1 and 1.6:1, respectively) than cluster 3 (7:1). Cluster 1 had a lower female to male ratio (including transgender; 1:2) than clusters 2 and 3 (1:7, each; Table 6).
Discussion
The purpose of this study was to determine whether subgroups of HCPs might benefit from attending ECHO training and, if so, to determine whether group membership predicts training attendance. Using in vivo coding of self-stated objectives for attending a telehealth educational training program by HCPs, 7 objective types emerged. Hierarchical cluster analysis yielded 3 distinct clusters of HCPs with similar patterns of objectives for attending ECHO trainings. Cluster 1 (knowledge), 38% of the sample, consists of HCPs interested in gaining, increasing, or updating their knowledge in HIV care. They include pharmacists, nurses, physicians, and administrators working in state health department or academic health centers. Cluster 2 (learn–expert), 32% of the sample, comprises HCPs interested in learning, obtaining, or providing expert opinion. They are predominantly nurses and physicians working primarily in community health centers. Cluster 3 (observe–apply–network), 30% of the sample, includes HCPs interested in observing the training, applying knowledge gained to practice, and networking. They are predominantly nurses/ANPs working in state health department or academic health centers. To our knowledge, this is the first study to determine whether definable subgroups of care providers exist based on the participants’ self-stated objectives for attending a telehealth educational training program.
Contrary to our expectation, there are no statistically significant differences in HCPs’ reaction to ECHO training, change in knowledge following the training, and on other HCP characteristics. Clusters 1 and 3 have lower participation rates than cluster 2, but the differences are not significant in implying that group membership is not predictive of training attendance. Although exploratory in nature, the clusters obtained may be useful in recruitment of future ECHO participants. For example, potential participants in ECHO training may be required to fill out a needs assessment form asking for a self-stated objective for attending the training. Based on the responses, a determination can be made regarding which cluster a particular HCP fits best, a proxy for the likelihood of attending ECHO training.
The 5% increase in the proportion of cluster 3 HCPs perceiving their knowledge level as beginner after the training is surprising. What these data suggest is not clear. It may be that for this cluster in which the interest is application of knowledge of HIV care, ECHO training is less beneficial to beginners but more beneficial to those at a competent level as shown by a comparatively higher increase. Compared to the other 2 clusters, the low frequency of attendance and the low percentage of HCPs agreeing or strongly agreeing that their interest was maintained throughout the session lend some support to the increase in proportion of HCPs perceiving their knowledge level as beginners after ECHO training.
Our study is not without limitations. Cognizant that different clustering algorithms may produce different results, we compared 2 clustering algorithms to ensure stability of the solution. Agglomerative hierarchical clustering forces each HCP into a cluster, but subgroups may exist within some clusters with unique combinations of objective types that differ from the majority of HCPs in that cluster. Due to small sample sizes, however, we were unable to uncover the existence of such outlying subgroups. Replicating these analyses in other settings and HCP populations is encouraged. Different or similar clusters may be obtained, which would and should inform recruitment strategies specific to those HCP populations.
Despite these limitations, our study has some methodological contributions. It demonstrates the technique of quantifying qualitative data (statement of objectives) to determine the existence of HCP subgroups targeted for participation in a telehealth training. Also, it demonstrates the utility of a cluster analysis technique to identify potential participants in telehealth training. We intend to reanalyze data after conducting more ECHO sessions. With increased sample size, the accuracy of our result will be enhanced.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the US Public Health Service's Health Resources Service Administration (HRSA) (DHHS-HAB Grant No. H4AHA00049).