Impact Evaluation of VKN–NIMHANS–ECHO Model of Capacity Building for Mental Health and Addiction: Methodology of Two Randomized Controlled Trials

Abstract

Background:

Bridging the alarming treatment gap for mental disorders in India requires a monumental effort from all stakeholders. Harnessing digital technology is one of the potential ways to leapfrog many known barriers for capacity building.

Aim and Context:

The ongoing Virtual Knowledge Network (VKN)–National Institute of Mental Health and Neurosciences (NIMHANS)–Extension of Community Health Outcomes (ECHO) (VKN–NIMHANS–ECHO: hub and spokes model) model for skilled capacity building is a collaborative effort between NIMHANS and the University of New Mexico Health Sciences Centre, USA. This article aims to summarize the methodology of two randomized controlled trials funded by the Indian Council of Medical Research (ICMR) designed to evaluate the effectiveness of the VKN–NIMHANS–ECHO model of training as compared to training as usual (TAU).

Methods:

Both RCTs were conducted in Karnataka, a southern Indian state in which the DMHP operates in all districts. We compared the impact of the following two models of capacity building for the DMHP workforce (a) the VKN–NIMHANS–ECHO model and (b) the traditional method. We use the SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) statement to describe the methods of these two trials.

Trial 1 is to evaluate the “Effectiveness of addition of Virtual-NIMHANS–ECHO tele-mentoring model for skilled capacity building in providing quality care in alcohol use disorders by the existing staff of DMHP districts of Karnataka.” Hub for trial 1 was set up at NIMHANS and the spokes were psychiatrists and other mental health professionals headquartered in the district level office. Trial 2 assesses the implementation and evaluation of the NIMHANS–ECHO blended training program for the DMHP workforce in a rural south-Indian district of Karnataka state. The hub for trial 2 was set up in the district headquarter of Ramanagaram. Hub specialists are DMHP psychiatrists, whereas spokes are the non-doctor workforce (including auxiliary nurse midwives [ANMs] and accredited social health activists [ASHA] workers) medical officers of primary health centers. The location of the HubHub differs in these two studies. Both trials are funded by the ICMR, Government of India

Discussion:

Both these trials, though conceptually similar, have some operational differences which have been highlighted. If demonstrated to be effective, this model of telementoring can be generalized and widely merged into the Indian health care system, thus aiding in reducing the treatment gap for patients unable to access care.

Key Messages:

1. This paper describe the methodology of two randomized controlled trials comparing two different capacity building models for scaling up public mental health services.

2. Trial-1 focuses on substance use disorders while trial-2 focuses on mental disorders in general.

3. Though conceptually similar, both trials are different operationally.

Psychiatric disorders are burdensome, and in India, the treatment gap ranges from 28% to 86%, as per the National Mental Health Survey 2016. ¹ The District Mental Health Program (DMHP), the operational arm of the National Mental Health Program, is the engine for bridging this deficit. ² There are 31 DMHP teams in Karnataka and each team comprises one psychiatrist, one psychiatric social worker, one clinical psychologist, two psychiatric nurses, and a record keeper. An essential objective of DMHP is integrating mental health care into the general healthcare. This is supposed to be achieved through the periodic training (in mental health) of all cadres of health workers (including doctors, nurses, pharmacists, auxiliary nurse midwives [ANMs], and accredited social health activists [ASHAs]) at the taluk (block) hospitals, community health centers (CHCs), and primary health centers (PHCs).

Three decades since its launch, the DMHP program has had limited success in reducing the treatment gap. Reasons for poor results include poor infrastructure, overburdened systems, inappropriate training, inadequate financial support for mental health, inherent difficulties with monitoring and continued hand-holding, and finally low technical inputs and guidance to revive community psychiatry programs at national levels. ³ Further, one of the major reasons for the low translational quotient of classroom training of primary care doctors is the absence of principles of adult learning, which in turn leads to low sustainability, acceptability, and generalizability. ⁴ NIMHANS, Bengaluru, is at the forefront of leveraging digital technology to overcome some of the above-mentioned barriers in capacity building for mental health. Under the umbrella of NIMHANS Digital Academy, many such capacity-building initiatives are underway for the past four years in states including Bihar, Karnataka, and Chhattisgarh. ⁵ One such initiative is the collaborative activity between NIMHANS and the University of New Mexico Health Sciences Center (UNMHSC), titled Project ECHO (Extension of Community Healthcare Outcomes).

Project ECHO: Project ECHO is a novel and innovative model of health care training and mentoring, developed at the UNMHSC that strives to enhance access to specialist care in underserved communities utilizing technology to leverage scarce resources. This low-cost, high-impact intervention is accomplished by linking multidisciplinary expert teams from an academic center (hub) with primary care clinicians (spokes) through videoconference-based Tele-ECHO clinics (Figure 1). During a tele-ECHO clinic, the experts comanage patient cases with the primary care clinicians and share their expertise via mentoring, guidance, feedback, and didactic education. ⁶ Multiple participants can join simultaneously for live Tele-ECHO clinics via their personal computers or mobile devices.

A typical tele-ECHO session begins with an introduction from all the participants, including those at the hub and the spokes followed by updates or announcements of relevant events. Subsequently, a brief didactic session related to mental health disorders is offered by a hub expert. This is followed by two to three de-identified case presentations by the participants. The clinic facilitator solicits clarifications and invites suggestions from the video participants and then by the team of experts at the hub. Finally, the recommendations made during the session along with additional points are summarized and forwarded to the presenters. ⁷

Figure 1.

Hub and Spokes of the ECHO Model

This model enables primary care clinicians to effectively manage patients with common yet complex diseases in their communities, which reduces travel costs, delays in treatment, and other avoidable complications. It has proved to be beneficial to both the service users and the health professionals.^8-10 For the DMHP, this technology has the potential to leapfrog the barriers in significant mental health care delivery. It can serve as a model for other academic institutions, departments of health, and primary care teams to provide specialized care to underserved populations.

To demonstrate this model’s effectiveness, two teams at NIMHANS have attempted to compare the ECHO tele-mentoring model of capacity building, with conventional model of capacity building for DMHP. The trials seek to examine whether the ECHO model of capacity building (as compared to the traditional methods of training health care professionals) results in (a) significant increase in the identification, screening, and assessment for alcohol use disorders (AUDs) and other mental disorders, (b) enhanced decision-making ability about appropriate referral of cases to higher centers, and (c) significant improvement in patient health outcomes such as symptom reduction, reduced level of disability, improved socio-occupational functioning, and an overall reduction in caregiver burden of patients, and eventually, the reduction in treatment gap. Lastly, we set out to examine the feasibility, sustainability, acceptability of this program among the ground level health care professionals. In the following paragraphs, the methodologies of these two trials are described. Both trials are approved by the NIMHANS Ethics committee and are registered in: www.ctri.in (reference number for registration number of trial 1 is CTRI/2019/12/022517 and the registration number for trial 2 is CTRI/2019/12/022245). The recruitment for both the trials is currently ongoing.

Trial 1: Effectiveness of Addition of Virtual NIMHANS–ECHO Tele-mentoring Model for Skilled Capacity Building in Providing Quality Care in AUDs by Existing Staff of the DMHP Districts of Karnataka

Participants

In Karnataka, DMHP has been implemented in 31 districts, and each DMHP has one qualified psychiatrist, a psychologist, a psychiatric social worker, and two staff nurses. In this study, 28 DMHP districts of Karnataka will be cluster-randomized to two groups, that is, the intervention (DMHP ECHO) group and control (service as usual) group. The DMHP located in BBMP, Bengaluru urban and Bengaluru rural were excluded due to the proximity to the NIMHANS hub, and hence the possibility of bias.

The inclusion criteria for the selection of DMHPs are as follows: (a) Located and administered by Department of Health and Family Welfare, Karnataka, (b) having a minimum of one psychiatrist and one psychologist/psychiatric social worker/psychiatric staff nurse/community nurse (spokes), and (c) having adequate internet bandwidth to support ECHO videoconference sessions. If the internet penetration is poor, then NICNET which is accessible at all district commission offices will be used.

The participants randomized into the intervention (i.e., DMHP ECHO) group will receive training offered by the NIMHANS team of addiction specialists (hub). The control group will continue service as usual, without additional training from NIMHANS. The course has been designed as a 12-month training module with a competency-based accreditation that is approved by the NIMHANS Board of Studies. The prerequisites for course completion and accreditation are two or more case presentations by the participants and maintaining 60% attendance during the live tele-ECHO sessions. They are also required to complete the pre-test and post-test, attain 75% scores in the periodically provided e-assessments, and submit online monthly reports of cases of SUDs seen by them at their respective healthcare practice. The participation in the course is entirely free of cost and the participants will not be paid any additional incentives for their participation.

Tele-ECHO clinics will be conducted weekly with a focus on iterative, case-based learning in which DMHP staff will present challenging clinical cases and receive guidance from the hum experts. In view of local needs, few alterations will be made to the existing model. The specific phases of this project are described below.

Phase 1: Hub Preparation

The intervention module will be developed by recruiting project staff. They will be trained in VKN–NIMHANS–ECHO model in taking the role of facilitator for the weekly Tele-ECHO clinics. The HubHub for this study is located at the already established Virtual Learning Centre, Centre of Addiction Medicine, NIMHANS. The team of experts at the HubHub will develop the curriculum and training material based on best practices, keeping in mind the need of the DMHP centers.

Phase 2: Implementation

After cluster randomization, the participants will be recruited for the study. The team of health professionals from DMHP ECHO group will undergo face-to-face on-site training regarding details of the research study, their role in the study along with brief didactic discussion about SUDs. During this training, they will be educated about getting linked to the NIMHANS–ECHO hub and using the addiction-related assessment tools required for the study. The tasks will be clearly defined for this study which will help in determining the competencies that should be focused on during the training sessions and will account for a systematic and fuss-free flow of work at the DMHP.

The DMHP health professionals will recruit cases by actively screening patients visiting their out-patients department at the district hospital for AUDs using the screening tools listed below. Patients testing positive will be entered into a record and referred to as DMHP for further management. An abbreviated version of clinical pro forma being currently used at NIMHANS will be used for assessment.

Tools:

AUD Identification Test (AUDIT) ¹¹

Patient Health Questionnaire (PHQ SADS)

WHO Quality of Life BREF (WHO QOL BREF) ¹²

Visual analogue scale to measure the status of improvement as per the patient and family members

Periodic semistructured assessments for knowledge, attitude, and motivation of health professionals

The assessments will be done every three months for patients and every six months for health professionals.

Phase 3: NIMHANS-–ECHO Tele-mentoring and Virtual Tele-ECHO Clinics

The online multipoint videoconference-based tele-ECHO clinic will take place at fixed times every week. The overall purpose of these tele-ECHO clinics will be implementing standard care as per SOP, discussing the challenges faced, resolving their doubts which will in turn encourage comanagement, collaborative care, and supervision. The tele-ECHO clinic will also serve as a forum for discussing the various challenges that participants face in implementation at the community level. The hub multidisciplinary team will have an addiction psychiatrist, a general adult psychiatrist, clinical psychologist, psychiatric social worker, and nurse. In instances where inputs from other psychiatric specialties (such as geriatric psychiatry, women’s mental health, forensic psychiatry) are required, the respective specialist will be requested to take part in the tele-ECHO clinic discussion. Apart from the routine case-based management, special attention will be given to counseling specific for AUDs such as the Feedback, Responsibility, Advice, Menu of options, Empathy and Self efficacy (FRAMES) ¹³ model of Brief Intervention, Develop Discrepancy, Avoid Argumentation, Roll with resistance, Express empathy, Support Self Efficacy (DARES) ¹⁴ approach for motivation enhancement, relapse prevention skills, enhancing coping skills, and use of stress reduction techniques.

Evaluation

The assessment for patient-related outcome will be conducted every 3 months and for the health professional outcomes every six months.

Primary Outcome: Patient Related

Evaluation was done in detail from two districts each from DMHP ECHO and DMHP SAU groups:

30% increase in the number of identification of cases of AUD at DMHP ECHO districts compared to that of DMHP SAU by nurse, social worker, and psychologist

Decrease in the AUDIT score of less than 8 and mean daily alcohol consumption in the past 14 days immediately preceding the 3-month outcome assessment in DMHP ECHO districts compared to DMHP SAU districts

Secondary Outcomes

Evaluation for health professionals was done in all districts based on Moore’s evaluation criteria ¹⁵ :

Service related: Increase in access to care, increased medicines dispensed, and improved treatment compliance.

Health professionals: Participation, satisfaction, learning, competence, performance—highlighting increase in their knowledge, attitude, empathy, skills, motivation, adherence to best practice care, professional satisfaction.

Clinical: Changes in disability and dysfunction reported by the patients as recorded by WHO QOL BREF during three-month outcome assessments.

Statistical Analysis and Data Management

There are 28 clusters, that is, DMHP spokes. Assuming an intra-cluster correlation coefficient of 0.04 and assuming loss to follow up of 50% over six months, and 1:1 allocation ratio, a trial size of 200 patients of AUD was established. This will have 85% power to detect the hypothesized effect between the DMHP ECHO and DMHP SAU groups.

Data storage and handling will be done at NIMHANS for tele-ECHO clinics in the NIMHANS server using a method to keep the data safe and encrypted. All the data stored at NIMHANS will be filed in a locked cabinet in a secure office (paper forms) and in encrypted files located on a secure, nonpublic, password protected computer (digital forms) in a locked room. Paper forms will be digitized to facilitate data analysis. Upon completion of the study period, all paper forms will be destroyed. All the nonsurvey data collected at the participating clinical sites will be collected in a manner consistent with clinical care of the patient and will be stored using the clinic’s already existing HIPAA-compliant medical record system (which may be digital or paper-based, depending on the clinic).

Data analysis will be carried out under the supervision of the principal investigator and the biostatistician and both quantitative and qualitative analysis will be conducted for the study.

Trial 2: Implementation and Evaluation of NIMHANS–ECHO Blended Training Program for the DMHP Workforce in a Rural South-Indian District of Karnataka State

Setting

This project hub will be located at Ramanagaram, a district head quarter which is located 50 km (30 miles) away from the NIMHANS, Bengaluru. Additionally, the district has the following peripheral health infrastructure: three taluk hospitals, three CHCs, and 61 PHCs. The PHCs are the first contact for patients in the district. Each PHC has one medical doctor (with basic qualification of MBBS), one pharmacist, a couple of nurses; 15 ANMs, and about 10 ASHAs. Around six PHCs in the district are managed solely by AYUSH (traditional Indian systems of medicine) doctors.

ANMs are known as the village health workers who are the primary contact between the health infrastructure and community. ASHAs are the latest human resource supplement to the public health. Both ANMs as well as ASHAs are well-equipped to satisfy the mental health needs of the communities. ¹⁶

For this project, three PHCs will be chosen (simple random sampling) to be the study group. ASHAs, ANMs, pharmacists, and doctors of these PHCs will be mentored and trained for a period of six months in running the DMHP program (details below). Three more PHCs will be selected as controls where the DMHP would run in the routine manner (i.e., sans the hand-holding program).

Implementation of the NIMHANS–ECHO Blended Training Program

Subjects: The ANMs, ASHAs, pharmacists, and doctors of PHCs will form the subjects for the current study. In total, 30 ASHAs, 12 ANMs, 2 pharmacists, and 3 doctors will be recruited for the study group and an equal number for the control group.

Setting up of the training “hub” at Ramanagaram: The district hospital at Ramanagaram (run by the health department of the Government of Karnataka, India) is the place where a training hub has been established. Physical space has been provided by the hospital.

Elements of the NIMHANS–ECHO-blended training program: The NIMHANS–ECHO training program will contain both online and onsite content.

• The onsite content: This content is a three-day training course for doctors and half-a-day orientation program for the nondoctor DMHP workforce.

• The online content: This will be designed keeping in mind the availability of time for the DMHP workforce. In principle, the content includes a one- or two-hour session once in three weeks. It would consist of an interactive session in which a clinical case conference and a discussion on a topic of clinical interest would be held.

The project involves onsite training of members of the study as well as the control groups. Mentoring and training through online sessions will be limited only to the study group. The mentoring exercise will continue for six months. The control group is expected to undertake the DMHP work as the rest of the state is doing.

Evaluation of the NIMHANS–ECHO-Blended Training Program

1. Short-term outcomes

2. Demonstration of improved knowledge and competence by the spokes—pre- and post-KAP assessments

3. Challenges faced by the spokes—feedback form

4. Challenges faced by the “hub” members in running the program—feedback form

5. Challenges and opportunities of setting up and running such a training module (to study the feasibility of running such program)—feedback form

Medium-term outcomes

Performance of spoke members in identification, pharmacotherapy, counseling, referral to higher centers—monthly monitoring reports

Sustainability of the program

Long-term outcomes

Patient health outcomes

Reduction in symptoms: Clinical Global Impression—schizophrenia (CGI-S), bipolar version, brief addiction rating scale (BARS), timeline follow-back (TLFB) method

Changes in the levels of disability: Indian Disability Evaluation and Assessment Scale (IDEAS), World Health Organization Disability Assessment Schedule 2.0 (WHODAS-2.0)

Improvement in work functioning: IDEAS, WHODAS-2.0, EuroQol

Improvement in social functioning: IDEAS, WHODAS-2.0, EuroQol

Reduction in family burden: Burden assessment schedule

Population health

Reduction in treatment gap: Treatment gap will be calculated by the formula given below. ¹⁷ The authors give the following definition:

The absolute difference between the true prevalence of a disorder and the treated proportion of individuals affected by the disorder. Alternatively, the treatment gap may be expressed as the percentage of individuals who require care but do not receive treatment. Estimating the treatment gap in a population depends on the prevalence period of the disorder, the time frame of the examination of service utilization, and the demographic representativeness of the study sample regarding the target population.

G = (S [1 – Sc] RcPc) / S (RcPc)

G = treatment gap, Sc = service utilization rate*, Rc = prevalence rate, and Pc = population

*Service utilization rate will be calculated by the number of patients with psychiatric disorders who are identified/treated by the ANMs/ASHAs and doctors after the project begins.

Comparison of the “study” group and the “control” group in terms of

Number of patients identified and treated

Reach in terms of delivering the social welfare benefits for persons with psychiatric disabilities

Cost of the entire program: The overall expenditure incurred during the implementation of the NIMHANS–ECHO-blended training program which includes the costs incurred for establishing the HubHub, travel expenditure, online mentoring for the study group, and onsite training for both the study and control group will be calculated.

Subsidiary outcomes

Onsite curriculum for nonspecialist DMHP workforce

Online curriculum for the nonspecialist DMHP workforce

Assessment Tools

1. Feedback form: The feedback form was developed on the basis of our previous experience in evaluating the effectiveness of the ECHO-blended training program in other states such as Chhattisgarh and Bihar.^18,19

2. Symptoms in others ²⁰ : “Symptoms in others” is a simple tool for screening psychiatric disorders. This can be applied by ground-level health care workers and has been shown to be successful in identifying nearly all psychiatric disorders in the communities. ²⁰ We have modified the tool by adding a couple of questions to identify substance use disorders and mental retardation. Totally, the modified tool contains 15 questions and it does not take more than five minutes to apply this instrument.

3. IDEAS ²¹ : The IDEAS had been originally developed for measuring and certifying disability for psychiatric patients in India. The IDEAS has good face validity, criterion validity, and internal consistency. Although originally meant for certifying disability of patients with psychiatric illnesses, it has been used for research purposes, and has been found sensitive in the identification of milder levels of disability as well.

4. EuroQol (EQ-5D) ²² : This is a standardized, nondisease-specific instrument for describing quality of life in health-related states. This is easy to administer and has been increasingly used to measure quality of life in psychiatric disorders especially in schizophrenia and related disorders.

5. Burden Assessment Schedule (BAS) ²³ : This instrument rates burden faced by the caregiver across 40 items. Each item is scored as 1 (no burden at all), 2 (burdensome to some extent), or 3 (very much burdensome). The scale assesses both objective and subjective burden experienced by the primary care givers of chronic mentally ill patients. Its criterion validity has been established and it has good inter-rater reliability (Kappa = 0.80).

6. World Health Organization Disability Assessment Schedule 2.0 (WHODAS 2.0) ²⁴ : WHODAS 2.0, by the WHO, supersedes the earlier version of the disability assessment schedule. It can be used to measure disability across all diseases including mental and substance use disorders. It covers six domains including cognition—understanding and communicating; mobility—moving and getting around; self-care—hygiene, dressing, eating and staying alone; getting along-interacting with other people; life activities, domestic responsibilities, leisure, work and school; participation—joining in community activities. We will translate the instrument into the local vernacular, Kannada. We will use the 12-item questionnaire for this study.

7. The CGI-S scale ²⁵ : This is a simple instrument to measure the diverse symptom dimensions of schizophrenia including positive, negative, and cognitive symptoms. Its psychometric properties have been well-established.

8. The Clinical Global Impressions Scale—Bipolar Version (CGI-BP) ²⁶ : Just as mentioned above for schizophrenia, the old CGI scale was modified for the purposes of tracking outcomes in bipolar affective disorders. CGI-BP gives a set of instructions to facilitate the tracking in terms of acute episodes and longer-term illness prophylaxis. The psychometric properties of this scale are good.

9. The BARS ²⁷ : The scale assesses problems due to addiction in different spheres. It has been used in the Indian setting.

10. The TLFB method ²⁸ : This technique measures and tracks outcomes of AUDs. The pen–paper version can be freely used. Patients are asked to provide an estimate of the amount of alcohol consumed on each drinking occasion during a specified time period. The TLFB has been used extensively in the research literature and has been found to have good psychometric properties.

It may be noted that these scales would be administered in a selected (random-cluster sampling: 1–2 villages by random selection) subgroup of patients only. Patients who are easily accessible (who regularly come to follow-up and those who can be easily reached by means of home visits) will be assessed using the mentioned scales. While it is difficult to commit as to how many patients (in each disorder category) would be assessed, reasonable efforts will be made to include at least 30 patients (for each disorder including schizophrenia, bipolar affective disorder, depressive disorders, and AUDs) in each group (“study” group as well as the “control” group). This is because covering all identified patients and bringing all of them under the ambit of these questionnaires requires huge logistic and financial support.

Statistical Analysis:

Descriptive statistics will be used to explain prevalence of the disorders, treatment gap, and level of symptoms, disability, burden, and outcome. In the two groups, outcomes would be compared using the chi-square test for categorical variables and t-test (Mann–Whitney test for non-normally distributed variables) for continuous variables. Alfa would be set at P < 0.05 with correction for multiple comparisons.

Discussion

Through the two randomized controlled trials described earlier, we intend to evaluate the impact of the VKN–NIMHANS–ECHO model of capacity building. One study focuses on substance use disorders, while the other focuses on mental illnesses in general. Both studies work with the common objectives of democratizing specialist knowledge and promoting multidirectional case-based learning, incorporating adult learning principles in the training of health professionals. The impact expected from this training model is not only the significantly improved patient outcome in these communities but also a heightened level of knowledge, skills, and confidence of the health professionals.

The operational differences between the two studies lie in the fact that trial 1 utilizes a tertiary care neuropsychiatric center (NIMHANS, Bengaluru) as the “hub” and the psychiatrists and other health professionals at the DMHP as the “spokes,” whereas trial 2 will have the “hub” situated in a district hospital (Ramanagaram, Karnataka) with the general health care workers (ANMs and ASHA workers) and PHC medical officers forming the “spokes.”

This model of tele-mentoring, if demonstrated to be effective, can be generalized and widely merged into the Indian health care system for providing an opportunity for continued sharing of knowledge and best practice guidelines amongst health professionals as well as capacity building in resource-scarce areas. Further with this model there tends to be least disruption in the routine work of the “spokes” which can further contribute to the economy in terms of cost-effectiveness and money spent along with improving the overall participation.