Translational Quotient as an Innovative Outcome Parameter to Assess the Effectiveness of Training Methods: A Feasibility Study in Assessing the Baseline Psychiatric Skills of Primary Care Doctors in their Real-world Live Outpatient Clinic

Abstract

Background:

The effectiveness of training methods in medical education is critical, particularly for primary care physicians (PCPs) who frequently encounter psychiatric issues in their practice. Traditional assessment methods often fail to evaluate skill acquisition in real-world clinical practice. The Translational Quotient (TQ) is proposed as an innovative outcome measure to assess PCPs’ ability to apply psychiatric skills in their live outpatient consultation among their general patients. This study aims to evaluate the feasibility of using the TQ in real-world outpatient clinics and to understand the baseline psychiatric skills among PCPs using TQ.

Methods:

Actively practicing MBBS-qualified PCPs enrolled in the Diploma in Primary Care Psychiatry (DPCP) program across various districts were invited to participate. Baseline TQ assessments were conducted in the PCPs’ live clinics, with psychiatrists observing live consultations either remotely or in-person for the first five consecutive general consultations. Each consultation was scored using a standardized TQ proforma, which evaluated six criteria: elicitation of psychiatric symptoms, clinical reasoning, medication choices, counseling, time management, and overall clinical skills.

Results:

A total of 25 PCPs participated, with an average baseline TQ score of 15.7% (4.72 out of 30). Most participants (72%) scored 5 or less, highlighting significant gaps in psychiatric skills. Analysis revealed no significant associations between TQ scores and PCP demographics such as age, gender, and prior psychiatric training, but regional differences were noted, with lower scores in Karnataka compared to Bihar.

Conclusions:

The TQ is a feasible and practical tool for assessing real-world psychiatric skills among PCPs. The baseline scores indicate a pressing need for targeted psychiatric training to bridge the treatment gap in primary care. Future training programs should focus on enhancing diagnostic accuracy, treatment planning, and patient communication to improve mental healthcare outcomes in primary care settings.

Keywords

Competency-based medical education training method effectiveness translational quotient psychiatric skills primary care

Key Messages:

This study introduces the Translational Quotient (TQ) as an innovative measure to assess the psychiatric skills of primary care physicians (PCPs).

The TQ proved feasible and practical, offering valuable insights for enhancing psychiatric training and improving care delivery in primary care settings.

Baseline TQ assessments revealed significant deficiencies, highlighting the need for targeted psychiatric training programs.

In India, primary care physicians (PCPs) are often the first point of contact for individuals experiencing psychiatric disorders.¹ However, inadequate psychiatric education in medical schools and limited access to psychiatric care contribute to a community treatment gap (individuals never receiving any form of psychiatric care who may not reach hospitals to receive treatment, have a psychiatric illness, do not reach hospitals, do not receive treatment) and a functional treatment gap (sufferers often reach primary health center with non-specific symptoms, receive symptomatic treatment rather that specific psychiatric treatment: have an illness, reach hospitals, do not receive psychiatric treatment).² A recent survey³ by the World Psychiatric Association (WPA) and the World Organization of Family Doctors (WONCA) further revealed a perceived gap in competencies in primary care psychiatry among PCPs. A primary reason for the functional treatment gap in primary care is the lack of psychiatric skills needed to diagnose and provide treatment among PCPs.

For many decades, in-service PCPs across India have been trained in psychiatry as part of DMHP in a classroom training format. However, the effectiveness of this training needs to be assessed systematically, especially in terms of the translation of skills in live consultations of primary health care (PHC) itself. However, indirect methods are used to assess the acquisition of knowledge, such as case vignettes, multiple choice questions (MCQs), pre-and post-knowledge, and Attitude and practice (KAP) questionnaires, which do not give an idea of the skills acquired for their live consultations. Measuring the effectiveness of training methods in medical education is of paramount importance. While various outcome measures have been utilized in existing medical literature, such as pre-and post-training knowledge assessments, objective structured clinical examinations (OSCEs), patient satisfaction surveys, and standardized patient (SP) simulations, these often evaluate the knowledge and fail to assess skill acquisition in the context of live consultations.^4,5,6 The ability to convert knowledge into clinical skills in real-world clinical practice is crucial, particularly for PCPs who frequently encounter psychiatric disorders. There needs to be studies that assess the skill acquisition required for the practice of doctors in their live outpatient consultations.

Manjunatha et al.^1,2 used the Translational Quotient (TQ) as an innovative outcome measure to evaluate the psychiatric skills present among PCPs even in live real-time clinical consultations with their patients (in-situ means own PHC itself). The TQ, as an innovative outcome measure, assesses a clinician’s ability to translate knowledge and clinical guidelines into real-world patient care, evaluating the psychiatric skills present among PCPs even in live, real-time clinical consultations with their patients. The TQ also evaluates their proficiency in applying evidence-based practices, adapting them to individual needs, and effectively communicating recommendations. The TQ’s core concept is to bridge the gap between knowledge and practice, ensuring that knowledge translates into clinical decisions for improved outcomes. However, the baseline TQ has yet to be assessed prior to the beginning of the training program.

To address this gap, the authors previously developed and implemented the TQ, an innovative outcome parameter designed to assess the ability of PCPs to translate knowledge into clinical skills effectively within their real-world in-situ clinical practice.¹ The TQ can help assess baseline clinical skills and monitor the progress of skill retention following specific training methods. The burden of common mental disorders like anxiety and depression on primary care systems underscores the need for PCPs to be proficient in managing these conditions, making the TQ, with its emphasis on real-world application, a potentially valuable tool for identifying areas where PCPs may need additional training or support.²

Baseline TQ assessments provide valuable insights for planning training programs, especially personalized training, to address specific areas of need. The TQ assessment in the digitally driven, real-world scenario course of the Diploma in Primary Care Psychiatry (DPCP) program offers a powerful measure for enhancing psychiatric competencies among PCPs.⁷

This study aims to assess the technical and operational feasibility of evaluating the TQ, an innovative outcome parameter for training programs, in real-world, live primary care clinical settings across India and to understand the baseline psychiatric skills of PCPs by examining their baseline TQ scores within their live outpatient clinic interactions.

Materials and Methods

Actively practicing MBBS-qualified PCPs enrolled in the DPCP program across two states (one each from the northern and southern states) of India were invited to participate in this study. The study was conducted in the PCPs’ own live general outpatient clinics to assess their baseline psychiatric skills in a real-world context among their consecutive general patients.

Feasibility Study of Baseline TQ Assessment

We understand that “feasibility study” is broadly used to include any preliminary research that helps prepare for full-scale intervention studies. Feasibility refers to the practicality of completing research within time, resource, and methodological constraints.

This study assessed the feasibility of TQ assessment during live outpatient consultations at primary healthcare centers (PHCs). The intervention involves innovative training programs likely not previously implemented in live PHCs. To evaluate the effectiveness of these interventions, we needed an outcome parameter to assess skills demonstrated during consultations. We developed the TQ concept to determine if these assessments could be done during busy consultations with minimal disruption to regular workflows. A key challenge was whether PCPs would allow psychiatrists to evaluate their skills before training, either through video streaming or in-person, in front of patients. This challenge typically decreases once the training begins and rapport is established.

Therefore, we explored both the technical and operational feasibility of conducting baseline TQ assessments. Technical feasibility examines if the project can be executed using live video streaming via PCPs’ smartphones or through the in-person presence of psychiatrists with PCPs. Operational feasibility looks at whether psychiatrists can perform assessments during the busy outpatient consultations of PCPs at PHCs, considering the limited consultation time available with general practitioners.

TQ Assessment Method

Before commencing the DPCP course, PCPs were invited to undergo a baseline TQ assessment after providing informed consent. Qualified psychiatrists with experience in primary care psychiatry, preferably those who have not trained that particular PCP to minimize the assessor bias, observed the PCPs’ live consultations either remotely (via video-based telemedicine) or in-person. The evaluating psychiatrists were fluent in the languages spoken by the PCPs and patients, with Hindi and Kannada being the languages used during the assessments. As a result, no communication issues were encountered. Each psychiatrist observed the consultations of PCPs in their five consecutive general patients or until at least two cases with psychiatric presentations were encountered. Following each observation session, the psychiatrist scored the PCP’s performance using the TQ assessment proforma (version 2.0).

TQ Assessment Proforma and its Scoring Method

The TQ assessment proforma consists of a six-item Likert scale that encompasses several key components crucial for evaluating clinical psychiatric skills. The elicitation of psychiatric symptoms and signs using the CSP screener* involves the structured and systematic identification of psychiatric symptoms to aid in accurate diagnosis. Clinical reasoning of psychiatric caseness assesses the clinician’s ability to integrate patient information, clinical findings, and psychiatric knowledge to determine if a psychiatric disorder is present. Appropriate decision-making in choosing psychiatric medication evaluates the clinician’s competence in selecting the most suitable psychiatric medications based on the patient’s specific needs, condition, and evidence-based guidelines. The assessment also considers whether the clinician has effectively covered the components of brief counseling for patients, ensuring that patients receive appropriate guidance, support, and information during consultations. Time management for psychiatric evaluation at the primary care level examines how efficiently the clinician conducts psychiatric evaluations, balancing thoroughness with the constraints of a primary care setting. Finally, overall clinical psychiatric skills are assessed, encompassing the clinician’s ability to integrate these various elements into effective, compassionate, and evidence-based patient care. This comprehensive assessment aims to ensure that clinicians possess the necessary skills and knowledge to deliver high-quality psychiatric care in a primary care setting.

The TQ assessment proforma was developed through consensus among two psychiatrists specializing in primary care psychiatry. The proforma consists of six criteria, as mentioned in Table 1. Each item was scored on a scale of 0 (worst) to 5 (best), resulting in a total possible TQ score ranging from 0 to 30.

Table 1.

Item-wise Analysis of Translational Quotient Proforma (N = 25).

No	Item	Highest Possible Score	Secured Minimum Score	Secured Maximum Score	Mean	Standard Deviation
1	Elicitation of psychiatric symptoms and signs using CSP* screener	5	0	2	0.60	0.58
2	Clinical reasoning of psychiatric caseness	5	0	4	0.76	1.09
3	Appropriate decision-making in choosing psychiatric medication	5	0	3	0.80	1.08
4	Did he/she cover the components of brief counseling for patients?	5	0	3	0.76	0.88
5	Time management for psychiatric evaluation at the primary care	5	0	3	1.04	1.06
6	Overall clinical psychiatric skills	5	0	4	0.88	1.24
	Total score	30	0	17	4.72	4.90

*CSP-Clinical Schedules for Primary Care Psychiatry.⁷

Descriptive statistics summarized the TQ scores and PCP demographics. Inferential statistics, including Mann-Whitney U and Kruskal-Wallis tests, were used to examine differences in TQ scores based on PCP characteristics and practice settings.

The study’s Ethical Committee approved it. Patients participating in the live consultations gave consent. The data collection period was between July 2023 and April 2024.

Results

A total of 25 PCPs participated in the study, and their data was analyzed in this study. The mean age was 37.4 years (SD 9.61). The most frequent age group was 30–39 years (44%); 76% were males, and 64% of them were from Karnataka. The majority of the study participants completed their MBBS after 2010 (64%), 60% of them were posted in a rural setting, and 76% of the sample size received no psychiatric training prior to DPCP (Table 2).

Table 2.

Frequency Distribution of the Socio-demographic Variables in the Primary Care Doctors (N = 25).

Variable		Frequency	Percentage (%)
Gender	Female	6	24
Gender	Male	19	76
Age group in years	20–29	5	20
	30–39	11	44
	40–49	6	24
	50–59	2	8
	60–69	1	4
State	Bihar	9	36
State	Karnataka (Belur & BBMP)	16	64
Setting	Urban	10	40
Setting	Rural	15	60
Registration year for MBBS degree	On or before 2000	2	8
	2001–2005	3	12
	2006–2010	4	16
	2011–2015	4	16
	2016–2020	9	36
	after 2020	3	12
Prior psychiatric training after MBBS internship	Yes	6	24
Prior psychiatric training after MBBS internship	No	19	76

All 25 PCPs (none refused) have readily undergone baseline TQ assessment successfully.

This study showed the average total TQ score of the group was 15.7% (4.72 out of 30), and 72% of PCDs (18 out of 25) scored less than equal to 5 out of 30.

The baseline TQ score of each criterion depicted that the maximum score in “Elicitation of Psychiatric Symptoms and Signs using CSP screener” was 2 (mean 0.60), “Clinical Reasoning of Psychiatric caseness” was 4 (mean 0.76), “Appropriate Decision in Choosing Psychiatric Medication” was 3 (mean 0.80), “Did he/she cover components of brief counseling for patients” was 3 (0.76), “Time Management for Psychiatric evaluation at the Primary level” was 3 (mean 1.04) and finally “Overall Clinical Psychiatric Skills” was 3 (mean 0.88). The maximum total score obtained by one of the participants was 17 out of 30 (56.67%) (Table 1).

The majority (84%) of the study participants scored less than 33% of the total score (10 out of 30) (Table 3). It included all six participants who had undergone prior psychiatric training. It was found that the average total score had no association with age, gender, setting, or experience of PCPs after completing an MBBS degree. However, a significant association was found between state and total score (p value .02). The association between evaluators and total score was also found to be significant (p value .006) (Table 4).

Table 3.

Baseline Translational Quotient Scores of all Participants.

Total TQ Score	Frequency	Percentage
0–5	18	72%
6–10	3	12%
11–15	3	12%
>15	1	4%
Total	25	100

Table 4.

Association of Total Translational Quotient Score with Characteristics of Primary Care Physicians and Evaluators.

Variable		Frequency	Median (IQR) of Total TQ score	p Value
Gender	Male	19	2 (1,5)	.13
Gender	Female	6	4.5 (2.75,14)	.13
State	Bihar	9	7 (2.5,13.5)	.02*
State	Karnataka (Belur & BBMP)	16	2 (0.5,4.5)	.02*
Setting	Urban	10	2.5 (2,3.5)	.66
Setting	Rural	15	5 (0,13)	.66
Prior psychiatric training after MBBS degree	Yes	6	3 (2,7)	.39
Prior psychiatric training after MBBS degree	No	19	2.5 (0,6.5)	.39
Age group in years	20–29	5	3 (1,10)	.83
	30–39	11	2 (1,5)
	40–49	6	4.5 (1.5,13.25)
	50–59	2	4.5 (1,– )
	60–69	1	7 (–, 7)
Registration year for MBBS degree	On or before 2000	2	5.5 (4, –)	.75
	2001–2005	3	2 (0,– )
	2006–2010	4	5 (2,5.75)
	2011–2015	4	3.5 (2,11)
	2016–2020	9	3 (0.5,13.5)
	after 2020	3	2 (0,–)
Evaluators	Evaluator 1	10	2.5 (2,3.5)	.006*
	Evaluator 2	4	13.5 (13,16.25)
	Evaluator 3	11	1 (0,5)

*Statistically significant (<.05).

Discussion

This study assessed the feasibility of the TQ as a tool for evaluating real-world psychiatric skills among PCPs in India, focusing on baseline TQ scores before the specialized training. Completing baseline TQ assessment among all 25 PCPs demonstrates its feasibility. This is the first study in which TQ is assessed at baseline among PCPs before any training, especially in live general outpatient primary healthcare clinics. The findings offer key insights for enhancing mental healthcare in primary care settings.

In this study, which included baseline TQ scores among 25 PCPs during the DPCP program held across states over the last decade, the findings indicate that TQ assessment is a practical and viable approach in real-world general outpatient clinical settings. The time required to complete the TQ assessment for each PCP ranged from 30 to 40 minutes, which is around 5 minutes for each patient. As the TQ is a core component and one of the key outcome parameters of the DPCP program, an assessment was carried out for every PCP. These data further demonstrate that TQ assessment is not only feasible but minimally disruptive to PCPs’ workflow and requires few resources in real-world clinical settings. Its seamless integration aligns with the shift toward competency-based medical education, which emphasizes practical, on-the-job training. Unlike traditional assessments, the TQ’s real-world focus more accurately reflects PCPs’ daily clinical skills, making it valuable for ongoing assessment and improvement of psychiatric competencies.⁷ Baseline TQ scores were extremely low, with most participants scoring 5 or less out of 30, that is, 17%. This substantial gap underscores the urgent need for targeted training, focusing on diagnostic accuracy, treatment planning, and patient communication—critical components of effective mental healthcare.⁸ The low scores point to systemic issues in medical education and clinical practice that need to be addressed through comprehensive training programs and designing a psychiatry curriculum for primary care doctors in India. The TQ’s real-world applicability helps identify areas where PCPs need further training and support.^9,10

Variations in TQ scores between regions (notably lower in Karnataka compared to Bihar) in Table 3 may be influenced by evaluator variability, highlighting the need for improved inter-rater reliability. Standardized training for evaluators, regular calibration sessions, and ongoing feedback mechanisms are essential. Future research should explore AI-driven assessment tools to minimize human bias and further standardize evaluations.¹¹

The concept of TQ may be adapted for both undergraduate and post-graduate medical training, providing a standardized, practical assessment of psychiatric competencies. By incorporating the TQ into baseline evaluations and at the end of clinical postings, educators can offer targeted feedback, potentially leading to significant improvements in patient care. This aligns with global recognition of the need for improved mental health competencies in primary care.¹² Authors suggest the generalizability of TQ across medical education by the National Medical Commission of India across specialties.

Study limitations include small sample size and focus on PCPs in a specific training program, potentially limiting generalizability. Voluntary participation may have introduced selection bias. However, the authors have initiated updates to the TQ proforma and efforts to improve inter-rater reliability. Future studies should expand to larger, more diverse samples to further validate the TQ’s effectiveness.

While the TQ may not fully capture the complexity of real-world psychiatric practice, it is a valuable complement to existing training and evaluation methods. To maximize its impact, we recommend standardized training for evaluators, AI-driven assessment tools, region-specific training programs, and advocacy for the TQ’s integration into health policies and curricula.

Conclusions

This study demonstrates the TQ’s feasibility and potential as a tool for assessing real-world psychiatric skills among PCPs in Indian live PHC outpatient clinics. The extremely low baseline TQ scores of all PCPs underscore the need for targeted training to address the identified gaps, particularly in regions with lower scores. By focusing on real-world scenarios and providing ongoing feedback and support, training programs can empower PCPs to effectively identify, diagnose, and manage mental health disorders, ultimately improving the mental health outcomes of individuals seeking care in primary care settings.

Footnotes

Acknowledgements

Evaluators of Translational Quotient, Primary Care Physicians from Bihar, Belur taluk & BMP.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Ethical Approval

Ethical approval for the study was obtained from the Ethical Committee of National Institute of Mental Health & Neurosciences NIMHANS, Bangalore.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Government of Bihar, Ashraya Hastha trust, and HCL Foundation.

Informed Consent

The consent from patients and primary care doctors were taken.

Use of Generative AI

Generative AI has been used for enhancement of language and the authors take full responsibility of the text generated in the manuscript.

References

Pahuja

Kumar

, Manjunatha

, Math

, Thirthalli

. An impact of a digitally driven primary care psychiatry program on the integration of psychiatric care in the general practice of primary care doctors. Indian J Psychiatry 2020; 62(6): 690–696.

Manjunatha

, Kumar

, Math

, . Designing and implementing an innovative digitally driven primary care psychiatry program in India. Indian J Psychiatry, 2018; 60(2): 236–244.

RMK

, Chan

, Herrman

, . What do psychiatrists think about primary mental health competencies among family doctors? A WPA-WONCA global survey. BJPsych Int, 2021; 18(1): 10–12.

Humphrey

, Marcangelo

, Rodriguez

, . Assessing competencies during education in psychiatry. Int Rev Psychiatry, 2013; 25(3): 291–300.

Garg

, Manjunatha

, Kumar

, . Case vignette-based evaluation of psychiatric blended training program of primary care doctors. Indian J Psychiatry, 2019; 61(2): 204–207.

Sriram

, Chandrashekar

, Isaac

, . Training primary care medical officers in mental health care: An evaluation using a multiple-choice questionnaire. Acta Psychiatr Scand, 1990; 81(5): 414–417.

Manjunatha

, Parthasarathy

, Paul

, . Karnataka telemedicine mentoring and monitoring program for complete integration of psychiatry in the general health care. Indian J Psychiatry, 2021; 63(2): 171–174.

Manjunatha

, Suchandra

, Subramaniyam

, . Telemedicine-driven under- and post-graduate medical education for transforming public healthcare delivery of India. J Psychiatry Spectr, 2022; 1(1): 15–20.

Sethi

, Abhilasha

and Moorthy

Psychiatry training in India—A resident’s perspective. J Psychiatry Spectr, 2023; 2: 46–49.

10.

Dinakaran

, Manjunatha

, Kumar

, . The rationale and guiding principles to design a psychiatry curriculum for primary care doctors of India. Natl Med J India, 2022; 35(1):32–37.

11.

Sethi

, Kumar

and Math

. The vanguard of psychiatry: Artificial intelligence as a catalyst for change. J Psychiatry Spectr, 2024; 3(1): 1–3.

12.

Gangadhar

, Kumar

, Sadh

, . Mental health programme in India: Has the tide really turned?. Indian J Med Res, 2023; 157(5): 387–394.