Sage Journals: Discover world-class research

Abstract

Objectives

To delineate areas of consensus and disagreements among practicing psychiatrists from various levels of clinical experience, hierarchy and organizations, and to test their ability to converge toward agreement, which will enable better integration of telepsychiatry into mental health services.

Methods

To study attitudes of Israeli public health psychiatrists, we utilized a policy Delphi method, during the early stages of the COVID pandemic. In-depth interviews were conducted and analyzed, and a questionnaire was generated. The questionnaire was disseminated amongst 49 psychiatrists, in two succeeding rounds, and areas of consensus and controversies were identified.

Results

Psychiatrists showed an overall consensus regarding issues of economic and temporal advantages of telepsychiatry. However, the quality of diagnosis and treatment and the prospect of expanding the usage of telepsychiatry to normal circumstances—beyond situations of pandemic or emergency were disputed. Nonetheless, efficiency and willingness scales slightly improved during the 2nd round of the Delphi process. Prior experience with telepsychiatry had a strong impact on the attitude of psychiatrists, and those who were familiar with this practice were more favorable toward its usage in their clinic.

Conclusions

We have delineated experience as a major impact on the attitudes toward telepsychiatry and the willingness for its assimilation in clinical practice as a legitimate and trustworthy method. We have also observed that the organizational affiliation significantly affected psychiatrists’ attitude, when those working at local clinics were more positive toward telepsychiatry compared with employees of governmental institutions. This might be related to experience and differences in organizational environment. Taken together, we recommend to include hands-on training of telepsychiatry in medical education curriculum during residency, as well as refresher exercises for attending practitioners.

Keywords

telemedicine mental health internet-based intervention telepsychiatry remote consultation rural health services computer-assisted diagnosis organizational culture high-income countries middle-income countries

Background

The fields of telemedicine and telepsychiatry have witnessed a substantial rise in practical and academic interest over the last decade, and especially following the outbreak of the COVID-19 pandemic.^1–4 Numerous studies highlighted the advantages of telepsychiatry, in terms of economic efficiency for patients and healthcare providers alike;^5,6 increase and ease of access in rural and remote localities;^7,8 as well as reducing congestion and alleviating the bottleneck of wait times.⁵

While telepsychiatry holds multiple advantages, it holds possible pitfalls as well. Despite the fact that interaction via videoconferencing has improved a lot during recent years, and the fact that its use became even more common during COVID-19 pandemic, certain populations do not have the ability to enjoy its benefits. Elderly people are not well acquainted with the technology, and some of the remote peripheral places still suffer from disrupted or slow internet communication, affecting the estimation of speech coherency and thought process. These limitations might impede the use of telepsychiatry for those who need it the most. In addition, telepsychiatry has inherent limitations even when communication is good and both therapist and patient are proficient in the technology. For example, the visual input which is limited to the face and neck and maybe upper torso might not disclose the full behavioral details of the patient, such as a fidgety leg, stereotypic hands movement, non-facial mannerisms, hallucinatory behavior and clothing details. It even enables the patient to manipulate the non-verbal data that is transferred to the therapist, by controlling the distance to the camera, thus obscuring visual details outside the face. Also, elements of smell, such as foul smell that discloses neglect are also missing in videoconferencing. A person who is used to cognitive assessment by the therapist can “cheat” by preparing written data that the camera does not disclose.^9,10

Indeed, studies among practitioners have also uncovered strongly conflicting views regarding the usage of telecommunication for mental healthcare. These included deep-seated objections and concerns among caregivers and psychiatrists regarding the quality of online examination,^11–13 the strength of therapeutic alliance^7,14; as well as its effects on social avoidance among patients.^8,12 Another area of concern focused on technical difficulties and limitations associated with online communication.^7,11

Some researchers hypothesized that such conflicting views are due to lack of practical experience with telepsychiatry, and that professionals’ objections will dissolve following such practical experience.^15–18 By contrast, other studies claimed these conflicts derive from distinctly different ethical and ideational perspectives regarding appropriate practices of psychological treatment; and are therefore not so easily negated.^19–21

The outbreak of the COVID-19 pandemic can serve as a natural experiment to test these hypotheses. The need to maintain social distance and enforce “stay at home” regulations has dramatically expanded the usage of telemedicine, and telepsychiatry more specifically. Mental healthcare professionals who have been so far reluctant to partake in telepsychiatry, were now obligated to practice and utilize this technology. Furthermore, practitioners who had limited previous experience with telepsychiatry, substantially increased their practice of telepsychiatry.^1–4,22

A previous qualitative study among Israeli mental healthcare professionals found indications that positive attitudes toward telepsychiatry are indeed correlated with levels of experience with this method.²³ Therefore, the purpose of the current study has been to examine longitudinal changes in attitudes toward telepsychiatry throughout the evolution of the COVID-19 pandemic in Israel, and to assess whether distinct sub-groups within the local mental healthcare professionals were more susceptible to such change. Furthermore, the current study sought to elaborate on possible points of consensus among local professionals regarding telepsychiatry. Hence, we aimed to delineate the areas of consensus and disagreements between practicing psychiatrists from various levels of clinical experience, hierarchy and organizations, and to test their ability to converge toward agreement, in order to enable better integration of telepsychiatry into mental health services.

Methods

A policy Delphi method has been chosen, being most suitable for exploring areas of consensus and disagreements among a panel of experts.^24,25 The Delphi method involves the systematic solicitation and collection of expert opinions on a particular topic, through sequential and interactive questionnaires. Essential components of the Delphi method are: (a) anonymity of participants, which allows free expression of opinion; (b) iterative rounds of response from the same panelists and (c) interactive process, which provides feedback to panelists, enabling them to reflect on their attitudes in light of the overall group response.^26,27 Originally developed to outline and explore areas of consensus, later versions included the exploration and development of policy alternatives and regions of descensus—in the form of Policy Delphi, which was utilized in the current study.^24,28,29

Questionnaire

A questionnaire was developed, following a thorough literature review, as well as in-depth interviews with Israeli mental healthcare professionals regarding their attitudes toward telepsychiatry.²³ Based on interviewees’ own phrasing, we distilled 30 statements covering the following issues: efficacy, efficiency, quality of treatment, alliance formation, diagnosis, compensation, legality, limitations, and barriers. This questionnaire was first pilot-tested on a small control group, checking for clarity of language and ease of use. An English translation of the original Hebrew questionnaire appears in Table A1.

Participants in the Policy Delphi study were asked to express their attitude toward each of the statements, using a 4-point Likert scale, where “1” indicates “strongly disagree,” “2” refers to “disagree,” “3” represents “agree” and “4” stands for “strongly agree.” Some of the items were framed as supportive of telepsychiatry, whereas others were framed as opposing, and were later reverse coded, so that all items scored higher marks in support of telepsychiatry usage. Due to the lack of a fixed scale point which represents a neutral value—between opposition and support—we defined a “neutral range” around the scale mean (2.2 > M > 2.8), whose values represent neither support nor opposition to the proposed statement.

Criteria for consensus

Several different criteria for measuring consensus have been offered by Delphi technique's practitioners.^30,31 The current study employed two of the more popular techniques: measures of central tendency (Mean) and dispersion (standard deviation [SD]). Mean represents the average attitude of all participants regarding a specific statement, while SD score reflects the level of individual's divergence/convergence around the mean. Items were considered consensual if they exhibit high level of convergence. Based on comparative reviews,^30,31 we will denote two threshold values for assessing the level of dispersion: high level of convergence (SD < 0.6) and high level of divergence (SD > 0.7). Regarding mean scores, items were considered consensual if they exhibit high level of support (M > 3.0) or high level of opposition (M < 2.0) to the specific statement included therein.

Procedure

The policy Delphi study included two rounds of online questionnaire distributed among mental healthcare professionals within the Israeli public healthcare system. These two rounds, conducted during 2020, corresponded with the outbreak and progress of COVID-19 in Israel. The 1st round was conducted between July and September 2020, corresponding with the rise of the second wave of COVID-19 in Israel. Government response during this period included partial weekend and local lockdowns, restrictions on social gatherings and mask mandate.¹ The questionnaire elicited participants’ attitudes toward structured statements regarding telepsychiatry; as well as questions regarding their previous experience in providing treatment through telepsychiatry, before and during the COVID-19 outbreak.

The 2nd round was conducted during November and December of 2020, when the COVID-19 pandemic was at a nadir, between the second and third waves. Government policy during this period included the lifting of most restrictions, except from specific localities in which high rates of infection still persisted.¹ For each of the statements in the questionnaire, participants were provided with feedback on the general distribution of responses in the previous round, and were asked to “reconsider their choice, in light of this new information.” Statements which achieved high levels of support during the 1st round (M > 3.0), were excluded from the 2nd round. Additionally, participants were probed regarding their experience with telepsychiatry during the pandemic, as well as their willingness to continue with telepsychiatry treatment in its aftermath.

Following the conclusion of the two rounds, exploratory factor analysis procedure was used to aggregate individual items into four scales which represent specific characteristics of common attitudes toward telepsychiatry. Items were first aggregated on a substantial and theoretical basis, to form rudimentary scales. These scales were then honed further through an exploratory factor analysis process, which eliminated some inconclusive and weaker correlations (eigenvalues below 0.35).

In order to assess the effects of the Delphi process, we compared the results from these two rounds, for individual items as well as aggregated scales. Two types of effects were expected, and tested for: (a) a substantial change in the mean score, representing an overall shift in attitudes toward telepsychiatry; measured through a paired t-test;² (b) a substantial change in the dispersion of participants’ answers, representing a “convergence around the mean”; measured by testing for equality of variance (both Levene's and F-test were used).³ A positive result on each of these measures indicated greater support and higher consensus on telepsychiatry, respectively (Figure 1).

Figure 1.

Study design.

Participants

Participants were recruited from among active mental healthcare professionals within the Israeli public healthcare system. A conscious effort was made to reach multi-level diversity within the sample, by allocating sampling quotas to regional, organizational, and hierarchical groups within the examined professional community. Over 300 email invitations were sent to mental healthcare professionals in hospitals, regional and local clinics, both in government service and Health Maintenance Organizations (HMO)⁴ employees. Specific attempts were made to recruit psychiatrists in managerial posts within government service. Overall, 56 professionals participated in the 1st Delphi round (19% response rate), and 49 completed questionnaires for both rounds (16% response rate).

Results

Table 1 details the main characteristics of the 49 participants, which constitute the sample population for this study.

Table 1.

Characteristic of participants.

		HMOs	Government	Total
		N (%)	N (%)	N (%)
Gender	Male	11 (22)	12 (25)	23 (47)
Gender	Female	16 (33)	10 (20)	26 (53)
Age	31–40	3 (6)	2 (4)	5 (10)
	41–50	14 (29)	7 (14)	21 (43)
	51–60	9 (19)	6 (12)	15 (31)
	61–70	1 (2)	7 (14)	8 (16)
Workplace^a	Local clinics	18 (36)	9 (19)	27 (55)
	Hospitals	6 (12)	11 (23)	17 (35)
	Management	5 (10)	4 (8)	9 (18)
Seniority	1–10 years	12 (25)	5 (10)	17 (35)
	11–20 years	10 (20)	9 (18)	19 (38)
	Above 20	5 (10)	8 (16)	13 (26)
Experience with telepsychiatry prior to the COVID-19 pandemic	No prior experience	12 (24)	16 (33)	28 (57)
	Prior experience	15 (31)	6 (12)	21 (43)
Total		27 (55)	22 (44)	49 (100)

Some of our participants (∼8%) hold two jobs within the public healthcare system.

Consensuses and disagreements regarding telepsychiatry in the Delphi process

An analysis of the Delphi rounds reveals patterns of consensus and disagreement among the participants. The 1st round exhibited several areas of consensus, together with distinct points of disagreement. The 2nd round expanded the consensus on some issues, while sharpening the disagreement around others. Summary statistics for the questionnaire items can be found in Table A1.

The 1st Delphi round included 30 statements. Six of these items were considered consensual—exhibiting high levels of both support and convergence—making a second round redundant (Q1, Q2, Q3, Q6, Q14, Q23). Two additional items displayed high levels of convergence (SD < 0.6), even though their mean score was not highly supportive (Q9, Q29). Another three items showed borderline levels of high support/opposition (M∼ = 3/M∼ = 2), combined with high levels of divergence (SD > 0.7; Q11, Q25, Q27). Furthermore, 18 additional items exhibited high levels of divergence in their dispersion rates (SD > 0.7), indicating a high level of disagreement of opinion among participants on these statements.

The 2nd Delphi round included 24 items. Three of these items showed borderline levels of high support/opposition (M∼ = 3/M∼ = 2), combined with high levels of divergence (SD > 0.7; Q11, Q25 and Q30). Ten items now exhibited high levels of convergence (SD < 0.6; Q9, Q10, Q12, Q16, Q17, Q18, Q22, Q27, Q28, Q29), while five others displayed high degree of disagreement between participants (SD > 0.7; Q5, Q11, Q13, Q15, Q26).

A comparison of results between the two Delphi rounds reveals that seven items exhibited a significant change in their mean score, with only two of these changes amounting to substantial effect (M > 3; Q11, Q30). Furthermore, 12 of the items showed a significant change in their dispersion scores, indicating increased convergence of respondents around the mean score. For seven of these items, the convergence effect was substantial enough to affect a high degree of consensus (SD < 0.6; Q10, Q16, Q17, Q18, Q22, Q27, Q28). These seven items refer to issues of limited willingness to practice telepsychiatry—under specific conditions; as well as to difficulties with traditional treatment, which are addressed through telepsychiatry.

By contrast, comparison also reveals six items which maintained high levels of divergence and disagreement between participants throughout the Delphi process (SD > 0.7; Q5, Q11, Q13, Q16, Q26, Q30). These six items refer to issues of quality of diagnosis and treatment with telepsychiatry; as well as willingness to practice telepsychiatry under normal circumstances, excluding situations of emergency or pandemic.

In order to better assess changes in attitudes, individual items were aggregated into four scales, representing specific groups of attitudes regarding telepsychiatry, using exploratory factor analysis: Efficiency scale incorporated items which referred to costs and benefits associated with telemedicine, time savings and higher efficiency among practitioners, etc. (Q4, Q9, Q10, Q12, Q29; Cronbach's α = 0.688, KMO = 0.63). Quality scale incorporated items which compared the quality of clinical diagnosis, treatment and therapeutic alliance between online and face-to-face treatment (Q7, Q13, Q15, Q16, Q20; Cronbach's α = 0.889, KMO = 0.821). Legitimacy scale incorporated items which questioned the legal and ethical boundaries of online treatment, and examined their legitimacy in the eyes of practitioners (Q5, Q19, Q22, Q25; Cronbach's α = 0.692, KMO = 0.709). While Willingness scale incorporated items that examined one's actual willingness to engage with telepsychiatry, personally (Q26, Q30; Cronbach's α = 0.887, KMO = 0.5). Additionally, these four scales were combined together onto a single scale, to form an overall assessment of support for telepsychiatry (Q4, Q5, Q7, Q9, Q10, Q12, Q13, Q15, Q16, Q19, Q20, Q22, Q25, Q26, Q29, Q30; Cronbach's α = 0.902, KMO = 0.772). Fuller details for the factor analysis can be found in Table A2.

An analysis of these aggregated scales reveals significant changes between rounds in their overall scores (Table 2). These results show that the overall mean scores for efficiency and legitimacy scales are significantly higher for the 2nd round, indicating a more positive attitude toward telepsychiatry in these attributes. Furthermore, Levene's test for equality of variance of the quality and willingness scales reveal a significant decrease in the dispersion of responses between rounds, indicating a convergence effect for these scales. However, the overall support scale does not show any significant change, indicating that, overall, the level of support for telepsychiatry remained unchanged.

Table 2.

Descriptive statistics, t-tests and Levene's test for combined indexes in two Delphi rounds.

	Round 1	Round 2	t-test	Levene's test
Items	Mean ± SD	Mean ± SD	ΔMean (p)	W (p)
Efficiency	2.75 ± 0.43	2.83 ± 0.36	0.081 (.022)	NS*^a
Quality	2.44 ± 0.7	2.42 ± 0.51	NS	5.629 (.02)
Legitimacy	2.41 ± 0.55	2.47 ± 0.47	0.061 (.042)	NS
Willingness	2.87 ± 0.83	2.9 ± 0.68	NS	3.522 (.064)
Overall support	2.62 ± 0.52	2.65 ± 0.41	NS	NS

Throughout this article, only statistically significant results (p < .1) are presented numerically, otherwise an “NS” label is included for statistical tests, representing a “non-significant” result. *NS refers to “non-significant” results.

The effects of COVID-19-induced experience on the use of telepsychiatry

In order to assess the effects of experience gained due to the COVID-19 pandemic, we divided the sample population into subgroups, comparing scores between sub-groups and between rounds. First, a preliminary comparison was made, between participants who had some experience with telepsychiatry before the advent of COVID-19 in Israel (21), and those who had no experience prior to the pandemic (28). This comparison revealed significant differences in aggregated scores between participants with and without experience (Table 3).

Table 3.

Differences between those with and without experience with telepsychiatry prior to Delphi.

		Round 1	Round 2	t-test	Levene's test
Prior experience	Items	Mean ± SD	Mean ± SD	Δ Mean (p)	W (p)
Without prior experience N = 28	Efficiency	2.71 ± 0.42	2.83 ± 0.29	0.116 (.028)	NS
	Quality	2.25 ± 0.62	2.26 ± 0.44	NS	3.585 (.064)
	Legitimacy	2.17 ± 0.45	2.29 ± 0.38	0.116 (.006)	NS
	Willingness	2.66 ± 0.68	2.7 ± 0.58	NS	NS
	Overall support	2.45 ± 0.43	2.52 ± 0.33	0.07 (.019)	NS
With prior experience N = 21	Efficiency	2.8 ± 0.46	2.83 ± 0.44	NS	NS
	Quality	2.7 ± 0.73	2.62 ± 0.52	NS	1.954 (.071)
	Legitimacy	2.73 ± 0.52	2.71 ± 0.48	NS	NS
	Willingness	3.14 ± 0.94	3.17 ± 0.71	NS	NS
	Overall support	2.84 ± 0.55	2.83 ± 0.45	NS	NS

Results from these comparative tests (t-test and Leven's test) show that the significant changes between rounds found in Table 2 are mostly associated with participants who had no prior experience with telepsychiatry. These participants exhibited a significantly positive increase in their mean scores for efficiency, legitimacy, and overall support scales. They also exhibited a marginally significant (p = .064) convergence effect in the quality scale. By contrast, participants who had some experience with telepsychiatry prior to the COVID-19 pandemic, exhibited no significant changes in mean scores for any of the examined scales, and only a marginally significant convergence effect on the quality scale.⁵

Next, we examined the effects of the COVID-19 pandemic on attitudes, distinguishing between participants who experienced / did not experience an increase in the volume of their tele-psychiatric practice during the pandemic. Comparing the attitudes of these sub-groups revealed that most of the significant changes between Delphi rounds can be attributed to those participants that substantially increased their practice of telepsychiatry during the pandemic (Table 4). This was reflected by the significant increase in the mean scores for efficiency, legitimacy and overall support scales, as well as marginally significant convergence effects on four of the examined scales (all but efficiency).

Table 4.

Change and continuity in the extent of telepsychiatry practice.

		Round 1	Round 2	t-test	Levene's test
Effect of COVID-19	Items	Mean ± SD	Mean ± SD	Δ Mean (p)	W (p)
No change in practice N = 16	Efficiency	2.94 ± 0.4	2.92 ± 0.43	NS	NS
	Quality	2.89 ± 0.73	2.7 ± 0.58	−0.187 (.032)	NS
	Legitimacy	2.64 ± 0.56	2.66 ± 0.54	NS	NS
	Willingness	3.19 ± 0.91	3.13 ± 0.87	NS	NS
	Overall support	2.91 ± 0.57	2.85 ± 0.51	−0.063 (.0833)	NS
Increase in extent during COVID-19 N = 33	Efficiency	2.66 ± 0.43	2.79 ± 0.32	0.127 (.009)	NS
	Quality	2.22 ± 0.58	2.28 ± 0.41	NS	3.171 (.079)
	Legitimacy	2.3 ± 0.52	2.38 ± 0.42	0.083 (.039)	3.512 (.066)
	Willingness	2.71 ± 0.75	2.79 ± 0.54	NS	3.482 (.067)
	Overall support	2.47 ± 0.43	2.56 ± 0.32	0.085 (.008)	1.812 (.049)

By comparison, participants that did not experience an increase in their practice of telepsychiatry, exhibited a significant decrease in their mean score for quality scale, as well as overall support for telepsychiatry. Interestingly, out of the 16 participants who did not report any change in their practice of telepsychiatry, half (8) were at the highest range of practice from the start; While the other half (8) did not practice telepsychiatry at all, neither before nor during the COVID-19 pandemic.

In an attempt to further pinpoint the effects of experience during COVID-19, we distinguished between those with and without prior experience, among the participants who experienced an increase in their practice of telepsychiatry during the COVID-19 pandemic (Table 5).

Table 5.

Comparison of participants with and without prior experience who reported increase in practice of telepsychiatry during COVID-19 (n = 33).

Increase in practice during COVID-19		Round 1	Round 2	t-test	F-test
Increase in practice during COVID-19	Items	Mean ± SD	Mean ± SD	ΔMean (p)	F (p)
Without prior experience “first timers” N = 20	Efficiency	2.66 ± 0.42	2.83 ± 0.27	0.165 (.015)	2.37 (.034)
	Quality	2.17 ± 0.59	2.24 ± 0.38	NS	2.478 (.027)
	Legitimacy	2.13 ± 0.52	2.27 ± 0.41	0.15 (.005)	NS
	Willingness	2.66 ± 0.61	2.75 ± 0.53	NS	NS
	Overall support	2.41 ± 0.4	2.52 ± 0.3	0.115 (.002)	NS
With prior experience N = 13	Efficiency	2.66 ± 0.46	2.73 ± 0.39	NS	NS
	Quality	2.31 ± 0.58	2.34 ± 0.46	NS	NS
	Legitimacy	2.56 ± 0.43	2.54 ± 0.39	NS	NS
	Willingness	2.77 ± 0.95	2.85 ± 0.59	NS	3.802 (.063)
	Overall support	2.57 ± 0.47	2.61 ± 0.36	NS	NS

The comparison—between participants who practiced telepsychiatry for the first time during the pandemic (“first timers”), and those who had prior experience with telepsychiatry, and increased their volume of practice during the pandemic—revealed significant changes between rounds among the “first timers.” These participants exhibited significant increases in their mean scores for efficiency, legitimacy and overall support scales. As well as significant convergence effects on efficiency and quality scales. By contrast, participants with prior telepsychiatry experience exhibited no significant changes, except a marginally significant convergence effect on the willingness scale.

Furthermore, within each round, we compared the attitudes toward telepsychiatry of “first timers” against participants with experience prior to the COVID-19 pandemic (Table 6). This comparison revealed that in each of the rounds, the mean scores for “first timers” were significantly lower on the Quality, Legitimacy, Willingness and overall support scales. These gaps were maintained between rounds, and not affected by changes in “first timers” scores, noted in Table 6.

Table 6.

Participants with any telepsychiatry experience (n = 41): comparing Delphi rounds between “first-timers” (n = 20) and participants with prior experience (n = 21).

	Round 1			Round 2
	Prior exp	First timers	t-test	Prior exp	First timers	t-test
	M ± SD	M ± SD	ΔM (p)	M ± SD	M ± SD	ΔM (p)
Efficiency	2.8 ± 0.46	2.66 ± 0.42	NS	2.83 ± 0.43	2.83 ± 0.27	NS
Quality	2.69 ± 0.73	2.17 ± 0.59	−0.525 (.008)	2.63 ± 0.53	2.24 ± 0.38	−0.379 (.006)
Legitimacy	2.73 ± 0.52	2.13 ± 0.52	−0.601 (.003)	2.71 ± 0.48	2.28 ± 0.41	−0.44 (.002)
Willingness	3.14 ± 0.94	2.68 ± 0.61	−0.468 (.034)	3.17 ± 0.71	2.75 ± 0.53	−0.417 (.02)
Overall support	2.84 ± 0.55	2.41 ± 0.4	−0.434 (.003)	2.83 ± 0.45	2.52 ± 0.3	−0.311 (.006)

Workplace differences

Finally, we examined specific employment groups, to investigate the effect of workplace on attitudes toward telepsychiatry. We compared scale scores between rounds for participants working in hospital wards, versus outpatient clinics in the community, (Table 7). Results show that employees of local clinics exhibited the most significant changes between rounds, with significant increases in mean score for efficiency, legitimacy and willingness scales. Hospital staff exhibited a significant increase only in mean score for efficiency scale, while also demonstrating a marginally significant convergence effect on quality scale.

Table 7.

Comparing attitudes among participants in different workplaces.

		Round 1	Round 2	t-test	F-test
Main workplace	Items	Mean ± SD	Mean ± SD	ΔMean (p)	F (p)
Hospital N = 17	Efficiency	2.69 ± 0.5	2.9 ± 0.41	0.203 (.008)	NS
	Quality	2.26 ± 0.64	2.27 ± 0.44	NS	2.265 (.074)
	Legitimacy	2.44 ± 0.57	2.49 ± 0.5	NS	NS
	Willingness	2.71 ± 0.66	2.74 ± 0.53	NS	NS
	Overall support	2.52 ± 0.49	2.6 ± 0.39	0.072 (.057)	NS
Outpatient clinics in the community N = 27	Efficiency	2.75 ± 0.49	2.83 ± 0.43	0.085 (.043)	NS
	Quality	2.4 ± 0.76	2.44 ± 0.59	NS	NS
	Legitimacy	2.37 ± 0.61	2.45 ± 0.55	0.083 (.024)	NS
	Willingness	2.87 ± 0.97	2.98 ± 0.79	0.111 (.068)	NS
	Overall support	2.6 ± 0.59	2.68 ± 0.49	0.079 (.017)	NS

When comparing attitudes toward telepsychiatry between HMO and government employees, the analysis revealed significant differences within each round. Results show that the mean scores for government employees are significantly lower for quality (ΔM = −0.404; p = .022), legitimacy (ΔM = −0.287; p = .035), willingness (ΔM = −0.502; p = .017), and overall support (ΔM = −0.342; p = .01) scales. These differences are carried and maintained throughout the Delphi process (both rounds).

Discussion

Consensus and dissensus in the Delphi process: Unveiling complex attitudes towards telepsychiatry

The comparison of the Delphi process regarding individual items revealed specific areas of consensus and agreement between participants, as well as regions of continued dissensus and disagreement. Some of the examined items showed strong tendencies of collective support and consensus early on—in the 1st round—with only minor changes during the 2nd round of the Delphi process. These items referred to issues of economic and temporal advantages to patients and practitioners alike (Q1, Q2, Q6, Q9, Q29), as well as the legal and technical difficulties associated with its practice (Q3, Q11, Q14, Q25, Q27).

The Delphi study also demonstrated that many items remained strongly contested, with high levels of disagreement among participants, throughout the Delphi rounds. These items exhibited mean scores in the “neutral” range, as well as high level of divergence between participants. Contested items referred mostly to issues regarding the quality of diagnosis and treatment, which types of patients were suitable for telepsychiatry, as well as to the practitioner's willingness to practice telepsychiatry and conditions for doing so. The 2nd round of the Delphi process exhibited a consolidation of consensus, in the form of convergence around the mean, on some of the finer points of these issues. While mean scores remained mostly neutral, a substantially high level of convergence was achieved on issues of willingness to practice, under specific conditions (additional compensation and better technology), as well as advantages for telepsychiatry in dealing with some problematic aspects of traditional treatment (threat of violence, no-shows and wait-times).

While these changes represent a “positive” outcome of the Delphi process, they do not herald any attitudinal change in support of telepsychiatry. Moreover, disagreement on other issues remained adamant. More specifically, the quality of diagnosis and treatment, as well as prospect of expanding the usage of telepsychiatry to normal circumstances—beyond situations of pandemic or emergency—remained a highly contested issue.

After aggregating individual items into combined scales, the results of the Delphi process for the entire sample population revealed mixed trends and complex attitudes toward telepsychiatry. Overall, the average scores for quality and legitimacy scales expressed a neutral position toward telepsychiatry, which did not change much between rounds. A statistically-significant increase in the score of the legitimacy scale actually advanced it into a more neutral position. Concomitantly, the average scores for the efficiency and willingness scales exhibited a mildly-positive attitude toward telepsychiatry, which was somewhat improved during the 2nd round (a more positive mean score for efficiency and a convergence effect on the willingness scale). However, these average scores fell short of widespread support (M > 3).

In regards to the role of experience and especially enhanced experience during the COVID-19 pandemic, our findings reveal that the largest and most significant differences in scores between sub-groups within each round are between participants with prior experience with telepsychiatry, and those participants’ which practiced this method of treatment only during the pandemic. First timers are more opposed regarding the quality and legitimacy of telepsychiatry, and this tendency was sustained in both Delphi rounds. In regards to willingness to practice telepsychiatry, participants with prior experience presented a strong positive consensus, while first timers exhibit positive but substantially lower levels of support.

However, our results also indicate that “first timers” are more open to change, and their attitudes toward telepsychiatry were altered over time. The analysis revealed a substantial positive shift in the mean scores of efficiency and legitimacy scales, which was entirely attributed to “first timers.” Furthermore, “first timers” were also responsible for the convergence effect around the mean for efficiency and quality scores. These changes may be attributed to the Delphi process, as they affected only a sub-group of participants.

Finally, a comparison of attitudes by workplace revealed that participants who worked in outpatient clinics in the community exhibited the most significant changes in attitudes between the Delphi rounds. The study also exposed significant differences in attitudes toward telepsychiatry between HMO and government employees, with the latter exhibiting consistently lower levels of support for the new method. These differences were not altered by the Delphi process.

Previous studies have suggested that additional indicators including level of development, geography, culture, and career stage, are possible effectors of attitudes of psychiatrists toward telepsychiatry. For example, Teixeira and colleagues suggested that early career psychiatrists were optimistic that the COVID-19 pandemic is an opportunity for expanding telepsychiatry.³² Furthermore, El Hayek and his colleagues showed that adoption of telepsychiatry varies a lot and largely depends on the level of development and affluence of the country.³³ Our study included a rather small sample size. Future studies should include a bigger sample size to allow stratification by additional categories. In addition, Israel is a small, developed country in which internet infrastructure is highly available, and cellular communication is widely used (Table A3).

Consensus, disagreement, and the role of practical and pandemic-induced experience

The results of the current study—regarding strong consensus and agreement between participants on specific issues related to telepsychiatry—corresponds with the general consensus reported previously in the literature, regarding its advantages in terms of economic costs and time-saving, as well as the difficulties associated with technological shortcomings.^1,6,12

Similarly, the adamant contention and disagreement regarding the quality of online examination and treatment, as well as its suitability for different types of mental illnesses—echoes similar findings in past research.^7,11,12,14 However, the current study represents one of the few examples where all these diverse aspects of telemedicine are investigated interconnectedly, in a systematic fashion.

In regards to the effects of practical experience and pandemic-induced experience on attitudes toward telepsychiatry, several studies have shown a correlation between practical experience and positive attitudes toward telemedicine, against the backdrop of the pandemic.^34–36 However, the present study is unique in its focus on mental healthcare, as well as more detailed and systematic in analyzing these relationships.

Furthermore, while several studies focused on attitudes toward telemedicine and telepsychiatry within specific sectors of the healthcare system (rural clinics, hospitals, VA clinics, etc.^12,37,38), the current study is unique in its systematic comparison of attitudes within and between different sectors of the Israeli mental healthcare system.

Limitations

The small sample size (49 participants) as well as its dissection into sub-groups, may be considered a liability in terms of statistical significance and representation. However, such level of participation is considered satisfactory in Policy Delphi studies, due to their recurrent and interactive nature.^39,40 Additional limitations may be associated with the co-occurrence of the COVID-19 pandemic, which exerted some undue influence on the Delphi process. However, the pandemic also created unique opportunities to examine the effects of experience, thereby mitigating some of these shortcomings.

Conclusions

COVID-19 added to the legitimacy of telepsychiatry, and dramatically accelerated its adoption. These circumstances are likely to increase acceptance of this tool among patients, psychiatrists and health organizations in the post-pandemic future.^3,41 The present study sought to examine longitudinal changes and professional differences in attitudes toward telepsychiatry among Israeli mental healthcare professionals, against the backdrop of the COVID-19 pandemic. Furthermore, this study endeavored to investigate the effect that practical experience with telepsychiatry may have on one's attitudes toward it.

An increasing significance to the assimilation of telepsychiatry into health systems is driven by the emergence of multiple mobile applications that provide additional data concerning patients’ health and generating tighter communication between the different caretakers, and even informal caregivers^.⁴² Thus, apps that are unique to telepsychiatry could facilitate multiple advancements such as joint meetings of a group of caretakers exchanging ideas and data,^43,44 streaming of data collected from mobile-based apps that provide data concerning voice modulation, movement, money spending, social interactions, frequency of phone calls, and other data,^45–48 and also enable psychotherapeutic groups sessions^49,50 omitting the logistical requirements within a clinical facility.

Results demonstrated a strong correlation between practical experience and positive attitudes toward telepsychiatry. Furthermore, the study showed that practitioners who experienced telepsychiatry for the first time during the COVID-19 pandemic, were more open to attitudinal change, and exhibited the most significant and positive change over time. Based on these findings we recommend that hands-on training and simulations of telemedicine and telepsychiatry will be included in medical education curriculum, as well as simulation and refresher training sessions provided for attending practitioners.

The study also revealed significant differences in attitudes toward telepsychiatry among different sectors of the Israeli mental healthcare community. More specifically, HMO employees exhibited a consistently more positive attitude toward telepsychiatry than practitioners employed in government-owned hospitals and clinics. Furthermore, participants working in local clinics demonstrated a significant and positive longitudinal change in their attitudes toward telepsychiatry, which was not visible in other sectors. We attribute these differences to greater exposure and positive experience with telepsychiatry among HMO-owned local clinics. Therefore, we recommend to enhance the integration of telemedicine into government-owned clinics, as well as expand its utilization and applicability in psychiatric hospitals’ outpatient services.

Additionally, the present study explored points of consensus and disagreement regarding the comparability of telepsychiatry with traditional face-to-face treatment. Results show a strong consensus among participants regarding the economic and convenience advantages usually associated with telemedicine, together with a common understanding regarding the undermining effects of technical difficulties associated with internet communications. In order to resolve some of these difficulties, we recommend improving the internet infrastructure available to public clinics, as well as the establishment of dedicated videoconference rooms at rural and remote communities.

Beyond these observed points of consensus, the findings of the present study revealed a significant area of contention and disagreement between participants regarding the quality of diagnosis and treatment with telepsychiatry, as well as the types of diagnoses suitable for such treatment and one's personal willingness to practice such method on a regular basis. Even when consensus was finally achieved on some of these issues, it consolidated on a “neutral” position which did not convey a positive or negative evaluation. These findings may indicate that the routinization of telepsychiatry may not be so easily achieved, and that practical experience under extreme circumstances (such as the COVID pandemic) may also have negative effects on attitudes. Further study is needed as to the effects of different types of experience on evaluation and attitudes toward telemedicine and telepsychiatry.

Despite the small-sized sample and the systemic differences between Israel and other Western countries in terms of healthcare system architecture, we believe that these findings may be applicable to other countries which employ a government-controlled and regulated healthcare system.

Footnotes

Acknowledgements

The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Contributionship: Conceptualization was contributed by MN and HK; methodology was contributed by TM; software was contributed by TM; validation was contributed by TM, MN, and HK; formal analysis was contributed by TM; investigation was contributed by TM; resources was contributed by MN and HK; data curation was contributed by TM; writing–original draft preparation was contributed by TM; writing–review and editing was contributed by MN and HK; visualization was contributed by TM; supervision was contributed by MN and HK; project administration was contributed by MN and HK; funding acquisition was contributed by MN and HK. All authors have read and agreed to the published version of the manuscript.

Data Availability

The data presented in this study is available on request from the corresponding author. The data is not publicly available due to confidentiality and anonymity considerations. Excepts used in the article were deducted, to ensure anonymity.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical Approval

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethics Committee of UNIVERSITY OF HAIFA (Authorization No. 327/19, Sep 05, 2019).

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Israel National Institute for Health Policy Research, (grant number 2018/11r).

Gurantor

All authors are gurantors.

Informed Consent

Informed consent, including agreement for publication, was obtained from all subjects involved in the study.

ORCID iDs

Maya Negev

Tamir Magal

Hanoch Kaphzan

Notes

Appendix

Table A3.

Additional items included in the Delphi questionnaire

Content of item	T1	T2
Demographics
Age	V
Sex	V
Geographical region of practice	V
HMO/Government employee	V
Professional qualifications
Where did you acquire your medical degree?	V
Where did you acquire your psychiatric training?	V
What is your primary place of employment, and role?	V
What is your secondary place of employment, and role?	V
Year of finishing psychiatric training	V
Experience with telepsychiatry
Have you ever treated patients via telepsychiatry within the public healthcare system?	V
How many patients have you treated via telepsychiatry before the COVID-19 pandemic/during 2019?	V	V
How many patients have you treated via telepsychiatry since the outbreak of the COVID-19 pandemic?	V	V
What percentage of your patients would you like to treat via telepsychiatry, following the conclusion of the COVID-19 pandemic?		V

References

Guinart

Patricia

Hauser

, et al. Mental health care providers’ attitudes toward telepsychiatry: a systemwide, multisite survey during the COVID-19 pandemic. Psychiatr Serv 2021; 72: 704–707.

Kannarkat

Smith

McLeod-Bryant

. Mobilization of telepsychiatry in response to COVID-19—moving toward 21st century access to care. Administration and Policy in Mental Health and Mental Health Services Research 2020; 47: 489–491.

O’Brien

McNicholas

. The use of telepsychiatry during COVID-19 and beyond. Ir J Psychol Med 2020; 37: 250–255.

Shore

Schneck

Mishkind

. Telepsychiatry and the coronavirus disease 2019 pandemic—current and future outcomes of the rapid virtualization of psychiatric care. JAMA Psychiatry 2020; 77: 1211–1212.

Hubley

Lynch

Schneck

, et al. Review of key telepsychiatry outcomes. World J Psychiatry 2016; 6: 269–282.

Mayworm

Lever

Gloff

, et al. School-based telepsychiatry in an urban setting: efficiency and satisfaction with care. Telemedicine and e-Health 2020; 26: 446–454.

Connolly

Miller

Lindsay

, et al. A systematic review of providers’ attitudes toward telemental health via videoconferencing. Clinical Psychology: Science and Practice 2020; 27: e12311.

Lindsay

Hudson

Martin

, et al. Implementing video to home to increase access to evidence-based psychotherapy for rural veterans. Journal of Technology in Behavioral Science 2017; 2: 140–148.

Shoib

Dass

de Filippis

, et al. Mental status via telepsychiatry: the potential pitfalls. Encephale 2022; 48: 712–713.

10.

Hailey

Ohinmaa

Roine

. Limitations in the routine use of telepsychiatry. J Telemed Telecare 2009; 15: 28–31.

11.

Cowan

McKean

Gentry

, et al. Barriers to use of telepsychiatry: clinicians as gatekeepers. Mayo Clin Proc 2019; 94: 2510–2523.

12.

Newman

Bidargaddi

Schrader

. Service providers’ experiences of using a telehealth network 12 months after digitisation of a large Australian rural mental health service. Int J Med Inf 2016; 94: 8–20.

13.

Werner

Karnieli

. A model of the willingness to use telemedicine for routine and specialized care. J Telemed Telecare 2003; 9: 264–272.

14.

Norwood

Moghaddam

Malins

, et al. Working alliance and outcome effectiveness in videoconferencing psychotherapy: a systematic review and noninferiority meta-analysis. Clin Psychol Psychother 2018; 25: 797–808.

15.

Bhattacherjee

Hikmet

. Physicians’ resistance toward healthcare information technologies: a dual-factor model. 40th Annual Hawaii International Conference on System Sciences (HICSS’07). Hawai: IEEE, 2007, 141–151.

16.

Grigsby

Brega

Bennett

, et al. The slow pace of interactive video telemedicine adoption: the perspective of telemedicine program administrators on physician participation. Telemedicine and e-Health 2007; 13: 645–656.

17.

Kuo

K-M

Talley

Lee

C-M

, et al. The influence of telemedicine experience on physicians’ perceptions regarding adoption. Telemedicine and e-Health 2015; 21: 388–394.

18.

Saigí-Rubió

Torrent-Sellens

Jiménez-Zarco

. Drivers of telemedicine use: comparative evidence from samples of Spanish, Colombian and Bolivian physicians. Implement Sci 2014; 9: 128–140.

19.

Barton

Brega

Devore

, et al. Specialist physicians’ knowledge and beliefs about telemedicine: a comparison of users and nonusers of the technology. Telemedicine and e-Health 2007; 13: 487–500.

20.

Gagnon

M-P

Godin

Gagné

, et al. An adaptation of the theory of interpersonal behaviour to the study of telemedicine adoption by physicians. Int J Med Inf 2003; 71: 103–115.

21.

Kim

DelliFraine

Dansky

, et al. Physicians’ acceptance of telemedicine technology: an empirical test of competing theories. Int J Inf Syst Change Mange 2010; 4: 210–225.

22.

Barney

Buckelew

Mesheriakova

, et al. The COVID-19 pandemic and rapid implementation of adolescent and young adult telemedicine: challenges and opportunities for innovation. Journal of Adolescent Health 2020; 67: 164–171.

23.

Magal

Negev

Kaphzan

. Attitudinal barriers hindering adoption of telepsychiatry among mental healthcare professionals: Israel as a case-study. International Journal of Enviornmental Research and Public Health 2021; 18: 1–12.

24.

Franklin

Hart

. Idea generation and exploration: benefits and limitations of the Policy Delphi research method. Innovative Higher Education 2007; 31: 237–246.

25.

Rayens

Hahn

. Building consensus using the Policy Delphi method. Policy, Politics, & Nursing Practice 2000; 1: 308–315.

26.

Hsu

C-C

Sandford

. The Delphi technique: making sense of consensus. Practical Assessment, Research, and Evaluation 2007; 12: 1–8.

27.

Vernon

. The Delphi technique: a review. Int J Ther Rehabil 2009; 16: 69–76.

28.

de Loë

Melnychuk

Murray

, et al. Advancing the state of Policy Delphi practice: a systematic review evaluating methodological evolution, innovation, and opportunities. Technol Forecast Soc Change 2016; 104: 78–88.

29.

Sossa

JWZ

Halal

Zarta

. Delphi method: analysis of rounds, stakeholder and statistical indicators. foresight 2019; 21: 525–544.

30.

Diamond

Grant

Feldman

, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol 2014; 67: 401–409.

31.

von der Gracht

. Consensus measurement in Delphi studies: review and implications for future quality assurance. Technol Forecast Soc Change 2012; 79: 1525–1536.

32.

Teixeira

ALS

Spadini

Pereira-Sanchez

, et al. The urge to implement and expand telepsychiatry during the COVID-19 crisis: early career psychiatrists’ perspective. Rev Psiquiatr Salud Ment 2020; 13: 174–175.

33.

El Hayek

Nofal

Abdelrahman

, et al. Telepsychiatry in the arab world: a viewpoint before and during COVID-19. Neuropsychiatr Dis Treat 2020; 16: 2805–2815.

34.

De Lott

Newman-Casey

Lee

, et al. Change in ophthalmic clinicians’ attitudes toward telemedicine during the coronavirus 2019 pandemic. Telemedicine and e-Health 2020; 27: 231–235.

35.

Kong

Azarfar

Ashour

, et al. Awareness and attitudes towards telemedicine among medical students in the United States. Cureus 2020; 12: e11574-e.

36.

Schinasi

Foster

Bohling

, et al. Attitudes and perceptions of telemedicine in response to the COVID-19 pandemic: A survey of naïve healthcare providers. Front Pediatr 2021; 9. Article 647937.

37.

Adler-Milstein

Kvedar

Bates

. Telehealth among US hospitals: several factors, including state reimbursement and licensure policies, influence adoption. Health Aff 2014; 33: 207–215.

38.

McClellan

Florell

Palmer

, et al. Clinician telehealth attitudes in a rural community mental health center setting. Journal of Rural Mental Health 2020; 44: 62–73.

39.

Hung

H-L

Altschuld

Lee

Y-F

. Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation. Eval Program Plann 2008; 31: 191–198.

40.

Trevelyan

Robinson

. Delphi methodology in health research: how to do it? Eur J Integr Med 2015; 7: 423–428.

41.

Fisk

Livingstone

Pit

. Telehealth in the context of COVID-19: changing perspectives in Australia, the United Kingdom, and the United States. J Med Internet Res 2020; 22: e19264.

42.

Oakley-Girvan

Yunis

Fonda

, et al. A novel smartphone application for the informal caregivers of cancer patients: usability study. PLoS Digital Health 2023; 1: 1–16.

43.

Herbert

Brennan

JMM

. Use of Mobile apps & stepped-care model for treating depression in primary care. Primary Care: Clinics in Office Practice 2023; 50: 11–19.

44.

. Digital technologies for mental health improvements in the COVID-19 pandemic: a scoping review. BMC Public Health 2023; 23: 1–10.

45.

Eisner

Berry

Morris

, et al. Exploring engagement with the CBT-informed actissist smartphone application for early psychosis. Journal of Mental Health 2023: 1–12.

46.

Montgomery

Neil

Cannell

, et al. The prevalence and perceived utility of mobile health technology among recently incarcerated homeless adults. Journal of Technology in Behavioral Science 2023: 1–9.

47.

Hilbert

Juarascio

Prettin

, et al. Smartphone-supported behavioural weight loss treatment in adults with severe obesity: study protocol for an exploratory randomised controlled trial (SmartBWL). BMJ open 2023; 13: e064394.

48.

Alamoudi

Breeze

Crawley

, et al. The feasibility of using smartphone sensors to track insomnia, depression, and anxiety in adults and young adults: narrative review. JMIR Mhealth Uhealth 2023; 11: e44123.

49.

Bock

Graf

Woeber

, et al. Radical acceptance of reality: putting DBT®-a skill groups online during the COVID-19 pandemic: a qualitative study. Front Psychiatry 2022; 13: e617941.

50.

Weinberg

. Obstacles, challenges, and benefits of online group psychotherapy. Am J Psychother 2021; 74: 83–88.

Attitudes of psychiatrists toward telepsychiatry: A policy Delphi study

Abstract

Objectives

Methods

Results

Conclusions

Keywords

Background

Methods

Questionnaire

Criteria for consensus

Procedure

Participants

Results

Consensuses and disagreements regarding telepsychiatry in the Delphi process

The effects of COVID-19-induced experience on the use of telepsychiatry

Workplace differences

Discussion

Consensus and dissensus in the Delphi process: Unveiling complex attitudes towards telepsychiatry

Consensus, disagreement, and the role of practical and pandemic-induced experience

Limitations

Conclusions

Footnotes

Acknowledgements

Data Availability

Declaration of conflicting interests

Ethical Approval

Funding

Gurantor

Informed Consent

ORCID iDs

Notes

Appendix

References