Incorporating telehealth into health service psychology training: A mixed-method study of student perspectives

Abstract

Objective

Telehealth is increasingly recognized as an avenue for enhancing psychologists’ capacities to meet the mental health needs of a diverse and underserved (due to barriers e.g., distance, transportation) public. The present study sought to inform training in telepsychology (i.e., telehealth delivery of psychological services) by using both quantitative and qualitative methods to explore the perspectives of doctoral students who have already been involved in such training.

Method

A total of 19 predoctoral students from two universities, with at least some experience in telepsychology training, provided their perspectives on two complementary research questions: (1) How do students perceive their level of competence in various domains of telepsychology?; and (2) What are students’ perspectives on the process of telepsychology competency development during their doctoral training?

Results

The results of our study provide early evidence that doctoral trainees are able to develop telepsychology competencies and suggest that a supportive, training-oriented environment and fit between telepsychology and existing programmatic areas of emphasis are likely key to success.

Conclusions

Continued efforts to enhance training in providing telepsychology services should focus on how to best define, measure, and promote competency development in this emerging specialty area.

Keywords

Telehealth telepsychology training competency health service psychology

Telehealth, which involves the use of technology to deliver health services at a distance,¹ is rapidly becoming a standard part of health care in many countries,² and its use has been accelerated as part of safety protocols for the ongoing COVID-19 pandemic.³ Now a nearly US $1 billion industry,⁴ telehealth has expanded to include all aspects of health care including primary care, specialty medical care, and mental/behavioral health.^5,6 The present study focused specifically on telepsychology, which is the use of telehealth technology to deliver psychological services.⁷ These psychological services include the provision of therapy, diagnostic interviewing and assessment, supervision, ethical decision-making, and respect for cultural diversity.^8,9

Research has consistently found that telepsychology has tremendous potential for improvements and innovations in care across a variety of clinical problems and demographic characteristics.^10–12 It offers an important avenue for making psychological services accessible to diverse populations who experience barriers (e.g., distance, transportation, work schedule) to in-person care, such as the 110 million Americans who live in mental health professional shortage areas¹³ and people whose work or family obligations preclude regular office visits during typical business hours. In fact, numerous experts have identified telepsychology as a critical area for future growth in the field.^14–16 This is further represented by an increasingly supportive policy environment, with emerging avenues that permit the practice of telepsychology across state lines without requiring licensure in each state.^17,18

The present study explored the experiences of individuals who could soon join the growing telepsychology work force: doctoral students in health service psychology (i.e., clinical, counseling, and school) programs. Given the rapid growth of the telepsychology market, some programs have begun preparing students to provide telepsychology services through dedicated training efforts.^19,20 Unfortunately, such training remains rare as a part of formal coursework or supervised clinical experiences.^21,22 Most psychologists are limited to the predoctoral internship, a one-year placement that serves as the capstone clinical experience of the doctoral student’s graduate program, or postdoctoral and post-licensure training in telepsychology. Furthermore, these trainings often rely on circumscribed didactics or continuing education programs (e.g., workshops, online courses) and lack expert supervision.²¹ Without greater attention to training in the clinical, ethical, and legal issues involved,²³ the number of psychologists who develop sufficient competence for independent practice in telepsychology will likely remain low – in turn limiting its tremendous potential impact.

Efforts to understand telepsychology training are best grounded in the broader understanding of competency development for telehealth, which has emerged over the past several decades. In this context, “competencies” refer to domains of knowledge, skills, values, and/or attitudes that must be used habitually and judiciously in effective professional practice.²⁴ Various health service fields such as medicine, dentistry, and nursing have each proposed sets of telehealth competencies; however, in order to provide a standard of care in the context of telehealth, common training and competencies are imperative. The skills and knowledge required for virtual interaction, compared to in-person care, can necessitate the addition or adjustment of professional behaviors. Such adjustments may include, for example, understanding and implementing changes for rapport building strategies, such as alternative behaviors for a handshake or increasing intentionality for providing eye contact through the video system. Numerous researchers have provided frameworks for defining telehealth care competencies.^25–28 For example, Hilty et al.²⁷ provided a summary and comparison of the telebehavioral competencies across professions, with nine domains: patient care, communication, systems-based practice, professionalism, practice-based learning, knowledge, technology, education/training, and other technologies (e.g., social media, apps, e-mail). Hilty et al. noted that various fields place more emphasis on different domains but that these categories provide a framework for understanding the implementation, measurement, and evaluation of competencies.

Articulation of specific telepsychology competencies is less advanced. Various guidelines for telepsychology are now available (see APA⁷) but each suggests a slightly different set of required competencies. As a step toward more unified and comprehensive guidance in this area, McCord and colleagues²⁹ synthesized all five available telepsychology guidelines into a Cube Model that identified 34 key foci of telepsychology training, organized into 9 practice domains, and their intersections with various modalities and settings of service delivery (all described in detail later). This model offers a useful descriptive framework for understanding and evaluating telepsychology training efforts. Now, there is a need for research that moves beyond identifying telepsychology training foci to understanding the processes by which training can be most successful. Such research will be especially useful for informing doctoral training efforts, but has implications for competency development in telepsychology at later stages of practice as well, given that many practicing psychologists never received training in telepsychology during their formal training years.

To that end, the present study explored factors that influence telepsychology training from the perspective of doctoral students who had already sought out such training. Students who are early adopters or “lead users,” such as those who are already learning to provide care via video-conference, often have valuable insights into a topic based on their high level of interest and motivation.³⁰ We collected and integrated data using mixed methods (i.e., quantitative and qualitative; see Palinkas et al.³¹) to answer two related, exploratory research questions: (1) How do students perceive their level of competence in various domains of telepsychology?; and (2) What are students’ perspectives on the process of telepsychology competency development during their doctoral training? More specifically, we took a QUAL + Quan approach involving simultaneous collection of qualitative and quantitative data for primarily qualitative, exploratory purposes (see Palinkas et al.³¹). Such research is meant to produce rich, detailed understanding of a phenomenon that can inform and/or complement more quantitatively focused research.^31–33

Method

Participants

Participants were 19 students who were enrolled in doctoral training programs in health service psychology at one of two universities with whom the authors were affiliated: the University of Arkansas or Texas A&M University. As only four U.S. doctoral programs are known to offer dedicated telepsychology training,¹⁹ this represents a considerable proportion of those programs. To be eligible for the study, students needed to have received some didactic and/or experiential training in telepsychology (i.e., they did not necessarily need to have delivered clinical services via telepsychology), be enrolled in their doctoral program, and not yet have started their predoctoral internship. Participants were 74% female and 26% male (ns = 14 and 5); identified as Non-Hispanic White/Caucasian (n = 12; 63%), Hispanic/Latinx (n = 4; 21%), or Asian/Pacific Islander (n = 3; 16%); and tended to be similar in age (M = 27.1; SD = 2.46). They were enrolled in PhD programs in clinical psychology (n = 10; 53%) or counseling psychology (n = 9; 47%). The majority (n = 10; 53%) of students had applied to predoctoral internship sites through the U.S. match system, which determines where they will complete the final phase of their doctoral clinical training, and all of those students who applied had matched at an internship training site. We do not provide additional details (e.g., breakdowns by training program) to protect participant confidentiality, given the small size of the sample.

Setting and training activities

There were important similarities and differences between the telepsychology training experiences offered at the two universities. The similarities included didactic training focused on telepsychology-related knowledge; on-site supervision, consultation, and technical assistance; and opportunities for supervised clinical service delivery using telepsychology modalities. Moreover, student trainees in both settings were enrolled in APA accredited doctoral programs with scientist-practitioner orientations.

The University of Arkansas is located within a predominantly rural service area. It offered telepsychology training in its standard clinical practicum for clinical psychology doctoral students. All activities took place in the program’s dedicated training clinic and were only available to students in that program. Prior to providing telepsychology services, students needed to complete didactic and general practicum (i.e., basic assessment and intervention skills) training requirements and obtain supervisor approval. Telepsychology services used a home-based delivery model, in which the client received services at their residence via synchronous video-conference. Individual and group supervision were conducted together for in-person and telepsychology services. The relationships between authors of this paper and the University of Arkansas are as follows: the first author was a supervising psychologist in the training clinic with primary oversight for telepsychology services (though other faculty also provided supervision of telepsychology cases and the first author was not a direct clinical or research supervisor for any student at the time of their participation) and the lead researcher for this study; the second and third authors were doctoral students in clinical psychology who provided administrative support during implementation of telepsychology services; and the fifth author was the clinic director.

In contrast, Texas A&M University was located in a predominantly urban area and offered a stand-alone, advanced training clinic that only provided telepsychology services. Practicum training at that clinic was available to students from all doctoral programs in health service psychology within the nearby geographic region of the university. Students applied and interviewed for placement at the clinic after having completed at least one semester of practicum training with their home institution. Admitted students received ongoing didactic training and individual/group supervision during the practicum. The telepsychology clinic primarily used a “hub-and-spoke” model, in which the client presented at a remote site (e.g., a community clinic or health center located in a rural area) where they received services via synchronous video-conference. However, clinicians also used telephone counseling when necessary (e.g., if the client was unable to travel to the remote site), which could be delivered anywhere in the state – including the client’s home. Supervision was provided in-person by the clinic director and a postdoctoral fellow, as well as via video-conference by off-site tele-supervisors (e.g., when a Spanish-speaking supervisor was needed for services provided in Spanish). The fourth author of this paper was the director of the clinic at Texas A&M.

Procedures

The Institutional Review Board of the University of Arkansas approved all recruitment and data collection procedures for the present study. For recruitment, the authors generated an initial list of students who were eligible based on their participation in practicum training experiences in telepsychology. The lead researcher sent email invitations to 31 eligible students. The recruitment emails emphasized that students’ participation was optional, that their decision to participate would not affect their standing in their programs, and that study data would not be used for student evaluations or otherwise shared with program faculty. Of those invited, 61% (n = 19) ultimately completed the study. Data collection lasted from March to June 2018.

Data collection consisted of a single session, conducted by the lead researcher either in-person (n = 13; 68%) or via a secure video-conferencing program (n = 6; 32%). During the research session, the participant first completed a verbal informed consent process that emphasized the same points noted above regarding participation. They then completed a demographic questionnaire and telepsychology competency rating scale via Qualtrics; and completed a qualitative interview with the lead researcher. The interviews (but not the remainder of the research sessions) were audio recorded for later transcription and analysis, and the researcher also took notes during interviews. Participants’ study materials were all identified using a unique, anonymous participant identification number to maximize confidentiality; only the first author had access to identifiable data (i.e., audio recordings) which were destroyed once they were no longer needed. Participants each received a $25 electronic gift card for their time.

Measures

Our use of mixed methods balanced breadth of information (quantitative measures) and depth of information (qualitative measures), allowing for a more comprehensive understanding of students’ experiences with telepsychology training than either method would provide alone. Under the taxonomy of mixed-method research proposed by Palinkas et al.,³¹ the present study simultaneously examined quantitative and qualitative data (QUAL + Quan), such that the data served complementary functions by elaborating on different aspects of the same topics.

Quan – Demographic questionnaire

On this questionnaire, participants provided the basic demographic characteristics reported previously and provided specific information about their previous experiences with telepsychology. First, participants rated their level of experience with telepsychology service delivery during their graduate training across various modalities (i.e., video-conference [including home-based vs. remote-clinic delivery], telephone, text-based, email, mobile application, and Web/internet-based) and settings (i.e., university counseling centers, hospitals, community clinics, private practice, schools, prisons and jails, military/Veterans Administration facilities) specified in the McCord et al.²⁹ Cube Model. Experience was rated as none, exposure only, minimal, moderate, or extensive (with behavioral anchors provided to guide ratings). The questionnaire also asked about hours of supervised experience with telepsychology (options were none, 1-50, 51-200, or >200), and the percentage of that experience involving services for children and adolescents. Finally, to consider the implications of telepsychology training for subsequent career stages, participants used a 5-point Likert scale (from Very Unlikely to Very Likely) to rate their likelihood of completing telepsychology training experiences during internship/postdoctoral training or use telepsychology in their future career.

Quan – Telepsychology Competency Rating Scale

Although some researchers continue to focus on defining competencies, others have highlighted the need for developing metrics to assess competencies.³⁴ There is no well-established rating scale for use with behavioral health providers to assess telehealth competencies. For this study, we developed a new 34-item, standardized rating scale for doctoral students to self-assess their telepsychology competencies, called the Telepsychology Competency Rating Scale (TCRS). A copy of the measure is available as Online Supplemental Material for this article. The TCRS was organized by the nine domains of McCord et al.’s Cube Model: service (6 items), supervision (1 item), technical skills (5 items), administrative skills (8 items), multicultural competence (2 items), research and evaluation (3 items), risk assessment (2 items), ethics and law (5 items), and assessment (2 items). For each item, the scale provided a name and brief description (all taken directly from the Cube Model), and then solicited a “general or average” rating of the student’s competence in that area (i.e., across all telepsychology training experiences). Ratings were completed using a 5-point Likert scale (0 = None, 1= Low, 2 = Moderate, 3 = High, 4 = Expert); we also provided behavioral anchors for each rating option (e.g., Moderate: “extensive experience and independence, but still requires regular supervision”). The lead researcher was available to answer participant questions during completion of the TCRS. As the TCRS was newly developed for this study, its psychometric quality is currently unknown, although we did conduct a preliminary examination of internal consistency.

Due to an error that occurred when transferring the TCRS into Qualtrics, the single item for supervision was inadvertently left out of data collection for the present study. Thus, we unfortunately did not collect participants’ ratings of that domain and therefore were unable to consider it when calculating average ratings of competence across all domains on the TCRS.

QUAL – Interview

We used a qualitative, semi-structured interview to elicit information about participants’ experiences with telepsychology training. The interview began by asking the participant what their experience has been like learning about and (if applicable) delivering telepsychology services. Next, a series of questions requested specific information regarding what was most helpful and most challenging in developing competency for each of the nine competency domains from the TCRS, as well as overall. A list of the domains, including relevant training foci (with definitions) as listed in the TCRS, was available to aid the participant in reflecting on each domain. Finally, the 10 students who had applied to internships were asked how their telepsychology training related to that next stage of their training and competency development. Interviews lasted an average of 59 min (SD = 11.25).

Analytic strategy

Quantitative data analysis

We calculated descriptive statistics (e.g., Ms, SDs, frequencies, ranges) for items on the demographic questionnaire and TCRS. Our small sample size precluded the use of more complex inferential statistics (e.g., testing for differences in TCRS ratings across domains or between subgroups). For the TCRS, we calculated average ratings for the items in each domain and an overall average rating across domains. The scale had high internal consistency (α = .946), although this is very preliminary evidence for reliability given the small sample and aforementioned missing item. Internal consistencies for most of the domain subscales of the TCRS also seemed acceptable (αs = .769– .921), except for two of the two-item subscales: multicultural competence (α = .491) and assessment (α = .603).

Rapid coding of qualitative data

Immediately following the completion of each qualitative interview, the lead researcher completed a set of brief, focused codes using a Rapid Evaluation and Assessment Method.³⁵ These codes summarized each participant’s perspective on three topics related to developing telepsychology competencies: what was most helpful, what was most challenging, and (when applicable) the relevance for internship. The lead researcher created three to five summary codes for each topic based on each interview, and organized those codes by participant within a de-identified matrix. As much as possible, the lead author recorded codes in participants’ own words to maximize validity and help to identify exemplar quotes.

Once all interviews were complete, we used an iterative and collaborative process to extract general themes summarizing the content of responses across participants (i.e., conventional content analysis)³⁶ from the matrix of codes. Initially, the doctoral student co-authors independently reviewed the matrix of codes and each generated a preliminary set of themes. When identifying a theme, they considered its cohesiveness and prevalence across participant responses, but also incorporated quotes and perspectives that were inconsistent (i.e., negative case analysis). They then combined their preliminary themes into a single set of themes through additional discussion and consensus. They also consulted with the lead researcher for feedback as needed during the analytic process. It should be noted that the coders explored for the presence of different themes across subgroups of training sites (e.g., model of telepsychology services, rural vs. urban service area) and trainees (e.g., amount of telepsychology experience), but found remarkable consistency in trainees’ perspectives and therefore focused on themes for the entire sample. The three authors agreed that the interviews had reached saturation (i.e., additional interviews were no longer resulting in meaningful incremental changes to the themes) and, therefore, further data collection was unnecessary. The two clinic director co-authors reviewed the initial set of themes and provided feedback, which the other authors addressed through additional refinements to produce a final set of themes. Once we had finalized themes, the lead author selected an exemplar quote to illustrate each theme and transcribed it verbatim from the audio recordings; the remaining authors reviewed and agreed to all selected quotes.

Results

Telepsychology training experiences

Participants reported varied amounts of supervised experience with telepsychology. Over half (53%) reported 50 hours or less, with one reporting zero; six (32%) reported 51-200 hours; and three (16%) reported over 200 hours (M = 317 hr). The majority of students’ hours of experience were typically devoted to work with adults. On average, 10.5% of hours were with children and adolescents, though this was highly variable (SD = 23.22%; range = 0–80%). Over half (n = 11; 58%) of participants reported no telepsychology experience with youth.

In terms of the types of telepsychology training, Table 1 presents the proportion of the sample reporting a given level of experience, by modality and setting. Among modalities, participants reported the most experience with video-conferencing (remote clinic delivery), telephone/audio call, and video-conferencing (home-based delivery). The proportions reporting moderate or extensive experience were 64%, 63%, and 26%, respectively. When considering settings, the most prominent was community clinics: 53% of the sample endorsed moderate or extensive experience there. Counseling centers were the next most common with 37% reporting some (i.e., minimal or greater) experience. Most participants had no experience providing telepsychology in the other modalities or settings.

Table 1.

Trainees’ level of experience with telepsychology training modalities and settings.

Type of experience	n (%) by level of experience
	None	Exposure	Minimal	Moderate	Extensive
By modality
Video-conferencing – home-based delivery	8 (42%)	3 (16%)	3 (16%)	5 (26%)	0 (0%)
Video-conferencing – remote clinic delivery	6 (32%)	1 (5%)	0 (0%)	6 (32%)	6 (32%)
Telephone/audio call	4 (21%)	0 (0%)	3 (16%)	8 (42%)	4 (21%)
Text message-based	16 (84%)	2 (11%)	0 (0%)	1 (5%)	0 (0%)
Email	14 (74%)	0 (0%)	2 (11%)	3 (16%)	0 (0%)
Mobile application	13 (68%)	1 (5%)	4 (21%)	1 (5%)	0 (0%)
Web/Internet-based	13 (68%)	0 (0%)	4 (21%)	1 (5%)	1 (5%)
By Setting
University counseling centers	10 (53%)	2 (11%)	3 (16%)	3 (16%)	1 (5%)
Hospitals	18 (95%)	1 (5%)	0 (0%)	0 (0%)	0 (0%)
Community clinics	4 (21%)	2 (11%)	3 (16%)	4 (21%)	6 (32%)
Private practice	16 (84%)	0 (0%)	1 (5%)	2 (11%)	0 (0%)
Schools	17 (89%)	0 (0%)	2 (11%)	0 (0%)	0 (0%)
Prisons and jails	16 (84%)	1 (5%)	1 (5%)	1 (5%)	0 (0%)
Military/Veterans Administration	17 (89%)	1 (5%)	1 (5%)	0 (0%)	0 (0%)

Regarding the likelihood of using their telepsychology training in internship and postdoctoral training, the average participant rating was 3.1 out of 5 (in the neither likely nor unlikely range). A considerable minority of students (n = 6; 32%) thought that this was somewhat or very unlikely. However, ratings for the likelihood of using their telepsychology training in their future career averaged a much higher 4.1 (in the somewhat likely range). Five participants (32%) rated this item as very likely; none rated it as somewhat or very unlikely.

Self-rated telepsychology competencies

Table 2 summarizes the participants’ ratings of their telepsychology competencies on the TCRS. The highest average rating was for multicultural competence (M = 2.4); that and four other domains (service, research and evaluation, risk assessment, ethics and law) averaged in the moderate range. The average overall competence score on the TCRS was in the same range (M = 2.0). The lowest average rating was for administrative skills (M = 1.2); that and two other domains (technical skills, assessment) averaged in the low range.

Table 2.

Self-rated telepsychology competencies, overall and by domain.

Domain	# of items	Ratings^a		Competencies included
Domain	# of items	M	SD	Competencies included
Service	6	2.3	0.61	Client appropriateness, informed consent, professional boundaries and communication, handle outages/downtime, be competent to provide the service, termination
Technical Skills	5	1.8	0.75	Fluency in technology, client communication, equipment use, privacy and confidentiality, technology disruption/outage
Administrative Skills	8	1.2	0.59	Verification of identity and location, privacy and confidentiality, record keeping, billing, collaborative partnership agreements, service evaluation and quality improvement, organization information, insurance/coverage
Multicultural Competence	2	2.4	0.49	Foundational multicultural competencies, special multicultural considerations for telehealth
Research and Evaluation	3	2.2	1.12	Have research/evaluation protocols, informed consent, information security of data
Risk Assessment	2	2.1	0.74	Knowledge of local resources, emergency planning
Ethics and Law	5	2.0	0.73	Refer to and enact relevant ethical codes, practice according to guidelines, assess remote environment, effectiveness of treatment, refer to and enact relevant laws
Assessment	2	1.6	0.96	Selection of assessment tools, protection of data and materials
Overall	33^b	2.0	0.57	all of the above^b

0 = None, 1 = Low, 2 = Moderate, 3 = High, 4 = Expert.

Does not include the single rating item for tele-supervision that was inadvertently left out of the survey.

Trainee perspectives on telepsychology competency development

We identified 12 qualitative themes from the summary matrix of participant responses, including five helpful aspects of training in telepsychology; five challenges in training; and two themes about internship. These themes are summarized in Table 3, which also provides an illustrative quote for each theme, and we describe each theme in detail next. Within each major topic area, themes are reported from most to least impactful (based on our assessment of their salience, prevalence, etc.) in the text and table. We refer to participants as “students” throughout this section to emphasize their role and context.

Table 3.

Themes about telepsychology competency development from trainees’ qualitative interviews.

Topic	Theme	Exemplar quote
Most helpful aspects of training	Supervisory and environmental support	“What’s been most helpful for me is consultation with peers as well as supervisors and staff… whenever there’s been challenges, I’ve been able to reach out [and get answers].”
	Learning by experience	“I think that [learning] just kind of comes through experience, trial and error with clients…I’ve seen a number of technical issues that I’ve had to kind of figure out on the fly…and just simply becoming more comfortable and confident in myself.”
	Congruence with training program goals and values	“Just about analyzing barriers and improve access to care…the culture with my training more generally has set me up to think about those things no matter what…I at least know where the importance is coming from.”
	Expansion of existing competencies	“There was a little bit of additional training for delivering telepsych, [but I was surprised that what] I was already doing in the clinic translated pretty well into telepsych.”
	Integration of research and practice	“This is a data-driven clinic, people do care about outcomes…that is beneficial because the clients understand that we value their input, and I’d say from there, it does help all parties.”
Most challenging aspects of training	Use of technology	“The technology itself isn’t perfect yet…in person, all I need is two chairs and a room, but you’re introducing more factors here [and] the more possibilities there are for failure, right?”
	New and rapidly changing area of practice	“Because it's a new area, that so much of it requires us to kind of brainstorm and find solutions… I think someday, hopefully, the guidelines and the ethics will all be more ironed out.”
	Domain-specific lack of exposure	“How to make technology work…I realize that I don’t know all those steps cause they have been dealt with by other folks…my knowledge definitely ends at a certain point.”
	Overextension of cultural competence	“We’re serving a greater area, every rural community is different…we talk to the community members so that we understand the values and culture more.”
	Tension between accessibility and appropriateness of care	“I think client appropriateness is, that’s sometimes a challenge in itself, especially if you’re considering the fact that there are no other resources…They might not be the best fit, but it’s like, are you going to just deny them services altogether?”
Relevance of training to internship	Influenced their training goals and interests	“I think it’s been career-altering if I can describe it that way… I’ve been learning so much about the potential that it has and, uh, the impact that it can have on my career moving forward.”
Relevance of training to internship	Sites viewed their experiences withtelepsychology favorably	“Having this experience is a huge asset. It provides a really unique training opportunity…really unique in terms of client population that you serve and service delivery. So I think it really kind of made you stand out.”

Most helpful aspects of training

Supervisory and environmental support

Breadth and depth in supervisory expertise, within a structured and well-resourced training environment, led to supportive climates in which trainees felt comfortable working on their knowledge and skills. Students also received more informal consultation and support from their peers. Students noted the importance of policies and procedures that were clear and specific, informed by student input, and disseminated in multiple ways (e.g., didactic training, clinic newsletter). They also appreciated the availability of program administrative staff who served as points of contact and helped solve challenges (related to technology and interpreting policies, for example). Despite the largely positive feedback about program supports, some students found that supervisors in their program were variable in their level of telepsychology knowledge. Also, a few students described community-based training sites as providing little telepsychology-specific training or support.

Learning by experience

Students perceived that they were able to learn firsthand through repeated practice – and that their supervisors encouraged them to approach the learning process in flexible, creative, and exploratory ways. Students said that this learning orientation helped them feel comfortable and supported (and, in turn, more open and willing to be vulnerable) during the learning process. Even students who had limited experience with telepsychology, such as working with a single case or indirect exposure (e.g., in group supervision), identified that experience as contributing to their initial competency development. Importantly, however, those same students still acknowledged their limited exposure as a barrier to competency development.

Congruence with training program goals and values

Students’ doctoral programs placed considerable importance on training students to work with diverse, underserved populations and on increasing access to mental health care. Thus, students found that their general clinical training backgrounds prepared them well for telepsychology; for example, they already had strong backgrounds in relevant multicultural, ethical/legal, and risk assessment domains. Students also described how their own values were aligned with their programs’ and, thus, they were a good fit for telepsychology on a personal level.

Expansion of existing competencies

Students described much of telepsychology as building on competencies for in-person service delivery, rather than representing brand-new learning. Many described being surprised to find that telepsychology services were “really similar” to in-person services in most ways, and thus they were able to build on foundational knowledge and skills from a variety of training areas (e.g., service delivery, administrative skills, assessment, research). Students described these competencies as common to doctoral training in psychology generally, though there some were program-specific (see “congruence with training program”). Furthermore, students noted that telepsychology training expanded their general competencies in important ways, such as using existing risk assessment skills over telephone or video-conference, as well as leading them to develop new competencies, such as technical skills.

Integration of research and practice

Several students stated appreciation for the central importance of data collection and analysis in their “data-driven” training clinic. A few also described how their telepsychology training influenced their later research interests or vice-versa.

Most challenging aspects of training

Use of technology

Many students noted that using technology during service delivery would solve some issues, such as a client’s lack of transportation, but also “adds a layer” of new issues. Examples included unfamiliarity or difficulty with the technology (i.e., lack of technical fluency among clinicians or clients); technological failures (e.g., loss of connectivity); and limited capabilities of the technology used (e.g., remote sites needing to collect paper forms from clients and fax them to the clinic). Despite the issues introduced by technology, students agreed that the benefits of telepsychology outweighed those challenges. Moreover, several students (especially one who said they “love technology”) noted that the prevalence of technology in daily life made telepsychology more relevant to them and their clients; they described use of technology as helpful, not challenging.

New and rapidly changing area of practice

Students discussed how even telepsychology experts, such as supervisors, are typically still learning the best ways to deliver services and provide training in this area. Research to inform telepsychology is also limited, and often unfamiliar or difficult to locate (e.g., published outside of mainstream psychology journals). Moreover, students noted how critical aspects of telepsychology – such as the technology used or the legal and policy contexts in which services are delivered – changed frequently, causing their knowledge to quickly become obsolete. Given all of these factors, students often felt unsure about how to proceed with telepsychology services and required frequent supervision and consultation. They reported the need for patience, creativity, and flexibility while developing competencies in telepsychology.

Domain-specific lack of exposure

A significant minority of students described how they had not pursued intensive exposure in certain domains due to lack of interest, particularly in the technical skills, administrative skills, and research/evaluation domains. Among the students who were interested in developing those domains, they also noted limited opportunities for exposure due to other individuals holding primary responsibility (e.g., service coordinators or Information Technology staff addressed serious technical issues; office administrative staff billed clients). That division of labor did allow students to focus on their clinical role, which they acknowledged as helpful for training. It is notable that students who worked in both service coordinator and clinician roles did not describe the same lack of exposure. Also, all students had limited exposure to domains that were less relevant to their settings (e.g., neither clinic accepted insurance).

Overextension of cultural competence

Some students expressed how telepsychology allowed them to engage more diverse individuals in services, and thus required the ability to effectively serve those diverse individuals. Given the large catchment areas served by the participating clinics, students were sometimes unfamiliar with their clients’ communities and cultural backgrounds, despite strong generalist training in multicultural competence. There is also the need to keep up with locally available resources in clients’ communities. In addition, students noted the limited availability of bilingual supervisors as a barrier, with services in Spanish remaining limited despite a high prevalence of Spanish-speaking clients.

Tension between accessibility and appropriateness of care

Students reported finding it difficult to make decisions about whether to initiate services with a client who was not a good fit for outpatient telepsychology services but also had no alternatives available in their community. Examples of factors contributing to a poor fit between the client and services included level of risk, severity of psychopathology, or the need for specific types of assessment that were not available for telepsychology administration. Related to the previous challenge on cultural competence, they noted how efforts to engage members of underserved communities in services are more likely to result in such dilemmas.

Relevance of training to internship

These themes summarize a subset of students’ (n = 10) views on the relevance of telepsychology training to predoctoral psychology internships.

Influenced their training goals and interests for internship

Exposure to telepsychology often shifted or focused students’ long-term career interests, which were reflected in the criteria they used to search for and evaluate the fit of internship sites. Settings for which they saw telepsychology as relevant included correctional/forensic sites (especially those that included community services) and a site offering Dialectical Behavior Therapy with phone coaching. Some students made additional telepsychology experience one of their highest priorities for internship training, as they wanted to either make telepsychology a central focus of their career or expand on the training received in their doctoral programs. Others focused on different training goals (e.g., ability to work with underserved populations) but still considered telepsychology to be a valuable training experience that was consistent with those training goals. It is worth noting that some students found it challenging to identify telepsychology-focused internships, or did not pursue such internships because they perceived their level of experience and competence as too limited to be competitive.

Internship sites viewed their experiences with telepsychology favorably

Students received indications of this theme directly from internship faculty, most often during interviews. Generally, students perceived that internship sites considered telepsychology training to be a unique, valuable experience that increased the student’s readiness for internship. Students reported sites indicated to them that such experience was a “huge asset,” “exciting,” and “very relevant.” This was true even of sites with no telepsychology training for interns, as the students thought that sites viewed telepsychology training as an indicator of other desirable student qualities (e.g., multicultural orientation, openness to different training experiences) related to readiness for internship. Even students with very minimal exposure to telepsychology reported receiving similarly positive reactions. However, students did note that telepsychology was not emphasized at some sites and that they had to communicate the value of their training to some internship faculty who were less familiar with telepsychology.

Discussion

In order to meet the growing need for a telehealth workforce,^14–16 doctoral programs in health service psychology must expand upon the currently limited training available in telehealth.^21,22 The need for expanded training has increased substantially this year, as telepsychology has been central to the mental health services response to the COVID-19 global pandemic.³⁷ The present study sought to inform such expansion efforts by capturing the perspectives of clinical psychology doctoral students regarding two complementary topics, using a QUAL + Quan exploratory approach³¹ to produce rich understanding of telepsychology training experiences.^32,33 First, participants tended to rate their telepsychology competencies as Moderate overall (M = 2.0) for the majority of domains (66.67%). Second, when describing the process of telepsychology competency development during their doctoral training, participants identified learning by experience, congruence with training program goals and values, and expansion of existing competencies as keys to their success; they also described high levels of interest, and success, in pursuing telepsychology-related experience in the next phase of their training (i.e., the predoctoral internship). The study had several methodological strengths, including (a) inclusion of “early adopter”³⁰ students, who offer a unique depth of insight into telepsychology; (b) collection of complementary mixed measures (QUAL + Quan; see Palinkas et al.)³¹ that allowed for elaboration on related topics from different perspectives; and (c) grounding of the study within the emerging literature on development of telepsychology practice domains (i.e., Cube Model; McCord et al.).²⁹

Despite the exploratory nature of this study, doctoral programs interested in developing students’ competencies in telepsychology, and telehealth more broadly, may find our results informative in their efforts to implement specific telehealth training experiences, either in-house or at external training sites. Our results suggest that telehealth training can be successful at the doctoral level, where moderate levels of competence are likely sufficient for a specialty area of practice (i.e., not expected of all program graduates).³⁸ It will be important for training programs to consider their ability to address areas identified as most helpful (e.g., “supervisory and environmental support,” “congruence with training program goals and values”) and challenging (e.g., “new and rapidly changing area of practice”) when deciding whether telehealth training fits well into their program. Dopp and colleagues¹⁹ describe a useful process for implementing telepsychology services in a doctoral training clinic and offer guidance on practical issues (e.g., development of policies and procedures, effective use of consultation, selection of equipment, ensuring HIPAA compliance) from an administrative perspective; while the article focused on telepsychology services, much of the guidance applies to implementation of telehealth training generally. Many doctoral training clinics in health services psychology programs recently began offering telepsychology services in response to COVID-19,^37,39 but additional efforts are likely necessary for those programs to develop associated training practices. Thus, the findings from this study could help guide the development of formalized telepsychology training efforts around the services that doctoral programs rapidly implemented in response to this global health crisis.

Programs that wish to provide focused training in telehealth, and telepsychology especially, should draw on existing literature for competency development, given that participants described the development of telepsychology competencies as sharing many characteristics with high-quality training more broadly. For example, psychology programs might consider the development of a variety of assessment tools (e.g., direct observation, written tests, role plays, self-assessment) that can triangulate evaluation of knowledge, skills, and attitudes involved in telehealth competencies, as with core/foundational competency assessment.⁸ It will also be necessary to establish expected levels for various telepsychology competencies (e.g., “minimum levels of achievement” in the language of the APA Commission on Accreditation),⁴⁰ especially those that are more unique to the specialty (e.g., technical skills). The TCRS has the potential to develop into one such self-assessment tool, but it is currently limited by unclear psychometric quality and a lack of detailed norms for different modalities and settings (e.g., our participants might have higher competencies for video-conferencing and telephone counseling in general outpatient settings, the areas in which they had the most experience) and different stages of doctoral training. The tendency for self-assessments of competence to be inflated also suggests that the TCRS should not be used in isolation.⁴¹ Literature on telehealth competency assessment could help inform improvement of the TCRS, although such research remains rare. In a study by Goran,⁴² the author examined competency assessment of tele-ICU nurses and found the most common method used was performance appraisal and observation, although peer review and self-assessment also were used. For behavioral telehealth, there is even less research on the assessment of competencies, but existing studies have used therapist fidelity measures (e.g., Gellis, Kenaley, & Have)⁴³ or video vignettes and self-report surveys (i.e., Giffords, Niles, Rivkin, Koverola, & Polaha).⁴⁴

Our results also suggest that, as in many specialties, doctoral training in telepsychology can usefully prepare students to pursue more advanced training opportunities. Across qualitative and quantitative (i.e., demographic) data, students described positive experiences with, and interest in, applying their telepsychology training during subsequent career stages (e.g., predoctoral internship, postdoctoral fellowship). Continued development in telepsychology is likely a necessary and desirable outcome of doctoral-level exposure to this specialty. Additional training experiences should be developed at the predoctoral internship and postdoctoral fellowship levels, as well as for mid/late-career psychologists wishing to respecialize. It would also be useful for that research to examine the specific settings (e.g., VA, prisons) where telepsychology training and practice are most applicable. Moreover, it will be useful to examine patterns of telehealth training in other health service professions outside psychology, as trainees in those professions could similarly benefit from telehealth experiences that are tailored to different career stages and settings.

Our findings generally support the benefits of doctoral-level training in telepsychology, yet they also suggest critical remaining challenges that also merit consideration for telehealth training more broadly. In particular, our participants tended to rate their competency as “low” in three domains on the TCRS – technical skills, administrative skills, and assessment – and described in the qualitative interviews how the “use of technology” introduced new problems and they had a “lack of domain-specific exposure.” The achievement of well-rounded telepsychology training may require doctoral students to serve in multiple roles with various emphases on clinical, administrative, and technical skills. Furthermore, it is important to keep in mind that some students appreciated the narrower focus on service-related competencies in their telepsychology training; it may not be necessary, or practical, for all telepsychology trainees to develop equivalent competency in all possible domains. These remaining questions illustrate how exploratory, primarily qualitative findings can inform future research; for example, some students described how the use of technology in their personal life was beneficial for developing technical competencies, which could inform efforts to develop (and test) predictive models of the characteristics of successful telepsychology trainees.

In contrast, addressing the lack of exposure to psychological assessment presents a different challenge due to the limited number of assessment tools that can be reliably and validly administered via telehealth.⁴⁵ Fortunately, computerized versions of widely used assessments, such as the Weschler Adult Intelligence Scale, are currently in development and show considerable initial promise for remote administration.⁴⁶ Tele-assessment thus provides an excellent example of telepsychology being a “new and rapidly changing area of practice.” Faculty supervisors or those in leadership roles, such as training directors and clinic directors, should strive to stay abreast of such developments in telepsychology so that the latest science can be incorporated into student training. At the systems level, professional organizations and state and federal funding agencies can invest in keeping up-to-date, research-based recommendations available for clinicians and training programs, and can serve a vital role in advancing competent and ethical telehealth service delivery.

Limitations

The present results should be interpreted in the context of several limitations. First, given that this study focused on depth of understanding,^32,33 our small sample from two training sites limits the generalizability of the findings. It would also be useful for future research to examine complementary, more quantitative issues around prediction and broad generalizability, for example by collecting data on telepsychology competencies from a large, diverse sample of trainees in health service psychology programs. Caution must also be taken when generalizing these findings beyond the place (we focused on programs in the United States) and time (we conducted this study prior to the COVID-19 pandemic) in which they were collected. Second, we may have reached different conclusions if we had collected information from others involved in telepsychology training efforts, such as supervisors, clinic directors, internship faculty, or the clients served by trainees. The research team represented some of these alternative perspectives, but we could not incorporate our own viewpoints into the analysis (e.g., during qualitative coding) without jeopardizing the validity of our findings. It will be important for future research to consider these different perspectives, as well as trends and factors influencing the attitudes of training faculty and clients toward telepsychology. Moreover, the disparity between the roles of the participants (primarily clinician roles) and research team (primarily administrative roles) may have limited our understanding of the participants’ perspectives, though we ameliorated this concern somewhat by including student coders and co-authors.

Third, doctoral students in health service psychology programs who are not “early adopters” might have different training needs (e.g., more didactic education about telepsychology) and challenges (e.g., lower motivation to develop telepsychology competencies). It is also possible that some students from the two universities in the study had negative experiences with telepsychology training, but declined our invitation to participate in this study. Fourth, our rapid coding procedure limited the depth of information obtained from qualitative interviews for this initial analysis, and limited the initial codes to a single researcher’s perspective (i.e., we could not examine interrater reliability or have participants engage in member-checking review of findings). In the future, we plan to conduct in-depth content analysis of the themes discussed by participants for each domain of telepsychology competence and to more fully explore subgroup differences. Fifth, the results of the TCRS should be interpreted with caution until additional research is able to examine the questionnaire’s psychometric strength, beyond the preliminary internal consistency examined in this study. Sixth, we were unable to make firm conclusions about tele-supervision due to our inadvertent exclusion of the relevant item from the TCRS. Research specifically focused on tele-supervision models and techniques will be important in the future because that is another rapidly changing area of telepsychology practice.^47,48 Finally, although we took extensive efforts to avoid bias through our recruitment and consent procedures, it will be important for independent researchers (i.e., not supervisors or clinic directors in the training programs being studied) to verify our findings in the absence of potential demand characteristics.

Conclusions

In conclusion, the results of our study provide early evidence about the conditions under which telepsychology training can be feasible at the doctoral level. A supportive, training-oriented environment and fit between telepsychology and existing programmatic areas of emphasis will likely be key to success. We hope that a growing number of programs will heed our call to provide such training. It will also be important to consider how supportive learning environments can be created for internship and postdoctoral training, as well as mid/late-career psychologists who seek to develop telepsychology competencies outside of a formal training environment. Moreover, continued research is needed to better understand how we can best define, measure, and promote telepsychology competency development. Finally, it is worth noting that understanding of telepsychology training should also advance efforts to articulate general competencies in telebehavioral health across a variety of professions (see e.g., Hilty et al.),²⁷ thus contributing to the development of a diverse and well-rounded telebehavioral health workforce as well as a clearly defined role for psychologists within that workforce.

Supplemental Material

sj-pdf-1-dhj-10.1177_2055207620980222 - Supplemental material for Incorporating telehealth into health service psychology training: A mixed-method study of student perspectives

Supplemental material, sj-pdf-1-dhj-10.1177_2055207620980222 for Incorporating telehealth into health service psychology training: A mixed-method study of student perspectives by Alex R Dopp, Ayla R Mapes, Noah R Wolkowicz, Carly E McCord and Matthew T Feldner in Digital Health

Footnotes

Contributorship

AD designed the study, with input from all other authors, and AD collected all study data. AD, AM, and NW conducted data analyses. CM and MF contributed to interpretation of findings. AD wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

The ethics committee of the University of Arkansas Institutional Review Board approved this study (protocol #2018-0102).

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This publication is supported by the Health Resources and Services Administration (HRSA) of the United States Department of Health and Human Services (HHS) as part of an award totaling $975,000 with no percentage financed with non-governmental sources (PI: McCord; grant number G01RH32158). The contents are those of the author(s) and do not necessarily represent the official views of, nor an endorsement, by HRSA, HHS, or the U.S. Government. This project was also supported by the Texas Child Mental Health Care Consortium funded by the Texas Legislature, and by faculty startup funds from the University of Arkansas Office of Research and Innovation, awarded to the first author (Dopp). The first author (Dopp) is an investigator with the Implementation Research Institute (IRI), at the George Warren Brown School of Social Work, Washington University in St. Louis; through an award from the National Institutes of Mental Health (5R25MH08091607) and the Department of Veterans Affairs, Health Services Research & Development Service, Quality Enhancement Research Initiative (QUERI).

Guarantor

AD.

ORCID iD

Alex R Dopp

Peer review

Dr Anna Chur-Hansen, The University of Adelaide - North Terrace Campus has reviewed this manuscript.

Supplemental material

Supplemental material for this article is available online.

References

American Telemedicine Association. Telemedicine practice guidelines, http://thesource.americantelemed.org/resources/telemedicine-practice-guidelines (2012, accessed 4 March 2020).

Dorsey

Topol

EJ.

Telemedicine 2020 and the next decade. Lancet 2020; 395: 859.

Smith

Thomas

Snoswell

, et al. Telehealth for global emergencies: implications for coronavirus disease 2019 (COVID-19). J Telemed Telecare 2020; 26: 309–313.

Wicklund

Report: telehealth will be a billion-dollar industry by 2018. mHealth Intelligence, https://mhealthintelligence.com/ (2016, accessed 2 December 2020).

Myers

Nelson

E-L

Rabinowitz

, et al. American telemedicine association practice guidelines for telemental health with children and adolescents. Telemed e-Health 2017; 23: 779–804.

Turvey

Coleman

Dennison

, et al. ATA practice guidelines for video-based online mental health services. Telemed e-Health 2013; 19: 722–731.

American Psychological Association. Guidelines for the practice of telepsychology. Am Psychol 2013; 2013: 68: 791–800.

Kaslow

Grus

Campbell

, et al. Competency assessment toolkit for professional psychology. Train Educ Prof Psychol 2009; 3: S27–S45.

Rubin

Bebeau

Leigh

, et al. The competency movement within psychology: an historical perspective. Prof Psychol Res Pract 2007; 38: 452–462.

10.

Hilty

Ferrer

Parish

, et al. The effectiveness of telemental health: a 2013 review. Telemed J E Health 2013; 19: 444–454.

11.

Rauch

SAM

Rothbaum

BO.

Innovations in exposure therapy for PTSD treatment. Pract Innov 2016; 1: 189–196.

12.

Slone

Reese

McClellan

MJ.

Telepsychology outcome research with children and adolescents: a review of the literature. Psychol Serv 2012; 9: 272–292.

13.

U.S. Department of Health and Human Services. Shortage Areas, https://data.hrsa.gov/topics/health-workforce/shortage-areas (2018, 4 accessed March 2020).

14.

Comer

Myers

Future directions in the use of telemental health to improve the accessibility and quality of children’s mental health services. J Child Adolesc Psychopharmacol 2016; 26: 296–300.

15.

Jones

DJ.

Future directions in the design, development, and investigation of technology as a service delivery vehicle. J Clin Child Adolesc Psychol 2014; 43: 128–142.

16.

Norcross

Pfund

Prochaska

JO.

Psychotherapy in 2022: a Delphi poll on its future. Prof Psychol Res Pract 2013; 44: 363–370.

17.

Association of State and Provincial Psychology Boards. PSYPACT: advancing the interjurisdictional practice of psychology: legislative updates, www.asppb.net/mpage/legislative (2019, accessed 27 October 2019).

18.

U.S. Department of Veterans Affairs. VA expands telehealth by allowing health care providers to treat patients across state lines, www.va.gov/opa/pressrel/pressrelease.cfm?id=4054 (2019, accessed 27 October 2019).

19.

Dopp

Wolkowicz

Mapes

, et al. Implementation of telepsychology services in a university training clinic: process and lessons learned. Behav Ther 2017; 40: 261–267.

20.

McCord

Saenz

Armstrong

, et al. Training the next generation of counseling psychologists in the practice of telepsychology. Couns Psychol Q 2015; 28: 324–344.

21.

Colbow

AJ.

Looking to the future: integrating telemental health therapy into psychologist training. Train Educ Prof Psychol 2013; 7: 155–165.

22.

Lustgarten

Colbow

AJ.

Ethical concerns for telemental health therapy amidst governmental surveillance. Am Psychol 2017; 72: 159–170.

23.

Barnett

Kolmes

The practice of tele-mental health: ethical, legal, and clinical issues for practitioners. Pract Innov 2016; 1: 53–66.

24.

Kaslow

NJ.

Competencies in professional psychology. Am Psychol 2004; 59: 774–781.

25.

Englander

Cameron

Ballard

, et al. Toward a common taxonomy of competency domains for the health professions and competencies for physicians. Acad Med 2013; 88: 1088–1094.

26.

Fisk

Necessary skills and knowledge for staff providing telehealth services. In: Jordanova

Lievens

(eds) Global telemedicine and eHealth updates: knowledge resources. Vol. 7, pp. 103–106. Switzerland: International Society for Telemedicine and eHealth, 2014.

27.

Hilty

Maheu

Drude

, et al. The need to implement and evaluate telehealth competency frameworks to ensure quality care across behavioral health professions. Acad Psychiatry 2018; 42: 818–824.

28.

Sharma

Nachum

Davidson

, et al. It’s not just FaceTime: core competencies for the medical virtualist. Int J Emerg Med 2019; 12: 1–5.

29.

McCord

Bernhard

Walsh

, et al. A consolidated model for telepsychology practice. J Clin Psychol 2020; 76: 1060–1082.

30.

Von Hippel

Lead users: a source of novel product concepts. Res Technol Manage 1989; 32: 773–907.

31.

Palinkas

Horwitz

Chamberlain

, et al. Mixed-methods designs in mental health services research: a review. Psychiatr Serv 2011; 62: 255–263.

32.

Braun

Clarke

Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77–101.

33.

Camic

Rhodes

Yardey

Qualitative research in psychology: expanding perspectives in methodology and design. Washington, DC: APA Books, 2003.

34.

Maheu

Drude

Hertlein

, et al. A framework of interprofessional telebehavioral health competencies: Implementation and challenges moving forward. Acad Psychiatry 2018; 42: 825–833.

35.

McNall

Foster-Fishman

PG.

Methods of rapid evaluation, assessment, and appraisal. Am J Eval 2007; 28: 151–168.

36.

Hsieh

H-F

Shannon

SE.

Three approaches to qualitative content analysis. Qual Health Res 2005; 15: 1277–1288.

37.

Pierce

Perrin

Tyler

, et al. The COVID-19 telepsychology revolution: a national study of pandemic-based changes in U.S. mental health care delivery. Am Psychol. Epub ahead of print 2020. DOI:10.1037/amp0000722

38.

Rodolfa

Bent

Eisman

, et al. A cube model for competency development: Implications for psychology educators and regulators. Prof Psychol Res Pract 2005; 36: 347–354.

39.

Perrin

Rybarczyk

Pierce

, et al. Rapid telepsychology deployment during the COVID-19 pandemic: a special issue commentary and lessons from primary care psychology training. J Clin Psychol 2020; 76: 1173–1185.

40.

American Psychological Association, Commission on Accreditation. Standards of accreditation for health service psychology, www.apa.org/ed/accreditation/about/policies/standards-of-accreditation.pdf (accessed 27 October 2019).

41.

Dunning

Heath

Suls

JM.

Flawed self-assessment: Implications for health, education, and the workplace. Psychol Sci Public Interest 2004; 5: 69–106.

42.

Goran

SF.

A new view: tele-intensive care unit competencies. Crit Care Nurse 2011; 31: 17–29.

43.

Gellis

Kenaley

Have

TT.

Integrated telehealth care for chronic illness and depression in geriatric home care patients: the integrated telehealth education and activation of mood (I-TEAM) study. J Am Geriatr Soc 2014; 62: 889–895.

44.

Gifford

Niles

Rivkin

, et al. Continuing education training focused on the development of behavioral telehealth competencies in behavioral healthcare providers. Rural Remote Health 2012; 12: 2108.

45.

Luxton

Pruitt

Osenbach

JE.

Best practices for remote psychological assessment via telehealth technologies. Prof Psychol Res Pract 2014; 45: 27–35.

46.

Vrana

DT.

Can a computer administer a Wechsler intelligence test?

Prof Psychol Res Pract 2017; 48: 191–198.

47.

Inman

Soheilian

Luu

LP.

Telesupervision: building bridges in a digital era. J Clin Psychol 2019; 75: 292–301.

48.

Rousmaniere

Abbass

Frederickson

New developments in technology‐assisted supervision and training: a practical overview. J Clin Psychol 2014; 70: 1082–1093.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.17 MB