Selection of criteria for a telemedicine framework for designing,implementing,monitoring and evaluating telemedicine interventions: Validation using a modified Delphi process

Abstract

Objectives

The call to scale up telemedicine services globally as part of the digital health transformation lacks an agreed-upon set of constructs to guide the implementation process. A lack of guidance hinders the development, consolidation, sustainability and optimisation of telemedicine services. The study aims to reach consensus among telemedicine experts on a set of implementation constructs to be developed into an evidence-based support tool.

Methods

A modified Delphi study was conducted to evaluate a set of evidence-informed telemedicine implementation constructs comprising cores, domains and items. The study evaluated the constructs consisting of five cores: Assessment of the Current Situation, Development of a Telemedicine Strategy, Development of Organisational Changes, Development of a Telemedicine Service, and Monitoring, Evaluation and Optimisation of Telemedicine Implementation; seven domains: Individual Readiness, Organisational Readiness, Clinical, Economic, Technological and Infrastructure, Regulation, and Monitoring, Evaluation and Optimisation; divided into 53 items. Global telemedicine specialists (n = 247) were invited to participate and evaluate 58 questions. Consensus was set at ≥70%.

Results

Forty-five experts completed the survey. Consensus was reached on 78% of the constructs evaluated. Regarding the core constructs, Monitoring, Evaluation and Optimisation of Telemedicine Implementation was determined to be the most important one, and Development of a Telemedicine Strategy the least. As for the domains, the Clinical one had the highest level of consensus, and the Economic one had the lowest.

Conclusions

This research advances the field of telemedicine, providing expert consensus on a set of implementation constructs. The findings also highlight considerable divergence in expert opinion on the constructs of reimbursement and incentive mechanisms, resistance to change, and telemedicine champions. The lack of agreement on these constructs warrants attention and may partly explain the barriers that telemedicine services continue to face in the implementation process.

Keywords

Telemedicine implementation science Delphi process tool delivery of healthcare

Introduction

The rapid adoption of digital telemedicine solutions played a critical role in responding to the enormous pressure experienced by healthcare services during the COVID-19 pandemic.^1–3 Since then, there have been increasing demands to strengthen, consolidate and scale up telemedicine services globally.^4–8 This is recognised as being a complex task that, besides technological deployment, requires an understanding of need, a strategic vision, skilful management of organisational change, well-considered integration into health service planning, and robust monitoring and evaluation mechanisms. However, despite the available literature of the added value that telemedicine brings to health service delivery^1,9–15 post-COVID health systems are struggling – due to implementation challenges – to integrate and sustain telemedicine services.^12,16–19 There is now a real risk that the promise of sustainable, equitable, high-quality telemedicine services at scale will not be realised.

A major obstacle facing those developing telemedicine services is an understanding of what are the essential components of implementing an integrated telemedicine service, as a part of a broader digital health transformation agenda.^20,21 Although numerous resources and tools are available to assist telemedicine implementers, a gap has been identified for an easy-to-use, comprehensive, evidence-based tool that delineates universally acknowledged essential requirements for the successful design, deployment and optimisation of telemedicine services.^22,23 The lack of an agreed-upon definition of telemedicine adds to the complexity.^24,25 This study adopts the widely used World Health Organization definition, which identifies telemedicine as part of digital health, emphasizing remote clinical synchronous or asynchronous communication between those separated by distance, either client-to-provider or provider-to-provider.²⁶

To address this gap in implementation resources, the Support tool to strengthen telemedicine: guidance for telemedicine assessment and strategy development (referred to as the ‘support tool’) is under development at the request of the World Health Organization Regional Office for Europe (WHO/Europe). The support tool is being developed in the spirit of knowledge mobilisation (the process of moving knowledge into action),²⁷ drawing on complex adaptive systems thinking (systems that can adapt and self-organize in response to changes), through a robust multi-phase implementation science research mixed-method study, comprising three phases. First, a literature review to identify evidence-based essential building blocks of telemedicine deployment.^12,28,29 Second, a consensus-building study to enlist agreement among telemedicine experts on the dimensions of telemedicine implementation. And third, field testing the support tool with telemedicine implementers on the ground. The findings of each study phase feed into the subsequent one and help to further refine and strengthen the support tool constructs and their use.

This article focuses on the second phase of the study, that is, to reach consensus among telemedicine experts on a set of constructs to assess the maturity of telemedicine services and guide the implementation process.

Methods

Study design

A modified Delphi process based on an online round-less real-time format³⁰ was adopted to evaluate the telemedicine implementation constructs that were identified in phase 1. This study approach best suited the needs of the research, which calls for expert judgement where there is a lack of consensus, and a consolidation of geographically dispersed expert opinion.^31,32 The modified Delphi process is characterised by online rounds of systematic, iterative, controlled feedback, statistical aggregation and anonymised expert opinion. The Delphi method has been used widely in relevant public health research – including digital health research – to reach consensus among experts when there is a lack of agreement and to assist with the development of knowledge tools.^32–44 The study received ethical approval from the Research Ethics Committee, Universitat Oberta de Catalunya (CE22-AA60, CE23-TE12).

Participant recruitment

An expert panel was recruited by purposive/convenience sampling using a multipronged selection strategy incorporating the snowball technique. This non-probability sampling method is conducive to a consensus-forming qualitative Delphi methodology, which does not require statistical significance sampling.⁴⁵ The primary criteria for selection were that candidates should be recognised, knowledgeable and experienced telemedicine implementation specialists. Diversity of experience was sought, including a variety of health professional backgrounds, comprising various levels of the health system, including public and private sector representation. Other considerations included geographic diversity, gender balance and ability to participate in the study in the English language. All participants were over 18 years of age. A list of potential experts was identified through the telemedicine global specialist network of WHO/Europe, other telemedicine professional networks, published relevant literature and their reference lists, telemedicine conference presenters, and recommendations from expert peers and the research team. The list of potential specialists was assessed by the research team, ranked according to the selection criteria and finalised. With an expected participation rate of 10% to 15% of those asked to take part,³² 247 participants who met the criteria were invited to do so. On 1 May 2023, a letter introducing the voluntary study was sent by the Unit Head, Data and Digital Health Division of Country Health Policies and Systems WHO/Europe, to those selected. A reminder was sent on 24 May 2023.

Dimensions and criteria

The telemedicine implementation constructs identified in the first phase (literature review) comprised three implementation levels: cores, domains and items. First, five cores considered to be the basis, or pillars, of a telemedicine service: Core 1 – Assessment of the Current Situation (C1); Core 2 – Development of a Telemedicine Strategy (C2); Core 3 – Development of Organisational Changes (C3); Core 4 – Development of a Telemedicine Service (C4); and Core 5 – Monitoring, Evaluation and Optimisation of Telemedicine Implementation (C5). Second, seven domains, which were defined by thematically grouping the barriers and facilitators identified in the literature:^{12,18,20,28,46–53} Domain 1 – Individual Readiness (D1); Domain 2 – Organisational Readiness (D2); Domain 3 – Clinical (D3); Domain 4 – Economic (D4); Domain 5 – Technological and Infrastructure (D5); Domain 6 – Regulation (D6); and Domain 7 – Monitoring, Evaluation and Optimisation (D7). The literature review identified that the domains were relational across the cores, and cross-cutting at the micro, mesa and macro levels of the health system (see Table 1 for a summary of the distribution of the domains across the cores). The monitoring, evaluation and optimisation construct was found to be both a core and a transversal domain with its own characteristics. And third, 53 items, which were defined by further articulating the barriers and facilitators within each core and domain. Each item was then formulated into an item question exploring the requirements of the item, which collectively form the determinants of the domain. For each item question, a more detailed description of the construct to be assessed was established.

Table 1.

Summary of the number of questions for consensus in the modified Delphi by core and domain classification.

Core	C1. Assessment of the Current Situation	C2. Development of a Telemedicine Strategy	C3. Development of Organisational Changes	C4. Development of a Telemedicine Service	C5. Monitoring, Evaluation and Optimisation of Telemedicine Implementation	Total
Domains	D1. Individual Readiness (n = 2)	D1. Individual Readiness (n = 2)	D1. Individual Readiness (n = 1)	D1. Individual Readiness (n = 1)	D7. Monitoring, Evaluation and Optimisation. (n = 6)
	D2. Organisational Readiness (n = 2)	D2. Organisational Readiness (n = 3)	D2. Organisational Readiness (n = 2)
	D3. Clinical (n = 1)	D3. Clinical (n = 1)	D3. Clinical (n = 1)	D3. Clinical (n = 2)
	D4. Economic (n = 1)	D4. Economic (n = 1)	D4. Economic (n = 1)	D4. Economic (n = 1)
	D5. Technological and Infrastructure (n = 2)	D5. Technological and Infrastructure (n = 1)	D5. Technological and Infrastructure (n = 3)	D5. Technological and Infrastructure (n = 11)
	D6. Regulation (n = 4)	D6. Regulation (n = 1)	D6. Regulation (n = 1)	D6. Regulation (n = 2)
No. of item questions	12	9	9	17	6	53

The support tool constructs were assessed by the telemedicine experts in the modified Delphi study at core and item levels. A total of 58 questions were developed, 5 questions asked respondents about the core constructs, and 53 questions evaluated item constructs. See Multimedia Appendix S1 for the list of Delphi questions.

Delphi software selection and questionnaire design

The Smart Delphi platform by Onsanity (www.smartdelphi.com) was chosen to carry out the modified Delphi process, which, in comparison to other platforms assessed using tested criteria,⁵⁴ provided: flexibility, strong data management, security, anonymity, ease of use, intuitive design, technical assistance and cost-effectiveness. The support tool questions were uploaded to the Smart Delphi platform, along with a definitional description for each question to reduce ambiguity and response bias, thereby strengthening reliability and validity. The questions were designed to be clear, to reduce cognitive load and to not overwhelm respondents. Five telemedicine specialists, not selected to participate in the study, pre-tested the survey instrument and platform for clarity, flow, navigational ease, time taken for completion, technical glitches, reliability and validity, along with syntax, semantics and typographical errors. The survey was finalised incorporating the findings of the pilot test.

Delphi process

In the invitation letter, participants received information about the study's purpose, instructions on how to participate, the login link and the access code. Once participants had entered the platform, they were asked to complete informed consent. After consenting, they provided general sociodemographic data (gender, age range, years of experience in telemedicine, professional profile, workplace and health sector). No personal identification information was collected during the study. Each member of the expert panel individually and independently participated in the study by answering the questions. The Smart Delphi platform design displayed one question on screen at a time. On entering each question, one evaluative query was put to the specialists, How important do you think this is for the implementation of telemedicine? The evaluative query was the same for all the questions in the survey. Respondents ranked the importance of the construct on a 6-point Likert scale, from not very important (1) to very important (6). Respondents were also encouraged to express their views on the construct by giving narrative comments on each question. See Figure 1 for a screen capture of the survey question.

Figure 1.

Screen capture of smart Delphi survey question.

Immediately after submitting a response to a question, participants were provided with anonymised aggregated data of all the responses at that time. The aggregated results for the question included the number of responses (and non-participants), the median, and the interquartile range (IQR). The data was displayed on the screen indicating an individual expert's response ranked within the total responses (see Figure 2), and these could be further broken down by sociodemographic groupings (see Figure 3). At this point, participants were asked to reflect further on their response considering the aggregated results and, if they wished, they could alter their response on the Likert scale. Respondents could also see the comments of others and were able to reply to an existing comment or add a new comment. The respondents’ comments were displayed verbatim on the screen and not moderated by the research team.

Figure 2.

Screen capture of smart Delphi total aggregated results.

Figure 3.

Screen capture of smart Delphi disaggregated by profession.

Once participants had finished giving their response to a question, they were invited to move on to the next question and work their way through all the questions, following the same process. The platform allowed respondents to move around the survey back and forth, revising their scores as often as they wished. At any time during the survey, they could also review the summary consensus portal comprising all their responses and comments, compared to the aggregated group results. The survey access link allowed the experts to leave the survey at any time they wished and return to it at their convenience to continue where they had left off. Once respondents had finished the survey, on the closing page they were provided with summary information (including their responses compared to aggregated responses, completion percentage, overall alignment percentage, number of adjustments, time taken for completion, and number of comments and replies), and they were again invited to return to the survey at a later stage to make further adjustments as they wished.

The potential bias associated with the initial condition effect was minimised during the survey by following good practice guidance,⁵⁵ with aggregated feedback only uploaded once a small critical mass of five responses had been completed. Throughout the study, the research team monitored the platform for potential technical glitches and configuration issues.

Analysis

Analysis commenced after the study was closed using the scores of the final round of ratings given by those who had fully completed the study. Descriptive statistical analysis and simplified narrative thematic analysis were performed. Percentages were calculated for sociodemographic data. Analysis included measuring central tendency and level of dispersion using the mean, median, standard deviation and IQR. The data were analysed using STATA 18.0. There is no universally agreed-upon definition of consensus in Delphi studies, however, informed by standard practice,⁵⁶ this study calculated consensus for every single item as the percentage of respondents who ranked the items as important (5) or very important (6). Three levels of consensus were established, high being above or equal to 80%, medium above or equal to 70% and below 80%, and low below 70%.^56–61 It was determined that items on which 70% consensus was not reached should be discarded from the final version of the support tool. Comparative quantitative analysis using percentiles was applied to explore performance across cores and domains, though this should be interpreted with caution given that items are not equally distributed across each dimension of the support tool. A simple thematic analysis was used to analyse the narrative comments and replies, by searching, reviewing, defining and naming themes.⁶²

Results

The Delphi study was conducted over 5 weeks from 1 May 2023 to 6 June 2023. Of the 247 specialists who had been invited to take part, 63 participated (25.5%); of these, 45 (71.4%) fully completed the study, and the remaining 18 partially completed it (28.6%). Experts posted 3195 ratings over the duration of the study.

Expert panel

Of the 63 experts who had participated in the study, more than two-thirds (71.5%) were over 40 years of age (see Table 2). The gender balance was slightly skewed towards male participation (57.1%). The extensive professional telemedicine expertise that specialists brought to the study was reflected by 84.1% of respondents reporting more than 4 years’ experience and almost half (47.6%) over 10 years’ experience. The respondents’ professional roles were academic/educational professions (38.1%), followed by medicine (34.9%), information technology (9.5%), policy-making (7.9%) and management (6.4%). Most participants came from the public sector (76.2%), some from the private sector (12.7%) and others from unspecified sectors. Several levels of the health system were represented, from primary care (15.9%), hospitals (12.7%) and public health services (17.5%); though the workplace of over half the experts was unspecified (52.3%).

Table 2.

Expert panel characteristics.

Characteristics	Respondents (n = 63), n(%)
Gender
Men	36 (57.1)
Women	27 (42.9)
Age (in years)
20–29	3 (4.8)
30–39	15 (23.8)
40–49	18 (28.6)
50–59	14 (22.2)
60–69	10 (15.9)
More than 69	3 (4.8)
Experience in telemedicine (in years)
Less than 1	5 (7.9)
1–3	5 (7.9)
4–5	11 (17.5)
6–10	12 (19.1)
More than 10	30 (47.6)
Professional role
Academic, educational	24 (38.1)
Medicine	22 (34.9)
IT systems	6 (9.5)
Policy-making	5 (7.9)
Management	4 (6.4)
Nursing	1 (1.6)
Others	1 (1.6)
Workplace
Public Health Service	11 (17.5)
Primary Care	10 (15.9)
General Hospital	8 (12.7)
Socio-Health Care	1 (1.6)
Others	28 (44.4)
Not applicable	5 (7.9)
Sector
Public	48 (76.2)
Private	8 (12.7)
Not applicable	7 (11.1)

Main findings

Of the 58 evaluative questions asked, comprising 5 core questions and 53 item questions, consensus was reached on 78% (45/58) of the constructs evaluated. In the 5 core questions (see Multimedia Appendix S2), there was high agreement among experts on all the core constructs of the draft support tool (≥80%). Monitoring, Evaluation and Optimisation of Telemedicine Implementation (C5) had the highest percentage of agreement and mean score, followed by Development of a Telemedicine Service (C4). The core with the lowest percentage of agreement (79.6%) was Development of a Telemedicine Strategy (C2).

Of the 53 item questions assessed, 40 questions (75.5%) ranked above the consensus threshold. Of these items, high consensus was reached on 21 (52.5%), with over 80% of respondents rating them highly, and medium consensus (above or equal to 70% but below 80%) was reached on 19 (47.5%). In Table 3, these items are ranked from highest to lowest percentage of consensus by core and domain.

Table 3.

Item questions – consensus ranking by core and domain.

Item number		Consensus % agreement (important to very important)		CORES
Item number		Consensus % agreement (important to very important)	Item question	Core 1 ASSESSMENT OF THE CURRENT SITUATION	Core 2 DEVELOPMENT OF A TELEMEDICINE STRATEGY	Core 3 DEVELOPMENT OF ORGANISATIONAL CHANGES	Core 4 DEVELOPMENT OF A TELEMEDICINE SERVICE	Core 5 MONITORING, EVALUATION AND OPTIMISATION OF TELEMEDICINE IMPLEMENTATION
HIGH CONSENSUS (≥80%)
47	TOP 10	95.6	Is telemedicine technology reliable?				D5
30		93.5	Are IT and technical support services available for the telemedicine service?			D5
37		91.1	Are data protection and security measures integrated into the telemedicine service?				D6
56		91.1	Is there a mechanism in place to assess patient outcomes (satisfaction and quality of life)?					D7
53		91.1	Is there a plan for the continuous monitoring and evaluation of the implementation of telemedicine services?					D7
39		91.1	How intuitive is the solution?				D5
45		88.9	Is there adequate broadband Internet to support the telemedicine service?				D5
55		88.9	Is there a mechanism in place to assess healthcare provider outcomes (healthcare professionals’ satisfaction and workload)?					D7
38		88.9	Are telemedicine standard operating procedures in place?				D5
48		88.9	Are the electronic devices related to the telemedicine service easily accessible, adaptable and self-configurable?				D5
8		86.8	Does the telemedicine service have the Information Systems and Technologies necessary for providing the service?	D5
35		86.7	Is the healthcare workforce capable of delivering quality telemedicine services?				D1
31		86.7	Are the healthcare delivery modalities adjusted to align with the telemedicine services?			D3
12		86.5	Is an adequate legal and regulatory framework in place for telemedicine services?	D6
50		84.4	Have clinical protocols, referral forms, policies and guidelines been developed to support the telemedicine service or programme?				D3
23		83.3	Does the strategy incorporate the regulatory, legal and ethical requirements of telemedicine?		D6
36		82.2	Is informed consent of the patient integrated into the telemedicine services?				D6
51		82.2	Is a clinical risk-assessment process in place?				D3
29		80.4	Is interoperability addressed in telemedicine implementation?			D5
42		80.0	Does the centre have a computer with audio and video capabilities and streaming devices for delivering telemedicine?				D5
54		80.0	Is there a mechanism in place to assess whether the telemedicine service is sustainable?					D7
MEDIUM CONSENSUS (<80% and ≥70%)
46		77.8	Are telemedicine service data interoperable with other health system databases?				D5
2		76.8	Are healthcare providers capable of providing telemedicine services?	D1
4		76.4	Does the organisation have a clear understanding of the needs and resources that the telemedicine service requires?	D2
44		75.6	Is there technology to support image sharing and peripheral devices for remote diagnosis?				D5
9		75.5	Is Internet connectivity adequate to meet the needs of the telemedicine service?	D5
7		75.5	Does the telemedicine service have adequate financial resources?	D4
20		75.0	Is the telemedicine strategy informed by the best available telemedicine clinical evidence?		D3
22		75.0	Does the strategy adequately consider the information, communication and technology needs of the telemedicine service?		D5
15		75.0	Is there a strategy to support telemedicine skills development of the health workforce?		D1
18		75.0	Is the telemedicine strategy aligned with the health system strategy?		D2
16		75.0	Is there a strategy to support patient and public engagement in the telemedicine service?		D1
5		74.1	Has the telemedicine service been adapted to the local context?	D2
6		74.1	Has the telemedicine service been clearly defined based on the health needs of the context?	D3
28		73.9	Are there mechanisms to upgrade technology according to the needs of the telemedicine service?			D5
49		73.3	Is a routine cost vs budget analysis of the telemedicine service performed?				D4
26		72.3	Is there a supportive organisational environment for telemedicine implementation?			D2
33		71.1	Are there communication mechanisms to inform and promote the telemedicine service to patients?			D6
57		71.1	Is there a mechanism in place to assess whether the telemedicine service is scalable?					D7
19		70.8	Does the telemedicine strategy include governance mechanisms?		D2
LOW CONSENSUS (<70%)
43	BELOW CONSENSUS THRESHOLD	68.9	Is there technology to support telemedicine clinical video conferencing?				D5
17		66.7	Is the telemedicine strategy informed by the best available evidence on how to implement telemedicine services?		D2
10		64.2	Is telemedicine integrated into the national health system architecture?	D6
27		63.8	Is resistance to change addressed in the telemedicine implementation process?			D2
13		62.7	Is there a national monitoring and evaluation framework in place to measure telemedicine service performance?	D6
3		62.5	Do healthcare providers have the intention to use telemedicine for routine clinical practice?	D1
21		62.5	Has an economic analysis been undertaken to determine the feasibility of the telemedicine service?		D4
32		60.0	Are there reimbursement and incentive mechanisms in place in the regulatory environment?			D4
11		57.7	Is there a national financing mechanism in place to resource telemedicine?	D6
25		51.1	Have telemedicine champions been identified to support implementation?			D1
41		46.7	Is there a data/cloud centre for storing telemedicine service data?				D5
40		46.7	Is there a designated telemedicine room which allows for confidential consultations?				D5
58		35.6	Is there a mechanism in place to assess whether the telemedicine service is transferable to other countries and across borders?					D7

Key

D1: Domain 1 – Individual Readiness

D2: Domain 2 – Organisational Readiness

D3: Domain 3 – Clinical

D4: Domain 4 – Economic

D5: Domain 5 – Technological and Infrastructure

D6: Domain 6 – Regulation

D7: Domain 7 – Monitoring, Evaluation and Optimisation

Analysing the top 10 items with the highest consensus, there was very high agreement on the importance of the constructs of telemedicine technology reliability (95.6%), followed by technical support services (93.5%). High consensus was also reached on items relating to data protection and security measures, intuition of the telemedicine solution, and assessing telemedicine outcomes, all with 91.1%. Slightly lower ranked in importance, though still within the top 10, were the constructs relating to adequate broadband, health provider outcome measures, telemedicine standardised operating procedures, and telemedicine device accessibility and adaptability, all with 88.9%.

Examining the top 10 ranked items by core, most items fall within C4 at 60% (6/10 items), followed by C5 at 30% (3/10) and C3 at 10% (1/10). No items from C1 or C2 were ranked in the top 10, moreover, high consensus was reached on only a few items from these cores. By domain, the top 10 in importance converge in the domains of D5 at 60% (6/10 items), followed by D7 at 30% (3/10) and, lastly, D6 at 10% (1/10). No items in D1, D2, D3 or D4 reached the top 10 in importance, though three items from D3 and one from D1 ranked in the high consensus group (above 80%).

Analysing the proportion of items on which consensus was reached by domain, D3 had the highest consensus at 100% (5/5 items), followed by D7 at 83% (5/6), D5 at 82% (14/17), D2 at 71% (5/ 7), D1 at 67% (4/6), D6 at 63% (5/8) and, lastly, D4 at 50% (2/4).

The 13 lowest ranked items on which consensus was not reached (below 70%) are somewhat evenly distributed across the cores, with most items found in C1 at 31% (4/13 items), followed by C3 at 23% (3/13), C4 at 23% (3/13), C2 at 15% (2/13) and, lastly, C5 at 8% (1/13). In terms of domains, the highest number of items on which consensus was not reached was found in D5 and D6, each with 23% (3/13 items). This was followed by D1, D2 and D4, each with 15% (2/13 items), and, lastly, D7 with 8% (1/13). D3 had no items that fell below the consensus threshold. Moreover, the transversal core-domain construct C5/D7 was highly rated, with only one item falling below the consensus threshold.

Qualitative analysis

Analysis of the narrative comments showed that, of the 58 questions posed, respondents provided further remarks on 21 questions (2 core questions and 19 item questions). In total, 29 remarks were posted (28 comments and 1 reply), with most of the 21 questions receiving only 1 comment, a few receiving 2 comments, and 1 generating the most discussion with 3 comments and one reply. The engagement in the comments section of the study was somewhat disappointing, though understandable, given the time constraints of the expert panel. Nevertheless, the remarks provided were extremely valuable in further elucidating the perspectives and priorities of respondents.

The questions falling into the medium consensus group received the greatest proportion of remarks (44.8%), followed by those in the lowest consensus group (31%) and, lastly, those in the highest consensus group (24.1%), where the least discussion was found. Analysing the number of remarks by core, C2 (27.6%) generated the most discussion followed by C1 (24.1%), C5 (24.1%), C4 (13.8%) and, lastly, C3 (10.3%). Across the domains, of the 27 item questions, D7 received the most remarks and D3 generated the least discussion, with the other domains generating comparable levels of discussion to one another.

Comments fell into three broad categories: first, those emphasising the importance of the telemedicine implementation construct under consideration, thereby attempting to influence the opinions of others; second, those further defining the item descriptors; and third, one suggesting improvements to the item question. See Multimedia Appendix S3 for a summary of the thematic analysis of the comments.

Reviewing the comments in more detail, remarks relating to the core questions emphasised the importance of integrating telemedicine into the health system, the risks of telemedicine silos, and the need for a supportive organisational culture. Analysing the comments by domain, in D1 they focused on the importance of building public trust, along with end-user (health workforce and patient) engagement, participation and digital literacy. The D2 comments reiterated the integration of telemedicine within the health system, along with a call for comprehensive regulatory systems and competitor analysis. The D3 ones focused on understanding the context and patient needs. The D4 ones stressed the importance of a good fit between available economic resources and telemedicine solutions, along with the need to conduct a comprehensive economic analysis (including cost-benefit, cost-utility, cost-effectiveness and opportunity costs). The D5 ones focused on interoperability and the importance of mitigating technological risks, such as the need for backup storage systems. Here, it was recommended that the interoperability question (item 29) be better defined to include semantic as well as technical interoperability. The D6 ones highlighted the importance of monitoring and evaluation frameworks, and integration was again reiterated, with a focus on care channels and secondary data use. The issue of an adequate level of consent based on emerging evidence was also raised, as was the value of telemedicine in addressing the quadruple aim (enhancing patient experience, improving population health, lowering costs and improving staff experience). And lastly, D7 was the domain in which there was the most debate, where the comments again focused on themes of health system integration, equity of access, and a call for definitional clarity of the meaning of sustainability in telemedicine implementation.

In addition to the construct comments, there were four general feedback responses provided by participants at the end of the survey. These again echoed the importance of embedding telemedicine into the health system and recommended that the support tool be designed to facilitate integration and interoperability, and not allow telemedicine services to be siloed. Another response suggested that despite the evidence base in support of telemedicine, there remains inadequate consolidation of telemedicine in the public sector, even in mature telemedicine post-COVID contexts. Lastly, one respondent challenged the professional community to revisit the term ‘telemedicine’, as it was felt that it did not adequately represent the increasingly diverse range of health professionals (i.e. nurses) providing virtual healthcare.

Discussion

The key finding of the modified Delphi study was that consensus was reached on 78% of the constructs evaluated, thus further validating their importance to the telemedicine implementation ecosystem. Telemedicine experts considered the most important core dimension to be Monitoring, Evaluation and Optimisation of Telemedicine Implementation (C5) and the least important to be Development of a Telemedicine Strategy (C2). Across the domains, the highest level of consensus was reached on the Clinical one (D3) and the least on the Economic one (D4).

To the best of our knowledge, there is no comparable research in the peer-reviewed literature that sets out to reach consensus on the constructs of a telemedicine framework for designing and implementing telemedicine interventions using the Delphi methodology or any other method of group consensus. Hence, this study offers a distinctive contribution to the global knowledge base on telemedicine implementation. There are, however, some systematic reviews that have synthesised common constructs across telemedicine implementation tools and attempted to rank these.^49,53,63 Differences in taxonomy and the heterogeneity of these studies make it difficult to compare them with the findings of this modified Delphi study. Nevertheless, the constructs and ranking in these studies are somewhat consistent with the results of our study. Like the findings of our study, all of the aforementioned studies identified technology and infrastructure to be the highest priority construct, with other constructs relating to strategy, change management, service design, individual and organisational readiness, economic and financial factors, and policy and regulatory mechanisms all featuring as important dimensions across the telemedicine implementation frameworks reviewed. However, it is imperative to underscore that this predominant focus on technology may, to some extent, account for the limitations faced by telemedicine in its implementation process. Constructs that were less frequently included were dimensions relating to the public and patient sphere, which, as Mauco et al.⁴⁹ emphasise, needs further attention given the rapid growth of telemedicine interventions in diverse cultural and socio-economic contexts. In this respect, in our study, consensus was reached on all the items relating to patient and public dimensions, which may reflect a growing appreciation of the importance of patient-centred perspectives in the implementation process. Interestingly, the thematic monitoring, evaluation and optimisation construct ranked highly in our study. However, this construct does not feature explicitly in any of the systematic review findings. The increasing focus on monitoring, evaluation and optimisation may stem from the demand from decision-makers for greater evidence supporting telemedicine services. Finally, all the studies are united in highlighting the weakness of the validity and rigour of existing implementation frameworks, calling on the scientific community to develop more robust evidence-based tools.²⁹

Comparing our study to the broader field of eHealth, two studies provide findings of interest. Cremers et al.⁶⁴ conducted a three-round Delphi study to develop an eHealth implementation guideline for interventions in daily practice. In contrast to our study, which focused on the health system, Cremers et al. only focused on the health facility. Nevertheless, the findings are consistent with those of our study, with consensus being reached on five comparable implementation constructs: technology, acceptance, finance, organisation and legislative and policy. Interestingly, consensus was not reached on the evidence-based medicine construct in that study, whereas in ours it was reached on all clinical dimensions, including evidence-based medicine. Rezai-Rad et al.³³ also conducted a Delphi study aimed at designing an eHealth readiness assessment framework for Iran. The importance of the technology construct was again emphasised in this study, but other dimensions were grounded in different taxonomies, making comparisons with our modified Delphi study meaningless.

Another noteworthy finding from our study is the discrepancy between the results of the evidence review which informed the draft support tool and those of our study in relation to three dimensions of implementation. While consensus was not reached on the constructs of reimbursement and incentive mechanisms (item 32), resistance to change (RTC) (item 27) and telemedicine champions (TCs) (item 25) in our study, they were emphasised as important dimensions in the literature review. The low ranking of the importance of telemedicine reimbursement and incentive mechanisms (TRIMs) in our study was striking given the evidence supporting the fact that TRIMs motivate end-user participation in telemedicine services.^{1,12,52,65–75} Indeed, some evidence suggests that they constitute the most important dimension of implementation.^3,76,77 Several studies have empirically shown a causal relationship between TRIMs and health provider/consumer acceptance of telemedicine⁵² and intention to use telemedicine.^1,70 Moreover, the strengthening of TRIMs by governments during the COVID-19 pandemic was recognised as contributing to the improved uptake of telemedicine services.^{3,65,73,75,76} The literature suggests that TRIMs are pivotal to the implementation ecosystem, though the design of TRIMs to fit the heterogeneity of health financing contexts is recognised as being complex.^68,72,77,78 Interestingly, in our study, the Economic domain (D4) had the lowest consensus ranking compared to other domains, which further reinforces the lack of attention experts pay to financial aspects of telemedicine implementation. This oversight may be attributed to several factors, including fragmented budget allocations,⁷⁹ inadequate financial planning,⁸⁰ limited cost-benefit analysis,⁸¹ and competing financial pressures, especially in lower and middle-income nations.²⁰

Developing strategies to address health workforce RTC in telemedicine interventions is another construct on which consensus was not reached in our study. Again, this is striking given the extensive literature on RTC as a barrier to telemedicine implementation.^{1,12,51,52,74,82,83} Moreover, the relationship between end-user acceptance of technological innovation as a condition of telemedicine uptake has been theoretically verified.^1,84–88 Absent or poorly designed TRIMs are also associated with RTC.^{51,52,69,83,89} Early identification and amelioration of RTC are recognised as reducing implementation risks and contributing to the success of telemedicine interventions.^{12,17,51,52,66,68–70,83,87,90}

The consensus threshold on the TCs construct was not reached either. The potential catalytic role that TCs play in managing change and mitigating RTC is emphasised across the literature,^{12,17,82,91–96} though the extent of their contribution is inconclusive.¹² Moreover, champions have been identified as a success factor in the diffusion of innovation and adoption of evidence-based practice in public health, with considerable research pointing to the positive, catalytic role that champions can play in managing change and mitigating RTC.^97–101 Nevertheless, the importance of the role that champions play in the telemedicine implementation process is inconclusive,¹² which in part is due to the diversity of the TC approaches used, making comparative analysis difficult,⁹⁸ and also to the challenge of determining TC attributes from other telemedicine implementation strategies.⁹⁶ Despite the growing evidence of the potential role TCs can play in guiding, leading, coordinating, legitimising, educating, and communicating the telemedicine vision and supporting stakeholder engagement,^17,82,90,94 it is recognised that the causal pathways and mechanisms between TCs and telemedicine implementation outcomes need further empirical testing.^91,96,99,100

Strengths and limitations

A notable strength of this study is that it addresses an identified knowledge gap in the telemedicine implementation field, thereby helping to reach consensus among telemedicine experts on a set of constructs for telemedicine implementation. Moreover, the constructs evaluated in this study are drawn from a robust evidence-based review, strengthening their validity and reliability. Another strength is the number of experts participating in the modified Delphi process thanks to the online format, together with the iterative process of this study. However, it's worth noting that this approach also posed limitations in terms of depth. Validity was also enhanced by including several types of profile on the expert panel. Nevertheless, some profiles were underrepresented and more than half of the sample did not provide information about their workplace, meaning that some cores/domains may not have been properly addressed, thereby compromising the validity and reliability of the process. Another weakness of the study is the lack of collection of geographic characteristics of respondents. However, it's worth noting that the geographic distribution of invitees (n = 247) represented a broad global diversity. To mitigate the potential for bias and further strengthen validity and reliability, various strategies were deployed during the study by following good practice guidance,¹⁰² including grounding the study in the literature, integrating qualitative and quantitative measurements, verifying and further validating findings with additional experts, along with the planned field testing of the support tool. Another limitation of the study is that only one evaluative query was put to the respondents. More queries exploring the level of difficulty of implementing the construct may have provided deeper insights. However, given the time-constrained expert panel, breadth was favoured over depth for this study. In this sense, open-ended answers were included in each section, thereby enabling experts to provide comments on included criteria or provide new information for consideration. Nonetheless, it's important to note that not all participants chose to engage in this aspect of the study, hence these comments may not fully represent the sample.

Future actions and conclusions

The study's findings reveal that, at the core level, monitoring, evaluation and optimisation were ranked as the most important aspects, while developing a telemedicine strategy was deemed the least important. Among the domains, the clinical domain exhibited the highest level of consensus, whereas the economic domain showed the lowest. The results of this study are important for telemedicine service decision-makers, practitioners, stakeholders and experts. The results address a critical knowledge gap identified in telemedicine interventions, that is, the lack of a robust tool to guide telemedicine design, implementation, monitoring and evaluation. The findings provide a strategic direction forward for telemedicine services, offering an evidence-based set of telemedicine implementation constructs that have been agreed upon by a panel of telemedicine implementation experts and developed into an easy-to-use support tool to guide the telemedicine implementation process. Given the exponential efforts to scale up telemedicine globally coupled with the persistent obstacles to the consolidation of telemedicine, a robust support tool that facilitates knowledge sharing and exchange represents a significant step forward for the field of telemedicine. Importantly, the findings of this study also point to several noteworthy areas of incongruity in the collective understanding of telemedicine implementation, which may contribute to the prevailing barriers experienced in telemedicine execution, namely in the implementation constructs of reimbursement and incentive mechanisms, RTC and TCs. These constructs warrant further attention by the telemedicine community to ascertain their degree of importance in the implementation process. Finally, the findings of this study suggest additional avenues for future research, including the validation and improvement of the support tool, as well as longitudinal studies to monitor the implementation outcomes of integrated telemedicine interventions within various contextual settings, contributing to the broader digital health agenda.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076241251951 - Supplemental material for Selection of criteria for a telemedicine framework for designing, implementing, monitoring and evaluating telemedicine interventions: Validation using a modified Delphi process

Supplemental material, sj-docx-1-dhj-10.1177_20552076241251951 for Selection of criteria for a telemedicine framework for designing, implementing, monitoring and evaluating telemedicine interventions: Validation using a modified Delphi process by Che Katz, Noemí Robles, David Novillo-Ortiz and Francesc Saigí-Rubió in DIGITAL HEALTH

Supplemental Material

sj-docx-2-dhj-10.1177_20552076241251951 - Supplemental material for Selection of criteria for a telemedicine framework for designing, implementing, monitoring and evaluating telemedicine interventions: Validation using a modified Delphi process

Supplemental material, sj-docx-2-dhj-10.1177_20552076241251951 for Selection of criteria for a telemedicine framework for designing, implementing, monitoring and evaluating telemedicine interventions: Validation using a modified Delphi process by Che Katz, Noemí Robles, David Novillo-Ortiz and Francesc Saigí-Rubió in DIGITAL HEALTH

Supplemental Material

sj-docx-3-dhj-10.1177_20552076241251951 - Supplemental material for Selection of criteria for a telemedicine framework for designing, implementing, monitoring and evaluating telemedicine interventions: Validation using a modified Delphi process

Supplemental material, sj-docx-3-dhj-10.1177_20552076241251951 for Selection of criteria for a telemedicine framework for designing, implementing, monitoring and evaluating telemedicine interventions: Validation using a modified Delphi process by Che Katz, Noemí Robles, David Novillo-Ortiz and Francesc Saigí-Rubió in DIGITAL HEALTH

Footnotes

Acknowledgements

The authors would like to thank all the telemedicine experts who participated fully in the study.

Contributors

All authors contributed equally.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Disclaimer

DNO is a staff member of the World Health Organization. The author alone is responsible for the views expressed in this article, and he does not necessarily represent the decisions, policies, or views of the World Health Organization.

Ethical approval

The study received ethics approval by the Research Ethics Committee, Universitat Oberta de Catalunya (CE22-AA60 8 November 2022 and CE23-TE12 17 April 2023). Informed consent was included in the invitation letter, and participants consented prior to engagement in the study. No individual identification data were collected, thus ensuring the confidentiality of participants.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by the WHO Regional Office for Europe.

Guarantor

The corresponding author, FSR.

ORCID iDs

Che Katz

Francesc Saigí-Rubió

Supplemental material

Supplemental material for this article is available online.

References

Rouidi

Elouadi

Hamdoune

. Acceptance and use of telemedicine technology by health professionals: development of a conceptual model. Digit Health 2022; 8: 1–9.

Yadav

Mistry

Ghimire

, et al. Recognizing the roles of primary health care in addressing non-communicable diseases in low- and middle-income countries: lesson from COVID-19, implications for the future. J Glob Health 2021; 11: 1–6.

Omboni

Padwal

Alessa

, et al. The worldwide impact of telemedicine during COVID-19: current evidence and recommendations for the future. Connected Health 2022; 1: 7–35.

OECD. The future of telemedicine after COVID-19. OECD 2023: 1–10. doi:https://doi.org/10.1787/2074319x

Blandford

Wesson

Amalberti

, et al. Opportunities and challenges for telehealth within, and beyond, a pandemic. Lancet Glob Health 2020; 8: e1364–e1365.

Bhaskar

Bradley

Chattu

, et al. Telemedicine across the globe-position paper from the COVID-19 pandemic health system resilience program (REPROGRAM) international consortium (part 1). Front Public Health 2020; 8: 644.

Bashshur

Doarn

Frenk

, et al. Telemedicine and the COVID-19 pandemic, lessons for the future. Telemed e-Health 2020; 26: 571–573.

Gudi

NJO

Konapur

John

, et al. Telemedicine supported strengthening of primary care in WHO South East Asia region: lessons from the COVID-19 pandemic experiences. BMJ Innov 2021; 7: 580–585.

Appuswamy

Desimone

. Managing diabetes in hard to reach populations: a review of telehealth interventions. Curr Diab Rep 2020; 20: 28.

10.

Goharinejad

Hajesmaeel-Gohari

Jannati

, et al. Review of systematic reviews in the field of telemedicine. Med J Islam Repub Iran 2021; 35: 1–16.

11.

Petrazzuoli

Kurpas

Vinker

, et al. COVID-19 pandemic and the great impulse to telemedicine: the basis of the WONCA Europe statement on telemedicine at the WHO Europe 70th regional meeting September 2020. Prim Health Care Res Dev 2021; 22: 1–5.

12.

Saigí-Rubió

Nascimento do

IJB

Robles

, et al. The current status of telemedicine technology use across the World Health Organization European Region: an overview of systematic reviews. J Med Internet Res 2022; 24: e40877.

13.

Totten

Womack

Eden

, et al. Telehealth: mapping the evidence for patient outcomes from systematic reviews. Agency for Healthcare Research and Quality (US); 2016. Available: http://europepmc.org/books/NBK379320

14.

World Health Organization. WHO guideline. Recommendations on digital interventions for health system strengthening. 2019. Available: https://www.who.int/publications/i/item/9789241550505

15.

Zhang

Leuk

JS-P

Teo

W-P.

Domains, feasibility, effectiveness, cost, and acceptability of telehealth in aging care: scoping review of systematic reviews. JMIR Aging 6:e40460. 2023.

16.

Chitungo

Mhango

Dzobo

, et al. Towards virtual doctor consultations: a call for the scale-up of telemedicine in sub-Saharan Africa during COVID-19 lockdowns and beyond. Smart Health 2021; 21: 100207.

17.

James

Papoutsi

Wherton

, et al. Scale-up, and sustainability of video consulting in health care: systematic review and synthesis guided by the NASSS framework. J Med Internet Res 2021;23: e23775.

18.

Thomas

Taylor

Ward

, et al. Beyond forced telehealth adoption: a framework to sustain telehealth among allied health services. J Telemed Telecare 2022; 30: 559–569.

19.

Thomas

Haydon

Mehrotra

, et al. Building on the momentum: sustaining telehealth beyond COVID-19. J Telemed Telecare 2022; 28: 301–308.

20.

Bali

. Barriers to development of telemedicine in developing countries. In: Heston

(ed.) In Telehealth. IntechOpen, 2018. doi:https://doi.org/10.5772/INTECHOPEN.81723

21.

Aminoff

Meijer

. Context and complexity in telemedicine evaluation: work domain analysis in a surgical setting. JMIR Perioper Med 2021; 4: e26580.

22.

Rangachari

. A holistic framework of strategies and best practices for telehealth service design and implementation. In: Pfannstiel

Brehmer

Rasche

(eds) Service design practices for healthcare innovation paradigms, principles, prospects. Cham: Springer International Publishing; 2022. pp.315–335. doi:https://doi.org/10.1007/978-3-030-87273-1_16

23.

Otto

Whitehouse

Schlieter

. Scaling up telemedicine initiatives: requirements for a new telemedicine maturity model. In: I. Management Association (ed) Research anthology on telemedicine efficacy, adoption, and impact on healthcare delivery. Hershey, PA: IGI Global; 2021. pp.240–255. doi:https://doi.org/10.4018/978-1-7998-8052-3.CH013

24.

OECD. The COVID-19 pandemic and the future of telemedicine. Paris: OECD; 2023. doi:https://doi.org/10.1787/ac8b0a27-en

25.

Sood

Mbarika

Jugoo

, et al. What is telemedicine? A collection of 104 peer-reviewed perspectives and theoretical underpinnings. Telemed e-Health 2007; 13: 573–590.

26.

World Health Organization. Consolidated telemedicine implementation guide. 2022. Available: https://www.who.int/publications/i/item/9789240059184

27.

Best

Holmes

. Systems thinking, knowledge and action: towards better models and methods. Evid Policy 2010; 6: 145–159.

28.

Pan American Health Organization. Framework for the implementation of a telemedicine service. 2016. Available: https://www.researchgate.net/publication/303373019

29.

WHO Regional Office for Europe. Support tool to strengthen telemedicine: guidance for telemedicine assessment and strategy development. Copenhagen; 2024. Licence: CC BY-NC-SA 3.0 IGO.

30.

Gordon

Pease

. RT Delphi: an efficient, “round-less” almost real time Delphi method. Technol Forecast Soc Change 2006; 73: 321–333.

31.

Dalkey

Helmer

. An experimental application of the DELPHI method to the use of experts. Manage Sci 1963; 9: 458–467.

32.

Beiderbeck

Frevel

von der Gracht

, et al. Preparing, conducting, and analyzing Delphi surveys: cross-disciplinary practices, new directions, and advancements. MethodsX 2021; 8: 101401.

33.

Rezai-Rad

Vaezi

Nattagh

. E-Health readiness assessment framework in Iran. Iran J Publ Health 2012; 41: 43–51. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3494230/.

34.

Ziade

Hmamouchi

el Kibbi

, et al. Telehealth in rheumatology: the 2021 Arab League of Rheumatology Best Practice guidelines. Rheumatol Int 2022; 42: 379–390.

35.

Robles

Puig

Gómez-Calderón

, et al. Evaluation criteria for weight management apps: validation using a modified Delphi process. JMIR Mhealth Uhealth 2020; 8: e16899.

36.

Gargon

Crew

Burnside

, et al. Higher number of items associated with significantly lower response rates in COS Delphi surveys. J Clin Epidemiol 2019; 108: 110–120.

37.

Almathami

HKY

Win

Vlahu-Gjorgievska

. Development and validation of a new tool to identify factors that influence users’ motivation toward the use of teleconsultation systems: a modified Delphi study. Int J Med Inform 2022; 157: 104618.

38.

Woodcock

Adeleke

Goeschel

, et al. A modified Delphi study to identify the features of high quality measurement plans for healthcare improvement projects. BMC Med Res Methodol 2020; 20: 8.

39.

Niederberger

Renn

. Delphi methods in the social and health sciences. In: Niederberger

Renn

(eds) Delphi methods in the social and health sciences, Springer Fachmedien Wiesbaden/Germany, 2023. doi:https://doi.org/10.1007/978-3-658-38862-1

40.

Jiménez-Rodríguez

Ruiz-Salvador

Salvador

MDMR

, et al. Consensus on criteria for good practices in video consultation: a Delphi study. Int J Environ Res Public Health 2020; 17: 1–16.

41.

Davies

van Gassel

RJJ

van de Poll

, et al. Core outcome measures for clinical effectiveness trials of nutritional and metabolic interventions in critical illness: an international modified Delphi consensus study evaluation (CONCISE). Crit Care 2022; 26: 240.

42.

Varndell

Fry

Elliott

. Applying real-time Delphi methods: development of a pain management survey in emergency nursing. BMC Nurs 2021; 20: 149.

43.

Powell

Waltz

Chinman

, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci 2015; 10. doi:https://doi.org/10.1186/s13012-015-0209-1

44.

Snyder-Halpern

. Indicators of organizational readiness for clinical information technology/systems innovation: a Delphi study. Int J Med Inform 2001; 63: 179–204.

45.

Hasson

Keeney

McKenna

. Research guidelines for the Delphi survey technique. J Adv Nurs 2000; 32: 1008–1015.

46.

Al-Samarraie

Ghazal

Alzahrani

, et al. Telemedicine in Middle Eastern countries: progress, barriers, and policy recommendations. Int J Med Inform 2020; 141: 104232.

47.

Joseph

West

Shickle

, et al. Key challenges in the development and implementation of telehealth projects. J Telemed Telecare 2011; 17: 71–77.

48.

Kidholm

Clemensen

Caffery

, et al. The model for assessment of telemedicine (MAST): a scoping review of empirical studies. J Telemed Telecare 2017; 23: 803–813.

49.

Mauco

Scott

Mars

. Critical analysis of e-health readiness assessment frameworks: suitability for application in developing countries. J Telemed Telecare 2018; 24: 110–117.

50.

Otto

Harst

Timpel

, et al. Defining and delimitating telemedicine and related terms – an ontology-based classification. Stud Health Technol Inform 2020; 268: 113–122.

51.

Sagaro

Battineni

Amenta

. Barriers to sustainable telemedicine implementation in Ethiopia: a systematic review. Telemed Rep 2020; 1: 8–15.

52.

Kruse C

Karem

Shifflett

, et al. Evaluating barriers to adopting telemedicine worldwide: a systematic review. J Telemed Telecare 2018; 24: 4–12.

53.

van Dyk

. A review of telehealth service implementation frameworks. Int J Environ Res Public Health 2014; 11: 1279–1298.

54.

Aengenheyster

Cuhls

Gerhold

, et al. Real-time Delphi in practice – a comparative analysis of existing software-based tools. Technol Forecast Soc Change 2017; 118: 15–27.

55.

Gnatzy

Warth

von der Gracht

, et al. Validating an innovative real-time Delphi approach – a methodological comparison between real-time and conventional Delphi studies. Technol Forecast Soc Change 2011; 78: 1681–1694.

56.

Niederberger

Spranger

. Delphi technique in health sciences: a map. Front Public Health 2020; 8: 57.

57.

Herdman

Rajmil

Ravens-Sieberer

, et al. Expert consensus in the development of a European health-related quality of life measure for children and adolescents: a Delphi study. Acta Pediatr Int J Pediatr 2002; 91: 1385–1390.

58.

Holey

Feeley

Dixon

, et al. An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Med Res Methodol 2007; 7: 1–10.

59.

Stewart

Gibson-Smith

MacLure

, et al. A modified Delphi study to determine the level of consensus across the European Union on the structures, processes and desired outcomes of the management of polypharmacy in older people. PLoS One 2017; 12: e0188348.

60.

von der Gracht

. Consensus measurement in Delphi studies: review and implications for future quality assurance. Technol Forecast Soc Change 2012; 79: 1525–1536.

61.

Diamond

Grant

Feldman

, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol 2014; 67: 401–409.

62.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77–101.

63.

Otto

Whitehouse

Schlieter

. On the road to telemedicine maturity: a systematic review and classification of telemedicine maturity models. In: 32nd Bled eConference Humanizing Technology for a Sustainable Society, BLED 2019. University of Maribor Press; 2019. pp.485–506. doi:https://doi.org/10.18690/978-961-286-280-0.26

64.

Cremers

Theunissen

Hiddink

, et al. Successful implementation of ehealth interventions in healthcare: development of an ehealth implementation guideline. Health Serv Manage Res 2021; 34: 269–278.

65.

Badawy

Radovic

. Digital approaches to remote pediatric health care delivery during the COVID-19 pandemic: existing evidence and a call for further research. JMIR Pediatr Parent 2020; 3(1): e20049.

66.

de Grood

Raissi

Kwon

, et al. Adoption of e-health technology by physicians: a scoping review. J Multidiscip Healthc 2016; 9: 335–344.

67.

O’Cathail

Ananth Sivanandan

Diver

, et al. The use of patient-facing teleconsultations in the national health service: scoping review. JMIR Med Inform 2020; 8: e15380.

68.

Castaneda

Ellimoottil

. Current use of telehealth in urology: a review. World J Urol 2020; 38: 2377–2384.

69.

Roig

Saigí

. Barreras para la normalización de la telemedicina en un sistema de salud basado en la concertación de servicios. Gac Sanit 2011; 25: 397–402.

70.

Rho

Choi

Lee

. Predictive factors of telemedicine service acceptance and behavioral intention of physicians. Int J Med Inform 2014; 83: 559–571.

71.

Lluch

. Incentives for telehealthcare deployment that support integrated care: a comparative analysis across eight European countries. Int J Integr Care 2013; 13. doi:https://doi.org/10.5334/ijic.1062

72.

Schmit

Schwitzer

Survance

, et al. Telehealth in the COVID-19 Pandemic. In: Burris

de Guia

Gable

, et al (eds) Assessing legal responses to COVID-19. Boston/USA: Public Health Law Watch. 2020, pp.123–132. Available: https://ssrn.com/abstract=3675922.

73.

Van Citters

Dieni

Scalia

, et al. Barriers and facilitators to implementing telehealth services during the COVID-19 pandemic: a qualitative analysis of interviews with cystic fibrosis care team members. J Cyst Fibros 2021; 20: 23–28.

74.

Dubin

Wyant

Balaji

, et al. Telemedicine usage among urologists during the COVID-19 pandemic: cross-sectional study. J Med Internet Res 2020; 22: e21875.

75.

Verma

Kerrison

. Patients’ and physicians’ experiences with remote consultations in primary care during the COVID-19 pandemic: a multi-method rapid review of the literature. BJGP Open 2022; 6: BJGPO.2021.0192.

76.

Burg

Pscherer

Opitz

. Digital communication in visceral medicine: regulatory framework for digital communication. Visc Med 2021; 96: 1–7.

77.

Hoffman

. Increasing access to care: telehealth during COVID-19. J Law Biosci 2020; 7: lsaa043.

78.

Becker

Dandy

Gaujean

, et al. Legal perspectives on telemedicine part 1: legal and regulatory issues. Perm J 2019; 23: 18–293.

79.

Lobley

. The economics of telemedicine. J Telemed Telecare 1997; 3: 117–125.

80.

Smith

. The influence of financial factors on the deployment of telemedicine. J Health Care Finance 2005; 32: 16–27.

81.

Khanal

Burgon

Leonard

, et al. Recommendations for the improved effectiveness and reporting of telemedicine programs in developing countries: results of a systematic literature review. Telemed e-Health 2015; 21: 903–915.

82.

Kho

Gillespie

Martin-Khan

. A systematic scoping review of change management practices used for telemedicine service implementations. BMC Health Serv Res 2020; 20: 15.

83.

Choi

Park

Choi

JYB

, et al. Stakeholders’ resistance to telemedicine with focus on physicians: utilizing the Delphi technique. J Telemed Telecare 2018; 25: 378–385.

84.

Davis

. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 1989; 13: 319–339.

85.

Venkatesh

Morris

Davis

, et al. User acceptance of information technology: toward a unified view. MIS Q 2003; 27: 425–478.

86.

Ajzen

. The theory of planned. Organ Behav Hum Decis Process 1991; 50: 179–211.

87.

Egerton

Nelligan

Setchell

, et al. General practitioners’ perspectives on a proposed new model of service delivery for primary care management of knee osteoarthritis: a qualitative study. BMC Fam Pract 2017; 18: 85.

88.

Harst

Lantzsch

Scheibe

. Theories predicting end-user acceptance of telemedicine use: systematic review. J Med Internet Res 2019; 21: e13117.

89.

Jafni

Bahari

Ismail

, et al. Exploring barriers that affect telerehabilitation readiness: a case study of rehabilitation centre in Malaysia. In: Advances in intelligent systems and computing. 843. Springer Verlag; 2019. pp.761–771.

90.

Østervang

Vestergaard

Dieperink

, et al. Patient rounds with video-consulted relatives: qualitative study on possibilities and barriers from the perspective of healthcare providers. J Med Internet Res 2019; 21: e12584.

91.

Kolltveit

BCH

Gjengedal

Graue

, et al. Conditions for success in introducing telemedicine in diabetes foot care: a qualitative inquiry. BMC Nurs 2017; 16: 2.

92.

Latifi

Azevedo

Boci

, et al. Telemedicine consultation as an indicator of local telemedicine champions’ contributions, health care system needs or both: tales from two continents. Telemed e-Health 2021; 27: 200–206.

93.

Payán

Frehn

Garcia

, et al. Telemedicine implementation and use in community health centers during COVID-19: clinic personnel and patient perspectives. SSM Qual Res Health 2022; 2: 100054.

94.

Wade

Eliott

. The role of the champion in telehealth service development: a qualitative analysis. J Telemed Telecare 2012; 18: 490–492.

95.

Kernan

Dryden

Nearing

, et al. Integrating CFIR-ERIC and e-Delphi methods to increase telegeriatrics uptake. Gerontologist 2023; 63: 545–557.

96.

Greenhalgh

Robert

Macfarlane

, et al. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004; 82: 581.

97.

Bonawitz

Wetmore

Heisler

, et al.

Champions in context: Which attributes matter for change efforts in healthcare?

Implement Sci 2020; 15. doi:https://doi.org/10.1186/S13012-020-01024-9

98.

Miech

Rattray

Flanagan

, et al. Inside help: an integrative review of champions in healthcare-related implementation. SAGE Open Med 2018; 6: 1–11.

99.

Morena

Gaias

Larkin

. Understanding the role of clinical champions and their impact on clinician behavior change: The need for causal pathway mechanisms. Front Health Serv 2022; 2: 896885.

100.

Santos

Graham

Lalonde

, et al. The effectiveness of champions in implementing innovations in health care: a systematic review. Implement Sci Commun 2022; 3: 1–48.

101.

Shea

. A conceptual model to guide research on the activities and effects of innovation champions. Implement Res Pract 2021; 2: 2633489521990443.

102.

Jünger

Payne

Brine

, et al. Guidance on conducting and reporting Delphi studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med 2017; 31: 684–706.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.01 MB

0.02 MB