The Store-and-Forward Telemedicine Service User-satisfaction Questionnaire: Development and validation of a questionnaire to monitor and assess health care providers' experiences

Abstract

Introduction

With wider adoption of store-and-forward telemedicine accelerating, particularly post-coronavirus disease 2019, it is essential to understand health care providers’ (HCPs) satisfaction with digital solutions offered by telemedicine organizations to (continuously) guarantee service quality. We developed the Store-and-Forward Telemedicine Service User-satisfaction Questionnaire to monitor and assess HCPs’ experiences with contracted telemedicine organizations.

Methods

Questionnaire construction (phase 1) consisted of exploratory literature search on validated telemedicine satisfaction questionnaires, a telemedicine domain and human factors expert focus group, stakeholder focus group (customer service employee and telemedicine account managers), and two pre-testing rounds among 18 HCPs. The pilot questionnaire (phase 2) was sent to 2179 HCPs for validity and reliability assessment.

Results

Phase 1: Two validated questionnaires (73 items overall) were used as input for Store-and-Forward Telemedicine Service User-satisfaction Questionnaire. Revisions resulted in 61 items. Phase 2: the pilot 61-item Store-and-Forward Telemedicine Service User-satisfaction Questionnaire instrument was completed by 181 of 2179 invited HCPs. Forty-one mandatory items of the pilot Store-and-Forward Telemedicine Service User-satisfaction Questionnaire rated on a 5-point Likert scale were included in psychometric analyses and resulted in six reliable scales: training, communication, organization policy and strategy, interaction platform, usage platform, and working conditions.

Discussion

The Store-and-Forward Telemedicine Service User-satisfaction Questionnaire is a reliable and valid questionnaire for measuring HCPs’ satisfaction with store-and-forward telemedicine services as part of a continuous quality improvement cycle. Reimbursement questions were excluded due to low response. As adoption of telemedicine may be impeded by financial compensation issues, this requires consideration in future telemedicine questionnaires. Store-and-Forward Telemedicine Service User-satisfaction Questionnaire including video consultation items is needed to monitor also synchronous services as these expanded in the coronavirus disease 2019 pandemic.

Keywords

Telemedicine health care providers satisfaction validation of measuring instrument telehealth questionnaire

Introduction

Telemedicine includes store-and-forward (asynchronous) or real-time (synchronous) digital communication in clinical practice and enables referring health care providers (HCPs) to diagnose and possibly treat patients with advice of remote HCPs (i.e. medical specialists). Due to the proven cost-effectiveness and care quality of store-and-forward telemedicine, these services now permeate a wide variety of fields, e.g. cardiology, pulmonology, mental health, ophthalmology, and dermatology.^1–6 As this wider adoption of store-and-forward services accelerates, particularly post-coronavirus disease 2019 (COVID-19), it will become essential to understand HCPs' satisfaction with aspects of digital solutions offered by telemedicine organizations in order to guarantee telemedicine service quality and ongoing success.

Prior constructed HCPs' satisfaction telemedicine questionnaires often focus on comfort, observed privacy, ease of use, technical quality of sound and images, and lack details on validation and development.^7,8 It is well appreciated that telemedicine is not just a technological device or platform implementation process but concerns changes at various levels: technological, human, organizational, economical, cultural, and legal.⁹ Factors impacting adoption and long-term continuation of telemedicine services are for example (a) telemedicine platform technology (perceived ease of use or usefulness, reliability, maintenance, serviceable equipment), (b) appropriate, up to date training and (technical) support of HCPs, (c) implementation policy (bottom-up vs. top-down), (d) organizational infrastructure (organization stability, integration workflow, resources, culture, and management), (e) interaction and communication strategies of telemedicine organization with HCPs and among HCPs, (f) financing (insurance eligibility, reimbursement of care, and equipment costs).^10–17 All these factors may affect HCPs' satisfaction with and motivation to use and to continue telemedicine use. HCPs' dissatisfaction or negative experiences can influence their commitment to a telemedicine organization or may result in abandonment of the platform.¹⁸ Therefore, involved and satisfied referring and performing HCPs are crucial for the adoption and sustainability of telemedicine services.^18,19

Different types of intermediate telemedicine organizations have set up technical telemedicine platforms with some also offering liability and financial insurance policies as covered policies for HCPs. In the Netherlands, the largest store-and-forward health care organization that contracts HCPs to perform telemedicine services through a platform is Ksyos.²⁰ For Ksyos (a privately owned health provider) it is likewise crucial to continuously ensure the quality of its telemedicine services and to have insight into aspects that contribute to successful deployment as these services expand.

However, telemedicine services are not monitored on a structural basis, and information on the service quality of telemedicine vendors and platforms is still unavailable.^14,15 HCPs' satisfaction with a telemedicine organization and the services it delivers is key for its success.

To our knowledge, validated instruments that provide insight into factors that influence the satisfaction of HCPs with contracted telemedicine services are not yet available. Therefore, a Store-and-Forward Telemedicine Service User-satisfaction Questionnaire (SAF-TSUQ) was developed and validated for monitoring (both referring and performing) HCPs’ satisfaction with store-and-forward telemedicine services. Deployment of this questionnaire will give insight into why and what makes HCPs embrace these innovative services, and how store-and-forward telemedicine organizations can improve their services.

Methods

The Medical Ethical Commission of the Amsterdam UMC (location AMC) provided a waiver that further approval was not required. The recommendations of Langbecker et al.⁷ on the development of appropriate questionnaires to assess telemedicine services were used as input to our questionnaire.

Phase 1: questionnaire development

Questionnaire development took place between October 2018 and June 2019.

Step 1. A wide exploratory literature study was conducted (JvB) with snowballing and reversed snowballing to detect validated questionnaires related to the themes of interest as broadly as possible. Existing validated questionnaires within and without the telemedicine domain and in the grey literature were searched. Search string used in PubMed was “((‘Telemedicine’[Mesh] AND (‘Patient Satisfaction’[Mesh] OR ‘Personal Satisfaction’[Mesh])) AND ‘Surveys and Questionnaires’[Mesh])” and article type filter “Validation Study.” Additionally, the carrot2 clustering tool²¹ (a platform using the web and PubMed as search engines) was used by combining the terms “employee, satisfaction, questionnaire, telehealth” of which the top 50 matches were examined on relevance. Further, the snowballing procedure was applied to search for additional papers. Inclusion criteria for all three searches were: (a) includes a questionnaire and describes a validation method, (b) items within the questionnaire focus on the employee, job or service satisfaction, not merely on patient satisfaction, (c) paper is accessible. The resulting questionnaires were scored (strong, medium, and low) on construct validity, relevance to telemedicine services and employee satisfaction (JvB), and a Strengths–Weaknesses–Opportunities–Threats analysis was performed (JvB). Questionnaires with high scores were used as input for questionnaire development. At the start, patient satisfaction was included in the search string to examine whether these patient satisfaction questionnaires also contained elements relevant for the assessment of service providers. After assessing the poor applicability of these questionnaires specifically focusing on the patient perspective, we decided to exclude questionnaires solely focusing on patient satisfaction.

Step 2. A structured first focus group was performed (ET and JvB) with a domain expert of Ksyos (J. de Wit) and a human factor engineering expert (LWPP) to identify requirements of the questionnaire and to appreciate prior included constructs from the literature. Participants reached a high-level consensus on the selection of revealed constructs and assessed content validity. Items of the included constructs that might be relevant were forward translated to Dutch at the main level by one researcher (JvB), and then reviewed and reformulated to the telemedicine context by two researchers (ET and Y. Broekhuizen). Afterwards, the items were backwards translated to English by the two researchers (ET and Y. Broekhuizen) and when there was no consensus the items were discussed with the other researchers.

Step 3. Four telemedicine account managers and one customer service employee participated in a second structured focus group (ET and Y. Broekhuizen) to assess the face validity of the Dutch instrument. In this session, construct items were discussed regarding question formulation (wording) and applicability and consensus was reached by having participants rate all items on relevance (1 = relevant to 3 = not relevant). Questions could be added if an item was missing or changed if it required rephrasing.

Step 4. The feasibility and comprehensiveness of the questionnaire were assessed (ET and Y. Broekhuizen) in two pre-testing rounds. In the first round, 18 random HCPs (medical specialists, paramedics, general practitioners (GPs)) working with and participating in the Ksyos test panel were invited. Participants reviewed each question's phrasing and their applicability to the telemedicine context. The preliminary self-administered questionnaire was then built (ET, Y. Broekhuizen and M. Roskam-Mul) in an online questionnaire tool (LimeSurvey) to reveal further needed revisions, check for technical problems with filling in the questionnaire and measure completion time. For the second round, an invitation link was sent to another 21 HCPs who had not previously participated. HCPs completing the questionnaire received gift cards of €10 and €20, respectively. After integrating the feedback of the pre-testing rounds, a pilot version of the questionnaire, the SAF-TSUQ, was constructed.

Phase 2: validation SAF-TSUQ

Subjects and data collection

The SAF-TSUQ was distributed to 2179 Dutch HCPs of different specialties affiliated with Ksyos. HCPs were included when they had an active teleconsultation account registered before March 2019 in the Ksyos store-and-forward telemedicine platform and for whom an email address was known. On registration of their account, these affiliated HCPs sign an agreement with Ksyos that they will participate in scientific research and that Ksyos monitors the quality of the provided care. The number of HCPs' sampled from each specialty was disproportionally stratified to also include sufficient respondents in the small HCP groups, such as optometrists, pulmonologists, and somnologists. HCPs were arranged on the registration date. Therefore, systematic sampling was applied within these HCP groups to include an equal number of novel and experienced participants. All these included HCPs received an invitation email with a personal URL link for the online SAF-TSUQ. Participants with unfinished questionnaires received reminders after 1 and 9 weeks. HCPs could unsubscribe from the questionnaire by email. Anonymous data collection for internal validation of the SAF-TSUQ questionnaire ran from July to September 2019 (ET). Four gift vouchers of €50 were raffled among respondents. Completion time of the finished questionnaires was calculated and outliers above 1 h were excluded.

Statistical analysis

The questionnaire was analyzed (ET and JvB) using R (version 3.6.1).²² Questionnaires of HCPs reporting not using the telemedicine platform were excluded.

Psychometric analyses were conducted to assess construct validity and the internal consistency reliability of the questionnaire. Items with answering categories other than the standard format or items not answered by ≥95% due to the prior screener question were excluded for psychometric analysis. To keep “I do not know or not applicable” responses in the psychometric analyses they were recoded as “3,” reflecting the neutral answer category.

The data set was assumed to be suitable for factor analysis (FA) if the Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy was at least 0.60 and Bartlett's test of sphericity was significant (p < 0.05).^23,24 Principal component analysis (PCA) with direct oblimin rotation, common FA and confirmative FA (CFA) were conducted. Items in the common FA needed a minimum factor load of 0.40, items below this threshold did not fit to a scale. The CFA was performed on the original theorized model (six factors); loadings needed to exceed 0.45. Spearman's rank correlation coefficient was calculated to find inter-item overlap between items (r_s > 0.70 indicating one of the items is redundant). Cronbach's alpha (α > 0.70) and item-total correlations (ITC > 0.40) were calculated to assess the internal consistency of the different questionnaire sections.

Results

Phase 1: questionnaire development

Figure 1 displays the number of items in each of the developing steps.

Step 1. The literature study resulted in 18 articles describing 34 questionnaires. Two validated questionnaires (i.e. construct validity confirmed, validated question set, strong survey to item correlation, validated scaling, and telemedicine and employee-related items) were further explored: the satisfaction questionnaire of the Basque Health Service and the Telehealth Usability Questionnaire (TUQ).^25,26 The TUQ²⁵ (21 items) is applicable to different telehealth system types and includes usability components (usefulness, ease of use, effectiveness, reliability, and satisfaction) from the Technology Acceptance Model (TAM),¹⁶ Post-Study System Usability Questionnaire (PSSUQ)²⁷ and Telemedicine Satisfaction Questionnaire (TSQ).²⁸ The Basque Health Service Questionnaire²⁴ consists of 52 staff satisfaction aspects (e.g. working conditions, pay, and policy and strategy). All items of these questionnaires were combined in a list of 73 potentially relevant items for our questionnaire.

Step 2. After the first focus group 57 items and one improvement item remained (Figure 1). Patient–clinician interaction questions of the TUQ (like I could easily talk to the clinician/patient using the telehealth platform) were considered as not relevant in the store-and-forward setting and excluded.

Step 3. Participants in the second focus group rated 17 out of 57 items as not relevant in the telemedicine context (e.g. items related to job security and working conditions). It was decided to keep these items for completeness as the participants in the focus group did not represent the entire target population. Four items were added (occupation, frequency of use, self-reported computer skills, recommend to colleague), one deleted (information systems used in your workplace and other computer applications, are useful tools) and two reformulated (communication with telemedicine organization, telemedicine organization is an innovative and renewing company).

Step 4. In pre-testing 10/18 (first round) and 8/21 (second round) HCPs participated. In the second round, excluding one outlier, the average completion time of the seven questionnaires was 8.4 min. Four background items were added (technology adoption, contacted and informed by telemedicine organization, and how they view the telemedicine organization (supplier, employer, partner, etc.)), and three items deleted (conditions regarding staff leave, employment stability, and acceptable way to receive health care).

Figure 1.

Steps in development and validation of the Store-and-Forward Telemedicine Service User-satisfaction Questionnaire (SAF-TSUQ). “+” indicates final open-ended question to provide any feedback or suggestions for improvement of the questionnaire. “ⱡ” indicates questions focusing on the working environment, not applicable to the telemedicine context or not based on the interaction of the patient with a clinician or between clinicians. “*” indicates factors such as age, gender, occupation, type of telemedicine services used, frequency of use, self-reported computer skills and technology adoption.

The pilot SAF-TSUQ included 61 items and a final open-ended question to provide any feedback or suggestions for improvement of the questionnaire. HCPs could also provide free-text comments in a separate textbox at the end of each section. Three communication questions and one working condition question did not follow the standard answering format and were proposed by the telemedicine organization for acquiring extra information. The payment section consisted of a screener question whereafter the subsequent three questions were only asked if the HCP received any reimbursement for providing the telemedicine service. Forty-four of the 49 items followed standard response options using a 5-point Likert scale ranging from 1 = “strongly disagree” to 5 = “strongly agree” and the non-substantive singular option “I do not know or not applicable” (NA). The 5-point Likert questions were all mandatory except for three reimbursement items. The “NA” option was introduced to adapt the questionnaire to HCPs of all disciplines.

Phase 2: validation SAF-TSUQ

Subjects and data collection

In total 224 of 2179 invited HCPs (10%) completed the questionnaire including 43 respondents reporting that they were not sure whether they worked with the Ksyos platform. The data of the remaining 181 HCP that completed all 61-items was eligible for analyses. Survey completion time was 9 min and 43 s on average (n = 173; SD: 6.6 min; Min = 3 min, Max = 46 min). Eight outliers with completion times of 1 (n = 6), 6 (n = 1), or 15 (n = 1) hours were excluded from this calculation.

Table 1 summarizes the characteristics of the responding 181 HCPs; 55% were female, 29% aged between 55 and 64 years, 34% were GPs, and 97% rated their computer skills as sufficient, good, or excellent. Most respondents worked daily or weekly (62%) with the platform and 38% had been working with the telemedicine organization for >5 years.

Table 1.

Characteristics of questionnaire respondents n = 181.

	Respondents (%)
Age (years)
18–24	5 (2.8)
25–34	23 (12.7)
35–44	45 (24.9)
45–54	48 (26.5)
55–64	53 (29.3)
65+	7 (3.9)
Gender
Female	99 (54.7)
Male	81 (44.8)
Prefer not to say	1 (0.5)
Occupation^a
General practitioner	61 (33.7)
Medical specialist	48 (26.5)
Physician assistant	7 (3.9)
General practice assistant	26 (14.4)
Optometrist	26 (14.4)
Care manager	2 (1.1)
Other	13 (7.2)
Frequency of use
Daily	41 (22.7)
Weekly	71 (39.2)
Monthly	35 (19.3)
A few times a year	32 (17.7)
Never	2 (1.1)
Working with telemedicine organization
<6 months	6 (3.3)
6–12 months	21 (11.6)
1–3 years	54 (29.8)
3–5 years	32 (17.7)
>5 years	68 (37.6)
Self-reported computer skills
Poor	6 (3.3)
Sufficient	65 (35.9)
Good	80 (44.2)
Excellent	30 (16.6)

Total response percentage exceeds 100% due to multiple answers allowed.

At least five respondents rated each questionnaire item as “NA,” with the highest percentage for Q17 (72%). Eighteen items had ≥25% “NA” responses which mostly related to the “organization” and “working conditions” themes. Roughly a quarter of the respondents (25%) indicated that Q30 was “NA” of whom 18% were GPs. Optometrists answered only five questions with ≥25% “NA.”

HCPs' remarks and suggestions for improvement mainly concerned employee and payment-related questions. These questions had a high percentage of “NA” as respondents did not feel they were employees of the telemedicine organization but consider the organization as a supply partner. Merely 6% of the respondents consider the telemedicine organization as their employer.

Statistical analysis

Spearman's rank correlation coefficient indicated five items as redundant. None of the items were extremely skewed. Three payment questions were completed by <95% of the respondents (62%) and therefore excluded from psychometric analyses. Responses to five items did not comply with the standard answering format and 12 general questions were removed before psychometric analysis. The remaining data set consisted of 41 psychometric items (Appendix 1 presents the Dutch original items). Ten out of 12 “not relevant” focus group items (step 3) had high percentages of “NA” and/or redundancy answers in the psychometric analysis.

Data was suitable for FA according to the overall KMO value (0.90) and Bartlett's test of sphericity (p < 0.05). All individual items had a KMO of >0.80. PCA confirmed a six-factor solution. Based on common FA, Q14 did not support any of the factors sufficiently (highest factor loading = 0.28). Six items had a factor loading between 0.30 and 0.40. The CFA showed sufficient loadings (>0.45) for almost every single item except for items Q8 (0.44) and Q23 (0.40).

As shown in Table 2, ITC was considered sufficient (>0.40) for all items. Factors 1 and 2 had the lowest Cronbach's alpha (0.77) and factors 3, 4, 5 the highest (0.89). None of the items had to be removed to increase the Cronbach's alpha to the 0.70 threshold.

Table 2.

Results SAF-TSUQ validation.

	Item description	NA ≥ 25%	r_s > 0.70	CFA loading	ITC	α
Factor 1: Training		α^t (CI) = 0.77 (0.72–0.82)
Q1	The training and explanation offered by [telemedicine organization] are sufficient to be able to use the [telemedicine platform] in my work	—	—	0.55	0.71	0.70
Q2	I am satisfied with the possibilities for additional and/or continuing education offered by [telemedicine organization]	45.3^a	—	0.46	0.61	0.75
Q3	I am satisfied with the tasks I perform within my profession in the [telemedicine platform]^b	—	—	0.71	0.76	0.69
Q4	By using the [telemedicine platform] I am able to develop my knowledge and skills adequately	—	—	0.67	0.75	0.69
Q5	I believe that [telemedicine organization] meets my wishes and expectations for telemedicine	—	—	0.76	0.78	0.68
Factor 2: Communication		α^t (CI) = 0.77 (0.72–0.83)
Q6	The information and manuals that are provided to me by [telemedicine organization] and the instructions given to perform my work are sufficient	—	—	0.79	0.80	0.70
Q7	I receive sufficient relevant information regarding any decisions, projects and activities of [telemedicine organization] that may affect me	—	—	0.78	0.82	0.71
Q8	I know how to contact [telemedicine organization] if I have questions about my work for [telemedicine organization]	—	—	0.44^c	0.64	0.75
Q9	There are sufficient information channels for suggestions or complaints at [telemedicine organization]	—	—	0.68	0.81	0.68
Factor 3: Organization, policy and strategy		α^t (CI) = 0.89 (0.86–0.91)
Q10	Good work is recognized and valued at [telemedicine organization]	46.4^a	—	0.69	0.70	0.84
Q11	I have a good working relationship with [telemedicine organization]	—	—	0.72	0.72	0.84
Q12	I am satisfied with the opportunities I have to participate in the (daily) decision making at [telemedicine organization] related to my work and working environment^b	44.2^a	—	0.61	0.64	0.84
Q13	[telemedicine organization] listens to my complaints or suggestions for improvement and takes these into account	44.8^a	—	0.52	0.57	0.85
Q14	The work processes for [telemedicine organization] are well organized	40.3^a	—	0.59	0.62	0.84
Q15	Collaborative processes between the various stakeholder groups of [telemedicine organization] are well documented^b	65.2	—	0.60	0.64	0.84
Q16	I have clear work objectives at [telemedicine organization]^b	52.5^a	—	0.54	0.59	0.84
Q17	The management of [telemedicine organization] functions to my satisfaction in terms of planning and organization^b	71.8	—	0.64	0.66	0.85
Q18	It is clear to me what the objectives, projects and intended results are for [telemedicine organization]	54.1^a	—	0.52	0.60	0.85
Q19	I know enough about the mission and vision of [telemedicine organization]	45.3^a	—	0.50	0.60	0.84
Q20	[telemedicine organization] is open to suggestions and contributions from its health care provider clients^b	43.1^a	—	0.73	0.76	0.84
Q21	[telemedicine organization] is an innovative and renewing company	30.9^a	0.71 (Q22)	0.71	0.73	0.84
Q22	[telemedicine organization] makes good use of new technologies when it comes to exchanging experiences, disseminating information and professional guidelines, etc.^b	43.6^a	0.71 (Q21)	0.71	0.72	0.84
Factor 4: Interaction telemedicine platform		α^t (CI) = 0.89 (0.86–0.91)
Q23	I have sufficient and appropriate resources (such as computers, holterdevices, equipment, etc.) to perform my (daily) work for [telemedicine organization]	—	—	0.40^c	0.49	0.88
Q24	It is easy to use the [telemedicine platform]	—	0.77 (Q25)	0.73	0.78	0.86
Q25	It is easy to learn how to use the [telemedicine platform]	—	0.77 (Q24)	0.72	0.76	0.86
Q26	The way I interact with the [telemedicine platform] (using input fields, menus, transitions, etc.) is pleasant	—	0.75 (Q27)	0.83	0.83	0.85
Q27	I like using the [telemedicine platform]^b	—	0.75 (Q26)	0.86	0.83	0.85
Q28	The [telemedicine platform] is simple and easy to understand^b	—	—	0.73	0.76	0.86
Q29	The [telemedicine platform] includes all functionalities I would expect	—	—	0.71	0.73	0.86
Q30	Whenever I make a mistake within the [telemedicine platform] I can easily and quickly fix it	25.4^a,d	—	0.51	0.60	0.88
Q31	The [telemedicine platform] gives error messages that clearly state how I can fix problems	40.9^a	—	0.65	0.74	0.86
Factor 5: Usage of telemedicine platform		α^t (CI) = 0.89 (0.86–0.91)
Q32	The [telemedicine platform] improves access to health care services	—	—	0.81	0.84	0.85
Q33	The [telemedicine platform] offers me what I need to deliver better care	—	—	0.78	0.82	0.86
Q34	I became productive quickly while working with the [telemedicine platform]	—	—	0.67	0.74	0.87
Q35	I think the care provided with the [telemedicine platform] are the same as in-person visits	—	—	0.56	0.69	0.88
Q36	I would use the [telemedicine platform] again	—	—	0.64	0.66	0.87
Q37	I would recommend the [telemedicine platform] to a colleague	—	0.76 (Q38)	0.89	0.89	0.84
Q38	Overall, I am satisfied with the [telemedicine platform] I use	—	0.76 (Q37)	0.83	0.82	0.86
Factor 6: Working conditions		α^t (CI) = 0.86 (0.82–0.89)
Q39	I am satisfied with the conditions in terms of working hours and working days within [telemedicine organization]^b	59.7	0.80 (Q40)	0.89	0.91	0.60
Q40	The circumstances regarding flexibility in working hours within [telemedicine organization] allow me to achieve a good work-life balance^b	63.5	0.80 (Q39)	0.90	0.91	0.55
Q41	I believe that the working conditions (my working day, leave, etc.) are better at [telemedicine organization] than in the regular health care sector^b	68.5	—	0.68	0.82	0.75

SAF-TSUQ: Store-and-Forward Telemedicine Service User-satisfaction Questionnaire; CFA: confirmative factor analysis; NA: “I do not know or not applicable” responses ≥ 25%; r_s: Spearman's rank correlation coefficient between items > 0.70; ITC: item-total correlation; α^t: Cronbach's alpha on total scale; CI: confidence interval; α: Cronbach's alpha of scale if item deleted.

<25% responses were missing for occupation optometrist.

Rated as not relevant (3) in the second focus group but retained in the questionnaire.

= Loading <0.45.

<25% responses were missing for occupation general practitioner.

Discussion

This study presents our development and validation approach of the 61-item SAF-TSUQ instrument measuring HCPs' satisfaction with contracted store-and-forward telemedicine organizations (their services and used platform). The SAF-TSUQ proved to be a reliable and valid questionnaire. Internal consistency of SAF-TSUQ was sufficient and FA confirmed six factors: training, communication, organization policy and strategy, interaction telemedicine platform, usage telemedicine platform, and working conditions.

Our study showed that aspects related to “organization, policy and strategy” of a telemedicine organization and “working conditions” were rated as having low relevance within the second focus group and in the validation phase. This is in line with the fact that very few respondents consider the telemedicine organization as their employer but mainly as a supply organization. Further, most questions on “organization, policy and strategy” generated over 25% “NA” responses. In contrast, optometrists (the largest group to complete these questions) generated over 25% “NA” responses for only 2 of the 13 “organization, policy and strategy” questions. This indicates that certain HCPs are more informed on these organizational issues or that these issues are more applicable to their health care setting. Whether or not to retain these “organization, policy and strategy” and “working conditions” questions in future applications of SAF-TSUQ depends on the aim of the study and the settings of invited HCPs. Further, certain SAF-TSUQ items were redundant. For example, Q21 and Q22 both focused on the innovativeness of the telemedicine provider. Certain questions overlapped in flexibility in working hours (Q39, Q40), or in how easy (Q24, Q25) or enjoyable (Q26, Q27) it is to use the platform. Lastly, Q37 and Q38 were both related to satisfaction with the platform and the possibility to recommend it to others. Factors of these redundant items had relatively high internal consistency (≥0.77). Deleting an item from one of these pairs results in a slightly lower Cronbach's alpha. These items were therefore retained in the SAF-TSUQ FA. Removal of these redundant or high “NA” items to shorten the questionnaire could be decided in discussion with stakeholders.

Different instruments have been developed to assess HCPs' satisfaction with telemedicine, but these focus merely on privacy, comfort, ease of use, or technical sound and image quality.⁸ Moreover, few of these instruments have been validated and their psychometric properties or construction details are often not reported.^7,8 Further, these studies had small respondent numbers, and mainly focus on usefulness, acceptance, overall satisfaction, technical evaluation of a telemedicine system. None of these combine telemedicine platform items with supplier related aspects like training, communication, organization policy and strategy, and working conditions such as the SAF-TSUQ. The used TUQ²⁵ focuses on telehealth usability components from the TAM,¹⁶ PSSUQ,²⁷ and TSQ,²⁸ and the Basque Health Service Questionnaire²⁶ focuses specifically on employee satisfaction in health care settings where SAF-TSUQ combines these themes. The SAF-TSUQ items have some overlap with the UTAUT model¹⁷ in behavioral intention, facilitating conditions, attitude toward using technology, and performance and effort expectancy (similar to perceived usefulness and ease of use in the TAM model¹⁶). Self-efficacy, social influence, and anxiety items of the UTAUT model were not revealed as relevant items for the SAF-TSUQ. These items seem to be less applicable to already deployed telemedicine platforms as in our study wherein most respondents had sufficient, good, or excellent computer skills, worked daily or weekly with the platform or had collaborated with the telemedicine organization for several years. These items might be considered in future studies with novel users. Law et al.²⁹ developed a conceptual framework of five elements (professionals' demographics, care settings, motivations, experiences, and overall satisfaction) to evaluate HCPs’ satisfaction in utilizing telemedicine. Only 12 physicians validated this framework, whereas we validated the SAF-TSUQ among 181 HCPs. The framework of Law and co-authors was published after we had developed SAF-TSUQ but despite some overlap in items, it focuses on HCPs’ satisfaction and motivation for utilizing telemedicine in general (patient benefits, professionals time, care quality etc.) and the care setting, whereas SAF-TSUQ also evaluates HCPs' satisfaction with the services of the contracted telemedicine organization and used platform.

The first strength of this study is that we applied a four-step structured methodology to develop the SAF-TSUQ-instrument, consisting of both qualitative and quantitative phases. Second, we included a mix of HCPs from a variety of (telemedicine) specialties to assure that SAF-TSUQ is applicable to a wide spectrum of disciplines. Third, we reported the psychometric properties and construction details of our SAF-TSUQ instrument. One of the limitations is that the current SAF-TSUQ is mainly based on store-and-forward (asynchronous) contexts of Ksyos. Second, bias in the selection of HCPs may have occurred, despite disproportionate stratification and systematic sampling from a wide variety of specialties. Third, some general, communication and financial compensation items of SAF-TSUQ were necessarily excluded from the psychometric analysis despite reimbursement being a major telemedicine service adoption barrier.^10–15 These questions should be reconsidered for inclusion depending on health care contexts. Fourth, if the SAF-TSUQ is used in another national setting, correct linguistic translation and cultural adaptation are needed to reassure the content validity of SAF-TSUQ.³⁰ SAF-TSUQ should be nationally tailored and validated again when used in English or other languages than Dutch.

Organizations that supply telemedicine services to HCPs have to proactively assess HCPs' needs and issues or gaps on their performance and platform as part of a continuous quality improvement cycle for which the SAF-TSUQ can be used. The outcomes can assist these organizations in identifying important issues as perceived by HCPs and developing strategies to address these issues and improve their telemedicine service quality. SAF-TSUQ can be used as a baseline measure against which the results of subsequent SAF-TSUQ administrations can be compared to monitor performance over time, making continuous quality improvement possible. Except for telemedicine organizations, findings may be of relevance and value to HCPs, decision-makers, health insurance companies, public health inspections and governments. Health insurance companies can for example enforce major telemedicine organizations to collect feedback on HCPs’ satisfaction with the service and use this as a precondition for reimbursement or take action on the poor quality provided by telemedicine organizations. In the Netherlands, Ksyos is the largest store-and-forward telemedicine organization. Especially, in other countries with many different telemedicine organizations, the rollout of a continuous quality monitoring system and a telemedicine service quality mark is desired to guarantee telemedicine organizations performance at national levels. Future studies should focus on such telemedicine service quality marks and demonstrate if the SAF-TSUQ can also be used for these purposes.

This study was conducted in summer 2019 and due to its focus on store-and-forward telemedicine, video consultation (synchronous) interaction items between patients–clinicians and clinicians themselves were excluded. At the onset of 2020, COVID-19 stimulated health care organizations and providers to take up and integrate telemedicine solutions in their practices to reduce virus transmission and the burden on hospital-based consultations. This sudden uptake of telemedicine solutions boosted the number of telemedicine organizations and types of (store-and-forward and synchronous) services. This swiftness in telemedicine uptake (activities that usually take months or years are done in weeks or even days) emphasizes the need for decision-makers, health insurance companies, public health inspection and governments to continuously assess and monitor the quality of synchronous services also. We, therefore, intend to adapt SAF-TSUQ, that after validation will allow continuous quality monitoring of synchronous telemedicine services as well.

Conclusion

SAF-TSUQ provides insight into HCPs' satisfaction with store-and-forward telemedicine organizations, including the platform and service aspects in a systematic and standardized way. Findings of the questionnaire highlight aspects that require extra consideration or improvement and best practices on why HCPs embrace the telemedicine service. Continued efforts in such a monitoring cycle will contribute to wider adoption of these store-and-forward services and are essential to guarantee telemedicine quality and its success. An adapted SAF-TSUQ including video-consultation items might be used to evaluate and improve synchronous telemedicine services also.

Footnotes

Acknowledgements

The authors would like to thank all experts participating in the focus groups and the pre-testing steps, J. de Wit for her support in the questionnaire development process, Y. Broekhuizen for contributing to the questionnaire construction, M. Roskam-Mul for the LimeSurvey support, all health care providers who completed the questionnaire, and Ksyos for sending the questionnaire.

Author contributions

Study conception and questionnaire design were created by ET, JvB, and LWPP. Acquisition of data was performed by ET. Analysis and interpretation of data were performed by ET, JvB, and LWPP. ET drafted the first manuscript version. LWPP was involved in writing the manuscript from the first version onwards. JvB, MWMJ, and LW were further involved in critical revision of the manuscript.

Declaration of conflicting interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: ET is employed by Ksyos and LW is the founder, (co)owner and director of Ksyos. The remaining authors state no conflicts of interest.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Esmée Tensen

Appendix 1: Dutch SAF-TSUQ items

References

Vodička

Naji

Zelko

. The role of telecardiology in dealing with patients with cardiac rhythm disorders in family medicine—systematic review. Zdr Varst 2020; 59: 108–116.

Clinical effectiveness and cost-effectiveness of teledermatology: where are we now, and what are the barriers to adoption?

J Am Acad Dermatol 2020; 83: 299–307.

Delgoshaei

Mobinizadeh

Mojdekar

, et al. Telemedicine: a systematic review of economic evaluations. Med J Islam Repub Iran 2017; 31: 113.

Langarizadeh

Tabatabaei

Tavakol

, et al. Telemental health care, an effective alternative to conventional mental care: a systematic review. Acta Inform Med 2017; 25: 240–246.

Di Cerbo

Morales-Medina

Palmieri

, et al. Narrative review of telemedicine consultation in medical practice. Patient Prefer Adherence 2015; 9: 65–75.

Seguí

Parella

García

, et al. A cost-minimization analysis of a medical record-based, store and forward and provider-to-provider telemedicine compared to usual care in catalonia: more agile and efficient, especially for users. Int J Environ Res Public Health 2020; 17: 2008.

Langbecker

Caffery

Gillespie

, et al. Using survey methods in telehealth research: a practical guide. J Telemed Telecare 2017; 23: 770–779.

AlDossary

Martin-Khan

Bradford

, et al. A systematic review of the methodologies used to evaluate telemedicine service initiatives in hospital facilities. Int J Med Inform 2017; 97: 171–194.

Alami

Gagnon

Fortin

. Some multidimensional unintended consequences of telehealth utilization: a multi-project evaluation synthesis. Int J Health Policy Manag 2019; 8: 337–352.

, et al. Only 15% of FPs report using telehealth; training and lack of reimbursement are top barriers. Am Fam Physician 2016; 93: 101.

, et al. Telehealth in health centers: key adoption factors, barriers, and opportunities. Health Aff 2018; 37: 1967–1974.

, et al. Evaluating barriers to adopting telemedicine worldwide: a systematic review. J Telemed Telecare 2018; 24: 4–12.

13.

Segato

Masella

. Telemedicine services: how to make them last over time. Health Policy Technol 2017; 6: 268–278.

, et al. Report on EU state of play on telemedicine services and uptake recommendations, https://ec.europa.eu/health/sites/health/files/ehealth/docs/ev_20171128_co09_en.pdf (November 2017, accessed January 2021).

15.

Sisk

Alexander

Bodnar

, et al. Pediatrician attitudes toward and experiences with telehealth use: results from a national survey. Acad Pediatr 2020; 20: 628–635.

16.

Davis

. User acceptance of information technology: system characteristics, user perceptions and behavioral impacts. Int J Man Mach Stud 1993; 38: 475–487.

17.

Venkatesh

Morris

Davis

, et al. User acceptance of information technology: toward a unified view. MIS Quart 2003; 27: 425–478.

18.

Kissi

Dai

Dogbe

, et al. Predictive factors of physicians’ satisfaction with telemedicine services acceptance. Health Informatics J 2020; 26: 1866–1880.

19.

Permwonguswa

Hajar

Cook

, et al. Provider satisfaction with telehealth: exploring variation across structural and demographic factors. Heal Manag Policy Innov 2018; 3. https://hmpi.org/2018/03/14/provider-empowerment-and-satisfaction-with-telehealth-exploring-variation-across-structural-and-demographic-factors/ (accessed January 2021).

20.

Ksyos health care organization, https://www.ksyos.nl/en/ (accessed January 2021).

21.

Carrot2, https://search.carrot2.org/#/search/web (accessed October 2018).

22.

R Core Team. R: a language and environment for statistical computing, Vienna, Austria, https://www.R-project.org/ (accessed October 2019).

23.

Bartlett

. Tests of significance in factor analysis. Br J Stat Psychol 1950; 3: 77–85.

24.

Kaiser

Michael

. Little jiffy factor scores and domain validities. Educ Psychol Meas 1977; 37: 363–365.

25.

Parmanto

Lewis

Jr Graham

, et al. Development of the telehealth usability questionnaire (TUQ). Int J Telerehabil 2016; 8: 3–10.

26.

Matthies-Baraibar

Arcelay-Salazar

Cantero-González

, et al.

Is organizational progress in the EFQM model related to employee satisfaction?

BMC Health Serv Res 2014; 14: 468.

27.

Lewis

. IBM Computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum Comput Interact 1995; 7: 57–78.

28.

Yip

Chang

Chan J, et al. Development of the Telemedicine Satisfaction Questionnaire to evaluate patient satisfaction with telemedicine: a preliminary study. J Telemed Telecare 2003; 9: 46–50.

29.

Law

Cronin

Schuller

, et al. Conceptual framework to evaluate health care professionals’ satisfaction in utilizing telemedicine. J Am Osteopath Assoc 2019; 119: 435–445.

30.

Beaton

Bombardier

Guillemin

, et al. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine (Phila Pa 1976) 2000; 25: 3186–3191.