Sage Journals: Discover world-class research

Abstract

Background:

The use of telemedicine for continuity of care during disruptions in health care delivery and routine primary care is now well known. Insufficient scientific evidence from assessing telemedicine persists, with widespread use of nonvalidated questionnaires resulting in the inability to pool data on the quality of telemedicine encounters from numerous small-sample studies. This study examines the comprehensiveness of the recently developed Telemedicine Assessment Toolkit (TAT), among articles published after the toolkit was created.

Methods:

We conducted a PubMed search for articles that used questionnaires to assess telemedicine encounters, published between November 1, 2021, and July 31, 2023. We extracted individual questions from nonvalidated questionnaires and analyzed them to determine their similarity with TAT items. We used a statistical proportion test to see if rates of inclusion were similar across the initial TAT questionnaires and the follow-up set. We calculated p-values for proportions.

Results:

The database search had an initial yield of 277 articles in which there were 21 articles each with its nonvalidated questionnaire and 348 individual questions. Test of proportions revealed no statistical difference between initial and follow-up articles tested, adjusted for multiple tests. Further analysis found that 85.6% (298 of 348) of the questions used in the follow-up articles match items in the TAT.

Conclusion:

These results provide insight into the comprehensiveness and potential suitability of the TAT as a toolkit for telemedicine assessment. Future work to standardize TAT involving expert, cognitive testing, and weighting toward a single score would improve telemedicine encounter assessment.

Introduction

Telemedicine is the exchange of medical information from one site to another via electronic communications to improve a patient’s clinical health status.¹ Despite the growing adoption of audiovisual telemedicine, insufficient scientific evidence assessing telemedicine is a persistent problem.² Despite several validated questionnaires specific to telemedicine such as the Telemedicine Satisfaction Questionnaire (TSQ), the Telehealth Usability Questionnaire (TUQ), the Telemedicine Satisfaction and Usability Questionnaire (TSUQ), the Telehealth Satisfaction Scale (TESS), and the Patient Assessment of Communication in Telemedicine (PACT), their widespread use has been limited for a variety of reasons. Neglect of predictors for telemedicine encounter success,^3,4 lack of inbuilt mechanisms for continual improvement,⁵ and lack of comprehensive assessment⁵ are examples of barriers to using validated questionnaires.

The identified drawbacks among validated questionnaires have led to the widespread use of nonvalidated, customized questionnaires to assess audiovisual telemedicine encounters.^5–7 Eschewing uniformly accepted measures reduces the capacity to pool samples and data from studies having similar designs but lacking. Also, nonvalidated questionnaires might exhibit technical issues and shortcomings such as lack of response variability and poor reliability. Other shortcomings include inappropriate scale design, lack of construct validity, inadequate selection of constructs, and ambiguous questions.

The Telemedicine Assessment Toolkit (TAT) has been developed to enhance the comprehensive assessment of audiovisual telemedicine encounters. A two-step conceptual approach was used to first identify potential domains and then to subsequently identify subdomains by qualitative analysis of a pool of questions from studies published from January 1, 2016, to November 1, 2021.⁸ The TAT toolkit is composed of 11 domains to assess audiovisual telemedicine encounters: “comparison to standard (in-person) care,” “patient costs,” “patient feeling,” “patient perspectives,” “patient–provider interaction,” “patient satisfaction,” “privacy,” “qualitative feedback,” “technology,” “telemedicine readiness,” and “usability.” These are further divided into 26 subdomains: audio quality, benefits, clarity of communication, clinical examination, comfort with encounter, comparison with standard care, convenience, device usability, ease of setup, ease of use, equipment readiness, future use intent, improvement suggestions, likelihood to recommend others, money saved, overall satisfaction, patient–provider relationship, preference, previous experience, provider’s skills and knowledge, respect for privacy, satisfaction with process, satisfaction with system, shared decision making, time saved, and video quality.

The TAT toolkit has a core block of 30 questions, preceded by a block aimed at retrieving demographic/patient characteristics (Block 1), and followed by a customizable clinical outcomes block (Block 3).

TAT was designed to apply to a wide range of telemedicine topics and thus be broadly accepted by researchers, with relevant questions (item measures) placed within coherent domains to provide a comprehensive set of measures to assess audiovisual telemedicine encounters.

The goal of this study is to examine the comprehensiveness of the TAT within the articles published between November 1, 2021, and July 31, 2023.

Methods

We conducted a systematic search of the PubMed database to identify articles that used questionnaires to assess patients’ experiences with synchronous, audiovisual telemedicine consultations. To achieve this, we followed the same methodology used in the development of the TAT,⁸ and employed a search strategy encompassed terms such as “telemedicine,” which is a medical subject heading for enabling semantic search, combined with the keywords “surve*,” “questionnaire*,” or “poll*” by a Boolean operator AND. We refined the search strategy by applying the subsequent filters: species—human, age +19, and English articles. The time frame targeted articles published between November 1, 2021, and July 31, 2023, which we term the “follow-up cohort” of articles published after the creation of the TAT. Two authors (G.D.) and (S.H.) screened the titles and abstracts of the identified articles, removing those that meet the exclusion criteria. Two authors (G.D.) and (S.H.) reviewed the full text of the remaining articles to ensure they were eligible articles, specifically primary research about patient experience of telemedicine. Finally, articles that used validated measures or were included in the initial development of the TAT were removed. Figure 1 illustrates the screening process.

FIG. 1.

Articles screening process.

The individual questions from the questionnaires used in the included articles were assessed to determine similarities with items from the TAT. Two authors (G.D. and R.A.) reviewed the questions and categorized each question as either similar to the thirty TAT core items or not within the scope of TAT. Items considered similar in text or meaning were then categorized with the associated TAT item and subdomain from the related TAT literature. To categorize the question stems, author 1 (G.D.) reviewed the nonsimilar items coded by author 2 (R.A.). Disagreements were resolved by consensus based on a discussion of the keywords associated with each TAT item. Items that were considered demographic or routinely provided by the system during usual care were categorized into Block 1, while clinical assessment items were categorized as Block 3 items. We identified questions not within the scope of telemedicine encounters by consensus among the authors (Table 3). For question matching items in the TAT, we combined the following TAT pairs: TAT 1 and 2, TAT 10 and 11, TAT 12 and 13, as well as TAT 17 and 18, to be consistent with the analysis grouping used in creating the TAT.⁸ For each question type, we used a statistical proportion test to see if rates of inclusion were similar across the initial set of questionnaires and the follow-up set. We calculated p-values for proportions and further compared them to values after adjusting for the number of tests. Statistical proportion test analyses were performed using R (version 4.2.0, R Foundation for Statistical Computing, Vienna, Austria). A p-value of 0.05 was considered significant. We also investigated which and how many TAT questions matched questions from one of the previously validated assessment tools.

Results

We identified 277 articles from the initial search. Screening of titles and abstracts removed 226, reading full text removed another 20. Of the remaining 31 full-text articles that used questionnaires, two articles were used among the initial set of articles used in creating the TAT, while eight articles (25.8%) used validated questionnaires specific for telemedicine and excluded these from analysis, being outside the objective of the current study. Ultimately, we found 21 articles each with its own nonvalidated questionnaire and 348 individual questions.

In reviewing the 348 individual questions, we found that 298 (85.6%) had similar meaning to question items in the TAT. Twenty questions (5.8%) elicit routine health system information collected in the precursor (Block 1) of the TAT. Thirty questions (8.6%) of the 348 questions are considered outside the scope of TAT. For instance, the question: Were you provided with a call back number in case you had an inquiry, was considered a question about “follow-up” activities not applicable for most telemedicine encounters in which respondents are surveyed before the end of the encounter session. Tables 1–3, respectively, list the questions for each type.

Table 1.

Description of Individual Questions That Match TAT with Authors’ Names, Publication Year, Specialty, and Country of Study

TAT matched item	Question	Article reference
TAT 1	Was your encounter successful in addressing your health needs today?	Dodeja, 2023, Cardiology, USA⁹
TAT 11	Any technical issues?
TAT 9	Are you likely to choose telephone/video encounters in the future instead of coming to the clinic?
TAT 22	It was easy to use the system to fill out the questionnaires at home.	Listorti, 2023, Rheumatology, Italy¹⁰
TAT 23	It was easy to learn how to use the system to fill the questionnaires at home.
TAT 9	Telemedicine improves access to health services.
TAT 1	Telemedicine has allowed me to receive specialized care that I otherwise would not have been able to access.
TAT 25	Telemedicine has saved me the travel time to go to the hospital.
TAT 1	Telemedicine meets my health needs.
TAT 27	I believe that communications with staff through the telemedicine system are equivalent to communications in face-to-face meetings.
TAT 29	I enjoy communicating with medical staff using telemedicine tools.
TAT 17	I believe that telemedicine tools improve my ability to talk to doctors (e.g. asking questions, telling concerns, etc.).
TAT 1	Telemedicine is an acceptable way to receive health services.
TAT 19	I am overall satisfied with the experience I have had in telemedicine.
TAT 10	Indicate which of these benefits you believe you have benefited most thanks to the telemedicine experience with the rheumatology department (you can choose more than one):A.Better medical treatment experience B. Saving of travel time, C. Saving of waiting time D. Immediate and direct communication through telephone contact with healthcare professionals E. Convenience for the home delivery of drugs F. Greater emotional and psychological support received from medical staff G. Other.
TAT 7	Indicates on a scale from 1 (disagree) to 5 (completely agree) which of these channels you would like to use in the future to receive health services from the rheumatology department: Video call with your reference doctor; Phone call from medical staff; Phone call from nursing staff; Home delivery of drugs; Interactions by e-mails or messages on the telephone with medical staff; Other.
TAT 6	If the rheumatology department reproposed you a telemedicine experience, how would you behave? A. I would definitely accept B. Maybe I would accept C. I don’t think I would accept D. I definitely wouldn’t accept.
TAT 9	(for those who answered question 24 with “I would definitely accept” or “Maybe I would accept”) If the rheumatology department reproposed you a telemedicine experience, please indicate what you would prefer: A. Keep the current frequency of access to the department, but with the possibility of also using telemedicine services B. Reduce the current frequency of access to the ward, replacing some accesses with telemedicine services C. Use only telemedicine services, without going to hospital D. Other.
TAT 19	How satisfied are you with the effect of your treatment?	Little, 2023, Otolaryngology, USA¹¹
TAT 2	How satisfied are you with the explanations the SLP [speech-laguage pathologist] has given you about the results of your treatment?
TAT 3	The SLP was very careful to check everything when examining you.
TAT 16	How satisfied were you with the choices you had in decisions affecting your health care?
TAT 25	The time you had with the SLP was too short.
TAT 19	Are you satisfied with the telehealth care you received?
TAT 22	For the first few times, the presence of a third party to log in was essential or important.	Lal, 2023, Psychiatry, Canada¹²
TAT 22	During the teleconsultations, did you ask for help from a professional/member of the treatment team, family member or friend to start a session or to solve technical problems?
TAT 19	Generally, to what extent are you SATISFIED with the consultations offered through the … platform?
TAT 6	To what extent are you favorable towards continuing to receive telepsychiatry services via the … platform with the JAP/SOL team as part of your follow-up?
TAT 22	In general, … was EASY to use.
TAT 12	The quality of the IMAGE was adequate.
TAT 14	The quality of the SOUND was adequate.
TAT 28	I felt SECURE when using … (confidentiality, etc.).
TAT 22	For the first few times, the presence of a third party to log in was essential or important.
TAT 22	During the teleconsultations, did you ask for help from a professional/member of the treatment team, family member or friend to start a session or to solve technical problems? A. Yes B. No C. Professional/treatment team D. Family member/friend E. I prefer not to answer.
TAT 22	For the first few times, the presence of a third party throughout the consultation was essential or important.
TAT 22	For what reason(s) did you receive TECHNICAL ASSISTANCE from a professional/member of the treatment team, family member or friend? (more than one response possible) A. Sound/camera/video problems B. To start session C. Internet connection problems D. Other (‘To end a session’; ‘password forgotten frequently’; ‘To connect devices [cables, microphones, etc.]’).
TAT 8	Would you recommend … to your friends?
TAT 6	Reasons for not being favorable towards continuing to receive telepsychiatry services via videoconferencing? (More than one response possible) A. Preference towards other formats such as in‐person consultations and telephone consultations if in‐person is not possible B. Preference against or issues accessing technology (‘I have difficulty in getting technology or computer applications to work’; ‘I do not like to use technology’; ‘I prefer other platforms that are more user‐friendly’) C. I prefer not to answer.
TAT 1	The availability of the … platform has had a positive impact on the continuity of my clinical care.
TAT 27	In your opinion, a videoconferencing tool such as … SHOULD BE USED: (more than one response possible A. To avoid having to travel to the hospital B. To facilitate/increase access to care C. As a last resort when in‐person meetings are impossible D. In the case of unexpected events or in an emergency E. To replace in‐person meetings F. Other (‘useful to protect yourself from COVID‐19’) G. I prefer not to answer
TAT 7	Is this your first experience with teleconsultation?	Alromaihi, 2023, Internal Medicine, Bahrain¹³
TAT 7	Is this your first experience with teleconsultation at …?
TAT 20	How satisfied are you with the scheduling of the appointment (i.e. date and time)?
TAT 12	How satisfied are you with the sound and image quality of the device used during the teleconsultation?
TAT 21	How satisfied are you with timing of the teleconsultation (i.e. were you contacted in an appropriate time)?
TAT 18	How satisfied are you with the physician in terms of introducing him/herself clearly?
TAT 3	How satisfied are you with the physician in terms of explaining the aim of the teleconsultation?
TAT 16	How satisfied are you with the physician in terms of treating you with care and respect?
TAT 3	How satisfied are you with physician explanation of your medical condition?
TAT 3	How satisfied are you with physician explanation of the investigation results?
TAT 18	How satisfied are you with physician explanation of the treatment plan?
TAT 30	I felt comfortable talking to my physician using teleconsultation.
TAT 22	I find telemedicine easy to use.
TAT 9	I felt safer using teleconsultation than attending the hospital during COVID 19 pandemic.
TAT 6	I would like to see telemedicine used after COVID 19 pandemic.
TAT 22	I find teleconsultation more convenient.
TAT 25	If you answered the previous question with yes, please explain: A. Saves waiting time in the clinic, B. Saves lost time from my job, C. Saves lost time from my family, D. Saves the need to find a babysitter E. Saves transportation / difficulty to commute to hospital F. Others.
TAT 28	I trust my personal information and privacy will be protected with telemedicine.
TAT 19	How satisfied are you with the overall quality of teleconsultation in internal medicine outpatient department received at King Hamad University Hospital?
TAT 8	Would you recommend teleconsultation at … to others seeking medical care?
TAT 22	Did you have to ask for help to have the videoconsultation?	Perrin, 2023, Orthopedic, France¹⁴
TAT 10	Did you have to ask for help to obtain the equipment needed for the videoconsultation (computer, webcam, or smartphone)?
TAT 7	Was this your first consultation with this surgeon? (you never saw the surgeon in her/his office or via videoconsultation)?
TAT 1	If you have already had surgery, are you satisfied with the result of the surgery?
TAT 7	Is this your first videoconsultation?
TAT 25	How long did you wait to start the videoconsultation (minutes)? A. 0-15 minutes, B. 15-30, C. 30-45, D.45 minutes or more
TAT 25	How long would it have taken you to go to the hospital in person? A. less than 30 minutes, B.30 minutes to 1 hour, C.1 hour to 2 hours, D.2 hours to 3 hours.
TAT 3	Please evaluate the knowledge and competence of the surgeon?
TAT 15	Please evaluate the courtesy of the surgeon?
TAT 25	Please evaluate the time spent by the surgeon?
TAT 1	Do you feel this videoconsultation provided an appropriate response to your health condition?
TAT 16	Please evaluate your ability to speak freely during the videoconsultation?
TAT 27	Please evaluate your videoconsultation experience compared to an in-person consultation?
TAT 19	Please evaluate your overall experience with the videoconsultation?
TAT 8	How likely are you to recommend a videoconsultation to someone else (score from 0 to 5)?
TAT 27	After this experience, if you had the choice, would you prefer? A. a videoconsultation B. an in-person consultation.
TAT 27	If your surgeon suggests another consultation after the lockdown is lifted, which would you prefer? A. a videoconsultation B. an in-person consultation.
TAT 28	Was someone with you during the videoconsultation?
TAT 11	Did you encounter any problems during the videoconsultation? A. none B. unstable video connection unstable audio connection C. noise at your home interruption by someone in your home D. noise in the surgeon’s office E. had difficulty or could not upload imaging studies (X-rays, CT scan, MRI) F. poor Internet connection (insufficient bandwidth or loss of connection).
TAT 24	Did you encounter any difficulties during the videoconsultation? A. none B. difficulties positioning the camera C. difficulties showing the painful region D. difficulties understanding the maneuvers requested by the surgeon E. difficulties performing the maneuvers requested by the surgeon.
TAT 25	On average, approximately how long have your virtual visit (s) been with your primary care physicians and/or nurse practitioner? 1, 5 min 2, 10 min 3, 15 min 4, 20 min 5, 25 min 6, 30 min 7, Longer than 30 min 8, Unsure/don’t know 9, Other 10, Not applicable If other, please elaborate.	Bhatia, 2022, Primary Care, USA¹⁵
TAT 30	Please answer the following questions. What three words or phrases come to mind thinking about your experience with virtual visits with your primary care physicians and/or nurse practitioner? Word 1: Word 2: Word 3:
TAT 18	Thinking about the virtual visit you had How much effort was made to help you understand your health issues?
TAT 18	How much effort was made to listen to things that matter most to you about your health issues?
TAT 16	How much effort was made to include what matters most in choosing what to do next.
TAT 9	Did your primary care physician and/or nurse practitioner recommend that you purchase anything to make your care at home easier (e.g. pulse oximeter, blood pressure cuff?) If yes, what were you asked to purchase?
TAT 12	Ability to see the images of the primary care physician/nurse practitioner on the monitor.
TAT 14	Ability to hear the primary care physician/nurse practitioner.
TAT 23	Ability to get connected with the primary care physician/nurse practitioner virtually by phone or video.
TAT 25	The duration of the visit. The treatment plan you received (e.g. any medication changes, next steps).
TAT 21	How well the staff answered your questions about the equipment/process needed to do virtual visits.
TAT 28	How well your privacy was respected.
TAT 29	Overall comfort in using virtual visits to receive care.
TAT 4	Overall convenience in using virtual visits to receive care.
TAT 19	Overall quality of health care you received virtually.
TAT 19	How satisfied are you with your virtual visit(s)?
TAT 30	How could we improve your satisfaction with virtual visits? Please describe when a virtual visit may be most useful.
TAT 2	I felt reassured about my health problems after talking with my primary care doctor and/ or nurse practitioner during my virtual visit
TAT 27	How did your virtual visit compare to a traditional visit? 1, Better than traditional visit 2, Just as good as traditional visit 3, Worse than a traditional visit 4, Not sure.
TAT 17	Ask your primary care doctor or nurse practitioner questions about multiple health problems?
TAT 29	Right before your appointment, how confident did you feel in your ability to participate in a video visit?	Chu, 2022, Family Medicine, Canada¹⁶
TAT 1	To what degree did the video visit help you with the health issue for which you needed the appointment?
TAT 27	On average, compared to an in-office visit how was your experience with receiving care through a video visit? A. Better than an in-person, B. Same as an in-person visit, C. Worse than an in-person visit, D. Not sure.
TAT 27	On average, compared to an in-office visit how did the video visit impact your ability to access care from you doctor? A. Better than an in-person, B. Same as an in-person visit, C. Worse than an in-person visit, D. Not sure.
TAT 27	On average, compared to a telephone visit how was your experience receiving care through a video visit? A. I have not previously had a telephone visit, B. Better than a telephone visit, C. Same as a telephone visit, D. Worse than a telephone visit, E. Not sure.
TAT 9	What would you have done if you were not able to see your doctor through a video visit? A. Walk in clinic, B. Emergency department, C. See/talk to my family doctor, D. Scheduled an in person visit with this doctor, E. I would not have sought care at that time.
TAT 25	Reflecting on your last video visit, did the video visit save you time (for example by saving you travel time or arranging care for dependents)?
TAT 11	Were there any issues with technology that made it difficult to participate in a video visit? If yes, what issues made it hard to participate?
TAT 8	How likely are you to recommend the virtual visits to a friend on a scale of 1-10?
TAT 22	How do you rate your comfort with using technology in your day-to-day life?
TAT 6	Would you like the option to continue having video visits with your healthcare providers after COVID-19?
TAT 30	Any suggestions for how to improve video visits for patient care?
TAT 18	You were satisfied with what the doctor told you about your complaint.	Rali, 2022, Ophthalmology, USA¹⁷
TAT 17	The doctor communicated with you effectively.
TAT 19	You were satisfied with the session you had with the doctor.
TAT 29	You were comfortable asking questions of the doctor.
TAT 13	It was easy to see the doctor on the monitor.
TAT 18	You were able to communicate what you wanted to say to the doctor.
TAT 27	You think the doctor did as good a job as if you had been in the doctor’s office.
TAT 27	You think it would have been better if you had seen the doctor in person.
TAT 27	You would feel much more comfortable in face-to-face meeting.
TAT 27	Not being in the same room with the doctor was a problem for you.
TAT 24	The equipment made you feel uncomfortable.
TAT 24	The equipment made it hard to understand the doctor.
TAT 5	It bothered you that the doctor at the remote site couldn’t touch you.
TAT 27	You liked seeing the doctor this way as much as seeing him/her in the ordinary way.
TAT 17	I understood what the specialist told me regarding my medical issues.	Pow, 2022, Radiation Oncology, Australia¹⁸
TAT 14	I had no issue or problem with hearing or seeing the specialist.
TAT 15	I felt the specialist maintained a professional relationship.
TAT 15	I was able to establish a rapport (mutual trust/understanding) with the specialist.
TAT 16	I was able to explain my physical and medical issues to the specialist.
TAT 17	I felt comfortable expressing my physical and medical issues.
TAT 17	I was able to explain my emotional/psychological/social issues to the specialist.
TAT 29	I felt comfortable expressing my emotional/psychological/social issues.
TAT 5	I felt it was important for the specialist to physically examine me.
TAT 18	I understood the role, rationale and logistics involved in the planning and delivery of the radiotherapy plan in my circumstance.
TAT 29	I felt comfortable expressing potential side effects from my Radiotherapy treatment.
TAT 19	How satisfied were you that the consultation met your medical care expectations?
TAT 19	How satisfied were you with the overall quality of care you received?
TAT 27	I felt comfortable giving consent to Radiotherapy treatment.
TAT 16	I felt I was able to talk and ask/answer questions freely to the specialist.
TAT 28	I felt it was safe to discuss my medical/physical issues or concerns.
TAT 28	How satisfied were you that your privacy and confidentiality were preserved during the consultation?
TAT 25	How long did it take to get to your appointment?
TAT 25	How far did you have to travel for the consultation?
TAT 26	How much money (including fuel, parking, accommodation, loss of income from work etc) did it cost to attend the consultation?
TAT 4	How convenient was the consultation for you?
TAT 4	I felt the consultation did not disrupt/interfere with my daily life.
TAT 19	Overall, how satisfied were you with your consultation?
TAT 12	How satisfied were you with the quality of the picture and sound used?
TAT 29	I felt anxiety using this form of technology
TAT 1	I felt my medical care was not compromised using this form of consultation.
TAT 25	I felt that the consultation saved me time and/or money over an in person consultation.
TAT 25	How far would you have had to travel if your consultation was done in person?
TAT 9	Telemedicine consultations are suitable/appropriate as an initial consultation.
TAT 9	Telemedicine consultations are suitable/appropriate as a follow up consultation.
TAT 29	I feel comfortable with this form of consultation for future consultations.
TAT 9	I would prefer a telemedicine consultation over an in person consultation if offered.
TAT 23	Did you have to setup or control the equipment yourself?
TAT 19	How satisfied are you with your telemedicine visit?	Giunta, 2022, Orthopedic Surgery, USA¹⁹
TAT 7	How many times have you used telemedicine before this most recent appointment?
TAT 25	How many minutes did you spend preparing for your telemedicine visit? Time preparing for your telemedicine visit includes discussing telemedicine with office staff, downloading and troubleshooting software, preparing questions for physician, etc.
TAT 25	Excluding travel time, how long do you prepare for your typical in-person orthopedic clinic visits? Time preparing for your telemedicine visit includes discussing your clinic visit with office staff, preparing questions for physician, etc.
TAT 25	What is your typical travel time (round trip) to an in-person orthopedic clinic visit?
TAT 9	If given an option of video or telephone, which would you prefer? (video, telephone)
TAT 10	If you used at telephone for your telemedicine visit, do you have access to a computer/device with video camera and high-speed internet access?
TAT 19	How would you evaluate the care you received at your telemedicine visit?
TAT 26	How would you evaluate the care you received at your last in-person orthopaedic clinic visit?
TAT 19	How would you rate your patient experience via telemedicine?
TAT 27	How would you rate your patient experience during your last in-person orthopaedic clinic visit?
TAT 9	In the future, for a first-time new visit to an orthopedic surgeon, what is your preference for visiting the surgeon via telemedicine.
TAT 9	In the future, for a follow-up visit to an orthopaedic surgeon, what is your preference for visiting the surgeon via telemedicine.
TAT 30	Please provide any additional comments/concerns about your telemedicine experiences.
TAT 3	A physician can get a good understanding of my medical problem using telemedicine.	Danila, 2022, Rheumatology, USA²⁰
TAT 28	Telemedicine can violate my privacy.
TAT 23	The use of the necessary equipment seems difficult to me.
TAT 27	I can be as satisfied talking to the physician over telemedicine as talking in person.
TAT 1	Telemedicine can improve my general health.
TAT 25	Telemedicine can save time for the physician.
TAT 1	Using telemedicine, the physician will be able to monitor my condition well.
TAT 4	Telemedicine is a convenient form of health-care delivery for me.
TAT 25	Telemedicine saves me time.
TAT 6	Telemedicine will be a standard way of health-care delivery in the future.
TAT 9	Telemedicine can be an addition to the regular care I receive.
TAT 26	Telemedicine can reduce the costs for the health-care agencies.
TAT 9	Telemedicine makes it easier for me to contact my physician.
TAT 26	Telemedicine cannot save me any money.
TAT 5	I don’t like that there is no physical contact during a home telemedicine visit.
TAT 5	A physician cannot examine me properly over telemedicine.
TAT 10	I cannot always trust the equipment to work.
TAT 4	The Telehealth appointment was convenient for me.	Manski-Nankervis, 2022, General Practice, Australia²¹
TAT 25	The Telehealth appointment saved me time.
TAT 25	What is the estimated time you have saved by attending via Telehealth compared to in-person? A. Less than 1 hour, B. 1 hour, C.2 hours, D. Half a day (4 hours), E.A whole day (8 hours), F. More than 1 day, G-Prefer not to answer.
TAT 25	How would you have normally spent that time? A. In a paid role or job, B. In an unpaid role or job, C. Prefer not to answer.
TAT 25	How many kilometres approximately do you drive to reach this clinic?
TAT 26	How much would you usually pay for parking? (in dollars and cents e.g. 10.50)
TAT 26	How much would you usually pay for your transport? (in dollars and cents e.g. 10.50)
TAT 25	Did the Telehealth appointment start at the scheduled time?
TAT 25	How long (in minutes) did you have to wait?
TAT 29	I was comfortable using the Telehealth technology.
TAT 23	Joining the Telehealth call was easy for me.
TAT 28	My safety and privacy was maintained during the consultation.
TAT 1	Telehealth is an acceptable way to receive healthcare services.
TAT 1	Telehealth improves access to healthcare services.
TAT 9	I would prefer to have the option of a Telehealth consultation when I need it.
TAT 9	I prefer an in-person consultation over Telehealth.
TAT 16	The Telehealth service gave me the opportunity to ask questions about my care.
TAT 16	I felt that my questions or concerns about my health condition(s) were adequately addressed during this Telehealth service.
TAT 27	The Telehealth service I received was as good as an in-person appointment.
TAT 7	Have you ever had a Telehealth appointment before today?
TAT 25	We are interested in costs saved by you attending a Telehealth appointment. Do you have any further comments about this?
TAT 25	Are there any additional costs you have incurred by attending a Telehealth appointment (e.g. accommodation, support workers)?
TAT 8	How likely is it that you would recommend a Telehealth consultation to a family member or a friend?
TAT 30	Do you have any suggestions for improving our Telehealth services, or any other comments about Telehealth?
TAT 23	I had no difficulty gaining access to the Telehealth encounter.	Jeraq, 2022, Endocrine Surgery, USA²²
TAT 17	I was able to communicate adequately with the provider today.
TAT 25	The provider was on time for the appointment.
TAT 12	The picture quality was good.
TAT 14	The sound quality was good.
TAT 28	My privacy and confidentiality were respected and protected during the consultation.
TAT 30	I was comfortable with the evaluation that was done.
TAT 25	The provider spent an appropriate amount of time with me during this visit.
TAT 9	Telehealth made it easier to get healthcare today.
TAT 27	The Telehealth visit was equivalent to an “in person” provider visit.
TAT 8	You would recommend this Telehealth experience to others.
TAT 15	The friendliness of my provider was very good.
TAT 18	Explanations the care provider gave you about your problem or condition were very good.
TAT 15	The provider showed ample concern over your questions or worries.
TAT 19	The provider included you in decisions about your treatment.
TAT 18	The provider gave ample information about any medications (if any).
TAT 18	The provider gave appropriate follow-up instructions (if any).
TAT 17	The provider used language you could understand.
TAT 18	You are very confident in this provider’s course of action.
TAT 8	You would recommend this provider to others.
TAT 15	I felt the provider presented him/herself appropriately during this encounter.
TAT 9	Does the ability to provide telemedicine consultation improve your confidence in your cancer care center?	Gondal, 2022, Oncology, Canada²³
TAT 1	Met your care needs.
TAT 19	Overall quality of care provided.
TAT 17	Ability to talk freely over telemedicine.
TAT 17	Ability to understand the recommendation made.
TAT 12	Quality of the visual image.
TAT 14	Quality of the audio sound.
TAT 19	Overall telemedicine consult experience.
TAT 25	The length of time with the neurologist during the visit?	Asukile, 2022, Neurology, Zambia²⁴
TAT 3	The thoroughness, skillfulness, and carefulness of the neurologist?
TAT 17	Your ability to explain your problem to the neurologist?
TAT 17	The neurologist’s understanding of your condition?
TAT 18	The explanation of your condition by the neurologist?
TAT 17	The explanation of your treatment by the neurologist?
TAT 19	The overall treatment experience?
TAT 29	How satisfied were you with: Your personal comfort and privacy during the visit?
TAT 14	The audio quality of the call? (ability to hear and be heard)
TAT 12	The video quality of the call? (ability to see and be seen) (If applicable)
TAT 19	The overall treatment experience with the video/phone visit?
TAT 26	How much money did the tele-visit cost you?
TAT 26	How much would you normally spend on an in-person visit -Including transport and lodging?
TAT 25	How much time did It take to set up and attend the televisit?
TAT 25	How much time would you usually spend on an in-person visit? (from the time you leave home, to the time you return)
TAT 6	For your next visit, which method would you prefer? A. In-person B. tele-visit C. Unsure Give a reason for your answer?
TAT 8	Would you recommend a teleneurology visit to another person with a similar condition? Give a reason for your answer?
TAT 30	Additional comments
TAT 14	I was able to hear (audio) without difficulty.	Brown,2022, Rehabilitation, Australia²⁵
TAT 12	I was able to see (visual) the screen without difficulty.
TAT 28	I felt my privacy was maintained in the telehealth appointment.
TAT 22	The telehealth technology was easy to use.
TAT 9	I would prefer to attend a face-to-face appointment even when considering the situation with the pandemic.
TAT 27	I would rate the telehealth session as being equal to a face-to-face session.
TAT 5	I would consider using telehealth for health services in the future after the pandemic is over.
TAT 4	Convenience of appointment time.	Drerup, 2021, Internal Medicine, USA²⁶
TAT 20	Registration staff friendliness.
TAT 25	Amount of time with the doctor.
TAT 16	Doctor listened to concerns.
TAT 18	Doctor explained care.
TAT 18	Communication of test results.
TAT 8	Recommend this practice to others?
TAT 25	Wait time to see the doctor A.0–30 min B.31–60 min C.>60 min.
TAT 3	What did the doctor do well? A. Doctor communicated well B. Doctor did not rush C. Visit was efficient D. Doctor showed concern E. General positive comment.
TAT 19	I was pleased with the quality of the encounter.	Dwyer, 2023, Urology, USA²⁷
TAT 28	I was able to share sensitive and/or personal information with my provider.
TAT 19	I was overall satisfied with my appointment.
TAT 5	I believe that the provider is able to do his or her job without an examination.
TAT 15	I have a trusting relationship with my provider.
TAT 6	I would choose to meet with my provider in this setting in the future.
TAT 7	Earlier use of technology for health A. Videoconferencing B. Email C. Text messaging D. Patient portal E. Mobile health F. Other G. None.	Harkey, 2021, Surgery, USA²⁸
TAT 6	Future choice for follow-up care A. Virtual visit B. In-person clinic visit C. No preference.
TAT 21	Ease of scheduling the appointment.
TAT 25	Amount of time the healthcare provider spent with me.
TAT 19	Overall, how would you rate the care that you received?
TAT 10	Did you have a technical issue with your visit?
TAT 22	Did you need to contact a service for assistance?
TAT 22	Did the issue prevent you from completing your virtual visit?
TAT 19	How would you rate your overall experience with telemedicine?	Ribaya, 2023, Family Medicine, USA²⁹
TAT 22	How confident do you feel in having a telemedicine visit without a staff member from the office helping you with the technology aspect of the visit?
TAT 22	How confident do you feel in having a telemedicine visit without a family member, caregiver, or friend helping you with the technology aspect of the visit?
TAT 26	Are you concerned about the cost of a telemedicine visit?
TAT 5	Do you plan to use telemedicine again after the pandemic?
TAT 5	How worried are you about returning to our practice for in-person office visits?
TAT 30	Please provide us any additional comments/suggestions to help us prepare the future of our practice.

TAT, Telemedicine Assessment Toolkit.

Table 2.

Patient/Encounter Characteristics Questions Matching Routinely Generated Health Systems Information (Block 1)

BLOCK 1 questions	Article reference
Teleconsultation method: A. Telephone call B. Video call C. WhatsApp message	Alromaihi 2023, Internal medicine, Bahrain¹
Which digital device did you use for videoconsultation? A. Smartphone, B. Computer	Perrin, 2023, Orthopedics, France²
What health condition is the consultation about? A. Fracture of the upper limb (shoulder, arm, elbow, forearm, hand) B. Fracture of the lower limb (hip, femur, knee, leg, ankle, foot) C. Shoulder condition (tendinitis, rotator cuff, osteoarthritis, ...) D. Elbow condition (tendinitis, osteoarthritis, ...) E. Hand condition (carpal tunnel, Dupuytren’s contracture, trigger finger, ...) F. Hip condition (osteoarthritis,...) G. Knee condition (osteoarthritis, ACL tear, PCL tear, meniscal tear, ...)
When are you having this consultation? A. before deciding whether to have surgery, B. After having surgery
Since March 2020, have you had any virtual visits where you tried to use video but ended up having the visit over the phone due to the video not working?	Bhatia, 2022, Primary care, USA³
If you had both a phone visit and a video visit, which, did you prefer? 1, Phone based 2, Video based 3, No preference 4, Do not know 5, Refused to answer 6, Not applicable	Bhatia, 2022, Primary care, USA³
To date, how many video visits have you had with your healthcare team at …? A. 1, B. 2-3, C. 3-4, D. 5+	Chu, 2022, Family Medicine, Canada⁴
How would you rate your overall health?	Chu, 2022, Family Medicine, Canada⁴
How many telemedicine consultations (including non-related to your cancer treatment) have you had? A.1, B. ≤5, C.≥5	Pow, 2022, Radiation Oncology, Australia⁵
How often do you use other video conferencing technology/software e.g. Skype/Facetime/Webex for communication?	Pow, 2022, Radiation Oncology, Australia⁵
Were you offered to conduct your telemedicine via telephone, video, or both? (telephone, video, both)	Giunta, 2022, Orthopedic Surgery, USA⁶
What type of device did you use for your telemedicine visit? (Laptop, Desktop Computer, Tablet, Smartphone, Landline Telephone, Other)	Giunta, 2022, Orthopedic Surgery, USA⁶
Was your Telehealth consultation: A. Conducted with video on for the entire consultation, B. Conducted with video on for part of the consultation; C. video stopped by my request Conducted with video on for part of the consultation; D. video stopped due to technical issues Conducted by telephone with no video	Manski-Nankervis, 2022, General Practice, Australia⁷
How would you usually travel to the clinic for an in-person appointment? A-Personal Car, B-Public, C-Transport, D-Taxi, E-Uber or equivalent, F-Other, G-Unsure, H-Prefer not to answer If other, please specify
Was the appointment today a new or review appointment: A-New (I have never seen this doctor/healthcare provider before, B-Review (I have seen this doctor/healthcare provider before, either in-person or via Telehealth), C-Unsure, D-Prefer not to answer
Is this the first time you have attended this clinic? (Please tick all that apply.) A-Yes, I am new to this clinic, B-No, I have attended over 12 months ago, C- No, I have attended in-person within the last 12 months, D-No, I have attended via Telehealth within the last 12 months
Phone Visit?	Drerup, 2021, Internal Medicine, USA⁸
Video Visit?	Drerup, 2021, Internal Medicine, USA⁸
Have you participated in a telemedicine visit with the Center for Healthy Aging (formerly Jefferson Geriatrics)?	Ribaya, 2023, Family Medicine, USA⁹
If you have not had a telemedicine visit with Jefferson Geriatrics, why not? (check all that apply):	Ribaya, 2023, Family Medicine, USA⁹

Table 3.

Questions Outside the Scope of TAT

Theme	Question	Article reference
RESPECT	How much of the time did you feel respected by the SLP?	Little, 2023, Otolaryngology, USA¹¹
TRUST	I trust the instructions of my physician provided through teleconsultation.	Alromaihi 2023, Internal Medicine, Bahrain¹³
FOLLOW-UP	Were you provided with a call back number in case you had an inquiry?
FOLLOW-UP	If you answered the previous question with yes, were you able to reach your physician after using the call back number/were your inquiries answered?
FOLLOW-UP	I felt comfortable putting into action the treatment plan I received from my primary care doctor and/or nurse practitioner during my virtual visit.	Bhatia, 2022, Primary Care, USA¹⁵
FOLLOW-UP	I felt that it was easy to approach my primary care doctor and/or nurse practitioner after my virtual visit if I had any questions about the treatment plan or anything else that was discussed	Bhatia, 2022, Primary Care, USA¹⁵
URGENCY	How urgent is the consultation in your opinion?	Perrin, 2023, Orthopedics, France¹⁴
TRAINING	The educational materials you received by your health care provider.	Bhatia, 2022, Primary Care, USA¹⁵
TRAINING	Did you look at the training information prior to conducting your first video visit?	Chu, 2022, Family Medicine, Canada¹⁶
BLOOD TEST RESULTS	I felt the relevant test results (blood test/imaging) were presented or discussed adequately.	Pow, 2022, Radiation Oncology, Australia¹⁸
TECH ABILITY	I am a confident user of applications (e.g., Facebook, Twitter) for online communication.	Pow, 2022, Radiation Oncology, Australia¹⁸
TECH ABILITY	How did you find your ability to do telemedicine with regards to technology?	Giunta, 2022¹⁹
REASON	Why did you choose to use Telehealth? A-I didn’t choose, a Telehealth appointment was made for me, B-Reduced travel time and cost, C-Reduced risk of contact with COVID-19, D-Difficulty with travel (e.g. due to illness or limited mobility), E-Satisfied with a previous telehealth experience, F-Reduced burden on family members, friends, or carers, G-Curiosity - I wanted to see how it works, H- Other E-Prefer not to answer If other, please specify?	Manski-Nankervis, 2022, General Practice, Australia²¹
ATTIRE	I felt the provider’s attire was appropriate for this encounter.	Jeraq, 2022, Endocrine Surgery, USA²²
ATTIRE	I feel as though it is appropriate for providers to wear casual medical attire (surgical scrubs) during the Telehealth Visit.
ATTIRE	I feel as though it is appropriate for providers to wear casual attire (t-shirt) during the Telehealth Visit.
ATTIRE	I feel as though providers must dress professionally (ex. shirt/tie and white coat) regardless of the type of encounter.
ATTIRE	I feel as though providers should wear a white lab coat during the Telehealth visit.
ATTIRE	I have no preference/did not notice the attire the provider wore during the Telehealth encounter.
ATTIRE	What my surgeon was wearing influences my opinion of the care that I receive during the Telehealth Visit.
VIDEO BACKGROUND	I feel as though the background of the video should be an office setting during the Telehealth Visit.
VIDEO BACKGROUND	I feel as though the background of the video should be a branded UHealth setting during the Telehealth Visit.
VIDEO BACKGOUND	I have no preference/did not notice background of the video during the Telehealth encounter.
VIDEO BACKGROUND	The background of the video during the Telehealth visit influences my opinion of the care that I receive during the visit.
OTHER	What were some of the best aspects of your telemedicine visit?	Ribaya, 2023, Family Medicine, USA²⁹
OTHER	What were some of the frustrating aspects of your telemedicine visit?
NOT RELEVANT	How well did you feel supported by Center for Healthy Aging (formerly Jefferson Geriatrics) during the pandemic?
NOT RELEVANT	What could Jefferson Geriatrics do to best support your needs during the pandemic.
NOT RELEVANT	Given the recent events surrounding issues of racism and social justice, what could Center for Healthy Aging (formerly Jefferson Geriatrics) do to best support your needs?
NOT RELEVANT	Do you have thoughts/suggestions how Center for Healthy Aging (formerly Jefferson Geriatrics) can support communities of color?

Table 1 lists each of the 21 articles, presenting information on the authors’ names, publication year, and country of publications, and each question specifies the matching TAT item. Table 2 includes questions assessing patient and encounter characteristics routinely generated from health systems. Table 3 includes all the questions that are outside the scope of audiovisual telemedicine encounters and do not meet the criteria for inclusion in the TAT.

Table 4 provides information on the TAT items and the distribution (percentages) of questions from the 21 included articles that match each. The TAT items that match the most questions were “Time saved” (n = 34; 11.4%), followed by “Overall satisfaction” (n = 24; 8.05%), “Clarity of communication” (27, 9.06%), and “Comparison with in-person care” (n = 21; 7.05%). For each question type, we used a statistical test of proportions to see if rates of inclusion were similar across the initial set of surveys and the follow-up set. p-Values for these tests are reported in Table 4. The test of proportions reveals no statistical difference between initial and follow-up articles for 24 of the 26 items tested. Two items, TAT 17 with 18 (combined) and TAT 25 are statistically significant at the unadjusted p < 0.05 level. However, these outliers are also not significantly different when analyzed using threshold p-values adjusted with Bonferroni–Holm corrections for multiple hypothesis testing: items 17 with 18 “Clarity of communication” (p_adj = 0.0019) and item 25 “Time” (p_adj = 0.002), respectively, and thereafter, reported as not statistically different.

Table 4.

Frequency of Items: Initial TAT Articles Versus Follow-Up Articles

			Total number (percent) of questions matching to TAT		p-Value (unadjusted chi-square test of proportion)	Difference in item frequency-initial TAT versus follow-up questions
			Initial set	Follow-up set	p-Value (unadjusted chi-square test of proportion)
	Total number of questions		613	298
Item ID	TAT Question	Question from Validated Assessment Tools
TAT 1	I find online consultation a positive means to achieve my health needs.		36 (5.87)	16 (5.37)	0.8766	0.50
TAT 2	I have a good understanding of the next steps in my care.		36 (5.87)	16 (5.37)	0.8766	0.50
TAT 3	I could see that the clinicians showed enough knowledge during the online consultation.		19 (3.10)	8 (2.68)	0.8900	0.41
TAT 4	I received care that suited me.		19 (3.10)	6 (2.01)	0.4683	1.09
TAT 5	I believe not receiving a physical examination affected my care.		13 (2.12)	8 (2.68)	0.7666	−0.56
TAT 6	I will use online consultation again.	TSQ: I will use telemedicine services again.	38 (6.20)	9 (3.02)	0.0607	3.18
TAT 7	I have used online consultation in the past.		15 (2.45)	8 (2.68)	1	−0.24
TAT 8	I would recommend online consultation to others.		23 (3.75)	9 (3.02)	0.7105	0.73
TAT 9	I obtained better access to health care services by using online consultation.	TSQ: I obtain better access to health care services by use of telemedicine.	22 (3.59)	19 (6.38)	0.08305	−2.79
TAT 10	I had the necessary equipment for the online consultation.		24 (3.92)	8 (2.68)	0.4504	1.23
TAT 11	I did not have any technical problems with the online consultation.		24 (3.92)	8 (2.68)	0.4504	1.23
TAT 12	I could see all people involved in the online consultations clearly for the majority of the videoconference time.	ITU scale: I could see the other people clearly for the majority of the videoconference time.	23 (3.75)	9 (3.02)	0.7105	0.73
TAT 13	I did not have trouble seeing the clinicians during the online consultation.	ITU scale: Did you have trouble seeing your specialist during the consultation?	23 (3.75)	9 (3.02)	0.7105	0.73
TAT 14	I could hear all people involved in the online consultations clearly for the majority of the videoconference time.	ITU scale: I could hear the other people clearly for the majority of the online consultation time.	20 (3.26)	7 (2.35)	0.5791	0.91
TAT 15	I experienced a personal connection with the clinician during the online consultation.		30 (4.89)	7 (2.35)	0.0996	2.54
TAT 16	I was asked by the clinician for my concerns pertaining to my health problems.	PACT: How much did your doctor try to find out about all your problems and concerns?	29 (4.73)	9 (3.02)	0.3007	1.71
TAT 17	I was able to communicate clearly with the clinician.		24 (3.92)	27 (9.06%)	0.0026^a	−5.15
TAT 18	I was able to understand the plan explained by the clinician.		24 (3.92)	27 (9.06%)	0.0026^a	−5.15
TAT 19	Overall, I was satisfied with the quality of service provided during the online consultation.	TSQ: Overall, I am satisfied with the quality of service being provided via telemedicine.	31 (5.06)	24 (8.05)	0.0124	−3.00
TAT 20	I was satisfied with the structural features of the online consultation.	TUQ: This system is able to do everything I would want it to be able to.	18 (2.94)	2 (0.67)	0.0514	2.27
TAT 21	I was satisfied with how the online consultation was operated by the clinic.		15 (2.45)	3 (1.01)	0.2256	1.44
TAT 22	I did not need assistance while using the online consultation system.	TSQ: I do not need assistance while using the system.	20 (3.26)	17 (5.70)	0.1157	−2.44
TAT 23	I found it easy to set up the online consultation.		15 (2.45)	6 (2.01)	0.8620	0.43
TAT 24	I was able to operate the devices needed for the online consultation.		13 (2.12)	3 (1.01)	0.3513	1.11
TAT 25	I saved time by using the online consultation.		31 (5.06)	34 (11.41)	0.0008^b	−6.35
TAT 26	I saved money by using the online consultation.		21 (3.43)	9 (3.02)	0.9013	0.41
TAT 27	I think the visit provided over the online consultation is the same as in-person visit.	TUQ: I think the visits provided over the telehealth system are the same as in-person visits.	43 (7.01)	20 (6.71)	0.9760	−0.30
TAT 28	I was able to share sensitive information without concern about unintentional disclosure.		23 (3.75)	11 (3.69)	1.000	0.06
TAT 29	I felt comfortable during the online consultation with the clinician.		27 (4.40)	10 (3.36)	0.5663	1.05
TAT 30	I would like to suggest one aspect for the improvement of the online consultations……… (Free format)		21 (3.43)	9 (3.02)	0.9013	0.41

Adjusted threshold for statistical significance for items 17 with 18 “Clarity of communication” (p_adj = 0.0019).

Adjusted threshold for statistical significance for item 25 “Time” (p_adj = 0.002).

We analyzed the relative relevance of individual TAT items to researchers. Compared to the initial group of articles used in creating the TAT, the percentages of 10 TAT items showed increased preference, but not statistically significant, for use by researchers in the follow-up set of articles: TAT 9, TAT 17, TAT 18, TAT 19, TAT 22, TAT 25, TAT 27, TAT 28, TAT 29, and TAT 30. These items relate to 10 subdomains linked to clarity of communication, preference for telemedicine, overall satisfaction, and ease of use of audiovisual telemedicine. Other preferred subdomains include time saved, money saved, comparison to in-person care, respect for privacy, comfort with encounters, and qualitative feedback (improvement suggestions).

Ten TAT questions (TAT 6, TAT 9, TAT 12, TAT 13, TAT 14, TAT 16, TAT 19, TAT 20, TAT 22, TAT 27) matched a question from one of the previously validated assessment tools. We found four questions from the TSQ, two questions from TUQ, one from PACT, and three questions from the International Telemedicine Union (ITU) real-time video communication scale. Among the 10 TAT questions above, we found 116 (38.9.%) questions in the follow-up articles and 244 (39.8%) in the initial set of articles.

Discussion

We reviewed 21 articles with nonvalidated questionnaires for telemedicine encounters out of 31 articles, reinforcing the need for standardized tools to solve the problem of small diverse studies with no standardized, uniformly adopted assessment questionnaire.^5,7,9–11 In our screening of the PubMed database, the proportion of articles that used validated questionnaires compared to nonvalidated or generic questionnaires: 25.8% (8 of 31) was also comparable to earlier studies which found 32%⁷ and 22.5%,⁶ respectively.

We observed that two articles with nonvalidated questionnaires that were initially screened for inclusion were found to have been included in the original cohort of articles used to create the TAT.^12,13 We attributed this recurrence to overlaps in journal publication dates (advance electronic vs. print publication dates) and removed them from the analysis of the results to avoid double counting with questions repeated in both cohorts.

We acknowledge that this research may not have captured all questionnaires during the search for articles and is limited to one database, PubMed. This may increase the likelihood of selection bias through the exclusion of studies not indexed within the database. Also, in defining the scope of this research, only patient-focused questionnaires were included while provider assessments were excluded.

We found that 85.6% (298 out of 348) of the questions used in the articles in the extended cohort from November 2021 to July 2023 match items in the TAT, establishing the comprehensiveness of and potential suitability of the TAT as a toolkit for researchers coming after the initial articles aggregated to develop the TAT.

Thirty questions (8.6%) were not within the scope of TAT, due to several reasons. Half (15 of 30) of these questions asked respondents about the “attire” and the “background” and “follow-up” after the encounter. Questions about encounter follow-up could not be expected to reasonably assess most telemedicine encounters in which respondents are surveyed before the end of the encounter session. Questions about the background and attire used during the encounter predominantly came from one study which was focused on aesthetics and was excluded to avoid nonrepresentative items that are also used for in-person visits. One question each was about trust, respect, and blood tests. Our opinion is that these questions were not specific to telemedicine, being questions relating to every type of visit, were not relevant to the toolkit, and could add more respondent burden without eliciting useful information. Importantly, these questions were not in comparison to in-person encounters, offering little value in evaluating telemedicine.

The TAT contains one-third of questions that come from previously validated instruments indicating that researchers have the need to use a larger set of questions to successfully evaluate telemedicine encounters. However, we found that both the initial and follow-up articles used approximately 40% of questions from validated questions suggesting that, in the future, the TAT questions may be weighted differently.

Considering the relevance of individual TAT items to researchers, the increase in relative percentage use of 11 TAT items suggests a high likelihood of being adopted by more recent researchers (TAT 9, TAT 17, TAT 18, TAT 19, TAT 22, TAT 25, TAT 26, TAT 27, TAT 28, TAT 29, TAT 30). These items relate to 10 subdomains linked to clarity of communication, preference for telemedicine, overall satisfaction, and ease of use of audiovisual telemedicine. Other subdomains linked to increased preference by authors include time saved, money saved, comparison to in-person care, respect for privacy, comfort with encounters, and qualitative feedback (improvement suggestions). Nine of these subdomains reflect major themes that are also found in the major existing validated telemedicine questionnaires, TSQ, TUQ, TESS, and PACT.⁸

The next logical steps for future research in standardizing TAT for uniform adoption should involve the evaluation of items by experts, with focus groups and cognitive interviews among the intended target population of audiovisual telemedicine encounter users. Additionally, the remaining questions should also be validated. Finally, the questions could be weighted to produce a single score.

Footnotes

Acknowledgments

The authors would like to thank the staff of the Karolinska Institutet, Sweden, and Greenblatt Library at Augusta University for the use of library materials and assistance in obtaining full-text articles.

Authorship Contribution Statement

All authors contributed meaningfully to the conceptualization, article selection, writing, reviewing, and editing of this article.

Author Disclosure Statement

The authors declare no competing, personal financial interests or relationships impacting this study.

Funding Information

This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.

Abbreviations Used

References

American Telemedicine Association (ATA). What is telemedicine, exactly?. 2021.[cited 2021 November 2]. Available from: http://www.americantelemed.org/about-telemedicine/what-is-telemedicine

Ekeland

, Bowes

, Flottorp

. Methodologies for assessing telemedicine: A systematic review of reviews. Int J Med Inform, 2012; 81(1):1–11.

Izower

, Liao

, Kim

, Quintana

., editors. A proposed patient-inclusive methodology for developing and validating telehealth surveys that include social determinants of health. In: AMIA Annual Symposium Proceedings. American Medical Informatics Association; 2021.

Jackson

, McClean

. Trends in telemedicine assessment indicate neglect of key criteria for predicting success. J Health Organ Manag, 2012; 26(4–5):508–523.

Agbali

, Balas

, Beltrame

, et al. A review of audiovisual telemedicine utilization and satisfaction assessment during the COVID-19 pandemic. Int J Technol Assess Health Care, 2021; 38(1):e2.

Agbali

, Balas

, Beltrame

, et al. A review of questionnaires used for the assessment of telemedicine. J Telemed Telecare, 2024; 30(10):1636–1666.

Hajesmaeel-Gohari

, Bahaadinbeigy

. The most used questionnaires for evaluating telemedicine services. BMC Med Inform Decis Mak, 2021; 21(1):36.

Agbali

, Andrew Balas

, Heboyan

, et al. Design and development of a Telemedicine Assessment Toolkit (TAT) for the assessment of audiovisual telemedicine encounters. J Telemed Telecare, 2023:1357633X231194381.

Dodeja

, Schreier

, Granger

, et al. Patient experience with telemedicine in adults with congenital heart disease. Telemed J E Health, 2023; 29(8):1261–1265.

10.

Listorti

, Ferrara

, Adinolfi

, et al. Joining telehealth in rheumatology: A survey on the role played by personalized experience from patients’ perspective. BMC Health Serv Res, 2023; 23(1):682.

11.

Little

, Russell

, Hwang

, et al. Applications of telemedicine in speech‐language pathology: Evaluation of patient satisfaction. Laryngoscope, 2023; 133(4):895–900.

12.

Lal

, Abdel‐Baki

, Lee

. Telepsychiatry services during COVID‐19: A cross‐sectional survey on the experiences and perspectives of young adults with first‐episode psychosis. Early Interv Psychiatry, 2023; 17(4):368–377.

13.

Alromaihi

, Asheer

, Hasan

, et al. Evaluation of patients’ satisfaction with the transition of internal medicine outpatient clinics to teleconsultation during COVID-19 pandemic. Telemed J E Health, 2023; 29(2):270–277.

14.

Perrin

, Mainard

, Limousin

, et al. Satisfaction and feasibility of videoconsultation (VC) in orthopaedic and trauma surgery in the context of the COVID-19 pandemic: Prospective study of 783 patients. Orthop Traumatol Surg Res, 2023; 109(1):103345.

15.

Bhatia

, Gilliam

, Aliberti

, et al. Older adults’ perspectives on primary care telemedicine during the COVID‐19 pandemic. J Am Geriatr Soc, 2022; 70(12):3480–3492.

16.

Chu

, Brual

, Fang

, et al. The use of telemedicine in older-adults during the COVID-19 pandemic: A weekly cross-sectional analysis in Ontario, Canada. Can Geriatr J, 2022; 25(4):380–389.

17.

Rali

, Partin

, Maa

, et al. Telemedicine-based approach to caring for patients with inherited retinal diseases: Patient satisfaction and diagnostic testing completion rates. Ophthalmic Genet, 2022; 43(5):641–645.

18.

Pow

, Iankov

, Shierlaw

, et al. A study in patient satisfaction regarding telemedicine consultations in radiation oncology. J Med Radiat Sci, 2022; 69(3):327–335.

19.

Giunta

, Paladugu

, Bernstein

, et al. Telemedicine hip and knee arthroplasty experience during COVID-19. J Arthroplasty, 2022; 37(8S):S814–S8. e2.

20.

Danila

, Sun

, Jackson

, et al. Satisfaction with modes of telemedicine delivery during COVID-19: A randomized, single-blind, parallel group, noninferiority trial. Am J Med Sci, 2022; 364(5):538–546.

21.

Manski-Nankervis

J-A

, Davidson

, Hiscock

, et al. Primary care consumers’ experiences and opinions of a telehealth consultation delivered via video during the COVID-19 pandemic. Aust J Prim Health, 2022; 28(3):224–231.

22.

Jeraq

, Mulder

, Kaplan

, et al. Telemedicine during COVID-19 pandemic: Endocrine surgery patient perspective. J Surg Res, 2022; 274:125–135.

23.

Gondal

, Abbas

, Choquette

, et al. Patient and physician satisfaction with telemedicine in cancer care in Saskatchewan: A cross-sectional study. Curr Oncol, 2022; 29(6):3870–3880.

24.

Asukile

, Chishimba

, Chomba

, et al. Implementation of a teleneurology clinic in Zambia during the COVID‐19 pandemic. Ann Neurol, 2022; 91(4):445–454.

25.

Brown

, Doherty

, Claus

, et al. In a pandemic that limits contact, can videoconferencing enable interdisciplinary persistent pain services and what are the patient’s perspectives? Arch Phys Med Rehabil, 2022; 103(3):418–423.

26.

Drerup

, Espenschied

, Wiedemer

, et al. Reduced no-show rates and sustained patient satisfaction of telehealth during the COVID-19 pandemic. Telemed J E Health, 2021; 27(12):1409–1415.

27.

Dwyer

, Holt

, Wolff

, et al. Patient‐centered outcomes of telehealth for the care of rural‐residing patients with urologic cancer. Cancer, 2023; 129(18):2887–2892.

28.

Harkey

, Connor

, Wang

, et al. View from the patient perspective: Mixed-methods analysis of post-discharge virtual visits in a randomized controlled trial. J Am Coll Surg, 2021; 233(5):593–605. e4.

29.

Ribaya

DNF

, Cunningham

, Hersh

, et al. Acceptability of telemedicine in a geriatric outpatient practice during the COVID-19 pandemic. Telemed J E Health, 2023; 29(6):921–926.

Comprehensiveness of the Telemedicine Assessment Toolkit for the Assessment of Audiovisual Telemedicine Encounters

Abstract

Background:

Methods:

Results:

Conclusion:

Introduction

Methods

Results

Discussion

Footnotes

Acknowledgments

Authorship Contribution Statement

Author Disclosure Statement

Funding Information

Abbreviations Used

References