Sage Journals: Discover world-class research

Abstract

Objectives

Telemedical applications are solutions to challenges in the healthcare system. However, it is unclear what intensive care unit healthcare professionals expect from such solutions. This study investigated the expectations and concerns of nurses and physicians when implementing telemedicine tools in intensive care units (tele-ICU).

Methods

The study was conducted in intensive care units in 2020 during the second wave of the COVID-19 pandemic. It used a mixed-methods approach targeted at physicians and nurses and involved 14 qualitative interviews and 63 quantitative questionnaires.

Results

The qualitative and quantitative data showed that both nurses and physicians were willing to use tele-ICU. Nurses recognised the advantages of real-time access to expertise offered by tele-ICU, but feared this would reduce physicians’ on-site patient time. Physicians, in turn, were concerned that they would be expected to be continuously on call. The majority in both groups agreed that any tele-ICU solution must be simple to use and integrate easily into existing organisational structures, networks, and work routines. Additionally, COVID-19 significantly influenced expectations: those who reported having more personal health concerns during the pandemic were more predisposed to favour the use of tele-ICU.

Conclusions

Overall, tele-ICU supports better care, but a successful implementation depends on its ease of use and context-sensitive approaches. Effectively integrating tele-ICU solutions into daily clinical routines requires input from nurses and physicians and their involvement in the implementation process from the outset, as well as consideration of existing organisational structures. Such measures will vastly increase the chance of acceptance and successful adoption of telemedical solutions in clinical practice.

Keywords

Telemedicine mixed-methods digital health intensive care technology implementation

Introduction

The COVID-19 pandemic revealed both the scale of the challenges posed by a global healthcare crisis and the fact that some national healthcare systems were ill-equipped and unprepared to handle them.^1–3 These challenges were further exacerbated by the demands on healthcare systems caused by technical, staffing, and infrastructural limitations, which predated COVID-19.^4–10 Although countermeasures were quickly taken to prevent healthcare systems from collapsing during the pandemic, COVID-19 showed that all over the world, and particularly in intensive care units (ICUs), healthcare systems were struggling with severe issues of resource scarcity. These included shortages of well-trained physicians and nurses and equipment.^11,12 Telemedicine, including telemedical tools for ICUs, is seen as a potential solution to some of these difficulties.^13–15

Telemedical tools allow healthcare to be delivered remotely. They enable a diverse array of applications, such as remote monitoring, transmission of medical data, and consultation with physicians.^16–18 The value of telemedical tools in crisis situations makes them ideal for use in critical care settings (tele-ICU), where they connect off-site specialists with on-site staff (e.g. during medical emergencies).^19–21 Tele-ICU typically involves the use of audio-visual technology and monitoring systems to assist physicians and nurses in the ICU, for example, by virtually connecting external specialists who can share their expertise during consultations on complicated cases.^22–24

Ever since the publication of the study of Rosenfeld et al.²⁵ in 2000 and its positive findings on tele-ICU outcomes, the body of literature analysing its potential benefits as well as its barriers has grown considerably. Positive outcomes include a reduction in mortality, shortened length of stay (LOS), reduction of costs, and better adherence to best-practice protocols.^26–30 Meta-analyses by Young et al.³¹ and Chen et al.³² supported these propositions. Other studies identified barriers, such as high costs or staff perceptions of telemedicine utility.^33–35 Khurrum et al., for example, showed how ICU healthcare professionals sometimes viewed tele-ICU critically.³⁶ This highlights the need to identify and consider both enabling and inhibiting factors when implementing telemedicine in order to realise the potential benefits of this technology.^37–39

Telemedicine access capabilities garnered increasing attention during the first phase of COVID-19 in 2020, when many medical wards, hospitals, and other medical services restricted in-person visits and consultations.^40,41 A survey carried out by Peine et al.⁴² showed that medical staff in Germany broadly accepted this measure during COVID-19. Telemedicine increasingly moved into the spotlight as a means of reducing contagion.⁴³ A subjectively perceived risk of contagion could, therefore, positively influence attitudes towards telemedicine.

Telemedicine is by no means the only technological driver for developments in ICUs. Particularly in recent years, due to rapid developments in artificial intelligence, robotics and the “Internet of things”, new models of intensive care units have emerged and called “intelligent care units” or “smart ICUs”. Many of these models are still based on the same fundamental consideration as telemedicine: obtaining information from experts who are not currently on site and acting accordingly.^44,45

Technology acceptance of healthcare professionals is the key for the successful implementation of technological solutions like telemedicine or even more advanced technological solutions. Previous professional experience may strongly influence attitudes towards telemedicine. Generally, senior healthcare professionals rely more on their own expertise than on external input (e.g. via telemedicine^46,47) and might, therefore, be more critical about its use. This is supported by Ahmed et al.⁴⁶ who identified more positive attitudes towards telemedicine among less experienced nurses than more senior nurses and supervisors.

Positive attitudes towards tele-ICU solutions could also be linked to more personal factors and attitudes. Some studies showed a greater willingness to use innovations like telemedicine, where job satisfaction is high,⁴⁸ although this relation could be more complex.⁴⁹ The willingness to implement tele-ICU solutions may also reflect a more general inclination to use technology.⁵⁰ Furthermore, nurses and physicians may view the same technology differently. Previous studies showed that different areas of expertise and working structures (e.g. due to profession) in the medical setting result in qualitatively different forms of use, motivation, and expectations of technological tools. This calls for a comparison of these two professional groups.⁵¹

Critically, the above list of potentially influencing factors is not exhaustive and could be more heterogeneous. In order to achieve a more nuanced and in-depth overview of the influencing factors, we opted to conduct this research by triangulating methods and combining qualitative and quantitative research. The qualitative interviews bring detailed insights into daily routines, challenges, and individual insights. The quantitative data provide an overview of the expectations associated with tele-ICU and how these may be influenced by the factors outlined above. By approaching the expectations associated with the use of tele-ICU from multiple research angles, we can arrive at more comprehensive, reliable, and valid conclusions.^52,53

Overall, using tele-ICU can bring positive health and economic outcomes and has great potential for alleviating some of the pressure in critical care settings. However, the expectations healthcare professionals have of the benefits and disadvantages of using these solutions are less clear. Consequently, the main aim of this study is to identify those expectations that encourage or inhibit the use of tele-ICU solutions. This is achieved by examining how these expectations are related to the level of professional experience, job satisfaction, areas of responsibility, and impact of COVID-19.

Methods

The study was conducted at the ICUs of the Medical University/General Hospital of Vienna (Allgemeines Krankenhaus: AKH). The AKH houses 140 systemised intensive care beds on 15 wards, representing the largest concentration of ICUs in Austria, and has not yet implemented any large-scale, standardised telemedicine solutions before the pandemic.⁵⁴ The study ran from October 2020–January 2021 and, therefore, included the second wave of the COVID-19 pandemic in Austria. The study was approved by the Ethics Committee and the Data Protection Commission of the Medical University of Vienna (ECS 1256/2020).

Both qualitative and quantitative studies focus on the expectations of telemedicine. We did not provide study participants with a set definition of “telemedicine” in the ICU. Given the exploratory nature of the study, we aimed to see telemedical application options the participants would or could envision both as providers and as recipients of telemedical support and supervision. All participants reported prior experiences with telemedicine, ranging from low-tech options like remote (phone) support to more high-tech support like expert boards with vitals or patient records shared.

Qualitative data collection and analysis

Participants

To obtain the qualitative data, we followed COREQ, the consolidated criteria for reporting qualitative research.⁵⁵ ME (PhD, female, extensive training in qualitative methods as a social scientist) conducted individual in-depth interviews until thematic saturation was successfully reached.⁵⁶ This resulted in a total of 14 interviews, seven with ICU physicians and seven with ICU nurses. The range of ages and years of professional experience were typical for AKH hospital staff: 30 to 60 years old, and 5 to over 30 years of experience. The interviews were held either in person or via Zoom, as chosen by the interviewee. Each interview lasted between 35 and 120 min and followed a predefined structure. Interviewees took part voluntarily following a general email request issued to ICU ward staff by the research team. Participants were assured that the data would be processed anonymously, and each signed a consent form prior to the interview.

Interview & analysis

The interviews were conducted using guided questions about current professional responsibilities, and expectations of and concerns about telemedicine. It intentionally allowed participants to describe the telemedicine solution that came to mind. All interviews were conducted, recorded, and transcribed in German. Selected quotes were translated into English for this article, with the accuracy of the translation checked by two native English speakers. We also followed the COREQ structure in the subsequent analysis. The coding was performed by a research team of three using printed versions of the transcribed interviews. In the first step, the researchers coded separately. The results were then compared and discussed. Where necessary, new codes and sub-codes were created following joint discussions between the three researchers. After familiarising ourselves with the data and obtaining a general overview, we established 44 final codes (10 main codes and 34 sub-codes, see Appendix (Table 1) to focus on the areas with the greatest impact on the research question.⁵⁷ Themes emerging from the codes and sub-codes were used to inform the section on qualitative results. The results of the interviews were structured into four overarching themes identified using the detailed coding process. Supporting quotations were directly marked.⁵⁵ We then discuss how these findings offer insights into issues requiring consideration when introducing tele-ICU.

With respect to the strategy of mixed-method analyses, we adopted a sequential approach and analysed the qualitative data first.⁵² Following the theoretical considerations outlined in the introduction, this data then informed decisions about the statistical analyses of the quantitative data.

Quantitative data collection and analysis

Quantitative questionnaire

A questionnaire set consisting of psychometrically standardised and validated questionnaires,^50,58 self-generated questions (tele-ICU expectations, personal health concerns, and distress), and socio-demographic questions was used to evaluate the expectations from tele-ICU and possible influencing factors.

The two standardised questionnaires were: 10 items from the Neyer et al.⁵⁰ short scale (Cronbach α=.791) to quantify willingness to use technology. Job satisfaction was measured using the short version of the Fischer and Lück⁵⁸ scale. Both used five-point Likert scales (1 = high, 5 = low).

Additionally, a panel of experts consisting of five researchers (psychologists, social scientists, physicians, and nurses) developed a questionnaire to capture the following four overarching aspects about telemedicine:

Expectations towards tele-ICU solutions: Expectations were measured using eight items on a five-point Likert scale (1 = low expectations, 5 = high expectations) and by computing an average score. Reliability was good (Cronbach α=.850).

Effects of telemedicine on expected patient numbers: Respondents were additionally asked whether they expected tele-ICU to impact patient numbers (1 = significantly more, 2 = slightly more, 3 = about the same number, 4 = slightly less, 5 = significantly less).

Benefits and concerns about telemedicine use: In two additional items, respondents indicated whether they considered tele-ICU solutions to be beneficial or whether they were giving rise to concerns, requiring responses in a binary format (yes/no).

Questions related specifically to COVID-19: With respect to the impact of COVID-19, questions were included to report both pre-pandemic and pandemic-related levels of health concerns and general distress. Answers were given on a 10-point Likert scale (1 = low distress/low concern, 10 = high distress/high concern).

Overall, the questionnaire was pilot-tested with a physician and nurses from our research group.

The final questionnaire consisted of 48 items and was completed online (q-set.de), taking participants around 15 min to fill in. Informed consent was obtained from each participant prior to starting the questionnaire.

Participants

Emails were sent to 580 ICU healthcare professionals on an AKH management distribution list. A total of 63 health professionals completed the questionnaire in full, representing an estimated response rate of 11%: 38 (60.3%) were nurses, and 25 (39.7%) were physicians. The mean age of the respondents was 40.4 ± 9.7 years, and 47.6% (n = 30) was female. The mean number of years spent working in an ICU was 13.6 ± 10.8 years. Participant characteristics are displayed in the Appendix (Table 2).

Analysis strategy for quantitative data

The quantitative data are reported in three sections. The first two sections were informed by two topics that emerged as a result of the qualitative analyses and theoretical considerations outlined in the Introduction, namely, (1) the relation between age/professional experience and telemedicine and (2) group differences between physicians and nurses.

Firstly, with respect to the relation between age/professional experience and telemedicine expectations, Pearson correlations were calculated (see Results 1). Secondly, to assess group differences between physicians and nurses, we calculated the group comparisons (independent t-tests) for job satisfaction, willingness to use technology, and expectations of telemedicine. Furthermore, for the two binary outcome variables, namely, telemedicine seen as beneficial or causing concerns, we report group comparisons (physicians vs. nurses) based on Chi-squared tests (see Results 2: Differences due to profession).

Thirdly, in the last analysis step (Results 3: Effects of COVID-19) we considered the possible effects of the COVID-19 pandemic. These were measured using self-reported health concerns and general distress before and during COVID-19. We calculated dependent t-tests to test for changes resulting from COVID-19. Finally, we report Pearson correlations to analyse the effects of these variables on telemedical outcome variables (i.e. expectations of telemedicine, benefits of use, and concerns about use).

All analyses were conducted in R and R-Studio.^59,60 Figures were plotted using the ggstatsplot package,⁶¹ which allows raw data to be shown together with visualisations of data centrality (e.g. mean, median, percentages) and data dispersion (e.g. combined box and violin plots). This package also directly performs statistical tests (e.g. Chi-squared test, t-tests, etc.). For reasons of parsimony, the statistical outcomes (i.e. significance tests and effect sizes) from this package are directly reported within the associated figures.

Findings

Results—qualitative analysis

The qualitative analyses identified four overarching themes: (1) age and professional experience in relation to technology acceptance; (2) professional differences (nurse vs. physician); (3) technical expectations; and (4) interpersonal and organisational networks.

Finding 1: technology acceptance related to age and professional experience

Our interviews reveal a complex matrix of expectations of tele-ICU solutions both between and within different age groups. Younger healthcare professionals often perceive opportunities, associating the use of tele-ICU tools with increased safety, both for themselves and their patients. The option to contact other possibly more experienced healthcare professionals would provide reassurance in their decision-making or help in choosing the best course of action. Older and more experienced healthcare professionals sometimes viewed tele-ICU measures more critically, for example, feeling obliged to use them and perceiving this as monitoring or control by superiors. Although a complex set of factors influences technology acceptance, of which age and professional experience are just one aspect, the interview data indicated that this must be considered as a highly relevant aspect. For example, one interviewee stated:

“New employees would most likely be grateful (for telemedicine). But there will also be many who say that this is not necessary for them. It makes it appear as if they don’t know something, and they wouldn’t want others to think that.” (Interview three, physician)

Finding 2: professional differences

Although nurses and physicians in ICU units are similar in being highly qualified and skilled professionals in their respective fields, accustomed to the high-pressure working environment, and working closely within multidisciplinary structures and teams, our qualitative data indicated noticeable differences in their attitudes towards telemedicine.

In our interviews, physicians, even those generally in favour of introducing tele-ICU, expressed concerns about the effects of a new technology on their workload. Specifically, they expressed the fear of being expected to be always “on call”, even outside work and on-call hours, and that nursing staff and/or less experienced physicians would overuse the option to contact them when they needed reassurance or a second opinion:

“Pushing the senior physician into the background, that he (sic!) will only be consulted in some cases as an expert and perhaps actually have more work, because the attitude is that, well, then we actually no longer need one physician for two ICUs, but he can do four at once because he will be able to sit in his room. This would be a little bit of a dangerous dynamic, so to speak.” (Interview seven, physician)

Nurses, by contrast, mostly noted the positive aspects of telemedicine. They emphasised that sharing expertise with physicians in a reciprocal manner, rather than a one-sided input from physicians to nurses, offers the potential to foster inter-professional exchange and improve patient care in the ICU. As the interviews were conducted during the first COVID-19 winter of 2020 and pre-vaccination, nurses also expressed positive attitudes towards telemedicine as a means of involving physicians in the work of the ICU whilst on-call, but not physically in the hospital. Some felt that on-call physicians were not always easily reachable when needed and saw the introduction of a telemedical tool as a means of improving communication.

One interviewee added that for this cross-professional exchange to work, some changes in the current system might be necessary:

“But you would also perhaps slightly flatten hierarchies, with more senior physicians willing to source expertise from nurses,” adding that, “respectful communication is a basic prerequisite for the successful use of telemedicine.” (Interview nine, nurse).

Nurses also expressed concerns that the technical and organisational work of managing the telemedical tool would fall on them, even when used by physicians to contact other physicians. This would increase nurses’ workload as they would perhaps be expected to prepare the tools and set up telemedical appointments. One of the interviewees commented succinctly:

“Fears regarding anything that is unclear need to be addressed, such as the possibility of an increased workload.” (Interview seven, physician)

Another study participant said that even the tone of voice can vary between disciplines and professions:

“If left unaddressed, differences in communication style could result in the use of telemedicine to access expertise being rejected, even where patients might benefit.” (Interview fourteen, physician)

Finding 3: technical expectations and requirements

Our data show that when new (telemedical) technologies have been introduced into intensive care units in the past, key factors influencing their acceptance, such as ease of use and suitability for clinical use (e.g. easy to disinfect), have often been overlooked, rendering the technology impracticable.

According to the interviews, specific technical requirements and criteria are crucial for telemedicine's acceptance in the ICU. They particularly stressed the importance of ease of use. Previous experience showed that new technologies introduced into the hospital setting were only used in daily routines if they were easy to log into, intuitive to use, and provided a clear benefit. Even tools that offered a medical benefit were often abandoned if the hardware or software was not easy to use. As one interviewee stressed:

“If [the tool] is easy and people like using it. If the number of technologies just increases and the path to using them becomes more onerous, people will have difficulties accepting them. People need to say, ‘It used to be terrible, but now it's better’.” (Interview five, physician)

Another interviewee added that the ease-of-use factor holds true, even for experienced ICU staff who are more accustomed to and capable of working with technologies than the average hospital professional (Interview nine, nurse).

Regarding suitability for clinical use, ICUs house extremely vulnerable patients:

“We treat patients with one or more non-functional organs.” (Interview three, physician)

High-risk patients in the ICU are either particularly sensitive to germs brought into their rooms or are highly infectious patients whose disease could be transmitted through technologies that are difficult or impossible to disinfect.

Finding 4: interpersonal and organisational networks supporting technology

One central finding regarding the successful implementation of tele-ICU is that a perfect technological solution and ease of use are only a part of the equation. The related structures, regulations, and networks that support the use of telemedicine must be in place if tele-ICU is to be truly established.

As one participant said:

“One would probably need to develop a certain framework, stating what the specific task of a telemedical practitioner is and what consequences a consultation or given advice has.” (Interview two, physician).

Several interviewees pointed out that in some ICUs, it is currently seen as a sign of competency to do as much as possible by oneself. This would not be a good basis for introducing a telemedical tool, the purpose of which is to increase the quality of patient care by incorporating expertise and advice from as many experts as necessary. As one participant stated:

“You have to prepare (the introduction of telemedicine) in such a way that it is not seen as failure or defeat to contact someone else.” (Interview four, nurse)

Another interviewee agreed, saying that:

“Creating clear structures, and addressing fears related to uncertainty” would be key steps in establishing any telemedical tool in an ICU (Interview seven, physician).

Whilst this kind of shift in perspective may not happen immediately, the study participants were optimistic that:

“It could easily be accomplished within a year, at the very latest.” (Interview four, nurse)

Whilst agreeing that the shifts needed to support telemedicine may happen eventually, others stated that it could take some time to establish regular use, especially for recruiting and organising the personnel structures supporting the technology:

“A lot of organisational work will be needed behind the scenes. After all, one can’t just consult arbitrary people.” (Interview five, physician)

The healthcare professionals provided a range of scenarios and “use cases”, which they envisioned for telemedical applications in their daily routines related to work in the ICU, both in receiving and providing telemedical support and supervision. Suggestions included the following: telemedical support for ambulances, (e.g. to assess if hospitalisation is necessary); interdisciplinary consults for complex ICU cases; sharing expertise with smaller and/or more remote centres; remote (robotic) surgeries (e.g. ones occurring rarely); expert boards to discuss complicated cases, including sharing vitals/imaging/patient records, etc.; help from off-duty physicians; help from other departments; on-call “centres” to receive questions from physicians and connect with available experts; sharing specialised nursing expertise (e.g. about wounds) with both nurses and physicians; telemedicine for quarantine rooms (to avoid more than one person having to dress in protective gear); or exchange on rapidly changing and/or complex diseases, such as COVID-19 at the time.

Results—quantitative analysis

Results 1: correlations—what variables influence expectations of telemedicine

Figure 1 shows the correlational results. Apart from substantive and significant correlations between years of professional experience (ICU and health system) and age, which are clearly dependent, the experience and age variable was barely related to expectations about telemedicine, benefits, concerns about telemedicine use, expectations of reductions in patient numbers caused by the use of telemedicine, or a general willingness to use technology. The only other significant correlations concerned general expectations about telemedicine, which went hand-in-hand with higher expected benefits and fewer concerns.

Figure 1.

Correlations between professional experience/age and expectations of telemedicine and technology.

Results 2: differences due to profession: physicians vs. nurses

There was no significant difference between nurses and physicians (Figure 2, t-tests, Welch procedure to account for unequal variances) with respect to the reported levels of job satisfaction, willingness to use technology, or expectations of telemedicine. Generally, in terms of individual outcomes, on average, both physicians and nurses reported reasonably high levels of job satisfaction and willingness to use technology. The range of individual ratings only varied within the upper and mid-points of the scale. Overall, job satisfaction and general willingness to use technology were generally high among both physicians and nurses.

Figure 2.

Differences in job satisfaction, willingness to use technology, and expectations of tele-ICU by profession.

Although the difference in expectations of telemedicine was not statistically significant, descriptively, nurses (mean = 2.50, median = 2.38) reported slightly higher expectations of telemedicine in ICU settings compared to physicians (mean = 2.71, median = 2.88). The range of individual ratings varied considerably from very high (1) to very low (5), suggesting highly idiosyncratic expectations.

When looking at general benefits and concerns related to telemedicine, overall, there were no significant differences between physicians and nurses in the rate of ‘yes’ and ‘no’ responses (Chi-squared test). For benefits, the rates within professions were very similar—almost all physicians (p < . 05) and almost all nurses (p < . 05) perceived benefits in using tele-ICU. For concerns, however, an interesting pattern emerged. The rates for physicians were at par (p = .84), whilst the majority of nurses reported no concerns (p < .05). See Figure 3.

Figure 3.

Benefits and concerns of using tele-ICU by profession.

Results 3: Did COVID-19 affect expectations?

Comparing COVID and pre-COVID levels, participants reported greater health concerns, t(62) = −6.5, p < .01, and a higher general level of distress, t(62) = −5.7, p < .01. The greater health concerns during COVID-19 were indeed correlated with higher expectations of telemedicine (Figure 4, r = −0.33, p < .05). There were no substantive correlations between telemedicine expectations and concerns before COVID (r = −0.13, n.s), distress before COVID (r = 0.15, n.s.), or distress during COVID (r = −0.15, n.s) and the variable of telemedicine expectations.

Discussion and recommendations

The main aim of the study was to identify favourable and inhibiting factors for tele-ICU solutions. By triangulating qualitative and quantitative data, our study identified a complex set of factors influencing expectations of the use of tele-ICU. The findings highlight factors relevant to decision-making processes and change management, which go beyond the technological solution of implementing tele-ICU. We discuss these findings and conclusions in detail.

Age and professional experience

In the first topic discussed in our study—the influence of age and professional experience on willingness to use tele-ICU solutions—qualitative and quantitative data showed some differences, showcasing the benefits of a mixed-methods approach. We did not find any substantial correlations between age/professional experience and willingness to use tele-ICU in the quantitative data, although previous studies have found this link.⁶² However, the qualitative data offer a more detailed and layered view as it allowed participants to factor in aspects of teamwork and organisational culture. Specifically, age and experience-related factors influence the use of telemedicine. This may be because more experienced employees strategically choose technology that will truly add value rather than using it merely due to its availability.⁶³ Similarly, our study found that a connection between age and willingness to use telemedicine is based on factors, which are not necessarily only related to technology use itself, but also to social structures, hierarchies within the team, and attitudes towards making mistakes. Our interview data showed that possible concerns about older individuals and the use of telemedicine do not necessarily stem from them being less willing or able to adapt to the use of new technology, but rather due to concerns about how using telemedicine might negatively reflect on their abilities. Hence, whilst younger and less experienced employees particularly see tele-ICU as a means of increasing patient safety and improving professional skills, older and more experienced employees see their expertise and professional independence threatened by unsolicited, external interference.

Commonalities and differences between professions—nurses and physicians

Our data show that professional differences—being a nurse or a physician—is an important aspect of expectations of tele-ICU. In general, both groups reported similarly high levels of job satisfaction and a high degree of willingness to employ technologies, such as telemedicine. This is not unexpected as using complex technological solutions are an inherent aspect of working in ICUs. Consequently, both professional groups were open to tele-ICU solutions and valued the option to contact other readily available or possibly more experienced healthcare professionals using tele-ICU, as indicated by both the qualitative and quantitative data.

However, there were also some notable differences. Overall, nurses adopted a slightly more positive attitude towards tele-ICU solutions. According to our data, nurses tend to see more benefit from sharing, whilst physicians, especially more experienced ones, tend to be more wary of interference. One repeatedly stated concern was having to be constantly available for (ad-hoc) consultations. This was also reflected in the quantitative data. These examples indicate the potential for disagreement between the two groups when negotiating the use of new technologies. Ideally, a successful implementation of telemedicine would involve a discussion of underlying professional differences, with possible solutions explored.

When asked about the benefits, both groups unequivocally recognised the benefits of tele-ICU solutions. However, with respect to concerns, the majority of nurses reported having no concerns, but this was the case only for roughly half of the physicians. This finding qualifies the study by Avdalovic and Marcin,⁶⁴ which showed that whilst some healthcare professionals see their expertise and professional independence threatened by unsolicited outside interference, and others see it as an opportunity to enhance patient safety and improve their professional skills, these differences can be primarily attributed to the professional group. The ambiguities described above—benefits of sharing, fear of interference, increased workload—also map well to the systematic literature review by Ramnath et al.: in their review, the perceptions of telemedicine ranged from improved teamwork and additional night-time support to poor communication and interruptions.²⁸

Technology acceptance and organisational structures

Throughout our qualitative interviews, three central topics emerged: ease of use; established organisational structures; and options for involvement in the implementation process. The interviewees stated that a new telemedicine application should be uniformly integrated into existing IT systems, show a clear advantage over existing technologies, and “appropriate organisational infrastructure needs to be provided”.⁶⁵ These are in accordance with previous studies (e.g. Gagnon et al.⁶⁵ and Deisz et al.)⁶⁶ It is important to involve various professional groups and ascertain specific expectations and views prior to purchasing and implementing solutions. Additionally, implementation processes need to be based on a functioning network and internal structures. Our data show that the necessary systems and network(s) should be prepared and established in advance to guarantee a smooth implementation process and long-term, successful use of the chosen tool. The importance of additional steps taken prior to implementation has already been addressed in several studies underlining the necessity of early-stage training and support systems.^31,64 Involving ICU employees at an early stage in deciding which telemedical tools to purchase can help ensure that the tools will be used. Existing organisational networks can also be leveraged during the implementation of new telemedicine processes to gain input on robust communication and familiarity with workflows,¹³ as found in our qualitative study. Consequently, clear structures and workflows are essential, with both nurses and physicians involved as early as possible, together with other relevant professions, in implementing tele-ICU solutions.

Significant work needs to be done by the management to connect the right people via telemedicine to create a robust telemedical network of experts. After all, telemedical tools are only of value when the relevant people can be reached with ease when needed. This could take different forms depending on the ICU setting in question, such as connecting several ICUs or linking ICU employees to experts working in other hospital departments or to experts outside the hospital. Assessing the type of expertise that would bring the most benefit to the ICU in question and building a suitable network of people to provide this expertise are perhaps the most important steps in establishing tele-ICU.

Additionally, our data highlighted the need to create an environment of trust when following the therapeutical recommendations provided via telemedicine. Accordingly, hospital policies need to be explicitly communicated and employees ensured of their validity. Clear communication of the policies for tele-ICU is a crucial step towards achieving broad acceptance and use of the chosen tool. What is also required is a greater willingness to ask for help.

There must be a network of experts with whom there is either an established relationship and/or an agreement for the exchange of expertise through telemedicine. If these relationships are in place, and it is clear when particular experts can be reached through on-call lists, for example, then there is a much higher chance that the telemedical technology will be used in the ICU. If these factors are considered, a well-established telemedical expert network could have multifold benefits to ICUs and beyond.

Effects of COVID-19

The study was conducted during the second wave of the COVID-19 pandemic in Austria. The hospital used in the study served as a primary location for COVID-19 intensive care patients, with two of the six ICUs dedicated to ICU COVID-19 care. Presumably, this increased awareness of the possible benefits of telemedical solutions. Indeed, in our study, a greater concern for personal health during this testing period was statistically related to higher expectations of telemedicine, as other studies have also shown.⁶⁷ This relation was not significant when participants were asked to consider their pre-COVID-19 health concerns. In the qualitative interviews, participants arrived at similar conclusions. Additionally, ICU employees were acutely aware that hospitals are in dire need of new strategies to foster resilience during (future) crises. Telemedicine may be one, although not the only solution for accessing external expertise when ICU employees are unavailable (sickness, quarantined during COVID-19, etc.). Thus, a functional telemedical system can help access expertise, potentially from a global network, to provide optimal treatment for complex medical cases, which are common in ICUs.

Limitations

This study has several limitations. The study was carried out in Austria's largest university hospital, which is known for its specialised centres and high patient volume. This setting ensured consistency in work practices, organisational structures, and allowed for easy access to numerous ICU healthcare professionals. This is necessary as return rates for (quantitative) online studies are generally low. Our study had an 11% return rate (63 participants for the quantitative analyses), which is within the range of similar studies.⁶⁸ The variation in ratings for expectations of telemedicine (Figure 2) extended across the entire possible range, with a higher density around the mean, as might be expected by random selection. The qualitative interviews also showed that opinions of tele-ICU among ICU healthcare professionals were diverse and differentiated. This suggests that our data were free of a bias that attracts primarily people with intrinsic interest in telemedicine. Another caveat is that the hospital is a university hospital; hence, a disproportionately high level of curiosity about innovation and an affinity for technology might be assumed. Furthermore, the study was conducted in a multi-centre, city-based hospital. In smaller, less specialised, or more rural hospitals with less patient throughput, different sets of background factors might influence attitudes towards tele-ICU. Consequently, our study findings and conclusions should be not uncritically extrapolated to other hospitals or clinical environments. Nonetheless, some identified factors, such as the need to incorporate solutions on an organisational and personal level, can be considered valid in a broad range of settings.

In order to maintain anonymity, we did not explicitly ask whether or not the participants had taken part in both sections of the study. Some participants mentioned that they had. We do not believe this influenced the outcomes as the quantitative and qualitative studies were not conducted in direct succession. Finally, whilst most interviews were conducted in person at the hospital (10 interviews), due to interviewee preferences, four of the interviews were conducted remotely via ZOOM. Although it was slightly easier to establish a personal connection during in-person interviews, interviewers did not feel this affected the outcome.

Conclusions

The main findings of our study can help aid future strategies for implementing telemedicine in intensive care settings. We found that the value of even the best potential telemedical ICU technology is highly dependent on the organisational network and structures in which it is embedded. Furthermore, an organisational shift is needed, which reframes requests for support and external opinions as a means of ensuring excellent patient care and perhaps fostering continuous professional development. Assuming these “soft” factors of organisational culture and human networks are in place, the telemedical technology must be integrated into existing processes and structures as seamlessly as possible, be easy to use, and simple to keep sterile. Our findings also highlight the importance of considering the different groups of ICU employees (and hospital employees more generally), and showing which implementation measures could be tailored to more hesitant groups. This will allow for targeted measures to support individual employees during the implementation phase of telemedicine, rather than providing a one-size-fits-all training for all employees. This might include voluntary measures, such as training, in technology use for those wishing to participate. Not only does this give employees agency, it also saves personnel resources in often-overworked ICU settings by not requiring the participation of employees already comfortable using telemedicine. Adopting these changes will make the health system more resilient to future labour shortages and health crises.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076241257042 - Supplemental material for Focusing on experts: Expectations of healthcare professionals regarding the use of telemedicine in intensive care units

Supplemental material, sj-docx-1-dhj-10.1177_20552076241257042 for Focusing on experts: Expectations of healthcare professionals regarding the use of telemedicine in intensive care units by Magdalena Eitenberger, Gernot Gerger, Sophie Klomfar, Marcus Alexander Gabriel, Maria Kletecka-Pulker, Eva Schaden, Atanas G Atanasov, Mathias Maleczek, Sabine Völkl-Kernstock and Elisabeth Klager in DIGITAL HEALTH

Supplemental Material

sj-docx-2-dhj-10.1177_20552076241257042 - Supplemental material for Focusing on experts: Expectations of healthcare professionals regarding the use of telemedicine in intensive care units

Supplemental material, sj-docx-2-dhj-10.1177_20552076241257042 for Focusing on experts: Expectations of healthcare professionals regarding the use of telemedicine in intensive care units by Magdalena Eitenberger, Gernot Gerger, Sophie Klomfar, Marcus Alexander Gabriel, Maria Kletecka-Pulker, Eva Schaden, Atanas G Atanasov, Mathias Maleczek, Sabine Völkl-Kernstock and Elisabeth Klager in DIGITAL HEALTH

Footnotes

Acknowledgements

We would like to thank all participants for taking time to contribute to our research amidst the COVID-19 pandemic.

Contributorship

ME: Conceptualization, Methodology, Data collection, Data analysis, Writing Original Draft, Project Management; EK: Conceptualization, Methodology, Data analysis, Writing Original Draft; MG: Research, Data analysis, Writing Original Draft; SK: Methodology, Data collection, Data analysis, Writing Original Draft; GG: Data analysis, Writing Original Draft; MK-P: Supervision, Review of Draft; ES: Supervision, Distribution of Online-survey, Review of Draft; AGA: Review of Draft; MM: Review of Draft; SV-K: Data collection, Supervision

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

The study was confirmed by the Ethics Committee and the Data Protection Commission of the Medical University of Vienna (ECS 1256/2020).

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Guarantor

ORCID iDs

Magdalena Eitenberger

Marcus Alexander Gabriel

Elisabeth Klager

Supplemental material

Supplemental material for this article is available online.

Appendix

Table 2.

Participant characteristics for quantitative data.

Variable	Total(n = 63)	Physicians(n = 25)	Nurses(n = 38)
Sex, n (%)
Female	30 (47.6)	8 (32.0)	22 (57.9)
Age (in years), M (SD)	40.44 (9.66)	39.61 (10.18)	40.97 (9.42)
Working in the healthcare sector (in years), M (SD)	17.25 (11.15)	14.84 (10.37)	18.83 (11.49)
Working at an ICU (in years), M (SD)	13.60 (10.85)	9.71 (9.80)	16.16 (10.86)
Average number of patients cared for by day, n (%)
< 4	33 (52.4)	0 (0.0)	33 (86.8)
4–6	6 (9.5)	3 (12.0)	3 (7.9)
7–9	21 (33.3)	19 (76.0)	2 (5.3)
10–12	0 (0)	0 (0.0)	0 (0.0)
> 12	3 (4.8)	3 (12.0)	0 (0.0)
Average time required per patient, n (%)
0–60 min	30 (47.6)	23 (92.0)	7 (18.4)
61–120 min	3 (4.8)	1 (4.0)	2 (5.3)
121–180 min	11 (17.5)	1 (4.0)	10 (26.3)
> 180 min	19 (30.2)	0 (0.0)	19 (50.0)
Extent of concerns about own health before COVID-19 pandemic^a, M (SD)	3.12 (1.84)	3.09 (1.62)	3.14 (1.99)
Extent of concerns about own health since COVID-19 PANDEMIC^a, M (SD)	5.07 (2.56)	5.13 (2.42)	5.03 (2.68)
Extent of personal stress before COVID-19 pandemic^a, M (SD)	4.75 (2.11)	5.13 (2.12)	4.50 (2.09)
Extent of personal stress since COVID-19 pandemic^a, M (SD)	6.14 (2.41)	6.17 (2.06)	6.11 (2.64)
Job satisfaction score^b, M (SD), (min–max)	33.73 (3.85) (23–40)	33.87 (3.99) (23–40)	33.64 (3.81) (25–40)
General willingness to use technology—total score^c, M (SD), (min–max)	41.08 (4.73) (30–49)	42.04 (4.22) (36–49)	40.47 (4.99) (30–49)
Technology acceptance score^d, M (SD), (min–max)	16.23 (2.91) (9–20)	16.87 (2.93) (10–20)	15.83 (2.87) (9–20)
Technology control beliefs score^e, M (SD), (min–max)	7.29 (1.53) (4–10)	7.13 (1.89) (4–10)	7.39 (1.27) (4–10)
Technology competence beliefs score^f, M (SD), (min–max)	17.56 (2.23) (13–20)	18.04 (1.99) (13–20)	17.25 (2.35) (13–20)
Expectations for the use of telemedicine in an intensive care unit, total score^g, M (SD), (min–max)	27.64 (5.46) (8–40)	26.52 (6.02) (8–39)	28.36 (5.02) (11–40)
Expectations regarding a change in the number of patients to be cared for
Fewer patients expected, n (%)	5 (7.9)	2 (8.0)	3 (7.9)
No change in number of patients expected, n (%)	46 (73.0)	13 (52.0)	33 (86.8)
More patients expected, n (%)	12 (19.0)	10 (40.0)	2 (5.3)

n = number, M = mean, SD = standard deviation.

Extent measured using a scale of 1 to 10 with 1 = lowest and 10 = highest extent.

Total score of the standardised scale to assess job satisfaction (score could range from 8 = low to 40 = high satisfaction).

Total score of the standardised scale to assess general willingness to use technology (score could range from 10 = low to 50 = high technology willingness).

Score of the technology acceptance subscale (score could range from 4 = low to 20 = high technology acceptance).

Score of the technology control beliefs subscale (score could range from 2 = low to 10 = high technology control beliefs).

Score of the technology competence beliefs subscale (score could range from 4 = low to 20 = high technology competence beliefs).

Total score of items measuring expectations for the use of telemedicine in an ICU (score could range from 8 = low (more negative) to 40 = high (more positive) expectations).

References

Society of Critical Care Medicine. ICU readiness assessment: we are not prepared for COVID-19. https://www.sccm.org/Blog/April-2020/ICU-Readiness-Assessment-We-Are-Not-Prepared-for (2020, accessed 04 April 2023).

Litewka

Heitman

. Latin American healthcare systems in times of pandemic. Developing World Bioeth 2020; 20: 69–73.

Mahendradhata

Andayani

NLPE

Hasri

, et al. The capacity of the Indonesian healthcare system to respond to COVID-19. Front Public Health 2021; 9: 649819.

Ahmedov

Azimov

Mutalova

, et al. Uzbekistan: health system review. Health Syst Transit 2014; 16: 1–137.

Behmane

Dudele

Villerusa

, et al. Latvia: health system review. Health Syst Transit 2019; 21: i–165.

Chevreul

Berg Brigham

Durand-Zaleski

, et al. France: health system review. Health Syst Transit 2015; 17: 1–218.

Economou

Kaitelidou

Karanikolos

, et al. Greece: health system review. Health Syst Transit 2017; 19: 1–192.

Marć

Bartosiewicz

Burzyńska

, et al. A nursing shortage — a prospect of global and local policies. Int Nurs Rev 2019; 66: 9–16.

Nardi

Gyurko

. The global nursing faculty shortage: status and solutions for change. J Nurs Scholarsh 2013; 45: 317–326.

10.

Sowada

Sagan

Kowalska-Bobko

, et al. Poland: health system review. Health Syst Transit 2019; 21: 1–235.

11.

Tabah

Ramanan

Laupland

, et al. Personal protective equipment and intensive care unit healthcare worker safety in the COVID-19 era (PPE-SAFE): an international survey. J Crit Care 2020; 59: 70–75.

12.

Wahlster

Sharma

Lewis

, et al. The coronavirus disease 2019 pandemic’s effect on critical care resources and health-care providers. A global survey. CHEST 2021; 159: 619–633.

13.

Bender

Hiddleson

Buchman

. Intensive care unit telemedicine. Innovations and limitations. Crit Care Clin 2019; 35: 497–509.

14.

Hong

, et al. Telemedicine during the COVID-19 pandemic: experiences from western China. J Med Internet Res 2020; 22: e19577.

15.

Mahtta

Daher

Lee

, et al. Promise and perils of telehealth in the current era. Curr Cardiol Rep 2021; 23: 15.

16.

Canfield

Galvin

. Bedside nurse acceptance of intensive care unit telemedicine presence. Crit Care Nurse 2018; 38: e1–e4.

17.

Deslich

Coustasse

. Expanding technology in the ICU: the case for the utilization of telemedicine. Telemed J E Health 2014; 20: 485–492.

18.

Vidal-Alaball

Acosta-Roja

Pastor Hernández

, et al. Telemedicine in the face of the COVID-19 pandemic. Atención Primaria 2020; 52: 418–422.

19.

Rolston

Meltzer

. Telemedicine in the intensive care unit: its role in emergencies and disaster management. Crit Care Clin 2015; 31: 239–255.

20.

Fuhrman

Lilly

. ICU telemedicine solutions. Clin Chest Med 2015; 36: 401–407.

21.

Yeo

Grass Ahrens

Wright

. A new era in the ICU. The case for telemedicine. Crit Care Nurs Q 2012; 35: 316–321. 2020.

22.

Binder

Cook

Gramze

, et al.

Telemedicine in the intensive care unit: improved access to care at what cost?

Crit Care Nurs 2018; 30: 289–296.

23.

Caples

. Intensive care unit telemedicine care models. Crit Care Clin 2019; 35: 479–482.

24.

Vranas

Slatore

Kerlin

. Telemedicine coverage of intensive care units: a narrative review. Ann Am Thorac Soc 2018; 15: 1256–1264.

25.

Rosenfeld

Dorman

Breslow

, et al. Intensive care unit telemedicine: alternate paradigm for providing continuous intensivist care. Crit Care Med 2000; 28: 3925–3931.

26.

Kahn

Barnato

, et al. ICU telemedicine and critical care mortality: a national effectiveness study. Med Care 2016; 54: 319–325.

27.

Kopec

. Impact of intensive care unit telemedicine on outcomes. Crit Care Clin 2019; 35: 439–449. 2019.

28.

Ramnath

Maggio

, et al. Centralized monitoring and virtual consultant models of tele-ICU care: a systematic review. Telemed J E Health 2014; 20: 936–961.

29.

Udeh

Rahman

, et al.

Telemedicine/virtual ICU: where are we and where are we going?

Methodist Debakey Cardiovasc J 2018; 14: 126–133.

30.

Yoo

B-K

Kim

Sasaki

, et al. Economic evaluation of telemedicine for patients in ICUs*. Crit Care Med 2016; 44: 265–274.

31.

Young

Chan

, et al. Impact of telemedicine intensive care unit coverage on patient outcomes. A Systematic Review and Meta-analysis. Arch Intern Med 2011; 171: 498–506.

32.

Chen

Sun

Yang

, et al. Clinical and economic outcomes of telemedicine programs in the intensive care unit: a systematic review and meta-analysis. J Intensive Care Med 2018; 33: 383–393.

33.

Kahn

Cicero

Wallace

, et al. Adoption of ICU telemedicine in the United States. Crit Care Med 2014; 42: 362–368.

34.

Kalb

. Increasing quality through telemedicine in the intensive care unit. Crit Care Clin 2015; 31: 257–273.

35.

Venkataraman

Ramakrishnan

. Outcomes related to telemedicine in the intensive care unit: what we know and would like to know. Crit Care Clin 2015; 31: 225–237.

36.

Khurrum

Asmar

Joseph

. Telemedicine in the ICU: innovation in the critical care process. J Intensive Care Med 2020; 36: 1377–1384.

37.

Deisz

Rademacher

Marx

. Telemedizin: chancen und grenzen in der intensivmedizin. Anästhesiol Intensivmed Notfallmed Schmerzther 2017; 52: 94–106. (German).

38.

Johnson

. Satisfaction in intensive care unit telemedicine programs. Crit Care Clin 2019; 35: 511–517.

39.

Weiss

Paul

Balzer

, et al.

Telemedicine in the intensive care unit: a vehicle to improve quality of care?

J Crit Care 2021; 61: 241–246.

40.

Colbert

Verner Venegas-Vera

Lerma

. Utility of telemedicine in the COVID-19 era. Rev Cardiovasc Med 2020; 21: 583–587.

41.

Hincapié

Gallego

Gempeler

, et al. Implementation and usefulness of telemedicine during the COVID-19 pandemic: a scoping review. J Prim Care Community Health 2020; 11: –7.

42.

Peine

Paffenholz

Martin

, et al. Telemedicine in Germany during the COVID-19 pandemic: multi-professional national survey. J Med Internet Res 2020; 22: e19745.

43.

Ahmed

Sanghvi

Yeo

. Telemedicine takes centre stage during COVID-19 pandemic. BMJ Innov 2020; 6: 252–254.

44.

Mao

Liu

, et al. Intelligent intensive care unit: current and future trends. Intensive Care Res 2010; 3: 182–188.

45.

Teng

Ding

See

. Use of robots in critical care. Systematic Review J Med Internet Res 2022; 24: e33380.

46.

Ahmed

Baig

Bashir

, et al. Knowledge, attitudes, and perceptions related to telemedicine among young doctors and nursing staff at the King Abdul-Aziz University Hospital Jeddah, KSA. Niger J Clin Pract 2021; 24: 464–469.

47.

Elstad

Lutfey

Marceau

, et al. What do physicians gain (and lose) with experience? Qualitative results from a cross-national study of diabetes. Soc Sci Medic 2010; 70: 1728–1736.

48.

Shipton

West

Parkes

, et al. When promoting positive feelings pays: aggregate job satisfaction, work design features, and innovation in manufacturing organizations. Eur J Work Organ Psychol 2006; 15: 404–430.

49.

Tsai

C-T

Peng

J-C

. Does job dissatisfaction induce employee innovative behaviors? The moderating roles of TC and PSC. Proc AMIA Annu Fall Symp 2016; 1: 11416.

50.

Neyer

Felber

Gebhardt

. Entwicklung und Validierung einer Kurzskala zur Erfassung von Technikbereitschaft. Diagnostica 2012; 58: 87–99. (German).

51.

Plunger

Eitenberger

Kletecka-Pulker

, et al. Using telemedicine in nursing homes during the COVID-19 pandemic: a multi-perspective view on the implementation process. Nurs Open 2021; 9: 1155–1163.

52.

Östlund

Kidd

Wengström

, et al. Combining qualitative and quantitative research within mixed method research designs: a methodological review. Int J Nurs Stud 2011; 48: 369–383.

53.

Fetters

Curry

Creswell

. Achieving integration in mixed methods designs—principles and practices. Health Serv Res 2013; 48: 2134–2156.

54.

Wien

AKH

. Strukturdaten (2022). https://www.akhwien.at/default.aspx?pid=790 (2022, accessed 01 August 2022). (German).

55.

Tong

Sainsburry

Craig

. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007; 19: 349–357.

56.

Charmaz

. Constructing grounded theory. 2nd ed. London: Sage Publications, 2014, pp.192–224.

57.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77–101.

58.

Fischer

Lück

. Allgemeine Arbeitszufriedenheit. ZIS 1997. https://doi.org/10.6102/zis1. (German).

59.

R Core Team 2021. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/. [R Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/ (2021, accessed April 2023)].

60.

RStudio Team, 2020. RStudio: Integrated Development for R. RStudio, PBC, Boston, MA URL http://www.rstudio.com/. [RStudio Team. RStudio: Integrated Development for R. http://www.rstudio.com/ (2020, accessed April 2023)].

61.

Patil

. Visualizations with statistical details: the ‘ggstatsplot’ approach. J Open Source Softw 2021; 6: 3167.

62.

Elias

Smith

Barney

. Age as a moderator of attitude towards technology in the workplace: work motivation and overall job satisfaction. Behav Inf Technol 2012; 31: 453–467.

63.

Singh

. We are not phobic but selective”: the older generation’s attitude towards using technology in workplace communications. Development and Learning in Organizations 2014; 28: 18–20. https://doi/10.1108/DLO-10-2013-0082

64.

Avdalovic

Marcin

. When will telemedicine appear in the ICU? J Intensive Care Med 2019; 34: 271–276.

65.

Gagnon

Desmartis

Labrecque

, et al. Systematic review of factors influencing the adoption of information and communication technologies by healthcare professionals. J MED Syst 2012; 36: 241–277.

66.

Deisz

Marx

. Telemedizin in der Intensivmedizin — Möglichkeiten und Grenzen einer innovation. Med Klinik Intensivmed Notfallmed 2016; 111: 723–728. (German).

67.

Grata-Borkowska

Sobieski

Drobnik

, et al. Perception and attitude toward teleconsultations among different healthcare professionals in the era of the COVID-19 pandemic. Int J Environ Res Public Health 2022; 19: 11532.

68.

Fan

Yan

. Factors affecting response rates of the web survey: a systematic review. Comput Human Behav 2010; 26: 132–139.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB

0.02 MB