Sage Journals: Discover world-class research

Abstract

Online symptom checkers (SCs) are eHealth solutions that offer healthcare organizations the possibility to empower their patients to independently assess their symptoms. The successful implementation of eHealth solutions, such as SCs, requires a supportive organizational culture and leadership. However, there is limited knowledge about the factors associated with leaders’ support for the use of SCs. The aim of the study was to identify the factors associated to primary care leaders’ support for SCs in triage and their experiences of the benefits and challenges related to the use of SCs. An online survey was used to collect data from 84 Finnish primary care leaders. The data were analyzed using statistical analysis methods and content analysis. Vision clarity, perceiving efficiency improvements, and considering the service to be beneficial for patients were associated with leaders’ support for the service (β ranging from 0.41 to 0.44, p < 0.001). Leaders’ support for the service was also associated with how well the leaders provided information about the service to their subordinates (β =0.22, p < 0.048). SCs present slightly more challenges than benefits regarding health professionals’ work. The developers of SCs should focus more on features that decrease health professionals’ workload as well as how the solution can benefit patients.

Keywords

eHealth leadership online symptom checkers triage advice support health professionals primary care

Introduction

Online symptom checkers (SCs) are eHealth solutions that offer healthcare organizations the possibility to empower their patients to independently, and rapidly, assess their symptoms. SCs have been used, for example, by the patients with different symptoms or diseases, such as inflammatory arthritis,¹ HIV and hepatitis,² and COVID-19.³ It has been suggested that SCs may save resources by streamlining how patients are directed to the right level of care,^4,5 and during the COVID-19 pandemic, they have been evaluated to provide more reliable information for the patients than search engines.⁶

However, according to earlier studies, there are some conflicting results concerning the accuracy of the SCs.^6–8 Munsch et al.,⁶ who evaluated and compared the diagnostic accuracies of web-based COVID-19 SCs, concluded that a good balance between sensitivity and specificity was achieved only by two SCs out of ten. Although the accuracy, sensitivity, and specificity of the SCs are not very high, they have been suggested to replace telephone triage lines,⁶ and patients seem to express high levels of satisfaction when using SCs^9,10. However, there exists only little evidence that SCs may actually reduce healthcare resource use⁸ or increase clinical effectiveness,⁹ nor there is sufficient evidence to draw any conclusions of the cost effectiveness of SCs.⁹ In addition, little evidence indicates whether or not SCs are detrimental to patient safety,⁹ and complex patient cases may need a specialized diagnosis, requiring SCs to have their diagnostic algorithms to account for such complexity.² A recent study notified that there might even be a potential risk that using patient-led assessment tools, such as SCs, may lead to worsen outcomes due to the delaying appropriate clinical assessment.¹¹

SCs employ computerized algorithms to ask patients a set of questions about occurring symptoms.⁷ Based on the patients’ input, the SCs may provide information and medical advice concerning the patients’ symptoms, as well as a list of possible diagnoses.⁷ SCs also often conduct a triage, that is, the prioritization of care based on the urgency of the symptoms, for example, to recommend whether self-care or professional care is required.⁷ In the current study, the examined SCs provide patients advice but no diagnoses and health organizations use the SCs for supporting triage.

The successful implementation of eHealth, such as SCs, requires a supportive organizational culture and supportive leadership.^12,13 Leaders are more likely to support the implementation and use of eHealth if they have a clear vision of the eHealth solution,¹⁴ and they recognize its efficiency improvements¹⁵ and benefits for the patients.¹⁶ When leaders support the eHealth solution, they are more likely to provide information to health professionals,¹⁴ allocate resources for eHealth use,¹⁷ motivate professionals to use eHealth, and work closely with their subordinates.¹⁸ Leaders’ supportive attitudes toward eHealth were found to be associated with professionals’ support for the service during the pre-implementation phase;¹⁴ hence, leaders’ supportive behaviors may also increase health professionals’ willingness to use the service after it has been implemented.

SCs have most commonly been studied from a viewpoint of the patients,^1,10,19 and less attention has been paid to the viewpoints of health professionals and leaders in cases where SCs are used to support triage in health organizations. However, understanding leaders’ viewpoint would be essential because it may impact on the success of SCs implementation and willingness to implement and promote them among health professionals.^13,14 Successful implementation and use then again may impact on patients’ experiences in using SCs. In addition, whereas earlier studies have mostly focused on the implementation phase,^14,17,20,21 this study investigates healthcare leaders’ perspectives and supportive behaviors during the post-implementation phase of SCs. Identifying and understanding the factors influencing leaders’ supportive behaviors is important to ensuring that implemented SCs are successfully used, thus increasing the benefits for the health professionals and patients.

The objective of this survey study was to identify the factors associated with primary care leaders’ support for SCs in triage and their experiences of the benefits and challenges related to the use of SCs. Findings provide a better understanding of how novel SCs can support healthcare in triage and how healthcare providers can facilitate the implementation and use of SCs.

The research questions were:

1. What factors are associated with leaders’ support for SCs?

2. Which benefits and challenges do leaders associate with the use of SCs?

Methods

A cross-sectional survey study was conducted to capture leaders’ experiences of two SCs used in Finnish primary care. The questionnaire was targeted at supervisors and leaders. The study protocol was approved by the Ethical Review Board of Finnish Institute for Health and Welfare. The current study focused on two algorithm-based SC solutions adopted in primary care in Finland. Patients can use both solutions to receive guidance about appropriate care.

Study setting

In Finland, municipalities are responsible for arranging and funding healthcare, and they can either produce health services alone or in collaboration with other municipalities, or procure services from private healthcare providers.²² Public health services are divided into primary care and specialized medical care,²² and the current study specifically focused on primary care services.

Omaolo Oirearvio is a publicly funded SC, which was designed to direct patients to obtain appropriate care or provide self-care instructions. Omaolo SCs assess the necessity and urgency of care, provide recommendations for action, and automatically guide to appropriate help via regional service coordination. Omaolo Oirearvio includes several SCs for specific conditions such as low-back pain, urinary tract infection, and upper respiratory tract infection. During the COVID-19 outbreak, COVID-19 symptom assessment was added to the SC selection.²³ At the time of the survey of this study, the number of SCs available within this solution had increased to 16, including a generic checker that is not specific to any certain symptoms. At the end of September 2020, Omaolo Oirearvio was adopted in 79 municipalities, including the largest cities in Finland.

The second SC solution, called Klinik Access, is a condition generic tool allowing symptom checking and urgency assessment to prioritize patient care. Patients report their symptoms in a structured format and submit the report to the healthcare center.⁵ Klinik Access detects the most probable diagnoses and assesses urgency. Based on the given information, professionals initiate the care pathway. This solution was adopted in 34 municipalities and private primary care providers between 2016 and 2019.

Questionnaire

A previous study concerning the pre-implementation phase of eHealth solutions included a questionnaire that addressed expectations of a patient portal, including SCs, as one service.¹⁴ The questionnaire used in this study was a revised version of the one used in the previous study (available here https://osf.io/a6r2b/). Leaders were asked to evaluate their current experiences of the SC solution; therefore, the wording was changed from the future to the present tense.

The questionnaire included five existing validated Information Systems Expectations and Experiences survey items that were used to measure leaders’ support for the SCs.^14,24–27 As Ingebrigtsen et al.²⁸ identified, communicating vision and goals is one of the leadership behaviors associated with successful implementation, and we used the three-item vision clarity scale²⁹ to measure how well leaders perceive the SCs to serve the organization’s strategy. The vision clarity scale was previously found to be associated with organizational readiness²⁹ as well as leaders’ support.¹⁴

As leaders play a key role in preparing health professionals to use SCs, they were additionally asked to rate whether they had performed activities related to informing, orientation, and encouragement. Leaders could also use the data produced by SCs to guide their work; as such, they were asked to rate whether they had utilized data derived from SCs to monitor and lead their care work.^14,28. In addition, the leaders were asked to rate whether the SCs had brought the promised benefits to their work and patient care. As in the previous questionnaire,¹⁴ the leaders were also given two open-ended questions related to the benefits and challenges of the studied SCs.

Data collection

The data were gathered from February to September 2020 using the web-based questionnaire tool Webropol (Webropol, Helsinki, Finland). The initial data collection period was supposed to last until April 2020 but was extended due to the effects of the COVID-19 epidemic on Finnish primary care. A project manager from each participating organization sent a link to the questionnaire via email to all supervisors and leaders of units using the Omaolo Oirearvio SC. The respondents were asked to tell about their experiences of the SCs and provide feedback. A manager at Klinik Healthcare Solutions (Helsinki, Finland), which developed Klinik Access, sent survey invitations to the healthcare organizations that use their solution. Participation was voluntary and anonymous. To encourage participation, 50 movie tickets were raffled off among the respondents (supervisors, leaders, and professionals) of Omaolo Oirearvio questionnaire who submitted their email address in a separate questionnaire. In addition, three tablet computers sponsored by Klinik were raffled off among the Klinik Access users (supervisors, leaders, and professionals).

Statistical analysis

Descriptive statistics and Cronbach’s alpha coefficients of internal consistency were computed for all variables. Bivariate associations between the variables were examined using Pearson correlation analysis and Wilcoxon rank sum tests. Linear regression analysis, adjusted for respondent age and the health checker solution in question (“0” = Klinik Access, “1” = Omaolo Oirearvio), was used to examine the associations between key variables.

The key variables of interest were support for the service, vision clarity, perceived efficiency improvements, benefits for patients, and providing information about the service (monitoring and orientation activities were not included in the analysis because they were not regarded as meaningful predictors). Support for the service and providing information about the service were treated as the outcome variables and analyzed in separate models. Because many of the predictor variables were highly correlated, their relationships were also examined in separate models to avoid problems caused by multicollinearity.³⁰ All of the analyses were conducted using R version 3.6.1.³¹ Missing data were handled by excluding cases pairwise (correlations) or listwise (regression).

Content analysis

The responses to open-ended questions were analyzed using content analysis, which can be applied to various types of unstructured or semi-structured data to describe human experiences and perspectives about a research phenomenon.³² First, one researcher (EL) read through the data several times before starting the actual analysis, during which the text was coded according to correspondence with the research questions. Short sentences were chosen as the analytical unit. Next, the researcher compared the similarities and differences between the identified open codes to form sub-categories. In compliance with the privacy statement and in order to protect participants’ privacy, another researcher (OK) joined the analytical process after the raw data had been described as sub-categories. In the final step, the sub-categories were grouped into categories and main categories, with both researchers performing this step independently. The researchers discussed the similarities and differences in the categories they had formed and ultimately agreed on the final version. The data were also quantified.

Results

Respondents

A total of 84 primary care leaders (54 using the Omaolo service and 30 using the Klinik service) agreed to participate in the study and filled in the questionnaire. The sample was 90% female and had a mean age of 51.8years (SD 8.20). The respondents worked as nurse leaders (81%, n = 68), physician leaders (11%, n = 9), or leaders representing other healthcare professions (e.g., physiotherapist, 8%, n = 7). The majority of the respondents worked in an immediate supervisor position (77%, n = 65), while the rest worked either in middle or senior manager positions. The respondents whose unit used the Omaolo Oirearvio SCs represented 24.6% of the estimated number of leaders. The number of leaders using Klinik Access was not known, but the estimated response rate was 17.1% when we compared the number of all health professionals with the sum of the health professionals and leaders responding to their own questionnaires.

Factors associated with leaders’ support for SCs

Descriptive statistics and correlations between the study variables are presented in Table 1 and the distributions of the variables at https://osf.io/s3e4v/. Among the predictor variables, vision clarity, perceived efficiency improvements, and perceived benefits for patients were highly correlated (r ranging from 0.67 to 0.73).

Table 1.

Descriptive statistics and Pearson correlations (p values in parentheses).

Variable	N	M	SD	α	1	2	3	4	5	6	7
1. Leaders’ support	82	4.63	0.61	0.87	1
2. Vision clarity	83	4.51	0.59	0.78	0.44 (<0.001)	1
3. Efficiency improvements	81	3.80	1.03	0.91	0.45 (<0.001)	0.73 (<0.001)	1
4. Benefits for patients	75	3.98	0.75	0.78	0.41 (<0.001)	0.67 (<0.001)	0.69 (<0.001)	1
5. Quality of informing	78	4.09	0.88	0.78	0.28 (0.01)	0.31 (0.006)	0.17 (0.14)	0.25 (0.03)	1
6. Monitoring	76	4.00	1.02	0.86	0.20 (0.08)	0.31 (0.006)	0.40 (<0.001)	0.44 (<0.001)	0.44 (<0.001)	1
7. Orientation activities	73	4.36	0.54	0.69	0.10 (0.41)	0.15 (0.22)	0.09 (0.46)	0.16 (0.21)	0.71 (<0.001)	0.55 (<0.001)	1

Note. M = mean, SD = standard deviation, α = Cronbach’s alpha. The scales of the variables ranged from 1 (fully disagree) to 5 (fully agree), and also included option 6 (I don’t know), which was removed from the analysis.

The results of the regression analysis, in which the associations between independent variables and both leaders’ support for SCs and quality of informing were investigated, are presented in Table 2. Vision clarity, perceived efficiency improvements, and perceived benefits for patients were associated with leaders’ support for the service (β ranging from 0.41 to 0.44, p < 0.001). The leaders’ support for the service was associated with the quality of informing, that is, how well the leader disseminated information about the service to their subordinates (β=0.22, p < 0.048). According to Wilcoxon rank sum test, the respondents rated the Klinik service more favorably than the Omaolo service in terms of their support for the service (M = 4.77 vs 4.55, p = 0.03), perceived efficiency improvements (M = 4.17 vs 3.60, p = 0.02), monitoring service use (M = 4.57 vs 3.64, p < 0.001), and providing support and orientation activities to encourage service use (M = 4.55 vs 4.24, p = 0.007).

Table 2.

Results of the linear regression analyses.

Predictor	Outcome						Outcome
	Leaders’ support						Quality of informing
	β	p	β	p	β	p	β	p
Leaders’ support							0.22	0.048
Vision clarity	0.44	<0.001
Efficiency improvements			0.43	<0.001
Benefits for patients					0.41	<0.001
Age	0.02	0.862	0.03	0.757	0.05	0.612	0.24	0.028
Solution (Omaolo)	−0.35	0.095	−0.14	0.543	−0.39	0.085	−0.35	0.124
N	81		79		73		77
R ²	0.23		0.21		0.20		0.16

Note. All continuous predictors were mean-centered and scaled by one standard deviation. Statistically significant (p<0.05) associations are shown in boldface.

Benefits and challenges associated with the SCs

Answers to open-ended questions (n = 69) regarding the perceived benefits and challenges of SCs resulted in 214 lines of text. Primary care leaders identified slightly more challenges than benefits when assessing the SCs. Most of the described benefits were related to health professionals’ perceived workload, workflow, and work processes. The most commonly mentioned benefit of using SCs was the decreased number of phone calls and other contacts (e.g., visits to health centers), which were linked to decreased perceived workload.

“When SCs are used effectively, they decrease the number of phone calls.”

“The time consumed on phone calls can be used for direct patient work.”

The use of SCs seemed to enhance the health professionals’ workflow and work processes in some cases. For example, using the SCs accelerated the triage process. Because the patients filled in the SCs prior to meeting health professionals, the professionals already had some information concerning the patients and their conditions instead of starting from scratch. Leaders perceived it was faster for health professionals to schedule appointment when they already had some information concerning the patients’ condition, and in some cases, professionals were able to utilize prefilled data when reporting to electronic health record. On the other hand, the most common perceived benefit for patients was improved access to health services. SCs provided patients a new way to contact their healthcare providers that involved a lower threshold than traditional care. In addition, instructions for self-care were perceived to empower patients.

“[SCs] lower threshold for care, for example, when the young patients are seeking care for sexually transmitted diseases.”

Based on the responses, the leaders felt that using SCs made their organization more modern, which was especially important in face of the COVID-19 pandemic, as SCs enabled organizations to better manage the challenges associated with this rapidly changing situation. The benefits for organizations and management were related to responsiveness and management support. Improvements in the quality of informing were mainly related to useful reports for managing service operations.

“This [the use of SCs] tells that the organization modernizes.”

Most of the described challenges were related to health professionals’ perceived workload. Although SCs had the potential to decrease perceived workload via a reduction in phone calls and visits, leaders occasionally felt that it was more effective for health professionals to contact patients via telephone to obtain more detailed information about their health status. Thus, SCs did not always provide enough information, which introduced further tasks that, in turn, increased the professionals’ perceived workload.

“The SC does not exclude the need for specifying things over phone…”

SCs were also occasionally experienced as an extra task which disturbed professionals’ concentration. The responding leaders perceived several patient-related challenges, notably, some patients with insufficient online skills or the lack of devices may not be able to use the service. In addition, SCs were perceived to be especially complicated for elderly people to use.

“The clients are elderly people, who are not used to smart devices.”

Discussion

Principal findings in comparison with prior work

Leaders play an important role in supporting the use of eHealth solutions such as SCs. This study aimed to describe primary care leaders’ experiences of using SCs. Vision clarity, perceived efficiency improvements, and perceiving that the service benefits patients were all associated with leaders’ support for the service.

Having a clear vision has previously been recognized as a pivotal component of the eventual success of eHealth implementation.²⁸ Our results are also in line with the earlier finding that leaders’ support for a service is associated with how well they provide information about the service to their subordinates.¹⁴ As all leadership levels should communicate a clear vision and the goals of the new service,²¹ it is important that all leaders support the service and communicate the clear vision further. In this study, SCs seemed to support an organization’s strategy as they made organizations more responsive to the current situation, that is, disruptions caused by COVID-19. According to another study, SCs have been widely used in response to the COVID-19 pandemic.³ It is important that leaders understand the vision so that they can inform health professionals about it.¹⁴ However, as Ceney et al.⁸ noted, there exists confusion about the actual purpose of SCs, whether their purpose is to help with patients’ information, improve the access to care or reduce service load, or the mixture of all the previously mentioned. In our study, it seems that the leaders expect the mixture since the perceived efficiency improvements and perceived benefits for the patients associated with their support for the SCs service.

In this study, leaders perceived that SCs both improved and reduced efficiency, with the challenges more commonly highlighted. In terms of benefits, leaders reported that SCs could decrease the number of phone calls and other contacts, a finding which is supported by other study.³³ However, it is important to note that the decreased number of phone calls was mentioned as a possibility, yet numerous leaders felt that it was not always realized. For example, SCs provided insufficient information in complex patient cases; as such, the health professionals would have to call the patient by phone. It seems as though the leaders did not always identify efficiency improvements, which may decrease their support towards SCs. This is in line with a previous study in which health professionals believed that SC caused extra work for them and was even a threat to their professional autonomy.³⁴ According to a recent systematic review, evidence on the clinical effectiveness of SCs remains scant, and a majority of empirical investigations have reported that these tools demonstrate poor diagnostic accuracy in relation to the skills of health professionals.⁹ The poor accuracy of SCs may explain why certain leaders felt that they increased workload of health professionals. In addition, there seems to be lacking knowledge of SCs benefits regarding, for example, healthcare resources.⁸

Leaders’ perceptions of SC benefits for patients were associated with leaders’ support in the current study. However, in a study concerning pre-implementation phase, the benefits for the patient did not have a strong association with leaders’ support.¹⁴ It might be that the benefits for the patients become more tangible and easier to see after the implementation. However, if the leaders would be able to see the benefits in the pre-implementation phase, they might be able to argue for the benefits of SCs for health professionals and provide them with more information. For health professionals, the benefits for the patients seem to be an important source of motivation to use eHealth solutions.^27,35 According to a recent study, a large patient group perceived SCs as useful.¹⁰ However, there exists limited evidence on how patients react to advice provided by SCs.⁹

In our study, leaders identified several ways through which SCs benefit patients, for example, increased accessibility to health services and a lower threshold for care. Increased accessibility and lower threshold for care actualized because SCs were available around the clock and it was easier and, in some cases, less embarrassing to use them instead of meeting health professional. However, these benefits were not perceived as being available for all patients due to differences in information technology skills and ownership of the required devices. For example, the leaders participating in this study felt that elderly patients infrequently use SCs. An earlier finding that the users of SCs are more likely to be young supports this result.^9,36 In addition to age, health literacy may also limit the use of SCs. For example, previous research has shown that patients with low health literacy are less likely to use eHealth or perceive it as useful than other patients.³⁷ It may be difficult for patients with low health literacy to understand complex medical terms, and this should be noted when developing SCs and marketing them to the general public. For example, the model presented by Bodie and Dutta³⁸ suggests that patients with low health literacy could be supported through alternative forms of online health information, such as content with text that is easier to understand.

The comparison of two SC solutions currently used in Finland revealed that one seemed to receive more support than the other based on both quantitative and qualitative analyses. This might be explained by several distinct differences between the solutions. For example, in contrast to Omaolo Oirearvio, the Klinik Access solution also provides healthcare personnel with a triage tool that suggests a diagnosis and urgency level for each patient. It also seemed to support strategic management by providing good reports for the leaders.

Practical implications and future research

As our study has proposed, leaders are more likely to support the use of SCs when they have a clear vision, and they perceive efficiency improvements and benefits for the patients. In addition, our study provides more detailed information about how leaders view SCs and their benefits in clinical work. However, although our study provides some insights of the benefits regarding the use of SCs, we suggest that more research should be targeted on this matter from the viewpoints of patients, health professionals, and organizations. Our study also provides some challenges concerning the use of SCs. Understanding these challenges is important to develop more efficient eHealth solutions that support both the work of the employees and the care of the patients and result in more evidence-based SCs and better standards for patients. In particular, SCs should support health professionals’ and leaders’ work tasks, decrease their workload, and benefit all the patients. The clinical work process should be redesigned to guide patients in an efficient manner.

The finding that leaders’ support towards SC may be influenced by the perceived benefits for patients means that it would be important to also study the use of SCs from a patient’s perspective. There seems to be a lack of knowledge regarding whether patients follow the advice that they gain from the SCs.⁹ Leaders might be more supportive of SCs if they would fully understand the benefits for patients. Moreover, this study provided evidence that SCs should be developed so that they decrease health professionals’ workload instead of increasing them.

Limitations

Because the study was cross-sectional in design, no indications of causality can be determined from the results. The relatively small sample size and use of self-reported measurements mean that the reported associations between variables may be artificially inflated. Furthermore, as there is no statistics of the leaders available, we do not know whether the demographic distribution was representative. However, the sample covered all age groups, main professional backgrounds, immediate supervisor, middle and senior manager positions, and most of the target units. It is important to note that the content analysis was conducted independently by two researchers, which strengthens the dependability of the analysis.³⁹ However, these results may not be transferable to other settings as various SCs are used across many healthcare contexts and for numerous purposes. Since there was a difference between the value of the incentives raffled between the two groups, it could have influenced the number of participants from each group. However, the value of the incentive did not seem to impact on the response rates of the groups.

Conclusion

Our findings suggest that vision clarity, perceiving efficiency improvements, and understanding the benefits for patients are associated with leaders’ support for the use of SCs. Furthermore, leaders who support the use of SCs are more likely to provide information about the SCs to their subordinates. However, the leaders participating in this study reported that they experience more challenges than benefits from the SCs currently implemented in their units. The presented results indicate that the developers of SC solutions should focus more on features that decrease and/or simplify health professionals’ workload as well as how the solution can benefit patients and healthcare organizations.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Strategic Research Council at the Academy of Finland (grant numbers 327145 and 327147).

ORCID iDs

Elina Laukka

Outi Kanste

References

Powley

Mcilroy

Simons

, et al.

Are online symptoms checkers useful for patients with inflammatory arthritis?

BMC Musculoskelet Disord. 2016;17(1):362 DOI: 10.1186/s12891-016-1189-2

Berry

Cash

Wang

, et al. Online symptom checker diagnostic and triage accuracy for HIV and hepatitis C. Epidemiol Infect. 2019;147(104):e104.

Perlman

Zilberg

Bak

, et al. Characteristics and symptoms of app users seeking COVID-19–related digital health information and remote services: Retrospective cohort study. J Med Internet Res. 2019;22(10):e23197.

Marco-Ruiz

Bønes

de la Asunción

, et al. Combining multivariate statistics and the think-aloud protocol to assess human-computer interaction barriers in symptom checkers. J Biomed Inform. 2017;74:104–122. DOI: 10.1016/j.jbi.2017.09.002

Tenhunen

Hirvonen

Linna

, et al. Intelligent patient flow management system at a primary healthcare center-The effect on service use and costs in decision support systems and education: Help and support in healthcare. Stud Health Technol Inform. 2018;255:142–146.

Munsch

Martin

Gruarin

, et al. Diagnostic accuracy of web-based COVID-19 symptom checkers: comparison study. J Med Internet Res. 2020;22(10):e21299.

Semigran

Linder

Gidengil

, et al. Evaluation of symptom checkers for self-diagnosis and triage: Audit study. BMJ. 2018;351:h3480.

Ceney

Tolond

Glowinski

, et al. Accuracy of online symptom checkers and the potential impact on service utilisation. PLoS One. 2021;16(7):e0254088. DOI: 10.1371/journal.pone.0254088

Chambers

Cantrell

Johnson

, et al. Digital and online symptom checkers and health assessment/triage services for urgent health problems: Systematic review. BMJ Open. 2019;9(8):e027743.

, et al. Patient perspectives on the usefulness of an artificial intelligence—Assisted symptom checker: Cross-Sectional Surv Study. 2020;22(1):e14679.

. Performance of national COVID-19 symptom checkers’: A comparative case simulation study. BMJ Heal Care Inform. 2021;28(1):1–7.

12.

Cresswell

Sheikh

. Organizational issues in the implementation and adoption of health information technology innovations: An interpretative review. Int J Med Inform. 2013;82(5):e73–e86. DOI: 10.1016/j.ijmedinf.2012.10.007

, et al. Identifying the roles of healthcare leaders in HIT implementation: A scoping review of the quantitative and qualitative evidence. Int J Environ Res Public Health. 2020;17(8):2865.

, et al. The role of frontline leaders in building health professional support for a new patient portal: Survey study. J Med Internet Res. 2019;21(3):e11413.

. State of content: Healthcare executive’s role in information technology adoption. J Serv Sci Manag. 2012;5(02):124–131.

, et al. Introduction of shared electronic records: Multi-site case study using diffusion of innovation theory. BMJ. 2008;337:a1786.

, et al. Towards successful digital transformation through co-creation: A longitudinal study of a four-year implementation of digital monitoring technology in residential care for persons with dementia. BMC Health Services Research. 2019;19(1):366–417.

, et al. Middle managers’ experiences and role in implementing an interactive tailored patient assessment eHealth intervention in clinical practice. CIN—Comput Inform Nurs.2015;33(6):249–257.

, et al. Patients’ utilization and perception of an artificial intelligence-based symptom assessment and advice technology in a British primary care waiting room: Exploratory pilot study. JMIR Hum Factors. 2020;7(3):e19713.

20.

Deokar

Sarnikar

Understanding process change management in electronic health record implementations. Inf Syst E-Business Manag. 2016;14:733–766.

, et al. The influence of context and process when implementing e-health. BMC Med Inform Decis Mak. 2009;9(1):9.

22.

Ministry of Social Affairs and Health . Social welfare and healthcare in Finland. 2019. Available from: https://stm.fi/en/social-and-health-services.

23.

Ministry of Social Affairs and Health . Digital coronavirus information service now open in the public. 2020. Available from: https://stm.fi/en/-/sahkoinen-oirearviointi-koronavirusepailysta-kayttoon-koko-maahan.

24.

Ward

Vartak

Schwichtenberg

, et al. Nurses’ perceptions of how clinical information system implementation affects workflow and patient care. CIN—Comput Inform Nurs 2011;29(9):502–511.

25.

Wakefield

Halbesleben

JRB

Ward

, et al. Development of a measure of clinical information systems expectations and experiences. Med Care. 2007;45(9):884–890.

26.

Kirkendall

Goldenhar

Simon

, et al. Transitioning from a computerized provider order entry and paper documentation system to an electronic health record: Expectations and experiences of hospital staff. Int J Med Inform. 2013;82(11):1037–1045. DOI: 10.1016/j.ijmedinf.2013.08.005

, et al. Health professionals’ expanding eHealth competences for supporting patients’ self-management. Stud Health Technol Inform. 2018;247:181–185.

, et al. The impact of clinical leadership on health information technology adoption: Systematic review. Int J Med Inform. 2014;83(6):393–405. DOI: 10.1016/j.ijmedinf.2014.02.005

29.

Paré