Assessing Public Perceptions of Virtual Primary Care During the COVID-19 Pandemic in the UK,Germany,Sweden,and Italy: A Topic Modeling Approach

Abstract

The COVID-19 pandemic has driven the transition from face-to-face visits to virtual care delivery. In this study, we explore patients’ perceptions of the benefits and challenges of using virtual primary care technologies during the pandemic, using machine learning approaches. A cross-sectional survey was conducted in August 2020 in Italy, Sweden, Germany, and the UK. Latent Dirichlet Allocation was used to identify themes of two open-ended questions. Comparisons between participant characteristics were made using Wilcoxon rank-sum test. 6,331 participants were included (51.7% female; 42.4% +55 years; 60.5% white ethnicity; 86.6% low literacy). The benefits extracted included: primary care delivery, infection control, reducing contacts, virtual care, timeliness, patient-doctor interaction, convenience, and safety. Participants from Sweden were most likely to mention “primary care delivery” (UK p = .007, IT p = .03, DE p < .001), from the UK “virtual care” (SE p < .001, IT p < .001, DE p < .001) and from Italy “patient-doctor interaction” (UK p < .001, SE p < .001, DE p < .001). The challenges included: diagnostic difficulties, physical examination, digital health risks, technical challenges, virtual care, data security and protection, and lack of personal contact. “Diagnostic difficulties” was most significantly mentioned in Sweden (UK p = .009, IT p < .001, DE p < .001), “virtual care” in the UK (IT p = .02, SE p = .001, DE p < .001), and “data security and protection” in Germany (UK p < .001, IT p = .019, SE p < .001). Our study reinforces the feasibility of using machine learning to explore large qualitative datasets. Our findings contribute to a better identification of the lessons learned during the pandemic and inform improvements in policy and practice.

Plain language summary

The COVID-19 pandemic has driven the transition from face-to-face visits to virtual primary care delivery and has made online consultations the “new normal” for healthcare providers and patients While there is some evidence as perceived by General Practitioners, patients’ voice has been seldom heard. In this study, we explore patients’ experiences and perceptions of the key benefits and challenges of using virtual primary care technologies during the COVID-19 pandemic, using machine learning approaches. A public, online cross-sectional survey was conducted in August 2020 in Italy, Sweden, Germany, and the UK. Latent Dirichlet Allocation was used to identify the main themes of two open-ended questions. The benefits extracted included: primary care delivery, infection control, reducing contacts, virtual care, timeliness, patient-doctor interaction, convenience, and safety. The challenges included: diagnostic difficulties, physical examination, digital health risks, technical challenges, virtual care, data security and protection, and lack of personal contact. The statistical comparison between age groups, gender, countries of residence, and eHealth literacy generally showed the most significant differences between countries. Our findings contribute to better identification of the lessons learned and inform rapid improvements in the policy and practice of virtual primary care. The results underline the increasing relevance of digital health technologies in primary care and the growing acceptance and demand among patients. However, the study also highlights the risks that arise from their use, such as threats to privacy, inaccurate diagnoses, or the lack of personal contact. While the algorithm provided an overview of representative topics, granular insights on patient perception, as well as contextual and domain-specific analysis were not possible.

Keywords

primary care telemedicine COVID-19 topic modeling Latent Dirichlet Allocation natural language processing

Introduction

Background

The COVID-19 pandemic has caused unprecedented disruption in many European countries. The requirement to implement strong social distancing measures such as restrictions on gatherings or lockdowns to protect patients and providers resulted in a massive adoption of virtual care in General Practice. As a response, many European countries have developed national guidance on reshaping primary care, removed regulatory barriers, thus further accelerating the uptake of virtual primary care (Richardson et al., 2020).

Germany’s deficits in the digitalization of the healthcare system were addressed by the Digital Health Care Act (Digitale-Versorgung-Gesetz) enabling remote consultations and digital prescriptions, promoting the integration of virtual care into the healthcare system and facilitating reimbursement for digital health services (Bertelsmann Stiftung, 2019; German Federal Ministry of Health, n.d.; Wosik et al., 2020). During the pandemic the German Association of Statutory Health Insurance Physicians and the Association of Health Insurance Funds have extended the possible number of cases and the scope of services that can be provided via video consultation (Gerke et al., 2020). Sweden has long been at the forefront of incorporating virtual care into its healthcare system, even before the pandemic, in areas such as remote monitoring, teleconsultations, primary care, triage, and advice services (Björndell & Premberg, 2021; Ekman et al., 2019; Gabrielsson-Järhult et al., 2021). In the United Kingdom, the National Health Service (NHS) Long Term Plan has promoted the integration of digital health technologies, including virtual care, to improve patient access and experience (NHS England, 2019). The NHS has also launched platforms such as NHS Digital and NHS App, which provide access to virtual consultations, online prescriptions, and digital health records (NHS Digital, n.d.). Although the availability and diffusion of large-scale remote care solutions were initially limited, the pandemic has prompted the issuance of various legislative acts (Clarke, 2023; Omboni, 2020). In Europe, initiatives and policies, such as the Digital Health and Care Strategy, the European Health Data Space and Digital Green Certificates (COVID-19 passports) have been launched to foster innovation, enhance health services and improve interoperability of digital solutions across countries (European Commission, n.d.). Virtual care has been embraced as a suitable means of bridging the gap in service delivery and improving healthcare delivery during the pandemic. However, the rapid adoption has raised concerns about the quality of care, potential inequalities in access to care, lack of trust among users and not replacing in-person consultations (European Patients’ Forum, 2020; Webster, 2020). While for many patients this shift was their first exposure to virtual care, this mode of delivery is likely to become a common service model in the post-pandemic future. Therefore, the pandemic can be seen as a unique opportunity to explore the potential long-term impact of virtual primary care on the quality and safety of care by learning from its main users: patients.

The integration of digital technologies into healthcare has led to changes in the way patients perceive and engage with health services. The introduction of digital health technologies has improved information sharing, facilitated decision-making processes, created pathways for information sharing and support, strengthened self-management skills and encouraged better health promotion (Frank, 2000). Research shows that patients’ ability to access and rely on digital information influences their decisions about healthcare providers (Vuong et al., 2022). Some studies reported high levels of satisfaction with virtual care services in primary care and outpatient settings during the pandemic (Holtz, 2021; Imlach et al., 2020; Morgenstern-Kaplan et al., 2021; Ramaswamy et al., 2020; Vosburg & Robinson, 2021). A cross-sectional study in the United States found that patients who experienced remote care first during the pandemic were motivated by avoiding long waiting times and the risk of infection (Holtz, 2021). However, in contrast to patients who already used remote services, they continued to wish for face-to-face visits (Holtz, 2021). A mixed-methods study conducted in New Zealand found that patients using remote care consultations during lockdown reported high user satisfaction (Imlach et al., 2020). The study also found that remote care was perceived to be most suited for scenarios where a patient-provider relationship already existed, for example in the context of routine check-ups and known health problems. Remote care was considered less appropriate when a physical examination was required, when the diagnosis was unclear and for patients who preferred face-to-face visits (Imlach et al., 2020). Most of the studies mentioned above focused on patient satisfaction only and on specific geographical areas, lacking demographic diversity, and scarcely characterizing the digital literacy level of the users. There is, therefore, an opportunity to explore patients’ experience and perspectives on the use of virtual care in European primary care systems, in particular, capitalizing on large-scale datasets and data mining approaches to extract novel insights about what worked, and what needs to be improved.

Topic modeling is a type of statistical modeling technique used to extract valuable information and patterns from large amounts of unstructured, text-based data. Various machine learning algorithms have been utilized for topic modeling, including Non-negative Matrix Factorization (NMF) or Probabilistic Latent Semantic Analysis (PLSA; Hofmann, 1999; Lee & Seung, 1999). Latent Dirichlet Allocation (LDA), introduced by Blei et al. (2003), is a probabilistic generative model used to discover latent topics and their distribution in large sets of unlabeled data (Blei et al., 2003). Topic modeling is gaining interest in qualitative research for analyzing of prescription data, clinical reports, patient feedback or databases, for example, to identify core ventilation technologies to be used during the pandemic (Arnold & Speier, 2012; Durmuşoğlu & Durmuşoğlu, 2022; Park et al., 2017). Due to its automated approach, topic modeling offers a quick and efficient way to conduct exploratory, unsupervised analysis surpassing other qualitative methods such as manual coding, grounded theory or content analysis. The data-driven nature of topic modeling enables the identification of emerging topics, tracking topic changes over time and assessing the topic prevalence (Griffiths & Steyvers, 2004). This can be particularly useful in emergencies such as a pandemic, when a rapid assessment is needed to inform policy and practice. However, topic modeling has limitations in terms of interpretability, as it may generate ambiguous or overlapping topics without specific domain expertise (Chang et al., 2009). Additionally, it lacks contextual understanding and semantic meaning of text, and requires significant pre-processing steps.

Aim

While there is some evidence assessing the benefits and challenges, as perceived by General Practitioners, the patients’ voice has been seldom heard. In this study, we systematically explore and synthesize patients’ experiences and perceptions of the key benefits and challenges of using virtual primary care technologies during the COVID-19 pandemic, using LDA as a machine learning technique for topic modeling. The findings allow us to learn from this major real-life experiment of virtual care provided during the pandemic and inform the delivery of virtual care in the post-pandemic future.

Methods

Study Design and Participants

This study is a secondary analysis of data collected in a cross-sectional online survey. The survey was hosted on the online platform Qualtrics. Participants were included if aged 18 years and over and able to speak, read and write in the official language of their country. Recruitment took place in August 2020 in partnership with YouGov, including via standard advertising, and strategic partnerships with a broad range of websites. Stratified sampling was used to recruit participants from four different healthcare settings (Germany, Italy, Sweden, and the UK). The samples were nationally representative in terms of age, sex, social class, and education level. The survey was conducted in the respective mother tongue of the country. The survey adhered to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guideline for cross-sectional studies (Elm et al., 2007). The detailed methodology has been published elsewhere (Neves et al., 2021).

Survey Development

The survey was developed based on a rapid review of the subject and expert consultation. Table 1 shows the four survey sections and their respective questions. The complete survey questions are provided elsewhere (Neves et al., 2021). This study focused on the analysis of the demographic data and two open-ended questions evaluating the patients’ perception of the main benefits and challenges of using virtual care during the COVID-19 pandemic and were used for topic modeling.

Table 1.

Survey Structure and Types of Questions.

#	Section	Questions (n)	Survey question types
1	Demographic data and digital literacy	13	Close-ended questions (multiple choice) and set of questions using a 5-point Likert scale (1: strongly disagree, 5: strongly agree) as part of the eHEALS
2	Use of virtual care before and during the COVID-19 pandemic	3	Close-ended questions (multiple choice) and question on patient experience using a 5-point Likert scale (1: very bad, 5: very good)
3	Impact on quality and safety of care	15	Close-ended questions (multiple choice)
4	Benefits and challenges	2	Open-ended questions

Data Collection

The patient characteristics included sex (male, female), age groups (18–24, 25–34, 35–44, 45–54, 55+), and country of residence (Germany, Sweden, Italy, the UK). Digital literacy was assessed using the eHealth Literacy Scale (eHEALS) tool, which identifies six core literacy skills: traditional literacy, health literacy, information literacy, academic literacy, media literacy, and computer literacy. Based on these competencies, the eHEALS tool assesses participants’ knowledge, comfort, and perceived ability to find, evaluate, and apply digital health information (Norman & Skinner, 2006). The validity and reliability of eHEALS were demonstrated across different health conditions and the score was translated into different languages (Paige et al., 2017; van der Vaart et al., 2011).

Perceptions of benefits and challenges of using virtual care during the COVID-19 pandemic were assessed using the following free-text questions: “In your opinion, what do you think would be the main benefits of using virtual technologies when accessing primary care during the COVID-19 pandemic?” and “In your opinion, what are the main challenges of using virtual technologies when accessing primary care during the COVID-19 pandemic?”

Data Preparation

Text-based responses were translated by the research team using the artificial intelligence language translation software DeepL (version 2.6.63019, DeepL, 2017). Translations were validated using back-translation. The eHEALS was calculated as the sum of all instrument items ranging from 8 to 40, with higher scores reflecting higher levels of eHealth literacy. The eHEALS score was categorized into high eHealth literacy (eHEALS ≥ 26) and low eHealth literacy (eHEALS < 26), in line with previous studies (Chung et al., 2018; van der Vaart et al., 2011).

Topic Modeling Using Latent Dirichlet Allocation

The two open-ended questions were analyzed using an LDA topic modeling approach. Figure 1 shows the consecutive steps of text-pre-processing, creating document-term matrices, determining the number of topics, LDA topic modeling, and manual labeling of the topics.

Figure 1.

The steps of data processing and modeling including respective R packages used for the creation of LDA models.

The text data underwent automated pre-processing using R Studio (version 1.4.1717, R Core Team, 2021). Initially, the data set was imported as a .CSV UTF-8 file. Unwanted noise, such as double spaces, multiple newlines, punctuation, numbers, and stop words was removed using the text mining framework of the tm package (version 11-7, Feinerer et al., 2023) The text was converted to lowercase using the stringr package (version 1.5.0, Wickham, 2023) The tidyr (version 1.3.0) and dplyr (version 1.1.2) packages were used to break each row of the data set down into a list of terms called “tokens” and to create sequences of tokens known as “n-grams” (Wickham, François, et al., 2023; Wickham, Vaughan, et al., 2023). For example, the tokens “reduce,”“risk,” and “infection” were used to create the bi-grams “reduce_risk” or “risk_infection” and the tri-gram “reduce_risk_infection.”

Text-pre-processing enabled the creation of document term matrices using the textmineR package (version 3.0.5, Jones et al., 2021). The matrix captures the frequency of tokens in each document and represents the corpus, the entire body of contextual data. Lemmatisation was applied during this process as well, using the textstem package (version 0.1.4) to shorten terms to their base dictionary form (Rinker, 2018). For example, “reduces,”“reduced,” or “reducing” were converted to their lemma “reduce.” As a result of all text processing so far, the sentence “the main benefit is undoubtedly the possibility of avoiding the risk of infection” was transformed into “main benefit undoubtedly possibility avoid risk infection.”

To select the best fitting LDA model, three model comparison metrics (Arun et al., 2010; Cao et al., 2009; Deveaud et al., 2014) were utilized applying the ldatuning package (version 1.0.2, Arun et al., 2010; Cao et al., 2009; Deveaud et al., 2014; Nikita & Chaney, 2020). The metrics serve as quantitative measures for assessing the coherence of the generated topics. The Arun et al. (2010) metric calculates the average log-likelihood of words in relation to topics and the average log-likelihood of topics in relation to documents. It provides a balance between model complexity and model coherence (Arun et al., 2010). The Cao et al. (2009) metric considers the trade-off between the log-likelihood of the model and the divergence between topics. A low value indicates better model quality, meaning that the topics are well-separated and easily interpretable (Cao et al., 2009). The Deveaud et al. (2014) metric assesses the relevance of the generated topics to the underlying data. It measures the probability of a word occurring in a document when it is associated with a topic. Higher values indicate more relevant and meaningful topics (Deveaud et al., 2014). Applying the three metrics helps to determine manually the number of topics that maximize coherence, interpretability, distinctiveness, and relevance.

In the final stage of the analysis, the LDA algorithm was applied to the document-term matrix to extract topics based co-occurrence terms using the topicmodels package (version 0.2-12, Grün et al., 2023).

The process of generating topics in LDA involves probabilistic modeling and iterative optimization that go beyond mere word frequency, identifying groups of words that co-occur and represent meaningful themes. The LDA yields two outputs: (a) the probability of each term belonging to a certain topic, and (b) the probability of each topic belonging to a row of the data frame. As the algorithm did not automatically provide topic labeling, manual labeling was conducted with other researchers. Drawing from our collective expertise, we labeled the topic-based on the top ten terms or n-grams with the highest marginal probability within each topic. One of the topics, referred to as “no answer provided” mostly covered responses that lacked meaningful information regarding the research questions and included responses such as “I do not know” and “no answer.” Furthermore, to provide an alternative visualization of the data, word clouds were also created from the document-term matrices. These show the frequency of additional n-grams in the matrices.

The LDA model is based on the assumptions that word order does not matter and thus the distribution of words in the data within each topic is independent of the documents, that each document has only a few topics with non-zero probabilities, and that each topic consists of only a small subset of words. These assumptions were manually checked by the research group by examining the topics for coherence, relevance, and interpretability.

Finally, a sensitivity analysis was conducted to examine a change in the number of topics (n = 9, n = 10) on the composition of the topics.

Statistical Analysis

Comparisons between participant characteristics on the frequency of responses for each topic were made using Wilcoxon rank-sum test. Summary statistics for the variables are presented as mean. p-values were adjusted for multiple comparisons using False Discovery Rate (FDR) correction when performing Wilcoxon rank-sum test. Results with p-values <.05 were considered statistically significant. The abbreviations used for the country names follow the ISO 3166 ALPHA-2: Germany (DE), Sweden (SE), Italy (IT), and the United Kingdom (UK).

Ethics

Ethical approval was granted by Imperial College London’s Ethics Research Committee (ICREC; Approval number: 20IC5956).

Results

Participant Characteristics

Table 2 summarizes the characteristics of the survey participants (n = 6,331). The gender distribution of participants was balanced (51.7% female, n = 3,272). Most participants were over 55 years old (42.4%, n = 2,687) and of white ethnicity (60.4%, n = 3,289). Most participants showed a low eHealth literacy (86.8%, n = 5,497). For participants with low eHealth literacy (eHEALS > 28, 18.82 ± 6.79), literacy was balanced across the eight items of the eHEALS score, with the lowest score for knowledge of using the internet to answer health questions (2.12 ± 0.97; Supplemental Table 2).

Table 2.

Characteristics of survey respondents.

Variables	Total (n = 6,331)
Sex
Male	3,059 (48.3%)
Female	3,272 (51.7%)
Age-groups
18 to 24	505 (8.0%)
25 to 34	998 (15.8%)
35 to 44	1,001 (15.8%)
45 to 45	1,140 (18.0%)
55+	2,687 (42.4%)
Ethnicity
White	3,827 (60.4%)
Mixed/multiple ethnic groups	150 (2.4%)
Asian	77 (1.2%)
Black/African/Caribbean	27 (0.4%)
Other	51 (0.8%)
Skipped due to GDPR	2,199 (34.7%)
Country
Germany	2,161 (34.1%)
United Kingdom	2,024 (32.0%)
Italy	1,131 (17.9%)
Sweden	1,015 (16.0%)
eHealth literacy
High literacy	834 (13.2%)
Low literacy	5,497 (86.8%)

Number of Topics

To examine the desired number of topics, different LDA models were fitted. Supplemental Figure 1 plots the changes in model comparison metrics with an increasing number of topics for the benefits and challenges of virtual primary care. The Cao et al. (2009) metric indicated LDA models with eight to ten topics to be best for further analysis. Due to their monotonic patterns, the other two metrics provided little guidance in determining the number of topics. Considering all three metrics on balance, an eight-topic LDA model was identified for subsequent analysis.

Benefits of Virtual Technologies in Primary Care

Based on the model comparison metrics, the LDA model identified eight topics related to the benefits of virtual care technologies, including (1) primary care delivery, (2) infection control, (3) reducing contacts, (4) virtual care, (5) timeliness, (6) patient-doctor interaction, (7) convenience and safety, and (8) no answer provided (Figure 2, 1–8). The world clouds (Figure 3, 1–8) serve as an alternative way to illustrate the extracted topics. They contain the ten most frequent terms of each topic in the center of the cloud, as shown in Figure 2, and beyond that, at the periphery of the cloud, other terms that were mentioned less frequently. For the topic “primary care delivery” particularly frequently mentioned terms such as “health,”“care,”“benefit,” or “digital” are in the center of the clouds, highlighted in color and shown in a larger font. Less frequently mentioned terms such as “main,”“provider,” or “protect” are shown toward the edge of the cloud.

Figure 2.

The bar graphs 1 to 8 depict the results of the eight-topic LDA regarding the benefits of virtual primary care; with the topic labels identified by the researchers and the ten most frequently mentioned terms (one-gram and two-gram) for each topic. Beta = probability distribution.

Figure 3.

Word clouds (1–8) corresponding to each topic identified by the Latent Dirichlet Allocation model regarding the benefits of virtual care.

The statistical analysis (Supplemental Figure 2, 1–4) showed little difference between men and women. When comparing age groups, it was found that the 18 to 24-year-old group was significantly more likely to mention the topic “infection” than the 35 to 44 (p = .006), 45 to 54 (p = .005), and 55+ (p < .001) age groups. They mentioned “timeliness” less than most age groups, however significantly less compared to the 45 to 54 age group (p = .02). The cross-country comparison showed that the topic “primary care delivery” was mentioned the most by participants from Sweden (UK p = .007, IT p = .03, DE p < .001) and the least by participants from Germany (UK p = .001, IT p < .001). Participants from Sweden also most frequently mentioned the benefit of infection control (UK p < .001, IT p < .001, DE p < .001). Participants from Italy wrote the least about this topic (UK p < .001, DE p < .001). Participants from the UK were most likely to write about “virtual care” (SE p < .001, IT p < .001, DE p < .001), while participants from Sweden were least likely (IT p < .001, DE p = .04). Participants from Italy wrote the most about “patient-doctor interaction” (UK p < .001, SE p < .001, DE p < .001), while participants from Sweden wrote the least about it (UK p < .001, DE p < .001). Participants from Sweden were also the least likely to mention “timeliness” (UK p < .001, IT p < .001, DE p < .001). When comparing eHealth literacy levels, participants with low eHealth literacy significantly less frequently contributed to the topics “virtual care” (p = .02), “patient-doctor interaction” (p = .03) and “no answer provided” (p = .04).

Challenges of Virtual Primary Care

The eight topics identified by the LDA concerning challenges of virtual care technologies were (1) diagnostic difficulties, (2) physical examination, (3) digital health risks, (4) technical challenges, (5) virtual care, (6) data security and protection, (7) lack of personal contact, and (8) no answer provided (Figure 4, 1–8). As described above, the word clouds (Figure 5, 1–8) provide an alternative visual representation of the topics identified by the LDA in Figure 4 showing additional, less frequently mentioned terms of the clusters at the edges of the clouds.

Figure 4.

The bar graphs 1 to 8 depict the results of the eight-topic LDA regarding the challenges of virtual primary care; with the topic labels identified by the researchers and the ten most frequently mentioned terms (one-gram and two-gram) for each topic. Beta = probability distribution.

Figure 5.

Word clouds (1–8) corresponding to each topic identified by the LDA regarding the challenges of virtual care.

The statistical analysis (Supplemental Figure 3, 1–4) indicated that women were significantly (p = .008) less likely to mention “technical challenges” than men. A comparison of age groups showed that 25 to 34-year-olds had a peak in responses to the topic “physical examination,” which was significantly higher compared to the 45 to 54 (p = .03) and 55+ (p = .01) age groups. The over-55-year-olds were significantly more likely to write about “virtual care” compared to the 25 to 34 (p = .004) and 35 to 44 (p < .001) age groups. The cross-country comparison showed that participants from Sweden mentioned “diagnostic difficulties” (UK p = .009, IT p < .001, DE p < .001), participants from the UK “virtual care” (IT p = .02, SE p = .001, DE p < .001) and participants from Germany “data security and protection” (UK p < .001, IT p = .019, SE p < .001) significantly more often than participants from other countries. When comparing eHealth literacy, participants with low eHealth literacy were significantly less likely to contribute to the topic “virtual care” (p = .047).

Sensitivity Analysis

The sensitivity analysis allowed an assessment of the impact of varying the number of topics of the LDA model on the quality and interpretability of the identified topics. When focusing on the benefits of virtual care technologies, an exploration of nine topics resulted in a new topic “health services” and some minor modification in the probabilities of the top terms for each topic, while maintaining consistency of the topic headlines (Supplemental Figure 4.1). Increasing the number to ten topics introduced two new topics, namely “keeping safe at home” and “avoiding contact in surgery” (Supplemental Figure 4.2).

Regarding the challenges of virtual care, the exploration of an LDA model with nine topics introduced two additional topics, namely “digital access” and “patient-doctor interaction.” The latter topic highlights the challenge of missing symptoms due to the absence of thorough physical examinations. Simultaneously, the topic “digital health risks” disappeared. However, increasing the number of topics to ten, reintroduced the latter topic (Supplemental Figure 5.2).

Interpretation of the connection between the terms became increasingly difficult with the exploration of a higher number of topics as the topic boundaries became less distinct. For example, labeling “avoiding contact in surgery” was difficult, as the terms “people,”“surgery,” and “infect” per se do not suggest a straightforward context. The topic thus also overlaps with the topic “reducing contacts” which raises the question of whether it is a meaningful, coherent, distinct theme (Supplemental Figure 4.2). Similarly, within the challenges, labeling the topic “patient-doctor interaction” was difficult as the terms “patient” and “doctor” do not signify a challenge. Other terms such as “examine,”“assess,” and “physically” can also associated with other topics such as “physical examination” and “diagnostic difficulties.”

Discussion

Key Findings

The analysis of the benefits of virtual care revealed eight prominent topics, namely primary care delivery, infection control, reducing contacts, virtual care, timeliness, patient-doctor interaction, convenience and safety, and no answer provided. The statistical comparison between age groups, sex, countries of residence and eHealth literacy generally showed the most significant differences between countries. Participants from Sweden were most likely to mention primary care delivery, from the UK virtual care and from Italy patient-doctor interaction. The benefit of infection control was mentioned most often by the youngest age group (18–24 years) and by participants from Sweden.

Regarding the challenges of virtual primary care, the identified topics encompassed diagnostic difficulties, physical examination, digital health risks, technical challenges, virtual care, data security and protection, lack of personal contact, and no answer provided. The country comparison again showed the most statistically significant differences within each topic. In Sweden diagnostic difficulties were most significantly mentioned, in the UK it was virtual care, and in Germany, it was data security and protection. Technical challenges were most significantly cited by male participants, and challenges of virtual care were mentioned most by respondents with high eHealth literacy.

Discussion and Comparison With Previous Literature

Systematic reviews have shown that, in general, there is a high acceptance and satisfaction among patients regarding the use of virtual care in primary health care (Bashshur et al., 2016; Carrillo de Albornoz et al., 2022; Versluis et al., 2022). Some also underlined the profound impact of the COVID-19 pandemic on virtual care patients and practices (Bashshur et al., 2016; Carrillo de Albornoz et al., 2022; Zimbroff et al., 2021).

Even before the pandemic, participants of qualitative studies have reported good satisfaction, convenience and timeliness of virtual care (Atherton et al., 2018; Johansson et al., 2017; Mold et al., 2019). Previous findings have also shown an increase in virtual care services with higher experience rates in men, participants with higher literacy and participants from Germany (Neves et al., 2021). Efficiency and timeliness were perceived to be most positively influenced by virtual care technologies (Neves et al., 2021). The results of our study have shown that the challenges of the COVID-19 pandemic translate into a new set of perceived benefits that were not previously expressed. Themes such as reducing physical contact and containing the spread of infections are now central to users’ concerns, potentially highlighting the increased public health and safety risk perception introduced by the pandemic (Schneider et al., 2021).

Among the challenges, prominent themes revolved around the lack of personal contact, physical examination and diagnostic difficulties, underlining that despite the digitalization of healthcare, a personal patient-provider relationship is essential and may be at risk through online communication. Virtual consultations may therefore not be equally suitable for all medical scenarios. Previous findings have shown that patients and healthcare providers consider online consultations unsuitable for initial diagnosis, and rare and unpredictable conditions (Greenhalgh et al., 2018; Wherton et al., 2020). It was found that patient preferences for virtual consultations were dependent on a variety of factors such as age, technology access, their situation of care, their expectations of care, the demands on the patient and the resources to allocate care (Gilbert et al., 2021; Reed et al., 2020).

Participants’ viewpoints on the benefits and challenges of virtual care technologies are shaped by their unique approaches to information processing (Vuong, 2023). How they absorb, process and interpret information is dependent on their existing e-Health knowledge, experiences with technology, cultural backgrounds, and cognitive biases. Furthermore, external factors, such as the implications of the COVID-19 pandemic, have the potential to trigger shifts in the perception of the benefits and challenges. For instance, the low eHealth literacy of 86.8% of participants may have influenced the perceived benefits and challenges of using virtual care. Participants with low eHealth literacy may have difficulty understanding and navigating digital health platforms and using virtual care services and therefore might have experienced more challenges in using virtual care technologies than benefits. There is also the possibility that participants with low eHealth literacy may have provided less detailed, accurate and comprehensive responses, which could have affected the LDA model topics by leading to a narrower or distorted representation of perceptions of virtual care.

Significant differences in the frequency of mentioning the different themes of benefits and challenges occurred between countries. This could be related to several factors, such as differences in the structure of the primary care system, the speed and extent of the national roll-out of virtual care during the COVID-19 pandemic, the respective status of digital maturity before the pandemic, the general policy response to COVID-19, or the survey administration in different languages. For example, in 2019, the UK and Sweden ranked ahead of Italy and Germany in the speed of digitizing their national health systems (Bertelsmann Stiftung, 2019). Having strategies to strengthen virtual care already years before the pandemic, respondents from the UK may have had more experience with virtual care and thus may have mentioned the lack of personal contact and risks to data and protection significantly less than respondents from Germany and Italy (NHS England, n.d.).

It remains to be answered to which extent the cross-country differences were influenced by the individual user preferences or the systemic factors mentioned above. Future research may focus on elaborating on these quantitative findings using qualitative approaches. More international studies that consider digitalization indices, indices on health systems performance and virtual care user profiles are needed to guide future policy decisions and identify best-practice examples.

Strengths and Limitations

This is the first large sample size international study to explore the patient perspective on the benefits and challenges of using virtual care in primary care during the COVID-19 pandemic. Our findings contribute to better identification of the lessons learned and inform rapid improvements in policy and practice of virtual primary care. The results underline the increasing relevance of digital health technologies in primary care and the growing acceptance and demand among patients. However, the study also highlights the risks that arise from their use, such as threats to privacy, inaccurate diagnoses, or the lack of personal contact. Future research may focus on the implications of the different technological modalities, examining changes in the patient-doctor relationship, and considering the long-term impact of virtual care beyond the pandemic.

While various variables on patient characteristics were included, data on implications of individual technologies (e.g., phone, video, web-based) was not collected. As the study did not collect data before the pandemic, it is difficult to examine possible changes in use and perspectives triggered by the pandemic or to make comparisons over time. Further, common limitations of survey designs such as social desirability, self-reported recall, and self-selection apply.

Topic modeling and statistical analysis emulated quantitative analysis to analyze the large data set. Human interaction in pre-processing, choosing the LDA model and topic labeling was necessary for the LDA model because the exclusive use of statistical tools and computation risks losing the semantic structure of the data. While the algorithm provided an overview of representative topics, granular insights on patient perception, as well as contextual and domain-specific analysis were not possible. Translating survey responses from different languages into English using artificial intelligence software may have affected the topic modeling. For example, the translation may have introduced errors or variations in the original language usage and cultural nuances, as well as changes in contextual information that were not captured by the algorithms and may have led to misinterpretation and loss of meaning of the information, resulting in bias or inaccuracy in the derived topics. Following previous criticisms of the use of thematic modeling, there may be cause for skepticism about the degree to which topics match textual statements in the LDA results (Mimno et al., 2011). Reasons for this could be that topics such as “primary care delivery,”“virtual care,” and “patient-doctor interaction” consist of a heterogeneous mix of words. However, it could also be argued that these topics underline the importance of different areas of application of virtual care and the variety of virtual services to patients. Future research should evaluate how the method compares with traditional, non-automated thematic analysis; this could be achieved by analyzing the responses using traditional thematic analysis and comparing the results in what concerns both content and granularity and exploring the potential for synergies using both methods as complementary approaches.

Overall, this study contributes to the growing body of knowledge on virtual care in primary care during the COVID-19 pandemic. It underscores the importance of incorporating the patient perspective in understanding the benefits and challenges of virtual care highlighting areas for improvement in policy and practice. Recommendations for its future practice include enhancing eHealth literacy in patients and healthcare professionals, introducing hybrid models that combine virtual and in-person care to balance out personal interaction and remote convenience, improving the user-centered design of virtual care technologies and defining clear medical needs for their application. Maintaining trust in virtual technologies is essential, thus data security and privacy are vital. Differences among age groups, gender, and countries of residence were observed, indicating that preferences and priorities regarding virtual care may vary based on these factors. Our findings underline the increasing relevance of topic modeling in identifying key issues as part of an automated exploratory approach. By learning from this major real-life experiment of digital care, we can better shape the future of virtual care delivery in a post-pandemic world.

Supplemental Material

sj-docx-1-sgo-10.1177_21582440241263147 – Supplemental material for Assessing Public Perceptions of Virtual Primary Care During the COVID-19 Pandemic in the UK, Germany, Sweden, and Italy: A Topic Modeling Approach

Supplemental material, sj-docx-1-sgo-10.1177_21582440241263147 for Assessing Public Perceptions of Virtual Primary Care During the COVID-19 Pandemic in the UK, Germany, Sweden, and Italy: A Topic Modeling Approach by Felix Machleid, Roberto Fernandez Crespo, Kelsey Flott, Saira Ghafur, Ara Darzi, Erik Mayer and Ana Luisa Neves in SAGE Open

Footnotes

Acknowledgements

We acknowledge infrastructure support from the National Institute for Health Research (NIHR) North West London Patient Safety Research Collaboration and from the NIHR Imperial Biomedical Research Centre. The work was enabled by the Imperial Clinical Analytics, Research and Evaluation (iCARE) Secure Data Environment, staff and data resources. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care.

Author Contribution

All authors contributed to the conceptualization of the study and the protocol development. ALN, SG, KF and EM contributed to data collection. FM, RFC and ALN contributed to data analysis. FM and ALN contributed to interpreting the findings and writing the manuscript. RFC, KF, SG, AD and EM reviewed and provided feedback on the manuscript. All authors had full access to all the raw data in the study. All authors approved the final version of the manuscript.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: AD is Chair of the Health Security initiative at Flagship Pioneering UK Ltd. All other authors have no competing interests.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded by the Imperial COVID-19 Response Fund, with support from the National Institute for Health Research (NIHR) North West London Patient Safety Research Collaboration, and the NIHR Imperial Biomedical Research Centre. ALN is also funded by the NIHR Applied Research Collaboration Northwest London (NIHR ARC NWL).

ORCID iDs

Felix Machleid

Erik Mayer

Ana Luisa Neves

Data Availability Statement

The anonymized data that support the findings of this study are available from the corresponding author, upon reasonable request.

Supplemental Material

Supplemental material for this article is available online.

References

Arnold

Speier

(2012). A topic model of clinical reports [Conference session] Proceedings of the 35th international ACM SIGIR conference on Research and Development in Information Retrieval (pp. 1031–1032). Association for Computing Machinery.

Arun

Suresh

Venni Madhavan

C. E.

Narasimha Murthy

M. N.

(2010). On finding the natural number of topics with latent dirichlet allocation: Some observations. In: Zaki

M. J.

J. X.

Ravindran

Vikram

(Eds.), Advances in knowledge discovery and data mining. PAKDD 2010. Lecture Notes in Computer Science (Vol. 6118), (pp. 391–402). Springer.

Atherton

Brant

Ziebland

Bikker

Campbell

Gibson

McKinstry

Porqueddu

Salisbury

(2018). Alternatives to the face-to-face consultation in general practice: Focused ethnographic case study. British Journal of General Practice, 68(669), e293–e300.

Bashshur

R. L.

Howell

J. D.

Krupinski

E. A.

Harms

K. M.

Bashshur

Doarn

C. R.

(2016). The empirical foundations of telemedicine interventions in primary care. Telemedicine and e-Health, 22(5), 342–375.

Bertelsmann Stiftung. (2019). Summary #SmartHealthSystems – Focus Europe. Retrieved November 10, 2021, from https://www.bertelsmann-stiftung.de/en/publications/publication/did/summary-smarthealthsystems-focus-europe

Björndell

Premberg

. (2021). Physicians’ experiences of video consultation with patients at a public virtual primary care clinic: A qualitative interview study. Scandinavian Journal of Primary Health Care, 39(1), 67–76.

Blei

D. M.

A. Y.

Jordan

M. I.

(2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.

Cao

Xia

Zhang

Tang

(2009). A density-based method for adaptive LDA model selection. Neurocomputing, 72(7–9), 1775–1781.

Carrillo de Albornoz

Sia

K.-L.

Harris

(2022). The effectiveness of teleconsultations in primary care: Systematic review. Family Practice, 39(1), 168–182.

10.

Chang

Gerrish

Wang

Boyd-Graber

Blei

(2009). Reading tea leaves: How humans interpret topic models. Advances in Neural Information Processing Systems, 22, 1–9.

11.

Chung

Park

B. K.

Nahm

E.-S.

(2018). The Korean eHealth Literacy Scale (K-eHEALS): Reliability and validity testing in younger ddults recruited online. Journal of Medical Internet Research, 20(4), e138.

12.

Clarke

(2023). Telemedicine in Italy takes next step following pandemic regulatory advanc. https://www.osborneclarke.com/insights/telemedicine-italy-takes-next-step-following-pandemic-regulatory-advance

13.

DeepL SE. (2017). DeepL. Retrieved July 14, 2023, from https://www.deepl.com/translator

14.

Deveaud

SanJuan

Bellot

(2014). Accurate and effective latent concept modeling for ad hoc information retrieval. Document Numérique, 17(1), 61–84.

15.

Durmuşoğlu

Z. D. U.

(2022). Remembering medical ventilators and masks in the days of COVID-19: Patenting in the last decade in respiratory technologies. IEEE Transactions on Engineering Management, 71, 1339–1373.

16.

Ekman

Thulesius

Wilkens

Lindgren

Cronberg

Arvidsson

(2019). Utilization of digital primary care in Sweden: Descriptive analysis of claims data on demographics, socioeconomics, and diagnoses. International Journal of Medical Informatics, 127, 134–140.

17.

European Commission. (n.d.). eHealth: Digital health and care. Retrieved August 12, 2023, from https://health.ec.europa.eu/ehealth-digital-health-and-care_en

18.

European Patients’ Forum. (2020). Digitalisation of healthcare: The new normal. Retrieved June 30, 2021, from https://www.cocir.org/fileadmin/Events_2020/Digitalisation_16_Nov/4._COCIR_event_16_Nov._2020_-_Michele_Calabro.pdf

19.

Feinerer

Hornik

& Artifex Software Inc. (2023). tm: Text Mining Package (version 0.7-11). Retrieved July 15, 2023, from https://cran.r-project.org/web/packages/tm/index.html

20.

Frank

S. R.

(2000). Digital health care—the convergence of health care and the Internet. The Journal of Ambulatory Care Management, 23(2), 8–17.

21.

Gabrielsson-Järhult

Kjellström

Josefsson

K. A.

(2021). Telemedicine consultations with physicians in Swedish primary care: A mixed methods study of users’ experiences and care patterns. Scandinavian Journal of Primary Health Care, 39(2), 204–213.

22.

Gerke

Stern

A. D.

Minssen

(2020). Germany’s digital health reforms in the COVID-19 era: Lessons and opportunities for other countries. npj Digital Medicine, 3(1), 94.

23.

German Federal Ministry of Health. (n.d.). Digital Healthcare Act (Digitale-Versorgung-Gesetz). Retrieved July 15, 2023, from https://www.bundesgesundheitsministerium.de/en/digital-healthcare-act.html

24.

Gilbert

A. W.

Jones

Stokes

May

C. R.

(2021). Factors that influence patient preferences for virtual consultations in an orthopaedic rehabilitation setting: A qualitative study. BMJ, 11(2), e041038.

25.

Greenhalgh

Shaw

Wherton

Vijayaraghavan

Morris

Bhattacharya

Hanson

Campbell-Richards

Ramoutar

Collard

Hodkinson

(2018). Real-world implementation of video outpatient consultations at macro, meso, and micro levels: Mixed-method study. Journal of Medical Internet Research, 20(4): e150.

26.

Griffiths

Steyvers

(2004). Finding scientific topics. Proceedings of the National Academy of Sciences of the United States of America, 101, 5228–5235.

27.

Grün

Horni

Blei

Lafferty

J. D.

Phan

X.-H.

Matsumoto

Nishimura

Cokus

(2023). topicmodels: Topic Models (Version 0.2-12). Retrieved July 15, 2023, from https://cran.r-project.org/web/packages/topicmodels/topicmodels.pdf

28.

Hofmann

(1999). Probabilistic latent semantic analysis[Conference session]. In: Kaskey

K. B.

Prade

(Eds.), Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence (pp. 289–296). Morgan Kaufmann Publishers Inc.

29.

Holtz

B. E.

(2021). Patients perceptions of telemedicine visits before and after the coronavirus disease 2019 pandemic. Telemedicine and e-Health, 27(1), 107–112.

30.

Imlach

McKinlay

Middleton

Kennedy

Pledger

Russell

Churchward

Cumming

McBride-Henry

(2020). Telehealth consultations in general practice during a pandemic lockdown: Survey and interviews on patient experiences and preferences. BMC Family Practice, 21(1), 1–14.

31.

Johansson

A. M.

Lindberg

Söderberg

(2017). Healthcare personnel’s experiences using video consultation in primary healthcare in rural areas. Primary Health Care Research & Development, 18(1), 73–83.

32.

Jones

Doane

Attbom

(2021). textmineR: Functions for Text Mining and Topic Modeling (Version 3.0.5). Retrieved July 15, 2023, from https://cran.r-project.org/web/packages/textmineR/index.html

33.

Lee

D. D.

Seung

H. S.

(1999). Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755), 788–791.

34.

Mimno

Wallach

Talley

Leenders

McCallum

(2011). Optimizing semantic coherence in topic models [Conference session]. In: Barzilay

Johnson

(Eds.), Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (pp. 262–272). Association for Computing Machinery.

35.

Mold

Hendy

Lai

Y. L.

de Lusignan

(2019). Electronic consultation in primary care between providers and patients: Systematic review. JMIR Medical Informatics, 7(4), e13042.

36.

Morgenstern-Kaplan

Rocha-Haro

Canales-Albarrán

S. J.

Núñez-García

León-Mayorga

(2021). An app-based telemedicine program for primary care and specialist video consultations during the COVID-19 pandemic in Mexico. Telemedicine and e-Health. Advance online publication. https://doi.org/10.1089/tmj.2021.0055

37.

Neves

A. L.

van Dael

O’Brien

Flott

Ghafur

Darzi

Mayer

(2024). Use and impact of virtual primary care on quality and safety: The public’s perspectives during the COVID-19 pandemic. Journal of Telemedicine and Telecare, 30(2), 393–401.

38.

NHS Digital. (n.d.). NHS App. Retrieved July 15, 2023, from https://www.nhs.uk/nhs-app/

39.

NHS England. (2019). The NHS Long Term Plan. Retrieved May 6, 2021, from https://www.longtermplan.nhs.uk/publication/nhs-long-term-plan/

40.

NHS England. (n.d.). Digital first primary care. Retrieved May 28, 2021, from https://www.england.nhs.uk/gp/digital-first-primary-care/

41.

Nikita

Chaney

(2020). ldatuning: Tuning of the Latent Dirichlet Allocation Models Parameters (Version 1.0.2). Retrieved July 15, 2023, from: https://cran.r-project.org/web/packages/ldatuning/index.html

42.

Norman

C. D.

Skinner

H. A.

(2006). EHEALS: The eHealth literacy scale. Journal of Medical Internet Research, 8(4), 1–7.

43.

Omboni

(2020). Telemedicine during the COVID-19 in Italy: A missed opportunity? Telemedicine and e-Health, 26(8), 973–975.

44.

Paige

S. R.

Krieger

J. L.

Stellefson

Alber

J. M.

(2017). EHealth literacy in chronic disease patients: An item response theory analysis of the eHealth literacy scale (eHEALS). Patient Education and Counseling, 100(2), 320–326.

45.

Park

Choi

Kim

Cha

Kim

Moon

I. C.

(2017). Identifying prescription patterns with a topic model of diseases and medications. Journal of Biomedical Informatics, 75, 35–47.

46.

R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing.

47.

Ramaswamy

Drangsholt

Culligan

P. J.

Schlegel

P. N.

J. C.

(2020). Patient satisfaction with telemedicine during the COVID-19 pandemic: Retrospective cohort study. Journal of Medical Internet Research, 22(9), 1–9.

48.

Reed

M. E.

Huang

Graetz

Lee

Muelly

Kennedy

Kim

(2020). Patient characteristics associated with choosing a telemedicine visit vs office visit with the same primary care clinicians. JAMA Network Open, 3(6), e205873.

49.

Richardson

Aissat

Williams

G. A.

Fahy

(2020). Keeping what works: Remote consultations during the COVID-19 pandemic. Eurohealth, 26(2), 73–76.

50.

Rinker

(2018). Textstem: Tools for Stemming and Lemmatizing Text (Version 0.1.4). Retrieved August 11, 2023, from https://cran.r-project.org/web/packages/textstem/index.html

51.

Schneider

C. R.

Dryhurst

Kerr

Freeman

A. L.

Recchia

Spiegelhalter

van der Linden

(2021). COVID-19 risk perception: A longitudinal analysis of its predictors and associations with health protective behaviours in the United Kingdom. Journal of Risk Research, 24(3–4), 294–313.

52.

Van der Vaart

Van Deursen

A. J.

Drossaert

C. H.

Taal

van Dijk

J. A.

van de Laar

M. A.

(2011). Does the eHealth Literacy Scale (eHEALS) measure what it intends to measure? Validation of a Dutch version of the eHEALS in two adult populations. Journal of Medical Internet Research, 13(4), e86.

53.

Versluis

Schnoor

Chavannes

N. H.

Talboom-Kamp

E. P.

(2022). Direct Access for patients to diagnostic testing and results using eHealth: Systematic review on eHealth and diagnostics. Journal of Medical Internet Research, 24(1), e29303.

54.

von Elm

Altman

D. G.

Egger

Pocock

S. J.

Gøtzsche

P. C.

Vandenbroucke

J. P.

; STROBE Initiative. (2007). Strengthening the reporting of observational studies in epidemiology (STROBE) statement: Guidelines for reporting observational studies. BMJ, 335(7624), 806–808.

55.

Vosburg

R. W.

Robinson

K. A.

(2021). Telemedicine in primary care during the COVID-19 pandemic: Provider and patient satisfaction examined. Telemedicine and e-Health, 28(2), 167–175.

56.

Vuong

Q. H.

T. T.

V. P.

Nguyen

M. H.

(2022). The psychological mechanism of internet information processing for post-treatment evaluation. Heliyon, 8(5), e09351.

57.

Vuong

Q.-H.

(2023). Mindsponge theory. Walter de Gruyter GmbH.

58.

Webster

(2020). Virtual health care in the era of COVID-19. Lancet (London, England), 395(10231), 1180–1181.

59.

Wherton

Shaw

Papoutsi

Seuren

Greenhalgh

(2020). Guidance on the introduction and use of video consultations during COVID-19: Important lessons from qualitative research. BMJ Leader, 4(3), 120–123.

60.

Wickham

(2023). stringr: Simple, Consistent Wrappers for Common String Operations (Version 1.5.0). Retrieved July 14, 2023, from https://cran.r-project.org/web/packages/stringr/index.html

61.

Wickham

Francxois

Henry

, et al. (2023). dplyr: A Grammar of Data Manipulation (Version 1.1.2). Retrieved July 15, 2023, from: https://cran.r-project.org/web/packages/dplyr/index.html

62.

Wickham

Vaughan

Girlich

, et al. (2023). tidyr: Tidy Messy Data (Version 1.3.0). Retrieved July 14, 2023, from https://cran.r-project.org/web/packages/tidyr/index.html

63.

Wosik

Fudim

Cameron

Gellad

Z. F.

Cho

Phinney

Curtis

Roman

Poon

E. G.

Ferranti

Katz

J. N.

(2020). Telehealth transformation: COVID-19 and the rise of virtual care. Journal of the American Medical Informatics Association, 27(6), 957–962.

64.

Zimbroff

R. M.

Ornstein

K. A.

Sheehan

O. C.

(2021). Home-based primary care: A systematic review of the literature, 2010–2020. Journal of the American Geriatrics Society, 69(10), 2963–2972.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

3.76 MB