“I am chatbot,your virtual mental health adviser.” What drives citizens’ satisfaction and continuance intention toward mental health chatbots during the COVID-19 pandemic? An empirical study in China

Abstract

Introduction

In order to address the psychological problems during the COVID-19 pandemic, mental health chatbots have been extensively used by public sectors. According to Theory of Consumption Values, this paper proposed an analytical framework to investigate the determinants behind users’ satisfaction and continuance intention toward mental health chatbots.

Methods

The empirical study was conducted through an online survey, facilitated by the use of questionnaire posted on the WeChat platform. Seven-point Likert scale and closed-ended questions were employed. Totally 371 valid samples were collected. The research data was tested via the partial least squares structural equation modeling. Gender, age, and income were included as control variables.

Results

Analysis of samples collected from 371 Chinese users suggested that personalization (functional value), enjoyment (emotional value), learning (epistemic value), and the condition of the COVID-19 pandemic (conditional value) have positive influences on user satisfaction and continuance intention, but such effects were weak. The findings also revealed that user satisfaction has weakly positive impact on continuance intention. However, voice interaction (functional value) was an insignificant predictor of user satisfaction and continuance intention.

Discussion

This study developed a critical perspective on the role of Theory of Consumption Values in the context of mental health chatbot usage, while Theory of Consumption Value has been increasingly employed to explain the use of AI-based public services. Thus, this research devotes to the enhancement of theoretical frameworks regarding the usage of mental health chatbots.

Keywords

Mental health chatbots COVID-19 pandemic user satisfaction continuance intention Theory of Consumption Values

Introduction

The ongoing coronavirus pandemic 2019 (COVID-19) has caused millions of deaths and infected hundreds of millions of people worldwide, in turn leading to a plethora of mental disorders.¹ Government agencies in many countries have used mental health chatbots to provide virtual therapies and services. Although the use of mental health chatbots is widespread, our knowledge about what influences citizens’ continuance intention to use chatbots is limited. In addition, our knowledge about the determinants of citizens’ user satisfaction with mental health chatbots is limited. Such knowledge gaps significantly prevent the implementation of mental health chatbots during the COVID-19 pandemic.

The main purposes of this research are thus twofold: (1) employing Theory of Consumption Values (TCVs) as a theoretical framework and testing whether this theory can reveal the determinants behind user satisfaction and continuance intention during the COVID-19 pandemic; and (2) examining precisely the relationship between user satisfaction and continuance intention during the COVID-19 pandemic. These objectives are noteworthy for several reasons. Theoretically speaking, with the wide implementation of AI-based chatbots, it is necessary to extend the theoretical frameworks to explain user satisfaction and continuance intention.² However, prior research on the adoption of AI-based chatbots has mainly relied on Technology Acceptance Model (TAM).³ Currently, the explanatory power of TCV has been widely demonstrated in the field of business, but this theory has rarely been employed to explain user behaviors of AI-based chatbots, especially mental health chatbots. Thus, this research may offer a critical insight into TCV by demonstrating whether it can be used to explain user satisfaction and continuance intention of mental health chatbots during the pandemic. Moreover, although many studies have revealed a positive relationship between user satisfaction and continuance intention when people use information systems (ISs),^4,5 there is a lack of theoretically driven hypothesis-based research on this relationship in the public health emergency context. Therefore, this research might extend the existing literature on the relationship between user satisfaction and continuance intention in the IS area. Practically speaking, although chatbots are extensively used in the digital era, people's continuance intention to use chatbots remains low.⁶ Expectation-Confirmation Theory (ECT), IS Continuance Theory (ISCT), and Technology Continuance Theory (TCT) have argued that people will be reluctant to continue using ISs if they feel dissatisfied with it.^7,8 Thus, a deeper understanding of the antecedents of chatbot satisfaction and continuance intention can help government agencies and chatbot engineers develop chatbot systems and ultimately improve the adoption rate of mental health chatbots.

Based on TCV, this research developed a research model to examine the determinants behind citizens’ user satisfaction and continuance intention. Furthermore, the relationship between user satisfaction and continuance intention to use chatbots was tested in this study.

Theoretical background and hypotheses development

Mental health chatbots

Chatbots, also known as virtual agents or intelligent agents, are generally defined as computer systems that use AI-based technologies to mimic humanlike behaviors and offer a task-oriented framework with evolving dialog, making it capable of participating in conversations.^9–11 A conversation between humans and chatbots is triggered by user input. The contextual awareness technologies allow chatbots to provide feedback based on receiving users’ messages.¹²

Mental health is considered as one of the areas that will benefit greatly from chatbot services.^11,13 Currently, mental health chatbots are increasingly being used for therapeutic and training objectives, such as suicide prevention¹⁴ and cognitive-behavioral therapy (Ail, 2018). In some cases, virtual agents, such as HARR-E and Wysa, are also tailored to certain groups.¹¹

Compared with manned hotlines, mental health chatbots are more accessible and flexible.¹⁵ It is believed that these advantages can offer great support to the existing mental healthcare system during the pandemic. For instance, mental health chatbots can simultaneously provide therapies and services for many more people than a manned hotline,¹⁶ in turn improving the effectiveness of service delivery. In addition, mental health chatbots are devoted to addressing the shortage of human resources, especially mental health advisers, during the COVID-19 pandemic.¹ Given that the mental healthcare systems could benefit greatly from mental health chatbots, government agencies in many countries such as Singapore, America, and China have deployed virtual agents to deliver 24-h psychological assistance during the COVID-19 pandemic.^16,17

Many previous studies have focused on the effectiveness and acceptability of mental health chatbots. For example, Fitzpatrick et al.¹⁸ proved that a chatbot called “Woebot” could effectively help college students with depression. Cameron et al.¹⁹ identified some limitations of a mental health agent named “iHepr” and gave suggestions to improve its functions. However, few studies have investigated the determinant factors behind users’ continuance intention to use mental health chatbots during the COVID-19 pandemic.

During the COVID-19-related lockdown, the government agencies in China then collaborated with their AI service contractors to launch a number of mental health chatbots to offer psychological assistance. For instance, a mental health chatbot named Psybot, which can listen, analyze, and talk, was employed by the local government in Hangzhou to help people express their true feelings and ease their moods. On encountering an emergency case, it sends out early warnings and asks professional human consultants to step in.²⁰ Another chatbot called Xiaoi, as deployed by the local government in Hubei Province. With its emotional computing and empathy module, Xiaoi could more clearly analyze citizens’ problems and provide a warm-hearted interaction.²¹ This study selected a mental health chatbot (Xiaolv) as the research target.

User satisfaction and continuance intention

User satisfaction is a central concept in behavioral research.^4,22 Expectation Disconfirmation Theory is an influential theory that explains the formation of user satisfaction by comparing expectations and post-experience belief in services.²³ According to this theory, individuals often set expectations before experiencing a service.²⁴ After experiencing this service, a post-experience belief is generated.²⁵ When comparing expectations and post-experience beliefs, individuals can feel a sense of disconfirmation.²⁶ If post-experience belief is better than expectation, the sense of disconfirmation will be positive, and individuals will feel satisfied.²⁷ In contrast, if post-experience belief is worse than expected, the sense of disconfirmation will be negative, and individuals will be dissatisfied.²⁸ Thus, user satisfaction typically refers to the fulfillment response that results from comparing a service's perceived expectation of performance.²² In the field of ISs, user satisfaction has been widely employed by researchers and practitioners to predict people's continuance intention.⁸

The notion of continuance intention is different from the concept of acceptance intention: the former emphasizes user loyalty, while the latter highlights initial acceptance.⁷ According to ECT, ISCT, and TCT, loyalty and long-term use lead to the success of IS services.^8,29,30 In this sense, continuance intention should be regarded as a key factor in IS success.³¹ Thus, when IS technologies were being used, researchers and practitioners tried to investigate why people continued to adopt them. This was the case for e-learning,³² online banking,³³ online shopping,³⁴ and online games.³¹

User satisfaction is often regarded as an important predictor that stimulates continuance intention to use IS product or service.²⁹ Influential theories such as ECT, ISCT, and TCT ascertain that continuance intention is determined primarily by user satisfaction, which means that people who have higher levels of satisfaction with ISs are more likely to stay loyal longer.^7,35 Generally, if individuals feel satisfied after using an IS product or service, they will believe that this product or service can also fulfill their expectations next time as well, and thus have greater motivation to continuously use it.³⁶

Theory of Consumption Values

TCV is a theory that reveals how people judge and make decisions to adopt a product or service.³⁷ TCV has been widely used to explain people's decision-making behavior in economics, business, and sociology.³⁸ According to TCV, products or services potentially contain multiple consumption values, including functional value, emotional value, epistemic value, conditional value, and social value.³⁹ Consumers attach different values to different products or services, which in return affects their intention to use.³⁸

Specifically, functional value is defined as the value acquired through the possession of utilitarian or physical attributes.³⁹ It is closely linked to practical factors such as reliability, effectiveness, and price.³⁸ In most cases, functional value is an important driver of consumers’ decisions.^37,40,41 Emotional value is the utility derived from an alternative capacity to evoke affective feelings.⁴² According to Sheth et al. (1991), products or services are frequently related to emotional responses, such as esthetics and feelings. These emotions and affections often influence customers’ motivation to use a product or service. Social value is defined as the utility derived from an alternative's relationship with specific groups (Yao et al., 2016). A product or service may represent a specific social group, customers can enhance a sense of belonging to that group through the consumption of such product or service.⁴³ Thus, people's decisions to use a product can be influenced by social value. Epistemic value is the utility of a product or service to arouse curiosity, or satisfy a desire for knowledge.³⁹ A product or service may be selected because people feel curious.⁴² Conditional value is the utility acquired from a specific situation involved in customer choice.⁴⁰ When people are involved in a specific event or situation, they might value the use of a product or service because it can help them solve problems. As a result, people tend to use the product or service in a specific event or situation.^39,44 For instance, people usually choose to use umbrella when it is raining. Conditional value arises suddenly, and consumers benefit from it temporarily.⁴⁵ Compared with other value components in TCV, conditional value more highlights the influence of condition on people's decisions.

The original TCV is used only to explain customers’ choice motivations. Recently, this theory has presented a wider view wherein consumption values also influence user satisfaction.²⁶ Consumption values arise before or during the purchase process, whereas user satisfaction is created at the post-purchase stage.³⁸ In this sense, consumption values may be the determinants of user satisfaction.

According to TCV, the relative consumption values vary with changing context.⁴¹ Thus, the value components in this research were contextualized based on the characteristics of mental health chatbots. According to Zhu et al.,²⁶ personalization and voice interaction are important functions of AI-based machines. As for mental health chatbots, personalization can help offer more appropriate therapies and thus improve the effectiveness of human–chatbot interaction. Voice interaction can give chatbots more human-like factors and make the human–chatbot interaction more natural. Thus, the functional values of mental health chatbots in this study are contextualized as personalization and voice interaction. As many previous studies have argued, enjoyment is a significant element of human–chatbot interaction.¹⁰ The effective interaction between users and mental health chatbots induces a feeling of enjoyment, which in turn helps address psychological problems such as anxiety and stress.⁴⁶ Therefore, emotional value is contextualized as enjoyment. In addition, learning is another important element of human–computer interaction.^40,41 Curiosity is one of our basic attributes and we are nearly oblivious of its pervasiveness in our lives.⁴⁷ Curiosity is also a component of subjective well-being.⁴⁸ As a significant notion in positive psychology, subjective well-being emphasizes that only individuals can assess their levels of well-being in any condition, including those who live with mental health conditions.⁴⁹ Thus, people can feel curious even if they encounter mental disorders. As a new AI-based technology, mental health chatbots can easily arouse curiosity and stimulate the desire to learn. Once individuals are motived to learn, they are more likely to continuously interact with chatbots to acquire knowledge. Meanwhile, through the learning process—that is, human–chatbot interaction—individuals can satisfy their desire for knowledge about mental health chatbots. As a result, user satisfaction will improve. Thereby, we employ learning to represent the epistemic value. People found themselves suddenly involved in a special condition (e.g. the national lockdown, overtaxed manned hotlines, and serious infection) due to the outbreak of the COVID-19 pandemic. Such a condition endows mental health chatbots with great utility, making them a valuable and available approach to offer psychological assistance.²⁶ In other words, the value of mental health chatbots can be better expressed and perceived by people during the COVID-19 pandemic. Thus, we contextualized the conditional value as the condition of the COVID-19 pandemic. However, social value is excluded in this study because a person is free of social bonds when interacting with a mental health chatbot.

Hypotheses development

Based on TCV, the functional values of mental health chatbots in this research are contextualized as personalization and voice interaction. Personalization refers to the ability of the system to provide personalized services that match users’ unique characteristics and needs.^34,50 Currently, the achievements of machine learning and data mining technology allow chatbots to deeply analyze users’ historical data. Based on historical data, chatbots can offer services that may meet users’ demands better.⁵¹ According to TCV, functional values are essential in forming user satisfaction²⁶ and continuance intention.⁴¹ This point has been supported by many empirical studies on AI-based applications.^52,53 In this study, personalization as a functional value can identify specific psychological problems accurately and offering personalized therapies or services. Thus, personalization may greatly help those who are with psychological issues during the COVID-19 pandemic. Based on the argument above, we formulated the following hypotheses: H1a:

Personalization is positively related to citizens’ user satisfaction with mental health chatbots.

H1b:

Personalization is positively related to citizens’ continuance intention to use mental health chatbots.

Voice interaction is another function value of AI-based chatbots. It refers to the communication between human and chatbots through voice.²⁶ Because voice is commonly used by a person to interact with others in daily life, when chatbots use voice to communicate with people, people tend to feel natural and friendly.⁵⁴ As a result, the overall user experience of chatbots can be improved.⁵⁵ In this sense, the important role of voice interaction is to bring a feeling of natural and friendly. As some previous studies have proved, voice interaction may be a key variable that affects people's user satisfaction⁵⁶ and continuance intention⁴⁰ when they interact with AI-driven applications. During the COVID-19 pandemic, voice interaction may help the interaction between users and chatbots become more natural, which in return improves the overall interaction quality and might ultimately enhance user satisfaction and continuance intention. Therefore, we proposed the following hypotheses: H2a:

Voice interaction is positively related to citizens’ user satisfaction with mental health chatbots.

H2b:

Voice interaction is positively related to citizens’ continuance intention to use mental health chatbots.

In this research, we expected that the emotional value of mental health chatbots (enjoyment) shows positive influences on user satisfaction as well as continuance intention. According to prior studies, a feeling of enjoyment is important to user satisfaction with AI-based chatbots.^2,57 As to mental health chatbots, when users interact with them, a feeling of enjoyment can contribute to addressing psychological issues such as anxiety and depression, which in return improves the overall user satisfaction.⁵⁸ Thus, we hypothesize that: H3a:

Enjoyment is positively related to citizens’ user satisfaction with mental health chatbots.

For many years, enjoyment was believed to have influence on the acceptance behavior of ISs.⁵⁹ A feeling of enjoyment usually generates positive feedback and thus encourage user adoption of ISs.^60,61 Prior studies based on TCV have proved that enjoyment as emotional value can stimulate people's motivation to adopt ISs.^41,62 Recently, Kasilingam⁵⁷ has found that citizens’ motivation to adopt chatbots for online shopping can be greatly affected by a feeling of enjoyment. When people interact with a mental health chatbot, they may feel enjoyable because this chatbot provides high-quality services and therapies. Such a feeling of enjoyment can help to reduce people's stress, anxiety, and depression,⁴⁶ which in return may encourage their continuance intention to use the chatbot. Thus, we hypothesize that: H3b:

Enjoyment is positively related to citizens’ continuance intention to use mental health chatbots.

As a basic element of our cognition, curiosity motivates learning and decision-making.⁴⁷ Users choose to adopt a product or service may be because they feel curious and intend to learn.³⁸ According to TCV, learning is an important value component that can satisfy users’ curiosity.⁴⁵ Commonly, if a product or service successfully arouses people's curiosity, people will take action to learn more about it. The learning process is usually completed through the continuous use of this product or service.⁶³ Thus, curiosity motivates the desire to learn, and learning drives the continuance intention to use. Moreover, if people perceive that their knowledge is enriched through the use of a product or service, they tend to feel more satisfied.⁶³ The research by Kim et al.⁶⁴ proved that those who improve their degree of knowledge through the use of mobile shopping applications will have a higher level of user satisfaction. In addition, Lee et al.⁴⁰ found that learning can positively affect citizens’ intention to reuse Web-based taxi services. Thus, we formulated the following hypotheses: H4a:

Learning is positively related to citizens’ user satisfaction with mental health chatbots.

H4b:

Learning is positively related to citizens’ continuance intention to use mental health chatbots.

According to TCV, the condition in which a product is used drives the conditional value.³⁸ Although a condition or event may be temporary, it can exert a great impact on user satisfaction and continuance intention.⁶⁵ The national lockdown, overtaxed manned hotlines, and serious infections triggered the special conditions of the COVID-19 pandemic. In such a condition, people tended to highly value the use of mental health chatbots because these virtual agents were available and helpful. As a result, citizens might be more likely to feel satisfied with mental health chatbots and show greater continuance intention. The research by Zhu et al.²⁶ noted that the condition of the COVID-19 pandemic is positively associated with user satisfaction when using mental health chatbots. Based on the arguments above, we hypothesize that: H5a:

The condition of the COVID-19 pandemic is positively related to citizens’ user satisfaction with mental health chatbots.

H5b:

The condition of the COVID-19 pandemic is positively related to citizens’ continuance intention to use mental health chatbots.

User satisfaction is considered as a key factor to form and retain continuance intention by many theories. The early theories such as ECT and ISCT hold that user satisfaction is a basic and pivotal predictor of continuance intention.^7,25,66 The recent theory such as TCT also ascertains that user satisfaction plays a great role in building user loyalty.²⁹ Based on these theories, existing literature has proved that there is a positive link between user satisfaction and continuance intention. To list a few, Kim et al.³⁶ argued that continuance intention is greatly influenced by user satisfaction when people use e-commerce systems. Nascimento et al.⁶⁷ proved that satisfaction has a great impact on continuance intention to adopt smartwatch. In line with these studies, we hypothesize that: H6:

User satisfaction is positively related to citizens’ continuance intention to use mental health chatbots.

All the hypotheses and the research model are shown in Figure 1.

Figure 1.

The research model in this study.

Method

Data collection

This is a cross-sectional study conducted using data collected from Wuhan, Chongqing, and Hangzhou, during the COVID-19 pandemic in China. A complete national lockdown was implemented in China from January to March 2020.²⁶ This serious lockdown made millions of people experience a sudden change. It also caused a shortage of living supplies and financial distress.⁶⁸ In such a condition, many people felt anxious, depress, and sleepless.

In order to help those who are with mental health problems, local governments launched an AI-based mental health chatbot called Xiaolv. People with issues of stress, anxiety, depression, and sleep could interact with Xiaolv through Wechat, a popular mobile application in China.²⁶ Figure 2 shows the typical mental health care supervised by Xiaolv. At the beginning, Xiaolv welcomes users via a self-introduction. Then, users can freely “talk” to Xiaolv about their worries, emotions, and feelings through both spoken and written languages. After analyzing users’ inputs, Xiaolv will organize languages to encourage and comfort users. For instance, when a user with depression input “Due to the pandemic, I have to stay at home. I am not in mod,” Xiaolv comforted this person by saying “Do not be nervous my dear, no matter what happened, I will always be with you.” Such a conversation could offer a feeling of warm to users, especially those who are in home isolation, and make them believe they find a good listener. Also, Xiaolv could provide other mental health supports to help users relax, such as game recommendations.

Figure 2.

A case of mental health care supervised by Xiaolv. It will analyze users’ problems via a two-way communication and offer personalized services accordingly.

From February to April 2020, many Chinese undergraduate students worked as volunteers to fight on the front line of urban and rural communities for epidemic prevention. Community health service centers randomly assigned several families to each student volunteer. The students then provided health services for the designated families. We contacted 49 volunteer students in Wuhan, Chongqing, and Hangzhou, they helped us to send the Xiaolv chatbot's link through WeChat groups to 166 families in 43 communities. After a week, the online questionnaire was uploaded on Wechat groups and potential respondents were requested to complete it.

Before the formal survey, several experts in the area of ISs were invited to give suggestions. In addition, we also conducted a pilot test with 23 volunteers. According to their revision advice, we modified the questionnaire, making it contain three parts. In the first part, we used three questions to identify those who had adopt Xiaolv: “could you please tell us the name of the chatbot?,” “have you interacted with the chatbot?” and “can this chatbot speak to you?.” Then, we collected demographic information of the respondents. Third, we measured consumption values, user satisfaction, and continuance intention through a set of items. As a response, we totally collected 445 samples. Then, we deleted those responses with missing data and those samples who indicated that they never interact with Xiaolv. Finally, 371 responses were considered as valid. The demographic information, user frequency, and duration are shown in Table 1.

Table 1.

The information of the research samples.

Category		Number	Percentage
Gender	Male	217	58.5%
	Female	154	41.5%
Age	18–30	63	17.0%
	31–40	212	57.1%
	41–50	79	21.3%
	Over 50	17	4.6%
Education	Under high school	71	19.1%
	High school	82	22.1%
	Bachelor degree	202	54.5%
	Master's degree and above	16	4.3%
Annual income	$0–$4500	77	20.8%
	$4501–$15,000	200	53.9%
	$ 15,001–$45,000	81	21.8%
	Over $45,001	13	3.5%
User frequency	Once within a week	43	11.6%
	Twice within a week	105	28.3%
	3–5 times within a week	173	46.6%
	More than 5 times within a week	50	13.5%
Duration	Less than 5 min once	118	31.8%
	5–10 min once	203	54.7%
	10–20 min once	41	11.1%
	Over 20 min once	9	2.4%

Measurement development

All measurement items used in this paper were adapted from relevant studies. Measurement items for personalization were adapted according to Roy et al.⁶⁹ Three items were used to measure voice interaction based on the research by Zhu et al.²⁶ Measurements for enjoyment were modified from Lee et al.⁴⁰ To measure learning, three items were used according to Teng.⁴¹ Items measuring the condition of the COVID-19 pandemic were adapted from Omigie et al..⁶⁵ User satisfaction was measured according to the research by Li and Fang.⁷⁰ Finally, continuance intention was measured based on three items from Ashfaq et al..² Each item in this study was evaluated by using a seven-point Likert scale, with anchors ranging from 1 (highly disagree) to 7 (highly agree). All the measurement items were shown in Appendix. In order to make the potential respondents understand our questions, we translated the questionnaire from English to Chinese. As Brislin⁷¹ has argued, translation is necessary in the formulation of questionnaires for cross-cultural research. In this research, we checked the back-translation according to the approach suggested by Brislin⁷¹ and Werner and Campbell⁷²: (1) simple sentences, (2) repetition of nouns rather than use of pronouns, (3) avoiding English passive tense, (4) avoiding metaphor and colloquialisms.

Control variables

Several commonly controlled variables such as age, gender, income, and education were considered as control variables in this research. Chen and Mitomo⁷³ believed that these specific variables might have impact on the empirical results. Thus, we included them as control variables and did not test their influences in this research.

Results

Measurement model assessment

In this study, we employed the partial least squares structural equation modeling (PLS-SEM) to examine the 11 hypotheses. According to Kasilingam,⁵⁷ PLS-SEM has the ability to compare two or more groups by specifying a permutation-based analysis of variance approach. Thus, it is widely used for hypothesis-testing analysis.

The measurement model was assessed through reliability, convergent validity, and discriminant validity. According to Table 2, the Cronbach's alpha values for all variables are higher than 0.70, suggesting that the measurement is reliable.⁷⁴ In addition, the composite reliabilities of all the cases are higher than the recommended value of 0.70.²⁶ Moreover, the factor loadings of all items are greater than the desirable level of 0.70 and reported as significant.⁷⁵ The average variance extracted (AVE) ranges between 0.628 and 0.728, exceeding the recommended value of 0.5.⁷⁶ Thus, the measurement model meets the conditions of reliability and convergent validity.

Table 2.

The reliability and validity of the measurement.

Constructs	Items	Factor loading	Mean	Cronbach's alphas	CR	AVE	VIF
Personalization (P)	P1	0.817	4.299	0.839	0.892	0.674	2.925
	P2	0.765	4.814				2.530
	P3	0.876	4.445				3.244
	P4	0.822	4.774				2.684
Voice interaction (VI)	VI1	0.791	4.647	0.702	0.835	0.628	1.445
	VI2	0.746	4.776				1.269
	VI3	0.838	4.752				1.636
Enjoyment (E)	E1	0.785	4.437	0.752	0.858	0.669	1.429
	E2	0.797	4.501				1.509
	E3	0.870	4.685				1.722
Learning (L)	L1	0.795	4.631	0.746	0.856	0.664	1.463
	L2	0.792	4.679				1.435
	L3	0.856	4.863				1.723
Condition (C)	C1	0.804	4.892	0.784	0.875	0.701	1.678
	C2	0.800	4.644				1.624
	C3	0.903	4.752				2.312
User satisfaction (SA)	SA1	0.831	4.655	0.805	0.886	0.721	1.845
	SA2	0.810	4.749				1.612
	SA3	0.904	4.580				2.365
Continuance intention (CI)	CI1	0.824	4.663	0.811	0.889	0.728	1.815
	CI2	0.818	4.690				1.746
	CI3	0.914	4.666				2.546

In order to test discriminant validity, we used cross-factor loadings and Fornell-Larcker criterion. Table 3 shows the cross-factor loadings of all variables, proving that they loaded highest on their respective construct.²⁶ According to Table 4, none of the intercorrelations of the constructs are higher than the square root of the AVE for the constructs, indicating that the condition of discriminant validity is met by the measurement model.⁷⁵ Moreover, there is an adequate model fit when the value of SRMR is less than 0.08.⁵⁷ In this research, SRMR is 0.07.

Table 3.

Cross-factor loadings of all variables.

	P	VI	E	L	C	SA	CI
P1	0.817	0.404	0.645	0.635	0.629	0.683	0.660
P2	0.765	0.404	0.508	0.516	0.588	0.510	0.640
P3	0.876	0.429	0.677	0.656	0.662	0.702	0.689
P4	0.822	0.430	0.552	0.555	0.614	0.523	0.651
VI1	0.393	0.791	0.369	0.412	0.414	0.421	0.412
VI2	0.424	0.746	0.408	0.388	0.410	0.387	0.429
VI3	0.385	0.838	0.411	0.396	0.405	0.388	0.399
E1	0.588	0.375	0.785	0.570	0.590	0.563	0.619
E2	0.551	0.410	0.797	0.550	0.582	0.573	0.592
E3	0.649	0.440	0.870	0.606	0.670	0.677	0.708
L1	0.595	0.418	0.555	0.795	0.620	0.604	0.634
L2	0.612	0.357	0.579	0.792	0.625	0.582	0.661
L3	0.562	0.458	0.585	0.856	0.611	0.601	0.672
C1	0.611	0.445	0.593	0.601	0.804	0.626	0.683
C2	0.620	0.430	0.625	0.625	0.800	0.655	0.672
C3	0.676	0.426	0.670	0.678	0.903	0.660	0.728
SA1	0.627	0.386	0.623	0.595	0.633	0.831	0.661
SA2	0.618	0.460	0.625	0.605	0.659	0.810	0.676
SA3	0.648	0.438	0.641	0.660	0.677	0.904	0.725
CI1	0.662	0.419	0.661	0.669	0.681	0.674	0.824
CI2	0.683	0.446	0.641	0.662	0.693	0.669	0.818
CI3	0.712	0.472	0.704	0.728	0.749	0.728	0.914

Table 4.

Discriminant validity.

	P	VI	E	L	C	SA	CI
P	(0.821)
VI	0.507	(0.792)
E	0.731	0.500	(0.818)
L	0.723	0.505	0.704	(0.815)
C	0.760	0.518	0.753	0.759	(0.837)
SA	0.743	0.504	0.742	0.731	0.773	(0.849)
CI	0.804	0.523	0.785	0.805	0.830	0.810	(0.853)

Multicollinearity

In order to check multicollinearity, we examined the variance inflation factor (VIF). Table 2 shows that all VIF values are lower than the recommended value of 5.0,²⁶ indicating that the research model is free from multicollinearity.

Structural model analysis

The proposed hypothesis and path coefficients were examined by applying bootstrapping procedure with 2000 subsamples and two-tailed test with a significance of 0.05 in Smart PLS 3.3. According to Figure 3, user satisfaction (β = 0.207, t = 2.547, p < 0.01) and continuance intention (β = 0.199, t = 2.811, p < 0.01) were positively influenced by personalization, offering support to H1a and H1b. Enjoyment is positively related to user satisfaction (β = 0.225, t = 2.471, p < 0.05) and continuance intention (β = 0.152, t = 2.232, p < 0.05), offering support to H3a and H3b. In addition, the influences of learning on user satisfaction (β = 0.185, t = 2.037, p < 0.05) and continuance intention (β = 0.227, t = 2.470, p < 0.05) offer support to H4a and H4b, respectively. Similarly, the condition of the COVID-19 pandemic is positively correlated with both user satisfaction (β = 0.281, t = 2.905, p < 0.01) and continuance intention (β = 0.235, t = 3.063, p < 0.01), supporting H5a and H5b. User satisfaction also shows positive impact on continuance intention (β = 0.197, t = 2.150, p < 0.05). Thus, H6 is supported. However, voice interaction shows insignificant effect on both user satisfaction (β = 0.048, t = 1.078, p > 0.05) and continuance intention (β = 0.011, t = 0.329, p > 0.05), H2a and H2b should be rejected. Overall, 9 out of the 11 hypotheses are supported, as shown in Table 5. According to Figure 3, there were weak betas (<0.30) but high R² values (>0.67) in the research model (Seo & Bernsen, 2016). This may be because the determinant factors were highly interrelated (shown in Table 4). While these interrelated variables were significantly correlated with the dependent variables on an individual level, together they might compete to explain the same variance.

Figure 3.

Hypotheses testing. Path significance: *p < 0.05, **p < 0.01.

Table 5.

Results of hypotheses testing.

Hypothesis	Path	T statistics	Result
H1a	P → SA	2.547	Supported
H1b	P → CI	2.811	Supported
H2a	VI → SA	1.078	Not supported
H2b	VI → CI	0.329	Not supported
H3a	E → SA	2.471	Supported
H3b	E → CI	2.232	Supported
H4a	L → SA	2.037	Supported
H4b	L → CI	2.470	Supported
H5a	C → SA	2.905	Supported
H5b	C → CI	3.063	Supported
H6	SA → CI	2.150	Supported

Discussion and implications

AI-based mental health chatbots have been extensively used to address the COVID-19 outbreak-related mental disorders across the world. This study investigated the determinant factors behind citizens’ user satisfaction and continuance intention when interacting with a mental health chatbot. In addition, it further explored the relationship between user satisfaction and continuance intention.

Specifically, voice interaction failed to bring greater satisfaction and stronger continuance intention to users. This result is consistent with the research by Zhu et al..²⁶ Moreover, although personalization showed positive influences on user satisfaction and continuance intention, the effects were not as strong as the research by Chen et al.³⁷ Similarly, enjoyment showed weakly significant influences on user satisfaction and continuance intention. This result is in line with some previous studies.^40,77 Meanwhile, learning was positively related to people's user satisfaction and continuance intention, but the relationship was not strong. This result is different to some prior research^26,41 that found a great impact of learning on both user satisfaction and user loyalty. The condition of the COVID-19 pandemic also had weakly significant influences on user satisfaction and continuance intention. This is consistent with the research by Zhu et al.²⁶ Finally, the positive relationship between user satisfaction and continuance intention was weakly supported in this research. This result is in line with the research by Ashfaq et al.² (weakly supported) but is different to the study by Kim et al.⁵ (strongly supported).

Thus, although the positive influences of personalization, enjoyment, learning, and the condition of the COVID-19 pandemic were successfully demonstrated in this research, their effects were all rather weak. One appropriate explanation for this is that these value components of TCV frame the therapy as something to be consumed or sold to consumers, but this might be an incorrect way to consider mental health service. As mental health service in this research was free of charge public service that offered warm, personable, and reflective care, it cannot be handled from a consumer's perspective with TCV. Thereby, although the determinants of TCV could explain user satisfaction and continuance intention, they failed to exert strong influences. Another potential explanation is that we failed to find the most appropriate value components to predict user satisfaction and continuance intention toward mental health chatbots. TCV allows researchers to contextualize value components according to the specific product or service.⁵ Hence, we contextualized five value components based on the features of mental health chatbots and prior literature. It is possible that some other value components may have greater influences but we failed to identify them in this research.

Theoretical implications

Although value components were not strongly related to user satisfaction and continuance intention, the findings in this research can make contribution to the strengthening of theory regarding the acceptance and use of chatbots. First, this research is one of the early attempts to investigate chatbot users’ satisfaction and continuance intention based on TCV. Currently, the wide use of mental health chatbots creates a demand to extend the theoretical frameworks to predict user satisfaction and continuance intention. While the existing studies have predominately used TAM,³ this study provides a new perspective by employing TCV.

Second, this study also paves the way to future studies. Through the contextualized theorizing process, we identified five value components to predict user satisfaction and continuance intention toward mental health chatbots. As the results shown, voice interaction failed to play a significant role, the influences of personalization, enjoyment, learning, and the condition of the COVID-19 pandemic were weakly significant. These findings offer empirical evidences to future studies when choosing value components.

Practical implications

For practices, the findings in this research had the potential to contribute to user satisfaction and continuance intention because it developed end-user awareness on what factors should be considered for better performance of mental health chatbot services. First, our findings are straightforward: the personalized service is the predictor of user satisfaction and continuance intention. Accordingly, in order to make users keep using mental health chatbots, chatbot developers and designers should ensure that their products can offer personalized, relevant, and effective mental services. The design process should take into account the chatbot's capacity to understand user's demands and provide appropriate therapy.

Second, if people feel enjoyable when interacting with a mental health chatbot, they will tend to keep using this chatbot. Recently, with the evolutionary development of entertainment technologies, gamification techniques and features have been increasingly used in human–computer interaction to generate a feeling of enjoyment.⁷⁸ In the context of mental health chatbot usage, chatbot developers and designers can added more gamification features into mental health chatbots to make the human–chatbot interaction more pleasant and enjoyable. For example, these chatbots may recommend more funny games to users. It is believed that gamification elements have potential to offer a feeling of enjoyment and ultimately improve user satisfaction and continuance intention.

Moreover, if people can enrich their knowledge and satisfy their desire to learn when using a mental health chatbot, they will tend to feel satisfied and continuously adopt it. Thus, chatbot developers need to ensure that people can satisfy their desire to learn through the human–chatbot interaction. For instance, chatbot developers and designers might embed subject-related knowledge about mental health chatbots, such as purposes, functions, and cases, into chatbot systems so that users can read the information and enrich their knowledge about mental health chatbots. Also, chatbot developers may conduct a survey to understand what people desire to know about mental health chatbots. This can help mental health chatbots offer information to meet people's curiosity and improve their user satisfaction and continuance intention.

Research limitations

This research is not without limitations. First, as mentioned above, one of the most obvious limitations in this study is the use of TCV because the value components of TCV fail to exert strongly positive influences on user satisfaction and continuance intention. Second, there are problems regarding the generalizability of the findings, as the research samples were only collected from Wuhan, Hangzhou, and Chongqing. Future studies should conduct international investigation to validate the findings of this research. Third, we only chose a mental health chatbot as a research target, the results may be influenced by the advantages or shortages of the chatbot. Thus, researchers should consider more research targets in the future. Fourth, most of the correlations between factors were over 0.7, indicating that these determinant factors may share commonalities. This may influence the findings. Finally, this study considered demographic factors as control variables. Future studies should aim to test the potential moderating effect of demographic factors on continuance intention.

Conclusion

The wide use of mental health chatbots during the COVID-19 pandemic provides an opportunity to explore the determinant factors behind users’ satisfaction and continuance intention toward these virtual agents. This paper is one of the early attempts to investigate the usage of a mental health chatbot through the lens of TCV. Although the findings outlined the significance of personalization, enjoyment, learning, and the condition of the COVID-19 pandemic, their effects were all rather weak. Also, the positive relationship between user satisfaction and continuance intention was weakly supported. On the other hand, this research found that voice interaction is an insignificant predictor of user satisfaction and continuance intention. All these findings developed a critical perspective on the role of TCV in the context of mental health chatbot usage, while this theory has started to be increasingly used in the context of AI-based public service delivery. Based on the findings, several managerial implications for chatbot developers and designers were suggested to improve user satisfaction and continuance intention of mental health chatbots.

Although there were limitations in this research, it might make considerable contributions to both existing and future studies. First, it filled in our knowledge gaps by identifying four determinants that influenced user satisfaction and continuance intention of mental health chatbots during the COVID-19 pandemic. Moreover, it developed a critical perspective on TCV that prior studies could not acknowledge. In addition, it provided empirical evidence to enhance our understanding on the relationship between user satisfaction and continuance intention. Future studies can benefit from the findings in this research when investigating the usage of mental health chatbots.

Footnotes

Acknowledgments

We are grateful to the editor and anonymous reviewers for their valuable comments on this research. In particular, we should appreciate professor Theo Arvanitis and editor Kaur Simran.

Conflict of Interest

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Contributorship

Yonghan Zhu wrote the original draft of the manuscript, completed revisions, participated in the conceptualization of the manuscript content, and conducted data collection and data analysis. Rui Wang completed revisions, conducted data collection, and assisted in the data analysis. Chengyan Pu conducted data collection and provided revisions on the manuscript. All authors approved of the final manuscript draft.

Ethnical approval

Not applicable.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Guarantor

Yonghan Zhu.

ORCID iD

Yonghan Zhu

Appendix

References

Ransing

Nagendrappa

Patil

, et al. Potential role of artificial intelligence to address the COVID-19 outbreak-related mental health issues in India. Psychiatry Res 2020; 290: 113176.

Ashfaq

Yun

, et al. I, chatbot: modeling the determinants of users’ satisfaction and continuance intention of AI-powered service agents. Telemat Inform 2020; 54: 101473.

Pillai

Sivathanu

. Adoption of AI-based chatbots for hospitality and tourism. Int J Contemp Hosp Manage 2020; 32: 3199–3226.

Deng

Turner

Gehling

, et al. User experience, satisfaction, and continual usage intention on IT. Eur J Inf Syst 2010; 19: 60–75.

Kim

Chang

Park

, et al. The effects of service interactivity on the satisfaction and the loyalty of smartphone users. Telemat Inform 2015; 32: 949–960.

Forbes

. How business can begin using chatbots the right way. 2019. Retrieved from https://www.forbes.com/sites/forbestechcouncil/2019/01/18/how-businesses-can-begin-using-chatbots-the-right-way/#1d3b68252183 (Assessed, 13 July 2020).

Bhattacherjee

. Understanding information systems continuance: an expectation-confirmation model. MIS Q 2001: 351–370.

Rahi

Khan

Alghizzawi

. Extension of technology continuance theory (TCT) with task technology fit (TTF) in the context of Internet banking user continuance intention. Int J Quality Reliab Manage 2021; 38: 986–1004.

Shawar

Atwell

. Chatbots: are they really useful? LDV-Forum 2007; 22: 29–49.

10.

Valtolina

Barricelli

Gaetano

. Communicability of traditional interfaces VS chatbots in healthcare and smart home domains. Behav Inf Technol 2019; 39: 108–132.

11.

Vaidyam

Wisniewski

Halamka

, et al. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. Can J Psychiatry 2019; 64: 456–464.

12.

Pearl

. Designing voice user interfaces: principles of conversational experiences. O’Reilly Media, Inc, 2016.

13.

Nadarzynski

Miles

Cowie

, et al. Acceptability of artificial intelligence (AI)-led chatbot services in healthcare: a mixed-methods study. Digit Health 2019; 5: 1–12.

14.

WIRED. Virtual therapists help veterans open up about PTSD. 2018. https://www.wired.com/story/virtual-therapists-help-veterans-open-up-about-ptsd/. Retrieved from.

15.

Smith

Thomas

Snoswell

, et al. Telehealth for global emergencies: implications for coronavirus disease 2019 (COVID-19). J Telemed Telecare 2020; 26: 309–313.

16.

Ahuja

Reddy

Marques

. Artificial intelligence and COVID-19: a multidisciplinary approach. Integr Med Res 2020; 9: 100434.

17.

Liu

Yang

Zhang

, et al. Online mental health services in China during the COVID-19 outbreak. Lancet Psychiat 2020; 7: 30077–8.

18.

Fitzpatrick

Darcy

Vierhile

. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment Health 2017; 4: e19.

19.

Cameron

Megaw

, et al. Assessing the usability of a chatbot for mental health care. Int Workshop Int Sci 2019, April: 121–132. Cham: Springer.

20.

Chinadaily. Supporting epidemic prevention through digital technology. Government in Hangzhou recommends mental health chatbot 2020. Retrieved from http://caijing.chinadaily.com.cn/a/202004/03/WS5e86c784a3107bb6b57aa9f9.html.

21.

Chinanews. The chatbot Xiaoi offers several AI-based services to support epidemic prevention. 2020. Retrieved from https://m.chinanews.com/wap/detail/zw/business/2020/02–17/9094091.shtml.

22.

Kotler

Keller

. A framework for marketing management. Essex: Pearson Education Limited. 2012.

23.

Oliver

. Effect of expectation and disconfirmation on postexposure product evaluations: an alternative interpretation. J Appl Psychol 1977; 62: 480–486.

24.

Olson

Dover

. Disconfirmation of consumer expectations through product trial. J Appl Psychol 1979; 64: 179–189.

25.

Oliver

. Measurement and evaluation of satisfaction processes in retail settings. J Retail 1981; 57: 25–48.

26.

Zhu

Janssen

Wang

, et al. It is me, chatbot: working to address the COVID-19 outbreak-related mental health issues in China. User experience, satisfaction, and influencing factors. Int J Hum Comput Interact 2021.

27.

Lankton

McKnight

Thatcher

. Incorporating trust-in-technology into expectation disconfirmation theory. J Strategic Infor Syst 2014; 23: 128–145.

28.

Zhou

Cai

, et al. Unravelling the relationship between response time and user experience in mobile applications. Internet Res 2020; 30: 1353–1382.

29.

Liao

Palvia

Chen

. Information technology adoption behavior life cycle: toward a Technology Continuance Theory (TCT). Int J Inf Manage 2009; 29: 09–320.

30.

Sun

Liu

Peng

, et al. Understanding Chinese users’ continuance intention toward online social networks: an integrative theoretical model. Electron Mark 2014; 24: 57–66.

31.

Wang

Goh

& Lim

, et al. Understanding continuance intention toward crowdsourcing games: a longitudinal investigation. Int J Hum Comput Interact 2020; 36: 1168–1177.

32.

Lin

. Perceived fit and satisfaction on web learning performance: IS continuance intention and task-technology fit perspectives. Int J Hum Comput Stud 2012; 70: 498–507.

33.

Hoehle

Huff

Goode

. The role of continuous trust in information systems continuance. J Comput Inf Syst 2012; 52: 1–9.

34.

Kang

Namkung

. The role of personalization on continuance intention in food service mobile apps. Int J Contemp Hosp Manage 2019; 31: 734–752.

35.

Xiang

Zheng

Zhang

KZK

, et al. Understanding consumers’ continuance intention to contribute online reviews. Ind Manage Data Syst 2018; 118: 22–40.

36.

Kim

Ferrin

Rao

. Trust and satisfaction, two stepping stones for successful e-commerce relationships: a longitudinal exploration. Inf Syst Res 2009; 20: 237–257.

37.

Chen

Guo

Gao

, et al. AI-based self-service technology in public service delivery: user experience and influencing factors. Gov Inf Q 2021: 101520.

38.

Yeo

Mohamed

Muda

. A study of Malaysian customers purchase motivation of halal cosmetics retail products: examining theory of consumption value and customer satisfaction. Procedia Econ Finance 2016; 37: 176–182.

39.

Sheth

Newman

Gross

. Why we buy and what we buy: a theory of consumption values. J Bus Res 1991; 22: 159–170.

40.

Lee

Kim

. A customer value theory approach to the engagement with a brand: the case of KaoKao talk plus in Korea. Asia Pacific J Infor Syst 2018; 28: 36–60.

41.

Teng

. Look to the future: enhancing online gamer loyalty from the perspective of the theory of consumption values. Decis Support Syst 2018; 114: 49–60.

42.

Sweeney

Soutar

. Consumer perceived value: the development of a multiple item scale. J Retail 2001; 77: 203–220.

43.

Mathwick

Wiertz

de Ruyter

. Social capital production in a virtual P3 community. J Consum Res 2008; 34: 832–849.

44.

Hung

Hsieh

. Searching the fit pattern between cultural dimensions and consumption values of mobile commerce in Taiwan. Asia Pacific Manage Rev 2010; 15: 147–165.

45.

Furukawa

Matsumura

Harada

. Effect of consumption values on consumer satisfaction and brand commitment: investigating functional, emotional, social, and epistemic values in the running shoes market. Int Rev Manage Market 2019; 9: 158–168.

46.

Cheng

Jiang

. AI-powered mental health chatbots: examining users’ motivations, active communicative action and engagement after mass-shooting disasters. J Contingencies Crisis Manage 2020; 28: 339–354.

47.

Kidd

Hayden

. The psychology and neuroscience of curiosity. Neuron 2015; 88: 449–460.

48.

Macaskill

Denovan

. Assessing psychological health: the contribution of psychological strengths. Br J Guid Counc 2014; 42: 320–337.

49.

Macaskill

. A feasibility study of psychological strengths and well-being assessment in individuals living with recurrent depression. J Posit Psychol 2012; 7: 372–386.

50.

Sheng

Nah

Siau

. Values of silent commerce: a study using value-focused thinking approach. Proc Eleventh Am Conf Inf Syst 2005, August: 11–14. NE.

51.

Shi

Wang

Chen

, et al. Conceptualization of omnichannel customer experience and its impact on shopping intention: a mixed-method approach. Int J Inf Manage 2020; 50: 325–336.

52.

Wang

, et al. Factors affecting hotels adoption of mobile reservation systems: a technology-organization-environment framework. Tourism Manage 2016; 53: 163–172.

53.

Kocaballi

Berkovsky

Quiroz

, et al. The personalization of conversational agents in health care: systematic review. J Med Internet Res 2019; 21: e15360.

54.

Wang

. Design of a smart storytelling toy based on voice interaction. Proceedings of the 2019 2nd World Conference on Mechanical Engineering and Intelligent Manufacturing, November 2019: pp. 229–233. IEEE.

55.

Mavridis

. A review of verbal and non-verbal human-robot interactive communication. Robot Auton Syst 2015; 63: 22–35.

56.

Kiseleva

Williams

Awadallah

, et al. Predicting user satisfaction with intelligent assistants. Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, July 2016: pp. 45–54. ACM.

57.

Kasilingam

. Understanding the attitude and intention to use smartphone chatbots for shopping. Technol Soc 2020; 62: 101280.

58.

Abd-alrazaq

Alajlani

Alalwan

, et al. An overview of the features of chatbots in mental health: a scoping review. Int J Med Inf 2019; 132: 103978.

59.

Zhang

Zhu

Liu

. A meta-analysis of mobile commerce adoption and the moderating effect of culture. Comput Human Behav 2012; 28: 1902–1911.

60.

Agrebi

Jallais

. Explain the intention to use smartphones for mobile shopping. J Retail Consum Serv 2015; 22: 16–23.

61.

Merhi

. Factors influencing higher education students to adopt podcast: an empirical study. Comput Educ 2015; 83: 32–43.

62.

Kaur

Dhir

Rajala

, et al. Why people use online social media brand communities. A consumption value theory perspective. Online Inf Rev 2018; 42: 205–221.

63.

Qaoumi

Masson

Weil

, et al. Testing evolutionary theory of household consumption behavior in the case of novelty – a product characteristics approach. J Evol Econ 2018; 28: 437–460.

64.

Kim

Wang

Roh

. Do information and service quality affect perceived privacy protection, satisfaction, and loyalty? Evidence from a Chinese O2O–based mobile shopping application. Telemat Inform 2020: 101483.

65.

Omigie

Rho

, et al. Customer pre-adoption choice behavior for M-PESA mobile financial services. Extending the theory of consumption values. Ind Manage Data Syst 2017; 117: 910–926.

66.

Anderson EW and Sullivan MW. The antecedents and consequences of customer satisfaction for firms. Marketing Science 1993; 12: 125–143.

67.

Nascimento

Oliveira

Tam

. Wearable technology: what explains continuance intention in smartwatches? J Retail Consum Serv 2018; 43: 157–169.

68.

Yuan

Xiao

Dai

, et al. Modeling the effects of Wuhan's Lockdown during COVID-19, China. Bull W H O 2020; 98: 484–494.

69.

Roy

Balaji

Sadeque

, et al. Constituents and consequences of smart customer experience in retailing. Technol Forecast Soc Change 2017; 124: 257–270.

70.

Fang

. Predicting continuance intention toward mobile branded apps through satisfaction and attachment. Telemat Inform 2019; 43: 101248.

71.

Brislin

. Back-translation for cross-cultural research. J Cross Cult Psychol 1970; 1: 185–216.

72.

Werner

Campbell

D. T

. (1970). Translating working through interpreters, and the problem of decentering. Handbook Cultural Anthropol. American Museum of Natural History, New York.

73.

Cheng

Mitomo

. The underlying factors of the perceived usefulness of using smart wearable devices for disaster applications. Telematics & Informatics 2017; 34: 528–539.

74.

Zhang

Zhu

. Comparing attitudes towards adoption of e-government between urban users and rural users: an empirical study in Chongqing municipality, China. Behav Inf Technol 2021; 40: 1154–1168.

75.

Liu

Feng

. Public and private value in citizen participation in E-governance: evidence form a government – sponsored green commuting platform. Gov Inf Q 2019; 36: 101400.

76.

Fornell

Larcker

. Evaluating structural equation models with unobservable variables and measurement error. J Retail Consum Serv 1981; 18: 39–50.

77.

Melián-González

Gutiérrez-Taño

Bulchand-Gidumal

. Predicting the intentions to use chatbots for travel and tourism. Curr Issues Tourism 2021; 24: 192–210.

78.

Cabrera-Sánchez

Villarejo-Ramos

Liebana-Cabanillas

, et al. Identifying relevant segments of AI applications adopters – expanding the UTAUT2's Variables. Telemat Inform 2021; 58: 101529.