While previous studies of customer service chat systems (CSCS) understood user satisfaction as individuals’ subjective perceptions and depended heavily on self-report methods for satisfaction measurement, this article presents an obtrusive chat log analysis that followed the established approaches of search log analysis and examined the relationships between dialog patterns and user satisfaction. An 81-day chat log was obtained from a real-world CSCS that involves both a chatbot and human representatives. A total of 75,918 chat sessions/147,972 sub-sessions containing 251,556 user messages and 349,416 system messages were extracted after data processing and analysed in terms of topic, length and path. As found in this study, the users were more likely to get satisfied on low-difficulty topics. The dialog between the CSCS and users was shallow in general. While human representatives’ elaboration contributed to user satisfaction, the chatbot was responsible for damaging user satisfaction. The significance of this study consists not only in providing objective evidence about user satisfaction in online chat but also in generating practical implications for the CSCS to improve user satisfaction.
FølstadANordheimCBBjørkliCA. What makes users trust a chatbot for customer service? An exploratory interview study. In: Proceedings of the 5th international conference on internet science (INSCI), St. Petersburg, Russia, 24–26 October 2018, pp. 194–208. Berlin; Heidelberg: Springer.
2.
ChungMKoEJoungHet al. Chatbot e-service and customer satisfaction regarding luxury brands. J Bus Res2020; 117: 587–595.
3.
McLeanGOsei-FrimpongK. Chat now… Examining the variables influencing the use of online live chat. Technol Forecast Soc2019; 146: 55–67.
4.
RappACurtiLBoldiA. The human side of human-chatbot interaction: a systematic literature review of ten years of research on text-based chatbots. Int J Hum: Comput St2021; 151: 102630.
5.
LiLLiCJiD. Deep context modeling for multi-turn response selection in dialogue systems. Inform Process Manag2021; 58: 102415.
6.
ShinD. How do people judge the credibility of algorithmic sources?AI Soc2022; 37: 81–96.
7.
Van der GootMJPilgrimT. Exploring age differences in motivations for and acceptance of chatbot communication in a customer service context. In: Proceedings of the international workshop on chatbot research and design, Amsterdam, 19–20 November 2019, pp. 173–186. Berlin; Heidelberg: Springer.
8.
VerhagenTVan NesJFeldbergFet al. Virtual customer service agents: Using social presence and personalization to shape online service encounters. J Comput-Mediat Comm2014; 19: 529–545.
9.
JansenBJ. Understanding user-web interactions via web analytics. San Rafael, CA: Morgan & Claypool Publishers, 2009.
10.
AnwarTAbulaishM. A social graph based text mining framework for chat log investigation. Digit Invest2014; 11: 349–362.
11.
DeviKN. Hotspot detection and analysis in chat environment. Asian J Res Soc Sci Human2016; 6: 1000–1007.
12.
ChenLFanAShiHet al. Region-action LSTM for mouse interaction sequence based search satisfaction evaluation. Inform Process Manag2020; 57: 102349.
13.
SalehiSDuJTAshmanH. Personalised Web search and user satisfaction: a user-centred evaluation. Inform Res2020; 25: 878.
14.
HassanAShiXCraswellNet al. Beyond clicks: query reformulation as a predictor of search satisfaction. In: Proceedings of the 22nd ACM international conference on information and knowledge management, San Francisco, CA, 27 October–1 November 2013, pp. 2019–2028. New York: Association for Computing Machinery (ACM).
15.
GnewuchUMoranaSAdamMet al. Faster is not always better: understanding the effect of dynamic response delays in human-chatbot interaction. In: Proceedings of the European conference on information systems (ECIS’2018), Portsmouth, 23–28 June 2018, p. 143975. Atlanta: AIS eLibrary.
16.
FeineJMoranaSGnewuchU. Measuring service encounter satisfaction with customer service chatbots using sentiment analysis. In: Proceedings of the 14th international conference on Wirtschaftsinformatik, Siegen, 24–27 February 2019, pp. 1115–1129. Atlanta: AIS eLibrary.
17.
KangLWangXTanC-Het al. Understanding the antecedents and consequences of live-chat use in electronic markets. J Org Comp Elect Com2015; 25: 117–139.
18.
ChattaramanVKwonW-SGilbertJE. Virtual agents in retail web sites: benefits of simulated social interaction for older users. Comput Hum Behav2012; 28: 2055–2066.
19.
McLeanGOsei-FrimpongK. Examining satisfaction with the experience during a live chat service encounter-implications for website providers. Comput Hum Behav2017; 76: 494–508.
20.
RietzTBenkeIMaedcheA. The impact of anthropomorphic and functional chatbot design features in enterprise collaboration systems on user acceptance. In: Proceedings of the 14th international conference on Wirtschaftsinformatik, Siegen, 24–27 February 2019, pp. 1642–1656. Atlanta: AIS eLibrary.
21.
ShinD. The perception of humanness in conversational journalism: an algorithmic information-processing perspective. New Media Soc. Epub ahead of print 12March2021. DOI: 10.1177/1461444821993801.
22.
Van den BroeckEZaroualiBPoelsK. Chatbot advertising effectiveness: when does the message get through?Comput Hum Behav2019; 98: 150–157.
23.
ShinD. Embodying algorithms, enactive artificial intelligence and the extended cognition: you can see as much as you know about algorithm. J Inf Sci. Epub ahead of print 12January2021. DOI: 10.1177/0165551520985495.
24.
ShinDChotiyaputtaVZaidB. The effects of cultural dimensions on algorithmic news: how do cultural value orientations affect how people perceive algorithms?Comput Hum Behav2022; 126: 107007.
25.
ElsholzEChamberlainJKruschwitzU. Exploring language style in chatbots to increase perceived product value and user engagement. In: Proceedings of the 2019 conference on human information interaction and retrieval, Glasgow, 10–14 March 2019, pp. 301–305. New York: Association for Computing Machinery (ACM).
26.
XuYShiehC-HVan EschPet al. AI customer service: task complexity, problem-solving ability, and usage intention. Australas Mark J2020; 28: 189–199.
27.
HillJFordWRFarrerasIG. Real conversations with artificial intelligence: a comparison between human–human online conversations and human–chatbot conversations. Comput Hum Behav2015; 49: 245–250.
28.
CortiKGillespieA. Co-constructing intersubjectivity with artificial conversational agents: people are more likely to initiate repairs of misunderstandings with agents represented as human. Comput Hum Behav2016; 58: 431–442.
29.
JansenBJSpinkABlakelyCet al. Defining a session on Web search engines. J Am Soc Inf Sci Tec2007; 58: 862–871.
30.
KacprzakEKoestenLIbáñezL-Det al. Characterising dataset search – an analysis of search logs and data requests. J Web Semant2019; 55: 37–55.
31.
LiaoZSongYHuangYet al. Task trail: an effective segmentation of user search behavior. IEEE T Knowl Data En2014; 26: 3090–3102.
32.
TeevanJRamageDMorrisMR. #TwitterSearch: a comparison of microblog search and web search. In: Proceedings of the 4th ACM international conference on web search and data mining, Hong Kong, China, 9–12 February 2011, pp. 35–44. New York: Association for Computing Machinery (ACM).
33.
JansenBJBoothD. Classifying web queries by topic and user intent. In: Proceedings of the extended abstracts on human factors in computing systems (CHI’10), Atlanta, GA, 10–15 April 2010, pp. 4285–4290. New York: Association for Computing Machinery (ACM).
34.
Al-MaskariASandersonM. A review of factors influencing user satisfaction in information retrieval. J Am Soc Inf Sci Tec2010; 61: 859–868.
35.
KamvarMKellarMPatelRet al. Computers and iPhones and mobile phones, oh my! A logs-based comparison of search users on different devices. In: Proceedings of the 18th international conference on World wide web, Madrid, 20–24 April 2009, pp. 801–810. New York: Association for Computing Machinery (ACM).
36.
SchultheißSLinhartABehnertCet al. Known-item searches and search tactics in library search systems: results from four transaction log analysis studies. J Acad Libr2020; 46: 102202.
37.
WalhoutJOomenPJarodzkaHet al. Effects of task complexity on online search behavior of adolescents. J Assoc Inf Sci Tech2017; 68: 1449–1461.
38.
HendahewaCShahC. Evaluating user search trails in exploratory search tasks. Inform Process Manag2017; 53: 905–922.
39.
TingHIClarkLKimbleCet al. APD-A tool for identifying behavioural patterns automatically from clickstream data. In: Proceedings of the international conference on knowledge-based intelligent information and engineering systems and the XVII Italian workshop on neural networks, Vietri sul Mare, 12–14 September 2007, pp. 66–73. Berlin; Heidelberg: Springer.
40.
JiangTChiYGaoH. A clickstream data analysis of Chinese academic library OPAC users’ information behavior. Libr Inform Sci Res2017; 39: 213–223.
41.
StolckeARiesKCoccaroNet al. Dialogue act modeling for automatic tagging and recognition of conversational speech. Comput Linguist2000; 26: 339–373.
42.
CarpenterTFujiokaE. The role and identification of dialog acts in online chat. In: Proceedings of the workshops at the 25th AAAI conference on artificial intelligence, San Francisco, CA, 7–11 August 2011, pp. 2–7. Menlo Park, CA: AAAI Press.
43.
ManikondaLPon-BarryHKambhampatiSet al. Discourse analysis of user forums in an online weight loss application. In: Proceedings of the joint workshop on social dynamics and personal attributes in social media, Baltimore, MD, 27 June2014, pp. 28–32. Stroudsburg, PA: Association for Computational Linguistics (ACL).
44.
TangSLiuQTanW-A. Intention classification based on transfer learning: a case study on insurance data. In: Proceedings of the international conference on human centered computing, Čačak, 5–7 August 2019, pp. 363–370. Berlin; Heidelberg: Springer.
45.
SharpeD. Chi-square test is statistically significant: now what?Pract Assess Res Eval2015; 20: 8.
46.
KoetterFBlohmMKochanowskiMet al. Motivations, classification and model trial of conversational agents for insurance companies, 2018, https://arxiv.org/abs/1812.07339
47.
CrescenziACapraRArguelloJ. Time pressure, user satisfaction and task difficulty. P Am Soc Inform Sci Tech2013; 50: 1–4.
48.
AgostiMCrivellariFDi NunzioGM. Web log analysis: a review of a decade of studies about information acquisition, inspection and interpretation of user interaction. Data Min Knowl Disc2012; 24: 663–696.
49.
AthukoralaKGłowackaDJacucciGet al. Is exploratory search different? A comparison of information search behavior for exploratory and lookup tasks. J Assoc Inf Sci Tech2016; 67: 2635–2651.
50.
ChangY-W. Influence of human behavior and the principle of least effort on library and information science research. Inform Process Manag2016; 52: 658–669.
