Most digital music repositories and services have mobile applications (apps) that facilitate convenient access for users via smartphones. Although China has one of the largest music listener populations in the world, there is little research evaluating Chinese online or mobile music services. To bridge this gap, this study evaluated mobile apps of three of the most popular Chinese music services from the user’s perspective, using usability testing and semi-structured interviews with a sample of active users in China. Nielsen’s 10 user experience heuristics and four criteria in recommender evaluation were examined. Results identified criteria that create a positive user experience, and those that need further improvement. This study contributes to the literature in user-centred evaluation in music information retrieval (MIR) and music digital libraries (MDL), and provides practical insights for music application design, use and evaluation.
UrbanoJSchedlMSerraX. Evaluation in music information retrieval. J Intell Inf Syst2013; 41: 345–369.
3.
KuhlthauCC. Inside the search process: information seeking from the user’s perspective. J Assoc Inf Sci Tech1991; 42: 361.
4.
ISO 9241-210:2010. Ergonomics of human-system interaction – part 210: human-centred design for interactive systems.
5.
HuXLeeJHBainbridgeDet al. The MIREX grand challenge: a framework of holistic user-experience evaluation in music information retrieval. J Assoc Inf Sci Technol2017; 68: 97–112.
6.
LeeJHPriceR. User experience with commercial music services: an empirical exploration. J Assoc Inf Sci Tech2016; 67: 800–811.
iiMedia Research. 2016–2017 Chinese mobile music application market research report [艾媒咨询: 2016–2017年中国手机音乐客户端市场研究报告], www.iimedia.cn/14875843996076279i.pdf (2016, accessed 20 December 2016).
9.
HuXLiuJ. Evaluation of music information retrieval: towards a user-centered approach. In: Proceedings of the 4th workshop on human-computer interaction and information retrieval (HCIR), New Brunswick, NJ, 22 August 2010.
10.
PauwsSEggenB. PATS: realization and user evaluation of an automatic playlist generator. In: Proceedings of the 3rd international society for music information retrieval conference (ISMIR), Paris, 13–17 October 2002.
11.
VignoliFPauwsSA. Music retrieval system based on user driven similarity and its evaluation. In: Proceedings the 6th international society for music information retrieval conference (ISMIR), London, 11–15 September 2005.
12.
PauwsSVan DeWijdeven S. User evaluation of a new interactive playlist generation concept. In: Proceedings of the 6th international society for music information retrieval conference (ISMIR), London, 11–15 September 2005.
13.
HoashiKHamawakiSIshizakiHet al. Usability evaluation of visualization interfaces for content-based music retrieval systems. In: Proceedings of the 10th international society for music information retrieval conference (ISMIR), Kobe, Japan, 26–30 October 2009, pp. 207–212. Canada: International Society for Music Information Retrieval.
14.
TurpinAScholerF. User performance versus precision measures for simple search tasks. In: Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval, Seattle, WA, 6–11 August 2006. Canada: International Society for Music Information Retrieval.
15.
HuXKandoN. User-centered measures vs. system effectiveness in finding similar songs. In: Proceedings of the 13th international society for music information retrieval conference (ISMIR), Porto, 8–12 October 2012, pp. 331–336.
16.
LeeJHCunninghamSJ. Toward an understanding of the history and impact of user studies in music information retrieval. J Intell Inf Syst2013; 41: 499–521. Canada: International Society for Music Information Retrieval.
DownieJSByrdDCrawfordT. Ten years of ISMIR: reflections on challenges and opportunities. In: Proceedings of the 10th international society for music information retrieval conference (ISMIR), Kobe, Japan, 26–30 October 2009, pp. 13–18. Canada: International Society for Music Information Retrieval.
19.
LeeJHHuXChoiKet al. MIREX grand challenge 2014 user experience: qualitative analysis of user feedback. In: Proceedings of the 16th international society for music information retrieval conference (ISMIR), Malaga, 26–30 October 2015, pp. 779–785. Canada: International Society for Music Information Retrieval.
20.
LeeDParkJYKimJet al. Understanding music sharing behaviour on social network services. Online Inform Rev2011; 35: 716–733.
21.
HuXLeeJHWongLKY. Music information behaviors and system preferences of university students in Hong Kong. In: Proceedings of the 15th international society for music information retrieval conference (ISMIR), Taipei, Taiwan, 27–31 October 2014, pp. 579–584. Canada: International Society for Music Information Retrieval.
GranićAMitrovićIMarangunićN. Exploring the usability of web portals: a Croatian case study. Int J Inform Manage2011; 11: 339–349.
24.
Suarez-TorrenteMDCConde-ClementePMartínezAet al. Improving web user satisfaction by ensuring usability criteria compliance: the case of an economically depressed region of Europe. Online Inform Rev2016; 40: 187–203.
25.
Gaona-GarcíaPAMartín-MoncunillDSánchez-AlonsoSet al. A usability study of taxonomy visualisation user interfaces in digital repositories. Online Inform Rev2014; 38(2): 284–304.
26.
OkhovatiMSharifpoorEAazamiMet al. Novice and experienced users’ search performance and satisfaction with Web of Science and Scopus. J Libr Inf Sci2017; 49: 359–367.
27.
ZhangY. Developing a holistic model for digital library evaluation. J Am Soc Inf Sci Tec2010; 61: 88–110.
PuPChenLHuR. A user-centric evaluation framework for recommender systems. In: Proceedings of the fifth ACM conference on recommender systems, Chicago, IL, 23–27 October 2011, pp. 157–164. New York: ACM.
30.
LeeJHWatermanNM. Understanding user requirements for music information services. In: Proceedings of the 13th international society for music information retrieval conference (ISMIR), Porto, 8–12 October 2012, pp. 253–258. Canada: International Society for Music Information Retrieval.
FaulknerL. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Method Instrum Comput2003; 35: 379–383.
33.
McHughML. Interrater reliability: the kappa statistic. Biochem Med2012; 22: 3276–3282.
34.
SchedlMKneesPMcFeeBet al. Music recommender systems. In: RicciFRokachLShapiraBet al. (eds) Recommender systems handbook. New York: Springer, 2015, pp. 453–492.
35.
NielsenJBudiuR. Mobile usability. Frechen: MITP-Verlags GmbH & Co. KG, 2013.
36.
CyrDHeadMIvanovA. Design aesthetics leading to m-loyalty in mobile commerce. Inform Manage2016; 43: 950–963.
37.
KnijnenburgBPWillemsenMCGantnerZet al. Explaining the user experience of recommender systems. User Model User-Adap2012; 22: 441–504.
38.
CastledineEEftosMWheelerM. Build mobile websites and apps for smart devices. Collingwood, VIC, Australia: Sitepoint, 2011.
39.
SlobodaJAO’NeillSAIvaldiA. Functions of music in everyday life: an exploratory study using the experience sampling method. Music Sci2001; 5: 9–32.
