User experience (UX) researchers in technical communication (TC) and beyond still need a clear picture of the methods used to measure and evaluate UX. This article charts current UX methods through a systematic literature review of recent publications (2016–2018) and a survey of 52 UX practitioners in academia and industry. Our results indicate that contemporary UX research favors mixed methods, and that usability testing is especially popular in both published research and our survey results. This article presents these findings as a snapshot of contemporary research methods for UX.
Bargas-AvilaJ. A.HornbækK. (2011). Old wine in new bottles or novel challenges: A critical analysis of empirical studies of user experience. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 93–102). ACM. http://doi.org/10.1145/1978942.1979336
2.
BeckC. E. (1992). Definition as central to the field of technical communication. Technical Communication, 39(3), 457–461. https://www.jstor.org/stable/43090118
3.
BoettgerR. K.FriessE. (2020). Content and authorship patterns in technical communication journals (1996–2017): A quantitative content analysis. Technical Communication, 67(3), 5–25.
4.
BoettgerR. K.LamC. (2013). An overview of experimental and quasi-experimental research in technical communication journals (1992–2011). IEEE: Transactions on Professional Communication, 56(4), 272–293. https://doi.org/10.1109/TPC.2013.2287570
5.
ChilanaP. K.KoA. J.WobbrockJ. O.GrossmanT.FitzmauriceG. (2011). Post-deployment usability: A survey of current practices. In CHI '11 proceedings of the SIGCHI conference on human factors in computing systems (pp. 2243–2246). Association for Computing Machinery, New York, NY. https://doi.org/10.1145/1978942.1979270
6.
ChongF. (2016). The pedagogy of usability: An analysis of technical communication textbooks, anthologies, and course syllabi and descriptions. Technical Communication Quarterly, 25(1), 12–28. https://doi.org/10.1080/10572252.2016.1113073
7.
CookeL. (2010). Assessing concurrent think-aloud protocol as a usability test method: A technical communication approach. IEEE Transactions on Professional Communication, 53(3), 202–215. https://doi.org/10.1109/TPC.2010.2052859
8.
de SáM.CarriçoL. (2011). Designing and evaluating mobile interaction: Challenges and trends. Foundations and Trends in Human–Computer Interaction, 4(3), 175–243. https://doi.org/10.1561/1100000025
9.
DoveG.HalskovK.ForlizziJ.ZimmermanJ. (2017). UX design innovation: Challenges for working with machine learning as a design material. In CHI '17 Proceedings of the 2017 CHI conference on human factors in computing systems (pp. 278–288). https://doi.org/10.1145/3025453.3025739
10.
ForlizziJ.BattarbeeK. (2004). Understanding experience in interactive systems. In DIS '04 Proceedings of the 5th conference on designing interactive systems: Processes, practices, methods, and techniques (pp. 261–268).
11.
GehannoJ.-F.RollinL.DarmoniS. (2013). Is the coverage of Google Scholar enough to be used alone for systematic reviews. BMC Medical Informatics and Decision Making, 13(7), 1–5. http://doi.org/10.1186/1472-6947-13-7
12.
GereaC.HerskovicV. (2015). Measuring user experience in Latin America: An exploratory survey. In Proceedings of the latin American conference on human computer interaction (pp. 19–14). ACM. http://doi.org/10.1145/2824893.2824914
13.
González-PérezL. I.Ramirez-MontoyaM.García-PeñalvoF. J. (2018). User experience in institutional repositories: A systematic literature review. International Journal of Human Capital and Information Technology Professionals, 9(1), 70–86. https://doi.org/10.4018/IJHCITP.2018010105
14.
GrayC. M. (2016). “It's more of a mindset than a method”: UX practitioners’ conception of design methods. In CHI '16 Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/2858036.2858410
15.
GrossA.BongartzS. (2012). Why do I like it? Investigating the product-specificity of user experience. In NordiCHI ’12 (pp. 322–330).
16.
HaoY.ChongW.ManK. L.LiuO.ShiX., (2016, March 16–18). Key factors affecting user experience of mobile crowdsourcing applications. In International MultiConference of Engineers and Computer Scientists 2016 Vol II, IMECS 2016, Hong Kong (pp. 1–6).
17.
HassenzahlM. (2018). The thing and I: Understanding the relationship between user and product. In BlytheM.MonkA. (Eds), Funology 2. Human–computer interaction series (pp. 301–313). Springer. https://doi.org/10.1007/978-3-319-68213-6_19
HenningT.BemerA. (2016). Reconsidering power and legitimacy in technical communication: A case for enlarging the definition of technical communicator. Journal of Technical Writing and Communication, 46(3), 311–341. https://doi.org/10.1177/0047281616639484
20.
HooperC. J.DixA. (2012). Web science and human-computer interaction: When disciplines collide. In WebSci '12 Proceedings of the 4th Annual ACM Web Science Conference, Evanston, IL (pp. 128–136). https://doi.org/10.1145/2380718.2380736
21.
IbargoyenA.SzostakD.BojicM. (2013). The elephant in the conference room: Let's talk about experience technology. In CHI 2013 Extended Abstracts (pp. 2079–2088).
IsmailN. A.AhmadF.KamaruddinN. A.IbrahimR. (2016). A review on usability issues in mobile applications. IOSR Journal of Mobile Computing & Application, 3(3), 47–52. https://doi.org/10.9790/0050-03034752
24.
JääskeläinenA.HeikkinenK. (2010). Divergence of user experience: Professional vs. end users. In Proceedings of I-UxSED 2010 (p. 30).
25.
KaasinenE.RotoV.HakulinenJ.HeimonenT.JokinenJ. P. P.KarvonenH.KeskinenT.KoskinenH.LuY.SaariluomaP.TokkonenH.TurunenM. (2015). Defining user experience goals to guide the design of industrial systems. Behaviour & Information Technology, 34(10), 976–991. https://doi.org/10.1080/0144929X.2015.1035335
26.
KastmanL. M.ZachryM.SpinuzziC. (2001). Usability instruction in technical communication programs: New directions in curriculum development. Journal of Business and Technical Communication, 15(2), 223–240. https://doi.org/10.1177/105065190101500204
27.
KimM. J.OhM. W.ChoM. E.LeeH.KimJ. T. (2013). A critical review of user studies on healthy smart homes. Indoor and Built Environment, 22(1), 1–11. https://doi.org/10.1177/1420326X12469733
28.
KimS. J. J. (2012). User study trends in augmented reality. In Proceedings of 2012 International Symposium on ubiquitous virtual reality (pp. 1–5). https://doi.org/10.1109/ISUVR.2012.17
29.
KjeldskovJ.GrahamC. (2003). A review of mobile HCI research methods. In L. Chittaro (Eds.), Mobile HCI 2003: Human-computer interaction with Mobile devices and services (pp. 317–335). Springer, Berlin, Heidelberg
30.
KurosuM.KobayashiT.YoshitakeR.TakahashiH.UrokoharaH.SatoD. (2004). Trends in usability research and activities in Japan. International Journal of Human-Computer Interaction, 17(1), 103–124. https://doi.org/10.1207/s15327590ijhc1701_8
31.
LachnerF.NaegeleinP.KowalskiR.SpannM.ButzA. (2016). Quantified UX: Towards a common organizational understanding of user experience. In NordiCHI ’16. https://doi.org/10.1145/2971485.2971501
32.
LallemandC.GronierG.KoenigV. (2015). User experience: A concept without consensus? Exploring practitioners’ perspectives through an international survey. Computers in Human Behavior, 43, 35–48. http://doi.org/10.1016/j.chb.2014.10.048
33.
LauerB.BrumbergerE. (2016). Technical communication as user experience in a broadening industry landscape. Technical Communication, 63(3), 248–264.
34.
LawE. L.RotoV.HazzenzahlM. (2009). Understanding, scoping, and defining user experience: A survey approach. CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. April 2009. Pages 719–728. http://doi.org/10.1145/1518701.1518813
35.
LawE.van SchaikP.RotoV. (2014). Attitudes towards user experience (UX) measurement. International Journal of Human-Computer Studies, 72(6), 526–541. http://doi.org/10.1016/j.ijhcs.2013.09.006
36.
MacDonaldC. M. (2015). User experience librarians: User advocates, user researchers, usability evaluators, or all of the above? In ASIST '15 Proceedings of the 78th ASIS&T annual meeting: Information science with impact: research in and for the community.
37.
MaiaC. L. B.FurtadoE. S. (2016). A systematic review about user experience evaluation. In MarcusA. (Ed.), Design, user experience, and usability: Design thinking and methods. DUXU 2016. Lecture notes in computer science (Vol. 9746, pp. 445–455). Springer.
38.
MarcusA. (Eds.). (2016). Design, User Experience, and Usability: Design Thinking and Methods. DUXU 2016. Lecture Notes in Computer Science, vol 9746. Springer, Cham.https://doi.org/10.1007/978-3-319-40409-7_49
39.
McNelyB.SpinuzziC.TestonC. (2015). Contemporary research methodologies in technical communication. Technical Communication Quarterly, 24(1), 1–13. https://doi.org/10.1080/10572252.2015.975958
40.
MelonçonL.St. AmantK. (2018). Empirical research in technical, & professional communication: A 5-year examination of research methods and a call for research sustainability. Journal of Technical Writing and Communication, 48(1), 1–28. https://doi.org/10.1177/0047281618764611
41.
MkpojioguE. O. C.AsuquoA. E. (2018). User experience of ATM users in Nigeria: A systematic review of empirical papers. JORIND, 16(1), 115–125.
42.
MaoJ.VredenburgK.SmithP. W.CareyT. (2005). The state of user-centered design practice. Communications of the ACM, 48(3), 105–109. https://doi.org/10.1145/1047671.1047677
43.
NieB.SunS. (2017). Using text mining techniques to identify research trends: A case study of design research. Applied Sciences, 7(4), 401–423. https://doi.org/10.3390/app7040401
44.
RajanenM.TapaniJ. (2018). A survey of game usability practices in North American game companies. In AnderssonB.JohanssonB.CarlssonS.BarryC.LangM.LingerH.SchneiderC. (Eds.), Designing digitalization (ISD2018 proceedings). Lund University. ISBN: 978-91-7753-876-9. http://aisel.aisnet.org/isd2014/proceedings2018/HCI/1
45.
RajanenD.ClemmensenT.IivariN.InalY.RızvanoğluK.SivajiA.RocheA. (2017). UX Professionals’ definitions of usability and UX – A comparison between Turkey, Finland, Denmark, France and Malaysia. In BernhauptR.DalviG.JoshiA.BalkrishanD. K.O’NeillJ.WincklerM. (Eds.), Human-computer interaction – INTERACT 2017. INTERACT 2017. Lecture notes in computer science, vol 10516 (pp. 218–239). Springer. https://doi.org/https://doi.org/10.1007/978-3-319-68059-0_14
46.
RedishG.BarnumC. (2011). Overlap, influence, intertwining: The interplay of UX and technical communication. Journal of Usability Studies, 6(3), 90–101.
47.
RobinsonJ.LaniusC. (2018). A geographic and disciplinary examination of UX empirical research since 2000. In SIGDOC '18 Proceedings of the 36th ACM international conference on the design of communication, Milwaukie, WI. https://doi.org/10.1145/3233756.3233930
48.
RobinsonJ.LaniusC.WeberR. (2017). The past, present, and future of UX empirical research. Communication Design Quarterly, 5(3), 10–22. https://doi.org/10.1145/3188173.3188175
49.
RotoV.VermeerenA.Väänänen-Vainio-MattilaK.LawE. (2011). User experience evaluation – which method to choose? In P Campos, N Graham, J Jorge, N Nunes, P Palanque, & M. Winckler (Eds.), Lecture notes in computer science (pp. 714–715). Springer, Berlin, Heidelberg. http://doi.org/10.1007/978-3-642-23768-3_129
50.
RudeC. D. (2009). Mapping the research questions in technical communication. Journal of Business and Technical Communication, 23(2), 174–215. http://doi.org/10.1177/1050651908329562
51.
SalgadoA. D. M.AmaralL. A.FreireA. P.FortesR. P. M. (2016). Usability and UX practices in small enterprises: Lessons from a survey of the Brazilian context. In SIGDOC '16 Proceedings of the 34th ACM international conference on the design of communication. https://doi.org/10.1145/2987592.2987616
SullivanP. (1989). Beyond a narrow conception of usability testing. IEEE Transactions on Professional Communication, 32(4), 256–264. https://doi.org/10.1109/47.44537
UgrasT.GülseçenS.ÇubukçuC.İli Erdoğmuşİ.GashiV.BedirM. (2016). Research trends in Web site usability: A systematic review. In MarcusA. (Ed.), Design, user experience, and usability: Design thinking and methods. DUXU 2016. Lecture notes in computer science, vol 9746 (pp. 517–528). Springer.
56.
VermeerenA.LawE.RotoV.ObristM.HoonhoutJ.Väänänen-Vainio-MattilaK. (2010). User experience evaluation methods: Current state and development needs. In Proceedings of the 6th Nordic Conference on human computer interaction: Extending boundaries (pp. 521–530). ACM. http://doi.org/10.1145/1868914.1868973
57.
VredenburgK.MaoJ.SmithP. W.CareyT. (2002). A survey of user-centered design practice. In CHI '02 proceedings of the SIGCHI conference on human factors in computing systems (pp. 471–478). ACM. https://doi.org/10.1145/503376.503460
58.
WilliamsS. D. (2007). User experience design for technical communication: Expanding our notions of quality information design. In 2007 IEEE International Professional Communication Conference (pp. 1–13). IEEE.
59.
YusopN. S. M.GrundyJ.VasaR. (2017). Reporting usability defects: A systematic literature review. IEEE Transactions on Software Engineering, 43(9), 848–867. https://doi.org/10.1109/TSE.2016.2638427
