Usability assessment methods evolved from traditional human factors/ergonomics techniques beginning in the early 1980s. Following a brief historical introduction, we describe the four major categories of these methods: usability testing; usability inspections; surveys, interviews, and focus groups; and field methods. We describe the basic techniques of these methods, their strengths and weaknesses, their measurement characteristics, their validity and reliability, and how they are being applied to product development and assessment. We discuss the new challenges to usability assessment: (a) using the methods thoughtfully, (b) expanding the scope of assessment beyond usability and productivity to affect end user experience, (c) and emphasizing design solutions to problems in addition to simply finding the problems themselves.
Get full access to this article
View all access options for this article.
References
1.
AndreT.HartsonH.WilligesR. (2003). Determining the effectiveness of the usability problem inspector: A theory-based model and tool for finding usability problems. Human Factors, 45, 455–482.
2.
BaileyR. W.AllanR. W.RaielloP. (1992). Usability testing vs. heuristic evaluation: A head-to-head comparison. Proceedings of the Human Factors Society 36th Annual Meeting (pp. 409–413). Santa Monica, CA: SAGE Publications.
3.
BakemanR.GottmanJ. M. (1986). Observing interaction: An introduction to sequential analysis. New York: Cambridge University Press.
4.
BarnumC. (2002). Usability testing and research. New York: Longman.
5.
BeyerH.HoltzblattK. (1998). Contextual design: Defining customer-centered systems. San Francisco: Morgan Kaufmann.
6.
BergS.TaylorA.HarperR. (2003). Mobile phones for the next generation: Device designs for teenagers. Proceedings of CHI ′03 (pp. 433–440). New York: ACM Press.
7.
BiasR. (1994). The pluralistic usability walkthrough: Coordinated empathies. In NielsenJ.MackR. (Eds.), Usability inspection methods (pp. 63–76). New York: Wiley.
BlytheM.WrightP. (2003). Introduction. In BlytheM.OverbeekeK.MonkA.WrightP. (Eds.), Funology: From usability to enjoyment (pp. xiii–xix). Boston: Kluwer Academic Publishers.
10.
BorenM.RameyJ. (2000). Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication, 43(3), 261–278.
11.
BrookeJ. (1996). SUS: A quick and dirty usability scale. In JordanP.ThomasB.WeerdmeesterB.McClellandI. (Eds.), Usability evaluation in industry (pp. 189–194). London: Taylor & Francis.
12.
CantaniM. B.BiersD. W. (1998). Usability evaluation and prototype fidelity: Users and usability professionals. Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting (pp. 1331–1335). Santa Monica, CA: SAGE Publications.
13.
ChapanisA. (1959). Research techniques in human engineering. Baltimore: Johns Hopkins University Press.
14.
CourageC.BaxterK. (2005). Understanding your users. A practical guide to user requirements. methods, tools, and techniques. San Francisco: Morgan Kaufmann.
DesurvireH.KondzielaJ.AtwoodM. (1992). What is gained and lost when using evaluation methods other than empirical testing. In MonkA.DiaperD.HarrisonM. (Eds.), People and computers VII (pp. 89–102). Cambridge, England: Cambridge University Press.
17.
DumasJ. (1998). Usability testing methods: Using test participants as their own controls. Common Ground, 8, 3–5.
18.
DumasJ. (2001). Usability testing methods: Think aloud protocols. In BranaghanR. (Ed.), Design by people for people: Essays on usability (pp. 119–130). Chicago: Usability Professionals' Association.
19.
DumasJ. (2003). Usability evaluation from your desktop. Association for Information Systems (AIS) SIGCHI Newsletter, 2(2), 7–8.
20.
DumasJ.MolichR.JeffriesR. (2004). Describing usability problems: Are we sending the right message? Interactions, 11, 24–29.
21.
DumasJ.RedishG. (1993). A practical guide to usability testing. London: Intellect Books.
22.
DumasJ.RedishG. (1999). A practical guide to usability testing (rev. ed.). London: Intellect Books.
23.
DumasJ.SorceJ.VirziR. (1995). Expert reviews: How many experts is enough? Proceedings of the Human Factors and Ergonomics Society 39th Annual Meeting, (pp. 309–312). Santa Monica, CA: SAGE Publications.
24.
EblingM.JohnB. (2000). On the contributions of different empirical data in usability testing. Proceedings of Designing Interactive Systems, (pp. 289–296). New York: ACM Press.
25.
EricssonK. A.SimonH. A. (1993). Protocol analysis: Verbal reports as data. Cambridge: MIT Press.
26.
FaulknerL. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers, 35(3), 379–383.
27.
GoodwinN. C. (1982). Effect of interface design on usability of message handling systems. Proceedings of the Human Factors Society 26th Annual Meeting (pp. 69–73). Santa Monica, CA: SAGE Publications.
28.
GrayW.SalzmanM. (1998). Damaged merchandise? A review of experiments that compare usability methods. Human-Computer Interaction, 13, 203–335.
29.
GrinterB.EldridgeM. (2003). Wan2tlk?: Everyday text messaging. Proceedings of CHI ′03 (pp. 441–448). New York: ACM Press.
30.
HackosJ.RedishG. (1998). User and task analysis for interface design. New York: Wiley.
31.
HancockP.PepeA.MurphyL. (2005). Hedonomics: The power of positive and pleasurable ergonomics. Ergonomics in Design, 13(1), 8–14.
HertzumM.JacobsenN. E. (2001). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 13, 421–443.
34.
HixD.HartsonH. (1993). Developing user interfaces: Ensuring usability through process and product. New York: Wiley.
35.
HongJ.LandayJ. (2001). WebQuilt: A framework for capturing and visualizing the web experience. Proceedings of the 10th International Conference on World Wide Web (pp. 717–724). Southampton, England: University of Southampton.
HughesJ.KingV.RoddenT.AndersenH. (1995). The role of ethnography in interactive systems design. Interactions, 2(2), 56–65.
38.
HurstW. (2003). Indexing, searching, and skimming of multimedia documents containing recorded lectures and live presentations. Proceedings of 11th Annual Conference of Multimedia Publisher (pp. 450–451). New York: ACM Press.
39.
International Organization for Standardization. (1998). Ergonomic requirements for office work with visual display terminals (VDTs)—Part 11: Guidance on usability (ISO 9241–11:1998[E]). Switzerland: Author.
40.
JacobsenN.HertzumM.JohnB. E. (1998). The evaluator effect in usability studies: Problem detection and severity judgments. Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting (pp. 1336–1340). Santa Monica, CA: SAGE Publications.
41.
JeffriesR.MillerJ.WhartonC.UyedaK. (1991). User interface evaluation in the real world: A comparison of four techniques. Proceedings of CHI ′91 (pp. 119–124). New York: ACM Press.
42.
JordanP. (2000). Designing pleasurable products. An introduction to the new human factors. New York: Taylor & Francis.
43.
JordanP. (2002). The personalities of products. In GreenW.JordanP. (Eds.), Pleasure with products (pp. 19–48). London: Taylor & Francis.
44.
JungY.PerssonP.BlomJ. (2005). DeDe: Design and evaluation of a context-enhanced mobile messaging system. Proceedings of CHI ′2005 (pp. 351–360). New York: ACM Press.
45.
KaratJ. (2003). Beyond task completion: Evaluation of affective components of use. In JackoJ.SearsA. (Eds.), The human-computer interaction handbook (pp. 1152–1164). Mahwah, NJ: Erlbaum.
46.
KirahA.FusonC.GrudinJ.FeldmanE. (2003). Ethnography for software development. In JackoJ.SearsA. (Eds.), The human-computer interaction handbook (pp. 415–446). Mahwah, NJ: Erlbaum.
47.
KirakowskiJ. (1996). The Software Usability Measurement Inventory (SUMI): Background and usage. In JordanP.ThomasB.WeerdmeesterB.McClellandI. (Eds.), Usability evaluation in industry (pp. 169–177). London: Taylor & Francis.
48.
KirakowskiJ. (2005). Summative usability testing: Measurement and sample size. In BiasR.MayhewD. (Eds.), Cost-justifying usability: An update for the internet age (2nd ed., pp. 519–554). San Francisco: Morgan Kaufmann.
49.
KirakowskiJ.CorbettM. (1988). Measuring user satisfaction. In JonesD.WinderR. (Eds.), People and computers, Vol. 4. Cambridge, England: Cambridge University Press.
50.
KirakowskiJ.CorbettM. (1990). Effective methodology for the study of HCI. New York: Elsevier.
KrahmerE.UmmelenN. (2004). Thinking about thinking aloud: A comparison of two verbal protocols for usability testing. IEEE Transactions on Professional Communication, 47(2), 105–117.
53.
KuniavskyA. (2003). Observing the user experience. A practitioner's guide to user research. San Francisco: Morgan Kaufmann.
54.
LawC.VanderheidenG. (2000). Reducing sample sizes when user testing with people who have and who are simulating disabilities: Experiences with blindness and public information kiosks. Proceedings of the XIVth Triennial Congress of the International Ergonomics Association and 44th Annual Meeting of the Human Factors and Ergonomics Society (pp. 4.157–4.160). Santa Monica, CA: SAGE Publications.
55.
LesaigleE. M.BiersD. W. (2000). Effect of type of information on real-time usability evaluation: Implications for remote usability testing. Proceedings of the XIVth Triennial Congress of the International Ergonomics Association and 44th Annual Meeting of the Human Factors and Ergonomics Society (pp. 6.585–6.588). Santa Monica, CA: SAGE Publications.
56.
LewisC.MackR. (1982). Learning to use a text processing system: Evidence from “thinking aloud” protocols. Proceedings of Human Factors in Computer Systems (pp. 387–392). New York: ACM Press.
57.
LewisJ. (1991). Psychometric evaluation of an after-scenario questionnaire for computer usability studies: The ASQ. SICCHI Bulletin, 23, 78–81.
58.
LewisJ. (1994). Sample size for usability studies: Additional considerations. Human Factors, 36, 368–378.
59.
LewisJ. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1), 57–78.
60.
LewisJ. (2001). Evaluation of procedures of adjusting problem-discovery rates estimated from small samples. International Journal of Human-Computer Interaction, 13, 445–480.
61.
MalhotraA.GosainS.HarsA. (1997). Evolution of a virtual community: Understanding design issues through a longitudinal study. Proceedings of the International Conference on Information Systems (pp. 59–73). Atlanta: Association for Information Systems.
62.
MaoJi-YeVredenburgK.SmithP. W.CareyT. (2005). The state of user-centered design practice. Communications of the ACM, 48, 105–109.
63.
McGeeM. (2003). Usability magnitude estimation. Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting (pp. 691–695). Santa Monica, CA: SAGE Publications.
64.
McGeeM. (2004). Master usability scaling: Magnitude estimation and master scaling applied to usability measurement. Proceedings of CHI ′2004 (pp. 335–342). New York: ACM Press.
65.
MedlockM.WixonD.McGeeM.WelshD. (2005). The rapid iterative test and evaluation method: Better products in less time. In BiasR.MayhewD. (Eds.), Cost-justifying usability: An update for the information age (pp. 489–517). San Francisco: Morgan Kaufmann.
66.
MedlockM. C.WixonD.TerranoM.RomeroR.FultonB. (2002). Using the RITE method to improve products: A definition and a case study. Proceedings of the Usability Professionals Association Annual Meeting [CD-ROM]. Chicago: Usability Professionals Association.
67.
MeisterD. (1999). The history of human factors and ergonomics. Mahwah, NJ: Erlbaum.
68.
MillenD. (2000). Rapid ethnography: Time deepening strategies for HCI field research. In DIS 2000, Designing Interactive Systems (pp. 280–286). New York: ACM Press.
69.
MolichR.BevanN.CursonI.ButlerS.KindlundE.MillerD.KirakowskiJ. (1998). Comparative evaluation of usability tests. Proceedings of the Usability Professionals' Association Annual Meeting [CD-ROM]. Chicago: Usability Professionals Association.
70.
MolichR.DumasJ. (in press). Comparative usability evaluation (CUE-4). Behaviour and Information Technology.
MurphyL.StanneyK.HancockP. (2003). The effect of affect: The hedonic evaluation of human-computer interaction. Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting (pp. 764–767). Santa Monica, CA: SAGE Publications.
74.
NielsenJ. (1992). Finding usability problems through heuristic evaluation. Proceedings of CHI ′92 (pp. 373–380). New York: ACM Press.
75.
NielsenJ. (1993). Usability engineering. New York: Academic Press.
76.
NielsenJ. (1994). Heuristic evaluation. In NielsenJ.MackR. (Eds.), Usability inspection methods (pp. 25–62). New York: Wiley.
77.
NielsenJ.MolichR. (1990). Heuristic evaluation of user interface. Proceedings of CHI ′90 (pp. 241–256). New York: ACM Press.
78.
NielsonJ.MackR. (1994). Usabilitiy inspection methods. New York: Wiley.
79.
OlsonJ.MoranT. (1995). Mapping the method muddle: Guidance in using methods for user interface design. In RudisillM.LewisC.PolsonP.McKayT. (Eds.), Human-computer interface designs: Success stories, emerging methods, and real world context (pp. 269–300). San Francisco: Morgan Kaufman.
80.
PagulayanR.KeekerK.WixonD.RomeroR.FullerT. (2003). User-centered design in games. In JackoJ.SearsA. (Eds.), The human-computer interaction handbook (pp. 883–906). Mahwah, NJ: Erlbaum.
81.
PalenL.SalzmanM. (2002a). Beyond the handset: Designing for wireless communications usability. In ACM Transactions on Computer Human Interaction, 9(2), 125–151.
82.
PalenL.SalzmanM. (2002b). Voice-mail diary studies for naturalistic data capture under mobile conditions. Proceedings of Computer Supported Cooperative Work 2002 (pp. 87–95). New York: ACM Press.
83.
PerkinsR. (2001). Remote usability evaluation over the Internet. In BranaghanR. (Ed.), Design by people for people: Essays on usability (pp. 153–162). Chicago: Usability Professionals Association.
QuesenberryW. (2004). Balancing the 5Es: Usability. Cutter IT Journal17(2), 4–11.
86.
QuesenberryW. (2005). The five dimensions of usability. In AlbersM.MazurB. (Eds.), Content and complexity: Information design in technical communication (pp. 81–102). Mahwah, NJ: Erlbaum.
87.
RobertsT.MoranT. (1982). Evaluation of text editors. Proceedings of Human Factors in Computer Systems (pp. 136–141). New York: ACM Press.
88.
RoseA.ShneidermanB.PlaisantC. (1995). An applied ethnographic method for redesigning user interfaces. DIS ′95, Designing Interactive Systems (pp. 115–122). New York: ACM Press.
89.
RosenbaumS.CocktonG.CoyneK.MullerM.RauchT. (2002). Focus groups in HCI: Wealth of information or waste of resources? Proceedings of CHI 2002 (pp. 702–703). New York: ACM Press.
90.
RosenbaumS.RohnJ.HumburgJ. (2000). A toolkit for strategic usability: Results from workshops, panels, and surveys. Proceedings of CHI 2000 (pp. 337–344). New York: ACM Press.
91.
RowleyD.RhodesD. (1992). The cognitive jog through: A fast-paced user interface evaluation procedure. Proceedings of CHI ′92 (pp. 389–395). New York: ACM Press.
92.
RubinJ. (1994). Handbook of usability testing. New York: Wiley.
93.
SawyerP.FlandersA.WixonD. (1996). Making a difference—The impact of inspection. Proceedings of CHI ′96 (pp. 378–382). New York: ACM Press.
94.
SearsA. (1997). Heuristic walkthroughs: Finding the problems without the noise. International Journal of Human Computer Interaction, 9, 213–234.
95.
SheppardS.KruesiE.BaileyJ. (1982). An empirical evaluation of software documentation formats. Proceedings of Human Factors in Computer Systems (pp. 121–124). New York: ACM Press.
96.
ShneidermanB. (1987). Designing the user interface: Strategies for effective human computer interaction. Reading, MA: Addison-Wesley.
97.
ShneidermanB. (1997). Designing the user interface: Strategies for effective human computer interaction (3rd ed.). Reading, MA: Addison-Wesley.
98.
SiegelD.DrayS. (2003). Living on the edges: User-centered design and the dynamics of specialization in organizations. Interactions, 10, 19–27.
99.
SklarA.GilmoreD. (2004). Are you positive? Interactions, 11, 28–33.
SnyderC. (2003). Paper prototyping. San Francisco: Morgan Kaufmann.
102.
SpencerR. (2000). The streamlined cognitive walkthrough method, working around social constraints encountered in a software development company. Proceedings of CHI ′92 (pp. 353–359). New York: ACM Press.
103.
SpinuzziC. (2000). Investigating the technology-work relationship: A critical comparison of three qualitative field methods. IEEE, 2000, 419–432.
104.
SullivanP. (1991). Multiple methods and the usability of interface prototypes: The complimentarity of laboratory observation and focus groups. Proceedings of CHI ′91 (pp. 106–112). New York: ACM Press.
105.
TeagueR.WhitneyH. (2002), What's love got to do with it? User Experience, 1, 6–13.
106.
TullisT.FlieschmanS.McNultyM.CianchetteC.BergelM. (2002). An empirical comparison of lab and remote usability testing of web sites. Proceedings of Usability Professionals Association (pp. 1–8). Chicago: Usability Professionals Association.
107.
TullisT.StetsonJ. (2004). A comparison of questionnaires for assessing website usability. Proceedings of Usability Professionals' Association (pp. 1–12). Chicago: Usability Professionals Association.
108.
VirziR. (1990). Streamlining the design process: Running fewer subjects. Proceedings of the Human Factors Society 34th Annual Meeting (pp. 224–228). Santa Monica, CA: SAGE Publications.
109.
VirziR. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34, 457–468.
110.
VirziR. A.SokolovJ.KarisD. (1996). Usability problem identification using both low- and high-fidelity prototypes. Proceedings of CHI ′96 (pp. 236–243). New York: ACM Press.
111.
VirziR. A.SorceJ. F.HerbertL. B. (1993). A comparison of three usability evaluation methods: Heuristic, think-aloud, and performance testing. Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting (pp. 309–313). Santa Monica, CA: SAGE Publications.
112.
WalkerM.TakayamaL.LandayJ. (2002). High-fidelity, paper or computer? Choosing attributes when testing web prototypes. Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting (pp. 661–665). Santa Monica, CA: SAGE Publications.
113.
WhartonC.BradfordJ.JeffriesR.FranzkeM. (1992). Applying cognitive walkthroughs to more complex user interfaces: Experiences, issues, and recommendations. Proceedings of CHI ′92 (pp. 381–388). New York: ACM Press.
114.
WhitesideJ.BennettJ.HoltblattK. (1988). Usability engineering: Our experience and evolution, In HelanderM. (Ed.), Handbook of human-computer interaction (pp. 791–817). Amsterdam: North-Holland.
115.
WixonD.RameyJ. (Eds.) (1996). Field methods casebook for software design. New York: Wiley.
116.
WixonD.RameyJ. (2002). Usability in practice: Field methods evolution and revolution. In Extended Abstracts for CHI ′02 (pp. 880–884). New York: ACM Press.
117.
WoodL. (1996). The ethnographic interview in user-centered work/task analysis. In WixonD.RameyJ. (Eds.), Field methods casebook for software design (pp. 101–121). New York: Wiley.
118.
WoodsD. (1996). Decomposing automation: Apparent simplicity, real complexity. In ParasuramanR.MoulouaM. (Eds), Automation and human performance: Theory and applications (pp. 3–17). Mahwah, NJ: Erlbaum.
119.
WoolrychA.CocktonG. (2002). Testing a conjecture based on the DR-AR model of UIM effectiveness. Proceedings of HCI 2002 (pp. 30–33). London: Springer-Verlag.
