Heuristic and Formative Evaluation: A Case Study Illustration of a New Technique

Abstract

The purpose of the present work was to carry out an evaluation of an interactive, instructional Website that teaches the basic tenets of human performance technology. The evaluation methodology was based upon a unique combination of heuristic and formative evaluation techniques. It involved measuring attitudinal reactions to the Website, learning gains from performance scores on practice exercises, and content, navigation, and design areas needing modification or revision. Evaluation data were gathered from five students pursuing graduate degrees in education at an urban university. Paper-based attitude surveys, think-aloud protocols, and heuristic response forms were utilized to collect data. Student evaluators found the content in the Website to be useful and interesting; however, in some instances the practice items were confusing. The site was found to be easy to navigate and, overall, evaluators enjoyed using it. The evaluation methodology was shown to be effective in assessing design, content, and attitudinal issues, although in the future think-aloud protocols may be optional because they do not provide sufficient data to warrant the time spent on their use. Data also revealed that measuring learning gains was critical to the accurate evaluation and educational effectiveness of instructional Websites.

Get full access to this article

View all access options for this article.

References

Belotti

(1988). Implications of current design practice for the use of HCI techniques. In Jones

D. M.

Winder

(Eds.), People and computers IV (pp. 13–24). Cambridge: Cambridge University Press.

Cory

M. D.

Frick

T. W.

Hansen

(1997). User-centered design and usability testing of a Web site: An illustrative case study. Educational Technology Research and Development, 45:4, 65–76.

Cummings

(1998). What stakeholders want to know. In Brown

S. M.

Seidner

(Eds.), Evaluating corporate training: Models and issues (pp. 41–62). Boston: Kluwer Academic Publishers.

Dick

Carey

(1996). The systematic design of instruction (4th ed.). New York: HarperCollins College.

Dumas

J. S.

Redish

J. C.

(1993). A practical guide to usability testing. Norwood: Ablex Publishing Corporation.

Hix

Hartson

H. R.

(1994). Developing user interfaces: Ensuring usability through product and process. New York: John Wiley & Sons Incorporated.

Molich

Nielsen

(1990). Improving a human-computer dialogue. Communications of the ACM, 33:3, 338–348.

Nielsen

(1989). Usability engineering at a discount. In Salvendy

Smith

M. J.

(Eds.), Designing and using human-computer interfaces and knowledge based systems (pp. 394–401). Amsterdam: Elsevier Science Publishers.

Nielsen

(1990). Big paybacks from “discount” usability engineering. IEEE Software, 7:3, 107–108.

10.

Nielsen

(1993). Usability engineering. Boston: Academic Press.

11.

Nielsen

(1994a). Guerrilla HCI: Using discount usability engineering to penetrate the intimidation barrier. In Bias

R. G.

Mayhew

D. J.

(Eds.), Cost-justifying usability. Boston: Academic Press.

12.

Nielsen

(1994b). Heuristic evaluation. In Nielsen

Mack

R. L.

(Eds.), Usability inspection methods. New York: John Wiley & Sons.

13.

Patterson

Bloch

(1987). Formative evaluation: A process required in computer-assisted instruction. Educational Technology, 27:6, 26–30.

14.

Pike

G. R.

(1995). The relationship between self-reports of college experiences and achievement test scores. Research in Higher Education, 36:1, 1–21.

15.

Rubin

(1994). Handbook of usability testing: How to plan, design, and conduct effective tests. New York: John Wiley & Sons, Incorporated.

16.

Scriven

(1967). The methodology of evaluation (AERA Monograph series on curriculum evaluation, No. 1). Chicago: Rand McNally.

17.

Smith

P. L.

Ragan

T. J.

(2000). Instructional design (2nd ed.). Upper Saddle River: Prentice-Hall Incorporated.

18.

Smith

Wedman

J. F.

(1988). Read-think-aloud protocols: A new data-source for formative evaluation. Performance Improvement Quarterly, 1:2, 13–22.

19.

Whiteside

Bennett

Holtzblatt

(1988). Usability engineering: Our experience and evolution. In Helander

(Ed.), Handbook of human-computer interaction (pp. 791–817). North-Holland: Elsevier Science Publishers.