Reader-Focused Text Evaluation

Abstract

This article presents a review of the literature on reader-focused text evaluation. First, an account is given of the document characteristics that can be evaluated. Then the possible functions of evaluations are considered, a distinction being made between verifying, troubleshooting, and choice-supporting research. Finally, an overview is presented of methods appropriate for the various document characteristics and evaluation functions. Relevant research findings on the methodological strengths and constraints of each method are discussed.

Get full access to this article

View all access options for this article.

References

Abeele, Piet vanden , and Ignace Butaye . Pretesting Advertisements: Empirical Validation of Pretest Methods. Report No. 7807. Leuven, Belg.: Catholic University of Leuven, Department of Applied Economics, 1978.

Ahlschwede, Margrethe P. Pre-Testing a Publication for Low-Income Homemakers. Report No. 10. Raleigh: North Carolina State University, Department of Agricultural Information, 1970.

Baker, Linda . “Metacognition, Comprehension Monitoring, and the Adult Reader.” Educational Psychology Review 1 (1989): 3-38.

Bertrand, Jane T. Communications Pretesting. Media Monograph 6. Chicago: University of Chicago, Communication Laboratory, Community and Family Study Center, 1978.

Bormuth, John R.

“Cloze Test Readability: Criterion Referenced Scores.”

Journal of Educational Measurement 5 (1968): 189-196.

Bormuth, John R. . “Comparable Cloze and Multiple-Choice Comprehension Test Scores.” Journal of Reading 11 (1967): 291-299.

Carter, Richard F. , et al.

“Application of the Signaled Stopping Technique to Communication Research.”

New Models for Mass Communication Research. Ed. Peter Clarke . Beverly Hills, CA: Sage, 1973. 15-43.

Cooper, Marilyn , and Michael Holzman . “Talking about Protocols.” College Composition and Communication 34 (1983): 284-293.

Couzijn, Michel , and Gert Rijlaarsdam . “Learning to Write by Reader Observation and Written Feedback.” Effective Teaching and Learning of Writing: Current Trends in Research. Ed. Gert Rijlaarsdam , Huub van den Bergh , and Michel Couzijn . Amsterdam: Amsterdam University Press, 1996. 224-252.

10.

Desurvire, Heather W.

“Faster, Cheaper!! Are Usability Inspection Methods as Effective as Empirical Testing?”

Usability Inspection Methods. Ed. Jakob Nielsen and Robert L. Mack . New York: John Wiley, 1994. 173-202.

11.

Dieli, Mary. Designing Successful Documents: An Investigation of Document Evaluation Methods. Diss. Carnegie-Mellon University, 1986.

12.

Dieli, Mary . “The Usability Process: Working with Iterative Design Principles.” IEEE Transactions on Professional Communication 32 (1989): 272-278.

13.

Duffy, Thomas M.

“Readability Formulas: What's the Use?”

Designing Usable Texts. Ed. Thomas M. Duffy and Robert Waller . Orlando: Academic Press, 1985. 113-143.

14.

Dumas, Joseph S. , and Janice C. Redish . A Practical Guide to Usability Testing. Norwood, NJ: Ablex, 1993.

15.

Elling, Rien . “Revising Safety Instructions with Focus Groups.” JBTC 11 (1997): 451-468.

16.

Entin, Eileen B. , and George R. Klare . “Some Inter-Relationships of Readability, Cloze and Multiple Scores on a Reading Comprehension Test.” Journal of Reading Behavior 10 (1978): 417-436.

17.

Ericsson, K. Anders . “Concurrent Verbal Reports on Text Comprehension: A Review.” Text 8 (1988): 295-325.

18.

Ericsson, K. Anders , and Herbert A. Simon . Protocol Analysis: Verbal Reports as Data. Cambridge: MIT Press, 1984.

19.

Flagg, Barbara N. Formative Evaluation for Educational Technologies. Hillsdale, NJ: Lawrence Erlbaum, 1990.

20.

Flower, Linda , John R. Hayes , and Heidi Swarts . “Revising Functional Documents: The Scenario Principle.” New Essays in Technical and Scientific Communication. Ed. Paul V. Anderson , R. John Brockman , and Carolyn R. Miller . New York: Baywood, 1983. 41-58.

21.

Goldman, Alfred A. , and Susan Schwartz McDonald . The Group Depth Interview. Principles and Practice. Englewood Cliffs, NJ: Prentice-Hall, 1987.

22.

Grice, Roger A. , and Lenore S. Ridgway . “A Discussion of Modes and Motives for Usability Evaluation.” IEEE Transactions on Professional Communication 32 (1989): 230-237.

23.

Haaland, Ane . Pretesting Communication Materials with Special Emphasis on Child Health and Nutrition Education: A Manual for Trainers and Supervisors. Rangoon, Burma: UNICEF, 1984.

24.

Hayes, John R. , and Linda Flower . “Uncovering Cognitive Processes in Writing. An Introduction to Protocol Analysis.” Research on Writing: Principles and Methods. Ed. Peter Rosenthal , Lynne Tamor , and Sean A. Walmsey . New York: Lawrence Erlbaum, 1983. 206-219.

25.

Hoeken, Hans . “Evaluating Persuasive Texts: The Problem of How and What to Measure.” Functional Communication Quality. Ed. Luuk van Waes , Egbert Woudstra , and Paul van den Hoven . Atlanta, GA: Rodopi, 1994. 76-87.

26.

Jackson, Kenneth M.

“Monitoring Communication Activity: New Techniques of Observation.”

Journalism Quarterly 51 (1974): 47-55.

27.

Jansen, Carel , and Michaël Steehouder . “Forms as a Source of Communication Problems.” Journal of Technical Writing and Communication 22 (1992): 179-194.

28.

Jansen, Frank , and Mariska Dee . “The Elicitation of Stylistic Evaluations from Readers of Functional Texts: Three Methods Compared.” Functional Communication Quality. Ed. Luuk van Waes , Egbert Woudstra , and Paul van den Hoven . Atlanta, GA: Rodopi, 1994. 88-103.

29.

Jong, Menno D.T. de , and Leo R. Lentz . “Expert Judgments versus Reader Feedback: A Comparison of Text Evaluation Techniques.” Journal of Technical Writing and Communication 26 (1996): 507-519.

30.

Jong, Menno D.T. de , and Peter Jan Schellens . “Focus Groups or Individual Interviews? A Comparison of Text Evaluation Approaches.” Paper submitted for publication.

31.

Jong, Menno D.T. de , and Peter Jan Schellens . “Towards a Valid Design for Pretesting and Revising Leaflets.” Functional Communication Quality. Ed. Luuk van Waes , Egbert Woudstra , and Paul van den Hoven . Atlanta, GA: Rodopi, 1994. 104-114.

32.

Karat, Claire-Marie . “A Comparison of User Interface Evaluation Methods.” Usability Inspection Methods. Ed. Jakob Nielsen and Robert L. Mack . New York: John Wiley, 1994. 203-233.

33.

Keitz, Beate von . “Eye-Movement Research: Do Consumers Use the Information They Are Offered?” European Research 16 (1988): 217-224.

34.

Klare, George R.

“Readability.”

Handbook of Reading Research. Ed. P. David Pearson et al. New York: Longman, 1984. 681-744.

35.

Krueger, Richard A. Focus Groups: A Practical Guide for Applied Research. 2nd ed. Thousand Oaks, CA: Sage, 1994.

36.

Lentz, Leo , and Menno de Jong . “The Evaluation of Text Quality: Expert-Focused and Reader-Focused Methods Compared.” IEEE Transactions on Professional Communication 40.3 (1997): Forthcoming.

37.

Lentz, Leo , and Henk Pander Maat . “Evaluating Text Quality: Reader-Focused or Text-Focused?” Studies of Functional Text Quality. Ed. Henk Pander Maat and Michaël Steehouder . Atlanta, GA: Rodopi, 1992. 101-114.

38.

Lewis, James R.

“Sample Sizes for Usability Studies: Additional Considerations.”

Human Factors 36 (1994): 368-378.

39.

Lunt, Peter , and Sonia Livingstone . “Rethinking Focus Group in Media and Communication Research.” Journal of Communication 46 (1996): 79-98.

40.

Mack, Robert , and Frank Montaniz . “Observing, Predicting, and Analyzing Usability Problems.” Usability Inspection Methods. Ed. Jakob Nielsen and Robert L. Mack . New York: John Wiley, 1994. 295-339.

41.

Micklos, David , and Walton Bishop . “The Measurement of Redundancy and Its Effects in Science Communication.” Annual Meeting of the Association for Education in Journalism and Mass Communication. Athens, OH, 25-28 July 1982.

42.

Morgan, David L. , ed. Successful Focus Groups. Advancing the State of the Art. Newbury Park, CA: Sage, 1993.

43.

Nathenson, Michael B. , and Euan S. Henderson . Using Student Feedback to Improve Learning Materials. London: Croom, 1980.

44.

Nielsen, Jakob . “Estimating the Number of Subjects Needed for a Thinking Aloud Test.” International Journal of Human-Computer Studies 41 (1994): 385-397.

45.

Nielsen, Jakob . Usability Engineering. Boston: Academic Press, 1993.

46.

Nielsen, Jakob , and Robert L. Mack , eds. Usability Inspection Methods. New York: John Wiley, 1994.

47.

Pander Maat, Henk . “Identifying and Predicting Reader Problems in Drug Information Texts.” Researching Technical Documents. Ed. Titus Ensink and Christoph Sauer . Groningen, Neth: University of Groningen, Department of Speech and Communication, 1996. 17-48.

48.

Queipo, Larry . “Taking the Mysticism out of Usability Test Objectives.” Technical Communication 38 (1991): 185-189.

49.

Rankin, Earl F. , and Joseph W. Culhane . “Comparable Cloze and Multiple-Choice Comprehension Test Scores.” Journal of Reading 12 (1969): 193-197.

50.

Rubin, Jeffrey . Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. New York: John Wiley, 1994.

51.

Russo, J. Edward , Eric J. Johnson , and Debra L. Stephens . “The Validity of Verbal Protocols.” Memory and Cognition 17 (1989): 759-769.

52.

Schellens, Peter Jan , and Menno de Jong . “Revision of Public Information Brochures on the Basis of Reader Feedback.” JBTC 11 (1997): 483-501.

53.

Schellens, Peter Jan , Menno de Jong , and Madelon Witteveen . “Functional Evaluation of Public Information about Laws and Regulations.” Discourse Analysis and Evaluation: Functional Approaches. Ed. Leo Lentz and Henk Pander Maat . Atlanta, GA: Rodopi, 1997. 157-175.

54.

Schriver, Karen A. Document Design as Rhetorical Action. Utrecht, Neth: Utrecht University, Faculty of Arts, 1995.

55.

Schriver, Karen A. . “Evaluating Text Quality: The Continuum from Text-Focused to Reader-Focused Methods.” IEEE Transactions on Professional Communication 32 (1989): 238-255.

56.

Schriver, Karen A. . “Plain Language through Protocol-Aided Revision.” Plain Language: Principles and Practice. Ed. Edwin R. Steinberg . Detroit, MI: Wayne State University Press, 1991. 148-172.

57.

Schriver, Karen A. . “Teaching Writers to Anticipate Readers' Needs: What Can Document Designers Learn from Usability Testing?” Studies of Functional Text Quality. Ed. Henk Pander Maat and Michaël Steehouder . Atlanta, GA: Rodopi, 1992. 141-157.

58.

Schumacher, Gary M. , and Robert Waller . “Testing Design Alternatives: A Comparison of Procedures.” Designing Usable Texts. Ed. Thomas M. Duffy and Robert Waller . Orlando: Academic Press, 1985. 377-403.

59.

Sienot, Matthijs . “Pretesting WEB Sites: A Comparison between the Plus-Minus Method and the Think-Aloud Method for the World Wide Web” JBTC 11 (1997): 469-482.

60.

Singh, Surendra N. , and Catherine A. Cole . “Advertising Copy Testing in Print Media.” Current Issues and Research in Advertising 11 (1988): 215-284.

61.

Smagorinsky, Peter . “The Reliability and Validity of Protocol Analysis.” Written Communication 6 (1989): 463-479.

62.

Steehouder, Michaël , and Carel Jansen . “Optimizing the Quality of Forms.” Studies of Functional Text Quality. Ed. Henk Pander Maat and Michaël Steehouder . Atlanta, GA: Rodopi, 1992. 159-172.

63.

Stewart, David W , et al.

“Methodological and Theoretical Foundations of Advertising Copy Testing: A Review.”

Current Issues and Research in Advertising 8 (1985): 1-74.

64.

Stratman, James F. , and Liz Hamp-Lyons . “Reactivity in Concurrent Think-Aloud Protocols.” Speaking about Writing: Reflections on Research Methodology. Ed. Peter Smagorinsky . Thousand Oaks, CA: Sage, 1994. 89-112.

65.

Swaney, Joyce H. , et al.

“Editing for Comprehension: Improving the Process through Reading Protocols.”

Plain Language: Principles and Practice. Ed. Edwin R. Steinberg . Detroit, MI: Wayne State University Press, 1991. 173-203.

66.

Taylor, Wilson L.

“Cloze Procedure: A New Tool for Measuring Readability.”

Journalism Quarterly 30 (1953): 415-433.

67.

Treistman, Joan , and John P. Gregg . “Visual, Verbal, and Sales Responses to Print Ads.” Journal of Advertising Research 19.4 (1979): 41-47.

68.

Ummelen, Nicole . Procedural and Declarative Information in Software Manuals: Effects on Information Use, Task Performance and Knowledge. Atlanta, GA: Rodopi, 1997.

69.

United States. Dept. of Health and Human Services . Pretesting in Health Communications: Methods, Examples and Resources for Improving Health Messages and Materials. NIH Pub. No. 84-1493. Bethesda, MD: National Institutes of Health, 1984.

70.

Velotta, Chris , ed. Practical Approaches to Usability Testing for Technical Documentation. Arlington, VA: Society for Technical Communication, 1995.

71.

Velte, Charles E.

“Document Usability through Objectives.”

IEEE Transactions on Professional Communication 32 (1989): 279-282.

72.

Virzi, Robert A.

“Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?”

Human Factors 34 (1992): 457-468.

73.

Weiss, Edmund H.

“Usability: Stereotypes and Traps.”

Text, ConText and HyperText. Ed. Edward Barrett . Cambridge: MIT Press, 1988. 175-185.

74.

Wiles, Julia A. , and T. Bettina Cornwell . “A Review of Methods Utilized in Measuring Affect, Feelings, and Emotion in Advertising.” Current Issues and Research in Advertising 13 (1991): 241-275.

75.

Wolfe, Harry D. , et al. Pretesting Advertising. Studies in Business Policy No. 109. New York: National Industrial Conference Board, 1963.

76.

Wright, Anita D.

“The Value of Usability Testing in Document Design.”

The Bulletin of the Association for Business Communication 57.4 (1994): 48-51.

77.

Wright, Peter C. , and Andrew F. Monk . “A Cost-Effective Evaluation Method for Use by Designers.” International Journal of Man-Machine Studies 35 (1991): 891-912.