Are technical writers able to predict the results of a reader-focused text evaluation? In this article we report a study with fifteen technical writers, who were asked to point out the reader problems in a public information brochure. The brochure was also evaluated with thirty readers from the target audience (using a combination of the plus-minus method, a questionnaire, and user protocols). The results of both kinds of text evaluation show little overlap. The technical writers only predicted a small proportion of the reader feedback, and produced a lot of new problem detections. In addition, there was little agreement among the technical writers with regard to their problem detections.
Get full access to this article
View all access options for this article.
References
1.
DumasJ. S. and RedishJ. C., A Practical Guide to Usability Testing, Ablex, Norwood, New Jersey, 1993.
2.
RubinJ., Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, John Wiley & Sons, New York, 1994.
3.
VelottaC. (ed.), Practical Approaches to Usability Testing for Technical Documentation, Society for Technical Communication, Arlington, Virginia, 1995.
4.
WrightP., Is Evaluation a Myth? Assessing Text Assessment Procedures, in The Technology of Text: Principles for Structuring, Designing, and Displaying Text, Vol. 2, JonassenD. H. (ed.), Technical Technology Publications, Englewood Cliffs, New Jersey, pp. 418–435, 1985.
5.
SchumacherG. M. and WallerR., Testing Design Alternatives: A Comparison of Procedures, in Designing Usable Texts, DuffyT. M. and WallerR. (eds.), Academic Press, Orlando, pp. 377–403, 1985.
6.
SchriverK. A., Evaluating Text Quality: The Continuum from Text-Focused to Reader-Focused Methods, IEEE Transactions on Professional Communication, 32:4, pp. 238–255, 1989.
7.
SwaneyJ. H., Editing for Comprehension: Improving the Process through Reading Protocols, in Plain Language: Principles and Practice, SteinbergE. R. (ed.), Wayne State University Press, Detroit, Michigan, pp. 173–203, 1991.
8.
JansenC. and SteehouderM., Forms as a Source of Communication Problems, Journal of Technical Writing and Communication, 22, pp. 179–194, 1992.
9.
de JongM. D. T. and SchellensP. J., Effects of Pre-Testing and Revising Brochures, in Satz—Text—Diskurs: Akten des 27. Linguistischen Kolloquiums, Münster 1992, Band 2, KönigP. P. and WiegersH. (eds.), Max Niemeyer, Tübingen, pp. 189–195, 1994.
10.
de JongM. D. T. and SchellensP. J., Towards a Valid Design for Pretesting and Revising Leaflets, in Functional Communication Quality, van WaesL.WoudstraE., and van den HovenP. (eds.), Rodopi, Atlanta, Georgia, pp. 104–114, 1994.
11.
WrightA. D., The Value of Usability Testing in Document Design, The Bulletin of the Association for Business Communication, LVII, pp. 48–51, 1994.
12.
SchriverK. A., Document Design as Rhetorical Action, inaugural lecture, Utrecht University, 1995.
13.
DieliM., Designing Successful Documents: An Investigation of Document Evaluation Methods, dissertation, Carnegie Mellon University, Pittsburgh, Pennsylvania, 1986.
14.
LentzL. and Pander MaatH., Evaluating Text Quality: Reader-Focused or Text-Focused? in Studies of Functional Text Quality, Pander MaatH. and SteehouderM. (eds.), Rodopi, Atlanta, Georgia, pp. 101–114, 1992.
15.
WrightP. C. and MonkA. F., A Cost-Effective Evaluation Method for Use by Designers, International Journal of Man-Machine Studies, 35, pp. 891–912, 1991.
16.
DesurvireH. W., Faster, Cheaper!! Are Usability Inspection Methods as Effective as Empirical Testing? in Usability Inspection Methods, NielsenJ. and MackR. L. (eds.), John Wiley & Sons, New York, pp. 173–202, 1994.
17.
KaratC. M., A Comparison of User Interface Evaluation Methods, in Usability Inspection Methods, NielsenJ. and MackR. L. (eds.), John Wiley & Sons, New York, pp. 203–233, 1994.
18.
MackR. and MontanizF., Observing, Predicting, and Analyzing Usability Problems, in Usability Inspection Methods, NielsenJ. and MackR. L. (eds.), John Wiley & Sons, New York, pp. 295–339, 1994.
19.
de JongM. D. T. and SchellensP. J., Pretest van de brochure Huursubsidie 1991–1992, Deelrapport 1 van het onderzoeksproject Pretesten (Pretest of the Brochure Rent Subsidy 1991–1992, Report no. 1 of the Research Project Pretesting), University of Twente, Dept. Applied Linguistics, Enschede, 1992.
20.
de HaasW., Research for Government Information Campaigns: Not a Final Judgement but a Management Tool, in Government Information in the Netherlands, KatusJ. and VolmerW. F. (eds.), Koninklijke Vermande, Lelystad, pp. 123–136, 1994.
21.
de JongM. D. T. and SchellensP. J., Betrouwbaarheid en steekproefomvang bij een pretest met de plus-en-minmethode, Deelrapport 14 van het onderzoeksproject Pretesten (Reliability and Sample Size for a Pretest with the Plus-Minus Method, Report no. 14 of the Research Project Pretesting), University of Twente, Dept. Applied Linguistics, Enschede, 1996.
22.
VirziR. A., Refining the Text Phase of Usability Evaluation: How Many Subjects is Enough?Human Factors, 34:4, pp. 457–468, 1992.
23.
WhartonC., The Cognitive Walkthrough Method: A Practitioner's Guide, in Usability Inspection Methods, NielsenJ. and MackR. L. (eds.), John Wiley & Sons, New York, pp. 105–141, 1994.
24.
de JongM. and van der PoortP., Towards a Usability Test Procedure for Technical Documents, in Quality of Technical Documentation, SteehouderM.JansenC.van der PoortP., and VerheijenR. (eds.), Rodopi, Atlanta, Georgia, pp. 229–238, 1994.
25.
de JongM. D. T. and SchellensP. J., Towards a Valid Design for Pretesting and Revising Leaflets, in Functional Communication Quality, van WaesL.WoudstraE., and van den HovenP. (eds.), Rodopi, Atlanta, Georgia, pp. 104–114, 1994.
26.
LentzL. and Pander MaatH., Evaluating Text Quality: Reader-Focused or Text-Focused? in Studies of Functional Text Quality, Pander MaatH. and SteehouderM. (eds.), Rodopi, Atlanta, Georgia, pp. 101–114, 1992.
27.
Pander MaatH. and LentzL., Patient Information Leaflets: A Functional Content Analysis and an Evaluation Study, in Functional Communication Quality, van WaesL.WoudstraE., and van den HovenP. (eds.), Rodopi, Atlanta, Georgia, pp. 137–148, 1994.
