In many literacy studies, it is important to establish the reliability of independent observers' judgments. Reliability most commonly is measured either by the percentage of agreement or the correlation between the observers' judgments. This article argues that the percentage of agreement measure is more difficult to interpret than are correlation measures because of the following: (a) the effects of chance agreement are not accounted for automatically by the percentage of agreement measure; and (b) rates of chance agreement are strongly influenced by the variability of the data, by “ceiling” and “floor” effects, and by the scoring of near agreement as perfect agreement. For these reasons, the authors recommend that the field of literacy research adopt correlation as the standard method for estimating the reliability of observers' judgments.
Get full access to this article
View all access options for this article.
References
1.
Allen, M. S.
(1995). Valuing differences: Portnet's first year. Assessing Writing, 2, 67-89.
2.
Almasi, J. F.
(1995). The nature of fourth graders' sociocognitive conflicts in peer-led and teacher-led discussions of literature. Reading Research Quarterly, 30, 314-351.
3.
Boice, B.
(1997). Which is more productive, writing in binge patterns of creative illness or in moderation?Written Communication, 14, 435-459.
4.
Chambliss, M. J.
(1995). Text cues and strategies successful readers use to construct the gist of lengthy written arguments. Reading Research Quarterly, 30, 778-807.
5.
Chapman, M. L.
(1994). The emergence of genres: Some findings from an examination of first-grade writing. Written Communication, 11, 348-380.
6.
Crammond, J. G.
(1998). The uses and complexity of argument structures in expert and student persuasive writing. Written Communication, 14, 230-268.
7.
Deegan, D.
(1995). Exploring individual differences among novices reading in a specific domain: The case of law. Reading Research Quarterly, 30, 154-170.
8.
Dole, J. A.
, Brown, K. J., & Trathen, W. (1996). The effects of strategy instruction on the comprehension of at-risk students. Reading Research Quarterly, 31, 62-88.
9.
Faber, B.
(1996). Rhetoric in competition. Written Communication, 13, 355-384.
10.
Gearhart, M.
, Herman, J. L., Novak, J. R., & Wolf, S. A. (1995). Toward the instructional utility of large-scale assessment: Validation of a new narrative rubric. Assessing Writing, 2, 207-242.
11.
Glass, G. V.
, & Hopkins, K. D. (1984). Statistical methods in education and psychology. Englewood Cliffs, NJ: Prentice Hall.
12.
Guthrie, J. T.
, Van Meter, P., McCann, A. D., Wigfield, A., Bennett, L., Poundstone, C. C., Rice, M. E., Faibisch, F. M., Hunt, B., & Mitchell, A. M. (1996). Growth in literacy engagement: Change in motivation and strategies during concept-oriented reading instruction. Reading Research Quarterly, 31, 306-332.
13.
Hartman, D.
(1995). Eight readers reading: The intertextual links of proficient readers reading multiple passages. Reading Research Quarterly, 30, 520-561.
14.
Haswell, R. H.
, & Haswell, J. T. (1996). Gender bias and critique of student writing. Assessing Writing, 3, 31-83.
15.
Johnson, T. M.
, Jones, G. A., Thornton, C. A., Langrall, C. W., & Rous, A. (1998). Students' thinking and writing in the context of probability. Written Communication, 15, 203-229.
16.
Mathison, M. A.
(1996). Writing the critique, a text about a text. Written Communication, 13, 314-354.
17.
Newell, G. E.
(1994). The effects of written between-draft responses on students' writing and reasoning about literature. Written Communication, 11, 311-347.
18.
Nystrand, M.
, Cohen, A. S., & Dowling, N. (1993). Addressing reliability problems in the portfolio assessment of college writing. Educational Assessment, 1, 53-70.
19.
Purnell-Gates, V.
(1996). Stories, coupons, and the TV Guide: Relationships between home literacy experiences and emergent literacy knowledge. Reading Research Quarterly, 31, 406-428.
20.
Schraw, G.
, & Bruning, R. (1996). Readers' implicit models of reading. Reading Research Quarterly, 31, 290-305.
21.
Smolkin, L. B.
(1997). Dealing with dialogue: Fifth graders' responses to reading a play. Research in the Teaching of English, 31, 240-266.
22.
Soliday, M.
, & Gleason, B. (1997). From remediation to enrichment: Evaluating a mainstreaming project. Journal of Basic Writing, 16(1), 64-78.
23.
Sperling, M.
(1994). Constructing the perspective of teacher-as-reader: A framework for studying response to student writing. Research in the Teaching of English, 28, 175-207.
24.
Sperling, M.
, & Woodlief, L. (1997). Two classrooms, two writing communities: Urban and suburban tenth-graders learning to write. Research in the Teaching of English, 31, 205-239.
25.
Stahl, S. A.
, Hynd, C. R., Britton, B. K., McNish, M. M., & Basquet, D. (1996). What happens when students read multiple source documents in history?Reading Research Quarterly, 31, 430-456.
26.
Stolarek, E. A.
(1994). Prose modeling and metacognition: The effect of modeling on developing a metacognitive stance toward writing. Research in the Teaching of English, 28, 154-174.
27.
Taylor, C. S.
(1997). Using portfolios to teach teachers about assessment: How to survive. Educational Assessment, 4, 123-147.
28.
Turner, J. C.
(1995). The influence of classroom contexts on young children's motivation for literacy. Reading Research Quarterly, 30, 410-441.
29.
van Gelderen, A.
(1997). Elementary students' skills in revising. Written Communication, 14, 360-397.
30.
Venezky, R. L.
, Bristow, P. S., & Sabatini, J. P. (1994). Meaning change in adult literacy programs: Enduring issues and a few answers. Educational Assessment, 2, 101-131.
31.
Wollman-Bonilla, J. E.
(1994). Why don't they “just speak?” Attempting literature discussion with more and less able readers. Research in the Teaching of English, 28, 231-258.
32.
Yageliski, R. P.
(1995). The role of classroom context in the revision strategies of student writers. Research in the Teaching of English, 29, 216-238.
33.
Zhu, W.
(1995). Effects of training for peer response on students' comments and interaction. Written Communication, 12, 492-528.
