An Assessment of Reliability and Validity of a Rubric for Grading APA-Style Introductions

Abstract

This article describes the empirical evaluation of the reliability and validity of a grading rubric for grading APA-style introductions of undergraduate students. Levels of interrater agreement and intrarater agreement were not extremely high but were similar to values reported in the literature for comparably structured rubrics. Rank-order correlations between graders who used the rubric and an experienced instructor who ranked the papers separately and holistically provided evidence for the rubric's validity. Although this rubric has utility as an instructional tool, the data underscore the seemingly unavoidable subjectivity inherent in grading student writing. Instructors are cautioned that merely using an explicit, carefully developed rubric does not guarantee high reliability.

Get full access to this article

View all access options for this article.

References

Andrade

H. G.

(2005). Teaching with rubrics: The good, the bad, and the ugly. College Teaching, 53, 27–30.

Crocker

Algina

(1986). Introduction to classical & modern test theory. Belmont, CA: Wadsworth.

Moskal

B. M.

(2000). Scoring rubrics: What, when and how. Practical Assessment, Research & Evaluation, 7(3). Retrieved April 27, 2007, from http://PAREonline.net/getvn.asp?v=7n=3.

Moskal

B. M.

Leydens

J. A.

(2000). Scoring rubric development: Validity and reliability. Practical Assessment, Research & Evaluation, 7(10). Retrieved July 16, 2007, from http://PAREonline.net/getvn.asp?v=7n=10.

Newell

J. A.

Dahm

K. D.

Newell

H. L.

(2002). Rubric development and interrater reliability issues in assessing learning outcomes. Chemical Engineering Education, 36, 212–215.

Peat

(2006). Integrating writing and research skills: Development and testing of a rubric to measure student outcomes. Journal of Public Affairs Education, 12, 295–311.

Stevens

D. D.

Levi

(2004). Introduction to rubrics: An assessment tool to save grading time, convey feedback, and promote student learning. Sterling, VA: Stylus.

Thaler

Kazemi

Huscher

(2009). Developing a rubric to assess student learning outcomes using a class assignment. Teaching of Psychology, 36, 113–116.

Tinsley

H. E. A.

Weiss

D. J.

(2000). Interrater reliability and agreement. In Tinsley

H. E. A.

Brown

S. D.

(Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 95–124). San Diego, CA: Academic.

10.

Zimmaro

D. M.

(2004). Developing grading rubrics. Retrieved September 29, 2008, from http://www.utexas.edu/academic/mec/research/pdf/rubricshandout.pdf.