Multifaceted Rasch measurement was used in the present study with 91 student and 4 teacher raters to investigate how self- and peer-assessments work in comparison with teacher assessments in actual university writing classes.
The results indicated that many self-raters assessed their own writing lower than predicted. This was particularly true for high-achieving students. Peer-raters were the most lenient raters; however, they rated high-achieving writers lower and low-achieving writers higher. This tendency was independent of their own writing abilities and therefore offered no support for the hypothesis that high-achieving writers rated severely and low-achieving writers rated leniently. On the other hand, most peer-raters were internally consistent and produced fewer bias interactions than self- and teacher-raters. Each of the four teachers was internally consistent; however, each displayed a unique bias pattern. Self-, peer-, and teacher-raters assessed Grammar severely and Spelling leniently. The analysis also revealed that teacher-raters assessed Spelling, Format, and Punctuation differently from the other criteria.
It was concluded that self-assessment was somewhat idiosyncratic and therefore of limited utility as a part of formal assessment. Peer-assessors on the other hand were shown to be internally consistent and their rating patterns were not dependent on their own writing performance. They also produced relatively few bias interactions. These results suggest that in at least some contexts, peer-assessments can play a useful role in writing classes. By using multifaceted Rasch measurement, teachers can inform peer-raters of their bias patterns and help them develop better quality assessment criteria, two steps that might lead to better quality peer-assessment.
Bachman, L.F., & Palmer, A.S. (1989). The construct validation of self ratings of communicative language ability. Language Testing, 6(1), 14-29.
2.
Bailey, K.M. (1998). Learning about language assessment. Cambridge, MA: Heinle & Heinle.
3.
Ballantyne, R., Hughes, K., & Mylonas, A. (2002). Developing procedures for implementing peer assessment in large classes using an action research process. Assessment & Evaluation in Higher Education, 27(5), 427-441.
4.
Brown, J.D., Hudson, T., Norris, J., & Bonk, W. (2002). An investigation of second language task-based performance assessments. Honolulu: University of Hawai'i.
5.
Bonk, W., & Ockey, G.J. (2003). A many-facet Rasch analysis of the second language group oral discussion. Language Testing, 20(1), 89-110.
6.
Caulk, N. (1994). Comparing teacher and student responses to written work. TESOL Quarterly, 28(1), 181-188.
7.
Cheng, W., & Warren, M. (1997). Having second thoughts: Student perceptions before and after a peer assessment exercise. Studies in Higher Education , 22(2), 233-240.
8.
Cheng, W., & Warren, M. (2005). Peer assessment of language proficiency. Language Testing, 22(1), 93-121.
9.
Conway, R., & Kember, D. (1993). Peer assessment of an individual's contribution to a group project. Assessment & Evaluation in Higher Education, 18(1), 45-54.
10.
Evans, A.W., Aghabeigi, B., Leeson, R., O'Sullivan, C., & Eliahoo, J. (2002). Are we really as good as we think we are?Annals of the Royal College of Surgeons of England, 84(1), 54-56.
11.
Evans, A.W., McKenna, C., & Oliver, M. (2002). Self-assessment in medical practice. Journal of the Royal Society of Medicine, 95(10), 511-513.
12.
Freeman, M. (1995). Peer assessment by groups of group work. Assessment & Evaluation in Higher Education, 20(3), 289-301.
13.
Goldfinch, J.M. (1994). Further developments in peer assessment of group projects. Assessment and Evaluation in Higher Education, 19(1), 29-35.
14.
Goldfinch, J.M., & Raeside, R. (1990). Development of a peer assessment technique for obtaining individual marks on a group project. Assessment & Evaluation in Higher Education, 15(3), 210-225.
15.
Hamp-Lyons, L. (1991). Scoring procedures for ESL contexts. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 241-276). Norwood, NJ: Ablex.
16.
Hanrahan, S., & Isaacs, G. (2001). Assessing self- and peer assessment: The students' views. Higher Education Research and Development, 20(1), 53-70.
17.
Hargreaves, A., Earl, L., & Schimidt, M. (2001). Perspectives on alternative assessment reform . American Educational Research Journal, 39(1), 69-95.
18.
Hughes, I.E., & Large, B.J. (1993). Staff and peer-group assessment of oral communication skills. Studies in Higher Education, 18(3), 379-385.
19.
Ikeno, O. (2002). The Japanese mind: Understanding contemporary culture. NorthClarendon: Tuttle .
Kondo-Brown, K. (2002). A FACETS analysis of rater bias in measuring Japanese second language writing performance. Language Testing, 19(1), 3-31.
22.
Kozaki, Y. (2004). Using GENOVA and FACETS to set multiple standards on performance assessment for certification in medical translation from Japanese into English. Language Testing, 21(1), 1-27.
23.
Kwan, K., & Leung, R. (1996). Tutor versus peer group assessment of student performance in a simulation training exercise. Assessment & Evaluation in Higher Education, 21(3), 205-215.
24.
Linacre, J.M. (1989/1994). Many-facet Rasch measurement . Chicago, IL: Institute for Objective Measurement.
25.
Linacare, J.M. (1999). FACETS: computer program for many faceted Rasch measurement (version 3.22). Chicago, IL: Mesa Press.
26.
Linacre, J.M. (2002a). What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions, 16(2), 878.
Linacre, J.M. (2002c). Construction of measures from many-facet data . Journal of Applied Measurement, 3(4), 486-512.
29.
Linacre, J.M., & Williams, J. (1998). How much is enough?Rasch Measurement Transactions, 12(3), 653.
30.
Lumley, T., & McNamara, T.F. (1995). Rater characteristics and rater bias: Implications for training. Language Testing, 12(1), 54-71.
31.
Lunz, M.E. (1992). New ways of thinking about reliability. Professions Education Researcher Quarterly, 13(4), 16-18.
32.
Lunz, M.J., & Linacre, J.M. (1998). Measurement designs using multifacet Rasch modeling . In G. A. Marcoulides (Ed.), Modern methods for business research (pp. 44-47). Mahwah, NJ: Lawrence Erlbaum.
33.
Lunz, M.E., Stahl, J.A., & Wright, B.D. (1994). Interjudge reliability and decision reproducibility . Educational and Psychological Measurement, 54(4), 914-925.
34.
Mangelsdorf, K. (1992). Peer reviews in the ESL composition classroom: What do the students think? ELT Journal, 46(3), 274-283.
35.
McNamara, T.F. (1996). Measuring second language performance. Harlow: Addison Wesley Longman.
36.
McQueen, J., & Congdon, P. (1997). Rater severity in large-scale assessment: Is it invariant? (ERIC Document Reproduction Service No. ED411303)
Nakamura, Y. (2002). Teacher assessment and peer assessment. (ERIC Document Reproduction Service No. ED464483)
39.
Oldfield, K.A., & Macalpine, J.M.K. (1995). Peer and self-assessment at tertiary level: An experiential report. Assessment & Evaluation in Higher Education, 20(1), 125-132.
40.
Orsmond, P., & Merry, S. (1997). A study in self-assessment: Tutor and students' perceptions of performance criteria. Assessment & Evaluation in Higher Education, 22(4), 357-370.
41.
Orsmond, P., Merry, S., & Reiling, K. (2000). The use of student derived marking criteria in peer and self-assessment. Assessment and Evaluation in Higher Education, 25(1), 23-38.
42.
Patri, M. (2002). The influence of peer feedback on self- and peer-assessment of oral skills. Language Testing, 19(2), 109-131.
43.
Perkins, K. (1983). On the use of composition scoring techniques, objective measures, and objective tests to evaluate ESL writing ability. TESOL Quarterly, 17(4), 651-671.
44.
Pope, N.K.L. (2005). The impact of stress in self- and peer assessment . Studies in Higher Education, 30(1), 51-63.
45.
Ross, S. (1998). Self-assessment in second language testing: A meta-analysis and analysis of experiential factors. Language Testing, 15(1), 1-20.
46.
Rudy, D.W., Fejfar, M.C., Griffith, C.H. III. , & Wilson, J.F. (2001). Self-and peer-assessment in a first-year communication and interviewing course. Evaluation and the Health Professions, 24(4), 436-445.
47.
Saito, H., & Fujita, T. (2004). Characteristics and user acceptance of peer rating in EFL writing classrooms. Language Teaching Research, 8(1), 31-54.
48.
Sluijsmans, D.M.A., Brand-Gruwel, S., & Marriënboer, G.V. (2002). Peer assessment training in teacher education: Effects on performance and perceptions. Assessment & Evaluation in Higher Education, 27(5), 443-454.
49.
Sullivan, K., & Hall, C. (1997). Introducing students to self-assessment. Assessment & Evaluation in Higher Education, 22(3), 289-306.
50.
Swanson, D., Case, S., & Van der Vleuten, C. (1991). Strategies for student assessment. In D. Boud & G. Feletti (Eds.), The challenge of problem based learning (pp. 269-282). London: Kogan Page .
51.
Swanson, D., Case, S., & Van der Vleuten, C. (1997). Strategies for student assessment. In D. Boud & G. Feletti (Eds.), The challenge of problem based learning (pp. 269-282). London: Kogan Page .
52.
Takada, N., & Lampkin, R. (1996). The Japanese way: Aspects of behavior, attitudes and customs of the Japanese. New York: McGraw-Hill.
53.
Taras, M. (2001). The use of tutor feedback and student self-assessment in summative assessment tasks: Towards transparency for students and for tutors . Assessment & Evaluation in Higher Education, 26(6), 289-306.
54.
Topping, K.J., Smith, E.F., Swanson., I., & Elliot, A. (2000). Formative peer assessment of academic writing between postgraduate students. Assessment & Evaluation in Higher Education, 25(2), 149-169.
55.
Wigglesworth, G. (1993). Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction. Language Testing, 10(3), 305-335.
56.
Williams, E. (1992). Student attitudes towards approaches to learning and assessment. Assessment & Evaluation in Higher Education , 17(1), 45-59.
57.
Yang, Wen-Ling. (2000). Analysis of item ratings for ensuring the procedural validity of the 1998 NAEP achievement-levels setting. (ERIC Document Reproduction Service No. ED440136)
