This study examined the effects of training on peer assessment and comments provided regarding oral presentations in EFL (English as a Foreign Language) classrooms. In Study 1, both the treatment and control groups received instruction on skill aspects, but only the treatment group was given an additional 40-minute training on how to rate performances. The results of the correlation difference analyses show no significant differences between the treatment and control groups, but the three instructors are misfitting, presumably because the frame of reference is set by the majority of student data. In the second study, the treatment groups received long training. Again, there are no significant correlation differences between the treatment and control groups; however, all the instructors are not misfitting, which indicates that the frame of reference in the data improved in Study 2. Analyses of comments reveal that the treatment groups are superior in both quality and quantity of comments. Along with a meta-analytic summary, it is argued that peer assessment is a robust system in which instruction on skill aspects may suffice to achieve a certain level of correlation with the criterion variable (instructor), but training may enhance student comments and reduce misfitting raters.
Berg, E.C. (1999). The effects of trained peer response on ESL students' revision types and writing quality. Journal of Second Language Writing, 8, 225-241.
2.
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education, 5, 7-68.
3.
Cheng, W., & Warren, M. (1997). Having second thoughts: Student perceptions before and after a peer assessment exercise. Studies in Higher Education , 22, 233-239.
4.
Cheng, W., & Warren, M. (1999). Peer and teacher assessment of the oral and written tasks of a group project. Assessment & Evaluation in Higher Education, 24, 301-314.
5.
Cheng, W., & Warren, M. (2005). Peer assessment of language proficiency. Language Testing, 22, 93-121.
6.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
7.
Cumming, A. (1990). Expertise in evaluating second language compositions . Language Testing, 7(1), 31-51.
8.
Devenney, R. (1989). How ESL teachers and peers evaluate and respond to student writing. RELC Journal, 20, 77-90.
9.
DiPardo, A., & Freedman, S.W. (1988). Peer response groups in the writing classroom: Theoretic foundations and new directions. Review of Educational Research, 58, 119-149.
10.
Falchikov, N. (1986). Product comparisons and process benefits of collaborative peer group and self assessments. Assessment & Evaluation in Higher Education, 11, 146-165.
11.
Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70, 287-322.
12.
Field, J. (2002). The changing face of listening. In J. C. Richards & W. A. Renandya (Eds.). Methodology in language teaching: An anthology of current practice (pp. 242-247). Cambridge, UK: Cambridge University Press.
13.
Fletcher, C., & Baldry, C. (1999). Multi-source feedback systems: A research perspective . International Review of Industrial and Organizational Psychology , 14, 149-193.
14.
Haaga, D.A. (1993). Peer review of term papers in graduate psychology courses. Teaching of Psychology, 20, 28-32.
15.
Harris, M.M., & Schaubroeck, J. (1988). A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. Personnel Psychology , 41, 43-62.
16.
Hu, G. (2005). Using peer review with Chinese ESL student writers . Language Teaching Research, 9, 321-342.
17.
Jafarpur, A. (1991). Can naive EFL learners estimate their own proficiency ? Evaluation and Research in Education, 5, 145-157.
18.
Liu, J., & Hansen, J.G. (2002). Peer response in second language writing classrooms . Ann Arbor, MI: The University of Michigan Press.
19.
Long, M.H., & Porter, P.A. (1985). Group work, interlanguage talk, and second language learning. TESOL Quarterly, 19, 207-228.
20.
Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19, 246-276.
21.
Lumley, T., & McNamara, T.F. (1995). Rater characteristics and rater bias: Implications for training. Language Testing, 12, 54-71.
22.
McGroarty, M.E., & Zhu, W. (1997). Triangulation in classroom research: A study of peer revision. Language Learning, 47, 1-43.
23.
McNamara, T., & Lumley, T. (1997). The effect of interlocutor and assessment mode variables in overseas assessments of speaking skills in occupational settings . Language Testing, 14(2), 140-156.
24.
Morahan-Martin, J. (1996). Should peers' evaluations be used in class projects? Questions regarding reliability, leniency, and acceptance. Psychological Reports, 78, 1243-1250.
25.
Patri, M. (2002). The influence of peer feedback on self- and peer-assessment of oral skills. Language Testing, 19, 109-131.
26.
Rosenthal, R. (1991). Meta-analytic procedures for social research (2nd ed.). Thousand Oaks, CA: Sage.
27.
Rothschild, D., & Klingenberg, F. (1990). Self and peer evaluation of writing in the interactive ESL classroom: An exploratory study. TESL Canada Journal, 8, 52-65.
28.
Saito, H. (1999). Dependence and interaction in frequency data analysis in SLA research. Studies in Second Language Acquisition, 21, 453-476.
29.
Saito, H., & Fujita, T. (2004). Characteristics and user acceptance of peer rating in EFL writing classrooms. Language Teaching Research, 8, 31-54.
30.
Schoonen, R., Vergeer, M.M., & Eiting, M. (1997). The assessment of writing ability: Expert readers versus lay readers. Language Testing, 14, 157-184.
31.
Shohamy, E., Gordon, C.M., & Kraemer, R. (1992). The effect of raters' background and training on the reliability of direct writing tests. The Modern Language Journal, 76, 27-33.
32.
Smithson, M.J. (n.d).: Scripts and software for noncentral confidence interval and power calculations: noncentral F files for SPSS. Retrieved December 20, 2005, from http://psychology.anu.edu.au/people/smithson/details/CIstuff/CI.html
33.
Somervell, H. (1993). Issues in assessment, enterprise and higher education: The case for self-, peer and collaborative assessment. Assessment & Evaluation in Higher Education, 18, 221-233.
34.
Stanley, J. (1992). Coaching student writers to be effective peer evaluators. Journal of Second Language Writing, 1, 217-233.
35.
Stefani, L.A. (1992). Comparison of collaborative self, peer and tutor assessment in a biochemistry practical. Biochemical Education , 20, 148-151.
36.
Thompson, B. (2002). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31, 25-32.
37.
Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational Research, 68, 249-276.
38.
Tornow, W.W. (1993). Perceptions or reality: Is multi-perspective measurement a means or an end? Human Resource Management, 32, 221-229.
39.
Webb, N.M. (1982). Student interaction and learning in small groups . Review of Educational Research, 52(3), 421-445.
40.
Weigle, S.C. (1994). Effects of training on raters of ESL compositions . Language Testing, 11, 197-223.
41.
Weigle, S.C. (1998). Using FACETS to model rater training effects. Language Testing, 15, 263-287.
42.
Wright, B., & Linacre, J.M. (1994). Reasonable mean-square fit values. Rasch Measurement: Transaction of the Rasch Measurement SIG, 8, 370.
43.
Yamashiro, A. (1999). Using structural equation modeling to validate a rating scale. Presented at the 21st Language Testing Research Colloquium, Tsukuba, Japan.
44.
Yamashiro, A.D., & Johnson, J. (1997). Public speaking in EFL: Elements for course design . The Language Teacher, 21, 13-17.
