Generalizability of writing scores has always been a longstanding concern in L2 writing assessment. A number of studies have been conducted to investigate this topic during the last two decades. However, with the introduction of new test methods, such as reading-to-write tasks, generalizability studies need to focus on the score accuracy of different task types. The current study attempts to examine how reading-to-write tasks are compara ble to independent tasks in terms of score generalizability. To look into this issue, 115 Egyptian university students were asked to write on two independent tasks and two reading-to-write tasks. The essays were holistica lly scored by three raters, who received training prior to scoring. The study design followed a fully crossed univariate design (p × t × r), and consequently GENOVA (Crock & Brennan, 1983) was used to analyze the results. Results showed that the reading-to-write tasks yielded as reliable scores as independent tasks. In addition, the result indicated that score generalizability is very low when using one writing task due to the large (pt) variance component. Implications and limitations of the study are presented.
Allison, D., Berry, V., & Lewkowicz, J. ( 1995). Reading-writing connections in EAP classes: A content analysis of written summaries produced under three mediating conditions. RELC Journal, 26(2), 25—43.
2.
Bachman, L. ( 2004). Statistical analyses for language assessment. Cambridge: Cambridge University Press.
3.
Bachman, L., & Palmer, A. ( 1996). Language testing in practice. Oxford : Oxford University Press.
4.
Brennan, R.L. ( 2001). Generalizability theory. New York : Springer.
5.
Brennan, R.L. ( 2000). Performance assessment from the perspective of generalizability theory. Applied Psychological Measurement, 24(4), 339—353.
6.
Brennan, R., Goa, X., & Colton, D. ( 1995). Generalizability analyses of WorkKeys Listening and Writing tests. Educational and Psychological Measurement, 55(2), 157—176.
7.
Campbell, C. ( 1990).Writing with others’ words: Using background r eading text in academic composition. In B. Kroll (Ed.), Second language w riting: Research insights from the classroom (pp. 211—230). New York: Cambridge University Press.
8.
Candell, G.L., & Ercikan, K. ( 1992). Assessing the reliability of the Maryland school performance assessment program using generalizability theory. The Annual Meeting of theAmerican Educational Research Association, San Francisco.
9.
Cansecon, G., & Byrd, P. ( 1989). Writing required in graduate courses in bu siness administration . TESOL Quarterly, 23(2), 305—316.
10.
Carrell, P. ( 1987). Readability in ESL. Reading in a Foreign Language, 4(1), 21—40.
11.
Carson, J. ( 2001). A task analysis of reading and writing in academic contexts . In D. Belcher & A. Hirvela (Eds.), Linking literacies: Perspectives on L2 reading-writing connections (pp. 48—83). Ann Arbor, MI: The University of Michigan Press.
12.
Cho, Y. ( 2003). Assessing writing: Are we bound by only one method? Assessing Writing, 8, 165—191.
13.
Connor, U., & Kramer, M. ( 1995). Writing from sources: Case studies of graduate students in business management. In D. Belcher & G. Braine (Eds.), Academic writing in a second language: Essays on research and pedagogy (pp. 155—182). Norwood, NJ: Ablex.
14.
Cooper, P.L. ( 1984). The assessment of writing ability: A review of research (GRE Board Research Report No. GREB 82—15R/ETS Research Report no. 84—12) . Princeton, NJ: Educational Testing Service.
15.
Crock, D., & Brennan, R. ( 1983). Manual for GENOVA (ACT Tech Bulletin No43). Iowa City, IA: ACT.
16.
Cumming, A., Kantor, R., Powers, D., Santos, T., & Taylor, C. ( 2000). TOEFL 2000 writing framework: A working paper (TOEFL Monograph Series, Report no. 18). Princeton, NJ: Educational Testing Service.
17.
Cumming, A., Kantor, R., Baba, K., Erdosy, U., Eouanzoui, K., & James, M. ( 2005). Differences in written discourse in independent and integrated prototype tasks for next generation TOEFL. Assessing Writing , 10, 5—43.
18.
Currie, P. ( 1998). Staying out of trouble: Apparent plagiarism and academic survival. Journal of Second Language Writing, 7(1), l—18.
19.
Educational Testing Services. (2005). Test of English as a foreign language. Retrieved September 8, 2005, from www.toefl.org.
20.
Eblen, C. ( 1983). Writing across the curriculum: A survey of a university faculty’s views and classroom practices. Research on the Teaching of English, 17, 343—349.
21.
Feldt, L.S., & Brennan, R.L. ( 1989). Reliability. In R. L. Linn (Ed.), Educational measurement, 3rd ed. (pp. 105—146). New York: Macmillan .
22.
Ferrara, S. ( 1993). Generalizability theory and scaling: Their roles in writing assessment and implications for performance assessments in other content areas. The Annual Meeting of the National Council on Measurement in Education, Atlanta.
23.
Gao, X., & Brennan, R. ( 2001). Variability of estimated variance components and related statistics in a performance assessment. Applied Measurement in Education, 14(2), 191—203.
24.
Gebril, A. ( 2006). Indpendent and Integrated academic writing tasks: A study in generalizability and test method. Unpublished doctoral dissertation, The University of Iowa, Iowa City.
25.
Gebril, A. ( 2009). Score generalizability in writing assessment: The interface between applied linguistics and psychometrics research. Saarbrücken, Germany: VDM Verlag Dr. Müller .
26.
Gebril, A., & Plakans, L. ( 2009). Investigating source use, discourse features, and process in integrated writing tests. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 7, 47—84.
27.
Grabe, W. ( 2001). Reading-writing relations: Theoretical perspectives and instructional practices. In D. Belcher & A. Hirvela (Eds.), Linking l iteracies: Perspectives on L2 reading-writing connections (pp. 15—47). Ann Arbor, MI: University of Michigan Press.
28.
Hale, G., Taylor, C., Bridgeman, J., Carson, J., Kroll, B., & Kantor, R. ( 1996). A study of writing tasks assigned in academic degree programs. Princeton, NJ: Educational Testing Service.
29.
Hamp-Lyons, L. ( 1991). Scoring procedures for ESL contexts. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 241—276). Norwood, NJ: Ablex.
30.
Hamp-Lyons, L., & Kroll, B. ( 1996). Issues in ESL writing assessment: An overview. College ESL, 6(1), 52—72.
31.
Hayes, J.R. ( 1996). A new model of cognition and affect in writing. In C. M. Levy & S. Ransdell (Eds.), The science of writing (pp. 1—30). Hillsdale, NJ: Erlbaum.
32.
Horowitz, D. ( 1986). Essay examination prompts and the teaching of academic writing. English for Specific Purposes, 5(2), 107—120.
33.
Huot, B.A. ( 1990). The literature of direct writing assessment: Major c oncerns and prevailing trends. Review of Educational Research , 60, 237—263.
34.
Lane, S., & Sabers, D. ( 1989). Use of generalizability theory for e stimating the dependability of a scoring system for sample essays. Applied Measurement in Education, 2(3), 195—205.
35.
Lee, Y., & Kantor, R. ( 2005). Dependability of new ESL writing test scores: Tasks and alternative rating schemes (TOEFL Monograph Series no. 1). Princeton, NJ: ETS.
36.
Leki, I., & Carson, J. ( 1997). Completely different worlds: EAP and the writing experiences of ESL students in university courses. TESOL Quarterly, 31, 39—69.
37.
Lehmann, R.H. ( 1987). Reliability and generalizabilitv of compositions. The Annual Meeting of theAmerican Educational Research Association , Washington, DC.
38.
Lewkowicz, J. ( 1997). The integrated testing of a second language. In C. Calpham (Ed.), Encyclopedia of language and education, Vol. 7: Language testing and assessment (pp. 121—130). Dordrecht, The Netherlands: Kluwer Academic Publishers .
39.
Lumley, T. ( 2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19(3), 246—276.
40.
Matalene, C. ( 1985). Contrastive rhetoric: An American writing teacher in China. College English, 47(8), 789—808.
41.
Messick, S. ( 1989). Validity. In R. L. Linn (Ed.), Educational measurement, 3rd ed. (pp. 13—103). New York: Macmillan .
42.
Messick, S. ( 1996). Validity and washback in language testing. Language Testing, 13(3), 241—256.
43.
Moore, T., & Morton, J. ( 1999). Authenticity in the IELTS academic module writing test: A comparative study of task 2 items and university assignments. IELTS Research Reports no. 2. Canberra: IELTS Australia.
44.
Parkes, J., Zimmaro, D., Zappe, S., & Suen, H. ( 2000). Reducing task-related variance in performance assessment using concept maps. Educational Research and Evaluation, 6(4), 357—378.
45.
Pecorari, D. ( 2003) Good and original: plagiarism and patchwriting in academic second-language writing. Journal of Second Language Writing , 12, 317—345.
46.
Pennycook, A. ( 1996). Borrowing others’ word: Text, ownership, memory, and plagiarism. TESOL Quarterly, 30(2), 201—230.
47.
Plakans, L. ( 2007). Second language writing and reading-to-write a ssessment tasks: A process study. Unpublished Doctoral Dissertation, The University of Iowa, Iowa, City: IA.
48.
Schoonen, R. ( 2005). Generalizability of writing scores: An application of structural equation modeling. Language Testing, 22(1), 1—30.
49.
Shavelson, R., Baxter, G., & Goa, X. ( 1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30(3), 215—232.
50.
Shohamy, E., Gordon, C., & Kraemer, R. ( 1992). The effect of rater background and training on the reliability of direct writing tests. The Modern Language Journal, 76(1), 28—33.
51.
Song, B., & Caruso, I. ( 1996). Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL Students? Journal of Second Language Writing, 5(2), 163—182.
52.
Spack, R. ( 1993). Student meets text, text meets student: Finding a way into academic discourse. In J. Carson & I. Leki (Eds.), Reading in the composition classroom: Second language perspectives (pp. 183—196). Boston, MA: Heinle & Heinle.
53.
Stuhlmann, J., Daniel, K., Dellinger, A., Denny, R., & Powers, T. ( 1999). A generalizability study of the effects of training on teachers’ ability to rate children’s writing using a rubric. Journal of Reading Psychology, 20, 107—127.
54.
Swartz, C.W., Hooper, S.R., Montgomery, J.W., Wakely, M.B., DeKruif, R.E.L., Reed, M. et al. (1999). Using generalizability theory to estimate the reliability of writing scores derived from holistic and a nalytical scoring methods. Educational and Psychological Measurement, 59(3), 492—506.
55.
Tierney, R.J., & Shanahan, T. ( 1991). Research on the reading-writing relationship: Interactions, transactions, and outcomes. In R. Barr , M. Kamil, P. B. Mosenthal, & P. D. Pearson (Eds.), Handbook of reading research, Vol. 2 (pp. 246—280). New York : Longman.
56.
Tierney, R.J., Soter, A., O’Flahavan, J.F., & McGinley, W. ( 1989). The effects of reading and writing upon thinking critically . Reading Research Quarterly, 24, 134—173.
57.
Watanabe, Y. ( 2001). Read-to-write tasks for the assessment of second language academic writing skills: Investigating text features and rater reaction. Unpublished doctoral dissertation, University of Hawaii, Manoa.
58.
Weigle, S.C. ( 1994). Effects of training on raters of ESL compositions. Language Testing, 11(2), 197—223.
59.
Weigle, S.C.( 1998). Using FACETS to model rater training effects. Language Testing, 15(2), 263—287.
60.
Weigle, S.C. ( 2002). Assessing writing. Cambridge: Cambridge University Press.
61.
Weigle, S.C. ( 2004). Integrating reading and writing in a competency test for non-native speakers of English. Assessing Writing, 9, 27—55.
