The purpose of the present study was to investigate convergent and divergent validity of inferences from selected measures of struture, coherence, and quality for compositions. Thirty sixth-graders wrote two stories each. Each story was scored six times, for each of three constructs (structure, coherence, and quality), using two ways of measuring (reader- and text-based). Patterns in correlation matrices were examined for the extent to which they conformed to expectations which would support convergent and divergent validity. The major conclusions of the study were: (a) Overall, the findings strongly supported validity of inferences from the structure measures, called into serious question the validity of inferences from the coherence measures, and weakly supported it for inferences from the quality measures; (b) some evidence was provided for the distinctiveness of the three constructs, structure, coherence, and quality; and (c) some method/halo bias emerged in the reader-based method of assessment.
References
1.
ArmstrongR. D. (1965). An objective measure of the quality of the written composition of fifth grade pupils (Doctoral dissertation, University of California, Berkeley, 1965). Dissertation Abstract International, 26, 7174.
2.
AtwellM. A. (1981). The evolution of text: The interrelationship of reading and writing in the composing process. Paper presented at the meeting of the National Council of Teachers of English, Boston. (ERIC Document Reproduction Service No. ED 211 993).
3.
BambergB. (1983). What makes a text coherent. College Composition and Communication, 34, 417–429.
4.
BambergB. (1984). Assessing coherence: A reanalysis of essays written for the National Assessment of Educational Progress, 1969–1979. Research in the Teaching of English, 18, 305–319.
5.
BeckI. L.McKeownM. G.OmansonR. G.PopleM. T. (1984). Improving the comprehensibility of stories: The effects of revisions that improve coherence. Reading Research Quarterly, 19, 263–277.
6.
BrelandH.JonesR. J. (1984). Perceptions of writing skills. Written Communication, 1, 101–109.
7.
BrittonJ.BurgessT.MartinM.McLeodA.RosenH. (1975). The development of writing abilities: 11–18. London: Macmillan.
8.
CampbellD. T.FiskeD. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105.
9.
ConlanG. (1986). “Objective” measures of writing ability. In GreenbergK. L.WienerH. S.DonovanR. A. (Eds.), Writing assessment: Issues and strategies (pp. 109–125). New York: Longman.
10.
CooperC. R. (1977). Holistic evaluation of writing. In CooperC. R.OdellL.Evaluating writing: Describing, measuring, judging (pp. 3–31). Urbana, IL: National Council of Teachers of English.
11.
CoxB. E.ShanahanT.TinzmannM. B. (1991). Children's knowledge of organization, cohesion, and voice in written exposition. Research in the Teaching of English, 25, 179–218.
12.
CTB/McGraw-Hill. (1981). Test of Cognitive Skills. Monterey, CA: CTB/McGraw-Hill.
13.
CTB/McGraw-Hill. (1986). California Achievement Test. Monterey, CA: CTB/McGraw-Hill.
14.
FitzgeraldJ. (1992). Towards knowledge in writing. New York: Springer-Verlag.
15.
FitzgeraldJ.SpiegelD. L. (1983). Enhancing children's reading comprehension through instruction in narrative structure. Journal of Reading Behavior, 15, 1–17.
16.
FitzgeraldJ.SpiegelD. L. (1986). Textual cohesion and coherence in children's writing. Research in the Teaching of English, 20, 263–280.
17.
FitzgeraldJ.TeasleyA. (1986). Effects of instruction in narrative structure on children's writing. Journal of Educational Psychology, 78, 424–433.
18.
FreedmanS. W. (1979a). How characteristics of student essays influence teachers' evaluation. Journal of Eductional Psychology, 71, 328–338.
19.
FreedmanS. W. (1979b). Why do teachers give the grades they do. College Composition and Communication, 30, 161–164.
20.
GereA. R. (1980). Written composition: Toward a theory of evaluation. College English, 42, 44–58.
21.
GergenK. J. (1990). From construction in context to reconstruction in education. Paper presented at the seminar series, “Constructivism and Education,” Athens, GA.
22.
GodshalkF. I.SwinefordF.CoffmanW. E. (1966). The measurement of writing ability. New York: College Entrance Examination Board.
23.
GreenbergK. L.WienerH. S.DonovanR. A. (1986). Writing assessment: Issues and strategies. New York: Longman.
24.
GrobeC. (1981). Syntactic maturity, mechanics, and vocabulary as predictors of quality ratings. Research in the Teaching of English, 15, 75–85.
25.
HarrisW. H. (1977). Teacher response to student writing: A study of the response patterns of high school English teachers to determine the basis for teacher judgment of student writing. Research in the Teaching of English, 11, 175–185.
26.
HasanR. (1984). Coherence and cohesive harmony. In FloodJ. (Ed.), Understanding reading comprehension (pp. 181–263). Newark, DE: International Reading Association.
27.
HillegasM. B. (1912). Scales for the measurement in English composition by young people. Teachers College Record, 13, 331–384.
28.
HopkinsK. D.StanleyJ. C.HopkinsB. R. (1990). Educational and psychological measurement and evaluation. Englewood Cliffs, NJ: Prentice-Hall.
29.
HudelsonE. (1923). English composition: Its aims, methods and measurements. In WhippleG. M. (Ed.), The twenty-second yearbook of the National Society for the Study of Education, Part One (pp. 1–100). Bloomington, IL: Public School Publisher.
30.
HuotB. (1990a). The literature of direct writing assessment: Major concerns and prevailing trends. Review of Educational Research, 60, 237–263.
31.
HuotB. (1990b). Reliability, validity, and holistic scoring: What we know and what we need to know. College Composition and Communication, 41, 201–213.
32.
HuxtableZ. (1929). Criteria for judging thought content in written English. Journal of Educational Research, 19, 188–195.
33.
JohnstonP. (1987). The process of assessment, in the Language Arts. In SquireJ. R. (Ed.), The dynamics of language learning research in Reading and English (pp. 335–357). Urbana, IL: National Council on Research in English and ERIC Clearinghouse on Reading and Communication Skills.
34.
KonoldC.JohnsonD. K. (1991). Philosophical and psychological aspects of contructivism. In SteffeL. P. (Ed.), Epistemological foundations of mathematical experience. New York: Springer-Verlag.
35.
KucerS. B. (1983). Text coherence from a transactional perspective. In NilesJ. A.HarrisL. A. (Eds.), Searches for meaning in reading/language processing and instruction. Thirty-second Yearbook of the National Reading Conference (pp. 104–110). Rochester, NY: National Reading Conference.
36.
Levi-StraussC. (1955). The structural study of myth. Journal of American Folklore, 68, 428–444.
37.
LomaxR. G.AlginaJ. (1979). Comparison of two procedures for analyzing multitrait multimethod matrices. Journal of Educational Measurement, 16, 177–186.
38.
MandlerJ. M.JohnsonN. S. (1977). Remembrance of things parsed: Story structure and recall. Cognitive Psychology, 9, 111–151.
39.
MarshH. W.HocevarD. (1983). Confirmatory factor analysis of multitrait-multimethod matrices. Journal of Educational Measurement, 20, 231–248.
40.
McCollyW. (1970). What does educational research say about the judging of writing. Journal of Educational Research, 64, 148–56.
41.
MessickS. (1989). Validity. In LinnR. L. (Ed.), Educational measurement (pp. 13–103). New York: American Council on Education and Macmillan.
42.
MeyerB. J. F. (1975). The organization of prose and its effect of memory. Amsterdam: North-Holland.
43.
MoeA. J. (1979). Cohesion, coherence, and the comprehension of text. Journal of Reading, 23, 16–20.
44.
MossP. A. (1992). Shifting conceptions of validity in educational measurement: Implications for performance assessment. Review of Educational Research, 62, 229–258.
45.
NeilsenL.PicheG. (1981). The influence of headed nominal complexity and lexical choice on teachers' evaluation of writing. Research in the Teaching of English, 15, 65–74.
46.
NystrandM. (1986). The structure of written communication: Studies in reciprocity between writers and readers. Orlando, FL: Academic Press.
47.
NystrandM. (1989). A social-interactive model of writing. Written Communication, 6, 66–85.
48.
PhelpsL. (1985). Dialectics of coherence: Toward an integrative theory. College English, 47, 12–29.
49.
ProppV. (1968). Morphology of the folktale (2nd ed.). (ScottL., Trans.). Baltimore, MD: Port City Press. (Original work published 1928).
50.
RentelV. M.KingM. L.PettegrewB.PappasC. (1984). A longitudinal study of coherence in children's written narratives, March 31, 1981–March 31, 1982. (ERIC Document Reproduction Service No. ED 237 989).
51.
RohmanG. (1965). Pre-writing: The stage of discovery in the writing process. College Composition and Communication, 29, 209–211.
52.
RuthL.MurphyS. (1988). Designing writing tasks for the assessment of writing. Norwood, NJ: Ablex.
SchonellF. J. (1935). Diagnostic tests in English. In UsillH. V. (Ed.), The year book of education (pp. 503–508). London: University of London, Institute of Education and Evans Brothers Ltd.
55.
SpiegelD. L.FitzgeraldJ. (1990). Textual cohesion and coherence in children's writing revisited. Research in the Teaching of English, 24, 48–66.
56.
Standards for Educational and Psychological Tests. (1974). Washington, DC: American Psychological Association.
57.
SteinN. L.GlennC. (1979). An analysis of story comprehension in elementary school children. In FreedleR. O. (Ed.), New directions in discourse processing (pp. 53–120. Norwood, NJ: Ablex.
58.
StewartM. R.GrobeC. H. (1979). Syntactic maturity, mechanics, and vocabulary and teachers' quality ratings. Research in the Teaching of English, 13, 207–215.
59.
StrawS. B. (1981). Assessment and evaluation in written composition: A commonsense perspective. In FroeseV.StrawS. B. (Eds.), Research in the language arts: Language and schooling (pp. 181–202). Baltimore, MD: National Council of Teachers of English.
60.
TiemeyR. J.MosenthalJ. H. (1983). Cohesion and textual coherence. Research in the Teaching of English, 17, 215–229.
61.
van DijkT. A. (1980). Macrostructures: An interdisciplinary study of the global structure in discourse, interaction, and cognition. Hillsdale, NJ: Erlbaum.
62.
van DijkT. A.KintschW. (1983). Strategies of discourse comprehension. New York: Academic Press.
63.
VaughanC. (1987, March). What affects raters' judgments? Paper presented at the meeting of the Conference on College Composition and Communication, Atlanta.
64.
VealL. R.HudsonS. A. (1983). Direct and indirect measures for large-scale evaluation of writing. Research in the Teaching of English, 17, 290–296.
65.
WhaleyJ. F. (1981), Readers' expectations for story structure. Reading Research Quarterly, 16, 90–114.
66.
WhiteE. M. (1985). Teaching and assessing writing. San Francisco: Jossey-Bass.
67.
WhiteE. M. (1990). Language and reality in writing assessment. College Composition and Communication, 41, 187–200.
68.
WilliamsJ. D. (1985). Coherence and cognitive style. Written Communication, 2, 473–491.
69.
WinterowdR. (1970). The grammar of coherence. College English, 31, 828–835.
70.
WitteS. P.FaigleyL. (1981), Coherence, cohesion, and writing quality. College Composition and Communication, 32, 189–204.
71.
WitteS. P.TrachselM.WaltersK. (1986). Literacy and the direct assessment of writing: A diachronic perspective. In GreenbergK. L.WienerH. S.DonovanR. A. (Eds.), Writing assessment: Issues and strategies (pp. 13–34). White Plains, NY: Longman.
