Development and Initial Validation of the Early Elementary Writing Rubric to Inform Instruction for Kindergarten and First-Grade Students

Abstract

This article describes the development of the Early Elementary Writing Rubric (EEWR), an analytic assessment designed to measure kindergarten and first-grade writing and inform educators’ instruction. Crocker and Algina’s (1986) approach to instrument development and validation was used as a guide to create and refine the writing measure. Study 1 describes the development of the 10-item measure (response scale ranges from 0 = Beginning of Kindergarten to 5 = End of First Grade). Educators participated in focus groups, expert panel review, cognitive interviews, and pretesting as part of the instrument development process. Study 2 evaluates measurement quality in terms of score reliability and validity. Data from writing samples produced by 634 students in kindergarten and first-grade classrooms were collected during pilot testing. An exploratory factor analysis was conducted to evaluate the psychometric properties of the EEWR. A one-factor model fit the data for all writing genres and all scoring elements were retained with loadings ranging from 0.49 to 0.92. Internal consistency reliability was high and ranged from .89 to .91. Interrater reliability between the researcher and participants varied from poor to good and means ranged from 52% to 72%. First-grade students received higher scores than kindergartners on all 10 scoring elements. The EEWR holds promise as an acceptable, useful, and psychometrically sound measure of early writing. Further iterative development is needed to fully investigate its ability to accurately identify the present level of student performance and to determine sensitivity to developmental and instruction gains.

Keywords

writing assessment kindergarten first grade

Get full access to this article

View all access options for this article.

References

Allen

A. A.

Jung

P. G.

Poch

A. L.

Brandes

Shin

Lembke

E. S.

McMaster

K. L.

(2020). Technical adequacy of curriculum-based measures in writing in grades 1–3. Reading & Writing Quarterly, 36(6), 563–587. https://doi.org/10.1080/10573569.2019.1689211

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing.

Applebee

(2000). Alternative models of writing development. Perspectives on Writing Research, Theory, and Practice, 90–110.

Beck

I. L.

McKeown

M. G.

Kucan

(2013). Bringing words to life: Robust vocabulary instruction. Guilford Press.

Bondie

R. S.

Dahnke

Zusho

(2019). How does changing “one-size-fits-all” to differentiated instruction affect teaching? Review of Research in Education, 43(1), 336–362. https://doi.org/10.3102/0091732X18821130

Burgin

Hughes

G. D.

(2009). Credibly assessing reading and writing abilities for both elementary student and program assessment. Assessing Writing, 14(1), 25–37. https://doi.org/10.1016/j.asw.2008.12.001

Cohen

(1988). Statistical Power Analysis for the Behavioral Sciences. Routledge Academic.

Coker

D. L.

Ritchey

K. D.

(2010). Curriculum-based measurement of writing in kindergarten and first grade: An investigation of production and qualitative scores. Exceptional Children, 76(2), 175–193.

Coker

D. L.

Ritchey

K. D.

(2014). Universal screening for writing risk in kindergarten. Assessment for Effective Intervention, 39(4), 245–256. https://doi.org/10.1177/1534508413502389

10.

Conway

J. M.

Jako

R. A.

Goodman

D. F.

(1995). A meta-analysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80(5), 565–579.

11.

Crocker

L. M.

Algina

(1986). Introduction to classical and modern test theory. Holt, Rinehart, and Winston.

12.

Curran

P. J.

West

S. G.

Finch

J. F.

(1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1(1), 16–29. https://doi.org/10.1037/1082-989X.1.1.16

13.

Espin

C. A.

Weissenburger

J. W.

Benson

B. J.

(2004). Assessing the writing performance of students in special education. Exceptionality, 12(1), 55–66.

14.

Feenstra

H. M.

(2014). Assessing writing ability in primary education: On the evaluation of text quality and text complexity [Doctoral dissertation]. https://research.utwente.nl/en/publications/assessing-writing-ability-in-primary-education-on-the-evaluation-

15.

Frey

B. B.

(Ed.). (2018). The SAGE encyclopedia of educational research, measurement, and evaluation. Sage Publications.

16.

Gansle

K. A.

VanDerHeyden

A. M.

Noell

G. H.

Resetar

J. L.

Williams

K. L.

(2006). The technical adequacy of curriculum-based and rating-based measures of written expression for elementary school students. School Psychology Review, 35(3), 435–450.

17.

Hall-Mills

Apel

(2015). Linguistic feature development across grades and genre in elementary writing. Language, Speech, and Hearing Services in Schools, 46(3), 242–255. https://doi.org/10.1044/2015_LSHSS-14-0043

18.

Hayes

A. F.

Krippendorff

(2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89.

19.

Henson

R. K.

Roberts

J. K.

(2006). Use of exploratory factor analysis in published research: Common errors and some comment on improved practice. Educational and Psychological Measurement, 66(3), 393–416.

20.

Huot

(2002). Toward a new discourse of assessment for the college writing classroom. College English, 65(2), 163–180.

21.

James-Burdumy

Mansfield

Deke

Carey

Lugo-Gil

Hershey

. . . Faddis

(2009). Effectiveness of selected supplemental reading comprehension interventions: Impacts on a first cohort of fifth-grade students. NCEE 2009-4032. National Center for Education Evaluation and Regional Assistance.

22.

Jeffery

J. V.

(2011). Subjectivity, intentionality, and manufactured moves: Teachers’ perceptions of voice in the evaluation of secondary students’ writing. Research in the Teaching of English, 46(1), 92–127.

23.

Jeong

(2017). Narrative and expository genre effects on students, raters, and performance criteria. Assessing Writing, 31, 113–125. https://doi.org/10.1016/j.asw.2016.08.006

24.

Juel

Griffith

P. L.

Gough

P. B.

(1986). Acquisition of literacy: A longitudinal study of children in first and second grade. Journal of Educational Psychology, 78(4), 243–255.

25.

Kaminski

R. A.

Good

R. H.

III . (1998). Assessing early literacy skills in a problem-solving model: Dynamic indicators of basic early literacy skills. In Shinn

M. R.

(Ed.), The Guilford school practitioner series. Advanced applications of curriculum-based measurement (pp. 113–142). Guilford Press.

26.

Keller-Margulis

M. A.

Ochs

Reid

E. K.

Faith

E. L.

Schanding

G. T.

Jr . (2019). Validity and diagnostic accuracy of early written expression screeners in kindergarten. Journal of Psychoeducational Assessment, 37(5), 1–14. https://doi.org/10.1177/0734282918769978

27.

Kim

Y. S.

Al Otaiba

Folsom

J. S.

Greulich

Puranik

(2014). Evaluating the dimensionality of first-grade written composition. Journal of Speech, Language, and Hearing Research, 57(1), 199–211. https://doi.org/10.1044/1092-4388(2013/12-0152)

28.

Kim

Y. S.

Al Otaiba

Puranik

Folsom

J. S.

Greulich

Wagner

R. K.

(2011). Componential skills of beginning writing: An exploratory study. Learning and Individual Differences, 21(5), 517–525. https://doi.org/10.1016/j.lindif.2011.06.004

29.

Kim

Y. S.

Al Otaiba

Wanzek

Gatlin

(2015). Toward an understanding of dimensions, predictors, and the gender gap in written composition. Journal of Educational Psychology, 107(1), 79–95. https://doi.org/10.1037/a0037210

30.

Krueger

R. A.

(1994). Focus groups: A practical guide for applied research (2nd ed.). SAGE.

31.

Lam

(2018). Portfolio assessment for the teaching and learning of writing. Springer.

32.

Landis

J. R.

Koch

G. G.

(1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.

33.

Lembke

E. S.

Garman

Deno

S. L.

Stecker

P. M.

(2010). One elementary school’s implementation of response to intervention (RTI). Reading & Writing Quarterly, 26(4), 361–373. https://doi.org/10.1080/10573569.2010.500266

34.

Lloyd-Jones

(1977). Primary trait scoring. Evaluating Writing: Describing, Measuring, Judging, 33–66.

35.

Lonigan

C. J.

Milburn

T. F.

(2017). Identifying the dimensionality of oral language skills of children with typical development in preschool through fifth grade. Journal of Speech, Language, and Hearing Research, 60(8), 2185–2198. https://doi.org/10.1044/2017_JSLHR-L-15-0402

36.

Lynn

M. R.

(1986). Determination and quantification of content validity. Nursing Research, 35, 382–385.

37.

MacCallum

R. C.

Widaman

K. F.

Zhang

Hong

(1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99.

38.

McCoach

D. B.

Gable

R. K.

Madura

J. P.

(2013). Instrument development in the affective domain. Springer. https://doi.org/10.1007/978-1-4614-7135-6_8

39.

McCrae

R. R.

Kurtz

J. E.

Yamagata

Terracciano

(2011). Internal consistency, retest reliability, and their implications for personality scale validity. Personality and Social Psychology Review, 15(1), 28–50. https://doi.org/10.1177/1088868310366253

40.

McKenna

Goldstein

Soto

Cheng

Troia

Ferron

(2021). Supplemental intervention improves writing performance of first grade students: A single case experimental design evaluation. Journal of Educational Research, 114(3), 278–293. https://doi.org/10.1080/00220671.2021.1923450

41.

McMaster

K. L.

Espin

(2007). Technical features of curriculum-based measurement in writing: A literature review. The Journal of Special Education, 41(2), 68–84.

42.

McMaster

K. L.

Shin

Espin

C. A.

Jung

P. G.

Wayman

M. M.

Deno

S. L.

(2017). Monitoring elementary students’ writing progress using curriculum-based measures: Grade and gender differences. Reading and Writing, 30, 2069–2091. https://doi.org/10.1007/s11145-017-9766-9

43.

Messick

(1975). The standard problem: Meaning and values in measurement and evaluation. American Psychologist, 30(10), 955–966.

44.

Muthen

L. K.

Muthen

B. O.

(2019). Mplus [computer software]. Muthén & Muthén.

45.

National Center for Education Statistics. (2012). The nation’s report card: Writing 2011. Institute of Education Sciences, U.S. Department of Education.

46.

National Commission on Writing. (2006). Writing and school reform. http://www.collegeboard.com

47.

Northwest Regional Educational Laboratory 6+1 Trait® Writing. (2014). http://www.nea.org/tools/lessons/59760.htm

48.

Olinghouse

N. G.

(2008). Student-and instruction-level predictors of narrative writing in third-grade students. Reading and Writing, 21(1–2), 3–26.

49.

Payan

A. M.

Keller-Margulis

Burridge

A. B.

McQuillin

S. D.

Hassett

K. S.

(2019). Assessing teacher usability of written expression curriculum-based measurement. Assessment for Effective Intervention, 45(1), 51–64. https://doi.org/10.1177/1534508418781007

50.

Philippakos

Z. A. T.

FitzPatrick

(2018). A proposed tiered model of assessment in writing Instruction: Supporting all student-writers. Insights into Learning Disabilities, 15(2), 149–173.

51.

Puranik

Duncan

Ying

(2020). Exploring the dimensionality of kindergarten written composition. Reading and Writing, 33, 2481–2510. https://doi.org/10.1007/s11145-020-10053-1

52.

Rios

Wells

(2014). Validity evidence based on internal structure. Psicothema, 26(1), 108–116. https://doi/10.7334/psicothema2013.260

53.

Romig

J. E.

Therrien

W. J.

Lloyd

J. W.

(2017). Meta-analysis of criterion validity for curriculum-based measurement in written language. The Journal of Special Education, 51(2), 72–82. https://doi.org/10.1177/0022466916670637

54.

Sturm

J. M.

Cali

Nelson

N. W.

Staskowski

(2012). The Developmental Writing Scale: A new progress monitoring tool for beginning writers. Topics in Language Disorders, 32(4), 297–318. https://doi.org/10.1097/TLD.0b013e318272159e

55.

Taylor

R. L.

(2006). Assessment of exceptional students: Educational and psychological procedures (7th ed.). Pearson Education.

56.

United Kingdom Department for Education. (2012). The research evidence on writing. Department for Education. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/183399/DFE-RR238.pdf

57.

Wagner

R. K.

Puranik

C. S.

Foorman

Foster

Tschinkel

Kantor

P. T.

(2011). Modeling the development of written language. Reading and Writing, 24, 203–220. https://doi.org/10.1007/s11145-010-9266-7

58.

White

K. M.

(2013). Associations between teacher–child relationships and children’s writing in kindergarten and first grade. Early Childhood Research Quarterly, 28(1), 166–176. https://doi.org/10.1016/j.ecresq.2012.05.004

59.

Willis

G. B.

(2004). Cognitive interviewing revisited: A useful technique, in theory. In Presser

Rothgeb

J. M.

Couper

M. P.

Lessler

J. T.

Martin

Singe

(Eds.), Methods for testing and evaluating survey questionnaires (pp. 23–43). Wiley.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

2.04 MB