Reliability of Ratings of an English Language Arts Curriculum With the Curriculum Evaluation Guidelines

Abstract

The current study examined the reliability of The Reading League Curriculum-Evaluation Guidelines (CEGs), which were developed to help school-based teams rate the presence of red flags when considering adopting specific literacy curricula. Coders (n = 30) independently used the CEGs to evaluate a free online English language arts curriculum. The results indicated strong internal consistency (a = 0.96) and high interrater reliability (H_M = .91, 95% CI = .89 to .93, p < .01). Overall, the CEGs hold the potential as a psychometrically sound tool for evaluating reading curricula. Limitations and implications for practice and research are discussed.

Keywords

curriculum English language arts rating scales

Get full access to this article

View all access options for this article.

References

Achieve. (2003). Educators evaluating the quality of instructional programs. https://www.achieve.org/our-initiatives/equip/equip

Aiken

L. R.

(1985). Three coefficients for analyzing the reliability and validity of ratings. Educational and Psychological Measurement, 45(1), 131–142. https://doi.org/10.1177/0013164485451012

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.

The Apple Group. (2014). Connections: OG in 3D. https://applegroupdyslexia.com/connections

Bogaerds-Hazenberg

S. T.

Evers-Vermeul

van den Bergh

(2022). What textbooks offer and what teachers teach: An analysis of the Dutch reading comprehension curriculum. Reading and Writing, 35(7), 1497–1523. https://doi.org/10.1007/s11145-021-10244-4

Chan

T. M. Y.

Grock

Paddock

Kulasegaram

Yarris

L. M.

Lin

(2016). Examining reliability and validity of an online score (ALiEM AIR) for rating free open access medical education resources. Annals of Emergency Medicine, 68(6), 729–735. https://doi.org/10.1016/j.annemergmed.2016.02.018

Clifford

R. M.

Reszka

S. S.

Rossbach

H. G.

(2010). Reliability and validity of the Early Childhood Environment Rating Scale. The University of North Carolina at Chapel Hill.

Crowe

E. C.

Connor

C. M.

Petscher

(2009). Examining the core: Relations among reading curricula, poverty, and first through third grade reading achievement. Journal of School Psychology, 47(3), 187–214. https://doi.org/10.1016/j.jsp.2009.02.002

Dougherty Stahl

K. A.

Flanigan

McKenna

M. C

. (2020). Assessment for reading instruction (4th ed.). The Guilford Press.

10.

EL Education. (2023). K-5 Language arts curriculum. https://curriculum.eleducation.org

11.

Hallinger

Wang

W. C.

Chen

C. W.

(2013). Assessing the measurement properties of the principal instructional management rating scale: A meta-analysis of reliability studies. Educational Administration Quarterly, 49(2), 272–309. https://doi.org/10.1177/0013161X124681

12.

Hollingsworth

(2023, April 20). Why more schools are embracing a new “science of reading.” PBS News Hour. https://www.pbs.org/newshour/education/why-more-u-s-schools-are-embracing-a-new-science-of-reading

13.

Hoover

W. A.

Gough

P. B.

(1990). The simple view of reading. Reading and Writing, 2, 127–160. https://doi.org/10.1007/BF00401799

14.

Hsu

W. Y.

Lin

S. S.

Chang

S. M.

Tseng

Y. H.

Chiu

N. Y.

(2015). Examining the diagnostic criteria for Internet addiction: Expert validation. Journal of the Formosan Medical Association, 114(6), 504–508. https://doi.org/10.1016/j.jfma.2014.03.010

15.

Iacono

Spong

Bagley

Garcia-Melgar

McKinstry

Hyett

Arthur-Kelly

(2023). Reliability testing of the reasonable adjustments for inclusive education rating scale. Journal of Research in Childhood Education, 38(1), 1–13. https://doi.org/10.1080/02568543.2023.2187488

16.

Ilgen

J. S.

I. W.

Hatala

Cook

D. A.

(2015). A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Medical Education, 49(2), 161–173. https://doi.org/10.1111/medu.12621

17.

Kane

(2013). The argument-based approach to validation. School Psychology Review, 42(4), 448–457. https://doi.org/10.1080/02796015.2013.12087465

18.

Klages

Scholtens

M. M.

Fowler

(2023). Evaluating literacy curriculum: Making sure elementary students learn to read. European Journal of Teaching and Education, 5(3), 1–23. https://doi.org/10.33422/ejte.v5i3.1043

19.

National Center for Educational Statistics. (2022). The 2022 nation’s report card. U.S. Department of Education Institute of Education Sciences.

20.

Norman

(2005). Checklists vs. ratings, the illusion of objectivity, the demise of skills and the debasement of evidence. Advanced Health Science Education Theory and Practice, 10, 1–3. https://doi.org/10.1007/s10459-005-4723-9

21.

Partelow

Shapiro

(2018, August 29). Curriculum reform in the nation’s largest school districts. Center for American Progress. https://www.americanprogress.org/article/curriculum-reform-nations-largest-school-districts/

22.

The Reading League. (2020). Curriculum evaluation tool.

23.

The Reading League. (2023). Curriculum evaluation guidelines. https://www.thereadingleague.org/curriculum-evaluation-guidelines

24.

Salvia

Ysseldyke

J. E.

Witmer

(2017). Assessment in special and inclusive education (13th ed.). Cengage.

25.

Scarborough

H. S.

(2001). Connecting early language and literacy to later reading (dis)abilities: Evidence, theory, and practice. In Neuman

S. B.

Dickinson

D. K.

(Eds.), Handbook of early literacy research (Vol. 1, pp. 97–110). The Guilford Press.

26.

Shanahan

Lomax

R. G.

(1988). A developmental comparison of three theoretical models of the reading–writing relationship. Research in the Teaching of English, 22(2), 196–212. https://doi.org/10.58680/rte198815553

27.

Steiner

(2018). Curriculum research: What we know and where we need to go. Standards Work.

28.

Vaughn

Scales

R. Q.

Stevens

E. Y.

Kline

Barrett-Tatum

Van Wig

Yoder

K. K.

Wellman

(2021). Understanding literacy adoption policies across contexts: A multi-state examination of literacy curriculum decision-making. Journal of Curriculum Studies, 53(3), 333–352. https://doi.org/10.1080/00220272.2019.1683233

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB