The current study examined the reliability of The Reading League Curriculum-Evaluation Guidelines (CEGs), which were developed to help school-based teams rate the presence of red flags when considering adopting specific literacy curricula. Coders (n = 30) independently used the CEGs to evaluate a free online English language arts curriculum. The results indicated strong internal consistency (a = 0.96) and high interrater reliability (HM = .91, 95% CI = .89 to .93, p < .01). Overall, the CEGs hold the potential as a psychometrically sound tool for evaluating reading curricula. Limitations and implications for practice and research are discussed.
AikenL. R. (1985). Three coefficients for analyzing the reliability and validity of ratings. Educational and Psychological Measurement, 45(1), 131–142. https://doi.org/10.1177/0013164485451012
3.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
Bogaerds-HazenbergS. T.Evers-VermeulJ.van den BerghH. (2022). What textbooks offer and what teachers teach: An analysis of the Dutch reading comprehension curriculum. Reading and Writing, 35(7), 1497–1523. https://doi.org/10.1007/s11145-021-10244-4
6.
ChanT. M. Y.GrockA.PaddockM.KulasegaramK.YarrisL. M.LinM. (2016). Examining reliability and validity of an online score (ALiEM AIR) for rating free open access medical education resources. Annals of Emergency Medicine, 68(6), 729–735. https://doi.org/10.1016/j.annemergmed.2016.02.018
7.
CliffordR. M.ReszkaS. S.RossbachH. G. (2010). Reliability and validity of the Early Childhood Environment Rating Scale. The University of North Carolina at Chapel Hill.
8.
CroweE. C.ConnorC. M.PetscherY. (2009). Examining the core: Relations among reading curricula, poverty, and first through third grade reading achievement. Journal of School Psychology, 47(3), 187–214. https://doi.org/10.1016/j.jsp.2009.02.002
9.
Dougherty StahlK. A.FlaniganK.McKennaM. C. (2020). Assessment for reading instruction (4th ed.). The Guilford Press.
HallingerP.WangW. C.ChenC. W. (2013). Assessing the measurement properties of the principal instructional management rating scale: A meta-analysis of reliability studies. Educational Administration Quarterly, 49(2), 272–309. https://doi.org/10.1177/0013161X124681
HsuW. Y.LinS. S.ChangS. M.TsengY. H.ChiuN. Y. (2015). Examining the diagnostic criteria for Internet addiction: Expert validation. Journal of the Formosan Medical Association, 114(6), 504–508. https://doi.org/10.1016/j.jfma.2014.03.010
15.
IaconoT.SpongJ.BagleyK.Garcia-MelgarA.McKinstryC.HyettN.Arthur-KellyM. (2023). Reliability testing of the reasonable adjustments for inclusive education rating scale. Journal of Research in Childhood Education, 38(1), 1–13. https://doi.org/10.1080/02568543.2023.2187488
16.
IlgenJ. S.MaI. W.HatalaR.CookD. A. (2015). A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Medical Education, 49(2), 161–173. https://doi.org/10.1111/medu.12621
KlagesC.ScholtensM. M.FowlerK. (2023). Evaluating literacy curriculum: Making sure elementary students learn to read. European Journal of Teaching and Education, 5(3), 1–23. https://doi.org/10.33422/ejte.v5i3.1043
19.
National Center for Educational Statistics. (2022). The 2022 nation’s report card. U.S. Department of Education Institute of Education Sciences.
20.
NormanG. (2005). Checklists vs. ratings, the illusion of objectivity, the demise of skills and the debasement of evidence. Advanced Health Science Education Theory and Practice, 10, 1–3. https://doi.org/10.1007/s10459-005-4723-9
SalviaJ.YsseldykeJ. E.WitmerS. (2017). Assessment in special and inclusive education (13th ed.). Cengage.
25.
ScarboroughH. S. (2001). Connecting early language and literacy to later reading (dis)abilities: Evidence, theory, and practice. In NeumanS. B.DickinsonD. K. (Eds.), Handbook of early literacy research (Vol. 1, pp. 97–110). The Guilford Press.
26.
ShanahanT.LomaxR. G. (1988). A developmental comparison of three theoretical models of the reading–writing relationship. Research in the Teaching of English, 22(2), 196–212. https://doi.org/10.58680/rte198815553
27.
SteinerD. (2018). Curriculum research: What we know and where we need to go. Standards Work.
28.
VaughnM.ScalesR. Q.StevensE. Y.KlineS.Barrett-TatumJ.Van WigA.YoderK. K.WellmanD. (2021). Understanding literacy adoption policies across contexts: A multi-state examination of literacy curriculum decision-making. Journal of Curriculum Studies, 53(3), 333–352. https://doi.org/10.1080/00220272.2019.1683233
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
