Knowledge tests used to evaluate child protection training program effectiveness for early childhood education providers may suffer from threats to construct validity given the contextual variability inherent within state-specific regulations around mandated reporting requirements. Unfortunately, guidance on instrument revision that accounts for such state-specific mandated reporting requirements is lacking across research on evaluation practices. This study, therefore, explored how collection and integration of validity evidence using a mixed methods framework can guide the instrument revision process to arrive at a more valid program outcome measure.
American Educational Research Association, American Psychological Association, National Council on Measurement in Education, and Joint Committee on Standards for Educational and Psychological Testing (U.S.). (2014). Standards for educational and psychological testing. American Educational Research Association.
2.
AzzamT.JacobsonM. R. (2015). Reflections on the future of research on evaluation. New Directions for Evaluation, 148, 103–116. https://doi.org/10.1002/ev.20160
BravermanM. T. (2012). Negotiating measurement: Methodological and interpersonal considerations in the choice and interpretation of instruments. American Journal of Evaluation, 34(1), 99–114. https://doi.org/10.1177/1098214012460565
5.
BrownT. A. (2006). Confirmatory factor analysis for applied research. Guilford Press.
6.
CanivezG. L. (2016). Bifactor modeling in construct validation of multifactored tests: Implications for multidimensionality and test interpretation. In SchweizerK.DiStefanoC. (Eds.), Principles and methods of test construction: Standards and recent advancements (pp. 247–271). Hogrefe.
7.
CarterY. H.BannonM. J.LimbertC.DochertyA.BarlowJ. (2006). Improving child protection: A systematic review of training and procedural interventions. Archives of Disease in Childhood, 91, 740–743. https://doi.org/10.1136/adc.2005.092007
8.
ChenO.Castro-AlonsoJ. C.PaasF.SwellerJ. (2018). Extending cognitive load theory to incorporate working memory resource depletion: Evidence from the spacing effect. Educational Psychology Review, 30(2), 483–501. https://doi.org/10.1007/s10648-017-9426-2
Child Welfare Information Gateway. (2019). Mandatory reporters of child abuse and neglect. U.S. Department of Health and Human Services, Children’s Bureau. https://www.childwelfare.gov/pubPDFs/manda.pdf
13.
CreswellJ. W.Plano ClarkV. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage.
14.
DaroD.DonnellyA.C. (2015). Reflections on child maltreatment research and practice: Consistent challenges. In DaroD.Cohn DonnellyA.HuangL.PowellB. (Eds.), Advances in child abuse prevention knowledge. Child maltreatment (Contemporary issues in research and policy (Vol. 5). Springer. https://doi.org/10.1007/978-3-319-16327-7_1
15.
DickensonT. S.GilmoreJ. A.PriceK. J.BennettH. L. (2013). Investigation of science inquiry items for use on an alternate assessment based on modified achievement standards using cognitive lab methodology. The Journal of Special Education, 47(2), 108–120. https://doi.org/10.1177/0022466911414720
16.
FoxS. E.LevittP.NelsonC. A. III. (2010). How the timing and quality of early experiences influence the development of brain architecture. Child Development, 81(1), 28–40. https://doi.org/10.1111/j.1467-8624.2009.01380.x
17.
GilbertR.KempA.ThoburnJ.SidebothamP.RadfordL.GlaserD.MacMillanH. L. (2009). Child maltreatment 2: Recognising and responding to child maltreatment. The Lancet, 373, 167–180. https://doi.org/10.1016/S0140-6736(08)61707-9
18.
GrinnellR. M.GaborP.UnrauY. A. (2012). Program evaluation for social workers: Foundations of evidence-based programs (6th ed.). Oxford University Press.
19.
HuL.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
20.
Institute of Medicine and National Research Council. (2014). New directions in child abuse and neglect research. National Academies Press. https://doi.org/10.17226/18331
21.
JohnsonR. B.OnwuegbuzieA. J.TurnerL. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133. https://doi.org/10.1177/1558689806298224
22.
Jones HardenB.BuhlerA.ParraL. J. (2016). Maltreatment in infancy: A developmental perspective on prevention and intervention. Trauma, Violence, & Abuse, 17(4), 366–386. https://doi.org/10.1177/1524838016658878
LangenfeldT.ThomasJ.ZhuR.MorrisC. A. (2020). Integrating multiple sources of validity evidence for an assessment-based cognitive model. Journal of Educational Measurement, 57(2), 159–184. https://doi.org/10.1111/jedm.12245
25.
LawsonJ. (2019). Domestic violence as child maltreatment: Differential risks and outcomes among cases referred to child welfare agencies for domestic violence exposure. Children and Youth Services Review, 98, 32–41. https://doi.org/10.1016/j.childyouth.2018.12.017
26.
LeeY.LeeH. (2011). Divided attention facilitates intentional forgetting: Evidence from item-method directed forgetting. Consciousness and Cognition, 20(3), 618–626. https://doi.org/10.1016/j.concog.2010.09.008
27.
LeightonJ. (2017). Using think-aloud interviews and cognitive labs in educational research. Oxford University Press.
28.
LeviB. H.BelserA.KappK.VerdiglioneN.MincemoyerC.DoreS.KeatJ.FieneR. (2019). iLookout for child abuse: Conceptual and practical considerations in creating an online learning programme to engage learners and promote behaviour change. Early Child Development and Care, 1–10. https://doi.org/10.1080/03004430.2019.1626374
29.
LeviB. H.CrowellK. (2011). Child abuse experts disagree about the threshold for mandated reporting. Clinical Pediatrics, 50(4), 321–329. https://doi.org/10.1177/0009922810389170
30.
LeviB. H.CrowellK.WalshK.DellasegaC. (2015). How childcare providers interpret ‘reasonable suspicion’ of child abuse. Child & Youth Care Forum, 44(6), 875–891. https://doi.org/10.1007/s10566-015-9302-5
31.
MathewsB.YangC.LehmanE.MincemoyerC.VerdiglioneN.LeviB. (2017). Educating early childhood care and education providers to improve knowledge and attitudes about reporting child maltreatment: A randomized controlled trial. PLoS One, 12(5), e0177777. https://doi.org/10.1371/journal.pone.0177777
32.
MathewsB.WalshK.CoeS.KennyM. C.VagenasD. (2015). Protocol for a systematic review: Child protection training for professionals to improve reporting of child abuse and neglect. Campbell Systematic Review, 11(1), 1–24. https://doi.org/10.1002/CL2.141
33.
McCruddenM. T.McTigueE. M. (2019). Implementing integration in an explanatory sequential mixed methods study of belief bias about climate change with high school students. Journal of Mixed Methods Research, 13(3), 381–400. https://doi.org/10.1177/1558689818762576
34.
McDonaldR. P. (1999). Test theory: Unified treatment. Lawrence Erlbaum Associates.
35.
McKeeB. E.DillenburgerK. (2012). Effectiveness of child protection training for pre-service early childhood educators. International Journal of Education, 53, 348–359. https://doi.org/10.1016/j.ijer.2012.04.008
36.
MessickS. (1989a). Validity. In LinnR. L. (Ed.), Educational measurement (3rd ed., pp. 13–103). American Council on Education, Macmillan.
37.
MessickS. (1989b). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5–11. https://doi.org/10.3102/0013189X018002005
38.
MorellL.TanR. J. B. (2009). Validating for use and interpretation: A mixed methods contribution illustrated. Journal of Mixed Methods Research, 3(3), 242–264. https://doi.org/10.1177/1558689809335079
39.
NastasiB. K.HitchcockJ. H. (2016). Mixed methods research and culture-specific interventions: Program design and evaluation. Sage.
40.
NastasiB. K.HitchcockJ. HSarkarS.BurkholderG.VarjasK.JayasenaA. (2007). Mixed methods in intervention research: Theory to adaptation. Journal of Mixed Methods Research, 1(2), 164–182. https://doi.org/10.1177/1558689806298181
41.
National Child Abuse and Neglect Training and Publications Project. (2014). The Child Abuse Prevention and Treatment Act: 40 Years of safeguarding America’s children. U.S. Department of Health and Human Services, Children’s Bureau. https://www.acf.hhs.gov/sites/default/files/cb/capta_40yrs.pdf
42.
PanlilioC.MiyamotoS.FontS. A.SchreierH.M.C. (2019). Assessing risk of commercial sexual exploitation among children involved in the child welfare system. Child Abuse & Neglect, 87, 88–89. https://10.1016/j.chiabu.2018.07.021
43.
PerryM. A.MillerY. S.FuscoR. S.FantuzzoJ. (2013). Parent voices: An investigation of the response process validity of the child behavior checklist with head start parents. NHSA Dialog, 16(2), 75.
44.
PoisterT. H. (2015). Performance measurement. In NewcomerK. E.HatryH. P.WholeyJ. S. (Eds.), Handbook of practical program evaluation (4th ed., pp. 108–119). John Wiley & Sons. https://doi.org/10.1002/9781119171386.ch5
45.
PowellB. J.McMillenJ. C.ProctorE. K.CarpenterC. R.GriffeyR. T.BungerA. C.GlassJ. E.YorkJ. L. (2012). A compilation of strategies for implementing clinical innovations in health and mental health. Medical Care Research and Review, 69(2), 123–157. https://doi.org/10.1177/1077558711430690
46.
ReiseS. P.MooreT. M.HavilandM. G. (2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92(6), 544–559. https://doi.org/10.1080/00223891.2010.496477
47.
RodriguezA.ReiseS. P.HavilandM. G. (2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21(2), 137–150. https://doi.org/10.1037/met0000045
RyanK.Gannon-SlaterN.CulbertsonM. J. (2012). Improving survey methods with cognitive interviews in small- and medium-scale evaluations. American Journal of Evaluation, 33(3), 414–430. https://doi.org/10.1177/1098214012441499
50.
StoneS. (2019). Maltreatment as a wicked problem: Implications for educational settings. In PanlilioC. (Ed.), Trauma informed schools: How child maltreatment prevention, detection, and intervention can be integrated into the school setting (pp. 79–92). Springer.
WeitzmanB. C.SilverD. (2013). Good evaluation measures: More than their psychometric properties. American Journal of Evaluation, 34(1), 115–119. https://doi.org/10.1177/1098214012461628
53.
WinterP. C.KoprivaR. J.ChenC.EmickJ. E. (2006). Exploring individual and item factors that affect assessment validity for diverse learners: Results from a large-scale cognitive lab. Learning and Individual Differences, 16(4), 267–276. https://doi.org/10.1016/j.lindif.2007.01.001
