Integrating Validity Evidence to Revise a Child Abuse Knowledge Test for Early Childhood Education Providers: A Mixed Methods Approach

Abstract

Knowledge tests used to evaluate child protection training program effectiveness for early childhood education providers may suffer from threats to construct validity given the contextual variability inherent within state-specific regulations around mandated reporting requirements. Unfortunately, guidance on instrument revision that accounts for such state-specific mandated reporting requirements is lacking across research on evaluation practices. This study, therefore, explored how collection and integration of validity evidence using a mixed methods framework can guide the instrument revision process to arrive at a more valid program outcome measure.

Keywords

mandated reporting child maltreatment validity mixed methods instrument revision child abuse knowledge test effectiveness outcomes

Get full access to this article

View all access options for this article.

References

American Educational Research Association, American Psychological Association, National Council on Measurement in Education, and Joint Committee on Standards for Educational and Psychological Testing (U.S.). (2014). Standards for educational and psychological testing. American Educational Research Association.

Azzam

Jacobson

M. R.

(2015). Reflections on the future of research on evaluation. New Directions for Evaluation, 148, 103–116. https://doi.org/10.1002/ev.20160

Barendse

M. T.

Oort

F. J.

Werner

C. S.

Ligtvoet

Schermelleh-Engel

(2012). Measurement bias detection through factor analysis. Structural Equation Modeling, 19(4), 561–579. https://doi.org/10.1080/10705511.2012.713261

Braverman

M. T.

(2012). Negotiating measurement: Methodological and interpersonal considerations in the choice and interpretation of instruments. American Journal of Evaluation, 34(1), 99–114. https://doi.org/10.1177/1098214012460565

Brown

T. A.

(2006). Confirmatory factor analysis for applied research. Guilford Press.

Canivez

G. L.

(2016). Bifactor modeling in construct validation of multifactored tests: Implications for multidimensionality and test interpretation. In Schweizer

DiStefano

(Eds.), Principles and methods of test construction: Standards and recent advancements (pp. 247–271). Hogrefe.

Carter

Y. H.

Bannon

M. J.

Limbert

Docherty

Barlow

(2006). Improving child protection: A systematic review of training and procedural interventions. Archives of Disease in Childhood, 91, 740–743. https://doi.org/10.1136/adc.2005.092007

Chen

Castro-Alonso

J. C.

Paas

Sweller

(2018). Extending cognitive load theory to incorporate working memory resource depletion: Evidence from the spacing effect. Educational Psychology Review, 30(2), 483–501. https://doi.org/10.1007/s10648-017-9426-2

Child Abuse Prevention and Treatment Reauthorization Act. (2010). P.L. 111-320, 42 U.S.C. 5101 § 3.

10.

Child Welfare Information Gateway. (2014). Domestic violence and the child welfare system. U.S. Department of Health and Human Services, Children’s Bureau. https://www.childwelfare.gov/pubs/factsheets/domestic-violence/

11.

Child Welfare Information Gateway. (2016). Definitions of child abuse and neglect. U.S. Department of Health and Human Services, Children’s Bureau. https://www.childwelfare.gov/pubPDFs/define.pdf#page=5&view=Summaries%20of%20State%20laws

12.

Child Welfare Information Gateway. (2019). Mandatory reporters of child abuse and neglect. U.S. Department of Health and Human Services, Children’s Bureau. https://www.childwelfare.gov/pubPDFs/manda.pdf

13.

Creswell

J. W.

Plano Clark

V. L.

(2018). Designing and conducting mixed methods research (3rd ed.). Sage.

14.

Daro

Donnelly

A.C.

(2015). Reflections on child maltreatment research and practice: Consistent challenges. In Daro

Cohn Donnelly

Huang

Powell

(Eds.), Advances in child abuse prevention knowledge. Child maltreatment (Contemporary issues in research and policy (Vol. 5). Springer. https://doi.org/10.1007/978-3-319-16327-7_1

15.

Dickenson

T. S.

Gilmore

J. A.

Price

K. J.

Bennett

H. L.

(2013). Investigation of science inquiry items for use on an alternate assessment based on modified achievement standards using cognitive lab methodology. The Journal of Special Education, 47(2), 108–120. https://doi.org/10.1177/0022466911414720

16.

Fox

S. E.

Levitt

Nelson

C. A. III

. (2010). How the timing and quality of early experiences influence the development of brain architecture. Child Development, 81(1), 28–40. https://doi.org/10.1111/j.1467-8624.2009.01380.x

17.

Gilbert

Kemp

Thoburn

Sidebotham

Radford

Glaser

MacMillan

H. L.

(2009). Child maltreatment 2: Recognising and responding to child maltreatment. The Lancet, 373, 167–180. https://doi.org/10.1016/S0140-6736(08)61707-9

18.

Grinnell

R. M.

Gabor

Unrau

Y. A.

(2012). Program evaluation for social workers: Foundations of evidence-based programs (6th ed.). Oxford University Press.

19.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

20.

Institute of Medicine and National Research Council. (2014). New directions in child abuse and neglect research. National Academies Press. https://doi.org/10.17226/18331

21.

Johnson

R. B.

Onwuegbuzie

A. J.

Turner

L. A.

(2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133. https://doi.org/10.1177/1558689806298224

22.

Jones Harden

Buhler

Parra

L. J.

(2016). Maltreatment in infancy: A developmental perspective on prevention and intervention. Trauma, Violence, & Abuse, 17(4), 366–386. https://doi.org/10.1177/1524838016658878

23.

Kenny

M. C.

(2007). Web-based training in child maltreatment for future mandated reporters. Child Abuse & Neglect, 31(6), 671–678. https://doi.org/10.1016/j.chiabu.2006.12.008

24.

Langenfeld

Thomas

Zhu

Morris

C. A.

(2020). Integrating multiple sources of validity evidence for an assessment-based cognitive model. Journal of Educational Measurement, 57(2), 159–184. https://doi.org/10.1111/jedm.12245

25.

Lawson

(2019). Domestic violence as child maltreatment: Differential risks and outcomes among cases referred to child welfare agencies for domestic violence exposure. Children and Youth Services Review, 98, 32–41. https://doi.org/10.1016/j.childyouth.2018.12.017

26.

Lee

(2011). Divided attention facilitates intentional forgetting: Evidence from item-method directed forgetting. Consciousness and Cognition, 20(3), 618–626. https://doi.org/10.1016/j.concog.2010.09.008

27.

Leighton

(2017). Using think-aloud interviews and cognitive labs in educational research. Oxford University Press.

28.

Levi

B. H.

Belser

Kapp

Verdiglione

Mincemoyer

Dore

Keat

Fiene

(2019). iLookout for child abuse: Conceptual and practical considerations in creating an online learning programme to engage learners and promote behaviour change. Early Child Development and Care, 1–10. https://doi.org/10.1080/03004430.2019.1626374

29.

Levi

B. H.

Crowell

(2011). Child abuse experts disagree about the threshold for mandated reporting. Clinical Pediatrics, 50(4), 321–329. https://doi.org/10.1177/0009922810389170

30.

Levi

B. H.

Crowell

Walsh

Dellasega

(2015). How childcare providers interpret ‘reasonable suspicion’ of child abuse. Child & Youth Care Forum, 44(6), 875–891. https://doi.org/10.1007/s10566-015-9302-5

31.

Mathews

Yang

Lehman

Mincemoyer

Verdiglione

Levi

(2017). Educating early childhood care and education providers to improve knowledge and attitudes about reporting child maltreatment: A randomized controlled trial. PLoS One, 12(5), e0177777. https://doi.org/10.1371/journal.pone.0177777

32.

Mathews

Walsh

Coe

Kenny

M. C.

Vagenas

(2015). Protocol for a systematic review: Child protection training for professionals to improve reporting of child abuse and neglect. Campbell Systematic Review, 11(1), 1–24. https://doi.org/10.1002/CL2.141

33.

McCrudden

M. T.

McTigue

E. M.

(2019). Implementing integration in an explanatory sequential mixed methods study of belief bias about climate change with high school students. Journal of Mixed Methods Research, 13(3), 381–400. https://doi.org/10.1177/1558689818762576

34.

McDonald

R. P.

(1999). Test theory: Unified treatment. Lawrence Erlbaum Associates.

35.

McKee

B. E.

Dillenburger

(2012). Effectiveness of child protection training for pre-service early childhood educators. International Journal of Education, 53, 348–359. https://doi.org/10.1016/j.ijer.2012.04.008

36.

Messick

(1989a). Validity. In Linn

R. L.

(Ed.), Educational measurement (3rd ed., pp. 13–103). American Council on Education, Macmillan.

37.

Messick

(1989b). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5–11. https://doi.org/10.3102/0013189X018002005

38.

Morell

Tan

R. J. B.

(2009). Validating for use and interpretation: A mixed methods contribution illustrated. Journal of Mixed Methods Research, 3(3), 242–264. https://doi.org/10.1177/1558689809335079

39.

Nastasi

B. K.

Hitchcock

J. H.

(2016). Mixed methods research and culture-specific interventions: Program design and evaluation. Sage.

40.

Nastasi

B. K.

Hitchcock

J. H

Sarkar

Burkholder

Varjas

Jayasena

(2007). Mixed methods in intervention research: Theory to adaptation. Journal of Mixed Methods Research, 1(2), 164–182. https://doi.org/10.1177/1558689806298181

41.

National Child Abuse and Neglect Training and Publications Project. (2014). The Child Abuse Prevention and Treatment Act: 40 Years of safeguarding America’s children. U.S. Department of Health and Human Services, Children’s Bureau. https://www.acf.hhs.gov/sites/default/files/cb/capta_40yrs.pdf

42.

Panlilio

Miyamoto

Font

S. A.

Schreier

H.M.C.

(2019). Assessing risk of commercial sexual exploitation among children involved in the child welfare system. Child Abuse & Neglect, 87, 88–89. https://10.1016/j.chiabu.2018.07.021

43.

Perry

M. A.

Miller

Y. S.

Fusco

R. S.

Fantuzzo

(2013). Parent voices: An investigation of the response process validity of the child behavior checklist with head start parents. NHSA Dialog, 16(2), 75.

44.

Poister

T. H.

(2015). Performance measurement. In Newcomer

K. E.

Hatry

H. P.

Wholey

J. S.

(Eds.), Handbook of practical program evaluation (4th ed., pp. 108–119). John Wiley & Sons. https://doi.org/10.1002/9781119171386.ch5

45.

Powell

B. J.

McMillen

J. C.

Proctor

E. K.

Carpenter

C. R.

Griffey

R. T.

Bunger

A. C.

Glass

J. E.

York

J. L.

(2012). A compilation of strategies for implementing clinical innovations in health and mental health. Medical Care Research and Review, 69(2), 123–157. https://doi.org/10.1177/1077558711430690

46.

Reise

S. P.

Moore

T. M.

Haviland

M. G.

(2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92(6), 544–559. https://doi.org/10.1080/00223891.2010.496477

47.

Rodriguez

Reise

S. P.

Haviland

M. G.

(2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21(2), 137–150. https://doi.org/10.1037/met0000045

48.

Roe

(2008). Question stem. In Lavrakas

P. J.

(Ed.), Encyclopedia of survey research methods (pp. 666–666). Sage. https://doi.org/10.4135/9781412963947.n429

49.

Ryan

Gannon-Slater

Culbertson

M. J.

(2012). Improving survey methods with cognitive interviews in small- and medium-scale evaluations. American Journal of Evaluation, 33(3), 414–430. https://doi.org/10.1177/1098214012441499

50.

Stone

(2019). Maltreatment as a wicked problem: Implications for educational settings. In Panlilio

(Ed.), Trauma informed schools: How child maltreatment prevention, detection, and intervention can be integrated into the school setting (pp. 79–92). Springer.

51.

U.S. Department of Health & Human Services, Administration for Children and Families, Administration on Children, Youth and Families, Children’s Bureau. (2019). Child maltreatment 2017. https://www.acf.hhs.gov/cb/research-data-technology/statistics-research/child-maltreatment

52.

Weitzman

B. C.

Silver

(2013). Good evaluation measures: More than their psychometric properties. American Journal of Evaluation, 34(1), 115–119. https://doi.org/10.1177/1098214012461628

53.

Winter

P. C.

Kopriva

R. J.

Chen

Emick

J. E.

(2006). Exploring individual and item factors that affect assessment validity for diverse learners: Results from a large-scale cognitive lab. Learning and Individual Differences, 16(4), 267–276. https://doi.org/10.1016/j.lindif.2007.01.001

54.

Zucker

Sassman

Case

B. J.

(2004). Cognitive labs. Pearson. http://images.pearsonassessments.com/images/tmrs/tmrs_rg/CognitiveLabs.pdf