Developing and Gathering Psychometric Evidence for a Fidelity Instrument

Abstract

Fidelity assessment has received renewed attention in recent years, particularly as distinctions have been made in implementation science between intervention fidelity and implementation fidelity. Considering both types of fidelity has been recommended when developing fidelity instruments. In the present article, we describe development of the pilot version of the Teaching Pyramid Observation Tool (TPOT-P) as a case example of designing a fidelity instrument for use in research and practice. The TPOT is a multimethod judgment-based rating scale designed to measure practitioners’ fidelity of implementation of practices associated with the Pyramid Model. We describe the structure of the TPOT-P in relation to Pyramid Model components and fidelity indicators. We summarize the measurement approaches grounded in generalizability theory and classical test theory that were used to investigate the psychometric properties of TPOT-P scores based on data collected by trained raters on three occasions in 50 preschool classrooms. Findings suggest the TPOT-P shows promise for dependably measuring teachers’ implementation of Pyramid Model practices.