Evaluators must employ research designs that generate compelling evidence related to the worth or value of programs, of which assessment data often play a critical role. This article focuses on assessment design in the context of evaluation. It describes the process of using the Framework for K-12 Science Education and Next Generation Science Standards (NGSS) to design assessments to evaluate the efficacy of a curricular intervention. The new science standards present a significant challenge to assessment designers and evaluators because these standards emphasize the integration of disciplinary core ideas, practices, and crosscutting concepts. This article presents the structure of a validity argument for such uses with an evidence-centered design perspective and unpacks the design decisions in developing and implementing these assessments in an efficacy study of a project-based science curriculum. Implications for designing NGSS-aligned assessments for program evaluation purposes are discussed.
AtkinJ. M.BlackP. (2003). Inside science education reform: A history of curricular and policy change. New York, NY: Teachers College Press.
2.
CenturyJ.RudnickM.FreemanC. (2010). A framework for measuring fidelity of implementation: A foundation for shared language and accumulation of knowledge. American Journal of Evaluation, 31, 199–218.
3.
CordrayD. S.PionG. M. (2006). Treatment strength and integrity: Models and methods. In BootzinR. R.McKnightP. E. (Eds.), Strengthening research methodology: Psychological measurement and evaluation (pp. 103–124). Washington, DC: American Psychological Association.
4.
DuschlR. A.SchweingruberH. A.ShouseA. W. (Eds.). (2007). Taking science to school: Learning and teaching science in grades K-8. Washington, DC: National Academies Press.
5.
FurtakE. M.ThompsonJ.BraatenM.WindschitlM. (2012). Learning progressions to support ambitious teaching practices. In AlonzoA. C.GotwalsA. W. (Eds.), Learning progressions in science: Current challenges and future directions (pp. 405–434). Rotterdam, the Netherlands: Sense Publishers.
6.
KaneM. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112, 527–535.
7.
KennedyC. A. (2012a). PBIS energy instrument psychometric analyses. Unpublished internal document, SRI, Menlo Park, CA.
8.
KennedyC. A. (2012b). New PBIS earth science analysis. Unpublished internal document, SRI, Menlo Park, CA.
9.
KolodnerJ. L.CampP. J.CrismondD.FasseB. B.GrayJ. T.HolbrookJ.…RyanM. (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting learning-by-design into practice. Journal of the Learning Sciences, 12, 495–547.
10.
LoveA. (2004). Implementation evaluation. In WholeyJ. S.HatryH. P.NewcomerK. E. (Eds.), Handbook of practical program evaluation (2nd ed., pp. 63–97). San Francisco, CA: Jossey-Bass.
11.
MeansB.PenuelW. R. (2005). Research to support scaling up technology-based educational innovations. In DedeC.HonanJ. P.PetersL. C. (Eds.), Scaling up success: Lessons from technology-based educational improvement (pp. 176–197). San Francisco, CA: Jossey-Bass.
12.
MessickS. (1989). Validity. In LinnR. L. (Ed.), Educational measurement (3rd ed., pp. 13–103). New York, NY: Macmillan.
13.
MessickS. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23, 13–23.
14.
MessickS. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.
MislevyR. J.HaertelG. D. (2006). Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 25, 6–20.
17.
MorganS. L.WinshipC. (2007). Counterfactuals and causal inference. London, England: Cambridge University Press.
18.
National Research Council. (2002). Scientific research in education. Washington, DC: National Academy Press.
19.
NGSS Lead States. (2013). Next generation science standards: For states, by states. Washington, DC: National Academies Press.
20.
PellegrinoJ. W. (2013). Proficiency in science: Assessment challenges and opportunities. Science, 340, 320–323.
21.
PellegrinoJ. W.ChudowskyN.GlaserR. (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academies Press.
22.
PellegrinoJ. W.WilsonM.KoenigJ. A.BeattyA. S. (2013). Developing assessments for the Next Generation Science Standards. Washington, DC: National Academies Press.
23.
QuinnH.SchweingruberH.KellerT. (Eds.). (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Academies Press.
24.
RivetA. E.KastensK. A. (2012). Developing a construct-based assessment to examine students’ analogical reasoning around physical models in earth science. Journal of Research in Science Teaching, 49, 713–743. doi:10.1002/tea.21029
25.
Ruiz-PrimoM. A.ShavelsonR. J.HamiltonL. S.KleinS. (2002). On the evaluation of systemic science education reform: Searching for instructional sensitivity. Journal of Research in Science Teaching, 39, 369–393.
26.
SamejimaF. (1994). Estimation of reliability coefficients using the test information function and its modifications. Applied Psychological Measurement, 18, 229–244.
27.
SchwarzC. V.ReiserB. J.DavisE. A.KenyonL.AcherA.FortusD.…KrajcikJ. (2009). Developing a learning progression for scientific modeling: Making scientific modeling accessible and meaningful for learners. Journal of Research in Science Teaching, 46, 632–654.
28.
SechrestL. B.WestS. G.PhillipsM. A.RednerR.YeatonW. H. (1979). Some neglected problems in evaluation research: Strength and integrity of treatments. In SechrestL. B.WestS. G.PhillipsM. A.RednerR.YeatonW. (Eds.), Evaluation studies review annual (Vol. 4, pp. 15–35). Beverly Hills, CA: Sage.
29.
SechrestL. B.YeatonW. H. (1979). Strength and integrity of treatments in evaluation studies. Washington, DC: National Criminal Justice Reference Service.
30.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton-Mifflin.
31.
ShepardL. A. (1993). Evaluating test validity. In Darling-HammondL. (Ed.), Review of Research in Education (Vol. 19). Washington, DC: American Educational Research Association.
