To maximize the impact of research on programs, this article proposes a ‘reaching-down—reachingup’ perspective in evaluation design, whereby it serves two functions simultaneously: the program improvement function, reaching down, and the knowledge development function, reaching up. This proposal frames applied research as a particular species of evaluation. As validity is a fundamental assessment of how well a design supports particular inferences related to specific uses, the article subsequently summarizes and integrates disparate validity perspectives to develop an argument-based approach to validity in a ‘reaching-down—reaching-up’ evaluation model case study. Lastly, the location(s) of validation is considered in order to highlight evaluator responsibilities.
Bollen KA and Curran PJ ( 2006) Latent Curve Models: A Structural Equation Perspective . Hoboken: John Wiley & Sons, Inc.
2.
Campbell DT ( 1957) Factors relevant to the validity of experiments in social settings. Psychological Bulletin54: 297-312.
3.
Campbell DT ( 1969) Reforms as experiments. American Psychologist24(4): 409-29.
4.
Campbell DT and Stanley JC ( 1963) Experimental and quasi-experimental designs for research on teaching. In: Gage NL (ed.) Handbook of Research on Teaching. Chicago, IL: Rand McNally.
5.
Cook TD ( 2000) Toward a practical theory of external validity. In: Bickman L (ed.) Validity and Social Experimentation: Donald Campbell’s Legacy, Vol. 1. Thousand Oaks, CA: SAGE, 3-43.
6.
Cook TD and Campbell DT ( 1979) Quasi-Experimentation Design and Analysis Issues for Field Settings. Boston, MA: Houghton.
7.
Cronbach LJ ( 1971) Test validation. In: Thorndike RL (ed.) Educational Measurement, 2nd edn. Washington, DC: American Council on Education, 443-507.
8.
Cronbach LJ ( 1980) Validity on parole: how can we go straight? New Directions for Testing and Measurement5: 99-108.
9.
Cronbach LJ ( 1982) Designing Evaluations of Educational and Social Programs . San Francisco, CA: Jossey-Bass .
10.
Cronbach LJ ( 1988) Five perspectives on validity argument. In: Wainer H and Braun HI (eds) Test Validity. Hillsdale, NJ: Lawrence Erlbaum Associates, 3-17.
11.
Cronbach LJ and Meehl PE ( 1955) Construct validity in psychological tests. Psychological Bulletin52(4): 281-302.
12.
Cronbach LJ and Suppes P (eds) (1969) Research for Tomorrow’s Schools: Disciplined Inquiry for Education. London: Macmillan.
13.
Cureton EE ( 1951) Validity. In: Lindquist EF (ed.) Educational Measurement. Menasha, WI: George Banta, 621-94.
14.
Hennessy M. and Greenberg J. ( 1999) Bringing it all together: modeling intervention processes using structural equation modeling. American Journal of Evaluation20(3): 471-80.
15.
Henry GT ( 2002) Choosing criteria to judge program success: values inquiry . Evaluation8(2): 182-204.
16.
House ER ( 1980) Evaluating with Validity. Beverly Hills, CA: SAGE.
17.
House ER and Howe KR ( 1999) Values in Evaluation and Social Research. Thousand Oaks, CA: SAGE.
18.
Johnson E., Kirkhart KE, Madison AM, Noley GB and Solana-Flores G. ( 2008) The impact of narrow views of scientific rigor on evaluation practices for underrepresented groups. In: Smith NL and Brandon PR (eds) Fundamental Issues In Evaluation. New York: The Guilford Press, 197-211.
Kane MT ( 2009) Validating the interpretations and uses of test scores . In: Lissitz RW (ed.) The Concept of Validity: Revisions, New Directions and Applications. Charlotte, NC: Information Age Publishing, Inc, 39-64.
21.
Kirkhart KE ( 2005) Through a cultural lens: reflections on validity and theory in evaluation. In: Hood S, Hopson RK and Frierson HT (eds) The Role of Culture and Cultural Context: A Mandate for Inclusion, the Discovery of Truth, and Understanding in Evaluative Theory and Practice. Greenwich, CT: Information Age Publishing, Inc, 21-39.
22.
Mark MM ( 2000) Realism, validity, and the experimenting society. In: Bickman L (ed.) Validity and Social Experimentation: Donald Campbell’s Legacy, Vol. 1. Thousand Oaks, CA: SAGE, 141-66.
23.
Mark MM, Henry GT and Julnes G. ( 2000) Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs. San Francisco, CA: Jossey-Bass.
24.
Messick S. ( 1989) Validity. In: Linn RL (ed.) Educational Measurement. New York: Macmillan, 13-103.
25.
Pawson R. ( 2002) Evidence-based policy: the promise of ‘realist synthesis’ . Evaluation8(3): 340-58.
26.
Pawson R. ( 2004) Evidence-based policy: a realist perspective. In: Carter B and New C (eds) Making Realism Work: Realist Social Theory and Empirical Research. London: Routledge.
27.
Pawson R. and Tilley N. ( 1997) Realistic Evaluation. Thousand Oaks, CA: SAGE.
28.
Schwandt TA ( 2007) Judging interpretations. New Directions for Evaluation114: 11-14.
29.
Scriven M. ( 2007) Key evaluation checklist. Available at: http://www.wmich.edu/evalctr/checklists
30.
Shadish WR, Cook TD and Campbell DT ( 2002) Experimental and Quasi-Experimental Designs For Generalized Causal Inference. Boston, MA: Houghton Mifflin.
31.
Yarbrough DB, Shulha LM, Hopson RK and Caruthers FA (eds) (2011) Joint Committee on Standards for Educational Evaluation: The Program Evaluaton Standards A Guide for Evaluators and Evaluation Users, 3rd edn. Thousand Oaks, CA: SAGE.
