This paper explores the gains from multiplying the number of sites used in experimental evaluation of the effects of employment and training programs. Using a multilevel (hierarchical) statistical framework, the authors analyze the role of site multiplication in three recent program evaluations. Although several experiments have involved substantial numbers of sites, the potential benefits from such strategies are largely unrealized. The authors argue for more involvement of the federal government in designing and implementing evaluations that will allow analysis of how the interaction of client, program, and environmental variables affects outcomes.
Get full access to this article
View all access options for this article.
References
1.
AmemiyaTakeshi. 1978. “A Note on a Random Coefficients Model.”International Economic Review, Vol. 19, No. 3, pp. 793–96.
2.
BlankRebecca. 1994. “The Employment Strategy: Public Policies to Increase Work and Earnings.” In DanzigerSheldon H.SandefurGary D.WeinbergDaniel H., eds., Poverty and Public Policy: What Do We Know? What Should We Do?Cambridge, Mass.: Harvard University Press, forthcoming.
3.
BloomHoward S.1991. The National JTPA Study: Baseline Characteristics of the Experimental Sample. Bethesda, Md.: Abt Associates Inc.
4.
BloomHoward S.OrrLarryCaveGeorgeBellStephen H.DoolittleFred. 1993. The National JTPA Study: Title II-A Impacts on Earnings and Employment at 18 Months. Bethesda, Md.: Abt Associates Inc.
5.
BrykAnthony S.RaudenbushStephen W., 1992. Hierarchical Linear Models. Newbury Park, Calif: Sage.
6.
BurtlessGaryOrrLarry. 1986. “Are Classical Experiments Needed for Manpower Policy?”Journal of Human Resources, Vol. 21, No. 4, pp. 606–39.
7.
CookThomas D.CooperHarrisCordrayDavid S.HartmannHeidiHedgesLarry V.LightRichard J.LouisThomas A.MostellerFrederick. 1992. Meta-Analysis for Explanation: A Casebook. New York: Russell Sage Foundation.
8.
DetskyAllan S.1989. “Are Clinical Trials a Cost-Effective Investment?”Journal of the American Medical Association, Vol. 262, No. 13, pp. 1795–1800.
9.
DetskyAllan S.1990. “Using Cost-Effectiveness Analysis to Improve the Efficiency of Allocating Funds to Clinical Trials.”Statistics in Medicine, Vol. 9, pp. 173–84.
10.
FriedlanderDanielGueronJudith M., 1992. “Are High-Cost Services More Effective Than Low-Cost Services?” In ManskiCharles F.GarfinkelIrwin, eds., Evaluating Welfare and Training Programs. Cambridge, Mass.: Harvard University Press, pp. 143–200.
11.
FriedlanderDanielRiccioJamesFreedmanStephen. 1993. GAIN: Two-Year Impacts in Six Counties. New York: Manpower Demonstration Research Corporation.
12.
GreenbergDavidMeyerRobertWisemanMichael. 1993. Prying the Lid from the Black Box: Plotting Evaluation Strategy for Welfare Employment and Training Programs. University of Wisconsin–Madison, Institute for Research on Poverty, Discussion Paper 989–93.
13.
GreenbergDavidWisemanMichael. 1992a. “What Did the OBRA Demonstrations Do?” In ManskiCharles F.GarfinkelIrwin, eds., Evaluating Welfare and Training Programs. Cambridge, Mass.: Harvard University Press, pp. 25–75.
14.
GreenbergDavidWisemanMichael. 1992b. What Did the Work-Welfare Demonstrations Do?University of Wisconsin–Madison, Institute for Research on Poverty, Discussion Paper 969–92.
15.
GueronJudith M.PaulyEdward. 1991. From Welfare to Work. New York: Russell Sage Foundation.
16.
HedgesL. V.1982a. “Estimation of Effect Size from a Series of Independent Experiments.”Psychological Bulletin, Vol. 92, pp. 490–99.
17.
HedgesL. V.1982b. “Fitting Continuous Models to Effect Size Data.”Journal of Educational Statistics, Vol. 7, pp. 245–70.
18.
HedgesL. V.1984. “Advances in Statistical Methods for Meta-Analysis.” In YeatsWilliam H.WortmanPaul M., eds., Issues in Data Synthesis. San Francisco: Jossey-Bass, pp. 25–42.
19.
HedgesL. V.1992. “Meta-Analysis.”Journal of Educational Statistics, Vol. 17, No. 4, pp. 279–96.
20.
HedgesL. V.OlkinI., 1985. Statistical Methods for Meta-Analysis. New York: Academic Press.
21.
HotzV. Joseph. 1992. “Designing an Evaluation of the Job Training Partnership Act.” In ManskiCharles F.GarfinkelIrwin, eds., Evaluating Welfare and Training Programs. Cambridge, Mass.: Harvard University Press, pp. 76–114.
22.
HsiaoCheng. 1986. Analysis of Panel Data. New York: Cambridge University Press.
23.
HunterJohn E.SchmidtFrank L., 1990. Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. Newbury Park, Calif.: Sage.
24.
KirkRoger E.1982. Experimental Design: Procedures for the Behavioral Sciences, 2nd ed.Pacific Grove, Calif.: Brooks/Cole.
25.
Manpower Demonstration Research Corporation. 1992. The JOBS Evaluation: Overview of the Design. New York: Manpower Demonstration Research Corporation.
26.
ManskiCharles F.GarfinkelIrwin. 1992. Evaluating Welfare and Training Programs. Cambridge, Mass.: Harvard University Press.
27.
PumaMichael J.BursteinNancy R.MerrellKatySilversteinGary. 1990. Evaluation of the Food Stamp Employment and Training Program: Final Report. Bethesda, Md.: Abt Associates, Inc.
28.
RiccioJamesFriedlanderDaniel. 1992. GAIN: Program Strategies, Participation Patterns, and First-Year Impacts in Six Counties. New York: Manpower Development Research Corporation.
29.
RosenthalRobert1991. Meta-Analytic Procedures for Social Research (Revised Edition). Newbury Park, Calif.: Sage.
30.
RosenthalRobertRubinD. B., 1982. “Comparing Effect Sizes of Independent Studies.”Psychological Bulletin, Vol. 92, pp. 500–504.
31.
RubinDonald B.1992. “Meta-Analysis: Literature Synthesis or Effect-Size Surface Estimation?”Journal of Educational Statistics, Vol. 17, No. 4, pp. 363–74.
32.
StiglerS. M.1986. The History of Statistics: The Measurement of Uncertainty Before 1900. Cambridge, Mass.: Harvard University Press.
33.
WisemanMichael. 1993. The New State Welfare Initiatives. Madison, Wis.: University of Wisconsin Institute for Research on Poverty, Working Paper 1002–93.
