A trial of impact evaluation utilizing effect size statistics: Its application to the evaluation of Japan’s foreign student policy

Abstract

This article proposes a method of evaluating effect, impact and efficiency of treatment using the standardized mean difference (SMD) of outcomes between the treatment and the control groups. Since the SMD is commensurable (divisible and measurable by a common standard), here it is given a unit name ‘effect’ as an expression of the ‘standardized effect’. Based on the assumption that impact is the sum total of the change brought about by the treatment in the population, this article asserts that impact can be calculated by multiplying the ‘standardized effect’ by the population under treatment given that the sample represents the population. Extending this logic, efficiency can be calculated by dividing the impact by the total input of the treatment expressed in a monetary value. An example of the application of this approach to the evaluation of Japan’s Foreign Student Policy toward Thailand is introduced and its merits and limitations are discussed.

Keywords

effect size efficiency impact standardized effect standardized mean difference (SMD)

Get full access to this article

View all access options for this article.

References

Abadie

Drukker

Leber Herr

. (2001) Implementing matching estimators for average treatment effects in stata. The Stata Journal 1(1): 1–18.

Aoyagi

(2006) Trend of impact evaluation in international development community. In: FASID, Issues and Prospects of Evaluations for International Development. Tokyo: Foundation for Advanced Studies on International Development (FASID), 87–153.

Bamberger

Fujita

(2003) Impact Evaluation of Development Assistance. Tokyo: FASID.

Barnette

(2007) Using effect size and association measures. handout at a Professional Development Workshop at the 21st Conference of American Evaluation Association, Baltimore.

Carson

(2008) The effective use of effect size indices in institutional research. URL (consulted 1 August 2008): http://www.keene.edu/ir/effect_size.pdf

Center for Global Development (2006) When will we ever learn? Improving lives through impact evaluation. URL (consulted 1 August 2008): http://www.cgdev.org/content/publications/detail/7973

Clements

Chianca

Sasaki

(2008) Reducing world poverty by improving evaluation of development aid. American Journal of Evaluation 29(2): 195–214.

Cohen

(1988) Statistical Power Analysis for the Behavioral Sciences second edition. New York: Lawrence Erlbaum Associates.

Dehejia

Wahba

(2002) Propensity score-matching methods for nonexperimental causal studies. The Review of Economics and Statistics 84(1): 151–61.

10.

Duflo

Kremer

(2003) Use of randomization in the evaluation of development effectiveness. URL (consulted 1 August 2008): http://www.povertyactionlab.org/research/rand.php

11.

Hansen

Rieper

(2009) The evidence movement: the development and consequences of methodologies in review practices. Evaluation 15(2): 141–63.

12.

International Initiative for Impact Evaluation (2010) URL (consulted 15 December 2010): http://www.3ieimpact.org/

13.

Khandker

SRI

Koolwal

Samad

(2010) Handbook on Impact Evaluation: Quantitative Methods and Practices. Washington, DC: The World Bank.

14.

Lachin

Matts

Wei

(1988) Randomization in clinical trials: conclusions and recommendations. Controlled Clinical Trials 9(4): 365–74.

15.

Levin

McEwan

(2001) Cost-effectiveness analysis: methods and applications, 2nd edn. Thousand Oaks, CA: SAGE.

16.

McEwan

(2010) Empirical research methods in the economics of education. In: Brewer

McEwan

(eds) Economics of Education. Oxford: Elsevier, 9–14.

17.

OECD (2003) Education at a glance 2003 – Tables. URL (consulted 19 October 2005): http://www.oecd.org/document/34/0,2340,en_2649_37455_14152482_1_1_1_37455,00.html

18.

Ministry of Foreign Affairs of Japan (2002) Nihon ni kansuru ASEAN (6 kakoku) Yoron Chosa (Poll survey on Japan in 6 ASEAN countries). URL (consulted 19 June 2005): http://www.mofa.go.jp/mofaj/area/asean/yoron.html

19.

Poverty Action Lab (2008) About the Abdul Latif Jameel poverty action lab. URL (consulted 1 August 2008): http://www.povertyactionlab.org/

20.

Radej

(2011) Synthesis in policy impact assessment. Evaluation 17(2): 133–50.

21.

Ravallion

(2003) Assessing the poverty impact of an assigned program. In: Bourguignon

Pereira

Silva

(eds) The Impact of Economic Policies on Poverty and Income Distribution–Evaluation Techniques and Tools. Washington, DC: The World Bank.

22.

Rossi

Lipsey

Freeman

(2004) Evaluation: A Systematic Approach, 7th edn. Thousand Oaks, CA: SAGE.

23.

Sasaki

(2006) Challenge of evidence-based evaluation in ODA sector: poverty action lab. The Japanese Journal of Evaluation Studies 6(1): 43–54.

24.

Sato

(2002) An impact of the Japanese government’s foreign student policy toward Indonesia: from the perspective of human resources development. Journal of International Development Studies 11(2): 201–19.

25.

UNESCO (1963–1999) Statistical Yearbook. Paris: UNESCO Publishing and Bernan Press.