American Journal of Evaluation

Proposals satisfy a number of functions in evaluation work. They may serve as a work plan, a basis for a contract, a request for commitment, evidence of ability, an evaluative criterion, and, most importantly, to justify the conduct of the study. This article examines how these functions differ depending on whether the proposal concerns a preordinate study in which methods are pre specified, or an emergent study in which decisions about methods are made as the study proceeds.

The argument advanced in this article is that advocacy in evaluation is inevitable. This is so when advocacy is understood not as program partisanship or contaminating bias, but rather as a value commitment to a particular regulative ideal (of rational decision making, interpretive meaning, community activism). The regulative ideal for evaluation advanced in this discussion is a commitment to democratic pluralism. These ideas are illustrated and substantiated with three case examples.

The question is not whether we should take sides, since we inevitably will, but rather whose side are we on? (Becker, 1967, p. 239).

A large government market has developed in evaluation over the past few decades. Governments let contracts to conduct evaluation studies, and these contracts shape the studies. Such contracting does not follow the idealized image of open markets. Rather, a few firms conduct most studies because of the dynamics of the contracting process. If we are to have honest, unbiased evaluations, evaluation ethics must take these market factors into consideration. The conditions under which evaluations are produced are as important as House how evaluation results are used.

In certain circumstances, the evaluation of programs designed to serve disadvantaged and minority target groups may actually be threatening to stakeholders in those groups. When such threats are real, participation by these target groups in the evaluation may cause undue personal stress or not serve their best interests. An important question therefore is what can be done to address the ethical dilemma that arises when participation by these target groups in the evaluation may put them in jeopardy? The present paper examines the issues involved in addressing this question, particularly the roles English that key stakeholders play throughout the evaluation process. Specifically, it is argued that involvement by these target groups in all phases of an evaluation as 'co-operative partners,' and not just at the stage of collecting data as 'bystanders,' is one way of addressing the dilemma that emerges when evaluators cannot guarantee that participation will not be threatening.

Analysis of qualitative—usually verbal—information from interview transcripts, field notes, and other sources has required huge amounts of labor. Some critics, moreover, maintain that such analysis is less than reliable, objective and tabula rasa. A new computer program, however, allows precisely repeatable analysis without a priori frameworks and without tedious and subjective categorizing of passages. The program finds repetitive association of words or ideas that characterize the verbal research corpus which evaluators may display and interpret. This article explains the Walberg program and illustrates its use in an evaluation of a collaborative botanic education project sponsored by the National Science Foundation.

In this essay, I discuss the importance of modeling effective evaluation practice as we teach others about program evaluation. Over the last few years I have experimented with using a tool and process called "critical incidents" in my graduate program evaluation courses. This activity asks students each week to reflect on their reactions to, and learning from the course, by responding to a set of evaluative questions. The students' cumulative responses to these questions demonstrated increased understanding of key Preskill evaluation concepts and an appreciation for their own and others' learning styles. The approach confirmed the importance of modeling formative evaluation throughout a course as a way to teach about evaluation practice.

In this paper, we object to Michael Scriven's claim that the basic logic of evaluation is criterial and standards-based. We note that valuing is an integral part of perception and that valuing within perception, repeatedly refined, is an even more basic logic of evaluation. We find unpersuasive his claim that making the final synthesis "governed" will diminish bias, noting that bias will find its way into the required statements of need, function, standards and weighting. We offer our alternative for disciplining the synthesis process, by urging more systematic and demanding critiques of Robert Stake emerging interpretations and values, and by more deliberately using competing conceptual organizers (e.g., goals, issues, decisions and elements of the rules Scriven advocates) as temporary and dialectical grounds for reconsidering the evolving meanings of the program, including its merit and shortcoming.

This study investigated internal and external evaluators' practices and concerns about communicating and reporting evaluation findings. Approximately three-quarters (72%) of a random sample of American Evaluation Association members responded to a survey on this topic. Most of those responding: (1) adhere to traditional reporting formats; (2) are only moderately satisfied with their communicating and reporting efforts; (3) found that insufficient time and political/organizational complexity impedes success in communicating and reporting; and (4) describe effective Rosalie T. Torres practice as typically entailing high stakeholder involvement. Internal evaluation was found to be not only equally as prevalent as external evaluation, but different in relation to certain communication and reporting practices.

Differential item functioning (DIF) is not often seen in the literature on attitude assessment. A brief discussion of DIF and methods of implementation is followed by an illustrative example from a program evaluation, using an attitude-towards-science scale with 1550 children in grades one through six. An item exhibiting substantial DIF with respect to gender was detected using the Mantel-Haenszel procedure. In a second example, data from workshop evaluations with 1682 adults were recoded to a binary format, and it was found that an item suspected of functioning George A. Johanson differentially with respect to age groups was, in fact, not doing so. Implications for evaluation practice are discussed.

Our recent work with grassroots organizations raises several questions about empowerment evaluation. Critics have focused on concerns such as lack of rigor and objectivity, but few have addressed deeper philosophical issues. One of these issues centers on evaluation standards. Empowered program stakeholders who set the research agenda in the empowerment model may have different ideas from those of the professional evaluator about the structure of inquiry and the use of evaluation. In some cases, the evaluator may question the ethics underlying their choices. One critical issue is

Fostering self-determination is the defining focus of empowerment evaluation and the heart of its explicit political and social change agenda. However, empowerment evaluation overlaps participatory, collaborative, stakeholder-involving, and utilization-focused approaches to evaluation in its concern for such issues as ownership, relevance, understandability, access, involvement, improvement, and capacity-building. A critical question becomes how to distinguish empowerment evaluation from these other approaches. Making such distinctions has become critical as the field debates the Michael Quinn Patton boundaries and implications of empowerment evaluation.

Review essay about Empowerment Evaluation: Knowledge and Tools for Self-Assess ment & Accountability, edited by David M. Fetterman, Shakeh J. Kaftarian, and Abraham Wandersman. Thousand Oaks, CA: Sage Publications, 1996.

It is suggested that there are some serious problems with defining the concept of empowerment evaluation, with its underlying assumptions, and with its proposed justification.

As part of a larger effort by members of the American Evaluation Association (AEA) Topical Interest Group on Evaluation Use (TIG- EU), we undertook an extensive review and synthesis of literature in evaluation use published since 1986. We observe several recent developments in theory, research and practice arising from this literature. These developments include: the rise of considerations of context as critical to understanding and explaining use; identification of process use as a significant consequence of evaluation activity; expansion of conceptions of use from the individual to the Lyn M. Shulha organization level; and diversification of the role of the evaluator to facilitator, planner and educator/trainer. In addition, understanding misutlilization has emerged as a significant focus for theory and to a limited extent, research. The article concludes with a summary of contemporary issues, particularly with regard to their implications for evaluation practice.

This article presents the results of a survey sent to Evaluation Use Topical Interest Group (TIG) members for the purpose of ascertaining their perceptions about and experiences with evaluation use. Fifty-four percent (n = 282) of the 530 members surveyed responded. These respondents agree that the major purposes of evaluation are to facilitate organizational learning, provide information for decision making, improve programs, and determine the merit or worth of the evaluand. Performance-results oriented Hallie Preskill evaluations, formative evaluations, as well as evaluations with a participatory emphasis, organizational learning emphasis, and practitioner-centered action research or empowerment approaches were all viewed as more important today than they were 10 years ago. Survey findings revealed that the most important strategies for facilitating use are planning for use at the beginning of an evaluation, identifying and prioritizing intended users and intended uses of the evaluation, designing the evaluation within resource limitations, involving stakeholders in the evaluation process, communicating findings to stakeholders as the evaluation progresses, and developing a communication and reporting plan. This survey represents a comprehensive effort to understand TIG respondents' views on evaluation use and should help further discussion on developing and advancing our theoretical and practical knowledge.

While there is growing interest in employing mixed methods in evaluation research, there are few documented examples describing how to implement this in practice. This article describes the use of a mixed method approach to evaluate a nonprofit agency that provides organizational consultation and other support services to nonprofit organizations in Israel. It uses conceptualizations proposed by Greene, Caracelli, and Graham (1989) and by Rossman and Wilson (1994) to discuss the unique benefits of such methodology and also addresses the limitations and disadvantages of mixed method evaluation.