Abstract
One of the cornerstones of academic legitimacy is the concept of peer review, in which any book, journal article or scholarly exploration that gets published is first assessed by academics from the same sphere of expertise, who are best placed to understand that work. This has not transferred into mainstream development practice, where most development projects (even those being implemented by the urban poor themselves) are not assessed by community groups and NGO supporters who are their peers, but by outside professionals who visit the project briefly. Although these professionals have no expertise in living in informal settlements on very low incomes or avoiding eviction or negotiating with local governments, they pronounce judgement on the project. These supply-driven kinds of assessments and the principle of “judgement by neutral outsiders” does not fit with the concept of demand-driven development processes that are implemented in different ways by different groups in different places, in response to very different local contexts, needs and capacities. The implementation of the Asian Coalition for Community Action (ACCA) programme has sought to build a new, more horizontal system for assessing, learning from and refining the hundreds of projects it supported in different countries. Teams of community leaders, and their partner NGOs who are actively implementing their own ACCA projects, assess the work of their peers in other nations through visits to ACCA projects and discussions with the people who are implementing them. This paper describes the six assessment trips organized so far and how this more demand-driven assessment process is helping adjust and correct problems in the implementation processes in various cities. This has also opened up a large new space for two-way learning, sharing and building mutual assistance links across Asia, and helping expand the range of what community people see as possible.
I. Introduction
This paper describes the assessment programme of the Asian Coalition for Community Action (ACCA) whose projects and initiatives are described in other papers in this issue. One of the characteristics of this programme’s three-year implementation has been the way it has been assessed as it goes along. When the programme was being designed, we looked at the kinds of assessment that usually go with conventional, supply-driven development projects, in which outside professionals are hired to assess the project, with or without the participation of the communities and implementing groups, according to some tick-list of objectives and outputs that were agreed upon in the original project document. These supply-driven kinds of assessments, and the principle of “judgement by neutral outsiders” that they represent, did not fit with the concept of demand-driven development processes that are implemented in different ways by different groups in different places, in response to very different local contexts, local needs and local capacities.
So a new kind of assessment process was developed that sought to build a new, horizontal system for comparing, assessing, learning from and refining the ACCA projects in different countries through a series of intense visits to ACCA projects within certain countries, and discussions with the people who are implementing them. These are “joint” assessment trips, since the visiting teams include a mix of community people, support professionals and sometimes a few supportive local government officials. Because they are all intensely and “jointly” involved in implementing their own ACCA projects back home, these visitors come with all kinds of questions, doubts, problems and ideas. They find themselves in a different country, seeing other urban poor community people also undertaking projects – some similar, some different. They may be critical of some of what they see, but they will learn from other things and be inspired to bring the idea back home.
In these ways, besides helping to adjust and correct problems in the implementation processes in various cities, the assessment trips also open up a large new space for learning, sharing and building mutual assistance links across Asia, and help expand the range of what community people see as possible. The learning in this new “university” is not academic or theoretical – it is rooted in action and in a shared belief that community-driven processes can bring about significant, structural change. In addition, this learning is not something that happens only at the end of projects; rather, it is taking place constantly, and most of the projects being visited on these assessment trips are all still in process and are incomplete. The objective is not to assess the neatly finished project but to bring a rich element of communal learning and communal adjusting and sharing into the process of implementation as it takes place.
To date, six assessment trips have been organized and the local groups who hosted them organized each a little differently. In the Philippines, Vietnam, Mongolia and Nepal, the visitors went to all the cities and projects together, in a single big group. In Cambodia, the visitors were divided into three groups, who went to cities in different parts of the country and then converged on Phnom Penh for more project visits and a joint meeting where each group presented. In Sri Lanka, the visitors started out together in Colombo and Moratuwa; they then divided into two groups, each one going to different cities, and then met up again in Colombo at the end to report back. All these visits were intense, tightly packed with project visits, discussions, meetings with local government officials, ribbon cuttings and ground breakings, with all kinds of trips by van, bus, train, air and boat in between. When the local groups who host these visits know that this big “international” team is coming to visit and assess their projects, there is a lot of preparation, itineraries to set, arrangements to be made, food to be organized – and perhaps a little tension that comes of wanting to show what they are doing in a way that they are proud of. The six assessment trips are outlined below.
Reports have been prepared on all these assessment trips, for the donors, for the website and for circulating among friends around the world.(1) But the rich interactions between people and the profound learning that takes place during these trips is something no report can adequately convey. The assessments have helped broaden and enrich the ACCA process, and in this paper we look at several aspects that have brought a new dimension to the process of change by people in the Asia region.
II. Peer Assessment, Peer Learning
One of the cornerstones of academic legitimacy is the concept of peer review, in which any thesis, dissertation, book, scientific study or scholarly exploration that gets published is first assessed by other academics working in the same field, who read the work and give their critical comments on it – sometimes confidentially, sometimes openly. The idea is that only those with a similar area of expertise, from a similar background and with a similar language of understanding can be in a position to truly understand a work of scholarship, to assess its good and bad points and to determine whether or not that work is legitimate. The concept of assessment by one’s own peers embodies a logic that is undeniable – for who would ever imagine that a Chinese potato farmer, for example, or an Amazonian tribal chief would be in a position to pronounce judgement on a Harvard academic’s PhD thesis about the Great English Vowel Shift?
This quite reasonable idea has not transferred into mainstream development practice, however. Development projects – especially those being implemented by the urban poor themselves – are usually assessed not by the community groups and NGO supporters who design or implement them, nor by other community groups and NGOs who do similar projects, but by outside professionals who are hired to visit the project briefly, put the people through their X-ray to determine whether they are doing properly the thing they promised to do, and whether this is right or wrong according to all the prevailing development theories. Then, based on their knowledge and their opinion, they pronounce judgement and fly home to write up the report.
These visitors are not the peers of the poor communities or the development teams that are actually carrying out the work. Their expertise is different; their educational and class backgrounds are different. They have never lived in an informal settlement, or experienced eviction or living on the margins of illegality. They have never negotiated with a difficult city government official to get secure land tenure or collectively designed and built their own new housing. Nor have they ever been part of a savings group or managed development budgets with their neighbours. On the contrary, they are usually well set up with secure jobs, good salaries and nice living arrangements. Yet on the strength of a few years of PhD research or with some other research experience, they are deemed to be “experts” on community development and therefore qualified to assess projects of all sorts around the world.
In the terms of academic peer review, this is something akin to getting that Chinese potato farmer to review that Harvard PhD thesis or the academic telling the farmer how he should grow potatoes. Yet this kind of practice remains common in development and is seldom questioned or challenged as making no sense. Almost all projects now have built into them an “assessment” component or a “monitoring and evaluation” section, in which budget and time are specifically set aside to facilitate this kind of review by outside “experts”. Accountability for this kind of assessment is upwards, towards the donor agencies that provided the budget, and not downwards, towards the poor communities whose lives the project was designed to improve. So the terms of the assessment are usually to make sure that the money was spent properly, to do what the project promised in the original proposal, and not whether the project brought about significant change in the lives of the poor in that context.
Assessing external assessments
Thus, it is not surprising that when the time comes for one of these assessments, both communities and their NGO partners brace themselves for an unfriendly onslaught. Under these terms, the assessment process becomes something akin to a policing, a judgement from above. And very often these “experts” fail to understand the real substance and quality of a complex, community-driven process – even as the “evaluations” are couched as visiting “experts” as neutral observers, called in “to facilitate a process of self-reflection” and to help groups “consolidate their learning”.
In the ACCA programme, we are trying to make assessment a more horizontal process, in which it is community people, their partner NGOs and the city – the “doers” who are actively implementing their own ACCA projects – who assess the work of their peers in the region. We believe these peer groups can use the assessment as a learning opportunity: they see what people are doing, they check, they compare, they analyze, they discuss and they learn together by seeing problems and solutions and by understanding how real politics can be made to work by and for people. And in the process of assessing others, these active groups assess themselves, so the assessment process is two-way, with lots of learning on both sides.
This two-way peer learning works in several ways. People have to want to learn in order to learn anything, and all the community people on these ACCA assessment trips – both the visitors and the visited – are at some stage of planning or implementing their own projects and their own citywide upgrading process. So they are eager to learn, hungry for ideas, looking around with eyes wide open. That hunger only comes when people are fully involved in the complications of actual citywide upgrading and actual housing projects, and so they are full of questions and ideas. At the same time, when community people and their city teams are able to do something, achieve something or make some change, they want to show and tell everyone about that achievement and make a full report, because they are proud of what they have accomplished. When this hunger to learn is combined with the hunger to tell, it intensifies the two-way learning between peers exponentially, and this is something we see happening all the time on these assessment trips.
“We, and our judgement, and all mortal things go on flowing and rolling unceasingly. Thus nothing certain can be established about one thing by another, both the judging and the judged being in continual change and motion.”(2)
Another important aspect is that community people are much better than the outside “experts” at being respectful of the communities they visit and bringing out their points of praise and criticism with a great deal of tact and understanding. This means they might actually be listened to and be useful in improving their understanding. When the visitors are from informal settlements themselves, and are struggling to do their own projects, they appreciate all the difficulties low-income communities face when they try to bring about any kind of change. They know from intimate experience the adversarial forces people have to negotiate with when they stay in informal settlements, and they can pick up on the complicated internal politics, leadership problems and stagnation that communities struggle with in developing and implementing these upgrading projects. All these layers may be difficult for foreign “experts” to understand or to factor into an assessment of the project, but poor community people can pick up on these things almost instinctively and share stories of their own.
This does not mean that community people are positive about everything; they are also critical of what they see. They challenge and argue and disagree, and the discussions that take place often turn into lively debates. But since the debaters see each other as friends who face the same enormous difficulties, they’re diplomatic about it, and find gentle, friendly and relaxed ways to get their views across. They also have the social sense to know what is appropriate to be aired in discussions with the community and what is best kept for internal reflection sessions later on.
III. Self-Assessment and Mutual Assessment
The ACCA assessments and the long experience of horizontal, people-to-people exchanges that preceded and inspired them offer proof that active urban poor communities and their city teams can be among the most insightful, subtle and constructive peer reviewers of the development projects being undertaken by other doers – and also by themselves. Learning how to look critically and constructively at their own development projects and processes (“self-assessment”) is perhaps the most important assessment of all, and one that should be part of any community-driven process. Ultimately, urban poor communities and their networks and allies in a given city are on their own, with little infusions of outside help and outside resources that may come and go. It is their process, they have to manage it and they should be the ones to keep assessing and adjusting it accordingly. So a habit of self-assessment is worth building. Outsiders who come to visit – even outsiders who are peers – can observe, comment, suggest and assess, but after those points have been made, the local groups are always allowed to make the decision about what form their change will take – because it is their change. This is a fundamental aspect of these assessment visits, that the visitors have to respect what the local groups do and how they do it.
To strengthen this self-assessment process and embed it in the larger learning, we sent e-mails to the implementing groups in each country to be visited before each of the assessment trips and suggested they organize some reflective self-assessment internally, at city and national level within their country, with their development partners, before the visitors arrived (Box 2). The idea was to bring together some of the key groups and community leaders – even if it was only a few key leaders and some of the people in the support organization in each city – to share their views, describe the things they felt had gone well with their ACCA process and what had not gone so well, and generally review what they had been doing. The idea behind trying to initiate these pre-visit self-assessments was that when the international groups arrived, the sharing would be enriched and more focused.
Self-assessment in the Philippines
Once a people’s process has been able to internalize a process of reviewing and reflecting on what they are doing, it may not matter whether their discussions ever get written up and shared. In many of our meetings, when we ask people to look back and assess this and that in a discussion, they invariably bring the truth and the main points to the discussion. Even if these points never get written up, they get understood and absorbed. Normally, meetings allow people another space in which to review, share, reflect and decide how to go forward.
IV. The importance of Seeing Projects
The assessment trips are planned around a series of visits to ACCA projects. We can listen to new project proposals being presented at the ACCA meetings and later read the reports and look at the figures the implementing groups send; but it is only by actually visiting the projects that we can begin to understand how these groups are thinking and working, understand their systems and their context, understand the weak points and the strengths in their strategies, and understand how the change is taking place. This understanding is possible because we go to the ground on these assessments, as a group: we see the city, meet the communities, meet with the mayor or the local government, visit the projects, talk with all the actors who took part in the process and observe the dynamic between them. This seeing and discussing of concrete actions on the ground is probably the most powerful and most immediate way to help us understand all these issues more clearly.
The projects themselves also tell us a lot – especially projects that are still being implemented, where all the complex aspects are in play. These show the politics and the nature of the relationships; the power of the people and how the finance systems work; how the groups conceptualize the housing. If the management system or the finance isn’t being handled correctly, this will show up in the details of the project. If a support agency does not really understand the strength of the people or the power of community control, this will show up in the project. It is not a question of a project being done correctly or incorrectly, but of what concepts and relationships inform it. The proof of all these issues is in the project. We have often found that a single project speaks for the larger city process: if there is a weakness in the system in one community we visit, that weakness will often reflect the larger system of implementation and will appear in all the other ACCA projects.
V. Seeking a little more Analytical Rigour in the Assessment Trips
The Asian Coalition for Housing Rights that initiated the ACCA programme has a long history of supporting and organizing “horizontal” exchange and exposure visits, where people from poor communities (and their support organizations) in one place travel to other places to see, to share, to learn and to support. Thirty years ago, the idea of poor people getting on a plane and travelling to another country was seen by many as preposterous and radical,(3) but now these kinds of people-to-people exchanges happen all the time, and exchange has become one of the key tools of learning, sharing and solidarity building within people’s movements in Asia. These assessment visits are more than an exposure visit, though. The groups are not just going to see and learn, but also to assess a set of ACCA projects and assess the community and city process within which these projects are being implemented. This should involve a deeper, more serious and more focused reflection on the way people are doing things in the cities they visit. When a community – or a network of communities – in a city does something important and new, they need at some point to reflect, review and assess this before they go forward. Assessment in this sense is positive and not to be feared – especially if it can be done with the assistance of friends from around the region. If the national group that hosts the assessment visit understands the meaning and the value of that, they can facilitate a deeper comparative discussion between the local groups and the international visitors.
The housing project in Salyani
The assessment trips usually start with a meeting on the first day, to review the schedule of visits and the travel arrangements and to discuss what this assessment process is about and how it differs from an exposure visit. On each of the six assessment trips so far, attempts have been made to build a little methodology and more structure into the visits by allowing the participants (both visitors and locals) to divide themselves into three groups, with each group taking charge of looking more rigorously at a certain key aspect of the projects to be visited, gathering that information and then reporting back to the group:
In practice, these attempts to focus the assessment visits on the programme’s key objectives and impose a little analytical discipline on a messy, vital and human exchange have never worked very well, at least not in that way. When people are on-site, looking at the projects and talking with the people, the questions and answers come from every direction and angle. But even though the participants may not restrict their questions to aspects related to their group assignment, they do create an energetic process of comparing what they are seeing with what they are doing back home, mostly in the informal small group discussions that take place during the project visits. When people see something different, their first reaction is: “Why do you do it like that?” Then they listen to the reasons people have for doing such a different thing and they try to understand those reasons. During that process they start thinking, and thinking makes new ideas take seed in their imaginations. The sequence is invariably
These assessment trips provide a structured opportunity for much unstructured interaction between community people and their city teams, who are intensely involved in doing their own projects. The main characteristic that interaction takes on is comparative: “We built our road like this, so why do you build your road like that? Why don’t you do savings that way? Why do you build your houses like that?” To understand something by comparing it to something similar is a fundamental method of analytical thinking, and when people from low-income communities visit housing and upgrading projects that their peers in other communities are implementing, and ask their questions, form their opinions and make their comparisons, they are doing the same thing that scientists and academics do all the time. This achieves one of the pedagogical goals of the ACCA assessment process, which is to get people to analyze and understand what they are doing in a larger context, to break out of the isolation of “my community” and “my city” and see their struggles and their projects in a wider context and as taking place along a very broad spectrum of new possibilities. Comparison is one of the simplest and most direct ways of doing this.
Comparing loans and grants
One of the reasons why it is external “experts” who are tasked with assessing development projects is because of the assumption that low-income people cannot analyze their own projects or the projects of others and cannot assess their effectiveness in a larger context that weighs the project investments against project outcomes. But this can be challenged. Poverty
During or after the visits, reflection sessions allow the visitors to give their impressions of the projects they have just seen. Sometimes, there is time for a summarizing review at the end, from which key conclusions can be drawn. But it is not always possible to record every conversation or interaction even though this is where the real assessment is actually taking place, where the substantive two-way learning is happening, and where the adjustments and new perspectives are being created.
VI. The Question of Accountability and Ownership
Like the larger projects they are organized to evaluate, assessments can also be divided into those that are supply driven and those that are demand driven. If an assessment is driven by the supply side, accountability is all upwards, to the development agency and to the donor that is funding it; and the questions will include: “Have you done what you promised to do in the funding proposal? Have you spent the money according to your budget plan? Why don’t your actual outputs match your projected outputs?” But if an assessment is driven by the demand side – by communities and their real needs – accountability is downwards, to those communities whose lives, housing conditions and position in the city the project is trying to improve. In this kind of demand-driven assessment, the questions will be quite different: “Why do you buy the blocks in the market when you can make them yourself much cheaper? How did you get that mayor to be on your side? How did you get that land? Why are there only a few families in the community savings?”
Using the assessment to work on weak spots
A revealing example of this supply vs. demand distinction is in the questions asked about the issue of community finance. When community people on an assessment trip visit a city and look at that city’s savings process and city development fund, what kinds of questions might they ask? A supply-driven line of questions would focus on the interest being earned on loans, as a technicality, and how much the fund is growing from that interest. A demand-driven line of questions might ask how that interest is being used creatively to address other needs in the community, such as supporting a welfare programme, subsidizing network activities or administrative costs, building a fund to cover late loan repayments, or adding to the fund’s lending capital.
This brings out the question of ownership. One of the key characteristics of the ACCA programme is ownership of the development process by local communities and local groups. In whatever activities the programme supports, the local people should be the owners, because ultimately it is their city, their problems and their process of change. The tools the programme offers are designed to assist them to build and strengthen their own process of change. In other words, the ACCA programme participates in their change process, not the other way round, as is sadly the case in most development programmes. With the ACCA assessment process, the emphasis is likewise on ownership. When community people and their partners from several countries go together and do the assessing as a team, it becomes a joint assessment process by the whole region and it belongs collectively to the people who are implementing the programme and using it to bring about real change in their cities.
This question of ownership is quite important because it defines the nature of relationships, and those relationships can constrain or liberate a people’s process. When a development agency on the supply side feels ownership of a project, it is to be expected that they would also feel ownership of the budget and feel justified in demanding to know whether the implementing groups on the ground have spent their money properly and carried out the project according to that development agency’s rules and objectives. This is the form most assessments take because they come out of a sense of ownership by the donors and a set of relationships that are vertical and hierarchical. The joint assessments within ACCA seek to re-cast those relationships in a more horizontal, more collaborative pattern, in which people who are doing things on the ground go together on a joint mission, to see and assist the change process of others who are like themselves and who are part of a larger people-driven development movement they all own together. So the design of the assessments has sought to allow the people on the ground (both visitors and visited) to feel responsible and important, to feel ownership, because the assessments are part of their movement and the relationships are horizontal.
The various tools and budget support that the ACCA programme offers are something that the Asian Coalition for Housing Rights, as a regional coalition, provides to the various groups, with trust, to allow them to go ahead with their larger process of change, which they own themselves. That task is important and substantive, so those groups very seldom betray that trust. These joint assessments follow a system of trust, where the work is done together, as a movement, and we jointly see whether things are going well or need to be adjusted; but we do it with trust in the groups who are the real owners of that change process.
VII. Support organizations and Governments also Learn on Assessment Trips
On all of the assessment trips to date, the visiting teams have been made up mostly of community leaders, but they always include a few representatives from their partner NGOs, who come not just as translators and facilitators but also as learners and assessors; and the assessment trips have also opened a large new space for learning and growth in that intermediate support layer.
Some of the visiting teams have also included local government officials who are working closely with the communities in their cities. All the same principles of expanding possibilities and learning happens with these officials, just as it does with community people. These regional assessment trips carry a little more weight than a local exchange trip in winning over these officials. In most cases, when they return home, they have doubled or tripled their energy in supporting the community-driven process. In Cambodia, the governors of Kampong Cham, Koh Kong and Serey Sophoan provinces have taken part in assessment visits, and all have become strong allies of the people’s process in their provinces and all have helped obtain free land for the people for housing. The CEO mayor of Bharatpur in Nepal, Ramji Prasad Baral, joined the Nepal team on the assessment trip to Cambodia and was so inspired by what he saw that on his return, he found five million rupees (US$ 70,000) in the municipal budget to offer as seed capital to the new Urban Community Support Fund, which was launched two months later, during the Nepal assessment trip. On that same Nepal assessment trip, another officer from the city of Dharan joined the team on all the visits, and during the course of the assessment he persuaded his CEO mayor (by mobile phone) to find municipal funding to start a similar community fund in Dharan. In Mongolia, district governors from Bayanchandmani, Tsenhermandal and Tunkhel accompanied the visiting team for the whole Mongolia assessment and have been key allies in the ACCA process in those districts. Likewise, in the city of Vinh, the vice-mayor, Nguyen Van Chinh, who has been instrumental in convincing the city to revise their building standards after the success of the ACCA housing project in Cua Nam ward, spent the whole day with the visiting assessment team and told them repeatedly and enthusiastically: “The people are my teacher.”
Footnotes
2.
de Montaigne, Michel (2005), The Complete Works, Volume 2, Chapter 12, translated and edited by Donald Frame, Everyman, London, page 553.
3.
4.
Jacobs, Jane (1969), The Economy of Cities, Vintage, New York, 288 pages.