Towards Better Project Evaluation for ACORD and CIEDEL Trainees
From 1st to 19th June, ACORD has engaged 18 junior evaluators (17 Kenyan and 1 Rwandese) aged between 28 and 45 years in an extensive 3-weeks practical training on project evaluation. "I look forward to improving the Maseno university programme and methodology in M&E based on the new skills that I have acquired", asserted one participant from Western Kenya.
This has been coupled with effective evaluation of 9 small projects funded by the French Embassy in Kenya under the Solidarity Development Fund (FSD). For this exercise, ACORD has been partnering with the French-based Research and Development Training institute CIEDEL (French acronyms for Centre International d'Etude pour le Développement Local) of the University of Lyon in France, and French Embassy in Kenya. The training held in Kenya was inspired by both ACORD and CIEDEL development approach, considering qualitative data of social change more important than the usual econometric and statistical evaluation methodologies.
ACORD and CIEDEL also believe that evaluation goes beyond simple method of measuring results. An evaluator is an actor and therefore capable of influencing social change, in a process that leads to greater commitment of the stakeholders who participate in it. In addition, the ACORD and IEPALA evaluation approach has implications because it makes transparent the interests of those involved in a project and also questions the activity as much on its purpose as on its results.
Indeed, many development projects across the African continent continue to face challenges of sustainability due to weak support to crucial monitoring and evaluation components. Additionally, they lack adequate beneficiary capacity development for them to be able to continue alone the process at the end of the project period. This management gap often results in wastage of capital following neglect and poor maintenance or lack of community involvement.
However, ACORD experience working with different projects demonstrates that proper follow-up, monitoring and evaluation can produce significant changes and inject vigour and active service in project functionality. For example, ACORD's findings, working with Parish orphans care associations in Uganda, have shown that involving the local communities in the monitoring process added significant value to the project's dynamism.
During theoretical session, the team settled down to actively engage in evaluation theory and methodology for a period of one week. Among key topics discussed by the participants included the Evaluation Path (identifying the stakes of each stakeholder, defining evaluation, criteria and evaluation questions and indicators), aspects of impact analysis, evaluation planning, data collection and processing, evaluation report writing and feed-back.
On 7th June the groups (with two evaluators each) set out for two-week long field evaluation assignments to visit 9 designated projects in different parts of Kenya. The projects covered included those located out of Nairobi such as the Kiangurwe Hydro-Power project in Kiandai, the The Rabai Kaya Conservation in Rabai, the Old City Conservation in Mombasa and a primary school construction project for Kogema kalanya Primary School in Homa Bay. Projects within Nairobi included the Urban Farming in Kibera, the Sport Coach Training Centre in Lang'ata, the Plastic Waste Recycling in Makadara, the Community Development Centre in Huruma and the WOFAK Centre for HIV/AIDS Counselling in Kayole.
Field assignments aimed at evaluating and documenting the relevance, effectiveness, coherence, level of participation of all stakeholders, efficiency sustainability and image of the French cooperation in the projects context. The evaluators applied the different data collecting skills including focus groups discussions, open individual interviews, and collection of secondary data to get the necessary information that helped them make their evaluation judgement. Feed-back sessions of the evaluations were organised in each project site involving the different stakeholders, to validate the evaluations findings and collect comments and additional information to feed in the evaluation reports.
To enhance data management, the team members re-convened for 5 more days to exchange feedback on their experiences, engage in comparative analysis, restitution, give recommendations and report back after 2 weeks of field work, community engagement, and project analysis.
Each pair of evaluators provided a final evaluation report of their selected project, and a PowerPoint synthesis. These were presented at the ‘graduation' ceremony presided by the French Ambassador, Her Excellency Mrs Elisabeth Barbier at the French Cultural and Cooperation Centre in Nairobi. Other cooperation agencies present included the European Union and the British High Commission in Kenya. Involvement of other cooperation agencies was meant to share the training and evaluation approach to influence evaluation practices from different donors in Kenya and in the region.
Criteria-wise, out of the 9 projects evaluated, the project for Community Development Centre in Huruma was ranked the best. Its assets included relevance to the local context and to the development priorities of both the French Cooperation in Kenya and the Kenyan Government, as reflected in the MDG targets fro Kenya. This project is also providing good internal and external coherence and has shown good effectiveness and an excellent level of participation by all stakeholders, which is a pillar for sustainability.
"We own the centre; we give suggestions and critic how to run the centre. The Management Committee members are just facilitators", said one of the beneficiaries interviewed. "Whenever we have an activity we are given a training workshop to know how to implement."
Because the project is a direct answer to the Mathare/ Huruma slum communities' needs, the number of community based organisations affiliated to it has increased from 60 in 2008 to 137 in 2009. These CBOs range from garbage collection groups, reformed prostitutes, traditional dancers and football teams among others.
Furthermore, the project has indirectly generated positive effects in the context of Kenya's post election violence as it has served as a peace-building anchor. According to one of the beneficiaries, "this is the only centre that has brought all ethnic groups together, without restrictions"
Overall, most projects demonstrated relevance in terms of their social utility to the local contexts and to the development priorities of both the French Cooperation in Kenya and the Kenyan Government, as reflected in the MDG targets for Kenya. However, some projects have shown weaknesses in terms of coherence.
A case in point is the primary school construction project for Kogema Kalanya in Homa Bay, for which the funds were handed over directly to the constructor instead of the school committee. However, the constructor, who reports directly to the consortium of donors involved in the project (i.e the South African High Commission, The French Embassy and Safaricom Limited) reported that he was only accountable to his contractors and not the school committee.
Other weaknesses witnessed in most projects and documented in evaluators' reports include low level of involvement of all stakeholders in the different projects, no steering committees in the projects that could help increase the ownership of the project by the beneficiaries as well as assuring a participatory monitoring process. Gender balance in leadership and administration in running the projects as well as among the beneficiaries was also poor, with men overwhelmingly taking the centre stage and women receiving second-hand responsibilities. Other weaknesses highlighted by evaluators include lack of technical, financial and organisational capacity building of beneficiaries. Some project operators, it was observed, had not sufficiently involved the final beneficiaries in the management, or had given up their responsibilities, leaving the latter to manage the projects alone.
These were some general observation highlighted by evaluators:
1. The project operators have the responsibility to ensure a minimum of follow-up and support to beneficiaries even after the end of the project period.
2. Participation of all stakeholders in local steering committees including local authorities and local public service departments will help increase the ownership of the project by the beneficiaries as well as assuring a participatory monitoring by all stakeholders.
3. A tri-partite funding contract is a better approach to ensure a proper ‘triangulation' of commitments between the donor, the operator and the beneficiaries.
4. The French Embassy needs to undertake a ‘quick assessment' based on key organisational performance criteria, before funding a project. However, the weakest link coming from the assessment of the project operators needs to feature in the project document as deliverables/aspects to work on, and be budgeted for.
From Training to Practice
The evaluation trainees decided to form a network and join the AFREA (African Network of Project Evaluators) as well as the Kenya National Evaluators Network, to promote experience and information sharing and support each other in development strong monitoring and evaluation consultancies. "I am going to work on improving the university content programme and methodology in monitoring and evaluation, based on the new skills that I have acquired", said a trainee who lectures at Maseno University in Kisumu.
ACORD involvement in this training-action in project evaluation through its monitoring and evaluation and knowledge management coordinator contributes to ACORD's need to share experiences with the external environment. ACORD is promoting policy and practice changes through a combination of practical activities, research, critical analysis, social action and advocacy.
The joint venture ACORD-CIEDEL created for this project has proven to be very effective for both organisations and the approach of training-action will be replicated by ACORD in the 17 countries where it is present. This will build internal capacities of project evaluators, as well as for local partners in these countries. In doing so, each area programme benefits from ACORD's organisational capital in terms of know-how, technical expertise, lessons learned in other countries as well as ACORD's pan-African network for bringing forth issues that require sub-regional or international attention.
For more information about the ACORD-CIEDEL Monitoring and Evaluation project contact firstname.lastname@example.org