Somalia: External Evaluation Consultant

1. Background

Terms of Reference concern the internal evaluation process of the PAH project in Somalia “Comprehensive WASH and livelihoods assistance to the most vulnerable IDP communities in Somalia 2016/2017”.

Present in Somalia since 2011, PAH has been specializing in Water, Sanitation and Hygiene activities, as this is the agency's main focus. Nevertheless, we also have experience with cross-cutting programs linking WASH to nutrition and health. So far, PAH has reached to over 110 000 people from the most vulnerable communities in different regions of Somalia. PAH has been gradually expanding activities in Somalia with a priority given to the most vulnerable beneficiaries and locations - especially IDP camps and settlements. In 2011, PAH opened a Country Office in Garowe and field office in Mogadishu in May 2012. In May 2015 we have restructured our mission and withdrew from Puntland to focus on the South Central Zone and WASH emergency response. Currently, our Country Office is located in Mogadishu, and we are one of the few INGOs with the CD and senior staff based in Somalia. We do not apply remote management. Additionally, we have two field offices - in Mogadishu and Jowhar (Middle Shabelle). The established modus operandi provides us with direct access to diverse communities and deep understanding of their needs.

Our primary focus is to provide safe drinking water by rehabilitating crucial water points. In our work we also focus on other WASH components that is improvement of sanitary conditions and raising awareness on hygiene and sanitation. That approach makes our WASH interventions fully comprehensive. All interventions include O&M components, provision of spare parts as well as distribution of hygiene items and WASH NFIs. Since 2011 we have implemented various projects in different locations. We implemented WASH for schools' program designed to support schools with access to water and sanitation by constructing school berkads, establishing school gardens and improving existing sanitation facilities. The project was implemented in cooperation with Puntland Ministry of Education and WFP. In 2012 and 2013 (for 15 months) we were also providing Emergency Nutrition in Mogadishu.

In 2013 PAH became an ECHO partner in Somalia. Our emergency WASH response for IDPs is implemented in Puntland - Garowe and in Mogadishu supported more than 20 000 IDPs in camps in Jowle and Mogadishu (K7-13).

In 2014/2015 and 2015/2016 our ECHO program focused on increasing access to water and sanitation facilities, included also livelihoods component in Jowle IDP camp and establishment of public/private partnership in latrine dislodging, support in establishing waste management system in the camp. In Mogadishu focus of the activities remains on emergency WASH response in K7-13 or other new areas, which face huge influx of IDPs or immediate needs due to secondary disasters, for example we assisted flood-affected communities during the October floods in Jowhar district. In the current ECHO we are focusing on EP&R in Middle Shabelle which is big addition to our WASH emergency response in Mogadishu K-Zones.

The now completed ECHO project was a continuation and development of a comprehensive WASH intervention in IDP camps in South Central Zone with main activities as follows:

• Provision of safe and sufficient water to local communities (rehabilitation of water infrastructure and direct provision of water);Improved access to adequate sanitation facilities (emergency latrines construction, desludging activities and hygiene promotion);

• Strengthening resilience and improving access to livelihoods of most vulnerable HHs in IDP communities (unconditional cash transfers and vocational trainings);

• Strengthening capacity of local of local institutions to react and coordinate response in case of a disaster (WASH DRR);

• Improved emergency preparedness and response to on-set events (EP&R).

The project in questions concerned the main activities of PAH in Somalia carried out in 2016-2017.

2. Evaluation aims and scope

This project evaluation serves to:

• Assess the appropriateness and coverage of the programme in relation to the situation of the affected communities and people in the PAH Area of Operations.
• Assist the programme team in overall analysis of the effectiveness of the programme i.e. the extent to which the results of the projects and the indicators were achieved?
• Together with the team assess the implementation process of the project, its strengths and weaknesses, particularly in relation to experiences, knowledge, good practices and improvements in the quality of humanitarian assistance.
• If possible, identify outcomes and impacts (positive and negative) of the project.
• Based on findings draw recommendations, lessons-learned for the future programmes of PAH and if appropriate, suggest changes in the project design in the future.

This project evaluation is guided by the OECD-DAC evaluation criteria adjusted for humanitarian assistance and adopts standards in evaluation of humanitarian action. Specifically, the evaluation of the project should include the criteria of relevance, coverage and effectiveness.

The focus of this evaluation is laid on three evaluation criteria and three aspects. Firstly, it aims to explore the understanding of appropriateness and coverage of PAH activities, to review the criteria for selecting the locations of the project, to verify and/or document the selection criteria of the beneficiaries and provide an analysis of the practices in this field.

Secondly, this evaluation serves to assist the programme team in analysis of effectiveness of the project in terms of attainment of objectives and indicators set out in the logical framework. With that, the evaluation process should assist the project/programme team with reporting the indicators set out in the project and particularly conduct quantitative analysis of selected indicators which are collected through household interviews. The evaluation report should include triangulation of findings from various sources and include qualitative research.

Thirdly, for the purpose of continuous learning and quality improvements, evaluation should depict implementation process of the project, its strengths and weaknesses. Particularly, this part should focus on analysis of documentation, practices of experiences, knowledge, good practices and improvements in the quality of humanitarian assistance that have taken or have not taken place at PAH. This part is meant to involve the team, to identify the various sources of knowledge such as Post-Distribution Monitoring, Evaluation Reports and other relevant files and to identify ways how these findings and recommendations were shared, promoted, implemented and followed-up in the course of the project.

In addition to that, if in the course of evaluation and qualitative research, outcomes and impacts are identified, they should be clearly identified and reported.

The main users of this evaluation will be: PAH Emergency Response Team, the M&E Coordinator in the PAH HQ, project coordinators in PAH, other internal stakeholders at PAH and wider external stakeholders of the project in Somalia and beyond.

3. Candidate evaluation questions

In order to respond to the aims of evaluation candidate evaluation questions were formulated. The list is not exhaustive and should to be reviewed by the evaluator upon commencing the work.

With the focus on relevance and coverage:

• How were the specific locations selected within the PAH Area of Operations?
• Who participated and who benefited from the activities/project?
• Were there any groups who were excluded from the assistance?
• What were the selection criteria for the beneficiaries? Did the selection of beneficiaries include the vulnerable communities and people? How were the vulnerable groups defined and selected?
• Did the vulnerability criteria change and if yes, how?
• Was the WASH coverage comprehensive?

With the focus on project achievements and effectiveness:

• To what extent were the project indicators attained?
• Is the water infrastructure in use and maintained? Are there any failures reported and where they fixed?
• The level of dependency generates by the PAH project should be assessed. For instance: are the service users sufficiently involve in the management of the WASH service? Appropriateness of the community approach/strategy of community involvement (with gradual hand over of tasks…) rolled out by PAH? Modalities or/and exit strategy effectiveness and relevancy?
• To what extend does the community feel ownership over the infrastructure delivered? How is it executed?
• What is the exit strategy and is it documented and respected by the team? Is there an exit strategy for introduction of water fees?
• Whether water trucking was a timely and sufficient response, what were the options, whether they were documented and analyzed?
• To what extent has the knowledge of hygiene practices and waste disposal increased?
• To what extent has the practice of proper hygiene behavior increased?
• To what extend has access to livelihoods improved among beneficiaries of the UCT and how sustainable was the change
• What was the outcome of the vocational training?
• Were the skills in which people were trained marketable?
• Was the training of sufficient quality and time to give beneficiaries’ an actual employment opportunity?
• What was the outcome of DRR interventions and how durable is it?
• Was the EP&R prioritizing the most vulnerable populations with highest needs?
• Was the hardware (construction) and software (behavioral change, capacity building) well synchronized in time?
• Was the intervention cost efficient? Were the best modalities of the intervention used to achieve the results?

With the focus on learning:

• What are the outcomes or impacts of the trainings for the local authorities and institution representatives?
• How were the affected communities and people involved and were they satisfied with their participation?
• Was the feedback and complaints mechanisms in place and was it effective?
• What feedback and complaints were collected?
• What is the feedback from the WASH cluster and other partners? How do they assess coordination?
• How PAH learn from past experience and how far they use such findings to improve the quality, sustainability and effectiveness of their interventions?

With the focus on PAH performance

• How do beneficiaries assess PAH staff performance in terms of communication – were they well informed about the planned intervention, changes and interruptions, could they freely provide feedback and were the ideas taken into account?
• How often did PAH staff visit the sites
• Did PAH staff encourage people to participate in the activities?
• In case of difficulties how was the staff handling the situation
• Where there any cases where beneficiaries felt abused, exploited or treated unfairly by PAH staff?
• Could the beneficiaries talk freely to PAH staff without the presence of gatekeepers
• From the perspective of beneficiaries were the selection criteria fair and clearly explained?
• Was the assistance delivered timely?
• Did the interventions take into account also the centrality of protection?

The final evaluation questions will be agreed between the MEAL Manager, Programme Coordinator and the selected evaluator based on time and resources dedicated for this evaluation.

3. Methodology, timeline and deliverables:

The suggested methodology includes the following initially identified methods to reach the aims of evaluation:

• Desk Review of all relevant information and project documentation
• Testing and finalization of methodology for qualitative and quantitative research
• Data and quality control in the quantitative survey based on HH interviews
• Analysis of qualitative information based on FGD and KII
• Case study analysis for section of target locations and the beneficiaries
• Benchmarking of data with standards in the sector
• Review of data and the project with the project team
• Drafting, consulting and finalisation of the evaluation report.

The timeline and the key milestones and deliverables of evaluation are as follows:

1^st week (tentatively starting from end of April)

Signing the contract and exchanging information

Updated ToR

1-2 weeks

Desk review, analysis of documentation and planning

Evaluation plan, with clear methodology, evaluation tools, sampling methods and action plan; evaluation matrix

2-3 weeks

Preparation for quantitative data collection and analysis; completion of questionnaire and operational provisions; data collection started

3-5 weeks

Data gathering; review of quantitative data; data control

Cleaned database for analysis

5-6 weeks

Analysis of quantitative data and preparation of initial findings

Draft report for HH survey

6-8 weeks

Visit to the mission in Mogadishu (approximately 10 days); presentation of initial findings; participation in qualitative research; participatory review of project activities together with the team; analysis of field team reports

Minutes of the meetings; summary of conclusions and agreements

Deadline: 16^th of June

Drafting of final evaluation report; incorporation of the final data and analysis collected at the later stage

Draft evaluation report

Deadline: 23^rd of June

Finalization of the report and the tasks

Finalized and completed evaluation report, with annexes and methodology part.

Team and implementation

Given the context of Somalia and the learning-focused design of this evaluation, the evaluator will be accompanied and assisted by PAH personnel from the mission and HQ. This includes but is not limited to:

• Sharing of project documentation and field reports;
• Briefing on past and current activities in the project locations;
• Prepare the team for quantitative and qualitative research;
• Assist-in verifying the evaluation questions;
• Support in necessary logistics depending on the needs for this assignment.

Evaluator will be tasked to combine the data and information collected by the filed team under Post-Intervention Monitoring, cross-check with primary data collected during evaluation and draw concluding judgements.

4. Qualification and requirements

PAH is looking for candidates who meet the criteria outline below.

Qualifications:

• Higher education in related field, minimum of Bachelor degree (higher degree is desirable).
• Experience in NGO and humanitarian assistance sector and projects (5-7 years).
• Experience in similar in M&E, MEAL tasks and assignments (minimum 2 years).
• Professional experience and knowledge of context of sub-Saharan African countries.
• Knowledge of IT software such as Kobo, IformBuider, SPSS or other relevant.
• Knowledge and experience in projects in WASH and/or livelihoods.
• Excellent skills in oral and written English is a must.
• Excellent analytical skills, attention to details and ability to synthesize reports and information.
• Excellent experience in data collection and analysis, data management and reporting for both qualitative and quantitative research.
• Capacity to carry out Data Quality Control.
• Knowledge of other languages, especially Somali is an asset.
• Previous experience of documented and positive work in Somalia is an asset.

Desired competencies:

• Strong focus on results and completing of tasks.
• Good organization of work and process management skills.
• High intercultural sensitivity and competencies.
• Strong skills in communication and cooperation.
• Excellent understanding and sensitivity in relation to humanitarian principles.
• Strong focus on quality and reliability of work.

5. Budget and practical preparations

Total budget for all costs related to this evaluation is estimated for 12 000 USD. This should include all international and local transport costs, hiring enumerators, accommodation, honorarium for evaluator and all other related costs. Please, include all categories in the budget and send it together with the application.

Accommodation during the visit is recommended to take place in Mogadishu Peace Hotel 1 to ensure smooth communication with the PAH team.

PAH may also assist in estimation of operational costs related to this assignment.

Candidates for this position should express their interests and send their applications before 20^th of April 2017.

To apply, please send:

1. Your updated CV in English.

2. Breakdown of all costs for evaluation.

3. Short outline of how you will deal with this assignment and respond to the provisions in this ToR.

4. Brief, maximum 1 page overview of anticipated limitations in this research and mitigation measures.

5. Exemplary evaluation report or analysis prepared in the previous assignments.

Please, send your documents to: desk.som@pah.org.pl.

Only complete applications received before the deadline will be considered. The organization reserves the right to reply only to selected candidates. **