TitleTerms of reference: Lead Evaluator for the evaluation of Let Us Learn Initiatives (LUL) PurposeTo provide technical support for the evaluation of Let Us Learn Initiatives in Afghanistan, Bangladesh, Liberia, Madagascar and Nepal, and an evaluation synthesis from the five discreet evaluationsReference Evaluation Timeline April 2014 through December 2014
A. Background
The Let Us Learn initiative (formerly known as the Basic Education & Equity initiative) is a unique private partnership that allows for flexible and innovative approaches to addressing inequities in education access and outcomes. Challenges and barriers to education, particularly amongst excluded and marginalized children and youth have been identified in five diverse country contexts, Afghanistan, Bangladesh, Liberia, Madagascar and Nepal. Each country is targeting the hardest to reach children by sharpening the equity focus in both programming and monitoring of results. 20 million USD have been designated for the period 2011-2014.
The programme is focused on three equity pillars: reaching out-of-school children, expanding girls’ education and improving quality outcomes for learners. Each of the participating Country Offices is working to address inequity through targeted approaches, adapted to address their specific contexts.
Afghanistan: Girl’s education is selected for its innovative approach towards establishing Accelerated Learning Programmes (ALPs) in community-based schools for girls to receive and finish primary education. This initiative provided educational opportunities for young girls as well as providing an innovative opportunity to recruit young women into the teaching profession.
Bangladesh: The project is targeting three primary areas, namely (1) improved access to pre- primary education for school readiness for children 5 years old. The other two interlinked components are: (2) Ability-Based Learning for OOSC children 8-14 years old; and (3) Communication for development and social mobilization for OOSC 5-14 years old. Overall, this evaluation focuses on Pre-Primary Education (PPE) school centers in underserved and marginalized communities.
Liberia: Projects are focused on innovation, girls’ education and improving learning environments. The CO is supporting girls’ education through the Girls Achievement Programme (GEEAP) which provides support to girls in schools through extra-curricular academic and life-skills activities. The WASH component focused on providing infrastructure to improve access to water and sanitation, education on good sanitation practices and supporting hygiene through girls’ clubs.Connecting Classrooms is an initiative to bring 21st century skills to junior secondary schools, training girls and boys on the use of computers and online learning and sharing platforms.
Madagascar: UNICEF is supporting the Ministry of Education to encourage girls to transition from primary into junior secondary school and progress upwards through a dynamic and innovative range of supports. These include scholarships, helping schools to become more ‘girl friendly’, and construction of classrooms to increase space for more young people to attend schools. A special ICT focused programme partners with environmental organizations to build young people’s capacities to share information and learn ICT skills.
Nepal: Multiple pathways to learning are provided for children and youth via both formal and non-formal education interventions. Initiatives aim to improve reading skills, provide access to schooling for out of school girls and help girls in junior secondary school succeed.
B. Rationale for the evaluation
The partners involved in launching the initiative, UNICEF Education, the German National Committee for UNICEF, and the private donors agreed that an evaluation would be carried out at the end of the programme. The evaluation effort was intended to ensure accountability, strengthen any future programming and contribute to the global equity dialogue. In order to capture information across a broad range of goals, a series of evaluation activities are being proposed. Evaluative activity is broadly defined to include evidence gathering, performance monitoring and measuring results. Outside of the technical assessment, consideration will also be given to improving organisational effectiveness by examining the development and implementation process, as well as programme supervision.
One primary focus of evaluation will be to engage all COs and HQ in a learning process around the Let Us Learn (LUL) initiative, documenting new efforts in monitoring for equity, innovations, partnership and good practice. A series of 5 discrete evaluations and a synthesis/consolidation are envisioned, along with participatory products such as testimonials and ‘most significant change’ stories involving beneficiaries.
We are aware that some of these analyses may be be challenged by a lack of baseline data and varied programme approaches across all of the participating COs. Activities should build evidence to fill information gaps and inform any next phase of the LUL, thus retroactive baselines may need to be established to measure change. Mixed methodologies will be utilized given the diverse range of activities taking place across COs but should fall within a global methodological framework.
The audience for the evaluation products will be internal (HQ, ROs, COs), national committees and the private donors. COs will have varying capacity to lead or support evaluative activities in their country. Each country has a small evaluation budget which will be supplemented by HQ. The final evaluation product needs to be completed by December 2014 and Education proposes joint management with the Evaluation Office to ensure well-coordinated and consolidated final products.
C. Evaluation approach and scope
From a global perspective, the objectives of the evaluation are twofold;
To examine the extent to which LUL projects are achieving intended outcomes at the country level; and,
To systematically document new learning generally, and to document new efforts in monitoring for equity, in particular.
The proposed approach is to conduct self-evaluation[1] exercise, to be executed in all five LUL participating countries, (spanning three regions of ESAR, ROSA and WCAR). We anticipate using a common evaluation/learning framework and methodology in each participating country. This will enable comparison across countries, especially in efforts towards monitoring for equity.
Activities included and excluded from evaluation. The self-evaluation study will examine only activities/results that are agreed upon with the respective country offices. Processes related to planning, coordination and monitoring will be included. Selected programme activities along each of the three pillars of the programme will be included as will thematic analysis related to equity and innovation.
Attribution and/or contribution. While evaluating outcomes also often determines if a programme/project has added value to country-level results as stated in the Country Programme, it is often difficult to attribute results to only one source of inputs, actions, or actors in a field where there are multiple actors and inputs. Therefore the evaluation will seek to outline, to the extent possible, LUL’s contribution to overall results through a contribution analysis.
D. Methodology
Sample: A draft stakeholder analysis has been conducted as part of preliminary planning for the evaluation, hence a sampling framework that reflects a range of activities, stakeholders and will be developed for each country.
Design and data collection methods: Data will be collected from multiple sources, using both qualitative and quantitative methods. A menu of methods and sources include a review of programme documents, interviews of key/elite informants, interviews and/or focus groups of important groups of beneficiaries, self-administered surveys, testimonials and ‘most significant change’ stories involving beneficiaries, and direct observations of activities with beneficiaries wherever possible.
· Review of documents and procedure:A desk-based review will be used as an important tool to examine the theory of change, restate and/or amend it to reflect what happened in practice (empirical ToC); assemble evidence of activities or outputs. LUL focal points in each country will avail key documents for a systematic desk review. These will include LUL programme documents (proposals, monitoring frameworks, reports, human interest stories, etc.), sector plans, other education evaluations and assessments as relevant.
· Data Collection:Primary data – to be collected through interviews and focus groups, self-administered surveys, and possibly small scale assessments – will add to the existing knowledge about the activities, and will be used to gauge perceptions of stakeholders on the utility and impact of interventions.
E. Evaluation responsibilities and management
The LUL global self-evaluation and global synthesis will be supported by:
· An international Lead evaluator (ILE) at global level;
· A national evaluator (NE) in each participating country; and,
· Reference groups at the country level.
The international lead evaluator (ILE),contracted by the UNICEF Evaluation Office, will be responsible for leading the evaluation efforts from a global perspective. Specifically, he/she will be responsible for the following:
· develop the evaluation approach and methodology, including the evaluation/learning framework;
· develop a self-evaluation module/guide and other materials as he/she sees fit;
· facilitate a virtual planning workshop for national consultants and focal points in each participating country;
· pilot the evaluation methodology in one country, and effect the necessary revisions;
· review data collection tools and protocols used at the country level;
· undertaking quality assurance missions during data collection in up to two participating countries;
· review country reports - draft and final reports;
· collect additional data at the regional level and ensuring that regional and global perspective are represented in the global synthesis;
· compile the synthesis report, draft and final versions;
· ensure that the evaluation managers (Evaluation Specialist and Education Specialist in New York) are regularly informed of the progress of the evaluation, possible causes of delay and issues to resolve.
The Evaluation Specialist and Education Specialist (in New York) will manage the work of the international consultant, provide overall guidance to the evaluation, and contribute directly to its quality assurance activities.
TheNational evaluator (NE) in each participating country will be responsible for the professional conduct of the evaluation in accordance with the terms of reference, the United Nations Evaluation Group (UNEG) Norms and Standards for Evaluation[2] and the UNEG Code of Conduct for Evaluation[3]. Each consultant will be contracted by the respective UNICEF Office to perform the following tasks:
· review and finalize the evaluation approach and work plan;
· customize, review and validate data collection tools, (the self-evaluation module, which may include questionnaires, interview questions and protocols);
· supervise data collection and data processing;
· design and facilitate consultative workshops and meetings;
· prepare of country reports – inception, draft and final reports, and develop PowerPoint presentations;
· attend and make a presentation at the global reporting workshop; and,
· ensure that the International Lead Evaluator is regularly informed of the progress of the evaluation and possible causes of delays and issues to resolve.
Countries and regions will support the self-evaluation by providing support to arrange for meetings, write letters of introduction and/or accompany consultants to meetings where necessary. Ideally, the evaluation should be managed by the Monitoring and Evaluation Officer (or equivalent) in the Country Office, who will provide overall guidance and contribute directly to its quality assurance activities.
Reference Group:Our recommendation is to establish a reference group in each participating country. Membership will be determined at each level (at HQ by Evaluation Office and Education Section), and within country for each of the five country offices.
The Reference Group Chair is responsible for oversight of evaluation activities, while members will advise on the following:
· Consultant’s terms of reference and work plan, including any adjustments required at inception or other phases
· Approval of all evaluation products, including the final report
· Decisions on a post-evaluation management response, action plan and dissemination strategy, and a consideration of how the evaluation findings are to be utilized.
F. Evaluation processes and products
Evaluation consultants, national evaluators (NEs) and the international lead evaluator (ILE) are each responsible for the following activities and productions:
· Self-evaluation module and other materials for the orientation and planning webinar and facilitating the workshop, (ILE);
· Overall inception report, revised evaluation approach and methodology, outline for the evaluation report, revised evaluation work plan, (ILE);
· Country inception reports, including revised evaluation methodology for their country, outline for the evaluation report, revised evaluation work plan, (NEs)
· Draft evaluation reports, both country reports and the overall synthesis report,(ILE and NEs);
· Dissemination workshop to consider initial evaluation results, (ILE and NEs);
· PowerPoint files for the presentation of final reports (ILE and NEs)
· Final Evaluation Reports, according to the UNICEF House Style and UNICEF standards for evaluation reports (ILE and NEs)
Evaluation Planning Workshop (via Webinar):TheInternational Lead Evaluator will develop initial draft of the evaluation approach, and convene an orientation and planning webinar for the national evaluators, to finalize the approach to the evaluation. The planning workshop is expected to agree on the methodology, including the sampling frame for each country, a set of evaluation tools with necessary customization to each country, and a tentative work plan that reflects all evaluation activities for country teams and for the international lead evaluator.
Inception Reports:An inception report will be produced at the global level by theInternational Lead Evaluator,* and adapted/customized for each country by the National Evaluator. In addition to presenting a short summary of the programme context, these reports will be used to confirm a common understanding of the description of what is being evaluated, and the logic or theory of change for each programme. The report will include,inter alia*,
Evaluation purpose and scope – a clear statement of the objectives of the evaluation and the main aspects of the evaluation
Evaluation criteria and questions – question to be answered by the evaluation, and the criteria which will be used to assess performance, including questions that address critical human rights and gender equality issues
Evaluation methodology– expanding on the methodology section in the TOR, a sampling strategy, a description of data collection methods and data sources (including a rationale for their selection), draft data collection instruments, a discussion on reliability and validity of the evaluation, and a discussion on the limitations of the methodology. This section should include instruments to assess relevant human rights and gender equality aspects.
Evaluation matrix and analysis plan– a mapping that identifies evaluation questions, how they will be answered through the selected methods, and a data analysis plan;
Evaluation work plan and timeline –a revised work and travel plan
Structure for the final reportwill be proposed in the inception report, and may be revised later in the evaluation
Resources requirements –detailed budget for the evaluation, tied to evaluation activities, work plan, deliverables.
The inception report will be 10-15 pages, including the revised work plan, and will be presented at a formal meeting of the in-country reference group, and LUL extended management team (HQ Manager plus Chiefs of Education/LUL focal points in participating countries).
Evaluation Report: The final evaluation report should include, but not limited to the following
· An analysis of concepts, trends and critical themes in innovating solutions for equitable provision of education in respective contexts;
· An analysis of overarching programme goals and themes (OOSCI, equity, innovation, etc.), nothing those with most demonstrable impact;
· An overview of each of the country evaluations, analyzing trends, areas of divergence and any lessons learned;
· An assessment of UNICEF’s mandate, strengths and weaknesses relating to UNICEF’s strategic and programmatic choices for equity, against a set of agreed evaluation criteria;
· An analytical framework which UNICEF can apply to improve the effectiveness of its support for equity interventions, at the country level, and an assessment of the role of the regional and global offices in support of equity in education;
· Derived from the findings of the evaluation, recommendations for improving coordination, and successful scale up of equity focused programming.
The evaluation report should not exceed 30 pages (excluding the executive summary and annexes). Annexes will include the TOR, description of methodology (including evaluation/learning framework), list of background materials used, list of people interviewed, PowerPoint presentations, and workshop materials.
Desired Competencies for Evaluators: Evaluators must each offer the following demonstrated experience, knowledge and competencies:
Significant knowledge and experience of evaluation concepts and approaches;
Good knowledge in gender in the Education Sector, and experience in evaluating equity and education innovations;
Facilitation skills, particularly design of stakeholder consultation exercises as well participatory methods;
Strong quantitative and qualitative data collection skills;
Strong quantitative and qualitative data analysis skills;
Excellent language and communication skills in English;
Demonstrated report writing skills, in English; and
Computer literacy in Word, Excel and PowerPoint;
The international lead evaluator must have experience of working cross-culturally in development, and demonstrated capacity in managing evaluation teams.
G. PROPOSED RESOURCE COMMITMENTS
It is anticipated that the resource commitments would as follows:
· 100 person days for the International Lead Evaluator
· 60 person days for Country Level Evaluators in each country
· 22 person days for the LUL focal point in UNICEF (for recruiting, facilitation for, and managing national consultants).
· Travel costs for one international mission and 4 days DSA for consultants (reporting workshop)
· Travel costs for one international mission and 4 days DSA for UNICEF/LUL focal point or UNICEF M&E Special (reporting workshop)
H. PROPOSED WORK PLAN AND TIMELINE
Tentative Schedule: A tentative schedule for major evaluation activities, and expected timeline is as follows:
Activity Deadline
Recruitment of International Lead Evaluator April 2014
Recruitment of National Evaluators April 2014
Developing the methodology for the evaluation April/May 2014
Submit/present inception reports to UNICEF May 2014
Piloting the evaluation approach and methodology May 2014
Execution of self-evaluation; data collection, analysis, etc. June-August 2014
Participation at reporting workshop 8-11 September 2014
Submit first draft of country reports 29 September 2014
Submit first draft of synthesis report 13 October 2014
Submit final country reports 3 November 2014
Submit final report of the global synthesis 17 November 2014
H. Expression of interest
All interested consultants should send an application packet including the following:
a. A completed Expression of Interest form (see Appendix A) and responses to the questions, including the professional fee/rate, per person day;
b. Completed Personal History Profile (P11)[4], if UNICEF Evaluation Office does not have it on file already;
c. Updated CV/Resume, and;
d. A sample evaluation report (or writing of a similar nature), with clear indication of the applicant’s contribution in the report.
The application packet should be transmitted via email to pdconsultants@unicef.org
Email Subject: Expression of Interest- International Lead Evaluator (LUL)
I. Closing date
The closing dates for this EoI is 21 April, 2014, 12:01 am (midnight), New York City time.
NB: Submissions have to file all four documents accordingly to be considered. Submissions do not that file all four documentswill notbe considered.
Appendix A: Expression of Interest Form
International Lead Evaluator for Evaluation of Let US Learn Initiatives (LUL)
Please fill-in page 1 of the form in its entirety and submit it to us electronically or via fax.
First Name:
Last Name:
User Salutation:
Mr. Ms. Mrs. Dr.
Job Title:
Mobile: (please include country & city code
Fax: (please include country & city code)
Email address:
Address:
City:
State:
Postal Code:
Country:
Alternate contact
Please respond to the questions below in a narrative not exceeding 3-4 pages.
1. Provide information which will enable us to determine whether you have relevant experience to conduct the proposed evaluation activities. Information should include:
a. The number of years of experience in evaluation research;
b. The number of evaluations you have carried out as lead investigator, and as a member of an evaluation team, and the number of evaluations commissioned by UN agencies or comparable organizations;
c. A description of your technical competencies, skills and expertise in evaluation concepts and approaches, and analytical skills to support both qualitative and quantitative research;
2. Provide information which will enable us to determine whether you have relevant specialized knowledge in the areas that are critical to this work. Information should include;
a. A description of programming experience in the education sector, and programming under the themes of girl’s education, out-of-school children, assessment of learning, and/or programming for equity.
b. A description of evaluation or programming work in complex settings, as education in emergencies programmes, and/or with vulnerable populations in conflict affected countries;
c. A description of actions/decisions that demonstrate your leadership and strategic thinking skills;
d. A description of ‘other’ skills or experiences that may be required to execute this evaluation, include facilitation skills, design and execution of stakeholder consultations, conducting team building exercises, time management skills, etc;
3. Provide any additional experience that may be critical to the success of the proposed work, including but limited to:
a. Affiliation to universities, university programmes, professional bodies, communities of practice, etc.
b. Any other information that you deem relevant to the this work that would give you an advantage over others competing for the same consultancy
4. Please provide your professional fee rate, per person day
<strong style="background: transparent; border: 0px; font-size: 14px; line-height: inherit; margin: 0px; ou