Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.
Rolling Five-Year Development Evaluation Work Plan 2012-2013 - 2016-2017
- List of Abbreviations
- Executive Summary
- 1.0 Background
- 2.0 CIDA’s Evaluation Function
- 3.0 Overview of Achievements and Challenges – FY 2011/2012
- 4.0 Strategy and Key Considerations in Developing the Work Plan
- 5.0 Evaluation Division Priorities for FY 2012/2013
- 6.0 Program Branches’ Evaluation Plans for FY 2012/2013
- 7.0 Resource Estimates and Challenges
- CIDA Evaluation Initiatives (FY 2012/2013 – FY 2016/2017)
- Evaluation Commitments to TBS
- Annex 1: Establishing the Evaluation Universe
List of Abbreviations
- Canadian International Development Agency
- Country Development Performance Framework
- Development Assistance Committee of the Network on Development Evaluation
- Development Effectiveness
- Departmental Performance Report
- Evaluation Committee
- Evaluation Division
- Federal Accountability Act
- Geographic Programs Branch
- Management Accountability Framework
- Multilateral and Global Programs Branch
- Multilateral Organizations Performance Assessment Network
- Development Assistance Committee of the Organization for Economic Development and Cooperation
- Performance Management Strategies
- Partnership with Canadians Branch
- Risk and Results-Based Management and Accountability Framework
- Report on Plans and Priorities
- Strategic Policy and Performance Branch
- Treasury Board
- Treasury Board Secretariat
- Terms of Reference
This Rolling Five-Year Evaluation Work Plan (“the Plan”) was prepared by the Evaluation Division (ED) of CIDA’s Strategic Policy and Performance Branch (SPPB). It is an updated version of the Plan from fiscal year (FY) 2011/12, which was approved by the Evaluation Committee (EC) in February 2011.
The Plan lays out the scheduled evaluations for FY 2012/2013 to 2016-2017. Execution of this Plan will enable CIDA to move towards achieving evaluation coverage of 100% of its direct program spending, as required by the 2009 Treasury Board (TB) Policy on Evaluation. The Plan provides an overview of the ED’s achievements in FY 2011/12. It outlines the evaluation work planned by the ED for fiscal years 2012/13 to 2016/17, with a focus on FY 2012-13, and a detailed list of resources required. It also includes a list of decentralizedFootnote 1 evaluations planned for FY 2012-13. This Plan was developed using the best available information regarding the availability of the Agency’s resources for evaluation in FY 2012/2013.
Evaluation Division’s Accomplishments – FY 2011/2012
The ED completed several evaluations in FY 2011/2012 and provided input to numerous corporate initiatives. Highlights of these accomplishments include:
- Initiation of reviews of the World Health Organization (WHO) and the Asian Development Bank (ADB), as well as evaluations of the Mali, ColombiaFootnote 2 and CaribbeanFootnote 3 country programs.
- Completion of the following evaluation activities:
- Ukraine, Honduras, and Peru Country Program Evaluations; the Regional Inter- American Program EvaluationFootnote 4; and CIDA’s Humanitarian Assistance Evaluation.Footnote 5
- Development Effectiveness Review of the United Nations Development Programme (UNDP)Footnote 6 and the World Food Programme's Humanitarian AssistanceFootnote 7; and a Review of Evidence of the Effectiveness of CIDA’s Grants and Contributions.
- Support to the renewal of CIDA’s terms and conditions, based on the review of the effectiveness of CIDA’s grants and contributions;
- Two joint evaluations with international donors: Mali and the Paris Declaration;
- A scoping/evaluability assessment of CIDA’s largest aid program (Afghanistan) and development of terms of reference for the evaluation;
- Development of evaluation strategies for the Muskoka Initiative and CIDA’s three thematic priorities;
- A strategy and work plan for the dissemination of evaluation knowledge;
- Completion of a quality assurance tracking system for decentralized evaluations, along with the provision of coaching, advice, and guidance to program branches, with a 113% increase in the number of inquiries during FY2011/12;
- Improvements to the generic terms of reference for decentralized evaluations and other guidance documents;
- Provision of quality at entry services to over 70 branch-led (decentralized) evaluations;
- Input into several corporate initiatives (e.g., Report on Plans and Priorities, Departmental Performance Report);
- Contribution to 8 TB submissions for new initiatives, with a total value of $528 million;
- Effective management of the evaluation standing offer and design of the upcoming new supply arrangement;
- Termination of three branch-led evaluation mandates due to insufficient quality of deliverables; and,
- Endorsement of a new approach for assessing the development effectiveness of multilateral organizations by the DAC-EVALNET.
Planned Evaluation Work – FY 2012/2013
In 2012-2013, the ED will initiate and/or complete the following evaluations:
- Countries of concentration and regional programs: Bolivia, Colombia, Mali, Tanzania, Pakistan, Indonesia, and Caribbean Regional
- Fragile States: Afghanistan and Haiti
- Multilateral Organizations: African Development Bank, Asian Development Bank, World Health Organization and UNICEF
- Partnerships with Canadians Branch: Governance programming
- Corporate evaluations: a meta-evaluation of branch-led evaluations, and a report on the state of performance measurement at CIDA
In addition the ED will:
- Provide advice and support to program branches on decentralized evaluations;
- Complete an annual report on CIDA’s evaluations, which will be entitled CIDA Learns;
- Continue to implement the dissemination strategy for evaluation knowledge; and,
- Serve as Secretariat for CIDA’s Evaluation Committee.
Implementation of this evaluation work plan will require a budget of approximately $2.2 million for FY2012/2013, including a base salary budget of $1.4 million for 18 FTEs. In order to achieve cost savings while meeting evaluation coverage requirements, the UNICEF review will be conducted jointly with the Netherlands, and coordination between centralized and decentralized evaluations will be maximized.
The Evaluation Division has a base salary budget of about $1.4 million for 18 FTEs. She also knows that the effects of the action plan for deficit reduction could impact on the resources needed to implement the plan. This could be changed to reflect a different scenario.
This document describes the Rolling Five-Year Evaluation Work Plan (“the Plan”) for the Evaluation Division (ED) of CIDA’s Strategic Policy and Performance Branch (SPPB). The Plan takes into account a range of considerations, including:
- The evolving international, Canadian and CIDA contexts;
- The achievements, experiences and lessons learnt from previous years;
- Treasury Board of Canada requirements;
- Accountability, value for money; and,
- A strategic assessment of riskFootnote 8.
The scope and content of the Plan are consistent with the requirements of the 2006 Federal Accountability Act (FedAA) and the 2009 Treasury Board (TB) Policy on Evaluation. The Plan proposes an approach that moves CIDA towards achieving 100% evaluation coverage of direct program spending as required by the FedAA and the 2009 TB Policy on Evaluation, while contributing to the Agency’s learning and decision-making needs and processes.
1.2 International Context
In 2005, the Paris Declaration on Aid Effectiveness emerged from a growing consensus on reforming donors and developing countries’ approaches to delivering and using aid in order to ensure more and better development results. The Paris Declaration committed developing countries and donors to hold each other accountable for implementing its principles at the country level through a set of clear indicators, with targets to be achieved by 2010.
In 2008, the first phase of the Paris Declaration evaluation assessed changes in behavior and identified better practices for partners and donors in implementing the Paris commitments. The results of this evaluation contributed constructively to the ongoing aid effectiveness policy debate and, in particular, to the 2008 Accra Agenda for Action. The Accra Agenda for Action further built on the Paris Declaration by promoting policies and actions that improve transparency and accountability in strengthening aid effectiveness. In 2011, the second phase of the Paris Declaration evaluation contributed towards the aid effectiveness debate, ahead of discussions at the 4th High Level Forum on Aid Effectiveness. The evaluation highlighted that most principles and commitments in the declaration have proven relevant to improving the quality of aid. However, evidence gathered on the implementation of the Paris Declaration shows that while progress has been made, it has not been to the extent and pace envisioned in 2005.
The 4th High Level Forum on Aid Effectiveness was held in Busan, Korea, at the end of 2011. The forum considered the results of implementing the Paris Declaration and charted the future course for aid effectiveness, particularly given changing global dynamics and political upheavals. At the Busan forum, decision-makers in developed and developing countries agreed on a framework for global co-operation supporting the achievement of the Millennium Development Goals. A global compact was also forged for effective development towards and beyond 2015.
1.3 Canadian Context
Similar to its peers in a globalizing world, Canada is grappling with the demands of economic recovery, demographic shifts, climate change, and fiscal restraint. The impact of social media, the special challenges in conflict and fragile states, and the influence of new types of donors active in development cooperation efforts are also having an impact on Canada and other donors’ approach to international development.
The Government of Canada has outlined its plan to return to a balanced budget by 2015–16, as indicated in the federal budget tabled on June 6, 2011. In this context, government departments underwent a strategic and operational review to inform the government’s deficit reduction decision-making process. Evaluations supported this objective by providing information on the relevance and performance of programs.
In addition, government departments and agencies have been focusing on the implementation of the 2009 TB Policy on Evaluation. The objective of this updated policy is to create a comprehensive and reliable base of evaluation evidence that is used to support policy and program improvement, expenditure management, Cabinet decision-making, and public reporting.
In line with the 2006 FedAA, the 2009 TB Policy on Evaluation requires that 100% of direct program spending be evaluated over a five-year cycle. This requirement will be in full effect by 2017-18. The TB Policy on Evaluation also introduces the use of flexible evaluation approaches guided by risk, scale, and scope. It requires departments to develop a strategically focused evaluation plan that is founded on an assessment of risk, departmental priorities, and government priorities. The expected results of this updated policy are a more robust evaluation function focused on value for money; accountability; and credible, timely and neutral information on the ongoing relevance and effectiveness of all direct program spending.
1.4 CIDA Context
With a view to improving aid effectiveness, The Budget Plan 2007: Aspire To A Stronger, Safer, Better Canada laid out a three-point program for enhancing the focus, efficiency and accountability of Canada’s international assistance efforts. To facilitate this objective, the government committed to "examining options to ensure the independent evaluation of our aid program, providing parliamentarians and Canadians with an objective assessment of the results we achieve with our international assistance" (pp. 263).
In response to this commitment, the Agency agreed to strengthen the independence of its evaluation function in a manner consistent with the framework of the FedAA and the TB Policy on Evaluation. Changes include strengthening the governance, oversight, and independence of CIDA’s Evaluation Committee (EC), which is now comprised of six members from outside government and five CIDA senior executives (including the President, who chairs the committee). The EC acts as an advisory body to the President on evaluation activities and is a central part of the Agency’s evaluation function.
Over the last three years, CIDA has re-examined its programs, policies and operations. The Agency has implemented a comprehensive aid effectiveness agenda, strengthened its geographic and thematic focus, completed a multilateral effectiveness review, and modernized its partnerships with Canadians programming.
In the context of the Government of Canada’s deficit-reduction action plan, CIDA continues to work towards achieving further efficiencies. The Agency’s business modernization initiative is an attempt to streamline the Agency’s core business processes through the implementation of a system to manage all of CIDA’s policies and procedures (e.g. a Guide to Rules and Tools), and decentralization of program management and project delivery functions to field offices.
This Plan has been developed using the best available information regarding the availability of Agency resources for evaluation in FY2012/2013.
2.0 CIDA’s Evaluation Function
Evaluation provides Canadians, parliamentarians, beneficiaries, ministers, central agencies, and CIDA’s senior management with evidence-based timely, credible, and neutral information on the relevance, effectiveness, sustainability, efficiency, coherence and cross-cutting issues of the Agencies policies and programs. The key objective of CIDA’s evaluation function is to create a comprehensive and reliable base of evaluative evidence for the purposes of accountability, expenditure management, decision-making and learning.
CIDA’s evaluation function is governed by the Agency’s Evaluation Policy, which was updated to reflect the new requirements outlined in the 2009 TB Policy on Evaluation. The policy is currently going through the Agency’s internal approval process. The policy is implemented through the Rolling Five-Year Evaluation Work Plan, which is approved by the President on advice from the EC.
The policy and the Plan are executed through the Agency’s centralized and decentralized evaluation functions. The ED leads the centralized evaluation function, focusing on corporate (Agency-level) and program-level assessments of relevance and performance. The evaluations led by the ED respond to the requirements of the 2009 TB Policy on Evaluation.
Program branches lead and manage (alone or jointly with other donors) decentralized evaluations, while the ED provides technical quality assurance services, fostering an environment of quality improvement. These evaluations cover investments below the sub-sub activity level of the Program Activity Architecture. Decentralized evaluations support decision- making, program and project improvement, mutual accountability and learning. While the results of these evaluations improve CIDA’s aid effectiveness, the primary beneficiaries are developing country partners. These decentralized evaluations also serve as building blocks for centralized evaluations.
Centralized and decentralized evaluations are integral to CIDA’s solid oversight function while simultaneously contributing to learning and the adoption of better practices. Recognizing the duality of the evaluation function within the Agency, this Plan will continue to focus on both centralized and decentralized evaluations, including some multi-donor initiatives.
The key responsibilities of both the Evaluation Division and program branches are outlined below.
2.1 Responsibilities of the Evaluation Division
The Evaluation Division is responsible forFootnote 9:
- ensuring that the evaluation function is appropriately staffed with competent professionals;
- developing a rolling five-year evaluation plan, which covers 100% of CIDA’s direct program spending;
- conducting centralized evaluations in a neutral and cost-effective manner;
- disseminating evaluation knowledge to increase the visibility, accessibility and use of evaluation results and lessons, both within and outside the Agency;
- tracking the implementation of commitments in management responses and action plans to address the recommendations of evaluations;
- strengthening CIDA’s evaluation function through evaluation quality assurance services, by providing: advice, coaching, training, guidelines, tools, and templates to the Agency’s branches;
- maintaining standing offers for evaluation services;
- building and maintaining strategic alliances with partners in Canada and internationally; and,
- supporting the Evaluation Committee to ensure effective governance and oversight.
2.2 Responsibilities of Program Branches
Program branches are responsible forFootnote 10:
- ensuring that credible and reliable performance data are being collected to effectively support evaluation;
- developing multi-year evaluation work plans (for decentralized evaluations) and submitting them to the ED for inclusion in CIDA’s Rolling Five-Year Evaluation Work Plan;
- managing decentralized evaluations, providing management responses to these evaluations, and ensuring follow-up and tracking of recommendations;
- disseminating the results of their decentralized evaluations, for learning and accountability purposes;
- preparing management responses and/or actions plans in response to centralized evaluations of their programs; and,
- implementing actions based on the approved management responses and/or action plans.
3.0 Overview of Achievements and Challenges - FY 2011/2012
3.1.1 Evaluations of Geographic Programs
During FY 2011/2012, the ED completed the Peru evaluation, which was presented to the EC during its February 2012 meeting. The joint evaluation of general budget support (Mali) was also completed, as well as the Ukraine, Honduras, and Inter-American program evaluations.
In terms of evaluations in fragile states, the ED completed a scoping/evaluability assessment of CIDA’s largest aid program (Afghanistan) and developed terms of reference for this evaluation.
3.1.2 Evaluations of Multilateral and Global Programs
The approach to assess the development effectiveness of multilateral organizations was finalized under the guidance of the Development Assistance Committee Network on Development Evaluation (DAC/EVALNET) of the Organization for Economic Development and Cooperation (OECD) and was endorsed by its members as an acceptable approach for this type of review.
The approach was applied to two multilateral organizations in FY 2011/2012, namely the United Nations Development Programme (CIDA-only initiative) and the World Food Programme (jointly with the Netherlands). The reports from these reviews were presented to the Evaluation Committee during its February 2012 meeting.
3.1.3 Corporate Evaluations
The ED completed A Review of Evidence of the Effectiveness of CIDA’s Grants and Contributions. This review covers CIDA’s recent investments provided through its three programming channels – bilateral, partnership and multilateral. The results of this review also fed into the renewal process of CIDA’s Terms and Conditions.
The Corporate Evaluation of CIDA’s Humanitarian Assistance was initiated in 2011 and its final evaluation report was presented to the EC during its February 2012 meeting.
The division finalized its participation in the joint evaluation of the Paris Declaration, including assisting Senegal and Ghana to complete their country evaluations as case studies while strengthening their evaluation capacity.
The ED completed evaluation strategies for the Muskoka Initiative, the Sustainable Economic Growth Thematic Priority Strategy, the Children and Youth Thematic Priority Strategy, and the Food Security Thematic Priority Strategy.
3.1.4 Evaluations of Partnerships with Canadians Programming
The ED has worked, in conjunction with PWCB, to develop an approach suitable for evaluating PWCB programming at the corporate level. The ED has identified five evaluation units within the new PWCB organization and thematic focus, all of which are integrated within this Plan. These evaluation units correspond to the themes of PWCB's logic model.
3.1.5 Support Provided to Program Branches
The ED continued to provide support and advice to program branches on decentralized evaluations, with a 113% increase in the number of inquiries during FY2011/12. Other support included:
- reviewing 75 evaluation TORs, 30 evaluation work plans, and 20 draft evaluation reports, of evaluations which range from smaller mid-term formative evaluations ($60,000 to $90,000) to large and complex evaluations ($200,000 to $400,00);
- providing support to programs in their selection of consultants for branch-led evaluations, ensuring the best fit between mandated requirements and the experience and knowledge of consultants;
- effective management of the evaluation standing offer and design of the upcoming new supply arrangement; and,
- improving the generic terms of reference for decentralized evaluations and other guidance documents.
All of these efforts are intended to support the program branches attempts to strengthen their decentralized evaluations, which will also create stronger “building blocks” for upcoming ED evaluations.
3.1.6 Support to Other Corporate Initiatives
The ED provided support and input to corporate initiatives such as the Report on Plans and Priorities (RPP), the Departmental Performance Report (DPR), the OECD/DAC Peer Review exercise, the Busan 4th High-Level Forum on Aid Effectiveness and the Management Accountability Framework (MAF). In FY2011/12 CIDA received a “strong” rating in the evaluation area of management, as part of the TBS MAF assessment. This marked the sixth consecutive year that CIDA’s evaluation function has received this rating. The ED also provided advice and recommendations for the review of proposals for grants and contributions under the authority of the SPPB, and continued to contribute to high-level decision-making by CIDA’s executives.
3.1.7 TB Submissions
In collaboration with the Performance Management Division, the ED assisted programs with preparation of their TB submissions. In particular, the ED supported preparation of 8 TB submissions this year. Table 3 (at the end of the Plan) provides a list of evaluations that are required as a result of TB submissions.
3.1.8 Dissemination and Use of Evaluation Results
The ED developed and initiated the implementation of a strategy and work plan for the dissemination of evaluation knowledge. The strategy and work plan are aligned with the Government of Canada’s commitment (in the 2007 budget) to improve the accountability to Canadian taxpayers of international assistance programs. The strategy and work plan will contribute to the Agency’s knowledge management and lessons learned agenda.
The accompanying work plan provides further details on the specific activities under each of the areas of focus to be undertaken during the next two fiscal years, including deliverables, costs and timelines.
The ED launched the first Evaluation Learning Café in September 2011 with the support of the Agency’s evaluation knowledge champion, the Vice-President of the Strategic Policy and Performance Branch. The learning event was on the evaluation of the Paris Declaration on Aid Effectiveness with a presentation by Bernard Wood, Team Leader for the evaluation. Approximately 100 participants from headquarters and the field attended the event, as well as colleagues from other departments. The learning café was the first in a series of events held in FY2011/12, which were designed to help disseminate evaluation results and promote learning within the Agency.
CIDA’s Dissemination Strategy
CIDA’s strategy for the dissemination of evaluation knowledge seeks to increase the visibility, accessibility and use of evaluation results and lessons, both within and outside the Agency.
CIDA’s dissemination practices will help ensure that information from evaluations is systematically provided to Agency staff and managers, partners, and Canadians.
The strategy outlines the objectives and expected results, target audience, roles and responsibilities, areas of focus, and implementation approach.
3.1.9 Building and Maintaining Strategic Alliances
The ED strengthened strategic alliances through domestic joint efforts, such as the horizontal evaluation of the Canadian Police Arrangement, which was led by the Department of Foreign Affairs and International Trade (DFAIT) with participation from CIDA, the Royal Canadian Mounted Police and, Public Safety Canada. Internationally, the ED has continued to actively collaborate with other donors, and participate in the DAC-EVALNET. The ED’s participation in the DAC-EVALNET has included promotion of the common approach for assessing the development effectiveness of multilateral organizations.
During the past year, the ED faced specific challenges that have impacted its work. These challenges constitute key considerations for developing the division’s work plan for FY 2012/2013 and beyond. These challenges include:
- Evaluation Capacity: The shortage of skilled evaluators (both consultants and staff) has been a challenge for the evaluation function in Canada; CIDA’s evaluation function is no exception. This problem is further exacerbated by the departure of experienced evaluators due to retirement. To address this challenge, the ED is currently working on the Agency’s standing offer arrangement to improve the capacity of consultants. Additionally, the ED intends to hire four junior evaluation officers, and provide them with a clear path towards career progression and development. These officers will receive training in evaluation methods, which will serve to enhance the ED’s capacity while contributing to a pool of qualified evaluators within CIDA and the public service at large.
- Evaluation Coverage: Another challenge is the fulfillment of evaluation coverage requirements with multilateral organizations. The division’s leading role in the current OECD/DAC joint donor exercise to develop and test a methodology for evaluating the development effectiveness of multilateral organizations is bringing to success in this area.
- Evaluation of Fragile States: The ED faces particular challenges in delivering evaluations on fragile and conflict-affected states (i.e., Haiti, Sudan, West Bank/Gaza, Afghanistan). Both natural disasters and ongoing security issues make field-based evaluations problematic and expensive. This challenge is being addressed in various ways, depending on the specific situation. Joint evaluations are one way to leverage the resources of multiple donors (e.g., Multi-Donor Evaluation of Conflict Prevention and Peacebuilding Activities in Southern Sudan) and mitigate security risks. In other situations, the ED will engage consultants with significant experience in fragile states (e.g., Afghanistan).
4.0 Strategy and Key Considerations in Developing the Work Plan
In developing this Plan, the Evaluation Division considered the following:
- the requirement to achieve 100% evaluation coverage of CIDA’s direct program spending over 5 years by 2017/2018;
- the evaluation approach used in different types of programming (geographic, fragile states, multilateral, and partnerships);
- assessing and managing levels of risk for specific evaluations;
- the duality of the evaluation function at CIDA (i.e., centralized and decentralized evaluations);
- the commitments in Treasury Board submissions; and
- the need to further promote the use of evaluation results within the Agency.
4.1 Evaluation Coverage
The first challenge in addressing the TB evaluation coverage requirement is to establish an evaluation universe that provides for full coverage of the Agency’s direct program spending during the 5-year cycle. A detailed description of the Agency’s evaluation universe is provided under Annex 1: “Establishing the Evaluation Universe”. In the current environment of fiscal restraint, the ED will give utmost priority to meeting the coverage requirements stipulated in the 2006 Federal Accountability Act and the 2009 TB Policy on Evaluation.
The second challenge is the appropriate sequencing of evaluations so that their usefulness and value are optimized. The sequencing is determined by factors such as program priorities; materiality (importance in financial terms); and the contribution of the evaluation to decision- making processes (e.g. renewal of the programming strategy or replenishment of funding to partner organizations and institutions). The sequencing of evaluations over a five-year cycle is outlined in Table 2.
Based on the Agency’s corporate risk profile, a preliminary risk assessment has been undertakenFootnote 11 to assess the operational, financial, reputational, and development risks of undertaking a specific evaluation. The ED’s risk assessments, which are listed in Annex 1, paid particular attention to stability, level of security, and complexity.
4.2 Evaluation Approach for CIDA’s Programs
Given the variety of CIDA’s programming channels, the ED has adopted a multi-pronged evaluation strategy:
4.2.1 Geographic Programs Branch (GPB)
The approach taken to cover geographic programs will continue to give priority to countries of concentration (priority level I). The Minister of International Cooperation has announced that CIDA will prioritize recipients by allocating 80% of its geographic program resources to 20 such priority countries. Country program evaluations in countries of concentration with greater political, social and economic stability (e.g., Vietnam, Mozambique, Senegal, Mali, Honduras) have been assessed as medium risk.Footnote 12
For selected countries that are not countries of concentration, the ED proposes to conduct cluster evaluations at the sub-regional level in order to maximize the cost-effectiveness of evaluation resources. As this category also includes regional programs, an approach and methodology to undertake these evaluations will be developed, based on an examination of themes across these countries through a sample of projects, using various sources of information and evaluations that have already been undertaken by the Program Branches. For these selected countries of more modest investment, the risk is considered low to medium. The level of priority for these same countries is low to medium (priority level II and III).
4.2.2 Fragile States
Fragile states evaluations include consideration of the extent to which the program is aligned with the OECD/DAC Principles for Good International Engagement in Fragile States and Situations. These principles direct international interventions in fragile states towards governance and state building for long-term stability. The Evaluation Division is developing generic TORs for program evaluations in fragile states that fully integrate the OECD/DAC principles and provide guidance on risk mitigation strategies. The ED considers fragile states evaluations a high priority (level I), given the large investment made in these programs. The level of risk to undertake program evaluations in fragile states has been assessed as high, given security concerns and/or the complexity of programming.
4.2.3 Multilateral and Global Programs Branch (MGPB)
MGPB provides support to a large number of international institutions and funds. The ED has developed, under the guidance of the DAC/EVALNET, a joint approach aimed at assessing the development effectiveness information from these organizations themselves, through their evaluation work and their performance reporting. The approach, which was endorsed by the DAC-EVALNET following two pilot tests, includes a common methodology, which allows greater coverage of more institutions over a shorter period, at less cost to each shareholder, and with greater leverage for affecting needed changes.
This approach is complementary to the work done by the Multilateral Organization Performance Assessment Network (MOPAN), which until 2011 focused mainly on organizational effectivenessFootnote 13. The levels of priority and risk for multilateral organization evaluations is low to medium (priority level II or III).
4.2.4 Partnerships with Canadians Branch (PWCB)
In September 2010, the branch underwent a transformation and refocused its approach. The new branch, renamed “Partnerships with Canadians Branch,” is comprised of two overarching programs: "Partners for Development" and "Global Citizens."
Through these programs, PWCB supports development projects and programs that are designed and implemented by Canadian partner organizations, in co-operation with their counterparts in developing countries. PWCB also aims to engage Canadians and build knowledge around international development.
The ED has identified five evaluation units within the new PWCB organization and thematic focus, all of which are integrated within this Plan. These evaluation units correspond to the themes of PWCB's logic model. The five evaluation units are:
- Economic Growth and Environmental Sustainability;
- Human Development;
- Engaging Canadians; and,
- Knowledge Creation and Sharing.
Country program evaluations will also cover investments by PWCB, in whole-of-agency assessments of CIDA program spending. Regarding the above-listed evaluation units, PWCB has been consulted on the order in which the units will be assessed. Given PWCB’s previous level of implementation and achievement of results in the area of governance, the first evaluation unit will focus on this thematic area. This evaluation will be implemented in FY 2012/2013 and will use decentralized evaluations and project reports generated by PWCB, among other sources of information. Although PWCB programming has limited financial implications, the level of risk and priority is considered medium due to high levels of public interest in Canadian partnership programming.
4.2.5. Corporate Evaluations
In addition to performance information at the program-level, the Agency also requires information on the performance of cross-agency initiatives such as policies, thematic issues and delivery mechanisms. Corporate evaluations include periodic meta-evaluations to assess the quality of program-led evaluations, and thematic evaluations, which can cut across various country programs and/or priority themes. Given the complexity and variety of situations emerging from corporate and policy evaluations, the level of risk has been assessed as medium. The priority for corporate evaluations is medium (priority level II).
4.3 Alignment between Centralized and Decentralized Evaluations
As mentioned above, decentralized (branch-led) evaluations are “building blocks” to centralized evaluations (ED-led). In addition, they complement centralized evaluations by providing a more detailed picture of the Agency’s performance.
Decentralized Evaluations as “Building Blocks”
There is a great deal of synergy between decentralized and centralized evaluations, which can be harnessed to maximize resources. Using decentralized evaluations, which are often smaller in scope, as “building blocks” for centralized whole-of-program evaluations is one way to do this.
For instance, CIDA’s Haiti program is currently conducting evaluations of six CIDA funded projects (worth $90 million), which cover the majority of CIDA’s health portfolio in Haiti. While three of these evaluations are being executed by UN agencies, and three by CIDA’s Haiti program, an overall report, complete with findings and conclusions, will be prepared and shared with Haitian stakeholders.
The timing of these decentralized evaluations leave them well placed to contribute to the centralized Haiti program evaluation, which will reduce its overall cost without sacrificing independence or methodological rigour. The centralized Haiti program evaluation is currently in the planning phase and will be presented the Evaluation Committee in the fall of 2013.
The ED is moving towards establishing better alignment by: (1) including the evaluation plans of the program branches within the ED’s Rolling Five-year Evaluation Work Plan and, (2) providing technical advice, coaching, and ensuring that decentralized evaluations are carried out in accordance with the OECD-DAC’s 2010 Quality Standards for Development Evaluation.
4.4 Commitments in Treasury Board Submissions
Over the past few years there have been several evaluation commitments made as a result of Risk and Results-Based Management and Accountability Frameworks (RRMAF) and TB Submissions for the approval of major initiatives. Many of these initiatives are program sub- components, which could have been covered through the respective program evaluations, instead of separate evaluations –as per commitments made to TB. These evaluation commitments may be fulfilled through existing planned program evaluations, rather than conducting separate evaluations. The level of risk of such evaluations depends on the country, program and institution under consideration. The numbers of these evaluations are expected to decline in the future. The ED will finalize discussions with TBS to facilitate this objective.
4.5 Optimizing the Learning / Knowledge Benefits from Evaluations
Evaluations at CIDA create a source of credible and neutral information, which feeds into strategic reviews, renewals of terms and conditions, Treasury Board submissions, the Report on Plans and Priorities, the Departmental Performance Reports, and memoranda to Cabinet.
Though CIDA has traditionally benefited from an evaluative culture, there increasingly remains a need to better optimize the learning and knowledge benefits from evaluations. The Agency’s strategy and work plan for the dissemination of evaluation knowledge will increase the visibility, accessibility and use of evaluation findings and lessons, both within and outside the Agency. An example is Agency committees, which can be a useful venue for discussing evaluation results and promoting learning across the Agency.
5.0 Evaluation Division Priorities for FY 2012/2013
Based on the evaluation context, the lessons and challenges from the previous year, and the considerations outlined in Section 4.0, the evaluation work plan (FY 2012/2013 –FY 2016/2017) will focus on the following broad areas:
- Evaluations planned for FY 2012/13;
- Provision of advice, support and evaluation quality assurance services to program branches;
- Dissemination and use of evaluation results; and
- Governance structure and strategic alliances.
5.1 Evaluations Planned for FY 2012/2013
5.1.1 Countries of Concentration and Regional Programs
In selecting countries to be the evaluated, the ED will give priority to the countries where CIDA has concentrated its efforts. These countries have received significant investments over the past five years. In FY 2012/2013, the ED will finalize the Mali, Colombia and Caribbean regional program evaluations, which were initiated in FY 2011/2012, and will launch and conduct country program evaluations in Tanzania, Bolivia, Indonesia and Pakistan.
5.1.2 Fragile States
The ED will undertake evaluations of the Afghanistan and Haiti programs, as per commitments made to the TB. These countries constitute a high priority given the importance they have for the Canadian government’s policy agenda and the magnitude of resources invested over the past five years.
Conducting evaluations in these countries is considered a very complex undertaking given the level of insecurity and instability and the number of actors involved (including CIDA, Foreign Affairs and International Trade Canada, the Department of National Defence, etc.). In preparation for the Afghanistan program evaluation, the ED completed a scoping/evaluability assessment, which determined the scope of the evaluation, and prepared TORs for the evaluation. The Afghanistan program evaluation will be implemented over two fiscal years, starting in July 2012 with a planned duration of 15 months. The evaluation of CIDA's Program in Haiti will be implemented during 2012/13, and the final report will be produced early in the following fiscal year.
5.1.3 Multilateral Programs
Multilateral partners are excellent candidates for joint evaluation with other bilateral shareholders, many of whom share Canada’s interests in finding out more about their effectiveness. In FY 2012/13, CIDA will conduct reviews of the African Development Bank and UNICEF, and finalize reviews of the Asian Development Bank and World Health Organization. The UNICEF review will be conducted jointly with the Netherlands, using the DAC/EVALNET common approach. Given the close linkage between the joint approach and the work of the MOPAN, the ED will continue to collaborate with the MGPB to maximize complementarities, avoid duplication of efforts, and consider how to institutionalize the approach.
The ED will continue to offer assistance and provide evaluation services to CIDA’s large multinational partners. For example, the ED will continue to comment on our multilateral partners’ evaluations policies and plans; evaluation mandates; TORs; and, evaluation reports and management responses, while continuing to participate in multi-donor evaluation committees.
5.1.4 Partnerships with Canadians Programming
In FY 2012/2013, the ED will conduct the first evaluation of partnerships with Canadians programming, following its modernization in 2010. This program evaluation will focus on PWCB’s governance programming, and serve to inform related thematic programming across CIDA.
5.1.5 Corporate Evaluations and Initiatives
In FY 2012/2013, the ED will conduct a meta-evaluation of branch-led evaluations, and complete a report on the state of performance measurement in the Agency. In addition, the ED will continue to provide advice in relation to the thematic evaluation strategy of the Muskoka Initiative.
5.2 Advice and Support to Program Branches
In FY 2012/2013, the ED will continue to provide evaluation quality assurance services, advice and support to program branches on ongoing evaluations, and other related needs. The ED will also develop new tools to ensure that CIDA’s evaluations properly assess gender equality performance and develop guidance documents for the implementation of project-level evaluations. In addition, the ED will help program branches streamline their evaluation plans and better align them with the ED Rolling Five-Year Evaluation Work Plan.
Following the deficit reduction action plan, a number CIDA programs will be closing. As a result, there will likely be an increase in demand for the ED to provide evaluation services and support to program branches. Additional resources may be required to support this demand.
5.3 Corporate Initiatives
The ED will also continue to provide support and input to corporate level initiatives, such as the Departmental Performance Report; the Report on Plans and Priorities; and, the Management Accountability Framework (MAF). The ED will also provide advice and recommendations for the review of proposals for grants and contributions under the authority of the SPPB. Moreover, the ED will continue to participate in various committees and working groups (e.g. Program Committee) that offer opportunities for the knowledge from ED’s evaluations to feed back into the policy and planning processes of the Agency.
5.4 Dissemination and Use of Evaluation Results
The ED will continue to implement the Agency’s strategy for the dissemination of evaluation knowledge, which seeks to increase the visibility, accessibility and use of evaluation results and lessons, both within and outside the Agency. The first phase of the implementation was initiated during the fiscal year 2011/2012 with a focus on evaluation knowledge from ED-led evaluations and on dissemination efforts within the Agency.
In 2012/2013, the ED will continue to phase in the implementation of the dissemination strategy. The first phase focused on evaluation knowledge from ED-led evaluations and on dissemination efforts within the Agency. The second phase will involve expanding the work on dissemination of evaluation knowledge to program-led evaluations and spreading dissemination efforts more widely to reach audiences outside the Agency. Priority will be given to the dissemination of evaluation knowledge within CIDA with a focus on improving CIDA’s planning, programming and performance.
Based on the accompanying work plan, one of the key deliverables for fiscal year 2012/2013 will be the preparation of the annual report on CIDA’s evaluation function for dissemination of evaluation knowledge from ED- and branch-led evaluations. The report will present key lessons learned from ED- and branch-led evaluations conducted during the previous fiscal year. Specifically, the report will contain an overview of key lessons learned from evaluations, improvements via ED-led evaluations, corporate evaluation resources, and one-to-two pagers on each ED-led and branch-led evaluation included in the analysis for the report, describing purpose, findings and lessons learned.
5.5 Governance Structure and Strategic Alliances
5.5.1 Support to the Evaluation Committee
The ED, serving as the Secretariat to the EC, will continue to dedicate the necessary human, technical and financial resources to ensure that the EC functions efficiently and effectively.
5.5.2 Building and Maintaining Strategic Alliances
Outside the Agency, the Canadian evaluation community of practice consists of the TBS Centre for Excellence in Evaluation (CEE), heads of evaluation from other government departments and the Canadian Evaluation Society. Internationally, the ED is a member of the DAC Evaluation Network (DAC-EVALNET), the Nordic Plus sub-group inside the DAC-EVALNET, and the Multilateral Organization Performance Assessment Network (MOPAN). CIDA has been particularly active within the DAC-EVALNET as a champion of the common approach for assessing the development effectiveness of multilateral organizations.
Liaising with these communities of practice is important to maintain and strengthen Canadian leadership in evaluation, and provides the ED with sources of best practices and lessons learned. Additionally, it fosters opportunities for joint evaluations with other donors on initiatives such as the evaluation on general budget support; the DAC initiative on multilateral effectiveness; the joint evaluation of the Paris Declaration; and the International Initiative for Impact Evaluation (3IE).
6.0 Program Branches’ Evaluation Plans for FY 2012/2013
As described in previous sections, CIDA maintains a dual evaluation function (centralized and decentralized). The centralized evaluation function, maintained by the ED, focuses on corporate (Agency-level) and program-level evaluations. The decentralized evaluation function, in the program branches, conducts evaluations to support decision-making at the program and project level (i.e. sub-sub-sub activity level). As the functional lead for evaluation across the Agency, the ED provides tools, advice, guidance, training and referral of evaluation expertise to program branches on their decentralized evaluations.
The following section describes the evaluation activities planned by the branches during the FY 2012/2013 period. Information for this section was provided directly by each respective program branch. The ED recognizes that not all branch-led evaluation activities can be classified as evaluations, as per the TB Policy on Evaluation. However, branch-led activities complement the ED’s evaluation work by providing performance assessments, monitoring data, and other relevant information.
6.1 Geographic Program Branch (GPB) – Decentralized Evaluations Planned for FY 2012/2013
GPB has approximately 30 branch-led evaluations currently scheduled for FY 2012/2013. These evaluations will include consideration of project relevance, efficiency, effectiveness, and sustainability. While GPB work plans may be modified prior to the start of the fiscal year, the list below provides provisionally planned evaluations.
|Africa—Southern and Eastern|
|Ethiopia||Agricultural Growth Program (AGP)|
|Mozambique||Support to Ministry of Health—Health Sector Common Fund (PROSAUDE)|
|South Africa||Gender Mainstreaming Project|
|South Africa||Public Sector and Anti-Corruption Training|
|Honduras||Strengthening Networks for Community Health (REDES)|
|Nicaragua||Nicaraguan Electrification Project (PELNICA)|
|Peru||Improvement of Basic Education (PROMEB)|
|Cuba||Development of the Forestry Sector|
|Colombia||Planning the Future|
|Haiti||Centre Semi Autonome|
|Haiti||Comp. Adaptation aux changements climatiques|
|Haiti||Health Management Capacity Building Support (PARC)|
|Haiti||Projet de coopération municipale|
|Haiti||Health System Development Project (PADESS)|
|Africa—West and Central|
|Regional Program in West and Central Africa (PRACO)||Initiative intégrée du Bassin du Niger|
|PRACO||Strengthening Traditional Health Systems for Prevention and Treatment of Malaria|
|République Démocratique du Congo (RDC)||Réhabilitation des services de santé - Province de Kinshasa (PROASKIN)|
|Nigeria||Response to Climate Change|
|Europe, Middle-East & Maghreb, Afghanistan and Pakistan (EMMAP)|
|Afghanistan||Aga Khan Foundation Canada (AKFC)—Enhancing the Livelihoods Opportunities in Northern Afghanistan (ELLONA)|
|Afghanistan||Afghanistan Community Renewal Program—Phase II (ACRP II)|
|Afghanistan||Integrated Alternative Livelihoods Program—Phase 2|
|Afghanistan||Rural Enterprise Development in Kandahar|
|Afghanistan||Accelerated District Reconstruction Program|
|Pakistan||Women-Focused Value Chain Development in Pakistan: Pathways and Purse Strings|
|Ukraine||Improving Competitiveness of the Diary Sector|
|Burma (Border Areas)||Burma Border Assistance Program: Building Social Capital|
|Nepal||Local Government and Community Development Program|
|China||Migrant Labour Organizational Health and Safety|
6.2 Partnership with Canadians Branch (PWCB) – Decentralized Evaluations Planned for FY 2012/2013
PWCB started several thematic evaluations in FY 2011-12, which will be completed and disseminated in FY 2012/2013. The thematic evaluations are: cooperatives; water and sanitation; sports for development; and, the volunteer cooperation program.
In FY 2012/2013, PWCB will initiate thematic evaluations on Basic Education and the Short- Term Volunteers Model.
6.3 Multilateral and Global Programs Branch (MGPB) – Decentralized Evaluations Planned for FY 2012/2013
At the end of FY2011/12 MGPB completed a mid-term formative evaluation of the Micro- Nutrient Initiative. This evaluation provided performance information for the Micronutrient Initiative to fulfill a Treasury Board Submission commitment. In FY 2012/2013, MGPB will also implement a mid-term evaluation of the $120 million TB Reach Program, which is housed within the Stop TB Partnership of the World Health Organization. No linkages have been identified between these decentralized evaluations and the corporate evaluations that will be led by the ED in the upcoming fiscal year.
7.0 Resource Estimates and Challenges
7.1 Non-Salary O&M
As shown in Table 2 at the end of this document, the Plan anticipates a budget of approximately $2.2 million for FY 2012/2013 to implement the proposed plan. This amount covers the costs of engaging external professional expertise, as well as other non-salary administrative costs such as staff travel, translation and printing. In order to achieve cost savings while meeting evaluation coverage requirements, the Plan takes into account that the UNICEF review will be conducted jointly with the Netherlands. Additionally, the ED will strive to improve coordination between centralized and decentralized evaluations in an effort to maximize resource efficiency.
7.2 Salary Budget
The division now has a base salary budget of approximately $1.4 Million and a staff complement of 18 FTEs: the Director, 16 evaluators and 1 support staff.
The Evaluation Division recognizes that the results of the deficit reduction action plan may have an impact on the resources necessary to deliver this Plan. Therefore, this Plan may be altered to account for a different scenario. Options may include proposing alternative implementation approaches or reducing the scope of planned evaluations.
CIDA Evaluation Initiatives (FY 2012/2013 – FY 2016/2017)
- Only initiatives for FY 2012/2013 are costed. Amounts include all non-salary O&M (such as professional services, printing, travel etc.).
- The Total Program Level of Expenditures amount reflects investments from FY 2006/2007 to FY 2010/2011. It does not include expenditures from closed programs.
- FY 2011/2012 is retained in the table order to provide a more comprehensive picture of the evaluation universe. However, it is understood that the 5-Year Rolling Plan includes FY 2012/2013 to FY 2016/2017.
- E = Led by Evaluation Division. (Evaluation activities for the upcoming FY are highlighted in green)
- S = Led by Program Branches and supervised (quality-controlled) by the Evaluation Division
- D = Dissemination / knowledge sharing.
- C = Support to the Corporate level (Evaluation activities for the upcoming FY are highlighted in blue.)
|Proposed Initiatives||Level of Risk||Total Program Expenditures (FY 06/07 to FY 10/11) in C$ millions||Program Operational Cost (FY 06/07 to FY 10/11) in C$ millions||G&C Amount (FY 06/07 to FY 10/11) in C$ millions||2011-2012||Evaluation Resources FY 12/1314||2012-2013||2013-2014||2014-2015||2015-2016||2016-2017||Considerations|
|14 The amounts listed in this table cover only disbursements for FY2012/13, even for evaluations whose contracts cover more than one fiscal year.|
15 Table shows multilateral institutions to be reviewed in 2012/13. The sequencing of institutions to be reviewed in future years is tentative and will be firmly identified in future updates of the Plan, to reflect agreements reached with other donors on the sequencing of upcoming reviews under MOPAN and the DAC Evaluation Network Approach.
|Bangladesh Program||Medium||299.02||8.66||290.37||E||D||Last evaluated in FY 08/09.|
|China Program||Low||246.54||10.96||235.58||E||D||CDPF expired in 2010. Never evaluated. Will be evaluated with other emerging countries (Brazil, South Africa, Russia, India).|
|Indonesia Program||Medium||238.55||1.64||236.90||$200,000||E||E/D||CDPF expired 2009. Evaluation was delayed because the program was being audited. Program expenditures include disbursements for Indonesia, Timor-Leste, and South Pacific.|
|Vietnam Program||Medium||134.16||1.51||132.65||E||D||Evaluation completed in FY 09/10.|
|Philippines, Sri Lanka, Cambodia, Nepal Program||Medium||265.83||0.11||265.72||E||D||Cluster program evaluation of middle-income countries of modest presence. Although Cambodia and Nepal are not middle-income countries, they have been clustered into this category for the purposes of having a manageable evaluation unit. Program expenditures include disbursements for: Cambodia, Burma border areas, Philippines, Nepal, Sri Lanka, Bhutan, Maldives, Mongolia.|
|Southeast Asia Regional Program||Med-Low||47.56||0.05||47.51||E||D||Evaluation of a regional program of modest presence.|
|West and Central Africa Programs|
|Benin, Burkina Faso, Guinea, Niger Program||Medium||162.67||4.91||157.77||E||D||Cluster program evaluation of low-income countries of modest presence. Program expenditures include disbursements for: Guinea, Liberia, Ivory Coast, Niger, Burkina Faso, Sierra Leone, Benin, Cameroon, Central African Republic, Chad, Gabon, Equatorial Guinea, Sao Tome and Principe, Congo, Burundi.|
|West and Central Africa Regional Program||Medium||73.24||2.17||71.07||E||D||Regional program evaluation.|
|DR Congo, Rwanda Program||Medium||96.25||1.98||94.27||E||D||Cluster program evaluation of low-income countries of modest presence.|
|Mali Program||Medium||381.27||3.28||378.00||E||$25,000||E/D||Last evaluated in FY 06/07. Joint-Evaluation of Budget support interventions undertaken in 2009-11.|
|Senegal Program||Medium||237.52||2.94||234.57||D||E||CDPF expired in 2011. Evaluation completed in FY 2010/11. Approved by Evaluation Committee in Feb. 2011. Program expenditures include disbursements for Cape Verde, Gambia, Senegal, Guinea-Bissau.|
|Ghana Program||Medium||396.13||3.51||392.62||E||D||Corporate evaluation in RRMAF for FY 11/12. Last evaluated in FY 07/08. Program expenditures include disbursements for Ghana and Togo.|
|Canada Investment Fund for Africa||Medium||94.03||4.01||90.01||E||D||E||D||TB Condition: mid-term evaluation completed. A summative evaluation is planned for FY 2014/15.|
|Southern and Eastern Africa Programs|
|Ethiopia Program||Medium||403.00||3.23||399.77||E||D||TBS commitment. Last evaluated in FY 09/10. Program expenditures include disbursements for Ethiopia and Djibouti.|
|Kenya, Malawi, Nigeria, Zambia, Zimbabwe Program||Medium||273.35||6.28||267.07||E||D||Cluster program evaluation of low-income countries of modest presence. Program expenditures include disbursements for Nigeria, Zambia, Malawi, Swaziland, Kenya, Madagasscar, Uganda, Somalia, Angola, Botswana, Zimbabwe, Lesotho, Namibia.|
|Mozambique Program||Medium||370.53||3.88||366.65||E||D||Last evaluated in FY 09/10.|
|Pan-African Regional Program||Medium||148.29||2.13||146.16||E||D||Regional program evaluation. Program expenditures include disbursements for Pan Africa and Africa Health Systems Initiative.|
|Southern and Eastern Africa Program||Medium||41.9||0.22||41.68||E||D||Program evaluation of regional programs of modest presence.|
|Darfur/Sudan Program||High||270.15||4.83||265.32||D||E||Multi-donor evaluation of Southern Sudan (led by the Dutch) completed in 2010/11. Case study in the evaluation of IHA Programming.|
|Tanzania Program||Medium||306.92||3.14||303.78||$100,000||E||E/D||Last evaluated in FY 04/05. Joint evaluation led by the European Commission for the budget support interventions in 2011-12 and 2012-13.|
|Bolivia Program||Medium||63.48||1.77||61.71||$260,000||E||D||CDPF expired in 2007. Last evaluated in FY 05/06.|
|Caribbean Regional Program||Medium||220.58||4.55||216.03||E||$21,000||E/D||Incl. Guyana, Jamaica. RDPF priority for 2007. Deferred at direction of Evaluation Committee. Program expenditures include disbursements for: Barbados, Caribbean Region, Lee & Windward Region, Guyana, Jamaica, Surinam, Trinidad & Tobago, Dominican Republic and Hispanolia.|
|Haití Program||High||591.58||6.12||585.46||$200,000||E||E/D||Corp. eval. in RRMAF for FY 10/11, in addition to Branch-level eval. in 2009. Last evaluated in FY 02/03. Deferred due to earthquake.|
|Honduras Program||Medium||76.99||1.83||75.16||E||D||E||Reviewed by Evaluation Committee in December 2011.|
|Colombia Program||Medium||55.96||1.09||54.87||E||$21,000||E/D||E||Never evaluated. To be presented to the Evaluation Committee June 2012.|
|Peru Program||Medium||68.98||1.34||67.64||E||D||E||Never evaluated. To be presented to the Evaluation Committee February 2012.|
|Reg. Inter-American Program||Med-Low||146.47||0.02||146.45||E||D||E|
|Cuba, Guatemala, Nicaragua and South American Regional Program||Medium||158.61||9.05||149.56||E||D||Sub-regional program evaluation of middle-income and low-income countries of modest presence. Program expenditures include disbursements for: Costa Rica, El Salvador, Guatemala, Mexico, Nicaragua, Panama, Belize, Argentina, Chile, Ecuador, Paraguay, Uruguay, Venezuela, South America Regional, Cuba, Deployment for Democratic Development.|
|Europe, Middle-East & Maghreb, Afghanistan and Pakistan (EMMAP) Programs|
|West Bank / Gaza Program||High||260.52||0.36||260.16||E||D||Based on discussions with the program, evaluation deferred to 2013/14.|
|Ukraine Program||Medium||95.73||1.70||94.03||D||E||D||Presented to the Evaluation Committee in June 2011. The Ukraine CDPF covers the period from 2009 to 2014.|
|Reg. Institutional Partnerships / Broader Middle East / North Africa (BMENA) Program||Med-Low||215.64||12.09||203.56||E||D||The BMENA program will close on March 31, 2013. Includes cost centres for Belarus, South Caucasus, Bosnia-Herzegovina, Balkans, BMENA, Mauritania, Middle East & Maghreb Region, Algeria, Tunisia, Yemen, Iraq Programs, Middle East Peace Process.|
|Egypt, Jordan, Lebanon, Morocco Program||Medium||158.38||0.37||158.01||E||D||Cluster program evaluation of middle-income low-income countries of modest presence.|
|Pakistan Program||Medium||186.77||0||186.77||$200,000||E||D||Last evaluated in FY 05/06. CDPF 2007-2015. Program expenditures include disbursements for Pakistan and Tadjikistan Operations.|
|Afghanistan Program||High||1,099.65||17.65||1,082.01||$420,000||E||E/D||Program evaluation in RRMAF for FY 09/10. Last evaluated in FY 07/08. Evaluability assessment conducted in FY2011/12. The total budget for this evaluation is $600k. Since the evaluation spans two fiscal years, $400k will be spend in FY 12/13 and the remaining $200k will be spend in FY 13/14.|
|Partnerships with Canadians Programs|
|Governance||Medium||85.77||2.28||83.49||$100,000||E||E/D||The cost estimate is based upon ED’s experience doing other hybrid evaluations. The ED will initiate this evaluation in late 2012 and will finalize it by mid 2013.|
|Economic Growth and Environmental Sustainability||Medium||162.27||4.75||157.52||E||D|
|Knowledge Creation and Sharing||Medium||Not available||E|
|Multilateral Programs15: Financial Institutions|
|Asian Development Bank||Medium||355.39||355.39||E||$40,000||E/D||Report to be completed based on results of the Multilateral Initiative pilot test, which used a methodology developed under the guidance of and endorsed by the DAC/EVALNET.|
|African Development Bank||Medium||510.92||510.92||$100,000||E||D||To be reviewed using the approach developed under the guidance of and endorsed by the DAC/EVALNET. Possibility of conducting the review jointly with other donors.|
|Caribbean Development Bank||Medium||125.50||125.50||E||D||To be reviewed using the approach developed under the guidance of and endorsed by the DAC/EVALNET. Possibility of conducting the review jointly with other donors.|
|Inter-American Development Bank||Medium||44.14||44.14||E||D||To be reviewed using the approached developed under the guidance of and endorsed by the DAC/EVALNET. Possibility of conducting the review jointly with other donors.|
|World Bank||Medium||584.42||584.42||E||D||To be reviewed using the approach developed under the guidance of and endorsed by the DAC/EVALNET. Possibility of conducting the review jointly with other donors.|
|Global Environmental Facility (GEF)||Medium||225.11||225.11||E||D|
|International Fund for Agriculture Development||Medium||100.78||100.78||E||D|
|Multilateral Programs: UN Development and Humanitarian Organizations/Programs, Commonwealth and Francophonie|
|United Nations Development Programme (UNDP)||Medium||307.82||307.82||E||D||E||To be presented to the Evaluation Committee February 2012.|
|World Health Organization (WHO)||Medium||260.75||260.75||E||$40,000||E/D||E||Report to be completed based on results of the Multilateral Initiative pilot test, which used a methodology developed under the guidance of and endorsed by the DAC/EVALNET.|
|World Food Program (WFP)||Medium||845.32||845.32||E||E/D||E||To be presented to the Evaluation Committee February 2012.|
|Office of the United Nations High Commissioner for Refugees (UNHCR)||Medium||215.70||215.70||E||D||To be reviewed using the approached developed under the guidance of and endorsed by the DAC/EVALNET. Possibility of conducting the review jointly with other donors.|
|Global Fund for Aids, Tuberculosis and Malaria (GFATM)||Medium||450.00||450.00||E||D|
|Consultative Group on International Agricultural Research (CGIAR)||Medium||121.69||121.69||E||D|
|United Nations Population Fund (UNFPA)||Medium||124.90||124.90||E||D|
|UNICEF||Medium||414.45||414.45||E||D||To be reviewed using the approach developed under the guidance of and endorsed by the DAC/EVALNET. Possibility of conducting the review jointly with other donors. This review will be conducted by ED staff (in-house).|
|International Organization of la Francophonie||Medium||48.11||48.11||E||D|
|Operational costs||N/A||17.08||Kept separate due to structure of CIDA not aligned by organizational spending for each one of the multilateral organizations listed above.|
|Corporate Evaluation of CIDA’s Humanitarian Assistance||Medium||E||D||E||To be presented to the Evaluation Committee February 2012.|
|Annual Report on the Evaluation Function||Medium||N/A||C|
|Muskoka Initiative Thematic Evaluation||Medium||N/A||C||E||D||A commitment in the TB submission to carry out a thematic evaluation. An Evaluation Strategy was developed in 2011/12 to lay the foundation for the evaluation to be carried out in 2015-16.|
|Muskoka Initiative Review||Medium||E||D||The ED will conduct a review of Canada’s contribution to the Muskoka initiative in order to report back to cabinet in 2015.|
|Meta-Evaluation of Branch-led Evaluations||Medium||$80,000||E||D||Assessment of the quality of branch-led evaluations.|
|Review of Evidence of the Effectiveness of CIDA’s Grants & Contributions||Medium||E/D||E||Input to the renewal of Agency’s Terms and Conditions, which expire on 31 March 2012. Submitted to TBS in November 2011.|
|Support to Branch-led Evaluations (incl. Tools, training, guidance documents, Standing Offers and Consultant referrals, etc).||Medium||E/S||$65,000||E/S||E/S||E/S||E/S||E/S||Provision of support and advice to program branches on decentralized evaluations, including developing new tools to ensure that CIDA’s evaluations properly assess gender equality performance.|
|Dissemination / Learning Strategy||Medium||C||C||C||C||C||C||To be implemented in consultation with SPPB.|
|Evaluation Committee Secretariat||N/A||C||$100,000||C||C||C||C||C|
|Other (travel, training, translation miscellaneous)||N/A||C||$308,000||C||C||C||C||C|
|Total Resources Evaluation Resources||$2,200,000|
|Total Program Expenditures||13,947.74||167.15||13,780.59|
|Evaluation Coverage of Program Expenditures||1,934.12 (13.86% of all direct program expenditures)||12.97||1,921.16 (13.94% of all G&Cs)||$607,000||The evaluations included in the calculation of coverage are those that are scheduled for completion in FY 2012/13, namely Mali, Bolivia, Caribbean Regional , Colombia , PWCB Governance programming, Asian Development Bank, African Development Bank, and World Health Organization (WHO). The operational expenses for ADB and WHO are not included since CIDA does not align organizational spending by multilateral organization.|
Evaluation Commitments to TBSFootnote 16
The table below presents only specific evaluation commitments indicated in the TB Submissions that were approved between FY 2006/2007 and FY 2010/2011. It excludes evaluation commitments that have already been fulfilled.
|TB Submission (abridged title)||Lead*||Evaluation Commitments made to TBS and TB Conditions|
|* PB: Program Branch|
ED: Evaluation Division
17 The Afghanistan Evaluation was postponed to FY 2011/12 in order to ensure that sufficient information was available to conduct a summative evaluation.
|1. Muskoka Initiative||ED||Thematic evaluation to be completed by 2015. A review of the initiative to be completed by 2014.|
|2. CIDA’s Humanitarian Assistance Program and Organizations||ED||Humanitarian Assistance program evaluation to be completed by 2011/2012.|
|3. Afghanistan Program||ED||Summative evaluation to be completed by FY 2009/201017.|
|4. Burkina-Faso||PB||Summative evaluation to be completed by 2015.|
|5. Canada Investment Fund for Africa||ED||Summative evaluation to be completed by FY 2013/2014.|
|6. Pakistan (Debt)||PB||Summative evaluation to be completed by FY 2013/2014.|
|7. Senegal (Education Sector)||PB||Summative evaluation to be completed by FY 2011/2012.|
|8. Tanzania (Education Sector)||PB||Meta-evaluation to be completed by FY 2012/2013.|
|9. UN Central Emergency Response Fund (CERF)||PB||Meta-evaluation to be completed by 2011. An independent five-year summative evaluation was completed in 2011.|
|10. Micronutrient Initiative||PB||Formative evaluation completed. Summative evaluation to be completed by 2014.|
|11. Rural Water Supply and Sanitation Initiative – RWSSI (AfDB)||ED||Joint CIDA-Donors Institutional Assessment to be completed by FY 2010/2011.|
|12. Canadian Foodgrains Bank||PB||Summative evaluation to be completed by FY 2011/2012. (Evaluation completed in 2010 due to accelerated disbursements.)|
|13. Global Partnership for Education (formerly “Education for All Fast Track Initiative”)||PB||Meta evaluation to be completed by 2013.|
|14. Canadian International Food Security Research Fund||PB||Summative Evaluation to be completed by FY 2014/2015.|
|Partnerships with Canadians Programs|
|15. Canadian Catholic Organization for Development and Peace||PB||Summative evaluation to be complete.|
|16. Jules and Paul Émile Léger Foundation||PB||Summative evaluation to be completed by FY 2011/2012.|
|17. Canadian Francophonie Scholarship Program||PB||Formative evaluation to be completed by FY 2011/2012. Summative evaluation to be completed by FY 2013/2014.|
|18. Volunteer Cooperation Program||PB||Formative evaluation to be completed by FY 2011/2012. Summative evaluation to be completed by FY 2013/2014.|
|19. International Youth Internship Program (IYIP)||PB||Formative evaluation to be completed by FY 2011/2012.|
Annex 1: Establishing the Evaluation Universe
Taking into consideration the PAA, the Performance Measurement Frameworks approved by TBS, and the commitments made in TB Submissions, the ED arrived at a tentative Evaluation Universe of 65 units for coverage during the next five (5) years:
- 5 evaluation units related to Fragile Countries and Crisis-Affected Communities;
- 14 evaluation units related to Low-Income Countries;
- 15 evaluation units related to Middle-Income Countries;
- 21 evaluation units related to Global Engagement and Strategic Policy;
- 5 evaluation units related to Canadian Engagement;
- 5 evaluation units related to Corporate Initiatives; and,
- specific commitments to the TB (see Figure 2), which will be covered as part of our Program and Corporate Evaluations.
The number of units in the evaluation universe have been reviewed and adjusted in the Plan, based on CIDA’s PAA, which was approved by TB on September 15, 2011.
Table 4 identifies the evaluation units stated above. Given the large number of evaluations to be completed over the next five years, the ED is exploring the possibility of a division of labor between CIDA and partner organizations. Some of the evaluations will be conducted jointly with other partners (Geographic and Multilateral Programs) while others (Partnerships) will necessitate the development of new methodologies to review the new organizational structure.
In the case of multilateral organizations, the ED will be using the common approach endorsed by the DAC-EVALNET.
Following the deficit reduction action plan, CIDA will need to, in consultation with TBS and legal services, determine its evaluation obligations in regard to the following:
- Closing the Cambodia, China, Malawi, Nepal, Niger, Rwanda, Zambia and Zimbabwe programs.
- Country programs, which are facing reduction, particularly: Bolivia, Pakistan, Mozambique, Ethiopia, Tanzania, and South Africa.
- Regional programs that are consolidating or reducing; in particular African programming will be consolidated into the pan-African regional program, and the Southeast Asia regional program will be reduced.
- Multilateral and global programs, which are being reduced or consolidated.
- Partnerships with Canadians programming that will be reduced or consolidated.
|Category of Evaluation Unit||Evaluation Units||Risk Assessment||Priority Level|
|Risk is considered “high” for countries in crisis; “medium” for countries of concentration; and, “low” for Institutional assessments|
1 “Middle-income country of modest presence” were reallocated to this group for purpose of creating manageable evaluation units
2 “Low-income country of modest presence” were reallocated to this group for the purpose of creating manageable evaluation units
|(1) Fragile countries and crisis-affected communities|
|Countries of Focus: Afghanistan, Haiti, Sudan, West Bank/Gaza||4||High||I|
|Humanitarian Assistance: Food Assistance, Non-Food Assistance||1||Med-High||I|
|Sub-total (Fragile countries)||5|
|(2) Low-Income Countries|
|Countries of Focus: Bangladesh, Ethiopia, Ghana, Mali, Mozambique,Pakistan, Senegal, Tanzania, Vietnam||9||Medium||I|
|Regional Programs: Sub-Saharan Africa Regional Programs (West and Central Africa, Pan-African, and Southern and Eastern Africa Regional Programs).||3||Medium||II|
|Countries of Modest Presence:||2||1-Medium|
|Sub-Total (Low-income countries)||14|
|(3) Middle-Income Countries|
|Focus Countries: Bolivia, Caribbean Regional, Colombia, Honduras, Indonesia, Peru, Ukraine||7||Medium||I|
|Regional Programs: Inter-American Program, Southeast Asia Regional Program Europe Regional Programs, Middle East and North Africa Regional Program.||4||Med-Low||II|
|Countries of Modest Presence:||4||1-Medium|
|Sub-Total (Middle-income countries)||15|
|(4) Global Engagement and Strategic Policy|
|Multilateral Strategic Relationships:||18||Med||I|
|Multilateral and Global Programming: Health Programming and Sectors/Themes other than Health||2||Medium||II|
|International Development Policy, Research and Analysis (to be determined)||1||Low||III|
|Sub-Total (Global engagement & strategic policy)||21|
|(5) Canadian Engagement|
|Economic Growth and Environmental Sustainability||1||Medium||II|
|Knowledge Creation and Sharing||1||Medium||II|
|Sub-Total (Canadian engagement)||5|
|(6) Corporate Initiatives|
|May include: Meta-Evaluations, Review of Program-Based Approaches, Review of CIDA’s implementation of the Paris Declaration, Review of the Effectiveness of the Agency’s Grants and Contributions, thematic evaluations such as the Muskoka Initiative and the Canada Investment Fund for Africa||6||Medium||III|
|Sub-Total (Corporate Initiatives)||6|
|Refer to the list of initiatives in Table 3||0||High||I|
- Date Modified: