3534-Rikka.pdf 6/20/05 2:51 PM Page i
Agriculture and Rural Development Discussion Paper 20
The World Bank
Monitoring and
Evaluation for
World Bank Agricultural
Research and
Extension Projects:
A Good Practice Note
Riikka Rajalahti
Johannes Woelcke
Eija Pehu
3534-Rikka.pdf 6/20/05 2:51 PM Page ii
� 2005 The International Bank for Reconstruction and Development / The World Bank
1818 H Street, NW
Washington, DC 20433
Telephone 202-473-1000
Internet www.worldbank.org/rural
E-mail ard@worldbank.org
All rights reserved.
The findings, interpretations, and conclusions expressed herein are those of the author(s) and do not necessarily reflect
the views of the Board of Executive Directors of the World Bank or the governments they represent.
The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denomina-
tions, and other information shown on any map in this work do not imply any judgment on the part of the World Bank
concerning the legal status of any territory or the endorsement or acceptance of such boundaries.
Rights and Permissions
The material in this work is copyrighted. Copying and/or transmitting portions or all of this work without permission
may be a violation of applicable law. The World Bank encourages dissemination of its work and will normally grant
permission promptly.
For permission to photocopy or reprint any part of this work, please send a request with complete information to the
Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978-
750-4470, www.copyright.com.
All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher,
World Bank, 1818 H Street NW, Washington, DC 20433, USA, fax 202-522-2422, e-mail pubrights@worldbank.org.
ii
3534-Rikka.pdf 6/20/05 2:51 PM Page iii
CONTENTS
ACRONYMS AND ABBREVIATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
PREFACE AND ACKNOWLEDGEMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
EXECUTIVE SUMMARY. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
1. CHALLENGES FOR M&E SYSTEMS IN AGRICULTURAL RESEARCH
AND EXTENSION PROJECTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2. THE WORLD BANK RESULTS FRAMEWORK REQUIREMENTS . . . . . . . . . . . . . . . . 6
3. DATA COLLECTION, REPORTING, AND DISSEMINATION REQUIREMENTS. . . . 16
4. PARTICIPATORY MONITORING AND EVALUATION. . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5. EVALUATION OF OUTCOMES AND IMPACT IN THE WORLD BANK. . . . . . . . . . . . 45
6. ECONOMIC EVALUATION OF COMPETITIVE GRANTS SCHEMES. . . . . . . . . . . . . 51
7. CONCLUDING REMARKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
8. REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Appendix 1. Narratives of New Approaches and Instruments in ARE Systems . . . . . . . . . . . 67
Appendix 2. Implementation of a Monitoring and Evaluation Plan in the
Bank's Project Cycle: Putting It All Together. . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Appendix 3. A Brief Description of the "Extended" Logical Framework . . . . . . . . . . . . . . . . 75
Appendix 4. List of Indicators and Data Sources for ARE Projects . . . . . . . . . . . . . . . . . . . . 77
Appendix 5. Project Monitoring and Evaluation Costs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Appendix 6. Economic Evaluation of Competitive Research Grants:
The Case of PROMSA, Ecuador . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Tables, Boxes, and Figures
TABLE 1.1: COMPLEMENTARY ROLES FOR MONITORING AND EVALUATION . . . . . . . . . . . . . . . . . . . . . 2
TABLE 2.1: DIFFERENCES BETWEEN THE WORLD BANK LOGICAL AND RESULTS FRAMEWORK . . . . . . . 6
TABLE 2.2: THE WORLD BANK'S RESULTS FRAMEWORK APPLIED TO AN ARE PROJECT . . . . . . . . . . . 7
TABLE 2.3: DEFINITIONSAND EXAMPLES ON AGRICULTURAL RESEARCH AND EXTENSION
PROJECT INDICATORS IN THE WORLD BANK LOG FRAME AND RESULTS FRAMEWORK . . . 11
TABLE 2.4: EXAMPLES ON OUTCOME INDICATORS DEPENDING ON THE PROJECT DEVELOPMENT
OBJECTIVE AND THE PERCEPTIONS OF THE DIFFERENT STAKEHOLDER GROUPS
INVOLVED INAGRICULTURAL RESEARCH AND EXTENSION PROJECTS. . . . . . . . . . . . . . . . 12
iii
3534-Rikka.pdf 6/20/05 2:51 PM Page iv
TABLE 3.1: ARRANGEMENTS FOR OUTCOME MONITORING FOR "CHINA AGRICULTURAL
TECHNOLOGY TRANSFER PROJECT" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
TABLE 3.2: THE MAJOR DATA SOURCES--MANAGEMENT INFORMATION SYSTEM,
EXISTING AND REGULARLY MAINTAINED DATABASES AND VARIOUS
SPECIAL STUDIES AND THE MAIN DATA COLLECTION METHODS . . . . . . . . . . . . . . . . . . 31
TABLE 4.1: MAJOR DIFFERENCES BETWEEN CONVENTIONAL AND PARTICIPATORY M&E . . . . . . . . . . 33
TABLE 4.2: DEGREE OFPARTICIPATION AT DIFFERENT STAGES OF PME PROCESS. . . . . . . . . . . . . . . 35
TABLE 6.1: KEY REFERENCE SOURCES ON THE THEORY AND EMPIRICAL TOOLS FOR ASSESSING THE
ECONOMIC IMPACT OF RESEARCH AND EXTENSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
TABLE 6.2: STATISTICAL REGRESSION MODEL FOR SOLATING
I PROJECT EFFECTS
INIMPACT EVALUATION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
TABLE 6.3: CALCULATION OF IRR FOR AN AGRICULTURAL RESEARCH AND EXTENSION PROJECT . . . 58
TABLE 6.4: SOURCES OF NFORMATION ON THE
I MINIMUM SET OF PARAMETERS
TO BEESTIMATED FOR A SOUND ECONOMIC IMPACT EVALUATION . . . . . . . . . . . . . . . . . 60
TABLE A1.1: POTENTIAL OF SELECTED INNOVATIVE INSTRUMENTS TO ADDRESS
VARIOUS FISCAL AND NSTITUTIONAL
I CHALLENGES ASSOCIATED WITH
AGRICULTURAL RESEARCH AND EXTENSION SYSTEMS.. . . . . . . . . . . . . . . . . . . . . . . . 68
TABLE A3.1: THE "EXTENDED" LOGICAL FRAMEWORK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
TABLE A5.1: A PROJECT COMPONENT TABLE FROM COSTAB 32. . . . . . . . . . . . . . . . . . . . . . . . . . . 90
BOX 1.1: WORLD BANK REQUIREMENTS FOR PROJECT AND PROGRAM
MONITORING AND EVALUATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
BOX 1.2: KEY FEATURES OF MPLEMENTATION
I -FOCUSED VERSUS RESULTS-BASED MONITORING . . . . 3
BOX 1.3: FOCUS ON NSTITUTIONAL
I DEVELOPMENT OR PRODUCTIVITY IMPACTS. . . . . . . . . . . . . . . . . 4
BOX 1.4: COMPETITIVE RESEARCH GRANT PROGRAM IN ECUADOR. . . . . . . . . . . . . . . . . . . . . . . . . . 5
BOX 2.1: PARTICIPATORY INDICATOR IDENTIFICATION IN AN EXTENSION PROJECT IN MEXICO. . . . . . . 11
BOX 2.2: TQQ OF DEFINING INDICATORS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
BOX 2.3: THE IMPLEMENTATION STATUS AND RESULTS (ISR) REPORT QUALITY CHECKLIST. . . . . . . 15
BOX 3.1: MANAGEMENT INFORMATION SYSTEM (MIS) PREREQUISITES FOR
AN EFFECTIVE M&E SYSTEM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
BOX 3.2: TECHNOLOGY OPTIONS IN DEVELOPMENT OF MANAGEMENT INFORMATION SYSTEM. . . . . . 20
BOX 3.3: KEY FEATURES OF THE MANAGEMENT INFORMATION SYSTEM (MIS) FOR
THE MODERNIZATION PROGRAM FOR AGRICULTURAL SERVICE PROJECT (PROMSA)
IN ECUADOR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
BOX 3.4: PROJECT INFORMATION MANAGEMENT SYSTEM NETWORK OF NDIA S
I ' NATIONAL
AGRICULTURAL TECHNOLOGY PROJECT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
BOX 3.5: SCENARIO ON HARMONIZING M&E IN NATIONAL SECTOR MINISTRIES . . . . . . . . . . . . . . . 24
BOX 3.6: WHAT TO DO WHEN THE PROJECT DOES NOT HAVE BASELINE DATA?. . . . . . . . . . . . . . . 27
BOX 3.7: RURAL WOMEN'S EMPOWERMENT THROUGH AGRICULTURAL EXTENSION IN VENEZUELA . . 29
iv
3534-Rikka.pdf 6/20/05 2:51 PM Page v
BOX 3.8: USING INTERVIEWS AND FOCUS GROUP DISCUSSIONS AT MID-TERM EVALUATION OF A
FARMER-LED EXTENSION SERVICE PROVISION PROJECT . . . . . . . . . . . . . . . . . . . . . . . . . . 30
BOX 4.1: BASIC STEPS INPARTICIPATORY ACTIVITY EVALUATION . . . . . . . . . . . . . . . . . . . . . . . . . . 37
BOX 4.2: USING CONVERSATIONAL INTERVIEWS WITH FIELD-FOCUSED APPROACH . . . . . . . . . . . . . . 38
BOX 4.3: MAIN TOOLS AND METHODS TO USE IN PME FOR ARE PROJECTS . . . . . . . . . . . . . . . . . 41
BOX 4.4: LESSONS LEARNED FROM THE UGANDA NATIONAL AGRICULTURAL
ADVISORY SERVICES PROJECT'S (NAADS) PME PILOT . . . . . . . . . . . . . . . . . . . . . . . . . 42
BOX 5.1: SPECIFIC HIGHLIGHTSFOR M&E DURING ICR PREPARATION. . . . . . . . . . . . . . . . . . . . . . 46
BOX 5.2: POTENTIAL FOCUS FOR EVALUATION IN AGRICULTURAL RESEARCH AND EXTENSION
PROJECTS/PROGRAMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
BOX 5.3: WHEN TO USE INTERNAL AND EXTERNAL REVIEWERS AND EVALUATORS?. . . . . . . . . . . . . 49
BOX 6.1: USING RANDOM SAMPLES IN EVALUATION OF MPACT OF
I COMPETITIVE GRANT SCHEMES . . . 54
BOX 6.2: ESTIMATING INTERNAL RATE OF RETURN FOR SUBPROJECTS IN COMPETITIVE
GRANT SCHEMES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
BOX 6.3: SUSTAINABLE FINANCING OF A COMPETITIVE FUND IN ECUADOR . . . . . . . . . . . . . . . . . . . 61
BOX A1.1: AGRICULTURAL INNOVATION SYSTEM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
FIGURE 6.1: ESTIMATION STRATEGY IN PRONATTA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
FIGURE A2.1: DIAGRAM OF M&E STEPS IN THE PROJECT CYCLE . . . . . . . . . . . . . . . . . . . . . . . . . 74
v
3534-Rikka.pdf 6/20/05 2:51 PM Page vi
ACRONYMS AND ABBREVIATIONS
ABF Annual Benefit Flow
ACF Annual Cost Flow
APL Adaptable Program Loan
AIS Agricultural Innovation System
ASTI Agricultural Science & Technology Indicators
BA Beneficiary Assessment
BP Bank Procedure
CAS Country Assistance Strategy
CDB Country Database
CGIAR Consultative Group on International Agricultural Research
CGP Competitive Grant Project
CGS Competitive Grant Scheme
CIAL Local Agriculture Research Committee
CIMMYT International Maize and Wheat Improvement Center
CRGP Competitive Research Grant Program
CRGS Competitive Research Grant Scheme
ER Expert Review
FAO Food and Agriculture Organization
FFS Farmer Field School
GDB Global Database
GDP Gross Domestic Product
GEF Global Environment Facility
ICR Implementation Completion Report
ICT Information and Communication Technology
IDA International Development Association
IFAD International Fund for Agriculture Development
IFPRI International Food Policy Research Institute
INB Increment in Net Benefits
IRR Internal Rate of Return
ISNAR International Service for National Agricultural Research
ISR Implementation Status Report
MAKIS Modernizing Agricultural Knowledge and Information Systems
M&E Monitoring & Evaluation
MIS Management Information System
NAADS National Agricultural Advisory Services
NATP National Agricultural Technology Project
vi
3534-Rikka.pdf 6/20/05 2:51 PM Page vii
NGO Non-governmental Organization
NRI Natural Resources Institute
PDB Project Database
OD Operational Directive
OED Operations Evaluation Department
OP Operational Policy
PRSP Poverty Reduction Strategy Paper
PAD Project Appraisal Document
PCN Project Concept Note
PCU Project Coordinating Unit
PCR Project Completion Report
PDO Project Development Objective
PIMSNET Project Information Management System Network
PIU Project Implementation Unit
PME Participatory Monitoring & Evaluation
PROMSA Modernization Program for Agricultural Services
PSR Project Supervision Report
PTD Participatory Technology Development
QAG Quality Assurance Group
R&E Research & Extension
RD Recommendation Domain
R&D Research & Development
SS Special Studies
TQQ Time, Quality and Quantity
TT Task Team
TTL Task Team Leader
UEFC Competitive Fund Management Unit
UOA Unit of Analysis
vii
3534-Rikka.pdf 6/20/05 2:51 PM Page viii
PREFACE AND ACKNOWLEDGEMENTS
This Monitoring and Evaluation Good Practice Note was prepared by Riikka Rajalahti, Eija Pehu,
and Johannes Woelcke. We would also like to thank Greg Traxler from Auburn University and
Gustavo Sain from the Inter-American Institute for Cooperation on Agriculture (Economic Impact
Analysis), and World Bank Consultants Remileku Rakey (M&E in the World Bank Project Cycle),
Ghazali Raheem (M&E costs), and Lawrence Salmen (Beneficiary Assessment) for their contribu-
tions to this report.
The authors would also like to thank reviewers Jody Kusek (AFTKL), Gary Alex (USAID), Jim
Hanson (Maryland University), Doug Horton (CGIAR), Willem Janssen (SAR), Indira Ekanayake
(LCR), Richard Chisholm (EAP), Matthias Grueninger (MNA), David Nielson (AFR), and Derek
Byerlee (ARD) for their time and useful comments.
The World Bank's Sustainable Agriculture Systems Knowledge and Institutions (SASKI) Thematic
Group is acknowledged for its financial support.
viii
3534-Rikka.pdf 6/20/05 2:51 PM Page ix
EXECUTIVE SUMMARY
Monitoring and Evaluation (M&E) are integral tools for managing and assessing the efficiency and
effectiveness of investments in agricultural research and extension (ARE) systems. However, moni-
toring and evaluation of ARE project performance, outcomes, and impact has been a significant chal-
lenge. Moreover, the increased focus of donors and borrowers on impact has resulted in a high
demand for expertise in M&E.
The objective of this Good Practice Note is to assist Task Teams of the World Bank and their col-
leagues in the client countries develop and implement effective M&E systems for ARE projects and
programs. The chapters in this Note provide a step-by-step guide for achieving that objective, with
emphasis on the World Bank M&E requirements and the specific nature of ARE projects.
Chapter 1 titled "Challenges for M&E Systems in Agricultural Research and Extension Projects" and
Appendix 1 titled "Narratives of New Approaches and Instruments in ARE Systems" discuss the M&E
challenges inARE projects; and provide additional information on the forces driving the changes inARE
systems as well as on the typical approaches and instruments applied in ARE systems and projects.
The focus of M&E in the Bank has shifted from monitoring implementation to tracking results.
Results-based systems build upon and add to traditional implementation-focused systems emphasiz-
ing project outcomes. Since 2003, the Bank has used the Results Framework, which is a simplified
version of the traditional log frame focusing on (a) the Project Development Objective (PDO) and
(b) intermediate outcomes expected from implementing each individual project component, which
all in turn contribute to the achievement of the PDO.
The Note emphasizes the World Bank's M&E Requirements at the design stage, namely the intro-
duction of Results Framework and the associated requirements in the Project Appraisal Document
(i.e., the final product of a successful project planning process). Chapter 2 describes the key issues
in the Results Framework which are (a) development of a clear PDO and Project Components, and
(b) the importance of developing measurable and realistic Project Outcome and Intermediate
Outcome indicators. In addition, Appendix 2 summarizes the Bank's M&E steps throughout the
Project Cycle and Appendix 3 provides further details and references on the Logical Framework. A
list of indicators, with emphasis on institutional, social, and environmental sustainability, and data
sources for ARE Projects are provided in Appendix 4.
After designing the Results Framework and selecting the key performance indicators, the next step is
to specify the M&E arrangements for data collection, reporting, dissemination, and use for decision-
making. Chapter 3 presents the Results Monitoring Arrangements. Effective M&E requires the devel-
opment of concise M&E Plan (see section 3.1 as presented in the PAD), the establishment of a
management information system (MIS), an internal system for collection, analysis, storing, and dis-
semination of pertinent information (see section 3.2) and good institutional and human capacity to
ix
3534-Rikka.pdf 6/20/05 2:51 PM Page x
implement the required actions (see section 3.3). Finally, sections 3.4 and 3.5 emphasize the impor-
tance of data collection and use (including baseline), list helpful data sources, describe methodology
for data collection and analysis, and indicate how much a good M&E system costs. Additional infor-
mation is provided in Appendix 4.
Participatory monitoring and evaluation (PME) is gaining importance in the development and mon-
itoring of demand-driven ARE systems and projects. The basic idea in PME is to allow active
involvement of key stakeholders in the M&E process in order for them to learn about and affect the
process and impact of a development intervention. Participatory M&E can be used in concert with
traditional M&E but should be considered a different process. If resources are limited, it is better to
identify carefully when and how to apply PME rather than sacrifice the quality of the process and
results generated. Chapter 4 provides further information on PME in ARE projects.
Evaluation is the periodic assessment of the relevance, performance, efficiency, and impact (both
expected and unexpected) of a development intervention in relation to stated objectives. Evaluation
measures achievements in relation to institutional policies, Bank-wide program objectives, and the
goals set for each operation. Evaluation at the Bank has two major dimensions: (a) self-evaluation
that consists of an interim evaluation and a terminal evaluation undertaken at the end of a project
(requirement for the Project Completion Report); and (b) independent impact evaluation usually
undertaken by Operations Evaluation Department several years after final disbursement; which
measures changes attributable to the project in terms of both direct and indirect causality. Chapter 5
presents the Bank evaluation requirements and evaluation of outcomes and impact. Chapter 6 high-
lights the key steps in the economic impact evaluation of Competitive Grants Schemes (additional
case study in Appendix 5) and describes the fiscal analysis recommended for ARE projects.
x
3534-Rikka.pdf 6/20/05 2:51 PM Page 1
1. CHALLENGES FOR M&E SYSTEMS IN
AGRICULTURAL RESEARCH AND EXTENSION PROJECTS
1.1 PURPOSE OF THE GOOD PRACTICE NOTE
High rate of growth in agricultural productivity is essential to promote broad-based economic
growth, reduce poverty, and conserve natural resources. In turn, productivity growth is based large-
ly on application of science, technology, and information provided through national research and
development systems and various extension and advisory services. Within this context, the success
of rural development projects and programs has been shown to depend largely on direct stakeholder
involvement in planning, implementation, and evaluation (Agriculture Investment Sourcebook).
Monitoring and evaluation (M&E) are integral tools for managing and assessing the efficiency and
effectiveness of investments in agricultural research and extension (ARE) systems. However, moni-
toring and evaluation of ARE project performance, outcomes, and impact has been a significant chal-
lenge. Moreover, the increased focus of donors and borrowers on impact has resulted in a high
demand for expertise in M&E.
To meet this challenge, the objective of this Good Practice Note is to assist Task Teams of the World
Bank and their counterparts in client countries develop and implement effective M&E of ARE pro-
gram performance, outcomes, and impact.1
Chapters 1 through 6 provide a step-by-step guide for developing an M&E system for ARE projects.
1. Challenges for M&E Systems in Agricultural Research and Extension Projects
2. The World Bank Results Framework Requirements
2.1. From the Log Frame to the Results Framework
2.2. Developing the Results Framework (including selecting indicators)
3. Data Collection, Reporting, and Dissemination Requirements
3.1. Monitoring and Evaluation Plan
3.2. Project Management Information System
3.3. Institutional and Human Capacity in Project M&E
3.4. Data Sources and Methodology for Data Collection and Analysis
3.5. Monitoring and Evaluation Costs
4. Participatory Monitoring and Evaluation
5. Evaluation of Outcomes and Impact in the World Bank
6. Economic Evaluation of Competitive Grants Schemes
1
3534-Rikka.pdf 6/20/05 2:51 PM Page 2
1.2 MONITORING AND EVALUATION
REQUIREMENTS INWORLD BANK PROJECTS
Monitoring and evaluation systems are designed to inform project management (World Bank Task
Teams and client's project implementation unit) whether implementation is going as planned and
whether corrective action is needed to adjust implementation plans. In addition, M&E systems should
provide evidence of project outcomes and justify project funding allocations. The roles of monitoring
and evaluation are summarized in table 1.1. The sources and explanations for the World Bank M&E
requirements, as determined in the Operational Manual, are briefly explained in box l.1. In addition,
the World Bank project cycle and the associated M&E requirements are summarized in Appendix 2.
1.3 EMPHASIZED FOCUS ONRESULTS IN THE WORLD BANK
The focus of M&E has shifted from monitoring implementation to tracking results. Traditionally
M&E systems were implementation-focused and included tracking of input mobilization, activities
undertaken and completed, and outputs delivered. However, the implementation-focused approach
Table 1.1: Complementary roles for monitoring and evaluation
Monitoring Evaluation
� Routine collection of information � Analyzing information
� Tracking project implementation progress � Ex-post assessment of effectiveness and impact
� Measuring efficiency � Confirming project expectations
� Measuring impacts
� Question: "Is the project doing things right?" � Question: "Is the project doing the right things?"
Source: Alex and Byerlee (2000).
BOX 1.1:WORLD BANK REQUIREMENTS FORPROJECTAND PROGRAM MONITORING AND EVALUATION
The World Bank Monitoring and Evaluation requirements of both program and project lending are pre-
sented in the Operational Directive 10.70 and Operational Policy (OP)/Bank Procedure (BP) 13.05.
The Operational Directive (OD) 10.70 defines Monitoring as "the continuous assessment of project
implementation in relation to agreed schedules and the use of inputs, infrastructure, and services by
project beneficiaries. Evaluation is defined as "the periodic assessment of the relevance, performance,
efficiency, and impact (both expected and unexpected) of the project in relation to stated objectives."
In addition, OD 10.70 sets out the concept of M&E, and provides general guidance on the design and
implementation of the information systems required for M&E activities.
The OP 13.05 explains the rationale for Bank's supervision of Bank-financed projects, which include
monitoring and evaluative review and reporting. It also explains the responsibilities of Task Teams (TT),
and requires that the borrower and the TT agree on implementation arrangements which include M&E
arrangements, and use of appropriate performance indicators.
Source: World Bank Operational Manual
2
3534-Rikka.pdf 6/20/05 2:51 PM Page 3
does not provide managers, stakeholders, or policy-makers with an understanding of failure or suc-
cess of the project in reaching the desired outcomes (Kusek and Rist 2004). Results-based systems
build upon and add to traditional implementation-focused systems, emphasizing project outcomes.
Box 1.2 illustrates some of the key features and differences between implementation- and results-
based M&E systems.
BOX 1.2: KEY FEATURES OF MPLEMENTATION FOCUSED
I - VERSUS RESULTS-BASED MONITORING
Elements of Implementation Monitoring (traditionally used for projects):
� Description of the problem or situation before the intervention;
� Benchmarks for activities and immediate outputs;
� Data collection on inputs, activities, and immediate outputs;
� Systematic reporting on provision of inputs;
� Systematic reporting on production of outputs;
� Directly linked to a discrete intervention (or series of interventions); and
� Designed to provide information on administrative, implementation, and management issues as
opposed to broader development effectiveness issues.
Elements of Results Monitoring (used for a range of interventions and strategies):
� Baseline data to describe the problem or situation before the intervention;
� Indicators for outcomes;
� Data collection on outputs and how and whether they contribute toward achievement of
outcomes;
� Timelines expressed such as at mid-term and end-term;
� More focus on perceptions of change among stakeholders;
� Systematic reporting with more qualitative and quantitative information on the progress
toward outcomes;
� Done in conjunction with strategic partners; and
� Captures information on success or failure of partnership strategy in achieving desired outcomes.
Source: Kusek and Rist (2004).
1.4 SPECIFIC CHALLENGES FOR M&E
IN AGRICULTURAL RESEARCH AND EXTENSION PROJECTS
A workable system for monitoring and evaluation of project performance and especially outcomes
and impact is a challenge for all development programs. The particular problems facing project man-
agement in the development of an M&E system for ARE projects include:
1. The natural uncertainty of research outcomes and technology adoption and difficulties
in establishing timetables for technology development and dissemination. This is espe-
cially true for basic research with very long time lags and indirect pathways to impact;
3
3534-Rikka.pdf 6/20/05 2:51 PM Page 4
2. The difficulties in measuring the diverse objectives of ARE programs, such as poverty
reduction, natural resource conservation, food security, export promotion, economic
growth, and reduction of social conflicts;
3. The problems related to cause-and-effect attribution of impact due to diverse external
factors (e.g., rural credit institutions, input supply systems, product marketing sys-
tems, macro-economic policies), which are outside of direct project control;
4. The lack of reliable national data and lack of awareness and managerial demand for
M&E data;
5. The lack of national capacity for M&E and low financial priority for M&E systems in the
context of low agricultural technology program funding and tight government budgets;
6. The dilemma of whether to develop an M&E system specific to the needs of the Bank
project, or to the broader needs of the (national) ARE program and institution(s). Both
are related and similar, but each has somewhat different requirements; and
7. What is the right balance in project M&E? Should the M&E system track both the
institutional development and the impact on productivity or prioritize based on the
cost of M&E? See box 1.3 for further discussion.
BOX 1.3: FOCUS ON NSTITUTIONAL
I DEVELOPMENT OR PRODUCTIVITY IMPACTS
Productivity impact is defined broadly to include any impact on the agricultural production system and
rural development, including environmental, economic, and social welfare.
Institutional development refers to the creation or the capacity of an institution to reflect systematically
and rigorously upon its role and function, and better enable them to carry out their responsibilities. It
reflects an attempt to introduce change and development in the way the institution is organized so that
it is better able to meet its mission.
Agricultural R&E projects typically seek productivity impacts, but these usually require institutional
development to bring about the desired impacts, and provide a basis for continued innovation and sus-
tainable impact on productivity (e.g., a multi-stakeholder priority setting mechanism for NARS).
Investments into institutional development, in turn, are not justified if they do not have productivity
impacts. However, a focus on quick productivity impact without the institutional development to sustain
it over time is not recommended.
The majority of Bank-supported projects combine both institutional and productivity impacts. Hence,
the inclusion of these two types of impacts needs to be considered when developing the Project
Development Objective (PDO) and the M&E system indicators.
Source: Alex and Byerlee (2000).
1.5 NEW APPROACHES AND NSTRUMENTS
I
IN AGRICULTURAL RESEARCH AND EXTENSION PROJECTS
The recent changes in the agricultural research and extension systems (i.e., decentralization, pri-
vatization, stakeholder participation, and increasingly separation of the functions of financing
4
3534-Rikka.pdf 6/20/05 2:51 PM Page 5
and delivery), as a response to satisfy the changing global and local demands2 placed on the ARE
system, have resulted in a gradual transition from supply-driven technology generation toward
building a more demand-driven agricultural innovation system (AIS). Agricultural innovation
system essentially links multiple sources of innovation and uptake pathways along the continu-
um from basic research to technology adoption recognizing the innovative capacity of all stake-
holders from producers to the marketplace. Thus, the focus is not only on the science suppliers
but rather on the totality of actors that are involved in innovation (see Appendix 1 for more details
on AIS).
Although the ARE systems have moved toward more responsive and effective AIS, there has also
been a shift to a clearer focus on poverty reduction which includes new partnerships for inclusion of
marginalized groups of people and an increased emphasis on co-financing with the private sector. In
this context, a number of instruments and approaches, such as contracting for services, competitive
research grants (CRGP), participatory and farmer-to-farmer R&E, and enhanced use of Information
and Communication Technology (ICT) (for details, see Appendix 1), are being introduced into many
ARE systems. These approaches and instruments are often used simultaneously. As an illustration
box 1.4 describes the complex nature of a CRGP arrangement in the Bank-supported Agricultural
Research Project in Ecuador.
BOX 1.4: COMPETITIVE RESEARCH GRANT PROGRAM INECUADOR
The Program for Modernization of Agricultural Services in Ecuador finances a competitive research grant
program (CRGP) that has funded 112 research and educational subprojects. The program has supported
strategic, applied, and adaptive research and education in a diverse range of thematic areas. These include:
organic agriculture, agro-forestry, agro-industry, variety development, management of plant genetic
resources, soil and water conservation, integrated pest management, animal production, and marketing.
The program introduced a new research culture and brought new organizations into the research system.
Research and educational subprojects are being conducted country-wide and are being executed by
45 different public and private organizations. These include: public and private universities, NGOs, pro-
ducer organizations, private enterprises, and the Instituto Nacional Autonomo de Investigaciones
Agropecuaris (INIAP). Subproject research costs averaged US$ 116,000, of which 54 percent is
financed by grants and 46 percent by executing agencies. By leveraging co-financing for subprojects,
the program helped to increase national research funding by 92 percent, amounting to approximately
0.54 percent of agricultural GDP.
Source: World Bank (2004a). Agriculture Investment Sourcebook.
Additional M&E challenges resulting from the new implementation context include:
� A high level of pluralism--multiple partners, including farmers, farmers' organizations,
and various public and private sector actors;
� Multiple project components, particularly with CRGPs and multiple timelines for imple-
mentation;
5
3534-Rikka.pdf 6/20/05 2:51 PM Page 6
� Non-formal communication and technology transfer routes, which are desirable but more
difficult to track;
� Highly decentralized decision-making and information flows; and
� Variable levels of capacity among partners involved in M&E.
6
3534-Rikka.pdf 6/20/05 2:51 PM Page 7
2. THE WORLD BANK RESULTS FRAMEWORK REQUIREMENTS
2.1 FROM THE LOG FRAME TO THE RESULTS FRAMEWORK
In 2003 the Bank shifted from the log frame as a mandatory requirement in project M&E to a two-
pronged strategy comprising of the results framework and the arrangements for results monitoring.
The results framework is a simplified version of the traditional log frame focusing on (a) the Project
Development Objective (PDO) and (b) intermediate outcomes expected from implementing each indi-
vidual project component, which contributes to the overall achievement of the PDO. The added empha-
sis on component outcomes helps focus project design and management on results. It also forces the
M&E system track these outcomes and impacts rather than simply reporting on production of outputs.
Table 2.1 summarizes the differences between the traditional log frame and the results framework.
Table 2.1: Differences between the World Bank Logical and Results Framework
World Bank Log Frame World Bank Results Framework
Goal: long-term, widespread -- - -
improvement in society
_ but only at
Outcome: intermediate effects of Project Development
outputs on clients Objective (PDO) level _ both at PDO
and component level
Outputs: products and services
- -
produced by the projects --
Activities: tasks personnel
undertake to transform inputs to -- - -
outputs
Inputs: financial, human, -- - -
and material resources
Note: - - = included in the framework and _= not included in the framework.
Source: Adapted from Binnendijk 2000.
Project Appraisal Document (PAD) is the final product of a successful project planning process. For
M&E related matters, the PAD requires the development of the results framework (as described in
chapter 2 of this Note) and description of the M&E arrangements (in chapter 3). The Results
Framework has the following structure:
7
3534-Rikka.pdf 6/20/05 2:51 PM Page 8
� Project Development Objective and Project Component Statements;
� Indicators for PDO outcome and intermediate component outcomes; and
� Explicit statement on how to use the outcome information.
The structure of the results framework as presented in the PAD and an example on a hypothetical
ARE project are shown in table 2.2.
Table 2.2: The World Bank's Results Framework Applied to an ARE Project
Project Development Objective Outcome Indicators Use of Outcome Information
Statement of Project List of outcome indicators at Specify use or purpose of monitoring
Development Objective PDO level specific indicators and use of findings
(PDO level)
Agricultural producers increase For example, percent or number of If the rate of technology adoption by
the adoption of profitable and women and men producers that women and men lower than x% by the
environmentally sound have adopted environmentally specified target dates, the components
technologies sound technologies by year x 1 and 2 must be reviewed for efficiency
Intermediate Outcomes Outcome Indicators
for each Component for Components Use of Outcome Monitoring
Statement of outcomes List of outcome indicators Specify use or purpose of monitoring
per component per component specific indicators and use of findings
(PDO level)
Component 1. Extension service For example percent of women and If the rate of satisfaction is lower than
providers have an improved men producers satisfied with access x% by the specified target dates, the
understanding of client needs and to and quality of extension services activities under component 1 will be
how to respond to them adjusted as needed
Component 2. National research For example, percent of priority If the proportion is lower than
institutes have improved capacity research funds allocated to research envisioned, the situation will be
to conduct research in identified in priority areas. analyzed and corrected.
priority areas
Source: Authors.
2.2 DEVELOPING THE RESULTS FRAMEWORK
When developing the framework, pay attention to the following:
Develop a strong and clear project development objective
A strong and clear Project Development Objective is essential for good project design. The PDO is
expected to illustrate the principal outcome of a change in end-user behavior, (i.e., that of the proj-
ect's primary target group). One of the main problems identified in the Bank's project design has
been lack of clarity and focus in the PDO. PDOs are expressed either too high (e.g., at the Country
Assistance Strategy level) or too low (e.g., at the activity level) and sometimes are not well articu-
lated (e.g., mixing goals, objectives, and strategies).
8
3534-Rikka.pdf 6/20/05 2:51 PM Page 9
Questions that help develop a clear and strong project development objective
(a)Who is the key recipient of project benefits?
(b)After the close of the project, what problem has been solved for the primary target group?
(c)What will the target group be doing differently after the project?
Pay close attention to the following:
� Focus on the outcome that the project can directly influence, given its duration, resources,
and approach;
� PDO should not encompass longer-term outcomes that depend on efforts outside the proj-
ect's scope;
� Keep PDO short, focused, and measurable; and
� PDO is not a restatement of the project's components or outputs.
Practice: In ARE projects the primary target group usually consists of the agricultural producers. An
example of a typical Project Development Objective is "Agricultural producers increase the adoption
of profitable and environmentally sound technologies." Another example could be "Small-holder
farmers in rain-fed areas of Allura actively participate in the development of and use of research pro-
grams results."
Make sure that the project components contribute to achieving
the project development objective
The results framework presents a statement of the key project components, the success of which
determines the achievement of the PDO.
� To track the success of the different components, the expected outcomes from each individual
component (i.e., intermediate outcomes) need to be linked to the achievement of the PDO.
� The outcomes of each project component should clearly specify the immediate target group
that receives the benefits from the component implementation. Thus, they describe the
changes for the secondary target group or the skills or capacities the secondary target
group must have acquired to better serve the primary target group.
Practice: Agricultural R&E projects very often have several secondary target groups. A typical Bank-
supported ARE project would include research and extension staff and institutions as secondary tar-
get groups. If the project improves their performance (e.g., through human capacity building or
institutional development), they will better serve the primary target group (e.g., small scale farmers
and women's groups).
Select indicators for the project development objective and each project component
Why indicators? Indicators are one of the crucial aspects of a project design. They are quantitative and
qualitative variables that provide a simple and reliable means to measure achievement, reflect changes
connected to an intervention, or help assess the performance of an organization against the stated targets.
Carefully selected indicators are needed to measure the success of achieving the PDO and the out-
come(s) of each project component. Specifically, indicators can help managers make informed and
9
3534-Rikka.pdf 6/20/05 2:51 PM Page 10
timely decisions by way of a balanced presentation of financial and operational performance, and
consequent outcomes and effects; and a summary of complex information "at a glance." Tracking
indicators can provide continuous feedback and streams of data throughout the lifetime of the proj-
ect (for more details see section on performance indicators). Establishment of the baseline and tar-
get values is essential for tracking the achievements.
Different indicator levels and types
Indicators are derived from log frames and results framework and provide for measuring perform-
ance at two levels: (1) implementation performance indicators and (2) project impact indicators. See
Table 2.3 for definitions and examples of different indicator levels.
1. Implementation performance indicators measure program inputs and outputs. Tracking
input use and direct project accomplishments often requires multiple indicators, but
they are relatively easy to measure.
2. Project impact indicators measure outcomes and impacts on social goals. Outcome
and impact indicators are more difficult to measure, as they track the results of project
activities and outputs, but are influenced by many other factors. The Results
Framework requires only the description of outcome indicators.
It is important to make a distinction between output, intermediate, and outcome indicators.
� The indicator associated with the Project Development Objective describes a change in end-
user behavior (e.g., farmers' access to, use of, or satisfaction with goods and services) where-
as the indicator of each project component describes the changes for the secondary target
group (e.g., improved skills or capacity among staff of research and extension organizations).
� It is also important to make a distinction between an outcome and output. Outcomes are
intermediate effects of outputs on target group and thereby address the issue of uptake (i.e.,
coverage or adoption) of project outputs resulting from project activities.
� This difference is also reflected in the indicators. An outcome indicator could be "Percent
of trainees implementing new practices" but it would not be "number of trainings conduct-
ed." An illustrative list of key indicators, with particular focus on Competitive Research
Grant Schemes and contracting extension services, is included in Appendix 4.
Indicators must be defined for each step of the results framework (Project Development Objective
and component outcomes). If an extended log frame is developed, indicators must be defined for
goal, Project Development Objective, component outcomes, as well as outputs and inputs.
Key questions to address when selecting and developing indicators
Prior to defining the indicators, ask the following questions:
1. What is the development objective of the project?
The selection of indicators depends on the Project Development Objective and the
focus of the research and extension program (i.e., poverty focus, agricultural market
development, institutional development, productivity change, capacity building, and
environmental sustainability. Table 2.4 includes examples on outcome indicators from
the perspectives of different stakeholders.
10
3534-Rikka.pdf 6/20/05 2:51 PM Page 11
in of
and total or
for client
reflect
are response the year services. and of
Framework. research
R&E vary needs technical and changes findings. developed,
problems, be of train number
similar may institutional per
of of of to and
These show and posted, and
esults are than new could
R time. system new
These institutional research agents
staff, curricula
(Secondary building
outcomes themselves, by number
and other usually lag or reviews
delivery on agents
projects. address efficiency conducted constructed, person-months equipment,
responsiveness or
indicators design. to peer indicators trained,
Frame factors technology funding, based extension research
measured service with disbursed,
Indicators) established.
indicators the institutions of extension
Log Development impact extension clients. be ability train
many considerable development project programs,
productivity, attitudes, with policy laboratory
and a on the outcome graduates
shift certified to funds
and by in with may staff,
of offices of assistance.
Bank Group)
with satisfaction trained.
proposed
World Institutional Target Sector-level research affected programs, only Institutional depending the changes institutional relations Changes relationships, stakeholder programs, collaborative satisfaction government Component number client Extension--Workshops extension regional Research--Laboratories workshops staff Education--faculty numbers training. Examples: construction, technical
the of
in
of may
of or little
include of range (i.e., (for in day
poverty rates be
or says work.
with rate in in investments deforestation program
may wide results farmers field the and
Indicators a (adoption) in change may
of projects
new from listeners. of
social trends two behavior adoption. include technology
or completed.
have in of investment. magazine building
Project associated reduction new decreases,
change further or completion
Indicators number answered, or
activities equipment,
(Primary may reflect change of decreases derived
are positive cost extension but
nutrition. effects rates campaign projects
that of significance
productivity, technology, often
behavior reforms) include or disbursed,
Extension Indicators indicators, radio
initial because questions research
newspaper
Oriented technology. outcomes behavior options. of increases,
new may number as planned, laboratory
and indicators factor improved on production adoption and
investment, and as funds
indicators activities--change of policy quality assistance.
Group) yield
to and farmer
level and possible effects of the
erosion,
Research Productivity Target Goal agricultural increased return environmental rates, Productivity possible program adoption) Examples adoption made management Initial include or effects technology Extension--Data reports trained, attendance, implemented, readership, Research--Research Challenging completed about Examples: construction, technical
be on as
due
the the each
goods
Agricultural in of
project cannot project Objective and
of outputs groups activities relatively (2000).
the of the on works,
PDO
Framework change indirectly to control project.
usually outputs target performance indicator/s indicator(s). physical
Bank-financed project
in of project. Byerlee
which are of
a effects full outputs. the
of
Examples and and of focuses
Results to of related Development funding, the and
measure
and or life under quality change by
goal project associated framework,
M&E Alex
measures
indicators project, institutional
the closely not the of Project outcome products control include
in with from
Frame the intermediate are results an the level
indicators are only
these to direct input provided
Definitions during are but reflect behavioral the has
Log in in are
in Goal They changes within
development "adoption" frame, outcome output quantitative and Adapted
2.3: as
an are services
the log
Level Sector broad contributes. Changes attributable measured project. Outcomes clients. activities, project. Outcomes produced, well to In has whereas component Outputs and Project simple completion. Budget and Source:
Table
11
3534-Rikka.pdf 6/20/05 2:51 PM Page 12
Table 2.4: Examples on Outcome Indicators Depending on the Project Development Objective and the
Perceptions of the Different Stakeholder Groups Involved in Agricultural Research and Extension Projects
Focus of Stakeholder in ARE project
the Project Donor-- Borrower-- Research or Farmer or
Development World Ministry of Extension Academic Private Producer
Objective Bank Agriculture Agency Institution Sector Group
Poverty x% increase x% increase x% increase x% increase Sales of crops Increase in
reduction in poor small in poor smal in farm in farm increased by number of
holders' lholders' income income x% meals with
household real household meat or fish
income income
Productivity x% increase in x% increase x% of farmers Number of Input sales Staple food
change1 small holder in small have access to staff with increased x% yield increase
crop pro- holder crop improved improved by year x
duction and production varieties technical
farm income skills
Institutional Number of Number of Generation Number of x% of the Extension
development2 regions regions and transfer of staff in farmers have agents visit
covered by having good quality collaborative access to rural
good quality decentralized technologies ARE company communities
ARE services ARE services to x% of programs inputs and/or frequently
farmers services
Market Farmers' Farmers' Demand for Change in Number of x% higher
development access to access to high-value farmers' production price from
markets markets commodity ability to contracts with vegetables
improved by improved by research market farmer groups
x% x% increased products increased
Capacity x% increase in x% increase in x% increase High level of Number of x% increase
building farmer groups autonomous in partici- participation contracts with in farmer
participation in farmer patory among farmer groups group
decision- groups' technology farmers increased membership
making in the formation in development
area the area
Environmental x% of farmers x% of farmers x% increase in Number of x% of farmers Reduction in
sustainability adopting adopting generation of trainings to trained in safe pesticide
environ. sound environ. sound environmen- extension staff pesticide use applications
production production tally sound in environ. or increase in
practices practices technologies sound land under
practices no-till
Source: Riikka Rajalahti, World Bank.
Note: Bank-supported agricultural R&E projects typically focus on two objectives: (1) institutional development to strengthen the
institutions and technology transfer system necessary to develop and disseminate improved technologies and management
practices in the agricultural sector; and (2) productivity change due to technological innovation introduced through the technology
system.
12
3534-Rikka.pdf 6/20/05 2:51 PM Page 13
2. Who are the stakeholders?
As evident from table 2.4, indicators need to be defined and monitored together with
the relevant stakeholders. Different stakeholders have different perceptions on desired
outcomes, which need to be tracked with different indicators.
3. What do we really need to know?
To save time and resources and to avoid confusion, it is important to carefully think
through the real and relevant information that the stakeholders will need. A good prac-
tice example on participatory stakeholder consultation and indicator selection in
farmer-to-farmer extension project is shown in box 2.1.
BOX 2.1: PARTICIPATORY INDICATOR IDENTIFICATION IN AN EXTENSION PROJECT IN MEXICO
In a farmer-to-farmer extension program in Mexico, the project team followed these steps to develop
indicators:
� Define broad indicator areas (based on PDO)
� Select currently used indicators for these areas
� Define stakeholder groups
� Select stakeholder groups to be consulted
� Develop indicators with different stakeholder groups
� Test the developed indicators across different stakeholder groups to assess their significance to
others and effectiveness at indicating change
� Agree on a priority list among indicator options
� Carry out fieldwork to gather data for indicators
� Create lists of indicators for full evaluation use with a focus on indicators with specific impor-
tance for different actors with a limit, (e.g., three key indicators for each stakeholder group)
The program team identified the range of different institutional and individual actors who affect and are
affected by the project. They then prioritized three stakeholder groups to be consulted for indicator
development in this trial phase: (a) farmers (participating and non-participating), (b) farmer-extension
agents and (c) funding agencies.
The research team initially proposed seven indicator areas. These were eventually narrowed down to four,
based on the groups' objectives: (1) changes to local, regional, political, and sectorpractice and policy (e.g.,
level of dependence on external resources, involvement of local people, growth of local institutions, and
changes in policy and practice); (2) dissemination impacts: extension to other localities or regions (e.g., hor-
izontal and vertical linkages with other projects, agencies and NGOs beyond the region); (3) changes to the
roles of the individuals in the project (primarily the coordinator, outside advisors, immediate project partic-
ipants, and NGO staff); and (4) changes in the institutional structure (within and beyond the actual project).
Source: Baluert and Quintanar (2000) in IFAD 2003.
4. What is the most efficient method of collecting this information?
To save time and resources, find a balance between the information needs and the
efforts required to collect the information. Each indicator requires arrangements for
monitoring--an M&E plan; and inclusion in the Management Information System.
The effort expended should match the improvement gained in decision-making.
13
3534-Rikka.pdf 6/20/05 2:51 PM Page 14
Not all potential indicators are practical. They may be either too difficult or too expensive to gather
the data. For example, measuring actual rural income levels may be difficult or costly. Instead, proxy
indicators3 such as the number of community bicycles or the number of televisions may be a more
practical, if less precise, indicator of the general trend. The client of the project is a useful source of
suggestions for proxy options. See chapter 3 for further information on data collection.
Principles for Selecting Indicators
Although the key questions listed above guide the selection and development of indicators, sound
indicators must also be characterized by TQQ: time (time-bound accomplishment), quantity (numer-
ically measurable), and quality (what level of quality or degree of achievement is desired), respec-
tively. See box 2.2 for an example on indicator development.
BOX 2.2: TQQ OF DEFINING INDICATORS
1. Set Time: Access to agricultural extension services for poor farmers in Quynh Luu District is
improved by the year 2007
2. Set Quantity: Access to agricultural extension services for poor farmers in Quynh Luu District is
improved by x% from the baseline by the year 2007
3. Set Quality: Access to good quality agricultural extension services for poor farmers in Quynh Luu
District is improved (where good quality is defined as . . . ) by x% from the baseline by the year
2007
Source: Riikka Rajalahti, World Bank.
The TQQ serves as a general good practice guideline for selecting indicators. More detailed princi-
ples for selecting key performance indicators require that the indicators are:
� Relevant to the development objective of the project;
� Unambiguous as to definition, valid and targeted in reflecting changing conditions, and
consistent and verifiable in measurement;
� Meaningful and of interest to project management and stakeholders;
� Significant and selective in that the monitoring system should limit the number of indica-
tors to a manageable number;
� Practical in terms of the ability to collect data in a timely and cost-effective way with a rea-
sonable degree of reliability;
� Informative of project impact disaggregated by gender,4 poverty status, age, ethnicity, or
other characteristic of the client group;
� Acceptable to the Borrower, the Bank, and other stakeholders;
� Quantifiable to the extent possible, or, if qualitative, based on a common understanding of
what constitutes success; and
� Responsive in reflecting changes due to project activities and sensitive in demonstrating
change in as a short time period as possible.
14
3534-Rikka.pdf 6/20/05 2:51 PM Page 15
Key lessons in developing indicators
1. Involve partners and target communities in the selection of indicators.
2. Avoid collecting too many indicators (e.g., one or two per Project Development Objective
and 1 to 3 per each project component). Look for coherence in the selected set.
3. Select indicators that are reliable proxies.
4. Look for unanticipated, negative and positive program or project effects.
Decide how tracking results would guide project implementation
The Results Framework includes a column titled "Use of Results Information."
� This column describes when and how to take corrective action if there are problems in
meeting the agreed targets. Clear and time-bound targets for the selected indicators mea-
sured throughout the project lifespan to support management and decision-making are
needed to reap the full benefit of this information.
� The level of detail and flexibility in these guidelines can vary. An example of a rather detailed
guidance is: If the rate of women and men farmers' satisfaction with access to services is less
than 50% two years after the implementation, the activities will be reviewed and adjusted.
Cover additional M&E features in the text of the project appraisal document
The results framework does not explicitly capture the CAS or sector goals and output levels, nor the
detailed project component activities and input levels. However, Bank-supported projects are
required to articulate a clear alignment between the project and higher order strategic, program, or
sector outcomes to which the project contributes. The "Strategic Context" section of the PAD
describes how the project contributes to these higher level objectives. Furthermore, project inputs,
activities, and outputs are now required to be captured in the main text of the PAD.
The results framework does not explicitly capture the critical assumptions (factors that the project
can not or does not want to control but which are essential for reaching the desired outputs and out-
comes of the project). These critical risks, assumptions and possible controversial aspects need to be
discussed in the main text of the PAD.
Practical implications
Although it is mandatory to develop the results framework and the arrangements for results moni-
toring, the project team may also develop full log frame that introduces an added level of detail for
component inputs, outputs, and outcomes.
� The extended log frame approach may help the project team and the client to precisely
define the different components, their inter-linkages and contributions to the Project
Development Objective.5
� The extended log frame can be a useful tool in facilitating discussions and development of
the project design with the national team.
� It may also assist and complement project management, and may support the development
of a sustainable M&E system beyond the lifetime of the project as it is often a better fit
with the requirements for M&E in the client countries. See Appendix 3 for an example of
an extended log frame.
15
3534-Rikka.pdf 6/20/05 2:51 PM Page 16
Supervision and monitoring of the project
After the project and the results framework have been designed, the M&E arrangements specified
and agreed on, it is essential to pay adequate attention to implementation and supervision of the over-
all project as well as monitoring in particular.
Responsibility for supervision of a Bank-financed project rests with a task team (TT) assigned to the
project. Preparation for supervision is a joint task with the client and begins during project preparation,
when the TT and client staff agrees on arrangements for project implementation. These arrangements
include: (a) monitoring and evaluating of progress in project implementation and achievement of devel-
opment outcomes, with a common set of criteria to be used for project monitoring and evaluation;
(b) use of performance indicators; and (c) reports such as audit reports, Project Monitoring Reports
(PMRs), and other reports required for effective project monitoring, evaluation, and disclosure--to be
provided by the client, along with their outlines, content, and format.
Unfortunately, supervision missions frequently do not pay adequate attention to M&E. To ensure that
M&E will be an integral part of the project activities, emphasis of the importance of the following
issues is useful during the project launch workshop(s) and later during the supervision missions.
These issues are:
� emphasis on sufficient M&E capacity from the beginning of the project;
� timely establishment of the project M&E system;
� collection and use (by TT, PIU, management, and government) of project M&E data from
the very beginning; and
� the need to remedy the identified deficiencies in a speedy manner.
The TT is expected to prepare the "Implementation Status and Results (ISR)" report, formerly
known as "Project Supervision Report (PSR)." The ISR is a result of streamlining of the PSR to
make it a more effective tool for overseeing the implementation of projects and monitoring their out-
comes. A new module has been introduced for reporting on project outcomes that can be monitored
during implementation. See box 2.3 for details on the ISR quality checklist on Project Development
Objective, Outcomes and associated indicators.
16
3534-Rikka.pdf 6/20/05 2:51 PM Page 17
BOX 2.3: THE IMPLEMENTATION STATUS AND RESULTS (ISR) REPORT QUALITY CHECKLIST
The Implementation Status and Results Report require reporting on the following issues related to the
Project Development Objective (PDO), Outcomes and associated indicators:
1. Indicators for PDO and Intermediate Outcomes
� At least one indicator entered for PDO and Intermediate Outcomes.
� Baseline values entered for all indicators.
� Target values entered for all indicators.
� If project is two or more years into implementation, is any useful information entered on indica-
tors' progress to date?
� Explanation if quality of outcome information is rated poor or is unavailable.
2. Project Development Objective and Outcomes
� PDO rating needs to be consistent with other information in ISR.
� PDO rating and explanation needs to be adequately justified taking into account: implementation
performance, major risks, achievement of major outputs, achievement of intermediate outcomes,
extent to which PDO and project design remain relevant.
� If PDO rating is unsatisfactory, relevant issues and actions need to be presented.
Source: Operations Policy & Country Services: Implementation Status and Results Report.
http://intranet.worldbank.org/WBSITE/INTRANET/UNITS/INTOPCS/INTINVLENDING/0,,contentMDK:20296324~hlPK:
430750~menuPK:388717~pagePK:64137152~piPK:64136883~theSitePK:388707,00.html
17
3534-Rikka.pdf 6/20/05 2:51 PM Page 18
3. DATA COLLECTION, REPORTING,
AND DISSEMINATION REQUIREMENTS
After key performance indicators have been selected, the next step is to specify the M&E arrange-
ments for data collection, reporting, dissemination and use for decision-making. The project team
together with the partners must identify the following:
1. Data collection arrangements, including data sources and the reliability of the infor-
mation provided and associated costs and responsibilities; and
2. Institutional issues, including how M&E complements and assists project manage-
ment, what are the implications for institutional responsibilities and would participato-
ry M&E lead to capacity building of communities involved;
3. Human and institutional capacity, including extent of local capacity for M&E and the
need for capacity building and the associated costs.
3.1. MONITORING AND EVALUATION PLAN
Data collection arrangements require the development of a comprehensive monitoring and evalua-
tion plan and a management information system (MIS). An important step in developing the plan and
MIS is the identification of data collection sources and methods.
Development of a comprehensive monitoring and evaluation plan becomes an essential part of the
project preparation and Project Appraisal Document. Elements of the M&E plan should include:
� What? Type of information and data to be consolidated;
� How? Procedures and approaches for M&E including methods for data collection and
analysis;
� Why? Identify how the collected data will support monitoring and project management;
� When? Frequency of data collection and reporting;
� Who? The responsible focal points/resource persons for M&E; their responsibilities and
capacities to conduct required M&E functions; and
� Where? Information/data flow chart from M&E focal points/resource persons (implement-
ing agencies); data flow from decentralized levels to the central M&E unit.
The above questions have to be carefully assessed when designing the project. Inadequate planning
for data collection and use has been one of the main challenges in the Bank's project design and
preparation--with negative consequences for timely implementation and project management as
well as M&E. The Project Appraisal Document requires a table for Arrangements for Monitoring' an
example of the China Agricultural Technology Transfer Project is presented in Table 3.1. It provides
the scope to compare baseline information with target values, and to specify data collection frequen-
cy and methods as well as reporting responsibilities.
18
3534-Rikka.pdf 6/20/05 2:51 PM Page 19
for
of PPMO
experts.
Collection
and and
Team Team/ Team/
Reporting Data
Responsibility
and PPMOs/CPMO PPMOs/CPMO Expert national international Consultant/PPMOs (Shaanxi Heilongjiang) PPMOs/CPMO Expert PPMOs/CPMO Expert Shaanxi
of
Collection
survey by
records. set at team
Collection a
Data assessment questions. survey
surveys
Instruments on Team
and
Data Expert
Household Provincial Expert based guiding Visitor conducted fairs technology markets. Member Expert Assessment Assessment the
Project"
Reports closing.
at
mid- convened
Transfer and and
and
closing. and
reports entry, Team
and
mid-term
Technology Frequency Survey project term Annual Expert at Annual Mid-term closing Mid-term closing
YR5 sites
60 10 70% 30% 60 70 Satis- factory
Agricultural Project
YR4
Values
"China Transfer
YR3 sites
for 20 4 70% 15% 60 70 Satis- factory
Target
YR2
Technology
Monitoring YR1
Agricultural
6
Outcome project project project
Baseline
Determined at entry Determined at entry Determined at entry 0 0
for
Document:
of
in the
targeted for in
in in size. to
of farmers. the assessed
of Appraisal
Indicators investment One: displayed centers successful successful Team.
models sites Two: Three:
successful by farm satisfaction as aligned strategies
of measured of
Arrangements
increase particularly poor rate by as
income of Team indicators associations provision and
Expert Project
3.1. component 7
fairs provincial
Outcome Company-farmer assessed exhibition member the
Percent farmer farmers, resource Government in partnership non-project province. Percent components/sub-projects as Expert Results each Component Adoption technologies in and Component Percentage farmer service members by rating Percentage sub-projects by Component Investments with policies Source:
Table
19
3534-Rikka.pdf 6/20/05 2:51 PM Page 20
3.2. PROJECT MANAGEMENT INFORMATION SYSTEM
Apart from and in addition to the above described data requirements to satisfy the World Bank's
information needs on project progress, effective project M&E requires the establishment of a proj-
ect management information system (MIS), the project's internal system for collection, analysis,
storing, and dissemination of project information.
The new ARE instruments, such as competitive research grant programs for research and extension,
have increased the importance of an effective MIS due to significant requirements on tracking of
implementation and amount of funding allocation for several different subprojects. For Bank-
financed projects, the MIS should be described in the Project Appraisal Document as the project's
internal data management system.
What is an MIS?
The primary purpose of an MIS is to support management in making timely and effective decisions
for planning, monitoring, and managing the project. An MIS is essentially a system that uses formal-
ized procedures to provide management at all levels with appropriate information from internal and
external sources (Vernon 2001).
MIS generally consists of accounting software and a database management system for planning
and non-accounting information. Besides software and hardware, MIS comprises four elements:
(a) the actors who take decisions on the project; (b) the data and information that is useful for
decision-making; (c) the procedures that determine how the actors relate to the data; and (d) the
tools that facilitate the collection, analysis, storage, and dissemination of the data (Lecuit et al.
1999).
What should the MIS for ARE project do and be like? One essential feature of any MIS is that it
should operate using the regular client processes as much as possible. Thus, design and planning of
the system requires systematic dialogue with the stakeholders.
Other key desired features of an MIS in general, and of a Competitive Research Grant project in par-
ticular, are the following (Srivastava et al. 2003) (for an example on MIS pre-requisites for an effec-
tive M&E system, see box 3.1):
1. The accounting system should be linked with the subproject monitoring;
2. The MIS tools should be organized modularly (different functions of the system are
managed by distinct modules, which in turn are integrated into a common, central
database);
3. The MIS should adapt to the decentralized organization and operation of the
Competitive Grant project;
4. Any new project components should be managed by the MIS in order to prevent the
multiplication of management tools;
5. The MIS should facilitate the management of impact information;
6. MIS tools should facilitate decision making; and
7. Information management should be secure.
20
3534-Rikka.pdf 6/20/05 2:51 PM Page 21
BOX 3.1: MANAGEMENT INFORMATION SYSTEM (MIS) PREREQUISITES FOR AN EFFECTIVE
M&E SYSTEM
A computerized Management Information System (MIS) is the window that captures the quantitative
data within the M&E system. It will capture both progress and process data for concurrent monitoring
of activities at every level within the project.
Starting point: The starting point of the design of an MIS is to identify the inputs, the outputs, and the
process to be followed. This should then be followed by a clear and lucid description of the project cycle
and the results and/or logical framework that includes an institutionally defined set of indicators. Indicators
can be categorized as progress (P) indicators, physical progress indicators (PP), and Financial Progress
indicators (FP). Process indicators can be classified as including input, output, and outcome indicators.
Critical elements in the design of an operational MIS
� Census Data or other national data on population along with geocode information on the classi-
fication by village, district, or other administrative borders need to be available for uploading
into the MIS. Demographic Data with habitation, income, and poverty level data at the lowest
level should also be available for loading directly into the MIS.
� Reports need to be organized as being accessible to the general public and those available for
project management purposes, only.
� In today's web connected world a Central Server that is either owned or leased needs to be
secured. All of the data will reside on this central server; access to this data will be based on pre-
defined protocols and compartmentalized rights of use.
� Hardware and software and peripheral equipment will be connected to the central server in order
to access and process information.
� Trained personnel will be needed at each level to manage and operate the equipment. At each
level within the project structure a minimum of two persons (coordinator and analyst) will be
required to input, process, report, and access information and data for that particular level.
Source: Ghazali Raheem, Consultant, World Bank.
How to develop MIS for ARE? Lecuit et al. (1999)8 provided detailed guidelines for MIS for social
funds, with principles, design features, and terms of reference, that are useful also for ARE proj-
ects-- especially those using competitive funds and contracting. Another good source is by Vernon
(2001) on MIS for agricultural research organizations.
When developing a MIS, the following issues require special attention:
1. Design and planning--the first steps:
� Ensure stakeholder dialogue in the design and planning.
� Take time for needs assessment, find out about local information technology capac-
ities, and what information is needed and by whom.
� Examine what has been done in other competitive funds and ARE projects.
� For technology options, see box 3.2.
21
3534-Rikka.pdf 6/20/05 2:51 PM Page 22
BOX 3.2:TECHNOLOGY OPTIONS IN DEVELOPMENT OF MANAGEMENT INFORMATION SYSTEM (MIS)
In an interconnected world the technology options available today are either web-based, stand alone,
web-enabled or Web-enabled systems using Excel templates. Connectivity is a critical issue in most
computerized MISs, however, most stakeholders will require that web-enabled M&E systems need to be
developed as connectivity will increase and improve in the future.
Hardware, software and options for connectivity
Equipment will consist of hardware, software, database, connectivity, and other peripheral tools. At each
level within the project, equipment needs to be available and trained personnel should be in position to
use the equipment. Training should be provided in overall management, database use, hardware opera-
tions, network management, and operations and software use.
Equipment National or Central State or District Levels
Hardware Owned or leased server with database Workstations, desktops and
support for application development laptop computers
& hosting.
Software Database programs along with spreadsheets, Internet Explorer, spreadsheet,
PC-based database and statistical software PC- based database programs
Database SQL server RDBMS none
Connectivity Leased line VSAT dish, leased line/dial up modems
Others Registered domain name Internet connection
Source: Ghazali Raheem, Consultant, World Bank.
2. Staffing and responsibilities
� Ensure adequate and qualified staffing.
� Designate responsibilities for compiling, checking, and follow-up on reporting.
� ARE projects generally requires M&E specialists of the Project Implementation Unit
(PIU) to carry out required data collection, analysis, and reporting functions.9 The
specialist group is expected to collect and summarize data from the appropriate insti-
tutions (e.g., research institution) at an appropriate level of detail in a project data-
base or MIS. Project Implementation Unit is also recommended to offer training
courses to subproj-ect leaders to ensure that the required information will be made
available and prescribed in the uniform reporting process. Attention to these details
may significantly simplify the data collection, analysis, and reporting for the PIU.
3. Data and targeting
� Quantify expected inputs and outputs.
� Establish indicators or milestones for each subactivity.
� Provide gender-disaggregated data where possible.
4. Adequate procedures
� Establish procedures for regular flow of information between decentralized imple-
mentation units and a central M&E unit.
22
3534-Rikka.pdf 6/20/05 2:51 PM Page 23
� Make it modular, integrate all information needs (across ministries, extension cen-
ters, research stations, and subprojects), including accounting, into a single flexible
system with a common central database (usually at the PIU).
� Be ready when the project starts by piloting and troubleshooting the system prior to
starting implementation, otherwise information will be lost and heavy workloads
will be required to catch up.
� Be careful in the transition from design to implementation; lack of continuity can
lead to delays.
5. Evaluation
Retain flexibility. Evolve with the project and evaluate once a year. Pick up problems
early.
Good practice examples. Babu et al. (1997) describe MIS for extension programs
(http://www.fao.org/docrep/ W5830E/w5830e0k.htm). Box 3.3 and Box 3.4 describe good practice
examples of an MIS in an agricultural research project in Ecuador and the Project Information
Management System Network in India.
BOX 3.3. KEY FEATURES OF THE MANAGEMENT INFORMATION SYSTEM (MIS) FOR THE
MODERNIZATION PROGRAM FOR AGRICULTURAL SERVICES PROJECT (PROMSA) IN ECUADOR
The PROMSA Project Implementation Unit (PIU) in collaboration with Natural Resources International
(NRI) and the Competitive Fund Management Unit (UEFC) developed a state of the art M&E system
for the competitive funding scheme in Ecuador.
For individual projects financed under the competitive fund, a simple system of objectives and mile-
stones (for inputs, activities, and outputs) was established, with monitoring based on information in
quarterly reports.
The annual evaluation of subprojects is based on annual reports by researchers; establishment of partic-
ipatory grupo de referencia of users (reference group) and other interested individuals for each project;
and bi-annual visits by UEFC to each project. In addition, the UEFC has recently solicited feedback
from project leaders through a survey of their opinions about the processes employed in the first three
calls. Data on all projects and milestones are being recorded in a database managed by UEFC.
The PROMSA Competitive Fund employs a system of "alert" signals or flags that indicate if a project has
become a "problem project." While in "alert" status, a project is not eligible for payments from the Fund.
M&E of the competitive fund as a whole is based on several mechanisms. These include:
(i) The annual operating plan of the UEFC combined with quarterly reports on implementation
according to the proposed work plan;
(ii) the project data base established by UEFC which will provide the PROMSA-PIU direct access to
data on implementation of projects, and overall performance of the project portfolio;
(iii)visits by PROMSA-PIU officer to projects and other UEFC-organized activities to observe first-
hand processes being employed in administering the competitive fund; and
(iv) special studies and external evaluation of the operation of the competitive fund and its outcomes
and impacts.
Source: World Bank (2004a). Agriculture Investment Sourcebook.
23
3534-Rikka.pdf 6/20/05 2:51 PM Page 24
BOX 3.4. PROJECT INFORMATION MANAGEMENT SYSTEM NETWORK
OF NDIA S
I ' NATIONAL AGRICULTURAL TECHNOLOGY PROJECT
Project Information Management System Network (PIMSNET) has been designed, developed, and
implemented under the subproject "Institutionalization of Research Priority Setting, Monitoring and
Evaluation and Networking of Social Scientists" of the National Agricultural Technology Project
(NATP). It is coordinated by the team in the Division of Computer Applications, Indian Agricultural
Statistics Research Institute.
The purpose of the PIMSNET is to:
1. Effectively assess research efforts against well-defined targets;
2. Avoid duplication of research efforts;
3. Provide feedback to research planning process; and
4. Help to establish link between performance evaluation and incentive mechanism.
PIMSNET is being used for M&E of 845 research projects under NATP. It has four modules: Data
Management, Monitoring, Reports, Query, and Administration. For technical details, see the website
www.pimsnet.gen.in.
The monitoring module acts essentially as a monitoring system for NATP.
� The module stores information on the progress of the projects for the different activities in terms
of the expected targets achieved (several quantifiable indicators), and if not, the reasons thereof
and the steps taken to achieve the targets as reported by the Principal Investigators.
� It is possible to monitor the half yearly and annual progress in terms of three types of indicators
for physical progress, financial progress, and scientific or technical progress.
� The research achievements, shortcomings, and difficulties faced by the researchers are captured
by the system and the management can effectively monitor the progress of all the projects right
from their desktop.
� The system has a built-in algorithm to determine the projects which are not performing well/well
performing/very well performing/excellent performing projects.
� Provision has also been made to review the progress report by the Site Committees, State Level
Committees and the Peer Review Teams.
The output modules for the reports and queries through which top and middle level management
may get reports and queries as per their specific needs. The system is able to produce the following
reports:
� General Reports are ready to use reports at macro level generated on the basis of single criteria,
such as projects running under a specific agro-ecosystem, on a particular thematic area, in a par-
ticular state, etc.
� Custom Reports meet the specific needs of the management and are generated on the basis of
options selected by the user based on multiple criteria selected from a ready-to-use list of
options for a desired project. The module generates reports meeting all the selected criteria. For
example, a selection option might be to get a report on projects running under Irrigated-agro
ecosystem program in the thematic area of Integrated Pest Management and having a budget of
more than Rupees 20 lakhs.
24
3534-Rikka.pdf 6/20/05 2:51 PM Page 25
� Specific Reports are related to the details of a particular project. These are of utmost
importance when management needs to find out specific details of a single project. These
reports contain all the approved items for the project, like equipment procured, contractual
services, training obtained, civil work carried out etc. as well as project objectives, budget,
and the experts involved.
� Administrative Reports act as a tool for the administration and management of the informa-
tion storage and user management.
Source: Dr. S. D. Sharma, Indian Statistics Research Institute.
3.3. INSTITUTIONAL AND HUMAN CAPACITY REQUIREMENTS
FOR PROJECT M&E
Adequate institutional arrangements and institutional and human capacity are essential for any M&E
project, including functioning of MIS. The level of skills required depends on the complexity of the
project. Competitive Research Grant Projects and projects using contracting and involvement of a
wide range of stakeholders across several R&E subprojects are demanding in M&E capacity.10 As
evident from the PROMSA example, the implementation of a well-functioning M&E system both at
the subproject and the overall program level can be a major challenge.
The following features are recommended for developing institutional arrangements for M&E of ARE
projects:
1. Establish a centralized M&E unit. In general, projects may either establish an M&E
unit which is integrated into the Project Implementation Unit or not have a centralized
M&E unit but share M&E tasks among the implementing partners and primary stake-
holders. For complex ARE projects, it is recommended that a centralized M&E unit be
established within the main implementing institution (e.g., the Ministry of Agriculture).
2. Link the centralized M&E unit to subunits. The centralized unit should collaborate
with M&E units in other co-implementing institutions (e.g., extension agencies,
research centers, private sector implementers, enterprise development centers) and in
decentralized regions (e.g., province, district, and county level centers) where project
activities take place or have influence. Similarly, sub-M&E systems within the
Agriculture Ministry, located at a Ministry's institutional or policy planning unit,
should be linked to the central M&E unit within the Ministry of Agriculture.
These first two steps aim to guarantee that all project components will (a) provide sufficient project
and program level M&E links and a scope for effective communication between and within projects;
(b) ensure adequate reporting at the national or program level, and pinpoint any gaps or shortfalls
that may not be detected by the M&E system of a single project; and (c) provide an avenue for var-
ious project teams or Team Leaders with designated projects under the Ministry of Agriculture to
collaborate, share lessons, and ensure desired results.
25
3534-Rikka.pdf 6/20/05 2:51 PM Page 26
M&E of competitive grant funds to private researchers have particular challenges due to the individ-
ual nature of the grant operation.
� Given that many private researchers are not under the umbrella of any research or edu-
cational institution, tracking and eventual assessment of grant operation and research
results may be very difficult.
� It is recommended that such individual researchers be monitored by focal M&E persons
under the project's technical and supervisory Ministry. This would ensure proper
administration of the grant and generation of the expected research results.
Contracting extension services to private groups may have similar requirements.
� M&E at the project level would focus on the contracted group which will provide
tracked and assessed extension outcomes on a more systematic basis.
� On the positive side, contracted extension service provider may be in a position to:
(a) attribute specific services to the improved position of farmers; (b) pinpoint the real
effects of extension in a particular area and eliminate a mismatch by factoring out nega-
tive effects which may occur due to attributes other than that of extension (annual rainfall
fluctuation or declining production); (c) capture the information on indirect beneficiaries;
and (d) be provided sufficient time to ascertain extension progress and results.
3. Consider institutionalizing an M&E system within implementing institutions
(extension and research centers). This will provide a permanent M&E system
beyond the life of the project as well as meet Bank needs. Box 3.5 illustrates a sce-
nario on how M&E efforts of different projects implemented by the Ministry of
Agriculture can be harmonized.
BOX 3.5. SCENARIO ON HARMONIZING M&E IN THE NATIONAL SECTOR MINISTRIES
This scenario responds to the broader needs of the national research and extension programs.
1. A centralized M&E unit within the Ministry of Agriculture, coordinated by highly qualified
local staff (e.g., Ghana's Ministry of Land and Forestry, which houses a Policy Planning & M&E
Unit), will have information on all projects in the country that are being implemented under the
Ministry.
2. During the implementation of given projects under the Ministry, data or duplicates of data from
all projects will be fed into the centralized system such that at any point in time, or at any stage
in implementation, the central M&E unit will be in a position to determine what results or inter-
mediate results have been achieved and likewise where expected results have fallen short of their
planned/scheduled achievements.
3. Each specific project will also have an M&E sub-unit within their respective Project Coordinating
Unit (PCU) in the implementing Ministry, closely collaborating with co-implementing agencies as
well as decentralized implementing bodies. Therefore, project level data is managed at the level of
each PCU, but will be fed into the institution's central M&E system at the Sector Ministry.
The outcome of such a scenario facilitates formulation of coherent sector priorities, policies, plans, and
programs, as well as enhanced knowledge sharing. This scenario can also provide a step toward donor
harmonization.
Source: Remileku Rakey, Consultant, World Bank.
26
3534-Rikka.pdf 6/20/05 2:51 PM Page 27
4. Allocate sufficient time and resources to guarantee adequate capacity. Adequate
institutional and human capacity is imperative for effective functioning of M&E
frameworks within implementing institutions.
5. Assess existing local capacity for M&E at the design stage (e.g., by using institution-
al assessment in chapter 4). At a minimum, capacity must include: (a) the ability to
successfully construct indicators; (b) the means to collect, aggregate, analyze, and
report on the performance data in relation to indicators and their baseline; and (c) have
the skills and understanding to use the information effectively.
1. Identify important capacity gaps to fill (i.e., number and skills of staff);
2. Estimate the costs of capacity-building in M&E; and
3. Develop and establish a capacity enhancement plan (to be partly implemented
by the project). Factors to include in an M&E capacity enhancement plan: (a) the
identified skill gap; (b) target persons in the central and subunits of the Ministry,
project components, implementing agencies and beneficiaries; (c) timeline for the
needed skills; and (d) appropriate training times, trainers, and costs.
3.4. DATA SOURCES AND METHODOLOGY FOR DATA COLLECTION AND ANALYSIS
The following section reviews good practices in data sources and data collection. The section starts
with a discussion on baseline data, which is lacking in a large share of the agriculture projects.
Why to collect baseline data? The baseline is the first critical measurement of the performance
indicators and is used as a starting point, or guide, by which to monitor future performance of proj-
ects or programs (Kusek and Rist 2004). Therefore, baseline data should be collected at least for each
identified outcome indicator.
Because the success of a project will be, in part, measured by comparing target values with achieved
or actual values, setting target values is a sensitive issue and should be taken seriously. One method
to establish targets is to start with the baseline indicator level, use historical data or another estimate
of the rate of change to set the desired level of improvement--taking into account available funding
and other resources over the target period--to arrive at the performance target. Although it is tempt-
ing to set targets relatively low to assure they are reached, it is important to set the targets high
enough to ensure project implementation momentum.
A comparison of baseline to achieved values can be done in the following manner:
1. By comparing the situation before the project started with the situation after it started.
This requires understanding which factors influenced the outcome.
2. By tracking changes with (inside the project area) and without a project presence (out-
side the project area). This requires finding comparable areas.
3. By comparing the difference between similar groups (inside the project area) between
a project group and a group outside the project influence. This requires finding com-
parable groups within the same area.
What are the key questions to ask? Often baseline studies suffer either from information overload
and lack of use of the information or a baseline does not exist at all with a burning need to assess
27
3534-Rikka.pdf 6/20/05 2:51 PM Page 28
project progress and impact. When collecting data to define the baseline level and later on to deter-
mine current levels the following questions are necessary:
1. What are the sources of data?
2. What are the data collection methods?
3. Who will collect the data?
4. How often will the data be collected?
5. What is the cost and difficulty to collect the data?
6. Who will analyze the data?
7. Who will report the data and in what form and forum should data be reported?
8. Who will use the data?
9. Capacity building needs?
Links to examples of baseline surveys
1. Grosh, M. and P. Glewwe. (2000). Designing household survey questionnaires for
developing countries: Lessons from 15 years of the Living Standards Measurement
Study Volumes 1�3. World Bank, Washington, DC. Designing household survey ques-
tionnaires for developing countries: Lessons from 15 years of the Living Standards
Measurement StudyVol. 1.
� On agriculture, volume 2 lays out the most important agricultural policy issues in
developing countries, discusses the data needs and measurement concerns, and
introduces three suggestions for an agriculture module.
� Volume 3 presents draft questionnaire modules on several topics, including issues in
agriculture, among others a separate questionnaire on access to and use of agricul-
tural extension services.
2. The World Bank South Asia Region has developed a "generic" baseline survey for
water resources and irrigation projects. These surveys are accessible in the SASAR
intranet website: http://wbln0023.worldbank.org/Internal/SAR/SARInternalWeb.nsf/
SASRD/B2330FE4633F93E185256FAC006EAD6F?OpenDocument
What to do without baseline data? The collection of baseline data has been very often neglected
in Bank-supported projects in the past. It is highly recommended that the operational units make sure
that a baseline survey will be carried out before the project will be presented to the Board. However,
when no baseline data exists, alternative means must be identified (see box 3.6).
From where and how to collect the data for different M&E needs? The M&E systems of ARE proj-
ects, be it for a baseline or other M&E needs, must balance data needs between the ideal (and costly)
and the practical, and will generally need to draw on data from various sources. The choice of a given
set of methods for data collection depends largely on the specific objectives of the given task.
Regardless of the method used for collecting information, during the preparation it is also essential
to pay attention to:
� The sampling frames; sample size and sampling methods;11 and
� Preparation of data collection materials (e.g., questionnaires and interview guides.)
28
3534-Rikka.pdf 6/20/05 2:51 PM Page 29
BOX 3.6: WHAT TO DO WHEN THE PROJECT DOES NOT HAVE BASELINE DATA?
Quite frequently, projects and programs have started or have to start without baseline data at hand. In
this situation, the project team can consider the following alternatives:
1. Track changes with (inside the project area) and without a project presence (outside the project
area) if comparable areas can be found.
2. Use first measurements as a starting point (e.g., production during the first implementation sea-
son).
3. Look for existing local and national agricultural databases. In some countries, authorities regu-
larly collect data on farm and off-farm income sources, production data, and use of inputs.
4. Carry out interviews with local key informants (e.g., village leaders, elderly, and extension
workers) or focus groups (e.g., women's and savings groups and producer associations) to detect
changes in the availability and use of agricultural services and inputs, and yield data--historical
trends and timelines.
5. Other existing records, such as
� Data collected by other donor projects or NGOs.
� Training records--village or community and extension centers' records on the past topics, tim-
ings, participants, possible evaluations on trainings and field events.
� Research program records--topics, events, farmer participation, and variety release and adop-
tion.
� Data on input sales (e.g., on specific varieties) from government and private sector records.
Source: Riikka Rajalahti, World Bank.
There are many different data sources and methods (see Alex and Byerlee (2000) and IFAD (2003)
for more details), but here we focus on the three major data sources: (a) Management Information
System; (b) existing/regularly maintained databases; and (c) special studies (see table 3.2). In prac-
tice, it is often necessary to use a combination of methods.
1. Management Information System
Project input-output data as well as data on agreed indicators are generally available from the proj-
ect MIS. The section above includes more details on MIS.
2. National and International Level Databases
As it is generally not cost-effective for a project to measure national change of goal-level indicators,
information needs at this level rely on national and international databases. These databases may pro-
vide data for indicators such as rural population below poverty line, daily per capita dietary energy
supply, agricultural GDP, or public expenditures on agricultural research as percentage of the agri-
cultural GDP. Quality of national data may be suspect to poor sampling procedures or intentional dis-
tortions, but it is readily available and comprehensive, represents considerable investment in data
collection, and is official, government-sanctioned data. For example, relevant data may be collected
by the Ministry of Finance, Ministry of Agriculture, or Ministry of Science. These data can be sup-
plemented by international databases.12
29
3534-Rikka.pdf 6/20/05 2:51 PM Page 30
3. Special Studies
Bridging the gap between indicators from routine reporting on project activities and from national
data sets is a challenge for ARE projects, and reflects the difficulty inherent in linking project
performance and eventual impacts. To bridge this gap, project teams could consider using special
studies.
When to use special studies? The need for special studies should be considered already during the
project preparation as they are of relatively high cost and management intensive. Special studies are
a practical tool:
� For providing data disaggregated by gender, ethnicity, or other key characteristic of client
population;
� For stakeholders to learn about and affect the project and M&E process (see chapter 4);
� For collecting information on agricultural systems, evaluating project impacts, and espe-
cially for assessing and measuring project outcomes; and
� To track specific planned and unplanned impacts or operational issues (e.g., technology
access by women or minority groups, and changes in land tenure and other social condi-
tions in the area). An example of a diagnostic special study on the impact of extension on
women is presented in box 3.7.
How to conduct special studies?
1. Build special studies into the regular work plan of ARE implementing institutions,
starting from the project start-up, mid-term review. For example, extension agencies
must track changes in farmer attitudes, knowledge, and technology adoption; research
programs need to know results of new technologies; and training institutions need to
know how graduates perform. An example of how to use interviews and focus group
discussions for a mid-term evaluation is shown in box 3.8.
2. Clearly identify the stakeholders involved and the purpose of the studies. Beneficiary
and stakeholder participation is highly recommended (see chapter 4);
3. Select an appropriate methodology (see table 3.2 and chapter 4);
4. Pre-test any questionnaires and surveys (with the target group and the survey imple-
menter) and adjust as needed;
5. Consider using an independent institution to avoid overburdening the implementing
institution or to avoid bias in monitoring change within the institution;
6. Consider financing special studies through a project office independent of the
implementing agency;
7. Verify the validity of the collected data (e.g., by cross-checking); and
8. Analyze the information and prepare corrective measures (as needed) to meet the
desired outcomes.
30
3534-Rikka.pdf 6/20/05 2:51 PM Page 31
BOX 3.7. RURAL WOMEN'S EMPOWERMENT THROUGH AGRICULTURAL EXTENSION IN VENEZUELA
In Venezuela, the Agricultural Extension Project (PREA) implemented by the CIARA Foundation from
1995�2004 was designed to establish a new decentralized extension service while focusing on social and
gender aspects with the direct involvement of the beneficiaries through the participation of rural grass
root organizations. The rationale for the gender perspective of this project was that the gender approach
encourages participation and personal growth of women, which influences individual and family well-
being, strengthens the capacity for team work and consolidation of the rural associations and organiza-
tions leading to benefiting the men, the family, the community, and the society as a whole.
What? In terms of M&E, the diagnosis, indicators for productivity, and family and community activi-
ties were implemented with a gender approach. This required dedicated continuous work with program
users and monitoring of the experiences as well as the incorporation of a gender focal team into the
Project Coordinating Unit. As an example, a comprehensive diagnostic study was conducted for the
project in order to analyze the impact of extension activities on women in rural areas.
How? The Project included a gender sensitive M&E during project implementation by incorporating
gender data in the producer's technical files and allowing the rural women to be registered as produc-
ers and heads of the households. With this data a comparison of gender-related differences in agricul-
tural production, and family and community activities were made possible. Other applications of
gender approach were in: (a) training in theoretical and methodological gender equity tools; (b)
preparation of the participatory diagnosis done from a gender perspective; (c) design of the annual
Plan of Extension oriented in gender perspective; and (d) execution of several of the productive and
community projects to apply gender tools learnt under this project.
Findings: The membership of Civil Associations of Extension or Producers Grassroots Organizations
(ACEs) in municipal extension in the project included 2,696 rural women out of a total of 10,370.
Over one quarter of the project direct beneficiaries (45,000) and members of ACEs were women
(11,700), indicating the success of the project's strategy to ensure women's access to extension serv-
ices. Women were involved mainly in activities such as in micro-enterprising and group activities and
expressed their desire for additional services. In training, more extensionists received training in gen-
der than in technical matters reflecting the project strategy on social aspects of community develop-
ment. Of the 561 extensionists trained 118 were women.
This project experience demonstrated how attention to the family needs, women in particular, gives such
an investment a high level of social relevance and is a source of incentive to social participation. Nearly
25 percent of extension agents are women, and extension services for women have emerged as an impor-
tant part of the overall municipal extension program.
This work highlighted the need for early engagement of gender approach, defining gender-based key
indicators, and step-wise monitoring.
Sources: The World Bank, ICR Report No. 29081, dated June 29, 2004 & Draft paper on "Rural Women as Axis of Family
and Community Empowerment in the Agricultural Extension Project of Venezuela" by Maria Magdalena Colmenares and
Andrea Pereira, with input from Ninoska Loaiza and Indu John Abraham (reviewer). Additional input: Indira J. Ekanayake,
World Bank.
31
3534-Rikka.pdf 6/20/05 2:51 PM Page 32
BOX 3.8: USING INTERVIEWS ANDFOCUS GROUP DISCUSSIONS ATMID-TERM
EVALUATION OF A FARMER-LED EXTENSION SERVICE PROVISION PROJECT
Why? To evaluate the effectiveness of farmer-led extension in improving access to services among
smallholders in Vietnam
How? Using semi-structured household and focus group discussions at household, village and com-
mune levels.
Who? Interviews by the staff of a non-governmental organization--can also be carried out by local
associations, extension center staff, PIU staff, consultants, etc.
Specific focus of the "study:" to determine farmers' (women and men) access to and use of extension,
changes in the sources of extension, and satisfaction with quality and access to services.
Key findings: Volunteer farmer trainers (FT) were not yet sufficiently effective and reliable in provid-
ing extension services to fellow farmers in their villages.
Why? The FTs tended to retain the information and knowledge to themselves and did not openly invite
other farmers to their multiple demonstration sites. Also, other farmers were hesitant to contact FTs.
Corrective action:
� Awareness raising among villagers and farmer trainers on the role of FTs
� Continued effort to improve FTs' skills to inform and support farmers and organize field days
and other events.
� Regular reporting and meetings between FTs and commune extension coordinator.
� Local level monitoring (by other farmers and commune extension coordinator) of FT activities.
Source: Riikka Rajalahti, World Bank
When to collect and use the data and by whom? The responsibilities for data collection vary from
project to project based on the country and project context and must be specified in the M&E plan.
Similarly, the capacity building needs must be identified early on and are based on the specific
M&E functions and the complexity of the project. See table 3.2 for a summary on:
� Suitable timing for using different data sources during the project life cycle and with differ-
ent types of instruments;
� Cost and capacity implications; and
� Suggestions on who will collect and use the data.
32
3534-Rikka.pdf 6/20/05 2:51 PM Page 33
and
all
external analysis
udies
St producer
Formal sample and
surveys or
appraisals extension
Yes No No No Yes Yes Yes Often Often Often Implementing agencies, M&E Potentially stakeholders Med-high Med-high regularly
and
Special rural reviews
Rapid
3 and research
peer organizations,
and maintained
peers 2. external
Various
donor agencies are The
and high.
analysis needed needed consultants 3.
reviews ases
External are
No Yes No No As As Yes Often Often Often External and PIU, impl. High High ys. overnmentg vernmental
datab
bases surve
non-go
Databases data include
all
external national
and may
and diagnostic
Formal surveys MIS
cost diagnostic needed consultants,
Maintained for agencies by
As Yes No No Yes/No Yes Yes Often Often Often Implementing agencies, M&E Potentially stakeholders Med-high High
and activities informal
out
5 and
source 2 all M&E
Regularly M&E implications carried
be
Data complexity 4.Implementing
and of Rapid rural benefic. routine cost
needed can
appraisals of
and observations,
As Yes No No Yes Yes Yes Often Often Always PIU, implementers, external Potentially stakeholders Med-high Med-high
Level part M&E
a analysis).
Existing all as
structured
complexity
govt., External
centers of
System, Other economic 5.
routine statistics needed needed needed needed
established
Yes No As As As As Yes Always Always Always Local beneficiaries, ARE Potentially stakeholders Low Low level
is discussions,
the ex-post groups.
1
and and (MIS) and
group
Information
govt. donor producer
needed needed focus
System ex-ante
databases assumptions, and
National international Yes No No No As As Yes Always Always Always Ministry other agencies PIU, implementers Low Low and
valid
NGOs,
not interviews,
and
, Information reviews,
1 4 are
Bank
MIS impl. donor agencies peer
these providers,
If community
Sources--Management World
Methods No No Always Always Always Yes Yes Always Always Always PIU, Agencies beneficiaries PIU, impl. Low Med-high
Management and program service
Data project.
grants R&E Rajalahti, the sector
research
Major Collection of informant
When Who How Cycle user capacity Riikka Assumptions: key (e.g., private
3.2. Data Outcomes evaluation collects? 1.
Baseline Inputs Outputs IR evaluation Contracting Services Costs Staff
Project Appraisal Monitoring Mid-term Final Instrument Competitive Participatory Who Primary Resources Source: Note. irrespective include include centers, organizations.
Table Main
33
3534-Rikka.pdf 6/20/05 2:51 PM Page 34
3.5. MONITORING AND EVALUATION COSTS
There are typically two types of costs that are required in every project and are crucial for M&E
activities: (a) costs relating to project preparation and implementation activities; and (b) costs relat-
ing to the formulation of a budget.13
Project Costs: Bank project costs are usually prepared using Costab 32, a PC-based project costing
tool. Costab 32 is the project costing component of a software series to improve efficiency and effec-
tiveness of Project Processing activities. See Appendix 5 for more details.
M&E Costs: Project M&E costs on the other hand need to follow realistic budget estimates. The
most important factors to consider when estimating an accurate budget for monitoring and evalua-
tion activities are scope, methodology, and stakeholder participation. Evaluations requiring the par-
ticipation of a large number of stakeholders would require more funds than the evaluation of a single
project.
In preparing the M&E work plan it is necessary to allow adequate provision at the outset of the proj-
ect for all activities related to M&E. These will include:
� Costs for collecting baseline data;
� Concurrent monitoring of implementation activities;
� Independent evaluations and studies;
� Training and capacity building workshops to include MIS design development and imple-
mentation.
Given the various types of interventions and the development circumstances in each project, it is not
practical to prescribe the percentage of the total budget of every project that should be allocated for
monitoring and evaluation actions.
Typically, M&E costs comprise 10�15% of a project's total budget. These costs should be included
in the project budget on the respective budget line. Standard costs include baseline values, ongoing
monitoring, mid-term evaluation, final evaluation and terminal report as well as project-specific
costs (i.e., surveys to collect baseline data, workshops, cost of hiring staff, and M&E specialists)
should be taken into account. See Appendix 5 for an example of a typical M&E budget plan, includ-
ing a breakdown of key expense categories, for an IDA/GEF Environmental project.
34
3534-Rikka.pdf 6/20/05 2:51 PM Page 35
4. PARTICIPATORY MONITORING AND EVALUATION
Focused attention to poverty reduction and socially sustainable development embracing the socio-
cultural diversity among target populations has increased the need to engage in direct dialogue with
different stakeholder groups involved in development projects. One critical dimension of this
engagement is M&E implemented in participation with stakeholders. This section gives an overview
of participatory M&E and guidance for its use in ARE projects.
4.1. WHAT IS PARTICIPATORY M&E?
Participatory monitoring and evaluation (PME) is the active involvement of key stakeholders in the
M&E process in order for them to learn about and affect the process and impact of a development
project. The nature of stakeholder involvement in PME may range from voice or consultation, to
implementation and use of information; thereby ensuring the needed collective process of reflection,
planning, and management for desired outcomes and impact.
At the heart of PME are four broad principles:
� Participation--which means opening up the design of the process to include those most
directly affected, and agreeing to analyze data together;
� Negotiation--to reach agreement about what will be monitored or evaluated, how and
when data will be collected and analyzed, what the data actually mean, how findings will
be shared, and what action should be taken;
� Learning--becomes the basis for subsequent improvement and corrective action;
� Flexibility--is essential because the numbers, roles, and skills of stakeholders, the external
environment, and other factors change over time.
PME can be used in concert with traditional M&E but should be considered a different process. In
conventional M&E, participation is often limited to working with primary stakeholders as informa-
tion sources, rather than as joint users of information. PME, in turn, incorporates ongoing M&E by
local stakeholders throughout the project cycle and after the project has ended. Table 4.1summarizes
the main differences between conventional and participatory M&E.
4.2 WHEN AND HOW TO USE PME INARE PROJECTS?
Beneficiary and stakeholder participation in M&E is critical: (a) to ensure that ARE projects and
programs are responsive to the genuine needs of intended clients; (b) to empower target beneficiar-
ies and strengthen ARE institutions through better programs, accountability and transparency; and
(c) to determine the impact (also unintended impacts) on the intended beneficiaries and stakehold-
ers from their respective perspectives.
35
3534-Rikka.pdf 6/20/05 2:51 PM Page 36
Table 4.1. The Major Differences between Conventional and Participatory M&E
Facet of M&E Conventional M&E Participatory M&E
Who plans and Senior managers and outside experts Primary stakeholders, project staff, managers
manages the process and other stakeholders, often helped by a
facilitator
Role of primary stakeholders Information provision only Designing and adapting the methodology,
collecting and analyzing data, sharing
findings, identifying lessons learned and
linking them to action
How success is measured Externally defined, mainly Internally-defined indicators, including
quantitative indicators more qualitative judgment and stories of
personal change
Approach Predetermined and fixed Indicative and adaptive
Source: Adapted from IFAD (2003).
Step 1--Decide when to use PME
PME can and is encouraged to be used at any stage of the ARE project; and the degree of participa-
tion can vary from stage to stage. However, it is better that the overall project planning is participa-
tory and that PME activities are initiated at the very beginning of the project as this will increase the
likelihood of mainstreaming PME in the project cycle.
If resources are limited--time, staff and finances--it is better to prioritize when to use PME rather
than to sacrifice the quality of the activities. During the planning process, or later when questions
arise and participatory M&E approach is likely to be more useful, the project team together with the
beneficiaries and implementers decides the timing of specific PME activities (see table 4.2).
Step 2--Decide who to involve in PME
PME in ARE projects usually involves all stakeholders in the various facets of farm and non-farm
activities. One has to assess the need for and affordable degree of participation by the possible stake-
holder groups (see table 4.2). The following questions can guide the decision-making (Guijt 1999):
1. What is the relevant degree of participation for each stakeholder group? Is the process
of collating and calculating the information important, or only the final information?
2. Who is going to use the final evaluation? Those who are to use it should understand
on what the data is based and how it was calculated.
3. What skills does the analysis require? The more difficult, the more caution should be
used in encouraging broad participation unless it is clear who it will benefit and how.
Step 3--Decide the scope of the work: What and how to do PME
After deciding when to use PME, who to involve and the degree of participation by different stake-
holders, it is essential to clarify the methodology to be used as well as the logistics and funding. The
process usually begins during the project planning or later during implementation when the need
arises with a workshop involving a facilitator and representatives of stakeholders. The initial work-
shop should bring clarity concerning the aim of the PME process, roles and responsibilities, logisti-
36
3534-Rikka.pdf 6/20/05 2:51 PM Page 37
out
of
costs results.
the rements
if high difficult community
crowding it patterns, of stakeholders
needs needs clear indicators requi
of project requirements
objectivity process
the
resources resources available of
not challenging opportunity of
make level
intermediate
of of high
time-consuming risk trends, and
Challenges and are requirements capacity a and or capacity high stakeholder design
effects involve
non-functional groups the
and pose and to
become true wider in
determine
Potential Availability Capacity-building Facilitation Availability Capacity-building Facilitation Easily objectives indicators Capacity Potentially Harmonization across Time Beneficiaries' can May the outcomes to outcomes, Time Requires commitment the Need also
local theme.
directly
authorities. the
preferences and findings
providers staff, on that
the
providers, local
and providing on
and issues
different different service providing Project in depends on
service borrower. service in varies.
role
have have ARE level.
ARE and to providers
ARE to role
providers, key information
a forum
feedback
bank staff. key local
in
likely service likely a involvement and
service in the
are receive
are are at
M&E borrower.
ARE are provide (1998).
ARE to format
authorities, with and
impact. purpose--farmers, should
borrower
on Gaventa
local farmers, bank the
information chance
and
Process Involve? stakeholders--farmers, stakeholders stakeholders--farmers, stakeholders farmers, on authorities a
them--the and
to key authorities key
PME indicators. local stakeholders--specific stakeholders have stakeholders. Guijt
of Who The providers, Different perceptions Essentially local The authorities, Different for Primarily Depends and collecting managers information. All All and concern All
from
Stages
adapted
needed
as needed needed
as as evaluation
stage
Different evaluation
at "questions"
When? Design Evaluation Design Implementation Design Evaluation Implementation Design Baseline Monitoring Evaluations Monitoring Evaluations Baseline Evaluations Monitoring Baseline Final Mid-term
in participation
project Present findings
Participation for data
drawing the met of
the
of
of the
process M&E indicators the on
to changes data evaluation and with project been Level
is for evaluate
define the
opinion and from
Degree Participation "impact" views which have
what by purpose, and
themes and and stakeholders?
feedback to Authors.
4.2. of their collecting analyzing their
methods
PME context results communicate
Level in Define meant Design and Define monitor Select Give history the Help monitoring Help conclusions and Share primary and Give degree objectives Source:
Table
37
3534-Rikka.pdf 6/20/05 2:51 PM Page 38
cal arrangements, schedule, potential capacity issues, and finally the data collection, analysis, and
methodology issues.
In this process, the following issues are important to keep in mind:
� Systematic dialogue and facilitation during PME: The degree and timing of involvement
may vary, but a systematic dialogue between farmers, scientists, extension personnel, govern-
ment officials, the business community and their designate institutions and agencies is essen-
tial throughout the process. It may be necessary to choose an appropriate and non-biased
facilitator to guide the process (in a gender-sensitive manner) and to guarantee that key stake-
holders agree on the rationale, objectives, and plans of any PME activity before it becomes a
part of the project implementation process; and that the level of stakeholders participation, tim-
ing, modes, relevant sources of information, and costs for the PME activity are being clarified.
� Foster use of local knowledge: Local knowledge includes perceptions of stakeholders,
individual and collective. The thematic areas include the productive environments of stake-
holders and target beneficiaries. All these elements may be translated into criteria for evalu-
ating research and extension results.
� Avoid too ambitious plans. Allow sufficient time to build local skills for indicator selection
and collection and other assessments. It is better to start simply and monitor only some
aspects of the project. As experience grows and capacities build, the system can be expanded.
For example, in Brazil, farmer-based research on agro-forestry systems started by monitoring
plant diversity, labor input and ground cover. After the first year, farmers and scientists decid-
ed to include soil nutrients and the research process itself (Guijt 1999).
� Create incentives for stakeholders to participate in PME and sustain the system.
The choice of methodology
When information on stakeholders' opinions, views, and perceptions on ARE services are needed, the
people become the major source of information.Various quantitative and qualitative tools can be applied.
Stakeholder participation can be used to define the questions and questionnaires, to provide and collect
data, and to communicate the findings, particularly when actual field level information is needed.
Many of the methods useful for traditional M&E can be used in either a participatory or non-
participatory way. The participatory impact comes with the way a method is used and in the knowl-
edge of who helped to select it. For different tool options, see chapter 3, Appendix 4 (data sources
and methods) and PME tools below.
A. Beneficiary and stakeholder assessments
The overall objective of a beneficiary assessment is to help beneficiaries and other local-level stake-
holders identify and design development initiatives, signal constraints to their participation, and give
feedback on these activities to those designing and managing a project or formulating policy.
Beneficiary assessment includes a set of commonly used tools: conversational interviews, focus
group discussions, and participant observation. For details on the necessary steps in ARE projects,
see Beneficiary Assessment for Agricultural Extension: A Manual of Good Practice.
38
3534-Rikka.pdf 6/20/05 2:51 PM Page 39
During the project design stage, a broad stakeholder group assessment can clarify the vision and per-
spective with which research and extension initiatives will be undertaken. The perspectives of vari-
ous stakeholders can contribute to determining the directions for change and their respective roles
toward supporting and strengthening such a change process. Therefore, all R&D projects are
designed as a consequence of such perspectives. This assessment helps to define the Project
Development Objective and the definition for impact.
The assessment can be used for evaluation to review and assess the assumptions behind the project
activities and indicators after a specific timeframe of implementation; the relevance of the project in
the context of changing realities; the implementation of project activities; and to what extent there
are deterring factors to reaching desired outcomes. Similarly, it can used to assess project impact and
outcomes by way of changes that occurred among target beneficiaries and their environments as a
result of project interventions and draw lessons for future activities. The target beneficiaries as well
as the main implementers are at the center of this evaluation. See box 4.1 for an example of the key
steps to take during a participatory activity evaluation.
BOX 4.1: BASIC STEPS INPARTICIPATORY ACTIVITY EVALUATION
I. Group discussion --Activity Participants II. Activity Evaluation Report
Inputs to discussion Background
Monitoring data Relation to goals and objectives
Experiences and observations Specific purpose of the activity
Results � outputs and outcomes
Discussion and Analysis
Major impressions Analysis � social, economic, environmental and
Beneficial and problem outcomes institutional issues
Constraints Inputs and outputs, costs and benefits
Reasons Intangible � benefits and problems
Gender and other cross-cutting implications Gender and other cross-cutting implications
What to do next
Conclusions
Potential for adoption
Potential constraints
Recommendations
Potential to improve
What to do next
Source: Adapted from Souvanthong, Pheng and R. Trethewie (2001).
Conversational interviews
Conversational interviews constitute the basic tool of inquiry for the PME practitioner. Conversational
interviews often take place in the homes of the interviewees, who are apt to be most comfortable there.
Interviews should be conducted in the local dialect in such a way that open-ended questions revolve
around a number of themes or topics that project management has selected. The objective is to gain
39
3534-Rikka.pdf 6/20/05 2:51 PM Page 40
in-depth information on beneficiary views in relation to a planned or ongoing activity by encouraging
beneficiaries to speak freely and bring to light issues of concern to project management. Interviews can
be conducted on a one-to-one basis or in focus groups. The advantages of individual interviews are that
people are likely to speak more freely, without worrying what peers or other community members may
think. Lower-status or introverted members of communities may not feel comfortable speaking out in
groups. See box 4.2 for an example on a conversational interview with a field-focus.
BOX 4.2. USING CONVERSATIONAL INTERVIEWS WITH FIELD-FOCUSED APPROACH INAFRICA
What? Purpose of the task was to identify the magnitude of information dissemination from exten-
sion worker through contact farmer group to non-contact farmer group
How? Using conversational interviews with both members and non-members of contact groups (CG)
in ten African countries1
Findings
� All but one (Mali) Contact Group generally failed in its task as diffuser of information received
from the extension worker.
� In Senegal, a number of farmers who were listed as belonging to CGs did not know that they
were in fact members.
� Only a third of the farmers in areas covered by the extension knew of the existence of CGs.
� Ninety percent of the non-CG farmers did not know anything about the existence of CGs.
Root causes
� Insufficient supervision and publicity was given to contact groups.
� Lack of incentives for the CG members to pass information on to his non-CG member farmers.
Corrective measures
� These insights regarding the degree of knowledge about the CG and the lack of motivation of the
CG member to relay extension messages to non-CG neighbors came directly out of the conversa-
tional interviewing done with samples of farmers.
� The importance of these insights can be seen in the decisions to strengthen CGs in several coun-
tries or downplay them, to the point of elimination in Senegal, and in a general trend toward
more participatory extension models involving entire communities.
Source: Lawrence Salmen, Consultant, World Bank.
Unguided discussion is apt to be vague and, therefore, of little use for decision-making; probing for
specificity is often required (e.g., if the intended beneficiary of an extension project stated that she
did not benefit from it, the interviewer should probe to find reasons for this dissatisfaction, prefer-
ably prioritizing reasons (up to three).
Focus group discussions
For PME, in addition to enabling a wider coverage of the beneficiary population in a given time,
interviews carried out in focus groups can serve as a cross-check to individual interviews. The
40
3534-Rikka.pdf 6/20/05 2:51 PM Page 41
groups should normally comprise six to twelve people with common characteristics (e.g., groups of
intended beneficiaries may be comprised of married women, male heads of households, youth from
15 to 35, and so on). However, there are times when it may be of use to purposefully mix the con-
stituents of a focus group--say, with members and non-members of contact groups--in order to bet-
ter appreciate the nature of conflict and communication between them, and provide the opportunity
for indigenous solutions.
The interview guide should be used in conducting these interviews. The interviewer takes on a facil-
itative role, guiding the discussion to cover topics from the thematic guide and ensuring that every-
one has an opportunity to participate. This will generally entail encouraging the more reticent,
introverted persons to speak up while providing less encouragement to those most apt to dominate
the discussion. A researcher should also be present to take notes. Although the difficulty of quan-
tifying focus group discussions may be considered a liability, their use as a cross-check and as a
fairly rapid and easy-to-read barometer of the mood of a community on many topics makes focus
groups a useful component of the PME approach.
Participant observation
This technique generally involves protracted residence in a targeted community. During this stay, it
is expected that the participant observer will establish enough rapport and involvement to help him
or her accurately represent the conditions within the community as they relate to project objectives.
The participant observer normally spends from one to three weeks in a given community. The
researcher will focus on the areas of concern identified in the interview guide. Particular issues relat-
ed to agricultural extension that have been the object of inquiry using participant observation
include: (a) the relationship between contact groups and other community associations; (b) the role
of gender in dissemination of the extension message; and (c) the nature and extent of interaction
between an extension agent and the community he serves.
During this stay in the community, the participant observer should prepare case studies of five to ten
households based on repeated visits and observation. Participant observation, being costly and time-
consuming, should be used selectively on topics of particular interest that are of a sensitive nature
and lend themselves to this form of intensive personal interaction.
B. Institutional assessment
In ARE projects an institutional assessment at the design stage can help to determine whether insti-
tutions and organizations involved in the project implementation have appropriate organizational
procedures, systems, and structures in place. This assessment helps to identify gaps and define some
of the main project components.
Institutional assessment can be used for evaluations. The process involves an evaluation of the
respective institutions' program activities and target interventions relevant to a project, and the extent
to which planned outcomes are being achieved. The review highlights shortfalls that relevant insti-
tutions have experienced. The institutions' staff, including field researchers, extension agents, and
contracted extension service providers, are actively involved together with farmers associations and
other players in the field. The exercise is expected to bring about common and shared understanding
of apparent problems and collective efforts to solve them.
41
3534-Rikka.pdf 6/20/05 2:51 PM Page 42
For further details on institutional assessment, see:
� http://lnweb18.worldbank.org/ESSD/sdvext.nsf/81ByDocName/ToolsandMethods
Institutionalanalysis
� http://lnweb18.worldbank.org/ESSD/sdvext.nsf/09ByDocName/ProjectPreparation
ImplementationParticipatoryAssessmentsInstitutionalAssessment
� Horton et al. (2003) "Evaluating capacity development: Experiences from research and
development organizations around the world".
� Morgan and Taschereau (1996): Capacity and Institutional Assessment: Framework,
Methods and Tools for Analysis. www.acdi-cida.gc.ca/INET/IMAGES.NSF/vLUImages/
CapacityDevelopment/$file/1996-06-02Tools(Wkshp6).pdf
C. Participatory indicator selection
Participatory indicator selection, identification of appropriate indicator proxies, and allocation of
ME responsibilities are essential features of the design stage. The stakeholder and institutional
assessments reflect the stakeholder perspectives and help define the indicators.
It is important to avoid imposing indicators and/or methods. Organizations keen to facilitate the
development of local monitoring systems often impose their ideas of useful indicators or meth-
ods to some extent. It is also necessary to build indicator collection system on what exists local-
ly, collect indicators at optimal moments (based on agreed timing, frequency and responsibilities)
and avoid collecting too much data. See box 2.1 (in chapter 2) for participatory indicator selec-
tion process.
D. Community score card
Community Score Card is a community-based monitoring tool--a hybrid of the techniques of
social audit, community monitoring, and citizen report cards. It has a strong focus on empowerment
and accountability as it includes an interface meeting between service providers and the communi-
ty that allows for immediate feedback on quality and adequacy of services provided in the commu-
nity. This tool can be used for tracking inputs or expenditures; generation of benchmark
performance criteria used by communities and service providers to assess their services; monitor-
ing the quality of services over time; comparison of performance across facilities and districts; gen-
erating feedback mechanisms between providers and users; and strengthening citizen voice and
community empowerment. For more details, see
http://info.worldbank.org/etools/docs/library/94360/Tanz_0603/Ta_0603/TheCommunityScoreCard
Process_June03.pdf.
For a summary in the main tools and methods used in PME for ARE projects, see box 4.3
Steps 4 and 5--Analyze and use the data and findings. After analyzing the data, it needs to be
disseminated and used for corrective action as needed. Findings can also help to strengthen the PME
process itself. Lessons Learned from the NAADS Participatory M&E Pilot in Uganda are described
in box 4.4.
42
3534-Rikka.pdf 6/20/05 2:51 PM Page 43
BOX 4.3: MAIN TOOLS AND METHODS USED INPARTICIPATORY MONITORING AND EVALUATION
FOR AGRICULTURAL RESEARCH AND EXTENSION PROJECTS
1. The choice of tools and methods for a specific survey or assessment depends on the country
context, individual project situation, resource constraints, time frame, available capacity as well
as the actual information need.
2. Most tools can be used for design, monitoring, and evaluation of the project. However, quantita-
tive methods are best-suited for baselines and evaluations.
3. In principle, any assessment must take account of the following issues: (a) setting objectives; (b)
selecting field researchers (with data collection and or communication skills for evaluation); (c)
establishing sampling frame and control groups; (d) preparing guidelines in a collaborative man-
ner; (e) applying an appropriate methodology; and (f) analyzing and communicating the data and
results.
Proposed modes of undertaking assessments may include:
1. Quantitative methods, such as administered questionnaires (e.g., household surveys), physical
field measurements, and records reviews are used to provide comprehensive and statistically
valid data on stakeholders, local conditions, program outcomes and impacts. See Grosh and
Glewwe (2000) for further details Designing household survey questionnaires for developing
countries: Lessons from 15 years of the Living Standards Measurement Study Vol. 3.
2. Qualitative methods, such as beneficiary assessment, stakeholder forums, focus group discus-
sions, and key informant interviews, are used to analyze stakeholders, their views, opinions, and
experiences, as well as institutional rules and behaviors.
� Beneficiary and stakeholder assessments: To identify and design development initiatives; signal
constraints to their participation; give feedback on these activities to those designing and manag-
ing a project or formulating policy; and define the Project Development Objective and definition
for impact.
� Stakeholder forums and workshops involving many stakeholders provide a basis for clarifying
questions and issues and exchange of views. For example, this is useful during the design stage.
� Focused group discussions are facilitated discussions among 8-12 participants with similar back-
grounds. Useful when the objective is to gain in-depth information on beneficiary views, per-
spectives, outcomes, and impact in relation to planned activities.
� Community Score Card is a community-based monitoring tool--a hybrid of the techniques of
social audit, community monitoring, and citizen report cards. It has a strong focus on empower-
ment and accountability as it includes an interface meeting between service providers and the
community that allows for immediate feedback on quality and adequacy of services provided in
the community. http://info.worldbank.org/etools/docs/library/94360/Tanz_0603/
Ta_0603/TheCommunityScoreCardProcess_June03.pdf
� Open-ended group interviews revolving around a number of themes selected and agreed on by
the participants.
� Key informant interviews are loosely structured, in-depth interviews of 15�35 people selected
for their first-hand knowledge about a specific topic. www.usaid.gov/pubs/usaid_eval/
pdf_docs/pnabs541.pdf
continued on next page
43
3534-Rikka.pdf 6/20/05 2:51 PM Page 44
BOX 4.3:--CONTINUED
� Participant and field observation, which is a time-consuming mode of enquiry, covers the active
involvement of observers within targeted communities to review and examine themes surround-
ing relationship among the respective contact groups; the roles of different social and economic
groups; the nature and extent of interaction between the various institutions and farming commu-
nities and ongoing activities.
Source: Authors.
BOX 4.4: LESSONS LEARNED FROM THE UGANDA NATIONAL AGRICULTURAL ADVISORY SERVICES
PROJECT'S (NAADS) PME PILOT
Although much of the Participatory M&E work on agricultural extension has been geared to serve the
needs of the rural development project manager, in the Uganda rural development NAADS project the
central actors became the farmers themselves.
What: A pilot PME exercise was conducted in 2002/2003 in a sub-county of the Mukono District in
which monitoring information was collected from six NAADS farmer groups; and evaluation data was
gathered from 60 randomly selected households. The implementers of this PME were 16 trained farm-
ers under the supervision of and with technical backstopping provided by sub-country technical staff
and persons trained in beneficiary assessment methodology from Makerere University (Kampala).
Findings
There was clear appreciation of this farmer-driven M&E process as (a) a source of information gath-
ered both by and from persons directly affected by the project; and (b) a generator of ownership for
the project, with the learning--as well as the project activities--being carried out by the farmers
themselves.
However, there were a number of challenges, or constraints, which serve as lessons for the improve-
ment of this farmer-driven approach to PME in the future:
� The time allotted for data tabulation and analysis (two days) was considered inadequate;
� Monetary allowances to local PME facilitators, for travel and expenses, was considered by these
persons as insufficient to meet their needs;
� Several of the communities had not been adequately informed about the PME, resented the PME
facilitators as privileged persons and failed to participate in the interviewing (individual and
group) as expected;
� Generally, the PME facilitators (interviewers) were seen as NAADS "representatives" rather than
independent farmers. This impression unduly biased the data; and
� Some PME facilitators experienced difficulty with the conversational interviewing approach.
Particularly problematic were: (a) maintaining respondents' attention span for long periods of
time (due in part to an excessively long interview guide); (b) probing for in-depth information;
(c) posing non-leading questions; and (d) controlling group discussions.
44
3534-Rikka.pdf 6/20/05 2:51 PM Page 45
Follow-up action
Aside from correcting monetary and logistic factors, it was determined as a result of this pilot to:
� Better inform communities about the PME and the nature of the facilitators;
� Place more stringent requirements regarding the qualifications necessary to be a facilitator;
� Contract external, trained interviewers for evaluation work while retaining farmers for
monitoring activity; and
� Emphasize capacity building that is tailored to persons with low education levels--involv-
ing visual aids, local language, practical field exercises, and alleviation of fears at the out-
set of training.
Conclusions
� The key feature and strength of participatory M&E of rural development projects puts tar-
get beneficiaries at the center of the PME process, proving the scope for feedback toward
improved project implementation as well as learning on the part of beneficiary farmers.
� On the target beneficiary assessment approach, voice is given primacy and there is a sys-
tematic listening to the farmer. In the more participatory, farmer-driven PME represented
by the NAADS case in Uganda, farmers not only acquire voice but decision-making
authority, both as implementers and intended contractors of the learning process.
� It is the catalytic transformation of learning into empowerment that lies at the heart of the
rationale and potential of participatory monitoring and evaluation as described in the Final
Report on this PME by Margaret Mangheni and Christopher Bukenya.
"Our experience from the PM&E methodology pilot phase, at least, has revealed renewed com-
munity enthusiasm and "social energy" in the NAADS process which are valuable outcomes for
a program which seeks among other important goals empowerment of target communities."
Source: Lawrence Salmen, Consultant, World Bank.
4.3. ADDITIONAL MATERIAL ON PME
For further details on PME in general:
1. World Bank--Participation and Civic Engagement--
http://www.worldbank.org/participation/home.htm
2. Participatory Monitoring and Evaluation--
http://www.worldbank.org/participation/pme/partme.htm
3. Eldis Participation Resource Guide--PME
http://www.eldis.org/participation/pme/index.htm
4. PME in Agricultural Research and Extension:
Guide to participatory extension methods used in community forestry in Laos
http://www.eldis.org/cf/search/disp/docdisplay.cfm?doc=DOC10101&resource=f1par
5. On participatory evaluation, see "Who are the question-makers?"
http://www.undp.org/eo/documents/who.htm
45
3534-Rikka.pdf 6/20/05 2:51 PM Page 46
PME tools:
1. World Bank--PME Tools and Methods
http://www.worldbank.org/participation/pme/partme2.htm
2. World Bank--Social Analysis Sourcebook http://www.worldbank.org/socialanalysissource
book/socialassess5.htm
3. World Bank--Monitoring & Evaluation: Some Tools, Methods & Approaches.
lnweb18.worldbank.org/. . ./a5efbb5d776b67d285256b1e0079c9a3/
$FILE/MandE_tools_methods_approaches.pdf
4. IFAD (2003)--Methods for Monitoring and evaluation.
http://www.ifad.org/evaluation/guide/appendixd/index.htm and
5. Gathering, Managing and Communicating information.
http://www.ifad.org/evaluation/guide/6/index.htm
6. Eldis Participation Resource Guide--Participatory Monitoring and Evaluation--Methods,
Tools and Manuals http://www.eldis.org/participation/pme/pme3.htm
7. Gubbels, P. and Koss, C. (2000). From the Roots Up. Strengthening Organizational
Capacity through Guided Self-assessment.
46
3534-Rikka.pdf 6/20/05 2:51 PM Page 47
5. EVALUATION OF OUTCOMES
AND IMPACT IN THE WORLD BANK
5.1. WHY AND WHAT TO EVALUATE IN THE WORLD BANK PROJECTS?
Evaluation is the periodic assessment of the relevance, performance, efficiency, and impact (both expect-
ed and unexpected) of the project in relation to stated objectives. Evaluation measures achievements in
relation to institutional policies, Bank-wide program objectives, and the goals set for each operation.
Evaluation is designed to:
� Provide an objective basis for assessing the performance of policies, programs, projects,
and processes;
� Help provide shared accountability for the achievement of the Bank's objectives; and
� Improve policies, programs, and projects by identifying and disseminating the lessons
learned from experience and by making recommendations drawn from evaluation findings.
Evaluation at the Bank has two major dimensions:
1. Self-evaluation (by the units responsible for particular programs and activities).
� An interim evaluation is undertaken by project management during implementation as a
first review of progress and a prognosis of the likely effects of the project. It is intended to
identify project design problems, and is essentially an internal activity undertaken for proj-
ect management. See section 5.2.
� Terminal evaluation, a similar process undertaken at the end of a project, is required for the
Project Completion Report (PCR; OP/BP 13.55, Implementation Completion Reporting). It
includes an assessment of the project's effects and their potential sustainability. Chapter 6
and Appendix 6 present the key steps and a case study on economic impact evaluation of a
competitive research grant scheme.
Terminal Evaluation: The implementation completion report is prepared to evaluate the performance
of the Bank and the borrower. Each party is required to prepare and submit an ICR. Guidelines for ICR
preparation can be found from http://siteresources.worldbank.org/INTINVLENDING/Resources/
ICRGuide.pdf. Box 5.1 provides specific highlights for M&E during ICR preparation and write-up.
2.Independent evaluation:
� The impact evaluation by OED is usually undertaken several years after final disbursement,
and measures changes attributable to the project in terms of both direct and indirect causali-
ty. This is normally undertaken by national authorities or donor agencies. For example, in
the Bank, impact evaluation is carried out by the Operations Evaluation Department (OED)
for its Project Performance Audit Report, and therefore planning and implementation for
such a system should be left to OED.
47
3534-Rikka.pdf 6/20/05 2:51 PM Page 48
� OED evaluates by three main factors: (a) The relevance of the project's objectives in rela-
tion to the needs of the country; (b) was the project effective in meeting its goals; and (c)
was the project efficient in terms of using no more resources than necessary.
BOX 5.1. SPECIFIC HIGHLIGHTS FOR M&E DURINGICR PREPARATION
In preparing the Bank Implementation Completion Report the following should be reflected in selected
sections of the Report:
� Principle performance ratings: Good reflection and cross-checking in line with implementa-
tion, results of M&E system, baseline data and subsequent updates should go into the ultimate
ratings. If a QAG was conducted at project effectiveness, depending on the rating, M&E should
be a feature for improving quality by project completion;
� Major factors affecting implementation and outcome--if M&E was a factor, either factors
subject to or outside of government/implementing agency/donor control, should be highlighted
with indications of how those factors were addressed;
� Sustainability ratings with specific regards to transition arrangements to regular operations
should reflect the mainstreaming of M&E system into the regular functions of the implementing
Ministry and plans for continuity;
� Bank and borrower performance should reflect perspectives of M&E implementation as con-
ducted by borrower and Bank parties highlighting good practices as well as shortfalls;
� Ratings for achievement of objectives, outputs of components speak of the project implementa-
tion in its entirety: including management, coordination and impact; M&E including independ-
ent assessments, case studies and participatory consultation features as key sources of
information for arriving at those ratings; M&E is therefore a major guide for the ultimate ratings
under this section of the ICR; and
� Lessons learned in the ICR should factor modifications or adjustments made in the M&E sys-
tems that better captured targeted outputs, outcomes, and impacts.
Other ratings during implementation including Project Supervision Report (PSR) ratings of outcome
and impact indicators namely--projected against actual targets--should factor quite strongly in the
implementation of M&E system during supervision and updates of PSRs.
Source: Remileku Rakey, Consultant, World Bank.
5.2. KEY ISSUESTOCONSIDER WHEN
EVALUATING AGRICULTURAL RESEARCH AND EXTENSION PROJECTS
The focus of the evaluations may be the social, institutional, environmental, and economic outcomes,
sustainability, and impact of the ARE interventions. Economic effect and impact evaluations are the
most common type of assessments. It is increasingly clear that the social and environmental effects
and impacts of interventions must also be assessed. However, it is challenging to monitor and eval-
uate environmental and social indicators and effects of ARE projects.
The primary purposes of economic, environmental and social assessments of ARE projects are to
provide: (a) an information base for understanding complex effects of ARE interventions (i.e., new
technologies, policies, and services); (b) an early warning system to prevent adverse effects and to
take mid-course corrective action; (c) a tool for communities to protect the public's interest; (d) a
48
3534-Rikka.pdf 6/20/05 2:51 PM Page 49
means of identifying alternative approaches, technologies and adaptations; and (e) a basis for plan-
ning that involves the relevant (and affected) stakeholders (Deshler 1989).
Social impact evaluations look into the potential effects of ARE interventions (events, trainings, gen-
erated/transferred/ technologies, and improved organizations) on specific groups of people (i.e., farm
community, poor segments of the community, women farmers, and disadvantaged groups). These eval-
uations require the identification of socio-economic variables relevant to a proposed activity; identifi-
cation of information sources; data collection and categorization; a community profile of affected
populations; projections of consequences for several alternative courses of action; and assessment of
impacts and evaluation of the social impact by community members (van Willigen in Score 1995).
Similarly, environmental impact evaluations will look into the potential environmental effects of
ARE interventions on the farming community and specific groups as well as the environment and
natural resource base. It is often necessary to establish indirect ways to measure the environmental
effects by selecting indicators that track a change in a physical measure that is considered to have
environmental consequences. Examples of this type of indirect indicators are "the change in the
length of Vetiver grass strips" and "the number of hectares under no-tillage practices"--they are
expected to indicate the degree of soil conservation. Box 5.2 provides examples on potential focus
of the evaluation (whether social, environmental, or economic) in ARE projects/programs. In addi-
tion, Appendix 4 includes a list of various ARE project indicators.
BOX 5.2. POTENTIAL FOCUS FOR EVALUATION
IN AGRICULTURAL RESEARCH AND EXTENSION PROJECTS/ PROGRAMS
National impacts: political stability, economic fairness, agricultural environmental sustainability.
Community change: changes in administration of justice; health, welfare, and quality of life; fairness
in the marketplace; change in human rights, status of women; change in economic and social indicators
for poor; change of indicators of sustainable agriculture and natural resource management; change in
communication patterns and access to education and news; change in relationships between farmers and
non-farm population; change in youth perceptions about farming career; public opinion change; fairer
distribution of land and other resources; improved inter-organization relations; evidence of conflict res-
olution; and cultural practice change.
Organizational change: group operation and management; economic performance; technical operation
and management; financial operation; group institutionalization and self-reliance, new groups of farmers
included, new organizational linkages and partnerships; change in staff performance, new service delivery,
new methods used, additional facilities and equipment; change in research priorities; diversification of
funding; cost-benefits improved; new philosophy, purposes, and goals; improved organizational culture.
Individual change: changes in knowledge, attitudes, skills; sustainable agricultural practice; change in
aspirations, self-image, perspectives; expenditure of effort and money; use of methods, services; inven-
tion of appropriate technology; increased production or use of tolls; compliance with or opposition to
public policy; patterns of communication, career directions, and family relationships.
Reactions: testimonials; reactions to the relevance of content; appropriateness of technology, helpful-
ness, perceived value of educational experience; reputation of the extension provider.
Participation: farmer access to extension services by social class, gender, and ethnic groups; intensity
of face-to-face contacts; extent of media-assisted contacts; type of participation (volunteering, planning,
recruiting, learning, experimenting, evaluating); indicators of commitment (attendance, continuity, fre-
quency); private sector involvement. continued on next page
49
3534-Rikka.pdf 6/20/05 2:51 PM Page 50
BOX 5.2:--CONTINUED
Activities: participatory rural appraisal; planning; local knowledge documentation; on-farm/farmer
experimentation; farmer-to-farmer knowledge sharing; farm tours; farmer organizing; master farmer
leadership training; farm demonstrations; exhibitions and fairs; residential workshops; marketing analy-
sis; farm policy education.
Inputs-resources: organizational sponsorship and networks; funds; organizational design facilities,
equipment; philosophy, mission, goals, objectives; staff, resource people, volunteers; local and external
research knowledge and relevance; cultural, economic and political context.
Source: Deshler 1997; Riikka Rajalahti, World Bank.
The following recommendations for conducting social, environmental and economic evaluations of
ARE projects build on recommendations by Hanson et al. (2004) who provided suggestions on prac-
tical and cost-effective activities when evaluating economic sustainability of extension programs:
1. Evaluate ARE project/program components/activities rather than complete programs--
the individual results can be used in successive improvements better than results from
an overall evaluation;
2. Integrate community's and farmers' perceptions of how activities are organized--it is
important that the activities have an internal consistency from the farmers' point of view;
3. Separate the evaluation of direct effects/benefits (e.g., income generation by farmers)
from indirect ones, (e.g., soil conservation and health benefits and effects on labor
division and community relationships);
4. Improve the use of indicators particularly when monitoring and evaluating indirect
environmental benefits. However, improved use of indicators must go hand-in-hand
with baseline data collection;
5. Take advantage of farmer groups in evaluations. The findings from a group are gener-
ally more reliable than from single farmers. However, it is important to train the inter-
viewee team properly; and
6. Carry out periodic ex-post analyses as they may contribute to mid-course corrections.
Besides the points above, the following steps are essential when planning evaluations:
1. Identify all relevant stakeholders (women and men farmers, research and extension
staff, local leaders and authorities, civil society, government, and donors) in this
process--those who have an interest in the evaluation results and those who should be
involved in the evaluation process;
2. Focus and priorities: It is important to narrow the choices and priorities of the evalu-
ation as early as possible. It is equally important to agree on a short list of evaluation
questions--these questions will guide the collection and analysis of data and informa-
tion as well as interpretation and presentation of the results. Note that the achievement
of the Project Development Objective and the intermediate outcomes presented in the
project's Results Framework must be part of the evaluation.
50
3534-Rikka.pdf 6/20/05 2:51 PM Page 51
3. Tools and actors: Based on the focus and priorities of the evaluation, the stakeholders
involved, and the time and resources available, select the appropriate tools and actors for
carrying out the evaluation. See box 5.3 for guidance on selection of internal or external
evaluators.
BOX 5.3. WHEN TO USE INTERNAL AND EXTERNAL REVIEWERS AND EVALUATORS?
Internal reviewers and evaluators are generally used when the objective of the evaluation is to identify
project design problems and when the objectivity of the evaluation is not sacrificed. These conditions
are generally in place for unplanned assessments, annual reviews, and interim evaluations. The major
benefits of using internal evaluators include cost-effectiveness, in-depth knowledge of the project and
the local context and the trust gained among the stakeholders. The major disadvantages of using inter-
nal evaluators may include subjectivity, over-emphasis on the positive findings, lack of commitment and
resistance to change. However, internal evaluators and/or M&E staff should be included in all evalua-
tions, with or without external evaluators.
External reviewers or evaluators can:
� Provide independent and constructive criticism that helps project stakeholders to reflect on and
identify lessons;
� Give a fair judgment of project progress and areas in need of improvement;
� Help identify priorities for the remaining time of the project to support the rational use of resources;
� Help unite diverse stakeholder perspectives.
The key problems associated with external evaluators may include:
� Due to time constraints, they often have limited dialogue with project stakeholders, outside the
staff, and may make superficial analyses or to generalize the situation.
� They can jump to conclusions without in-depth knowledge of local realities;
� If foreigners, language constraints and limited cultural insights may affect the analysis;
� They can be excessively strict in their methodology;
� They can be inclined to focus more on funding agency requirements than what a project needs
for improved impact;
� Reports often highlight the negative aspects of the project and do not give due emphasis to posi-
tive aspects.
Source: IFAD 2003; Riikka Rajalahti, World Bank.
� Quantitative methods, such as administered questionnaires (e.g., household surveys),
physical field measurements, and records reviews, are used to provide comprehensive
and statistically valid data on stakeholders, local conditions, program outcomes and
impacts.
� Qualitative methods, such as beneficiary assessment, stakeholder forums, focus
group discussions, and key informant interviews, are used to analyze stakeholders,
their views, opinions, and experiences, as well as institutional rules and behaviors.
51
3534-Rikka.pdf 6/20/05 2:51 PM Page 52
� Chapters 3 (Data collection, reporting, and dissemination: section 3.4) and 4
(Participatory M&E) in this note presented many of the tools that can be used for eval-
uation.
4. Analyze the data and findings and bring in the perspectives of relevant stakeholders.
Make sure to cross-check the information and results to guarantee the validity of the
findings.
5. Communicate the evaluation findings to the relevant stakeholders in an appropriate
format.
� External stakeholders, such as donors, usually require a formal evaluation report
that should be circulated widely enough. Internal stakeholders, such as beneficiaries
and implementing partners, may have different priorities (identified at the begin-
ning) and interests, and therefore different reporting expectations. The reporting for-
mat should therefore be tailored to the needs of the stakeholder--verbal reporting in
workshops and meetings may be sufficient for some issues whereas other selected
issues may be presented in reports.
6. Use evaluation results. Project evaluation results are expected to provide information
for decision making on e.g., policies, management and future projects. Frequently,
however, this has not been the case. It is essential to pay adequate attention to identify-
ing the relevant stakeholders and the evaluation priorities to improve the use of evalua-
tion results. In addition, evaluation can be used as a learning process.
52
3534-Rikka.pdf 6/20/05 2:51 PM Page 53
6. ECONOMIC EVALUATION
OF COMPETITIVE GRANTS SCHEMES
Competitive Grants Scheme (CGS) programs implemented as a component for agricultural research and
extension projects seek to divest some planning and oversight functions to local entities with closer links
to end users. Each CGS is comprised of a large number of subprojects carried out by different execut-
ing institutions. As the inclusion of CGS in Bank's ARE programs has increased significantly during the
past decade, this chapter is meant to help in economic assessment of such schemes. This chapter builds
on two previous World Bank publications: "Ex Ante EconomicAnalysis inAKIS Projects" by Horstkotte-
Wesseler et al. (2000) and "Monitoring and Evaluation for AKIS Projects" by Alex and Byerlee (2000).
For a case study on economic evaluation of Competitive Research Grant in Ecuador, see Appendix 6.
6.1. ECONOMIC IMPACT EVALUATION OF A CGS
The theory and empirical tools for assessing the economic14 impact of Research and Extension are
well developed (see table 6.1).
� The objective of the analysis is to identify changes in productivity that has been induced by
the research effort, and compare these benefits to the cost of the research.
� The key indicator of research efficiency is the internal rate of return (IRR) of the invest-
ment in the CGS.
� Because a CGS is comprised of many subprojects, adjustments must be made to the stan-
dard procedures for calculating the IRR, primarily with respect to procedures for data col-
lection. But the goal is still to compare the discounted benefit and cost streams.
Information on intermediate indicators of research output should also be collected and analyzed. It
will take from five to twenty years for the economic impact of a successful research project to occur.
� A rough rule of thumb might be that extension projects could begin to have a measurable
impact within three years, an adaptive research project within five years, and strategic
research within 5�10 years. A successful basic research project may not even generate a
specific identifiable technology, yet may make a large contribution to the society.
� Indicators should be collected to examine: (a) whether the grant fund has been competently
administered; and (b) the scientific merit of the funded projects (i.e., whether a sufficient
amount of good science is being conducted by grantees).
� A variety of indicators can be used. Administrative efficiency can be assessed by consider-
ing factors such as the number of grant applications received, the percentage of proposals
funded, and the share of overhead costs in total expenditures. Scientific merit is indicated
by the number of publications, particularly in refereed or international journals, or the num-
ber of papers presented at international conferences.
� At some point in the evaluation process, external scientific experts should be called upon to
assess the quality of science. It is very difficult for non-experts to do such an assessment.
53
3534-Rikka.pdf 6/20/05 2:51 PM Page 54
Table 6.1. Key Reference Sources on the Theory and Empirical Tools for
Assessing the Economic Impact of Research and Extension
Reference Books
Alston, J., G. Norton, and P. Pardey. 1995. Presents the principles and practice of ex-post and ex-ante
Science Under Scarcity. Ithaca, NY: Cornell economic evaluation methods and their use in research priority
University Press. setting. A wide range of approaches are reviewed, synthesized,
and assessed using a unifying conceptual framework.
Horton, D., P. Bellantyne, W. Peterson, B. Uribe, A sourcebook that provides a synthesis of literature and
D. Gapasin, and K. Sheridan. 1993. Monitoring experience on research monitoring and evaluation principles,
and Evaluating Agricultural Research: A processes, and methods.
Sourcebook. Wallingford, UK: CAB International.
Manuals
CIMMYT, 1993. The Adoption of Agricultural An excellent manual for practitioners that explains step by step
Technology: Mexico: CIMMYT the approaches to estimating technology adoption.
Masters, W.A., B. Coulibaly, D. Sanogo, A guidebook intended to provide a concise summary of the
M. Sidib�, and A. Williams. 1996. The Economic tools needed to conduct persuasive impact studies, which
Impact of Agricultural Research: A Practical enable researchers to quantify the economic benefits and costs
Guide. Department of Agricultural Economics, of their work. Three spreadsheet-based computer exercises help
Purdue University, West Lafayette, IN. E-mail: apply the methods described in the manual.
Masters@AgEcon.Purdue.edu
Janssen, W. and A. Kissi. 1997. Planning and A practical approach to combine natural resource management
Priority Setting for Regional Research. Research and productivity concern; introduces a methodology to choose
Management Guidelines 4. International Service regional priority research subprojects based on economic
for National Agricultural Research, The Hague. considerations.
Available online at www.cgiar.org/isnar
Collion, M.H. and A. Kissi. 1995. Guide to An approach to research program planning by objective, based
Program Planning and Priority Setting. Research on a series of steps which include benefit-cost analysis as an
Management Guidelines No. 2E. ISNAR, The approach to priority setting.
Hague. Available online at www.cgiar.org/isnar
Software
Dream�--Dynamic Research Evaluation for www.cgiar.org/ifpri
Management This is a menu-driven computer program developed by ISNAR
to facilitate application of the economic surplus model under a
variety of market situations. The latest version of this program
(June 1998) is available from IFPRI (contact: Stanley Wood,
s.wood@cgnet.com).
MODEXC--Model for Economic Analysis Http://www.ciat.cgiar.org/inslinks/modexc.htm
This is a spreadsheet based economic analysis model developed
by CIAT, available on the World Wide Web, that allows to
calculate the NPV, IRR and B/C ratio of investments in
agricultural research. The model can be used for economic
analysis of technical change both ex-ante and ex-post under
alternate scenarios of market situation.
54
3534-Rikka.pdf 6/20/05 2:51 PM Page 55
Web sites
ISNAR's information and discussion forum Http://www.cgiar.org/isnar/Fora/Priority/index.htm
for agricultural research priority setting This web site provides the process, steps, and methods of
research priority setting.
CIAT impact sub-project Http://www.ciat.cgiar.org/inicio_in.htm, the CIAT web site
features abstracts on the impact of agricultural research, data
bases, trends, as well as a download version of MODEXC (see
above).
Source: Byerlee, D. 2003. "Conceptual and Practical Issues in Measuring Impacts of Sub-projects in Competitive Grants Scheme
(CGS)." Unpublished manuscript. 12p.
6.2. RESEARCH COSTS
Due to the de-centralized nature of the CGS, the cost of each subproject is composed of at least three
sources:
1. Central CGS administrative overhead;
2. Direct costs incurred by the executing institution and cooperating institutions; and
3. The cost incurred by the CGS for administration and monitoring and evaluation of the
subproject (usually considered as an overhead).
Collecting information on research costs is relatively straightforward, so attention here will focus on
identifying and measuring productivity changes.
6.3. SAMPLING
To keep impact assessment within reasonable costs, only a sample of subprojects can be evaluated.
One approach is to evaluate a random sample of subprojects (see box 6.1 for an example). The prin-
ciple advantage is that results are representative whereas the major disadvantage can be that a random
sample may become more costly to implement if subprojects are widely disbursed geographically.
A more serious drawback is that a random sample may miss the most successful project. Agricultural
research success has a very skewed distribution. Most projects will have modest or negative economic
impacts, but a small number of projects will make important discoveries. An important discovery by a
single research project can generate sufficient economic benefits to cover the entire project investment.
Therefore it is important to closely examine the benefits generated by projects that are thought to have
been successful, perhaps supplemented by a random sample of remaining projects.
Procedure for random sampling:
� The random sample can be stratified by grouping sub-projects by theme, commodity, target
population group, executing agency, or geographic criteria (e.g., crop varieties, small farm-
ers, IPM, post-harvest, pastures, and basic grains).
55
3534-Rikka.pdf 6/20/05 2:51 PM Page 56
BOX 6.1: USING RANDOM SAMPLES INEVALUATION OF MPACT OF
I COMPETITIVE GRANT SCHEMES
During its lifetime of 6 years (1994�2000) PRONATTA carried out six rounds of project funding and
co-funded a total of 635 sub-projects. The evaluation of PRONATTA had to be done in a short time and
with limited human and financial resources. More importantly, the unstable political and social situation
in many rural areas made the collection of survey data impossible. In some areas it was not even possi-
ble to conduct field interviews with farmers.
A random sample of funded projects was evaluated. Sample sizes for the first and second phases (n1 and
n2) were 56 and 50, representing 17% of the sub-projects in each phase. The samples were discriminated
by two stratum--the region where the subproject was carried out, and the type of subproject. The evalu-
ation team expected these two factors to influence project success.
The evaluation was done in two stages, taking advantage of the temporal sequence of finalization of the
subprojects. This strategy allowed for a learning process for the evaluation team. Experiences in the first
phase illustrated the possible range of values of the needed parameters, while results from the second
phase served to validate the procedures applied in the first phase. See figure 6.1.
Source: Greg Traxler (Auburn University) and Gustavo Sain (The Inter-American Institute for Cooperation on Agriculture).
Figure 6.1. Estimation strategy in PRONATTA. Source: Greg Traxler (Auburn University) and Gustavo Sain
(The Inter-American Institute for Cooperation on Agriculture).
1994
2001 2003 2004
First sampling Second sampling
N = 338 Estimation N = Estimation
n = 56 (17%) n = 50 (17%)
First sample Second sample
distribution distribution
of TIRs of TIRs
Combined samples
Estimator of TIR
LEARNING PROCESS
56
3534-Rikka.pdf 6/20/05 2:51 PM Page 57
� This allows common methods to be developed, use of relevant experts, and a more mean-
ingful aggregation of impact results.
� Care is needed to ensure that sampled sub-projects are representative, and that within group
sample sizes are large enough to be meaningful.
� Once a sample is determined, the IRR for each of sub-projects is calculated, and an aggre-
gate IRR is estimated.
6.4. MEASURING BENEFIT STREAMS FOR SUBPROJECTS
In order to measure project benefits, data are required on three key variables: (a) technologies gen-
erated; (b) diffusion levels; and (c) per unit economic benefits.15
1. The evaluator must first identify the technologies that were generated by the research
effort. For example, were new crop varieties or new methods of pest control devel-
oped. This can generally be done by reviewing project reports, possibly with addition-
al information obtained by interviewing researchers and farmers.
2. The level of diffusion of each innovation must be estimated.
3. The productivity impact of each innovation must be measured.
Procedures for the steps 2 and 3 are discussed below. The calculated economic impact of a research
sub-project is the sum of the individual innovation productivity increases multiplied by the total
hectares covered by each new technology.
1. Estimating the diffusion of each innovation
Many studies have found that technological diffusion follows a logistic curve pattern. This curve can
be estimated by statistical means, but this would require information on the adoption percentage at
several points in time. Such information is unlikely to be available, so a shortcut method has been
developed. This method requires information on:
1. The initial proportions of farmers using the innovation when the subproject started
(p0). Baseline surveys will often not be available mandating that adoption information
be elicited through expert opinion. The evaluation team can ask farmers (as members
of the panel of experts) to recall production conditions at the time the sub-project was
initiated. In many cases estimates of the values of p0 will be equal to zero--diffusion
would not have occurred had it not been for the research project.
2. The proportion that have adopted the innovation at the moment of the evaluation (p1); and
3. The maximum expected adoption percentage (K).
With this information it is possible to approximate the logistic curve (Mart�nez and Sa�n 1983). The
annual adoption percentages are then multiplied by the total potential adoption area to come up with
the total number of units (hectares) benefiting from the innovation.
57
3534-Rikka.pdf 6/20/05 2:51 PM Page 58
2. Estimation of the incremental net benefit (INB) per unit of analysis (farm or hectare) from
adoption of the recommended practice
Calculation of the INB requires that the economic value of adopting the research recommendations
be estimated. Research projects generally attempt to increase land productivity (yield), reduce pro-
duction cost, or increase output quality.16
� Technical information about the effect of adoption must reflect the real impact at the field
(not experimental) level.
� Prices used in the valuation must reflect those received for products and paid by farmers
for inputs and services.
� Family labor should be valued at its opportunity cost, approximated by wages in local labor
markets.
� Special care should be taken in estimating the cost of working capital. If farmers do not
have access to formal credit, the interest rate must reflect the interest rate prevailing in
informal credit markets.
In order to calculate the INB, a counterfactual point of reference, representing the state of tech-
nology absent of the research project, must be established. Three main options exist for establishing
the counterfactual:
1. Collecting a baseline counterfactual data set: Counterfactual data can be collected by
surveying the target population at the initiation of the project and then again at the
time of the evaluation.
2. Alternatively, a survey at the time of the evaluation that contains producers affected by
the project and farmers not affected by the project can also be used.
These baseline data approaches require a significant investment in conducting and
analyzing field survey data, but make it possible to accurately disentangle the effects
of the research or extension intervention using the statistical regression model illus-
trated in table 6.2.
3. In most cases it will not be feasible to conduct a comprehensive survey. The most com-
mon method of estimating the INB is to use partial budgets17 based on experimental
results, making adjustments in yield and costs to reflect expected on-farm performance.
Table 6.2: Statistical Regression Model for Isolating Project Effects in Impact Evaluation
Statistical Model Parameter of Interest
Y = 0 + 1D1 + 2 D2 + Other factors Estimation is carried out using the pooled sample from
Y = parameter of interest (i.e., Yield) baseline and evaluation surveys.
D1 = 1 for observations at the moment of the evaluation 2 is the parameter of interest. It measures the net impact
= 0 for baseline observations attributed to the sub-project (Value E in table 2.1)
D2 = 1 for observations affected by the subproject
result (adopters)
= 0 for baseline observations
Source: Greg Traxler (Auburn University) and Gustavo Sain (The Inter-American Institute for Cooperation on Agriculture).
58
3534-Rikka.pdf 6/20/05 2:51 PM Page 59
6.5. ESTIMATION OF THE IRR
Estimated annual benefit flows (ABF) and cost flows (ACF) of each sampled sub-project are needed
to estimate the IRR. The ABF is estimated by direct multiplication of the incremental net annual ben-
efit (ABF-ACF) by the adoption level.
For most projects, it will be necessary to project ABF and AFC for several years into the future in
order to get a realistic estimate of the subproject's impact. This presents a quandary; to omit these
projected benefits and costs would seriously distort the research impact, but it is difficult to project
future values of a relatively young project. If the research phase of the project has already finished
and the flow of benefits continues beyond the project termination date, a future maintenance cost
that is often estimated as 10% of total cost of the project should also be included. The ABF and AFC
must be expressed in constant monetary units using a common deflator factor.
Once individual project IRRs are computed, the IRR for the CGS is obtained as the sample average
of the IRRs. Using the sample standard deviation, confidence intervals for the IRR estimate can also
be obtained. Average IRRs and confidence intervals can be calculated for each project category.
Subproject IRRs can be examined to identify factors associated with successful or unsuccessful proj-
ects. See box 6.2 for an example.
Estimating the NPV and IRR
Table 6.3 summarizes the steps typically involved to arrive at the Net Present Value (NPV) and
Internal Rate of Return (IRR) of a research and/or extension project.18
� The first column simply indicates the number of years the technology under consideration
is expected to last. It is not easy to precisely determine the life of the innovation. It is rec-
ommended that the analyst try to identify the diffusion pattern of comparable technologies
adopted in the past and discuss these issues with experts, such as local agronomists and
extension experts.
Table 6.3: Calculation of IRR for an Agricultural Research and Extension Project
1 2 3 4 5 6 7 8 9 10
Number Incremental Benefits Benefits Total
of benefits per with without direct Indirect Total Net
Year adopters adopter/year project project benefits benefits benefits Costs benefits
1 Column 2*3 Column 4-5 Column Column
6+7 8-9
2
3
...
14
15
NPV
IRR
Source: Greg Traxler (Auburn University) and Gustavo Sain (The Inter-American Institute for Cooperation on Agriculture).
59
3534-Rikka.pdf 6/20/05 2:51 PM Page 60
BOX 6.2. ESTIMATING INTERNAL RATE OF RETURN FOR SUBPROJECTS
IN COMPETITIVE GRANT SCHEMES
Subproject IRRs were estimated for the PRONATTA project in Colombia. IRR estimates were grouped
by type of research project to see whether one type of project was more successful than another (see
table below) This identifies means for improving future CGS programs. The average IRR for the CGS
was 86% and was fairly similar across types of research, even though the IRRs for individual subpro-
jects varied widely (see figure below).
Subproject frequency distribution of IRRs
Subproject IRR by type of research
Type of research IRR (%)
Mean SD Max Min
Capacity Development 79 129 122 42
Adaptive Research 86 114 125 50
Applied Research 98 115 143 53
Entire CGS 86 106 112 65
Source: Greg Traxler (Auburn University) and Gustavo Sain (The Inter-American Institute for Cooperation on Agriculture).
� The second column reflects the number of adopters at each point in time and the third col-
umn specifies the incremental benefits per adopter and year.19It is important to note that
the incremental benefits are not necessarily constant over time. For example, perennials
may have a specific production cycle with significant yield fluctuations.
60
3534-Rikka.pdf 6/20/05 2:51 PM Page 61
� In the next column the benefits with the project are estimated by multiplying the values in
column 2 and 3. For a more precise estimate of the expected benefits of the project it is
important that a realistic "without project" scenario is captured (column 5). If we take the
example of a project which aims at breeding a high-yielding rice variety and at promoting
its adoption, the calculation of project benefits should account for increases in rice yields
which would have been achieved without the activities of the project. It is not likely that the
yields would remain constant over time without the project. However, the estimation of
these benefits without the project is hypothetical and based on many assumptions. Once
again, the analyst should base his judgment on average yield increases in the past and on
discussion with local experts.
� The estimation of total direct benefits (column 6) is then simply based on subtracting the
values in column 4 and 5.
� The indirect benefits (column 7) accounts for spillover effects to non-targeted farmers.
The effects capture the adoption of technology by farmers who have not been directly tar-
geted by the project. Thus, total benefits (column 8) capture both total direct and indirect
benefits.
� Column 9 accounts for project costs over time. A subproject of a competitive grant
scheme for example should not only include the direct costs of the respective subproject,
but also a percentage of the overhead costs of the overall administration of the grant
scheme.20
� By subtracting the costs from the total benefits the analyst receives the net benefits (col-
umn 10). Finally, the values of the net benefits over time can be used to calculate the NPV
and the IRR.
6.6. SOURCES OF NFORMATION
I
When conducting a full survey is not feasible, the practitioner must resort to alternative sources of
information for estimating values for the required parameters:
1. Subproject information. Among the relevant subproject information the evaluation team
must be familiar with the following information: Subproject Proposal, Advance, Technical
(particularly experimental data), and Final reports as well as budgetary aspects.
2. Secondary information. Market prices and other economic variables including govern.
ment policies are usually obtained from secondary sources.
3. Expert opinion. Expert opinion is a valuable source of information for the evaluation
in the absence of a survey. It refers to a participatory method of eliciting information
on the value of one or two parameters of interest through group or individual inter-
views.21 Table 6.4 shows the minimum set of parameters to be estimated for a sound
economic impact evaluation (ex-post economic evaluation) as described in previous
sections.
61
3534-Rikka.pdf 6/20/05 2:51 PM Page 62
Table 6.4: Sources of Information on the Minimum Set of Parameters to be Estimated for a Sound Economic
Impact Evaluation
Parameter Information source and procedure
Size of the RD Subproject documentation. If a proper monitoring system is in place the
information should be readily accessible from advance reports. Another
source of useful information are the technical personnel that participated
in carrying over the subproject
INB = increase in Net Benefits per UOA. Technical information: Project documentation and Experts Opinion
Technical information on induced (farmers and technicians)
changes and sources of information Economic information: Secondary sources and Expert Opinion
Economic information on prices and (farmers and local merchants)
markets
Diffusion pattern. Requires three P0 from subproject documentation
parameters: P1 and K from Expert Opinion
P0 = proportion at the subproject start
P1= proportion at the evaluation time
K = Adoption potential (ceiling)
AFC= annual flow of subproject costs Subproject documentation
in constant monetary terms Secondary information
Source: Greg Traxler (Auburn University) and Gustavo Sain (The Inter-American Institute for Cooperation on Agriculture).
6.7. FISCAL ANALYSIS
Sustainability of financing
Even though a project may be economically and financially justified, it may not be sustainable if the
fiscal support is not available. A crucial step in assessing the long-term potential for competitive
grants, or any other project, is to assess the fiscal impact of the project. This should be done early in
the project cycle. Fiscal analysis has become increasingly important because of stringent constraints
on government budgets in recent years. Because of the diffuse nature, and the long lags that charac-
terize research benefits, research and extension projects are often hard hit by austerity measures.
The Project Appraisal Document should include a fiscal analysis aimed at assessing the likelihood
that sustainable local resources and funds will be available at the end of the project to take over from
funds provided under the project. This analysis should estimate the long-term impact of the proposed
project on public expenditures and assess whether the project can be sustained with local funding
resources. It is important that the analysis attempts to look beyond the implementation period.
Obtaining sustainable funding has been a difficult challenge for past research projects to overcome.
See box 6.3 for further details.
The key steps in the fiscal analysis:
1. The first step in the fiscal analysis is: collect data on recent public expenditures for
agricultural research and extension.
2. The next step is: estimate the level of public expenditures needed to sustain project
activities after project completion.
62
3534-Rikka.pdf 6/20/05 2:51 PM Page 63
BOX 6.3: SUSTAINABLE FINANCING OF A COMPETITIVE FUND INECUADOR
The Ecuador competitive fund project began full operation late in 1999 and continued until 2004. Prior
to the project virtually all agricultural research funding came for the federal government budget as ded-
icated funding to the National Agricultural Research Institute (INIAP). The project was very successful
in attracting counterpart funding from participating research institutions, attaining a high of $2.6 mil-
lion in 2003. The project induced a large increase in total research expenditures, but half or more of total
expenditures were from loan proceeds.
At the end of the project, research funding remained dependent on a highly politicized federal budget
allocation process. Although the 2004 government agricultural research budget was the largest ever, all
funds were dedicated to INIAP and the competitive fund program was closed. Much of the momentum
from a project that was very successful in increasing agricultural research output has therefore been lost,
and the future of the competitive fund remained uncertain.
Govt. Ecuador World Bank Counterpart Total
$US $US $US $US
Year millions % millions % millions % millions
1995 3.4 100% 0.0 0% 0.0 0% 3.4
1996 4.3 100% 0.0 0% 0.0 0% 4.3
1997 4.5 98% 0.1 2% 0.0 0% 4.6
1998 3.3 95% 0.2 5% 0.0 0% 3.5
1999 1.2 47% 1.3 53% 0.0 0% 2.5
2000 1.7 44% 2.2 56% 0.0 0% 3.9
2001 2.9 47% 3.1 49% 0.3 4% 6.3
2002 3.4 37% 4.1 45% 1.7 18% 9.2
2003 2.8 30% 3.8 42% 2.6 28% 9.1
2004 5.9 63% 1.9 21% 1.5 16% 9.2
1995-
1998 avg.3.9 98% 0.1 2% 0.0 100% 3.9
1999-
2004 avg.3.0 45% 2.7 44% 1.0 100% 6.7
Source: Johannes Woelcke, World Bank.
63
3534-Rikka.pdf 6/20/05 2:51 PM Page 64
3. Total project costs should be separated into investment costs and recurrent costs.
Computed factors can help to translate project costs into projected incremental public
expenditures per year by multiplying the factors by the average annual project costs.
To calculate a factor that reflects the proportion of the investment costs to be taken
over by the government after the project ends, the investments can be categorized
according to maintenance and replacement versus one-off investments such as foreign
technical assistance.22 The calculation of a factor for recurrent costs can be based on
the relation of the recurrent costs in the last year of the project to average annual
recurrent project costs.
4. Once the annual incremental investment expenditures and recurrent costs have been
estimated for each project component, they can be added up to estimate the projected
total incremental public expenditure per year that will be required to sustain the
research project.
Once the projected incremental public expenditure needs are estimated, the source and form of the
financing mechanisms must be considered. Cost recovery mechanisms have often been incorporated
in competitive fund projects, but have generally fallen short of generating the amount of funding need-
ed to fully support future activities.
PROJECT IMPACT ON TAX REVENUE
The analyst should attempt to capture the project's impact on tax revenue, and reflect this impact in
the projected incremental public expenditure. The project contributes to increased general tax rev-
enues if, for example, the use of taxed inputs or if a tax is imposed on farm profits (and the project
is expected to increase profits). Because the actual level of cost recovery through the mechanisms
mentioned above is uncertain, the analyst should perform some sensitivity analysis assuming differ-
ent levels of cost recovery.
In many cases, continuation of the project after the loan is closed would represent a significant
increase in government commitment to agricultural research and extension. In times of tight pub-
lic budgets these increases might be difficult to defend against competing uses of government
funds. Hence, the analyst should include a section on the importance of the agricultural sector, and
in particular of agricultural research and extension for achieving national goals, such as econom-
ic growth and poverty reduction. The analyst might also refer to the high return on investment in
research and extension that has been achieved internationally. In addition, the analyst could set the
estimated public expenditures for agricultural research and extension in relation to total public
expenditures and compare it to the expenditures of neighboring countries, and other developing or
developed countries. In most cases, public expenditures on agricultural research and extension are
still comparatively low.23
64
3534-Rikka.pdf 6/20/05 2:51 PM Page 65
7. CONCLUDING REMARKS
Monitoring and evaluation are integral tools for managing and assessing the efficiency and effective-
ness of investments in ARE systems and projects. Monitoring and evaluation of ARE project per-
formance, outcomes, and impact has, however, been a significant challenge. However, successful
project M&E can be carried out in a number of ways.
Besides the mandatory use of Results Framework, description of results monitoring arrangements,
and the official evaluations, the project management together with the relevant stakeholders can
choose different M&E tools and implementers as well as the level of desired technical sophistica-
tion. The choices depend on the Project Development Objective, stakeholders and their priorities,
and particularly the resources available (i.e., time, staff, capacity, and finances).
Besides the mandatory requirements, the six most important things to remember when planning and
implementing project and program M&E include:
At the design stage:
1. A strong and clear Project Development Objective is essential for good project design;
2. Select only a few key indicators--that are Time, Quantity and Quality (TQQ) bound,
and that are realistic to collect considering the resources available;
3. Prepare the project M&E plan carefully. Give due consideration to stakeholder priori-
ties, available resources, and capacity requirements;
During the project implementation:
4. Pay attention to M&E issues during the supervision missions;
5. Implement M&E--collect and use M&E data for the benefit of project primary bene-
ficiaries, project implementers, and project management; and
6. Integrate participatory M&E into the project M&E as much as possible, giving consid-
eration to the priorities and available resources.
65
3534-Rikka.pdf 6/20/05 2:51 PM Page 66
8. REFERENCES
Alex, G., and D. Byerlee. 2000. Monitoring and Evaluation for AKIS Projects. Framework and Options. Agricultural
Knowledge and Information Systems (AKIS). Good Practice Note. Washington, DC: World Bank.
http://lnweb18.worldbank.org/ESSD/ardext.nsf/26ByDocName/MonitoringandEvaluationforAKISProjectsFramework
andOptions/$FILE/m&e.pdf
Anderson, J., and G. Feder. 2004. "Agricultural Extension: Good Intentions and Hard Realities." The World Bank
Research Observer 19 (1).
Babu, A., Y. Singh, and R. Sachdeva. 1997. "Establishing a Management Information System." In: Improving
Agricultural Extension: A Reference Manual. Rome: Food and Agricultural Organization of the United Nations.
Bantilan, M. C. S., and P. K. Joshi.1997. Integrating Research and Evaluation Efforts. Proceedings of an International
Workshop. India: INCRISAT.
Bellon, M. R. 2001. Participatory Research Methods for Technology Evaluation. A Manual for Scientist Working with
Farmers. Mexico: International Maize and Wheat Improvement Center. CIMMYT, 2001.
Binnendijk, A. 2000. "Results-Based Management in the Development Cooperation Agencies: A Review of
Experience." Paper prepared for OECD/DAC Working Party on Aid Evaluation.
Braun, A., G. Thiele, and M. Fernandez. 2000. "Farmer Field Schools and Local Agricultural Research Committees:
Complementary Platforms for Integrated Decision-making in Sustainable Agriculture." AgREN Network Paper
No. 105.
Byerlee, D. 2003. "Conceptual and Practical Issues in Measuring Impacts of Sub-Projects in Competitive Grants Scheme
(CGS)." Unpublished manuscript.
Casley, D. J., and K. Kumar. 1987. Project Monitoring and Evaluation in Agriculture. A Joint Study of the World Bank,
IFAD, and FAO. Baltimore, MD: John Hopkins University Press.
CIMMYT 1988. From Agronomic Data to Farmer Recommendations: An Economics Training Manual. revised. ed.
Mexico: CIMMYT.
Deshler, D. 1989. "Future Risk Assessment: Issues Programming Opportunity." Journal of Extension 27 (1): Spring 1989.
Deshler, D. 1997. "Evaluating Extension Programmes." In Improving Agricultural Extension. A Reference Manual. FAO.
Fort, L., B. Martinez, and M. Mukhopadhyay. 2001. Toolkit: Integrating a Gender Dimension into Monitoring &
Evaluation of Rural Development Projects. Washington, DC: World Bank.
Goels V. K., E. Koryukin, M. Bhatia, and P. Agarwa. 2004. "Innovation Systems: World Bank Support of Science and
Technology Development." Working Paper No. 32, World Bank, Washington, DC.
Grosh, M., and P. Glewwe. 2000. Designing Household Survey Questionnaires for Developing Countries: Lessons from
15 years of the Living Standards Measurement Study Vol.1. Washington, DC: World Bank.
Grosh, M., and P. Glewwe. 2000. Designing Household Survey Questionnaires for Developing Countries: Lessons from
15 years of the Living Standards Measurement Study Vol. 2. Washington, DC: World Bank.
Grosh, M., and P.Glewwe. 2000. Designing Household Survey Questionnaires for Developing Countries:: Lessons from
15 years of the Living Standards Measurement Study Vol. 3. World Bank, Washington, DC.
Gubbels, P., and C. Koss. C. 2000. "From the Roots Up. Strengthening Organizational Capacity Through Guided Self-
Assessment." World Neighbours Field Guide: Capacity Building. 2nd ed. Paragon Press.
Guijt, I. 1999. "Participatory Monitoring and Evaluation for Natural Resource Management and Research." Socio-
economic Methodologies for Natural Resources Research Best Practice Guidelines. Chatham, UK: Natural
Resource Institute. http://www.nrsp.co.uk/Nrspweb/SEM/SEM%20BPGs/BPG04_Guijt_PM&E.pdf
Guijt, I., and J. Gaventa. 2000. "Learning from Change. An Introduction." In Learning from Change: Issues and
Experiences in Participatory Monitoring and Evaluation. M. Estrella, J. Blauert, D. Compilan, J. Gaventa,
J. Gonsalves, I. Guijt, D. Jonson, and R. Ricafort (eds.), (pp. 1�15). Ottawa, Canada: Intermediate Technology
Publications, London, UK, and International Development Research Centre.
66
3534-Rikka.pdf 6/20/05 2:51 PM Page 67
Hanson, J., R. Just, and J. Lainez.. 2004. "Evaluating a Publicly Funded, Privately Delivered Agricultural Extension
System in Honduras." Working Paper 2003-04, Department of Agricultural and Resource Economics,
University of Maryland, College Park, MD.
Horstkotte-Wesseler, G., M. Maredia, D. Byerlee, and G. Alex. 2000. "Ex-Ante Economic Analysis in AKIS Projects."
Methods and Guidelines for Good Practices. Agricultural Knowledge and Information Systems (AKIS). Good
Practice Note. Washington, DC: World Bank.
http://lnweb18.worldbank.org/ESSD/essdext.nsf/26DocByUnid/5A74F2C4E3F9372185256B9E00146BA9/$FILE/Exan
te_Analysis.pdf
Horton, D., P. Ballantyne, W. Peterson, B. Uribe, D. Gapasin, and K. Sheridan, 1993. Monitoring and Evaluating
Agricultural Research: A Sourcebook. Wallingford, UK: CAB International.
Horton, D., R. Mackay, A. Anders, L. Dupleich.2000. Evaluating Capacity Development in Planning, Monitoring and
Evaluation: A Case from Agricultural Research. ISNAR Research Report 17. The Hague: International Service
for National Agricultural Research.
IFAD 2003. A Guide for Project M&E. Managing Impact for Rural Development.
http://www.ifad.org/evaluation/guide/index.htm
Kumar, K. 1993. Rapid Rural Appraisal Methods. World Bank, Washington, DC. http://www.fao.org/docrep/
W3241E/w3241e09.htm
Kusek, J. and R. Rist. 2004. Ten Steps to a Results-Based Monitoring and Evaluation System: A Handbook for
Development Practitioners. Washington, DC: World Bank.
Lecuit, L., J. Elder, C. Hurtado, F. Rantua, K. Siblini, and M. Tovo. 1999. Demystifying MIS--Guidelines for
Management Information Systems in Social Funds. Washington, DC: World Bank. http://imagebank.world
bank.org/servlet/WDS_IBank_Servlet?pcont=details&eid=000094946_99092405344364
Maredia, M., D. Byerlee, and J. Anderson. 2000. "Ex Post Evaluation of the Economic Impacts of Agricultural Research
Programs: A Tour of Good Practice." Paper presented to a workshop on "The Future of Impact Assessment in
the CGIAR: Needs, Constraints, and Options," CGIAR Technical Committee, May 3-6, 2000. Rome: FAO.
Mosse, R., and L. Sontheimer.1996. "World Bank's Performance Monitoring Indicators Handbook." World Bank
Technical Paper No. 334, World Bank, Washington, DC.
Murphy, J., and T. Marchant.1988. "Monitoring and Evaluation in Extension Agencies." World Bank Technical Paper
No.79, World Bank, Washington DC.
Rajalahti, R., J. Woelcke, and E. Pehu. 2005. "Development of Research Systems to Support the Changing Agricultural
Sector." Proceedings of a Research Workshop, World Bank, Washington, DC.
Salmen, L. F. 2000. Beneficiary Assessment for Agricultural Extension. A Manual of Good Practice. Agricultural
Knowledge and Information Systems (AKIS). Good Practice Note. Washington, DC: World Bank.
Score, M. 1995. "Social Impact Assessment in Extension Educational Programming." Journal of Extension. 33(6): 4 p.
http://www.joe.org/joe/1995december/tt2.html
Souvanthong, Pheng, and R.Trethewie.2001. Guide to Participatory Extension Methods Used in Community Forestry in
Laos. Mekonginfo, 2001.
Srivastava, J., J. Kampen, G. Baker, D. Byerlee, and M. Sehga. 2003. Good Practices in Monitoring & Evaluation
and Management Information Systems for Competitive Grant Programs in ECA Countries. Washington, DC:
World Bank. http://imagebank.worldbank.org/servlet/WDS_IBank_Servlet?pcont=details&eid=000090341_
20040331111629
UNDP. 1997. Who are the Question-makers? A Participatory Evaluation Handbook. OESP Handbook Series.
http://www.undp.org/eo/documents/who.htm
Veldhuizen, L., A.Waters-Bayer, and H. Zeeuw. 1997. Developing Technology with Farmers: A Trainers Guide for
Participatory Learning. London: Zed Books Ltd.
Vernon, R. 2001. Knowing Where You're Going: Information Systems for Agricultural Research Management. The
Hague: International Service for National Agricultural Research.
67
3534-Rikka.pdf 6/20/05 2:51 PM Page 68
Wood, S., and P. Pardey. 1997. "Agro-ecological Aspects of Evaluating Agricultural R&D: EPTD Discussion Paper
No.23, International Food Policy Research Institute (IFPRI), Washington, DC.
World Bank. Operational Manual. http://wbln0018.worldbank.org/institutional/manuals/opmanual.nsf/.
World Bank. 1992. "Designing and Implementing Agricultural Extension for Women Farmers." A Technical Note. Women
in Development Division, Population, and Human Resources Department, World Bank, Washington DC.
World Bank. 1997. The Logframe Handbook. A Logical Framework Approach to Project Cycle Management.
http://wbln1023/OCS/Quality.nsf/Main/MELFHandBook/$File/LFhandbook.pdf
World Bank. 2001. Thematic Brief: Agricultural Knowledge and Information Systems; Agricultural Research,
Extension, and Education, 2001.Washington, DC: World Bank.
World Bank. 2003a. Extension and Rural Development--Converging Views on Institutional Approaches? Workshop
Summary. Agriculture and Rural Development Department, World Bank, Washington DC.
World Bank. 2003b. "Monitoring and Evaluation Component in Agricultural Operations. A Toolkit. Agriculture and
Rural Development." Department Draft, World Bank, Washington, DC.
World Bank. 2004a. Agriculture Investment Sourcebook. Washington, DC: World Bank.
World Bank. 2004b. Investments in Agricultural Science and Technology. Agriculture Investment Sourcebook. Module
2. Washington, DC: World Bank.
68
3534-Rikka.pdf 6/20/05 2:51 PM Page 69
APPENDIX 1. NARRATIVES OF
NEW APPROACHES AND INSTRUMENTS IN ARE SYSTEMS
The recent changes in the agricultural research and extension systems are primarily driven by
changing global and local needs as well as the overall need to address poor fiscal sustainability, dif-
ficulties with attribution of impact, low political commitment, insufficient local content, weak
accountability or transparency, and weak linkages between research and extension, generally asso-
ciated with ARE systems.24The changes, such as decentralization, privatization, democratization,
and increasingly separate functions of financing and delivery, have resulted in a slow transition
from supply-driven technology generation toward building a more demand-driven agricultural inno-
vation system (AIS). Agricultural innovation system essentially links multiple sources of innova-
tion and uptake pathways along the continuum from basic research to technology adoption
recognizing the innovative capacity of all stakeholders from producers to the marketplace. Thus,
focus is not on the science suppliers but on the totality of actors that are involved in innovation. See
box A1.1 for further details.
BOX A1.1: AGRICULTURAL INNOVATION SYSTEM
A national innovation system is made up of the institutions, enterprises, and individuals that demand and
supply knowledge and technologies; and the rules and mechanisms by which these different agents are
interacting. The focus is not on the science suppliers but on the totality of actors that are involved in
innovation.
The ultimate objective of a well functioning innovation system is to serve the needs of the economy by
achieving better integration of the S&T infrastructure with production needs; by increasing private sec-
tor participation in technology development; and by developing stronger linkages between industry, uni-
versities, and research institutions.
The past approaches (e.g., NARS and AKIS) have strengthened research systems that increase the avail-
ability of new knowledge and new technologies, but not necessarily the number of innovations that will
be implemented; also the role of market and private sector has not been very strong. A lot of changes,
many of which have tried to ensure that new knowledge is relevant in the market context and that the
role of the private sector in the development and diffusion of new knowledge is explicitly recognized,
have taken place.
Source: Goel V.K., E. Koryukin, M. Bhatia, and P. Agarwal 2004.
A number of instruments and approaches, such as contracting for services, competitive research
grants (CRGP), participatory and farmer-to-farmer R&E, and communication media, are being
introduced into many ARE systems in order to facilitate the change. The approaches and instruments
are often used simultaneously and may overlap to some extent. Table A1.1 summarizes the potential
of selected instruments to address the various challenges.
69
3534-Rikka.pdf 6/20/05 2:51 PM Page 70
Table A1.1. Potential of Selected Innovative Instruments to Address Various Fiscal and Institutional
Challenges Associated with Agricultural Research and Extension Systems
Challenges with Competitive Use of
Agricultural Research Contracting Research Participatory Farmer-to- Mass Media
and Extension Systems Services Grants R&E Farmer25 and ICT26
Fiscal sustainability M-H M-H M M-H M-H
Attribution of impact H H H M-H M-H
Political commitment M M M M L
Responsiveness--local content H M-H H H M-H
Accountability---transparency M-H H M-H M H
Weak linkages between R&E M H H H L
Poverty reduction M-H H M-H M M
Specific targeting H H H M-H L-M
Enhanced private sector role H H M M M-H
Note. Potential of the instrument/approach to address the challenge: low (L), medium (M), and high (H).
Source: Riikka Rajalahti, World Bank.
Contracting extension services
Contracting for agricultural extension by the public sector takes many forms and may involve con-
tracts with public sector agencies, non-governmental organizations, universities, extension consult-
ing firms, or rural producer organizations. In contracting systems, in general, the agency--such as
public funding agency--draws up terms of reference and details of services to be provided, and then
contracts for them, usually on a competitive basis. Services to be contracted are usually identified in
consultation with users, although the programs tend to be longer term and more program-oriented.
Contracts may be administered by national governments, the national government in collaboration
with lower level government, or by governments with an NGO that then contracts with private com-
panies. There can be some "reverse" contracting of public extension agents, often subject matter spe-
cialists, by NGOs or private sector and farmer organizations.27
Competitive research grant programs
In CRGPs research providers are selected on a competitive basis, using call for proposals and scien-
tific peer review mechanisms to allocate funding. CRGP complement "core" funding or "block"
grant funding, which annually allocates funds to specific public research organizations for their core
research programs, infrastructure, and human resources. CRGPs are used as financing mechanisms
to mobilize a wide range of research institutions for work on priority areas, develop institutional link-
ages and research capacities across organizations, and to link scientists with users of new technolo-
gies. In the context of agricultural extension, governments typically continue to finance many rural
extension services, but provision of services is more commonly contracted to private advisory serv-
ice firms, NGOs, universities, producer organizations, and other groups. Competitive procedures can
improve quality of services, make providers more accountable for results, and improve efficiency
whereas contracting allows for specialization and selection of service providers according to their
individual comparative advantage.28
70
3534-Rikka.pdf 6/20/05 2:51 PM Page 71
Participatory research and extension
Participatory research and extension comes with many names, such as participatory technology
development (PTD) and local agricultural research committees (CIAL), and with many levels of par-
ticipation. Participatory technology development29 refers to the process in which development work-
ers facilitate the generation and dissemination of agricultural innovations together with rural women
and men. Through this interaction, the partners try to increase their understanding of the main traits
and dynamics of the local farming systems, define priority problems and opportunities, and experi-
ment with a selection of "best-bet" options for improvement. The options are based on ideas and
experiences derived from both indigenous knowledge and formal science. This process of technolo-
gy development is geared toward finding solutions to current problems and developing sustainable
agricultural practices that conserve and enhance the natural resources. An important aspect is that
PTD should strengthen the capacity of farmers and rural communities to analyze the ongoing
processes and develop relevant, feasible, and useful innovations.
Local agricultural research committees30 are participatory platforms for improving decision-making
capacity and stimulating local innovation for sustainable agriculture. They are a permanent agricul-
tural research service staffed by a team of four or more volunteer farmers elected by the communi-
ty. The committees create a link between local and formal research. The CIAL process is iterative
and consists of a number of steps that are all supported by facilitation, monitoring, and evaluation.
The strength of the CIALs lies in their systematic evaluation of technological alternatives and their
ability to influence the research agendas of formal research and extension systems.
Participatory farmer-to-farmer approaches
Farmer-led extension or farmer-to-farmer extension refers to extension methodology that relies
either on informal or formal training or transfer of technologies, or advice by a farmer to another
farmer or a group of farmers. Usually the key farmer receives some technical and methodological
training by extension services and may be partially or fully compensated for this service in kind by
monetary remuneration.
Farmer Field School (FFS)30 is likely the best known farmer-to-farmer extension method. FFS offers
community-based, non-formal education to groups of 20�25 farmers, who are expected to transfer
their knowledge and skills to other farmers in their communities. Discovery-based learning is relat-
ed to agroecological principles in a participatory learning process throughout a crop cycle. FFS fill
gaps in local knowledge, conducts holistic research on agroecosystems and increases awareness and
understanding of phenomena that are not obvious or easily observable. FFS require significant insti-
tutional commitment and support, usually provided by the national extension service.
71
3534-Rikka.pdf 6/20/05 2:51 PM Page 72
APPENDIX 2. IMPLEMENTATION OF
A MONITORING AND EVALUATION PLAN IN
THE BANK'S PROJECT CYCLE: PUTTING IT ALL TOGETHER
THE BANK'S PROJECT CYCLE AND MPLICATIONS FOR
I M&E
The Bank's projects are financed and implemented according to a well-documented project cycle.
The cycle is an eight-step process that provides participants and stakeholders with a full view of the
project development process, planning, and achievement of outcomes from the identification and
concept stage to the post-project completion stage, including full implementation. Therefore, plans
for M&E should be evident at every stage of the eight-step project cycle.31 See Figure A2.1 for a dia-
gram of M&E Steps in the World Bank Project Cycle.
Although M&E are the responsibility of the borrower's project management team, the Bank task team
has a critical role in assisting the client team in setting and achieving M&E objectives. These include:
(a) help identify information needs and specify monitoring objectives;
(b) ensure that all documents in the project cycle clearly establish M&E plans and expect-
ed results for project activities and project coordination;
(c) review the design and organizational arrangements of M&E information system along
with needed special studies to ensure feasibility;
(d) help to finance technical assistance and equipment required for the design and imple-
mentation of M&E activities; and
(e) ensure that training and relevant capacity needs are adequately addressed.
Guidelines for Sound M&E in Innovative ARE Projects
For Agricultural Research and Extension projects, M&E is often needed as permanent elements of
country programs in view of the institution building objectives of some programs. Therefore, an
M&E system must contribute substantially to enhance program sustainability and set the pace for
establishing M&E permanency within country programs.
Each stage of the Bank's eight-step project cycle requires specific action planning and special atten-
tion to the development and evolution of a project M&E system and use of indicators. The follow-
ing are necessary elements that must be visible at all stages of the project cycle:
� During the preparation, sufficient time and resources must be allocated toward establish-
ment of a gender-disaggregated baseline;
� The Bank task team must identify and secure an M&E resource person in its core team to
provide support in the evolution and implementation of M&E systems; where financial
resources are a constraint, M&E must be featured in the TOR of an appropriate mission
member;
� The client team must include a political champion or leader from an implementing
Ministry whose role would be to: (a) promote results-based M&E system and ensure the
institutionalization and sustainability of developed systems or convince of its importance;
and (b) provide the scope for in-service training and capacity building programs;
72
3534-Rikka.pdf 6/20/05 2:51 PM Page 73
� Targeting different impact areas, such as gender, ethnicity, socio-economic equity, partici-
pation, and consultation should be considered at all stages of the project cycle and can sig-
nificantly influence the project design. Indicators sensitive to such targeted areas may
determine the extent to which the extension and research services take these issues serious-
ly and ensure an efficient functioning of systems developed;
� Sufficient donor support in the implementation and funding of M&E activities such as
modernizing or developing MIS systems and statistical capacity increases the value of
M&E.
The following are proposed means of implicating M&E at each stage of the project cycle:
1. CAS level identification: The Bank prepares lending and advisory services for a specific country
based on selectivity framework and areas of comparative advantage, targeted to country PRSP efforts.
The process begins when sector preparation groups identify specific sector deficiencies and a range
of possible areas or themes for project intervention to be initiated over a three-year CAS period.
2. Identification: Projects are identified that support strategies and that are financially, economical-
ly, socially, and environmentally sound. Developed strategies are analyzed. Diagnostic case studies
are conducted using rapid/participatory rural appraisal to assess problems and potential constraints,
opportunities, special issues with farmers and other stakeholders in a given agro-ecological area. The
project concept note (PCN) sets the pace for developing M&E indicators, and subsequently provides
the scope and elements of the project M&E system. Results of case studies/theme proposals feed into
project baseline information, identifies opportunities, special scenarios, and other stakeholders.
3. Preparation: The Bank provides policy and project advice along with financial assistance.
Clients conduct studies and prepare final project documentation. More comprehensive quantita-
tive surveys are carried out to determine the extent and need for intervention and importantly,
provide baseline data for the research environment and for selected extension areas. These can
either be narrowly focused to capture specific themes relevant to project components and relat-
ed project activities, or broadly targeted to capture different but related themes and opportunities
that warrant the need for subsequent or additional interventions. They may include situation
analysis and participatory needs assessment to appraise current farm households, local condi-
tions as well as institutional/organizational issues in the research/extension continuum. Within
the framework of such studies, it will be necessary to assess prospects for institutional reforms
and technological changes to impact the current production environment. All preparation relevant
to M&E, including preliminary discussions of all study results, should be undertaken and com-
pleted by pre-appraisal.
4. Appraisal: The Bank assesses the key parameters for investment and prepares the PAD and
draft legal documents. Preparation follows a pre-appraisal phase and subsequently an apprais-
al phase. In between the two phases, the project appraisal document (PAD) is developed. The
most important feature for M&E in the PAD is the logical framework which establishes outputs,
outcomes, and impact indicators with established baselines and controls. Although the Bank has
now shifted from the log frame to the results framework, which is a more simplified version of
the log frame, it is strongly recommended for ARE projects that an extended log frame be pre-
pared with indicators to capture the outputs and outcomes of all activities by component. A
results framework may then be derived from the log frame to satisfy Bank requirements. The PAD
73
3534-Rikka.pdf 6/20/05 2:51 PM Page 74
should also provide a description of M&E arrangements, including a M&E plan and Management
Information System, for the entire project phase.
Pre-appraisal phase: M&E arrangements initiated at pre-appraisal and firmed up at appraisal
include the following:
� Development of M&E framework and detailed M&E plan which will include:
� M&E system and how it functions in the context of the project
� M&E information flow, sources, and uses of data
� Key players in M&E project activities
� Reporting formats for M&E
� Timeframe for key information requirements and follow-up studies
� Frequency for baseline updates and control group monitoring as needed;
� Development of M&E manual of procedure;
� Initiate an assessment of M&E institutional and human capacity including capacity to inte-
grate gender and relevant target areas into project activities;
� Develop a plan for M&E capacity enhancement; and
� Develop M&E training program.
Appraisal phase: During appraisal, the Bank and the borrower will:
� Discuss and agree on (a) log frame/results framework indicators and identified targets for
outputs, outcomes and impact and (b) established baselines and controls;
� Through a validation workshop, finalize and adopt (a) the M&E framework and (b) the
M&E manual; and
� Review with client, the results of the capacity assessment study and agree on a capacity
enhancement plan and staff training program.
5. Negotiation and board approval: The Bank and borrower agree on loan/credit proceeds and
presents to the board for approval. The implementation of the loan/credit is negotiated based on
agreed upon plans and presented to the board for approval.
6. Implementation and supervision: The borrower implements the project and the Bank supervis-
es loan with due regard for economy, efficiency and effectiveness.
Pre-implementation: Between board approval and actual implementation start-up, there is a period
during which the project becomes effective. The following will need to be accomplished during the
pre-implementation or project effectiveness period:
� M&E Sub-unit within the implementing/coordinating unit of the project need to be estab-
lished and fully equipped;
� M&E focal points established at decentralized locations of project implementation;
� Because M&E requires the participation and contribution of all project staff, at least the
initial M&E training program for project staff should be completed; and
� Workshop held for project staff to internalize M&E manual.
74
3534-Rikka.pdf 6/20/05 2:51 PM Page 75
Implementation: Indicators that have been developed are now used to monitor implementation
progress by activity. Using the information flow chart and the reporting formats agreed upon, feed-
back is circulated to support program management and stakeholders. The central data processing and
information consolidation point is the M&E subunit of the project coordinating/implementation
unit--usually under the implementing Ministry. The M&E focal points in the respective implement-
ing agencies must provide routine reports on progress (inputs, outputs, outcomes) to the M&E sub-
unit within the project management unit. Consolidated reports and analyzed data complemented with
information on progress from periodic assessments, case studies, and activity appraisals, are put at
the disposal of project management for dissemination and action as appropriate. Periodic updates of
baseline studies are necessary to monitor and capture changes in selected areas resulting from proj-
ect interventions and other factors.
Supervision: Supervision/implementation support missions must include an M&E resource person
or a core team member with the ability to review: (a) M&E activities; (b) the functionality and ade-
quacy of M&E systems; and (c) ensure that project indicators are being monitored adequately by
M&E focal points at central and decentralized levels. Indicators and their respective targets should
be reviewed to ensure that planned project activities are on track in terms of planned levels of out-
put realizations, achievement of outcomes, planned timeframes, and use of allocated resources.
Specific roles of M&E during the supervision missions include:
� Systems adjustment: M&E systems once designed and operational, should not be an end
in itself; it should have some scope for adjustment in terms of (a) new issues that might
emerge during implementation, and (b) lessons on the system operation and the functioning
that may be incorporated into the design. Supervision missions will provide the scope for
such adjustments;
� Capacity adequacy: Missions will review, assess, and analyze capacity adequacy for M&E
support team and propose training needs and additional skills as necessary;
� Response capacity: Missions will review and assess the response capacity of high-level
client/partner officials on M&E implementation process and propose ways of enhancing
apparent deficiencies;
� Sensitivity to targeted areas: Mission will pay particular attention to progress achieved in
gender/ethnicity/poverty sensitive indicators. In particular, the extent to which the project
has succeeded in integrating these target areas into project implementation; and
� Guidance to Bank Task Team: on any salient M&E issue that might emerge from either
client, wider stakeholder community, or members of the Bank core team.
7. Implementation and completion: The implementation completion report is prepared to evaluate
the performance of both Bank and the borrower. Each party is required to prepare and submit an
Implementation Completion Report (ICR). Institutionalization of M&E actions within implementing
Ministries as well as harmonizing the M&E systems of various projects within sector Ministries has
specific implications for project completion and post-project activities. M&E must not end at proj-
ect completion--mainstreaming M&E within implementing Ministries establishes M&E as an ongo-
ing function of government policy units and technology programs. Lessons must continuously be
drawn from evaluation efforts throughout project implementation. A key indicator for project sus-
tainability and continuity or mainstreaming project activities with regular functions of the Ministries,
75
3534-Rikka.pdf 6/20/05 2:51 PM Page 76
is the extent to which the project M&E system has been established and used at project completion
and for post-project activities. Box 5.1 in chapter 5 provides specific highlights for M&E during ICR
preparation and write-up.
8. Evaluation: The Bank's independent OED audits and evaluates the project. Analysis feeds into
future project design. OED should pay particular attention to the ratings that have relevance to M&E
implementation and outcomes.
Figure A2.1. Diagram of M&E Steps in the World Bank Project Cycle.
8
Project Evaluation
7. 1.
Independent OED audits
Implementation And CAS level
And evaluates project
Completion identification
ICR evaluates performance Bank prepares lending
Of both Bank and Borrower and advisory services
M&E FOR AR & E
6.
2.
Implementation by borrower
In the Bank's Project identified in line
Supervision by Bank with
CAS and PRSP
due regard to efficiency
Project Cycle PCN prepared
& effectiveness
5. 3.
Negotiations and Project Preparation
Board Approval 4. Bank provides policy
Loan/credit proceeds Appraisal and Project advice
are agreed upon Pad and draft legal along with financial asst
documents prepared
by Bank
Source: Adapted from
http://intranet.worldbank.org/WBSITE/INTRANET/INTCOUNTRIES/INTLAC/0,,menuPK:257867~pagePK:64089870~piPK:6408
9874~theSitePK:257804,00.html
76
3534-Rikka.pdf 6/20/05 2:51 PM Page 77
APPENDIX 3. A BRIEF DESCRIPTION
OF THE "EXTENDED" LOGICAL FRAMEWORK
What is log frame?
� Log frame is a program management technique used by the World Bank (since 1997) and
other development organizations to manage the complete project cycle from design, imple-
mentation, monitoring, and evaluation.
� The log frame is a "cause and effect" model of project interventions to create desired
impacts for beneficiaries.
Why use it?
� It is used to develop the overall design of the project, to improve project implementation
monitoring, and to strengthen period project evaluation. It is a tool that has the power to
communicate the essential elements of a project clearly and succinctly throughout the proj-
ect cycle.
� Developing the log frame helps the project team and the client to precisely define the dif-
ferent components and how they are linked and contribute to the Project Development
Objective (PDO).
Components of the log frame:
� In the Bank's log frame, the multi-step process by which investments lead to impacts is
compressed into a four-step format under a 16-box matrix as presented in Table A3.1 below
(excluding the rows on project component statements).
� The matrix presents the cause-effect-logic between inputs (financial, human, and material
resources), activities (tasks personnel undertake to transform inputs to outputs), outputs
(products and services produced), Development Objective (effects of outputs on target
groups), and Country Assistance Strategy goal (long-term, widespread improvement in
society--impact).
� In addition, consideration is given to the critical assumptions--the external risk and
enabling factors outside the direct control of the project.
� The basic logic is that when the critical assumptions hold, the project interventions through
inputs and activities generate component outputs. The outputs, in turn, are expected to
result in desired impacts (project objective level outcomes and impacts) for beneficiaries.
To track the achievement of each objective, a set of performance indicators will be set for
each objective level.
� Information regarding the use of log frame in program design can be found in detail in: The
Log frame Handbook. A Logical Framework Approach to Project Cycle Management.
http://wbln1023/OCS/Quality.nsf/Main/MELFHandBook/$File/LFhandbook.pdf
77
3534-Rikka.pdf 6/20/05 2:51 PM Page 78
Why "extended" logical framework?
� Although it is mandatory to develop the results framework and the arrangements for results
monitoring, the project team may also develop full log frame that introduces an added level
of detail for component inputs, outputs and outcomes.
� The "extended log frame" approach may help the project team and the client to precisely
define the different components and how they are linked and contribute to the PDO.
� The extended log frame can be a useful tool in facilitating discussions and development of
the project design with the national team.
� It may also assist and complement project management, and may support the development
of a sustainable M&E system beyond the lifetime of the project as it often better fits the
requirements for M&E in the client countries.
� Table A3.1 below provides an example of the "Extended" Logical Framework.
Table A3.1. The "Extended" Logical Framework
Monitoring &
Narrative Summary Performance Indicators Evaluation Plan Critical Assumptions
CAS Goal Sector goal level indicator The program evaluation system. CAS Goal to
Super Goal
Project Development Outcome indicator People, events, processes, sources PDO to CAS Goal
Objective (PDO) of data for organizing the project
evaluation system.
Project Component Intermediate Outcome People, events, processes, sources Project Component
Statement of data for organizing the project to PDO
M&E system.
Outputs Output indicator People, events, processes, sources Output to Project
of data--supervision and Component
monitoring system for project
implementation.
Project Component Inputs (budget for each People, events, processes, sources Component Activity
Activities component/activity) of data and monitoring system to Output
for project design.
Source: Authors. Adapted from The Log frame Handbook. A Logical Framework Approach to Project Cycle Management.
78
3534-Rikka.pdf 6/20/05 2:51 PM Page 79
APPENDIX 4. LIST OF INDICATORS
AND DATA SOURCES FOR ARE PROJECTS
A. DIFFERENT INDICATOR LEVELS
Budget and input level: Tracking budgets and input use is not only an important element of
sound project planning and monitoring, but also crucial element in the evaluation of project suc-
cess. During the planning process budgets and input requirements are established as part of pri-
ority setting. These cost projections (budgets) must be linked to expected outcomes and are an
essential element for evaluation of any program, as benefits are assessed in relation to program
cost. Accurate and timely budget and expenditure data are important to the efficiency and effec-
tiveness of a program evaluation. Expenditure data are commonly used and helpful in tracking
implementation. The required information for the preparation of the log frame or results frame-
work can be easily extracted from the cost tables. If used alone, however, it is an inappropriate
and insufficient indicator of project implementation progress and performance. Based on the pro-
jected and actual budgets two important evaluation methods must be highlighted here: ex ante
and ex post economic analysis. Ex ante economic analysis relating project costs and benefits is
a standard means of evaluating whether the proposed investments are financially and economi-
cally justifiable. Ex-post economic analysis uses similar methodology to evaluate investment
impacts (see chapter 6).
Output level: Outputs are direct products of project activity and within the control of the project.
Project output level M&E focuses on relatively simple quantitative measures of productivity and
physical completion rather than on the qualitative assessment of the goods produced. Data on out-
puts should come from the management information system (see Chapter 3).
Bank-supported agricultural R&E projects typically focus on two objectives: (a) institutional devel-
opment to strengthen the institutions and technology system necessary to develop and disseminate
improved technology and management in the agricultural sector; and (b) productivity change due to
technological innovation introduced through the technology system
The two objectives are related and mutually supportive, but require different indicators for outputs
and outcomes. An illustrative list of key indicators, with particular focus on CRGS and contracting
extension services, is included in section B.
Outcome level: Outcomes can be defined as the intermediate effects of outputs on clients but are not
under full control of the project. Although the measurement of outputs is mainly based on simply
quantifiable indicators, the measurement of outcomes adds a quality dimension which is more diffi-
cult to quantify. The newly introduced results framework requires the definition of outcomes and out-
come indicators not only at the PDO level, but also at the component levels.
Component level: Outcomes and outcome indicators specify the expected benefit for specific
target group from component implementation. In AIS projects, the target group of components
are very often researchers and extension agents or institutions whose performance are expected
to be improved to better serve their respective clients, agricultural producers. Hence, combined
outcomes at the component level contribute to the desired outcome at the PDO level. At the PDO
level, outcome indicators reflect the initial results or intermediate impacts of the overall project.
79
3534-Rikka.pdf 6/20/05 2:51 PM Page 80
While formulating the outcome statements of the different components it is important to note that
the various outcomes are not independent of each other but instead reinforce each other to con-
tribute to the overall objective of the project.
In many recent Bank-supported ARE projects, the project components focus on institutional
development outcomes, and the PDO focuses on productivity outcomes. However, it is critical to
recognize that a PDO can also focus on institutional development outcomes directly. Institutional
development outcomes vary considerably depending on the technology system needs and the pro-
posed project design. These may reflect changes in the institutions themselves, institutional pro-
ductivity, or institutional relations with clients. Project outcomes cannot always be reported on a
routine basis by a project management information system. For this reason, special studies are
usually necessary to measure outcomes. Chapter 3 and section C below provide more details on
data sources.
Sector goal indicators: Sector goal indicators are not required any longer for Bank-supported proj-
ect M&E when a results framework is being developed. However, selection of sector goal indicators
may still be useful (a) as project documents have to describe how the project contributes to higher
level objectives and (b) when the project team and in particular the counterparts in client countries
decide to implement a permanent M&E system with a long-term perspective based on the log frame
approach.
Sector goal indicators measure change in the broad development goal to which the project con-
tributes. Changes in these indicators are indirectly attributable to the project, and usually can-
not be measured during the life of a Bank-financed project. Although demonstrating direct
project impact on a goal level indicator may not be within the "manageable interests" of a proj-
ect, it is important the project provides a plausible hypothesis of how project activities will lead
to change at the level of the sector goal. This often requires other sources of research and analy-
sis to build a plausible link between project activities and the social goal. Ex post impact analy-
sis are important to demonstrate past success and guide future investments. The trick with
impact assessment is that of comparing the "with project" situation with the counterfactual
"without project" scenario.
Sector-level impact indicators are similar for research and extension projects. Despite the difficul-
ties with attribution and detecting change, R&E projects should link activities to impacts on indica-
tors of social goals as a basis of justifying investment. Models may be useful in relating R&E
programs to changes in social goals and in estimating impacts that are not otherwise readily meas-
urable (Wood and Pardey 1997). Methodologies for use of such models are still under development,
and more work may be needed to develop practical models that bridge the gap between measurable
outcomes (adoption and effects of adoption) and impacts on social indicators (Maredia, Byerlee, and
Anderson 2000). Development and verification of such models are important tools for assessing var-
ious influences on agricultural production processes, and may be appropriate for financing under
R&E projects. Such modeling is likely to be more appropriate and useful as a research activity with-
in a country research program, and less practical for estimating impacts for a Bank-financed project
M&E system.
80
3534-Rikka.pdf 6/20/05 2:51 PM Page 81
B. INDICATOR LIST
The following is a menu of possible goal, outcome, and output indicators for agricultural research
and extension projects. The list includes social, environmental, and institutional-capacity-building
indicators. It is important to note that the list is not exhaustive but provides suggestions.
� Goal level indicators are not mandatory any more for World Bank-supported projects.
However, these indicators may be of interest if the Bank team chooses to use the log frame
approach as well as for the counterparts in the client countries who would like to imple-
ment an M&E system like the project.
� Due to cost and time considerations it is recommended that only a limited number of rele-
vant indicators should be selected to meet the needs of specific projects.
� The level (especially for PDO outcomes vs. component outcomes; and outputs vs. out-
comes) at which a specific indicator is relevant may vary depending on the specific project
or program objectives.
1. Goal level indicators (typically not monitored by projects)
All ARE Projects
Goal Indicator Measurement Source
� Rural population below (national) poverty line % number GDB/CDB
� Rural population with incomes < $1/< $2 per day % number GDB/CDB
� Daily per capita dietary energy supply Calories/capita GDB/CDB
� Malnutrition prevalence (rural and urban) % number GDB/CDB
� Agricultural GDP value, % growth / year GDB/CDB
� Agricultural GDP / agricultural worker value, % growth / year GDB/CDB
� Agricultural GDP / rural worker value, % growth / year GDB/CDB
� Agricultural exports value, % growth / year GDB/CDB
� Agricultural employment (full-time; part-time) % or number GDB/CDB
� Freshwater withdrawal for agriculture Water use / agricultural GDP; GDB/CDB
total; % of total available
� Deforestation % per year; GDB/CDB
% change in rate over time
� Agricultural land loss/total arable area % change in rate over time/ GDB/CDB
% per year
� Land use pattern % change in rate over time GDB/CDB
Source: Authors. Adapted from Alex and Byerlee (2000).
81
3534-Rikka.pdf 6/20/05 2:51 PM Page 82
2. Outcome indicators (Level: Component and Project Development Objective)
Extension Projects--Institutional Development
Outcome Indicator Measurement Source
� Co-financing of extension service by farmers (disaggregated by % of total cost PDB
farm size, gender, age, farm type, income)
� Co-financing of extension service by providers (disaggregated by % of total cost PDB
type of service providers, e.g., private enterprise, NGO, producer
organization, public agency)
� Agencies providing extension services (disaggregated by type of Number, % of total number PDB
service provider, e.g., private enterprise, NGO, producer of subprojects
organization, public agency)
� Regions covered by (decentralized) extension system (or covered Number of projects in each PDB
by extension projects) region; % of total
� Women, youth, and minority groups reached by extension Number, % of total PDB / SS
services or participating in extension subprojects
� Priority areas captured (as defined at national or regional level) Number of extension activities/ PDB
through extension system / extension projects subprojects per priority area
� Quality/skills of extension providers (natural resource Number of certified PDB / SS
management, participatory technology development, enterprise extension agents
development & marketing, organization skills)
� Environmental topics captured through extension system Number of extension PDB
activities/area
� Motivation expressed by the extension staff % change PDB / SS
� Perceived fit (by extension staff) between supply and demand % change PDB / SS
for services (priorities/geographical area/target groups covered)
� Intensity of extension coverage Families/agent or area/agent PDB / SS
� Extension cost per farmer, per hectare, or per unit value Cost per unit PDB / SS
of production
� Efficiency of extension activity (demonstrations, training, field days) Number/agent, Cost / activity PDB / SS
� Gender balance of extension staff % women PDB / SS
� Ethnic balance or minority language capability of extension staff % minority or language capable PDB / SS
� Non-salary operational costs as a percentage of total extension % of total budget CDB / PDB
budget (national and project level)
� Service provider awareness of competitive fund for extension Number of application per call PDB
service provision and capacity to apply successfully (disaggregated and number or % accepted
by type of service provider)
� Farmer evaluation of extension providers Ratings SS
� Farmer willingness to share extension costs Number or % change, % total SS
� Farmer requests for extension services (disaggregated by gender, Number, % of total PDB / SS
minority groups)
82
3534-Rikka.pdf 6/20/05 2:51 PM Page 83
Extension Projects--Institutional Development--continued
Outcome Indicator Measurement Source
� Farmer awareness of extension activities (disaggregated by % of farmer SS
gender, minority groups)
� Farmer, private sector, and civil society representation on extension
management committees (by sex, ethnicity, farm size, wealth) Number, % SS / PDB
� Functioning M&E system with active participation
of executing institutions Number of impact studies; PDB / ER /
number of progress reports; SS
general assessment of system
� Efficiency of fund administration unit Overhead costs (total and PDB
% of total project costs);
Average number of weeks
or month from call for proposals
and signing contract
Extension Projects--Technology Dissemination
Outcome Indicator Measurement Source
� Adoption of new practices or technologies (disaggregated by Total number of adopters PDB/SS
participants and non-participants); including studies of rationales (including spillovers);
for adoption / non-adoption by gender, minority group, age % of targeted farmers
and all farmers
� Change in income, welfare, costs, yield, productivity, as a result % change from base SS
of new technologies or practices (disaggregated by gender,
minority groups, age)
� Change in farm/household labor needs (disaggregated by gender, % change from base SS
minority groups, age)
� Change in relative income from on-farm and off-farm activities % change from base SS
(disaggregated by gender, minority groups, age as appropriate)
� Adoption of environmentally-sound technologies Area (ha); number of farmer; PDB/SS
reduced use of agrochemicals
(in liter or kg); reduced erosion
(soil loss in kg per area)
� Change in farmers organizations (autonomy, leadership, funding, % change, Number SS
membership, participation, planning, activities)
� Change in farmers/organizations/communities' ability to % change from base, Number SS
market products
� Changes in use of natural resources (water, forest, land) Rate of use; area SS
� Yield gap between farmers' yields and on-farm trials % difference SS
� Yield gap between national yields and neighboring countries % difference CDB
continued on next page
83
3534-Rikka.pdf 6/20/05 2:51 PM Page 84
Extension Projects--Technology Dissemination
Outcome Indicator Measurement Source
� Time-series or case studies of change in farmer knowledge, % change from base SS
skills, attitudes, or understanding of technologies and practices
� Change in land area used for sustainable, integrated production Area (ha) PDB/SS
� Farms/farmers participating in environmental farm plan programs Area (ha); number of PDB/SS
farms/farmers
Research Projects--Institutional Development
Outcome Indicator Measurement Source
� Institutions engaged in execution of research projects (in Number, % of total PDB / SS
CGS subprojects); disaggregated by type (e.g. public universities,
private universities, private enterprises, national research
organization, producer organization)
� Co-financing of research projects (CGS) by executing research Total or % of project costs PDB / SS
institutions; disaggregated by type (e.g. public universities, private
universities, private enterprises, national research organization,
producer organization)
� Awareness of competitive research fund and capacity to Number of application per call PDB
apply successfully (disaggregated by type of research institute) and number or % accepted
� Extent of inter-institutional, regional, and international Number of projects; PDB
collaboration (CGS) % of budget
� Identified research priority areas covered (CGS) Number of projects per PDB/CDB
priority area; % of total budget;
scientist time allocated
� Disciplinary, product and regional balance of research projects Number of projects per priority PDB/CDB
(CGS) area; % of total budget;
scientist time allocated
� Balance of strategic, applied, and adaptive research (CGS) Number of projects per category PDB/CDB
� The degree to which the objectives and activities reflect % change PDB/SS
stakeholder priorities
� Graduates from education programs / projects (CGS) Number of MSc or BSc PDB
� Competitive funding share of total national research funding Total; % of total budget PDB/CDB
� Stakeholder participation in governance, priority setting, and Assessment of participation SS /PDB
execution (e.g., stakeholder representation
on boards)
� Farmer representation on research management committees Number, % PDB/SS
(by sex, ethnicity, farm size, wealth, age)
� Research scientists by level of qualification (by sex; by priority area) Number and qualification CDB/SS
(MSc and PhD)
84
3534-Rikka.pdf 6/20/05 2:51 PM Page 85
Research Projects--Institutional Development--continued
Outcome Indicator Measurement Source
� Research scientists by technical specialty--economics, social Number CDB/SS
sciences, natural resource management, agronomy, business
development (by sex; by priority area)
� Recurrent costs or personnel costs relative to total research % of total budget CDB/PDB
budget (CGS)
� Peer reviewed publications, research reports, conference paper, Total number; number per PDB/SS
patents (by priority area) year per researcher
� Ratio of professional to support staff and trained technician Ratio PDB/SS
(by sex)
� Staff time devoted to research (CGS) % of time SS/PDB
� Research-extension linkages (CGS) % of staff time devoted to SS/PDB
extension activities; number
of joint activities (workshops,
field days, and field trials)
� Research programs applying participatory technology Number or % PDB/SS
development methodology/farmer participation in research activities
� Annual research staff turnover % of all staff CDB/SS
� Functioning M&E system with active participation of executing Number of impact studies; PDB/ER/
institutions number of progress reports; SS
general assessment of system
� Efficiency of fund administration unit Overhead costs (total and PDB
% of total project costs);
Average number of weeks or
month from call for proposals
and signing contract
Research Projects--Technology Development
Outcome Indicator Measurement Source
� Adoption of new technologies and management recommendations Total number of adopters PDB/SS
(disaggregated by participants and non-participants); including (including spillovers); % of
studies of rationales for adoption / non-adoption by gender, targeted farmers and all farmers
minority group, youth
� Change in income, welfare, costs, yield, productivity, as a result % change from base SS
of new technologies or recommendations (disaggregated by
gender, minority groups, youth)
� Change in farm/household labor needs (disaggregated by gender, % change from base SS
minority groups, age)
� Change in relative income from on-farm and off-farm activities % change from base SS
(disaggregated by gender, minority groups, age as appropriate)
continued
85
3534-Rikka.pdf 6/20/05 2:51 PM Page 86
Research Projects--Technology Development--continued
Outcome Indicator Measurement Source
� Client satisfaction with new technologies or management Rating (% of positive response) SS
recommendations
� Adoption of environmentally-sound technologies Area (ha); number of farmer; PDB/SS
reduced use of agrochemicals
(in liter or kg); reduced erosion
(soil loss in kg per area)
� Changes in use of natural resources (e.g., water, forest, land) Rate of use; area SS
� Research programs applying participatory technology Number or % PDB/SS
development methodology
� Yield gap between farmers' yields and on-farm trials % difference SS
� Yield gap between national yields and neighboring countries % difference CDB
Source: Authors. Adapted from Alex and Byerlee (2000).
3. Project output indicators
Extension Projects--Institutional Development
Output Indicator Measurement Source
� Regional/decentralized offices established Number PDB
� Extension provider staffing (disaggregated by region; by sex) Number and qualifications PDB
� Extension staff assigned in the field %; number of agents living
in communities they serve
� Extension service strategies and plan completed Number PDB
(disaggregated by regions)
� Extension agents training support (e.g., induction courses, and Number of staff trained; PDB
in-service training) training days per year
� Extension materials preparation unit established (by region) Number PDB
� Extension agents trained on NRM issues Number of staff trained PDB
� ICT support capacity (by region) Number of offices/agents PDB
with computer (internet) access;
number of computer/telephones
per agent
� Transport capacity (by region) Number of vehicles/bicycle PDB
per agent
� Extension management committees established Number
� Competitive fund management unit established Assessed adequacy PDB/ER
� M&E system (including management information system) established Assessed adequacy PDB/ER
86
3534-Rikka.pdf 6/20/05 2:51 PM Page 87
Extension Projects--Technology Dissemination
Output Indicator Measurement Source
� Farmer training courses conducted (per region) Number PDB
� Farmer trained, contacted, visited (by sex, per region) Number PDB
� Frequency of face-to-face meeting with farmer (by region, by sex) Average number per month; PDB/SS
average time spent
� Demonstration established (on-station and on-farm per region) Number PDB
� Farmer contests conducted Number PDB
� Number of producers organizations established Number PDB
� Farmer group meetings held (per region) Number; number of participants PDB
� Information material produced and disseminated (per region) Number of brochures and PDB
pamphlets
� Extension service projects (CGS) Number PDB
� Environmental farm program trainings conducted Number PDB
� Farmer-led trainings conducted Number PDB
Research Projects--Institutional Development
Output Indicator Measurement Source
� Research projects (CGS) Number PDB
� CGS promotion workshops/seminars Number PDB
� Graduate courses established (PhD, MSc, BSc, short courses) Number PDB
� Curricula revised or developed Number PDB
� Costs of graduate programs Cost per graduate PDB
� Proposal review committee established (CGS) Assessed adequacy PDB/ER
� Competitive fund management unit established Assessed adequacy PDB/ER
� Research priority setting mechanisms/committee implemented Assessed adequacy PDB/ER
� Training support for researcher in priority areas Number of workshops, seminars PDB
� ICT support capacity (e.g., computer/internet access; access to Number of researcher with PDB
electronic journals) computer/internet access/
electronic journals; number of
computers per researcher
� Journals/publications acquired Number PDB
� Laboratories equipped Assessed number and qualityPDB
� Research-extension linkages established Number of full-time SMS; PDB
number of joint seminars/
workshops
continued
87
3534-Rikka.pdf 6/20/05 2:51 PM Page 88
Research projects--Institutional development--continued
Output Indicator Measurement Source
� Regulatory and legal frameworks for research and technology Assessed adequacy PDB/ER
importation established
� Researchers trained in participatory technology development Number PDB
� Natural resource management researchers hired Number PDB
Research Projects--Technology Development
Output Indicator Measurement Source
� New varieties developed Number PDB
� Extension recommendations published Number PDB
� Other research products completed; research tools / Number PDB
methods developed
� On-farm research projects (CGS) Number PDB
� Papers published Number PDB
� Environmentally sound technologies developed Number PDB
Source: Authors. Adapted from Alex and Byerlee (2000).
4. Input indicators--All ARE projects
Input Indicator Measurement Source
� Civil works completed Number and value PDB
� Goods (e.g., equipment and vehicles) procured Value PDB
� Technical assistance provided Person-weeks and value PDB
� Unit costs of inputs Cost per unit PDB
Source: Authors. Adapted from Alex and Byerlee (2000).
Abbreviations: GDB = Global Database; CDB = Country Database; PDB = Project Database; SS = Special Studies; and ER =
Expert Review.
88
3534-Rikka.pdf 6/20/05 2:51 PM Page 89
C. SOURCES FOR BASELINE DATA AND NDICATORS
I
Agricultural Innovation Systems project M&E systems must balance data needs between what is
ideal and also costly, and what is practical, and will generally need to draw on data from various
sources. These sources will be described briefly in the following paragraphs.32
Management information system
Project input-output data are generally available from a project management information system
(MIS), the project's internal system for collection, analysis, and dissemination of project information.
National and international level databases
As it is generally not cost-effective for a project to measure national change in goal-level indicators,
information needs at this level rely on national and international databases. These databases may pro-
vide data for indicators such as rural population below poverty line, daily per capita dietary energy
supply, agricultural GDP, and public expenditures on agricultural research as percentage of the agri-
cultural GDP. Quality of national data may be suspect to poor sampling procedures or intentional dis-
tortions, but it is readily available and comprehensive, represents considerable investment in data
collection, and is official, government-sanctioned data. Changes in national-level indicators are usu-
ally gradual and not observable over a typical five-year Bank-financed project. It may be advisable
to dampen the fluctuations because agricultural statistics are subject to wide year-to-year fluctuations
due to weather and commodity price changes, tracking three-year averages and statistical trends
rather than annual data. For example, relevant data may be collected by the Ministry of Finance,
Ministry of Agriculture, or Ministry of Science. These data can be supplemented by international
databases.33
Special studies
Bridging the gap between indicators from routine reporting on project activities and from national
data sets is a challenge for ARE projects, and reflects the difficulty inherent in linking project per-
formance and eventual impacts. Special studies are a practical tool for collecting information on
agricultural systems, program impacts, and especially for assessing and measuring outcomes. In gen-
eral, special studies should provide for disaggregating by gender, ethnicity, or other key characteris-
tics of client populations.
In addition to surveys or case studies to assess outcomes, special studies might be needed to track
specific planned and unplanned impacts or program operational issues (e.g., technology access by
women or minority groups, and changes in land tenure and other social conditions in the area).
Beneficiary and stakeholder participation in M&E is critical to ensure that R&E projects are
responding to genuine needs of intended clients. This can be accomplished by using survey tech-
niques or participatory evaluation (see chapter 4).
Special studies can usually be built into the regular work plan of ARE institutions, as extension agen-
cies must track changes in farmer attitudes, knowledge, and technology adoption; research programs
need to know results of new technologies; and training institutions need to know how graduates per-
form. In some cases, it may be best for monitoring to be done by an independent institution to avoid
89
3534-Rikka.pdf 6/20/05 2:51 PM Page 90
overburdening the implementation institution, or to avoid bias in monitoring change within the insti-
tution. The alternative is to finance special studies through a project office independent of the imple-
mentation agency. No matter how special studies are carried out, there is a danger of them being
neglected during project start-up, as studies are of relatively high cost, management intensive, and
easily put off until data are needed for a Mid-Term Review or other critical project evaluation. The
major types of special studies used in ARE programs include:
a) Sample surveys. Formal sample surveys are useful to provide comprehensive and statistically valid
data on program outcomes and impacts. However, formal sample surveys also have disadvantages in
that many institutions lack capacity to conduct them, wide fluctuations in agricultural production
make it difficult to establish statistically significant trends in production data over short time peri-
ods, and survey data collection costs are high. Large-scale national surveys are probably best avoid-
ed, unless they are linked to a regular program of national agricultural statistics. More focused
regional or localized surveys may be more manageable, and are useful in characterizing agricultural
conditions and changes in a particular area.34 Grosh and Glewwe (2000) provide detailed advice on
how to design multiple-topic household surveys, that include defining the objectives of the survey,
identifying the data needed, and drafting questionnaires.35
b) Rapid rural appraisal. Rapid rural appraisals are a compromise between formal surveys (which
generate statistically valid, quantitative data but are costly and time consuming) and informal field
visits (which provide anecdotal information) (Kumar 1993). They are low-cost, quick, open-ended,
and flexible, but are not statistically valid and may be open to charges of bias. Five core methods of
rapid rural appraisal approaches are: key informant interviews, focus group interviews, community
interviews, structured direct observation, and informal surveys.
c) Special topic studies. Special topic studies are often useful to analyze technology program impacts
and develop an analytical base for routine M&E data collection. Ex ante economic analysis and eco-
nomic models of the impact of technological innovation are common types of special topic studies.
Others might focus on specific impacts on women, land tenure, food prices, living standards, or
nutrition.
d) Diagnostic case studies. Diagnostic case studies may provide a holistic, in-depth descriptive
assessment of a program or local situation. These may combine information from various sources
and can be useful to: (a) analyze potential social, technical, and financial factors affecting uptake of
new agricultural technologies; (b) assess potential and economic benefits from technological inno-
vation; (c) assess differential impacts of programs on different groups; and (d) identify key issues for
effective extension program operations. Agricultural Innovation Projects should frequently build rou-
tine field case studies into M&E systems. In fact, it may be difficult to envision an efficient technol-
ogy system without a functional system for such studies to assess how well a project is doing and
how well technologies are being accepted. A key challenge is that of relating "with-project" changes
to the counterfactual "without-project" scenario.
e) External peer reviews. Peer reviews are used in highly technical program areas, most commonly
in research. External peer reviews are useful to obtain technical experts' assessment of research
activities from three perspectives: (a) a prospective review (to assess implications and relevance of
research); (b) an in-process review (to assess efficiency of research and performance against targets
for outputs), and (c) a retrospective review (to assess scientific or technical quality of work complet-
90
3534-Rikka.pdf 6/20/05 2:51 PM Page 91
ed). They are also useful in developing linkages between scientists and institutions working in relat-
ed fields. As an example, an external review was conducted for the Agricultural Research and
Extension Project (Adaptable Program Loan) in Peru to evaluate the progress of the first project
phase, with the specific task of verifying if triggers for the specific phase had been reached (the trig-
gers or measures of project implementation progress must be met prior to any phase will be passed
for approval and execution of the following phase can begin).
91
3534-Rikka.pdf 6/20/05 2:51 PM Page 92
APPENDIX 5. PROJECT AND
MONITORING & EVALUATION COSTS
Project Costs: Bank project costs are usually prepared using Costab 32 a PC-based project costing
tool. Costab 32 is the project costing component of a software series to improve efficiency and effec-
tiveness of Project Processing activities. Costab 32, upgraded in 2001 by ADB, and is designed to
operate under Windows 2000 as well as earlier 32-bit versions of Windows. Costab 32 also incorpo-
rates several new features which include:
� Automatic calculation of financial charges during implementation;
� Summary table format for both ADB and World Bank;
� Option to compute economic costs in either domestic economic prices or border prices; and
� Other operational parameters.
Costab 32 helps task team leaders, project economists, and financial analysts organize and analyze
project costing data in the course of project preparation and appraisal calculates physical and price
contingencies, taxes and foreign exchange displays data in detailed cost tables coverts financial costs
to economic costs for use in economic analysis. Costs should be broken down by component and by
type of input. They should be expressed in local currency and foreign exchange, specifying the source
of financing and, where applicable, co-financing arrangements. See table A5.1 for an example.
It is important to ensure that all activities shown in a detail table are incorporated in the preparation
of M&E tables such as a standard Log Frame or the results framework arrangements matrix.
Table A5.1. A Project Component Table from Costab 32.
Unit Cost 05/06 06/07 07/08 08/09 09/10 Total
I. Investment Costs
A. Review of national acts,
policies and regulations
Review of existing
Land mgt. guidelines 329,787.234 134,553.2 104,951.5 109,149.5 � � 348,654.2
Awareness raising and IEC 329,787.234 134,553.2 104,951.5 109,149.5 � � 348,654.2
Field surveys and validation 329,787.234 168,191.5 174,919.1 � � � 343,110.6
SLM policy regulatory
capacity building 329,787.234 168,191.5 174,919.1 � � � 343,110.6
Subtotal review of national
acts, policies and regulations 605,489.4 559,741.3 218,299.1 � � 1,383,529.7
B. Ssatellite imagery of
3 Dzongkhags
Satellite image of Radhi 1,000,000 � 1,060,800.0 � � � 1,060,800.0
Satellite image of Nangkhor 1,000,000 510,000.0 530,400.0 � � � 1,040,400.0
Satellite image of Phuentsholing 1,000,000 510,000.0 530,400.0 � � � 1,040,400.0
Subtotal satellite imagery of
3 Dzongkhags 1,020,000.0 2,121,600.0 � � � 3,141,600.0
1,625,489.4 2,681,341.3 218,299.1 � � 4,525,129.7
Source: Ghazali Raheem, Consultant, World Bank.
92
3534-Rikka.pdf 6/20/05 2:51 PM Page 93
Below is a typical M&E budget plan for an IDA/GEF environmental project, including a breakdown
of key expense categories.
Marine and Coastal Environment Management Program
United Republic of Tanzania
Indicative M&E Budget
Unit Price PY1 PY2 PY3 PY4 PY5
Core Staffing and
Administration Quantity Cost Quantity Cost Quantity Cost Quantity Cost Quantity Cost
Staffing
M&E Coordinator PCU 1 $ 1,172 1 $ 1,172 1 $ 1,172 1 $ 1,172 1 $ 1,172
MIS Analyst PCU 1 $ 956 1 $ 956 1 $ 956 1 $ 956 1 $ 956
Statistician/Analyst PMU 1 $ 109 1 $ 109 1 $ 109 1 $ 109 1 $ 109
Office equipment
Computers $1,500 3 $ 4,500 0 $ -- 0 $ -- 0 $ -- 0 $ --
Printers $ 250 3 $ 50 0 $ -- 0 $ -- 0 $ -- 0 $ --
UPS $ 100 3 $ 300 0 $ -- 0 $ -- 0 $ -- 0 $ --
Furniture $1,000 2 $ 2,000 0 $ -- 0 $ -- 0 $ -- 0 $ --
MIS Applications
Applications $8,600 1 $ 8,600 1 $ 8,600 1 $ 8,600 1 $ 8,600 1 $ 8,600
Training
Intro to Computing
Computing
applications
Statistical analysis
Report preparation
Data base and MIS
Network principles
Beneficiary training $ 3,4001 $ 3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400
Field visits $1,500 1 $ 1,500 1 $ 1,500 1 $ 1,500 1 $ 1,500 1 $ 1,500
Workshops $3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400 1 $ 3,400
Assessments, Studies, and Surveys
Baseline surveys $4,800 1 $ 4,800 0 $ -- 0 $ -- 0 $ -- 0 $ --
Annual M&E
Review and
Reports $3,600 1 $ 3,600 0 $ -- 0 $ -- 0 $ -- 0 $ --
M&E workshop $2,200 1 $ 2,200 0 $ 2,200 0 $ 2,200 0 $ 2,200 0 $ 2,200
Safeguards report $4,500 1 $4,500 0 $ -- 0 $ -- 0 $ -- 0 $ --
Social and Environ
Impact
Assessments $5,500 1 $ 5,500 0 $ -- 0 $ -- 0 $ -- 0 $
continued
93
3534-Rikka.pdf 6/20/05 2:51 PM Page 94
Marine and Coastal Environment Management Program--continued
Unit Price PY1 PY2 PY3 PY4 PY5
Core Staffing and
Administration Quantity Cost Quantity Cost Quantity Cost Quantity Cost Quantity Cost
Mid-term review $1,200 1 $ 1,200 0 $ -- 0 $ -- 0 $ -- 0 $ --
Annual audit $3,200 1 $ 3,200 0 $ -- 0 $ -- 0 $ -- 0 $ --
Operations and
Maintenance
Supplies and
Maintenance $3,042 1 $ 3,042 1 $ 3,042 1 $ 3,042 1 $ 3,042 1 $ 3,042
Installation $4,000 1 $ 4,000 1 $ 4,000 1 $ 4,000 1 $ 4,000 1 $ 4,000
Total $58,729 $28,379 $28,379 $28,379 $28,379
Source: Ghazali Raheem, Consultant, World Bank.
94
3534-Rikka.pdf 6/20/05 2:51 PM Page 95
APPENDIX 6. ECONOMIC EVALUATION OF COMPETITIVE
RESEARCH GRANTS: THE CASE OF PROMSA, ECUADOR
OBJECTIVES, EXPECTED RESULTS,
AND MAIN CHARACTERISTICS OF PROMSA
From the time leading up to the implementation of a CGS in 1999 Ecuador's national research sys-
tem was dominated by a centralized public research institute, Instituto Nacional Autonomo de
Investigaciones Agropecuaria (INIAP). Because public agricultural research funds were dedicated
almost exclusively to supporting INIAP, financing was scarce for other potential suppliers of agri-
cultural research such as private and public universities, NGOs, and private firms. The number and
level of training of agricultural scientists in Ecuador was very low as well. In 1998 the total number
of agricultural scientists holding a Ph.D. degree was just 16, and only 71 held an M.Sc. INIAP link-
ages with domestic and international research entities were weak, limiting expertise and technology
spill-ins. In 1998, INIAP had only 6 national, and 7 international research agreements with total
funding of less than $200,000.
The original objective of World Bank-support for agricultural research in Ecuador was to strengthen
the national capacity to do agricultural research, with the aim of increasing agricultural productivi-
ty and output by (a) introducing a competitive research grant program; (b) developing research and
education exchange partnerships with international science institutions; and (c) strengthening the
national agricultural research institute, INIAP.
Ecuador's Competitive Grants System provides grants for individual research projects, and for
research and educational exchange partnerships with international institutions. The former grant type
was implemented as a mechanism for (a) increasing the efficiency of fund allocation for agricultur-
al research; (b) assuring the participation of scientists in agricultural research from diverse institu-
tions; and (c) strengthening linkages between public and private sector entities. The latter grant type
was implemented to finance the formation of research and education partnerships between research
agencies, universities, key NGOs, and international scientific institutions to facilitate the importa-
tion of applicable technology, strengthen the research capacity of Ecuadorian research institutions,
and strengthen graduate education programs in Ecuadorian universities.
By 2004 the National Modernization Program for Agricultural Services (PROMSA) had financed
108 individual research projects carried out by national research entities, and 14 international
alliances for research and educational programs. All programs follow the same competitive granting
procedure. Individual research projects supported by the fund were not to last more than three years
or cost more than $100,000, and were to have co-financing from other sources of at least 25%. The
maximum size of international alliances was set at $500,000. A Project Implementation Unit (PIU)
was created at project start-up and an international private firm was contracted to set up a monitor-
ing and evaluation system (M&E). In addition to monitoring the progress of funded projects, the PIU
is responsible for preparing annual budgets and work plans, making payments from the loan account
for project expenditures, and preparing financial and progress reports.
95
3534-Rikka.pdf 6/20/05 2:51 PM Page 96
Monitoring a competitive grant system
The management system that has been implemented in Ecuador is sophisticated and has contributed
substantially to the success of the competitive fund mechanism in Ecuador. The PIU prepares an
annual operating plan and quarterly reports on implementation of the proposed work plan. PIU field
officers visit each project at least twice each year to observe first-hand processes being employed in
administering the competitive fund. The computerized Monitoring and Evaluation system is state of
the art. The project database provides ready access to data on implementation of projects, and over-
all performance of the project portfolio. The database is relatively simple to maintain and use to gen-
erate information for reports.
Good practice: Adequate resources must be assigned to M&E activities. Resources should be ade-
quate to staff an M&E unit, establish a database, and provide consistent and timely feedback to
subproject scientists.
A system of objectives and milestones (for inputs, activities, outputs) with projected dates for mon-
itoring project implementation has been established and forms the basic elements for a log frame
approach. The M&E system is relatively simple to understand and manage. It is primarily based on
quarterly progress report by the project researchers. Data on all projects and milestones are record-
ed in the M&E database. PIU prepares an annual evaluation of subprojects based on (a) annual
reports by researchers, and (b) formal and informal reports by grupos de referencia (GR) (locally run
individual project advisory boards) for each project, (c) bi-annual visits by PIU specialists to each
project.
Good practice: Formal project milestones must be established and monitored.
A GR has been formed to oversee the implementation of each subproject. GRs are an innovative
approach to decentralized M&E in which beneficiaries and researchers collaborate. Each GR is com-
prised of 4�8 stakeholders (other researchers, extension staff, and agribusiness). The exact group com-
position and function are flexible, but ideally the GR is expected to participate in project design,
planning, and implementation. The groups meet several times a year to consult on project issues. The
PIU has also solicited feedback from project leaders through a survey of their opinions about the CGS
process. Funded researchers report that the PIU M&E reporting system is a substantial improvement
over the lengthy reporting often required by their own institutions or other funding agencies.
Good practice:Advisory groups should be organized. These groups can provide valuable feedback
into the M&E process.
Good practice: Reporting requirements should be designed to collect a minimum data set in an
efficient manner. The needs for M&E information must be balanced with a respect for the burden
that it places on busy scientists.
The PROMSA M&E system employs a system of "alert" signals or flags that indicate if a project has
become a "problem project." There are four ways a project can be flagged as a problem: (a) quarter-
ly or annual reports that are overdue by more than a month, (b) completion of less than 80 percent
of the quarterly milestones, (c) a rating by PIU field personnel of less than 2.5 on a 4-point scale, or
(d) general problems in quality of implementation. While in alert status, a project is not eligible for
new disbursements from the Fund. If the project does not clear its alert status by correcting deficien-
cies or rescheduling milestones within a specified time frame, the project can be terminated.
96
3534-Rikka.pdf 6/20/05 2:51 PM Page 97
Good practice: An alert system should be implemented to enforce communication between
researchers and CGS administrators. The criteria for triggering an alert should be clearly under-
stood and uniformly applied.
Evaluation of a competitive grant system
Most funded projects are not expected to have results diffuse to end users until five or more years after
the start of research. This makes it impossible to conduct a realistic economic evaluation of the CGS.
Therefore, evaluation must focus on success in meeting overall CGS objectives, in creating necessary
research institutional infrastructure, in measuring progress in implementing the proposed subproject
research, and in generating research outputs such as publications and technologies. This evaluation
relies on indicators that can be extracted from the PIU database (see the list below). There are a large
number of potential indicators of CGS success, some of which are more important than others. Not
all of those listed in the appendix are necessary for each evaluation. Indicators should allow evalua-
tion of the degree of success in institutional, administrative, and scientific aspects of the CGS.
Good practice: When the CGS funds research projects of longer gestation economic analysis will
not be meaningful until the project has been active for several years. During this period indicators
that allow evaluation of institutional, administrative and scientific progress should be monitored.
List of potential M&D indicators
1. Research Project Indicators (both at project type and portfolio level)
� Number of research project applications
� Number of research projects approved
� Number of research projects established
� Number of research projects by priority area
� Number of projects involving basic, strategic, applied, adaptive research
� Number of projects financed in small-scale agriculture
� Number of technical and scientific publications (differentiate according to quality such as
peer-reviewed articles, pamphlets, or manuals for farmers)
� Number of workshops
� Number of international alliances established
� Labor-capital ratio (e.g., person months--value of equipment)
� Recurrent costs or personnel costs relative to research budgets
� Cost per researcher
� Number of technologies transferred to farmers fields
� Rates of technology adoption by farmers
2. Education Project Indicators (both at project type and portfolio level)
� Number of short-term course, MSc. Program, Ph.D. program application
� Number of short-term courses, MSc. Programs, Ph.D. programs approved
97
3534-Rikka.pdf 6/20/05 2:51 PM Page 98
� Number of short-term courses, MSc. Programs, Ph.D. programs established
� Number of short-term courses, MSc. Programs, Ph.D. programs by priority area
� Number of MSc. degrees, Ph.D. degrees awarded
� Number of fellowships awarded
� Number of MSc. degrees, Ph.D. degrees awarded by international institutions
� Number of MSc. degrees, Ph.D. degrees awarded by national institutions
� Number of international alliances established
� Length of time required to complete degree programs or obtain certificates
� Cost of training programs per trainee
� Training program costs recovered from payments by users
� Employment rate of graduates
3. Overall Institutional Indicators (both at project type and portfolio level)
� Total number of project applications, projects approved, projects established
� Number of grant applications by INIAP, universities, private sector, others, joint projects
(define type of partnership)
� Number of grants awarded by INIAP, universities, private sector, others, joint projects
(define type of partnership)
� Number of projects established by INIAP, universities, private sector, others, joint projects
(define type of partnership)
� Amount of resources (%) captured from INIAP, universities, private sector, others, joint
projects (define distribution of partnerships)
� Ratio of project/counterpart funding
� Amount of resources from government of Ecuador
� Ratio of professional to support staff and trained technicians
� Resource allocation in relation to established priorities
� Competitive funding as share of total research funding
98
3534-Rikka.pdf 6/20/05 2:51 PM Page 99
NOTES
Chapter 1. Challenges for M&E Systems in Agricultural Research and Extension Projects
1. The Note is a follow-up to earlier Bank publications in M&E (Monitoring and Evaluation for
AKIS Projects by Alex and Byerlee, 2000, Ex-Ante Economic Analysis in AKIS Projects by
Horstkotte-Wesseler et al., 2000, Beneficiary Assessment for Agricultural Extension by Salmen,
2000, and Ten Steps to Results-based Monitoring and Evaluation by Kusek and Rist, 2004) by
accounting for recent changes in the design of World Bank-supported research and extension (R&E)
projects and the trend toward results-based M&E.
2. The driving forces behind change in ARE systems stem from deficiencies experienced in tradi-
tional systems: poor fiscal sustainability, difficulties with attribution of impact, low political com-
mitment, insufficient local content, weak accountability/ transparency, and weak linkages between
research and extension. Anderson, J. and G. Feder, 2004. Agricultural extension: good intentions and
hard realities. The World Bank Research Observer, vol. 19, no 1. and Rajalahti, R., J. Woelcke, and
E. Pehu, 2005. "Development of Research Systems to Support the Changing Agricultural Sector."
Proceedings of a Research Workshop, World Bank, Washington, DC.
Chapter 2. The World Bank Results Framework Requirements
3. World Bank, 1997. "Proxy indicators." The Log Frame Handbook: A logical framework approach
to project cycle management. World Bank, Washington, DC.
4. A World Bank Toolkit: Integrating Gender Dimension into Monitoring & Evaluation of Rural
Development Projects; and a Thematic Brief: Agricultural Knowledge and Information Systems;
Agricultural Research, Extension, and Education, 2001 provide further examples on gender-
disaggregated indicators.
5. Alternatively, the task team could present the component outcomes as combined outcomes at the
PDO level. This was common practice in the past. The advantage may be that it is in some cases eas-
ier just to define the combined effects of the components at the PDO level.
Chapter 3. Data Collection, Reporting and Dissemination Requirements
6. Baseline and targets based on weighted average of farmer income in the different sub-projects
from information provided in the proposals.
7. Association deemed successful if over 60% of the farmers are satisfied with the services provid-
ed by the Association.
8. Guidelines for Management Information Systems in Social Funds.
9. Relevant Bank publications in this context include: Murphy and Marchant (1988) discussion of
issues in planning M&E systems for extension agencies; Casely and Kumar (1987) discussion of
issues for agricultural sector programs in general; World Bank (2003b)--the most recent Bank pub-
lication on M&E. in agricultural operations; and Kusek and Rist (2004)--the most recent discussion
of M&E system for development programs in general--with particular focus on results-based M&E
systems.
99
3534-Rikka.pdf 6/20/05 2:51 PM Page 100
10. The thematic areas may include variety development; soil and water conservation; integrated pest
management; agro-forestry; marketing; and livestock production in different agro-ecological zones.
11. Poor sampling or inadequate mix of information may introduce significant bias which might
result in a disconnect between the objectives and true outcomes of the project.
12. Examples include the one by the World Bank (http://www.worldbank.org/data), FAO
(http://www.fao.org/waicent/portal/statistics_en.asp) and IFPRI ASTI at http://www.asti.cgiar.org/.
13. This section written by Ghazali Raheem, Consultant, World Bank.
Chapter 6. Economic Evaluation of Competitive Grants Schemes
14. It is usual in the empirical literature on project evaluation to classify projects' impacts, either
potential or realized, in three main categories: economic, institutional, and environmental. This chap-
ter deals with impacts within the first category--those that can be measured in monetary terms.
15. Details of practical procedures of incremental cost benefits analysis can be found in: CIMMYT
(1988). From Agronomic Data to Farmer Recommendations: An Economics Training Manual.
Revised Edition. CIMMYT: Mexico.
16. CIMMYT (1998) provides an excellent discussion about how to estimate the incremental net
benefit between technological alternatives for annual crops. Extension to more than one period for
perennial crops or technologies with longer beneficial effects is straightforward.
17. A partial budget compares estimated revenues and costs for a typical producer with and without
the new technology.
18. It is recommended to transfer this simple table into EXCEL which would allow for a convenient
calculation of NPV and IRR.
19. Methods for estimating the diffusion and the estimation of incremental net benefits have been
explained in section 6.4.
20. For more details see section 6.1: economic impact evaluation of a CGS; and section 6.5: estima-
tion of IRR.
21. Expert Opinion is often used in conjunction with simulation analysis. In this case the experts are
asked for their estimation of three values of the parameters of interest: its most likely value, the max-
imum, and the minimum.
22. The specific assignment of factors to the different categories of investment costs clearly depends
on the nature of the project and no general recommendation can be made. However, as an example,
the following factors have been assigned to the different categories of investment costs for the
Modernizing Agricultural Knowledge and Information Systems Project (Romania): civil works-
factor: 0.1; equipment-factor: 1; vehicles-factor: 1; technical assistance-factor: 0; training-factor: 0;
competitive grant: 1.
23. A commonly used indicator for the level of public financial support of agricultural research is
public expenditure for agricultural research as percentage of the agricultural GDP (AgGDP). Figures
100
3534-Rikka.pdf 6/20/05 2:51 PM Page 101
for public research expenditures can be obtained from the Ministry of Agriculture. Another recom-
mendable source is the Agricultural Science and Technology Indicators (ASTI) provided by the
CGIAR (http://www.asti.cgiar.org).
Appendix 1. Narratives of New Approaches and Instruments in ARE Systems
24. Anderson, J. and Feder, G. 2004. "Agricultural extension: Good intentions and hard realities."
The World Bank Research Observer, vol. 19, no 1. and Rajalahti, R., J. Woelcke, and E. Pehu 2005.
Development of Research Systems to Support the Changing Agricultural Sector. Proceedings of a
Research Workshop, Washington, DC: World Bank.
25. Farmer-to-farmer methodology is one type of participatory research and extension methodolo-
gy. Here, it includes Farmer Field Schools, CIAL, use of volunteer contact farmers, and producer
organizations.
26. Mass media and new ICT applications can be integrated into the strategic extension programs.
Media applications include both "new and old" communication devices and methods, such as
posters, booklets, radio (and enter-edutainment), television, drama, internet, and mobile phones.
27. World Bank (2003a). "Extension and rural development--Converging views on institutional
approaches?" Workshop Summary. Agriculture and Rural Development Department, World Bank,
Washington DC.
28. World Bank (2004b). Investments in Agricultural Science and Technology. Agriculture Investment
Sourcebook. Module 2. World Bank, Washington, DC.
29. Veldhuizen, L., A. Waters-Bayer, and H. Zeeuw 1997. Developing technology with farmers. A
trainers guide for participatory learning. London: Zed Books Ltd.
30. Braun, A., G. Thiele, and M. Fernandez 2000. "Farmer Field Schools and Local Agricultural
Research Committees: Complementary platforms for integrated decision-making in sustainable agri-
culture." AgREN Network Paper No. 105.
Appendix 2. Implementation of a Monitoring and Evaluation Plan in the Bank's Project
Cycle: Putting It All Together
31. This chapter written by Remileku Rakey, Consultant, World Bank.
Appendix 4. List of Indicators and Data Sources for ARE Projects
32. The following section is taken mainly from Alex and Byerlee (2000).
33. Examples include (e.g., the one by the World Bank http://www.worldbank.org/data) and FAO
(http://www.fao.org/waicent/portal/statistics_en.asp).
34. Important data sources include the CGIAR centers which regularly conduct relevant surveys in
developing countries (e.g., IFPRI in Uganda).
101
3534-Rikka.pdf 6/20/05 2:51 PM Page 102
35. Grosh, M. and P. Glewwe (2000). Designing household survey questionnaires for developing
countries: lessons from 15 years of the Living Standards Measurement Study Volumes 1�3. World
Bank, Washington, DC. On agriculture, volume 2 lays out the most important agricultural policy
issues in developing countries, discusses the date needs and measurement concerns, and introduces
three versions of a draft agriculture module. Volume 3 presents draft questionnaire modules on sev-
eral topics. On agriculture, the issues covered include land holdings, farm capital inventory, disposi-
tion of crops, input purchases and sources, access and use of agricultural extension services, farm
capital inventory and transactions, crop output, household member labor inputs, hired labor and non-
labor inputs, input prices, expenditures and sources, and livestock.
102