Supporting the Implementation of Public Investment Management Reform in Serbia Monitoring Public Investment Projects & Setting Financial and Social Discount Rates Ministry of Finance workshop Date: 15-16 December 2020 Objectives of the Workshop 1 To inform and educate on the nature, the purpose and the means of conducting the monitoring of public investment projects at a project level and a portfolio level 2 To inspire a positive desire to develop a ‘good global practice’ monitoring regime in Serbia To provide the knowledge that Serbian officials will need to develop and later operate an effective 3 monitoring system To provide the knowledge and techniques for identifying - through monitoring data - and managing 4 projects facing difficulties in implementation 5 To familiarize PIMOCU staff with the purpose, objectives and available methods for calculating FDR and SDR; and 6 To guide PIMOCU staff through the appropriate method and calculate FDR and SDR for Serbia. Expected Results / Outcome Knowledge disseminated Short Term Capacity and capability developed FDR and SDR for FY of 2021 published timely Serbian PIM monitoring system developed Longer Term Serbian PIM monitoring system effective 2 Capacity to calculate FDR and SDR established Agenda Day One: Tuesday 15th December 2020: 09.00 – 13.00 Session 1 Monitoring of public investment– what it is and 09.00-10.45 why it is important Design of the monitoring system Break 10.45 – 11.00 Sessions 2-3 Case Study and Demonstration 11.00-13.00 Using monitoring data to review and rationalize projects Day Two: Wednesday 16th December 2020: 09.00 – 15.00 Session 4.1 Quiz 09.00-10.30 Practical Work: How to identify and review projects in difficulties 3 Agenda (cont’d) Break 10.30 – 10.45 Session 4.2 Practical Work: How to identify and review projects in difficulties (continued) 10.45-11.30 Session 5 Methodological steps on calculating financial discount rate (FDR) 11.30-12.15 Lunch Break 12.15 – 13.00 Sessions 6-7 Methodological steps on calculating social 13.00-15.00 discount rate (SDR) Reference periods for project appraisals Q&A 4 Session 1 Monitoring of public investment– what it is and why it is important Different types of monitoring in the Public Investment Management sphere (project, portfolio, strategy and internal policies / KPIs) Monitoring and evaluation (M&E): are they the same thing? Benefits of monitoring, including using monitoring to improve other aspects of the PIM system. Monitoring can often be viewed as a negative practice (to seek out poor performance and punish) whereas the positive aspects are often overlooked Design of the monitoring system What aspects should be monitored; and why? What data is required? How frequently? Who should do the monitoring? Design of data collection and reporting templates to facilitate analysis IT systems for effective monitoring International examples Design issues in the Serbian context? 5 Monitoring and Evaluation (M&E) A commonly used single phrase but not a single subject • Monitoring is a continuous process to ensure a plan is achieved in terms of: – financial - expenditure and (where applicable) income – non financial - realisation of activities, outputs, results • Evaluation is an analysis of the completed project to check that resources have been employed efficiently and effectively and with relevance to the objectives of the program or project and to ensure that lessons are learned through the experience of the project or program • Why do we do Budget Monitoring of Projects? It ensures that resources are used for their planned purposes and are properly accounted for to internal or to external bodies. But there has to be a purpose: No point monitoring and collecting data if you don’t use it or ignore warning signs 6 M&E means different things in different contexts Level Application Monitoring Evaluation - National Development Plan Financial -To what extent did the program - Sector Plans -Are the resources deployed so far to complete and fulfill its aims? - Supra-National Agreements (eg meet the program aims sufficient? WBIF, Environmental -Are the future resources needed to -Were the resources deployed an Strategic commitments) fully achieve the program aims going effective and efficient use of public to be sufficient? money? (Program level) (All of these are ‘Programs’) Non-Financial Will the objectives of the program be achieved on target? Financial Did the project achieve its The Implementation of Individual - Current costs versus approved objective(s)? Projects estimates - Absorption rates Was the project delivered efficiently? Operational Non-Financial (Project level) - Physical completion rates What lessons can be learned for - Is the project on schedule? future projects? - Is the project in good shape? - Any interventions required? - Internal Capacity What could have been done better? - Internal Policies - Performance of Project Managers Internal - Internal Practices and other staff What changes are needed to - Internal Compliance - Contract performance improve future projects? 7 What Should be Monitored? What Should be Evaluated? Program Program Objectives Outcome Evaluation Project Project Objectives Outcome Performance Operations / Monitoring Service Delivery Inputs Activities Outputs Project Monitoring 8 Operational and Internal Monitoring within the overall structure National Development Plan 9 M&E Does Not Exist in Isolation ….nor does it exist for its own sake - It is only effective when it is part of a fully functioning Program / Project system – a Public Investment Management (PIM) framework They are mutually dependent on each other • Without M&E the PIM framework cannot be fully efficient and effective • Without a solid PIM framework, the M&E cannot be fully efficient and effective – ‘garbage in = garbage out’ When one or the other is missing = Weaker than expected results 10 Monitoring sits at the hub of the Project Cycle – it affects everything directly or indirectly Program Planning Ex-post Project Evaluation Planning Monitoring Project Operation Prep Selection and Implementation Budgeting 11 Effective monitoring can identify problems… Specific weaknesses ex-ante -Poor or inadequate preparation -Lack of quality checking -Incorrect assumptions resulting from poor or lack of information- Procurement Issues: -Bids are above expectation -Poor contractual fulfilment Corrupt Practices / Fraud Under-execution - Low absorption rates - Delays in physical progress Weak Budgeting: Quality issues -Inadequate allocations -Incorrect allocations 12 Benefits of a Functioning M&E Framework Identifies problems early as they are developing Provides reliable indicators for …Allows corrective action Early warning for when projects and programs need to be taken early to potential fraud to be reviewed. Some countries prevent small problems have trigger points becoming big problems Good and bad practices can be Smarter Budgeting: Identifies under- highlighted for the benefit of spending that might be vired to other future program and project better performing projects / programs design therefore maximising execution rates Develops costing data by which future Clear data on on-going commitments programs and projects can be better helps better understanding of fiscal estimated (how much does it cost to space for new projects provide one school place etc)…. Information feeds back to quality Overall provides a means to checkers in the ex-ante stages measure progress in the development of the country 13 When does Monitoring Begin? …. and end? From the government perspective? From the MDA perspective? From the Project Manager perspective? 14 Group Task Mapping Responsibilities to Current Serbian Institutions We will discuss who does what in the Serbian system and then complete the matrix as shown below ? ? ? ? ? ? ? ? ? ? ? ? 15 Institutional Responsibility for Monitoring in Serbia? A Discussion 16 Institutional Tasks • Performing public investment • Process coordination portfolio monitoring and review • Formulation of resolution • Suggesting project resolution Capital strategy strategy for non-performing Investment • Implementation monitoring of projects Commission project resolution activities • Acting as back-office to CIC Ministry of Finance - Implementing Agencies PIMOCU Initial review of implementation • Project implementation • MDAs plan and progress reports • Data collection (Line Ministries – • Preparation and submission of • Timely submission of these Authorized project implementation plan and documents to MOF Proposers) • Elaborating and coordinating project progress reports implementation of project • Implementation of project resolution plan resolution plan 17 Institutional Responsibilities for Monitoring in Serbia? - Responses What data is Who gathers Who receives Who reports Type of currently and sends the the data? the data? Monitoring monitored? data? This is not clear to the This is not clear to the This is not clear to the This is not clear to the participants participants participants participants Strategic According to the Implementing entities PIM Unit at MoF PIM Unit Operational Rulebook This is not clear to the Implementing entities Sponsoring (owning) This is not clear to the participants Internal entities participants 18 Aids for Monitoring Projects: 1 Project Implementation Plan Project Implementation Plan 1. General information about the project sponsors (i.e. line ministry/DBB and Implementing Agency/Investor) 2. Project name, project code and sector code 3. Implementation start date and estimated end date with intermediate key ‘milestones’ Project Implementation 4. Approved project value Plan 5. Source of financing and cash flow requirements 6. Breakdown of project costs by implementation activity including contingency reserve 7. Procurement plan for all necessary resources 8. Quarterly implementation schedule for project activities 9. Quarterly financial implementation schedule 10. Risk Register 19 Aids for Monitoring Projects: 2 Project Progress Reports Project Progress Reports 1. General information about the project sponsors (i.e. line ministry/DBB and Implementing Agency/Investor) 2. Basic project data 3. Verification (or update) of key implementation inputs, including estimated total project cost and estimated completion date Project 4. Review of activities undertaken during the reporting Progress Reports period including those related to physical implementation (referring to the milestones defined in the implementation plan) and financial implementation (i.e. activities with planned versus realized expenses with reference to the planned laid out in project implementation plan) 5. Disbursement / absorption rate in reporting period and total 6. Physical completion (total) 7. Update to risk register 20 Typical Indicators used for Monitoring • Time against plan • Cost against plan – Absorption Rate – Likelihood of Full absorption • Physical progress against plan • Granular level too • Risk Register 21 Typical Monitoring Data The most common monitoring data and where they can be found Data Source Project starting date Project Implementation Plan Original project cost estimate Project Implementation Plan New project cost estimate Project Progress Report Sum of project appropriations Budget Management Information System Disbursements in each reporting period Project Progress Report Original estimated project completion date Project Implementation Plan Revised estimate of project completion date Project Progress Report Estimate of physical completion, current period (in %) Project Progress Report Estimate of physical completion, total (in %) Project Progress Report Risk Register Project Implementation Plan 22 Reporting – Project Level • Purpose / benefits – Essential management information for project managers – Accountability to project board / minister – Rapid visibility on what’s going well / not so well – Highlights need for any corrective actions – Data demonstrates future cash-flow requirements – Important tool to manage the contract / contractor(s) – Internal control • Frequency – Continuous if possible (‘real-time’) – Usually monthly or quarterly – anything longer than 3 months is poor practice • Coverage – Financial – Non-financial (time/ schedule / milestones / quality – Regulatory compliance checks – Project Risk Register • General Rule: ‘Stay with the key messages’: Examples: ‘Project on Schedule / Project has spent only 5% of its allocation in the first month of the FY / Forecast spend by year end = 75% / Reason for variance is …… / Variation in risk profile since previous report……’ 23 Public Investment Portfolio Reporting • Public Investment Portfolio Brief (quarterly) i) Overall portfolio information (value, sector breakdown, funding, disbursement levels) ii) Historical performance per financial and delay-related indicators iii) Current performance developments (for the last quarter) iv) Other important information (e.g. legislative changes, funding opportunities, policy briefs, etc.) v) Specific adverse implementation risks (or systemic risks emerging) • Public Investment Portfolio Report (annual) The elements are the same as for the Brief, except that it should refer to the entire year and that there should be a specific chapter after the portfolio performance assessment (i.e. current developments) which would discuss application of the portfolio rationalization mechanism (if required) and follow-up actions of the CIC and MoF. 24 Systems to Support Operational Monitoring • Governments are increasingly developing project monitoring platforms often known as PIM Information Systems (PIMIS) that allow tracking and monitoring of projects through all phases of the PIM cycle • A unique project code identifies the project through all phases • Developed often with the assistance of development partners • Objective is to track projects from ‘cradle to grave’ and compare the plan with results (evaluation) • Can permit monitoring of individual projects during implementation and aggregation at a portfolio level to provide a capital baseline for budget planning • Algorithms can automatically ‘red-flag’ distressed projects based on nationally agreed criteria 25 Typical PIM Information System (PIMIS) 26 Common Challenges in Developing Systems • Over ambition of the government / over ambition of the promoting development partner • Under-estimating the time it takes to develop • A desire for the system to do everything from day one – a monitoring module could be the most useful to start a new system • Failure to start simple and build in phases • Failure to prototype system logic with easy to adjust open source programs such as MS Excel • Reliance on proprietary systems that have large licence fees and charge exorbitantly to adjust / maintain • Easy to under-estimate the interfaces with existing systems • Poor stakeholder management – other institutions of government that ‘own’ systems that need to be interfaced are usually reluctant to allow others into their systems for no benefit to themselves • Failure to properly roll-out data input training to all stakeholders in all MDAs (garbage in = garbage out) 27 How Good is Monitoring in other Countries? 28 Source: IMF’s PIMA Assessments: average of 51 countries Parameters for Monitoring - Canada • Comparison of actual costs to budget estimates • Progress of implementation against schedule • Regulatory and Quality checks Reporting: • Details of scope changes and related costs (within monitoring period and total) • Explanations for variances above the established thresholds for the project • Reason(s) for delays; and remedial action already taken 29 A Different Model for Monitoring - Saudi Arabia • A top down approach • Independent body formed in 2016 – ‘ADAA’ • Reports to the Prime Minister • Only task is M&E – most effort so far is focussed on monitoring • They are responsible for collecting contract performance data and developing tools to support the task • Web-based platform viewable by the public but real- time inputting not yet possible 30 Online Monitoring 1 - Colombia 31 Online Monitoring 2 - Colombia 32 Common Challenges to Monitoring in Other Countries Lack of Resource Lack of understanding of the importance or the benefits = low political motivation In MDAs it is rarely considered important (often viewed as ‘bureaucracy’) It enjoys brief moments of prominence and is then easily forgotten Reliability and accuracy of data 33 Session 2 Country Case Study: Uganda - monitoring with early results Case Study – PIM Monitoring in Uganda Previous Challenges Included: Very little information on projects under implementation Projects not receiving required budget funding No capital budget baseline Projects in distress not monitored – inadequate data to diagnose problems 35 The motivation for action: a need to rationalize the portfolio and reprioritize capital expenditure due to the current global crisis Questions: Do we know our starting point? How is this possible without the necessary data? How do we avoid cutting the good projects by If we want to stop mistake? projects in distress, how do we know which ones they are? 36 Actions Taken A spreadsheet (MS Excel) model of all projects under implementation was developed Monitoring Criteria and a Scoring System were agreed Projects were entered into the system at a sector / MDA level and the spreadsheet was able to aggregate data to a portfolio level Preliminary Results Less than 2 months to develop The model requires monitoring data to be collected systematically for the first time There was a high level of compliance / co-operation It highlights projects in distress The model was used to make recommendations on which projects to continue, review or suspend It shows the fiscal impacts of different decisions regarding public investment projects 37 Demonstration of the tool used 38 Session 3 Using monitoring data to review and rationalize projects Monitoring and Evaluation (M&E) - What do the Rulebooks say? (THE MONITORING RULEBOOK) • Based on the information from Implementation (5) and Progress Reports (6), each quarter within 30 days from reception of the Progress Report, PIMOCU calculates three main PERFORMANCE INDICATORS: 1. Time-to-complete (TTC) – showing how much the project deviates from the planned completion period having in mind its implementation profile. Threshold – 25% deviation from the planned time of completion. 2. Adequacy of Funding (PAF) - reveals the quality of cost estimates associated with project implementation submitted by project sponsors. Threshold – if the projects spends less than 10% of annual appropriation two quarters in a row, while the remaining appropriation is more than 50%. 3. Dormancy – identifies projects which are chronically underfunded or under- disbursing serving as an early warning signal for delays in project completion. Threshold – if the dormant status is not envisaged by the plan. 40 Project Level Performance Indicators Name and Description Data needed Threshold values Overall Adequacy of Funding (OAF) -original project cost estimate The lower the better. Levels around 10-15 Quality of the cost estimates associated percent are considered tolerable (depending with project implementation on type of project) -new project cost estimate In-year Adequacy of Funding (IAF) The share of appropriation spent. Shows -sum of last year's project The lower the better. Again, type of project adequacy of funding and annual financial appropriations would define in-year procurement plan which planning needs to be refered to. -disbursements in each reporting period Time-to-complete (TTC) -latest reporting cut-off date Shows how much time is required to -project start date Depends on the originally planned project life. complete the project at current rates -project cost estimate Values of more than 30 percent call for -disbursements in each attention. reporting period Dormancy -latest reporting cut-off date Project is considered “dormant� if its -project start date Dormancy is indicated by aggregate disbursement level is much below levels -estimated project completion disbursement rates of 10%, 20% or 30% given expected by its duration date that project has been active for at least 25%, -project cost estimate 50% or 70% of planned implementation -disbursements in each period. reporting period 41 Monitoring and Evaluation (M&E) What do the Rulebooks say? (THE RATIONALIZATION RULEBOOK) PIMOCU information Project explanation Rationalization Assessment Sponsor procedure Capital Rationalization Rationalization Investment Plan List Committee Review 42 Most common data issues • Overall - project start date - project end date - anything else? • Implementation Plan (i.e. obrazac 5) - point 4.1. whether financial or physical implementation? - point 4.2. whether financial commitment or actual payment? • Progress Report (i.e. obrazac 6) - physical plan cannot be followed since it is indicated by “X� - how to introduce financing source – sensitive monitoring? 43 Discussion on design of reports • A discussion will be facilitated on quality issues in reporting. • Key question: how can reporting better support monitoring, review and rationalization as defined in rulebook? • The discussion is based on the feedback from MoF on the issues with structure and content of implementation reports (Obrazac 5) and progress reports (Obrazac 6). • Examples of reports with quality issues will be referenced as needed (link to examples can be accessed here) • The discussion will be geared toward finding a practical solution for each of the identified issues. Problem identification will be in plenary while solutions will be discussed in smaller groups. • Necessary changes to the reporting Rulebook and forms of reports will be compiled. 44 End of Day One Thank you for your attention and see you tomorrow! 45 Day 2 Session 4 How to identify and review projects in difficulties …but first… Monitoring or Evaluation? Monitoring? Evaluation? I. Checking the overall performance of a project II. Checking whether a project is on schedule III. Checking whether a project has been delivered efficiently IV. Checking to see if the project has fulfilled its objectives V. Checking that the allocated budget has been executed A benefit of Monitoring? Yes or No? Yes No I. Better project costing information II. The project manager does not have to worry about anything III. You will be able to identify bad project ideas IV. You will be able to identify problems during implementation V. Helps with better budgeting 47 True or False? True False I. Monitoring is at the center of any good PIM system II. A good monitoring system reduces the need for evaluation III. Monitoring parameters are decided by the construction company IV. Monitoring can only work at the level of individual projects because different MDAs have different systems V. Monitoring can be automated Only to aggregate and ‘red flag’ trigger values VI. KPIs should be supplemented by field data and interviews with users Only when necessary such as ‘red flag’ situations VII. Rationalization should be triggered automatically 48 What does monitoring data tell us about projects under stress? • A health check on the project (good and bad) • Therefore an early warning system for project stress • Indicators of project stress include: • Delays to start date • Slower than expected progress after the start date • Stop / start activity on the site • Physical progress is falling behind plan on a continuous basis • Inadequate budget allocations • Very low absorption rate in the first year • Lower than expected absorption rate overall 49 Diagnosis: Important to see the full picture I am exhausted Effects I cannot sleep Manage and Rectify the Causes Core Problem My tooth hurts not the Effects I have a cavity (decay) Cause I eat too much chocolate 50 Steps to Remedy Projects under Stress • Diagnose the nature and root cause of the problem • Verify data and consult stakeholders • Examine possible resolution options with consequences for each one – Legal (specifically existing contracts) – Financial – Economic – Environmental – Social • Choose the most appropriate option and consult again with all stakeholders • Implement 51 Let’s look at some hypothetical project case studies 52 Performance Assessment and Rationalization: Instructions for Case studies 1-3 For each of the following three case study the training participants are invited to: 1. observe and analyze the values of the three relevant KPIs within the overall project context, 2. consider the need and sources of supplementary information 3. develop a hypothetical project rationalization plan 53 Performance Assessment and Rationalization CASE STUDY 1: Reconstruction of an existing railway section which is part of the Traffic Master Plan for the current period. The project is implemented by the MoCTI. The project was received well by the local communities along the section, but it brings a range of environmental issues since Government’s decision was to install diesel-run trains to overcome the issue of weak electricity supply which would cost a lot to overcome. For that reason, the project couldn’t get financing from IFIs and instead secured funding from budgetary sources. Total value of the project is EUR 50 million. The reconstruction started 2.5 years ago. We are now in the third year of implementation and have to decide how to proceed with the project given that the budget preparation process for the next year has begun and needs to be adjusted for the project performance assessment. Information from the Implementation Plan and quarterly progress reports are given here. 54 Performance Assessment and Rationalization CASE STUDY 2: Customs Administration is running a large scale equipment purchase which is worth EUR 75 million and is scheduled to take place in phases over the period of 3 years. The need is well established and comes from the necessity to adjust with the EU standards. The project is thus co- financed by the EU with 50%. The project started 2 years ago and is severely underperforming. Budget request for the next year just came in and it is an aggregate figure that exceeds all previous appropriations assuming project will be able to catch up and align with the original schedule. Information from the Implementation Plan and quarterly progress reports are given here. 55 Performance Assessment and Rationalization CASE STUDY 3: GoS is financing a large scale (i.e. EUR 200 million) national high- education center financed from a mix of national and international financial sources. A very high commitment fee of 0.75% is imposed on the EUR 150 million financed externally. The project is well received and a long awaited one. Its completion is supposed to bring all technical faculties in Belgrade under one roof and provide them with state-of-the- art working environment. The project started 2 years ago. After 6 months of very successful implementation, a dispute with the construction company stopped the works completely. The negotiations between the Ministry of Education and the company are very intensive and receive a lot of media attention. In the meantime the project accumulated over EUR 2 million in commitment fees paid to the banks. Information from the Implementation Plan and quarterly progress reports are given here. 56 57 Performance Assessment and Rationalization: Instructions for Case Studies 4-6 For each of the following three case studies the training participants are invited to consider the following questions: • What scenario is the monitoring information presenting? • What type of possible conditions or issues can be considered to have caused such a scenario? • What would be the first course of action when presented with this scenario? Also please consider the following questions: • Is the information presented sufficient to guide further actions? • If not, what additional information would be required to guide a response? • Would the necessary additional information be easy to access and would it be reliable and accurate? • What courses of action would be open to respond to each set of circumstances? • Who would be responsible for undertaking the review and what stakeholders would be involved? 58 Project Stress: Additional indicators that could be considered Stress Indicator Possible Causes Potential Remedies 1. Delays to start date 2. Slower than expected progress after start date 3. Stop / start activity 4. Physical progress continuously falling behind plan 5. Inadequate budget allocations 6. Very low absorption rate in the first year 7. Lower than expected absorption rate 59 overall Performance Assessment and Rationalization CASE STUDY 4: A new road was first planned 15 years ago and approved 8 years ago. It was first allocated budget funding three fiscal years ago. Since then, the project has received its full budget request allocation every fiscal year. Budget allocations over the past three years have exceeded 90% of the initial project estimate already but absorption rates have been poor: in the 1st year: 22% absorption; 2nd year: 52%; 3rd year: 65%. The original time estimate for building the road was 4 years and yet the road is still physically less than half complete. According to the implementing authority, current estimates put the road 20% over budget at completion, and behind schedule by 15 months. The implementing authority has informed the budget department that it will request an increase in its planned allocation for next year in order to speed up the implementation. 60 Performance Assessment and Rationalization CASE STUDY 5: A new cultural venue is under construction. The original cost estimate has already been exceeded by 15% and delays to the completion of the project have meant that a major cultural event cannot be staged next year as planned. Even though the building structure is physically complete, it is not yet fitted out technically to make it suitable for receiving an audience and performers. The implementing authority estimates that the completed cost of the project will be 20% above the original estimate and will be around 10 months late in completion. 61 Performance Assessment and Rationalization CASE STUDY 6: Three new primary healthcare clinics are being built in a regional center under a single contract. The clinics were meant to be built one after the other. The first one is already completed and operational and the second one has been operational for 4 months. The first clinic was completed at a cost which was 10% over the original estimate; the second one was 6% over the estimate. The third clinic is in the early stages of construction with only the groundwork being completed to date. The numbers of patients at the first two clinics are approximately 30% below expectations and health officials are concerned about the number of visits to the 3rd clinic being even lower due to a smaller local population than the other two locations. 62 Много хвала 63