ICRR 13244 Report Number : ICRR13244 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 02/24/2010 PROJ ID : P035672 Appraisal Actual Project Name : National Transport US$M ): Project Costs (US$M): 91.13 120.60 Program Support Project Country : Chad Loan/ US$M): Loan /Credit (US$M): 67.00 69.57 Sector Board : TR Cofinancing (US$M): US$M ): 0 0 Sector (s): Roads and highways (89%) Central government administration (10%) Health (1%) Theme (s): Rural services and infrastructure (40% - P) HIV/AIDS (20% - S) Rural markets (20% - S) Nutrition and food security (20% - S) L/C Number : C3426 Board Approval Date : 10/26/2000 Partners involved : Closing Date : 08/02/2006 01/31/2008 Evaluator : Panel Reviewer : Group Manager : Group : Hernan Levy Fernando Manibog IEGSE ICR Reviews IEGSE 2. Project Objectives and Components: a. Objectives: According to the Credit Agreement, the objective of the Project is to support the implementation of Chad's National Program, which aims at reducing poverty and rural isolation and improving national food security through improved year-round access to markets and services especially in the rural areas . The ICR and the PAD use the same formulation, a minor difference in the PAD being that it is more explicit about services, as it notes that both social and administrative are covered. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): The project comprised six components, with about 90 percent going for the first two components, road investment and road maintenance. Actual costs below are shown net of taxes, to make them comparable to appraisal estimates . Road Investment (Appraisal cost US$60.90 million, actual cost US$68.06 million). This component intended to rehabilitate or upgrade 390 kilometers of key links in the National Road Network (NRN). It comprised two subcomponents: (i) road upgrading and paving of the 104-kilometer Ngoura-Bokoro road and (ii) rehabilitation of about 286 kilometers of existing paved road to extend their useful life . Pilot Program for Performance-based Management and Maintenance of unpaved roads (Appraisal cost US$19.57 million, actual cost US$22.96 million). This component was intended to test a new performance -based contracting method on unpaved roads over a four -year period. The pilot contract covered the initial upgrading and maintenance of 441 kilometers of unpaved main road along the main East -West road axis between N’Djamena and Abeche. Road Safety (Appraisal cost US$2.04 million, actual US$0.33 million. This component intended to promote road safety in Chad through a combination of the establishment of a traffic database, road safety awareness campaigns, treatment of blackspots (road sections with a high concentration of traffic accidents ) and carrying out a national road safety study and action plan This component was severely reduced in scope at the Mid -Term Review. Rural Transport Strategy Development (Appraisal cost US$1.24 million, actual US$0.98 million). This component aimed at the development of a rural transport strategy also comprised a series of elements, including studies, workshops, study tours, publicity to share experience and disseminate information and establishing a Rural Transport and Travel Program (RTTP) secretariat in Chad. Institutional Support (Appraisal cost US$6.27 million, actual US$5.99 million). This component intended to strengthen capacity of the Ministry of Public Works, Transport, Housing and Urban Development (MTPHU) and other stakeholders in the transport sector HIV/AIDS and Transport (Appraisal cost US$0.61 million, actual US$0.57 million). This component aimed to contribute to reducing the risk of increased HIV /AIDS infections along major domestic and international roads in Chad, through the promotion of AIDS awareness and education campaigns d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Cost: The actual cost in Section 1 above is shown with taxes, since the tax -free exemption existing at appraisal was reversed during implementation (see Financing below). Total project cost without taxes was $ 98.90 million. The strong depreciation of the dollar over the project period increased the unit cost (measured in USD) of the components, especially of the road investment and the pilot maintenance contract . The cost of rehabilitation of paved roads was further increased due to a change in the technical solution to provide a more robust design, especially after important road sections that were in good condition at appraisal suffered catastrophic pavement failures soon thereafter due to heavy traffic by overloaded trucks used for the construction of another road further south . Financing. An amendment early during project execution in 2001 increased the disbursement percentage in three categories (civil works, goods and consulting services ) to compensate for changes in the Chadian tax legislation . These changes made it mandatory that all contracts, including donor -financed contracts that originally were tax exempt, include all taxes (including a 18 percent value added tax) and customs duties, if applicable . Borrower Contribution. The cost increase of the pavement rehabilitation subcomponent made it impossible to fund upgrading of the road Ngoura-Bokoro from the IDA Credit. The government agreed to fund this work with its own funds, and it did, increasing the Borrower contribution from US$ 24 million at appraisal to US$51 million by project closing. Dates. The closing date of the project was extended in October 2005 by 18 months to complete the large works contract for the rehabilitation of Guelengdeng -N’Djamena-Dandi road. This work was delayed due to cancellation of the original contract required because the contractor was unable to execute the works . Preparation of the ICR was extended by almost 12 months beyond the regulation date, as civil unrest in Chad caused the closure of the Country office, preventing interaction between the Bank and the Government that was necessary to complete the ICR . Suspension of Disbursements . Following changes to the Petroleum Revenue Management Law ( law that governed the use of oil revenues generated when the Chad -Cameroon oil pipeline opened in 2003) in December 29, 2005 that weakened programs intended to improve the lives of poor people, the Bank suspended disbursement of IDA funds . The sanctions were in place for six months and were lifted following a Memorandum of Understanding between the Bank and the Government signed on July 13, 2006. 3. Relevance of Objectives & Design: Reducing rural isolation through improved year round access to markets and services was and remains a highly relevant objective for Chad, and likely to help reducing poverty and increasing food security . The high poverty levels in Chad are related to the country ’s enormous size, its often harsh environment and its population distribution in regional enclaves. A result of the non-availability of reliable year-round rural access is the creation of many agricultural enclaves, where the high cost of transport hinders the inflow of inputs and consumer goods to rural areas and the outflow towards food-deficit regions and urban centers . The project supported the CAS goal (CAS discussed at the Board on May 23, 2000, and was prevailing at the time of appraisal ) of removing infrastructure constraints to private-sector led growth and poverty alleviation, a goal that is also stated in the most current CAS (FY2004-FY2006) and in Chad’s National Transport Project, which the project supported . Excessive project complexity compromised the quality of project design, which is rated modest. Six components and institutional responsibilities shared between several administrations, in the context of a weak institutional environment, were bound to affect the implementation of some components, especially those requiring inter -agency coordination. This was the case of the HIV/AIDS and the road safety components . (The ICR highlights this as one of the lessons learned). The road investment and maintenance components, the largest project components, were and remain relevant today, for they would have a direct impact on the quality of the road network, improving accessibility and reducing transport costs. Subcomponents were well selected . Selection criteria of the road sections under the road investment component, covering some 390 kilometers of key links, benefited from lessons from past Bank interventions in the sector, which focused on the construction /rehabilitation of individual road sections without taking into account regional aspects and the impact on other sectors . This was corrected in the project under review . The pilot program for performance-based road management and maintenance contract, covering initial upgrading and maintenance of 441 kilometers of unpaved roads over a four -year contract was an innovative approach to maintaining unpaved roads. The other four components were mostly capacity -building, rather than capital investments . Each had a plausible reason, although taken together made it a complex project . Overall relevance of objectives and design is rated substantial. 4. Achievement of Objectives (Efficacy): Since the formulation of the development objective reflects that of the PRSP including a reduction in transport costs and an improvement in food security, which could not alone be due to improved road access to markets and services especially in rural areas, the review focuses on the improved accessibility aspects . The improvement in rural accessibility was achieved, and by 2007 1,900 kilometers of the National Road Network provided year-round access (a key outcome indicator, which projected that 1,800 kilometers of the NRN would provide such accessibility by 2005). The rural accessibility achievement depended mainly on the road investment components being carried out as planned, and this happened . The 104 kilometer-road Ngoura-Bokoro was upgraded and paved as planned, and rehabilitation of existing paved roads was carried out for 286 kilometers, while the target was a minimum of 200 kilometers. The pilot program to test performance -based maintenance contract met expectations, as upgrading and maintenance works were carried out for 441 kilometers, which was the original target. As reported in the ICR, technical evaluations conducted by Bank engineers and a technical auditor financed by the project praised the quality of the work, specifically for its good riding quality, drainage, safety signs at river and village crossings. The Bank phased out financing for this component in 2005, and similar contracts have been concluded with domestic contractors under local financing . The success of this component, starting with a large pilot and the replications that followed, widening the reach of the approach, is a major achievement of the project . The capacity-building components had a more uneven efficacy . The scope of the road safety component, which was a mix of ‘hard and soft’, was reduced because the data collection and the investments in physical improvements on the N’Djamena streets were not carried out, and the same happened with awareness campaigns, accident data collection and analysis and staff training . A national strategy on road safety was prepared . Physical improvements were limited to a few black spots on the Bokoro -Bitkine road. The component on rural transport strategy development was scaled down and instead a Priority Investment program for rural roads was prepared . Under the institutional support and capacity building component a second -generation Road Maintenance Road Fund, was established, leading to an increase in the road maintenance funding and better prospects than before that road maintenance funding could reach satisfactory levels . This is an important achievement . The HIV/AIDS component carried out most of the planned activities, but a baseline study was not carried out to determine the objective of the component and to select appropriate M&E indicators . Further, the better road accessibility resulted in an improved access to market and services, especially in rural areas . Previous studies showed that a lack of access prevented agricultural produce from reaching markets in the rainy season. Since the road improvements were completed a survey has indicated that trading has increased significantly and improved access to administrative and social services has been achieved . Two road links which were previously not passable for most of the year are now "all weather" facilities. Overall, efficacy is rated substantial. 5. Efficiency (not applicable to DPLs): The ICR estimates the economic rate of return (ERR) at completion, covering the road upgrading and paving, the rehabilitation of existing paved roads and the road maintenance, or about 90 percent of the project cost, at 14 percent, slightly below the 16 percent estimated at appraisal . By individual components, the before and after ERRs are as follows: Paving program 11 & 13 Rehabilitation Program 20 & 14 Pilot maintenance 22 & 14 Overall project 16 & 14. The ERR estimated at completion was close to the return estimated at appraisal, despite significant cost overruns, thanks to unexpected, large increases in the level of traffic on all roads, ranging from 71 percent on the N'Djameena-Guelendeng section to 625 percent on the Walia-Nguele road. These unusually large traffic increases, are not explained in the ICR. They may be due to the significant increase in the country's economic activity following the start of operations of the Chad -Cameroon pipeline in 2003, and a jump in th level of government resources . The former TTL suggested that supplying Darfur refugee camps also generated a significant amount of traffic . Yet it is unclear whether these factors would be enough to generate traffic increases of 625 percent. This raises doubts about the quality of the economic analysis . Further, the statement in the ICR that the ERR calculation at completion was done without estimating the roughness through the IRI (international roughness indicator ) of the roads improved by the project is confusing . The IRI is an essential input to assess the ERR in the models used both at appraisal and at completion . Efficiency of the project was detracted to some extent, (in addition to the higher costs ), by the longer time it took to complete project components and by the fact that the components on rural transport strategy, institutional support and HIV/AIDS achieved less than expected, although completed at a cost barely below (excluding taxes) the appraisal costs. Overall, the ERR, 14 percent is lower than expected despite unusually large traffic increases not foreseen at appraisal, and there are doubts about the methodology followed to estimate the ERR . Moreover There are additional negative factors relating to the delays in completing the investment components and the less -than-expected achievement of the soft components . Overall, efficiency is rated modest. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 16% 88% ICR estimate Yes 14% 92% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The relevance of the development objective was high at appraisal and remained so until project completion, but the design was too complex for the country ’s institutional capacity and rated modest ..Overall relevance is rated substantial. Efficacy was also substantial, since the targets for road upgrading and paving, and road maintenance were achieved, that for rehabilitation was exceeded by 43 percent, and the road maintenance target was met . Had it not been for the underachievement in the soft components, efficacy could have been rated higher . The scope of the road safety component was reduced, the component on rural transport strategy development was scaled -down and the HIV/AIDS initiative lacked a baseline study . Efficiency is rated modest, mainly because of doubts about the credibility of the 14 percent ERR presented in the ICR. a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The level of risk depends mainly on the capacity and political will to provide funding for road maintenance at the required levels. The establishment of a second generation Road Fund, a type of Fund known to be less subject to political interference and to lead to less diversion of road maintenance funds to other uses, is a good step . Another good step is the move towards performance -based maintenance contracts, started during the project, since generally governments are more reluctant to cut funding for such contracts . However, these two steps are still relatively new, and experience elsewhere has shown that it is difficult to ensure long -term sustainability of funding for road maintenance a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: Key factors for judging quality at entry are : (i) inadequate M&E design (Section 10, below), (ii) weak risk analysis, since the PAD failed to take into account risks associated with governance, counterpart funding, cost overruns and implementation delays, all of which turned out to be important risk factors, and (iii) the complexity of the project that involved five different government agencies . The maintenance by contract pilot was novel and an important contribution to modernizing and improving the efficiency of road maintenance . Its success suggests that it was well designed. The former TTL noted that designing the contract during preparation taking into account conditions in Chad required several iterations and complicated fine -tuning of procurement arrangements . The inclusion in the project design of an external technical auditor was useful . The Bank supervised the project closely and with participation of a wide range of professionals to cover different aspects of the project. There was good cooperation between the Bank team and the government . The supervision effort was continuous over the project period, and helped resolve day to day issues . Reporting by the supervision missions was timely and detailed, keeping both the government and Bank management well informed on the progress and issues with implementation . During the last three years of the project, serious disbursement delays resulted from the transition period when the Bank decentralized handling of disbursements from Washington to South Africa, causing serious liquidity problems to contractors and consultants . The Bank team played a key role in the coordination of a large number of donors in the National Transport Program that was supported by the project. The supervision team provided technical advice on road design for the upgrading of paved roads and followed closely the contractor performance in the pilot maintenance contract . at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: The government strongly supported the project during preparation and appraisal, which helped having a fast processing schedule up to credit effectiveness (less than one year between appraisal and effectiveness ). The government fulfilled its commitment to institutional reform by establishing early during project implementation the second generation Road Fund . Financial problems arose during implementation . The most critical performance problem was the delay in the provision of counterpart funding . Another problem was the government decision, shortly after project approval, that all contractors under donor -financed projects would be obliged to pay value added taxes, which were then to be reimbursed by the government . As reported by the ICR, by project closing, a large part of the reimbursements had not yet been made by the government . b.The implementing agency, the Ministry of Infrastructure, carried out its responsibilities mainly through its technical departments of roads and of land transport . The Ministry was supported by long term technical assistance financed by the project and by other donors . Implementation of the physical investments by contractors was generally correct . The Ministry was also supported by the CISCP, an interministerial unit charged with the monitoring and coordination of projects . The CISCP was equipped with experienced staff, most of whom had worked on earlier projects and were conversant with Bank procedures . Execution of the institutional components that required inter -agency coordination, such as road safety component and rural transport strategy, was more complicated and coordination was not as good as it should have been . There were some issues with financial management (Section 11, below). a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: Design. The performance indicators were generally well selected, although one of the three outcome indicators, a decrease in transport rates expected to happen following the improvement of the roads, has proven in previous projects to be impractical. This is because transport rates depend on several factors in addition to vehicle operating costs that are reduced when roads are improved . Chief among such factors are a competitive market and the relative length of the road improved compared to the key origin -destination points in a transport itinerary where rates can be observed. Another design problem was that not all components had indicators (notably the small components) and that not all indicators had baseline data . On the other hand, the performance based contract, which was designed with an emphasis on results rather than on inputs (the classical road maintenance contract ), allowed the supervising and monitoring personnel to focus on service delivery and check the achievement of specified road service levels rather than the input quantities . The ICR notes that the Bank’s Africa Action Plan (AAP) now offers an M&E framework that did not exist at the time of project appraisal, and that it would have helped prevent inconsistencies or gaps in performance indicators . Implementation: Project monitoring was the responsibility of the Ministry of Infrastructure through its technical departments and the CISCP. The performance indicators were reported in a single consolidated quarterly report covering all major activities by the CISCP . Utilization: The performance indicators were used by the supervision missions to verify achievement of targets and recommend remedial actions as needed . The indicators also helped monitor achievements under the National Transport Program. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Environment: The project was classified as Category B . Ratings during implementation were ‘highly’ or just ‘satisfactory’. The ICR rates safeguards compliance as ‘moderately satisfactory’, although it states that no major environmental concerns were noted during implementation . The ICR notes, however, that the Environmental Management Plan included the establishment of an Environmental Protection Unit within the Department of Roads, whose primary role would be ensure compliance by contractors with environmental safeguards in the civil works contracts. The unit was established but did not fulfill its mission adequately because of capacity constraints . Some issues resulting from the civil works included : inadequate restoration of parking lots and contractors ’ campsites, and increased deforestation and bush meat business along the roads under the road maintenance contract . The resettlement safeguard policy was not triggered because none of the physical components (or subcomponents) altered the layout of the roads to be improved, required destruction of dwellings or of collective or private property . Fiduciary: Financial management was generally good in the early years (in 2003 it was rated as the best of all Bank projects in Chad) but deteriorated towards the end of the project . The FY06 audit report was qualified. At the time of project closing in January 2008, the 2007 audit report had not been submitted to the Bank . During implementation, several unaudited Interim Financial Reports (IFRs) were submitted with various delays . At least one claim submitted during the grace period after project closing was paid 13 months after closing, under a special authorization by the Loan Department. 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately Project design was overly complex . Satisfactory The scope of the road safety and rural strategy development components were scaled-back. The HIV/AIDS initiative lacked a baseline study . Efficiency was rated modest due to problems with the quality of economic analysis and consequent credibility of the ERR. Risk to Development Significant Significant Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: In countries with weak institutions and a high risk environment, project should have simple designs, and avoid components that spread across ministries or independent agencies . The introduction of performance -based maintenance contracts for unpaved roads proved successful and is being expanded widely across the Africa region . The experience could be improved by better studying the level of service that will be demanded from the contractor, by documenting lessons learned and by developing capacities of local small and medium enterprises to carry out this type of contract . An effective M&E requires not only the definition of a good set of performance indicators during project preparation, but that it be supported by a well articulated monitoring and evaluation framework . Continuous and close monitoring is essential to readily identify and address issues that surface during implementation . Institutional components require that initiatives be demand -driven and developed in a participatory manner . This would help limit the shortfalls observed on components that required changes in practices such as those on road safety and HIV/AIDS. 14. Assessment Recommended? Yes No Why? Yes. To assess risks to development outcome, including the sustainability of the Road Maintenance Fund and of the use of performance -based contracts for road maintenance, taking into account, on the one hand, the substantial oil revenues and, on the other, the volatile political environment in Chad . 15. Comments on Quality of ICR: The ICR contains a frank discussion of issues of quality in most areas, especially the PAD but also Bank operational problems such as with disbursements following decentralization of this function . The ratings are well supported by the analysis. The ICR includes summary information on a Beneficiary Survey, which is useful although such surveys are not required by Core ICRs. There are, however, some issues with the ICR, which are mainly presentational : -Rating of Quality at Entry in the Data Sheet is different from that in Section 5.1 on Bank Performance. -Efficiency: ERRs analysis and values only reported in an Annex, not in text section - Economic analysis: the presentation raises doubts about the quality of the analysis . No estimate of IRIs after works completed and lack of explanation for the very large, unexpected increases in road traffic . - Section 4: Assessment of Risk to DO. The ICR rates it as moderately unsatisfactory . This is not a rating available for this category. The ICR really rates the quality of risk analysis in the PAD rather than the risk to development outcome. a.Quality of ICR Rating : Satisfactory