Document of The World Bank Report No: ICR00003342 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-78240 TF-91176) ON AN ADAPTABLE PROGRAM LOAN IN THE AMOUNT OF US$ 20 MILLION TO THE REPUBLIC OF COLOMBIA FOR A JUSTICE SERVICES STRENGTHENING PROJECT – PHASE 1 (APL1) IN SUPPORT OF THE FIRST PHASE OF THE JUSTICE SERVICES STRENGTHENING PROGRAM June 8, 2015 Governance Global Practice Colombia and Mexico Country Management Unit Latin America and the Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective as of November 5, 2009) Currency Unit = Peso (COP$) COP $1.00 = US$ 0.000504796 US$ 1.00 = COP$ 1,981 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS APL Adjustable Policy Loan CONPES Council of National Economic and Social Policy (El Consejo Nacional de Política Económica y Social) CPS Country Partnership Strategy DNP National Planning Department (Departamento Nacional de Planeación) IADB Inter-American Development Bank IEG Independent Evaluation Group ICR Implementation Completion Report PHRD Japan Policy and Human Resources Development Fund LIL Learning and Innovation Loan M&E Monitoring and Evaluation MIJ Ministry of Interior and Justice NDP National Development Plan PAD Project Appraisal Document PDO Project Development Objective Vice President: Jorge Familiar Country Director: Gerardo Corrochano Senior Global Practice Director: Mario Marcel Sector Manager: Arturo Herrera Project Team Leader: Jorge Luis Silva ICR Team Leader: Onur Erdem REPUBLIC OF COLOMBIA Justice Services Strengthening Project CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 5 3. Assessment of Outcomes .......................................................................................... 10 4. Assessment of Risk to Development Outcome......................................................... 16 5. Assessment of Bank and Borrower Performance ..................................................... 16 6. Lessons Learned ....................................................................................................... 18 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 19 Annex 1. Project Costs and Financing .......................................................................... 20 Annex 2. Outputs by Component ................................................................................. 21 Annex 3. Economic and Financial Analysis ................................................................. 25 Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 26 Annex 5. Beneficiary Survey Results ........................................................................... 28 Annex 6. Stakeholder Workshop Report and Results................................................... 29 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 30 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 31 Annex 9. List of Supporting Documents ...................................................................... 32 MAP A. Basic Information Justice Services Country: Colombia Project Name: Strenghtening Project ID: P083904 L/C/TF Number(s): IBRD-78240,TF-91176 ICR Date: 04/29/2015 ICR Type: Core ICR REPUBLIC OF Lending Instrument: APL Borrower: COLOMBIA Original Total USD 20.00M Disbursed Amount: USD 18.55M Commitment: Revised Amount: USD 20.00M Environmental Category: C Implementing Agencies: Judiciary of Colombia Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 11/09/2006 Effectiveness: 05/16/2010 03/09/2010 03/27/2013 Appraisal: 11/04/2009 Restructuring(s): 11/11/2013 06/25/2014 Approval: 12/17/2009 Mid-term Review: 08/15/2012 04/01/2013 Closing: 12/31/2013 12/31/2014 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Negligible to Low Bank Performance: Moderately Satisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory Implementing Quality of Supervision: Moderately Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Moderately Closing/Inactive status: Satisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Law and justice 100 100 Theme Code (as % of total Bank financing) Access to law and justice 15 15 Judicial and other dispute resolution mechanisms 85 85 E. Bank Staff Positions At ICR At Approval Vice President: Jorge Familiar Calderon Pamela Cox Country Director: Gerardo M. Corrochano Gloria Grandolini Practice Arturo Herrera Nicholas Paul Manning Manager/Manager: Project Team Leader: Jorge Luis Silva David F. Varela ICR Team Leader: Onur Erdem ICR Primary Author: Onur Erdem F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The Project Development Objective is to strengthen the capacity of the Judiciary and the Ministry of Interior and Justice (MIJ) to deliver timely, efficient, effective and quality dispute resolution services to citizens. Revised Project Development Objectives (as approved by original approving authority) N/A (a) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Backlog in the civil, family and labor courts of Barranquilla, Bogota, Indicator 1 : Bucaramanga, Cartagena, Cali and Medellin reduced by approximately 300,000 cases resolved or archived (Number, Custom) Value quantitative or 1634000.00 1334000.00 547963.00 Qualitative) Date achieved 11/06/2009 12/18/2009 12/31/2014 Comments Over-achieved. The current inventory of the civil, family and labor courts in the (incl. % six cities is 547,963 cases, reducing the number of cases by nearly 1,086,000. achievement) Productivity in the civil, family and labor courts of Barranquilla, Bogota, Indicator 2 : Bucaramanga, Cartagena, Cali and Medellin increased by an average of 50 cases resolved per judge per year. (Number, Custom) Value quantitative or 600.00 650.00 989.00 Qualitative) Date achieved 11/06/2009 12/18/2009 12/31/2014 Comments Over-achieved. The average number of cases resolved for the six cities included (incl. % in the Project was 989. Except from Bogota (average 638 cases per judge), the achievement) rest of the cities clearly exceeded the proposed target. Indicator 3 : Certified mediation centers provide services to 20,000 users (Text, Custom) Value quantitative or 0 20000.00 165000.00 Qualitative) Date achieved 11/06/2009 12/18/2009 12/31/2014 Comments Over-achieved. The technical standard for quality and mediation centers has been (incl. % completed and delivered by ICONTEC. To date around 165,000 users have been achievement) attended (825%). 50 percent of the Judiciary staff in the civil, family and labor courts is trained and Indicator 4 : evaluated in accordance with the policies and processes developed under the project. (Text, Custom) Value quantitative or 0 50% 88.54% Qualitative) Date achieved 11/06/2009 12/18/2009 12/18/2009 Comments (incl. % To date 1700 officers have been trained from a total of 1920. achievement) Decentralized units in Barranquilla, Bogota, Bucaramanga, Cali, Cartagena and Indicator 5 : Medellin provide services to 350,000 users. (Text, Custom) Value 0 350.000 N/A quantitative or Qualitative) Date achieved 11/06/2009 12/18/2009 12/31/2014 Comments (incl. % Not measured. achievement) Court users of civil, family & labor courts of report increased satisfaction levels Indicator 6 : (up to 85 percent) in terms of the in terms of the quality, timeliness and effectiveness of services delivered. (Text, Custom) Value quantitative or 75% 85% N/A Qualitative) Date achieved 11/06/2009 12/18/2009 12/31/2014 Comments (incl. % Not measured. achievement) Court users of the civil, family and labor courts in report increased satisfaction Indicator 7 : (up to 80 percent) in terms of accountability and transparency (Text, Custom) Value quantitative or 0 80% N/A Qualitative) Date achieved 11/06/2009 12/18/2009 12/31/2014 Comments (incl. % Not measured achievement) (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Management model operational in the 100 labor courts in Barranquilla, Bogota, Indicator 1 : Bucaramanga, Cartagena, Cali and Medellin. (Text, Custom) Value (quantitative 0 100 courts 102 courts or Qualitative) Date achieved 11/06/2009 12/18/2009 12/31/2014 The management model had to be adapted to the new General Judicial Process Comments Code. Up to date, the model has been implemented in all the labor courts in (incl. % Barranquilla, Bogotá, Bucaramanga, Cartagena, Cali and Medellin (102 courts, achievement) 100%). Management model operational in the 134 civil and family courts in Indicator 2 : Barranquilla, Bogota, Bucaramanga, Cartagena, Cali and Medellin. (Text, Custom) Value 0 134 courts 380 courts (quantitative or Qualitative) Date achieved 11/06/2009 12/18/2009 12/31/2014 The management model had to be adapted to the new General Judicial Process Comments Code. Up to date, the model has been implemented in all the civil and family (incl. % courts in Barranquilla, Bogotá, Bucaramanga, Cartagena, Cali and Medellin (380 achievement) courts, 283%). A new administrative management model is adopted by EJRLB, including a new Indicator 3 : methodology for virtual training and applied legal research (Text, Custom) Value 1 management 1 management (quantitative 0 model model implemented or Qualitative) implemented Date achieved 11/06/2009 12/18/2009 12/31/2014 The administrative management model has been adopted by the EJRLB. Model Comments was delivered to the Administrative Chamber of the Superior Council of (incl. % Judicature and was adopted by EJRLB in July 2013. The model started to fully achievement) operate the 14th of November, 2013 3 specific training modules to support the management model are operational and Indicator 4 : benefit judicial staff (Text, Custom) Value (quantitative 0 3 training modules 3 training modules or Qualitative) Date achieved 11/06/2009 12/18/2009 12/31/2014 Comments The design and development of 3 training modules have been implemented and (incl. % are operational (100%). The design of the models was contracted in 2012 and achievement) implemented in 2013. New mechanisms and parameters are developed and approved by CSJ for the Indicator 5 : evaluation of the Judiciary’s staff under the management model, and staff is evaluated accordingly (Text, Custom) Evaluation Evaluation parameters and Value parameters and mechanisms was (quantitative 0 mechanisms implemented and or Qualitative) developed used to evaluate 1700 officers. Date achieved 11/06/2009 12/18/2009 12/31/2014 Comments The consultancy for the implementation of an evaluating mechanism has already (incl. % been presented to the magistrates of the Administrative Court. To date, 1700 achievement) officers have been evaluated. Indicator 6 : Justice Services Map updated annually. (Text, Custom) Value Map updated Map updated (quantitative 0 annually annually or Qualitative) Date achieved 11/06/2009 12/18/2009 12/31/2014 Comments Justice Services Map has been updated. The map includes the justice service (incl. % offer in the whole country, not only regarding jurisdictional offer but also achievement) alternative dispute resolution methods. Indicator 7 : First national basic unmet justice needs carried out. (Text, Custom) Value 0 Survey carried out Survey carried out (quantitative in 2013 or Qualitative) Date achieved 11/06/2009 12/18/2009 12/31/2014 First national basic unmet justice needs survey was completed and its results Comments were disseminated in August 23, 2013. (incl. % The survey was applied to 5,866 persons in 14 cities (100%), with a trust level of achievement) 95% and error margin of 3%. 13 decentralized facilities are operational in Barranquilla, Bogota, Bucaramanga, Indicator 8 : Cali, Cartagena and Medellin. (Text, Custom) Value 13 decentralized 14 decentralized (quantitative 0 facilities facilities are or Qualitative) operational operating Date achieved 11/06/2009 12/18/2009 12/31/2014 The refurbishing works have concluded, and the IT equipment and management Comments models have been delivered. To date there are 14 operational facilities (2 in (incl. % Barranquilla, 2 in Bogotá, 2 in Bucaramanga, 1 in Cartagena, 3 in Cali and 4 in achievement) Medellin) 50 mediation centers certified under the new quality standards norm (Text, Indicator 9 : Custom) 50 mediation centers 55 mediation Value certified under centers have been (quantitative 0 the new certified under the or Qualitative) quality new quality standards standards norm. norm Date achieved 11/06/2009 12/18/2009 12/31/2014 Comments 55 mediation centers have been certified, and the Project has also supported other (incl. % 30 centers. The support to the other mediation centers consisted of the achievement) presentation of good practices to improve performance in these centers. Judiciary establishes permanent user information offices to disseminate: Indicator 10 : standards of accountability, transparency, competency and effectiveness applicable to justice sector; & (ii) citizens’ rights and duties (Text, Custom) Value 14 permanent # of information (quantitative 0 information offices offices established or Qualitative) were established. Date achieved 11/06/2009 12/18/2009 12/31/2014 Comments The Project exceeded the initial target of 5 information offices and established 14 (incl. % permanent offices in Barranquilla (2 offices), Bogotá (2 offices), Bucaramanga achievement) (2 offices), Cartagena (1 offices), Cali (3 offices) and Medellin (4 offices). G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 06/11/2010 Satisfactory Satisfactory 0.00 2 02/15/2011 Moderately Satisfactory Moderately Satisfactory 1.00 Moderately Moderately 3 08/10/2011 1.00 Unsatisfactory Unsatisfactory 4 02/27/2012 Moderately Satisfactory Moderately Satisfactory 1.00 5 11/05/2012 Satisfactory Satisfactory 4.97 6 06/25/2013 Satisfactory Satisfactory 14.67 7 12/29/2013 Satisfactory Satisfactory 14.67 8 06/28/2014 Satisfactory Satisfactory 16.89 9 12/12/2014 Moderately Satisfactory Satisfactory 18.55 H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions Changes in some of the activities were made to adapt the project to the needs of the Judiciary given a new reform 03/27/2013 S S 14.67 process that included the introduction of a New Procedural Code. No changes were made to the indicators. Extended the closing date to 11/11/2013 S S 14.67 June 30, 2014. Extended the closing date to 06/25/2014 S S 16.89 December 31, 2014. I. Disbursement Profile 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. Justice and citizen security have constituted the centerpiece of the evolution of the Colombian state. Following a series of legal changes in the first half of the 20th century, a new Constitution was adopted successfully in 1991. The new Constitution emphasized inclusiveness, acknowledged indigenous people, introduced new rights, strengthened the Judiciary and sought a more balanced distribution of powers, all to bring peace to a country long torn by violence. 2. Providing for the three traditional branches of the government (executive, legislative and judicial), the Constitution introduced a complex organization for the justice sector that continues to characterize the sector today. From the Executive branch, the Ministry of Interior and Justice (MIJ) articulates sector policies, with a particular focus on access to justice services. 1 The Legislative branch engages in functions relating to the impeachment of high State officials. The Judiciary is the backbone of the sector, and includes a Constitutional Court and Superior Judicial Council that acts as the governing body of the Judiciary. The court system is divided into five jurisdictions: ordinary, administrative, constitutional, peace, and indigenous communities. Ordinary is the largest jurisdiction and encompasses criminal, civil, labor, family, and agrarian cases. It is exercised by Municipal Courts, Circuit Courts, Superior Courts, and the Supreme Court. 3. The legal reforms that the new Constitution introduced included a right of the citizen to protest the violation of his or her constitutional rights, provisions to accelerate the transition to an accusatorial system of criminal justice, and the creation of out-of-court conflict resolution mechanisms. The core objective of these initiatives was to address the historically complex nature of legislation that emphasized form over substance, referred to as the “written system” of conducting court procedures. Often confusing, with meticulous attention to detail and reliance on written correspondence, the written system had made it difficult for lawyers to comply with its requirements and overwhelmed the capacity of judges to respond to the citizens’ demands, eroding citizen confidence in the sector and resulting in persistent backlogs in the court system. 4. Major initiatives were introduced in 2005 and 2008, aimed at decongesting the court system of these backlogs. The most important one was a gradual transition from the written procedural system to oral in criminal, labor, civil, and family courts. The transition to the oral system, while treated in the backdrop of the Project Appraisal Document (PAD), eventually became the crux of the Judiciary’s strategy to resolve persistent congestion in the court system, impacting major project decisions during Project implementation. 5. The failure of the justice system to respond to citizens’ demands in a timely and effective manner, demonstrated by persistent backlogs in the court system and lack of decentralized access to justice services, has been an important item on the Government’s development agenda. The Vision Colombia Bicentennial 2019, the National Development Plan for 2006-2010, and the 1 The Ministry of Interior and Justice underwent an institutional restructuring during the period that the Project was implemented. Today, it is called the Ministry of Justice and the Law. 1 Judiciary Strategic Plan all emphasized the importance of consolidating a model of democratic governance to bring effective justice services closer to citizens. 6. Against this backdrop, the World Bank Country Partnership Strategy (CPS) for the period 2008-2011 (Report No 42847-CO), under its “Peace Pillar”, aimed at helping the Government strengthen the foundations for national reconciliation and lasting peace. The Peace Pillar envisioned that the Project would support this objective by helping to remove real or perceived barriers to access in the justice system, through improved court performance and enhanced access to justice services. The CPS further envisioned, under its “State in the Service of its Citizens: Efficient and Effective Government Pillar,” that the Project would support the authorities’ efforts to improve services in the justice sector by scaling up the “management model” that had been developed under the Judicial Conflict Resolution Improvement Learning and Innovation Loan (LIL) (P057369), a Bank funded effort implemented by the Judiciary that had closed in June 2006. 1.2 Original Project Development Objectives (PDO) and Key Indicators 7. The Project Development Objective (PDO) was to strengthen the capacity of the Judiciary and the Ministry of Interior and Justice (MIJ) to deliver timely, efficient, effective and quality dispute resolution services to citizens. 8. The PAD lists key project indicators as follows (as presented in the ICR datasheet): 1. Backlog in the civil, family and labor courts of Barranquilla, Bogota, Bucaramanga, Cali, Cartagena and Medellin targeted by the project reduced by approximately 300,000 cases resolved or archived. 2. Productivity in the civil, family and labor courts of Barranquilla, Bogota, Bucaramanga, Cali, Cartagena and Medellin targeted by the project increased by an average of 50 cases resolved per judge per year. 3. Certified conciliation/mediation centers provide services to 20,000 users. 4. 50 percent of the Judiciary staff in the civil, family and labor courts of Barranquilla, Bogota, Bucaramanga, Cali, Cartagena and Medellin is trained and evaluated in accordance with the policies and processes developed under the Project. 5. Decentralized units in Barranquilla, Bogota, Bucaramanga, Cali, Cartagena and Medellin provide services to 350,000 users. 6. Court users of the civil, family and labor courts of Barranquilla, Bogota, Bucaramanga, Cali, Cartagena and Medellin report increased satisfaction levels (up to 85 percent) in terms of the quality, timeliness and effectiveness of the services delivered. 7. Court users of the civil, family and labor courts of Barranquilla, Bogota, Bucaramanga, Cali, Cartagena and Medellin report increased satisfaction (up to 80 percent) in terms of accountability and transparency. 1.3 Revised PDO and Key Indicators 9. There were no revisions to the PDO or key indicators during the life of the Project. 1.4 Main Beneficiaries 10. The main beneficiaries of the Project were citizens who would benefit from the improvements in dispute resolution services triggered by the Project, as well as judges in civil, 2 family and labor courts of the six cities, whose productivity would improve as a consequence of the infrastructure, training, and technical assistance activities provided by the Project. 1.5 Original Components 11. Component 1 - Strengthening of judicial case management (US$ 11.48 million). The objective of this component was to strengthen the managerial capacity of the Judiciary to improve the performance of civil, family, and labor courts in Barranquilla, Bogota, Bucaramanga, Cali and Medellin. The activities to be financed under this component included: Subcomponent 1.1. Efficiency of justice services management (US$ 10.87 million) • Design of a management model for the selected courts aimed at reducing congestion and implementing oral procedures, based on the management model developed under the LIL for the circuit civil courts (Justicia XXI). • Implementation of the management model in the selected courts including remodeling and refurbishing of office space in existing court buildings. • Implementation of an M&E system for court performance focusing on quality and productivity, including the provision of the ICT infrastructure and training. • Screening of the case inventory of the selected courts, for the purposes of determining active and inactive backlogs, and reallocating inactive backlogs. • Design and implementation of automated procedures for delegation orders (despachos comisorios) in the civil courts of the city of Bogota, including the remodeling and refurbishing of office space in existing court buildings. Subcomponent 1.2. Communication systems between citizens and the Judiciary (US$ 0.61 million) • Implementation of the Judiciary’s internal and external communications strategy to support the management model. • Establishment of administrative accountability mechanisms for judicial personnel, aimed at enhancing the role of users in the improvement of justice services. 12. Component 2 – Development of judicial human resources (US$ 1.73 million). The objective of this component was to develop the Judiciary’s human resources on the basis of periodic performance evaluations and capacity building. The activities to be financed under the second component included: Subcomponent 2.1. Judicial staff training (US$ 1.63 million) • Design of a new administrative management model for CSJ’s Judicial School (EJRLB), based on an evaluation of its current model. • Implementation of the new administrative management model for EJRLB, through the development of: (i) virtual training methodologies and contents (including the refurbishing of the EJRLB’s video-conference room); and (ii) a new program on applied legal research for judges. • Design and implementation of specific training programs to support the management model developed under Component 1 aimed at streamlining oral-based arguments. Subcomponent 2.2. Judicial staff performance evaluation (US$ 0.1 million) • Design and implementation of standard performance evaluation mechanisms and parameters for judicial personnel. 3 13. Component 3 - Facilitating access to justice services (US$ 2.67 million). The objective of this component was to facilitate access to justice services by vulnerable groups, through the generation and dissemination of reliable information, the extension of geographic coverage and the strengthening of mediation services. The activities to be financed under this component included: Subcomponent 3.1. For all justice services (US$ 0.52 million) • Strengthening of justice services supply-demand analysis tools, including: o Development of the technological platform to update the justice services map; and o Replication of the survey on basic unmet dispute resolution needs at the national level. Subcomponent 3.2. For formal justice services (US$ 1.86 million) • Design and implementation of decentralization strategies for the management model in the selected courts, to facilitate: o Decentralization of the provision of selected justice services (e.g. filing/service of process, submissions of claims/counterclaims, written pleadings, etc.) in partnership with other State agencies; and o Physical decentralization of court offices through office remodeling and refurbishment of office space in existing buildings to provide accessible service entry points to citizens, including user information kiosks. Subcomponent 3.3. For alternative dispute resolution mechanisms (US$ 0.29 million) • Strengthening of mediation services, through: o Development of a technical quality norm for mediation centers services; and o Implementation of the technical quality norm in selected mediation centers. 14. Component 4 – Project coordination (US$ 0.81 million). The objective of this component was to achieve successful Project implementation, through standard Project coordination, and specific M&E activities. The activities to be financed under this component included the operation of a Project Coordination Unit (PCU) responsible for administrative, financial management, procurement and operational activities; and Project M&E. 1.6 Revised Components 15. There were no official revisions to the original Project components in the three restructurings that took place during the life of the Project. 1.7 Other significant changes 16. The first restructuring, performed in March 2013, introduced a reallocation of loan proceeds to respond to the evolving Government priorities, in particular to allow for the adequate financing of the Justice Services Map to be implemented by the Ministry of Justice and the initiation of the e-justice initiative in the civil and commercial courts of Colombia’s six major cities. The second restructuring, approved in November 2013, extended the Project closing date by six months to allow the Project to reorient its activities to support the Judiciary’s introduction of the New Procedural Code, which brought the transition to the oral system. The reorientation of the activities under the first and second restructuring did not require a change in the legal description of the project components. The third restructuring, which took place in June 2014, 4 extended the Project closing date for another six months, in order to allow for the completion of the initiatives launched during the second restructuring and the preparation of a follow on project as had been anticipated in the PAD. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 17. The PAD reflects country priorities as established in official Government and Bank documents, previous assistance activities supported by the Bank and other donors in the sector, and a comprehensive follow up diagnostic of the Justice sector that was performed during project preparation. The PDO stemmed from the strategic priorities set forth in the Vision Colombia Bicentennial 2019, the National Development Plan for 2006-2010, and the Judiciary Strategic Plan, as well as the different pillars of the World Bank CPS for the period 2008-2011 (Report No 42847-CO) that related to justice and security. The decision to focus Project activities on civil, family, and labor courts rather than criminal courts was made based on the fact that to date, most reform efforts, including those traditionally supported by the donor community, had targeted criminal justice, with positive results in terms of improved court and case management that had left similar types of success desirable in non-criminal courts. 18. The technical orientation of the Project around consolidating and instituting an integrated management model in the court system as the primary means to improve court performance originated from the Judiciary’s request to scale up the preliminary results of the Judicial Conflict Resolution Improvement LIL (P057369), the Bank’s earlier (and first) project with the Judiciary. Initial results of the LIL had suggested promising reductions in backlogs, partly attributable to the new management model that the project had introduced. The lessons learned from this loan informed the preparation of the Project, which, at the request of the Judiciary, was undertaken through a follow up diagnostic of the sector funded through a grant through the Japan Policy and Human Resources Development Fund (PHRD). 19. The lessons learned from the LIL, and later the diagnostic performed through the Japan PHRD Grant, provided a robust basis for Project preparation. Despite this advanced level of analysis, the ICR rates Quality at Entry as moderately satisfactory. The ICR observes, with the benefit of hindsight, that the Project’s PDO may have been too broad and that the link between Project outcomes and the activities to be financed was not always entirely clear. Moreover, in various places the PAD included language that set out to promise the achievement of rather broad goals that in one way or another sounded different than the official PDO, burdening the Project with ambitious promises in a historically complex sector. In this sense, the PAD promised a comprehensive reform effort, one that would not only “strengthen the capacity of the Judiciary and the Ministry of Justice to deliver timely, efficient, effective and quality dispute resolution services to citizens,” but also “lessen the effects of an expected economic downturn on economic players under financial distress,” “advance law-based policies and practices for the resolution of cases that have economic impact,” and “produce swifter enforcement of contracts.” 20. On the other hand, an examination of the logic behind the design of Project Components suggests that the majority of these goals were expected to be achieved by interventions through the first Component, the institutionalization of the integrated management model. In this sense, considerable expectations were built into not only the premise that the institution of the management model would be the ideal means to enhance timeliness, efficiency, effectiveness, and quality in the court system, but also that the management model would be implemented 5 successfully across six major cities. This premise does not seem well-anchored in a solid foundation, as the PAD itself indicated that the positive results observed in backlog reductions after the LIL were only partly attributable to the management model, which falls short of definitive evidence. 21. The rationale for the chosen lending instrument, Adaptable Program Loan (APL), also placed a heavy role on the potential impact of the management model in streamlining court performance. A lending instrument that is now under the Investment Project Financing instrument as a Series of Projects type of investment operation, the APL was an instrument at the time of Project preparation that enabled a programmatic approach to investments over the medium-term. Through this mechanism, the successful completion of the Project would trigger the preparation of the second phase of the program (i.e. a second APL). The conditioning of the second phase on the successful achievement of the PDO further intensified the expectations from and dependence on the management model’s success under the Project. 22. Finally, the implementation arrangements introduced in the last Component would later lead to significant delays in project implementation. It was conceived originally that a single Project Management Unit (PMU) would manage the implementation of the reform efforts supported by the World Bank and the Inter-American Development Bank (IADB) in the Judiciary, where the cost of sustaining the PMU would be shared between the two institutions. Project management experience suggests, however, that such efforts to coordinate donor support often lead to implementation delays, largely based on the difficulties that PMUs face in harmonizing their practices across the different requirements of different international financial institutions. In the case of the Project, these differences led to a delay of about one and a half years before the activities could commence. 23. Project preparation included an extensive treatment of critical risks to Project success. The risks identified included political economy considerations and factors relating to resistance to change and lack of coordination across different units and institutions. The Project aimed at mitigating these risks through an explicit orientation of the Project around reforming those operational aspects of court management that were deemed to be non-controversial. 24. What the PAD did not seem to have considered, however, was the possibility of comprehensive changes in legal codes, which would have inevitably had consequences for the procedural aspects of court management that the Project was preparing to influence. In this context, the Project seems to have tied its hands further by indexing its success to the definitive implementation of the integrated management model, especially in an environment that was in flux as it set out to transition from the written system of conducting court procedures to the oral. 2.2 Implementation 25. The Loan Agreement was signed in February 2010. From this point onwards, coordinating the operations of the two justice sector reform projects, one supported by the World Bank and the other by the IADB, presented challenges that effectively blocked the commencement of activities. The PMU could not be staffed fully during this period. As these two projects had entered the medium-term planning instrument (CONPES) of the National Planning Department (DNP) together in the form of an integrated program aimed at reforming the justice sector, separating them implied a bureaucratic process that further caused delays. These challenges were eventually overcome, the projects separated, and the Project commenced implementation by mid-2011, representing an overall delay of almost one and a half years since the time of signing the loan. 6 26. At that stage, all anticipated public works, equipment purchases, and consultancies began to be contracted, executed, and completed with remarkable speed and efficiency. Outcome indicators started showing the expected results, with all indicators on course to achieving and in some cases exceeding the anticipated targets. In 2012, the Project was featured as an exemplary project in the National Plan of the DNP. By December 2013, one year before the revised closing date of the Project, a high percentage of all planned activities had been completed, and the Project was on its way to achieving the results set forth in the PAD as triggers for the design of the second programmatic phase. 27. By the time the Project picked up steam in 2011, the gradual introduction of the oral system to non-criminal courts was a certainty and the Judiciary was already in the process of formulating its transition plan. The consultancies that the Project contracted for the elaboration of the integrated management model found themselves in an environment that was relatively uncertain in terms of what the operational implications of the gradual transition to the oral system would be. The transition to the oral system in the execution of the legal code, along with the official introduction of the policy of “decongestion” aimed at fast-tracking certain cases that had been in the backlog for some time, became the landmark initiatives that constituted the core of the Judiciary’s efforts to improve the performance of the court system. Faced with this reality, Project components reoriented themselves to adapt to and complement these initiatives. 28. Facing these structural changes that had not been anticipated during Project preparation, the PMU gradually adjusted its approach from a definitive development and comprehensive institution of an integrated management model to facilitating the transition of the non-criminal court system to the oral system in the six cities. While consultancies to define and develop an integrated management model were executed, the products did not go much farther than becoming reference documents as the Judiciary itself struggled with managing the dynamics of this comprehensive reform process across the country. Through two restructurings in 2013, the Project reallocated loan proceeds towards investments in infrastructure, equipment and training to support the Judiciary’s technological reform agenda as well as its introduction of the New Procedural Code, which formalized the transition to the oral system. Aligning itself with the Judiciary’s reform agenda, the Project invested in refurbishing buildings to accommodate court rooms and hearings, the purchase and installation of the associated audio, video, and IT equipment, and the training of Judiciary personnel. 29. One other major factor that impacted implementation took place in the context of the Project’s emphasis on the decentralization of selected justice services, under Component 3. Here, the Project ran into considerable challenges as the Judiciary had to negotiate extensively with the selected municipalities regarding finding locations and getting the necessary permissions for opening the new service centers. The Judiciary also faced challenges finding and appointing judges to these centers. As a result, most of these service centers only became fully operational in the last year of the Project, which makes measuring citizen satisfaction with these services challenging. Initial statistics on usage by citizens in many of these centers, however, are signaling an upward trend, and the judges interviewed related anecdotal evidence of citizen satisfaction. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 30. Colombia has a rich tradition of utilizing data in development planning and implementation. The Project rightly chose from the beginning to build on the Judiciary’s ongoing efforts in data-driven performance improvement, and incorporated into its M&E framework progress indicators towards which the Statistics Department of the Judiciary had already been 7 collecting information routinely. This was particularly true for Component 1, where Project interventions were expected to lead to improvements in the timeliness, efficiency and quality of justice services, and the progress towards this outcome was designed to be measured through backlog and case clearance rates, two indicators for which historical information across the country was available through the Judiciary. 31. Despite this effort to build on already existing data platforms, the ICR finds the M&E framework for the Project moderately unsatisfactory. Key outcome indicators are a combination of outcome, output and qualitative indicators that while at first sight offer reasonable insights on the direction of progress, do little to attribute the impact of the Project towards the achievement of that progress. 32. A closer look at the key outcome indicators will help explain this point. While it seems that the Project expected to measure court performance (a proxy for timeliness and efficiency of conflict resolution services) primarily through the two outcome indicators on backlog and case clearance rates (key outcome indicators 1 and 2), these indicators were based on generic ones for which data was already being collected by the Judiciary. Backlog and case clearance rates are influenced by a number of factors, and to expect that any major improvements in these rates were going to be clearly attributable to the impact of the Project would have been overly optimistic. Indeed, as the ICR will elaborate further under Section 3: Assessment of Outcomes, while throughout the life of the Project these indicators showed considerable improvements, which translated into “satisfactory” ratings in ISRs, a closer examination of the backlogs during this period reveals trends that possibly correlate with factors that the Project influenced only indirectly. 33. The other key elements of the PDO, namely the effectiveness and quality of justice services, were to be measured primarily through user satisfaction surveys (key outcome indicators 6 and 7). Satisfaction surveys can be costly exercises, and finding clear trends in service quality requires repeated observations over time. The Project’s infrastructure, equipment and management-related interventions were undoubtedly going to take time to be completed and to bear fruit, limiting the scope and available time period for such surveys. In the end, these surveys could not be carried out. 34. The other three key outcome indicators aren’t really outcome indicators but rather output indicators. While valuable in their own right and clearly important outputs that would have been the direct products of the interventions anticipated under Project Components 2 and 3, the training of judges and the physical establishment of decentralized service centers in and of themselves should probably not have constituted key Project outcome indicators. 35. The flaws in the design of the M&E framework had consequences for its implementation and utilization. The first two indicators were regularly updated based on data received directly from the Statistics Department of the Judiciary, but seem to have offered little use in informing implementation decisions, as the ICR could find no perceivably direct link between Project interventions and the nature of the data being collected. The indicators based on satisfaction surveys were never updated as these surveys were never realized. And the updating of the output indicators regarding the establishment of decentralized service centers came with a lag (and in some cases not measured) as this Component faced implementation delays. In any case, the target for this indicator seems to have been based primarily on the general populations of the areas where the centers were expected to be built – in that sense, once the center was built and operational, it would technically be providing services to virtually any number of citizens, thus signaling the successful achievement of this outcome, whereas this “outcome” in itself does not 8 mean much for the actual number of citizens visiting and benefiting from these center, neither the quality of the services offered. 2.4 Safeguard and Fiduciary Compliance 36. No issues were observed in safeguard compliance. Financial management performance was assessed moderately unsatisfactory during the early stages of the project implementation, but adequate corrective measures were taken by the client, leading to a satisfactory rating for the rest of the Project. Audit reports were submitted on time and did not signal issues. Similarly, acceptable interim unaudited financial reports were submitted regularly and in line with the loan agreement. Procurement performance was rated moderately satisfactory for the life of the project, except in the first ISR, which was due initially to the delay in contracting a Procurement Specialist and later to weaknesses in procurement planning and execution as noted previously. 2.5 Post-completion Operation/Next Phase 37. The primary objective of Component 1 was absorbed fully into the overall reform program of the Judiciary. Going forward, the Judiciary plans to develop a “unified management model,” one that integrates the model for non-criminal courts and the model that had been under development in parallel for the high courts. The decision to reorient organizational reform towards the development of a “unified management model,” (and the software associated with it, which would replace the case management system currently in place), was made in the middle of the Project, around the same time as the Judiciary was strengthening its efforts to transition to the oral system. At the time of writing this ICR, the Judiciary had a concrete transition plan to the oral system for all non-criminal courts, as well as a relatively well-developed conceptual idea of what it expected from a “unified management system.” 38. While the ICR team confirmed the Judiciary’s plans to continue its efforts towards improving organizational efficiency, at the time of drafting this ICR there was no evidence of a concrete financing plan which would help the Judiciary achieve these plans. The ICR observed that the Judiciary had been counting heavily on the second phase of the APL, which, as described above under Project design, had been the original plan and therefore built into expectations from the beginning. 39. As the Project achieved the triggers for the design of a second phase, the Government requested additional financing for the continuation and scaling up of activities beyond the closing date of the Project. By November 2014, one month before the official closing of the Project, Bank management had decided to no longer pursue the additional financing request. This decision was made on the basis that the technical nature of the activities and the desired outcomes that the program documents described justified the consideration of a new investment lending operation rather than an additional financing request to continue the same project. Moreover, it was decided that the size of the additional financing request, at $20 million, did not align fully with the current World Bank strategy in the country. Consequently, the Project proceeded to closing, and the Bank team entered discussions with the Judiciary, MIJ and World Bank leadership regarding the possibility of designing a new investment lending operation. At the time of writing this ICR, no concrete plans had been made regarding a new project, although discussions were underway. Given the relevance of the original PDO in Colombia, and the positive momentum achieved in justice sector reform over the past several years, the ICR recommends continued dialogue that focuses not only on strengthening case management as interventions traditionally have, but also, 9 and perhaps more importantly, the decentralization of justice services, which the ICR team observes has the greater potential to benefit the citizens in the short to medium term. 40. With regards to the sustainability of the public works and the equipment purchased under the Project, the ICR has observed that the infrastructure has been fully integrated into the Judiciary’s routine maintenance plans, which should ensure their longevity. 41. The Judiciary has a relatively robust M&E framework to collect statistics, especially on case clearance performance. The ICR expects that the interventions financed under the Project will continue to contribute to the Judiciary’s reform agenda, whose overall performance is monitored through this M&E framework. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation Rating: High 42. Judging by the development priorities and circumstances prevailing at the time of the ICR, including the Project’s consistency with current country and Bank strategies, the ICR finds the Project highly relevant. 43. In December 2010, the Government launched an ambitious new National Development Plan (NDP), called Prosperity for All 2010–2014. The overarching goals of the NDP were to increase employment, reduce poverty, and improve security. The NDP had three main objectives: Sustainable Growth and Competitiveness, Equality of Opportunities for Social Prosperity, and Consolidation of Peace. It highlighted four cross-cutting themes: Relevance of International Relations, Environmental and Disaster Risk Management, Good Governance, and Regional Development and Integration. 44. In the years leading up to the development of the NDP, including the period during which the Project was prepared, Colombia had taken significant steps toward the consolidation of peace. The difficulties Colombia’s judicial system faced in contributing to the consolidation of the peace process continue to be acknowledged widely, and efforts to strengthen peace and security in the new NDP stem from a broader initiative involving the whole of Colombian society, promoting a citizenship culture based on respect for peaceful coexistence and human rights, and strengthening and holding accountable democratic institutions and the justice system. 45. Coinciding with the priorities established in the NDP, the Bank’s CPS for the period 2012-2016 had three strategic themes: Expanding Opportunities for Social Prosperity; Sustainable Growth with Enhanced Climate Change Resilience; and Inclusive Growth with Enhanced Productivity. Each of these strategic themes had three areas of results with specific outcomes, which included improved citizen security. 46. Against this backdrop, the Project’s development objectives, as well as its design and implementation, remain highly relevant. The design of project interventions around the institution of procedural and systematic improvements to facilitate access to justice, and their implementation taking into account the evolving nature of the reform agenda in this context, continue to be the desired way forward for reforming the judicial sector in Colombia. This is demonstrated by the Judiciary’s continuing commitment to advancing its decongestion policies, 10 transition to the oral system, and expansion of decentralized justice service centers across the country. 3.2 Achievement of Project Development Objectives Rating: Substantial 47. Despite recent advances in the literature, assessing the impact and efficacy of governance and public sector interventions continues to be a challenging exercise. Short of historical data and resources to conduct a full-fledged impact assessment, the ICR team adopted a combination of approaches to assess Project efficacy. These approaches included 1) an analysis of the Project’s most relevant key outcome indicators, 2) the Economic and Financial Analysis included in the PAD, 3) the Project’s midterm review, and 4) an examination of the “triggers” from the PAD that had been established as conditions for the design of the second phase. 48. Analysis of the Project’s key outcome indicators. The positive trends that the ISRs observed in all key outcome indicators (except for those not measured), point towards the successful achievement of the PDO. And for the unmeasured indicators that were to be based on user satisfaction surveys, anecdotal evidence that the ICR team collected from judges in key cities and the decentralized service centers that the team visited, the case processing data that the team observed in the decentralized service centers, as well as brief interviews conducted with users of the decentralized service centers all signaled positive trends in usage and user satisfaction with the services provided. Judges and users alike seemed particularly satisfied with the reduction in case processing times, the opportunity to appear before a judge, and the ability to process routine cases without having to travel to big cities. 49. The ICR team performed an analysis of the case processing data for the period 2009-2014 first as a sum of total inventory in all six cities, an indicator that the Project had chosen as key. In addition to basing the analysis on a combination of all three different types of court cases that the Project set out to impact (civil, family and labor), which had shown considerable improvements during implementation, the team disaggregated the data by case type, presented in the chart below. 11 This analysis yielded an interesting result: the majority of the improvements in case load during this period seems to have been influenced by the dramatic change in case processing rates for civil cases. 50. Interviews that the ICR team conducted with judges across the three different cities visited during the ICR missions (Bogota, Medellin, and Bucaramanga), as well as those conducted with Judiciary counterparts, suggest that the majority of the improvements observed in civil case inventory during this period was due to the “decongestion” policies that the Judiciary had introduced. The decongestion policies made it easier for judges to fast-track, resolve, or drop certain cases based on their nature and the time the case had been in the court system. In family and labor courts, judges attributed the majority of the improvements in case processing to the transition to the oral system, which had reduced processing times dramatically, in many cases from an average of two years to a few months. Since the transition to the oral system has been gradual and so far partial, it is not surprising that the full effect has not been observed yet. Against this backdrop, the ICR concludes that the reorientation of Project Components during this period enabled the Project to complement the Judiciary’s efforts to establish the physical infrastructure and human capacity required to transition to the oral system, which in turn contributed positively to a reform process that has already shown strong potential to reduce backlog issues and improve the timeliness and the quality of justice cervices. 52. The Economic and Financial Analysis (EFA) included in the PAD. The EFA from the PAD makes a reference to the Project’s potential to contribute positively to the governance agenda by “providing the powerful deterrents of permanent transparency and accountability mechanisms.” It also points out the Project’s potential impact on doing business by increasing business confidence and reducing the time and cost of contract enforcement. 53. While the links between the regulatory environment and the business climate is clear, the ICR team has not been able to find a perceivably direct link between Project interventions and improvements in the business environment. The 2015 Doing Business report ranks Colombia 168 in enforcing contracts, at 1,288 days. This is the same number of days as in the 2014 report, which ranked the country 167. For the previous four years there is no ranking available, and the number of days was steady at 1,346. There is no particular information about enforcement of contracts in the main cities of Colombia. The subnational data available only includes information on starting a business, dealing with construction permits, registering property, and paying taxes. Against this backdrop, the team finds it difficult to assess the Project’s potential impact on business climate. 54. On the other hand, the team observes that the primary impact of this Project has been an increase in the overall efficiency, timeliness, effectiveness, and quality of dispute resolution services. In this sense, the more relevant measure of the return on the investment would be the efficiency gains in courtroom operations, which, as the EFA in the PAD claimed, would allow for a more cost-effective allocation of resources. 55. The EFA in the PAD states that justice sector investments are not “subject to conventional financial analysis since a substantial part of their outcomes are difficult or impossible to measure, and are more related to critical social values like legitimacy of, and confidence in, State institutions.” While this statement is valid, there have been considerable advances in recent years in attempting to quantify the impact of policy interventions in institutional reform. Colombia’s own process of collecting and analyzing rigorously case management data is an example of an attempt to understand and inform policy and its impacts on citizen welfare in the justice sector. 12 56. While an impact analysis of this nature has been beyond the capacity and scope of this ICR effort, the team has attempted to calculate the sample, average savings achieved in the system during this period, through anecdotal evidence from interviews with judges and other Judiciary stakeholders (see Text Box). The fact that the transition to the oral system has in many cases reduced average case processing times from two years to a few months in itself is a substantial, promising outcome. Furthermore, the establishment of decentralized service centers, which constitute relatively inexpensive investments in Average Savings in Case Processing infrastructure, are already reducing significantly the legal The implementation of the oral system started in fees and the transportation costs Colombia in January 2013. Of the six cities that the that citizens would have had to Project focused on, two had started implementing the oral incur under the old system, where system fully since that date (Cali and Barranquilla), while its implementation is still underway in other cities. To they would have had to perform identify the sample average savings per court during this even the most routine transactions period, the team used as basis the civil circuit courts in by traveling to a city and finding Cali. The team first calculated the average cost of legal intermediaries to help them processing a case before the introduction of the oral process their paperwork. system, dividing the total annual operating costs of one civil circuit court (COP$432,458,204 or approximately 57. The Project’s Midterm US$180,000) by the average final case inventory from Review. The DNP commissioned 2009 to 2012 (928), yielding an approximate average cost an independent midterm review of of COP$466,011 or US$195 per case. The team then the Project, which was carried out calculates the difference between the 2013 and 2014 final inventories to find the additional number of cases that in 2013 by G Exponencial and were processed after the introduction of the oral system. Ipsos Napoleon Franco and was The case inventory decreased by 474 cases during this reflected in the Bank’s midterm period, signaling savings around US$92,000 per court per review report. This detailed year. While simplistic in its methodology, these independent midterm review calculations point to significant potential savings, assesses the relevance, operational considering that the Project impacted 234 courts across efficiency and implementation six cities during their transition to the oral system. progress of the various contracts and activities executed under the Project. The review at first sight fails to identify a direct, causal connection between the activities of the Project and the positive results that the Judiciary achieved in case clearance rates during that period. Nevertheless, the review acknowledges the impressive execution progress that the Project achieved especially once the PMU became fully operational, finds that the nature of the activities being executed were in line with the original objectives of the Project, and validates the emphasis that the Project had begun placing on supporting the Judiciary’s decongestion policies and transition to the oral system. 58. Achievement of the “triggers” for the design of the second phase. The APL approach adopted for the design of this Project included the formulation of triggers, whose achievement would justify the preparation of the second phase of the program. The Project team prepared its additional financing request based on the successful achievement of these triggers, which included: Trigger Achieved? Comments Management model implemented in Yes While a Project-sponsored 234 civil, family and labor courts in integrated management model did Bogota, Barranquilla, Bucaramanga, not fully materialize in the form Cali, Cartagena and Medellin that had been anticipated originally, 13 Performance M&E system Yes the reorientation of Project implemented in the 234 courts that Components during restructurings operate the management model enabled the Project to complement the Judiciary’s efforts to establish the physical infrastructure and human capacity required to transition to the oral system and the implementation of the management and performance M&E procedures it brought about. Case inventory screening completed Yes in the 573 courts that will operate the management model First national unmet basic justice Yes The results of the first demand for services survey completed justice services survey was extremely well-received, and yielded important insights for the Ministry of Justice’s ongoing strategy of and plans to continue the expansion of alternative dispute resolution services, a key objective under the National Development Plan. The Ministry of Justice updated this first survey the following year, an indication of the usefulness of the first initiative. Design of the special access module Yes to facilitate decentralization completed Technical quality norm for Yes conciliation centers approved 70% of the proceeds of APL 1 Yes disbursed 59. The systematic achievement of these triggers for the design of a second phase, as validated through Bank supervision, further exemplifies the indirect outcomes that the Project substantively impacted, which are not captured by the Project’s PDO or the M&E framework. 60. Deconstructing the PDO against this backdrop, Project efficacy can be summarized as follows: “Strengthening the capacity of the Judiciary and the Ministry of Interior and Justice to deliver timely, efficient, effective, and quality dispute resolution services to citizens..” Elements Achieved? Comments of PDO Timely Substantially Interviews with judges consistently confirmed a dramatic reduction in typical case processing times, from an average of two years to a few months, as a result of the transition to the oral system. Efficient Substantially The refurbishing of buildings to accommodate court rooms, the building of decentralized justice centers, and the introduction of IT, 14 audio and video equipment through the Project have facilitated significantly case management under the new system. Effective Substantially While citizen satisfaction could not be measured under the Project, Quality Moderately ad hoc interviews with users and anecdotal evidence that the ICR team collected indicated initial and growing satisfaction with the effectiveness of the services offered under the new system, including in the decentralized service centers. Citizens interviewed appreciated especially the opportunity to appear before a judge in person, a situation that was uncommon under the old system, as well as the speed with which routine procedures were being resolved. In this context, while the ICR team has been able to make an informed judgment regarding the substantial achievements in the effectiveness of the Judiciary’s interventions, the lack of information regarding service quality per se leads the team to judge the “quality” aspect of the PDO as moderate. 61. Based on these findings, the ICR observes substantial efficacy in the ability of the Project to achieve the objectives premised in the PDO. 3.3 Efficiency Rating: Moderate 62. The costs involved in achieving Project objectives seem reasonable in comparison with recognized norms (“value for money”). All contracts were executed in line with Bank guidelines, and the resulting outputs seem in line with modern standards that would be reasonably expected from similar public works, refurbishments and equipment. The first Bank team, which led the design and the initial stages of implementation, noted that the client’s insistence on purchasing rather than leasing the IT equipment anticipated in the Project may have resulted in unnecessarily large costs for the Judiciary in the long run. Bank experience suggests that particularly in countries where the public sector does not have an efficiency-driven culture of maintenance of equipment, there is a case to be made for leasing rather than purchasing IT equipment for public sector use. While in this particular case the ICR does not see a reason to argue that Project efficiency was greatly compromised, the team feels that this is a valid point that warranted closer consideration during implementation. 63. In the absence of economic rate of return estimates emanating from an Economic and Financial Analysis, the ICR team has attempted to assess the Project’s efficiency through an analysis of cost per unit of output. Component Original Budget End of Project Component I $ 21.510.000 $ 21.501.139 Component II $ 1.730.000 $ 1.807.145 Component III $ 2.670.000 $ 3.242.584 Component IV $ 1.180.000 $ 3.420.955 64. As a proxy for “output,” the ICR takes the average operational savings per court (US$92,000), which was utilized under the efficacy section above, with the assumption that operational savings are desired outputs that signal more efficiency in the court system, a key Project outcome. Considering that the Project impacted 234 courts across six cities, and spent 15 US$29,971,823 in total, average “spending” per court was approximately at $128,000, which is still greater, but not too far from, the average operational savings achieved per court during this period. It should also be noted in this context that Component IV of the Project, which financed project management and is as such more removed from directly influencing Project outcomes, overspent considerably compared to its original budget. Against this backdrop, the ICR rates the Project moderately efficient. 3.4 Justification of Overall Outcome Rating Rating: Moderately satisfactory 65. The ICR rates the Project highly relevant and moderately efficient, and observes substantial efficacy in its ability to achieve the expectations premised in the PDO. Based on the harmonized evaluation criteria for ICR and IEG evaluations, the ICR suggests an overall outcome rating of moderately satisfactory for the Project. 4. Assessment of Risk to Development Outcome Rating: Negligible to low 66. The Judiciary has absorbed all Project activities into its reform agenda. The public works and the equipment purchased under the Project have been fully integrated into the Judiciary’s routine maintenance plans, and the decentralized service centers are becoming a centerpiece of the Judiciary’s access to justice reform strategy. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately satisfactory 67. The ICR finds the detailed level of analysis performed to inform Project preparation highly satisfactory, but observes that the PDO was too broad and originally relied too heavily on the potential successful implementation of the management model. Furthermore, the decision to establish a single PMU that would manage the implementation of the reform efforts supported by the World Bank and the IADB caused significant delays. (b) Quality of Supervision Rating: Moderately satisfactory 68. The Bank was consistently responsive to the needs of the Judiciary in general and the PMU in particular. There is reason to believe, however, that more proactive supervision especially during the first half of the Project may have helped address the initial delays that the Project faced. In the second half of the Project, the restructurings could have benefited from a formal revision of Project Components, in order to formalize moving away from the emphasis on the institution of a management model, towards a well-structured support for the Judiciary’s 16 evolving reform agenda. A revision of the Project’s results framework accordingly would have improved the monitoring and evaluation of Project activities. (c) Justification of Rating for Overall Bank Performance Rating: Moderately satisfactory 69. While the Project was conceived based on a highly relevant development objective and stemmed from an extensive analytical foundation, and benefited positively from responsive and focused supervision, overall Bank performance is rated moderately satisfactory largely based on the shortcomings in Project design and original implementation arrangements described above. Furthermore, at Project end, the Project would have been served well by a more effective management of expectations as the Bank went through the internal motions to evaluate the case for the continuation of Project activities. 5.2 Borrower Performance (a) Government Performance Rating: Moderately satisfactory 70. Government commitment to achieving development objectives remained solid throughout Project implementation. The Project benefited from the fact that the broad nature of its PDO coincided well with the reform priorities of the Government and the Judiciary. At the same time, the comprehensive reform agenda that the Judiciary took on during this period proved to be complicated, creating confusion across all stakeholders. The cultural and organizational changes that the reform agenda instigated complicated the Project’s plans to formulate an integrated management model that would align with the reform agenda, causing the Project to struggle to deliver until it reoriented itself to support the transition to the oral system. At the Ministry of Justice, the Project’s midterm review noted that a continuing turnover in officials caused delays in the implementation of the different aspects of the Project’s interventions through this institution. And in the decentralization agenda, it took a long time for the Judiciary to negotiate with other state institutions the locations for the anticipated justice service centers, which caused a delay of about one year. While in general the Government and the Judiciary maintained a positive, collaborative and progressive environment within which reform was unfolded, certain aspects of the reform agenda complicated Project implementation, therefore leading the ICR to rate Government performance during the Project moderately satisfactory. (b) Implementing Agency Performance Rating: Moderately satisfactory 61. Despite the long delay in commencing Project activities, the PMU performed an impressive job gearing up and intensifying efforts to fast-track Project implementation. The PMU leadership was able to position the Project as an important reform agent within the Judiciary, and succeeded to guide the Project through a cost-effective execution of all anticipated activities. On the other hand, in speeding up execution in order to make up for lost time, the PMU may have slightly lost sight of the original design of the Project and the expectations that were built from the outset, especially into the successful implementation of an integrated management model. In this context, the PMU could have been more proactive in instigating a comprehensive 17 restructuring of the Project and its results framework as it sought to align more closely with the changing priorities. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately satisfactory 62. Client ownership remained solid, counterparts were proactive in adjusting to changing circumstances, and the implementing agency was able to execute all planned Project activities and the associated financing in a satisfactory manner. However, certain aspects of the Judiciary’s reform agenda and the PMU’s leadership in reorienting Project activities without seeking a formal, comprehensive restructuring may have contributed to the moderate shortcomings observed in Project implementation. 6. Lessons Learned 63. Lessons from Project experience can be grouped into two themes: operational and technical. 64. From an operational perspective, Project experience confirms the challenges Governments often face when PMUs set out to manage projects financed by more than one institution. Creation of a PMU for one particular project, or ideally reliance on the Government’s own resources for project management, are alternatives that should be evaluated closely based on the nature of the project and the capacity of the Government to execute it. 65. The challenges that the ICR faced in validating the achievement of the PDO based on the results framework confirms the attention that Project preparation needs to pay to establishing a clear causal chain between project objectives, outputs, and results indicators. In this context, the M&E framework would have benefited from a more consistent treatment of measurable outcome indicators. The M&E framework would have been served better by a more robust framework for collecting different levels and layers of case processing data, which would have been good proxy indicators for measuring the timeliness and efficiency of justice services. 66. From a technical perspective, a number of lessons learned emerge. ICR interviews validated the significant potential impact of decentralized service centers on making justice services more accessible to citizens. The decentralization agenda has tremendous potential to alleviate grievances and enhance citizen confidence in the justice sector, all at a reasonable cost that involves largely the establishment of relatively inexpensive infrastructure and the appointment of a qualified judge. In this sense, the Project could have considered placing additional emphasis on the establishment and institutionalization of more decentralized service centers especially during the restructurings. The ICR strongly recommends future operations to explore this theme further. 67. The confusion that justice sector stakeholders, ranging from judges to lawyers and citizens, seem to have experienced during the transition to the oral system confirms that any comprehensive reform initiative should be accompanied by a strong change management and training agenda. In this context, interviewees have pointed out that the training that was done for judges should have been extended to administrative personnel, lawyers, and even citizens. 18 68. Interviews have also suggested that while the initial trainings on conceptual aspects of the oral system had been useful, further guidance on the administrative and procedural aspects of conducting cases under the new system would have been beneficial. The ICR understands that the unified management model that the Judiciary is setting out to establish may serve this need. 69. Lastly, the analyses that the ICR team conducted with the case processing data provided by the Judiciary, as well as interviews with judges in different cities, have shown varying degrees of success in the capacity of each city to absorb the reform agenda. Future interventions could consider including initiatives where different cities learn from each other’s experiences and even assist each other in addressing bottlenecks in reform implementation. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies Neither the Borrower nor the implementing agency has raised substantive issues with regards to the findings outlined in this ICR. (b) Cofinanciers N/A (c) Other partners and stakeholders N/A 19 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Actual/Latest Appraisal Estimate Percentage of Components Estimate (USD (USD millions) Appraisal millions) Component I 11.48 11.51 100.26 Component II 1.73 1.81 104.62 Component III 2.67 3.25 121.72 Component IV 0.81 3.42 422.22 Total Baseline Cost 16.69 20.00 119.83 Physical Contingencies 0.00 0.00 0.00 Price Contingencies 0.00 0.00 0.00 Total Project Costs 16.69 20.00 119.83 Front-end fee PPF 0.00 0.00 .00 Front-end fee IBRD 0.00 0.00 .00 Total Financing Required 16.69 20.00 119.83 (b) Financing Appraisal Actual/Latest Type of Estimate Estimate Percentage of Source of Funds Cofinancing (USD (USD Appraisal millions) millions) Borrower 0.00 0.00 .00 International Bank for Reconstruction 16.69 20.00 119.83 and Development 20 Annex 2. Outputs by Component (a) Strengthening of Judicial Case Management Objective: Strengthen the management capacity of the Judiciary to improve the performance of civil, family and labor courts in Barranquilla, Bogotá, Bucaramanga, Cali, Cartagena and Medellin. Component Activity Product Design of a management • Purchase of TV screens for the model that helps reduce the implementation of the backlog in courts and management model in labor implements the oral courts. (2011) procedure for Circuit level • Purchase of audio and video Civil Courts and equipment for the civil, family implementation of the and labor courtrooms in management model for Barranquilla, Bogota, courts, including the Bucaramanga, Cartagena, Cali execution of civil works and Medellin. (2012 and 2013) for the relocation and • Purchase of technological refurbishment of office equipment to support the spaces in labor, civil and implementation of the oral family courts. procedure in labor, civil and family courts of Cali, which includes 2 servers, 1 storage system, 1 tape library and 1 rack. (2012) • Purchase of desktop computers, printers, scanners, laptops and Efficiency in the justice software licenses to the support services management the implementation of the oral procedure in civil, family and labor courts of Barranquilla, Bogota, Bucaramanga, Cartagena, Cali and Medellin. (2012 and 2013) • Consultancy for the diagnosis of the physical and technological infrastructure needs for the implementation of the oral procedure in labor, civil and family courts of Barranquilla, Cartagena, Cali, Bogotá, Medellin and Bucaramanga. (2011) • Civil works for civil, labor and family courts, which included the setting up of 67 courtrooms, 14 courts and 2 service centers. (2011 and 2012) • Civil works for the Hernando 21 Morales building, which included the setting up of 40 courtrooms and 1 services center. (2013) • Consultancy for the review and adjustment of the management models for the civil, labor and family jurisdictions oriented to the implementation of oral procedure nationwide, which includes the strengthening of the human talent capacities, job profiles, procedures, organization and nature of judicial operators to strengthen an efficient service providing planning, execution an control tools. (2011) • Consultancy for an organizational diagnostic of the management model of the administrative chamber of the Superior Council of Judicature and the Executive Director of judicial administration. (2013) Review of court’s • Consultancy to review the workload to identify active statistics collected by the and inactive cases and Judiciary and design of new reassignment of inactive statistical parameters for the cumulative cases. decision making process, as well as the criteria and standards for its management. (2013) Design and implementation • Consultancy to adjust the of automated procedures management model of the for despachos comisorios despachos comisorios in the civil in civil courts of Bogota, jurisdiction of Bogota. (2011) including execution of civil • Purchase of the required works to refurbish office technological components for the spaces in existing implementation of the installations. management model of the despachos comisorios. (2011) • Establishment of • Consultancy to design an administrative mechanisms accountability model for the for Judiciary’s staff Judiciary. (2013) accountability, addressed to strengthen the role of users in the improvement of justice services (b) Development of Judicial Human Resources 22 Objective: Develop Judiciary’s HR based on periodic performance evaluations and capacity building. Component Activity Product Design of a new • Consultancy to design the administrative management methodology to measure the model for the Judicial impact of the existing training School Rodrigo Lara programs of the EJRLB through Bonilla (EJRLB), based on the creation of labor, civil and a review of the current family observatories. (2011) model. • Consultancy to design, develop and implement a software for the management of an impact evaluation observatory for the existing training programs of the EJRLB. (2013) • Consultancy to adjust the administrative, organizational and management models of the EJRLB. (2013) Implementation of the new • Purchase of desktop computers, administrative management printers, scanners, laptops and model through the software licenses to the support development of: (i) virtual the implementation of the EJRLB Judiciary’s staff training methodologies for training new administrative management and its technical content model. (2013) (including the execution of • Purchase of air conditioning and civil works for the work stations for the EJRLB. refurbishment of the (2013) video/conference room of • Purchase of the technological the EJRLB); and (ii) a new solution to guarantee the wifi applied legal research supply for all the EJRLB. (2013) program for judges. • Consultancy for the design, development and implementation of a virtual training software for the entry to the national training plan of the EJRLB. (2011) Design and implementation • Consultancy to design, develop of specific training and implement an education programs as a support of the management software. (2012) new administrative • Consultancy to design 3 training management model, modules applied to oral procedure focused in simplifying and in labor, family and civil courts. efficiently manage oral (2012) procedures. Design and implement the • Consultancy to design and mechanisms and standard implement a connectivity Judiciary’s staff parameters for the mechanism between SINEJ, performance evaluation performance evaluation of SIERJU and Justicia siglo XXI judicial staff. systems. (2011) 23 • Consultancy to design and implement a judicial management monitoring system. (2011) • Consultancy for the implementation of performance evaluation mechanisms of the judicial staff focused in quality and productivity. (2011) • Purchase of technological equipment to support the implementation of the judicial management monitoring system, which include 11 servers, 1 storage system, 1 tape library, 1 rack and 2 TV screens. (2012) (c) Facilitating Access to Justice Services Objective: Facilitate access to justice services for vulnerable population through the generation and dissemination of reliable information, geographical coverage expansion, and strengthening of conciliation services. Product • Purchase of desktop computers, printers, scanners, laptops and software licenses to the support the implementation of juzgados de pequeñas causas. (2013) • Purchase of air conditioning and work stations for the juzgados de pequeñas causas in Ciudad Bolivar. (2013) • Adaptation and functioning of 12 juzgados de pequeñas causas in Cartagena (1), Cali (3), Barranquilla (2), Medellin (4) and Bucaramanga (2). (2013) 24 Annex 3. Economic and Financial Analysis The Economic and Financial Analysis included in the PAD makes a reference to the Project’s potential to contribute positively to the governance agenda by “providing the powerful deterrents of permanent transparency and accountability mechanisms.” It also points out the Project’s potential impact on doing business by increasing business confidence and reducing the time and cost of contract enforcement. While the links between the regulatory environment and the business climate is clear, the ICR team has neither found a perceivably direct link between Project interventions and improvements in the business environment, nor felt the necessity to do so. The team has observed that the primary impact of this Project has been an increase in the overall efficiency, timeliness, effectiveness, and quality of conflict resolution services. In this sense, the more relevant measure of the return on the investment would be the efficiency gains in courtroom operations, which, as the PAD claimed, would allow for a more cost-effective allocation of resources. The PAD stated that justice sector investments are not “subject to conventional financial analysis since a substantial part of their outcomes are difficult or impossible to measure, and are more related to critical social values like legitimacy of, and confidence in, State institutions.” While this statement is largely valid, there have been considerable advances in recent years in attempting to quantify the impact of policy interventions in institutional reform. Colombia’s own process of collecting and analyzing rigorously case management data is an example of an attempt to understand and inform policy and its impacts on citizen welfare in the justice sector. While an impact analysis of this Average Savings in Case Processing nature has been beyond the capacity and scope of this ICR The implementation of the oral system started in effort, the team has attempted to Colombia in January 2013. Of the six cities that the calculate the sample, average Project focused on, two had started implementing the oral savings achieved in the system system fully since that date (Cali and Barranquilla), while during this period, through its implementation is still underway in other cities. To anecdotal evidence from identify the sample average savings per court during this interviews with judges and other period, the team used as basis the civil circuit courts in Judiciary stakeholders (see Text Cali. The team first calculated the average cost of processing a case before the introduction of the oral Box). The fact that the transition system, dividing the total annual operating costs of one to the oral system has in many civil circuit court (COP$432,458,204 or approximately cases reduced average case US$180,000) by the average final case inventory from processing times from two years 2009 to 2012 (928), yielding an approximate average cost to a few months in itself is a of COP$466,011 or US$195 per case. The team then substantial, promising outcome. calculates the difference between the 2013 and 2014 final Furthermore, the establishment of inventories to find the additional number of cases that decentralized service centers, were processed after the introduction of the oral system. which constitute relatively The case inventory decreased by 474 cases during this inexpensive investments in period, signaling savings around US$92,000 per court per While simplistic in its methodology, these infrastructure, are already year. calculations point to significant potential savings, reducing significantly the legal considering that the Project impacted 234 courts across fees and the transportation costs six cities during their transition to the oral system. that citizens would have had to incur under the old system, where they would have had to perform even the most routine transactions by traveling to a city and finding legal intermediaries to help them process their paperwork. 25 Annex 4. Bank Lending and Implementation Support/Supervision Processes ((a) Task Team members Responsibility/ Names Title Unit Specialty Lending Jorge Luis Silva Mendez Public Sector Specialist GGODR TTL Maria Gonzalez de Asis Senior Public Sector Specialist LCSPS TTL David F. Varela Senior Public Sector Specialist LCSPS TTL Santiago Rene Torres Procurement Specialist GGODR Sandra Ximena Enciso Gaitan Procurement Specialist GGODR Luz A. Zeron Financial Management Specialist GGODR John Factora Senior Operations Officer LCCCO Alonso Zarzar Casis Safeguards Specialist GSURR Jairo A. Arboleda Safeguards Specialist GSURR Camilo Andres Avila Ceballos ET Consultant GGODR Eguiar Lizundia Consultant GGODR Francisco Lazzaro Research Analyst GGODR Diomedes Berroa Senior Operations Officer OPSOR Carmen Machicado Operations Officer SEGM2 Supervision/ICR Onur Erdem Public Sector Specialist GGODR Jorge Luis Silva Mendez Public Sector Specialist GGODR TTL Maria Gonzalez de Asis Senior Public Sector Specialist LCSPS TTL David F. Varela Senior Public Sector Specialist LCSPS TTL Eguiar Lizundia Consultant GGODR Fernando Monge Counsel GGODR Oscar A. Florez Consultant LCSPS Javier Said Yueng Consultant LCSPS Camilo Andres Avila Ceballos ET Consultant GGODR Francisco Lazzaro Research Analyst GGODR (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY04 9 75.25 FY05 10 75.58 FY06 9 91.93 FY07 19 112.44 FY08 10 146.24 FY09 27 133.27 FY10 13 101.72 Total: 97 736.43 26 Supervision/ICR FY10 7 73.86 FY11 13 162.95 FY12 17 109.25 FY13 35 225.44 FY14 13 87.78 FY15 20 90.49 Total: 105 749.77 27 Annex 5. Beneficiary Survey Results N/A 28 Annex 6. Stakeholder Workshop Report and Results N/A 29 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Following a period of preparation during which the Bank team worked in close collaboration with the counterparts assigned by the Borrower for the ICR process, the draft ICR was sent to the Client for review on June 8, 2015. The Bank received a response from the Borrower on June 12, 2015. The Borrower noted that the Government had no substantive comments, and that the ICR presented an objective picture of Project performance. 30 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders N/A 31 Annex 9. List of Supporting Documents Country Partnership Strategy for the Republic of Colombia for the Period FY08-FY11, March 4, 2008 Country Partnership Strategy for the Republic of Colombia for the Period FY2012-2016, June 12, 2011 Implementation Supervision Reports, Colombia Justice Services Strengthening Project. Interim Financial Reports, Colombia Justice Services Strengthening Project. Interim Progress Reports, Colombia Justice Services Strengthening Project. Project Appraisal Document on a Proposed Adaptable Program Loan in the Amount of US$ 20 Million to the Republic of Colombia for a Justice Services Strengthening Project – Phase I (APL1) in Support of the First Phase of the Justice Services Strengthening Program, November 6, 2009 Plan Nacional de Desarrollo 2006 – 2010 “Estado Comunitario: desarrollo para todos.” Plan Nacional de Desarrollo 2014-2018 “Todos Por Un Nuevo País.” 32