Econometric Analysis of Framework Agreements in Brazil and Colombia June, 2021 Copyright© 2021 The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org Disclaimer This work is a product of the staff of the World Bank. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the World Bank, its Board of Executive Directors, or the governments they represent. This report is not meant for criticizing working of any government, judiciary, institution, or agency. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to: Office of the Publisher The World Bank 1818 H Street NW Washington, DC 20433 USA Fax: 202-522-2422 Email: pubrights@worldbank.org i|P a g e Acknowledgements This study has been carried out by a World Bank team led by Shanker Lal (Lead Procurement Specialist), the Task Leader. Other team members were Serena Sara Daniela Cocciolo (Young Professional), Sushmita Samaddar (Extended Term Consultant), and Danilo Pereira de Carvalho (Senior Procurement Specialist). This report was prepared under the overall guidance of Elmas Arisoy, Procurement Manager (South Asia), and Vinay Sharma, Global Director (Procurement), World Bank. Comments and inputs at various stages of preparation from the following World Bank persons are gratefully acknowledged: Felipe Goya (Procurement Manager, Africa West), Chris Browne (Lead Procurement Specialist), Shaun Moss (Lead Procurement Specialist), S.M. Quamrul Hasan (Lead Procurement Specialist), Jurgen Rene Blum (Senior Public Sector Specialist), Vincenzo Di Maro (Senior Economist), Francisco Rodriguez (Senior Procurement Specialist) and colleagues from India Procurement Team. Additional research and data collection support was provided by Short Term Consultants namely Leandro Justino Pereira Veloso, Maria Arnal Canudo and Stephen Shisoka Okiya. Administrative support from Payal Malik Madan (Program Assistant) is also acknowledged. The team is also thankful to Global Procurement Partnership Multi Donor Trust Fund (GPP MDTF) contributed by the following donors for financing this study. ii | P a g e Table of Contents Acronyms and Abbreviations…………………………………………………………………………………….…….. 1 Executive Summary…………………………………………………………………………..…………. 2 Background………………………………………………………………………………….………………….…… 2 Sample for Analysis………………………………………………………………………………….……….…… 2 Major Findings……………………………………………………………………………………….……….…… 3 Lessons Learned………………………………………………………………………………………………… 4 Chapter 1: Introduction……………………………………..…………………………….…………… 6 1.1 Background and Objectives of the Study………………………………….………………………… 6 1.2 Approach and Methodology………………………………….…………………………………………… 7 1.3 Literature Review………………………………….………………………………………………………… 8 1.4 Limitations of the Study…………………………………………………………………………………… 10 1.5 Structure of the Report……………………………………………………………………………………… 11 Chapter 2: Data Analytics - Brazil ………………………………………………………………… 12 2.1 Introduction…………………………………………………………………………………………………… 12 2.2 Data Description and Sources…………………………………………………………………………… 13 2.3 Research Questions and Methodology ……………………………………………………………… 17 2.4 Procurement Data Analysis……………………………………………………………………………… 19 2.5 Conclusions……………………………………………………………………………………………………… 29 Chapter 3: Data Analytics: Colombia………………………………………..…………………… 32 3.1 Introduction…………………………………………………………………………………………………… 32 3.2 Data Description and Sources…………………………………………………………………………… 33 3.3 Research Topics ……………………………………………………………………………………………… 36 3.4 Procurement Data Analysis……………………………………………………………………………… 37 3.5 Conclusions……………………………………………………………………………………………………… 48 Annexures to the Report..…………………….………………………………………….…………… 50 Annexure-A: A brief introduction to Regression Analysis ………………………………………… 51 Annexure-B: Supporting Data/Information - Brazil………………………………………………… 52 Annexure-C: Supporting Data/Information - Colombia…………………………………………… 55 iii | P a g e Acronyms and Abbreviations BRL Brazilian Real CATMAT Catalog of Materials CATSER Catalog of Services CCE Colombia Compra Eficiente COP Colombian Pesos FA Framework Agreement FC Framework Contract ID Identification (Number) OECD Organisation for Economic Co-operation and Development OEM Original Equipment Manufacturers RAIS The Regulatory Authority Information System SECOP El Sistema Electrónico para la Contratación Pública SME Small and Medium Enterprise TVEC Tienda Virtual del Estado Colombiano UNSPSC United Nations Standard Products and Services Code VfM Value for Money 1|P a g e Executive Summary Background Procurement of commonly used items is a challenge for government agencies. There are many examples of abuse and failure to achieve value for money in such procurement, in some cases attributable to the relatively unstructured, non-competitive, and non-transparent methods commonly used for the procurement of items of relatively low value. Where more structured, competitive, and transparent procurement processes are used, the time and costs involved are likely to be disproportionate to the price of the items, and may be particularly troublesome in emergency response situations. If the items are repeatedly purchased in one-off fashion, so that the total volume is significant, there may be potential problems like loss of Economy of Scale, loss of efficiency, lower Competition, and no long-term partnership with suppliers. Framework Agreements (FAs) have emerged as a potential solution for above issues. Many countries (particularly in Americas and Europe) have used FAs successfully, though the use of FAs by countries outside these regions is still very low. Hence there is tremendous potential for scaling-up the use of FAs in developing countries. At the same time, the use of FAs is not necessarily as a panacea to solve inefficiencies and integrity gaps in public procurement. In addition, FAs might have, by design, unintended consequences on some dimensions of public procurement, such as participation of SMEs, advantages to incumbent firms and market concentration. Lack of credible data about potential advantages and challenges of FAs is one of the reasons for reluctance to use them. This study tries to fill-up this gap by analyzing the data collected from two countries. Sample for the Study This study uses public procurement data from Brazil and Colombia, two major users of FAs. The subsequent chapters will describe the data used for the analysis, the methodology and the findings. Even though FAs are a significant proportion of total procurement spends of these economies, it is not possible to exactly arrive at monetary values of use of FAs due to federal nature of these countries and non-availability of entire contract award data in public domain. The country contexts, designs of FAs, available data and research questions vary across Brazil and Colombia (Table 1), and therefore the empirical findings are not comparable between these two countries. For each country case, the analysis provides insights on the benefits and costs of using FAs and useful lessons that can be informative for other countries that are considering adopting or strengthening the use of FAs with similar design. Table 1: FA design and available data for the study Parameter Brazil Colombia FA design Entities in charge of establishing Procuring entities, Central Procurement FAs independently or jointly Authority Obligation to use established FAs Yes, for entities that established Yes, only for some entities the FAs 2|P a g e Parameter Brazil Colombia Process to use established FAs Direct orders Direct orders through e- catalog and online platform Available data Data used for this study Public procurement micro-data Public procurement micro- data Coverage of procurement micro- Federal procurement All procurement data Unit prices available Yes No Main comparison for the analysis FA vs. other procurement FA vs. other procurement methods methods Major Findings A. Brazil: • The results from Brazil indicate that FAs are associated with a reduction in unit prices in comparison to other open procurement methods (average saving of about 10%), and these economic savings are primarily explained by the effect of buying in larger quantities. Another factor explaining this result is competition, as the number of bidders is higher under FAs relatively to both other open procurement methods (1 bidder more on average) and closed/limited procurement methods (5 bidders more on average). This empirical evidence indicates that FAs can be effective to increase efficiencies and exploit economies of scale. • Despite increasing competition, the use of FAs leads to higher market concentration i.e. larger share of contracts awarded to smaller number of firms. • On average, the total quantity purchased under FAs in Brazil is 65% of the total quantity initially established in the FAs, and this under-utilization of established FAs may create distortions in firms’ behavior and inefficiencies in the market. • Setting up FAs requires significant efforts (which is reflected in more than double time taken in initial setting of FAs, in comparison to other open procurement methods). • The share of SME firms among bidders is very similar for FAs than for other non-FA methods. However, the probability of awarding the contract to an SME in FA is 2% lower than in non- FA (open) methods. B. Colombia: • The analysis of public procurement data in Colombia shows that the use of FAs by non-obligated entities is more than those of entities obliged by the law to use established FAs. This shows the attractiveness of FAs even among procuring entities that are not obligated to purchases under FA, perhaps due to lower unit prices or convenience of using FA over non-FA methods. • Establishing an FA requires a much longer processing time than establishing a single-order standard contract. For example, in case of motorized cycles it takes more than double the time to establish an FA in comparison to establishing a single-order standard contract, while for non- alcoholic beverages the time taken to establish an FA is five times the time taken to establish a single- order standard contract. 3|P a g e • However, once the number of purchase orders issued under each FA is taken in to account, the average processing time per purchase order (total time involved in setting up the FA divided by the number of purchase orders issued under that FA) is only 0.32 days under an FA in comparison to the 31 days on average to establish a standard contract. This demonstrates the potential of FAs to reduce efforts required for repetitive procurement processes. • The share of the total contracts awarded to SMEs is higher under FA (34%) than under non-FA methods (12%), which shows that, in the context of Colombia, FAs do not appear to discourage or disadvantage SMEs but actually help them. • Issue of market concentration was also noted in Colombia. For most products, the share of the main supplier under FAs is above 60% but below 30% under non-FA methods. This finding is expected, given the nature and design of FAs, but it is relevant to highlight given its potential long- term implications on market structure and competition. Lessons Learned 1. FAs result in better Value for Money (VfM): Even though many factors contribute to VfM 1, cost saving is a major contributor. The results from Brazil clearly show the monetary saving due to use of FA in comparison to non-FA procurement methods. Assuming that other parameters (like quality and sustainability) will be same for commonly used items purchased through FA and non-FA methods, these results support the argument of achieving better VfM through FA. Even though large volume of FA data was available in Colombia, unfortunately corresponding data for non-FA methods was not available. So it was not possible to calculate savings due to use of FAs in Colombia. 2. Procurement under FAs is more efficient than non-FA methods: Based on analysis of data from both Brazil and Colombia, it is quite clear that the process for establishing an FA is longer than establishing a single-order standard contract, but there are clear gains in terms of per-order total processing time through FAs. In addition, the processing times for the purchase of products under an already established FA is much shorter than through stand-alone contracts issued through non-FA methods. 3. Establishing FAs require significant efforts: The results from Brazil and Colombia clearly show that the time taken in setting-up FA is considerable. Not only does this point out to need for keeping adequate lead time before items are required, it also underlines why FAs should have reasonable validity time (for example, FAs in Brazil currently have one-year validity, which might be to be too short). 4. FAs can be extremely useful for emergency situations: Despite the fact that this study did not specifically cover emergency procurement, some characteristics of FAs can make them particularly useful during emergencies. The process of issuing purchase orders under already established FAs in Brazil and Colombia is almost instantaneous (as there is no second stage competition involved). Thus, already established FAs are quite useful for fast-tracking purchases, which is also demonstrated by the large-scale use of FAs in Colombia for procurement of items for COVID-19 response. 5. FAs can generate uncertainties on the exact demand volume for suppliers: By design, FAs establishes umbrella arrangement that binds suppliers to supply goods and services at a fixed price (or within a given price range) but without establishing ex-ante the purchased quantities. This can create uncertainties for firms and risks in planning production and supply capacity. The case of Brazil demonstrates these uncertainties, as the realized demand is often below the expected demand indicated in the tender document, 1As per Government of U.K., Best value for money is defined as the most advantageous combination of cost, quality and sustainability to meet customer requirements (https://www.finance-ni.gov.uk/articles/definition-best-value-money) 4|P a g e but sometimes it can also exceed it. These uncertainties might generate delays in delivery times, as well as reduce the saving potential of FAs if firms take them into account when bidding. 6. FAs may boost competition: The analysis in Brazil shows that the use of FAs leads to higher number of firms participating (by 1 bidder on average) when comparing with other open procurement methods, and by 5 bidders on average when comparing with other closed procurement methods. The team was not able to corroborate this result with the analysis in Colombia due to lack of adequate data. 7. Impact of FAs on SME participation is country-specific: In Colombia, suggestive evidence indicates that SMEs are not less likely to be awarded a contract under FAs than through other non-FA methods. However, the opposite was observed in Brazil, where even though participation of SMEs in FAs is higher than for non-FA methods, the likelihood of an SME winner is lower for FAs than non-FA methods. This may suggest the need to have complementary incentive policies for SME bidders when establishing FAs. 8. The impact of FAs varies across products: FAs are mostly useful for routine and standardized goods, purchased in high quantities and at high frequency. The empirical findings presented in this report are specific to the set of products used for each country case. The analysis for Brazil and Colombia reveals that, even within the same country, the use of FAs generates different effect for different products. This indicates that it is necessary to identify the products best suited for this procurement method and regularly monitor and evaluate the effects generated by the adoption of FAs for each product. 9. Data Availability is key to optimized use and analysis of FAs: The analysis in this report is limited by data constraints and there are several important aspects which the team was not able to investigate due to lack of the necessary data 2. A necessary condition in order to assess the benefits and costs of using FAs is to have rich data on purchases for similar products done both through FA and through non-FAs. This is necessary in order to compare the realized procurement outcomes for similar products purchased through different procurement methods. Another necessary step to further optimize the use of FAs would be to complete a holistic assessment on how FAs are currently implemented and the contract management stage. For example, the available data in Brazil does not contain information on delivery times, on when the FA was signed, or on all procuring entities that participated in the establishment of the FA. Similarly, for Colombia it will be useful to ensure systematic use of IDs for entity, supplier and product across datasets; and that for each FA it is possible to connect information on purchase orders with the supply condition details as reported in the e-catalogs. 2 More details on limitations are covered in next Chapter 5|P a g e Chapter 1 Introduction 1.1 Background and Objectives of the Study Procurement of commonly used items is a challenge for government agencies. If the items are repeatedly purchased so that the total volume is significant, the following missed opportunities also arise: • Loss of Economy of Scale. Multiple separate contracts issued for the same items may result in higher unit rates than those that could have been secured if the procurements were aggregated • Loss of efficiency. Placing separate multiple contracts involves proportionately more transaction time and cost than aggregating volumes into one procurement, and is particularly challenging where procuring entities have limited capacity • Lower Competition: Low-volume contracts may not attract the more competitive suppliers, and Original Equipment Manufacturers (OEM) • No long-term partnership with suppliers: benefits associated with transparency about future procurement (such as better planning by suppliers, value engineering 3, performance improvement etc.) are lost. FAs have emerged as a potential solution for above issues. Many countries (particularly in Americas and Europe) have used FAs successfully, though FAs should not be considered as a silver bullet to address all the above challenges, and there remain risks and constraints in their use. In addition, the use of FAs by countries outside these regions is still very low – confined to a handful of countries elsewhere. Hence there is tremendous potential for scaling-up the use of FA in developing countries in particular. Although FAs have been the subject of academic commentary (see paragraph 1.3 below for example), lack of credible data about advantages and challenges of FA is one of the reasons for reluctance to use them. This is a major hindrance for adoption of FAs by developing countries, which are the Clients of the World Bank. Current study tries to fill-up these gaps by analyzing data collected from two countries and providing a full picture of the potential advantages and disadvantage of FAs which can support Clients of the World Bank to take well informed decisions on whether to adopt or how to strengthen FAs. Commonly-used definitions of FAs include the following: Under Article 33(1), of Directive 2014/24/EU (the European Procurement Directive), “A framework agreement means an agreement between one or more contracting authorities and one or more economic operators, the purpose of which is to establish the terms governing contracts to be awarded during a given period, in particular with regard to price and, where appropriate, the quantity envisaged.” 4 The UNCITRAL Model Law 5, which has worldwide application but is most commonly used in developing countries and countries in transition, has a similar definition: “a framework agreement procedure is……“conducted in two stages: a first stage to select supplier (or suppliers) … to be a party (or parties) to a framework agreement with a procuring entity, and a second stage to award a procurement contract under the framework agreement to a supplier … party to the framework agreement.” The purpose of the FA itself is to 3 Though it may be argued that there is not much scope for value engineering in case of simple common use items 4 The OECD has a similar approach: see https://www.oecd.org/gov/ethics/manual-framework-agreements.pdf 5 UNCITRAL Model Law on Public Procurement (2011), Official Records of the General Assembly, Sixty-sixth Session, Supplement No. 17 (UN document A/66/17), https://uncitral.un.org/en/texts/procurement. The definitions are in Article 2(e). 6|P a g e establish “the terms upon which purchases will be made (or [to establish] the main terms and a mechanism to be used to establish the remaining terms or refine the initially established terms).” The United States’ Federal Acquisition Regulation 6 has a related definition, for its equivalent of FAs: “‘Delivery-order contract’ means a contract for supplies that does not procure or specify a firm quantity of supplies (other than a minimum or maximum quantity) and that provides for the issuance of orders for the delivery of supplies during the period of the contract. ‘Task-order contract’ means a contract for services that does not procure or specify a firm quantity of services (other than a minimum or maximum quantity) and that provides for the issuance of orders for the performance of tasks during the period of the contract.” Another related term used in USA is “Indefinite Delivery, Indefinite Quantity (IDIQ)” Contracts, which are often used for service contracts and architect-engineering services 7. As per World Bank’s Procurement Regulations, 8 an FA is “...an agreement with one or more firms that establishes the terms and conditions that will govern any contract awarded during the term of the Framework Agreement (Call-off Contract)”. Centralized procurement is not necessarily FA, unless they follow the two stage approach mentioned in above definitions. While centralized procurement may offer the benefit of economy of scale, it does not provide other advantages mentioned on previous page. FAs can be set-up and managed using manual or electronic means (for example, e-Catalogues are a type of e- FA as explained in UNCITRAL Guide 9). 1.2 Approach and Methodology The Methodology broadly involves: 1.2.1 Selection of Sample of Countries: Availability of transactional micro-data on FAs and non-FA procurement methods was the main criteria used to select Brazil and Colombia for this study. In these countries FAs are implemented using different designs and approaches, therefore the empirical results reported in this study can offer useful insights and guidelines to a large spectrum of countries. 1.2.2 Literature Review: While the sample of countries was being finalized, the team in parallel conducted extensive review of existing literature (see section 1.3) to understand the work already done as well as to review the research approaches used in other studies on the topic. 1.2.3 Research Approaches used for current Study: Public procurement data in Brazil and Colombia was collected from open sources. This data was analyzed to assess the costs and benefits of using FAs. Regression analysis was used to study the relationship between the 6 Federal Acquisition Regulation, Code of Federal Regulations (CFR) Title 48, Subpart 16.5 - 16.501-1, Definitions. FAR 2005-83/07-02-2015, available at https://www.acquisition.gov/?q=browsefar. 7 https://www.gsa.gov/buying-selling/new-to-gsa-acquisitions/how-to-sell-to-the-government/indefinite-delivery- indefinite-quantity-contracts 8 http://pubdocs.worldbank.org/en/178331533065871195/Procurement-Regulations.pdf 9 https://uncitral.un.org/sites/uncitral.un.org/files/media-documents/uncitral/en/guide-enactment-model-law- public-procurement-e.pdf 7|P a g e use of FAs and procurement outcomes of interest. In case the reader is interested to get a brief introduction on what regression analysis is, please refer to Annexure-A. 1.2.4 Data Sources: (a) Brazil: The data used for this analysis was downloaded from two open data government portals, i) “portal.transparencia” and ii) “compras.dados”. This data is public and it is updated monthly with a broad range of information on tenders and contracts, such as data on suppliers, purchase methods, entities, prices, dates, and quantities. The data is structured in three main datasets: the Procurement Processes Dataset, the Bids Dataset, and the Contracts Dataset. These three datasets contain information of each item included within each procurement process or contracts. The main analysis for Brazil uses these datasets at the item level in order to compare procurement processes and contracts for similar products. Matched employer-employee (RAIS, Regulatory Authority Information System) administrative register has also been used in order to identify SME suppliers based on the firm size. (b) Colombia: The Government of Colombia uses the Colombia Compra Eficiente (CCE) platform for all procurement purchases 10 through any procurement method, including FAs. This is the main source of data for the analysis for Colombia. FAs were established as a contracting modality in 2007. In Colombia, FAs are established through open competition, which selects a few suppliers for each FA and generates an e-catalog that buyers can use in order to issue purchase orders. The analysis for Colombia uses data from the tendering stage, from the e-catalog and from the ordering stage, and it covers purchases made from 2016 through 2020. A feature of FAs in Colombia is that some procuring entities are obligated to purchase through FAs while for others this is voluntary, and this feature is used for some of the empirical analysis. 1.3 Literature Review The first step in this study was a stock-taking exercise of work done earlier. The team identified some reports and research papers on the subject, some of which are listed below: An Evaluation of Performance Improvement within Public Sector Construction Framework Agreements 11 (by Keith Gale, Anglia Ruskin University, 2013): This study from U.K. establishes that there were significant improvements in procurement performance with use of FAs, but that no significant additional costs were incurred. The study also confirmed long-term relationships, financial and non–financial incentives and stronger communication were sociological behavior factors driving performance for FAs. Mid Term Review of the Framework Agreement for SIDA Reviews, Evaluations and Advisory Services on Results Frameworks 12 (SIDA, 2014): This study assesses use of FA by Swedish International Development Cooperation Agency (SIDA) for various consultancy services and concludes that FA may have a negative impact on the long-term availability and quality of Evaluation services to SIDA mainly because of dominance of one service partner, given the diversity of evaluation needs, the quickly developing standards associated with methodologies, tools, and approaches, the plethora of innovations and new tools that emerge internationally, and the access of different evaluators to development evaluations at SIDA, simply implies to much of a constriction on what is a dynamic and quickly changing field and market. This study is important to understand when to not use FA. 10 Procurement purchases are made in Colombian pesos (COP). On November 11th, 2020, 1 COP = 0.000258299 US Dollars 11 https://core.ac.uk/download/pdf/77282988.pdf 12 https://www.transtec.be/website2/pixportal.nsf/pGen00/TRAC-9PUDDV/$FILE/mid-term-review-of-the- framework-agreement-for-sida-reviews-evaluations-and-advisory-services-on-results-frameworks---final-re_3743.pdf 8|P a g e A Contemporary Examination of Framework Agreements 13 (by Stuart Tennant and Scott Fernie, 2010): This paper discusses the changing dynamics of relationship between the Suppliers and Purchasers using FA at the time of 2009 recession, including the tendency of purchasers to use options outside of FA and need for renegotiations of terms and conditions of FA. Influence of Framework Contracting on Procurement Performance of Geothermal Development Company, Kenya 14 (by Waswa V. Nyongesa and Juma Wagoki, 2014): This paper investigates the impact of framework contracting on procurement performance at Kenya Geothermal Development Company. It was established that framework contracting techniques had a moderate positive relationship with procurement performance. Further, it was found out that Early Supplier Involvement and Supplier Relationship Management had moderate to strong positive relationship with procurement performance. It was concluded that implementing framework contracting reduced cost and promoted procurement performance in the Company. The study recommends increased awareness towards adoption of framework contracting as a means of cutting on total procurement costs. Performance of Framework Contract Projects in South Africa 15 (by Calistus Ayegba and David Root, 2020): In this study, the performance of Framework Contracts (FC) and the variables that influences the performance were investigated empirically. A qualitative research methodology was adopted for the study. Data were collected via semi-structured interviews with selected client organizations employing FC in South Africa. From the findings, FC is indicated to have performed spectacularly in dealing with unnecessary cost overruns, time delays, and improvement in the quality of delivery. However, it was reported that the approach work best for repetitive works and when a client has the intention of a long-term programme of work. The results provide substantial evidence that FC can perform better than the conventional approaches to short- term relationships and contracts. Estimating the cost effects of purchasing centralization—Empirical evidence from framework agreements in the public sector 16 (by K. Karjalainen, 2015): This interesting paper discusses the empirical evidence of the specific cost effects (e.g. lower prices and economies of processes) from using centralized FAs in the Finnish government. The empirical study is twofold: the cost effects of centralization are estimated by comparing the costs of centralized and decentralized tendering processes and potential price savings. The estimation of process costs is done by surveying the time spent on the tendering both for the decentralized operating model and the centralized operating model in the government and estimating the cost of those times. Potential price savings are estimated by comparing the central FA prices to market prices in two selected product categories . The results on the price savings provide empirical evidence of the academic consensus that significant volume discounts are available from pooling. Additionally, the size of savings potential in process costs shows that the number of units centralizing their purchasing process does not need to be very high before economies of process already become evident. Empirical Analysis of Procurement Auctions 17 (by Cinthia Konichi Paulo, University of Pennsylvania, 2016): This study empirically analyzes two types of procurement auctions using data from government procurement auctions in Brazil. The first in which suppliers must decide their bids based on expectations about how future market conditions will affect their costs. It was found that suppliers are risk averse and therefore include a risk premium in the prices they bid, which can reach 38% of the price for some goods. The second is jump bidding, when a bidder places a bid that is larger than necessary to outbid the current winning bid. Models that explain this type of behavior say that jump bidding arise as a signaling strategy to communicate strength to competitors. Based on analysis of data, it was found that winners place smaller jumps on average. 13https://www.arcom.ac.uk/-docs/proceedings/ar2010-0685-0694_Tennant_and_Fernie.pdf 14 https://www.ijsr.net/archive/v4i10/SUB159243.pdf 15https://www.matec-conferences.org/articles/matecconf/pdf/2020/08/matecconf_eppm2018_02010.pdf 16Journal of Purchasing & Supply Management 17 (2011 ) 87-97 17 https://repository.upenn.edu/cgi/viewcontent.cgi?article=3606&context=edissertations 9|P a g e The Influence of Framework Agreement System to the Procurement Process Cost: A Case of Government Procurement Services Agency (GPSA) 18 (by Msongole Emmanuel M., 2013): This research from Tanzania found that FA System for Common use Items and Services minimizes costs as well as shorten delivery time of goods and services to both procuring entities and suppliers. However involvement of too many levels of approvals and clearances in FA System for Common use Item and Services may lead to delay in tendering procurement process and contract award by GPSA. Framework Agreements in Procurement: An Auction Model and Design Recommendations14 (by Yonatan Gur, Lijian Lu, Gabriel Y. Weintraub, 2017): The study provides a formal understanding of FAs, with a particular focus on the cost uncertainty bidders face over the FA time horizon. The study introduced a novel model to carry out analysis on FAs especially the frequently used ones in the world. The study established that FAs have gained popularity due to the fact that they obviate administrative costs of having to run first-price auctions repeatedly and they carry out a screening of the more economical suppliers. Pooled procurement of drugs in low and middle-income countries15 (by Pierre Dubois, Yassine Lefouili and Stephane Straub): This study evaluated how a centralized procurement system influenced the price of drugs in low- and middle-income countries. The study established that lower prices were achieved from using a centralized system in the public sector. However higher concentration on the supply side led to smaller induced price reduction. Pre-purchasing with option contract and coordination in a relief supply chain16 (by Xihui Wang, Feng Li, Liang, Zhimin Huang and Allan Ashley, 2015): The study determined that in humanitarian relief operations, purchasing supplies in advance with an option contract from a supplier far outweighs instant purchasing based on a return policy. Further, the study argued that in an option contract, the distribution of profit is more acceptable to the supplier and the buyer. The study proposes a purchasing model that solves the dilemma arising from instant or pre-purchasing in humanitarian operations management. Active and Passive Waste in Government Spending: Evidence from a Policy Experiment, American Economic Review 19 (by Oriana Bandiera, Andrea Prat, Tommaso Valletti): This paper studies the use of FAs in Italy. The empirical findings show that procuring entities that higher prices when purchasing through FAs is not feasible are more likely to use FAs when they are given the chance, highlighting that procuring entities self-select into using or not FAs. These results highlight the importance of correctly interpreting estimates from observational data in absence of an Impact Evaluation Design, given self- selection and other biases. 1.4 Limitations of the Study For Brazil and Colombia, data available in the public domain was used. These two country case studies are not necessarily representative for other countries. However, for each country case the analysis provides insights on the benefits and costs of using FAs and useful lessons that can be informative for other countries that are considering adopting or strengthening the use of FAs. Another important limitation while comparing performance of FA with non-FA procurement method is the purchase of similar items under both FA and Non-FA methods. Once again this was not always possible as some countries use only FA for procurement of some items, while other use only non-FA methods. However, to ensure rigor in the estimates of price savings in cases where the data was comprehensive, the team added 18 http://scholar.mzumbe.ac.tz/bitstream/handle/11192/788/MSc_PSCM_Msongole%20Emmanuel%20M._2013.pdf?se quence=1 14 https://pubsonline.informs.org/doi/10.1287/msom.2017.0623 15 https://www.cgdev.org/publication/pooled-procurement-drugs-low-and-middle-income-countries 16 https://ideas.repec.org/a/eee/proeco/v167y2015icp170-176.html 19 https://www.aeaweb.org/articles?id=10.1257/aer.99.4.1278 10 | P a g e product and buyer level fixed effects to be able to compare the same items between different procurement methods. Specifically, the team used this method in the unit price analysis for Brazil but was not able to achieve the same for Colombia due to data limitations. Even when the data and the regression model allow to estimate the most conservative and reliable models, it is still not possible to fully claim causality of the estimated correlations and relationships in the data. By design, this report uses observational data and it does not rely on an Impact Evaluation design. This means that the empirical findings do not allow to conclude that FA causes the observed correlations, as they might be driven by selection bias or omitted variable bias. Each chapter discusses the identifying assumptions that would allow a stronger interpretation of the empirical results. 1.5 Structure of the Report Details are presented Chapter-wise in this report. Chapters 2 and 3 cover data analytics from Brazil and Colombia respectively. Additionally, following Annexures are part of this report: Annexure-A: A brief introduction to regression analysis Annexure-B: Supporting Data/Information - Brazil Annexure-C: Supporting Data/Information - Colombia 11 | P a g e Chapter 2 Data Analytics: Brazil 2.1 Introduction In this Chapter, the procurement methods have been categorized into three main categories depending upon the regulations and restrictions on firm participation: • Open methods: Competitive Bidding (concorrência pública) and Reverse Auction (pregão). • Closed/Limited methods: Contest (concurso público), Invited Bidding (Convite), Unenforceable Biddings (inexigibilidade de licitacao), direct purchase (dispensa de licitação), and restricted bidding (tomada de precos). • FA: Framework Agreement (pregão registro de preços). Brief description of these methods is included in Annexure-B. FAs are called “price registrations” in Brazil, and for FAs, suppliers are selected through a competitive process and the participating firm that offers the lowest price on procurement items wins the FA and becomes the provider of those items for a fixed period of time up to a year. An FA can be established by one contracting entity or jointly by a group of entities that pooled their procurement demands. Within the limits established in the FAs, the procuring entities have the autonomy to determine the quantity to be purchased and the number of orders they issue within an established FA during the agreement period, and it is even possible for procuring entities to not place any order. However, procuring entities that established an FA for a group of products, cannot use other procurement methods in order to procure those products. Procuring entities that did not participate in the establishment of an FA are allowed to purchase under that FA, as long as the total purchased quantities remain below the legal maximum. 20 This study covers the years 2013-2019. Table 2.1 presents the number of purchases and contracting volumes by procurement methods in this period, and Table 2.2 the main characteristics of bidding processes by procurement method. Between 2013 and 2019; 834,736 bidding processes were initiated through Direct Purchases and 136,766 for the establishment of FAs. In 2019, 10% of purchases were administered through FAs, corresponding to 18% of the total contracting volume. Closed/Limited methods are the largest category representing 81% of purchases and 45% of the total contracting volume in 2019. With an average of 21 bidders per bidding process, FAs are the procurement method with the highest participation of firms. In comparison, open methods have on average 13 bidders per tender and direct purchases have 2 bidders on average per tender. At the same time, the number of different items procured within the same bidding process is on average 47 for FAs, much larger than for other methods, both open and closed/limited. This report uses publicly available procurement data for federal entities that was downloaded from two open data government portals; “portal.transparencia” and “compras.dados”, in order to evaluate the use of FAs in Brazil. The analysis in this chapter relies upon publicly available data. The empirical analysis compares FAs with other open methods (reverse auctions) and closed/limited methods (invited biddings, direct purchases, and restricted biddings). In order to compare purchases across procurement methods that are as similar as possible, the empirical analysis focuses only on goods and it controls for a series of process, product, procuring entity and firm characteristics. 20 For each FA, the umbrella agreement established the maximum quantity that each participating entity can purchase within the FA for each product. Additional entities can also purchase through the FA, for a maximum of five times the total established ex-ante for the participating entities. 12 | P a g e Table 2.1: Volume and share of various procurement methods Proportion of Total Number of Purchases Proportion of Total Contracting Volume Volume Closed/Limited Closed/Limited BRL Years Open methods Open methods Methods Methods (billions ) 21 FA non-FA FA non-FA 2013 8.40% 12.00% 79.60% 33.10% 22 28.50% 38.30% 37.60 2014 8.90% 12.20% 78.90% 13.60% 25.10% 61.30% 36.62 2015 11.00% 11.70% 77.40% 14.10% 25.10% 60.80% 29.01 2016 11.20% 11.90% 77.00% 18.10% 21.20% 60.70% 45.35 2017 11.80% 12.10% 76.10% 17.70% 27.40% 54.90% 32.32 2018 10.60% 10.10% 79.20% 22.90% 29.10% 48.00% 32.47 2019 10.10% 8.90% 81.00% 17.60% 37.50% 44.90% 22.04 Source: Contracts dataset Table 2.2 :Characteristics of Procurement Methods in Brazil No. of processes average number average average number Purchase method type method per method of bidders per number of of items per [2013,2019] bidding process winners bidding process Competitive Bidding Open 3,609 1.09 1.09 23 2.49 Reverse Auction Open 142,907 12.73 2.05 10.44 FA FA 136,766 20.91 6.24 46.95 Contest Closed/limited 140 9.33 9.33 10.34 Invited Bidding Closed/limited 1,197 1.06 1.06 2.29 Direct Purchase Closed/limited 834,736 2.11 1.12 2.89 Ineligibilities Closed/limited 162,207 1.76 1.76 2.43 restricted bidding Closed/limited 4,990 1.04 1.04 1.44 Source: Procurement processes dataset 2.2 Data Description and Sources The data used for this analysis was downloaded from two open data government portals, i) “portal.transparencia” and ii) “compras.dados”. This data is public and it is updated monthly with a broad range of information on tenders and contracts, such as data on suppliers, purchase methods, entities, prices, dates, and quantities. The data is structured in three main datasets: the Procurement Processes Dataset, the Bids Dataset, and the Contracts Dataset. These three datasets contain information of each item included within each procurement process or contracts, and they are described in more details in the next sections. The main analysis in this report uses these datasets at the item level in order to compare procurement processes and contracts for similar products. The team also used the matched employer-employee (RAIS) administrative register in order to identify SME suppliers based on the firm size. 24 21 The average exchange rate in 2013 was 1 USD = 2.15 BRL 22 For the processes started in 2013, there is an exceptional purchase of aeronautic gas in the value of 5.6 billion reais using FA. This single contract explains the higher share of the total contracting volume through Framework Agreements in 2013. 23 For Competitive Biddings, the data include only information about the winners, therefore making the number of winners the same as the number of bidders. It is unclear whether this correctly represent the bidding process for Competitive Biddings or it is an error in the data registration system. Since Competitive Biddings are not used for the main analysis, there is no risk that potential data errors drive the main results. 24 Firm size is calculated according the number of employees in the establishment, following the categorization in the Labor yearbook 2013 by Sebrae & Dieese. Annexure-B provides the categorization of firms in micro, small, medium, and large enterprises based on firm size. SMEs are defined as micro, small or medium enterprises. 13 | P a g e 2.2.1 Procurement Processes Dataset and Bids Dataset The Procurement Processes Dataset includes information on the bidding processes and the purchased items, and it is possible that multiple items are procured within the same procurement process. The Bids Dataset contains information about all bids received for each item within each procurement process. 25 Participants to a procurement process are not obliged to submit a bid for each item included in the tender, and instead they can choose which item to submit a bid for. Figure 2.1 illustrates the relationship between the bidding process, the items procured within each process and the participants for each process- item. By matching the Procurement Processes Dataset and the Bids Dataset, both at item level, the team obtained an extensive dataset of approximately 50 million observations. Figure 2.1: Bidding process illustration 2.2.2 Contracts Dataset The Contracts dataset includes information about the contracts signed within each procurement process. A procurement process can generate multiple contracts in case different suppliers are selected for different items, but it can also generate no contract. By linking the Procurement Processes Dataset and the Contracts Dataset the team was able to verify the number of contracts associated to each tender and the share of procurement processes that result into a contract. Table 2.3 shows that only 12.6% of FA processes generated at least one contract, which is the lowest proportion within open methods. The share of bidding processes that generated at least one contract is also particularly low for direct purchases (4%), despite the majority of the contracting volume is procured through direct purchases. Only for competitive biddings, invited biddings and restricted biddings the majority of bidding processes generated at least one contract, but the purchase methods represent a small share of the total procurement processes and contracts in Brazil. We validated the data extensively before drawing these conclusions, but various conversations with procurement experts with knowledge of the Brazil system confirmed that this is an expected feature in this context. 25Except for Competitive Biddings, for which the Bids Dataset only include participating firms that were awarded a contract. 14 | P a g e Table 2.3 : Bidding to contract award Purchase Method Number of Number of Proportion of processes processes with processes that result contracts in a contract Competitive Bidding (Concorrencia 3608 2898 80.3% publica) Invited Bidding (Convite) 1197 756 63.2% Direct Purchase (dispensa de licitacao) 834013 29894 3.6% Ineligibilities (inexigibilidade de 162207 24236 14.9% licitacao) Reverse Auction (pregao) 133326 45579 34.2% FA (pregao-registro de preco) 135958 17149 12.6% restricted bidding (tomada de precos) 4990 4458 89.3% Source: Procurement processes and contracts dataset. 2.2.3. Product categories The procurement processes dataset and the contracts dataset contain detailed information on purchased items. Items are classified using the CATSER catalog for service, and the CATMAT catalog for goods. 26 Using these classifications, the team categorized items in goods, services, and works. Table 2.4 shows the contracting volume by purchase method and product category. While FA has a large volume of purchase for goods and services, the other open methods are more common for service and works projects. Table 2.4 : Goods volume (in BRL billions) Volume Volume Volume Methods goods services works FA 29.67 14.69 1.76 Open 5.95 40.38 17.06 Closed/Limited 56.83 63.25 5.02 Source: Contract dataset at item level. Figure 2.2 presents the ten goods with largest contracting volumes and the share of contracting volume procured through FAs, other open methods, and closed/limited methods for each of them. FAs are best suited for products with large contracting volume, and indeed Figure 2.2 shows that for almost all of these top 10 products purchased at least 40 times for FAs, other open methods and closed/limited method, FAs is the method representing the larger contracting volume share. 26Goods are detailed on 6 digits levels, and services/works 5 digits level. Both classifications are aggregable by 2, 3, and 4 digits level. The main analysis is on most disaggregated level. CATMAT is the Catalog of Materials, and CATSER is the Catalog of Services. 15 | P a g e Figure 2.2: Proportion of Total Contracting Volume on top 10 items by procurement method Source: Contracts dataset at item level; restricted to high frequency products in FAs, other open methods, and closed/limited method Figure 2.3: Top 20 products with highest proportion of FA Source: Contracts dataset at item level; restricted to high frequency products in FAs, other open methods, and closed/limited method 2.2.4. Use of FAs This section provides an overview on how FAs are used in practice. On average, there are 2,127 procuring entities that purchase through an established FA and 378,793 purchases completed within the umbrella contract. On average, the total quantity purchased under FAs is 65% of the total quantity initially established in the umbrella contract. The figure below shows the log of realized total quality by items over log of expected quantity by item. This figure signifies the relationship between the total quantity that was ultimately purchased under the FA and the expected quantity of purchase as stated in the FA. As seen in the figure, in 49% of cases the total quantity purchased is lower than estimated in the FA umbrella contract (Figure 2.4), but there are also 4.44% of cases for which it is higher. Given that participating entities cannot purchase more 16 | P a g e that the estimated amounts indicated in the umbrella contract, these cases can be explained by additional procuring entities that purchase through the FA even if they did not participate in establishing it. Figure 2.4: Total expected quantity by item over total effective purchase for FA. Source: Contracts dataset at item level, only goods. Only FAs. 2.3 Research Questions and Methodology The main analysis focuses on the comparison between FAs and other procurement methods on key procurement outcomes. Table 2.5 summarizes the research questions explored in this chapter and the outcome indicators used to answer each one. Table 2.1 : Research question summary Research Question Outcome Indicators 1. How do unit prices change when FAs are 1.1 Logarithm of unit price used? 2. How does the processing time change when 2.1 Decision period (number of months between FAs are used? bids opening and award publication) 3. Does competition change when FAs are used? 3.1 Number of bidders 3.2 Share of SME bidders 4. Do the characteristics of a winner change 4.1 SME status of the supplier awarded the when FAs are used? contract Table 2.6 presents the averages for outcomes of interest by procurement method. FAs have on average lower unit prices, longer time between publication and signature of the first contract, and higher participation of small/micro/medium companies (SMEs). These results provide a first description and overview of the main outcomes of interests, but they cannot provide solid insights on the potential impact of using FAs relative to other procurement methods. For each purchase, the choice of the procurement method is driven by the procurement demand, product and entity characteristics and other considerations on a case-by-case basis. 17 | P a g e For this reason, in this study the team relied on regression analysis in order to compare FAs with other procurement methods for purchases that are as similar as possible except for the choice of the procurement method. Table 2.6 Outcomes of interests across procurement methods Outcome FA Open Closed/Limited Average unit price of item (log) 3.94 4.33 2.43 Average Decision Time (months) 1.65 0.60 1.70 Share of micro/small/medium firms 0.88 0.88 0.83 Average number of bidders 2.07 1.00 0.64 Source: Contracts dataset at item level, only goods. Only FAs, reverse auctions, invited biddings, direct purchases and restricted biddings. In order to compare items purchased across procurement methods that are as similar as possible, the empirical analysis focuses only on goods and FAs are compared only against reverse auctions, invited biddings, direct purchases and restricted biddings. The team excluded from the analysis Competitive Biddings, Unenforceable Biddings, and Contests because these methods are used for specific types of procurement and not comparable with purchases done through FAs. Competitive Biddings are used for the purchase of high price items (above 3.3 million BRL), while FAs are typically used for standardized and common items. Ineligibilities are used when there is only one supplier in the market and therefore for very specific type of products and contracts. Contests are not used for goods, but only for highly specific contracts such as for art projects. The empirical strategy for this study relies on linear regression models with multiple levels of fixed effects in order to identify highly comparable purchases. In case the reader is interested to get a brief introduction on what regression analysis is, please refer to Annexure-A. The team used two separate regression models in order to compare (i) FAs vs. open methods and (ii) FAs vs. closed/limited methods. In both cases, the regression model is: = + + + + Ɛ where FA is an indicator for FA purchases/processes and the comparison group consists of open methods or closed/limited methods separately. The regression model is estimated on the Contracts Dataset, except for the analysis on the number of bidders and share of SME bidders, which uses the Procurement Processes Dataset matched with the Bids Dataset. Some fixed effects are used for all regression models: year, quarter, and product code. By using these fixed effects, the regression analysis allows us to compare the outcome of interests across procurement methods between purchases for the same product, in the same year and in the same quarter. In order to increase the precision and reliability of the empirical results, the team used following regression models: • Model A: Compare FAs and non-FA methods for the purchase of the same product by the same entity. This is achieved by using fixed effects for the pairs product-procuring entity. • Model B: Compare FAs and non-FA methods for the purchase of the same product from the same firm, while keeping into account contact characteristics of the procuring entity. This is achieved by using fixed effects for the pairs product-firm and procuring entity fixed effects. These models are used to estimate the differences in procurement outcomes when comparing FA vs. Open and Closed/Limited methods. These models ensure that FA vs. open method and FA vs. closed/limited method were compared for similar purchases. By showing results across these models, the team also ensured that the main results are robust across different specifications. 18 | P a g e These regression models are the most conservative that we could estimate given the available data, in absence of an Impact Evaluation design. However, the regression coefficients should not be interpreted as the causal effect of using FAs. Model A compares FAs and non-FA methods for the purchase of the same product by the same entity, and therefore it is not confounded by selection bias or omitted variable bias at entity-product level. However, selection bias and omitted variable bias might happen across contracts even for the same product by the same entity, for example if the choice of establishing FAs and integrity risks vary throughout the year. Model B compares FAs and non-FA methods for the purchase of the same product from the same firm, but this model this model should be interpreted with caution because firm participation and winner selection can also be a consequence of the procurement method used. Table 2.7 provides information on the samples used when comparing FAs with open methods and closed/limited methods, such as the number of items, goods, services, works, bids process, entities, contracts, and the total volume for each sample. For instance, when comparing FA and Open methods, the team used 22,638 distinct products, and representing BRL 96.31 billion of purchases. Table 2.7 : Sample descriptions Measures FA vs. Open FA vs. Closed/ methods Limited methods # distinct items ordered 24,213 21,433 # distinct items 3 digits ordered 588 552 # distinct items 2 digits ordered 110 108 # goods 22,638 19,953 # services 1,473 1,373 # works 101 106 # bids process (goods) 18,472 13,958 # entities (goods) 1,817 1,664 # establishments (goods) 11,994 11,269 # establishments x entities (goods) 39,523 32,896 # contract item – FA methods (goods) 378,793 378,793 # contract item – Open methods (goods) 202,004 - # contract item – Closed/Limited (goods) - 77,372 # contract item (goods) 580,797 456,165 total goods values (reais-Billions) 96.31 99.71 Source: Contracts dataset at item level. Only FAs, reverse auctions, invited biddings, direct purchases and restricted biddings. 2.4 Procurement Data Analysis This section discusses the results of comparing key procurement outcomes (unit prices, processing time, winner characteristics, competition, and market share of the top suppliers) between FAs and other procurement methods, focusing only on the procurement of goods. The results indicate that FAs are associated to a reduction in unit prices with respect to closed/limited methods as well as other open methods, and these economic savings are primarily explained by the effect of buying in larger quantities. Competition is also one of the reasons that may explain these results, as the team found that there is higher participation of firms under FAs than other open methods and, especially, closed/limited methods. However, the use of FAs is associated to longer processing times when set-up initially, lower probability that the contract is awarded to SMEs and higher market concentration. 2.4.1 Unit price 19 | P a g e As standard for price analysis and modeling, the team used the logarithm of the unit price as primary outcome of interest. 27 As an illustration, Figure 2.5 shows the (log) unit prices for a few selected products across procurement methods. The graph already provide suggestive evidence, for each product, that unit prices are lower for FAs relative to other procurement methods. It also shows that there is a large variation in average unit prices across products, therefore making it necessary to compare FAs with other procurement methods for purchases of similar products. With the regression analysis, the team compared FAs with other procurement methods for a larger set of products and the estimation framework ensures that average price difference across products are taken into account and the regression estimates only capture differences across procurement methods for purchases of similar products. Figure 2.5 also highlights that the effect of using FAs relative to other procurement methods varies across products. This indicates the importance of identifying the products best suited for this procurement method as well as regularly monitoring and evaluating the effects generated by the adoption of FAs for each product. Figure 2.5 : Box plot of select goods across procurement methods Source: Contracts dataset at item level. Only FAs, reverse auctions, invited biddings, direct purchases and restricted biddings. Table 2.8 shows that unit prices are on average lower for FA contracts than for other procurement methods. The results for Model A indicate that unit prices under FAs are lower than open and closed/limited methods even when comparing purchases done by the same procuring entity for the same product across methods. The results for Model B show that unit prices under FAs are lower than open and closed/limited methods 27Annexure-B contains some statistics characterizing the distribution of unit prices and motivating the need to use the logarithmic transformation for the regression analysis. 20 | P a g e even when comparing purchases done for the same product from the same firm across methods. The estimates reported for both Model A and Model B indicate that the reduction in prices through FAs is higher with respect to closed/limited methods than open methods. The economic savings associated to the use of FAs are primarily explained by the effect of buying in larger quantities, as indicated by the estimates reported in column (2) and (4) in Table 2.8. 28 On average, FAs are established for larger quantities, and this have implications on unit prices through various channels, such as competition and firm selection. Table 2.11 and Table 2.13 explore these two channels by comparing the number of bidders and the probability that the awarded firm is an SME across methods. Table 2.8: Log of Unit Price Row N0. VARIABLES Model A Model A Model B Model B With Quantity With Quantity 1 Panel A: FA vs. Open 2 FA -0.0919** 0.0491 -0.211*** 0.0182 3 (0.0407) (0.0514) (0.0311) (0.0438) 4 Observations 172,605 172,364 174,770 174,462 5 R-squared 0.910 0.933 0.940 0.954 6 Panel B: FA vs. Closed 7 FA -0.295*** -0.302*** 8 (0.0314) (0.0840) 9 Observations 136,919 141,131 10 R-squared 0.924 0.949 11 quantity ex-ante No Yes No Yes control 12 year FE Yes Yes Yes Yes 13 quarter FE Yes Yes Yes Yes 14 item FE No No No No 15 UG FE No No Yes Yes 16 item x UG FE Yes Yes No No 17 item x Firm FE No No Yes Yes 18 Robust standard errors in parentheses, *** p<0.01, ** p<0.05, * p<0.1 Model Specifications • Model A: Comparing FAs and non-FA methods for the purchase of the same product by the same entity. With and without controlling for ex-ante quantity. • Model B: Comparing FAs and non-FA methods for the purchase of the same product from the same firm. With and without controlling for ex-ante quantity. Reading Table 2.8: Row 2 and 7: Regression Coefficient: This provides the expected change in the dependent variable (here: Log of Unit Price) for a procurement process conducted under FA in comparison to non-FA (for Panel A the comparison is with Open, and for Panel B the comparison is with Closed). Row 3 and 8: Standard Error: The standard error is the estimate of the standard deviation of the coefficient. Row 4 and 9: Observations: The number of observations used for the model Row 5 and 10: R-Squared: The statistical measure of how close the data is to the fitted regression. A high R-squared means that the model explains more of the variation in the data. 28Column (2) and (4) can be estimated only for the comparison between FAs and other open methods because ex-ante estimated quantities are not available for closed/limited methods. 21 | P a g e Row N0. VARIABLES Model AModel A Model B Model B With Quantity With Quantity Rows 11 – 17: “Yes” indicates which control is included in each regression model. Source: Contracts dataset at item level; restricted to goods. Only FAs, reverse auctions, invited biddings, direct purchases and restricted biddings. 2.4.2 Processing time In order to compare FA and non-FA methods with respect to processing time, the Team looked at decision period 29, which is defined as the number of months between tender opening and award publication. In available data, this measure is well defined only for open tenders and therefore for this analysis the team did not compare FAs with closed/limited methods. Figure 2.6 shows that decision time is on average higher for FAs (while setting-up initially) than other open methods, and this result is confirmed by the estimates reported in Table 2.10. This is not unexpected and bids for FAs may be more complex to evaluate and as noted in the next section, the number of bids for FAs may be higher than other methods. Given the large number of items per bidding process for an FA (See Table 2.2), a longer decision period may also be a reflection of different bids for different types of items being evaluated at the same time for the FA. However, the longer decision time for FAs is amortized over multiple items and orders, and therefore the total decision time per order-item is approximately similar for FAs and reverse auctions (Figure 2.7). Figure 2.6 : Decision time Source: contract item dataset; restricted to goods. Only FAs, reverse auctions, invited biddings, direct purchases and restricted biddings. Table 2.9 : Number of months between bid opening to award publication date Model A Model B Row VARIABLES Decision time Decision time No. 1 Panel A: FA vs. Open 2 FA 0.649*** 0.643*** 3 (0.0371) (0.0350) 4 Observations 180,611 182,220 5 R-squared 0.817 0.884 6 year FE Yes Yes 7 quarter FE Yes Yes 29 Annexure-B contains some statistics characterizing the distribution of processing time for the regression analysis. 22 | P a g e Model A Model B Row VARIABLES Decision time Decision time No. 8 item FE No No 9 UG FE No Yes 10 item x UG FE Yes No 11 item x Firm FE No Yes 12 UG x Firm FE No No 13 Robust standard errors in parentheses, *** p<0.01, ** p<0.05, * p<0.1 Model Specifications • Model A: Comparing FAs and non-FA methods for the purchase of the same product by the same entity. With and without controlling for ex-ante quantity. • Model B: Comparing FAs and non-FA methods for the purchase of the same product from the same firm. With and without controlling for ex-ante quantity. Reading Table 2.9: Row 2: Regression Coefficient: This provides the expected change in the dependent variable (here: Number of months between bid opening to award publication date) for a procurement process conducted under FA in comparison to other open methods. Row 3: Standard Error: The standard error is the estimate of the standard deviation of the coefficient. Row 4: Observations: The number of observations used for the model Row 5: R-Squared: The statistical measure of how close the data is to the fitted regression. A high R- squared means that the model explains more of the variation in the data. Rows 6 – 12: “Yes” indicates which control is included in each regression model. Source: contract item dataset; restricted to goods. Only FAs, reverse auctions, invited biddings, direct purchases and restricted biddings. Figure 2.7: Decision time, total and per contract Source: process dataset; restricted to goods. 2.4.3 Competition To measure competition, the team used the number of bidders for each item within each procurement process. Open methods receive on average a higher number of bids than closed/limited methods, and even more so in case of FAs (Figure 2.8). 23 | P a g e Figure 2.8: Number of bidders Source: procurement processes dataset at item level; restricted to goods. Only FAs, reverse auctions, invited biddings, direct purchases and restricted biddings. SME firms were classified using the labor yearbook classification by Sabrae (2013) and employer-employee data in Brazil (see Annexure-B). Average share of SMEs are calculated as a proportion of bidding firms who are classified as small, medium, and micro enterprises over the total firms that bid on the procurement process. As seen from Figure 2.9, the share of SME bidders has remained more or less stable at 92 to 95 percent for open and FA methods. For closed methods the proportion of SME bidders has been steadily decreasing over the years, from 73 percent in 2013 to 69 percent in 2018. Figure 2.9: Share of SME Source: procurement processes dataset at item level; restricted to goods. Only FAs, reverse auctions, invited biddings, direct purchases and restricted biddings. The regression results reported in Table 2.12 confirm that the number of bids is on average higher for FA than closed/limited methods or other non-FA open methods by almost 1 bid when comparing FAs with other open methods and by almost 5 bids when comparing to closed/limited methods. Further, the team found that the average share of SMEs participating in bidding process for FAs is slightly higher when comparing with average SME bidders in non-FA open methods and limited/closed methods. 24 | P a g e Table 2.10 : Number of bidders and Share of SMEs by item process Row Model A Model A Nr VARIABLES Number of bidders Share of SME in bidding process 1 Panel A: FA vs. Open 2 FA 0.993*** 0.00634*** 3 (0.0121) (0.000431) 4 Observations 5,497,727 4,844,197 5 R-squared 0.686 0.505 6 Panel B: FA vs. Closed 7 FA 4.866*** 0.181*** 8 (0.00949) (0.000840) 9 Observations 5,804,329 5,042,998 10 R-squared 0.722 0.635 11 year FE Yes Yes 12 quarter FE Yes Yes 13 item FE No No 14 UG FE No No 15 item x UG FE Yes Yes 16 item x Firm FE No No 17 Robust standard errors in parentheses, *** p<0.01, ** p<0.05, * p<0.1 Model Specifications Model A: Comparing FAs and non-FA methods for the purchase of the same product by the same entity. With and without controlling for ex-ante quantity. Reading Table 2.10: Row 2 and 7: Regression Coefficient: This provides the expected change in the dependent variable (here: Number of bidders and Share of SMEs by item process) for a procurement process conducted under FA in comparison to non-FA (for Panel A the comparison is with Open, and for Panel B the comparison is with Closed). Row 3 and 8: Standard Error: The standard error is the estimate of the standard deviation of the coefficient. Row 4 and 9: Observations: The number of observations used for the model Row 5 and 10: R-Squared: The statistical measure of how close the data is to the fitted regression. A high R-squared means that the model explains more of the variation in the data. Rows 11 – 16: “Yes” indicates which control is included in each regression model. Source: procurement processes dataset at item level; restricted to goods. Only FAs, reverse auctions, invited biddings, direct purchases and restricted biddings. 2.4.5 Winner Characteristics For this analysis, the team focused on the SME status of the awarded supplier. Table 2.11 shows the distribution of firm size for proportion of participants and winners. When considering all processes, Micro firms are the largest groups among participating firms as well as among winning firms. However, the winning rate (i.e. the rate of wins per participation) is lower for Micro firms than other categories of firms. 25 | P a g e Table 2.11: Participation and Winning rate of Firms All Processes SME proportion of participants proportion of winners Non-classified industry 8.13% 6.63% Micro 76.10% 71.38% Small 12.72% 16.37% Medium 1.96% 3.37% Large 1.09% 2.25% Source: Procurement processes dataset and contracts dataset at item level; Table 2.12 shows that the probability that an SME is awarded the contract is on average lower for FAs with respect to other open methods, and similar to the one observed for closed/limited methods. This result is not surprising, since FAs are best suited for firms with high production capacity. Table 2.12 : SME Model A Row VARIABLES SME winner No. 1 Panel A: FA vs. Open 2 FA -0.0198*** 3 (0.00582) 4 Observations 166,399 5 R-squared 0.566 6 Panel B: FA vs. Closed 7 FA -0.00499 8 (0.00901) 9 Observations 126,624 10 R-squared 0.626 11 year FE Yes 12 quarter FE Yes 13 item FE No 14 UG FE No 15 item x UG FE Yes 16 item x Firm FE No 17 UG x Firm FE No 18 Robust standard errors in parentheses, *** p<0.01, ** p<0.05, * p<0.1 Model Specifications • Model A: Comparing FAs and non-FA methods for the purchase of the same product by the same entity. With and without controlling for ex-ante quantity. Reading Table 2.12: Row 2 and 7: Regression Coefficient: This provides the expected change in the dependent variable (here: Number of bidders and Share of SMEs by item process) for a procurement process conducted under FA in comparison to non-FA (for Panel A the comparison is with Open, and for Panel B the comparison is with Closed). Row 3 and 8: Standard Error: The standard error is the estimate of the standard deviation of the coefficient. Row 4 and 9: Observations: The number of observations used for the model. Row 5 and 10: R-Squared: The statistical measure of how close the data is to the fitted regression. A high R-squared means that the model explains more of the variation in the data. Rows 11 – 17: “Yes” indicates which control is included in each regression model. 26 | P a g e Source: contract item dataset; restricted to goods. Only FAs, reverse auctions, invited biddings, direct purchases and restricted biddings. 2.4.6 Market Share of top suppliers This section explores whether and how the use of FAs shapes market characteristics, such as the number of suppliers and market concentration. Figure 2.10 shows that, for the top 10 products with the largest total procurement volume, the number of suppliers contracted through open methods is always higher than under closed/limited methods and for 6 of these top 10 products there are more suppliers contracted through FAs than other open methods. Figure 2.10 shows, for these top 10 products, the market share of the top supplier, calculated separately for each procurement method. For 5 of these products, the market share of the top supplier is the highest for closed methods, while for other 5 it is higher under FAs than other open methods. Figure 2.10: Number of suppliers on top 10 items by type method Source: contract item dataset; restricted to goods. Only FAs, reverse auctions, invited biddings, direct purchases and restricted biddings. 27 | P a g e Figure 2.11: Proportion of Total Contracting Volume of the top supplier of the good for the top 10 items by type method Source: contract item dataset; restricted to goods. Only FAs, reverse auctions, invited biddings, direct purchases and restricted biddings. The use of FAs can share the markets in two opposite ways. On the one hand, there is a negative correlation between competition and market concentration, and therefore it is possible that the use of FAs is associated to lower market concentration given its positive effect on the number of bidders. On the other hand, by design, FAs are contracts established by one or more procuring entities with only one supplier for a given period of time (up to one year). In order to explore the relationship between the use of FAs and market concentration, the team estimated the following regression analysis: ℎ = _ + _2 + _ + _2 + Ɛ where ℎ is the market share of the top k Suppliers for the j good, _ is share of volume using FA methods, and _ the share of volume using open method. In Table 2.13 shows the marginal effects of a 1% increase in the share of contracting volume through FA or other open methods with respect to the share of contracting volume through closed/limited methods. The estimates indicate that market concentration is lower for products with a larger use of open methods, while it is higher for products with a larger use of FAs. Despite increasing competition, the use of FAs lead to higher market concentration, and this dimension should be taken into account when considering in which contexts and for which products to promote the adoption of FAs. 28 | P a g e Table 2.13: Marginal effect of proportion of FA and Open procurement Marginal effect Marginal effect Marginal effect Market share 1 Market share 3 Market share 5 VARIABLES highest highest highest supplier suppliers suppliers 1 2 Share of procurement using FAs 0.302*** 0.222*** 0.142*** 3 (0.0150) (0.0110) (0.00831) 4 Share of procurement using open methods -0.135*** -0.0905*** -0.0499*** 5 (0.0228) (0.0167) (0.0126) 6 7 Observations 6,411 6,411 6,411 8 Standard errors in parentheses, *** p<0.01, ** p<0.05, * p<0.1 Reading Table 2.13: Row 2 and 4: Regression Coefficient: This provides the expected change in the dependent variable (here: Marginal Effect of Market Share for Highest Supplier) for an increase in the share of procurement using FA or Open. Row 3 and 5: Standard Error: The standard error is the estimate of the standard deviation of the coefficient. Row 7: Observations: The number of observations used for the model Source: Item dataset; restricted to goods with at least 5 orders in contract dataset. 2.5 Conclusions In Brazil, the team found several signs that FAs have the potential to increase the efficiency of public procurement. FAs are associated to a reduction in unit prices with respect to closed/limited methods as well as other open methods, and these economic savings are primarily explained by the effect of buying in larger quantities and increased competition. However, the use of FAs is also associated to longer initial processing times, lower probability that the contract is awarded to SMEs and higher market concentration. These trade- offs might explain why the use of FAs has not been increasing over time in Brazil, as well as the current management structure and practices related to their implementation. On average, the total quantity purchased under FAs is 65% of the total quantity initially established in the umbrella contract, and in 49% of cases the total quantity purchased is lower than estimated in the FA umbrella contract. Transactional costs associated with higher competition through the use of FAs and preparation time for tenders advertising FAs are not considered in this analysis, but are potential trade-offs of using FAs. It would be appropriate to review the existing incentives and management structure for FAs and identify strategies to reduce inefficiencies and ensure the potential benefits of FAs are fully realized. Table 2.14: Findings on FAs in Brazil Parameter Hypothesis Finding With respect to With respect to open methods closed/restricted methods Unit prices FAs result in lower price Yes Yes Decision-making FAs require longer decision- Yes Not measured period making period Total contracting FAs require longer time to Not measured Not measured time establish the contract 29 | P a g e Parameter Hypothesis Finding With respect to With respect to open methods closed/restricted methods Time for placing Once established, FAs require Not measured Not measured orders less time to send orders Level of FAs result in higher number of Yes Yes competition bidders (number of bidders) SME participation FAs discourage SME No No participation SME winning FAs contract are less likely to Yes No be awarded to SMEs Market FAs result in more market Yes Yes concentration concentration 2.5.1 Recommendations for data quality improvements The public procurement data for Brazil has several unique feature that allowed a precise and advanced econometrics analysis. The data are available at both the tender level and the contract level, the datasets can be linked with well-defined IDs for tenders, contracts, bids and items, the information can be disaggregated at the product level and products are classified in small and precisely defined categories. Nonetheless there are some data limitations that constrained the analysis and there are various improvements in data availability and data quality that could be introduced in order to enable a more complete monitoring of public procurement in Brazil and specifically of FAs. These data quality improvements include (but are not limited to): • The data available on centralized open public procurement portals for Brazil only include public procurement administered by federal entities. At sub-national level, there are independent public procurement systems, legislation, and portals for each state, but the information is not centralized nor homogenized across states and therefore is was not possible to include sub-national public procurement in this study. • While each FA umbrella contract includes the full list of participating procuring entities for each product included in the FA and the expected quantity to be purchased by each of them for each product, this information is not available in the data spreadsheets downloaded from “portal.transparencia” and “compras.dados” and used for this study. This implies that it is impossible to investigate whether the expected quantities indicated in the umbrella contract are closely aligned with the quantities purchased in practice, nor compare the use of established FAs by participating vs. non-participating entities. There is suggestive and qualitative evidence that there are large discrepancies between expected and realized procurement demands, and this aspect should be investigated further in order to identify how to optimize the implementation of FAs. • Some key dates and milestones in the procurement process are not included in any of the datasets, such as the date when the FA umbrella contract is signed, expected delivery date, and realized delivery date. It would be necessary to measure public procurement processing times in order to draw a more complete picture on the monetary and administrative costs associated to the use of FAs. • Firm characteristics are not included in the datasets, such as location, sector, or firm size. For this analysis, the team was able to recover firm size by matching the public procurement data with the employer-employee (RAIS) administrative register, since firms are assigned the same unique ID in both data sources. However, additional firm characteristics would have enriched the analysis, for example on the extent of local preferences and whether the use of FAs limits them. • Some patterns in the data are surprising and it is unclear whether they correctly represent public procurement processes in Brazil or they are explained by data errors. For example, by matching the 30 | P a g e Procurement Processes Dataset and the Contracts Dataset it is possible to estimate the share of processes that are initiated but that do not result in any contract. This information is presented Table 2.3, but the data indicates that a very large share of FAs, direct purchases, unenforceable biddings and reverse auctions did not originate any contract. As another example, it seems that, only in the case of Competitive Biddings, the Bids Dataset includes only the bid of the awarded suppliers and not the bids received by all bidders. The analysis presented in this report should not be affected by these potential inconsistencies, but it would be relevant to correct the data errors or, if the data are correct, further investigate these unusual patterns. • The public procurement data for Brazil are rich of information, but the lack of a properly maintained data dictionary and data map did not facilitate the construction of the dataset and the interpretation of key variables for the analysis. 31 | P a g e Chapter 3 Data Analytics: Colombia 3.1 Introduction 3.1.1 Overview In Colombia, public procurement is conducted through three platforms, wherein SECOP I and II are used for all public procurement modalities and Tienda Virtual Del Estado Colombiano (TVEC) is used for making purchases through FAs (see Section 3.1.3 for further information). During the period 2017-May 2020, the volume purchased through non-FA modalities is notably higher than through FAs (Figure 3.1). The volume purchased through FAs slightly increased over years, but its share relative to the total procurement volume remains lower than 11% over the entire period. Figure 3.1 Volume purchased over years by platforms (in Million COP) Figure 3.2: Share of volume purchased by platform over years Figure 3.3 Percentage of purchase value by platform over the total purchased value (2017-2020) 32 | P a g e 3.1.2 Procurement processes in Colombia Colombia Compra Eficiente (CCE) is the national public procurement agency in Colombia, and SECOP II is the e-procurement platform currently in use. The use of FAs in Colombia was enabled in 2007 with the Law 1150 2007 and the Decree 4170 of 2011 indicated CCE as the only public institution entitled to design, organize and execute FAs. There are two types of operations under FAs: 1) Principal Operations, in which the CCE identifies the goods and services with common characteristics for which an FA can be established. In preparation of FAs, the CCE conducts market research considering both international and national offers and the demand of goods and services. For each FA, the CCE manages an open bidding competition in order to select suppliers and define supply conditions for a given time window, typically between 2 and 5 years, which then are published and made available to all contracting entities through an e-catalog. 2) Secondary Operations, through which contracting entities purchase the goods and services under an established FA. Contracting entities can submit a purchase order directly on a platform called Tienda Virtual del Estado Colombiano (TVEC). The CCE website provides application forms, the active e- catalogs, and instructions on how to place an order on TVEC. For certain products and entities, purchases under FA are mandatory, while for others they are optional (See Section 2.2 for the empirical analysis based on this feature). There are three types of instruments in Colombia to consolidate procurement demands: 1) Acuerdos Marco de Precios 30 (FA), 2) Grandes Superficies (department stores), 31 and 3) Otros instrumentos de Agregación de la Demanda (Other instruments for demand aggregation). In this study, the team focused on Acuerdos Marco (see Annexure-C). Most of the information on Colombia’s e-procurement system is public domain, such as data on tenders, contracts, and purchase orders. For FAs, CCE publishes information on the tender process that established the FA, the e-catalog with the list of suppliers, products and contract conditions, and all purchase orders. For other procurement methods, the CCE publishes tender and contract details. 32 3.2 Data Description and Sources For this study, three main sources of data used: the SECOP platform, the TVE platform and the e-catalogs. All these data sources are open access and SECOP and TVEC are updated daily. • The data extracted from the SECOP platform provides information on all tendering processes (FA and non-FA modalities) established by procuring entities or by CCE. The data on the contracting process are however published only for non-FA contracts on this platform. The SECOP dataset is at contract level with each contract referring to only one product. There are two different platforms to conduct e-procurement, SECOP I and SECOP II, and procuring entities can freely choose which one to use on a case by case basis. For each contract, the data downloaded from the SECOP platform includes the tender details, the supplier and buyer name, the total contract value, the SME status of the supplier and some key dates throughout the procurement cycle, but not unit prices. • The data extracted from the TVEC platform records all purchase-orders made under FAs established in Colombia. The TVEC dataset is at the purchase-order level, with each order including multiple 30 Acuerdos Marco is a modality of Instrumentos de Agregacion de la Demanda 31 Through Grandes superficies, the CEE has set up agreements with several department stores and supermarkets where contracting entities are able to purchase at lower cost. 32 The publicly available information for FAs and non-FA purchases is not necessarily the same for all FAs nor for all contracts. 33 | P a g e products. For each order, this dataset includes the supplier and buyer name, the list of purchased products, the total order value and the date and time of the order, but not unit prices. • The data extracted from the e-catalogs shows the supply conditions for each supplier and for each product included in that FA. The dataset is at supplier-product level, and the products included in the e-catalogs are listed in specific details, such as brand, model and year of production. For most FAs, the dataset includes unit prices for each product offered by each supplier. 33 Thus, for the purpose of this analysis, the data from TVEC was matched with the data from the e-catalog to recover unit prices. This was possible only for some FAs for which the e-catalog contained unit prices, was well- structured and readable in standard statistical software. Table 3.1: Sources of information Parameter SECOP 34 TVEC E-CATALOG Time frame 2016-May 2020 2016-May 20202 2016-May 2020 coverage Unit of Contract level Order level (Every Supplier-Product level analysis order includes several products) Data Contracts established through FA orders Unit price per product every procurement modality and supplier for each (including processes to FA establish FAs Main • Tender details • Supplier and buyer • Supplier name indicators • Supplier and buyer name name • Product description • Total contract value • List of purchased • Product details: • SME status of the supplier products per order brand, model, and • Some key dates throughout • Total order value year of production the procurement cycle • Date of the order • Unit price for each and time product offered by • Obligated/non- each supplier obligated status of the procuring entity Limitations • Not unit prices • Not unit prices • Each e-catalog has a • No obligated/non-obligated • No UNSPSC codes different structure status of procuring entities for products • No UNSPSC codes • No SME status of for products suppliers • No SME status of suppliers 3.2.1 Construction of the data Since the goal of the analysis is to compare purchases under FAs and other modalities, a process of cleaning and standardization of key indicators between SECOP and TVEC data was required to proceed with the analysis. The following steps were taken to clean and construct a standardized dataset with data from both FAs and non-FAs. The analysis conducted in this report is based on products that belong to Acuerdo Marco de Precios. At the time of downloading the data from the CCE website, there were 55 active FAs and available on the website. For this study, the team selected those FAs for which it was feasible to connect TVEC, SECOP and e-catalogs 33 This data is not available for all e-catalogs. For example, for one FA, Tiquetes aereos I, the e-catalog includes discounts offered by each supplier for each product item rather than the final offered unit price. 34 There are two SECOP platforms, SECOP I and SECOP II, whose difference is the newest and some indicators on purchases. Currently entities can freely choose which one to use on case by case basis. 34 | P a g e data. The final sample covers 11 FAs (see Annexure-C) for a total of 36 products (4 FAs include more than 1 product) and a timeframe of 2016 to 2020. The exact criteria for selection of products considered for analysis as following: • Availability of unit prices in the e-catalog and feasibility to merge the data on unit prices with the FA orders dataset (TVEC). • Availability of the associated UNSDP product code in the SECOP data to compare purchases of similar products in the SECOP data and in the TVEC data. • Estimation of time to clean the order-description variable in the TVEC dataset and split it into products. 35 Next subsections describe the exact steps in order to construct the final datasets used for the empirical analysis in this report, the criteria for the sample selection of FAs for the analysis, the main limitations, and their implications for the analysis. 3.2.2 Converting the TVEC dataset to the product level Due to the differences between the TVEC dataset and the SECOP dataset (Table 3.1), a disaggregation of the TVEC data at the product level was required to match the data from SECOP with the data from TVEC and compare purchases under FAs and non-FAs for the same products. To that end, the most accurate indicator to use in order to match the two datasets was the UNSPSC product codes at class level since the data from the SECOP platform included them. These product codes, however, were not included in the TVEC dataset nor in the e-catalogs. For each of the FAs selected for this study, the team split the order description in order to identify all products included in each order, and then manually assigned the UNSPSC code class to each product based on its description. The assignment of UNSPSC codes to products was conducted using the information from the United Nations official classification. Since the UNSPSC codes were assigned to product names manually in the TVEC, there may be a few deviations from the de facto categorization of these products. 3.2.3 Matching data on suppliers and unit prices from the e-catalog The goal of the disaggregation of the TVEC data was also to match the TVEC orders data with the e-catalog data for an analysis of unit prices and suppliers. Because product IDs and supplier IDs are not included in the TVEC dataset nor in the e-catalog dataset, this match could only be performed using supplier and product names. However, there are several inconsistencies between the data extracted from the TVEC and the e- catalogs (for example, different spellings for names of the suppliers, different product descriptions, or orders for products/suppliers that are not listed in the e-catalogs). Moreover, information in the e-catalogs are not organized in a standardized format for all FAs and thus, for some FAs it was not possible to convert the e- catalogs data into a format and structure that could be used for the analysis. 36 Because of these challenges, the analysis on unit prices was only performed for some FAs as for remaining products it was not possible to establish a link between the TVEC data and the e-catalog data. 3.2.4 Adding details on the FA from the FA website The dataset downloaded from the TVEC website did not include some information necessary for the empirical analysis but the team was able to recover through additional research on the CCE website. First, the dataset 35 The order-description variable lists all the products purchased in a purchase-order. This variable was cleaned by splitting the variable into several products to transform the TVEC data from order level to product level. 36 For example, for one FA, Tiquetes aereos I, the e-catalog includes discounts offered by each supplier for each product item rather than the final offered unit price. 35 | P a g e included Acuerdos Marco de Precios (FA), Grandes Superficies (department stores), 37 and Otros instrumentos de Agregación de la Demanda (Other instruments for demand aggregation) but without a systematic classification of the orders administered through these different instruments. Thus, orders under FAs were identified manually based on the FA title and cross-checking with the information on the CCE website. Second, information regarding the beginning and end of the FA, and identification of the FA were included manually as well using the information published on the CCE website. 3.2.5 Matching SME and Obligated status from SECOP to TVEC data Whether a supplier is a small-medium enterprise (SME) or not is a characteristic that is included in the data extracted from the SECOP platform but not in TVEC dataset. As a result, to conduct the analysis on the SME status of suppliers, the team needed to recover the SME status of suppliers registered in the e-catalogs and TVEC dataset from the information provided in SECOP. This process was complicated by the fact that TVEC data and SECOP data do not include supplier IDs. Therefore, suppliers information from these two datasets could be matched only using supplier names, which makes this match more cumbersome and less precise. As a result of this limitation in the data, there might be SME firms that only supplied goods through TVEC and not through SECOP for which the team was unable to determine their SME status. On the other hand, whether the procuring entity is obligated to procure under FAs or not (obligated status) is only recorded in the TVEC data because it is a characteristic of each FAs. Hence, the analysis of obligated vs. non-obligated entities covers only FA orders. Besides, it is not possible to replicate the obligated and non- obligated entities for non-FA purchases because of the legal definition of obligated entities (see Section 3.2) and the inconsistency in entities names in SECOP and TVEC data. 38 3.2.6 Implication of data limitations for the analysis The analysis of procurement outcomes under FAs vs. other procurement modalities is hugely limited by the data quality and availability of procurement data in Colombia. Given the many restrictions in standardizing different data sources into one cohesive dataset, there were only a few FAs that were finally used for the analysis (11 out of 55). This suggests that the analysis may suffer from selection bias as only FAs with better data availability on products were finally selected for the analysis. Due to limitations in data availability, there are some key procurement outcomes that cannot be consistently measured for both FAs and other procurement methods, such as unit prices or contract execution time, therefore the evaluation of FAs cannot be performed on these dimensions. The following analysis, however, suggests some preliminary advantages of using FAs over other procurement modalities, that can be further studied through more rigorous methods after comprehensive data integration of the Colombia procurement data management system. 3.3 Research Topics • Use of FAs by non-obligated entities • Time taken in setting up FAs in comparison to non-FA approaches • Participation of SMEs in FA and non-FA approaches • Unit prices for FAs • Top suppliers in FA and non-FA approaches 37 Through Grandes superficies, the CEE has set up agreements with several department stores and supermarkets where contracting entities are able to purchase at lower cost. 38 Procuring entities are obligated to use established FAs if they are: 1) part of the legislative branch 2) national entities or 3) under the General Statute of Contracting of the Public Administration. Because the procurement data available on CCE does not include information on the third criteria, the team was unable to determine the obligated/non-obligated status for the procuring entities included in SECOP but not in TVEC. 36 | P a g e 3.4 Procurement Data Analysis This section describes the main findings from the empirical analysis and it includes: • A general overview of procurement volume under FA vs. non-FA over time • An analysis of the use of FAs by obligated vs. non-obligated entities • A comparison of processing times between FA and non-FA methods • SME suppliers for purchases under FA and non-FA • A comparison of unit prices within FAs between SMEs and non-SME firms • A comparison of market concentration under FA and non-FA Most of the empirical analysis will present results by FAs and products and show that the effect of using FAs relative to other procurement methods varies across products. This indicates the importance of identifying the products best suited for this procurement method as well as regularly monitoring and evaluating the effects generated by the adoption of FAs for each product. 3.4.1 Overview There were in total 30,626 orders conducted under FAs between 2016 to May 2020. Considering only the products selected for this analysis, between 2016 and May 2020 there were 8,916 orders registered in the TVEC platform amounting to 3.7 million COP (Figure 3.4). The volume procured through FAs for these selected products continuously increased between 2016 and 2019, narrowing down the gap in the procurement volume through FA vs. non-FA methods (Figure 3.4). 39 Figure 3.4 Volume procured over years by FA and non-FA (in million COP) Overall, only 4 product groups 40 out of 11 present a higher share of volume procured purchased through FAs than through non-FA methods (Figure 3.5): Tiquetes aereos I and II (Flight tickets), Motocicletas, Cuatrimotos y Motocarros (Motorcycles, ATVs and Motorcars), Material de Intendencia I (Intendance Material II) and Refrigerios PAE Bogota (Refreshment PAE Bogota). 39Note: data for 2020 covers until May. 40Product groups are defined as groups of product categories (among ones selected for the analysis) that are procured under the same FA. Some FAs contain more than one product. Other products are associated to more than one FAs, covering different time periods. In these cases, the comparison between FA and non-FA methods is done considering purchases within the right time period. 37 | P a g e Figure 3.5 Share total volume purchased by type of products for FA and non-FA 3.4.2 Analysis of Obligated and Non-Obligated Entities According to the Colombia legislation, national entities that belong to the Executive branch and are under the General Statute of Contracting of the Public Administration must purchase through FAs if an FA for the product is established (Table 3.2) 41. Non-obligated entities can also purchase through these established FAs, but are not required to do so. In this section TVEC data has been used in order to compare the volume procured through FAs by obligated vs. non-obligated entities. Table 3.2: Obligated and Not-Obligated entities (FA) Entity Branch Entity No Obligated Total obligated Ejecutiva 3106 4535 7641 Judicial 89 60 149 Legislativa 43 2 45 Organismos autonomo o 835 125 960 independientes Total 4073 4722 8795 3.4.3 Non-Obligated entities also procure through FAs The volume of purchases through FAs increased over time both for obligated entities (Obligada) and for non- obligated entities (No obligada) (Figure 3.6). It is remarkable that FAs are substantially used even by non- obligated entities, and in 2017 and 2018 the procurement volume purchased through FAs was higher for non- obligated entities than for obligated entities. The significant use of FAs even when not mandatory by regulation is a sign that procuring through FAs may be considered convenient by procuring entities. However, it would be necessary to analyze the share of the contracting volume procured through FAs for each procuring 41 Because the team did not have the full list of procuring entities under the General Statute of Contracting of the Public Administration, it was unable to determine the obligated/non-obligated status for the procuring entities included in SECOP but not in TVEC. 38 | P a g e entity in order to draw a full picture on the extent to which each procuring entity uses FAs relative to other methods. 42 Figure 3.6 Volume purchased over years by obligated entities and non-obligated (in million COP) 3.4.4 Obligated entities have higher share of volume procured Figure 3.7 shows that for most of the FAs, obligated entities administer the majority of the procurement volume through FAs, except for Refrigerios PAE Bogota and Motocicletas, Cuatrimotos y Motocarros, which is specifically designed for Bogota and no other entities are expected to use it. This could be partially explained by the total number of entities obligated to purchase through that FA (which varies across FAs), but not entirely. For example, there is only one entity obligated to use Refrigerios PAE Bogota and only three entities obligated to use Material de Intendencia II (Intendance Material II), but still the prevalence of purchases done by non-obligated entities is very large for the former but minimal for the latter. For Combustible (Fuel), Vehicles II (Vehicles II), and Computadores y Perifericos (Computers and Peripherals), obligated and non- obligated entities have a similar procurement volume through FAs. Further research can be conducted to understand why non-obligated entities purchase through FAs for some product groups but not others, for example through surveys. 42This analysis is not feasible with the available data since it would be necessary to match procuring entity data from the SECOP dataset and the TVEC dataset. 39 | P a g e Figure 3.7 Share of purchase value by FA for obligated entities and non-obligated 3.4.5 Processing time Processing time is calculated as the difference between the process initiation date and the final contract signing date. This can be interpreted as the total time taken to complete the procurement process for a purchase. It is important to note that this processing time only captures the time to establish a contract or an order, but not the order-processing time or the delivery time. For FAs, only the products registered in SECOP and for which the necessary date variables are available are covered in this section (five out of the 11 selected FAs). It would have been interesting to calculate the time taken by contracting entities to send an order through TVEC once an FA is established, but unfortunately the TVEC data does not include the necessary dates/times for this calculation. Data on delivery times was not available in SECOP nor in TVEC and therefore the team was not able to compare delivery times for FA and non-FA orders. 3.4.5.1 Per-purchase processing time is significantly smaller for FA As expected, establishing an FA requires a much longer processing time than establishing a single-order standard contract. For example, in case of motorized cycles it takes more than double the time to establish an FA in comparison to establishing a single-order standard contract. For non-alcoholic beverages the time taken to process an FA is five times the time take to establish a single-order standard contract (Figure 3.8). However, one of the advantages of FAs is to exploit economies of scale, for example by centralizing and consolidating the procurement process for frequent purchases. As the total processing time is amortized over many purchase-orders within each FA, the per-order processing time for FAs is much smaller than the average processing time for non-FA contracts, representing a concrete benefit of these centralized procedures (Figure 3.9). 43 When considering the per-order processing time for an FA, the team found that the average processing time for each purchase order made under an FA is only 0.32 days in comparison to the 31 days on average to establish a standard contract. The consolidation of contracting terms under one established FA generates substantial savings in processing times. 43Figure 3.8 compares the processing time to establish each FA and the average processing time to establish a contract for the same products but through non-FA methods. For this calculation, it was assumed that the time taken to make a purchase order under an established FA is equivalent to zero days (given that the established FA already creates the contracting terms under which the order is made). Thus, the per-order processing time is calculated by taking the total processing time for establishing the FA and dividing that period by number of orders made under the FA. Even if purchase order processing may require some time, the average time taken for FAs is still likely to much lower than non-FA methods. 40 | P a g e Figure 3.8 Processing time for FA and average processing time for non-FA Figure 3.9 Per-order processing time for FA and average processing time for non-FA 3.4.6 SME suppliers 3.4.6.1 Share of volume purchased through SME suppliers is higher for FA than for non-FA One of the potential concerns related to the use of FAs is that they are best suited for firms with high capacity and therefore they might discourage the participation of SMEs in public procurement. However, Figure 3.10 suggests that this is not the case in Colombia, at least for the products selected for this analysis. The share of procurement volume awarded to SMEs through FAs is 22 percent points higher than through non-FA 41 | P a g e methods (from 12% under non-FAs to 34% under FAs), and Figure 3.11 confirms that this finding applies to almost all product categories considered in the analysis. 44 Figure 3.10 Volume purchased to SME for FA and non-FA (in Million COP) While SMEs do not represent the majority of the procurement volume for any product group when non-FA methods are used, the share of procurement volume awarded to SMEs under FAs is higher than 50% for Tiquetes Aereos, Seguros vehiculos II and Combustible (Figure 3.11 and Figure 3.12). For only three product groups (Material de Intendencia, Seguros vehiculos, Vehiculos) the share of procurement volume awarded to SMEs is higher under non-FA methods. These are products where it is reasonable to expect that SMEs may not be highly competitive suppliers under FAs due to resource and financial capacity constraints. Indeed, for these products, SMEs represent a very low share of the procurement volume even when considering non-FA methods, as under FA no SME was selected as a supplier in the first place. Figure 3.11 Share of volume purchased through SME suppliers under FA 44This effect is not explained by regulatory reasons, since the tender documents for the FAs used in this analysis do not include specific incentives (e.g. margins of preferences or evaluation criteria to favor SMEs) to encourage the participation of SMEs. 42 | P a g e Figure 3.12 Share of volume purchased through SME suppliers under non-FA methods 3.4.6.2 The share of purchases from SMEs decreased under FA but not under non- FA The share of the volume procured from SMEs under FAs decreased from 42% in 2016 to 29% in 2019 (Figure 3.13). In comparison, when looking at the contracting volume under non-FAs, the share procured from SME suppliers increased from 4% in 2016 to 12% in 2019. However, these trends may be affected by the type of product group considered for the analysis. While Tiquetes Aereos is a product of quick consumption (one ticket expires once it is used), other products, such as Seguros de Vehiculos, have a longer consumption period and therefore are purchased more sporadically. Further, complete data from 2020 would be required to consolidate the pattern of the previous years. Figure 3.4 Share of volume procured from SMEs over time, FA vs. non-FA (in million COP) 3.4.6.3 Top obligated entities report high volume procured through SMEs As seen in the previous section, approximately 35% of the total FA procurement volume is supplied by small- medium enterprises (SMEs). Figure 3.14 shows that the share of the procurement volume supplied by SMEs is even higher for the obligated entities with the largest procurement volumes. This suggests that among the top obligated entities, the success of SME suppliers is higher. 43 | P a g e Figure 3.5 Volume purchased to SMEs by the top obligated entities (in million COP) 3.4.7 Unit prices This section only covers FAs since unit prices can be only recovered from the FA e-catalogs and are not available for contracts established under other modalities and registered in SECOP. Thus, in this analysis, it is not possible to compare unit prices for the purchase of similar products under FAs vs. other methods. 3.4.7.1 On average, SMEs offer higher unit prices than non-SMEs under FAs For 16 UNSPSC products, the team was able to recover the SME status of suppliers and the unit prices from the e-catalogs and therefore able to compare unit prices for the same products by SME status of the supplier. Overall, the average unit price offered by SMEs for products procured under FAs is 17% higher than by non- SMEs for the same products (Figure 3.15). This could be explained by the nature of FAs, which allows aggregating demand and giving a comparative advantage to larger firms with greater economies of scale. However, for six UNSPSC products, Passenger motor vehicles, Printing and writing paper, Computers, Removable storage media accessories, Folders and bides and indexes, Insurance services for structures and properties and possessions, SMEs offer on average lower unit prices than non-SMEs. 44 | P a g e Figure 3.6 Unit price offered by SMEs vs. non-SMEs for FA (% difference) 3.4.7.2 Unit prices drive changes in the volume purchased The FA for Seguros de Vehiculos (Figure 3.16) has two terms, Seguros de Vehiculos I and Seguros de Vehiculos II. The number of suppliers and the average unit price decreases from term I to II (from 4 to 2 firms and from 407,243 COP to 280,955 COP respectively). Since the number of selected suppliers is proportional to the number of qualified participants, 45 it is interesting to note that in this case the average unit price decreases from term I to term II, despite a decrease in participation and competition. For term I, it was observed that the supplier that offers the lowest unit price (La Previsora) received a higher share of orders and procurement volume (Figure 3.17 and Figure 3.18). For term II, the team again observed that a higher share of procurement volume for Seguros de Vehiculos is procured through the supplier that offered the lower unit price, in this case an SME (La Solidaria). The transparency of the e-catalogs seems to help procuring entities in directing their purchases towards the most convenient suppliers. In both term I and in term II, the largest share of the total procurement volume for Seguros de Vehiculos is supplied by the firm that offers the lowest unit prices on average. While this supplier was La Previsora in term I, procuring entities shifted to the new most convenient supplier in term II (La Solidaria), despite La Previsora was again selected as available supplier for term II. While in non-FA procurement it is often observed that existing suppliers have an advantage in attracting a higher share of purchases and procurement volumes, it seems that this is not the case when public procurement is administered through FAs and unit prices can be easily compared from the e-catalogs. If data allowed, it would have been interesting to expand this analysis in order to verify whether similar trends are observed for other FAs and to compare FA vs. non-FA dynamics. 45For several FAs there are specific rules that indicate how the number of selected suppliers should vary depending on the number of bidders. 45 | P a g e Figure 3.7 Supplier unit price for FA Seguros vehiculos I & II Figure 3.8 Share of frequency of orders by supplier for the FA Seguros vehiculos Figure 3.9 Share of volume purchase by supplier for the FA Seguros vehiculos 46 | P a g e 3.4.8 Top Suppliers 3.4.8.1 Share of purchases through the main supplier is significantly higher for FA than for non-FA The share of the main supplier is much higher for FA than for non-FA methods (Figure 3.19), with the only exception of the Refrigerios PAE Bogota FA, which is specifically designed for Bogota and no other entities use it. For most products, the share of the main supplier under FAs is above 60% but below 30% under non- FA methods. There are various factors that might explain this finding. First, the number of suppliers is much lower for FA than non-FA for all products (Figure 3.20). Second, this higher concentration of purchases on few suppliers under FA might be a positive sign that the transparency of the e-catalogs helps procuring entities to direct purchases towards the most convenient suppliers and that these suppliers have enough capacity to satisfy this demand volume. Figure 3.10 Share of volume purchased for the main supplier by FA Figure 3.20 Number of suppliers per product for FA and non-FA 47 | P a g e While Figure 3.19 and Figure 3.20 compare the market share of the top supplier and the number of suppliers between FAs and other methods, Figure 3.21 investigates the relationship between the use of FAs and market concentration at product level. Considering all purchases across all methods, the market share of the top supplier is lower for products with a higher share of contracting volume procured through FAs, suggesting that the use of FAs contributes reducing market concentration. Figure 3.21 Share of the main supplier and volume procured through FA, product level 46 3.5 Conclusions In Colombia, the team found that FAs are helping to increase the efficiency of public procurement. The volume procured through FAs has increased over time, including for entities non-obligated to use them and whose choices indeed reveal a preference for these instruments. The processing time per purchase is substantially shorter for FAs, given that the (longer) procurement process to establish an FA is then amortized over many purchase orders. The share of procurement volume awarded to SMEs is higher under FA than non-FA methods and it seems that the transparency of the e-catalogs enables procuring entities to direct more of their procurement demands towards suppliers that offer the most convenient unit prices. The analysis is limited by data constraints, but the findings seem to depict a successful application of FAs in Colombia. Indeed, FAs were even used for COVID-19 related procurement already a few months after the start of the Pandemic, an effective way to coordinate demand, increase bargaining power vis-à-vis suppliers, standardize supply conditions and avoid internal competition within the same country. Table 3.3: Findings on FAs in Colombia Parameter Hypothesis Finding Choice of using FAs Non-obligated entities also use FAs Yes Total processing time With respect to other methods, FAs require Yes longer time to establish the contract Per-order total With respect to other methods, the per-order Yes processing time total processing time is lower for FAs SME winning With respect to other methods, the share of No spending awarded to SMEs is lower for FAs Market concentration FAs result in more market concentration Yes 46 The graph is at product level. Annexure-C includes the detailed list of the 37 products included in the analysis. 48 | P a g e 3.5.1 Recommendations for data improvements There are various improvements in data availability and data quality that could be introduced in order to facilitate monitoring of public procurement in Colombia and support advanced econometric analysis for policy evaluations. These data quality improvements include (but are not limited to): • For both FAs and non-FAs, data should be provided at product level. • Entity, supplier, and product IDs, as well as UNSPSC commodity codes, should be provided consistently in all datasets available in the CCE website. This would allow to effectively and precisely match the information registered in different platforms on different procurement modalities and on different procurement stages, therefore improving the possibility for monitoring and evaluation. • Unit prices are available in the e-catalogs for FAs, but they are not available for purchases through other methods, which are registered in the SECOP dataset. Therefore, it is currently impossible to assess the impact on unit prices of using FAs relative to other procurement methods. • The structure of the e-catalogs and the available data varies across FAs. For example, unit prices and the contract agreement are not available for all FAs and some e-catalogs cannot be easily transferred into a data spreadsheet for quantitative analysis. This impedes to compare public procurement outcomes between FAs and relative to other procurement methods. • Some important variables for the analysis are only included in the SECOP dataset but not in the TVEC nor in the e-catalogs, such as SME status of suppliers. It would be necessary to increase the performance indicators and data fields included in all datasets in order to increase data completeness, facilitate monitoring and enable comprehensive comparisons between different procurement methods. • There are various public procurement performance indicators that are not available for Colombia, such as bidding, ordering stages, offer prices, expected execution time, quantity purchased, bid validity, or bid duration. These variables are necessary in order to provide a comprehensive picture of the overall performance of the public procurement system in Colombia and they would have notably enriched the analysis and findings of this report, such as calculation of the time to place an order. 49 | P a g e Annexures to the Report 50 | P a g e Annexure-A: A brief introduction to Regression Analysis Regression analysis is a statistical technique used to determine whether there exists a relationship between a dependent variable and one or more explanatory variables. The regression equation is of the form: Y = b1X1 + b2X2 + ... + ε where Y is the dependent variable, X’s are explanatory variables and b’s regression coefficients. In the context of this FA study, procurement outcomes and procurement characteristics are dependent variables and explanatory variables respectively. Hence, we are trying to establish whether there is any relationship between procurement characteristics and procurement outcomes. Regression results summarizes the nature of relationship in case it exists. Some of the statistics produced from regression include F-statistic, R-squared, regression coefficients, p-values etc. The sign and magnitude of regression coefficients are critical in understanding the relationship. Positive value indicates an increase in the value of explanatory variable results to an increase in dependent variable. In contrast, negative value implies an increase in the value of explanatory variable leads to a decrease in dependent variable. The magnitude of regression coefficients gives information on how large or small the change of dependent variable is whenever there is a unit increase in the explanatory variable. For instance, if a variable X has a regression coefficient of -2 implies that if variable X increases by 1, the value of explanatory variable will decrease by 2. The goal of regression analysis is to know whether explanatory variables do have a genuine effect on dependent variable, or the effect is due to random chance. In other words, establishing whether there exists a statistically significant relationship. This is achieved by determining whether the regression coefficients of independent variables are really different from 0 and if they are, whether it can be considered not to be out of random chance. The null hypothesis states that each explanatory variable has absolutely no effect on the dependent variable i.e. has a coefficient of 0 Example In the example above, the regression analysis explores the relationship between student grades (dependent variable) and class size and school type (in this case whether its private) explanatory variables. Each explanatory variable has two values – regression coefficient and standard errors. Standard errors are in brackets whereas regression coefficients aren’t. The asterisks indicate whether the explanatory variables are statistically significant or not with the number of asterisks giving information on whether the p-value is less than 0.01, 0.05 and 0.1. The R-squared is a statistical measure that represents the proportion of the variance of the dependent variable that is explained by the explanatory variables in a regression model. Class size has a coefficient of -4.196. The negative sign implies a negative relationship with a unit increase in class size resulting to a 4.196 decrease in student’s grade. School is private has a coefficient of 7.292 which implies a student from private school is probably to have a grade that is 7.292 higher than a student from a public school. The three asterisks in the regression coefficient of class size can be interpreted as being statistically significant at a p-value of 0.01. On the other hand, school is private can be understood to have no statistical significance since it has no asterisk. 51 | P a g e Annexure-B: Supporting Data/Information – Brazil B.1 Procurement procedures Competitive Bidding (concorrência pública) – This procedure is directed at high-value purchases open to the participation of any party. Competitive Bidding is for procuring works and engineering services whose value exceeds BRL 3,300,000.00 and goods or services that exceed BRL 1,430,000.00. Reverse Auction (pregão) – In 2002, this method was legislated. It mainly aimed to purchase commons goods and services. It can happen physical or electronic way and the criteria. The criteria to select the supplier is the lowest price. There is no limit value of a purchase value. Invited Bidding (Convite) – Only firms invited can submit a bid, where it is necessary to at least three invitations to start a tender. This method is applied for procuring works and engineering services whose value is up to BRL 150,000.00 and goods or services that are up to BRL 80,000.00. Restricted bidding (tomada de precos) – This method requires previously-registered parties bid, and it is applied for procuring works and engineering services whose value up to BRL 3,300,000.00 and goods or services that up to BRL 1,430,000.00. Contest (concurso público) – Focus in scientific and art purchases. FA: Framework Agreement (pregão registro de preços) – The price registration system is allowed to use for Auction and Open Action. It selects suppliers for items for a limit of 12 months by criteria of the lowest price using prices listed. That method is used to acquire goods and standardized services of any value. Unenforceable Bidding (inexigibilidade de licitacao) – This method is used if there is a unique possible supplier to sell a product. It is direct negotiation between public administration and the seller. Direct purchase (dispensa de licitação) – This method is used for a few dozens of pre-established by Federal law 7.892/2013. It is direct negotiation between public administration and the seller. 52 | P a g e B.2 Unit price distribution Table B.1 presents the distribution of the unit price and its logarithm. The average unit price is R 150.00 for each item. Given that the quantiles show a large variation in the unit price, the team used its logarithm. Table B.1: unit price distribution variable log_unit_price unit_price N_obs 106928 106928 Average 3.46 7,883.85 SD 2.89 112,400.12 min -13.53 0.00 p1 -0.65 0.52 p5 0.25 1.28 p10 0.69 2.00 p25 1.25 3.50 p50 2.55 12.77 p75 5.22 184.14 p90 7.75 2,323.07 p95 9.12 9,130.00 p99 11.76 128,000.02 max 16.06 9,399,971.00 Source: Contract item dataset; restricted to goods and sample A B.3 Processing time Table B.2 shows that the decision period (time difference between time of bid opening and time of award publication) takes an average of 1.11 months, and in 95% of cases it takes less than four months. Table B.2: Time outcomes Decision period (months) Number of months between result bid and first contract signed N_obs 106928 106928 Average 1.11 1.46 SD 1.74 1.93 min 0.00 0.00 p1 0.00 0.00 p5 0.03 0.03 p10 0.07 0.10 p25 0.20 0.33 p50 0.57 0.80 p75 1.27 1.73 p90 2.77 3.53 p95 4.13 5.60 p99 6.60 9.83 max 50.87 17.73 Source: Contracts dataset at item level; restricted to goods and sample A 53 | P a g e B.4 SMEs categorization Size firm definition Micro: [0,19] in manucfactorying and construction; [0,9] service and comerce Small: [20,99] in manucfactorying and construction; [10,49] service and comerce Medium: [100,499] in manucfactorying and construction; [50,99] service and comerce Large: [500,19] in manucfactorying and construction; [100,inf] service and comerce non-classified indothers Source: labor yearbook - Sebrae & Dieese 54 | P a g e Annexure-C: Supporting Data/Information – Colombia Descriptive statistics of the sample TVEC SECOP I SECOP II sum count mean sum count mean sum count mean Combustible 475607.054 2439 195.001 1390866.854 14735 94.392 421063.007 1080 389.873 Computadores y 413911.198 1648 251.160 1697684.566 19532 86.918 75209.956 859 87.555 perifericos Derivados del 14559.836 142 102.534 171871.418 9919 17.327 8477.279 320 26.491 papel carton y corrugado Material de 92233.843 164 562.401 58865.649 1521 38.702 3822.062 140 27.300 Intendencia II Motocicletas, 264372.631 501 527.690 202179.104 605 334.180 112.141 15 7.476 Cuatrimotos y Motocarros Refrigerios PAE 635066.575 386 1645.250 580911.800 4334 134.036 11827.140 229 51.647 Bogota Seguros 99396.477 229 434.046 544841.404 7580 71.879 6999.897 47 148.934 vehiculos Seguros 15609.605 57 273.853 182257.116 3954 46.094 3449.707 84 41.068 vehiculos II Tiquetes aereos 592058.786 991 597.436 87828.802 1445 60.781 6342.442 26 243.940 Tiquetes aereos 270424.515 511 529.206 60906.277 656 92.845 1401.389 42 33.366 II Vehiculos II 866870.111 1848 469.086 1990497.674 3618 550.165 88611.894 67 1322.566 Total 3740110.631 8916 419.483 6968710.665 67899 102.633 627316.914 2909 215.647 Observations 8916 67899 2909 UNSCDP Products per Framework Agreement Framework Agreement Product (UNSPSC class level) Combustible Petroleum and distillates Computer data input devices Computer display accessories Computer hardware maintenance and support Computadores y perifericos Computer printers Computers Removable storage media accessories Software maintenance and support Desk supplies Folders and binders and indexes Derivados del papel carton y Organizers and accessories corrugado Printer and photocopier and facsimile accessories Printing and writing paper Workstations and office packages Boots Camping and wilderness facilities Material de Intendencia II Clothing accessories Shirts and blouses Shoes 55 | P a g e Uniforms Bedclothes Towels Motocicletas, Cuatrimotos y Motocarros Motorized cycles Cheese Citrus fruit puree Desserts and dessert toppings Fresh fruit byproducts Refrigerios PAE Bogota Milk and butter products Nonalcoholic beverages Packaged combination meals Processed cereals Snack foods Insurance services for structures and property and Seguros vehiculos possessions Insurance services for structures and property and Seguros vehiculos II possessions Tiquetes aereos Travel agents Tiquetes aereos II Travel agents Passenger motor vehicles Vehiculos II Product and material transport vehicles 56 | P a g e Framework Agreements published on the website 47 Name Acuerdo Marco para el suministro de Combustible de Aviación Active Acuerdo Marco para la contratación de Mesa de Servicio Active Arriendo ETP Aseo y Cafeteria III Active Aseo y cafeteria Aseo y cafeteria II Blindaje Catálogo Servicio de Desinfección – IAD Emergencia COVID-19 Active Centro Datos / Nube Privada Centro de contacto Combustible Combustible Nacional Active Combustible Nacional antiguo Computadores y perifericos Conectividad Conectividad II Active Consumibles de impresion Derivados del papel carton y corrugado Active Dotaciones de Vestuario Dotaciones de Vestuario II Dotaciones de Vestuario III Active Dotaciones escolares Dotaciones escolares II Active ETP Active Elementos para Emergencias Active Emergencia COVID-19 Active Enfermedad renal cronica Active Equipos biomedicos Google Google II Hemofilia Active Intermediarios de seguros Material de Intendencia Material de Intendencia II Active Microsoft Microsoft II Motocicletas, Cuatrimotos y Motocarros Motocicletas, Cuatrimotos y Motocarros II Active Nube Privada II Nube Privada III Active Nube publica Nube publica II Nube publica III Active Papeleria I Papeleria II Refrigerios PAE Bogota 47 Those FA highlighted in purple are included in this analysis 57 | P a g e SOAT II Seguros vehiculos Seguros vehiculos II Active Servicios BPO Active Servicios de distribucion Servicios de impresion Tiquetes aereos Tiquetes aereos II Active Tratamiento VIH Active Vehiculos II Vehiculos III Active Vehiculos blindados Vehiculos blindados II 58 | P a g e