GOVERNANCE GOVERNANCE EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT GovTech Procurement Practice Note Supported by the GovTech Global Partnership Republic of Korea © 2021 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO), http://creativecommons.org/licenses/by/3.0/igo. Under the Creative Commons Attribution license, you are free to copy, distribute, transmit, and adapt this work, including for commercial purposes, under the following conditions: Attribution—Please cite the work as follows: 2020. GovTech Procurement - Practice Note. EFI Insight-Governance. Washington, DC: World Bank. Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in this translation. Adaptations—If you create an adaptation of this work, please add the following disclaimer along with the attribution: This is an adaptation of an original work by The World Bank. Views and opinions expressed in the adaptation are the sole responsibility of the author or authors of the adaptation and are not endorsed by The World Bank. Third-party content—The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third- party-owned individual component or part contained in the work will not infringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to reuse a component of the work, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, figures, or images. All queries on rights and licenses should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; e-mail: pubrights@worldbank.org. Cover design and layout: Diego Catto / www.diegocatto.com >>> Contents Acknowledgments 4 Acronyms 5 1. Executive Summary 7 2. GovTech and Procurement 10 2.1 GovTech Procurement and Key Challenges 10 2.2 World Bank Public Procurement Principles 12 2.3. GovTech and the Procurement Process 13 3. GovTech Procurement Framework 14 3.1. STEP 1 – Technology Readiness Assessment and Gap Analysis 15 3.2. STEP 2 – Requirements Definition 21 3.3. STEP 3 – Procurement Strategy Development 27 3.4. STEP 4 – Implementation and Execution 39 3.5. STEP 5 – Ongoing Management and Technology Evolution 40 4. Disruptive Technologies in Procurement 42 Bibliography 46 ANNEX I: Sample Technology Assessment Interview and Survey 48 Questions ANNEX II: e-Procurement Toolkit Templates 51 ANNEX III: Example Service-Level Agreement (SLA) Terms 60 >>> Acknowledgments This Practice Note was prepared under the leadership of Hunt LaCascia (Senior Procurement Specialist, Governance Procurement) with contributions from Blandine Wu Chebili (Senior Pro- curement Specialist, Governance Procurement), Justin Valentine (Valent – Procurement Consul- tant), Travis Mells (Valent – Procurement Consultant), and Kelly Flanagan (Valent – Procurement Consultant). Overall guidance for the report was provided by Vinay Sharma, Ed Olowo-Okere, Tracey Lane and Adenike Oyeyiola. The Practice Note benefited immensely from the participation, assistance, and insights from other experts. The team is especially grateful for the support of Governance Procurement, the Information and Technology Solutions (ITS) Technology and Innovation Lab, and the prior work of the procurement colleagues in both Governance Procurement and Operations Policy and Country Services (OPCS). EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 4 >>> Acronyms ACT – IAC American Council for Technology and Industry Advisory Council AI Artificial Intelligence Bank The World Bank BPA Blanket Purchase Agreement BPSCM Budget, Performance Review, and Strategic Planning Vice Presidency Corporate and Middle Office CapEx Capital Expenditure Circular A model of production and consumption that involves sharing, leasing, Economy reusing, repairing, refurbishing, and recycling existing materials and products as long as possible CLMS Contract Lifecycle Management System COTS Commercial off-the-Shelf CSO Civil Society Organization DEC Development Economics Vice Presidency Disruptive T Emerging technologies that result in a step change in the cost of or access echnologies to products or services or that dramatically change how people gather information, make products, or interact DDOS Distributed Denial-of-Service DOS Denial-of-Service EAP East Asia and Pacific ECA Europe and Central Asia eConsultant2 World Bank Electronic Government Procurement solution for advertising, tendering, and contract award of consulting services eGP Electronic Government Procurement EO Economic Operator EOI Expression of Interest e-Procurement Electronic procurement, referring to the process of requisitioning, ordering, and purchasing goods and services online ETT The Emerging Technologies Team, which operates within the United Nations Office of Information Communications Technology EU European Union FCV Fragility, Conflict, and Violence Framework A contracting mechanism in which long-term contracts provide the terms Agreement and conditions under which smaller repeat purchasing orders (or call-off orders) may be issued for a defined period of time GGP Global Governance Practice GIS Geographic Information System GP Global Practice GPDR General Data Privacy Regulation EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 5 GPPD Global Public Procurement Database GPS Global Positioning System GPVP Global Practice Vice Presidency IaaS Infrastructure as a Service IBRD International Bank for Reconstruction and Development ICT Information and Communications Technology IDA International Development Association IoT Internet of Things IP Intellectual Property ITS Information and Technology Solutions KPI Key Performance Indicator LAC Latin America and Caribbean LLI Leadership, Learning, and Innovation LTA Long Term Agreement MDTF Multi-Donor Trust Fund MENA Middle East and North Africa NLP Natural Language Processing OICT United Nations Office of Information and Communications Technology OpEx Operating Expenditure PaaS Platform as a Service PII Personal Identifying Information PPP Public-Private Partnership PPSD Project Procurement Strategy for Development RFI Request for Information RFP Request for Proposal RPA Robotic Process Automation SaaS Software-as-a-Service SAR South Asia SBIR Small Business Innovation Research: programs that encourage small businesses to engage in federal research/research and development with the potential for commercialization SLA Service-Level Agreement SQL Structured Query Language, a programming language that is typically used in relational database or data stream management systems TCO Total Cost of Ownership: an estimate of the total costs associated with a solution over the whole of the operational life, including final disposal UN United Nations VfM Value for Money VPU Vice Presidential Unit WIPO World Intellectual Property Organization XSS Cross-site scripting, a type of injection security attack in which an attacker injects data, such as a malicious script, into content from otherwise trusted websites EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 6 1. >>> Executive Summary The objective of the GovTech Procurement Practice Note is to explore how to better procure GovTech solutions. The note provides recommendations to World Bank teams and client coun- tries on how to assess and prepare for the application of GovTech systems. GovTech is defined as the use of technology to support government operations, service delivery and transparency. It is a mechanism for whole of government public sector modernization that places the citizen at the center of the reform (World Bank 2020). As an example, e-Government Procurement (eGP), which is discussed in detail in this paper is a specific technology sub-category of GovTech that is applied to manage and automate public procurement processes. The GovTech procurement practice note aims to inform clients and teams for more efficient GovTech procurement by: • • Providing a five-step framework to systematically prepare and execute a procurement pro- cess for GovTech solutions. Identifying key requirements necessary for implementing GovTech solutions. • Identifying procurement options for supporting client countries in different areas with the procurement of disruptive technologies. • Utilizing best practice approaches using disruptive technologies in the public procurement area and inform the Multi-Donor Trust Fund (MDTF) and the Global Governance Practice (GGP) about opportunities for scaling. • Offering strategic direction for the coordination of donor and government resources on dis- ruptive technologies in procurement, providing strategies to integrate procurement reform actions using disruptive technologies. • Introducing innovative technologies that will help government procurement systems to in- crease competition and identify business process efficiency and technological solutions. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 7 Governments face several key challenges in the GovTech pro- • Stage 2 – Requirements Definition - When defining curement process, which include: technology requirements, there are four foundational items that should be considered – (i) Business Model (e.g. 1. Lack of industry knowledge about the complexities and Government Owned and Operated, Government Man- unique characteristics of public sector operations. aged Service, Public-Private Partnership (PPP)), (ii) Sys- 2. Identification of data sources for proper current state as- tem Implementation Approach (e.g. Software as a Service sessment and that inform an implementation plan. (SaaS), Commercial Off the Shelf (COTS), Custom-Build), 3. Interoperability requirements with a multitude of legacy (iii) System Security, and (iv) Hosting Solutions. After ad- systems can be complex and complicate solution options; dressing each of the foundational items for consideration, this includes standards of data communication with both a requirements document should be compiled. The docu- national and international systems. ment should capture all the requirements in a format that 4. Lack of existing technology infrastructure and resource can be shared with vendors for their review and response. capacity can inhibit the ability to implement certain solu- tions. • Stage 3 – Procurement Strategy Development - Pro- 5. Lack of understanding of Total Cost of Ownership (TCO) curement strategy development begins with market re- or limited ability to budget for systems maintenance can search. Market research involves the collection, record- inhibit sustainability of the solution. ing, and evaluation of data about a specific market that 6. Large technology providers that can move technology offers a desired product or service. The primary purpose market innovation tend to adhere to their existing technol- of market research is to inform the decision-making pro- ogies and are less flexible than smaller firms in generat- cess, leading to a gain in competitive advantage, reduced ing Proofs of Concept (PoCs) outside of their established risks, and increased productivity in a future investment. technology infrastructure. Common private sector procurement practices, such as 7. Existing technology solutions are rarely based on open competitive dialogue and pre-procurement market en- source technology, which can lead to vendor-lock in and gagement, are critical to developing both the right solution make it difficult to innovate away from proprietary solu- requirements and procurement strategy approach. Gov- tions and technologies. Tech solutions may also require non-traditional procure- ment strategies, such as competitions, funded research This note enables countries to confront these challenges and programs (e.g. Small Business Innovation Research to assess and implement potential GovTech solutions and to (SBIR)), or PPPs to promote new technologies and pro- fill the knowledge gap regarding a usable and replicable ap- vide a government-sponsored avenue for micro, small, proach to exploring and procuring GovTech to improve gov- and medium enterprises to participate in GovTech pro- ernment services and core information systems. A five-step curement opportunities. GovTech procurement framework is presented that guides readers through a strategic and holistic plan for the design, • Stage 4 – Implementation and Execution - The imple- procurement, and implementation of GovTech solutions. The mentation and execution of GovTech procurement begins steps in this framework are listed below and discussed in with the preparation of tender documents and ends with the greater detail in section 3 of this document. integration of the system. After the acquisition is executed, the focus shifts to supplier performance monitoring. • Stage 1 – Technology Assessment and Gap Analy- sis - A technology readiness assessment is a system- • Stage 5 – Ongoing Management and Technology Evo- atic analysis of an organization’s ability to success- lution - Ongoing support, maintenance, and development fully implement transformational processes or change. is critical to long-term implementation and service offer- The assessment identifies the potential challenges that ing. Contract management is therefore an important lever might arise when introducing new technology and any for a procurement strategy to ensure the long-term results accompanying procedures, structures, and processes of any GovTech solution. As GovTech solutions progress within a current organizational context. As part of a readi- through the procurement process, it is recommended that ness assessment, a gap analysis is performed to iden- solutions focus on total cost of ownership (TCO) over the tify gaps or risk areas that should be addressed and life of the project and incorporate rigorous service-level remedied before or as part of the implementation plan. agreements (SLAs) to qualify and evaluate suppliers as well as manage mission-critical service performance. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 8 This practice note puts a further subject matter focus on Gov- problem is an method used by the Information Technology So- Tech as a specific means to improve the public procurement lutions (ITS) Technology and Innovation Lab and the Global process, using a template that can be applied to other func- Governance Practice (GGP) procurement units collaborated tional areas and sectors. An additional focus on innovation to test the possibility and feasibility of using blockchain to in- and disruptive technologies further positions the initiative at crease the transparency and efficiency of eGP systems. Ac- the forefront of the digitization of government. tivities included problem framing and preparation of a disrup- tive technology research plan, followed by case analysis and Disruptive technologies can be defined as emerging technolo- delivery of a prototype solution, which can then be used to gies that result in a step change in the cost of or access to develop procurement requirements. products or services, or that dramatically change how people gather information, make products, or interact. Disruptive tech- Procurement requirements can be as classified conformance nologies will fundamentally change the way in which public or performance specifications. Conformance specifications procurement value is realized over the next decade. Existing are used where a thorough understanding of the requirements supply chain delivery mechanisms will be enhanced by drone already exists, and there is little desire for a supplier to inno- delivery (e.g., medicine to inaccessible locations), supply vate. Performance specifications are used where the under- chains will be increasingly and securely monitored via Geo- standing of what is required in terms of outcomes can be de- tagging, and contracts and payments will be made transparent scribed, but it is less certain as to the best process or method via blockchain. The value creation for developing countries is to deliver the requirements, or there is a desire for suppliers unlimited; however, to capture this value, countries need to to innovate. For the procurement of disruptive technologies, understand when and how best to implement disruptive tech- which by definition are new and innovative, performance nologies based on their unique operating environments. specifications should be used wherever possible to maximize the possibilities for innovation as developed in a Design Think- The procurement of disruptive technologies involves more ing approach. complex approaches than traditional tendering methods. De- sign Thinking, which moves tasks through a complex business EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 9 2. >>> GovTech and Procurement 2.1 GovTech Procurement and Key Challenges Criticality of GovTech Procurement Government services are the lifeblood of the citizens and businesses that fuel a country’s growth. The delivery of services—or lack thereof—has a direct impact on both the economy and citizen well- being. Governments and international institutions globally have recognized the criticality of GovTech in improving government service delivery and have moved to institute both smart and resilient city programs. As ongoing digital transformation efforts work to modernize the public sector, the appro- priation of public funds via procurement or open competitions is even more critical. Because citizens rightly hold digital government services to the highest standards, a robust procurement process is crucial to achieving successful outcomes. However, many governments, businesses, and regulators historically have taken a cautious approach to technology, delaying or even inhibiting its adoption out of fear of job losses and in the process, hindering entrepreneurship and the scaled development of technologies. A well thought out GovTech procurement strategy can provide the right programming and mitigation strategies to allow the government to more confidently develop and use all the leading and disruptive technologies. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 10 Key Challenges in GovTech Procurement Additionally, there are many challenges identified in World Governments face several key challenges in the GovTech pro- Bank GovTech project procurement and implementations, curement process. According to studies conducted in Europe which include: by consulting firms Accenture (Masson and Margot-Duclot 2018) and PwC (PWC 2016), these include: 1. Lack of industry knowledge about the complexities and unique characteristics of public sector operations. 1. Excessive controls on spending and burdensome pro- 2. Identification of data sources for proper current state as- curement procedures can lead to delays that inhibit the sessment and that inform an implementation plan. adoption of new technologies. 3. Interoperability requirements with a multitude of legacy 2. Governments may be locked into fixed, long-term con- systems can be complex and complicate solution options; tracts with obsolete technology systems that impede com- this includes standards of data communication with both mercial partnerships with new software providers. national and international systems 3. There is a significant digital skills gap, as recent Accen- 4. Lack of existing technology infrastructure and resource ture research has found that 59 percent of agencies in capacity can inhibit the ability to implement certain solu- Europe have trouble finding people with the right skills to tions support innovation. 5. Lack of understanding of Total Cost of Ownership (TCO) 4. Sales cycles are lengthy, as selling technology to govern- or limited ability to budget for systems maintenance can ment and the public sector requires sourcing leads, sub- inhibit sustainability of the solution mitting proposals and organizational and financial data, 6. Large technology providers that can move technology and waiting for official and public comments and frequent- market innovation tend to adhere to their existing technol- ly delayed outcomes. ogies and are less flexible than smaller firms in generat- ing Proofs of Concept (PoCs) outside of their established technology infrastructure 7. Existing technology solutions are rarely based on open source technology, which can lead to vendor-lock in and make it difficult to innovate away from proprietary solu- tions and technologies EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 11 2.2 World Bank Public Procurement Principles The GovTech procurement practice note will reinforce the World Bank’s established public procurement principles (World Bank 2017a), which are outlined below. > > > F I G U R E 1 - World Bank Public Procurement Principles 1. Value for Money (VfM). In simple terms, VfM refers to the net value of total lifetime benefit minus total lifetime cost. An economic use of resources requires an evaluation of the relevant costs and benefits. Included for consideration should be transactional, transition, and contingency costs. Contract manage- ment risks should also be considered. If aiming to achieve the greatest VfM, the lowest price or highest quality product or service may not be the most desirable. 2. Fit for Purpose. Fit for purpose implies that a purchased product and service must be able to deliver its intended purpose and provide a satisfactory level of service, taking into account the risk, value, and complexity of the procurement. 3. Economy. Economy takes into consideration non-price attributes, such as sustainability, quality, and environmental and social concerns. 4. Efficiency. Procurement must be proportional to the value and risks of underlying project activities. Procurement arrangements are generally time sensitive and strive to avoid delays. 5. Integrity. The integrity of the procurement process assures confidence in the procurement system. All stakeholders, including potential suppliers, must be able to rely on any information disseminated by the procuring entity. Additionally, all personnel involved in the procurement process must operate with in- tegrity and professionalism and observe the highest standards of ethics throughout, including refraining from fraud and corruption. 6. Transparency. Procurement information, including spending and transaction data, should be regularly reported to and accessible by the general public unless there are legal or security reasons for maintain- ing confidentiality. 7. Fairness. The procurement process should provide an equal opportunity for all potential suppliers to win business contracts. There should also be a transparent means for stakeholders to voice any concerns and for the procurement organization to provide an adequate response in a timely fashion. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 12 2.3 GovTech and the Procurement Process In addition to establishing procurement principles, the World Bank has also documented the key stages in the procurement pro- cess, which act as helpful guidelines for both those new to procurement and for World Bank procurement specialists, Task Team Leaders, client country government officials and international experts. The figure below depicts the key stages of a standard procurement process as documented by the World Bank (World Bank 2019b): > > > F I G U R E 2 - Key Stages in Procurement 1 IDENTIFICATION Identify develop- CHECK ment need/s, Post implemen- outcome/s to be tation review. achieved and inital Has the project delivered VfM? time and cost con- 2 6 What are the straints, ANALYSIS lessons learned? Research and analyze the supply market. Choose appropriate KEY STAGES selection method and approach to market IMPLEMENT IN options. Prepare PPSD Proactively and Procurement Plan. PROCUREMENT manage contract implementation. REQUIREMENTS 5 SOURCE Specifiy the requirements, and 3 Approach the market. Select develop evaluation Most Advantageu- methodology. ous Bid/Proposal. Prepare to go to Award contract. market. 4 Specific activities in each procurement process stage include: analysis, negotiations with potential suppliers may be necessary, resulting in the awarding of a contract. 1. Identification. Activities in this stage involve identifying 5. Implement. The supplier’s performance is monitored for procurement needs and gaining an understanding of cost continuous improvement and a supplier performance re- constraints and time limitations. view is carried out. 2. Analysis. In this stage, research is conducted to identify 6. Check. In this stage, the overall procurement process is potential suppliers, and the sourcing strategy is selected reviewed, and lessons learned are documented to inform and documented. the next procurement operation. 3. Requirements. Here, specific requirements that the ac- quired product or service needs to meet are generated. Although this process may be implemented to purchase the A request for information (RFI) may be released at this most common products and services, it is suggested that a stage to learn more about the market and suppliers’ ability more robust and specialized framework be implemented for to meet the requirements. The evaluation criteria that will GovTech that is more customized to the unique characteris- be used to evaluate supplier proposals are also created. tics of procuring technology for governments. This framework, 4. Source. This stage includes the creation and deployment discussed in section 3, utilizes elements from the standard of a request for proposals (RFP) and the analysis of pro- procurement process but tailors activities specifically to Gov- posals within the context of the evaluation criteria. After Tech acquisition. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 13 3. >>> GovTech Procurement Framework The figure below displays a five-step GovTech procurement framework that provides a strategic and holistic plan for the design, procurement, and implementation of GovTech solutions, with key activities and deliverables for each step. Execution details and considerations for each step are contained in the succeeding sections. Prior to undertaking a GovTech acquisition, it is imperative to establish a business case for the project that considers the following items. 1. 2. 3. 4. 5. Anticipated project benefits and business value including how it aligns to country priorities. Projected costs and proposed high-level implementation plan. Project assumptions, dependencies, and key stakeholders. Project risks and potential mitigation strategies. Expected outcomes if the project is not funded. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 14 > > > F I G U R E 3 - Five-Step GovTech Procurement Framework STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 Technology Requirements Procurement Implementation Ongoing Manage- Assessment and Definition Strategy and Execution ment/Technology Gap Analysis Development Evolution ACTIVITIES • Perform a • Research and • Determine • Develop tender • Establish service- technology assess requirements method(s) to documents level agreements readiness drivers: conduct market • Execute sourcing (SLAs) assessment • Business model research event • Establish key • Create map of • System • Determine • Conduct performance systems and implementation method(s) for negotiations and indicators (KPIs) software affected approach market engagement select vendors • Establish by technology • System security (e.g., prize • Integrate and requirements for implementation • Hosting solutions competitions and embed vendors vendor monthly • Document use of framework performance requirements agreements) reports DELIVERABLES • Technology • Analysis of • Documented • Finalized sourcing • Tracked SLAs and readiness requirements market research strategy KPIs assessment results drivers • Finalized approach • Procurement • Vendor monthly • System Map • Requirements to market analysis performance • Technology documentaion engagement • Final savings reports roadmap calculations • Final terms and conditions 3.1 STEP 1 – Technology Readiness Assessment and Gap Analysis A technology readiness assessment is a systematic analysis of An example of a technology readiness assessment is the “Elec- an organization’s ability to successfully implement transforma- tronic Government Procurement (e-GP) Readiness Self-Assess- tional processes or change. The assessment identifies the poten- ment,” which is “intended to assist a jurisdiction to conduct a tial challenges that might arise when introducing new technology high level review of its procurement environment to determine and any accompanying procedures, structures, and processes its level of readiness to make a transition to e-GP in a sustain- within a current organizational context. As part of a readiness as- able manner” (ADB, IADB, and World Bank 2004). A summary of sessment, a gap analysis is performed to identify gaps or risk the strategic foundations and accompanying components of the areas that should be addressed and remedied before or as part readiness assessment are contained in the table below. Each of the implementation plan. component is assessed on a scale from 1 (no readiness) to 4 (adequate degree of readiness). EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 15 > > > T A B L E 1 - eGP Readiness Assessment Strategic Foundations and Components STRATEGIC FOUNDATIONS COMPONENTS • Government Leadership: vision, sponsorship, resources, stakeholder and INSTITUTIONAL CAPACITY implementation support The capacity of government to set directions and • Human Resource Management: education, skills development, expertise, lead and resource the changes required and career development • Planning and Management: strategic planning and re-engineering of GOVERNANCE management protocols and processes Putting in place the rules, management support, • Policy: setting intent and guidelines that can be consistently applied and performance monitoring and evaluation to • Legislation and Regulation: supporting rules and the external and support eGP internal monitoring of efficiency, performance, and compliance in relation to the total approach to eGP • Infrastructure and Web Services: ensuring the reasonable access BUSINESS FUNCTIONALITY AND STANDARDS to and quality of e-services and their sustainable development and Sustainable infrastructure, support services, and maintenance common standards developed to ensure that ac- • Standards: development of management, procurement, and technical cessible, integrated, and consistent procurement standards to ensure the consistency of the approach to eGP and services can be put in place interoperability across the systems involved THIRD-PARTY INVOLVEMENT • Private Sector Integration: suppliers are enabled and have incentives to Ensuring the private sector can participate in eGP participate in eGP APPLICATION OF TECHNOLOGY • Systems: the planning, selection, development, implementation, and Appropriate, integrated, sustainable, and modifi- support of e-Procurement systems to provide tendering, contract able technology phased in to provide tendering, management, and purchasing services contract management, and purchasing services The outputs of a technology readiness assessment can take many forms. It should conform to a dashboard format that allows for decision-level data to be communicated effectively to leaders. The below figure depicts a “traffic-light” dashboard assessment1 of a tech- nology organization showing the relative performance status of different areas of the organization. It is equally important to demonstrate to leaders what is working and to identify any gaps to be solved via the introduction of new GovTech technologies. > > > F I G U R E 4 - Readiness Assessment Framework (Illustrative Example) Change Structure Process People Leadership Infrastructure Management Procurement Market Vision and Capability to Alignment Procurement Research Resources Strategic Implement w/ Strategic Planning / Business Plan Change Objectives Intelligence Executive Required Organization Category Procurement Skill-sets Buy-In from Resources Structure Management Technology the Business for Change Procure Governance Training to Pay Contract Strategic Performance Communica- / Vendor tions Management Operating at a high standard Some opportunity for improvement High opportunity for improvement Focus Area 1. Developed by Valent. See https://www.valent.com/. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 16 The dashboard-level output is created through the collection of both qualitative and quantitative data. An enterprise survey is a key tool that can be used to generate this type of data and is accurate and hyper-local to the situation being evaluated. A survey can be used to determine gaps as part of a readiness assessment. The figure below shows the results of a survey2 that identified key gaps and areas for improvement for an IT organization that directly helped generate the dashboard in figure III. A sample list of a technology assessment interview and survey questions can be found in Annex I. > > > F I G U R E 5 - Survey Results as Part of a Readiness Assessment (Illustrative Example) Number of Interviewees: Overall Acg. Score: 48 3.9 Perception of Procurement 4.0 as Strategic Partner KEY FINDINGS • High level satisfaction with current IT General Level of organizational changes and customer- 4.2 Satisfaction Amongst Peers centric service approach • Local resources are key assets that Overall Level of 4.2 understand the unique needs of the Satisfaction scientific communtiy • Desire for increased availability of Vendor 3.5 training for common systems and Management applications Procurement Project 3.9 Management Availability/Quality of KEY CHALLENGES 2.7 • Departments operate independently Training from [Field Location] - decentralized Sufficiency of Resources governance/policy, budgets and 4.1 and Skills priorities set by Director • Adoption of new systems and Quality of Procuremen applications may require a cultural 3.7 Org Structure shift among scientific community Procurement 4.0 Communications Governance 2.6 KEY PRIORITIES Procedures • Update infrastructure to enable reliable deliverence of services Alignment w/ Company 4.1 • Continue high level of customer service Strategic Objectives to enable scientists and support administrative staff IT Disaster Recovery/Backup 3.9 • Leverage technology to modernize procurement operations Performance in Developing 3.9 New Systems/Applications 2. Ibid. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 17 Another important exercise is the mapping of any systems and software that will be affected by the incoming technology. The graphic below is a notional example of the mapping of a procurement system environment. > > > F I G U R E 6 - Example e-Procurement System Map CLIENT SYSTEM CLMS SUPPLIER SYSTEM Contract Management Data Interface Groupware System & Supplier Portal Options Option 1: Manual entry Option 2: e-Procurement System Manual download of a CSV- document and manual import Option 3: Real EDI- communication ERP Database / File System EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 18 Once a system map and gap analysis are completed, the next step is to create a digital transformation and modernization road- map. One way to sequence the roadmap is to use a screening process to prioritize GovTech initiatives. The screening process is most effective when the GovTech team creates customized screening criteria that consider the resources and capabilities avail- able to execute projects. The below figure presents an example of how to prioritize initiatives in pursuit of a digital transformation and modernization roadmap.3 > > > F I G U R E 7 - Technology Screening Process Application A Solution A Application B Solution B Application C Solution C Generate List Requirements Operations and Ease of Solutions for of Potential / Functionality Maintenance Implementation High Priority Technology Solutions Screen Screen Cost Screen Screen Areas Roadmap • Utilize results • Deep dive to • Determine ex- • Examine • Gauge extent • Recommend • Prioritization of interviews, ensure that tent to which overall cost to which cur- appropriate of invest- benchmark- key require- solution may (implementa- rent systems solution(s) for ments, imple- ing, and IT ments can be require exten- tion costs, / applications, each high- mentation sector eper- met and key sive ongoing annual fees, infrastruc- priority area schedule, and tise to iden- functionality maintenance support costs, ture, pro- governance tify potential is available and develop- etc.) to elimi- cesses, and changes solutions for • Consider ment (e.g., nate solutions workflow will high-priority compatability customizable that may not be affected areas with existing vs. configu- be realistic • Consider applications rable) due to budget length of im- • Focus on • Take into ac- plementation quality of ap- count desires, and training plication and such as main- requirements outputs (e.g., taining a thin dashboards) administra- tion layer 3. Ibid. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 19 The result of the screening process should be a prioritization of initiatives based on unique criteria developed by the GovTech team to ensure that the roadmap is customized to the organization’s actual needs, requirements, and implementation capabilities rather than to general market recommendations. The output of the roadmap should be simple enough to communicate to internal and external stakeholders; it should also be a dynamic document that can be modified throughout the duration of a GovTech project. The below figure presents an example of a potential GovTech digital transformation and modernization roadmap.4 > > > F I G U R E 8 - Digital Transformation and Modernization Roadmap (Illustrative Example) Source Complete PriorityImplementation Functional of All GovTech Areas Not Functional Transformation Already Areas Procurement of Project Covered GovTech leading #3 to improvement Sourcing in public Implement procurement Project #2 and contract Project 2022 Review management #2 solutions Implement and Sourcing Project Review and Update #3 Update Technology Implement Technology Roadmap Project #1 Generate Functional Roadmap Requirement for 2021 Project #3 Project #1 Sourcing Generate Functional Establish Requirement for Governance Project #2 Council Generate Functional Requirement for Project #1 Perform Approve/Modify Milestone Marker Key Technology Technology Assessment Roadmap Gather Programmatic Governance Functional Requirements Upgrade 2020 The development of the complete set of documents in Step 1 – Technology Assessment and Gap Analysis will directly lead to improved outcomes throughout the rest of the process. Establishing an agreed understanding of gaps and a roadmap to the future is critical to the long-term success of any GovTech initiative. 4. Ibid. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 20 3.2 STEP 2 – Requirements Definition When defining technology requirements, there are four foun- When the PPP business model is implemented, the operator dational items that should be considered (World Bank n.d.). of the service defines the charges that will be levied against the government to cover system sustainment. The govern- Foundation #1 – ment should review and understand all costs to ensure that they are fair and equitable. The government may rely on ex- Business Model ternal expertise or project benchmarking firms to confirm costs After the initial funding phase, a government should embrace are in line with similar PPP projects and business models prior a specific business model to ensure the sustainability of oper- to the execution of the PPP. ations, including software/hardware maintenance, promotion/ marketing activities, help desk operations, training services, When implementing any of the three business models (gov- and the continuous evolution of strategic objectives. Obtaining ernment owned and operated, government-managed ser- funding to support the system must be a shared responsibility vice, or PPP shared service), the government is exclusively among the various stakeholder groups that use the system. responsible for elaborating a plan that ensures sustainability. When designing the model, it is critically important to ensure While sustainability and operational efficiency requirements that all the underlying costs of running the system are under- can be introduced into bidding documents and vendor con- stood and accounted for. tracts, long-term sustainability outcomes rely on political and leadership commitment to change management efforts with The most common business models used by governments to a long-term outlook and focus on project and organizational establish GovTech systems include the following: sustainability. Operations may become sustainable through a combination of fixed capital funding and ongoing operational • Government Owned and Operated. The government revenue generation. From a human resource perspective, a owns and operates the GovTech system, which is built by combination of government employees as well as specialist its own information and communications technology (ICT) service providers can be used to ensure a smooth operation. team. Alternatively, external partners may be used dur- These providers may be involved in many aspects of the sys- ing the development or support phases of the project, but tem, including technical operations and maintenance, end- these partners have no rights to any part of the system. user training, and end-user support through a dedicated help They are compensated for their services based on con- desk. Regular upgrades to the underlying technology plat- tracted fees. forms should be planned to ensure that the system continues • Government-Managed Service. In this approach, to operate over a lifetime that exceeds that of the physical the system is operated, supported, and owned by a third- technology components. party partner, but the government retains ownership of all the data and support services, such as a user help desk and training. A dedicated unit within the government is Foundation #2 – responsible for managing the service, including the per- System Implementation Approach formance of the third-party partner as well as any other The table below summarizes the advantages and disadvan- government departments that provide support services tages of three acquisition approaches to establishing a Gov- such as training. Tech system, including implementation of a software-as-a-ser- • Public-Private Partnership (PPP). In this model, the vice (SaaS), commercial-off-the-shelf (COTS), or custom-built GovTech system is owned and operated by a third-par- solution. The information is applicable to a broad range of ty service provider. The intention of this model is that at GovTech systems. some time in the future as specified in the agreement, the service will be transferred to the government. In the in- terim, the service provider is compensated either through a fixed monthly fee or transactional revenues based on the usage of the system. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 21 > > > T A B L E 2 - SaaS/COTS/Custom Build System Acquisition Considerations ATTRIBUTE SAAS COTS CUSTOM BUILD • System based on an already tested • System based on an already tested • Software is owned by government and used GovTech platform and used GovTech platform • Software is built for purpose • Vendor has know-how on analyzing, • Vendor has know-how on analyzing, • System development know-how configuring, and rolling out GovTech configuring, and rolling out GovTech would be accumulated systems systems • Government may have difficulties in • Mature product as a foundation for • Mature product as a foundation for retaining ICT experts in its team since a system that can be maintained a system that can be maintained the private sector can offer more virtually locally competitive salaries • Lower risk of technological obsoles- • Lower risk of technological obsoles- • The possibility of risks to the success cence cence of system development • Limited customization because of the • System may impose functional con- • Lack of proven skills (offset by includ- Quality other users of the platform straints as it is already pre-built ing foreign specialists) • System may impose functional con- • System may adopt standards that • Potential of fragmented develop- straints as it is already pre-built may be incompatible with existing ments between agencies unless • System may adopt standards that government systems and/or infra- carefully managed by the steering may be incompatible with existing structure committee government systems and/or infra- • Risk of lock-in by module vendors, • Moderate risk of lock-in by local structure mitigated by simplicity and service- developers (mitigated by systems • Risk of lock-in by module vendors, ability of modules portability) mitigated by simplicity and service- • If government has proven, internal ICT ability of module capacity, high-quality product can be achieved; otherwise, in-house develop- ment may be risky • Core system functions are already • Core system functions are already • Time required for the development of available in the vendor’s GovTech available in the vendor’s GovTech from-scratch implementation can be platform platform expected to be more than for a COTS- • Very short period required for rel- • Relatively short period required for based system evant and practical functionality to relevant and practical functionality • If government has proven internal ICT Time be operational. to be operational capacity, analysis, development, and • Vendor may require more time than • Vendor may require more time than rollout of the system can be achieved in-house staff to understand the in-house staff to understand the in reasonable time; otherwise, exces- country-specific requirements country-specific requirements sive time may be required • Economies of scale created by a • Purchase would only be for the func- • Maintenance/evolution of the system shared service often make it a very tionality for which there is capacity will be cost efficient, since all required cost-effective choice to utilize expertise related to the system archi- • Purchase would only be for the func- • Low initial implementation costs tecture will already exist tionality for which there is capacity • Low-risk system development and • Government may need to make to utilize maintenance costs special financial arrangements for • Low initial implementation costs • Government will be contractually retaining experts for the development • Low-risk system development and bound to the vendor and system; if and maintenance/support of the maintenance costs a clear financial arrangement is not system Cost • Government will be contractually defined right from the start, mainte- bound to the vendor and system; if nance/evolution cost may be large a clear financial arrangement is not • At the end of the contractual term defined right from the start, mainte- with the vendor, process for contract- nance/evolution cost may be large ing and migrating to a new GovTech • At the end of the contractual term system may be costly with the vendor, process for contract- ing and migrating to a new GovTech system may be costly • Solution must be hosted in vendor’s • Solution can be hosted on-premise or • Solution can be hosted on-premise or IT environment, which often resides Security and in domestic cloud to control access in domestic cloud to control access outside of client country’s geographical Access and conform to national data privacy and conform to national data privacy borders, which may violate security or laws laws data privacy laws EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 22 A universal best practice for a GovTech system acquisition General Data Privacy Regulation (GDPR), and any GovTech does not exist; rather each model suggested in the above system should have the ability to meet this requirement, or table has different pros and cons that are dependent on the at a minimum meet the established data privacy laws in the country environment and implementing agency technical and implementing country. Other security requirements include: financial capacity. • The system should be able to store sensitive information in encrypted formats. SaaS models present the most efficient time-to-market op- • Information must be made available only to authorized tion if the implementing country can accept configurations users. This should be ensured using a role-based secu- vs. customizations to meet technical and functional require- rity model that will limit user access following a pre-de- ments. Existing system training materials can also quickly be fined set of access rights. This approach will ensure that adapted to increase the pace of implementation and roll-out. the data are not disclosed to any unauthorized person A competitive procurement action can result in market-based or process. pricing for an initial period in production, which should keep • All communications with end-user browsers must be per- costs reasonable. formed over a secure and encrypted communications pro- tocol (e.g., HTTPS). COTS models can also be quick to implement but allow for • The system must be safeguarded against all known greater individual client customization. This can be both security threats (e.g., buffer overflows, Structured Que- beneficial as the system can be adapted to specific func- ry Language [SQL] injection, cross-site scripting [XSS], tional requirements and business process needs, but there denial-of-service [DoS], distributed denial-of-service is both a cost and implementation delay associated with this [DDoS], etc.). practice. System Operations & Maintenance (O&M) costs may • A proven anti-virus protection system should also increase significantly over time as the number of customiza- be deployed. tions increases. Foundation #4 Custom-Build will take the longest period to implement but Hosting Solutions is the most flexible model to conform the system to specific Cloud technologies are now mainstream, as they are country functional and technical requirements. Cost may also used heavily in the private sector and gaining traction in the be quite efficient if the local IT labor market is robust and costs public sector. The cloud technology to be considered for re- are low. The uses of external system development consultants placing a data center is platform as a service (PaaS), which could drive costs up significantly during the custom build and allows customers to create, run, and manage applications over the lifecycle of the system. without the complexity of building and maintaining the infra- structure typically associated with developing and launching A client country’s current and forecasted IT technical capac- such applications. ity is also an important factor in considering different types of system acquisition models. SaaS systems are the easiest to By using the PaaS model instead of designing, building, own- manage over time as the majority of the system IP and sup- ing, and operating a data center, government agencies can port resides with the system provider or a 3rd party service move from capital expenditures (CapEx) to operating expen- provider. COTS systems may require more in-house IT ex- ditures (OpEx). By using cloud technologies, agencies can pertise to manage the system and any customizations. Cus- “leap-frog” the need to build their own ICT server infrastruc- tom Build systems require the most internal IT capacity as the ture. One excellent feature of cloud technologies is that as systems must be maintained by the implementing agency and one pays only for what is used, there is no wasted process- meet operational standards such as continuous uptime, which ing or storage. Cloud technologies can easily scale. Any Gov- requires both technical expertise and personnel resources. Tech procurement strategy should investigate the use of cloud services and have a cloud category strategy to manage the Foundation #3 evolving use of the space. Governments may consider build- System Security ing a domestic cloud solution to be compliant with any national Security is an essential feature of any GovTech system, espe- security or data privacy considerations. cially one that hosts confidential, commercially sensitive infor- mation or personal identifying information (PII). The European A critical enabler of GovTech solutions is an appropriate legal Union (EU) has taken the global lead in data privacy with its and regulatory framework that enables adherence to interna- EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 23 tional standards on open government and data transparency, Requirements Documentation while recognizing country specific constraints posed by na- After addressing each of the foundational items for consid- tional security and data privacy laws. Sustained political and eration, a requirements document should be compiled. The leadership commitment are key to both the creation and evo- document should capture all the requirements in a format that lution of these frameworks as well as providing the financial can be shared with vendors for their review and response. resources to allow for both the build and sustainable main- Requirements documentation is especially important if there tenance of GovTech systems. Policymakers have dual re- is a need to pursue a custom-built solution. sponsibility to create an environment suited to both GovTech innovation and conducive to significant change management To evaluate if existing SaaS or COTS systems meet require- that must occur to take advantage of the benefits of public ments, a fit gap analysis can be performed. Fit gap analysis in- sector modernization. volves comparing existing requirements to the functionality of systems to identify which requirements are met (fit) and which are not met and may require customization or an alternative solution (gap). The figure below shows a representation of fit- gap analysis5. > > > F I G U R E 9 - Fit Gap Analysis Buziness System Fits Requirements Functionality Gaps 5. Adapted from Ancveire 2018 EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 24 Alternatively, the requirements documents can be shared with quirement that should be explained. The requirements docu- vendors of SaaS or COTS systems to collect their feedback ment is critical as it will drive decisions around the eventual on how well the software meets requirements, as shown in procurement of a system. The figure below captures a sample the figure below6. Typically, vendors are allowed to respond of requirements for a contract lifecycle management system that 1) they can meet the requirement, 2) they cannot meet (CLMS) and vendor responses. the requirement, or 3) there is a notable “exception” to the re- > > > F I G U R E 1 0 - System Requirements Document with Vendor Responses (Illustrative Example) Vendor Vendor Vendor Vendor ID Category Sub-Category Requirement Description 1 2 3 4 Business Charts and The system has the ability to generate graphs and 1 Intelligence/ Comply Comply Comply Comply Graphs charts (exporting data to Excel is not necessary) Reporting Business Data The system allows restricted users to generate reports 2 Intelligence/ Comply Comply Comply Comply Restrictions based on their role based data access Reporting Business Data The system provides read-only and drill down access 3 Intelligence/ Comply Comply Comply Comply Restrictions for specified end users (e.g., Contract Officers) Reporting Business Data The system provides dynami reporting (drag and drop Excep- 4 Intelligence/ Comply Comply Comply Restrictions capabilities) tion Reporting Business Fields and The system can generate reports using any available 5 Intelligence/ Comply Comply Comply Comply Dimensions field or dimension (including data flags) Reporting Business The system allows users to schedule when customized Scheduled 6 Intelligence/ reports are run and provides notification via email that Comply Comply Comply Comply Reports Reporting the report was run Business The system can provide a self-service dashboard for 7 Intelligence/ Self-Service end users (e.g., contract versus actual, spend by ADB Comply Comply Comply Comply Reporting group, spend by vendor, etc.) Document Document The system has the ability to associate any scanned Does Not Excep- 8 Comply Comply Management Assocation document with any particular record Comply tion The system provides the capability to store, retrieve (e.g., transaction clicking), display and print imaged Document Document Does Not Excep- 9 documents. The solution must provide the capability to Comply Comply Management Storage Comply tion receive scanned documents from document scanners having industry-standard connection protocols The system can accept scanned or electronic docu- Document Document Excep- 10 ments in any format (.pdf,.xlsx,.doc) and link them to a Comply Comply Comply Management Storage tion data record The system can generate report files in delimited, Document Excep- 11 Electronic Files ASCII, JSON, PDF, MS Word, MS EXCEL, and XML Comply Comply Comply Management tion formats Document Record The system can produce an electronic file that will main- 12 Comply Comply Comply Comply Management Retention tain records per ADB’s Retention Schedules The system has the ability to link to files located in Document System Does Not 13 a document management system (e.g. SharePoint, Comply Comply Comply Management Integration Comply OneDrive etc.) The system provides field level edits to ensure validity of 14 General Audit Comply Comply Comply Comply the data being entered into the system 6. Developed by Valent EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 25 The requirements are critical to determining a comparative most technically compliant, while also allowing for a ranking technical evaluation of GovTech functionalities and technical of all participating vendors. This evaluation is critical to the architecture. The below figure shows an example of a sum- overall procurement selection process. mary table of an evaluation7 indicating that Vendor 4 is the > > > F I G U R E 1 1 - Example Summary Technical Evaluation (Illustrative Example) Vendor 1 Vendor 2 Vendor 3 Vendor 4 Requirements Does Does Does Does Excep- Excep- Excep- Excep- Categories Comply Not Comply Not Comply Not Comply Not tion tion tion tion Comply Comply Comply Comply Business Intelligence/ Reporting 6 1 0 7 0 0 7 0 0 7 0 0 7 Requirements Contract Managements 8 0 0 6 2 0 5 1 2 8 0 0 8 Requirements Document Management 3 1 2 3 3 0 5 0 1 6 0 0 6 Requirements General (e.g. Audit) 23 2 0 23 2 0 24 1 0 25 0 0 25 Requirements Procurement 14 12 2 0 14 0 0 13 0 1 14 0 0 Requirements Information Security 6 2 0 8 0 0 6 1 1 8 0 0 8 Requirements Total 58 8 2 61 7 0 60 3 5 68 0 0 Total 85% 12% 3% 90% 10% 0% 88% 4% 7% 100% 0% 0% Percentage 7. Ibid. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 26 3.3 STEP 3 – Procurement Strategy Development 3.3.1 Conducting Market Research Procurement strategy development begins with market re- In 2016, the World Bank issued strategy paper entitled, search. Market research involves the collection, recording, and “Guidelines for Conducting Market Analysis for e-Procure- evaluation of data about a specific market that offers a desired ment Systems” (World Bank 2016). Although this document product or service. The primary purpose of market research is was developed specifically for e-Procurement systems, the to inform the decision-making process, leading to a gain in com- underlying process and templates can be extended to other petitive advantage, reduced risks, and increased productivity in technologies. There are several methods available for market a future investment. Conducting market research increases the research as summarized in the table below. The method that awareness of the involved stakeholders and assists them in is selected should depend on such factors as the availabil- understanding a market’s presence, capacity, product or ser- ity of information (e.g., via internet research), the amount of vice offerings, and associated costs. A successful undertaking time a buyer has to accomplish the task, and the availability of market research contributes to the definition and refinement of financial and personnel resources to complete the research of product or service requirements and to a better understand- activities. Although completing a single approach may provide ing of the available vendors and their offerings. By conducting enough information for decision-making purposes, completing market research, governments can better position themselves multiple approaches will provide the most comprehensive in- to successfully execute a public procurement action. sight into a market. > > > T A B L E 3 - Methods for Conducting Market Research METHOD DESCRIPTION • Involves investigation and analysis of information that is readily available and easily accessible, Desk-Based mainly through the internet. Published reports, industry/government/supplier websites, interviews Research with industry experts, and discussions with institutional representatives in other countries are among the sources of information that can be used. This method is both time and cost effective. • Surveys and questionnaires are effective means to collect information when other methods (e.g., desk-based research) do not yield the required information. Surveys and questionnaires allow for Surveys/ quantitative measurements for processing (such as the number of potential local vendors that Questionnaires support a specific requirement). Also, web-based surveys are easily distributed, and respondents can reply via e-mail. • System demonstrations allow vendors to present the functionality of their system to the potential System buyer in an interactive manner. Demonstrations may be performed in face-to-face meetings or Demonstration remotely (e.g., WebEx). Demonstrations also help buyers understand a systems graphical user interface (GUI), usability, translation capabilities, and help features. • Direct communication with vendors is a powerful technique in performing market research. Several processes and tools can be used to formally or informally communicate with vendors, as outlined below: • Formal communication methods include issuing a request for an expression of interest and a request for information (RFI) to collect information on the vendors and their ability to deliver a desired Communication solution. Issuing a request may also stimulate interest in potential vendors. Methods with • Informal communication methods include e-mail communications, advertisements in publications, Vendors and social media (e.g., Twitter, Facebook, LinkedIn), which can be used to leverage any interest in and familiarization with potential vendors regarding the advertised project. In this context, potential vendors are informed about a country’s intention to procure a product or service and can closely follow any future announcements related to the project. Involvement and/or • External consultants and specialists can typically complete some or all market research activities Full Assignment to in a timely and accurate manner. Use of consultants and specialists may not be as cost effective as External Consultant/ completing market research activities with internal resources. Specialist EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 27 The primary activities involved in completing market research referenced in the table below. The resulting analysis of these include developing a product or service profile, researching market components can be used to gain an understanding of suppliers, and identifying key market indicators. Conducting the external environment in which specific products are pro- market research also involves gaining an understanding of duced and sold and will enable the creation of a comprehen- industry trends and analyzing industry forces. A number of sive and in-depth supply market strategy. research analyses and tools used extensively in industry are > > > T A B L E 4 - Key Market Research Components MARKET COMPONENT DESCRIPTION Category Supplier An overview of the supplier market detailing different characteristics and their varying impact on the over- Summary all market. Insightful conclusions should be drawn based on a full understanding of the researched data. Market Overview/ Summary and descriptions of market characteristics and conditions, company market position, Segmentation potential challenges, industry success factors, industry cost structure, etc. Methodology and diagram detailing five competitive forces that shape each industry: rivalry among Five Forces Analysis existing competitors, bargaining power of suppliers, threat of new entrants, bargaining power of buyers, and threat of substitute products or services Analysis addressing such topics as general market conditions and potential supplier impact (e.g., technology advancements, capacity issues); competitor analyses (e.g., entrance of new industry Market Trends players); economic impacts; economic demand for a product/service; market trends and forecasts; financial analyses and reports (e.g., overall market revenue, profit margins); government regulations Supplier Chain/ Investigation of different distribution channels; supplier market size; distribution landscape; product/ Distribution Market service alternatives Analysis identifying the cost of each activity within a company and determining the importance of each Value Chain/Cost activity through the entire process (e.g., manufacturing, transportation). Additionally, the value chain Structure details the overall cost of sourcing a product/good and helps to identify links between activities. Major Players Detailed list of industry leaders as well as their financial profiles and product/service offerings Profiles of Major Company profiles include general company overview (e.g., mission, headquarters, flagship products/servic- Suppliers es), financial profile (e.g., profits, historical performance, assets), company capabilities, company leaders The table below contains key areas to investigate and corresponding questions that will assist in forming a comprehensive view of the supply market. > > > T A B L E 5 - Key Areas for Market Research MARKET COMPONENT DESCRIPTION • How large is the market for this category? Market Size and • What is the growth history and predicted growth of this category? Growth • Is this market regional, national, global? • Are there acceptable substitute products for this category? Product Substitution • Has a formal “value analysis” of category substitutes been completed? Industry Profitability • How profitable is the industry supplying this category? • How concentrated or fragmented is this category’s industry? Industry Structure • Do a few leaders dominate the industry? • Are there technology leaders? Technology Trends • What has been the impact of technology on costs, pricing, quality, and service? • What effects do customers have on the competitive environment? Buyer Profiles • Do a few large customers dominate the suppliers? • How high are switching costs for the customers? • Who are the main suppliers in the market? Supplier Profiles • Does the supply base in this category create any type of supply risk? • Are there any issues with delivery, cost, quality, or responsiveness from suppliers? EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 28 In the event that additional market research recommenda- • Market Analysis Report Template. The results of market tions are required, the World Bank’s “Guidance for Conduct- research activities should be documented in a way that ing Market Analysis for e-Procurement Systems” includes a points to the main conclusions. This template comprises number of templates that a buyer can use to aid in conducting four proposed sections: purpose of the analysis and its market research activities (World Bank 2016). Excerpts from participants, a brief description of the methodology, core each template can be found in Annex II of this document. The results, and main conclusions. templates include: • Request for Expression of Interest. This template is • Invitation Template. This template can be tailored and a suggested letter from the buyer to vendors that invites sent to a select set of vendors invited to participate in mar- them to express their interest in implementing the desired ket analysis activities. It provides the vendors with impor- product or service. tant information, including the purpose and objectives of the analysis and any procedural and contact information. Established market research can be used to accelerate the • Questionnaire Template. This template includes a sug- understanding of a key market. As an example, a Gartner gested set of questions for vendors. It contains ques- Magic Quadrant Analysis is shown in the figure below (Gart- tions related to the vendors’ profile (experience, geo- ner 2019). The analysis provides a graphical competitive posi- graphical coverage, etc.) and to their system offering. tioning of four types of technology providers in markets where growth is high and provider differentiation is distinct. > > > F I G U R E 1 2 - Gartner Magic Quadrant, Procure-to-Pay Suites (June 2019) CHALLENGERS LEADERS Challengers execute Leaders execute well today or may well against their dominate a large current vision and segment, but do are well positioned Coupa not demonstrate an for tomorrow Ivalua SAP (Ariba and Fieldglass) understanding of GEP market direction Oracle JAGGAER Basware Zycus SynerTrade Wax Digital Niche Players focus Determine Visionaries successfully on a understand where small segment, or Tradeshift the market is going ABILITY TO EXECUTE are unfocused and do or have a vision for not out-innovate or Proactis changing market outperform others rules, but do not yet execute well NICHE PLAYERS VISIONARIES COMPLETENESS OF VISION As of June 2019 © Gartner, Inc Additional sources that may be useful in conducting market research include Forrester, Kennedy Information, Factiva, IBISWorld, and industry papers and trade association research. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 29 3.3.2 Competitive Dialogue as a Process for GovTech Procurement For complex contracts, the use of competitive dialogue should • Does the procuring organization want to encourage and be considered. Competitive dialogue is an interactive multi- allow innovation and refrain from defining the means stage selection process that facilitates dynamic engagement through which the needs should be delivered? between buyers and vendors. In 2017, the World Bank issued • Could the needs be met through several different solutions? procurement guidance entitled, “Competitive Dialogue – How • Are there several potential options that could be adopted to Undertake a Competitive Dialogue Procurement Process” to provide the commercial element of the overall solution? (World Bank 2017b), which outlines that this kind of dialogue • Is the contract unique or unusual, that is, no previous pro- is most suitable for undertaking procurements where: curements have been undertaken by the procuring orga- 1. A number of alternative solutions that satisfy require- nization for similar requirements? ments may be possible, and the detailed technical and • Is the procuring organization sure that other procurement commercial arrangements required to support those so- selection methods and arrangements do not allow for the lutions require discussion and development between the required level of collaboration between the procuring or- procuring organization and the supplier. ganization and supplier to enable the development of an acceptable solution? 2. Due to the nature and complexity of the procurement, the • Does the procuring organization have sufficient resources procuring organization is not objectively able to: to devote to an intensive procurement process that may a. Adequately define the technical or performance spec- last 12–18 months and require a high level of inputs, re- ifications and scope to satisfy its requirements; or sources, and costs (especially in relation to preparation, b. Adequately specify the legal and/or financial arrange- rounds of dialogue, and proposal evaluation)? ments of the contract. • Is there the potential for a high level of market interest and therefore strong competition? The tests used to establish if a competitive dialogue is the op- • Have other procurement selection methods and arrange- timum fit-for-purpose procurement approach include: ments, such as the use of an RFP, with a negotiation • Are the needs clear, but the means of delivering these stage, been assessed and discounted as not appropriate needs undefined? for the contract? > > > F I G U R E 1 3 - Six Stages in the Competitive Dialogue Procurement Process STAGE NAME DESCRIPTION Planning for a Before buyers begin a Competitive Dialogue, they should ensure that they have fully STAGE Competitive planned for the complexity and duration of the process. This includes establishing 1 Dialogue appropriate work streams, appropriate resources, and dedicated teams. STAGE Early market Buyers should consider how to prepare the market and stimulate competition 2 engagement The use of Initial Selection is mandatory in a Competitive Diaglogue. Rated STAGE Advertise and criteria are used and only the best Applicants (normally not less than 3 and not 3 Initial Selection more than 6) are Initially Selected Request Interim Proposers submit Interim Proposals that address the buyers “needs”. One-on-one STAGE Proposals and dialogue between the buyer and each vendor leads to a refinement of the proposed 4 Dialogue solutions. Request Final STAGE Once that dialogue stage closes, the buyer invites the submission of final Proposals, Evaluation 5 proposals and undertakes final evaluation of the proposals. and Contract Award STAGE Contract During contract implementation, the buyer needs to have strong oversight in 6 Implementation place, supported by active and well-resourced contract management. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 30 As with an RFP, the procurement arrangements for competitive • Reach beyond the “usual stakeholders” to increase the dialogue need to be combined with pre-market engagement, the number of perspectives working to develop solutions for choice of specification type (performance), the pricing and cost- a specific problem. ing mechanism, the total cost of ownership (TCO) methodology, • Foster interdisciplinary collaboration and perspectives, and the evaluation methodology in order to complete the full con- reflective of the modern research environment. tract strategy. There are no award value limitations to competitive • Inspire the next generation of scientists, engineers, and dialogue for acquisitions; a single large award or multiple smaller entrepreneurs to work on difficult and important problems. proof-of-concept award can be made using this method. The U.S. government has set up the website challenge.gov, the official site for prize competitions across the federal gov- 3.3.3 Government Prize Competitions ernment. Since 2010, nearly 1,000 challenges have been Government prize competitions are a way to bring scientific posted by more than 100 federal agencies. Among the chal- and technological innovation to benefit citizens by offering lenge types listed on the website are: monetary and other benefits (e.g., recognition) to the winning • Analytics: Analytics, visualization, and algorithm chal- participants. Prize competitions have a long history in both the lenges focus on finding better ways to interpret or com- public and private sectors and have risen in popularity in re- municate data. cent years. Prize competition awards are typically under US$1 • Software: In a software and app development challenge, million for winning submissions. an organization asks solvers to create a software appli- cation to solve an existing problem or draw attention to United States potential uses of available datasets. In the United States, the America COMPETES Act, passed by • Technology: Technology demonstration and hardware Congress in 2010, gives all federal government agencies the challenges seek prototypes or fully developed solutions authority to conduct prize competitions, making “it dramatically to catalyze and demonstrate breakthrough technical in- easier for agencies to use prizes and challenges to spur inno- novations. vation, solve tough problems, and advance their core missions” (White House 2010). According to government leadership, the Challenges are structured into five phases as outlined in the use of prizes and challenges allows the federal government to: figure below, which is presented from the perspective of a gov- • Pay only for success and establish an ambitious goal ernment agency.8 without having to predict which team or approach is most likely to succeed. > > > F I G U R E 1 4 - Five Phases Challenge Structure PHASE 1 PHASE 2 PHASE 3 PHASE 4 PHASE 5 Prepare Develop Conduct Award Transiction • Framing of the • Determine the cal- • Execute your • Work with internal • Analyze and problem lenge structure and communications teams to expedite document results, • Estimate necessary implementation plan payment and outcomes and impat resources timeline • Accept submissions document processes of the challenge and consider • Work with internal and interact with • Explore non- • Stay engaged with partnerships groups to estab- respondents monetary incentives participants within the outside lish eligibility and • Evaluate for participants • Consider ways government submission require- submissions and to transition • Identify goals and ments, terms and select winners and incorporate desired outcomes conditions, and judg- promising solutions • Determine if a ing criteria into government challenge is the • Develop announce- appropriate tool for ments and ongoing achieving results outreach strategy to engage potential respondents 8. See https://www.challenge.gov/toolkit/. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 31 European Union The Budget, Performance Review, and Strategic Planning The EU has established a number of awards and cash prizes Vice Presidency Corporate and Middle Office (BPSCM) de- to spur innovative solutions for society.9 In particular, the Ho- termines the budget cap for any individual prize and the ag- rizon Prizes are “challenge prizes (also known as inducement gregate budget cap for a vice presidential unit (VPU) for prizes prizes) offering a cash reward to whoever can most effectively or awards per fiscal year. The budget cap for any single prize meet a defined challenge. The aim is to stimulate innovation is US$100,000 per fiscal year, and the aggregate budget cap and come up with solutions to problems that matter to Euro- per fiscal year is as follows: pean citizens.”10 The key benefits for society, as outlined by • US$1 million for the Global Practice Vice Presidency the European Commission, include: (GPVP) (all GPs combined) • Providing a way to advance innovations that would not or- • US$100,000 for each region, Development Economics dinarily develop through traditional routes, such as grants Vice Presidency (DEC), and Leadership, Learning, and or procurement. Innovation (LLI) • Attracting a wider range of innovators, not the ‘usual sus- pects’, because they have low entry barriers including not Any expenses incurred as part of the contest or prize program requiring a track record and because they are often excit- by the Bank are exempted from the award cap amounts. Priz- ing and inspiring. es and awards are intended to foster innovation from third par- • Providing an opportunity for innovators to take a risk and ties, and Bank staff and consultants are ineligible for awards to forge new partnerships. under this program. • Leading to sustainable new products and services spur- ring interest in a particular issue11. Country Example GovTech Polska, the Polish government’s technology agen- The World Bank cy, recently launched a virtual prize competition with the aim The World Bank recognizes the use of prizes and awards as of combatting the effects of the coronavirus epidemic13. The an incentive for innovation and collaboration with industry, competition was attended by almost 2,000 developers who academia, and civil society organizations (CSOs) in pursuit were competing for a prize pool of EUR 21,920. Participants of common development objectives and permits their use developed solutions over the course of three days for five via the provision of Bank administrative funds and/or Trust key challenges areas, including security during an epidemic; Funds for International Bank for Reconstruction and Devel- business, science and technology during the crisis; e-com- opment (IBRD) or International Development Association merce and logistics; education and leisure; and freestyle (IDA) activities.12 (open category). 9. For more information, see the European Commission’s “EU Awards and Cash Prizes for Innovative Solutions to Societal Challenges,” https://ec.europa.eu/info/research-and-innovation/funding/funding-opportunities/prizes_en. 10. See https://ec.europa.eu/info/research-and-innovation/funding/funding-opportunities/prizes/horizon-prizes_en. 11. Ibid. 12. This process is outlined in World Bank (2014a) and (2014b). 13. For more information, see https://www.thefirstnews.com/article/polish-government-launches-virtual-hackathon-to-fight-coronavirus-11641 EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 32 3.3.4 Small Business Innovation Research Funding and Grant Opportunities The highly competitive Small Business Innovation Research ture, education, energy, health care, and transportation. A nota- (SBIR) program encourages small businesses to take part ble benefit for the small businesses that participate in an SBIR in research and development activities with the potential for program is access, via the government and large businesses commercialization, providing an incentive for participation.14 that partner with the government, to local knowledge and logis- The mission of an SBIR program is to support scientific excel- tical strengths to scale-up innovative solution pilots and proof- lence and technological innovation through the investment of of-concepts across large geographical regions of the country, research funds in areas of critical importance to the funding for which start-ups do not typically have the relevant experience organization. From a government perspective, typical goals of or the logistical strength. Direct sub-contracting with large busi- an SBIR program include: nesses can help small businesses scale quickly and efficiently. • Stimulating technological innovation SBIR-like programs have been established in several countries • Meeting government research and development needs to spur innovation, including India15, Japan, and Sweden. • Fostering and encouraging participation in innovation and entrepreneurship The United States is widely considered the most successful in • Increasing the private sector commercialization of inno- establishing an SBIR. The program was launched in 1982 and vations derived from government research and develop- currently supports activities in 12 agencies across the gov- ment funding ernment, including the Department of Energy, Department of Health and Human Services, Environmental Protection Agen- SBIR programs can be implemented to develop products and cy, and National Science Foundation. The SBIR program is services across many sectors, including technology, agricul- structured in three phases, as shown in the figure below. > > > F I G U R E 1 5 - Phases of the U.S. SBIR Program PHASE I PHASE II PHASE III Objective Objective Objective • 1. Establish the technical merit, • Continue the R&B efforts initiated in • Pursue commercialization objecives feasibility, and commercial Phase I resulting from R&B activities potential of the proposed research & • Funding is based on the results conducted in Phases I and II development (R&D) efforts achived in Phase I and the scientific • The SBIR program does not fund • 2. Determine the quality of and technical merit and commercial Phase III performance of the small business potential of the project proposed in • Phase III may involve follow-on awardee organization prior to Phase II non-SBIR funded R&B or production providing further federal support in • Only Phase I awardees are eligible contracts for products, processes, or Phase II for a Phase II award services intended for use by the US • Awards normally do not exceed US$ • Awards normally do not exceed US$ Government 150,000 total costs for 6 months 1,000,000 total costs for 2 years Under the most recent SBIR policy, a program participant has 20 years of data rights and IP protections from the results of an SBIR award. During this time, the government has limited rights to use the data or software internally while the participant can pursue op- portunities in the private sector. At the end of the 20 years, the participant maintains ownership of all data and software, while the government obtains a royalty-free license to use the data for government purposes, including the right to authorize other organiza- tions to have access to it (Morlock 2019). As an example of an SBIR program success story, the U.S. National Institutes of Health funded Sanaria,16 a biotechnology company that was working on a malaria vaccine. SBIR funding supported the company in obtaining laboratory and office space and hiring per- sonnel. As a result of the initial funding, the company transitioned to securing the additional funds required to scale up and optimize manufacturing, plan for approval from the U.S. Food and Drug Administration and other regulatory authorities and launch the vaccine as a commercial product. 14. For more information, see https://www.sbir.gov. 15. For more information, see https://birac.nic.in/desc_new.php?id=217 16. See https://www.sbir.gov/node/1193461. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 33 3.3.5 Public Private Partnerships for GovTech Implementation The World Bank and client partners can explore technol- ing a complex GovTech initiative, a pre-procurement market ogy improvements in partnership with private sector technol- approach should be taken that allows the procuring entity to ogy firms and CSOs engaged in government modernization interact with industry in an open and transparent manner while without compromising the integrity of the public procurement avoiding conflict of interest. process. A PPP can be formed when working with technol- ogy firms. In this partnership, the technology developed by One method of executing this market approach is to hold a the firm is deployed within its partner organization, but the firm pre-procurement conference at which the procuring entity will maintains ownership and is responsible for technology opera- present a: tions. The intention is that at some future time as specified in I. Project summary the agreement, the technology will be transferred to the public II. Project timeline, including anticipated solicitation dates sector partner. In the interim, the technology firm is compen- III. Draft procurement package of technical and functional re- sated either through a fixed monthly fee or transactional rev- quirements for industry comment. enues based on the usage of the technology. Although the pre-procurement conference schedule and logis- Due to the technological complexity typically associated with tics are unique to each project, the common objectives of each a GovTech initiative, it is imperative that the project team en- conference are: gage with the market early and often to amplify the procure- 1. To inform the market of the opportunity ment opportunities for market leaders to help craft GovTech 2. To solicit market feedback on the draft functional and solutions with optimal outcomes. Equally critical to any public technical requirements procurement initiative is the need to adhere to well-established regulations, policies, and procedures aimed at preserving pro- Pre-procurement conferences should be held in person when curement principles, such as fairness and transparency, and possible, with participating industry members able to review to avoid conflict of interest. physical and electronic copies of technical and functional specifications in order to provide input and comment. The Once a formal public procurement process has begun, an procuring entity, at its sole discretion, can choose to alter or arms-length relationship is necessarily established between enhance specifications based on industry feedback prior to the procuring entity and industry. Therefore, when undertak- engaging in the formal procurement process. > > > T A B L E 6 - Pre-Procurement Market Engagement Process PRE-PROCUREMENT MARKET SPECIFIC ACTIVITIES ENGAGEMENT PROCESS Pre-Procurement 1. Outreach to market leaders Market Engagement 2. Outreach to industry associations and chambers of commerce (3 weeks) 3. Outreach to peer organizations and development banks Pre-Procurement 1. Presentation of project summary Conference 2. Presentation of procurement package opportunity (1-Day Event) 3. Discussion of draft functional and technical requirements Industry Feedback and Review 1. Process and publish industry comment and feedback (TBD, based on results of 2. Update requirements conference) 3. Prepare procurement documents for formal procurement process The result of the pre-procurement market engagement should be both improved functional and technical requirements via expert industry feedback and the greater participation of market leaders in an eventual procurement. Both factors will directly enhance the procurement outcome of any GovTech initiative. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 34 An example of a partnership with a CSO is the American work agreements may go by several different names, including Council for Technology (ACT) and Industry Advisory blanket purchase agreements (BPAs), long-term agreements Council (IAC).17 ACT-IAC is a non-profit PPP dedicated to (LTAs), call-off contracts, or umbrella contracts. Framework improving government through the application of information agreements make it easier for a procurement organization to technology. ACT-IAC provides a forum in which government fill the requirements for products that are routinely purchased and industry exchange information and collaborate on tech- over a specified period of time. For GovTech procurement nology issues. The ACT-IAC group holds numerous confer- purposes, framework agreements can be established across ences each year focused on improving GovTech and provides countries, states, and local jurisdictions that share similar Gov- a platform for professionals to interact and share best prac- Tech requirements. For example, the World Bank and certain tices. ACT-IAC creates working groups focused on particular UN agencies are authorized to take advantage of schedules challenges or new technologies and their application to Gov- of the U.S. General Services Administration, which are frame- Tech. One of the more recent working groups on blockchain work agreements for various products and services. technology developed a rubric to assess whether blockchain should be used in GovTech applications and made it available The process of acquisition via framework agreement occurs in globally. The rubric helped to push the World Bank’s Global two stages. In the first stage, all suppliers in a market can be Public Procurement Database (GPPD)18 away from block- engaged and respond to a buyer’s RFP. Proposal analysis is chain toward more traditional technologies. then performed, and a down selected list of suppliers is add- ed to a framework agreement. The second stage is initiated 3.3.6 Use of Framework Agreements when a purchase needs to be completed. The buyer will then The use of framework agreements should be considered an issue requirements to the suppliers who have signed on to instrument for GovTech procurement. Framework agreements the framework agreement and receive quotes for review and can be defined as a “contracting mechanism in which long- supplier selection. Once the supplier is selected, an award term contracts provide the terms and conditions under which is made and structured via a call-off contract. The figure be- smaller repeat purchasing orders (or call-off orders) may be low19 is an illustrative depiction of how a framework agreement issued for a defined period of time” (Arney et al. 2014). Frame- works. > > > F I G U R E 1 6 - Framework Agreement Illustration Suppliers in the Market 1st Stage Downselection Supplier Participants in Framework Agreement 2st Stage Award / Call-Off Contract Buyers Requirements Buyer Buyer 17. For more information, see https://www.actiac.org/. 18. See https://www.worldbank.org/en/topic/governance/brief/global-public-procurement-database. 19. Adapted from Consip (2017). EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 35 There are several key advantages to implementing framework • Cost savings can be captured via economies of scale and agreements, including: competition among suppliers. • The administrative burden is reduced as the sourcing pro- cess is shortened, resulting in saved time and resources The figure below shows how efficiencies are gained via the (e.g., reduction in supplier research and issuance of RFPs). use of framework agreements and call-off orders (Arney et al. • The shortened sourcing process leads to financial effi- 2014). Many of the steps in a standard procurement process ciency as time is shaved from each acquisition and the are skipped when completing a call-off order as they were al- buyer has the ability to purchase products as they are ready achieved when the framework agreement was conclud- needed, reducing product holding costs. ed. As such, greater efficiencies are gained as more call-off • Under the framework agreement, suppliers compete for orders are executed. each procurement, resulting in a transparent and fair pro- curement process. > > > F I G U R E 1 7 - Framework Agreement Efficiency Illustration STANDARD PROCUREMENT FRAMEWORK CONTRACT START Review Review Estimates Estimates Operational Plan Operational Plan Financial Approval Financial Approval Communicate Requirements Communicate Requirements Procurement Packages and Procurement Packages and Plans Plans SKIPPED STEPS 16-18 months Specifications Specifications Bidding Documents Bidding Documents Bidding Bidding Bid Opening Bid Opening Evaluation Evaluation Award Award CALL-OFF ORDER Contract Contract Transfer Funds Transfer Funds Manufacture Manufacture Pre-Shipment Inspection Pre-Shipment Inspection Shipping Shipping FINISH Delivery Delivery An example of a successful deployment of a framework agreement for GovTech procurement comes from Nigeria, where the Kaduna State Public Procurement Authority served as the lead purchaser and represented a group of purchasers to procure an SaaS electronic government procurement (eGP) suite.20 The framework agreement was structured for award to a single vendor for a period of three years. By using the framework agreement approach, the purchasing group was able to streamline the procure- ment procedure for all 36 states in Nigeria. As a result, instead of each state running its own individual procurement process, all Nigerian states can sign call-off contracts with the winning vendor. 20. See Kaduna State Procurement Authority, “Request for Bids, Framework Agreement(s) for Procurement of Information Service – Software as a Service,” December 18, 2019, https://lagosppa.gov.ng/wp-content/uploads/2019/12/SaaS-e-Procurement-Solution-Invitation-for-Bids-IFB-Final-Version.pdf EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 36 3.3.7 Technology and 3.3.8 Cost Estimates in GovTech Intellectual Property Transfer Cost estimates are critical to any organization as it must effi- A key area to consider for GovTech is the process by which ciently use its limited resources to achieve its objectives. This technology and intellectual property (IP) are managed and is even more important in GovTech as procurers are govern- shared for the benefit of government, industry, and society. ments or international organizations that are mandated to ef- The World Intellectual Property Organization (WIPO),21 a UN fectively and efficiently use their funds. Agency established in 1967 with 193 country members, holds the global mandate for services, policy, information, and coop- Cost estimation procedures may differ based on whether the eration as it relates to IP. Its mission is to “lead the develop- procurement is for goods, services, technology, plant, or works. ment of a balanced and effective international IP system that Specifically, for complex GovTech solutions, the cost estimation enables innovation and creativity for the benefit of all.”22 process should take into account the following concepts: • Total cost of ownership WIPO provides several services that can help organizations • Sustainability that are exploring GovTech solutions to navigate any technol- ogy or IP access and/or transfer issues. The WIPO Technol- TCO involves all direct and indirect costs for the acquisition ogy and Innovation Support Center program provides inno- and use of a system or product. This includes any upfront de- vators in developing countries with access to locally based, velopment and CapEx as well as any operations and mainte- high-quality technology information and related services, help- nance expenditure or other OpEx over a defined period. ing them to exploit their innovative potential and to create, pro- tect, and manage their IP rights. WIPO is also in the process Oftentimes procurement teams may balk at the initial cost of a of developing international guidance on how artificial intelli- GovTech solution without realizing that operations and main- gence (AI) and machine learning (ML) are addressed within tenance costs over the planned life of a project can make an the context of IP. WIPO publishes several resources regard- initially cheaper CapEx investment more expensive over time. ing technology transfer, including “Technology Transfer in In the below simplified example of TCO, although Solution B Countries in Transition: Policy and Recommendations.” is initially cheaper, it incurs operations and maintenance costs that are three times higher than Solution A, making it a more Countries should consider the ownership of IP and future tech- expensive solution over a five-year period. As GovTech solu- nology transfer as part of their GovTech procurement strategy tions often have long project life spans to continuously deliver so that they can embed IP Ownership requirements in bidding government services, a focus on a TCO evaluation model is documents. critical to determining VfM. > > > T A B L E 7 - GovTech Solution Total Cost of Ownership (TCO) Example CapEx OpEx Costs (Operations and Maintenance) Costs GOVTECH SOLUTIONS Initial Total Purchase Year 1 Year 2 Year 3 Year 4 Year 5 Cost GovTech System $ 100,000 $ 5,000 $ 5,000 $ 5,000 $ 5,000 $ 5,000 $ 125,000 Solution A GovTech System $ 75,000 $ 15,000 $ 15,000 $ 15,000 $ 15,000 $ 15,000 $ 150,000 Solution B 21. See https://www.wipo.int/. 22. See https://www.wipo.int/about-wipo/en/. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 37 Sustainability cost estimates take into account concepts leading to better management decisions and preventing pro- such as the circular economy, defined as a model of produc- curement cost overruns. If tasks are not completed in speci- tion and consumption that involves sharing, leasing, reusing, fied days, reminder e-mails are sent out to help prevent delays repairing, refurbishing, and recycling existing materials and in project activities that often lead to cost overruns. products as long as possible. In this way, the life cycle of prod- ucts is extended. A sustainability cost estimate would involve Assigned managers can approve, reject, or ask for addition- exploring alternatives to the simple purchase and disposal ap- al information about the authorization request at each stage proach; rather, it allows for other purchase strategies, such as of the approval process. Every stakeholder is thus informed the purchase of after-market or refurbished equipment. about the progress of the request. Use of Public Procurement Technologies to Achieve Gov- eGP would be beneficial in achieving the following for CapEx Tech Cost Estimates procurements: • significantly increase the speed of the approval process eGP can be a tool to improve the project procurement without any manual routing process through the structured management of CapEx re- • improve productivity by eliminating system delays and quests and approvals throughout the duration of a project and other routing errors their systematic integration with the supporting budget and • promote transparency in CapEx request status (pending documents. approval, completion, or rejection) • certify systematic documentation of all processed CapEx The use of such eGP tools as request management and ap- requests for later reuse proval authorities can further project monitoring and oversight, • ensure the project’s process and audit compliance EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 38 3.4 STEP 4 – Implementation and Execution The implementation and execution of GovTech procurement begins with the preparation of tender documents and ends with the integration of the system. Key activities and deliverables related to the process are illustrated in the figure below. > > > F I G U R E 1 8 - Key Activities and Deliverables for Implementation and Execution CONDUCT DEVELOP TENDER EXECUTE INTEGRATE AND NEGOTIATIONS AND DOCUMENTS SOURCING EVENT EMBED VENDORS SELECT VENDORS ACTIVITIES • Identify purchase • Develop negotiation • Select vendors for • Pilot of new supply plan commitments/contracts strategy negotiations • Implement supply ramp- and assess impact • Develop vendor evaluation • Establish pricing model as up plan • Identify sourcing levers criteria baseline • Develop vendor-balanced » Manage vendors • Develop & issue RFP • Conduct final negotiations scorecards » Reduce total cost • Perform RFP analysis • Evaluate final offers • Implement vendor » Improve process • Conduct reverse auction • Assess vendor switching performance review » Optimize demand (optional) costs and risks process • Select & document • Develop total cost of • Complete total cost • Execute call-off contracts commodity sourcing ownership analysis (if using framework strategy • Define target price, • Select vendor agreement) • Refine savings estimates payment terms, and • Monitor vendor for • Develop and issue RFI service levels continuous improvement (optional) • Periodically re-evaluate • Select vendors for RFP category and vendor performance DELIVERABLES • Sourcing strategies • Vendor short list • Preferred vendor list • Executed call-off including • RFI/RFP • Final savings calculation contracts » Key levers • RFP analysis • Final terms and conditions • KPI and continuous » Approach • Auction analysis • Framework agreement improvement measures » Future state model • Total cost analysis • Savings realized • Sensitivity analysis After the acquisition is executed, the focus shifts to supplier performance monitoring. Note that the implementation and execution process is applicable whether using a standing tendering process or implementing a framework agreement. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 39 3.5 STEP 5 – Ongoing Management and Technology Evolution As with any technology solution, ongoing support, mainte- • Regulatory Compliance: Supplier’s hosted services nance, and development is critical to long-term implementa- should remain in compliance with all applicable laws. tion and service offering. Contract management is therefore an important lever for a procurement strategy to ensure the One of the most critical service levels for any GovTech initia- long-term results of any GovTech solution. tive is system availability. As more and more industry service offerings are in the cloud or provided as a service (e.g., SaaS, The primary accepted methodology is the use of service-level PaaS, or infrastructure as a service [IaaS]), connectivity and agreements (SLAs) and key performance indicators (KPIs) availability are the most critical service levels. It is imperative to measure and monitor contract performance. Both SLAs and that this service-level requirement be properly documented KPIs should be carefully designed to measure performance and put in as a mandatory requirement in solicitation docu- in order to meaningfully impact the ongoing management of ments so industry can appropriately scope a solution. the solution. Any SLA methodology and governance frame- work should allow for the addition, deletion, and continuous In order to do so, it is helpful to understand how the system improvement of service levels over the course of a contract. availability measurement is calculated and measured. This process can be built on specific calculations that are based on best practices in both the public and private sectors Service Availability as an SLA Example: as well as customized by the government and its technology Performance Criteria. Hosted services will be available to the suppliers. Service levels are typically measured and reported client in all material respects and as measured against the cri- monthly. Suppliers, as part of their monthly performance re- teria listed below (collectively the “availability requirements”): ports, should provide electronic reports generated from sys- • During any 24-hour period, the services will be available tems where available or delivered manually in the absence of 99.95 percent of the time. a system measurement. The inclusion of SLAs and KPIs into • During any seven-day period, the services will be avail- GovTech contracts is critical to their long-term success and able 99.95 percent of the time. sustainability. Therefore, it is highly recommended to defines • During any 30-day period, the services will be available SLAs and KPIs during the development of tender documents 99.95 percent of the time. so that suppliers can be informed upfront and their proposed solution can be evaluated against these requirements. To determine whether a service level is achievable, it is com- mon practice to ask for historical data from prospective suppli- The following concepts are recommended for incorporation ers in order to evaluate and qualify them for the specific proj- into any GovTech SLA: ect. In the event the GovTech initiative is for a new technology • Service-Level Credits: These allow for financial com- or solution where historical data may not exist, one option is pensation when service levels are not met. to ask prospective suppliers to provide system availability his- • Earn backs: These allow suppliers to earn back financial torical data across existing clients (anonymized) in order to penalties for improved performance. evaluate their technical solutions, including their performance. • Termination for Repeated Service-Level Failure: Gov- The below table presents an illustrative example of the sys- ernment should have the ability to terminate service for tem availability performance of a single supplier over a six- any repeated failure of service levels. month period across 10 clients. It is important for evaluation • Updates and Corrections: Supplier should be respon- purposes that the historical data request cover both a broad sible for any updates or corrections required to meet ser- client base and lengthy time period so a proper performance vice levels. evaluation can be carried out. • Investigation: Supplier should be responsible for root cause analysis and investigation of any service-level failures. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 40 > > > T A B L E 8 - Example of System Availability Performance Data (Illustrative Example) CLIENT JANUARY FEBRUARY MARCH APRIL MAY JUNE A 100.000 100.000 100.000 100.000 100.000 100.000 B 99.901 99.956 99.950 99.959 99.899 99.952 C 100.00 100.000 100.000 100.000 100.000 100.000 D 99.995 99.995 100.000 100.000 99.991 99.990 E 100.000 100.000 99.948 100.000 100.000 F 99.953 99.992 100.000 100.000 100.000 100.000 G 99.983 99.967 100.000 99.768 99.963 100.000 H 99.958 100.000 99.943 99.965 100.000 100.000 I 99.875 99.927 99.972 100.000 100.000 100.000 J 100.000 100.000 100.000 99.645 100.000 100.000 The calculation for system availability itself is very straight- A simplified example of the above service-level calculation for forward, and the measurement should come from system- a single day would be the following: generated (and not manual) data. This is a common practice and easily achievable automatically via existing measurement (B) Actual system uptime: 23 hours 50 minutes / (C) sched- systems and technologies. uled uptime: 24 hours in a day x 100% = 99.31% system availability Measurement. System availability expressed as a percent- age of the scheduled system availability for the measurement For certain low-priority GovTech systems (e.g., traffic ticket window (i.e., availability % = ((actual uptime) / (scheduled up- payment processing), a lack of system availability of 10 min- time)) x 100%). utes per day may be acceptable. However, for more mission critical systems, such as citizen purchases of pharmaceuticals Compliance Calculation. Supplier shall calculate the system via a country’s public health system, 10 minutes of downtime availability service level for each month in accordance with the may be unacceptable. It is therefore important when devel- following formula: oping GovTech solution requirements to understand and con- tract for the right levels of system availability. The purpose of the SLA is to provide a monitoring framework to ensure that the contracted GovTech solution is meeting the A = the service level; performance standards required to deliver the service to the B = actual measured system availability in milliseconds in end user—often a country’s citizens, business community, or the measurement window month; and both. The framework also allows for the quick resolution of C = the total number of available milliseconds in a mea- system performance issues that may otherwise go unnoticed surement window month if not vigorously monitored with an incentive/penalty frame- work in place to ensure and incentivize compliance. In a world of complex financial dealings where thousands of transactions are happening every second of every day, avail- An example SLA framework for system availability and prob- ability is extremely critical and often measured in milliseconds. lem support terms, including measurement and remedies, is To better understand the concept, however, it may be more contained in Annex III. useful to think of system availability occurring over a single 24- hour period when the system is not accessible for 10 minutes within that period. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 41 4. >>> GovTech and Procurement Disruptive technologies can be defined as emerging technologies that result in a step change in the cost of or access to products or services, or that dramatically change how people gather information, make products, or interact. Disruptive technologies will fundamentally change the way in which public procurement value is realized over the next decade. Existing supply chain delivery mechanisms will be enhanced by drone delivery (e.g., medicine to inaccessible loca- tions), supply chains will be increasingly and securely monitored via Geotagging, and contracts and payments will be made transparent via blockchain. The value creation for developing coun- tries is unlimited; however, to capture this value, countries need to understand when and how best to implement disruptive technologies based on their unique operating environments. GovTech Procurement Practice Note for Disruptive Technologies A GovTech strategy focused on disruptive technologies in procurement should use the GovTech procurement framework as described in section 3 of this document to promote procurement system improvements and innovation. Countries and purchasing organizations can apply the framework to assess and implement potential procurement and contract management solutions using blockchain, drones, GPS tagging, AI, robotic process automation (RPA), big data, GIS mapping, and other technologies. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 42 The GovTech procurement framework assists in the procure- ruptive Technologies and the World Bank Group: Creating Op- ment of disruptive technologies by addressing the need to: portunities—Mitigating Risks” (World Bank 2018), discussed • Identify key requirements necessary for implementing dis- the Bank’s ability to assist client countries in harnessing disrup- ruptive technologies in the public procurement space tive technologies while also helping them to address the down- • Identify investment opportunities for supporting client side risks and impacts of that disruption. The paper lays out the countries in different areas of procurement reform using “Build-Boost-Broker” framework, which requires support from disruptive technologies the private sector and development partners, as listed below. • Utilize best practice approaches using disruptive technol- ogies in the public procurement area and inform the Multi- • Build the infrastructure and regulatory foundations to ex- Donor Trust Fund (MDTF) and the Global Governance pand the diffusion of and access to new technologies. Practice (GGP) about opportunities for scaling • Boost the capacity of individuals, institutions, business- • Offer strategic direction for the coordination of donor and es, and governments to pursue new opportunities and to government resources on disruptive technologies in pro- thrive in the face of change. curement, providing strategies to integrate procurement • Broker the use of technologies to address specific devel- reform actions using disruptive technologies opment challenges and to shape the global dialogue and • Introduce innovative technologies that will help government standard setting associated with disruptive technologies. procurement systems to increase competition and identify business process efficiency and technological solutions In 2019, an additional Development Committee paper, “Main- streaming the Approach to Disruptive and Transformative Technologies at the World Bank Group” (World Bank 2019a), Disruptive Technology and improved the “Build-Boost-Broker” framework by identifying five mechanisms that will help to mainstream the Bank’s ap- Performance Specifications proach to disruptive technologies: 1. Support country diagnostics that help chart the new driv- Technology requirements may be classified as either ers of growth. conformance or performance specifications. 2. Support the formulation and implementation of agile regu- lations for the new economy. Conformance specifications are used where a thorough un- 3. Scale up universal, affordable digital connectivity. derstanding of the requirements already exists, and there is 4. Support the provision of transparent, efficient, and ac- little desire for a supplier to innovate. Performance specifica- countable digital government services. tions are used where the understanding of what is required in 5. Support the development of skills and capabilities for the terms of outcomes can be described, but it is less certain as to new economy. the best process or method to deliver the requirements, or there is a desire for suppliers to innovate. For the procurement of Recognizing that technological advancement is central to its disruptive technologies, which by definition are new and inno- mission as technology impacts all aspects of the global econ- vative, it is highly recommended to use performance specifica- omy, the Bank is working with global partners to implement its tions wherever possible to maximize the possibilities for innova- disruptive technology agenda. tion. Conformance specifications may also be used for certain solution components where the requirements are known or Within the World Bank, the Information and Technology So- where there is a known reliance on existing technologies. lutions (ITS) Technology and Innovation Lab is a centralized department that is accessible to staff to test, prototype, and deploy pilot projects using leading edge and developing tech- nologies (e.g., AI, ML, blockchain). The purpose of the lab is to Disruptive Technology provide an organizational structure and resources to efficiently Innovation at the World Bank deploy and share knowledge around technological solutions that can improve the delivery of Bank and client services. The World Bank has released key policy papers in recent years In relation to the GovTech procurement practice note, the ITS that illustrate that effective use of disruptive technologies is a lab can be used to test and assist with the development of dif- priority. In 2018, a Development Committee paper titled, “Dis- ferent technology solutions, leading to the generation of require- EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 43 ments that can then be plugged into a procurement plan. For how the United Nations system will support the use of new example, the ITS lab tested both the feasibility and applicability technologies like AI, biotechnology, blockchain, and robotics of blockchain and AI technologies for the GPPD; the results of to accelerate the achievement of the 2030 Sustainable De- these prototype and feasibility tests were then incorporated into velopment Agenda and to facilitate their alignment with the the final procurement technical and functional requirements be- values enshrined in the UN Charter, the Universal Declara- fore the Bank formally issued a competitive solicitation. Specifi- tion of Human Rights, and the norms and standards of inter- cally, ITS prototype and feasibility tests showed that blockchain national law.” (UN 2018). The Emerging Technologies Team technology was not the right requirement fit for the GPPD, and (ETT)24 within the UN Office of Information and Communica- this technical architecture and functionality was eliminated from tions Technology (OICT) investigates technologies and their the product and solicitation requirements. applications that can facilitate the core work of the UN. ETT has investigated technologies in four areas: The ITS Lab also participated in a project on behalf of the World Bank’s Sindh Agricultural Growth Project for Pakistan is to im- 1. Artificial Intelligence. ETT focuses on AI subdomains, prove the productivity and market access of small and medium such as ML and natural language processing (NLP). Pro- producers in important commodity value chains. The ITS lab in totypes have been built for various applications, includ- collaboration with World Bank and client country teams proto- ing Unite Cognition Extractive Text Classification, Unite typed a Blockchain based platform to help manage the agricul- Cognition Answer Finder, and Unite Cognition Extractive tural equipment supply chain. The prototype served as a PoC Video Summarization. and the basis for a market competition that was launched with 2. Computer Vision. ETT applies computer vision techniques to an initial product implementation occurring in June 2020. better understand environments and situations while expos- ing data in unique ways. Prototypes have been built for vari- It is recommended that governments and other organizations ous applications, including Low Connectivity Asset Tracking that are investigating the potential use of GovTech establish and Unite Cognition Extractive Video Summarization. an organization like the Bank’s ITS lab to explore innovation, 3. Distributed Ledger Technology. Defined as a peer-to-peer prototyping, and feasibility tests using disruptive technologies. network of interaction and transaction that is replicated Centralizing this knowledge in a single lab allows multiple gov- and distributed among many participants, ETT has built ernment agencies to access this resource without having to prototypes for Low Connectivity Asset Tracking and Unite duplicate it across multiple agencies. Token (cryptocurrency). 4. Internet of Things (IoT). Defined as the interconnection The World Bank chose to centralize its lab within its ITS de- via the internet of computer devices embedded in objects, partment as the enterprise-level department responsible for IT allowing them to send and receive data, ETT is current- implementations across the entire WBG. Dependent on coun- ly developing IoT solutions for Low Connectivity Asset try context and purpose of the lab, it can be centralized into Tracking and Unite Cognition Smart Cities. an existing independent agency or placed within the appropri- ate ministry. For example, the Government of Singapore has The UN has already deployed disruptive technologies in sup- created a Government Technology Agency23 that sits directly port of its mission (UN 2018). For example, in Malawi, the UN under the Prime Minister’s Office and is responsible for imple- has established the world’s largest drone test corridor for hu- menting GovTech solutions and operating a variety of Gov- manitarian applications covering over 13,000 square kilome- Tech focused labs that are similar to the World Bank’s ITS lab. ters. In Zambia, the UN is supporting the creation of “virtual farmers markets,” where an app-based e-commerce platform connects farmers’ surplus with buyers’ demand for crops. The Disruptive Technology UN has also recently established the United Nations Tech- nology Bank for Least Developed Countries, a global orga- at the United Nations nization dedicated to enhancing the contribution of science, technology, and innovation to sustainable development in the world’s most impoverished states.25 In 2018, the Secretary-General of the UN issued a Strategy on New Technologies. The goal of the strategy was to “define 23. See http://tech.gov.sg 24. See https://unite.un.org/emerging-tech. 25. For more information, see https://www.un.org/technologybank/. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 44 World Bank Case Study – Blockchain Use in eGP The ITS Technology and Innovation Lab and the GGP procure- 3. Suppliers can input their work experiences and other ca- ment units collaborated to test the possibility and feasibility of pability information in one place, after which bid submis- using blockchain to increase the transparency and efficiency sion can be done with less effort. Consequently, the sup- of eGP systems (World Bank forthcoming). The expected out- plier community would find working with the government come and benefits of the concept developed include: much easier, resulting in greater competition for govern- 1. The experience and validity of suppliers is recorded in ment tenders. a standardized format that could help create a sense of competition among the supplier community, potentially The exploration team leveraged design thinking methodology, leading to improved ratings and better quality in govern- as shown in the figure below26. Implementing the double dia- ment contracting. mond design approach, which moves tasks through a complex 2. Expedited procurement and reduced transaction costs business problem, helped the team to better explore the case. are likely as purchasing agencies will find it easier to Initial activities included problem framing and preparation of evaluate suppliers. Automated bid evaluation is possible the research plan, followed by case analysis and delivery of a when the data required for an assessment of a supplier’s tested prototype. expertise can be pulled in a standardized format from au- thenticated sources. > > > F I G U R E 1 9 - Design Thinking Approach Engage Explore Incubate Scale Manage Scale the 1 2 3 4 5 Market Prepare Discover Define Develop Deliver Incubate Monitor Scale the Business Discover the Problem Discover the Solution Incubate the Solution Scale the Solution Goal: Learn how to access Goal: Learn how to access Goal: Learn how to rapidly Goal: Learn if the product is the opportunity of and idea the opportunity of and idea launch and test MVP’s in the a business worth growing market with customers Develop/ Phase Prepare Discover Define Develop Deliver Plan Grow Customise Activity Framing Exploration Interpretation Ideation, concept Evaluation Architecture Agile Implementation, planning & & analysis & generation and refinement & Viability development and launch & preparation understanding synthesis prototyping experimentation Project customisation monitoring Planning MVP Validation Design Question, Unstructured Insight report, Ideas, Validated Blueprints Manage activation Outcomes challenge, research and problem concepts and answer, MVP to referral, scale activity brief findings definition & prototypes business model, internal systems HMW question product, service The team agreed to implement an approach that provides access to both purchasing agencies and suppliers via global blockchain for the purposes of a prototype. Through the selected design solution, the team anticipates that supplier information will not be duplicated. The ITS lab recommends planning for the de-duplication of past supplier information migration separately, as part of project implementation. 26. Figure is sourced from forthcoming World Bank document EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 45 >>> Bibliography ADB (Asian Development Bank), IADB (Inter-American Development Banks), and World Bank. 2004. “Electronic Government Procurement (e-GP) Readiness Self-Assessment.” ADB, IADB, and World Bank, Washington, DC. Ancveire, Ieva. 2018. “Fit Gap Analysis Methods for ERP Systems – Literature Review.” IEEE 12th International Symposium on Applied Computational Intelligence and Informatics. Timiso- ara, Romania, May 17–19. Arney, Leslie, Prashant Yadav, Roger Miller, and Taylor Wilkerson. 2014. “Strategic Contracting Practices to Improve Procurement of Health Commodities.” Global Health Science and Practice 2 (3): 295–306. Consip. 2017. “Framework Agreements for Centralized Procurement: Consip’s Experience.” Pa- per prepared for the South Asia Region Public Procurement Conference, “e-GP in South Asia: Achievements, Opportunities, Challenges.” Colombo-Kandy, February 20–23. Gallo, Marcy E. 2020. “Federal Prize Competitions.” CRS Report R45271, Congressional Re- search Service, Washington, DC. https://fas.org/sgp/crs/misc/R45271.pdf. Gartner. 2019. “Magic Quadrant for Procure-to-Pay Suites.” Gartner, Stamford, CT. Masson, Bernard, and Alex Margot-Duclot. 2018. “GovTech: Europe’s Next Opportunity.” Ac- centure and Public. https://www.accenture.com/_acnmedia/pdf-90/accenture-govtech-pov.pdf. Morlock, Gabrielle M. 2019. “SBA Announces a Major Change to SBIR/STTR Data Rights Pro- tection.” The Law Offices of Snell & Wilmer, August 30. https://www.swlaw.com/publications/legal-alerts/2607. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 46 OICT (Office of Information and Communications Technology, United Nations). 2020. “Emerging Technologies.” OICT, New York. https://unite.un.org/emerging-tech. PWC. 2016. “GovTech: The Power to Transform Public Services in the UK.” PWC, London. https://www.pwc.com/gx/en/psrc/united-kingdom/assets/govtech-report.pdf. UN (United Nations). 2018. “UN Secretary-General’s Strategy on New Technologies.” United Nations, New York. https://www.un.org/en/newtechnologies/images/pdf/SGs-Strategy-on-New-Technologies.pdf. White House. 2010. “Congress Grants Broad Prize Authority to All Federal Agencies.” The White House (blog), December 21. https://obamawhitehouse.archives.gov/blog/2010/12/21/congress- grants-broad-prize-authority-all-federal-agencies. World Bank. 2014a. “Bank Directive on Prizes and Awards Expense.” World Bank, Washington, DC. ———. 2014b. “Bank Procedure – on Prizes and Awards Expenses.” World Bank, Washington, DC. ———. 2016. “Guidelines for Conducting Market Analysis for e-Procurement Systems.” World Bank, Washington, DC. http://eprocurementtoolkit.org/sites/default/files/2016-10/Market_Analy- sis-Rapid_e-Procurement_Toolkit.pdf. ———. 2017a. “Bank Policy – Procurement in IPF and Other Operational Procurement Matters.” World Bank, Washington, DC. https://policies.worldbank.org/sites/ppf3/PPFDocuments/a3656cb78847417b886f11fa0235216e.pdf. ———. 2017b. “Competitive Dialogue – How to Undertake a Competitive Dialogue Procurement Process.” World Bank, Washington, DC. http://pubdocs.worldbank.org/en/412401507743078456/Competitive-Dialogue-Guidance-2017.pdf. ———. 2017c. “Procurement in Investment Project Financing: Goods, Works, Non-Consulting and Consulting Services.” World Bank, Washington, DC. http://pubdocs.worldbank.org/en/659511533066042959/Procurement-Regulations-2017.pdf. ———. 2018. “Disruptive Technologies and the World Bank Group – Creating Opportunities – Mitigating Risks.” Development Committee, World Bank, Washington, DC. ———. 2019a. “Mainstreaming the Approach to Disruptive and Transformative Technologies at the World Bank Group.” Development Committee, World Bank, Washington, DC. ———. 2019b. “Procurement Guidance – Medical Diagnostic Imaging (MDI) Equipment: Under- standing How to Procure Medical Diagnostic Imaging Equipment.” World Bank, Washington, DC. ———. 2020. “GovTech Launch Report and Short-Term Action Plan” World Bank, Washington, DC. ———. forthcoming. “Use Case Viability Report: e-GP Global Procurement.” World Bank, Wash- ington, DC. ———. n.d. “e-Procurement Brief for Nigeria.” World Bank, Washington, DC. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 47 >>> ANNEX I: Sample Technology Assessment Interview and Survey Questions EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 48 Interview and Survey Questions iii. Additional comments: ______________________ c. Activity #3: _________________________________ i. Application/System Name (if one exists): _______ ii. Quality Rating (1=poor, 5=excellent, or “n/a” if no Note. For most questions, participants will be asked to provide application exists):________________________ responses on a scale of 1 to 5, where 1 is the weakest score iii. Additional comments: _____________________ and 5 is the strongest score. The actual scale will be specified for each question. 3. How would you rate the IT organization’s performance in developing new systems and applications? Con- In the instructions, we will also specify that participants should sider such elements as alignment with customer require- provide their evaluations and feedback based on: ments; development cycle time; application roadmaps; • Their experience at application quality and performance; application support. • Their experience and knowledge of other similar IT orga- (1=poor; 5=excellent) nizations • Their expectations based on common practices and com- Please explain and provide specific examples/instances municated performance standards if possible. In the upfront collection of background details, we will ask 4. How satisfied are you with the level and quality of for the individual’s level of interaction with the IT organization the IT disaster recovery and backup systems and ap- (1 = very limited, 5 = very extensive). plications? Consider such elements as response times; ability to store and recover critical data. (1=poor; 5=excellent) Infrastructure and Operations Please explain and provide specific examples/instances if possible. 1. What common activities do you primarily perform in the completion of your job? Please specify the top 3 IT Strategy and Leadership activities where you spend a significant amount of time: a. Procurement b. Travel Management c. Reporting 5. How would you rate the IT organization’s alignment d. Grants Management with ’s overall strategic objec- e. Human Resources and Payroll tives? Consider such elements as the technology road- f. Finance and Accounting map; alignment with overall g. Investment Management mission; how well the IT strategy is communicated; how h. Facilities or Asset Management well it is executed. i. Time and Attendance (1=poorly aligned; 5=very well aligned) j. Other (please specify) Please explain and provide specific examples/instances if possible. 2. For each of the top 3 activities that you specified in Question #1, please rate the quality of the current 6. How would you rate the IT organization’s Governance IT systems and applications that you leverage to procedures? Consider elements of IT governance, such perform each activity. as creating IT technology standards, decisions on IT in- a. Activity #1: _________________________________ vestment prioritization, allowance of beta tests (e.g., Pro- i. Application/System Name (if one exists): _______ curify), IT, and SW asset management. ii. Quality Rating (1=poor, 5=excellent, or “n/a” if no (1=distributed governance; 5=centralized governance) application exists):________________________ Please explain and provide specific examples/instances iii. Additional comments: _____________________ if possible. b. Activity #2: ________________________________ i. Application/System Name (if one exists): _______ ii. Quality Rating (1=poor, 5=excellent, or “n/a” if no ap- 7. How would you rate the IT organization’s Communi- cations? Consider elements of IT governance, such as plication exists):___________________________ EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 49 communications on: service interruptions, new product extent to which projects are delivered on time and on introductions, new service implementations, IT strategic budget; status reporting; issues management; whether IT vision/technology roadmaps. is keeping the lights running or introducing new projects. (1=poor communication; 5=excellent communication) (1=poor; 5=excellent) Please explain and provide specific examples/instances Please explain and provide specific examples/instances if possible. if possible. 12. How would you rate the quality of vendor manage- People and Organization ment in the IT organization? Consider such elements as the responsiveness of vendors, adjustment in vendor performance based on IT department feedback, quality of IT vendors. 8. [FOR IT PERSONNEL ONLY] How would you rate the (1=poor; 5=excellent) quality of the organizational structure in the IT orga- nization? Consider such elements as your perception of Please explain and provide specific examples/instances how a distributed IT organization impacts service levels if possible. versus your knowledge of IT organizational structure at other organizations. (1=poor; 5=excellent) Customer and Stakeholder Please explain and provide specific examples/instances Relationships if possible. 9. Do you think the IT organization has sufficient re- 13. What is your overall level of satisfaction with the ser- sources and skills to meet the needs of your orga- vices provided by the IT department? nization? Consider such elements as the availability of (1=very dissatisfied, 5=very satisfied) resources across the IT organization. Do resource avail- ability and skill sets differ across the organization (i.e., re- Please explain and provide specific examples/instances sources differ between P Street and departments)? if possible. (1=poor; 5=excellent) 14. What is your perception of the general level of satis- Please explain and provide specific examples/instances faction with the services provided by the IT organiza- if possible. tion amongst your peers? (1=very dissatisfied, 5=very satisfied) 10. How would you rate the availability and quality of train- ing from the IT organization? Consider such elements as Please explain and provide specific examples/instances your perception of the availability and quality of the training. if possible. Is training available in person or online (e.g., when new applications or systems are released, do you feel you have 15. What is your perception of the IT Organization as a been adequately trained on how to utilize them)? Do train- Strategic Partner? ing manuals exist? How is IT knowledge transferred? (1=not considered strategic partner, 5=considered strate- (1=poor; 5=excellent) gic partner) Please explain and provide specific examples/instances Please explain and provide specific examples/instances if possible. if possible. 16. General comments and feedback. We welcome any ad- ditional comments or feedback that you would like to pro- Project/Portfolio Management vide regarding your experience with the IT organization at . Any suggestions and/or recommendations for improvement that you would like to 11. How would you rate the quality of project management share are also welcome and greatly appreciated. in the IT organization? Consider such elements as the EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 50 >>> ANNEX II: e-Procurement Toolkit Templates The e-Procurement Toolkit can be found at: http://eprocurementtoolkit.org/. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 51 Invitation Template Recipient’s address: Sender’s details: Name Name Department/Organization Department/Organization Address Address Country Country [Date] Dear Sir/Madam, Our organization, in line with its mandate to modernize the [Information Technology (IT) system environment], is undertaking a market analysis exercise in the field of [Information Technology (IT) systems], aiming to obtain a better understanding of the market landscape. The ultimate ob- jective of the market analysis is to refine our requirements in preparation for a tendering procedure to contract with the most suitable vendor(s) to implement the [country]’s IT system]. The tendering procedure is planned to take place in the [number] quarter of [year].1 Your company is asked to participate in the market analysis exercise because of its expertise, reputation, and past performance record in the field. The mission of our organization is to modernize the [Information Technology (IT) system environment] in the whole country, with the goal of im- proving in five strategic areas: governance, institutional capacity, functionality and infrastructure, interoperability, and adoption. All five strategic areas are being addressed in an organized and thorough manner, as per the elaborated Action Plan published at [website], which also describes the achievements so far. The market analysis is a central component of the third strategic area of the Action Plan, that of functionality and infrastructure. A solid market analy- sis on [Information Technology (IT) systems] and their respective vendors is crucial to our understanding of the current state of the market, including the latest trends, best practices, industry capabilities, delivery approaches, and likely costs. The analysis will be based primarily on a questionnaire that raises five topics for profiling vendors (such as experience and methodology) and five topics for profiling e-Procurement solutions (such as sys- tem modules and maintenance support). In addition to these specific topics, you may submit any supplementary information you consider important for our better understanding of the market, including brochures, case studies, methodologies, pricing schemes, and so forth. I wish to emphasize that the market research is not a tendering competition. Any information obtained will be treated as confidential and will be used solely by government officials for the purposes outlined above. Furthermore, the identity of the vendors invited and those that respond will not be disclosed, and no formal evaluation of the submitted information will be performed. Finally, this invitation does not constitute a commit- ment for future business cooperation, and vendors will not be financially compensated their response. Any contractual arrangement with any organization(s) may be concluded only following the forthcoming tendering exercise. If you are interested in participating in the market analysis, please visit the following Internet location in order to obtain access to the terms, list of requirements, and questionnaire. [website] Your responses should be sent via e-mail to: [e-mail address] Any questions, objections, and/or concerns should be communicated via e-mail to the aforementioned e-mail address. I wish to thank you in advance for your cooperation and look forward to receiving your response. Sincerely [name] [role] [signature] Notes 1. If the time plan for the tendering exercise is not decided yet, this sentence should read “There is currently no specific date for com- mencing the tendering exercise.” 2. If the material is not intended to be published, this sentence should read “If you are interested in participating in the market analysis, please send a respective e-mail to [e-mail address]; we will then respond back to you, attaching all relevant material.” EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 52 Questionnaire Template Vendor Questionnaire 1. Profile 1.1 Name 1.2 Website 1.2 Field of expertise (free text up to 200 words) 2. Relevant Experience Country Client Project Budget 2.1 Experience in public sector systems (last 5 projects) Country Client Project Budget 2.2 Experience in public sector information systems (last 5 projects) Country Client Project Budget 2.3 Experience in private sector information systems (last 5 projects) 3. Eligibility to Participate in Public Procurement Proceedings 3.1 Eligibility in public procurement proceedings (i.e., not convicted of participation in any criminal activity, corruption, I do not confirm fraud, terrorist offenses, money laundering, or child labor; I confirm that the that the not subject to exclusion due to nonpayment of taxes or social organization is eligible. organization is eligible. security; not in the process of bankruptcy; not convicted of misconduct, etc.) (check as appropriate) 3.2 Average annual turnover in the past 5 years (number) 3.3 Average annual manpower (number) Independent until 3.4 Certificates by independent bodies Certificate Valid Body EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 53 4. Geographical Coverage Continent X Country Y 4.1 Physical presence in the region 4.2 Experience in delivering software solutions in region Continent X Country Y (If you cover [country], then simply check that box. If not, write the main countries covered in [region].) 4.3 Experience in delivering end-user training services in region Continent X Country Y (If you cover [country], then simply check that box. If not, write the main countries covered in [region].) 4.4 Experience in delivering software support/maintenance Continent X Country Y services in region (If you cover [country], then simply check that box. If not, write the main countries covered in [region].) 4.5 Experience in delivering help desk services in region Continent X Country Y (If you cover [country], then simply check that box. If not, write the main countries covered in [region].) 5. Engagement Methods with Clients and Typical Service-Level Agreement (SLA) 5.1 Delivery of software ownership and source codes Yes No (check as appropriate) 5.2 Delivery of software-as-a-service (SaaS) Yes No (check as appropriate) 5.3 Engagement in a public-private partnership (PPP) Yes No (check as appropriate) 5.4 Delivery of service in the form of a shared service Yes No (check as appropriate) 5.5 Indicative SLA provided Yes No (e.g., response time for minor/major issues, availability %, re- sponse to security issues, etc.) (free text up to 200 words) 6. Profile 6.1 Name 6.2 Website 6.3 Description of the system (free text up to 200 words) EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 54 7. System Modules (Note.* This section provides examples of eGP Modules. Template users should substitute actual IT System Modules based on their unique project design.) 7.1 e-Registration Supported (Yes/No) Supported (Yes/No) Process Management 7.2 e-Procurement Plan Group/consolidation Publication Supported (Yes/No) Tender workspace creation and workflow setup 7.3 e-Publication/Notification Officer association Tender documentation Tender questionnaire Completion Publication/ Activation Supported (Yes/No) Questions/answers 7.4 e-Tendering Creation/submission of bids Securities/guarantees Supported (Yes/No) Bid opening Offline tenders 7.5 e-Evaluation/e-Awarding Evaluation Complaints Award 7.6 e-Reverse Auctions Supported (Yes/No) Supported (Yes/No) Contract workspace management Contract negotiations, amendments, renewals 7.7 Contract Management Deliverables Actions Key Performance Indicators Financials Supported (Yes/No) Catalogue workspace management 7.8 e-Catalogues Catalog browsing Cart management Supported (Yes/No) Catalogue preparation Catalogue submission 7.9 Catalogue Management Catalogue approval Catalogue versioning Catalogue activation EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 55 Supported (Yes/No) Requisition Quotation 7.10 e-Purchasing Purchase order Invoice Payment Good receipt note Supported (Yes/No) e-Attestations 7.11 Vendor Management Prequalification Vendor performance Supported (Yes/No) Notifications Auditing 7.12 Procurement Monitoring Business Intelligence Reporting and Reporting & Dashboard Open Contracting Data Standard (OCDS) World Bank Indicators 8. Nonfunctional Features 8.1 Programming language(s), database, supported operating system and technologies(bulleted list) 8.2 Technical characteristics of special interest (free text up to 200 words) 8.3 Use of standards (bulleted list) 8.4 Interoperability capabilities and technical setup for integrations (free text up to 200 words) 8.5 Adaptation/customization capabilities and methodology used (free text up to 200 words) 9. Support/Maintenance Procedures 9.1 Methodology to support system and ensure a responsive, reliable, and available service (free text up to 500 words) 9.2 Methodology for risk management and disaster recovery (free text up to 500 words) 10. Nonfunctional Features 10.1 Indicative system baseline/license cost (USD range in thousands) 10.2 Indicative system annual license and maintenance fee (USD range in thousands) 10.3 Indicative cost for adaptation (USD range in thousands) 10.4 Indicative cost for SaaS model EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 56 Supplemental Information Vendors are invited to deliver any further documentation deemed relevant to this inquiry or any additional information that may supplement the provided answers. This can include: • Vendor profile brochures • System brochures • Case studies on the practical application of the system • Technical documentation of the system • Operational/procedural documentation on employed methodologies • Pricing scheme Market Analysis Report Template Note.* Example data analysis and charts for various sections of the Market Analysis Report Template can be found in the e-Procurement Toolkit. 1 INTRODUCTION 1.1 Purpose of the report Description of the current market analysis (i.e., the incentive for performing it and next steps) 1.2 Stakeholders and audience Description of the personnel involved in the market analysis and its intended audience 1.3 Structure of the report Description of the report chapters 2 METHODOLOGY 2.1 Market analysis phases Brief description of the phases followed for conducting the market analysis: • Inception • Material preparation • Vendor involvement/communications • Consolidation/synthesis • Conclusions 2.2 Material shared with vendors Documentation of the material provided to vendors 2.2.1 Invitation Attachment 2.2.2 Description of the requirements Attachment 2.2.3 Questionnaire(s) Attachment(s) 2.2.4 Other material Attachment(s) 2.3 Consolidation/synthesis Description of the processing performed to synthesize the collected data and extract the information needed for the next steps (including attachments of any “consolidation” templates used) EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 57 3 SYNTHESIZED INFORMATION 3.1 Relevant experience Relates to information collected from section 2 of the questionnaire. Conclusion to be drawn: do the economic operators (EOs) that provided data have experience in delivering public sector IT systems and other such large-scale projects? 3.2 Eligibility Relates to information collected from section 3 of the questionnaire. Conclusion to be drawn: are the EOs that responded to the market analysis eligible to support an IT system implementation project? 3.3 Geographical coverage Relates to information collected from section 4 of the questionnaire. Conclusion to be drawn: do the EOs that responded to the market analysis provide the necessary regional coverage to support the full cycle of an IT system implementation project? 3.4 Engagement method Relates to information collected from section 5 of the questionnaire. Conclusion to be drawn: which engagement methods are used most by the EOs that responded to the market analysis? 3.5 System modules Relates to information collected from section 7 of the questionnaire. Conclusion to be drawn: which system modules are widely available on the market and which would most probably require a substantial implementation/configuration effort as per the information provided by the EOs that responded to the market analysis? 3.6 Nonfunctional features Relates to information collected from section 8 of the questionnaire. Conclusion to be drawn: which are the most prominent programming languages and the most commonly used standards as per the information provided by the EOs that responded to the market analysis? 3.7 Support/maintenance procedures Relates to information collected from section 9 of the questionnaire. Since responses here involve free text provided by the EOs, a graph is not envisaged. Still, a conclusion should be drawn as to the most common management standards used by the EOs that responded to the market analysis. Support methodologies, tools, and communication mediums should be captured in order to identify commonalities, trends, and best practices. 3.8 Financial considerations Relates to information collected from section 10 of the questionnaire. Conclusion to be drawn: what are the envisaged costs for configuration/deployment, annual maintenance fees, and typical adaptation costs as per the information provided by the EOs that responded to the market analysis? 4 MAIN CONCLUSIONS This section should list the main conclusions of the market analysis, typically covering: • Number, eligibility/technical/financial status, and experience of EOs that responded to the market analysis, which can indicate the EO interest and capacity that could be expected in the tendering process for the e-Procurement system (unless wider advertisement schemes or improved communication methods are used) • Geographical coverage of EOs that responded to the market analysis, which can indicate whether the • tendering process could explicitly require a local presence, subcontracting with a local partner, specific • methodologies for long-distance cooperation, and so on • Engagement methods that are most preferred or used by vendors, which can be taken into account when determining the exact requirements for the tendering process • Commonalities and expertise in functionalities, technologies, and standards that can also be taken into • account in concluding the tender’s exact requirements • Rough financial estimates that can be used to estimate the budget for the tendering process Request for Expression of Interest Template EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 58 REQUEST FOR EXPRESSION OF INTEREST [COUNTRY] [PROGRAM NAME] [PROJECT/TENDER NAME] [PROJECT/TENDER REFERENCE NUMBER] The [agency] of the Government of [country] intends to obtain ICT consulting services for the establishment of the country’s national IT system (hereafter “IT system”). The consulting services would include analysis, design, configuration/ parameterization, testing, and delivery of the IT system. In addition to the above, the service provider should expect to engage in various support activities, such as pilot operation, rollout, end-user training and capacity building, promotion/marketing, help desk operation, hosting, and system support/maintenance. The solution must be based on a prebuilt, commercial-off-the-shelf (COTS) system that will need to be configured/parameterized to meet the specific project requirements. The [agency] now invites eligible firms to express their interest in providing the requested consulting services. Interested consultants should present evidence of their qualification status and demonstrate their financial and technical capacity. Eligible consultants will be short-listed using the following criteria: Experience in public sector ICT projects Evidence of having delivered at least one national public e-Procurement system of similar size Evidence of adhering to industry-accepted quality methods By submitting their Expression of Interest (EOI) in the current request, consultants implicitly confirm that there is no conflict of interest in relation to their past, current, and/or forthcoming assignments. If there is a potential conflict of interest, together with the EOI, consultants are required to submit a detailed description of the potential conflict to the agency so that we can investigate further and take it into consideration during the qualification/shortlisting process. Consultants may submit their EOI as individual firms or in a joint venture with other firms in the form of consortia or subconsultants. In the latter case, one expression must be submitted on behalf of the entire joint venture, which should include evidence of the qualification status of all the firms involved. Furthermore, a thorough description of the roles and responsibilities of each of the firms should be provided, along with a clear justification of each firm’s involvement and the value it brings to the project implementation. Following the qualification/short-listing process, successful consultants will be invited to submit their bids, which will be evaluated using the Quality and Cost-Based Selection method. EOIs must be delivered in person, by mail, by fax, or by e-mail to the addresses below by [date] [time] [time zone]. Physical address: [physical address] E-mail address: [e-mail address] Fax number: [fax number] Detailed information on the process, terms and conditions, and requirements can be found on the following website: [website] EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 59 >>> ANNEX III: Example Service-Level Agreement (SLA) Terms EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 60 Service Environ- Service Level Standard Description/Measurement Comments ment Credit Target Solution 99.75% The Client’s production database, Production 12.5% of • Exceptions (communicated through Availability processing environment, and Recurring ASP alerts): solution are designated to be Monthly • Scheduled monthly server patching available for online access Service charge (Sunday 12:00 to 5:00 window) 24 hours per day, 7 days per for month in • Scheduled backups (a few moments week, as measured by software which failure pre- and post-cycle) each night through SiteScope. This standard occurred • Scheduled application code promotions represents the average of uptime (Tuesday mornings before 6 :00) across all Omni-hosted products. • Annual/semi-annual maintenance tasks, such as database reorganization and history archive • Other scheduled outages (as agreed between Client and Supplier on a case- by-case basis) Backups 100% Supplier shall be responsible for Production 12.5% of Upon termination, all Supplier database performing master file backups Recurring backup processes and storage obligations of the production environment Monthly to the Client shall cease in accordance in accordance with the following Service charge with the provisions of the Agreement. schedule: for month in • A minimum of 34 daily backups which failure (daily rolling backups in occurred production) • A minimum of 18 monthly backups (monthly rolling backups in production) • A minimum of 7 annual backups (annual rolling backups in production) • Production backups delivered to the off-site storage facility each business day Full or 100% In the event that the restore of Production 12.5% of • Supplier maintains a local copy of the Partial one or more files or the database Recurring database refreshed within the prior Database in the production environment is Monthly twenty-four (24) hours. Recovery required, Supplier shall perform Service charge • Restoration of production data in 6 the restore of the file(s) or for month in business hours from notification from database. which failure Client if backup is <24 hours old. Initiate restore of production occurred • Restoration of production data older data in 2 business hours from than 24 hours (excluding holidays and notification from Client if backup non-business days) will be restored is <24 hours old. within 2 business days. Disaster 100% In the event that a regional or Production 12.5% of The Recovery Point will include all Recovery data center–wide outage occurs Recurring pended transactions through the and a disaster is declared by Monthly Recovery Point. Full recovery is subject Supplier, Supplier shall have Service charge to: Client’s production database for month in • Availability of interface feeds and restored and available for which failure source data. processing at a Supplier Disaster occurred • Source systems that reside in Client’s Recovery site within twenty-four environment or on Client’s desktops (24) hours of the declaration.  The will not be available at Supplier’s Recovery Point Objective (RPO) disaster recovery site. Supplier will is four (4) hours.  ensure connectivity to all external sites included in the Solution. • TEST or DEV databases are not hosted at Disaster Recovery sites. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 61 Service Environ- Service Level Standard Description/Measurement Comments ment Credit Target Disaster 100% This Service Level measures --- --- --- Recovery Supplier’s successful test or Test actual execution of the Disaster Recovery Plan, as required by the Order, consistent with the requirement for a Recovery Point Objective (RPO) of four (4) hours. Network 99.9% The network, for purposes of this Production 12.5% of Exceptions (communicated through ASP Availability document, is defined as those Recurring alerts): portions of the network that are Monthly • Scheduled maintenance (defined as monitored and maintained by Service charge 12:00 to 5:00 daily for non-customer Supplier up to the point of entry for month in impacting changes; 12:00 to 5:00 into the Client’s network. The which failure on Sunday’s for customer impacting network providing connectivity occurred changes) into Client’s processing • Scheduled outages for testing or environment is designated to installation of new equipment (as be available 24 hours per day, 7 agreed between Client and Supplier on days per week, as measured by a case-by-case basis) Supplier’s corporate wide area network monitor. Solution Response The client’s production application Production 12.5% of Exceptions (communicated through ASP Perfor- times for ports will respond to requests in a Recurring alerts): mance Omni ports timely manner 24 hours per day, Monthly • Scheduled monthly server patching must be less 7 days per week, as measured by Service charge (Sunday 12:00 to 5:00 window) than 2.0 software through SiteScope. The for month in • Scheduled backups (a few moments seconds for standard represents the average which failure pre- and post-cycle) each night more than response time across Omni- occurred • Scheduled application code promotions 97.5% of the hosted products. (Tuesday mornings before 6:00) available • Annual/semi-annual maintenance time in the tasks, such as database reorganization month and history archive • Other scheduled outages (as agreed between Client and Supplier on a case- by-case basis) Batch Expected: This Service Level measures Production --- The parties will agree and develop a Job/ File TBD on-time processing of batch jobs detailed listing of critical batch jobs and Transfer Minimum: and file transfer operations. The file transfers with associated processing TBD Service Level will be stated in completion deadlines once details terms of either the percentage regarding the technical solution have of number of batch jobs and been finalized. file transfers completed on time or the number of minutes late Supplier is allowed in processing the batch jobs and file transfers. Batch Expected: This Service Level measures the Production --- --- Job/ File 100% number of minutes utilized by Transfer Minimum: Supplier to complete processing –Extended 100% of any specific batch job or file Delay transfer beyond the processing completion deadline identified for that batch job or file transfer versus a requirement that no single delay for any batch job or file transfer exceed the number of minutes determined for the Batch Job/ File Transfer SLA. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 62 Service Environ- Service Level Standard Description/Measurement Comments ment Credit Target Message Expected: This Service Level measures the Production --- --- Processing 100% percentage of inbound messages Reject Rate Minimum: that are rejected or fail to be 99% processed by the Omni Platform, measured as messages diverted to the reject queue. Reject Expected: This Service Level measures Production --- --- Queue 99% Supplier’s percentage Message Minimum: compliance with Client’s service Processing 90% requirement to process reject queue messages within X hours measured monthly. For purposes of this Service Level, Supplier shall consider a reject queue message to have been processed once Supplier has: (i) identified the error or problem that resulted in the message being diverted to the reject queue, and (ii) corrected the error or problem that resulted in the message being diverted to the reject queue, and (iii) processed/actioned / responded to such message in a manner consistent with the manner in which the service would have processed/ actioned/ responded to such message had it not been diverted to the reject queue. Service Expected: This Service Level measures N/A --- --- Level 100% the completeness and on-time Report Minimum: delivery by Supplier of the Delivery 100% monthly Service Level Report. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 63 Problem Support Incident Standard Environment Service Credit Comments Type Priority 1 Refers to any Solution Examples of Priority 1 Issues: • Client must phone issue • ---Supplier will utilize all downtime that prevents • Covered application is into support and document necessary resources until the accomplishment of an unavailable (e.g., OMNI specifics in a support the incident has been operational or mission- DC). ticket. addressed and a resolution critical function as well as • Expected port access is • Supplier shall acknowledge or workaround has been any material functionality unavailable (e.g., web receipt of the notice within provided (24x7, including inoperable or functioning ports are down). thirty (30) minutes of its nights, weekends, and improperly. Production critical • Mission critical jobs are not delivery. Supplier will utilize holidays). or Solution down issues that providing workable results all necessary resources • Client will receive timely affect Client and/or multiple (e.g., trade files are not until the problem has updates on the status of production users. generating). been addressed and a the reported failure. resolution or workaround has been provided. Client will receive timely updates on the status of the reported failure. Priority 2 An incident that is not a Examples of Priority 2 Issues: Supplier shall acknowledge Supplier shall employ Priority 1 Incident that • Response time degraded receipt of the notice within 2 continuous efforts until either (i) adversely impacts from standard. hours of its delivery. the incident has been Client’s use of any material • Performance degradation addressed and a resolution or essential component of prohibits users from using or workaround has been the Solution or (ii) renders the Solution efficiently. provided during supplier’s Client’s use of any immaterial • Solution performance or normal business hours. Client or non-essential component port is operational but will receive timely updates of the Solution impossible or unstable. on the status of the reported impracticable. • Client and/or Third-Party failure, and Client agrees to Users have impacted test and install the Solution access to covered resolution. application. • Some jobs did not provide preferred results (e.g., certain reports are running late). EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 64 Incident Standard Environment Service Credit Comments Type Priority 3 An incident that is not a Examples of Priority 3 Issue: • Supplier shall • Upon supplier’s receipt Priority 1 Incident or Priority • Any incident that impacts acknowledge receipt of of reasonably sufficient 2 incident that has a high a single user’s ability to the notice within one (1) documentation evidencing degree of urgency that perform their job function business day of its delivery. the issue, supplier will requires immediate attention. • Any recurring incident • Tickets opened outside schedule correction of the Moderate-High business that causes the user Supplier’s normal incident to correspond with impact to any environment inconvenience but does the nature of the incident business hours will be that requires a response. not impact their ability to do and the timing of an prioritized accordingly. their job available release. • Any recurring incident that • Client will receive a is cosmetic in nature and recommended workaround has no direct impact on the for use until a fix is service’s functionality or available, as well as the user’s ability to perform the release in which their job function the resolution will be • Client and/or its Third- provided. Party Users have no or unacceptable access to the test website or VRU • Client and/or its Third-Party Users have functionality questions or other research queries • Client and/or its Third- Party Users require software/patches for a non-emergency break-fix situation. • Test Environment restore/ backup. • Need a password reset. Priority 4 • Any other outages, Examples of Priority 4 Issue: • Supplier shall Upon supplier’s receipt technical problems, or Minor Solution acknowledge receipt of reasonably sufficient technical questions with a Enhancements documentation describing the of the notice within two low degree of urgency. • Add link to Spanish Plan issue, supplier will provide (2) business days of its • Priority 4 issues may affect Highlights written notice to the Client a single functional aspect • Change to work order delivery. of either their intention of the application and a listing in DVW • Requests received to change the Solution workaround may be put in • Change to system outside of supplier’s or provide an alternative place. notifications or alerts normal business hours resolution. • Priority 4 issue may be • Systemic change to will be taken into a problem that can exist accommodate for off-cycle consideration on the for more than a few days payroll schedules in any application with • Support Items following business day minimal impact to the • Update to non-financial or as soon as feasible Client and/or the Third- reports based on the impact of Party Users. • Activity and transaction log the issue. • Moderate-low business changes impact to any application. • Adding an OMNI data element to transactional folders. • Update tags on an XML file EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT | GOVTECH PROCUREMENT PRACTICE NOTE <<< 65 Supported by the GovTech Global Partnership Republic of Korea