GOVERNANCE NOTES JUNE 2020 No.23
CHATBOTS FOR SERVICE DELIVERY
MONITORING: A CIVICTECH PILOT IN
MADAGASCAR
Prepared by Olivia Rakotomalala, Tiago Peixoto and Saki Kumagai1
Growing evidence confirms that citizen engagement COUNTRY CONTEXT: DECENTRALIZED SERVICE
is key to improving the delivery and quality of DELIVERY
public services, management of public finances, Madagascar is the fifth largest island in the world with
and promoting social inclusion, tangibly improving a population of 25.6 million. In spite of its considerable
people’s lives (World Bank Group 2014). The advent natural resources, it is one of the poorest countries
and availability of innovative technologies provide in the world, with an estimated poverty rate of 74.1
new opportunities to reach citizens, aggregate their percent in 2019—far exceeding the regional average of
“voice” and demands, help governments respond, and 41 percent. The country’s human capital index ranking
partner with citizens to find and implement solutions is among the lowest worldwide with an estimated 1.4
collectively. With the right approach, CivicTech enables million children having dropped out of primary school
citizens to overcome income, social, and geographical in 2012.2 The government of Madagascar adopted a
barriers to interact with governments and participate national development plan to address the country’s
at the local or national level. development challenges, focused on three pillars:
Launched in February 2019, the CivicTech pilot in (i) improving basic social services; (ii) strengthening
Madagascar supported the development of a Facebook governance and democracy; and (iii) promoting economic
chatbot to enable third-party and citizen monitoring growth. Decentralization is identified as a core feature to
of service delivery operations for the Madagascar improve service delivery nationwide. The PAPSP supports
Public Sector Performance Project (PAPSP). This this vision by helping to improve revenue management—
Governance Note documents the experience gained and boosting the government’s revenue mobilization efforts
lessons learned. A similar approach can potentially be and public financial management effectiveness—and local
replicated for operations through community-driven service delivery nationwide as well as in the education
development and local government and decentralized sector within selected regions (World Bank 2016).
service delivery operations that benefit from third- Madagascar’s decentralization framework establishes
party and citizen monitoring with a multi-channel three levels of government: provinces, regions, and
engagement structure (e.g., engagement offline and communes. Communes are the level of subnational
with mobile and web resources). government led by elected officials that is closest to
1
This Governance Note was prepared by Olivia Rakotomalala, Tiago Peixoto, and Saki Kumagai (Governance Global Practice (GGP), World
Bank). The pilot was partially supported by the GGP’s programmatic analytics and advisory service (ASA), “Citizen Engagement: Re-building
the State and Citizen Social Contract.” The ASA aims to provide analytical insights, knowledge, and learning the implementation of the next
phase of the World Bank Group’s Strategic Framework for Mainstreaming Citizen Engagement in World Bank Group Operations. The authors
express their appreciation to Asmeen Khan and Helene Grandvoinnet for their guidance and comments, Kimberly Johns for peer review
comments, and Barbara Rice for editorial support.
2
For the overview, see the World Bank website at https://www.worldbank.org/en/country/madagascar/overview.
1
GOVERNANCE NOTES JUNE 2020 No.23
citizens in urban and rural areas. In September 2014, the The rollout of the toll-free 3-2-1 hotline and participatory
government adopted new decentralization legislation that budgeting mechanisms have been successful in
increases the responsibilities of communes for service increasing access to information on development issues
delivery, particularly the rehabilitation of education and engaging citizens at the local level.
and health facilities and investments in small-scale
infrastructure, such as rural roads and markets. Yet FACEBOOK CHATBOT FOR THIRD-PARTY AND
local governments have limited capacity for revenue CITIZEN MONITORING
mobilization, and the budget remains highly centralized. Understanding the local context is important to design
Only 5 percent of total government revenues are suitable and sustainable CivicTech approaches and tools.
channeled outside the Analamanga region, where the A key element is deciding whether to explore using an
capital city is located, despite these combined regions existing open source solution or developing a customized
accounting for 90 percent of the country’s population. In solution. For this pilot, the team took into account
addition, funds being transferred to local governments the following elements: types of communication and
often experience significant delays. interaction needed on the CivicTech tool, cost and time
To alleviate these challenges, the PAPSP provides required to develop it, and uptake. The project identified
grant transfers to rural communes nationwide through the Facebook Messenger chatbot as a tool to explore
the Local Development Fund (FDL). The FDL was created third-party and citizen monitoring of decentralized
in 2007 by the government to support sustainable service delivery and civil works (see box 1).
socioeconomic development though subnational
government capacity building and funding for local
Box 1. Identification of a Suitable CivicTech
public investment projects. The PAPSP grant transfer
Solution for the Pilot
program is designed to encourage citizens to engage in
For this CivicTech pilot for third-party and citizen
the process of budget allocation decision-making through
monitoring of decentralized service delivery in
local participatory budgeting. While this engagement
Madagascar, the following considerations were made:
constitutes a significant landmark for decentralization in
Madagascar, the central government and FDL are unable • Built-in chatbot functions. Chatbots can support
to monitor the allocation and use of these funds in over very targeted and personalized interactions with
1,700 remote communes. This ultimately poses a threat users. They use a self-learning process to monitor
to the credibility of the decentralization process and the user interactions and preferences, and adapt
effective delivery of public services. their responses accordingly. This allows them to
The CivicTech pilot for third-party and citizen offer relevant services to the user and proactively
monitoring aimed at addressing this concern by engaging approach them on a regular basis.
with citizens—and youth in particular. The chatbot was
designed to be linked to the PAPSP’s grievance redress • Cost and timeframe. Building new applications,
mechanism, composed of an “issues-tracker,” in which and promoting their uptake, is more costly than
citizens’ reports were uploaded. These reports were building on an existing, popular platform such as
then sorted and sent to relevant institutions, which Facebook.
were tasked with formulating a response that would be
reported back to citizens. An existing toll-free “3-2-1” • Why Facebook rather than a customized solution?
automated hotline, which delivers information on a wide Many internet users in developing countries are
range of development topics, was leveraged to direct familiar with the Facebook interface. Sixty-five
callers to the PAPSP hotline to find more information developing countries worldwide benefit from
about local government investments through the FDL. Facebook’s “Free Basics” program. It allows users
2
GOVERNANCE NOTES JUNE 2020 No.23
equipped with a smartphone to access a range of ACHIEVEMENTS TO DATE AND CHALLENGES
free basic services, including chats. The majority The CivicTech pilot in Madagascar was rolled out in two
of their internet users—and particularly those who phases: (i) development of the chatbot and integration
cannot afford data plans—navigate the internet of the issues-tracker function; and (ii) deployment of
exclusively through Facebook. This is the case in targeted outreach campaigns on Facebook. The total
Madagascar. While a customized solution may pilot budget was $80,000,3 which was supported by the
provide a more versatile CivicTech application that PAPSP and the GGP’s Citizen Engagement ASA. Once the
matches the scope and scale of grant activities in product is fully developed, additional deployment of the
over 1,700 communes, it would run potential risks chatbot in another project is estimated to cost $10,000
of higher cost and longer solution development per project.
time frames together with uptake challenges. Under the pilot, the chatbot was prototyped,
developed, approved by Facebook4 and tested by the
• Uptake of CivicTech applications. Many users client (the Public Sector Reform Unit within the Presidency
in developing countries have limited Wi-Fi of the Republic of Madagascar) between May and
connectivity, access to mobile data, and on-board August 2019. It was implemented in a Facebook page
memory, which makes it difficult to introduce and compatible with Facebook versions on any device,
and install an entirely new CivicTech application. browser, and handset, including on Facebook Free Basics.
Users might not see any value in downloading The platform supporting the chatbot interacts with the
an unfamiliar new app. Evidence suggests that Facebook application program interface to automatically
a quarter of all downloaded apps are abandoned respond to the feedback of end users, i.e., citizens, in
after a single use. The already existing user real time. End users access the chatbot in the Facebook
population and uptake were considerations Messenger app on their smartphones. The interactions
addressed by the project team. take place through text commands or by selecting
predefined menu options, and the chatbot answers with
Given the context, mobilizing the user base of the
text, voice, and visual or audiovisual content. It prompts
Facebook Messenger chatbot for content generation
Facebook users to monitor public works in their area,
and diverse engagement on service delivery appeared
allowing users to:
to be a more convenient way to target participants
for third-party monitoring. Users are more familiar • Receive information about public works projects in their
with the Facebook interface and more likely to be area to be monitored (e.g., type of work, cost, timeline);
responsive if asked to download a new app. • Submit information about the status of public works;
Source: World Bank.
Note: There are an estimated 2.4 million Facebook users in • Anonymously report potential irregularities;5 and
Madagascar, out of 2.6 million total active internet users, with
• Invite other Facebook friends to join the monitoring
a quarterly growth of 4.3 percent in January 2019. For more
information see the Digital 2019 Madagascar website at https:// efforts.
www.slideshare.net/DataReportal/digital-2019-madagascar-
january-2019-v01.
The chatbot provides user-driven interaction flow along
a predefined decision tree. Users are walked through the
decision tree and can opt into different options, responding
to the chatbot by typing text or numeric answers (“I want
more info”) or clicking on menu options (“More info”).
3
The total cost includes two contracts for the development of chatbot, the integration of the issues-tracker function, and the deployment of
outreach campaigns; a World Bank short-term consultancy; and World Bank staff cost.
4
Chatbots must go through an approval process by Facebook to ensure that the chatbot meets the platforms’ standards and terms of use.
5
Information can also be provided through a hotline that is integrated with the issues-tracker.
3
GOVERNANCE NOTES JUNE 2020 No.23
A two-week promotion effort in May 2019 used Despite these achievements, the chatbot is not yet
the 3-2-1 hotline to encourage users to receive more entirely operational. Delays were caused initially when the
information from the PAPSP hotline. More than 46,000 tech team struggled to make information available for
users heard the promotional message, and 83 percent the more than 1,700 communes, sometimes with several
(more than 38,000 users) expressed an interest in ongoing or closed projects within the same commune.
receiving a call back from the PAPSP hotline. This The off-the-shelf product required further development.
overwhelmingly positive response was unexpected and In addition, in June 2019, Facebook unexpectedly disabled
resulted in a budgetary shortfall with so many users to the “share your location function” for all chatbots, which
call back. As a result, only a sample of 1,863 call backs underlies the commune look-up feature, and allowed
were made, of which 40 percent were successful. Of users to obtain more information on the PAPSP-funded
the successful calls, 40 percent asked to speak to an projects in their vicinity. This complication contributed to
operator. More than half of the users called were between delays while the tech team tried to develop alternative
the ages of 18 and 24, and overwhelmingly male (see proposals to make the look-up feature work.
tables 1 and 2). Once this stage is finalized, the government plans
to organize testing and feedback sessions with users
Table 1. Feedback from the PAPSP Hotline Promotion, and civil society. They will focus on highlighting any
May 15–31, 2019
context-based improvements in the overall design
Feedback Number Percentage and deployment of the chatbot as well as highlighting
any technical challenges that might prevent the user
Would Yes 38,682 83%
you like to from having a seamless experience. The tech team will
receive a call integrate possible adjustments to the chatbot as required
No 7,854 17%
back from
the PAPSP based on the participants’ feedback.
hotline? Total responses 46,536 100%
LESSONS LEARNED
Unsuccessful call
1,094 59% Six key lessons learned during the implementation of this
(no response)
pilot have important ramifications for replication in other
Successful call 769 41% projects and countries.
Call-back
Call not completed 470 25% Creating accountability requires leadership and
statistics
coordination to close the feedback loop. Transparency
Call fully completed 299 16%
through access to information is essential, but not
sufficient to achieve accountability. Projects must
Total 1,863 100%
focus on closing the feedback loop by ensuring the
government’s responsiveness to citizens’ concerns
Table 2. Demographics of Successful Call Backs
through the chatbot or other third-party monitoring
Age in Years Female Male Total tools. Establishing the right institutional framework
18–24 296 720 1,016 is key and requires high-level leadership. This can help
25–34 54 201 255
ensure the allocation of adequate resources to respond
to citizens’ concerns. It also signals to civil servants that
35–44 23 84 107
they are given sufficient latitude to respond transparently
Over 44 9 34 43 to citizens’ feedback. Full or partial outsourcing of
Under 18 138 304 442 services to professional call centers is an option.
Total 520 1,343 1,863 Coordination with anti-corruption agencies is essential to
4
GOVERNANCE NOTES JUNE 2020 No.23
follow-up and investigate any allegations or anomalies. CivicTech in particular. The pilot has demonstrated that
This requires allocation of dedicated resources. this approach can be highly effective, with promotions
on the 3-2-1 hotline generating many more calls
Conducting user testing is essential to deploy user than initially foreseen. However, they also generated
centric CivicTech solutions. User testing was done using significant subsequent work to respond to the sudden
task scenarios, where users were expected to accomplish and unexpected demand, for which government was
specific tasks, in this case requesting information or unprepared. When adopting such a mobilization strategy,
reporting a case to the authorities. This is only one of projects should be ready for the response and sudden
a series of tests that should be continued as the tool is rise in demand. Training should be conducted before the
refined and released to the wider public. User testing is launch, and financial and human resources adequately
recommended for future development of any CivicTech allocated.
solution in different settings. Testing user experience and
flow of engagement is essential to create a successful Relying on an off-the-shelf CivicTech solution can have
experience for the end user. The design aspects of drawbacks. While it is most useful to rely on a platform
modifications and user testing should be an ongoing effort, that already has a strong audience such as Facebook,
with several iterations to make needed improvements the project should be prepared for possible changes
for the overall user-engagement experience. This is in the policy of the solution provider(s) over which the
particularly important since chatbot experts are not user project has no control. This is the case, for example, with
and content experts, and the government tends to adopt the “share your location” function that was withdrawn
an institutional approach that does not promote uptake of by Facebook, which can lead to very costly alternative
CivicTech initiatives, especially among youth. In addition, development. Budget should be allocated during planning
it is essential to test the product within the target for unforeseen development costs.
user’s familiar environment, where natural constraints
(e.g., connectivity) can be assessed and they will feel Allocating the maintenance and update costs
comfortable providing candid feedback. up-front can contribute to the sustainability of
CivicTech activities. Even with an off-the-shelf
Factoring in the complexity of multiple points of sale solution, digital technology is a living product that
is critical to design, plan, and roll out the CivicTech should not be considered simply as a capital cost. It also
activities. Projects should take into consideration the requires recurrent costs for maintenance and constant
complexity of designing digital technology with a large updating. This is especially the case when leveraging
number of points of sale. Madagascar’s PAPSP targets other platforms (such as Facebook) and facing possible
more than 1,700 communes. The scale—the number changes in policy over which the client has no control.
of points of sale—was an unexpected challenge for the Solutions should be considered during project planning
technical team and generated delays given the off-the- and budgeting for the financing of recurrent costs to
shelf solution initially discussed lacked the capacity ensure sustainability.
to manage such a large volume of information. Such
complexities should be factored up-front into project
planning and budgeting.
Leveraging a successful point of interaction works
but be prepared for the demand. The common practice
in the private sector is to use a successful point of
interaction with users to promote another service. Yet,
this approach is rarely used in the public sector and for
5
GOVERNANCE NOTES JUNE 2020 No.23
REFERENCES
World Bank Group. 2014a. Strategic Framework for
Mainstreaming Citizen Engagement in World Bank Group
Operations: Engaging with Citizens for Improved Results.
Washington, DC: World Bank Group. http://documents.
worldbank.org/curated/en/266371468124780089/
pdf/929570WP0Box380ategicFrameworkforCE.pdf.
World Bank. 2016. “Madagascar — Public
Sector Performance Project.” Program Appraisal
Document Report 1547, World Bank, Washington,
DC. http://documents.worldbank.org/curated/
en/513681468197098764/pdf/PAD1547-PAD-P150116-
OUO-9-IDA-R2016-0116-1.pdf.
GOVERNANCE GLOBAL PRACTICE
Guiding Results through Public Institutions
Governance Notes captures knowledge derived from
World Bank engagements and technical and financial
assistance requests.
The views expressed are those of the authors and do not
necessarily reflect the views of the World Bank.
For more information, contact: govgplearns@worldbank.org.
6