“Iterate, again” is one of our core values at Evidence Action. We invest in monitoring and evaluation of our programs to regularly validate whether our intended outcomes are achieved. In our Dispensers for Safe Water program, which provides access to safe drinking water by installing chlorine dispensers near community water sources in Kenya, Uganda, and Malawi, we verify their use by measuring the adoption rate of chlorine among the households that our dispensers serve.

In addition, we recently conducted a survey to confirm the number of people served by our dispenser footprint. Beyond embedding monitoring and evaluation practices into our program approach, this is part of our carbon monitoring plan, which requires a refreshed survey of our population served every two years. Through the survey conducted in late 2017, we found that our population reached declined from 4.7 million, which is based on surveys conducted for each new dispenser installation during program scale-up from 2012-2016, to 4.0 million people program-wide today.

We believe it is important to be transparent about changes in our data, whether positive or negative, given the opportunity for learning and the resulting adjustments to our program. A few reasons may explain the drop, though we are still analyzing how big each driver might be:

Refining data collection methodologies:

  • The program’s initial population reach was based upon a household list compiled when each new dispenser was installed during scale-up of the program between 2012 and 2016; prior to dispenser installation, we approached a village elder or landowner to list all households using a specific water point. This number may have been slightly inflated if these individuals felt the need to do so for a dispenser to be installed, if households were recorded in overlapping catchment areas of different water points, or simply because the individuals asked were unfamiliar with the exact households using a particular water point.

  • Today, our community promoter lists all households using a particular water point with an existing dispenser; they are less likely to inflate the number or list households in overlapping catchment areas as they are more familiar with the households using their particular water point. Village elders/landowners are still engaged for their approval of new dispensers but their input is no longer used to assess existing dispenser population reach.

New Community Points-of-Access:

  • New water points appear in the villages that we serve, meaning that some households may switch to using more easily accessible water points that do not have a dispenser installed. This can be driven both by the passage of time (e.g., due to changing weather and water patterns) as well as by rising incomes, since certain households or communities may now be able to afford the installation of their own boreholes. The latter, a sign of positive socio-economic progress, may impact dispenser reach; we typically choose not to move dispensers to such boreholes as we require that a minimum of 10 households use the water source and that the water source be accessible to all community members.

Migration to new areas:

  • The decline may also be driven by migration from rural to urban areas over time, as some individuals move to urban areas for better jobs, education, and other opportunities. We are investigating whether this is a potential reason through a survey of our community promoters.

Community Willingness

  • There is a natural attrition of dispensers as certain units may be repeatedly vandalized (3+ times), which our staff takes to indicate that the community does not want the dispenser there. This only explains a small amount of the decline, as this has reduced our dispenser count and population served by about 3% (roughly 120,000 people) since the peak in mid-2016. We have actively addressed the issue by re-engaging community promoters this year to reinforce awareness of the importance of treating drinking water so they actively safeguard and encourage use of their community’s dispenser.

As always, we go where the evidence and data takes us. We are analyzing these potential causes of population decline to determine the magnitude of each one, and we will use the results to iterate on program delivery. For instance, we may find it is most cost-effective to install new dispensers at or move existing dispensers to more heavily used water points.

Moving forward, we believe that there is little likelihood of a similar decline in the future: by understanding the drivers of this change, we will be able to monitor our reach more effectively and mitigate the impact of changes where necessary.

Despite the decline, the intervention remains highly cost effective. Dispensers for Safe Water was named a “Standout” charity in 2017 by GiveWell because of its cost-effectiveness in averting childhood deaths caused by unsafe water. Adoption rates remain high, at 58% program-wide in May/June 2018, which is over 5x higher than comparable water treatment interventions. We continue to have full confidence in our program’s impact on the millions that it serves, providing safe water for only ~$1.25 per person per year.