How Much Evidence Is “Enough”? Our 5 Factors for Assessing Impact

Evidence Action works on programs for which there is a solid evidence base of positive impact, often in the form of randomized control trials. We develop the business models to scale these evidence-based programs so they benefits millions of the most poor and marginalized people. Evidence Action doesn’t work exclusively on a particular sector, like water or microfinance, or has a commitment to a particular kind of service delivery model. We are guided by evidence of impact first.

WHAT DO WE CONSIDER WHEN ASSESSING INTERVENTIONS TO EXPLORE OR SUPPORT?

We have written previously about the evidence that supports chlorine dispensers and deworming that illustrate how we think about this question. When we consider scaling an intervention that seems to be promising, we are looking at these five characteristics:

• Peer-reviewed rigorous evaluations that demonstrate a clear causal effect between the intervention and the desired impact;

• Related evaluations that add weight and context to the findings of the main research line;

• Evidence from multiple settings and contexts that inform our understanding of the resilience of a particular finding;

• A compelling explanation of the mechanisms at play, whether they come from economic theory, market research, or medicine;

• Confidence that a result holds in less controlled environments, and thus in real life, and persists over time.

We think if this list as a bit like a set of puzzle pieces; when you put them together, they create the full picture. Thinking through how to generalize from results of randomized trials is not a mechanical process. Replications of particular evaluations are great, but there are several other ways to get to a solid understanding of why a specific intervention reliably creates a particular impact, the confidence that this impact holds over time, and that the intervention can be implemented in a real life situation for millions of people.

Take, for example, the fact that chlorine from Dispensers for Safe Water is free of charge. The evidence for this comes from randomized control trials performed at dispensers, but it is also strongly bolstered by findings from other settings that demonstrate that even small positive prices for preventative health products depress demand. Dispensers for Safe Water might work because we know from related contexts that people are subject to habit formation and peer effects, and that the dispenser technology takes advantage of these tendencies. A range of evidence helps us understand the underlying mechanisms behind our business model – the reason why people behave the way they do.

We are big fans of the growing evidence base of promising approaches to tough challenges in global health and development. This evidence guides us as we strengthen our main programs and experiment with new ones. And one thing you can be sure of: We’ll report back as we test how to go from specific promising results to impact at scale.