• Post author:
  • Post published:July 23, 2015

The case for mass school-based deworming in endemic countries stands on two legs: First, the body of rigorous evidence that supports mass deworming as a cost-effective intervention. Second, mass treatment without first testing for infection is cheap, safe, and an efficient strategy for reaching lots of kids quickly. Claims in a recent paper by Calum Davey and colleagues and the Cochrane Review by members of the same research group that have been picked up in the press are based on flawed analysis, and could threaten an emerging public health success story.

Controversy Sells? 

Alexander Aiken, Calum Davey, James Hargreaves and Richard Hayes were tasked by the International Initiative for Impact Evaluation (3ie) to replicate a randomized controlled trial of the provision of deworming pills in schools in Kenya by Edward Miguel and Michael Kremer. Replication and transparency in science are great, and we need more of it. However, we also need to be careful what incentives we create for the replicators—controversy sells after all, right?

And indeed, Aiken et al. look to have done their darndest to generate controversy. When they follow the same statistical strategy in the original paper, which passed peer review of perhaps the top economics journal, Econometrica, they find a few errors but arrive at the same fundamental result that deworming increases school attendance. This replication effort is tremendously useful.

Poor Science 

Then, Aiken et al set out to parse the data differently than Kremer and Miguel. And sure enough, if you make enough changes to how you do the analysis, you can make the key result go away. Is this really surprising? Unfortunately, what Aiken et al do is statistically not best practice. They chop up the data into smaller subsamples for analysis; they include data on kids before deworming treatment starts in the deworming treatment sample. No surprise: These choices, which your Stats 101 professor would counsel against, make the key result weaker.  

The reparsing of the data to get a different result is not a “failure to replicate”. And at the time of the 3ie publication this point was made by several commentators. Kremer and Miguel and their colleagues noted this. Berk Ozler discussed this on his blog. Michael Clemens wrote a paper urging us to be careful about the use of the word “replication” when authors are going far beyond that in their analyses. Other researchers that had been part of the 3ie replication effort have independently questioned the value of “replications” that go far beyond that remit and then suggest that a weakness in the original research had been uncovered.

It’s unfortunate that the peer review process at the International Journal of Epidemiology did not encourage Davey and colleagues to substantively address the feedback that their data parsing choices are statistically flawed compared to the choices made in the original research by Kremer and Miguel. This has resulted in a manufactured controversy that could threaten real programs that help real kids.

Mass Treatment of Kids is Most Effective 

That brings us to the second leg of school based deworming: Mass treatment of all kids in places where worms are endemic is the cheapest and most efficient way to reach infected kids. Treating uninfected kids has no side effects. The medication is very cheap. The tests are very expensive. The Cochrane Review gets this one wrong when they suggest that we need to test before treatment to get health benefits. Presumptive treatment is an operational, pragmatic approach to treating as many sick kids as possible with a limited amount of money.

We have made incredible progress over the past few years in getting more kids to have the chance to live worm-free lives. We can not let weak scholarship and a flawed peer review process — let alone expensive treatment strategies — get in the way of this and hurt kids in the poorest countries around the world.

This Post Has 4 Comments

  1. Robert

    Right on point! Weak scholarship and poor peer reviewing are such a risk to authentic evidence-informed development works. Thanks for speaking out and candidly so.

  2. Julia Wise

    I just want to say thank you for releasing the data. I’m sure it wasn’t an easy choice, since so few researchers do it.

  3. Jim Killon

    Not to put too fine a point on it but figures lie and liars figure. If we were talking about exactly how many bricks it takes to build a wall this high by this long we can have an exact answer with empirical certainty. If we can all agree on the doctor’s oath, to "Do no harm" deworming children is a Very Good Idea. As inexpensive as it is, you can hardly miss hitting a target if your aim is to eradicate parasites that are literally eating children and killing them from the inside out. While the "Worm Wars" are raging between researchers and the Ut-oh squads about exactly what details may or may not be holding up under a new light of scrutiny, children in 3rd world countries are suffering. Here in Peru where we are deworming children at Changes for New Hope ubiquitously, children are benefiting. Whether they are benefiting a little bit or a lot is akin to arguing if a life boat is made out of oak or pine ignoring the fact that the ship is sinking under your feet.

  4. David Roodman

    A suggestion: I think it’s usually best to avoid theorizing or making implications about the motives of those you disagree with in debates like this. States of mind are hard to observe.

Leave a Reply