Monthly Archives: November 2013

Post by Eva Vivalt, @evavivalt.

AidGrade is starting another crowdfunding effort. You might wonder: why now? You’re not new so why should I support you?

There have been a lot of interesting things coming out of this work. Dare I say, more interesting than when we launched last year.

For one, we are looking more closely at the issue of generalizability. How much can you generalize from an impact evaluation’s results and are there any factors that improve your ability to make out-of-sample predictions?

This is a huge, important topic, and without having painstakingly collected the data from hundreds of impact evaluations, I’m not sure how you could answer it. Preliminary working paper which contains errors here. By adding impact evaluations to the data set, we’ll be able to say a lot more.

We’re expanding our previous analyses to look more closely at study quality and will put out white papers on each of the topics.

And we are looking at the issue of specification searching and publication bias. So far, it seems like it’s not as bad as you would think.

So, yes, we’re asking for more money. It’s because we’re doing a whole lot more than we did when we started (even more at the link). And frankly, it’s not a lot of money relative to the importance of the questions answered, and you will not find such a cost-effective group elsewhere.

A lot of people seem to like to focus on what we know. I’d like to focus for a second on what we don’t know. Particularly on the big, important questions which we can, with your help, answer.

Post by Eva Vivalt, @evavivalt.

We’ve stayed away from recommending specific charities. AidGrade gathers data from impact evaluations and analyzes that data, focusing more broadly on the effects of different types of programs and how they vary across contexts. Very few NGOs do any kind of impact evaluation, so we’re naturally a bit distant from that. Further, other organizations already have a lot to say about individual charities. Whenever someone asked me about how to best contribute to the relief efforts in the Philippines, for example, I would recommend they check out this old GiveWell post.

But people keep asking and asking for ways they can help, so we’re now letting you click through the “donate” links under “Examine a Program” and “Compare Programs by Outcome” (under the “Donors” tab), linking every type of intervention to a specific charity.

The links aren’t exact. How would you go about donating to a conditional cash transfer program, for example, when they are typically run by governments? You might think that some kinds of child sponsorship programs or scholarship programs are in effect conditional cash transfers, but there remain some similarities. In the absence of a good match, AidGrade is instead directing those interested in conditional cash transfer programs to GiveDirectly, which provides unconditional cash transfers.

With this caveat, how did we come to decide which organizations to feature for each intervention?

We followed several rules of thumb:

1. It should be an organization that does the work itself instead of being largely focused on advocacy. Advocacy work can be highly important, so we may revisit this in the future, but the concern is that it’s very hard to measure when advocacy is having a real effect and our findings are on programs themselves and not on advocacy for programs. It’s simply a closer match.

2. It should be a one-program organization as much as possible, to avoid the fungibility problem whereby if an organization does 10 things, and you donate to support 1 of them, they end up redirecting funds to support the other 9.

3. Where possible, go with the clear frontrunners. For example, regarding insecticide-treated bed nets to prevent malaria, the Against Malaria Foundation is one of the top charities of any kind recommended by both GiveWell and Giving What We Can. We look at their recommendations for charities that focus on a particular type of program.

4. All else equal, the organization should care about evaluation, like Evidence Action does.

These rules of thumb can sometimes conflict, but we adhered as closely as we could. The image attached to this post is misleading: nothing is so pristine as the straight road pictured. Undoubtedly, some of our matches are inexact, but it’s a step in the right direction. Bear in mind, since we are trying to link all the programs to specific NGOs, we are even providing links for those programs which don’t seem to be the most effective at achieving a particular goal.

That said, here are the matches we came up with. Would you suggest anything different?

Bed nets: Against Malaria Foundation
Conditional cash transfers: GiveDirectly
Deworming: Evidence Action: DeWorm the World
Improved cookstoves: Global Village Energy Partnership
Microfinance: Kiva
Safe Water Storage: Evidence Action: Dispensers for Safe Water
Scholarships: Pratham
School Meals: World Food Programme
Unconditional cash transfers: GiveDirectly
Water Treatment: Evidence Action: Dispensers for Safe Water

Coming soon: new topics!

We are making good progress on the next set of topics and hope to release them one by one over the fall. This set of topics includes:

Contract teachers
Financial literacy
HIV education
Irrigation
Micronutrient supplementation
Micro-health insurance
Mobile phones
Performance pay
Rural electrification
Women’s empowerment programs

Several of these topics actually comprise more than one intervention that are separately grouped (for example, different kinds of micronutrient supplementation). So far, eight of the topics have been fully coded and reconciled.

Also, remember to try our new and improved meta-analysis app! You can now view results by paper and download them. Special thanks to our web development genius Alex Robson!

Post by Eva Vivalt, @evavivalt.

I recently had the chance to sit down with Elizabeth Tipton, assistant professor at Columbia University.

Talking about the pros and cons, she made an interesting point. While some people argue that one shouldn’t combine studies because they are all so different, in practice we already do this in our heads all the time. One difference between meta-analysis and this informal mental process, and an advantage of it, is that all the assumptions are spelled out clearly up front and examined. People have all sorts of psychological biases that could affect how they are combining results in their heads.

One might wonder how much one should weight certain studies and how much results can generalize from one context to another. This is an empirical question that can only be answered by looking at the data. We need more study, not less. Will post soon about work being done on this subject.