How effective development programs are has traditionally been guesswork. NGOs and aid organizations have also had every incentive to exaggerate their claimed benefits. Many now have “monitoring and evaluation” schemes to track their outputs, such as the number of textbooks they distributed, and while these systems are a step in the right direction, they do not tell the full story.
There are two types of evaluations that can be done of an organization’s program: causal or non-causal. Monitoring and evaluation (M&E) usually refers to non-causal evaluation, whereas impact evaluation specifically refers to evaluation that can attribute a particular effect to the program itself.
What’s the difference? Suppose one observes that after a program provided textbooks, test scores rose. This would be observable through monitoring and evaluation. However, if one does not know what would have happened to test scores in the absence of the program – the counterfactual – one cannot say that test scores rose because of the program. Perhaps test scores would have risen for everyone, regardless of whether they or their class received textbooks. Impact evaluation allows one to say whether a program itself was responsible for an observed outcome.
The diagram below illustrates. In this example, test scores rise with textbooks, but they rise more for the control group, which did not receive textbooks. The segment shaded in orange shows the extent to which the group that received textbooks fell short of the group that did not. Perhaps the textbooks were in a language not spoken locally, or at the wrong level. The evidence on programs providing textbooks is not great, so this outcome is plausible. Yet monitoring alone would have suggested that the textbooks were a great success.
