A cautionary remark must be made regarding the selection of filters. There is a great concern in the development economics community that researchers can sometimes “game” results. By searching across many different specifications – in other words, running many regressions – a researcher can find one particular regression that gives the results he or she wants, and only report the results of that one regression. This problem is not limited to economics; it also plagues medicine and many other fields. The specification searching is not even always necessarily intentional; researchers can easily convince themselves that the “best” result they obtained is the “real” result.
The relevance of this fact for us, apart from making us more cautious in interpreting and extracting data from papers, is that by opening up the data in this way, we need to be careful not to accidentally engage in the meta-analysis equivalent of specification searching. This means that we made sure to specify which filters we would use in advance.
Similarly, if you try out our online meta-analysis tool, we must caution you to first determine which filters you would like to select and to make this a one-time decision rather than modifying your selections after you view the results to try to shift your results in the direction you would prefer. It is a natural part of human nature, to believe what one wants to believe, and one must fight against it by strict adherence to binding rules.
Thank you for your attention to this important matter!