Blog

The last few decades have seen explosive growth in the number of impact evaluations—randomized evaluations in particular—that seek to guide policy decision-making in low- and middle-income countries. However, there are important gaps in translating this ever-expanding wealth of knowledge into actionable and credible policy insights. When moving from evidence generation to aggregation, translation, and adaptation, a key challenge is the lack of standardization and comparability of the main results produced by each study. 

AidGrade is excited to collaborate with the World Bank as part of a global consortium of institutions to fill this gap by constructing an Impact Data and Evidence Aggregation Library (IDEAL). Hosted by the World Bank, IDEAL will be fully open-access and free for the world to use. Users can easily search, compare, and visualize relative effect sizes on a range of outcomes and of many different policy interventions across countries and contexts. We expect IDEAL to facilitate quantitative meta-analyses as well as more qualitative systematic reviews, whether such efforts are centered around an intervention, policy, population, or outcome. 

Importantly, IDEAL will help guide the standardization of key information from randomized evaluations, enabling more meaningful comparisons across studies while unlocking the potential to answer a wide range of methodological questions.

The IDEAL team will begin with a pilot phase, during which we will extract a minimum set of fields for a curated initial set of studies, and develop protocols for the library’s ongoing growth and maintenance. Once established, we will invite others to join, guided by consistent protocols for adding new content and rigorous quality control. Ultimately, we aim to scale across many sectors and programs in global development and develop a set of pedagogical resources for mainstreaming evidence aggregation methods across the research and policy community.

If you are interested in learning more about AidGrade’s contributions to IDEAL, and how to support and/or collaborate with us, please contact info@aidgrade.org.

AidGrade is hiring for a full-time research assistant / researcher. AidGrade is a non-profit research institute that focuses on generating and synthesizing evidence in international development. Recently, it has focused on using natural language processing to facilitate meta-analysis, collecting prior beliefs about the impacts of various programs, and research relating to how evidence is interpreted and used.

The research assistant will work closely with Dr. Eva Vivalt, assistant professor at the Australian National University, on several projects relating to prior beliefs and belief updating. Responsibilities will include survey design, data analysis, and a small amount of grantwriting. Experience with impact evaluations and some graduate study is desired. Co-authorship may be available for some projects depending on the research assistant’s level of engagement. It is possible to work remotely, and salary will be commensurate with experience.

Please apply by Jan. 31 by submitting a CV and cover letter to eva.vivalt@anu.edu.au.

We are hiring for two positions:

1) Short-term work involving travel, to help develop and implement a survey. For this role, we are looking for someone who can begin immediately. As the travel will be limited to short stints, this work could be done concurrently with studies, for people past the coursework phase of a research degree.

2) Long-term work assisting with several projects. Tasks will depend on the applicant’s strengths but may involve reading academic papers, developing and implementing surveys, and data analysis. Applicants should have a quantitative background. A master’s degree is preferred, but exceptional undergraduates should also apply. Work can be done remotely and we will consider both full-time and part-time applicants with a minimum of 20 hours per week.

To apply, please send cover letter and CV by email to: info@aidgrade.org. Please indicate which position you are applying for.

Deadline: Early application is encouraged, with a deadline of Feb. 10 for the short-term position and Feb. 24 for the long-term position.

We are looking for someone with great communication and interpersonal skills who is also quantitatively-oriented. This person will assist with outreach to policymakers and staff at international organizations.

We are willing to consider both part-time and full-time applicants, who may work remotely from any geographic location. If you are interested, please send an e-mail with CV and cover letter by Oct. 3 to: info@aidgrade.org.

We focus on gathering data from impact evaluations and synthesizing that information through meta-analysis.

The hardest part of doing a meta-analysis is gathering the data. Each paper must be read by two different people, who manually extract information from the papers; if they disagree on any aspect, a third person is called in to arbitrate.

This process doesn’t scale very well and it is a large part of why meta-analyses are often out of date. With many new studies coming out, we would like to be able to streamline this process.

Machine learning allows us to do so. We can extract information about the programs and studies as well as information about the effect sizes they found. For each extracted piece of information, we will also generate a probability that the information is correct.

At the very minimum, this will greatly reduce the amount of time it takes to identify basic characteristics of studies, such as where they were done and which methods they used. It is also the only way to really stay abreast of the latest research for a variety of areas. Given that the methods should be scalable to much of health, education, and economics, we will build this tool in a general way so that its results can inform policy even in developed countries.

You can think of this as a ScienceScape for meta-analysis.

To support this initiative, we have a crowdfunding campaign. Please consider donating.