We follow certain conventions in extracting the results from papers. Short answer: we typically use the authors’ preferred specification but look for those results that have the fewest controls (which could be used to bias results). Long answer: it’s complicated, and full details can be found in the Coding Manual.
Sometimes results are presented in a paper in a way that doesn’t seem at first glance to line up with what is on the website. There are a few possible reasons for this:
1. The results were presented in an alternative format.
For example, a paper might have presented the mean attendance rate, the standard deviation of the attendance rate, and the number of observations for each of the treatment group and the control group. These six numbers can be converted for combining with numbers presented in a different format – for example, a regression coefficient and standard error.
2. The results were presented for different subgroups.
If a paper reported results for several different subgroups, we collect the results for each subgroup separately, but for the sake of presenting on the website we use the aggregate result. This risks Simpson’s paradox, but ensures we treat papers equally. Our own meta-analyses can go into more detail.
3. We made a mistake.
It’s always possible, though reasons #1 and #2 are much more likely since each result was double-coded and a third researcher reconciled any disputes. If you think we collected the wrong data for a certain paper, please let us know! This is a living website and updates are part of the strategy to present you with the most accurate results.