Replication is basically the independent repetition of scientific analyses by other scientists, with the aim of controlling the results. The principle is well accepted in the natural sciences. However, it is far less common in empirical economics, even though non-reproducible research can barely be considered a contribution to the consolidated body of scientific knowledge.
When talking about replications, it is important to distinguish different notions of replication and reproducibility.
In the narrow sense, replicability means that the raw data for an analysis can be accessed, that the transformation from the raw data to the final data set is well documented, and that software code is available for producing the final data set and the empirical results. Basically, this comes down to a question of data and code availability, but nonetheless it is a necessary prerequisite for replication. A successful replication would then indicate that all of the material has been provided and that the same results were obtained when redoing the analysis.
In the wider sense, a replication could go much further by challenging previous analyses via changing the data sources (such as changing countries, switching time periods, or using different surveys), altering the statistical or econometric model, or questioning the interpretational conclusions drawn from the analysis. Here, the scientific debate really starts, since this type of replication isn’t concerned with simply redoing exactly the same analysis as the original study. Rather, the goal is to rethink the entire analysis, from data collection and operationalization to the interpretation of results and robustness checks.
Unfortunately, very few journals in economics have mandatory online archives for data and code and/or strict rules for ensuring replicability. Moreover, the incentive for making your own research reproducible, and for reproducing research done by others, is low.
In this respect, there are several interesting lessons to be learned by the Reinhart/Rogoff case.
One is that the impact of replication can actually be quite high, especially when replicating papers that have been influential in recent scientific debate. In other words, there is a reputation to be gained by making replications.
Still, it is important to remember that replications that question earlier results are not the only ones that are of value. It is also helpful to know if a specific study could be replicated without any major changes to the results, since this increases the empirical evidence for the results claimed by the authors.
Another important lesson is that involving students in replications can significantly change attitudes towards replication. For students, a replication is a perfect opportunity to perform their own analyses based on an already available paper. They get to learn how experienced scientists tackle applied-research questions and they also learn that the consolidated body of scientific knowledge is constantly changing as it is questioned and transferred to new contexts.
Finally, it is very important for the raw data to be made available so that every step up to the final results of a study can be replicated. Here, Reinhart and Rogoff provide a very good example and do not shy away from the debate about their paper. Indeed, without commenting on the debate itself, the Reinhart/Rogoff case demonstrates that replication can considerably stimulate scientific debate.
In recent years, we have been teaching replication to students at all levels (ranging from undergraduates to Ph.D. candidates) and have set up a large global network to support the idea of replication by students. In our experience, it is unfortunately not possible to replicate all results exactly (although in most cases this does not change the main story of a paper.) One particular issue is that it is rarely transparent how data sets used for the analyses were obtained from raw data.
The main problem with setting up a replication seminar is the identification of studies that are suitable for students at a given level. The necessary material (data and code) has to be available as a minimal requirement, but there also are limitations to the complexity that can be dealt with by students.
As a part of our Institute project on empirical replication, we therefore are collecting and sharing a large dataset of empirical studies. These studies are all potential candidates for replication that meet the minimal requirements for replicability. Information on these studies, as well as additional information about already published replications, is available in a wiki shared with collaborators who join our teaching initiative. Moreover, we will soon provide via the same wiki website additional resources to support teaching replication seminars. We also started a working paper series on replication so that replication papers can be published as reports and provide a forum for discussing replicability as another part of the wiki website.
We welcome you to join our efforts to provide students with these important opportunities. The Reinhart/Rogoff controversy is just one example of the role replication can play in making economics more effective and more transparent. You can find more information and contact us here and here.
