S4: Advice and Information on Computational Reproductions

WEBINAR REPLICATE REPLICATE: HOW YOU CAN TAKE PART

SESSION 4: Advice and Information on Computational Reproductions: Students May Find Them Messier Than Expected

Drawing on experience teaching and conducting replications this talk focuses on the most basic form of replication: the computational reproduction. This means simply trying to reproduce the numerical results of a previous study using the same data and statistical routines from that study. As it turns out, there are many potential pitfalls that come from both the original study and the replicator, that could lead to different results. Different computing environments or data versioning and formatting are also culprits.

This talk reviews these pitfalls using examples from published literature and several attempts across disciplines to computationally reproduce findings published in journals. It also reports results from a crowdsourced replication, where eighty-five independent teams attempted a computational replication of results reported in an original study that is central to economics and many other disciplines on the links between policy preferences and immigration. Although when teams had the original data and code they were able to achieve a high rate of similar results (95.7%), a random half of teams did not get the original code and struggled (89.3%). What was more surprising was that exact numerical reproductions to the second decimal place were far less common (76.9% and 48.1%). This has wider implications for science obviously, but it should serve as a lesson for students using replication that things are not a clean cut as one might hope. (full paper on the crowdsourced study).
Please watch the pre-recorded video for this session here. The live session will then focus on discussion.
For discussion please use the session's page on the ReplicationWiki.

SHORT BIO
Nate Breznau, Postdoctoral Fellow at the University of Bremen. Researcher at the Comparative Research Center “The Global Dynamics of Social Policy”. Principal Investigator of the research project, “The Reciprocal Relationship of Public Opinion and Social Policy”. Principal Investigator of “The Crowdsourced Replication Initiative”. Open Science advocate. Open Science Fellow at Wikimedia with the project, “Giving the Results back to the Crowd”. User of preprints. Crowdsourcing and Mertonian-norm advocate. Author of the open science and crowdsourcing blog “Crowdid”.