Using prediction markets and forecasting surveys to predict 28 replication outcomes of classic articles in social psychology and judgement and decision making

Abstract

Can researchers predict if classic findings published in the field of social psychology and judgement and decision making replicate? We set up prediction markets and a forecasting survey for predicting replications of 28 experiments of classic well-cited articles. Forecasters predicted if the original results would replicate, where a successful replication was defined as an effect in the same direction as the original and a signal (p-value lower than 0.05). Of the 28 original studies, 16 (57%) met the replication success criteria, compared to a predicted replication rate of 70% in the prediction markets and 65% replication rate in the survey. We concluded only suggestive evidence for associations of replication outcomes with prediction market prices (r = 0.43, 95% CI [0.07, 0.69]) and average survey beliefs (r = 0.26, 95% CI [-0.12, 0.58]). The prediction market effects were similar to observed effects in previous prediction market studies and suggest that prediction markets can to some extent predict replication outcomes, yet predictions are far from perfect and conducting replications is much more informative about the credibility of published findings. Data and code are available at https://doi.org/10.17605/OSF.IO/2KMH7.