Severity Testing: A Primer

Abstract

This is a primer on Mayo's severity testing technology, briefly explaining step by step the implications of severity in the context of rejection and retention of point nil hypotheses, of (conceptually broader) null hypotheses, and of confidence intervals. I finish proposing the use of a confidence interval heuristic for assessing Mayo's severity straightforwardly. In the present article we shall not concern ourselves with wars, whether statistical or philosophical. Instead, we shall work on a philosophical concept being put forth by Mayo in the last two decades (e.g., Mayo, 1983, 1991, 1996; Mayo & Spanos, 2006, 2010), more recently so with her book Statistical Inference as Severe Testing (Mayo, 2018). Severity testing stands for a procedure a researcher can avail of to falsify (frequentist) hypotheses; yet it may spill beyond that bin to become a broader philosophical approach that can serve to also falsify (Bayesian) models (Gelman & Shalizi, 2013), even entire theories. The article I present here is going to be a primer on Mayo’s severity concept in the frequentist realm. However, I sympathise with Gelman and reckon the unstated goal is to advance such primer as a step towards using severity in line with Gelman’s ideas and further, including Taleb’s own use of falsification as a tool to get more acquainted with what our models and theories prevents us from learning, such as about extreme events and Black Swans (e.g., Taleb, 2005, 2010).