1. InterModel Vigorish (IMV): A novel approach for quantifying predictive accuracy with binary outcomes

With Ben Domingue, Jessica Faul, Jeremy Freese, Klint Kanopka, Alexandros Rigos, Ben Stenhaug and Ajay Tripathi. Working paper available here, code library available here.

Abstract: Understanding the ``fit’’ of models designed to predict binary outcomes has been a long-standing problem across the social sciences. We propose a flexible, portable, and intuitive metric for quantifying the change in accuracy between two predictive systems in the case of a binary outcome: the InterModel Vigorish (IMV). The IMV is based on an analogy to weighted coins, well-characterized physical systems with tractable probabilities. The IMV is always a statement about the change in fit relative to some baseline model—which can be as simple as the prevalence—whereas other metrics are stand-alone measures that need to be further manipulated to yield indices related to differences in fit across models. Moreover, the IMV is consistently interpretable independent of baseline prevalence. We contrast this metric with alternatives in numerous simulations. The IMV is more sensitive to estimation error than many alternatives and also shows distinctive sensitivity to prevalence. We showcase its flexibility across high-profile examples spanning the social sciences. The IMV allows for precise answers to questions about changes in model fit in a variety of settings in a manner that will be useful for furthering research and the understanding of social outcomes.

2. From a Seed of Doubt Grows a Forest of Uncertainty

With Jiani Yan and Mark Verhagen. Code is available here. Some slides from recent talks available here. Preprint coming soon!

Abstract: The current best practice in ‘Open’, ‘Reproducible’ and ‘Responsible’ research is to hide the variation caused by pseudo-random number generators (PRNGs) through the arbitrary use of a ‘seed’ or ‘random state’ in algorithmic pipelines. However, in the process of doing so, we argue that researchers index into the scientific record an insurmountable number of outcomes with seeming certainty, when they are in fact anything from certain: eliminating this variation is the opposite of what responsible researchers and practitioners should be doing. PNRGs are almost ubiquitous in some research areas – occurring in a large proportion of quantitative and computational research designs – and the potential variation in the estimand or outcome of interest is hitherto significantly under-appreciated. We undertake a series of substantial replication projects of highly published work, primarily in the form of Monte Carlo simulations, Machine Learning, and more traditional inferential designs. We show just how large the variation caused by the instantiation of PRNGs can be. We conclude with recommendations on how to embrace this variatiaon for the betterment of scientific society, and how to responsibly conduct research designs which legitimately reduce it where possible.

3. The Legacy of Longevity: Persistent inequalities in UK life expectancy

With Aaron Reeves, Felix Tropf and Darryl Lundy. Working paper coming soon!

Abstract: That global life expectancy has more than doubled within the previous two centuries is–by any objective standard–something miraculous to behold, and the academic literature across the fields of economics, demography, public health and evolutionary biology have all contributed to our understanding of the mechanisms behind the regional variations in the demographic transitions in mortality. We focus on the effect of the income differential on health gradients through the life expectancies of the tertiary universe of descendants of the British aristocracy and the general population. We use a dataset of 127,523 offspring up to three generations deep, meticulously curated from 7,161 individual sources including 6,756 instances of direct correspondence with aristocratic families. Using this unstructured free-text data on date of birth and death and information on the general population, we develop lifetable based methodologies to provide five distinct findings. We first fail to replicate and generally rally against the so called `peerage paradox’: that lifespans between aristocrats (and their families) was equivalent to the general population until the turn of the 19th century. Secondly, the mortality transition of elites occurred around 100 years earlier than for the general public (with considerable relative improvements of approximately 30\% during the industrial revolution(s)). Thirdly, male aristocratic offspring fared less well than the general population during both the Great War and the Second World War, consistent with the existing evidence base. Fourthly, life expectancies equalized at the same time as the introduction of the National Health Service Act 1946. Finally, tentative evidence suggests that this gap has, however, begun to re-emerge since the 1980s.

4. A Grid Based Approach to Analysing Spatial Weighting Matrix Specification

Code library available here, with a link to the working paper version here (I hope to finish this paper one day, but it’s going to involve rewriting a lot of MatLab code into Python).

Abstract: We outline a grid-based approach to provide further evidence against the misconception that the results of spatial econometric models are sensitive to the exact specification of the exogenously set weighting matrix (otherwise known as the ‘biggest myth in spatial econometrics’). Our application estimates three large sets of specifications using an original dataset which contains information on the Prime Central London housing market. We show that while posterior model probabilities may indicate a strong preference for an extremely small number of models, and while the spatial autocorrelation parameter varies substantially, median direct effects remain stable across the entire permissible spatial weighting matrix space. We argue that spatial econometric models should be estimated across this entire space, as opposed to the current convention of merely estimating a cursory number of points for robustness.

5. Introducing RobustiPy: An efficient multiversal library with model selection, averaging, resampling, and out-of-sample analysis

With Daniel Valdenegro and Jiani Yan. Code library available here. Project website available here. Working paper coming soon!

Abstract The specification curve literature brings integrity into the scientific record, but further work remains to be done to adequately integrate researcher-based degrees of freedom. Here we present RobustiPy, an efficient, stable and highly reproducible tool with a user-friendly interface which allows researchers to conduct advanced multiversal analysis. It introduces several new features; it computes confidence intervals using bootstrap re-sampling; it undertakes combinatorics across possible dependent variables; it undertakes model selection and averaging, and it conducts Joint-Inference tests. It also conducts out-of-sample analysis and compares predictive models against two baselines (null and fully specified models), as well as computing feature importantance We demonstrate the utility of this library through applications to three well-published examples in the academic literature.