Arms races and intelligence explosions (extended abstract)

Carl Shulman, Singularity Institute of Artificial Intelligence
Stuart Armstrong, InhibOx

I. Introduction

A number of researchers (Chalmers, 2010; Good, 1965; Kurzweil, 2005; Moravec, 1999; Sandberg, 2009; Solomonoff, 1985; Vinge, 1993; see also Baum and Goertzel, 2010) have argued that sometime in this century humanity will develop Artificial Intelligence (AI) programs capable of substituting for human performance in almost every field, including AI research, and that this will greatly accelerate technological progress as AIs design their successors. That hypothetical event has been described as an “intelligence explosion” or “technological singularity.” While the term “singularity” is sometimes taken to refer to broader claims of accelerating change or limits of prediction (Yudkowsky, 2007), and has been elaborated in diverse formal models (Sandberg, 2009), we will rely on the recent overview in Chalmers (2010) for its account of the intelligence explosion.

Chalmers notes that even if it is technically feasible for humanity to produce an intelligence explosion, we may not exercise that capacity because of “motivational defeaters,” choosing to restrict, slow, and manage the development of advanced AI technologies to reduce risk. On the other hand, since a lead in AI technology may translate into overwhelming military advantage, an arms race dynamic may give states incentives to pursue even very dangerous research in hopes of attaining a leading position.