The only way of discovering the limits of the possible is to venture a little way past them into the impossible (Arthur C. Clarke's 2nd law)

Monday, 11 October 2010

(Joshua Fox) Informal bibliography on the Intelligence Explosion and Friendly AI

This is an informal bibliography on the Intelligence Explosion and Friendly AI.

It does not cover:

* material related to other definitions of the Singularity including technological acceleration and the superintelligence event horizon (see Yudkowsky, "Three Major Schools"),
* predictive futurism,
* existential risk,
* and the many relevant fields such as decision theory, cognitive neuroscience, and ethics.

There is an emphasis on academic work. However, the field is new and small, and much high-quality writing appears only in informal forums. Some of the best of this is listed below.

I'd like to make the bibliography comprehensive within its narrow field, so please add more items in the comments to help in future revisions


  1. Also: I.J. Good, "Speculations Concerning the First Ultraintelligent Machine," Advances in Computers, 6, 1965. Available at

  2. Also:

    Kaj Sotala: "Evolved altruism, ethical complexity, anthropomorphic trust: three factors misleading estimates of the safety of artificial general intelligence". Proceedings of the 7th European Conference on Computing and Philosophy (ECAP 2009). 2009.

    Kaj Sotala: From Mostly Harmless to Civilization-Threatening: Pathways to Dangerous Artificial Intelligences. Proceedings of the VIII European Conference on Computing and Philosophy. Edited by Klaus Mainzer (Munich: Verlag Dr. Hut, 2010). Pp. 443-450. 2010.

  3. Hall, JS & Waser, M. 2009. Ethics for Recursively Self-Improving Machines. Presentation at the 2nd AGI Conference.

  4. Waser, M. 2009. A Safe Ethical System for Intelligent Machines. In AAAI Technical Report FS-09-01. Menlo Park, CA: AAAI Press.

  5. Waser, M. 2010. Deriving a Safe Ethical Architecture for Intelligent Machines. In Proceedings of the VIII European Conference on Computing and Philosophy. (Powerpoint)

  6. Waser, M. 2010. A Game-Theoretically Optimal Basis for Safe and Ethical Intelligence. In Biologically Inspired Cognitive Architectures 2010: Proceedings of the First Annual Meeting of the BICA Society. (Powerpoint)

  7. Thanks for compiling this! A few housekeeping notes: the link to Shane Legg's 'Friendly AI is Bunk' is broken, and the link to Joshua Fox and Carl Shulman's 'Superintelligence Does Not Imply Benevolence' is split in two, the first half of which links to a nonsense URL.

  8. For those interested, I've developed a more thorough bibliography of Friendly AI, which I'll keep updated, here: