The only way of discovering the limits of the possible is to venture a little way past them into the impossible (Arthur C. Clarke's 2nd law)

Friday, 18 November 2011

The Singularity and Machine Ethics (Abstract)

The Singularity and Machine Ethics

Luke Muehlhauser and Louie Helm, Singularity Institute of Artificial Intelligence

Many researchers have argued that a self-improving artificial intelligence (AI) could become so vastly more powerful than humans that we would not be able to stop it from achieving its goals. If so, and if the AI’s goals differ from ours, then this could be disastrous for humans. One proposed solution is to program the AI’s goal system to want what we want before the AI self-improves beyond our capacity to control it. Unfortunately, it is difficult to specify what we want. After a brief digression concerning human intuitions about intelligence, we offer a series of "intuition pumps" in moral philosophy for our conclusion that human values are complex and difficult to specify. We then survey the evidence from the psychology of motivation, moral psychology, and neuroeconomics that supports our position. We conclude by recommending ideal preference theories of value as a promising approach for developing a machine ethics suitable for navigating the Singularity.

1 comment:

  1. What are these proposed "intuition pumps"? At first reading, I understood them to be a mechanism by which the AI's morality can evolve.

    And if that's not what you meant, I think it IS what you should mean! What a cool idea!