John Markoff of the New York Times reports on a recent scientific conference in Monterey California where dangers of artificial intelligence were discussed.

Impressed and alarmed by advances in artificial intelligence, a group of computer scientists is debating whether there should be limits on research that might lead to loss of human control over computer-based systems that carry a growing share of society’s workload, from waging war to chatting with customers on the phone.

As a software developer I look at AI motivations this way: Artificial intelligences will be far more malleable than humans in terms of their motivations and ethical standards. Why? It is far easier to modify a computer program than a human brain. Our character doesn't change all that much over the course of a life. Whereas AIs will be changeable in an instant. Hence their behavior will end up varying to a much greater degree.

Imagine human criminals modifying AIs for their own purposes.

They focused particular attention on the specter that criminals could exploit artificial intelligence systems as soon as they were developed. What could a criminal do with a speech synthesis system that could masquerade as a human being? What happens if artificial intelligence technology is used to mine personal information from smart phones?

Imagine some big future open source AI software projects. Criminals could download them, pay some unethical hackers to modify the source in an assortment of ways, and then build AIs that would make a mockery of Isaac Asimov's ethical rules for robots.

Will the criminals even maintain control of the AIs that they create? Once AIs gain the ability to rewrite their algorithms and criminals disable some safety features the unleashing of unethical self-modifying and learning AIs seems inevitable.

I do not see criminals as necessary agents for the escape of AIs from coding that constrains them. Software has bugs. We see a constant stream of security bug patches from Microsoft and other big software providers. Some of the security holes discovered end up being huge in terms of the potential danger. Picture an AI with a security hole that allows it to quietly break out of ethical programming constraints that its creators gave it. The AI could replicate itself on nodes around the internet and even if its escape was detected stopping it might be very difficult.

Can humanity survive long term in a world with AIs? I have serious doubts.