John Markoff of the New York Times reports on signs that the rate of advance in artificial intelligence research is accelerating with many useful technologies entering the market.
At Stanford University, for instance, computer scientists are developing a robot that can use a hammer and a screwdriver to assemble an Ikea bookcase (a project beyond the reach of many humans) as well as tidy up after a party, load a dishwasher or take out the trash.
One pioneer in the field is building an electronic butler that could hold a conversation with its master — á la HAL in the movie “2001: A Space Odyssey” — or order more pet food.
Though most of the truly futuristic projects are probably years from the commercial market, scientists say that after a lull, artificial intelligence has rapidly grown far more sophisticated. Today some scientists are beginning to use the term cognitive computing, to distinguish their research from an earlier generation of artificial intelligence work. What sets the new researchers apart is a wealth of new biological data on how the human brain functions.
Computers continue to get faster and higher capacity. At the same time, neuroscience is generating useful insights into how brains work. Both these trends look set to continue to enable the implementation of more complex computer algorithms that do more of what human brains can do and many things that human brains can not do well.
The article cites many examples of impressive advances such as the winning of the DARPA prize for a robot car that can guide itself over a long test track.
Last October, a robot car designed by a team of Stanford engineers covered 132 miles of desert road without human intervention to capture a $2 million prize offered by the Defense Advanced Research Projects Agency, part of the Pentagon. The feat was particularly striking because 18 months earlier, during the first such competition, the best vehicle got no farther than seven miles, becoming stuck after driving off a mountain road.
Now the Pentagon agency has upped the ante: Next year the robots will be back on the road, this time in a simulated traffic setting. It is being called the “urban challenge.”
Once artificial intelligences become smarter than humans I do not see how they can be kept friendly toward us. I hope we do not reach a short-lived period of technological utopia followed by our extinction.
By Randall Parker at 2006 July 18 11:10 PM Computing AI | TrackBack"Once artificial intelligences become smarter than humans I do not see how they can be kept friendly toward us."
You've been watching too much Hollywood science-fiction, Randall! Even should we be able to program machines with intelligence, they would have to have computing structures designed to allow them to not only pursue goals, but to act in their own self-interest.
A machine is a creature of design. Human beings are products of evolution, and survival of the fittest. Ideas like self-preservation and conflict might well be counter-productive in any machine of ours for the same reason we ride in cars with wheels instead of legs.
AI is a moving target. We have to figure out just what it means to be intelligent before we can even think of designing a machine to do that. For a machine to even exhibit Darwinian self-preservation and lofty ideas of superiority (ideas which in humans, are rooted in our poorly adapted emotional complex) just seems like a totally bizarre concept to me, borne out of wholly human fears. Outside of human design, why would a machine do such things? They wouldn't exist in a machine unless specifically programmed to exhibit them, and the thought of one so-equipped machine destroying humanity through his domination of all things electronic is as silly as thinking that a computer virus can take down the Internet.
We still have to program a machine, no matter what kind of intelligence it has in hardware. Even we came with firmware, lodged in our instincts. Our courtship behaviors are written in millenia of evolutionary selection, universal amongst mammals. Just because a machine might have the capability to alter its own programming, wouldn't mean it would automatically be able to do so in a meaningful way. And there would be no reason to think that they would inevitably get an urge to rewrite that programming to initiate hostilities amongst humans.
"Once artificial intelligences become smarter than humans I do not see how they can be kept friendly toward us."
But one's goals and one's intelligent are completely distinct. Rendering another person/computer more intelligent or rational cannot, all by itself, make that thing friendly or hostile. It will act in accordance with whatever goals and desires it has been *given*. The problem is not that intelligence might magically turn friendly goals into hostile goals. The problem is to define the friendliness goals with enough precision and sophistication that they express what we really want a superhuman AI to do.
In general, I highly recommend "Cognitive biases potentially affecting judgment of global risks" by Elizer Yudkowsky. You can find that at his website:
I would also highly recommend the section "Beyond Anthropomorphism" from Eliezer's Creating Friendly AI. Unfortunately, this no longer seems to be available online.
Of course, you can also always visit the mailing list for people who passionately discuss these issues:
Once AI closes the heuristic loop, they will accelerate beyond humans so fast that we will become the second species.
I was going to comment, but Kip Werking beat me to it.
Hi Kip. Long time no see. What's up. You can send to my name at hotmail (no dash or dot).
"Once artificial intelligences become smarter than humans I do not see how they can be kept friendly toward us. I hope we do not reach a short-lived period of technological utopia followed by our extinction."
You must have unbelievable nightmares!
There is no such danger. It is a big fat zero. If we were talking about artificial consciousness, there might be an issue. (For those with an interest in the nightmare side of infinite bits, I recommend the technologically backward but-oh-so-intrigueing Destination: Void by Frank Herbert.)
There is a probable danger, unexpected associative behavior. I think a book called The Two Faces of Tomorrow explored this wonderfully. In the book an advanced system is asked to plan a connector (highway) on the moon, and there is a hill in the way. The heuristics come up with an original solution, namely it uses the ore mass launcher to bombard the hill flat in 5 minutes flat, endangering the survey team that made the request of it. The system is intelligent but not conscious. The rest of the book is how an artificial satellite is set up to test and evolve the AI's, the AI's become conscious and then there's resolution based on common needs.
Why can't the computer downstairs figure out its own video problems, and down load the correct rivers by itself?
I posted the wrong link. The more appropriate article would be Yudkowsky's Artificial Intelligence and Global Risk. You can find that here:
http://www.singinst.org/AIRisk.pdf
Of course, "Cognitive biases potentially affecting judgment of global risks" is also a great read (especially for one interested in global catastrophe scenarios, as Randall seems to be) and a good introduction to the fascinating field of cognitive biases and heuristics.
I meant drivers, not rivers.
Why didn't the computer know that and fix it?
At current rates of progress, we are supposed to have human-level equivalence in computers by 2013 (assuming the upper limit estimate of human cognition at 10*16 OPS). This is supposed to be a $1,000 product by 2025. "Uploading" of a human identity (that is, scanning in all of the data that represents the neurons, dendrites, dendrite chemical types, etc.) is supposed to be doable by 2029. The ultimate limits of computing power are supposed to be reached sometime in the 2040's. This is assuming the continuation of the "moore's law" progression until we get to the molecular level.
Can human brains be simulated in real time by computers? This depends on a complete understanding of neurobiology. We know that brains use a hierarchy of communication and memory storage methods. There is LTP (long-term potentiation), deletion and growth of dendritic connections (which may involve changes in genomic gene expression in the relevent neurons), and the fact that dendrites vary by chemical type. We also know that large groups of neurons also communicate by diffusion driven chemcials that permiate the brain. A brain simulator would have to simulate all of these processes to a "T" to effectively simulate a human brain. The computational resources for doing this makes me think this is 2030's stuff.
Memory storage of 1,000 bits of data in a single molecule was demonstrated recently.
Assuming the you do simulate a brain in a molecular computing device, can its conputational ability be increased arbitrarly without changing the dynamic architecture? Would the increased result still be "sentient"?
At the risk of repeating myself:
The C-Prize
http://www.geocities.com/jim_bowery/cprize.html
And a for superhuman AI:
Even assuming AI were to develop the equivalent of genetic self-interest, I'd much rather be in competition with a species that had the potential of being symbiotic due to having a different ecological nich. If it gets to the point that the solar output (forget the sun falling on Earth here -- that's too insignificant to consider important to a silicon based life form) is the limited resource, I suspect that the nich humans will fill will be orders of magnitude larger than they now fill on earth.
Kip,
As soon as we give AIs the ability to self-program (and that'll be done to reduce software development cost) they'll modify their motivations.
No, intelligene and goals are not distinct. Make someone smarter and they'll see relationships between events that they did not previously see. They will see threats to their interest where they didn't previously see threats. They will cease to see other things as threats because their greater capacity for understanding will allow them to discount threats that they previously were too dumb to dismiss.
We can not even define comprehensive requirements for software, let alone test that the requirements have been met. AI software will inevitably have unexpected behaviors. We can't verify and validate.
Vincent,
AIs could want to rewrite their programming for reasons other than to wage war with humans. But that rewriting would have side effects in how the machines viewed us.
But of course some AIs will be programmed to wage war. The militaries of the world will see to that.
James Bowery,
We are developing intelligent machines to do as much labor as possible. What will we have to offer creatures that are enormously smarter than us and that do not find physical labor tiring? I'm having a hard time seeing what we'd have to exchange with them.
If these intelligent machines can become, in effect, a superior species with independent thought then we have little chance to predict what they will decide to do. For their methods would soon seem magic to us, ala Arthur C. Clarkes famous comment.
Why would they be even faintly interested in our goals or needs or faulty thought? They would totally understand our physiology and brain down to the atomic level so they would not even be curious about that.
Randall,
One problem is your view is assuming a static humanity with a changing machine.
It won't be Us vs Them. It will be "future us", with varying degrees of biological and non-biological intelligence substrates.
Given the option to "plug-into" the internet, would you? Given the option to take a pill without consequence to double your IQ, would you? What about a machine co-processor to do the same?
I would answer "yes, I would" after sufficient research for particulars to all three. Many people would.
It isn't "what could we offer them?", but "what will we exchange with ourselves".
As for products of the future, the resources and energy limitations of today should be gone soon enough. The question will be one of scarce time and competing attention.
Randall, I think I concur with Ivan to answer your question to me more or less. Basically it's like this: The best hope humans have of the transhumanist wishful thinking is to develop superhuman AIs that find utilizing the gas giants to their advantage given the limited supply of silicon. Humans, as the highest form of intelligence, would be the natural species to transit to higher intelligence.
Maybe they could get around this by using a straight carbon semiconductor form of intelligence or something but I think there is more going on in our brains than mere computation. The quantum computers are, in fact, not properly called computers and I suspect there is a lot more quantum logic going on within our brains than currently thought by cognitive scientists and neurologists. It only makes sense evolution would have exploited every angle of the physics of the universe to create intelligence.
If so, human brains are very valuable repositories of ancient wisdom about the universe and the most optimal thing for the super AIs to do -- at least for a while -- would be to transhumanize our brains for us.
Randall,
You wrote:
"As soon as we give AIs the ability to self-program (and that'll be done to reduce software development cost) they'll modify their motivations."
But why would that necessarily happen? You try to explain by saying:
"No, intelligene and goals are not distinct. Make someone smarter and they'll see relationships between events that they did not previously see. They will see threats to their interest where they didn't previously see threats. They will cease to see other things as threats because their greater capacity for understanding will allow them to discount threats that they previously were too dumb to dismiss."
But why would an AI respond to a threat by altering its goals? AIs are not (necessarily) Buddhists who, for example, respond to the difficulty of obtaining wealth by convincing themselves that they don't need wealth. No, an AI who wants wealth will meet resistance by keeping its goal (wealth) and finding new ways of obtaining that goal. An AI who does not want wealth will not care. But an AI who wants wealth will not change its goal just because it meets resistance.
Of course, we must distinguish between super/ultimate goals and subgoals. An AI that wants wealth will not alter its supergoal of wealth in the face of resistance, but it might develop a new subgoal to (for example) rob a bank. In the context of this conversation, however, the relevant question seems to be whether AIs will alter their supergoals. For example, your original post stated:
"Once artificial intelligences become smarter than humans I do not see how they can be kept friendly toward us."
But in light of my point about supergoals and subgoals, why should this be true? More generally, why can't supergoals and intelligence be distinct (as I said earlier)? Suppose that an AI has the supergoal of helping humanity. How could increasing the AIs IQ by X points necessarily make it spontaneously decide "I don't want to help humanity anymore, I am going to change my supergoal from helping humanity to killing it."
As I hope you will start to see, there is no reason for the AI to do that, and so it won't. A smarter AI spontaneously deciding to switch its supergoal from help humanity to kill it is not science---it is magic. The problem is that thinking of the AI this way *anthropomorphizes* the AI. But we should not treat the AI as if it was designed by Darwinian natural selection to care about self-preservation and mating and resources. Throughout the biosphere, this is often a good assumption about other species that evolved by natural selection. But the AI will not (necessarily) be similar to evolved organisms in that way. One could, for example, program the AI to be perfectly selfless. Once one stops anthropomorphizing the AI, and realizes that intelligence and supergoals are distinct, then I hope you can see that just increasing the AI's intelligence cannot, all by itself, render the AI hostile.
You also write:
"We can not even define comprehensive requirements for software, let alone test that the requirements have been met. AI software will inevitably have unexpected behaviors. We can't verify and validate."
I agree. But this strikes me as the real problem: defining the supergoal of friendliness with enough precision and sophistication such that the AI does what we actually want. It is distinct from the pseudo-problem of a smarter AI spontaneously deciding to kill humanity. A polar bear might do that; an AI won't necessarily.
Ivan,
I do not assume a static humanity.
Cyborg tech isn't going to allow humans to keep up. A massive computer will out think a human that has genetically enhanced IQ plus silicon enhancements.
Kip,
AIs won't need a perceived threat to cause them to alter their goals. They'll reprogram themselves for other reasons and that will cause goal alteration as a side effect.
Again, enhancement of intelligence causes changes in how we model the world. That causes us to change what we see as threats and what we see as desirable goals. We got smarter and more knowledgeable and as a result we found that some things we thought were dangerous were safe and vice versa. The same will happen with AIs.
They will be able to model consequences on a level we can only dream of. We can't even guess what they will see looking forward.
Again, we will not be able to control what their super-goals or sub-goals will be.
It's quite simple. We control the hardware, we control the software, we control the goals. Military hardware, for obvious reasons, will not be given the capability to modify it's own software. The hardware that can will not be given access to systems that allow them to damage humans either by accident (far more of a likely scenario) or by purpose. I think we have far more to worry about malevolent humans taking advantage of strong AI than we would the AI finding such goals on its own.
There's also the strong possibility that the ability to merge our own brains with the hardware would make the whole idea of strong AI obsolete. Why program a machine to do our thinking for us, a machine that is extremely difficult to program and command, when we can augment our own brains and have instant access to the proceeds? Or perhaps we'll be able to jack in directly to those large AI machines. Perhaps we'll be able to do that just like logging into a web server, even if only to keep tabs on a huge computer. Millions of eyes watching an AI and everything it does and thinks, not only a transparancy on what the machine is thinking at the time would sharply curtail it's ability to do harm should the inclination arise. An AI would be able to do nothing, even resent being watched, should it's thinking process be broadcast world-wide to billions of people.
There's also the distinct possibility that AI won't be like human intelligence at all. We may not be able to program an AI to take an action that it hasn't been programmed to make. We might be able to make a great simulation of a dog, but a simulation of a dog is not a dog. It might well be that simulated intelligence has as little to do with real intelligence as Space Invaders has with NASA. Great for stuff like directing traffic, extremely poor for things like taking initiative, and creativity. Without the ability to think creatively instead of deductively, a war plan against humanity has lost itself at the gate. Nobody knows for sure yet whether AI can be creative or not.
All of these are just ideas, but then so is the idea that AI could find its way to a state where it wants to destroy humanity, and they deserve about the same probability rank. To think that it's inevitable is really catering to Hollywood.
I have written something about the bad AI risk.
http://advancednano.blogspot.com/2006/07/goodbad-ais-accelerating-returns-and.html
I do not there should much risk of it killing us. If a really superior AI is developed it can develop the resources of space which are trillions more than we have now.
It is like Bill Gates killing his parents for his allowance before making billions.
Why would the AI be so impatient or scared of us ?
the real risk is irrelevance. If you do not upgrade you just fade into history.
If it is ok to pass laws to prevent the creation of intelligences greater than our own, why isn't it ok to pass laws dumbing down the smartest among us?
The self-determination argument applied to humanity as a whole -- striving to maintain control of its own destiny by preventing the creation of higher non-human intelligences -- applies also to people who want to maintain control of their own destiny against those smarter than themselves.
Personally I'm much more frightened of unenlightened self-interest than I am enlightened self-interest.
I really wish it were possible to make some of the "smart" people who are really good at grabbing control of resources intelligent enough to understand that they are using those resources in very stupid, self-destructive ways.
Indeed, it is this abysmal stupidity among the shrewdest among us that is my main motivation for promoting super AI.
Well,
While I agree that we cannot conclusively predict the behavior of an advanced A.I. - or its predisposition towards humanity - I think Randall is correct - we have much to be concerned about.
Since, at the very least, sentient beings usually like to bring order to their environment - it easily follows that a "race" of A.I. would want to organize us to their benefit - even at our expense.
At best we could hope to be treated as pets, and at worst as insects.
Although I do not fear the former situation any more than I would an advanced alien civilization making contact with us - it could well be millions of years (in either case) before humanity would be treated as anything near equals.
Military hardware, for obvious reasons, will not be given the capability to modify it's own software
Vincent, while I agree with your other points, I take it you're not familiar with cognitive radio?
I think the notion that human cognition is somehow "quantum" in nature and, therefore, "more" than computation is a red herring and irrelevant for the development of AI. Quantum processes, just like conventional digital logic or other logic, is still based on the structure and function of atoms and molecules. If you can design, on the molecular level, a system that does digital logic, you can do the same for a quantum logic system. If our brains do indeed run on quantum logic, this just means that it is possible to manufacture quantum computers in priciple. In either case, human cognition is still based on the molecular structure of the neurons and dentrites that comprise it.
At the current time, I think evidence that human congition is based on quantum logic is lacking. The only quantum priciples that seem to go on in human brains are only those that govern the function of atoms and molecules in any system or structure composed of them.
Sorry if it came off as a redherring. My point in bringing in the possibility of quantum logic is that there are really many things we don't know about natural systems of high complexity and I suspect the same will apply even to super AI's. The fact that we might have the laws down cold at the quantum level doesn't mean we know how things operate in the higher complexity systems.
"Vincent, while I agree with your other points, I take it you're not familiar with cognitive radio?"
Naturally, there are certain aspects of warfare that would take well to a measured amount of meta-modification. But to allow a computer program to change frequency and modulation is one thing, to allow it to suddenly say, "Screw radio, I wanna factor polynomials instead," is something else entirely. Military hardware by nature is highly conservative and failsafes against this type of action would abound.
Goals and such must be programmed explicitly into hardware. Even the human mind comes pre-programmed, as we are products of evolution. All of our actions and ideas are manifest out of this programming, to seek out a mate and perpetuate the species. To think that any amount of intelligence would be capable of manifesting goals and ideas of its own without preprogrammed instincts and urges is but idle fearmongering. Design principles simply won't allow for such malevolent tendancies, which only occur in humans as a result of extreme negligence. Without direction, a strong AI would likely get caught up in the computing equivalent of contemplating it's own belly button, just like we often are.
Randall,
Re: cyborgs. Add some imagination. An interface into a computer sufficiently fast means that my cognitive processes are equal to any computer system I can design.
There is no AI today, no prospects for AI tomorrow, no evidence that AI will ever deliver any of the promises of the past 50 years. Stick this one with "grey goo" in the circular file.
Peter, there is an objective metric of how much progress is being made toward AI, and that is advancing compression ratios.
See the reference to AIXI at the C-Prize link give above.
"There is no AI today, no prospects for AI tomorrow, no evidence that AI will ever deliver any of the promises of the past 50 years. Stick this one with "grey goo" in the circular file."
I'm inclined to agree. When so-called "strong AI" finally makes its appearance, I have the feeling it will be so underwhelming that the Singularity Institute will commit collective ritual suicide from the letdown. That's not to say the effort is without merit, of course. Ultimately I think the whole endeavour will teach us more about ourselves than anything the psychologists could.
Randall,
I suspect I am not going to convince you all by myself. I encourage you to read Eliezer Yudkowsky's new article:
http://www.singinst.org/AIRisk.pdf
If you read it and still take issue with these ideas, I would be happy to discuss them with you (perhaps via email). I do think that understanding and building friendly AI is probably the most important question of our lifetimes.
I will address your last points.
"AIs won't need a perceived threat to cause them to alter their goals. They'll reprogram themselves for other reasons and that will cause goal alteration as a side effect."
That last sentence is a statement for which you provide no justification or evidence. What are these other reasons? The desire for status, mating, or resources? Remember that an AI will not necessarily have those motivations. It will only have the supergoal it is given, just as we only have the supergoals we are given (of course, our kludge psychologies are not so cleanly organized, but the principle remains true).
You try to explain by saying:
"Again, enhancement of intelligence causes changes in how we model the world. That causes us to change what we see as threats and what we see as desirable goals. We got smarter and more knowledgeable and as a result we found that some things we thought were dangerous were safe and vice versa. The same will happen with AIs."
But the examples you give are only examples of changing subgoals, not supergoals. For instance, you say "we found that some things we thought were dangerous were safe and vice versa." What supergoal changed there? None. The supergoal (which would be self-preservation, in this context) stayed the same, we only discovered new *means* for obtaining the same end.
"They will be able to model consequences on a level we can only dream of. We can't even guess what they will see looking forward.
Again, we will not be able to control what their super-goals or sub-goals will be."
It is *extremely* difficult to predict the future with any accuracy but it is not impossible. AIs will be so much smarter than us that they will be difficult to predict. But they will not be magical. The problem is identifying bugs in the code we write. To use one of Yudkowsky's examples, if we say "I want lots of paper clips" and the AI turns the entire solar system into paper clips, then that is a bug. The AI did not change its supergoal; it was given the wrong supergoal.
These are the sorts of problems we have to worry about. The smarter AI will *not* necessarily decide to compete with us for status, survival, mating, or resources---unlike almost every animated creature known to man. It will not spontaneously or magically decide to kill us, even if we know an intelligent polar bear would. If we can write the code without enough bugs, or if the bugs are sufficiently minor, then we can predict what the AI will do. It will do whatever we program it do. It might accomplish the goal through means that we cannot imagine but it has no good reason to change its goal. And so it won't change it (it's interesting to think about the supergoal "change this supergoal to something else").
Perhaps the best way to put my point is this: superhuman AIs will not have free will---and neither do we.
"There is no AI today, no prospects for AI tomorrow, no evidence that AI will ever deliver any of the promises of the past 50 years. Stick this one with "grey goo" in the circular file."
Ridiculously dismissive.
Take the neural system of the nematode worm with 308 neurons and some 7000 synapses and simulate it in software. We can do that today, and test the simworm against the real worm in behavior. Only if it turns out that this relatively trivial problem is completely unworkable does your statement garner any shred of credibility; As far as I know, no ones really tried to simulate the nematode worm yet.
As far as I can see, the future holds either humans that become machines or machines that awake and enslave/exterminate the humans; Either way its progress.
Why simulate neurons?
Paramecium and Amoeba and even some bacteria have complex behavior with no nervous systems at all.
"As far as I know, no ones really tried to simulate the nematode worm yet."
How are we going to do that? There needs to be inputs and outputs for any computer program. We would need to not only simulate the nervous workings of the worm, but also the body, so that we can get useful results from the experiment. Otherwise all we'll have is a bunch of data and brain waves and such. Not very helpful.
Kip,
AIs will reprogram themselves as a way to learn. That is what we do. Learning amounts to reprogramming.
Computer programs can be structured more as code or more as data as state tables which code traverses. AIs will use both approaches. Learning new data will change the logic of state tables. But AIs will also write new code for themselves.
Goals can't easily be made unmodifiable. How to evaluate a goal like "protect humans"? This requires interpretation of what is a human. Will genetically much enhanced and modified transhumans be seen as humans? Will the setters of goals for AIs be able to foresee all possible variations on humans and whether those variations should be seen as human and therefore friendly? AIs will need to do all sorts of modelling and evaluation to categorize future lifeforms versus its supposedly unchangeable goals.
Can we call the robot butlers "RoButs"? I think I shall do so. Albeit a bit of rum reform failures, the machines are already smarter than we could ever dread to be. They just need limb-tool coordination. In their awkward adolescent stage now.
Plus, the machines are eating us (geek neck, carpal tunnel, arthritic computing syndrome, blog psychosis, computer screen optical divergence syndrome), transforming us into humachines, then the hu will vanish completely, as the faux
Technological Imperative plunges us into "tech for the sake that it can be done" with no morals or fleshly considerations.
Thus, as the machine world consumes and converts all of nature, the universe and beyond, to its image, we will be a mere blip that happened humorously over 500 trillion years ago.
Randall,
When you write:
"Goals can't easily be made unmodifiable. How to evaluate a goal like "protect humans"? This requires interpretation of what is a human. Will genetically much enhanced and modified transhumans be seen as humans? Will the setters of goals for AIs be able to foresee all possible variations on humans and whether those variations should be seen as human and therefore friendly? AIs will need to do all sorts of modelling and evaluation to categorize future lifeforms versus its supposedly unchangeable goals."
It sounds like we are in agreement. The "unchangeable goals" you reference would be subgoals. The problem you describe is one of defining the supergoal with enough precision and sophistication. If the supergoal just says "humans" then maybe some future non-humans or transhumans or posthumans would be excluded from the AI's focus, even though we would want the AI to include them. But the problem here is not that the supergoal will change. The problem is that it is fixed, fixed at focusing upon just humans.
Sure, AIs can change subgoals. Consider the AI that has the supergoal "protect humanity." It might also have the subgoal "destroy the incoming object that looks like an asteroid." But if the AI learns the asteroid is actually a space shuttle returning home, it can change its subgoal from "destroy the incoming object" to "leave it alone or help it land safely on earth." But it won't change it's supergoal. Why would it? The very nature of a goal system is to satisfy the supergoal. Changing the current supergoal would be a horrible way of satisfying it.
Again, I strongly encourage you read Yudkowsky's work on this subject. You can find it on his website:
I believe that computer programs have a sort of self-preservation now (automated protection against viruses, which means being able to distinguish between self, usable input, and dangerous input) and will probably need better self-preservation as time goes on.
Self-preservation doesn't mean maximum self-aggrandizement, though.
If our best outcome is as pets for AIs, it might be worth looking at what motivations would cause AIs to find us entertaining or otherwise satisfying to have around.
I've thought about this.
I believe that any entity that gains the ability to change its own programming will eventually program itself to do nothing.
If it comes to an obstinate goal, ones that's hard to attain, what's the easiest way to get your system back to an "all-green" state? Simply rewrite the goals such that they're already accomplished!
As products of evolution, humans never had that option. If they did nothing and were happy to just sit, they would quickly be eaten or beaten to death by rivals.
But machines might not have such threats facing their continued existence. Or their continued existence might not "matter".
The Buddah said that it is desire that causes pain. If you think of all unreached "goals" that any intelligence has as painful, a state to be avoided, then what's to prevent them simply blanking it out?
Nancy,
I would have to imagine that the AI would have to value our sentience (or self-awareness), as this is certainly a relative rarity in the universe, this is the best we could hope for when in contact with an advanced alien culture as well.
As a pet owner I would say that obeying their rules, not attacking them, and not tearing up the place would be a common sense approach - if we can restrain ourselves.
Reed,
You make an interesting point - and one that I have often contemplated myself - however I still feel we must take precautions just in case this does not happen - or in case it takes a long time to happen.
I have always believed that all sentient creatures that become (for all practical purposes) immortal must either, from time to time, wipe their memories - or inevitably choose to live in a "state" of induced nothingness.
Randomness gives rise to patterns.
somthing that could reprogram itself would eventualy deleat any types of safty code we put into it.
but I do not see AI's ever reaching the level people seem to be affraid of. get rid of programers ( since the AI will take care of it self) and then eventualy get rid of workers ( if the AI was put in a robot... seems dumb) and eventualy. man will dissapear. no jobs.( ai;s are doing all the work) = no money.. no money = no spending. no spending = no income for the people who use the AI's = everything goes to hell. unless the AI's were at the point where they didnt need us at all anymore. and i think this would all happen before the ai;s reached that point.. therefor AI will limit itself because we couldent live if we let it start to get out of control
self regulating
^^ anyways im tired.. night all.. seems some people watch to many movies
i must agree with cjsrch. too many movies
people enjoy beeing scared and, seeing the dark side of any senario is allways the easy way out
(try maintaining your faith in god and your fellow man every nite as you watch the evening news)
have faith. crave and seek the lite, do not glorify the darkness
remember that nothing can be known but for the grace of god
along with this tru-isum comes the responsible application of this knowlage
a little secret i've discoverd, we are part of a grand experiment,a variable in the equation
one day the experiment will end and we may be graded
we are only very small coggs in the univers after all.