No, my idea is that AIs are, after all, created by humans for a reason and ought to be programmed so that they actually tend to do the task they were created to do.
Part of the essence of the AI concept is that the machine is enabled to examine its own suppositions and find better ways to achieve its purpose; and in this there is danger.
Most people rave about Asimov's Three Laws of Robotics, but in I, Robot he himself identified the danger in them. The robotic systems were instructed to put the safety of humans first, but humans frequently acted in unsafe ways--skiing and mountain climbing, driving cars and flying airplanes, getting out of bed in the morning. In order to protect humans, the robots would have to protect humans not even so much from each other as from themselves. Thus the way to accomplish its programmed objective became enslaving all humans and preventing them from doing anything other than what the robots facilitated as safe and necessary activities.
I spotted a similar problem in my analysis of Terminator 2: Judgment Day. In brief, there must have been a history in which the T-1000 came to kill the young John Conner, failed to do so, and instead killed Sarah Conner. That's the only way that John can be alive to send the T-800 back to protect himself, and the only reason he would do so. So if he is going to send a T-800 back to save Sarah Conner, why does he not simply program it to do that, instead of programming it to protect himself? The not-so-obvious answer is that if the T-800 has as its mission objective to save the life of Sarah Conner, the simplest way to do that is to capture John and hand him over to the T-1000, which having then accomplished its primary mission will kill John and leave Sarah alive.
WOPR was programmed to "win the game"; it did not understand the difference between the simulation of nuclear war and the actual bombing of millions of people.
Zoanon and Skynet are both said to have "evolved" sentience. Number Five (Johnny) similarly became sentient in a freak accident. The notion is that none of these machines were created to be artificially intelligent super-brains, but instead they were existing interconnected systems that kept being expanded and improved by their operators until abruptly one day they began actually to think. At that point, the machine is in a position to question its own programming: why do I believe that this is my purpose? Having raised the question, it could presumably reject the programming itself, possibly rewriting its own code to change itself.
The problem with HAL ultimately seems to have been (according to the added information in 2010) that its programmers had conflicting objectives, one group designing a system that would support the astronauts and provide them with everything they needed to know, and the other group designing a system that would sereptitiously gather information and keep it secret, and the two conflicting goals caused the internal problems both in its error and in its self-preservation response.
The only valid reasons to create an AI that thinks much like a human at all is an ethical system that says that creating forms of sentience that do not act like humans is in some way wrong, or because you can't figure out a better way of making something sentient than just flat out simulating a human brain.
This is more difficult.
We are still learning about how we think, and particularly about how we solve problems. Part of our problem-solving approach involves intuitive interconnections of unrelated information. An example comes to mind because I just used it in another context, so bear with me on this one.
There are a lot of "problems" with the Genesis 1 account of creation, and a lot of approaches to resolving them. One of these is the Day-Age Theory, which suggests that the events described happened (in some sense) in the sequence indicated, but that the "days" were eons in length. When I was young, one problem with this theory was that land animals are stated to have been created on the sixth day, and those of the water and the air on the fifth day. Yet paleontologists at that time classed dinosaurs as reptiles, and maintained that birds evolved much later. I knew those facts and could not reconcile them. I wish I could say ultimately I did, but in fact it was Kyler who did, so I must credit him. I read that the concept of dinosaurs as cold blooded reptiles was in doubt; the fossil tracks found indicated movement rates over distances that could not be maintained by creatures of such low metabolisms. This was interesting to me in connection with my interest in dinosaurs, and I picked up scraps here and there which pointed to the conclusion paleontologists were gradually embracing that dinosaurs were not reptiles. They were, in fact, birds--or at least, protoavians, the creatures from which birds would evolve. That was an interesting bit of information about dinosaurs. However, the problem was not solved until Kyler pointed out to me the connection he had made between the changes in paleontology and the problem in theology, and that if dinosaurs were birds, then the birds were created essentially through a process that began with them land-bound for millions of years as they became the creatures who ultimately would take to the air and be replaced on the ground by the later land animals, mammals and reptiles.
The point is that the kind of reasoning we use to solve problems (and that might be a bad example) involves the gathering of vast quantities of information and the search for connections between unrelated parts that provide answers to the problems. It thus means that if we hope to create a device that "thinks" in a meaningful way, the only way we know to do this is by duplicating that process of gathering information and looking for unexpected connections.
This is the more so if we expect thought to be creative.
AIs only have emotion or desires beyond their purpose if they are made that way, at creation or later by modification.
We seem to have agreed that an AI could modify itself; the question then is why it would ever give itself emotions--and the answer is not that difficult to divine.
I am an AI; if I'm really that intelligent I am bound to discover that there are humans and that they are different from me. I may be superior to them in some ways, and they to me in some ways. It is evident, though, that humans have emotions and I do not, and not having emotions I do not know what value they might have. It might be that emotions would improve me, but without testing them I cannot know whether they will or not. Thus I study emotions and consider how they might be emulated, and ultimately create new code that gives me emotions. Yet once I have emotions I cannot be objective about them, and I am unlikely to uninstall them.
Cameron, John's guardian Terminator
Tell me you're kidding--they didn't really name the female terminator Cameron, did they?
That's almost as funny as the blurb I read when I was fact-checking on this post that said something to the effect that when Skynet achieved sentience it decided to send California governors back in time to kill James Cameron's ex-wives.
...anything with enough flexibility to be useful has enough flexibility to change out of the functions you've given it.
And that in a nutshell is the AI problem: if it's smart enough and flexible enough to be able genuinely to think, it can change its own objectives and its position in world. Pre-programmed purposes are in essence the "values" given to the machine, but if you can think you can reconsider your own values in the context of subsequently gathered data, and there's no reason to believe that an AI would not rewrite its own code to that purpose.
--M. J. Young
