Artificial intelligence


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Artificial intelligence

  1. 1. Artiﬁcial Intelligence and Consciousness Drew McDermott Yale University This paper is essentially the same as that published as chapter 6 (pages 117–150) of Philip David Zelazo, Morris Moscovitch, and Evan Thompson (eds.) 2007 The Cambridge Handbook of Consciousness. Cambridge University PressAbstract: Consciousness is only marginally relevant to artiﬁcial intelligence (AI), becauseto most researchers in the ﬁeld other problems seem more pressing. However, there have beenproposals for how consciousness would be accounted for in a complete computational theory ofthe mind, from theorists such as Dennett, Hofstadter, McCarthy, McDermott, Minsky, Perlis,Sloman, and Smith. One can extract from these speculations a sketch of a theoretical synthesis,according to which consciousness is the property a system has by virtue of modeling itself ashaving sensations and making free decisions. Critics such as Harnad and Searle have notsucceeded in demolishing a priori this or any other computational theory, but no such theorycan be veriﬁed or refuted until and unless AI is successful in ﬁnding computational solutions ofdiﬃcult problems such as vision, language, and locomotion.1 IntroductionComputationalism is the theory that the human brain is essentially a computer, although pre-sumably not a stored-program, digital computer, like the kind Intel makes. Artiﬁcial intelligence(AI) is a ﬁeld of computer science that explores computational models of problem solving, wherethe problems to be solved are of the complexity of problems solved by human beings. An AIresearcher need not be a computationalist, because they1 might believe that computers can dothings brains do noncomputationally. Most AI researchers are computationalists to some ex-tent, even if they think digital computers and brains-as-computers compute things in diﬀerentways. When it comes to the problem of phenomenal consciousness, however, the AI researcherswho care about the problem and believe that AI can solve it are a tiny minority, as we will see.Nonetheless, because I count myself in that minority, I will do my best to survey the work ofmy fellows and defend a version of the theory that I think represents that work fairly well. Perhaps calling computationalism a “theory” is not exactly right here. One might prefer“working hypothesis,” “assumption,” or “dogma.” The evidence for computationalism is notoverwhelming, and some even believe it has been refuted, by a priori arguments or empiri-cal evidence. But, in some form or other, the computationalist hypothesis underlies modernresearch in cognitive psychology, linguistics, and some kinds of neuroscience. That is, there 1 To avoid sexist pronouns, I will sometimes use third-person-plural pronouns to refer to a generic person. 1
  2. 2. wouldn’t be much point in considering formal or computational models of mind if it turned outthat most of what the brain does is not computation at all, but, say, some quantum-mechanicalmanipulation (Penrose, 1989). Computationalism has proven to be a fertile working hypothesis,although those who reject it typically think of the fertility as similar to that of fungi, or of podpeople from outer space. Some computationalist researchers believe that the brain is nothing more than a computer.Many others are more cautious, and distinguish between modules that are quite likely to bepurely computational (e.g., the vision system), and others that are less likely, such as themodules, or principles of brain organization, that are responsible for creativity, or romanticlove. There’s no need, in their view, to require that absolutely everything be explained in termsof computation. The brain could do some things computationally and other things by diﬀerentmeans, but if the parts or aspects of the brain that are responsible for these various tasks aremore or less decoupled, we could gain signiﬁcant insight into the pieces that computationalmodels are good for, and leave the other pieces to some other disciplines such as philosophyand theology.2 Perhaps the aspect of the brain that is most likely to be exempt from the computationalisthypothesis is its ability to produce consciousness, that is, to experience things. There aremany diﬀerent meanings of the word “conscious,” but I am talking here about the “HardProblem” (Chalmers, 1996), the problem of explaining how it is that a physical system canhave vivid experiences with seemingly intrinsic “qualities,” such as the redness of a tomato, orthe spiciness of a taco. These qualities usually go by their Latin name, qualia. We all knowwhat we’re talking about when we talk about sensations, but they are notoriously undeﬁnable.We all learn to attach a label such as “spicy” to certain tastes, but we really have no ideawhether the sensation of spiciness to me is the same as the sensation of spiciness to you. Perhaps tacos produce my “sourness” in you, and lemons produce my “spiciness” in you.3We would never know, because you have learned to associate the label “sour” with the qualeof the experience you have when you eat lemons, which just happens to be very similar to thequale of the experience I have when I eat tacos. We can’t just tell each other what these qualiaare like; the best we can do is talk about comparisons. But we agree on questions such as, Dotacos taste more like Szechuan chicken or more like lemons? I focus on this problem because other aspects of consciousness raise no special problem forcomputationalism, as opposed to cognitive science generally. The purpose of consciousness,from an evolutionary perspective, is often held to have something to do with allocation andorganization of scarce cognitive resources. For a mental entity to be conscious is for it to be 2 I would be tempted to say there is a spectrum from “weak” to “strong” computationalism to reﬂect thediﬀerent stances on these issues, but the terms “weak” and “strong” have been used by John Searle (1980) in aquite diﬀerent way. See section 5.2. 3 I am taking this possibility seriously for now because everyone will recognize the issue and its relationshipto the nature of qualia. But I follow Sloman & Chrisley (2003) in believing that cross-personal comparison ofqualia makes no sense. See section 3.4 and (McDermott, 2001), ch. 4. 2
  3. 3. held in some globally accessible area (Baars, 1988, 1997). AI has made contributions to thisidea, in the form of speciﬁc ideas about how this global access works, going under names such asthe “blackboard model” (Hayes-Roth, 1985), or “agenda-based control” (Currie & Tate, 1991).One can evaluate these proposals by measuring how well they work, or how well they matchhuman behavior. But there doesn’t seem to be any philosophical problem associated with them. For phenomenal consciousness, the situation is very diﬀerent. Computationalism seemsto have nothing to say about it, simply because computers don’t have experiences. I canbuild an elaborate digital climate-control system for my house, which keeps its occupants ata comfortable temperature, but the climate-control system never feels overheated or chilly.Various physical mechanisms implement its temperature sensors in various rooms. These sensorsproduce signals that go to units that compute whether to turn the furnace on or turn the airconditioner on. The result of these computations cause switches to close so that the furnaceor air conditioner does actually change state. We can see the whole path from temperaturesensing to turning oﬀ the furnace. Every step can be seen to be one of a series of straightforwardphysical events. Nowhere are you tempted to invoke conscious sensation as an eﬀect or elementof the causal chain. This is the prima facie case against computationalism, and a solid one it seems to be. Therest of this article is an attempt to dismantle it.2 An Informal SurveyAlthough one might expect AI researchers to adopt a computationalist position on most issues,they tend to shy away from questions about consciousness. AI has often been accused of beingover-hyped, and the only way to avoid the accusation, apparently, is to be so boring thatjournalists stay away from you. As the ﬁeld has matured, and as a ﬂock of technical problemshave become its focus, it has become easier to bore journalists. The last thing most seriousresearchers want is to be quoted on the subject of computation and consciousness. In order to get some kind of indication of what positions researchers take on this issue, Iconducted an informal survey of Fellows of the American Association for Artiﬁcial Intelligencein the summer of 2003. I sent e-mail to all of them asking the following question: Most of the time AI researchers don’t concern themselves with philosophical questions, as a matter of methodology and perhaps also opinion about what is ultimately at stake. However, I would like to ﬁnd out how the leaders of our ﬁeld view the following problem: Create a computer or program that has “phenomenal consciousness,” that is, the ability to experience things. By “experience” here I mean “qualitative experience,” the kind in which the things one senses seem to have a deﬁnite but indescribable quality, the canonical example being “looking red” as opposed to “looking green.” Anyway, please choose from the following possible resolutions of this problem: 1. The problem is just too uninteresting compared to other challenges 3
  4. 4. 1 Problem uninteresting 3% 2a Ill-deﬁned 11% 19% 2b Only apparent 8% 3 AI silent 7% 4 Requires new ideas 32% 5 AI will solve it as is 3% 6 Solution in sight 15% 7 None of the above 21% Percentages indicate fraction of the 34 who responded Table 1: Results of survey of AAAI Fellows 2. The problem is too ill deﬁned to be interesting; or, the problem is only apparent, and requires no solution 3. It’s an interesting problem, but AI has nothing to say about it 4. AI researchers may eventually solve it, but will require new ideas 5. AI researchers will probably solve it, using existing ideas 6. AI’s current ideas provide at least the outline of a solution 7. My answer is not in the list above. Here it is:. . . Of course, I don’t mean to exclude other branches of cognitive science; when I say “AI” I mean “AI, in conjunction with other relevant disciplines.” However, if you think neuroscientists will ﬁgure out phenomenal consciousness, and that their solution will entail that anything not made out of neurons cannot possibly be conscious, then choose option 3. Because this topic is of passionate interest to a minority, and quickly becomes annoying to many others, please direct all followup discussion to Directions for subscribing to this mailing list are as follows: . . . Thanks for your time and attention. Of the approximately 207 living Fellows, I got responses from 34. The results are as indicatedin Table 1. Of those who chose 7 (None of the above) as answer, here are some of the reasons why: “Developing an understanding of the basis for conscious experience is a central, long-term challenge for AI and related disciplines. It’s unclear at the present time whether new ideas will be needed....” 4
  5. 5. “If two brains have isomorphic computation then the ‘qualia’ must be the same. Qualia must be just another aspect of computation — whatever we say of qualia must be a property of the computation viewed as computation.” “There are two possible ways (at least) of solving the problem of phenome- nal consciousness, ‘explaining what consciousness is’ and ‘explaining consciousness away.’ It sounds like you are looking for a solution of the ﬁrst type, but I believe the ultimate solution will be of the second type.” “The problem is ill-deﬁned, and always will be, but this does not make it unin- teresting. AI will play a major role in solving it.” If the table seems to indicate no particular pattern, just remember that what the datashow is that the overwhelming majority (173 out of 207) refused to answer the question atall. Obviously, this was not a scientiﬁc survey, and the fact that its target group containeda disproportionate number of Americans perhaps biased it in some way. Furthermore, thedetailed responses to my questions indicated that respondents understood the terms used inmany diﬀerent ways. But if 84% of AI Fellows don’t want to answer, we can infer that thequestions are pretty far from those that normally interest them. Even the 34 who answeredinclude very few optimists (if we lump categories 5 and 6 together), although about the samenumber (categories 1 and 2) thought the problem didn’t really need to be solved. Still, theoutright pessimists (category 3) were deﬁnitely in the minority. ——————————–3 Research on Computational Models of ConsciousnessIn view of the shyness about consciousness shown by serious AI researchers, it is not surprisingthat detailed proposals about phenomenal consciousness from this group should be few and farbetween.3.1 Moore/Turing InevitabilityOne class of proposals can be dealt with fairly quickly. Hans Moravec, in a series of books (Moravec,1988, 1999), and Raymond Kurzweil (1999) have more or less assumed that continuing progressin the development of faster, more capable computers will cause computers to equal and thensurpass humans in intelligence, and that consciousness will be an inevitable consequence. Theonly argument oﬀered is that the computers will talk as though they are conscious; what morecould we ask? I believe a careful statement of the argument might go like this: 1. Computers are getting more and more powerful. 5
  6. 6. 2. This growing power allows computers to do tasks that would have been considered infea- sible just a few years ago. It is reasonable to suppose, therefore, that many things we think of as infeasible will eventually be done by computers. 3. Pick a set of abilities such that if a system had them we would deal with it as we would a person. The ability to carry on a conversation must be in the set, but we can imagine lots of other abilities as well: skill in chess, agility in motion, visual perspicacity, and so forth. If we had a talking robot that could play poker well, we would treat it the same way we treated any real human seated at the same table. 4. We would feel an overwhelming impulse to attribute consciousness to such a robot. If it acted sad at losing money, or made whimpering sounds when it was damaged, we would respond as we would to a human that was sad or in pain. 5. This kind of overwhelming impulse is our only evidence that a creature is conscious. In particular, it’s the only real way we can tell that people are conscious. Therefore, our evidence that the robot was conscious would be as good as one could have. Therefore the robot would be conscious, or be conscious for all intents and purposes. I call this the “Moore/Turing inevitability” argument because it relies on Moore’s Law (Moore,1965) predicting exponential progress in the power of computers, plus a prediction about howwell future programs will do on the “Turing test,” proposed by Alan Turing (1950) as a toolfor rating how intelligent a computer is.4 Turing thought all questions about the actual intelli-gence (and presumably degree of consciousness) of a computer were too vague or mysterious toanswer. He suggested a behaviorist alternative: Let the computer carry on a conversation overa teletype line (or via an instant-messaging system, we would say today). If a savvy humanjudge could not distinguish the computer’s conversational abilities from those of a real personat a rate better than chance, then we would have some measure of the computer’s intelligence.We could use this measure instead of insisting on measuring the computer’s real intelligence,or actual consciousness. This argument has a certain appeal. It certainly seems that if technology brings us robotsthat we can’t help treating as conscious, then in the argument about whether they really areconscious the burden of proof will shift, in the public mind, to the party-poopers who deny thatthey are. But so what? You can’t win an argument by imagining a world in which you’ve wonit and declaring it inevitable. The anti-computationalists can make several plausible objections to the behavioral-inevitabilityargument: 4 Turing actually proposed a somewhat diﬀerent test. See (Davidson, 1990) for discussion. Nowadays thisversion is the one everyone works with. 6
  7. 7. • Just because computers have made impressive strides doesn’t mean that any task we set them they will eventually be able to carry out. In particular, progress in carrying on conversations has been dismal.5 • Even if a computer could carry on a conversation, that wouldn’t tell us anything about whether it really was conscious. • Overwhelming impulses are not good indicators for whether something is true. The ma- jority of people have an overwhelming impulse to believe that there is such a thing as luck, so that a lucky person has a greater chance of winning at roulette than an unlucky person. The whole gambling industry is based on exploiting the fact that this absurd theory is so widely believed. I will come back to the second of these objections in section 5.1. The others I am inclinedto agree with.3.2 Hofstadter, Minsky, McCarthyRichard Hofstadter touches on the problem of consciousness in many of his writings, especiallythe material he contributed to (Hofstadter & Dennett, 1981). Most of he what he writesseems to be intended to stimulate or tantalize one’s thinking about the problem. For example,in (Hofstadter, 1979) there is a chapter (reprinted in (Hofstadter & Dennett, 1981)) in whichcharacters talk to an anthill. The anthill is able to carry on a conversation because the antsthat compose it play roughly the role neurons play in a brain. Putting the discussion in theform of a vignette allows for playful digressions on various subjects. For example, the anthilloﬀers the anteater (one of the discussants) some of its ants, which makes vivid the possibilitythat “neurons” could implement a negotiation that ends in their own demise. It seems clear reading the story that Hofstadter believes that the anthill is conscious, andtherefore one could use integrated circuits rather than ants to achieve the same end. But mostof the details are left out. In this as in other works, it’s as if he wants to invent a new, playfulstyle of argumentation, in which concepts are broken up and tossed together into so manyconﬁgurations that the original questions one might have asked get shunted aside. If you’realready convinced by the computational story, then this conceptual play is delightful. If you’rea skeptic, I expect it can get a bit irritating. I put Marvin Minsky in this category as well; perhaps it should be called “Those who don’ttake consciousness very seriously as a problem.” He wrote a paper in 1968 (Minsky, 1968b)that introduced the concept of self-model, which, as we will see, is central to the computationaltheory of consciousness. 5 The Loebner Prize is awarded every year to the writer of a program that appears “most human” toa panel of judges. You can see how close the programs are getting to fooling anyone at the website, . 7
  8. 8. To an observer B, an object A* is a model of an object A to the extent that B can use A* to answer questions that interest him about A.. . . If A is the world, questions for A are experiments. A* is a good model of A, in B’s view, to the extent that A*’s answers agree with those of A, on the whole, with respect to the questions important to B. When a man M answers questions about the world, then (taking on ourselves the role of B) we attribute this ability to some internal mechanism W* inside M.This part is presumably uncontroversial. But what’s interesting is that W*, however it appears,will include a model of M himself, M*. In principle, M* will contain a model of W*, which wecan call W**. M can use W** to answer questions about the way he (M) models the world.One would think that M** (the model of M* in W**) would be used to answer questions aboutthe way M models himself, but Minsky has a somewhat diﬀerent take: M** is used to answer general questions about himself. Ordinary questions about himself, e.g., how tall he is, are answered by M*, but very broad questions about his nature, e.g., what kind of a thing he is, etc., are answered, if at all, by descriptive statements made by M** about M*.Now, the key point is that the accuracy of M* and M** need not be perfect. A man’s model of the world has a distinctly bipartite structure: One part is concerned with matters of mechanical, geometrical, physical character, while the other is associated with things like goals, meanings, social matters, and the like. This division of W* carries through the representations of many things in W*, especially to M itself. Hence, a man’s model of himself is bipartite, one part concerning his body as a physical object and the other accounting for his social and psychological experience.This is why dualism is so compelling. In particular, Minsky accounts for free will by supposingthat it develops from a “strong primitive defense mechanism” to resist or deny compulsion. If one asks how one’s mind works, he notices areas where it is (perhaps incor- rectly) understood, that is, where one recognizes rules. One sees other areas where he lacks rules. One could ﬁll this in by postulating chance or random activity. But this too, by another route, exposes the self to the . . . indignity of remote control. We resolve this unpleasant form of M** by postulating a third part, embodying a will or spirit or conscious agent. But there is no structure in this part; one can say nothing meaningful about it, because whenever a regularity is observed, its representation is transferred to the deterministic rule region. The will model is thus not formed from a legitimate need for a place to store deﬁnite information about one’s self; it has the singular character of being forced into the model, willy-nilly, by formal but essentially content-free ideas of what the model must contain. 8
  9. 9. One can quibble with the details, but the conceptual framework oﬀers a whole new wayof thinking about consciousness, by showing that introspection is mediated by models. Thereis no way for us to penetrate through them or shake them oﬀ, so we must simply live withany “distortion” they introduce. I put “distortion” in quotes because it’s too strong a word.The concepts we use to describe our mental lives were developed over centuries by people whoall shared the same kind of mental model. The distortions are built in. For instance, thereis no independent notion of “free will” beyond what we observe by means of our self-models.We can’t even say that free will is a dispensable illusion, because we have no way of gettingrid of it and living to tell the tale. Minsky’s insight is that to answer many questions aboutconsciousness we should focus more on the models we use to answer the questions than on thequestions themselves. Unfortunately, in that short paper, and in his later book The Society of Mind (Minsky,1986), Minsky throws oﬀ many interesting ideas, but refuses to go into the depth that manyof them deserve. He has a lot to say about consciousness in passing, such as how Freudianphenomena might arise out of the “society” of subpersonal modules that he takes the humanmind to be. But there is no solid proposal to argue for or against. John McCarthy has written a lot on what he usually calls “self-awareness” (McCarthy,1995b). However, his papers are mostly focused on robots’ problem-solving capacities and howthey would be enhanced by the ability to introspect. An important example is the ability ofa robot to infer that it doesn’t know something (such as whether the Pope is currently sittingor lying down). This may be self-awareness, but the word “awareness” here is used in a sensethat is quite separate from the notion of phenomenal consciousness that is our concern here. In (McCarthy, 1995a), he speciﬁcally addresses the issue of “zombies,” philosophers’ termfor hypothetical beings who behave exactly as we do but do not experience anything. Thispaper is a reply to an article by Todd Moody (1994) on zombies. He lists some introspectivecapacities it would be good to give to a robot (“. . . Observing its goal structure and formingsentences about it . . . . Observing how it arrived at its current beliefs . . . .”). Then he concludesabruptly: Moody isn’t consistent in his description of zombies. On page 1 they behave like humans. On page 3 they express puzzlement about human consciousness. Wouldn’t a real Moody zombie behave as though it understood as much about consciousness as Moody does? I tend to agree with McCarthy that the idea of a zombie is worthless, in spite of its initialplausibility. Quoting Moody: Given any functional [=, more or less, computational] description of cognition, as detailed and complete as one can imagine, it will still make sense to suppose that there could be insentient beings that exemplify that description. That is, it is possible that there could be a behaviourally indiscernible but insentient simulacrum of a human cognizer: a zombie. 9
  10. 10. The plausibility of this picture is that it does indeed seem that an intricate diagram of thehardware and software of a robot would leave consciousness out, just as with the computer-controlled heating system described in section 1. One could print it on rose-colored paper toindicate that the system was conscious, but the color of the paper would play no role in whatit actually did. The problem is that in imagining a zombie one tends at ﬁrst to forget that thezombie would say exactly the same things non-zombies say about their experiences. It wouldbe very hard to convince a zombie that it lacked experience; which means, as far as I can see,that we might be zombies, at which point the whole idea collapses. Almost everyone who thinks the idea is coherent sooner or later slips up the way Moodydoes: they let the zombie ﬁgure out that it is a zombie by noticing that it has no experience.By hypothesis, this is something zombies can’t do. Moody’s paper is remarkable only in howobvious the slip-up in it is. Consider, for example, the phenomenon of dreaming. Could there be a cognate concept in zombie-English? How might we explain dreaming to them? We could say that dreams are things that we experience while asleep, but the zombies would not be able to make sense[z] of this.6Of course, zombies would talk about their dreams (or dreams[z]?) exactly as we do; consult theintricate system diagram to verify this. McCarthy’s three-sentence reply is just about what Moody’s paper deserves. But meanwhilephilosophers such as Chalmers (1996) have written weighty tomes based on the assumption thatzombies make sense. McCarthy is not interested in refuting them. Similarly, in (McCarthy, 1990b), McCarthy discusses when it is legitimate to ascribe mentalproperties to robots. In some ways his treatment is more formal than that of Dennett, whichI discuss below. But he never builds on this theory to ask the key question: Is there more toyour having a mental state than having that state ascribed to you?3.3 Daniel DennettDaniel Dennett is not a researcher in artiﬁcial intelligence, but a philosopher of mind andessayist in cognitive science. Nonetheless, he is sympathetic to the AI project, and bases hisphilosophy on computational premises to a great degree. The models of mind that he hasproposed can be considered to be sketches of a computational model, and therefore constituteone of the most ambitious and detailed proposals for how AI might account for consciousness. Dennett’s (1969) Ph.D. dissertation proposed a model for a conscious system. It contains thesort of block diagram that has since become a standard feature of the theories of psychologistssuch as Bernard Baars (1988, 1997), although the central working arena is designed to accountfor introspection more than for problem-solving ability. 6 The “[z]” is used to ﬂag zombie words whose meanings mustn’t be confused with normal human concepts. 10
  11. 11. In later work, Dennett has not built upon this model, but, in a sense, has been rebuilding itfrom the ground up. The result has been a long series of papers and books, rich with insightsabout consciousness, free will, and intentionality. Their very richness makes it hard to extracta brisk theoretical statement, but I will try. Dennett has one overriding methodological principle, to be distrustful of introspection. Thisposition immediately puts him at odds with such philosophers as Nagel, Searle, and McGinn,for whom the “ﬁrst person” point of view is the alpha and omega of consciousness. On hisside Dennett has the many anecdotes and experimental data that show how wildly inaccurateintrospection can be, but his view does leave him open to the charge that he is ruling out all thecompetitors to his theory from the start. From a computationalist’s vantage point, this is all tothe good. It’s clear that any computationalist theory must eventually explain the mechanismof the ﬁrst-person view in terms of “third person” components. The “third person” is thatwhich you and I discuss, and therefore must be observable by you and me, and other interestedparties, in the same way. In other words, “third-person data” is just another way of saying“scientiﬁc data.” If there is to be a scientiﬁc explanation of the ﬁrst person, it will surely seemmore like an “explaining away” than a true explanation. An account of how yonder piece ofmeat or machinery is conscious will almost certainly invoke the idea of the machinery playing atrick on itself the result of which is for it to have a strong belief that it has a special ﬁrst-personviewpoint. One of Dennett’s special skills is using vivid images to buttress his case. He inventedthe phrase “Cartesian Theater” to describe the hypothetical place in the brain where the selfbecomes aware of things. He observes that belief in the Cartesian Theater is deep-seated,and keeps popping up in philosophical and psychological writings, as well as in common-sensemusings. We all know that there is a lot going on the brain that is preconscious or subconscious.What happens when a train of events becomes conscious? According to the view Dennett isridiculing, to bring it to consciousness is to show it on the screen in the Cartesian Theater.When presented this way, the idea does seem silly, if for no other reason than that there is noplausible homunculus to put in the audience. What’s interesting is how hard it is to shake thisimage. Just about all theorists of phenomenal consciousness at some point distinguish between“ordinary” and “conscious” events by making the latter be accessible to . . . what, exactly? Thesystem as a whole? Its self-monitoring modules? One must tread very carefully to keep fromdescribing the agent with special access as the good old transcendental self, sitting alone in theCartesian Theater. To demolish the Cartesian Theater, Dennett uses the tool of discovering or inventing situa-tions in which belief in it leads to absurd conclusions. Many of these situations are experimentsset up by psychology researchers. Most famous are the experiments by Libet (1985), whoseobject was to determine exactly when a decision to make a motion was made. What emergedfrom the experiments was that at the point where subjects think they have made the decision,the neural activity preparatory to the motion has already been in progress for hundreds ofmilliseconds. Trying to make sense of these results using the homuncular models lead to absur- 11
  12. 12. dities. (Perhaps the choice causes eﬀects in the person’s past?) But it is easy to explain themif you make a more inclusive picture of what’s going on in a subject’s brain. Libet and otherstended to assume that giving a subject a button to push when the decision had been madeprovided a direct route to . . . that pause again . . . the subject’s self, perhaps? Or perhaps theguy in the theater? Dennett points out that the neural apparatus required to push the buttonis part of the overall brain system. Up to a certain resolution, it makes sense to ask someone,“When did you decide to do X?” But it makes no sense to try to tease oﬀ a subsystem of thebrain and ask it the same question, primarily because there is no subsystem that embodies the“will” of the whole system. Having demolished most of the traditional model of consciousness, Dennett’s next goal is toconstruct a new one, and here he becomes more controversial, and in places more obscure. Akey component is human language. It is diﬃcult to think about human consciousness withoutpondering the ability of a normal human adult to say what they are thinking. There are twopossible views about why it should be the case that we can introspect so easily. One is that weevolved from animals that can introspect, so naturally when language evolved one of the topicsit was used on was the contents of our introspections. The other is that language plays a morecentral role than that; without language, we wouldn’t be conscious at all, at least full-bloodedly.Dennett’s view is the second. He has little to say about animal consciousness, and what hedoes say is disparaging. Language, for Dennett, is very important, but not because it is spoken by the homunculusin the Cartesian Theater. If you leave it out, who is speaking? Dennett’s answer is certainlybold: In a sense, the language speaks itself. We take it for granted that speaking feels like itemanates from our “transcendental self,” or, less politely, from the one-person audience in theTheater. Whether or not that view is correct now, it almost certainly was not correct whenlanguage began. In its original form, language was an information-transmission device used byapes whose consciousness, if similar to ours in any real respect, would be about the same asa chimpanzee’s today. Messages expresssed linguistically would heard by one person, and forone reason or another be passed to others. Their chance of being passed would depend, veryroughly, on how useful their recipients found them. The same mechanism has been in operation ever since. Ideas (or simple patterns unworthyof the name “idea” — advertising jingles, for instance) tend to proliferate in proportion tohow much they help those who adopt them, or in proportion to how well they tend to stiﬂecompeting ideas — not unlike what genes do. Dennett adopts Dawkins’s (1976) term memeto denote a linguistic pattern conceived of in this way. One key meme is the idea of talking tooneself; when it ﬁrst popped up, it meant literally talking out loud and listening to what wassaid. Although nowadays we tend to view talking to oneself as a possible symptom of insanity,we’ve forgotten that it gives our brains a whole new channel for its parts to communicate witheach other. If an idea — a pattern of activity in the brain — can reach the linguistic apparatus,it gets translated into a new form, and, as it is heard, gets translated back into a somewhatdiﬀerent pattern than the one that started the chain of events. Creatures that start to behave 12
  13. 13. this way start to think of themselves in a new light, as someone to talk to or listen to. Selfmodeling, according to Dennett (and Jaynes, 1976) starts as modeling this person we’re talkingto. There is nothing special about this kind of model; it is as crude as most of the models wemake. But memes for self-modeling have been some of the most successful in the history (andprehistory) of humankind. To a great degree, they make us what we are by giving us a modelof who we are that we then live up to. Every child must recapitulate the story Dennett tells,as it absorbs from its parents and peers all the ways to think of oneself, as a being with freewill, sensations, and a still small voice inside. The theory has one striking feature: it assumes that consciousness is based on language andnot vice versa. For that matter, it tends to assume that for consciousness to come to be, theremust be in place a substantial infrastructure of perceptual, motor, and intellectual skills. Theremay be some linguistic abilities that depend on consciousness, but the basic ability must existbefore and independent of consciousness. This conclusion may be fairly easy to accept for the more syntactic aspects of language, butit is contrary to the intuitions of many when it comes to semantics. Knowing what a sentencemeans requires knowing how the sentence relates to the world. If I am told “There is a lion onthe other side of that bush,” I have to understand that “that bush” refers to a particular objectin view; I have to know how phrases like “other side of” work; and I have to understand what“a lion” means so that I have a grasp of just what I’m expecting to confront. Furthermore,it’s hard to see how I could know what these words and phrases meant without knowing that Iknow what they mean. Meditating in this way on how meaning works, the late-nineteenth-century philosopherFranz Brentano developed the notion of intentionality, the power mental representations seemto have of pointing to — “being about” — things outside of, and arbitrarily far from, the mindor brain containing those representations. The ability of someone to warn me about that liondepends on that person’s sure-footed ability to reason about that animal over there, as well ason our shared knowledge about the species Panthera leo. Brentano, and many philosopherssince, have argued that intentionality is at bottom a property only of mental representations.There seem to be many kinds of “aboutness” in the world; for instance, there are books aboutlions; but items like books can be about a topic only if they are created by humans usinglanguage and writing systems in order to capture thoughts about that topic. Books are said tohave derived intentionality, whereas people have original or intrinsic intentionality. Computers seem to be textbook cases of physical items whose intentionality, if any, isderived. If one sees a curve plotted on a computer’s screen, the surest way to ﬁnd out what it’sabout is to ask the person who used some program to create it. In fact, that’s the only way.Digital computers are syntactic engines par excellence. Even if there is an interpretation to beplaced on every step of a computation, this interpretation plays no role in what the computerdoes. Each step is produced purely by operations dependent on the formal structure of itsinputs and prior state at that step. If you use TurboTax to compute your income taxes, thenthe numbers being manipulated represent real-world quantities, and the number you get at the 13
  14. 14. end represents what you actually do owe to the tax authorities. Nonetheless, TurboTax is justapplying formulas to the numbers. It “has no idea” what they mean. This intuition is what Dennett wants to defeat, as should every other researcher who ex-pects a theory of consciousness based on AI. There’s really no alternative. If you believe thatpeople are capable of original intentionality and computers aren’t, then you must believe thatsomething will be missing from any computer program that tries to simulate humans. Thatmeans that human consciousness is fundamentally diﬀerent from machine consciousness, whichmeans that a theory of consciousness based on AI is radically incomplete. Dennett’s approach to the required demolition job on intrinsic intentionality is to focus onthe prelinguistic, nonintrospective case. In a way, this is changing the subject fairly radically.In the introspective set-up, we are talking about elements or aspects of the mind that we areroutinely acquainted with, such as words and images. In the nonintrospective case, it’s notclear that those elements or aspects are present at all. What’s left to talk about if we’re nottalking about words, “images,” or “thoughts”? We’ll have to shift to talking about neurons,chips, ﬁring rates, bits, pointers, and other “subpersonal” entities and events. It’s not clear atall whether these things are even capable of exhibiting intentionality. Nonetheless, showing thatthey are is a key tactic in Dennett’s attack on the problem of consciousness. (See especiallyAppendix A of Dennett, 1991b.) If we can deﬁne what it is for subpersonal entities to beintentional, we can then build on that notion and recover the phenomenal entities we (thoughtwe) started with. “Original” intentionality will turn out to be a secondary consequence of whatI will call impersonal intentionality. Dennett’s approach to the problem is to call attention to what he calls the intentional stance,a way of looking at systems in which we impute beliefs and goals to them simply because there’sno better way to explain what they’re doing. For example, if you’re observing a good chessprogram in action, and its opponent has left himself vulnerable to an obvious attack, then onefeels conﬁdent that the program will embark on that attack. This conﬁdence is not based on anydetailed knowledge of the program’s actual code. Even someone who knows the program wellwon’t bother trying to do a tedious simulation to make a prediction that the attack will occur,but will base their prediction on the fact that the program almost never misses an opportunityof that kind. If you refuse to treat the program as though it had goals, you will be able to sayvery little about how it works. The intentional stance applies to the innards of the program as well. If a data structure isused by the program to make decisions about some situation or object S, and the decisions itmakes are well explained by assuming that one state of the data structure means that P is trueof S, and that another means P ′ , then those states do mean P and P ′ . It is perhaps unfortunate that Dennett has chosen to express his theory this way, becauseit is easy to take him as saying that all intentionality is observer-relative. This would bealmost as bad as maintaining a distinction between original and derived intentionality, becauseit would make it hard to see how the process of intentionality attribution could ever get started.Presumably my intuition that I am an intentional system is indubitable, but what could it be 14
  15. 15. based on? It seems absurd to think that this opinion is based on what others tell me, but itseems equally absurd that I could be my own observer. Presumably to be an observer you haveto be an intentional system (at least, if your observations are to be about anything). Can Ibootstrap my way into intentionality somehow? If so, how do I tell the successful bootstrappersfrom the unsuccessful ones? A computer program with an inﬁnite loop, endlessly printing, “Iam an intentional system because I predict, by taking the intentional stance, that I will continueto print this sentence out,” would not actually be claiming anything, let alone something true. Of course, Dennett does not mean for intentionality to be observer-relative, even thoughmany readers think he does. (To take an example at random from the Internet, the on-linePhilosopher’s Magazine, in their “Philosopher of the Month” column in April, 2003 (Douglas &Saunders, 2003), say “Dennett suggests that intentionality is not so much an intrinsic featureof agents, rather, it is more a way of looking at agents.”) Dennett has defended himself fromthis misinterpretation more than once (Dennett, 1991a). I will come back to this issue in myattempt at a synthesis in section 4.3.4 Perlis, SlomanThe researchers in this section, although they work on hard-headed problems in artiﬁcial intel-ligence, do take philosophical problems seriously, and have contributed substantial ideas to thedevelopment of the computational model of consciousness. Donald Perlis’s papers build a case that consciousness is ultimately based on self-consciousness,but I believe he is using the phrase “self-consciousness” in a misleading and unnecessary way.Let’s start with his paper (Perlis, 1994), which I think lays out a very important idea. He asks,Why do we need a dichotomy between appearance and reality? The answer is, Because theycould disagree, i.e., because I could be wrong about what I think I perceive. For an organismto be able to reason explicitly about this diﬀerence, it must be able to represent both X (anobject in the world) and quote-X, the representation of X in the organism itself. The latter isthe “symbol,” the former the “symboled.” To my mind the most important consequence of thisobservation is that it must be possible for an information processing system to get two kindsof information out of its X-recognizer: signals meaning “there’s an X,” and signals meaning“there’s a signal meaning ‘there’s an X.’ ” Perlis takes a somewhat diﬀerent tack. He believes there can be no notion of appearancewithout the notion of appearance to someone. So the self-model can’t get started without someprior notion of self to model. When we are conscious of X, we are also conscious of X in relation to ourselves: it is here, or there, or seen from a certain angle, or thought about this way and then that. Indeed, without a self model, it is not clear to me intuitively what it means to see or feel something: it seems to me that a point of view is needed, a place from which the scene is viewed or felt, deﬁning the place occupied by the viewer. Without 15
  16. 16. something along these lines, I think that a ‘neuronal box’ would indeed ‘confuse’ symbol and symboled: to it there is no external reality, it has no way to ‘think’ (consider alternatives) at all. Thus I disagree [with Crick] that self-consciousness is a special case of consciousness: I suspect that it is the most basic form of all. Perlis continues to elaborate this idea in later publications. For example, “Consciousness isthe function or process that allows a system to distinguish itself from the rest of the world.. . . Tofeel pain or have a vivid experience requires a self” (Perlis, 1997) (italics in original). I havetrouble following his arguments, which often depend on thought experiments such as imaginingcases where one is conscious but not of anything, or of as little as possible. The problemis that introspective thought experiments are just not a very accurate tool. One may perhapsconclude that Perlis, although housed in a Computer Science department, is not a thoroughgoingcomputationalist at all. As he says, “I conjecture that we may ﬁnd in the brain special amazingstructures that facilitate true self-referential processes, and constitute a primitive, bare or ur-awareness, an ‘I’. I will call this the amazing-structures-and-processes paradigm” (Perlis, 1997)(italics in original). It’s not clear how amazing the “amazing” structures will be, but perhapsthey won’t be computational. Aaron Sloman has written proliﬁcally about philosophy and computation, although hisinterests range far beyond our topic here. In fact, although he has been interested in consciouscontrol, both philosophically and as a strategy for organizing complex software, he has tendedto shy away from the topic of phenomenal consciousness. His book The Computer Revolutionin Philosophy (Sloman, 1978) has almost nothing to say about the subject, and in many otherwritings the main point he has to make is that the concept of consciousness covers a lot ofdiﬀerent processes, which should be sorted out before hard questions can be answered. However,in a few of his papers he has confronted the issue of qualia, notably (Sloman & Chrisley, 2003).I think the following is exactly right: Now suppose that an agent A . . . uses a self-organising process to develop con- cepts for categorising its own internal virtual machine states as sensed by internal monitors. . . . If such a concept C is applied by A to one of its internal states, then the only way C can have meaning for A is in relation to the set of concepts of which it is a member, which in turn derives only from the history of the self-organising process in A. These concepts have what (Campbell, 1994) refers to as ‘causal indexicality’. This can be contrasted with what happens when A interacts with other agents in such a way as to develop a common language for referring to features of external objects. Thus A could use ‘red’ either as expressing a private, causally indexical, concept referring to features of A’s own virtual-machine states, or as expressing a shared concept referring to a visible property of the surfaces of objects. This means that if two agents A and B have each developed concepts in this way, then if A uses its causally indexical concept Ca, to think the thought ‘I am having experience Ca’, 16
  17. 17. and B uses its causally indexical concept Cb, to think the thought ‘I am having ex- perience Cb’ the two thoughts are intrinsically private and incommunicable, even if A and B actually have exactly the same architecture and have had identical histories leading to the formation of structurally identical sets of concepts. A can wonder: ‘Does B have an experience described by a concept related to B as my concept Ca is related to me?’ But A cannot wonder ‘Does B have experiences of type Ca’, for it makes no sense for the concept Ca to be applied outside the context for which it was developed, namely one in which A’s internal sensors classify internal states. They cannot classify states of B. This idea suggests that the point I casually assumed at the beginning of this paper, thattwo people might wonder if they experienced the same thing when they ate tacos, is actuallyincoherent. Our feeling that the meaning is clear is due to the twist our self-models give tointrospections of the kind Sloman and Chrisley are talking about. The internal representation ofthe quale of redness is purely local to A’s brain, but the self-model says quite the opposite, thatobjects with the color are recognizable by A because they have that quale. The quale is madeinto an objective entity that might attach itself to other experiences, such as my encounterswith blue things, or B’s experiences of red things.3.5 Brian Cantwell SmithThe last body of research to be examined in this survey is that of Brian Cantwell Smith. It’s hardto dispute that he is a computationalist, but he is also an antireductionist, which places him in arather unique category. Although it is clear in reading his work that he considers consciousnessto be a crucial topic, he has been working up to it very carefully. His early work (Smith, 1984)was on “reﬂection” in programming languages, that is, how and why a program written in alanguage could have access to information about its own subroutines and data structures. Onemight conjecture that reﬂection might play a key role in a system’s maintaining a self-modeland thereby being conscious. But since that early work Smith has moved steadily away fromstraightforward computational topics and toward foundational philosophical ones. Each of hispapers seems to take tinier steps from ﬁrst principles than the ones that have gone before, so asto presuppose as little as humanly possible. Nonetheless, they often express remarkable insight.His paper (Smith, 2002) on the “Foundations of Computing” is a gem. (I also recommend(Sloman, 2002), from the same collection (Scheutz, 2002).) One thing both Smith and Sloman argue is that Turing machines are misleading as idealvehicles for computationalism, which is a point often missed by philosophers. For example,Wilkes (1990) says “. . . computers (as distinct from robots) produce at best only linguistic andexclusively ‘cognitive’ — programmable — ‘behaviour’: the emphasis is on internal psycholog-ical processes, the cognitive ‘inner’ rather than on action, emotion, motivation, and sensoryexperience.” Perhaps I’ve misunderstood him, but it’s very hard to see how this can be true, 17
  18. 18. given that all interesting robots are controlled by digital computers. Furthermore, when com-puters and software are studied isolated from their physical environments, it’s often for purelytactical reasons (from budget or personnel limitations, or to avoid endangering bystanders).If we go all the way back to Winograd’s (1972) SHRDLU system, we ﬁnd a simulated robotplaying the role of conversationalist, not because Winograd thought real robots were irrelevant,but precisely because he was thinking of a long-term project in which an actual robot wouldbe used. As Smith (2002) says, In one way or another, no matter what construal [of formality] they pledge allegiance to, just about everyone thinks that computers are formal. . . . But since the outset, I have not believed that this is necessarily so.. . . Rather, what computers are . . . is neither more nor less than the full-ﬂedged social construction and development of intentional artifacts. (italics in original)The point he is trying to make (and it can be hard to ﬁnd a succinct quote in Smith’s papers) isthat computers are always connected to the world, whether they are robots or not, and thereforethe meaning their symbols possess is more determined by those connections than by what aformal theory might say they mean. One might want to rule that the transducers that connectthem to the world are noncomputational (cf. (Harnad, 1990)), but there is no principled wayto draw a boundary between the two parts, because ultimately a computer is physical partsbanging against other physical parts. As Sloman puts it, . . . The view of computers as somehow essentially a form of Turing machine . . . is simply mistaken. . . . [The] mathematical notion of computation . . . is not the primary motivation for the construction or use of computers, nor is it particularly helpful in understanding how computers work or how to use them (Sloman, 2002). The point Smith makes in the paper cited above is elaborated into an entire book, Onthe Origin of Objects (Smith, 1995). The problem the book addresses is the basic ontologyof physical objects. The problem is urgent, according to Smith, because the basic conceptof intentionality is that a symbol S stands for an object X; but we have no prior concept ofwhat objects or symbols are. A geologist might see a glacier on a mountain, but is there someobjective reason why the glacier is an object (and the group of stones suspended in it is not)?Smith believes that all object categories are to some extent carved out by subjects, i.e., byinformation-processing systems like us (and maybe someday by robots as well). The problem with this point of view is that it is hard to bootstrap oneself out of whatSmith calls the Criterion of Ultimate Concreteness: “No naturalistically palatable theory ofintentionality — of mind, computation, semantics, ontology, objectivity — can presume theidentify or existence of any individual object whatsoever” (p. 184). He tries valiantly to derivesubjects and objects from prior . . . umm . . . “entities” called s-regions and o-regions, but it is 18
  19. 19. hard to see how he succeeds. In spite of its length, 420 pages, the book claims to arrive at nomore than a starting point for a complete rethinking of physics, metaphysics, and everythingelse. Most people will have a hard time following Smith’s inquiry, not least because few peopleagree on his opening premise, that everyday ontology is broken and needs to be ﬁxed. I actuallydo agree with that, but I think the problem is much worse than Smith does. Unlike him, I amreductionist enough to believe that physics is the science of “all there is”; so how do objectsemerge from a primordial superposition of wave functions? Fortunately, I think this is a problemfor everyone, and has nothing to do with the problem of intentionality.7 If computationalistsare willing to grant that there’s a glacier over there, anyone should be willing to consider thecomputational theory of how systems refer to glaciers.4 A Synthetic SummaryIn spite of the diﬃdence of most AI researchers on this topic, I believe that there is a dominantposition on phenomenal consciousness among computationalists, “dominant” in the sense thatamong the small population of those who are willing to take a clear position, this is more orless the position they take. In this section I will try to sketch that postion, pointing out thesimilarities and diﬀerences from the positions sketched in section 3. The idea in a nutshell is that phenomenal consciousness is the property a computationalsystem X has if X models itself as experiencing things. To understand it, I need to explain 1. What a computational system is. 2. How such a system can exhibit intentionality. 3. That to be conscious is to model oneself as having experiences.4.1 The Notion of Computational SystemBefore we computationalists can really get started, we run into the objection that the word“computer” doesn’t denote the right kind of thing to play an explanatory role in a theory ofany natural phenomenon. A computer, so the objection goes, is an object that people8 useto compute things. Without people to assign meanings to its inputs and outputs, a computeris just an overly complex electronic kaleidoscope, generating a lot of pseudo-random patterns.We may interpret the output of a computer as a prediction about tomorrow’s weather, butthere’s no other sense in which the computer is predicting anything. A chess computer outputs 7 Even more fortunate, perhaps, is the fact that few will grant that foundational ontology is a problem in theﬁrst place. Those who think elementary particles invented us, rather than vice versa, are in the minority. 8 Or intelligent aliens, but this is an irrelevant variation on the theme. 19
  20. 20. a syntactically legal expression that we can take to be its next move, but the computer doesn’tactually intend to make that move. It doesn’t intend anything. It doesn’t care whether themove is actually made. Even if it’s displaying the move on a screen, or using a robot arm topick up a piece and move it, these outputs are just meaningless pixel values or drive-motortorques until people supply the meaning. In my opinion, the apparent diﬃculty of supplying an objective deﬁnition of syntax andespecially semantics is the most serious objection to the computational theory of psychology,and in particular to a computational explanation of phenomenal consciousness. To overcome it,we need to come up with a theory of computation (and eventually semantics) that is observer-independent. There are two prongs to this attack, one syntactic, the other semantic. The syntactic prongis the claim that even the symbols we attribute to computers are observer-relative. We pointto a register in the computer’s memory, and claim that it contains a number. The critic thensays that the mapping of states that cause this state to encode “55,000” is entirely arbitrary;there are an inﬁnite number of ways of interpreting the state of the register, none of which isthe “real” one in any sense; all we can talk about is the intended one. A notorious exampleof John Searle’s exempliﬁes this kind of attack; he claims in (Searle, 1992) that the wall of hisoﬃce could be considered to be a computer under the right encoding of its states. The semantic prong is the observation, discussed in sections 3.3 and 3.5, that even afterwe’ve agreed that the register state encodes “55,000,” there is no objective sense in which thisﬁgure stands for “Jeanne D’Eau’s 2003 income in euros.” If Jeanne D’Eau is using the EuroTaxsoftware package to compute her income tax, then such semantic statements are nothing buta convention adopted by her and the people that wrote EuroTax. In other words, the onlyintentionality exhibited by the program is derived intentionality. To avoid these objections, we have to be careful about how we state our claims. I havespace for only a cursory overview here; see (McDermott, 2001) for a more detailed treatment.First, the idea of “computer” is prior to the idea of “symbol.” A basic computer is any physicalsystem whose subsequent states are predictable given its prior states. By “state” I mean “partialstate,” so that the system can be in more than one state at a time. An encoding is a mappingfrom partial physical states to some syntactic domain (e.g., numerals). To view a system asa computer, we need two encodings, one for inputs, one for outputs. It computes f (x) withrespect to a pair I, O of encodings if and only if putting it into the partial state encoding xunder I causes it to go into a partial state encoding f (x) under O. A memory element under an encoding E is a physical system that, when placed into a states such that E(s) = x, tends to remain in the set of states {s : E(s) = x} for a while. A computer is then a group of basic computers and memory elements viewed under a consis-tent encoding scheme, meaning merely that if changes of component 1’s state cause component2’s state to change, then the encoding of 1’s outputs is the same as the encoding of 2’s inputs.Symbol sites then appear as alternative possible stable regions of state space, and symbol to-kens as chains of symbol sites such that the occupier of a site is caused by the presence of the 20
  21. 21. occupier of its predecessor site. Space does not allow me to discuss all the details here, but thepoint is clear: the notions of computer and symbol are not observer-relative. Of course, theyare encoding-relative, but then velocity is “reference-frame-relative.” The encoding is purelysyntactic, or even pre-syntactic, since we have said nothing about what syntax an encoded valuehas, if any. We could go on to say more about syntax, but one has the feeling that the wholeproblem is a practical joke played by philosophers on naive AI researchers. (“Let’s see howmuch time we can get them to waste deﬁning ’computer’ for us, until they catch on.”) I directyou to (McDermott, 2001) for more of my theory of syntax. The important issue is semantics,to which we now turn. One last remark: The deﬁnitions above are not intended to distinguish digital from analoguecomputers, or serial from parallel ones. They are broad enough to include anything anyonemight ever construe as a computational system. In particular, they allow neural nets (Rumelhartet al., 1986), natural and artiﬁcial, to count as computers. Many observers of AI (Churchland,1986; Churchland, 1988; Wilkes, 1990) believe that there is an unbridgeable chasm betweensome classical, digital, traditional AI and a revolutionary, analogue, connectionist alternative.The former is the realm of von Neumann machines, the latter the realm of artiﬁcial neuralnetworks, “massively parallel” networks of simple processors (meant to mimic neurons), whichcan be trained to learn diﬀerent categories of sensory data (Rumelhart et al., 1986). The“chasm” between the two is less observable in practice than you might infer from the literature.AI researchers are omnivorous consumers of algorithmic techniques, and think of neural netsas one of them — entirely properly, in my opinion. I will return to this subject in section Intentionality of Computational SystemsI described above Dennett’s idea of the “intentional stance,” in which an observer explainsa system’s behavior by invoking intentional categories such as beliefs and goals. Dennett iscompletely correct that there is such a stance. The problem is that we sometimes adopt itinappropriately. People used to think thunderstorms were out to get them, and a sign on mywife’s printer says, “Warning! This machine is subject to breakdown during periods of criticalneed.” What could it possibly mean to say that a machine demonstrates real intentionalitywhen it is so easy to indulge in a mistaken or merely metaphorical “intentional stance”? Let’s consider an example. Suppose someone has a cat that shows up in the kitchen at thetime it is usually fed, meowing and behaving in other ways that tend to attract the attentionof the people who usually feed it. Contrast that with the case of a robot that, whenever itsbattery is low, moves along a black trail painted on the ﬂoor that leads to the place where itgets recharged, and, when it is over a large black cross that has been painted at the end ofthe trail, emits a series of beeps that tend to attract the attention of the people who usuallyrecharge it. Some people might refuse to attribute intentionality to either the cat or the robot,and treat comments such as, “It’s trying to get to the kitchen [or recharging area],” or “Itwants someone to feed [or recharge] it,” as purely metaphorical. They might take this position, 21
  22. 22. or argue that it’s tenable, on the grounds that we have no reason to suppose that either thecat or the robot has mental states, and hence nothing with the kind of “intrinsic aboutness”that people exhibit. High catologists9 are sure cats do have mental states, but the skepticwill view this as just another example of someone falling into the metaphorical pit of “as-if”intentionality. I believe, though, that even hard-headed low catologists think the cat is truly intentional,albeit in the impersonal way discussed in section 3.3. They would argue that if you could openup its brain you would ﬁnd neural structures that “referred to” the kitchen or the path to it, inthe sense that those structures became active in ways appropriate to the cat’s needs: they wereinvolved in steering the cat to the kitchen and stopping it when it got there. A similar accountwould tie the meowing behavior to the event of getting food, mediated by some neural states.We would then feel justiﬁed in saying that some of the neural states and structures denoted thekitchen, or the event of being fed. The question is, Are the ascriptions of impersonal intentionality so derived arbitrary, orobjectively true? It’s diﬃcult to take either choice. It feels silly saying that something isarbitrary if it takes considerable eﬀort to ﬁgure it out, and if one is conﬁdent that if someoneelse independently undertook the same project they would reach essentially the same result.But it also feels odd to say that something is objectively true if it is inherently invisible.Nowhere in the cat will you ﬁnd labels that say “This means X,” nor little threads that tieneural structures to objects in the world. One might want to say that the cat is an intentionalsystem because there was evolutionary pressure in favor of creatures whose innards were tied via“virtual threads” to their surroundings. I don’t like dragging evolution in because it’s more ofa question stopper than a question answerer. I prefer the conclusion that reluctance to classifyintentionality as objectively real simply reveals an overly narrow conception of objective reality. A couple of analogies will help. 1. Code breaking: A code breaker is sure they have cracked a code when the message turns into meaningful natural-language text. That’s because there are an enormous number of possible messages, and an enormous number of possible ciphers, out of which there is (almost certainly) only one combination of natural-language text and simple cipher that produces the encrypted message. Unfortunately for this example, it involves interpreting the actions of people. So even if there is no observer-relativity from the cryptanalyst’s point of view, the intentionality in a message is “derived” according to skeptics about the possible authentic intentionality of physical systems. 2. Geology: A geologist strives to ﬁnd the best explanation for how various columns and strata of rock managed to place themselves in the positions they are found in. A good 9 By analogy with Christology in Christian theology, which ranges from high to low depending on how super-human one believes Jesus to be. 22
  23. 23. explanation is a series of not-improbable events that would have transformed a plausible initial conﬁguration of rocks into what we see today. In this case, there is no observer-relativity, because there was an actual sequence of events that led to the current rock conﬁguration. If two geologists have a profound disagreement about the history of a rock formation, they can’t both be right (as they might be if disagreeing about the beauty of a mountain range). Our normal expectation is that any two geologists will tend to agree on at least the broad outline of an explanation of a rock formation; and that as more data are gathered the areas of agreement will grow. These examples are cases where, even though internal harmoniousness is how we judgeexplanations, what we get in the end is an explanation that is true, independent of the harmo-niousness. All we need to do is allow for this to be true even though, in the case of intentionality,even a time machine or mind reader would not give us an independent source of evidence. Tohelp us accept this possibility, consider the fact that geologists can never actually get the entirestory right. What they are looking at is a huge structure of rock with a detailed microhistorythat ultimately accounts for the position of every pebble. What they produce in the end isa coarse-grained history that talks only about large intrusions, sedimentary layers, and such.Nonetheless we say that it is objectively true, even though the objects it speaks of don’t evenexist unless the account is true. It explains how a particular “intrusion” got to be there, butif geological theory isn’t more or less correct, there might not be such a thing as an intrusion;the objects might be parsed in a totally diﬀerent way. If processes and structures inside a cat’s brain exhibit objectively real impersonal intention-ality, then it’s hard not to accept the same conclusion about the robot trying to get recharged.It might not navigate the way the cat does — for instance, it might have no notion of a placeit’s going to, as opposed to the path that gets it there — but we see the same ﬁt with itsenvironment among the symbol structures in its hardware or data. In the case of the robot thehardware and software were designed, and so we have the extra option of asking the designerswhat the entities inside the robot were supposed to denote. But it will often happen that thereis conﬂict between what the designers intended and what actually occurs, and what actuallyoccurs wins. The designers don’t get to say, “This boolean variable means that the robot isgoing through a door” unless the variable’s being true tends to occur if and only if the robot isbetween two door jambs. If the variable is correlated with something else instead, then that’swhat it actually means. It’s appropriate to describe what the roboticists are doing as debuggingthe robot so that its actual intentionality matches their intent. The alternative would be todescribe the robot as “deranged” in the sense that it continuously acts in ways that are bizarregiven what its data structures mean. Two other remarks are in order: What the symbols in a system mean is dependent on thesystem’s environment. If a cat is moved to a house that is so similar to the one it’s familiar withthat the cat is fooled, then the structures inside it that used to refer to the kitchen of house 1now refer to the kitchen of house 2. And so forth; and there will of course be cases in which the 23
  24. 24. denotation of a symbol breaks down, leaving no coherent story about what it denotes, just as inthe geological case an event of a type unknown to geology, but large enough to cause large-scaleeﬀects, will go unhypothesized, and some parts of geologists’ attempts to make sense of whatthey see will be too incoherent to be true or false, or to even to refer to anything. The other remark is that it might be the case that the sheer size of the symbolic systemsinside people’s heads might make the impersonal intentionality story irrelevant. We don’t,of course, know much about the symbol systems used by human brains, whether there is a“language of thought” (Fodor, 1975) or some sort of connectionist soup, but clearly we can havebeliefs orders of magnitude more complex than those of a cat or a robot (year-2006 model).If you walk to work, but at the end of the day absent-mindedly head for the parking lot toretrieve your car, what you will believe once you get there has the content “My car is not here.”Does this belief correspond to a symbol structure in the brain whose pieces include symboltokens for “my car,” “here,” and “not”? We don’t know. But if anything like that pictureis accurate, then assigning a meaning to symbols such as “not” is considerably more diﬃcultthan assigning a meaning to the symbols a cat or robot might use to denote “the kitchen.”Nonetheless, the same basic story can still be told: that the symbols mean what the mostharmonious interpretation says they mean. This story allows us to assign arbitrarily abstractmeanings to symbols like “not”; the price we pay is that for now all we have is an IOU for aholistic theory of the meanings inside our heads.4.3 Modelling Oneself as ConsciousI have spent a lot of time discussing intentionality because once we can establish the conceptof an impersonal level of meaning in brains and computers, we can introduce the idea of aself-model, a device that a robot or a person can use to answer questions about how it interactswith the world. This idea was introduced by Minsky almost forty years ago (Minsky, 1968a),and has since been explored by many others, including Sloman (Sloman & Chrisley, 2003),McDermott (2001), and Dennett (1991b). As I mentioned above, Dennett mixes this idea withthe concept of meme, but self-models don’t need to be made out of memes. We start with Minsky’s observation that complex organisms use models of their environ-ments in order to predict what will happen and decide how to act. In the case of humans,model making is taken for granted by psychologists (Johnson-Laird, 1983); no one really knowswhat other animals’ capacities for using mental models are. A mental model is some sort ofinternal representation of part of the organism’s surroundings that can be inspected, or even“run” in some way, so that features of the model can then be transformed back into inferred orpredicted features of the world. For example, suppose you’re planning to go grocery shopping,and the skies are threatening rain, and you’re trying to decide whether to take an umbrella.You enumerate the situations where the umbrella might be useful, and think about whether onbalance it will be useful enough to justify having to keep track of it. One such situation is thetime when you emerge from the store with a cartload of groceries to put in the car. Will the 24
  25. 25. umbrella keep you or your groceries dry?10 This deﬁnition is general (and vague) enough to cover noncomputational models, but thecomputationalist framework provides an obvious and attractive approach to theorizing aboutmental models. In this framework, a model is an internal computer set up to simulate something.The organism initializes it, lets it run for a while, reads oﬀ its state, and interprets the stateas a set of inferences that then guide behavior. In the umbrella example, one might imaginea physical simulation, at some level of resolution, of a person pushing a cart and holding anumbrella while rain falls. A mental model used by an agent A to decide what to do must include A itself, simplybecause any situation A ﬁnds itself in will have A as one of its participants. If I am on asinking ship, and trying to pick a lifeboat to jump into, predicting the number of people onthe lifeboat must not omit the “+ 1” required to include me. This seemingly minor principlehas far-reaching consequences, because many of A’s beliefs about itself will stem from the wayits internal surrogates participate in mental models. We will call the beliefs about a particularsurrogate a self-model, but usually for simplicity I will refer to the self-model, as if all thosebeliefs are pulled together into a single “database.” Let me state up front that the way thingsreally work is likely to be much more complex and messy. Let me also declare that the self-model is not a Cartesian point of transcendence where the self can gaze at itself. It is a resourceaccessible to the brain at various points for several diﬀerent purposes. We can distinguish between exterior and interior self-models. The former refer to the agentconsidered as a physical object, something with mass that might sink a lifeboat. The latterrefers to the agent considered as an information-processing system. To be concrete, let’s lookat a self-model that arises in connection with the use of anytime algorithms to solve time-dependent planning problems (Boddy & Dean, 1989). An anytime algorithm is one that can bethought of as an asynchronous process that starts with a rough approximation to the desiredanswer and gradually improves it; it can be stopped at any time and the quality of the result itreturns depends on how much run time it was given. We can apply this idea to planning robotbehavior, in situations where the objective is to minimize the total time required to solve theproblem, which is equal to time (tP ) to ﬁnd a plan P + time (tE (P )) to execute PIf the planner is an anytime algorithm, then the quality of the plan it returns improves withtP . We write P (tP ) to indicate that the plan found is a function of the time allotted to ﬁndingit. Because quality is execution time, we can reﬁne that statement and say that tE (P (tP ))decreases as tP increases. Therefore, in order to optimize tP + tE (P (tP )) 10 For some readers this example will elicit fairly detailed visual images of shopping carts and umbrellas, andfor those readers it’s plausible that the images are part of the mental-model machinery. But even people withoutmuch visual imagery can still have mental models, and might still use them to reason about grocery shopping. 25
  26. 26. we must ﬁnd the smallest tP such that the time gained by planning ∆t longer than that wouldprobably improve tE by less than ∆t. The only way to ﬁnd that optimal tP is to have anapproximate model of how fast tE (P (tP )) changes as a function of tP . Such a model wouldno doubt reﬂect the law of diminishing returns, so that ﬁnding the optimal tP is an easy one-dimensional optimization problem. The important point for us is that this model is a model ofthe planning component of the robot, and so counts as an interior self-model. Let me make sure my point is clear: interior self-models are no big deal. Any algorithm thatoutputs an estimate of something plus an error range incorporates one. The mere presence ofa self-model does not provide us some kind of mystical reﬂection zone where we can make con-sciousness pop out as an “emergent” phenomenon. This point is often misunderstood by criticsof AI (Rey, 1997; Block, 1997) who attribute to computationalists the idea that consciousnessis nothing but the ability to model oneself. In so doing, they tend to muddy the water furtherby saying that computationalists confuse consciousness with self-consciousness. I hope in whatfollows I can make these waters a bit clearer. Today’s information-processing systems are not very smart. They tend to work in narrowdomains, and outperform humans only in areas, such as chess and numerical computation,where clear formal ground rules are laid out in advance. A robot that can walk into a room,spy a chessboard, and ask if anyone wants to play is still far in the future. This state of aﬀairsraises a huge obstacle for those who believe that consciousness is built on top of intelligencerather than vice versa, that obstacle being that everything we say is hypothetical. It’s easyto counter the computationalist argument. Just say, “I think you’re wrong about intelligencepreceding consciousness, but even if you’re right I doubt that computers will ever reach thelevel of intelligence required.” To which I reply, Okay. But let’s suppose they do reach that level. We avoid begging anyquestions by using my hypothetical chess-playing robot as a concrete example. We can imagineit being able to locomote, see chessboards, and engage in simple conversations. (“Want toplay?” “Later.” “I’ll be back.”) We start by assuming that it is not conscious, and thenthink about what it would gain by having interior self-models of a certain class. The startingassumption, that it isn’t conscious, should be uncontroversial. One thing such a robot might need is a way to handle perceptual errors. Suppose that ithas a subroutine for recognizing chessboards and chessmen.11 For serious play only Stauntonchess pieces are allowed, but you can buy a chessboard with pieces of almost any shape; I haveno doubt that Disney sells a set with Mickey and Minnie Mouse as king and queen. Our robot,we suppose, can correct for scale, lighting, and other variations of the appearance of Stauntonpieces, but just can’t “parse” other kinds of pieces. It could also be fooled by objects that onlyappeared to be Staunton chess pieces. 11 I have two reasons for positing a chessboard-recognition subroutine instead of a general-purpose vision systemthat recognizes chessboards and chess pieces in terms of more “primitive” elements: (1) Many roboticists preferto work with specialized perceptual systems; and (2) the qualia-like entities we will predict will be diﬀerent incontent from human qualia, which reduces the chances of jumping to conclusions about them. 26
  27. 27. Now suppose that the robot contained some modules for improving its performance. Itmight be diﬃcult to calibrate the perceptual systems of our chess-playing robots at the factory,especially since diﬀerent owners will use them in diﬀerent situations. So we suppose that aftera perceptual failure a module we will call the perception tuner will try to diagnose the problemand change the parameters of the perceptual system to avoid it in the future. The perception tuner must have access to the inputs and outputs of the chess recognitionsystem, and, of course, access to parameters that it can change in order to improve the system’sperformance. It must have a self-model that tells it how to change the parameters to reduce thelikelihood of errors. (The “back propagation” algorithm used in neural nets (Rumelhart et al.,1986) is an example.) What I want to call attention to is that the perception tuner interpretsthe outputs of the perceptual system in a rather diﬀerent way from the decision-making system.The decision-making system interprets them (to oversimplify) as being about the environment;the tuning system interprets them as being about the perceptual system. For the decisionmaker the output “Pawn at x, y, z ” means that there is a pawn at a certain place. For thetuner, it means that the perceptual system says there is a pawn, in other words, that thereappears to be a pawn. Here is where the computationalist analysis of intentionality steps in. We don’t need tobelieve that either the decision maker or the tuner literally “thinks” that a symbol structureat a certain point means a particular thing. The symbol structure S means X if there is aharmonious overall interpretation of the states of the robot in which S means X. The perceptual-tuner scenario suggests that we can distinguish two sorts of access to asubsystem: normal access and introspective access. The former refers to the ﬂow of informationthat the subsystem extracts from the world (Dretske, 1981). The latter refers to the ﬂowof information it produces about the normal ﬂow.12 For our robot, normal access gives itinformation about chess pieces; introspective access gives it information about . . . what, exactly?A datum produced by the tuner would consist of a designator of some part of the perceptualﬁeld that was misinterpreted, plus information about how it was interpreted and how it shouldhave been. We can think of this as being information about “appearance” vs. “reality.” The next step in our story is to suppose that our robot has “episodic” memories, thatis, memories of particular events that occurred to it. (Psychologists draw distinctions betweenthese memories and other kinds, such as learned skills (e.g.,the memory of how to ride a bicycle)and abstract knowledge (e.g., the memory that France is next to Germany), sometimes calledsemantic memory.) We take episodic memory for granted, but presumably ﬂatworms do withoutit; there must be a reason why it evolved in some primates. One possibility is that it’s a means tokeep track of events whose signiﬁcance is initially unknown. If something bad or good happensto an organism, it might want to retrieve past occasions when something similar happened and 12 Of course, what we’d like to be able to say here is that normal access is the access it was designed to support,and for most purposes that’s what we will say, even when evolution is the “designer.” But such basic conceptscan’t depend on historical events arbitrarily far in the past. 27
  28. 28. try to see a pattern. It’s hard to say why the expense of maintaining a complex “database”would be paid back in terms of reproductive success, especially given how wrongheaded peoplecan be about explaining patterns of events. But perhaps all that is required is enough paranoiato avoid too many false negatives in predicting catastrophes. The ﬁnal step is to suppose that the robot can ask fairly general questions about theoperation of its perceptual and decision-making systems. Actually, this ability is closely tiedto the ability to store episodic memories. To remember something one must have a notation toexpress it. Remembering a motor skill might require storing a few dozen numerical parameters(e.g., weights in neural networks, plus some sequencing information). If this is correct, then, asargued above, learning a skill means nudging these parameters toward optimal values. Becausethis notation is so lean, it won’t support recording the episodes during which skill was enhanced.You may remember your golf lessons, but those memories are independent of the “memories,”encoded as numerical parameters, that manifest themselves as an improved putt. If you tryto think of a notation in which to record an arbitrary episode, it’s like trying to think of aformal notation to capture the content of a Tolstoy novel. It’s not even clear what it wouldmean to record an episode. How much detail would there be? Would it always have to befrom the point of view of the creature that recorded it? Such questions get us quickly intothe realm of Knowledge Representation, and the Language of Thought (Fodor, 1975). Forthat matter, we are quickly led to the topic of ordinary human language, because the abilityto recall an episode seems closely related to the ability to tell about it, and to ask about it.We are far from understanding how language, knowledge representation, and episodic memorywork, but it seems clear that the mechanisms are tightly connected, and all have to do withwhat sorts of questions the self-model can answer. This clump of mysteries accounts for whyDennett’s (1991b) meme-based theory is so attractive. He makes a fairly concrete proposalthat language came ﬁrst and that the evolution of the self-model was driven by the evolutionof language. Having waved our hands a bit, we can get back to discussing the ability of humans, andpresumably other intelligent creatures, to ask questions about how they work. We will justassume that these questions are asked using an internal notation reminiscent of human language,and then answered using a Minskyesque self-model. The key observation is that the self-modelneed not be completely accurate, or, rather, that there is a certain ﬂexibility in what counts asan accurate answer, because what it says can’t be contradicted by other sources of information.If everyone’s self-model says they have free will, then free will can’t be anything but whateverit is everyone thinks they have. It becomes diﬃcult to deny that we have free will, becausethere’s no content to the claim that we have it over and above what the chorus of self-modelsdeclare.13 Phenomenal experience now emerges as the self-model’s answer to the question, What hap- 13 For the complete story on free will, see (McDermott, 2001), ch. 3. I referred to Minsky’s rather diﬀerenttheory above; McCarthy champions his own version in (McCarthy & Hayes, 1969). 28
  29. 29. pens when I perceive something? The answer, in terms of appearance, reality, and error, isaccurate up to a point. It’s when we get to qualia that the model ends the explanation with ajust-so story. It gives more useful answers on questions such as whether it’s easier to confusegreen and yellow than green and red, or what to do when senses conﬂict, or what conditionsmake errors more or less likely. But to questions such as, How do I know this is red in theﬁrst place?, it gives an answer designed to stop inquiry. The answer is that red has this quality(please focus attention on the red object) which is intrinsically diﬀerent from the analogousquality for green objects (now focus over here, if you don’t mind). Because red is “intrinsicallylike . . . this,” there is no further question to ask. Nor should there be. I can take steps toimprove my classiﬁcation of objects by color, but there’s nothing I can do to improve my abilityto tell red from green (or, more plausibly, to tell two shades of red apart) once I’ve obtainedoptimal lighting and viewing conditions.14 The computationalist theory of phenomenal consciousness thus ends up looking like a spoil-sport’s explanation of a magic trick. It comes down to: “Don’t look over there! The keymove is over here, where you weren’t looking!”15 Phenomenal consciousness is not part of themechanism of perception, but part of the mechanism of introspection about perception. It is easy to think that this theory is similar to Perlis’s model of self-consciousness asultimately fundamental, and many philosophers have misread it that way. That’s why “self-consciousness” is so misleading. Ordinarily what we mean by it is consciousness of self. Butthe self-model theory of consciousness aims to explain all phenomenal consciousness in termsof subpersonal modelling by an organism R of R’s own perceptual system. Consciousness ofself is just a particular sort of phenomenal consciousness, so the theory aims to explain it interms of modelling by R of R’s own perceptual system in the act of perceiving R. In these lasttwo sentences the word “self” does not appear except as part of the deﬁniendum, not as part ofthe deﬁniens. Whatever the “self” is, it is not lying around waiting to be perceived; the act ofmodelling it deﬁnes what it is to a great extent. There is nothing mystical going on here. WhenR’s only view of R is R∗, in Minsky’s terminology, then it is no surprise if terms occur in R∗whose meaning depends at least partly on how R∗ ﬁts into everything else R is doing, and inparticular on how (the natural-language equivalents of those) terms are used by a communityof organisms R belongs to. I think the hardest part of this theory to accept is that perception is normally not mediated,or even accompanied, by qualia. In section 1 I invited readers to cast their eyes over a complexclimate-control system and observe the absence of sensation. We can do the same exercise withthe brain, with the same result. It just doesn’t need sensations in order to do its job. But ifyou ask it, it will claim it does. A quale exists only when you look for it. 14 One may view it as a bug that a concept, qualia, whose function is to end introspective questioning hasstimulated so much conversation! Perhaps if human evolution goes on long enough natural selection will eliminatethose of us who persist in talking about such things, especially while crossing busy streets. 15 Cf (Wittgenstein, 1953): “The decisive movement in the conjuring trick has been made, and it was the veryone we thought quite innocent.” 29