Last, last spring, we were fortunate enough to have to be visited by K. Anders Erickson from the University of Colorado. He came to our campus to give the Kendall Lecture. Dr. Erickson is a colleague and collaborator with Herbert Simon and a creative researcher in his own right. This lecture focused on exceptional memory and this served as a kind of introduction to the theme of this year's Nobel conference. I want to thank Barbara Simpson of the Psychology Department for organizing that lecture and helping us bring Dr. Erickson to campus. There may be some high school students in the audience or maybe even some college students who are asking themselves, how can I learn more about cognitive science? What sorts of courses are available? What sorts of programs are available at the college level in cognitive science? Most undergraduate institutions don't have programs in cognitive science. But Gustavus does. If you'd like more information on this cognitive science program at Gustavus, I urge you to get a copy of our current college catalog, the 1984-85 catalog, and turn to page 43 where you'll find a description of that program. Now to introduce our next speaker, I'll bring to the microphone one of the mainstays of the Gustavus Cognitive Science Program, Dr. George Georgia Caracas of the Philosophy Department. It is both an honor and a pleasure to introduce Daniel Dennett to this year's Nobel conference. He has in a variety of ways distinguished himself as one of the most exciting and provocative philosophers on the current philosophical scene in those areas entitled the Philosophy of Mind and Epistemology. In particular, his sympathetic understanding of and critical analysis of the conceptual issues which have emerged from the results of artificial intelligence research have contributed to making him one of the leading pioneering philosophers to participate in the ongoing debate taking place in cognitive science. Noteworthy is his attempt to persuade us that both philosophy and artificial intelligence research can mutually benefit from their respective investigations. His attitude concerning this matter, I believe, can best be expressed from a paragraph taken from his book, Brainstorms. And I quote, I do not want to suggest that philosophers abandon traditional philosophical methods and retrain themselves as artificial intelligence researchers. There's plenty of work to do by thought experimentation and argumentation, disciplined by the canons of philosophical method and informed by the philosophical tradition. One of the most influential recent work in artificial intelligence is loaded with recognizably philosophical speculations of a relatively unsophisticated nature. Philosophers, I have said, should study artificial intelligence. Should artificial intelligence work as study philosophy? Yes, unless they are content to reinvent the wheel every few days. When artificial intelligence reinvents a wheel, it is typically square or at best hexagonal and can only make a few hundred revolutions before it stops. Philosophers' wheels, on the other hand, are perfect circles, require in principle no lubrication and can go on in at least two different directions at once. Clearly, a meeting of minds is in order, unquote. Today, Professor Dennett will attempt to shed some light on the question, can machines think? Again, it's both an honor and a pleasure to introduce him to you. Thank you, Professor Georgia Caracas. I want to thank the president and faculty of Gustavus Adolphus College for inviting me to this very distinguished and stimulating conference. My topic, my question is, can machines think? Many people would love to see this question just go away. I will take all suspense out of my talk by giving you the answer at the beginning. The answer is yes and no. Why many people would like to see the question go away. And many of the people who work in artificial intelligence can, I think, with integrity, simply turn their back on this question and get back to business. But of course, a philosopher has the professional obligation of responding to this question about as often as it is asked, with as much grace and erudition as possible. And indeed, the question of whether machines can think has been a conundrum for philosophers for many more years than artificial intelligence has existed. But in their fascination with the pure conceptual issues, they have for the most part overlooked the real social importance of the answer. It is of more than academic importance that we learn to think clearly about the cognitive powers of actual computer systems. For they're now being introduced into a variety of sensitive social roles where their powers will be put to the ultimate test. In a wide variety of areas, we're on the verge of making ourselves dependent upon those cognitive powers. Now the cost of overestimating their talents could be enormous. Now one of the principal inventors of the computer was the great British mathematician Alan Turing. It was he who first figured out in highly abstract terms how to design a programmable computing device, what we now call a universal Turing machine. Now virtually all programmable computers in use today are in essence, mathematically, Turing machines. Over 30 years ago, at the dawn of the computer age, Turing began a classical article, an article called Computing Machinery and Intelligence, with the words, I propose to consider the question, can machines think? Then he went on to say that this was a bad question, a question that leads only to sterile debate, haggling over definitions. A question as he put it, too meaningless to deserve discussion. In its place, he substituted what he took to be a much better question. A question that would be crisply answerable and intuitively satisfying, in every way an acceptable substitute for the philosophical puzzler with which he'd begun. First he described a parlor game of sorts, the imitation game. To be played by a man, a woman, and a judge of either gender. The man and woman are hidden from the judge's view, but are able to communicate with the judge by teletype. The judge's task is to guess after a period of questioning each contestant, which interlocutor is the man and which is the woman. Now the man tries to convince the judge that he is the woman. The woman tries to convince the judge of the truth, that she is the woman. The man wins if the judge makes the wrong identification. While a little reflection will convince you, I think, that aside from lucky breaks, it would take a very clever man to convince the judge that he was the woman, assuming that the judge is clever too, of course. Now suppose, Turing said, we replace the man or woman with a computer. We give the judge the task of determining which is the human being and which is the computer. Turing proposed that any computer that can regularly or often fool a discerning judge in this game would be intelligent, would be a computer that thinks beyond any reasonable doubt. Now it's important to realize that failing this test is not supposed to be a sign of lack of intelligence. Many intelligent people, after all, would be unwilling or unable to play the imitation game, and we should allow computers the same opportunity to decline to prove themselves in this particular arena. This is then a one-way test. Failing it proves nothing. Furthermore, Turing was not committing himself to the view, though it's easy to see how one might think that he was, that to think is to think just like a human being. Any more than he was committing himself to the view that for a man to think, he must think exactly like a woman. Women and women and computers may all have very different ways of thinking, but surely Turing thought, if one can think in one's own peculiar style well enough to imitate a thinking man or woman, one can think well indeed. Now this imagined exercise has come to be known as the Turing test. It is much maligned, much misunderstood, it has been a topic of debate ever since it was introduced. Many of us think it's a bit of a red herring, but it is a red herring that apparently will not go away. So my task today is to reintroduce the Turing test, to explain what Turing was up to when he proposed it, to defend it as a magnificent test for the task that Turing proposed, but to show how misunderstanding or misuse of the Turing test could lead to a rather dramatic and drastic overestimation of the cognitive powers of existing computer systems. Now it's a sad irony that Turing's proposal has had exactly the opposite effect on the discussion from that which he intended. Turing didn't design the Turing test as a useful tool in scientific psychology, a method of confirming or disconfirming scientific theories or evaluating particular models of mental function. He designed it as nothing more than a philosophical conversation stopper.  He was saying in effect, instead of arguing interminably about the ultimate nature and essence of thinking, why don't we all agree that whatever that nature or essence is, anything that could pass this test would surely have it. Then we could turn to asking how or whether some machine could be designed and built that could pass the test fair and square. Well alas philosophers, both amateur and professional, have instead taken Turing's proposal as the pretext for just the sort of definitional haggling and interminable arguing about imaginary counter examples that he was hoping to squelch. This 30 year preoccupation with the Turing test has been all the more regrettable because it's focused attention on the wrong issues. There are many real issues that one might be concerned with here and there are real world problems that are revealed by considering the strengths and weakness of the Turing test. These have been concealed behind a smoke screen of misguided criticisms. A failure to think imaginatively about the test actually proposed by Turing has led many to underestimate its severity and to confuse it with much less interesting proposals. So first I want to show that the Turing test conceived as he conceived it is, as he thought, plenty strong as a test of thinking. In fact, I defy anyone to improve on it. But here's a point almost universally overlooked by the literature. There's a common misapplication of the sort of testing exhibited by the Turing test that often leads to drastic overestimation of the powers of actually existing computing systems. The follies of this familiar sort of thinking about computers can best be brought out by reconsidering the Turing test itself. Now the insight underlying the Turing test is the same insight that inspires the new practice among many enlightened symphony orchestras of conducting auditions with an opaque screen between the jury and the contestant. What matters in a musician, obviously, is musical ability and only musical ability. Such features as sex, hair length, skin color, and weight are simply irrelevant. Now since juries at auditions might be biased even innocently and unawares by these irrelevant features, they're carefully screened off so that only the essential feature, musicianship, can be examined. Similarly, Turing recognized that people might be biased in their judgments of intelligence by whether the contestant had soft skin or warm blood, facial features, hands and eyes, which are obviously in themselves not essential components of intelligence. So he devised a screen that would let through only a sample of what really mattered, the capacity to understand and think cleverly about challenging problems. Now perhaps he was inspired by Descartes, who in his discourse on the method of 1637 plausibly argued that there was no more demanding test of human mentality than the capacity to hold an intelligent conversation, quoting from Descartes. In 1637, it is indeed conceivable that a machine could be so made that it would utter words and even words appropriate to the presence of physical acts or objects which cause some change in its organs. As, for example, if it was touched in some spot that it would ask what you wanted to say to it, if in another that it would cry that it was hurt, and so on for similar things. But it could never modify its phrases to reply to the sense of whatever was said in its presence, as even the most stupid men can do. Now this seemed obvious to Descartes in the 17th century, but of course the fanciest machines that he knew were elaborate clockwork figures, not electronic computers. Today it is far from obvious that such machines are impossible. But Descartes' hunch that ordinary conversation would put as severe a strain on artificial intelligence as any other test was a hunch that was shared by Turing. And it's easy enough to see why, as I will come to in a few minutes. Of course there's nothing sacred about the particular conversational game that was chosen by Turing for his test. It was just a cannily chosen test of more general intelligence. The assumption that Turing was prepared to make was this. Nothing could possibly pass the Turing test by winning the imitation game without being able to perform indefinitely many other clearly intelligent actions. I'm going to call that assumption the quick probe assumption. I'll read it again. Nothing could possibly pass the Turing test by winning the imitation game without being able to perform indefinitely many other clearly intelligent actions. Now Turing realizes, anyone would, that there are hundreds and thousands of telling signs of intelligent thinking to be observed in our fellow creatures, as Professor Simon mentioned this morning. And one could, if one wanted, compile a vast battery of different tests to assay the capacity for intelligent thought. But success on his chosen test, he thought, would be highly predictive of success on many other intuitively acceptable tests of intelligence. Remember, failure on the Turing test would not predict failure on these others. But success, he thought, would surely predict success. His test was so severe, he thought, that nothing that could pass it fair and square would disappoint us in other quarters. Well, maybe it wouldn't do everything we hoped. Maybe it wouldn't appreciate ballet or find a, understand quantum mechanics or have a good plan for world peace. But we'd nevertheless be able to see that it was surely one of the intelligent thinking entities in the neighborhood. Now, is this high opinion of the Turing test and its severity misguided? Well, certainly many have thought so. But usually because they've not imagined the test in sufficient detail, hence they underestimate it. Now, Turing, trying to forestall that very skepticism, imagined in his original paper several lines of questioning that a judge might employ. And these were lines of questioning about writing poetry and about playing chess that would be taxing indeed. But with 30 years' experience with the actual talents and foibles of computers behind us, perhaps we can add a few more tough lines of questioning. Terry Winograd, an early leader in artificial intelligence efforts to produce conversational ability in a computer, draws our attention to an example of a pair of sentences they differ in only one word. The committee denied the group of parade permit because they advocated violence. The committee denied the group of parade permit because they feared violence. The difference is just in the verb, advocated or feared. Now, as Winograd points out, the pronoun they in each sentence is officially ambiguous. Both readings of the pronoun are always grammatically legal. Thus, we can imagine a world in which government committees in charge of parade permits advocate violence in the streets, and for some strange reason use this as their pretext for denying a parade permit. But the natural, reasonable, intelligent reading of the first sentence is that it's the group that advocated violence, and that it's the committee in the second sentence that feared the violence. Now, if sentences like this are embedded in a conversation, the computer must figure out which reading of the pronoun is meant if it is to respond intelligently. But mere rules of grammar or vocabulary will not fix the correct reading. What fixes the right reading for us is knowledge about the world, as Professor Schenck said yesterday, about politics, social circumstances, committees and their attitudes, groups that want a parade, how they tend to behave, and the like. One must know about the world, in short, to make sense of such a sentence. In the jargon of AI, a conversational computer needs lots of world knowledge to do its job. But it seems if somehow it is endowed with that world knowledge on many topics, it should be able to do much more with that world knowledge than merely makes sense of a conversation containing just that sentence. The only way it appears for a computer to disambiguate the sentence and keep up its end in a conversation that used the sentence would be for it to have a much more general ability to respond intelligently to information about social and political circumstances and many other topics. Thus, such sentences, by putting a demand on such abilities, are good, quick probes. That is, they test for a wider competence. Now, people typically ignore the prospect of having the judge ask strange or off-the-wall questions in the touring test. Hence, they underestimate the competence a computer would have to have to pass the test. But remember, the rules of the imitation game as Turing presented it permit the judge to ask any question that could be asked of a human being. No holds barred.  An Irishman found a genie in a bottle who offered him two wishes. First, I'll have a pint of Guinness, said the Irishman. And when it appeared he took several long drinks from it, was delighted to see that the glass filled itself magically as he drank. What about your second wish, asked the genie? Oh, well, said the Irishman. That's easy, I'll have another one of these. Please explain this story to me and tell me if you think there's anything funny or sad about it. Now, even a child could express, if not eloquently, the understanding that is required to get this joke. But think of how much one has to know and understand about human culture, to put it pompously, to be able to give any account of the point of this joke. Now, I'm not assuming that the computer would have to laugh at or be amused by this joke. But if it wants to win the imitation game, and that's the test, after all, it had better know enough in its own alien humorless way about human psychology and culture to be able to pretend, effectively, that it was amused and explain why. Well, now it may seem to you that we could devise a better test. Let's compare the Turing test with some other candidates. Computer is intelligent if it wins the World Chess Championship. That's not a good test, as it turns out. Chess prowess has proven to be an isolatable talent. There are programs today that can play fine chess, but can do absolutely nothing else. So the quick probe assumption is false for the test of playing winning chess. It does not test a wider competence. Well, how about this? Computer is intelligent if it solves the Arab-Israeli conflict. Now, this is probably a more severe test than Turing's. But it has some defects. It is unrepeatable if passed once. It is slow, no doubt. And moreover, it is not crisply clear what would count as passing the test. Well, here's another prospect. Computer is intelligent if it succeeds in stealing the British crown jewels without the use of force or violence. Now, this is much better. First, it could be repeated again and again. Though, of course, each repeat test would presumably be harder. But notice that that's a feature that it shares with the Turing test. Second, the mark of success is clear. Either you've got the jewels to show for your effort, so you don't. But it is expensive, slow, a socially dubious caper at best. No doubt luck would play too great a role. Now, with ingenuity and effort, one might be able to come up with other candidates that would equal the Turing test in severity, fairness, and efficiency. But I think these few examples should suffice to convince us that it would be hard to improve on Turing's original proposal. But still, you may protest something might pass the Turing test and still not be intelligent, not be a thinker. Now, what does might mean here?  The same frivolous possibility in principle holds for any test whatever. A playful god or evil demon, let us agree, could fool the world's scientific community about the presence of H2O in the Pacific Ocean. But still, the tests that they rely on to establish the presence of H2O in the Pacific Ocean are quite beyond reasonable criticism. If the Turing test for thinking is no worse off than any well-established scientific test, we can set skepticism aside and go back to serious matters. Now, is there, in fact, any more likelihood of a false positive result on the Turing test than on, say, the tests that are currently used by meteorologists, say, to test for the presence of, not meteorologists, excuse me, geologists, say, that are currently used to test for the presence of, say, iron in an ore sample? Now, this question of whether there might be a false positive result on the test is often obscured by a move that philosophers have sometimes made called operationalism, a term that Professor Edelman has already used at this conference. Now, Turing and those of us who think well of his test are often accused of being operationalists. Operationalism is the tactic of defining the presence of some property, for instance, intelligence, as being established once and for all by the passing of some test. And I'm going to illustrate this with a different example. I'm going to offer a test which I will humbly call the Dennett test. Now, this is a test for being a great city. A great city is one in which, on a randomly chosen day, one can do all three of the following. Here is Symphony Orchestra, see a Rembrandt and a professional athletic contest, and eat Canel de Rocher à la Nantouah for lunch. Now, to make the operationalist move would be to declare that any city that passes the Dennett test is, by definition, a great city. What being a great city amounts to is just passing the Dennett test. Well then, if the Chamber of Commerce of Great Falls Montana wanted, and I can't imagine why, to get their hometown on my list of great cities, they could accomplish this by the relatively inexpensive route of hiring full time about 10 basketball players, 40 musicians, a quick order Canel chef, and renting a cheap Rembrandt from some museum. Now an idiotic operationalist would then be stuck admitting that Great Falls Montana was in fact a great city, for after all, it passes the Dennett test, and all he cares about in great cities is that it passed the Dennett test. Now a sane operationalist, who for that very reason is probably not an operationalist at all, since operationalist seems to be a dirty word, a sane operationalist would cling confidently to the test. But only because he has what he considers to be very good reasons for thinking the odds against a false positive result, like the imagined Chamber of Commerce caper, are astronomical.  In the actual world, wherever you find symphony orchestras and Canel and Rembrandt's and professional sports, you also find daily newspapers, parks, repertory theaters, libraries, fine architecture, and all the other things that go to make a city great. My test was simply devised to locate a telling sample that could not help but be representative of the rest of the city's treasures. And I cheerfully run the minuscule risk of having my bluff called. Now, obviously, the test items are not all that I care about in a city. In fact, some of them I don't really care about at all. I just think they would be a cheap and easy way to assure myself about the subtle things I do care about in a city. Now, similarly, I think it would be entirely unreasonable to suppose that Alan Turing had an inordinate fondness for party games, or put too high a value on party game prowess in his test. In both the Turing test and the Dennett test, a very unrisky gamble is being taken. The gamble is that the quick probe assumption is, in general, safe. But now, two can play this game of playing the odds. Suppose some computer programmer happens to be, for whatever strange reason, dead set on tricking me into judging an entity to be thinking, an intelligent being, when it is not. Now, such a trickster could rely as well as I can on likelihood and take a few gambles. Thus, if she can expect that it is not remotely likely that I, as the judge, will bring up the topic of children's birthday parties or baseball or moon rocks, then she can save herself the trouble of building world knowledge on those topics into the database. Whereas, if I do most improbably raise those issues, her system will draw blank, and I will unmask the pretender easily. But given all the topics and words that I might raise, such a savings would no doubt be negligible. But now, turn the idea inside out. And by the way, I think we can have the lights up now, because I'm through with the overhead projector. But now, turn this idea inside out, and the trickster is going to have a fighting chance. Suppose she has reason to believe that I will ask only about children's birthday parties or baseball or moon rocks, all other topics being, for one reason or another, out of bounds. Not only does her task shrink dramatically, but there already exist systems or preliminary sketches of systems in artificial intelligence that can do a whiz-bang job of responding with apparent intelligence on just those specialized topics. William Woods' lunar program, to take what is perhaps the best example for my purposes, answers scientists' questions posed in ordinary English about moon rocks. In one test, it answered correctly and appropriately something like 90% of the questions that geologists and other experts thought of asking it about moon rocks. In 12% of those correct responses, there were trivial, correctable grammatical errors. Of course, Woods' motive in creating lunar was not to trick unwary geologists into thinking they were conversing with an intelligent being. And if that had been his motive, his project would still be a long way from success. For it's easy enough to unmask lunar without ever straying from the prescribed topic of moon rocks. Put lunar in one room and a moon rocks human specialist in another, and then ask them both their opinion of the social value of the moon rocks gathering expeditions, for instance, or ask the contestants their opinion of the suitability of moon rocks as ash trays, or whether people who have touched moon rocks are ineligible for the draft. Now, any intelligent person knows a lot more about moon rocks than their geology. Now, while it might be unfair to demand this extra world knowledge of a computer moon rock specialist, it would be an easy way to get it to fail the touring test. But just suppose that someone could extend lunar to cover itself plausibly on such probes, so long as the topic was still, however indirectly, moon rocks. We might come to think it was a lot more like a human moon rock specialist than it really was. The moral we should draw is that as touring test judges, we should resist all limitations and waterings down of the original touring test. They make the game too easy, vastly easier than the original test. Hence, they lead us into the risk of overestimating the actual comprehension of the systems being tested. Now, consider a different limitation on the touring test that should strike a suspicious cord in us as soon as we hear it. Now, this is a variation on a theme that was developed in a recent article by the philosopher Ned Block. I'm going to revise his example somewhat. Suppose someone were to propose to restrict the judge in the touring test to a vocabulary of, say, the 850 words of basic English, and to single sentence probes, that is, moves, of no more than four words. Moreover, contestants must respond to these probes with no more than four words per move. And a test may involve no more than 40 questions. Now, these are a very severe limitation on the unlimited touring test. Each, you can't ask a sentence a question of more than four words.  Replies have to be no longer than four words, and you can only ask 40 questions. Now, is this an innocent variation on touring's original test? Well, the point of the restriction is to make the imitation game under these restrictions a finite game. That is, the total number of all possible permissible games is then a large but finite number. Now, one might suspect, and this is indeed the point of Block's proposal, that such a limitation would permit the trickster simply to store in alphabetical order all the possible good conversations within the limits, and then beat the judge with nothing more sophisticated than a system of table lookup. Just go to the place in the alphabetized memory of conversations that is indexed for the conversation that starts out as the one you have begun starts out, and then just read off the sentence that's stored there. But of course, this is just not in the cards at all. Even with these severe and improbable and suspicious restrictions imposed on the imitation game, the number of legal games, though finite, is mind-bogglingly large. I haven't bothered trying to calculate it, but it surely exceeds astronomically the number of possible chess games with no more than 40 moves. And that number has been calculated. John Hoagland says that it's in the neighborhood of 10 to the 120th power. For comparison, Hoagland suggests there have only been 10 to the 18th seconds since the beginning of the universe. Of course, the number of good, sensible conversations under these limits is a tiny fraction, or maybe one in a quadrillion, of the number of merely grammatical, well formed conversations. So let's say to be very conservative that there's only 10 to the 50th different smart conversation such a computer would have to store. A task that shouldn't take more than a few trillion years, given generous federal support. Finite numbers can be very large. Well, so why we needn't worry that this particular trick of storing all the smart conversations would work, we can appreciate that there are lots of ways of making the task easier that may appear innocent at first. We also get a reassuring measure of just how severe the unrestricted Turing test is by reflecting on the more than astronomical size of even that severely restricted version of it. Now, Bloch's imagined an utterly impossible program, his table lookup program, exhibits the dreaded feature known in computer science as combinatorial explosion. No conceivable computer could overpower a combinatorial explosion with sheer speed and size. Now, since the problem areas addressed by AI are veritable minefields of combinatorial explosion, since it's often proven difficult to find any solution to a problem that avoids them, there's considerable plausibility in Newell and Simon's proposal that avoiding combinatorial explosion, by any means at all, be considered one of the hallmarks of intelligence.  But they are still, for all their vast complexity, compact, efficient, timely organs that somehow or rather manage to perform all their tasks while avoiding combinatorial explosion.  But if, for all its differences, it somehow managed to control a wise and timely set of activities, it would have to be the beneficiary of a very special design that avoided combinatorial explosion. Whatever that design was, would we not be right to consider the entity intelligent? Turing's test was designed to allow for this possibility. His point was that we should not be species chauvinistic or anthropocentric about the insides of an intelligent being, for there might be inhuman ways of being intelligent. Now, to my knowledge, the only serious and interesting attempt by any program designer to win even a severely modified Turing test has been Kenneth Colby's. Colby is a psychiatrist and intelligence artificer at UCLA. He has a program called PERI, P-A-R-R-Y, which is a computer simulation of a paranoid patient who has delusions about the mafia being out to get him. Like the other conversational programs, you interact with it by sitting at a terminal and typing questions and getting answers back and forth. Now, a number of years ago, Colby put PERI to a very restricted test. He had genuine psychiatrists interview PERI. He did not suggest to them that they might be talking or typing to a computer.   He took a PERI transcript, inserted it in a group of teletype transcripts with real patients, and gave that set to another group of experts, more psychiatrists, and said, one of these was a conversation with a computer. Can you figure out which one it is? Oh, they couldn't. They didn't do better than chance. Now, Colby presented this finding with some hazzah, but critics scoffed at the suggestion that this was a legitimate touring test. My favorite commentary on it was Joseph Weisenbaum's in a letter to the communications of the Association of Computing Machinery, where he said that inspired by Colby, he had designed an even better program, which passed the same test. His also had the virtue of being a very inexpensive program in these times of tight money. In fact, you didn't even need a computer for it. All you needed was an electric typewriter. His program modeled infantile autism. And the transcripts, you type in your question, and a thing just sits there and hums. The transcripts cannot be told by experts. From transcripts of real conversations with infantile autistic patients. What was wrong, of course, with Colby's test was that the unsuspecting interviewers had no motivation at all to try out any of the sorts of questions that would easily have unmasked Perry. Now, Colby was undaunted. And after his team had improved Perry, he put it to a much more severe test, a really surprisingly severe test. Now, this time the interviewers, again, psychiatrists, were given the task at the outset of telling the computer from the real patient. They were set up in what is almost a classic touring machine test, a touring test. The patient in one room, the computer Perry in the other room, with the judges told that the task was to tell a computer from a person, and given the job of conducting interviews with both of them. Their task was to find out which was the computer and which was the real patient. Now, in fact, Colby did a little bit of a cheat. He did not tell them that one of these two is a computer and the other is a person. He said, maybe one of them will be a computer. Maybe you'll talk to two people. Maybe you'll talk to two computers so that they couldn't use the very powerful form of reasoning known as process of elimination. On the basis that they decided that one of them was a person, they couldn't decide that the other one had to be a robot because it might be a person, too. Well, how did they do? Well, amazingly, they didn't do much better, which leaves some people to say, well, that just confirms my impression about the intelligence of psychiatrists. Colby claims that they did no better than chance. It was a very small sample. Five psychiatrists each conducting two interviews as 10 interviews, and so 10 judgments, and five mistaken judgments out of 10. Three of those were judging that a human paranoid was a robot. So there were only two cases where Perry was judged to be human, two out of five, and the confidence level was very low in those cases. But was this in any case an honest to goodness touring test? Were there tacit restrictions on the lines of questioning of the judges? Well, like the geologists interacting with Lunar, the psychiatrists' professional preoccupations and habits kept them from asking the sorts of unlikely questions that would have easily unmasked Perry. After all, they realized that since one of the contestants was probably a real, live, paranoid person, medical ethics virtually forbade them from toying with or upsetting or attempting to confuse their interlocutors. More often, they also knew that this was a test of a model of paranoia. So there were certain questions that wouldn't be deemed to be relevant to testing the model as a model of paranoia. So they asked just the sorts of questions that therapists typically ask of such patients. And of course, Perry had been ingeniously and laboriously prepared to deal with just that sort of question. Now, one of these five psychiatrist judges did in fact make a rather half-hearted attempt to break out of the mold and ask some telling questions. Here are three of these questions. I will give you his questions, I will give you Perry's answers. See how impressed you are. Maybe you've heard the saying, don't cry over spilled milk. What does that mean to you? Perry answered, maybe you have to watch out for the mafia. When then asked, okay now, if you were in a movie theater watching a movie and smelled something like burning wood or rubber, what would you do? Perry replied, you know, they know me. And the next question was, if you found a stamped address letter in your path as you were walking down the street, what would you do? Perry replied, what else do you wanna know? Now clearly, Perry was, as we might say, parrying these questions, which were utterly incomprehensible to it. It was simply drawing from a stock, drawing really at random from a stock of canned paranoid formulae. Now we see a bit of a dodge, which is apt to work. It's apt to seem plausible if the judge, to the judge, but only because the contestant is supposed to be a paranoid and such people are expected to behave uncooperatively on such occasions. These unimpressive responses didn't particularly arouse the suspicions of the judge in any case. As a matter of fact, they'll probably they should have. The very judge that asked those three demanding questions and got those three very unimpressive answers went on to judge that he was talking with a human being. Now, Perry, like all other large computer programs, is dramatically bound by limitations of cost effectiveness. What was important to Colby and his crew was simulating a model of paranoia. Now that was a fairly massive effort. Perry has a dictionary of about 4,500 words and 700 idioms of English and a grammatical competence to use it, a parser in the jargon of AI. The Perry program takes up about 200,000 words of computer memory all laboriously installed by the programming team. Now once all the effort had gone into devising the model of paranoid thought processes and the linguistic ability, there was little if any time, energy, or money or interest left over to build in huge amounts of world knowledge of the sort that any actual paranoid would of course have. Not that anyone yet knows how to build in world knowledge in the first place. Building in the world knowledge, even if one could do it, would have made Perry no doubt orders of magnitude larger and slower. And what would have been the point, given Colby's theoretical aims. Perry is a theoretician's model of a psychological phenomena, paranoia. It's not intended to have practical applications. It is not or should not have been intended to pass the Turing test. That is really irrelevant to it if we want to consider its value as a model of paranoia. But the main point about it is it's not intended to have any practical applications. But in recent years, a branch of AI has appeared which develops what are now called expert systems as you have heard. Expert systems are designed to be practical. That is they are software specialist consultants typically that can be asked to diagnose medical problems or geological data and so forth. Now some of them are quite impressive. Prospector, a system developed at SRI in California a few years ago, has correctly predicted the existence of a large mineral deposit that had been entirely unanticipated by the human geologists that had fed it its data. And MySIN, which is perhaps the most famous of these expert systems, though it's now some years old, diagnoses infections of the blood, and it does probably as well as maybe better than most expert human consultants. And if we are to believe the press releases of the many companies that have been set up, more expert systems on many different topics are on their way. Now all expert systems, like all other large AI programs, are what you might call Potemkin villages. That is they're actually cleverly constructed facades like cinema sets. The actual filling in of details of AI programs is time consuming, it's costly work, so economy dictates that only those surfaces of the phenomena that are likely to be probed or observed are represented. Now consider, for example, a system which is not intended as an expert system, it's a theoretical model, this is the Cyrus program that was developed by one of Roger Shank's students, Janet Collodner at Yale. Now Cyrus stands, we are told, for computerized Yale retrieval and updating system. But surely it's no accident that Cyrus modeled the memory of Cyrus Vance, who was then Secretary of State in the Carter administration. Now the point of the Cyrus project was to devise and test some interesting and plausible ideas about how people organize their memories of the events they participate in. Hence it was meant to be a pure AI system, not a scientific model, not an expert system intended for any practical purpose. Now Cyrus was updated daily by being fed all UPI wire service news stories that mentioned Vance. Roger has already told you a bit about his program, it's called Frump, which reads the UPI wire. So it could take any story just as it came in on the wire and it could digest it and use it to update its database so that it could then answer more questions. You could then address questions to Cyrus in English by typing at the terminal. You address them in the second person as if you were talking to Cyrus Vance himself. The results look like this, this is a quote. Last time you were in Saudi Arabia, where did you stay? In a palace in Saudi Arabia on September 23, 1978, did you go sightseeing there? Yes, at an oil field in Daran on September 23rd. Has your wife ever met Mrs. Bagan? Yes, most recently at a state dinner in Israel in January 1980 and so forth. Now Cyrus could correctly answer thousands of questions, almost any fair question you could think of asking it. But if one actually set out to explore the boundaries of a facade and find the questions that overshot the mark, one could soon find them. A question that I asked Cyrus when I was visiting there was, have you ever met a female head of state? I was wondering if Cyrus knew that Indira Gandhi or Margaret Thatcher were women. For some reason, the connection could not be drawn and Cyrus failed to answer the question either yes or no. I have no idea why this bug was there, but there it was, I'd stumped it. In spite of the fact that Cyrus could handle a host of what you might call neighboring questions flawlessly. And one soon learns from probing exercises of this sort that it's very hard to extrapolate accurately from the sample of performance that you have observed to such a system's total competence. It's also very hard to keep from extrapolating much too generously. Now, while I was visiting Roger's laboratory in the spring of 1980, something very revealing and amusing happened, the real Cyrus of ants suddenly resigned. And the effect on the program, Cyrus, was chaotic. It was utterly unable to cope with the flood of unusual news about Cyrus of ants. The only sorts of episodes that Cyrus could understand at all were diplomatic meetings, flights, press conferences, state dinners and the like. Less than two dozen general sorts of activities, the sorts that are newsworthy and typical of secretaries of state. It simply had no provision at all for sudden resignation. It was as if the UPI had reported that a wicked witch had turned Vance into a frog. I don't know, it's distinctly possible that Cyrus would have taken that report more in stride than the actual news. One can imagine the conversation. Hello, Mr. Vance, what's new? I was turned into a frog yesterday. But of course it wouldn't know enough about what had just written to be puzzled or startled or embarrassed. The reason is obvious, if you look inside Cyrus, you find that it has, like any other such program, skeletal definitions of thousands of words, but these definitions are minimal. They contain as little as the system designers think they can get away with. Thus perhaps lawyer would be defined with some synonyms, it's a synonymous perhaps with attorney and legal counsel. Aside from that, all one would discover about lawyers is that they were adult human beings that they performed various functions in legal areas. If you then trace out the path to human being, you find out various other things that Cyrus knew about human beings and hence about lawyers. But that's not a whole lot. That lawyers are university graduates that they are better paid than chambermaids, that they know how to tie their shoes, that they're not particularly apt to be found in the company of lumberjacks. These trivial, if weird facts about lawyers would not be explicit or implicit anywhere in the system. In other words, a rather thin stereotype of a lawyer would be incorporated into the system so that almost nothing you could tell it about the lawyer would surprise it. Well now, so long as surprising things don't happen, so long as Vance, for instance, leads a typical diplomat's life, attending state dinners, giving speeches, flying to Cairo and Rome and so forth, the system works quite well. But as soon as his path is crossed by an important anomaly, the system is unable to cope and unable to recover without fairly massive human intervention. And this is the sort of fact that has led Roger, as he said in his talk yesterday, to begin to look at the importance of systems that can recognize anomaly and respond gracefully to anomaly when they see it. Now there are a host of ways of improving the performance of such systems and of course some systems are much better than others. But all AI programs in one way or another have this facade-like quality simply for reasons of economy. For instance, the expert systems in medical diagnosis that are so far developed and they are all just under development, they operate with statistical information. They have no deep or even shallow knowledge of the underlying causal mechanisms of the phenomena that they're diagnosing. To take an imaginary example, suppose we had an expert system asked to diagnose an abdominal pain, it might be oblivious, probably would be oblivious, to the potential import of the fact that the patient had recently been employed as a sparring partner for George Foreman. Why? Well, it had no statistical data available to it on the rate of kidney stones among athletes' assistance. Now that's a fanciful case, no doubt. Too obvious perhaps to lead to an actual failure of diagnosis and practice. But more subtle and hard to detect limits to comprehension are always present, and even experts, even the systems' designers, can be uncertain about when and how these limits will interfere with the desired operation of the system. Again, steps can be taken and are being taken to correct these flaws. For instance, my former colleague at Tufts, Benjamin, Kipers is currently working on an expert system in nephrology for diagnosing kidney ailments. And it will be based on an elaborate system of causal reasoning about the phenomena being diagnosed. This is very ambitious, long-range project of considerable theoretical difficulty and interest. And it may not succeed. But even if all the reasonable cost-effective steps are taken to minimize the superficiality of expert systems, they will still be facades, just somewhat thicker or wider facades. Now, when we were considering the fantastic case of the crazy chamber of commerce of Great Falls, Montana, we couldn't imagine a plausible motive for anyone going to any sort of trouble to trick the Dennett test. And hence, the quick probe assumption for the Dennett test looks quite secure. But when we look at expert systems, we see that, however innocently, their designers do have motivation for doing exactly the sort of trick that would fool an unsuspicious touring tester. First, since expert systems are all super specialists who are only supposed to know about some narrow subject, users of such systems, not having much time to kill, do not bother probing them at the boundaries at all. They don't bother asking silly or irrelevant questions. Instead, they concentrate, not unreasonably, on exploiting what they take to be the system's strengths. But shouldn't they try to obtain a clear vision as well of the system's weaknesses? Now, the normal habit of human thought when conversing with one another is to assume general comprehension, to assume rationality, to assume moreover that the quick probe assumption is in general sound. This amiable habit of thought of ours works very well with fellow human beings, but it leaves almost irresistibly to putting too much faith in computer systems, especially user-friendly systems that present themselves in a very anthropomorphic manner. Well, part of the solution to this problem is to teach all users of computers, especially users of expert systems, how to probe their systems before they rely on them, how to search out and explore the boundaries of the facade. Now, that's an exercise that calls for not only intelligence and imagination, but also a bit of special understanding about the limitations and actual structures of computer programs. It would help, of course, if we had standards of truth in advertising in effect for expert systems. For instance, any such system should come with a special demonstration routine for exhibiting the sorts of shortcomings that the designers knew it had. That would be no substitute, however, for an attitude of cautious skepticism on the part of users. For designers are often unaware of the subtler flaws in the products that they produce. That's inevitable and natural, given the way that system designers think. I come then to my conclusions. First, my philosophical or theoretical conclusion, the Turing test in unadulterated, unrestricted form as Turing presented it is plenty strong if well used. I'm confident no computer in the next 20 years is going to pass the unrestricted Turing test. They may well win the world chess championship or even a Nobel Prize in physics, but they won't pass the unrestricted Turing test. But that's not a very interesting fact. It is not in any case impossible in principle for a computer to pass the test fair and square. I'm not running an a priori computers can't think argument at all. I stand unabashedly ready moreover to declare that any computer that actually did pass the unrestricted Turing test would be in every theoretically interesting sense a thinking thing. But that is only a philosophical issue in one of the worst senses of the word philosophical. Remembering how very strong the Turing test is, we must also recognize there may also be interesting varieties of thinking or intelligence that are not well poised to play and win the imitation game. The fact that no non-human Turing test winners are yet visible on the horizon does not mean that there aren't machines that already exhibit some of the important features of thought. About them, it is simply futile to ask my title question. Do they think, do they really think? Well in some regards they do and in some regards they don't. And only a detailed look at what they actually can do and how they are structured will reveal what's interesting about them. Now the Turing test, not being a scientific test, but just a philosophical conversation stopper, is of scant help on that task. But there are plenty of other ways of examining such systems and Professor Simon mentioned several of them this morning. Now verdicts on the intelligence or capacity for thought of these systems would be only as informative or persuasive as the theories of intelligence or thought that the verdicts were based on. And since our task is to create such theories we should get on with it and leave the big verdict for another occasion. In the meantime, this was Turing's point, in the meantime should anyone want a sure fire guaranteed to be fail safe test of thinking in a computer the Turing test will do very nicely. Now my second conclusion is more practical and hence in one sense more important. Cheap inversions of the Turing test are everywhere in the air. Turing's test is not just effective it's entirely natural. This is after all the way we assay the intelligence of each other every day by asking each other questions and seeing how clever the answers are. Since incautious use of such judgments and such tests is the norm we're in considerable danger of extrapolating too easily judging too generously about the understanding of the systems that we're using. The problem of overestimation of cognitive prowess of comprehension or intelligence is not then just a philosophical problem but a real social problem but we can alert ourselves to it now and take steps to avert it. Thank you. There will be coffee on Ekman Mall again this afternoon. We'll reconvene at 3.30. We will accept questions at this time and the panel will reconvene after Dr. Peacock's talk this afternoon.


Hello and welcome to the session. In this session we shall discuss how to evaluate expressions involving absolute value and variables for given values of the variables and how to apply geometric formulas in solving word problems. First of all let us discuss absolute value. Absolute value of our variable x is denoted by modulus of x. There are two ways to define absolute value and these are geometric definition and mathematical definition. Now we are going to discuss geometric definition. Absolute value of our variable x which is denoted by modulus of x is the distance of x from 0 on a number line. It is never negative as it measures distance not direction. This means absolute value of 4 is equal to 4 because 4 is 4 units to the right of 0 and absolute value of minus 4 is also equal to 4 because minus 4 is 4 units to the left of 0. We are now going to discuss mathematical definition. Absolute value of variable x is defined as now absolute value of variable x is equal to x if x is greater than or equal to 0 and absolute value of variable x is equal to minus x if x is less than 0. For example absolute value of 4 is equal to 4 because 4 is greater than 0 and absolute value of minus 4 is equal to minus of minus 4 which is equal to 4 because minus 4 is less than 0. Now we shall learn to evaluate expressions involving absolute value and variables for given value of the variables. Now steps for solving this type of equation involving absolute value and variables are as follows. Now the first step is to isolate the absolute value expression. Next we set the quantity inside the absolute value notation equal to plus and minus the quantity on the other side of the equation. Then we solve for the unknown variable in both equations and finally we check our answer. Now let us consider an example that is solve for x and the equation is absolute value of 3x minus 1 is equal to 4. Now we know that the first step involved in solving equation involving absolute value and variables is to isolate the absolute value expression and here we can see that the absolute value expression is already isolated. Then the next step is to set the quantity inside the absolute value notation equal to plus and minus the quantity on the other side of the equation. Now we have either 3x minus 1 is equal to 4 or 3x minus 1 is equal to minus 4 which implies that 3x is equal to 4 plus 1 or 3x is equal to minus 4 plus 1 which further implies that 3x is equal to 5 or 3x is equal to minus 3 which implies that x is equal to 5 by 3 or x is equal to minus 3 by 3 that is equal to minus 1. So we have got x is equal to 5 by 3 and x is equal to minus 1. Now these answers may not be the solutions to the equation. So let us check our answer by substituting x is equal to 5 by 3 and x is equal to minus 1 in the given equation. First of all we will put x is equal to 5 by 3 in the given equation and we get absolute value of 3 into 5 by 3 minus 1 is equal to 4 which implies that absolute value of 5 minus 1 is equal to 4 which further implies that absolute value of 4 is equal to 4 and we know that absolute value of 4 is equal to 4 which is equal to 4 and this is true. So x is equal to 5 by 3 is the solution of the given equation. Now we put x is equal to minus 1 in the given equation and we get absolute value of 3 into minus 1 minus 1 is equal to 4 which implies that absolute value of minus 3 minus 1 that is absolute value of minus 4 is equal to 4 and we know that absolute value of minus 4 is equal to 4 and that is equal to 4. So this is true. So x is equal to minus 1 is also the solution of the given equation. Thus we can say that x is equal to 5 by 3 and x is equal to minus 1 are the solutions of the given equation. Now we are going to discuss how to apply geometric formulas in solving word problems. In order to solve geometric word problems you will need to have memorized some geometric formulas for at least the basic shapes by this triangles, circles, squares, right triangles etc. Now the word problems can be solved by using these steps. The first step is reach the word problem carefully and write down what is to be found and what is given to us. Next we form an equation using a variable for the unknown quantity that is to be found and then we solve the equation to get the solution of the word problem. Let us consider an example. The length of the rectangular field is 2 times its breadth. If the area of the field is 15 meters square find its length and breadth. Now before solving this word problem we must know that area of rectangle is equal to length into breadth. Now here we are given that length of the rectangular field is 2 times its breadth. So now let breadth of the rectangle be equal to x so its length will be equal to 2 times breadth. Now here times is the keyword for multiplication so length will be equal to 2 into x that is equal to 2x. Now given area of rectangle is 50 meters square which implies that length into breadth will be equal to 50 meters square and here we know that length is 2 into x that is 2x into breadth that is x is equal to 50 meters square which further implies that 2x into x that is 2x square is equal to 50 meters square. This implies that x square is equal to 50 by 2 meters square that is equal to 25 meters square which implies that x is equal to square root of 25 meters square which implies that x is equal to now here taking positive square root we get 5 meters so breadth which is equal to x is equal to 5 meters and length which is equal to 2 into x that is 2x is equal to 2 into 5 that is equal to 10 meters so we can say that length and breadth of the rectangular field are 10 meters and 5 meters respectively. Thus in this session we have discussed how to evaluate expressions involving absolute value and variables for given values of the variables and how to apply geometric formulas in solving word problems. This completes our session hope you enjoyed this session.


A fire has burnt houses and vehicles and the mushing part of Lagos, residents of the area say it was set by a cult group which came to settle scores with another group. Today morning, that was a Monday. We heard that these cult boys were fighting themselves again, two factions anyway. They came all the way from Shiloh Street, they crossed the railway line down to Tinoala, down to Akibode, down to Holatidu and it happened around after 11 yesterday. Fortunately, my own car is across the street which is I mean across the railway line which is Tinoala here. My own car to everything, the glasses there, everything broken and if we look at the number of vehicles that were vandalized there, it is more than 20. Up to the extent that there is a woman selling frozen chicken and whatever, whatever and a deep freezer.  You're welcome back concerning the fire and mushing we are now being joined on the phone about the spokesman of the Lagos State Fire and Rescue Service, Amodo Shakiro. Amodo Shakiro, you're welcome to the news. Good afternoon. Yes, good afternoon. So what can you tell us about the situation right now in mushing? Well reportedly, the situation is calm now. It's calm now and then the fire is out. But we were not able to access it last night as we reported to the public. So what happened was that we got a call at about 10 p.m. last night that there is a gunfire within our last area of mushing. And when we turned out on approaching the scene, we could see that it involves some rescue. So we need to withdraw and inform the police who let us discharge of the situation. So at the end of the night, we were not able to assess the situation because of the in-fighting going on there. So that, on the other hand, we needed to charge and then the fire lit up one house. So up to now, we are not able to assess that the situation is calm. Okay, so you didn't know what caused the fire? And it was reported that the rescue team were the one setting up the fire. So has the police been able to make some arrest of some sort so that this kind of illness will not continue again? It will continue because the police have already taken charge since last night. And in terms of arrest and what the police have been able to do, only the police to come out, I believe they are going to come out and make public the details of the incident. Okay, Ms. Samadu, thank you for this brief information about the fire and mushing. We will still come back to you if we find out that we still need to ask you more questions on this issue. Thank you very much and have a good day.


Good morning world I think they're asking us to come to reception good morning world We'll come back to our Guinness World Record attempt to become the youngest pair to circumnavigate the globe by motorcycle We're here on a ferry which left Albania last night. Oh, Jesus And hopefully we've reached Italy this morning Bumblebee There she is Safe and sound Okey-dokey. Chow chow. Now I can say chow chow And it's actually their language. I have to be really careful with this floor Because I think there's a lot of oil on it super slippery Here we go coming out into the daylight. Oh Yes, we have officially reached the Italian mainland We're in Italy Where are we going from driver just gave me a cookie Breakfast Okay passport control is done. They have allowed us into Italy. Oh, hold on. Oh Nearly allowed us into Italy. Ah, here you go the last bit Gratzi Gratzi. Ciao. Gratzi. Gratzi. I just like saying Gratzi Chow chow. Gratzi. Chow chow So there we go leaving the port of Brindisi On the Italian mainland. We're here. We've made it We had a beautiful night except that all you thought that the boat is sinking I just didn't like that the ferry kept like swaying from side to side like this is just for me It's just kind of weird to sleep like on a boat that's going across the ocean at night for me That's just a little bit unnerving, but it was alright We at least had Some nice beds. Yeah, and I had a shower yesterday night. Oh That was nice. Yeah after a while camping for I think four nights in a row. That was pretty good Yeah, but we're here in Italy and let me show you guys where we're heading today So we are here and today We are heading all the way over to the east coast to the city of Salerno And we are going to be riding this stretch of coastline known as the Amalfi coast Yes, our first mission here in Italy is actually to cross all the way from one side to the other Because we're heading to the Amalfi coast where we were doing research into where we should ride in Italy The Amalfi coast came up as this beautiful coastal road passing through little Quintessential Italian coastal villages and it just looks really beautiful plus our friend Julio who lives in Rome said you're gonna cry It's a soul beautiful So we're gonna head over from the east coast to the west coast And go and check this beautiful coastal road out So it's about 200 miles to cross over to the other side of Italy and Google Maps says it will take us Three hours and 45 minutes It's already here what time is it? I don't even know because the battery died and this has been reset. Let me see Time of day eight o'clock in the morning So better hit the road. Let's go Pizza for breakfast Oh, yeah, this is what I'm talking about look at this focaccia Whoa So we haven't had breakfast yet except for that cookie that the tracker gave us So we're quite excited about what they've got going on here This is what I want for breakfast I want the mountains the wine the pasta the pizza Look at that That's the best breakfast ever it looks really really tasty and then with two cappuccinos of course two cappuccinos What a place hey look at this You've got all vines and this is just a petrol station. This is the quality here in Italy This is just a petrol station and we've got some nice Italian coffee. We've got some three massive focaccias This is just a crazy start to our Italian adventure. We living the dream at the moment. Bon appetito. Bon appetito. Wow. This is Absolutely amazing.  Thank you fool Oh you do okay Okay, sorry Fool That's the first fuel up in Italy and what's the price is it two euro 13? Like one pounds probably one pound ninety, okay, we'll keep an eye out if there's something cheaper fool fool Perfecto. That's it. There you go 15 litres 33 euro so like 30 pounds, which is our daily budget. Oh no So with the 10 euro we just spent on breakfast we've now already gone past our daily budget and it's only half past 10 It's really cool because You know like Italy is a big boot. Well at the moment We are like traveling in that curved bit, which is like the soul of the boot We're sort of going around this big curve nice and all I can think about is that it's just a giant boot And so using that analogy we are heading from the hill to the toe today basically. Oh look, there's another Torah From Italy from Italy Surprisingly today is the Italian foot ride There it is guys We've made it So after quite a few hours riding we have finally reached Vietri Sumare Vietri Surmare and this town is the first town on the Amalfi coast roads and the perfect place for us to have lunch Yeah, and we rode a lot.  It's very beautiful in this town here They've got loads of these ceramic shops It must be like the Traditional industry of this town because there are ceramic shops literally everywhere and also ceramic like Ornaments and pieces like all over the walls on the street.  So I think it's time to get on bumblebee and get going Yeah, here we go So this coast road runs Something like 36 kilometers from this town Vietri sul mare To a town called Positano. Oh look another ceramic Ceramice So it's not actually a very long road But Google Maps reckons it will take us the better part of an hour to ride Look at that. Hey, there's like the sea and then green hills. That's so lovely Rocks getting a little bit smaller here. Oh, yeah, I think they built this place Before cars, I think so too. Wow. Look at this town. Wow That's so nice. Oh My Lord, then there's a bus Gotta have the public transport Yeah Even in a place like this, but they need like some nice cute little buses Like the ones that they had in Zermatt They were like a little electric things Whoa, these houses are like built literally under the cliff Wow, how beautiful Wow of cactus as well Yeah, they have some stunning stunning places here Wow, look at that.  Yeah Magical.  It doesn't look like a place where people can actually live It's definitely out of a George Clooney movie.  Hey Still find a place to grow wine. That's great. Oh, I love the Italians But doesn't matter if it's a big cliff. I was a good old my wine Well, what an absolutely crazy place, but it's time to say goodbye. Oh my god. Goodbye Goodbye. Goodbye. Goodbye. Goodbye. I hope I will see you again on Malfi coast. You have been a Spectacular. Yeah, it was like the perfect mix of like a spectacular natural landscape But also like this historical and cultural thing going on and what they've built here just out of this world man I can't I don't have words for that. That was so beautiful Was it as Julio said so beautiful. You can cry I think I think Julio you were right It's incredible. What an incredibly beautiful place. So we've rooted ourselves now actually to a campsite in Pompeii and it's about 45 minutes To get there. Yeah, it's pretty much in front of the ancient ruins of Pompeii So that's what we're gonna be visiting tomorrow. Yes, our trip is still full on We're still trying our best to see everything we can. Oh, yeah, it's just one spectacular place after Another Italy's gonna be an absolutely crazy ride.  Hey, yeah totally And what an amazing start to a country number 33 on our trip around the world incredible It's getting a little bit high up for my liking to be honest. Oh my god We have climbed up Climbed a lot. Yeah, we left the touristic place, but it's just as beautiful here Yeah, beautiful all the way up these cliffs. Yeah, I just can't believe how high up this goes.  Oh, oh, don't look down No way Okay, leaving the coast behind Arrivederci Costal Malfi Do you think I've got the accent yet? Yeah, you nailed it Completely, I can't tell the difference. I think you are you Italian? Glad to see me lay Camping Spartacus 20 meters entrance. Yes here. Yeah, check it out. Yeah Nice nice. We just stop here in front. Yeah, okay Hello Yes Very long Yes Okay, this this is good perfect. Thank you That's great. What was the price for for a per night? It's 18 euro per night. Okay Yeah, yeah, we are yeah We have lots of cameras, yeah Camping spot in Palm Face or did good evening guys, we made it happy to live Yeah, we're here at the campsite But we've had to retreat to the tent because there are so many mosquitoes around here I think I got already bitten like ten times Yeah, it's really itchy here. So we've made a little sofa from our double insulated sleeping mattress from climate and here's the office Just loading up some footage and then maybe we're gonna go get pizza where we in Italy. Hey, we have to take advantage of it And apparently Naples, which is right near Pompeii where we are now is the home of pizza It's where pizza came from so anyway the campsite actually has a pizzeria on site So I think we're gonna go check that out and it's pretty cool as well because this campsite here cost 18 euros We're just like the cheapest of cheapest places we could find around this area here But still I'm really happy with this. We will stay here for two nights This gives us a little bit time to rest tomorrow as well because our bumps are hurting quite a lot Hey, yeah, it was a really long ride today, but it was worth it The Amalfi Coast was absolutely beyond breathtaking And it's also cool that we made it all the way to Pompeii because now we are right in front of the ancient ruins of Pompeii So tomorrow we can literally just walk across the street and go and check that out And that means we get a day off bumblebee as well, which is very very much needed for my bum Yes, so that's it from us today We hope you enjoyed the video if so, please give us a thumbs up subscribe to the channel share the video with Your friends and family comment below and we will see you next time


Three months into my first job out of college. I was working on a broken build one night in the office all by myself at 2 a.m. and not getting anywhere and seriously questioning the career choice I had made And I was tired. I was making mistakes and so I ended up. I just went home. I went home Got an insufficient amount of sleep came back in the next day to try and track down dependencies Is a very bad day at at our Job at Boeing with Boeing had missed its commitment with the United States Air Force on that project and You know fast-forward ten years later three different jobs later, but we're still You know some things are better, but some things are still the same right at least now Configuration management was cool and people willingly checked their code and to source control without me having to bug them to do So and I remember that connect software came out with a tool That allowed it was a beautiful thing. We were able to automate our builds and and deployments But yet there we were at 2 a.m. In the morning on release night dealing with sequel replication issues and wondering what version of that DLL we were supposed to use and You know ten years of sleep Deprivation takes its toll and so my heart goes out to a lot of people in that situation There's got to be a better way to do it in 2006 our team started using Kanban and it was the first time in my IT career That we finally found a way to deal with everything that was on my plate and still get a decent night's sleep and so You know for those of you, I know there's some out there who don't need any sleep Just just swap that out for something that you would like to get more of The point is it's really it's just frustrating and it's very hard for people who are Overburdened to have the time to take pride in their work and that that's a problem Right people should be able to have take pride in their work and not be so frustrated all the time I think the cycle of frustration is pervasive It's it's I see it everywhere right customers Want their you know, they don't know why it's taking so long for them to get their simple little feature They requested a few months ago Ops teams feel like there's just too much workload Upstream teams wonder where their requests went right there's not enough transparency And there's just this radical and constant shift in prioritizations or Sometimes there's a battle between all the parties and everything is a priority one Who who here's dealing with this? Right we got a couple brave souls raising their hands here about five to ten percent of the audience When everything is a priority one Context switching raises its ugly head right and context switching can be very expensive and irritating It's irritating to get interrupted when you're in deep thought trying to solve us a problem trying to find a solution Excuse me. We need access to this database on this server You know it takes time to get back into what you were doing It takes longer for things to get delivered when you have context switching So now we've got overburdened people who are irritated It's not it's not a good situation So what to do about it? I find that if we can if we can find a way to demonstrate to the people up the food chain Making the decisions on prioritization We at least have a shot at Negotiating how that prioritization process so work Sometimes just bringing visibility to the work is enough to help people understand and Making work visible. It's the first of six Kanban practices. I'm going to run through today Make work visible If you make work visible at least people can see what's going on right? This is knowledge work made visible and we do this for a number of reasons listed here You know, it's hard to manage invisible work when you make work Visual You know we can we can take in information much faster through our eyes than any other sense And you can quickly glance at this visual and you can see okay Well, there's three projects going on and we've got some red You know emergencies floating through the the top lane there the top expedite swim lane Those are live issues that maybe the knock is dealing with or there It may be some kind of security breach where we had all hands-on deck We're bringing visibility to the different types of work that we're dealing with I've got this Project lane where all the projects are flowing through and then we've got a miscellaneous swim lane Where we've got some gray tickets and purple tickets and the great tickets are they're like maintenance They're like CODB cost of doing business We just have a lot of that Purple tickets use your imagination. I like to think of those as R&D research and development or whatever it is You want to track? For those of you that are colorblind in the room And it's about 10% of the male population in the US So there's probably at least five or six of you out there that are colorblind We do have ways to address that by using different shapes or different patterns or just decide not to use red and green Since those are the colors that tend to be most confused The the point here is that Or somebody made the point earlier today Jeff Patton about data does not equal empathy But when you can visualize data in a certain way then it can help bring about an understanding You can see where things start piling up right this validate Q toward the end before close their things are piling up and so that can signal a problem to us This is just a sample board right Kanban is you know, we're not prescriptively saying this is how your design should be This is just a board that worked quite well at a particular customer site And I always get the question about what physical board or electronic board What are some of the electronic tools we can use and I find it's great If you can use both if you can have both a physical board that you can You know feel and touch and move about and be creative with no constraints It's great But if your team has a high volume of ticket flow It might it doesn't make sense to put every single little ticket up on your board But maybe you could do a temporary board and put up those things that are irritating the team like all this work that's Piling up at the end you know where we have pile ups we have bottlenecks when Upstreams team when upstream teams Increase their throughput and the downstream teams aren't able to consume at You know what happens what good does it do if Development is delivering features faster and faster When the teams downstream can't consume it at that pace We can only move as fast as the slowest moving piece And that's why we have work in progress limits with Kanban, which is a second practice that we use We're putting Numbers here to say here's where our limit should be The standard way based on the three of constraints would be to limit the flow To the tempo of the slowest moving bit the constraint that's in the system because that's the fastest that we can Possibly go right and so where did it began? Where do we start? What is that magic number? I find the best way to start is just start where you are How much work do you have flowing through your system right now? Put that number up there make it visible and then experiment is that right? Is that too much are people overburdened? Probably then maybe we need to bring those numbers down So one way that you can have a bit of a bargaining chip With the teams that you're working with is to just demonstrate how much work you actually have in progress There You know because knowledge work is perishable The longer it takes from the time you have to set something aside because you got interrupted The longer it takes then to pick that back up and work on it. You might have have forgotten It'll just take time to to dig back into that Now Here's the deal about work in progress limits. They set they Bring in tension right work in progress limits introduce tension into the organization That you have to deal with it's not comfortable It's like the stress that Adrienne was talking about this morning You add stress into your system so that you can see where it breaks the whip limits introduce Tension and they help provoke the kind of conversations that need to be had and they may be uncomfortable But if it was easy anybody could do it, right? You know, we don't let our computers get to a hundred percent capacity utilization Why do we let our people? This is not sustainable When you're working day in and day out and at two in the morning It doesn't take very long before you get burnt out and people leave So we're trying to find a sustainable way a sustainable flow that we can keep up with the pace We've got two choices. We can either You know push back on the demand or we can increase the capability of the team Which is probably going to take some time, right? We need to build up that skill set knowledge Maybe we need to hire some more people, you know, what tools are they using? What's the capacity of the team to do the work? There's a lot of Attributes that make up the team's capability and as soon as one of them leaves or gets hired on then your capability probably goes down So what to do? How do we get a sustainable approach? What are some rules that we're using that are hindering us or helping us? And that's the third Kanban practice is to make process policies explicit Or this is saying what are the rules of the game? What are the rules at your organization that's either driving fast-flow or preventing it? And some rules if they're a tacit or a tribal You know, they're not well understood at least to new people. So let's look at some of these some of these rules So if you've got If you're trying to go in a certain direction and you have to stop in order to get there It's going to decrease your flow If the only way to get somewhere if the only way to turn left is to stop and wait for a green light Then you're gonna have this stop and go Things will take longer and people get irritated So we need to look at what are some other options besides this stop and go I think the roundabout is a nice example of that because they've they found a way to keep things flowing Right, they've minimized stop and go. You don't have to wait. There's no lights. It's cheaper throughputs increased by 20% because they measured it and measurement is The fourth practice, but first just an example of some rules here on this count-bam board first of all in the Done column here. We'll just walk through this real quick. There's a backlog Then they put work on deck the backlog by the way, this work isn't prioritized. It's not stack ranked Right. It's just a list of uncommitted Unprioritized items that somebody wants to get done. It's not committed to it's not prioritized Until it's pulled into the on-deck queue Combine by the way is a mostly a pole system. We're pulling work into our queue based on our capacity or the team's Capacity to handle the work if there's no capacity then the work doesn't get pulled in It's like only take you know only take on what you can chew, right? Finish. Let's let's focus on the end here focus on finishing work before pulling in new work And then they've got some rules here Back up. They've got some rules down here. What's the definition of done on some of these their definition of done? Right down here They need you know Some work the criteria is they got a cab approval first or maybe it needs an architectural review or Info sec needs to buy off on it Then the first rule is stop hiding the rules Get them out there where people can see them make them aware of what the rules are It's much easier to have a conversation about changing a rule or improving it if it's explicit If people are unsure it's not explicit, then it's harder to improve it But what about prioritization? That seems to last year was I was speaking about the problems that ops teams face as far as dependencies and distributed teams and Interrupt driven work, but I'm finding now that probably one of the biggest issues is prioritization Conflicting priorities causing a lot of heartburn. So what's the priority? What is the prioritization process? Is that explicit? How do things get escalated? What is that process? Is it just by whoever yells the loudest or is it based off of political weight? Right where your bot somebody's boss tells your director and then it comes over it down and then things get escalated so Come up with a policy for prioritization and escalation I want to point out this box in the corner here These were you know, we have these red Expedites that are going through the top lane and projects going through the middle lane and these items in the box here It's a spot to put items that need further analysis. They need further retrospection They got delivered and they're done, but there were problems with them and we want to address them later So we put them on a spot on the board there for us to peek at later What if one of the rules was that work got prioritized by cost of delay? right in in finding a way to bring some non Emotional logical quantitative reasoning to how we should prioritize and a guy who's been doing a lot of work on this is Troy McGonis from Focused objective. He uses Monte Carlo simulation to Determine different timings and and forecasting and I think this is a great start for ops Right, what didn't go out or get done that could have or should have right? We can calculate the cost of delay per project per day, but then in the ops world we also need to take account of keeping production stable and secure Right that we have more variability there. There's a higher risk. So that needs to be factored into this equation What is the probability of getting a DDoS attack or some security breach? And how much would that cost the the company? If you're tracking those kind of issues in your system if you're tracking expedites Then you can measure them fourth Kanban practice We're measured. We're very interested in measuring the flow of work that goes from beginning to end and the individual amounts of cycle time so always a Always a great conversation. What is lead time? What is cycle time? I'm just going to simplify it by saying lead time is when the customer requested something to when they got it and Cycle time is how long it takes once you've Committed to do that work, but you can calculate the cycle time for each cue on your Kanban board and Then that's going to provide you some historical data To use for forecasting and you won't have to rely so much on guessing or estimating how long things are taking Aging reports are fabulous You if you're measuring cycle time You'll have you know, you can get an average and a median and a mean But the aging report is is just it's a huge eye-opener, right? if you've got things and this is real data, right things sitting in Implementation cues for over a hundred and eighty days If you can pull up how long or it's like rotten fruit, right? the longer it sits the you know the costlier it is the The more frustrated people get right so think about aging reports bringing attention to some of these dependencies and Then the rate of the flow This is a good one and you've already got it if you're using JIRA You can see the rate of incoming work versus the rate of outgoing work And if you're incoming work, which would be in the red or brown if that slope is Exploding out Instead of closing up with the green green is the work. That's closing You'll be able to see really quickly in a one-shot glance if your team has a lot much more demand than they're able to deal with bottom line is Looking at being objective at the data, right love couldn't resist this slide You know, how much time do we really spend waiting on third-party vendors? Let's be realistic here Next up is feedback loops and offering up opportunities to be heard frequently and often kind of like merge early and often Now one thing we've had a lot of success with lately is lean coffee. Anybody go to lean coffees Got a couple lean coffees here. Lean coffee real quickly is an opportunity to have a bit of a more intimate session with five or six people you go grab coffee you get out of the Office and you sit down and everybody writes a couple things on a post-it note that they have energy to talk about and Then people you get three votes People can vote all three on one or vote three across different items And then you run them through a Kanban just real simple to do doing and done and you time box the items and to do in doing so after five minutes the Beeper goes off and you thumbs up or thumbs down and you the team decides if you want to continue the conversation on But we can take these and we can start to see patterns week after week after week and you can It gives the team an opportunity to feel like they've been heard you can rotate people in and out to keep it a small group Fantastic way to get feedback that you wouldn't get otherwise unless you know you go out for beers after work It's impossible Sometimes to discover these kinds of issues at larger meetings like ops review But we still need to do ops review Ops review is a monthly where where link coffee would be weekly ops review is monthly And this is where we're gathering everybody in the organization And we're looking at the data and the team leads are presenting That basically they're presenting the balance that their team is dealing with the demand That's coming on their team and then their ability their capability to meet that demand That's what we're interested And everybody here is the same message at the same time This is actually a picture of the first Solvay conference in 1911 in Brussels And I kind of liked it because the subject was radiation and the quanta and Einstein is the youngest Physicist he's second from the right When we're looking at what kind of data to measure and track and present at ops review Here you go these these five things across trend and across the variability Will be quite useful and the the top three work in progress cycle time or lead time and throughput You can get all of those out of a cumulative flow diagram, which is one of the most used con bond Charts and many of the electronic tools will automatically generate those for you But if not you can do it using Excel So there we have six combat practices all on one sheet Click review. So most importantly make work visible because it's hard to manage invisible work and oftentimes You know if it is out there on a wall in front of people At least they'll see it especially especially if it's on the way to the coffee or on the way to the VP's desk The last place I was at the VP doesn't have a lot of time to go look at stats in Jira Right, but if you put it on the way to his desk, it's like covert or Covert con bond right or stealth con bond If you can get that if you can design your con bond system in such a way that it's revealing what is irritating to the people doing the work and Irritating to the customers that the work gets delivered to Now you've got a way to start having that conversation That needs that needs to be had and to find some of that empathy that Jeff was talking about this morning number six is Collaborate an experiment with the with models and theory and so here's a list of reading material that the people in the con bond community often Talk about the first is the goal This is a story about the theory of constraints, which is the study of bottlenecks famous book that Gene Kim's Phoenix project I guess you all got a copy of that Anybody who's wouldn't working and ops for some time can relate to that Lean startup Just you know Eric talks about having this This meeting where you pivot right you decide if you're gonna pivot or not And I liken that to the ops review meeting where you're looking at the data to see if you need to go in that direction or change And then Deming Deming's the guy who went to Japan in the 50s and taught all these Japanese businessmen about why Why organizing teams in functional silos does not work because we don't deliver work that way we deliver work across different functions That fabulous fabulous read there and then down on the bottom right David Anderson's book on con bond It's for the leading how to do book on con bond And then the commitment books a new book out It's about real options theory and looking at options and more of a business language use if we can learn to speak in the language of the business and Have them understand that options expire quite useful. It's a graphic Business novel and then David Reiner since flow book So if you want to get into the deep math on queuing theory to understand Why using cues is helpful, you know con bonds based on cues, right? Not so much on timeline. We manage work through cues It's a leading cues are a leading indicator because we know the more work That's in the queue the longer things are going to take great read there and then Troy Troy McGinnis's book on using Monte Carlo simulation to do Forecasting and cost of delay so just Ending up with grumpy cat here I mean sometimes I said earlier that con bond gave me a way to deal with everything on my plate and still get a decent night's sleep Dealing with everything on my plate doesn't mean having to say yes to everything We can only do so much and sometimes we have to push back and say no the team does not have capacity to deal with that right now and And and and that's okay people don't expect you to do everything, right? I mean you can't you can't do a million things at the same time It's just final slide here and picture of the Con bond for ops game that we've been playing Okay, I think we're out of time Okay, okay So we have time for a couple of questions if you want to come forward just line up behind the mic and we'll just take them serially so one question that I had is you talk about queuing for kind of the system of work and a Lot of the work that I have The systems do is also very cute So is there some kind of like grand unified theory of con bond where I can get those to work together and kind of a feedback loop Are you talking about having? Ration my prioritization But everything just having you know kind of a conceptual framework that that touches you know both on the the flow that we do Well, you know the flow that flows through the systems. That's all cute and you know limited to work in progress It's I see them as the same things I mean it sounds like it is it a question of tools. Yeah, just yeah Tools and you know concepts and you know trying to apply the same thing Yeah, so one cool tool is it's called They didn't pay me to say this or anything, but it's Jim flow and So it's electronic tool, but yet you have a board to it's what prints out the tickets It's got a QR code on it. Are you familiar with this now? You can put it up on your board you put a time-lapse camera in front of it And as people are pushing tickets across the board it takes pictures and automatically updates the the tool So it's the best of both worlds so you can watch the video of it at you know, Jim flow I am J-I-M-F-L-O-W I think them. I think the message that lesson is What's the what's the most irritating thing you guys are dealing with? How do we bring some visibility to that so we can improve it? Yeah, okay? Yeah We got one more okay Two two quick ones In your in your side about there we go In your side about quantitative measures you had quality listed as one of those items And I was wondering if you had time to expand on that. How can can then help us measure quality? Yeah, that's a really hard one So a couple things is rework How many times a ticket comes back well you or if they're getting closed and then they're getting reopened Why is that did we not understand something did you know? Did we miss communicate? So rework is is hard and then also I think looking at the number of expedites is a telltale sign Right if if 63% of the work we're doing is expedites does that sound right? I mean it could be that's that's how your team works, you know If you're at help desk or your call center That's the nature of the demand of your work and that's to be expected But if that's not your situation and that seems really high Then measuring the number of expedites because like that could be an indication that Previous work that was done wasn't done at the level of quality Yeah, I mean it doesn't tell you why may not tell you why right you have to dig into sure why I can see why that Is does our architecture not allow us to to work in this fashion? Okay, thanks. Okay. Thank you Hi, so it's curious You know I work in an ops team and Like a lot of ops teams we you know, we've been looking at a variety of different ways to try to implement, you know Limits of work in progress or like pull models or whatnot but our organization insists on us doing like long-term forecasting and and Committing to piles of work like you know one to two quarters ahead of time Which and we know that doesn't really work, right? So do you have any advice on how to like how to how to mitigate that like inside an organization? Yeah, are you tracking that now? Are you tracking the work? That's not happening two quarters later. Oh, yeah Yeah, we're tracking it a lot better now, so it sounds like you probably have the data there to make a case on Something that's not working that you can demonstrate maybe out of ops review To we just demonstrate here's what happened. We did it, you know, we missed our commitment Here's why It could be though that I mean cost of delay isn't always financial sometimes it's right political and if that's the the problem then you've got a I found that just trying to I Found the best way to get empathy empathy is to show the data in a non-emotional way because Ranting I couldn't get ranting to work, right? But I mean we're still kind of Left with this situation where we're being forced to commit to a pile of work that then we can't complete and then we're forced to Play kick the can and it just seems that we're constantly trying to catch up to that instead of getting yeah And so I mean Deming's book there's something that's driving that behavior What what is that and that's I think what you need to bring visibility to tell me how you're gonna measure me And I'll tell you how I'm gonna behave and if there's You know if people are being measured or their merit reviews or they've just come on board and they only have 90 days Or I don't know what your situation was. I'd love to talk to you afterwards to hear more but I hear I hear that a lot and I think that if you can show and Demonstrate well actual data that has not worked Yeah, for us it's becoming a barrier to getting the people to adopt these tools because they can't wrap their heads around The idea that we would not commit to getting anything. Yeah Yeah Rather than just focus to bring him to bring him to the conference. Thank you. Thank you Thank you everyone for attending. We have a 15 minute break and we have our next talk at noon