Chapter 22 of The Clue of the Twisted Candle This is a LibriVox recording. All LibriVox recordings are in the public domain. For more information or to volunteer, please visit LibriVox.org Recording by Ian Skillan Glasgow The Clue of the Twisted Candle by Edgar Wallace Chapter 22 After a while, Lexman resumed his story. I told you that there was a man at the Palazzo named Salvolio. Salvolio was a man who had been undergoing a life sentence in one of the prisons of southern Italy. In some mysterious fashion he escaped and crossed the Adriatic on a small boat. How can I find him? I don't know. Salvolio was a very uncommunicative person. I was never certain whether he was a Greek or an Italian. All I am sure about is that he was the most unmitigated villain next to his master that I've ever met.  It was he who gave me the scar, John Lexman pointed to his cheek. In his master's absence he took upon himself the task of conducting a clumsy imitation of Kara's persecution. He gave me too the only glimpse I ever had of the torture of poor grace on the wind. She hated dogs and Kara must have come to know this. And in her sleeping room she was apparently better accommodated than I. He kept four fierce beasts so chained that they could almost reach her. Some reference to my wife from this low brute maddened me beyond endurance and I sprang at him. He whipped out his knife and he struck at me as I fell and I escaped by a miracle. He evidently had orders not to touch me for he was in a great panic of mind as he had reason to be. Because in Kara's return he discovered the state of my face started an inquiry and had Salvolio taken to the courtyard in the true Eastern fashion and bashed in adode until his feet were pulp. You may be sure the man hated me with a malignity which almost rivaled his employers. After Grace's death Kara went away suddenly and I was left to the tender mercy of this man. Evidently he had been given a fairly free hand.  Salvolio began his persecution by reducing my diet. Fortunately I ate very little. Nevertheless the supplies began to grow less and less and I was beginning to feel the effects of this starvation system when there happened a thing which changed the whole course of my life and opened to me a way to freedom and to vengeance.  He would bring up dancing girls from Durazzo for his amusement and invite prominent men in the neighbourhood to his feasts and entertainments for he was absolutely lord of the plato when Kara was away and could do pretty well as he liked. On this particular night the festivities had been more than usually prolonged for as near as I could judge by the daylight which was creeping in through my window it was about four o'clock in the morning when the big steel sheet of door was opened and Salvolio came in more than a little drunk. He brought with him as I judged one of his dancing girls who apparently was privileged to see the sights of the palace. For a long time he stood in the doorway talking incoherently in a language which I think must have been Turkish for I caught one or two words. Whoever the girl was she seemed a little frightened I could see that because she shrank back from him. His arm was about his shoulders and he was half supporting his weight upon her. There was fear not only in the curious little glances he thought it made from time to time but also in the averted face. Her story I was to learn. She was not of the class from when Salvolio found the dancers who from time to time came up to the palace for his amusement on the amusement of his guests. She was the daughter of a Turkish merchant of Skutari who had been received into the Catholic Church. Her father had gone down to Durazzo during the first Balkan war and then Salvolio had seen the girl unknown to her parent and there had been some rough kind of courtship which ended her running away on this very day and joining her ill favoured lover at the Palazzo.  As I say, the girl was frightened and made as though to go from the dungeon. She was probably scared both by the unkempt prisoner and by the drunken man at her side. He however could not leave without showing to her something of his authority. He came lurching over near where I lay his long knife balanced in his hand ready for emergencies and broke into a string of vituperations of the character to which he was quite hardened. Then he took a flying kick at me and got home in my ribs but again I experienced neither a sense of indignity nor any great hurt. Salvolio had treated me like this before and had survived it in the midst of the tirade looking past him. I was a new witness to an extraordinary scene. The girl stood in the open doorway shrinking back against the door looking with distress and pity at the spectacle which Salvolio's brutality afforded. Then suddenly there appeared beside her a tall Turk. He was grey-bearded and forbidding. She looked round and saw him and her mouth opened to her to a cry but with a gesture he silenced her and pointed to the darkness outside. She cringed past him her sandals feet making no noise. All this time Salvolio was continuing his stream of abuse but he must have seen the wonder in my eyes for he stopped and turned. The old Turk took one stride forward and circled his body with his left arm and there they stood grotesquely like a couple who were going to waltz. The Turk was a head taller than Salvolio and as I could see a man of immense strength. They looked at one another face to face Salvolio rapidly recovering his senses and then the Turk gave him a gentle punch in the ribs. That's what it seemed like to me but Salvolio coughed horribly went limp in the other's arms and dropped with a third to the ground. The Turk lent down soberly and wiped his long knife on the other's jacket before he put it back in the sash at his waist. Then with a glance at me he turned to go but stopped at the door and looked back thoughtfully. He said something in Turkish which I couldn't understand. Then he spoke in French. Who are you? he asked. In a few words as possible I explained. He came over and looked at the manacle about my leg and Chucky's head. You'll never be able to get that on, they said. He caught hold of the chain which was a fairly long one bound at twice round his arm and steadying his arm across his thigh.  There was a smart snap as the chain parted. He caught me by the shoulder, pulled me to my feet. Put the chain about your waist offender, he said. And he took a revolver from his belt and handed it to me. You may need this before we get back to Yoratso, he said. His belt was literally bristling with weapons. I saw three revolvers besides the one I possessed and he had evidently come prepared for trouble. We made our way out from the dungeon into the clean smelling world without. It was the second time I had been in the open air for 18 months. My knees were trembling under me with the weakness and excitement. The old man shut the prison door behind us and walked on until we came to the girl waiting for us by the next side. She was weeping softly. And he spoke our few words in a low voice in our weeping ceased. This daughter of mine will show us the way, he said. I don't know this part of the country, she knows it too well. To cut a long story short, said Lexman, we reached Yoratso in the afternoon. There was no attempt meant to follow us up and neither my absence nor the body of Salvolio were discovered until late in the afternoon. You must remember that nobody but Salvolio was allowed into my prison and therefore nobody had the courage to make any investigations. The old man got me to his house without being observed and brought me a brother-in-law or some other relative of his to remove the anklet. The name of my host was a sane offender. That same night we left with a little caravan to visit some of the old man's relatives. He was not certain what would be the consequence of his act and for safety's sake took this trip which would navel him if need be to seek sanctuary with some of the wilder Turkish tribes who would give him protection. In that three months I saw Albania as it is. It was an experience, never to be forgotten. If there's a better man in God's world than he, Abum, has sane offender, I've yet to meet him. It was he who provided me with money to leave Albania. I begged from him to the knife with which he had killed Salvolio. He had discovered that Kara was in England and told me something of the Greek's occupation which I had not known before. I crossed to Italy and went on to Milan. There it was that I learnt that an eccentric Englishman who had arrived a few days previously in one of the South American boats at Genoa was in my hotel desperately ill. My hotel I need to hardly tell you was not a very expensive one and we were evidently the only two Englishmen in the place. I could do no less than go up and see what I could do for the poor fellow who was pretty well gone when I saw him. I seem to remember having seen him before and when looking around for some identification I discovered his name. It was George Galerko who had returned from South America. He was suffering from malarial fever and blood poisoning and for a week with an Italian doctor I fought as hard as any man could fight for his life. He was a trying patient, John Lexman smiled suddenly at the recollection, vitriolic in his language, impatient and imperious in his attitude to his friends. He was, for example, terribly sensitive about his lost arm and wouldn't allow either the doctor or myself to enter the room until he was covered to the neck, nor would he eat or drink in her presence. Yet he was the bravest of the brave, careless of himself and only fretful because he had not time to finish his new book. His indomitable spirit did not save him. He died on the 17th of January of this year. I was in general at the time having gone there at his request to save his belongings. When I returned he had been buried. I went through his papers and it was then that I conceived my idea of how I might approach Kara. I found a letter from the Greek which had been addressed to Buenos Aires to await her arrival. And then I remembered in a flash how Kara had told me he had sent George Gather called to South America to report upon possible gold formations.  Even as he had planned my downfall, scheming every step and covering his trail so did I plan to bring about his death that no suspicion should follow me. I knew his house. I knew something of his habits. I knew the fear in which he went when he was in England and away from the feudal guards who had surrounded him in Albania. I knew of his famous door with its steel latch and I was planning to circumvent all these precautions and bring him not only the death he deserved but a full knowledge of his fate before he died. Gather called on some money about £140. I took £100 of this for my own use knowing that I should have sufficient London to recompense his heirs. The remainder of the money with all such documents as he had saved those which identified him with Kara, I handed over to the British consul. I was not unlike the dead man. My beard had grown wild. I knew enough of Gather called's eccentricities to live the part. The first step I took was to announce my arrival by inference. I'm a fairly good journalist with a wide general knowledge and with this corrected my reference to the necessary books which are found in the British Museum Library. I was able to turn out a very respectable article on Patagonia. This I sent to the Times with one of Gather called's cards and as you know it was printed. My next step was to find suitable lodgings between Chelsea and Scotland Yard. I was fortunate in being able to hire a furnished flat, the owner of which was going to the south of France for three months. I paid the rent in advance and since I dropped all the eccentricities I had assumed to support the character of Gather called, I must have impressed the owner who took me without references. I had several suits of new clothes made, not in London, it's my health, but in Manchester and again I made myself as trim as possible to avoid after identification. When I'd got these together in my flat I chose my day. In the morning I sent two trunks with most of my personal belongings to the Great Midland Hotel. In the afternoon I went to Cadogan Square and hung about until I saw Cara drive off. It was my first view of him since I'd left Albania and it required all my self-control to prevent me spraying at him in the street and tearing at him with my hands. Once he was out of sight I went to the house adopting all the style and all the mannerisms of poor Gather called. My beginning was unfortunate for the shock I recognised in the valley a fellow convict who had been with me in the warder's cottage in the morning of my escape from Dartmoor. There was no mistaking him and when I heard his voice I was certain would he recognise me, I wondered in spite of my beard and my eyeglasses. Apparently he did not. I gave him every chance, I thrust my face into his and visited challenge him in the eccentric way which poor old Gather called had to test the grey of my beard. For the moment however I was satisfied with my brief experiment and after a reasonable interval I went away returning to my place off Victoria Street and waiting till the evening. In my observation of the house whilst I was waiting for Cara to depart I had noticed that there were two distinct telephone wires running down to the roof. I guessed rather than knew that one of these telephones was a private wire and knowing something of Cara's fear I presumed that the wire would lead to a police office or at any rate to a Guardian of some kind or other. Cara had the same arrangement in Albenea connecting the Palazzo with the John Durham poster in Lasso. This much the Sen told me. That night I made a reconnaissance of the house and saw Cara's window was lit and at ten minutes past ten I rang the bell and I think it was then that I applied the test of the beard. Cara was in his room the valley told me and led the way upstairs. I had come prepared to deal with this valley for adding a special reason for wishing that he should not be interrogated by the police. On a plain card I had written the number he bought in Dartmoor and I added the words I know you get out of here quick. As he turned to lead the way upstairs I flung the envelope containing the card on the table in the hall. In an inside pocket as near to my body as I could put them I had the two candles. How should I use them both? I had already decided. The valley ushered me into Cara's room and once more I stood in the presence of the man who had killed my girl and blotted out all that was beautiful in life for me. There was a breathless silence when he paused.  The chief commissioner with a heavy frown and pushed lips sat stroking his moustache and looking under his shaggy eyebrows at the speaker. The French police officer his hands thrust deep in his pockets his head on one side was taking in every word eagerly the cellophase Russian impassive of face might have been a carved ivory mask. O'Grady the American the stump of a dead cigar between his teeth shifted impatiently with every pause as though he had hurried forward in an oomol. Presently John Lacksman went on he slipped from the bed and came across to meet me as I closed the door behind me. Mr. Gather Coley said in that silky tone of his and held out his hands. I didn't speak. I just looked at him the sort of fierce joy in my heart the like of which I had never before experienced and then he saw in my eyes the truth and half reached for the telephone. By that moment I was on him he was a child in my hands all the bitter anguish he had brought upon me all the hardship of starred days and freezing nights it strengthened and hardened me I'd come back to London disguised with a false arm and this I shook free it was merely a gauntlet of thin wood which I'd made for me in Paris I flung him back in the bed and half knelt half laid in him Kara I said you are going to die a more merciful death than my wife died he tried to speak his soft hands gesticulated wildly but I was half lying in one arm and held the other I whispered in his ear nobody will know who killed you Kara think of that I shall go scot-free and you will be the centre of a fine mystery all your letters will be read all your life will be examined and the world will know you for what you are I released his arm for just as long as it took to draw my knife and strike I think he died instantly John next one said simply I left him where he was and went to the door I had not much time to spare I took the candles from my pocket they were already ductile from the heat of my body I lifted up the steel latch of the door and propped up the latch of the smaller of the two candles one end of which was on the middle socket and the other beneath the latch the heat of the room I knew would still further soften the candle and let the latch down in a short time I was prepared for the telephone by his bedside though I didn't know whether it led the presence of the paper knife decided me I balanced it across the silver cigarette box so that one end came under the telephone receiver under the other end I put the second candle which I had to cut to fit on top of the paper knife at the candle end I balanced the only two books I could find in the room and fortunately they were heavy I had no means of knowing how long it would take to melt the candle to a state of flexion which would allow the full weight of the books to bear upon the candle end of the paper knife and fling off the receiver I was hoping that Fisher had taken my warning and had gone when I opened the door softly I heard his footsteps in the hall below there was nothing to do but finish the play I turned and addressed an imaginary conversation to Kara it was horrible but there was something about it which aroused me a curious sense of humour and I wanted to laugh and laugh and laugh I heard the man coming up the stairs and closed the door gingerly what length of time would it take for the candle to bend to completely establish the alibi I determined to hold Fisher in conversation and this was all the easier since apparently they had not seen the end I had left on the table downstairs I hadn't long to wait for suddenly with a crash I heard the steel latch fall in its place under the effect of the heat the candle had bent sooner than I expected I asked Fisher what was the meaning of the sound and he explained I passed down the stairs taking all the time I found a cabbage lawn square and drove to my lodging underneath my overcoat I was partly dressed in evening kit ten minutes later after I entered the door of my flat I found a beardless man about town not to be distinguished from the thousand others who would be found that night walking the promenade of any of the great music halls from Victoria Street I drove straight to Scotland Yard there was no more than a coincidence that whilst I should have been speaking with you all the second candle should have bent and the alarm be given in the very office in which I was sitting I assure you all in all endlessness that I did not suspect the cause of that ringing until Mr. Manson spoke there gentlemen is my story he threw out his arms you may do with me as you will Kara was a murderer died a hundred times in innocent blood I have done all that I set myself to do that and no more that and no less I had thought to go away to America but the nearer the day of my departure approached the more vivid became the memory of the plans which she and I had formed my girl my poor martyred girl he sat at the little table that his hands clasped before him his face lined in white and that is the end he said suddenly with her eyes smile and not quite TX it was Belinda May who spoke I can carry it on she said she was wonderfully self-possessed thought TX but then TX never thought anything of her but that she was wonderfully something or other most of your story is true Mr. Lexman she said this astonishing girl oblivious of the amazed eyes that were staring at her but Kara deceived you in one respect what do you mean asked your Lexman rising unsteadily to his feet for answer she rose and walked back to the door with the chinch curtains and flung it open there was a wait which seemed an eternity and then through the doorway came a girl slim and grave and beautiful my god this was TX Grace Lexman and of Chapter 22 Recording by Ian Skelan Chapter 23 of The Clue of the Twisted Candle This is a LibriVox recording or LibriVox recordings are in the public domain for more information or to volunteer please visit LibriVox.org Recording by Larissa Jaworski Brisbane Australia March 2007 The Clue of the Twisted Candle by Edgar Wallace Chapter 23 She was alone, two people who found in this moment a heaven which is not beyond the reach of humanity but which is seldom attained to Belinda Mary had an eager audience all to her very self of course she didn't die she said scornfully Kara was playing on his fears all the time he never even harmed her in the way Mr. Lexman feared he told Mrs. Lexman that her husband was dead just as he told John Lexman his wife was gone what happened was that he brought her back to England who asked TX incredulously Grace Lexman said the girl with a smile you wouldn't think it possible but when you realise that he had a yacht of his own and that he could travel up from whatever landing place he chose to his house in Cadogan Square by motor car and that he could take her straight away into his cellar without disturbing the household you'll understand that the only difficulty he had was in landing her it was in the lower cellar that I found her you found her in the cellar demanded the chief commissioner the girl nodded I found her and the dog you heard how Kara terrified her and I killed the dog with my own hands she said a little proudly and then she shivered it was very beastly she admitted and she's been living with you all this time and you've said nothing asked TX incredulously Belinda Mary nodded and that is why you didn't want me to know you will see she was very ill she said and I had to nurse her up and of course I knew that it was Lexman who killed Kara and I couldn't tell you about Grace Lexman without betraying him so when Mr Lexman decided to tell his story I thought I'd better supply the grand denouement the men looked at each other what are you going to do about Lexman asked the chief commissioner and by the way TX how does all this fit your theories fairly well replied TX Cooley obviously the man who committed the murder was the man introduced into the room as Gathacole and obviously it was not Gathacole although to all appearance he had lost his left arm why obvious asked the chief commissioner because answered TX Meredith the real Gathacole had lost his right arm that was the one era Lexman made the chief pulled his mustache and looked inquiringly around the room we have to make up our minds quickly about Lexman he said what do you think Karl know the Frenchman shrugged his shoulders for my part I should only import tune your home secretary to pardon him but I should recommend him for a pension he sent flippantly what do you think Svorsky the Russian smiled a little it is very impressive story he said dispassionately it occurs to me that if you intend bringing this gentleman to judgment you are likely to expose some very pretty scandals incidentally he said striking his trim little mustache I might remark that any exposure which drew attention to the loyal conditions of Albania would not be regarded by my government with Svavor the chief commissioner's eyes twinkled and he nodded that is also my view said the chief of the Italian bureau naturally we are greatly interested on the Adriatic literal it seems to me that Cara has come to a very merciful end and I am not inclined to regard a prosecution of Mr. Lexman with equanimity well I guess the political aspect of the case doesn't affect us very much said O'Grady but as one who was once mighty near the exfixiated by stirring up the wrong kind of mud I should leave the matter where it is the chief commissioner was deep in thought Linda Mary eyed him anxiously tell them to come in he said bluntly the girl went and brought John Lexman and his wife and they came in hand in hand supremely and serenely happy whatever the future might hold for them the chief commissioner cleared his throat Lexman were all very much obliged to you he said for a very interesting story and a most interesting theory what you have done as I understand the matter it is a theory not only as to how the murder was actually committed but as to the motive for that murder it is I might say a remarkable piece of reconstruction he spoke very deliberately and swept away John Lexman's astonished interruption with a stern hand please wait and do not speak until I am out of hearing he growled you have got into the skin of the actual assassin and have spoken most convincingly one might almost think that the man who killed Remington Cara for that piece of impersonation we are all very grateful he glared around over his spectacles at his understanding colleagues and they murmured approvingly he looked at his watch now I am afraid I must be off he crossed the room and put out his hand to John Lexman I wish you good luck he said and took both Grace Lexman's hands in his one of these days he said paternally your husband shall tell me another happier story he paused at the door as he was going out and looking back caught the grateful eyes of Lexman by the way Mr Lexman he said hesitatingly I don't think I should ever write a story called the clue of the twisted candle if I were you John Lexman shook his head it will never be written he said by me the end of the clue of the twisted candle


So I'm so happy to see so many people actually I was I had a nightmare just today that was only going to be Mauricio Me and maybe Whitney So yeah, appreciate you I know there was no We have to do the talk in Spanish in Spanish. See we can do it in Spanish. Yeah, no So I like I said, I know there was a lot of talks at this time So I really appreciate so many people coming in to know about the Kennedy project And the open source aspect of it, and we're going to talk a little bit about joining CNCF and a little bit about the the function so Just to survey the room Who who have contributed to to k-native a dog or code So we have a few folks who have used k-native before Great, that's who was who was yesterday here in the day to k-native con Okay, so don't miss out the the next one. We're going to have one in Detroit And the CFP will come in will come out soon So if this you have a chance by if you're not you're learning k-native or using k-native Those are the type of talks that are the program committee and part of the current committee that we're looking for So don't don't be afraid if you're known contribute to k-native one of the main talks that we're looking for is End-user companies or end-user organizations. How are you using k-native? How k-native is solving your issues? And also if you're an user company, please reach out to me because I'm doing interviews just to get your story about How you using k-native? What are the things that is working for you? What are the things that are missing so we can have feedback into the community and I can serve that into the user experience group To make it better and showcase your company as a case study because that's part of the CNCF Journey if we become CNCF other companies we joined who have Confidence that the budget will stick around for a while and it will have more features and will be robust and it'll be enterprise ready So with that and if you are into k-native, we have another talk this afternoon actually on k-native from Mauricio Yeah, I'm presenting with Thomas here and we are talking about like functions and all the stuff that it's coming there And how do you feel like a you know a function as a service kind of platform? Yeah, so we go deeper on on the k-native functions So they're going to talk a little bit about the project so people get get involved and hopefully By Detroit I want to see more hands for people contributing To k-native and again contribution is not code, right Mauricio. We always say it could be a piece of blog post It could be a tutorial it could be Just joining slack and answering questions on stack overflow creating an issue creating an issue Explaining like I have this issue and this is a Reproduce that is easy to reproduce sometimes people just say they have an issue and that's it But we all ask those excellent folks that write all the information like this is how to reproduce it in a in a simpler way So that's that's everything counts. I think actually those counts into contributions for voting for our members So currently if you are contributing those those type of things you can vote in the TOC elections the technical oversight committee is having elections right now and You can go online and vote if you are you are a contributor and we count Those contributors in the dev stacks like we the dev stacks that we use for Kubernetes out on the other CNCF projects. I Think that you're forgetting something important that is that we have a stickers. Oh, yes We have stickers. So if you have questions, you have deeps on on stickers So you have good questions you get two stickers But yeah, there's stickers around for everyone even has here and I have more here So if and take and take the stickers out right to give to your friends or if you are doing meet-ups in your country Where are you from? About k-native right you giving a talk, you know, you can give stickers out And we're going to have a CNCF store So if that's something that you want to continue doing like running meet-ups and be an ambassador or an advocate for k-native Now we're in CNCF. We have funds. We have we have money Now we can use to give swag. So we were trying to get the store up up up and running I think we're going to first thing is going to be t-shirts We didn't have them for here, but for Detroit. I hope we hopefully we have both stickers t-shirts and If Lindsey can get me socks, I really want socks k-native socks. I don't know. That's good. Everybody has cool socks and then we don't have socks So that was a long introduction We're burning five minutes on that one. So introductions. Who are the two people random people up here? My name is Carlos Santana. I play the guitar make millions and now I heard the Kubernetes is Awesome and people are making more millions out of Kubernetes So I need to to get in there. I'm an architect in IBM. I work with a lot of customers insurance company bank companies trying to make Their software and IBM software run on Kubernetes the best way possible with DevOps and upstream I work on k-native and steering committee and in Kubernetes I'm in the sick release Recently joined there and the release team lead for release notes 125 help in 124 So if you are in the also in Kubernetes upstream come say hi because I'm trying to meet and learn that that journey about all the six and Marysio yeah, so my name is Marysio Salatino. I will keep it super short I'm working for VMware upstream k-native as well working on functions a lot these days Working also a little bit on k-native eventing, but I am truly passionate and close to developers So I'm really pushing forward to get like a great developer experience there on the functions project It's pretty exciting. We will be showing a bit of that now. Yeah, and follow Marysio on Twitter He has a cool weekly k-native newsletter that he does of his experience of working with k-native and the k-native community I I read it every week. So it's it's keep doing it. So people are actually reading it You don't have metrics get metrics In there so yeah, so let's we have something fun for this for this talk It's maintainers track usually are like dry, but we want to do a game Yes, so basically we are going to play a game at the end of the session, right? But I wanted to show you how the game looks like so we can all play at the end because it's a little bit fast It's like a just a quiz game pretty simple stuff But if you don't see it before you play it might be too fast for for some some people there So usually like this is a quiz game you go there You just generate a player's name because we don't want you to enter your crazy names And then when you have one that you like then you start playing right as you can imagine This is being built by functions, but that basically means that each question Processing kind like the inputs that you're providing It's just a function that will be get started in a Kubernetes cluster, right? So the moment you start the question you will see here a timer and you have 10 seconds to answer if you save some Time for example, five seconds there That's going to come count in your your points experience, right like an I corrected Maybe kind of like okay, so I get ten points there You can see here that you know a function was created there and bootstrapped in the cluster I can do that for question two which is a little bit different It has some pressing buttons and stuff and you will see that as soon as the timer runs out a new function We'll be created in the cluster, right? If we are all playing on the same time We will just be sending a lot of you know Putting a lot of demand on the system so functions will be a scale up Dynamically by K native well and what I wanted to explain here is in the last like in the last the screen here You have like this tweet button Basically that when you click it it will take you to Twitter and if you tweet and you rank high in the leaderboard We will be you know sharing some swag in there as well So now you know how it works. Let's go through the presentation and then we will be playing together at the end Carlos yes, so it's a competition and it's not a competition. It's yes. It's a fun competition Valencia trivia so for you that haven't seen a came here before Is the project is around? Serving which is gets a lot of attention to it for the serverless containers on Kubernetes and we also have the event driven apps or EDA It's good to spell it out event driven applications, which is a sync operations a sync cloud events It's based on cloud events, which is another CNCF project On those we have a reference implementation, but K native is also an API So you can take an API for developer experience. We made the API v 1.0 So you say it's a stable API that we feel confident going out to do conformance You can have an implementation some vendors have an implementation behind it That is not Kubernetes, but most of the people here today are going to run it in Kubernetes as the implementation of it and that's what makes it a Kubernetes native right CRD is the way you manage so a lot of questions when you have for running K native Usually the answer is like you do it the Kubernetes way, right? So we put the developer experience that abstraction But at the end of the day a lot of people like to use it because some of the lower level questions They get answered like you do it either the cut Kubernetes way or a cloud provider way K native features, we can say the simpler abstractions that get very appealing to folks is the yaml I didn't do them scrolling right we have Twitter Twitter Twitter them scrolling We have yaml through scrolling when you start with a yaml with the ingress and the cert manager and the deployment and the Service and get the labels correct and then you have the HPA and you can have maybe some Service mesh routing team percentages and weight by the end of the day you have a lot of yaml So what we try to do is like abstract that into a simpler Attraction that it literally is like 10 lines of code that you can get started with a serving yaml file Would you specify? Here's the container and here's the port and if the port is 8080 You can just omit that the other one is auto scaling like this morning We're talking about the in the keynote of the energy efficiency on how much CPU is wasted So what we want to do is have as many pods right that being packing into into auto scaling so you can have maximum efficiencies to get as many deployment definitions right and those deployment definitions will be sometimes scaled to zero and Would not consume energy and then you have other definitions that are running So you can have a lot of definitions of these applications in a current discussor But depending what is the demand in the traffic and what are the types of the type of applications are running? They will scale to out and then when there's no traffic can scale to zero or one depending if your Your call starts and those type of things progressive rollouts is some feature that It's it has like a an initial implementation I will say but we're looking for the community to come up with the better and more clever ways of doing these rollouts Because there's a lot of there's a lot of I will say a lot of information or innovation going out in your companies on How do you do rollouts and some of the feedback that I hear the community? Which whole-way track is awesome for that is I don't use the standard rollout because it doesn't work for my company Well, we want to see what is working for companies and share that and build that in but at least in Incaneated we have one that is time-based basically you give it like I want to roll out this the new revision in one hour So it would it was split that one hour in 100 percent increments and it will start Running out the percentage of the traffic right from zero to a hundred because you don't want to like Suddenly like send all the traffic to the new revision So you have the option to say like well take your time to roll out These this revision and maybe for one region and then you move to the next one The other one is event integrations and this is where we have a need for sources So we have the the we have channels and subscriptions that is kind of a direct communication messaging Platform, but we also have a kind of a hub or broker pattern that you can have consumers or Sinks that tap into that into those cloud events based on different filters And those can be dynamic filters, which is a it's behind a feature flag So if you have seen demos or examples of being like strict to some attributes of that kind of event There's a dynamic filter for that For that event integration, but we need event sources more from the community And then handling events is like somebody produced events. It could be a cloud provider It could be pops up from from Google It could be through your mesh has a lot of integrations They can send you events from different providers and cloud providers or it could be your own events in your application So you're handling events and producing events and that's there was a great talk yesterday By Pablo that explains. How do I handle events? Why reply to a reply with an error, which is an interesting aspect to that The other one is pluggable since we are using Kubernetes We are Careful of choosing the abstractions, but also we are needing help from the community in terms of like What are the things that we expose into those in those type of things by using? Kubernetes right you can have controllers webhooks that annotate certain things. We have to leave that alone because you have the need for In introducing kind of specific configurations in those in those deployments But it's pluggable in the terms of you can have cloud events or you can have for example I was mentioning the broker so right now if you use your data core systems of events is rabid and Q blogging If you're doing dev in a kind cluster in your computer You can have the in-memory one or if your system you already established your Infrastructure on Kafka you can tap you can plug the Kafka so the abstraction Is pluggable the things that you do and you don't have to change the definition of the trigger or the handler for that So we try to make it very very pluggable This is this is one that I added Last minute which is came out out of this week and we still have some Information of K native but people have heard in the past and I just wanted to have some misbusters So the first one I think it comes out a lot like I can use k native because it's used Istio And I don't know how to use Istio and it's just so big that I can like burns my laptop up right and I cannot figure out You don't have to use Istio. Istio was the first implementation But at the when we have we heard that feedback we we right away Created an abstraction that we were able to plug something else So we was I think it was ambassador or a misery glue, but right now the one that are stable our contour We have our own Ingress that is smart enough that implements the the minimum things that we need is called courier And we have contour and I need to still there is still without the mesh and with the mesh But you don't have to use Istio Not saying that these two is bad or anything It's like not anyone not everyone is using Istio, but the key aspect is I always say if you're using Istio Don't introduce another one right use Istio We k native the other one was people are installing a k native in their computer, right? It was serving an event in an event independent on serving CRDs So we separated that so we could have serving by itself a small footprint or eventing and actually says a very small Footprint that we get it running in kind In less than five minutes You see another one that is that is recently is PVC Where some people were like try k native and because they needed some Was targeted for stainless applications, but maybe they did it empty there for example very very simple PVC or real PVC There's a feature flag that got introduced recently So you can have PVC but take into account if you create a PVC it will take time to the storage which Certain applications can can handle that Sidecarts you can we can support multiple user containers, so you need a sidecar that you're doing something special with your app you can add that I think yesterday somebody asked about arm and Like we run Raspberry I'm a fan of running k native or Raspberry Pi because I have a little bit a lot of small applications there But not not running so they run when there's an event in the like in my house in auto Automation and it runs so it supports our Controllers are building go and they're the images are multi architecture, so you can run it in arm I already mentioned the dynamic filters and header based routing You can have the DNS all that function or service have an extra sub domain that you can target the URL But there's applications that you want to do certain things by header for a front end application That want to get to a revision or the last revision you can have a header and then have the routing to that to that service So it's kind of the misbusters that some people try k native I mean they were not able to get it into their Production application because they were missing one or two features So I just wanted to point out a few of them that give it another shot and if you find something that is blocking you You know tell us and we can help you out It could be like a simple web hook for now or work around and we can put it in the backlog and then we have Different working groups that will be evaluating depending what where's the gap? Adopters this is something that comes along a lot is like who's using k native right? No one wants to be like this the single company and the first company to use k native in production But we have some companies in production some companies building a platform see people like red hat IBM We're Google where are the the main the first ones and trigger match which is a it's a startup that basically started out of the innovation of K native and car events But we have case studies in the website you can check them out and those are the case studies that I want to continue doing We have three one from deep C for for AI and gaming outfit seven Anyone knows Tom Tom the cat that company actually I was surprised to see them in slack And I asked them like what you guys? What are you doing with with k native and I got I got to meet them had an interview and learn about what their Problem that they're solving and the case study made it to the to the website. So what case studies are very good to feel Confident to other companies to and also learn what there's what are the problems they're solving? So I asked in the community if you have your end user company, please reach out to to me Evan or one of our Maintainers because we want to capture that and see this is maintainers track If you have your open open source project that you want to get into CNCF case studies is one of the requirements that the TOC would evaluate We need to Yeah Major milestones that we did recently was the v1.1 that I said joining CNCF And these are like kind of like the the major milestones. We started the project Stats we have 11,000 github stars We have two github repos. So we have many repos in the core and also we have a github repo call k native sandbox There's a multiple PRs coming in and there's a lot of reviewers But we need help in the community to get more people into positions that they can help review PRs And we have a a ladder on you getting started as a member of And also to a lead position into one or the working group leads And about CNCF projects since this is a CNCF Event we were able to get into CNCF. We did the the work and the paperwork And we steering is kind of working out getting all those last minute transitions into getting all the CNCF Benefits from sum accounts to implement the the CLA and replace the Google CLA that we had in place To replace it with a CNCF one and that people can contribute The the project structures these are the we have a steering committee. We have a technical committee Steering works with more of the community and the health of the project The oversight technical committee we having I mentioned at the beginning we have in elections So if you can vote go ahead and vote for one of the candidates and working groups We're dividing into working groups. So depending on your preference or your skills or the things that you want to learn Join a working group and you can just join. There's no requirement You can join the meeting and start learning what they're working on from productivity That will work on infra to serving and things like venting I think there's a new one functions to say is a hot one that is just became a working group Roadmap we're looking into API Gateway Kubernetes ingress v2 Things that we presented yesterday that container freezer and a security audit that is kind of a check mark saying that We it's a record. It's one of the many requirements to go into CNCF graduation We started like that process seven is it's working on that and also if you are a security person We would like to see you join So we have an open community that we want to hear what are the things maybe we put in that RFP And also when the outcome of like what are the security findings dealing with like the TLS version and the Cyphers, you know, so you in that space we need people like that like to help us out Clicker there you go. Okay. So, yeah, I think that like following kind of that message and the state of the project I would say that K native functions is a very good place to start contributing We are actively looking for new contributors And this is the good thing about functions is that it's like a polyglot place no matter which language, you know You can definitely contribute to the project if you are into go You can contribute into the CLI and that's what I want to quickly show you kind of what you can do with functions today It's pretty simple and the idea of functions again It's just to build simple applications that do a single thing pretty easily and we are focusing here into the developer experience So basically what I can do can you read it in the back? Yeah, is that okay? Yeah, okay. So basically I'm creating a Function here, which is just a directory and then what I will use is the funk CLI And in this case, I'm creating a function using the goal language, right? So I've just created like a language like a function here that I would can open in my ID Pretty pretty simple stuff. Again. This funk CLI allows you to scaffold the project This is just a very simple simple source code here like nothing. It's just saying, you know This is a function. It's going to do something here This is where you write your code and this is the output of the function, right? So hey maintainers Right, so pretty simple function. It's not going to do much But the experience behind these is like the idea of here like a developer I can just go back to the the terminal here That's quite a bit better and then basically just have this running in a Kubernetes cluster, right? So I have a Kubernetes cluster already configured And funk will take care of building this function into a container no matter the language that I use, right? and then build a container push that container to a registry and then basically deploy that into a Knative installation without having to write a docker file or a jammel file in this case So Kennedy funk in this case the function project is taking care of all these developer experience And there is a lot of work in that area, right? Yesterday in Kennedy con We also showed how you can do the building process instead of locally with your docker You know with your docker demon in your laptop You can do it remotely in a cluster and what we are seeing here is that the function, you know The container was built it's now push pushing this function container to the docker register in this case docker hub with my You know with my account and then the next step it's going to be deploying that into The Kubernetes cluster that is running so as a developer I can focus on writing code the code of that function I can connect that function to a database or do whatever you want to do in the function And then just use the funk CLI just to move that into a cluster That can be like the your development cluster for example Now it's deploying the function and at the end of this kind of like running this command What I'm expecting to have is just a URL that I can basically access by just doing an HTTP request, right? So I went from having no project whatsoever to a function that runs in side Kubernetes It's leveraging the entire, you know framework of k-native in this case It's automatically a scaling based on demand and also it's going to be downscale to zero if I don't use it for a while In just three minutes, which is pretty nice in my opinion and again This is polyglot so I can create like a function using Java or I can create a function using Rust or any other You know energy efficient language We do not support C right now, but we should I'm bash and yes. I think it's like I was talking about Jokes jokes aside That's kind of nice, but that's a yes You can guess that's a simple hello world kind of thing, right? And that's the main reason why we build kind of the function like the quiz game just to show a little bit more How can like a normal application will look like? So the function game that you are going to play in your phones I guess or your laptops if you have a laptop it's kind of following this architecture, right? So we have a Kubernetes cluster with k-native installed and as I mentioned before We have one function per level in this case per question and these functions are connecting to readies to store the state Like when you score like the answer of like a player You're going to just store store that in in redis And then you will have like we will have like a leaderboard screen Which basically has a different function that goes to readies get all this goes for your your your play your gameplay And then just crunch that data and then send it to you know like a view that it's basically a ranking When you start building these applications you start to have how much time do we have we have five minutes, okay? When you start building this kind of like application you start realizing that for example in this case if Your functions are like the ones that I show that are synchronous You need to like send an HTTP request to the function so the function can do something You will start dealing with latency issues right like in this case every time that I want to evaluate the score for a level I need to go to the function calculate the score then go to readies store You know the score and then go back with the answer to the to the user, right? So all that kind of thing takes time so it's like when you start building functions the way of You know the mental model that you use in order to code these functions It's a little bit different to normal applications If you are working with lambdas or any other you know function platform It's pretty much the same, but you need to kind of like be aware as a developer The function that we have used here for the levels are all synchronous So they are all have this kind of like latency problem You need to wait for the functions to return in order to continue playing the game in this case But because For that we are only using K native serving right like the serving part of the project We can also start using the eventing part of the project which will give us like a more reactive approach Right, so in this case what we have done for the eventing side of things is we are using the K native Eventing broker and triggers to basically route cloud events inside the architecture So every time that we score like one of these levels We are emitting a cloud event to a K native broker Which is a clouds event router and that event is being routed to the front end Which is then forwarded to the user interface, you know, that's client side. It's in my browser running here So we have cloud events running all over the like moving around all over the place And we have this very very genetic infrastructure that we can wire dynamically for our applications We can connect more pieces if you want to we can route those events to multiple functions to keep doing processing and stuff In my opinion this provides you this gives you a very nice developer experience because when you are building functions In order to use all this messaging and cloud events and all that stuff as a developer I only need to use HTTP I don't need to just you know connect to a message broker or connect to Kafka or have an extra dependency in my application to Generate cloud events and in this case we are using our socket as like a bidirectional channel from the you know From the cluster to the browser in this case So pretty simple and complicated stuff at the same time We are planning to put this you know this this example publicly available so people can play around with it and also you know use it Like your own trivia right exactly so you can make your own trivia You can also show it like around the people can do the rubber cup in Detroit We can do that we can do that we can do a hackathon as well, right? Yeah Before I start and share this playing I wanted to just run a set of commands I think that that's pretty much it. I want to check that the leaderboard is there You get the QR code. Yes, I will just share it in a bit So as soon as this is you know working which is now right now I will just share the QR code with you folks. So if you have your phone Open your phone You can scan this code. I want to have everyone a picture of everyone playing If the QR code doesn't work there is a link there as well yesterday, I think that we hit the Like the limit for a free service QR code my bad now using an NPM CLI A thousand dependencies. I can see people playing. So I will take off the the QR thing, right? Is that all right? Everybody got it. Everybody's up Who can guess where those names are coming from? Generating names five minutes all right and I will switch here and Look at that.  Sorry. I Can show the pots on the cluster. That's very good. Evan. Thank you very much And that's we're not using lambda. We're not using cloud around. So scuba net is playing simple Yeah, look at that. So you have all the pots there Like CPU for the front and it's going hard because we have a single replica there We are going to change that pretty soon And then I can go back to the web. We're using WebSocket. So one thing that I think I mentioned when the milestone was yeah, some people Use cloud function for others, but we can use WebSocket's HTTP to gRPC Look at that. We have like 115 people playing there. Yeah, I told you to share. Yes So yeah, so if you finish you can tweet your score And he has a link right so people can find it on Twitter also play if they're on Twitter That's pretty cool. Who's winning? nice Rosalinda Rosalina 2 and second place happy Carver 4 and third zeroes made there too. This is good This is a score 68. Who has it? Who has a perfect score? No one? I think that yeah nice Rosalina Rosalina is okay So she means where my might be from Valencia from Spain from Spain cool stuff. All right. I think that's Do we have an extra slide? Yeah, I think the end the end slide right we do have an extra slide. That's true Yeah, so it's a call out for the community to join us like I said you can help in any capacity You're good writing docs. You're good advocating. You're good Organizing and working with the community we need help in that and also security network all the aspects of the of the Project so we we work on everyone that wants to either learn or contribute Yeah, and I want to say like functions is definitely an easy entry point for people that it's already coding different kind of applications It's a polyglot place. So if you want to come just reach out There are a bunch of K native function maintainers here as well So, you know, just get in touch and templates they can help with templates templates for examples And the rising in your in your company if you have like we build functions like this You can create your own template and then your developers just take that template and that's your standard in your company to write the functions Yep, whatever is the monetary agent or thing that you have to have in that library that you want You don't want to explain to people how to get it the template actually gets gets it for you Questions one one question on a line that you have a secret now. Yeah, at the end at the end you can come come here Yes, any other questions other questions Do virtual oh no from virtual we don't have anything Guys, thanks a lot. It was great. I want to complain about the game because it was bias for the people that are from Spain from Yes No, I'm joking. That's great great stuff, you know what? I'm coming from sort of different words the telco world I was on Vodafone and we are running now a new a lot of new activities on cloud and I got into coming K native from colleagues, you know supporting us other companies, and it's really really very interesting I want to do we want to do things with K native But you know in our world everything stops when you reach Operations, so I was wondering Observability areas. Yeah, like when we launch that and we start spinning up functions and services How does the operations people don't go nuts? I mean they can observe they can monitor and so on That's that's my question a bit of information on that Thanks, can we be the question? Yes, so the question is about like observability and what are we doing around observability inside the project? What Efficiently how are you doing efficiently? Okay, so so the question is you as a platform team in charge of the cluster and giving The developers access to to the functions how the best to run a monitor that cluster I guess Right. So one level is the Kubernetes level that we're just launching pods launching deployments So those you just took your your standard monitoring stack and then we have even can talk a little bit more After we finish in details about the open sensors and the open telemetry of the controllers of K native I think that's also you want to watch is because it's that's that's your owning those controllers So we have a couple of controllers like auto scaler the main controller the auto scalers They have metrics endpoints that you can grab information from them. That's a what And and the Q proxy cycle so those are like K native Components that you want to tap into observability to see how are the health and how do you scale them? So there's there's aspects of like making the HPA of those components so they can have redundancy and scaling So those those type of type of things are in the website some of that information is in the website But you can reach out to us and there's also some Grafana Community-driven Grafana dashboards and the stack that you can tap tap into it and get started on on that But a lot of people use all their monitoring services, right? Yeah To your point is like if you are already monitoring Kubernetes, you just keep doing yeah But I think people ask like well, what about the others the native components Those are built like cloud native way and have instrumentation into it But definitely we need people like you folks of like what are the best patterns for that to have more Documentation and examples on how to integrate maybe we do with third parties. That's a good question Yep, I think we're out of time. We're out of time, but feel free to come And stickers like I can see Maria there picture


I am Albert Einstein. I am Albert Einstein. Ever been called weird or different? That's what they thought I was.  Is that normal? Don't worry. He'll be just like everyone else. Doc, I wouldn't be so sure about that. He didn't get easier. I did things my own way, in my own time. I didn't speak until I was three years old. And when I did, my speech was so odd. Our maid used to call me DER DEPETER. What's DER DEPETER mean? That dopey one. Lady, I wouldn't be so sure about that. Some say I took longer to speak because I didn't think in words. I thought in pictures. And when I did speak, I'd practice each sentence in my head silently, moving my lips and whispering to myself until I had every word right. Public speaking, one-o-one. MUM DER DEPETER. MUM DER DEPETER. Albert, what are you saying? Are you sick? Son, what's wrong? I don't understand. Are you hungry? Do you need vinegar, Schnitzel? Does something hurt? Do you have to go to the bathroom? MUM DER DEPETER. KU KU. Flying? You want to fly? Wait, he's trying to say something. My hair is so awesome. When I was little, my cousins ran around and played games outside. I liked playing alone. I did puzzles, fed the pigeons, or just watched my toy boat sail in a water pail. When they saw me, other people called me Father Boar. GOOTY GOOTY. What are you doing, LB? Watching it sail. That's it? It helps me think. I like thinking. But the biggest moment in my young life came when I was four or five years old and sick in bed. To cheer me up, my father... Sample complete. Ready to continue?


Welcome to my lecture on experimental design in simulation. We have to remember that once the model building is over and is validated and we have a valid model to work with, to simulate with, the job isn't really done. The whole point of the experimental design phase is to make note of the fact that we are going to analyze the results of the simulation. We're going to use the simulation to generate data and to learn even more about the system than we did during the model building phase. When we analyze the data, we're going to try to extract as much information as possible from it. Of course the information has to be in there to be found and that's where the design comes in. The goal of the design of the experiment is to make sure that the data contains as much relevant information as possible. And of course we know that simulation projects have a lot of time and budget constraints and so we're going to make sure we're practical as well as trying for the max. It's always interesting to compare simulation experiments to statistical experiments in what we might call the real world. Sometimes you might call it a field experiment. When we have to go out into the world and collect data, whether the data involves human subjects or not, machine parts. And we have to collect data sufficiently to construct an experiment in order to test the responses, the response variables that we're interested in. It often requires a large amount of data. Often it requires a lot of data that we really can't get easily, maybe not even at all. Simulation experiments allow us to manipulate randomness. They allow us to control randomness. We can control the environment, the random phenomena so that we make sure that we have this exactly the same environment at every single design point. That's something you can't do in the real world. It makes it very interesting to design experiments for simulation. Expanding in the real world is so expensive. Each end value, each data item that's collected adds an additional expense to the study. And so we try very hard to keep the experiment small in the real world instead of large enough or maybe only large enough. In simulation analysis, on the other hand, a lot of the expense goes into building the model in the first place. Once the model is built and we can use it to generate data, it's a very simple thing to create a lot of different factor levels, a lot of different design points, even looking at many different response variables. And it doesn't really add very much to the cost of the experiment. Unlike in the real world where every data value has a great deal of variable cost that it adds to the experiment. Once we know what the objective of the simulation study is, we can start to identify the variables that we're interested in. Some of the variables will be control variables, will be independent variables, which we'll call factors in experimental design terminology. These independent variables have values that we set beforehand, values that we're interested in studying. We're typically going to be setting the values of independent variables rather than just measuring them. The response variable is the dependent variable and that's the variable we're interested in. That's the reason we're doing the study in the first place. And whenever we're interested in learning about, it's going to have something to do with the response variable, with the dependent variable. What you might have been calling the Y variable and we still will call it that again. Factors can be quantitative and it can be qualitative. Quantitative factors naturally have numeric values like the number of machines in a workstation, the number of tellers in a bank. Qualitative factors represent structural assumptions and they might be categorized. For instance, Q-discipline could be either LIFO, FIFO, priority, and so on. The shortest job first, there's a good one. So sometimes they're qualitative and they'd be coded. We might be referring to this variable as having values one, two, and three for the different values of the Q-discipline. There are two types of variables in the area of experimental design that are not quite factors and certainly not responses. We don't worry about them as much in simulation because these variables in real life are variables that can't really be controlled. But in simulation, of course, everything is controlled. Nuisance variables affect the behavior of the system but can't be controlled directly. And intermediate variables are affected by the factors, by the independent variables. But we can't control them, we measure them. So between measuring and monitoring in the real world, we do the best we can with these other variables that are not quite factors and not quite responses. Sometimes the goal of the simulation implies a particular experimental design must be used. Sometimes the goal of the simulation means that there are a number of experimental designs that can be used to achieve that goal. Some of the more commonly used designs or a better way to look at it might be goals, objectives of the simulation study are on this slide. And this is what we're going to be continuing to look at for the rest of the lecture. We might want to estimate a parameter. We might want to look at different systems and compare the value of the estimated parameter. We might want to look at more than just a one-way comparison and that would mean we're looking at factorial designs. We might want to look at a lot of different alternative systems, one-way, two-way, whatever, and optimize the value of the response variable. Find the best system for what we're studying. Sometimes we want best and second best and maybe third best and then we'd be looking at ranking the alternative systems. Sometimes we're more interested in isolating the important factors instead of looking only at the response variable, factor screening. And sometimes we want to look at the entire relationship between the inputs and the outputs between the factors and the responses. And we can do that very nicely with the simulation metamodel. Parameter estimation is one of the most common goals of any experiment, including the simulation experiment. By this point in the course, you've already done some of your own simulation homework and you've looked at estimating performance measures, measures of effectiveness from the results of the simulation run. So this probably speaks to you very clearly. Sometimes the parameter estimation is even taken as a generic. When we talk about simulation output analysis, we assume, oh, we're outputting the values of a measure that is going to be averaged at the end of the run. And we're going to use that to estimate the parameter, the real-world system. We don't know the real-world system when we're simulating much in the same way that we don't know the population when we collect a sample from the real-world system and use it to estimate a parameter. Simulation is a little different because we can make it more or less complex as we wish. And also, we can control runs by controlling the randomness, the random numbers or the random variables that we input into it. So that just makes the experiment a little bit more interesting, but in the end, if we're estimating a parameter, we're estimating a parameter. Probably the second most popular or most typical experimental design in a simulation is one where we're comparing alternatives. It might be alternative systems that we're thinking of building. It might be alternative strategies, things we may be thinking of changing. If we have an automobile repair shop or an airplane maintenance section center, we could be thinking of how many servers do we need? How much machinery do we need to process the large vehicles that come in? So we could try different configurations of the system before we build it, and it's certainly better doing it that way in a simulation than doing it in the real-world. It's not even possible in the real-world. And also, there are all kinds of behavioral issues in the real-world. Suppose we want to test two tellers versus five tellers in a bank with various other inputs. Hiring people, moving them around just for the purpose of an experiment, you have to train them. Sometimes it's just too much getting in the way. We're just changing too much to do this experiment in the real-world. But in simulation, that's not a problem. Now you might think, similar to the real-world, that if I run my simulation or my system with one strategy one week and then do that with the other system the next week, I can't compare anyway because I have different configuration of demands coming in, different service times perhaps. The thing is with simulation, I can control all that because I can control the input distribution. I can control the input variables that are generated from the input distribution that come into the simulation. Basically, I'm controlling the arrivals, I'm controlling the customers, the entities, the cars, the trucks, the airplanes that are coming in. And I can make sure that every time I run a new configuration, it's the same sequence of arrivals coming in. Let's look at a small example. Suppose we're interested in comparing two alternative systems. System one, we arrange for multiple queues, one queue in front of each server. And with system two, we just have a single queue and the individual at the front of the queue goes to the next available server. We have run 26 independent replications for each of these systems, each system alternative. And we're examining the response variable we're looking at is wait time. How long does an average customer have to stand or sit or wait in the queue before being served? So in this case, we're looking at it from the customer perspective. There might be other opportunities where we're looking at something similar but let's say examining utilization of each server and how the two different systems react. At any rate, you see the data for this example on the left, the raw data, well the averages from each run. And we've got on the right the summary statistics, sample size is 26 in both. The mean for system one 6.12 the mean for system two, not that much different 6.18 and the standard deviations point 016 and 013. So the question is, are these different, or are these basically two different samples from the same ultimate population, which would mean that there's really no difference in waiting time. The population parameter is the same, no matter what queuing type or queuing arrangement structure that we're using. Here are the results from MS Excel, Microsoft Excel, you could be using just about any statistical software for this. This is just one example of what you might use in order to analyze this using a t test with assuming equal but unknown variances. It's sort of halfway down, you've got what's called t stats. That's the computed value of the t statistic. And if you know anything about t tests, a t statistic of negative 14.85 is is huge. And we don't even have to look at the p value anymore I mean we know that there's a large significant difference between the two systems, even though when we looked at the data. It didn't look like a very large difference, but don't forget the standard deviations were quite small. The p value, if we're looking at a two tail test, the p value is 5.7 times e to the minus 20. I mean there's a lot of zeros in there 0.000000. If you're working at the oh five level of significance it's going to be way way less than oh five. And you even given the look up at the table look up the critical value from the T with 50 degrees of freedom. And 0.05 alpha equals 0.05 is 2.00 or whatever. And so the conclusion is there is a significant difference between these two systems with regard to time spent waiting in queue. A factorial design is similar to the experiments of comparison. But affect those are one way. Usually that's that we think of them as one way. A factorial design is not one way it's a cross study. So for example, in the simple case where we're looking at two control variables and one response variable. We might be looking at single queue versus multiple queues of one in one in front of each server. We might also be looking at traffic in the system help the customers may come in very very quickly very small interval time. They may come in spread more spread apart. So we could be looking at three levels of average into arrival time. And that gives us a three by two factorial design with. So that's six cells. And in each cell, that's going to be the result of the run of the simulation. Let's say we're looking at time and system. And for any customer in the system. So each one of those is the result the average output from a single run. We could then have replications in each cell also. So that's the essence of a factorial design. But we're going to look at a different types of factorial designs that are used in general and specifically in simulation. There are several specific types of factorial designs that are studied in this field. For the most part, they're trying to allay the disadvantage, the main disadvantage that the number of cells in a complete factorial design grows very quickly. And you saw the previous slide in a simple two variable three by two we end up with six cells. Three by three would be nine. And what if we had another variable? What if we had five variables control variables that we're looking at? So the number of runs, you know, the number of the sample size grows very, very quickly. And that's only if we want to do one run in each cell. What if we what if we want replications? It turns out though that this is more of a problem in the real world than in the simulation world as we saw before. Because the main expense of the simulation comes before we actually generate the runs. Sometimes there's a simulation that's very, very complicated and takes a very long time to run and having to generate extra runs would make a difference there. But for the most part, this is more of a problem in the real world. However, because of the goals of typical simulation projects, we do often use specific types of factorial designs. And we're going to look at 2k factorial designs and fractional factorial designs. First, let's look at what we might call the one at a time approach where we have more than one factor. We want to look at the contribution or the effect at each of these factors on the response variable of interest. We could do this one at a time, take each independent variable, each control variable that we're calling X, take X1, look at it separately, do the analysis, take X2, do the analysis and so on. There are some statistical issues with this, but probably one of the most important is that we won't be able to test to see if there is an interaction effect. We're only looking at these separately in terms of the main effect. And in addition, of course, we need to make many, many simulation runs in order to get the precision that we need. But probably more important from an overall point of view is this notion of maybe we want to test for interactions. We're going to see this a little bit more depth on the next slide. Here's an example. Suppose we're looking at two different control factors in a system, a queuing type of a system. We're looking at how experienced the server is and we have only two design points for that. We are looking at what we call experienced servers, those who have more than or greater than or equal to five years of experience. And we're looking at comparing them to the inexperienced servers, the trainees who have less than one year experienced. On the other hand, the other factor we're interested in is seeing how the servers handle low traffic. In other words, a system that's not terribly crowded as opposed to heavy traffic. And we know from our study of queuing systems that we can identify a low traffic system as something with large inter-arrival times. The span between successive customers entering the system is large and heavy traffic would have small inter-arrival times. The responsive variable here is something based on time in minutes. And let's say what we're looking at is not necessarily time in the system, but we're looking actually on server performance. So we're interested in the service time, the actual service time, how long it takes the server to complete the transaction. Now if you look at the data and you just look at averages of each factor ignoring the other. Experienced versus inexperienced, the totals are very, very close. Probably you won't see any difference between experienced and inexperienced servers with regard to service time. Low traffic versus heavy traffic, you look at the totals, same thing. You won't see that much of a difference between the service times in low traffic system or heavy traffic system. And yet there is a difference. And the difference is all in the interaction. The experienced servers in low traffic take longer to complete their transaction. Perhaps they're chatting, enjoying the social interaction with the customers. And yet in heavy traffic, they speed up in order to move things along. And then they have an average of 5.5 minutes service time. The inexperienced servers on the other hand, in low traffic, they're about what they should be probably 6.1 minutes. Compared to the low average of 5.5 minutes that the experienced servers have in the heavy traffic. So the inexperienced workers do their job 6.1 minutes on the average in low traffic. But all of a sudden when the system gets crowded, they start to panic. They know they're supposed to be moving people through, but they're not quite sure how to do it. And they don't have the self-confidence to do that quickly. And they slow down in order to make sure to get it right. There's the interaction without testing for an interaction effect that won't be picked up. A better approach, and one that still includes some layers of efficiency, is what's called 2K factorial designs. Where for every control factor that you have, you have two levels, much like we did in the previous slide in the previous example. You have two levels of each X of each control factor. And you set them far enough apart so that you'll be examining the effect in a realistic manner and also coming up with enough wiggle room to look at the interactions. So you choose two levels for each factor. All together, those are the design points. You assume that the response is approximately linear over the range of the factor. Because you want to make sure that when you interpolate, that you're relying on it on the data to follow your assumptions. You might be missing something. If the actual data output is not linear, if the actual data is not linear, you're looking at only two points of the factor. You might actually be missing something important. So here's what a 2 to the 3, where K is 3, factorial design might look like. This is called a design matrix. We've got, in this case, three factors, factor 1, factor 2, factor 3. We've got the response. Factors 1, 2, and 3 are what we might call the X's factor. The response is what we might call the Y. And remember, in every statistical study, it's the response that we're interested in. That's what we're studying. So if we have three factors, and each factor has two points that we're studying, a low and a high, signified by, noted by negative and the positive, the minus sign and the plus sign. We end up with eight design points. The first design point has factor 1, factor 2, factor 3, all at the low point. And then the second design point has a plus and then a minus minus. So factor 1 is at the high point, 2 or 3. And you can see that basically this is a permutation. And we've got eight design points with the eighth one being plus, plus, plus, the high levels of all of the factors. So it's a very nice, simple way to set up a factorial design where we're studying all the main effects and the interactions. And we're also making use of efficiencies in collecting our data. You can probably see from the example in the previous slide that as K gets large, the number of design points pretty much explodes. And so what you might start to think is, can I reduce the size of my experiment so that I have fewer design points? And still get practical information out of it. And when you do that, that's called a fractional factorial design where you're not looking at the full 2K design. And you're probably giving up on the opportunity to study some interaction effects, but you will be able to study certain of them. So this involves much more expertise in order to reduce the size and the complexity of the design of the study and also be within budget. Sometimes we're specifically examining the factors. We not only want to study what makes our response variable best, what gives our best performance for the measure of effectiveness. But we're specifically interested in which factors contribute the most amount of information to studying the response variable. And that's called factor screening. We might be using 2K factorial designs, fractional factorial designs, other types of factor screening designs. But the ultimate goal is really to examine which control factors are important and must be left in and which can be left behind and considered part of the random variation or noise factor. Optimization is one of the most common objectives in a simulation. When we say optimization, what we mean is we want to come up with the levels of the control factors that produce the best response, the best value of Y. So if you imagine a regression equation, although I have to say we're not usually imagining a linear function for simulation output. But if you imagine a regression equation, you pretty much get the idea and you've done this in regression, I'm sure. In nonlinear systems, in simulation systems, in large data systems, this might be called response surface methodology. We're not going to look at this from a practical point of view. If you want to go into it, at least you know what it's called and you can learn more about it. We can do this with a regression model. That's the topic of regression metamodels and we're going to look at that in its own lecture. And in addition, let's not forget that we could also really be doing what if analysis here. It may not be the most sophisticated way of going about it, but it's certainly something that's doable, easy, and something that everyone understands. Just as in real life, we may want to optimize and find the best possible solution, but we may be satisfied with the best possible solution in a price range in our budget. And so we may not only want to look at the optimal, we may want to rank alternative systems with regard to the response variable of interest. So that we could look at the second best or the third best and certainly we can satisfy. And that's exactly what the design called ranking and selection of alternative systems is doing. Most of the experimental designs we've looked at here assume an underlying linear model, which on closer examination may or may not be true with our simulation data. A metamodel is where we're explicitly looking at a model, whether it's a linear model or a nonlinear model to represent our simulation. We know that a simulation is a transformation of inputs to outputs, right? And if we can take our simulation inputs and our simulation outputs and generate an analytic model, think in your head of a regression model, although the form might be slightly different. Then we can learn a lot more about the interactions between the inputs and the outputs and about the interactions among the Xs and even among the Ys because we're assuming there would be more than one response variable. And we can learn a lot just by looking at the form that the equation takes. The metamodel itself as an experimental design technique we're going to look at in its own lecture coming up soon. You may think from looking at the experimental designs studied here and from what you've done in your earlier statistics courses that we always are looking at only a single response variable. There's always just one Y that we're studying. It might be waiting time in queue. It might be total time in system. It might be the utilization of the servers. It might be maximum queue size. Whatever it is, it seems like we have to pick one, but we know that that's not realistic. That doesn't really happen in the real world. In the real world, there's always more than one thing that we want to study for one thing. These are complicated systems. We're putting a lot of work into it and a lot of money into it and we might as well study more than one variable for that for the money. So now we have to consider can we just go along merrily doing what we've been doing and just do it repeatedly. I have the analysis that I want to do, the design that I've created for a single response. I'll just do it over and over again. The same thing for waiting time. The same thing for size of the queue. The same thing for utilization of server. And we'll have independent sections in our report for the analysis. Well, the fact of the matter is there are several things that are wrong with this. The most obvious is the significance level for these tests. And if you think you're performing each test at a particular significance level alpha, where alpha is the probability that you're rejecting the null hypothesis when really you should not reject it. You're really operating at a different one because you have to look at the entire experiment. And you, these individual univariate tests are not independent because they're all on the same set of data. And if you're increasing the number of tests, then you're naturally increasing the probability that you'll reject falsely and so you're very likely to reject something. Or if you do several tests at a probability of type one error of point one, and you do many, many of these univariate tests, the chance that you're going to reject something just keeps increasing. And you can actually compute that you compute the experiment wise alpha rate, but that doesn't help you. If all you can do is compute it and report it. It's something, but it's not exactly the end of the matter. Let's see what happens on the next slide. So if we know the alpha error we would like to be working at, and we can compute the experiment wise error rate. So what we would really like to do is use that experiment wise error rate, take the number of univariate tests we are doing and manipulate this algebraically or by the Bonferroni approach in order to come up with the level of significance alpha that we should be using for each univariate test. So that's one way to do it. It's still leave something to be desired because it doesn't look globally at the inter-correlation among the various why variables. Now, if you've taken any course in multivariate statistics, you see that there are techniques that have been developed specifically for this reason. And even if you haven't taken a test yet, a course like that yet, you really should or you should try to do this on your own if you can, but that's the solution to the multiple response problem. And just about every univariate technique that we've discussed or that you've studied in your statistics course has a multivariate equivalent. Sometimes the researcher wants to avoid using multivariate techniques, which I don't recommend, but one way that they might do that is to say, well, let's take all of the responses and make one out of it. Take all of the possible responses, combine it into some sort of a formula and have one overall metric that represents the system. It could be some sort of a utility function. You lose a lot of interesting and important information in this way, but it is one way of simplifying and solving the problem. Although, as I mentioned a minute or so ago, using the multivariate techniques that were created for exactly this situation gives you much more information about the responses that you've gotten. As far as is possible and practical, you do want to validate the experimental design, even before you generate the simulation runs to collect the data. You can do that by making assumptions, run, use constants and see what happens, try your design, have made up data for all of your design points. You can perhaps run one single simulation experiment, one single simulation run, rather, and use that data to come up with your values for testing. It's sort of like testing the stress of the experiment, the experimental design. What you want to do is you want to look for issues that might come up when you get your actual data. One way of testing, of testing the stress on a system is using weird numbers, put in very, very large values. Imagine that you had very large metrics, large MOAs that came out. Imagine that they were small, imagine some things are zero, and then put that into the experiment as if that were the data that you came up with, and test. These are some ways that we validate an experimental design. In this lecture, we looked at experimental design as one important phase of a simulation project. We know that when we design the experiment, it's so that it will contain the most possible information. When we actually do the data analysis, that's when we would like to extract all this information from the data. We have looked at several different experimental designs that are typically used in simulation experiments. And we have looked at issues such as the variables that are involved, the responses, the factors, whether we have a single variable, a univariate solution, or multiple response variables, the multivariate problem analysis. And even though this is a very, very long lecture, what it probably did for the most part is point you in a direction where you could go and study further and relate it to whatever statistics courses you might already be taking or have taken. Thank you for joining me in this lecture.