We'll continue our review of basic integer operations. And again, this is a review. It assumes that at some point you learned how to do arithmetic with side numbers, but you need some sort of refresher. And again, we have a set of rules for dealing with sign numbers followed by examples. This is the worst way to learn mathematics. You cannot learn mathematics by memorizing a set of rules and following some examples. If you haven't already learned the arithmetic of sign numbers, you won't be able to learn it by watching this video. Instead, I recommend that you watch these videos to learn the rules of sign numbers. So let's take a look at this and do a quick review. Given two integers A and B, I have the following rules. The negative of the negative of A is the same as A itself. The product of two sign numbers, if I have two negative numbers, the sign drops, the quotient. Likewise, if I have two negatives, the sign drops. If I only have one, that negative can attach to any of the factors. Likewise for the quotient, that negative can attach to any of the dividend or the divisor. That negative also can be pulled entirely outside the quotient or factor or product. And if I use the commutative and associative properties of multiplications, I can transform any statement involving sign numbers into a different statement involving sign numbers that I might be able to do. For example, negative five times three. So my first property, my property of the integer is negative A times B. Well, that's the same as negative A times B. So I can remove that negative to the outside, and this is going to be five times three. And now I have a set of parentheses. And the parentheses say do me first. So five times three is 15. Parentheses say do the stuff inside first. And there's nothing to do at this point. So I don't need those parentheses anymore. 15 divided by negative three. Again, I have a very similar property. Negative A divided by B is the same as A divided by negative B. So this negative can attach to either of the factors, and either the terms. And so I'll go ahead and move that to the numerator. This is negative 15 over 3. Another property says the negative of a quotient is the negative of the quotient. So I can move the group that 15 divided by 3. And again, parentheses say do this first. 15 divided by 3 is 5. And there's nothing more to do inside the parentheses. So I can leave the problem at this point. Five times a negative 8 divided by 2. Multiplication and division are co-precedent, which means I have to do them in the order that I see them. I multiply by negative 8 first, then I divide by 2. So I can begin by using the property. If I have a product that negative can attach to either factor, that negative that is attached to the 8. I can detach it and attach it to the 5. And again, my next property, the negative of a product, is the negative of the product. So that's five times 8 negative. Do the stuff inside the parentheses first. Negative 40. And again, if I have a quotient, negative something divided by something, that negative, I can separate out the whole number portion of the arithmetic. 40 over 2. Do that first. And I don't need the parentheses anymore.


Welcome to All Hands Update, I'm Petty Officer Brandi Wills with your headlines from around the fleet. The Navy announced June 4th, the establishment of the 21st Century Sailor Office. The office is responsible for the integration of the Navy's objectives concerning numerous areas including personal and family readiness and suicide prevention. For more information, read Navadman 153-13. The Department of Defense launched Safe Help Room May 30th in response to Safe Helpline users who identified a need for peer support services. Safe Help Rooms are available twice a week in two-hour sessions. To see the schedule, go to safehelpline.org. You can read more about these stories at navy.mil. From All Hands Update, thanks for watching.


Good evening. Tonight is the second night of the 2021 scary stories reading and in a few minutes I'm going to start reading one of my favorite favorite creepy ghost stories Now in the meantime, we'll give you a few minutes to climb on in sit down get comfy Pull up a chair turn out the lights and get ready to listen But I'll give people a few memories to do this I'm going to share this with a few different parts in social media Gotta bother the people on Twitter. I have to hit up my good friends on Facebook and even on LinkedIn Let's see how they respond. In the meantime, how are you all doing? I hope you're all safe and sound I hope the season is treating you well. I Hope you're all comfortable Yet ready to get made a little nervous To be a little creeped out while I go if you'd like to say something in the chat, please feel free The chat is there for you now To get things rolling a few words, what's nice story Emperor James was a British academic who on the side and purely for his unentertainment Wrote some of the best short stories in the ghost story vein in the English language These are notable for being partly funny very satirical, especially of Cambridge Don's But also very creepy in a delicate way where the story builds up meticulously until the Creepy horror is hinted at just very very gently But tonight's example is one that I'm very fond of for a lot of reasons and one is that it's partly about multimedia And this is a little story from 1904 from his first great collection ghost stories of antiquity Called the Met's intent some time ago I believe I had the pleasure of telling you the story of an adventure which happened to a friend of mine By the name of Dennis Dune during his pursuit of objects of art for the museum at Cambridge He did not publish his experiences very widely upon his return to England But they could not fail to become known to a good many of his friends and among others to the gentleman who at that time presided over an art museum at Another university it was to be expected that the story should make a considerable Impression on the mind of a man his vocation lay in lines similar to Dennis Dune's Now he should be eager to catch any explanation of the matter which tended to make it seem improbable They should ever be called upon to deal with so agitating an emergency It was indeed somewhat consoling to him to reflect that he was not expected to acquire ancient manuscripts for his institution That was the business of the Shelburne and library The authorities at that institution might as they pleased ran sec obscure cores the continent for such matters He was glad to be obliged the moment to confine his attention to enlarging the already unsurpassed Collection of English topographical drawings and engravings possessed by his museum Yet as it turned out even a department so homely and familiar as this may have its dark corners And to one of those mr. Williams was unexpectedly introduced Those who have taken even the most limited interest in the acquisition of topographical pictures are aware There is one London dealer whose aid is indispensable to their research is mr. JW Brittnell publishes its short intervals very admirable catalogs of a large and constantly changing stock of engravings plans and old sketches of Mansions churches and towns in England in Wales These catalogs were of course the ABC of his subject to mr. Williams But as his museum already contained an enormous accumulation of topographical pictures He was a regular rather than a copious buyer And he rather looked mr. Brittnell to fill out gaps in the rank and file of his collection than to supply him with varieties Now in February of last year there appeared upon mr. Williams desk at the museum a catalog of mr. Brittnell's Emporium and accompanying it was a type written communication from a dealer himself The latter ran as follows Dear sir, we beg to call your attention a number of 978 in our accompanying catalog Which will she be glad to send on approval. You are as faithfully JW Brittnell To turn to number 978 the accompanying catalog was with mr. Williams as he served to himself The work of a moment in a place indicated. He found the following entry 978 unknown interesting mezzo tint view of a manor house early part of the century 15 by 10 inches black frame two pounds two shillings It was not especially exciting and the price seemed high However, as mr. Brittnell who know his business and his customers seem to set store by it Mr. Williams wrote a postcard asking for the article to be sent on approval along with some other engravings and sketches Which appeared in the same catalog and so he passed without much excitement of anticipation to the ordinary laborers of the day a Parcel of any kind always arrives a day later than you expected and that a mr. Brittnell approved as I believe the right phrase goes no exception to the rule It was delivered at the museum by the afternoon post of Sunday Saturday after mr. Williams and left his work and it was accordingly brought round to his rooms in college by the attendant in Order that he might not have to wait over Sunday before looking through it and returning such of the contents as he did not propose to keep And here he found it when it came in the tea with a friend The only item which I am concerned was the rather large black frame that's a tint of which I have already quoted the short description Give the mr. Brittnell's catalog Some more details it will have to be given but cannot hope to put before you the look of the picture as Clearly as it is present to my own eye Very nearly the exact duplicative that may be seen in a good many old-in parlors or in the passages of undisturbed Country mansions at the present moment It was a rather indifferent mezzotint and an indifferent mezzotint is perhaps the worst form of engraving No, you presented a full-faced view of a not very large manor house of the last century With three rows of plain sashed windows with rusticated masonry about them a parapet with balls or balls or vases at the angles and a small portico in the center On either side were trees and a front a considerable expanse of lawn the legend AWF sculpts it was engraved in the narrow margin and there was no The whole thing gave the impression that it was the work of an amateur What in the world mr. Brittnell could mean by affixing the price of two pounds to be shillings There's such an object as more than mr. Williams could imagine He turned it over with a good deal of contempt upon the back was a paper label the left hand half of which had been torn off All that remained were the ends of two lines of writing the first of the letters Inley Hall and the second Sussex Or just sex it would perhaps be just worthwhile to identify the place represented Which he could easily do the help of a gazetteer and he would send it back to mr. Brittnell with some marks reflecting upon the judgment of that gentleman He lighted the candles for now. It was dark made the tea and supplied the friend with whom he'd been playing golf For I believe the authorities of the University. I write of indulge in that pursuit by way of relaxation and Tea was taken to the accompaniment of discussion which Golfing persons can imagine for themselves, but which the conscientious writer has no right to inflict upon any non golfing persons The conclusion arrived at was that certain strokes may have been better than certain emergencies Not our player had experienced that amount of luck which a human being had a right to expect He was now the friend. Let us call him professor banks took up the afframed engraving and said What is this place Williams? Just what I'm going to try to find out said Williams going to the chauffeur gazetteer Look at the back something Lee Hall either in Sussex or Essex half the names gone You see you don't happen to know what I suppose it's from that man Britain. Oh, I suppose isn't it said banks? Is it for the museum? Well, I think I should buy it if the price was five shillings said Williams But for some honorific reason he wants two guineas for it. I can't conceive why it's a wretched engraving There aren't any even figures to give it life That's not worth two guineas. I should think said Berks banks But I don't think it's so badly done the moonlight seems rather good to me And I should have thought there were figures or at least a figure just on the edge in front Let's look to Williams. Well, it's true. The light is rather cleverly given. Where's your figure? Oh, yes, you're just the head in the very front of the picture and indeed there was Hardly more than a black dot on the extreme edge of the engraving the head of a man or a woman a good deal muffled up The back turned the spectator and looking towards the house Williams had not noticed it before Still he said though, it's a cleverer thing that I thought I can't spend two guineas a museum money on a picture of a place I don't know Professor Binks had his work to do and soon went and very nearly up to whole time Williams was engaged in a vain attempt to identify the subject of the picture If the vowel before the NG had only been left it would have been easy enough He thought but as it is the name of the anything from guesting lead a line Lee And there are many more names ending like this than I thought and this rotten book has no index of terminations Hole in Mr. Williams college was at seven It need not be dwelt upon the less so as he met their colleagues who have been playing golf during the afternoon And words of which we have no concern. We're freely bandied across the table merely golfing words. I should hasten to explain I suppose an hour or more to have been spent in what is called common room after dinner Later in the evening some few retired to Williams room And I have little doubt that whisk was played in tobacco smoked during a loan those operations Williams picked up the message in from the table without looking at it and had lit to a person mildly interested in art Telling him where to come from the other particulars, which we already know The gentleman took it carelessly looked at it and then said in a tone of some interest It's really a good piece of work Williams. It has quite a feeling of the romantic period The light is admirably managed It seems to me and that figure though. It's rather too grotesque is somehow very impressive Yes, isn't it to the Williams who was just then busy giving whiskey and soda to what others the company It was unable to come across the room look at the view again It was by this time rather late in the evening and the visitors around the move after they went Williams was obliged to write a letter or two And clear up some odd bits of work at last sometime before past midnight He was disposed to turn in he put out his lamp after lighting his bedroom candle The picture lay face upwards in the table with the last man who looked at it put it and it called his eye as he turned lamp down What he saw made him very nearly drop the candle on the floor and he declares now that if you've been left in the dark at that Moment he would have had a fit But as that did not happen. He was able to put down the light in the table and take a good look at the picture It was indubitable Rankly impossible no doubt, but absolutely certain in the middle of the lawn in front of the unknown house There was a figure where no figure had been five o'clock that afternoon It was crawling on all fours towards the house And it was muffled in a strange black garment with a white cross on the back I do not know what is the ideal course to pursue a situation of this kind You can only tell you what mr. Williams did He took the picture by one corner and carried it across the passage to a second set of rooms Which he possessed there he locked it up in a drawer Sported the doors of both sets of rooms and retired a bit But first he wrote out and signed an account of the extraordinary change which the picture had undergone since it came into his possession Sleep visited him rather late. It was consoling to reflect the behavior of the picture did not depend upon his own unsupported testimony Evidently the man who looked at it the night before had seen something of the same kind as he had Otherwise you might be tempted to think that something gravely wrong was happening either to his eyes or his mind This possibility being fortunately precluded two matters awaited him on the morrow He must take stock of the picture very carefully and call no witness for the purpose and He must make a determined effort to ascertain what house it was that was represented He would therefore ask his neighbor nizbet to breakfast with him and he would subsequently spend a morning over the gaze of tear Nizbet was disengaged and arrived around 9 30 his house was his host was not quite dressed I'm sorry to say even at that late hour during breakfast Nothing was said about the mess attempt by Williams say that he had a picture on which he wished for nizbet's opinion But those who are familiar with the University of life can picture for themselves the wide and delightful range of subjects Over which the two fellows of Canterbury College is likely to extend during a Sunday morning breakfast Are they topics left unchallenged from golf to long tennis? They're bound to say that Williams is rather distraught for his interest naturally centered in that very strange picture Which is now reposing face downwards in the drawer in the room opposite The morning pipe was at last lighted and the morning and the moment had arrived which he looked well Very considerable almost tremulous excitement He ran across Unlock the drawer and extracting the picture still face downward ran back and put it into nizbet's hand Now he said nizbet. I want you to tell me exactly what you see in that picture Describe it if you don't mind rather minutely. I'll tell you why afterwards Well said nizbet. I have your view of a country house English I presume by moonlight moonlight. You're sure of that Oh, certainly the moon appears to be on the wane if you wish for details and there are clouds in the sky Alright go on. I'll swear I did Williams as in the side. There was no moon when I saw it first Well, there's not much more to be said nizbet continued bass has one two three rows of windows Five in each row except to the bottom where there's a porch instead of the middle one and what about figures past Williams With market interest well, there aren't any I said nizbet But what no figure in the grass in front not a thing. You'll swear to that Certainly, I will but there's just one other thing. But why one of the windows on the ground floor Left of the door is open. Is it really so my goodness? He must have got in said Williams with great excitement And he hurried to the back of the sofa which nizbet was sitting and catching the picture from him verify the matter for himself It was quite true. There was no figure and there was the open window Williams after a moment of speechless surprise went to the writing table and scribbled for a long time Then he brought two papers to nizbet asked him first to sign one It was his own description of the picture which you've just heard and then to read the other which is Williams a statement about the night before What can it all mean said nizbet exactly said Williams? Well, one thing I must do or three things now that I think of it I must found from Garwood. This was his last night's visitor What he saw that I must get the thing photographed before it goes further that I must find out what the place is Oh, I can do a photographing myself said nizbet and I will but you know It looks very much as if we were assisting at the working out of a tragedy somewhere The question is has it happened already or is it going to come off? You must find out what the place is Yes, he said looking at the picture again. I expect you're right. He has got him If I don't mistake, there'd be the devil to pay one of the rooms upstairs I'll tell you what said Williams. I'll take the picture across old green This is the senior fellow of the college who've been bursar for many years. It's quite likely. He'll know it We have property now six and saw sex they must have been over the two counties a lot in his time Quite likely he will sit nizbet, but just let me take my photograph first But look here. I rather think green isn't up today. He wasn't in a haul less than I think he said he was going down for Sunday, and that's true, too. So Williams. I know he's gone to Brighton Well, if you'll photograph it now I'll go across to Garwood and get his statement and you keep an eye on what I'm gone I'm beginning to think two guineas is not a very exorbitant price for it now You know short time he had returned and brought Mr. Garwood with him Garwood statement was to the effect of the figure when he had seen it was clear of the edge of the picture I had not grown across the lawn He remembered a white mark on the back of its drapery, but he could not be sure it was a cross Document to this effect was drawn up and signed in his bed proceed to photograph the picture Now what do you mean to do? He said are you gonna sit and watch it all day? Well, no, I think not siblings I'd rather imagine we're meant to see the whole thing You see between the time I saw it last night and this morning there was time for lots of things to happen But the creature only got into the house You could easily have got through its business in the time and gone to its own place again But the fact of the window being open I think must mean it's in there now So I feel quite easy about leaving it and besides I have a kind of idea It won't change much if at all in a daytime We might go out for a walk this afternoon and come into tea or whenever it gets dark I shall leave it out on the table here and sport the door my skip can get in but no one else The three agreed this would be a good plan and further if he spent the afternoon together There'd be less to talk about the business to other people for any rumor such a transaction as what's going on would bring the whole Lafazmatological society about their ears We may give them a respite until five o'clock At or near the hour the three were entering William staircase They were first slightly annoyed to see the door of his rooms was unsported But in a moment he was remembered that on Sunday the skips came for orders and hours so earlier than a week Days however a surprise was awaiting them The first thing they saw was the picture leading up against a pile of books in the table as it had been left and the next was William skip Seated in a chair opposite gazing at it with undisguised horror How was this? Mr. Filtcher the name is not my own invention was a servant of considerable standing and set the standard of etiquette To all his own college into several neighboring ones and nothing to be more alien to his practice than he found sitting on his master's chair We're appearing to take any particular notice of his master's furniture or pictures Indeed he seemed to feel this himself He started vying on him the three men were in the room and got up with a marked effort Then he said I asked her pardon sir for taking such a freedom as to set down not at all Robert and opposed Mr. Williams I was mean to ask you for some time what you thought of that picture Well, sir, of course, I don't set up my opinion against yours, but ain't the picture I should hang up where my little girl could see it sir. Wouldn't you rubber? Why not? No, sir. Why the poor child I recollect once she saw a doer Bible with pictures Not half what that is and we add this up with her for three or four nights afterwards if you believe me And she was just catch a sight of the skeleton in here whatever it is carrying off the poor baby She'd be in for taking you know how it is with children our nervous They get for the little thing at all which I say it don't seem a right picture to be laying about sir Not where anyone that's liable to be startled could come on it. She'd be wanting anything this evening, sir Thank you, sir With these words the excellent man went to continue the round of his masters And you may be sure the gentleman whom he left lost no time in gathering around the engraving There was the house as before under the waning moon the drifting clouds the window That had been open was shot and the figure was once more on the lawn But not this time crawling cautiously out of hands and knees now It was erect and stepping swiftly with long strides towards the front of the picture The moon was behind it and the black drapery hung down over its face so that only hints of that could be seen And what was visible may the spectator is profoundly thankful that they could see no more than a white dome like Farhead and a few straggling hairs The head was bent down and the arms were tightly clasped over an object Which could be dimly seen and identified as a child whether dead or living. He was not possible to say The legs of the appearance alone could be plainly discerned and they were horribly thin From five to seven the three companions sat and watched the picture by turns, but it never changed They agreed at last that it would be safe to leave it. They would return after a haul of a wait for the developments When they assembled again at the earliest possible moment the engraving was there But the figure was gone and the house was quiet under the moonbeams There was nothing for it to spend the evening over gaze at tears and guidebooks Williams was the lucky one at last and perhaps he deserved it at 11 40 p.m. He read from Murray's guide to Essex the following lines 16 and a half miles Anningly the church had been an interesting building of Norman date But it was extensively classicized in the last century It contains the tomb of the family of Francis whose mansion Anningly Hall a solid Queen Anne house Sends immediately beyond the churchyard in the park of about 80 acres The family is now extinct the last air having disappeared mysteriously in infancy in the year 1802 The father mr. Arthur Francis was locally known as a talented and amateur engraver in Mezzotin After his son's disappearance He lived in complete retirement in the hall and was found dead in his studio I'm the third anniversary of the disaster having just completed an engraving of the house Impressions of which are a considerable rarity This look like business and indeed mr. Green on his return at once identified the house was Anningly Hall Is there any kind of explanation for the figure green was a question which Williams naturally asked I Don't know. I'm sure Williams What used to be said in the place when I first knew it which was before I came up here was just this Old Francis is always very much down on those poaching fellows And whenever he got a chance used to get a man whom he suspected of it turned off the estate and by degrees He got rid of them all but one Squires could do a lot of things then that they dare and think of now Well, this man that was left was what you find pretty often the country the last remains of a very old family I believe there were Lords of the Mounder at one time. I recollect just the same thing in my own parish Well, like the man on tests of the doobie reels Williams put in yes, I dare say it's not a book I could ever read myself But this fellow could show row tombs in the church there that belonged to his ancestors And all that went to sour him a bit, but Francis they said could never get at him He always kept just on the right side of the law until one night the keeper has found him at it in a wood Right at the end of the estate. I could show you the place now It marches with some land that used to be to an uncle of mine And you can imagine there was a row and this man gaudy. That was a name to be sure gaudy I thought I should get it gaudy. He was unlucky enough for a chap to shoot a keeper Well, that was a Francis wanted and grand juries You know what they would have been then and poor God He was strung up a double quick time and I've been shown the place who was buried in the north side of the church You know the way in that part of the world anyone's been hanged or made away with themselves They bury them at that side and the idea that there was some friend of God He's not a relation because he had none poor devil He was last of his line kind of spes ultimagentus must have planned to get hold of Francis his boy and put an end to his Lining too. I don't know. It's a rather out of the way thing for as exposure to think of but you know I should see it looks more as if old gaudy had managed the job himself Oh, I hate to think of it. Have some whiskey Williams The facts are communicated by Williams Deniston and by him to a mixed company of which I was one And the side you're saying professor o theology another I'm sorry to say the latter when asked what he thought of it only remarked Ah, those bridgeford people will say anything the sentiment which met with the reception it deserved I should only have to add the pictures now on the ashley and museum There has been treated to a view of discovering whether sympathetic ink had been used on it But without effect the mr. Britnell knew nothing of it say that he was sure was uncommon And that though carefully watched It has never been known to change again Oh, there's so much going on that story that I admire and I enjoy And partly is the the multimedia function of it that we it's a text based story But it's so carefully based on this work of art with its strange properties I love that that mixture of poking fun at these characters while also scaring them and scaring the reader too And there's this great thing that james always does of just barely hinting at the horror Just just living the outside of it so that we don't quite see the figure We get the moonlight. We get the gothic sense. We get the building We get the child, but we never quite know exactly what happened And you think if the child was living What happened to it after the figure took it? and just How old was that figure and what happened to it? It's one of those stories that succeeds by these hints and insinuations rather my outright right Wow That's a sample of mr. James for all of you to enjoy. I hope you liked it Please let me know in the comments what you thought If you'd like to add any suggestions for my next readings, please go ahead We've got quite a stack up ahead and I'd love to hear your thoughts In the meantime watch out for paintings and pictures to make sure they don't change on you or tell you stories that you might not appreciate Stay safe. Stay warm and cozy and we'll see you next time for scary stories Good night


I'd like to thank everyone for welcome to today's CNCF live webinar, real shooting of Kate's applications. I'm Libby Schultz and I'll be moderating today. I'm going to read our code of conduct and then hand over to a local co-founder and CTO of Ops crews and Nick Lee, CTO of Megazone Cloud. A few housekeeping items before we get started. During the webinar, you are not able to talk as an attendee, but there's a chat box on the right-hand side of your screen. Hello, thank you, and continue to do so, and also leave your questions in the same spot and wait to as many as we can at the end. This is an official webinar of the CNCF, and as such is subject to the CNCF code of conduct. Please do not add anything or questions that would be in violation of that code of conduct, and please be respectful about fellow participants and presenters. Please also note that the recording and slides will be posted later today to the CNCF online programs page at community.cncf.co under online programs. They're also available via your registration link that you use today, and the recording will be on our online programs YouTube playlist. With that, it's over to Alok and Nick to kick off today's presentation. You all take it away. Hey, thanks, Louis. Thanks for the introduction, and thanks everybody for joining today's webinar, live webinar demo, and let Nick give a shout out as well. Good morning, everyone. This is Nick. Thanks for time. All right. I'm going to jump straight ahead to the topic at hand. Hopefully, you guys can see my screen. Everything looks good. Today's topic is going to be an interesting one. It's going to be about how can we do real-time troubleshooting of Kubernetes applications, especially when those applications start showing problems and performance issues. I'm going to dispense with the legal notice, et cetera, and thought we should give you a little bit of background who we are in case you don't know who ops through this. We are a relatively young company based out of the Bay Area, but our focus is almost exclusively on how to provide observability for Cloud-native applications. Because it's Cloud-native, we are an active participant and member of the CNCF community, and as you will see, we are pretty much built totally on CNCF and open-source instrumentation. I don't want to read through this, but you can see we've been working with a number of customers, number of partners. We are venture-backed. The other thing I'll point out, because we are focused so much on open-source and CNCF, Prometheus being one of the first projects, Julius Volts, who, if you guys are following Prometheus and CNCF instrumentation, is on our advisory post. We are glad to have him. So we'll go straight into a little bit of background on Megazone. So I'm going to hand this off to Nick. Again, as I said, you can obviously find out more while looking at our website, ops.com. Nick, if you want to give a little background about self and Megazone Cloud. Sure thing. So Megazone Cloud was founded in 1998. We mainly focused on helping the customers to utilize Cloud better. So headquartered in South Korea. We have an office here in Palo Alto, Toronto, Canada, Tokyo, Hong Kong, Vietnam, and Shanghai. And we recently opened one in Australia as well. That's the latest office. So our main focus is using Cloud for the customers and helping them to use it properly. So we have AWS, Google, Microsoft partnership, and we're the largest AWS partner in APEC and top three in globally, as well as we have original Clover service providers such as Neighbor Platform and KT Cloud. We try to help the customer from the day one to operational, continuous operational. So consulting, development, any migration they need from the on-prem or another cloud to the other cloud and operation after the migration or development. We work with various partners such as Ops Cruise, ISV partnerships in Korea. And one of the things that I do here in Palo Alto is looking for that leading edge technology companies to bring their technologies to Asia to help the customers in Korea as well as reduce the gap between the US and other part of the country. We also provide a service called the Hyperbilling to help the customer with the billing on their cloud usage. So multi-cloud building services, as well as a space one also named as a cloud for it, which is a Linux foundation project that helps managing the multiple cloud in a single portal. Next slide. So the reason why we are working with Ops Cruise was that internally we were challenging, we were facing the challenges. Like I mentioned, we have a cloud for it and a space one, which is provided as a SaaS product. We wanted to make sure that we are within the SLOs and we provide a right level of equality of service to the customers. And we saw the same pattern with other customers as well, our own customers. So one of the largest mobile telecom company and mobile service provider in Korea was actually working with us and asking for help on their Kubernetes environment. So we're trying to solve their problem because they were facing to use multiple Kubernetes tools. It's solid and it's hard to maintain and operate. And they need to translate the metrics and perform the complex correlation. The problem with this correlation is that unless you know exactly what you wanna do and you know every aspect of the metrics, it's very difficult to have that. And getting those information and providing it to DevOps, to enhance the DevOps practice and skills is not an easy thing. And it takes way too long to have the newcomers to get trained and start using the environment and the tools properly. And keeping up with the new release and a various open source project, it's not an easy task. And it's not just us, it's our customers and also globally. It's a hard to keep the talents. When you hire new operation people or DevOps engineers, it takes at least three to six months for them to understand what it is and for them to understand what they need to do to keep up the same environment and make it better and upgrade. So the solutions that we're looking for is that's something that we can automate but easily adopted and we can train others easily. As well as we get a single pane of glass with all the integration and telemetry coming into that environment. And it doesn't hurt to get a machine learnings and AI assist at troubleshooting because we all know we don't have enough people to all the troubleshooting by ourselves. If we can get help from others, that's always better. And easily understood SLO and a quality of service for our DevOps and service owners. So that's where we were a partner up with ops crews and trying to help the customers with these challenges. Hello, it's all yours now. Thanks Nick, I think you said the stage and hopefully many of you can recognize or empathize with the issues that Megazone and their clients were facing in Kubernetes. I would say having been working in this area for six, seven years, it's not an easy problem, right? So to get right to the heart of it, Kubernetes application performance troubleshooting is not just about Kubernetes, right? It's about everything that sits above it and below it, right? Think about just what has happened with cloud native services, just the number of objects, containers, services. We've seen 3,000, 5,000 containers in a single cluster, large number of nodes. And then it's not just that, right? You have service to service calls, you have the SaaS entities that are not being managed by Kubernetes, could be external calls to APIs. And then of course, the reason you go into cloud native and agile is you can constantly make changes. You can make changes to the services you can scale out, scale in, change the code version, so on and so forth. So you have now on top of these large complexities and scale dynamic changes that are affecting everything because every time you add a new service or take it out, your dependencies and who's talking to whom has changed. And if you don't do the deployment right, how do you know that causes a problem? Or runtime, something happens, you didn't know about the infrastructure or something else that changed or the configurations you set. All of that is not just a Kubernetes problem, it goes all the way up, obviously the dependency on infrastructure, but also all the way up to the microservice and therefore applications. So the question is if the application does fail, how do you know is it at the code level, something else on a third party service or it's Kubernetes or it's the infrastructure? And so that makes it complex, right? The good news is, and this is part of the reason we love working inside the CNCF ecosystem is all of the telemetry need metrics from Prometheus, right? Flows, we are looking at level four bytes and packets or even level seven, whether it's from Istio or EBPF, Extended Berkeley packet filter that tells us request rates, response times, error rates, that events from Kubernetes state metrics. So you know in changes that you made, you can capture that. Logs that are coming in, whether it's the application level or at the specific container level issues and of course traces, open telemetry. So if you look at that between Prometheus, Istio, Kubernetes, Fluenty and of course open source like EBPF and Loki, Yeager, Zipkin, all of these telemetry as well as information on the configuration all there. What's the challenge? Just like the number of objects for every object, there is metrics, flows, events, logs and traces. So if you think about it, there's a huge cardinality problem. Technically speaking, right? If you analyze it, a large Kubernetes application is really a highly dynamic complexity of spatial dependencies and temporal dependencies. And you're only getting certain of these metrics every 30 seconds or one minute depending on how you're scraping that. So we have a fundamental cardinality problem when we are trying to debug and trying to figure us out in real time. How do we help ops know what's going on to troubleshoot a problem if something's slowing down in Kubernetes? That's the focus, right? Now as I said, and I'm just re-emphasizing all of the things that we are looking at, all of the metrics and telemetries already available. So I'm going to just quickly be look at that if you look at things like all the standard metrics, you recognize most of these logos, but of course you also want to capture cloud level metrics because that gives you the infrastructure information on the VMZ are using or the persistent volume and the storage, right? What we want to do is leverage all of these and this is what you should think about. All of that being available, leverage it to figure out, hey, how are they tied together? Because that's going to tell you contextually what those things mean across their different telemetry. Where is the dependency? Because dependency usually leads to what could be the causal paths? And then if you knew what to expect, what we call predictive behavior, then you know what to look for and we'll talk about that briefly in the context of the demo. And this all ties together and leads to the whole idea of trying to do causal analysis, which is not an easy problem as we mentioned. So what we recommend and what we have done as an example is as you can see on the left-hand side, leverage all of the open telemetry, right? Especially the CNCF ones. And effectively what you want to do is stream processing. I'm not going to go to hold out a detail here. You can always look this up and maybe some of you are doing something similar. Collect all the telemetry, bring them contextually so you know how they link to each other. So you're not looking at metrics or logs or traces of flows or any events happening across the infrastructure Kubernetes and app in isolation because then you are doing all the work. If you can pull that together and get the topology, meaning who's talking to whom at the application level down to the dependence to the infrastructure, that gives you better context to knowing what's going on, right? And then of course, look at flow, understanding behavior and all the way as the sequence says, able to isolate the cause. So we have all this information. So the context across this telemetry configuration changes is very important and I'll emphasize this again. The whole idea is to get enough information across this. So we know what is the state of the application at the time when this problem was detected, right? This is where analytics comes in. This is where you would need automation because there is no way one or more SRVs are going to be able to do this manually without having some automation. And that's the whole point. The point to make here is all of the data is available as close to real-time as you can. Can you pull the insights you need? So by the time a problem does happen, you can able to figure out what the problem is. So let's talk a little bit about what does that mean in automating cause isolation? If you're a computer science geek and I've looked at this problem, there's something called non-polynomial complete or NP-hard problem. What does that mean? It means that the number of possible combinations of data that you have across and trying to get a sense of what is happening in time and space is very, very hard. But you know what's interesting, I've talked to customers and people who are using this, they say some of their best sources of isolating a problem is their senior SRE guy who's seen it all. So because it's not a simple problem and highly non-linear, think about how we solve this problem. We leverage information about the IT style. We know that when a container is working and it is sitting on a node, it is using its resources. We know that those resources are coming from the cloud. We know there's a shared service and that shared service can become a bottleneck when multiple things are talking about it, especially things like databases, et cetera, right? These are things, these are aspects of knowledge that really good SREs use. They also look for what I call, as you can see in the second bullet follow the breadcrumbs, they will look at the dependencies. Who is talking to whom? Because they know if the alert is happening somewhere and they're slowed on, if something is in the path that is gonna be, anything is, that's not in that path but probably not to be relevant, right? They also know when there's a problem in Kubernetes, is this readiness or liveness? They know the meanings of that. So they're using all of this knowledge together, looking at the alerts, then looking at the metrics, logs and places. And let's say if you have a service that has a certain kind of behavior, like let's say is IO intensive, they'll start looking at saying, hey, they did this problem. So expert SREs who do cause isolation use all of this information. So why not follow that instead of trying to look at everything and throwing all the information without not able to narrow it down? The whole idea for an automated system has to follow these breadcrumbs, understand, use knowledge, use the information properly and narrow down to a very few set of objects which gets closer to the likely cause. I would venture to say perfect cause isolation in real time is not theoretically possible. However, if you have enough information and you have used the information correctly, you can get to it very, very quickly. And that decision system is what most SREs use. So kind of a block diagram on the right is basically saying, you know, when you see the alert, looking at where the alert is in the source and then start looking at, say, hey, what's around it? What is the performance? Who was it interacting with? Was there an issue depending on the type of alert in Kubernetes? Was there an infrastructure? Was there problems and saturation? And based on that, you would start looking and eliminating the possible cases that you don't have to look at, right? So that's what the dynamic decision system is. And that's what you're gonna talk about, how to leverage all the information, extract insights that will help us isolate the problem. So before I go into the demo mode and show you how this works as an example, I wanna give you an idea what we want to, what you're gonna see in the demo, what instrumentation is being used. So our deployment architecture is shown here. Effectively, all the blue that you're seeing as you'll recognize it are open source instrumentation sitting inside the Kubernetes cluster. So CAdvisor node exporter, your familiar Prometheus components deployed as demon sets in the cluster. And Promtail is being used to collect the logs from Loki. So that's also a demon set. I would add that in the node exporter in order to understand flows and not just bytes and packets coming in and out of the node into and out of the containers, we are leveraging EBPF. So our node exporter also uses an EBPF collector so we can look at what L7 metrics resource, sorry, request rates, error rates, response times are happening at the level of a container within a node, right? We can get that information. We also have the, most of our environments have Yeager. So we collect, we can collect traces. So the three primary four things that you're seeing here are Yeager for traces, Prometheus for the metrics and flows, Loki for the logs. And then because we want to look at changes that are coming, we are also capturing Kubernetes state metrics. This is one way to see changes that have happened within the cluster. So then what we do essentially is once this is instrumented and you can do this yourself with the Helm chart, we basically have four plus one additional, what we call pod, each for a type of telemetry. So we are collecting the metrics from the orange metric you see Prometheus gateway to collect all the metrics from Prometheus. The log gateway pod is basically collecting all the logs through the Loki collector. Kubernetes gateway is collecting Kubernetes state metrics and changes. So we know what the configurations are and changes being made. And then finally the fifth one you're seeing is the cloud gateway because we want to know what is the infrastructure and so do we understand the dependence. All of these essentially collect data and just as you may be familiar with a scraping at a classic interval that you set up whether it's 30 seconds, one minute depending on your scenario and bandwidth. They're collected, compressed and sent to a SAS service which is shown below in the orange hexagon which is where our controller is. The controller is basically getting all that information which I showed in that previous staged pipeline and processing and extracting and doing that information basically bringing them into context, discovering the topology, figuring out what the dependencies are both at the service level down from Kubernetes to infrastructure and then looking at the data, figuring out using machine learning what they expected behavior. So that's kind of what the scenario is. So today the specific use case that I want to talk about in the next probably 20 or so minutes is an application slowdown that we have detected and how do we can analyze especially in the Kubernetes kind of scenario or Kubernetes problem that caught as an application slowdown and how do we do that? Now I want to make a note here that the case that we are looking at here we are not using tracing. Clearly there are ways of using tracing for doing that. There's a whole nother problem and that's a separate issue on something that we use called trace path. In fact, there is a CNCF live webinar on that if you're interested, you can follow up with us. But today's case we're going to look at doing root cause analysis for Kubernetes affecting application but there's no tracing label. All right, so at this point I'm going to switch my screen to the demo tab. Let me see if I can do that again. All right, so let me go back to sharing. All right, I think that's the one. If you guys, Libby or Nick screen showing up, just confirm. Looks good. Excellent. So folks, what you're seeing here is what we call our application map. You can look it up. I'm not going to spend too much time here because our focus is going to be on the automated cause analysis. And what you're seeing effectively here is I can zoom in and out is basically everything put together that shows at the, what I would call the service to service level. So in fact, if I hover down and you can see this basically says request coming in. Let me give an example. Load balancer talking to an engine X card load balancer going into this container, going into another service, et cetera, going into the whole thing all the way down to, hey, there is a Postgres database and it's running actually on AWS. So this application map that I'm showing you here is being built by the data that we are getting. And in fact, obviously can organize it by labels on the application, by the namespaces that are running on it. In fact, there are different applications running on it here that I'll show in a minute. It's running on a five node Kubernetes cluster. I'll show that in a few minutes, the different pods. And this is a really small application. This is the what test bed that we try with pods, containers, the SaaS services. This is in fact running on AWS as you can see it, load balancers and there are actually multiple clusters but they're connected together. It's a multi-cluster environment but we won't focus on that today. And if you wanna know what are the different namespaces, I think I can hear on the screen here. Let me share this tab. If you guys can see this, I can actually search by namespace. And in fact, what you're seeing here is, this is the application, a chopping guard as a small e-commerce application. You can see a robot trap application which is a IBM application. The ob screws. I'm actually not seeing anything right now. Oh, did it not? There you go. There you go. Sorry, there is a slowness here. I might just be on delay. Yeah. There we go. So I'll start again. So what I've done is, I've actually tried to show you guys that different namespaces of the different application namespaces and or what's being deployed in this cluster. And what I was showing here was today we'll focus on this little e-commerce app called shopping cart. And there is more there. There is our ob screws deployment. It's only namespace, robot shop, which is another e-commerce application, online booting that's used for tracing all of these. One reason for showing that is to tell you that we can filter down, et cetera. But for today again, and I'm switching screens again and hopefully back to this. Hopefully the screen switched. Can you guys see that again? Always a delay. Yes. Might be a want to be my check. So to give you an idea of why this matters here, these, when I look at any of these containers, for example, engine X coming in from the service. In fact, if you look at this, as I highlighted, you can see because we collect not only Prometheus metrics by, but also the flow metrics, you can actually see average response time between this and its corresponding service. So this is actually very useful, right? If you think about it, a lot of enterprises that use this environment, this kind of environment ecosystem for monitoring obstacles can see that dependency right away. And why do I say that? Because when we're doing the root cause analysis, if I'm not involved in this card service and there's a problem here, then I don't have to look here. Or in some other application service like here, I don't need to look at this. But if the data is flowing in and there's a problem I know what to look at. I have narrowed down the focus and I can see visually what's going on. So that's one key part, right? Knowing the topology and the dependency is the first thing that most of us do. This is what an expert SRE will say. I don't need to look here if the problem is here. Well, how do I know that? In real time, as Kubernetes is changing, I should be able to see this dependency, how the data is flowing. So let's go into one more depth here. So I'm going to zoom in here in the shopping cart app. Hopefully you can see what I'm doing there. And if I look at that, and I'm going to shift my screen here so we can see, the key here is able to see for this container, what are the telemetry in context? And which is what I was saying earlier. I'm going to move this up a bit here. So metrics, what is the metrics coming in? Obviously, back from Prometheus, et cetera, right? I can get that. What are the, if there's no events, I know this one doesn't have it, but there's logs, what are the logs in there? I can look for specific logs. For example, is there a problem on this container, on this engine X, something has happened, right? Anything that I'm recording, I can look for errors, et cetera. I'm not going to go back, but I had that in context, what we call a quick view. What is it talking to? In case you're not seeing it, I can see because I am capturing, because we are capturing those flow metrics as well as null and we can disambigate Kubernetes namespaces, we can say, hey, what is the IP address it's coming from? It's inbound. You can see that it's coming from this engine X controller, bytes, et cetera. Sorry, look, I'm not seeing it again. It might just need to reload, but I just want to make sure. Yeah, because I might be going ahead and things haven't updated. I could see it. I could see the screen right now. Can you see it, maybe? Yeah, I could see it. Yeah, all right. Let's ask audience. Yeah, audience, if you can pop in the chat because I'm moving it. Okay, they can see it. It's my connection then. All right, I need to probably get into a better connection, sorry, Libby. So what I was getting at is putting in context, we talked about metrics, logs, but also the connections because this is what we want to know when we are trying to see who's talking to whom and what is the problem. So in this case, just to quickly summarize, I know what is inbound. I can disambiguate that. I even know how much data is coming. You know what else is coming. And actually there is, if you look at this, if I click on this, let me see if I can find the right way. Might be hard to do it because I'm trying to show. This on this port itself, there are multiple connections because there are different ports involved in there. And outbound, as you can see, is talking to this other engine next controller. And in fact, you can see on the screen. So what we were trying to point out is, and if there are trace paths involved, if I want to go look at the TCP address, I don't have one explicitly here, we can get trace paths as well as in service performance, right? More interestingly, what about the configuration? We can pull in the actual Kubernetes manifesto so we know what has been designated, how much is the resources, what volumes is talking to, are things healthy? What is the rate at which we are scraping and the timeout settings? All of that information, including namespace, everything is right at your fingertips. So you don't have to go switch and do cube couple commands, right? This is important. And so any changes that happen, we will update and present this. So having everything together in context is very, very useful. I'll do one more switch before I get into the root cause problem. So for this application, and as I showed there is multiple containers in this environment, as I showed, what does the node map say? Node map says, where are those applications sitting? So for example, I was looking at nginx, and if I want to find it, I can look at and figure out there are five nodes here. And I can see which of these nodes have what containers and in fact, more interestingly, how much are being used? So another view that we be able to pull together from that is the usage of requests. And I feel that numbers are the highest requests. And what is the request limit, et cetera, both at CPU and memory and on a node by node basis. So we can see whether you're over-provisioning, under-provisioning, right? So for example, here, the request on this one, this primitive node exporter is set at 200 and the requests are already exceeding. And so the reason it's red is because it's worstable and it might be prone to eviction. So that gives you another view both to right-size the environment, right? So going back to this application, what happens when we have a problem? And in order to do that, I'm gonna jump into what's called our, not this, but the alert view. So I'm gonna switch the screen again. Let me know if it's showing up. I actually picked up an alert 105.915. So Nick, I'm gonna rely on you. Can you see my screen has changed to the alert window? Yes, it has. All right. So the one, the primary example that we use today is an RCA analysis that we are doing on a service level objective breach. We do this automatically because we're collecting flow metrics on that ingress on the shopping cart I've showed you. And it was run a little while ago. So I can just go through this. If I click on it, this is where things start getting interesting in terms of how this is automated. So we capture alerts automatically based on explicit alerts from Kubernetes infrastructure, predictive using ML, which I'll talk about in a few minutes, but also if there is delays on the service level indicators, and I will go back in a minute and show that to you. So actually I should do that. Let me go back to this example here and show you that for this service, you can see the one feeding into that SLO on the ingress, there is something called SLO SLI. You see that something has been set. The suggested is this value and that's done automatically by analysis by the system. You can also look at what the current max is. Someone has manually, the user has set it at four seconds and you can obviously change that. If that's because if you're looking at an outbound connection to a customer facing, you can set that. So that's where we can set SLOs right in this app map. And so what we are looking at now, sorry, I'm switching back the tab here. Can you see my tab again? I think I switched this tab and I'm maybe. Yes. It shows the alert detail detect. Yeah. So I'm gonna share this tab. What I was gonna show you, I think I've missed it. On this tab is that the data coming in into on the ingress side on that, that we can detect an SLO SLI. And while we can do this by using machine learning and what the expected should be, someone has set it at four seconds. And this is what we'll use as an example of where the breach is. So the service level objective for this application on the ingress side has been set at four seconds. All right. Now I'm gonna switch tabs, so bear with me. And then I'm back to that alert that I showed you here. And I apologize, I'm jumping back and forth, but I'm trying to give you context of what was the SLO and what does the system do in this kind of automated. So for that SLO breach on that shopping cart app, there is a breach you detect because on the flow. So if you see that, it says it's automatically detected because as you can see the four second SLO, we are at 6.657. And there's a lot more detail here. Obviously this one is very simple rule. I can see you can set this up. The max across what's coming in that graph based on the response time and the flow can, if it's more than 4,000 milliseconds, that's what triggered it, right? It's a latency based alert. And all of the details and the aspects of that that we already have in context are given here. But the most important thing from the user's perspective is we saw an SLO breach. The question is, how do I know what caused the SLO breach? So what we've done in this system, the decision tree that's running in the background, the AI engine does this and analyze. And this is where it starts getting interesting. So Nick, I'm assuming screen has changed. Can you guys see the screen change? I'm on the analyze tab. So now this is interesting. What you're seeing here, this is automatically be done in the background whenever an SLO kicks off. Remember decision plan is kicked when we detect a problem. It is actually saying, okay, for that, there are five connection paths. And the latency, the highest latency path has been pulled out. The whole idea is that if you know the context, can you narrow down the path? And everything that's read here is basically saying all of them are high latency along that path. So I'm gonna close that. And you can actually see laid out here in this high latency path, engine X coming into it's bought and container to the service for web server, bought and container for that to a cart cache, a caching element, bought and container, the cart server, bought and container down to the database server. And then, oh, I think I am actually on, am I on the right one? I might be on the wrong one actually. Let me go back and see if I got the right alert here. The one I'm gonna look at, let me see. Let me just check if I have the right alert. Okay, I think I have a different alert here. But the one I wanna show you is slightly different. So let me go back and retrace. That one also is an interesting one, but that one does not a Kubernetes one. So the one I wanna use, I apologize. I'm gonna do this real time. I think it's this one that has a Kubernetes related issue. This one also has a breach. This one even higher. Same case at a different time, 15.454 seconds and exceeding the four second service level. When you look at the analyze, same thing happens, slightly different. And what it's saying is in this case, there were three connection paths. The highest latency path was pulled by the system by using that topology. If I close that, in fact, you'll see one, two, three, four, five, six, seven, eight alerts detected in red. And you can see where the breach happened during what time while this was happening. Now, starting here, as I said, similar to what I was showing earlier, going from NGENX to its pod and container to web server pod and container and so on. So put all the way to the backend, the SCART server. What's interesting to note is that is not the only path. So normally a user would actually go and say, hey, what are the possible paths on which that NGENX container, sorry, the service is dependent on and there are three paths, right? You could have more, you could have 10, 15, depending on the complexity application. But the reality is what slowed down is the path that has the highest latency path. And not surprisingly, that's the one that's got other alerts. The system can then isolate and saying, let me just show you the one that's really relevant and that's key. So that's the first thing the decision system and automatic causal analysis should do. So let's kind of walk through that. So obviously this is a situation where it was slow. We've already detected. If I go to the web server, it says, hey, this is slow because it's higher. So the service is slow. That's not surprising. If you analyze it, it'll say, yep, it's slow because the response time downstream coming back has slowed things down, right? So that's not surprising. What about the next error? Because this slowed down, expected. How about the next one? That also is high. This one also has an SLO breach, this card cache. Let's look at that again. This one also has its own analysis tab. This is also higher than expected. And this one, I believe, is an ML-based alert or actually it's response time has been higher. And it captured that. It says, okay, further dependencies further down. So slowing everything down, going back, going back again to that source again, let's go down to the next one. What about this? So this was slow. What about this card cache? This says network metrics are not normal. And if I click on it and you'll see there is something called an RCA tab. That's being analyzed besides just being high, meaning triggered based on the expected service level of response time. If I click on it, this is where it starts getting interesting. What you're seeing here is that this one has what we call our fishbone analysis. That means it's categorizing that container based on memory, CPU, file system, if there's any IO dependence. Demand side meaning what is coming in and the responses to that. Supply means in our case, what is going on downstream and even configuration changes. But if you look at one of the things that sticks up is, hey, the number of errors actually has increased in this case. That's one. Second, this packets here decreased. The data that was going outbound has decreased from 100% decrease. There's nothing coming in and neither is anything going out. So this is a suspicion why the network level metrics. And this has been detected by just having an expected behavior model that was learned over the past. The reason is if you think about threshold based alerts, usually are high water marks. And the challenge with that is when things drop below that, you would normally won't detect it. But if you knew the expected behavior data comes in but the data doesn't go out or received on the downstream side, this is where an expected predicted behavior model helps. So going back to the detect, the reason we have this and the analysis says there's something wrong, the fact that there are more errors coming in. So going back now, that chain again. So that's what we detected here. What does the cart server say? We're starting to see something here. If you see it says cart server does not have any pod to serve requests.  Well, if there is no pod here, there is no data coming back. Neither is any requests coming in. So that itself is telling us that the reason why this network metric detected on that ML side and this pod not showing serving requests are connected. So we have a problem that's further away downstream until we come here. So if I go to this container, it says, well, in fact, I already gave you the answer. There's an image pullback error here. And if you look at this, it says, hey, the cart server container is terminated. In fact, if I click on it, it'll just like say actually, I don't have a container on that. It's terminated. So if it's terminated, of course, what that says is I'm not gonna have, I'm not gonna have anything from the cart server responding back. If I go to the container now, now we can start looking at what has happened. This analysis now will now say what is going on here. There is no pod running. And now if I click on the analyze, we would use the same fishbone except it's not for a container. This is specific to the Kubernetes problem. And if you look at this, the same fishbone that you saw for the container side, it has two classes, pod schedule failures, node failures, startup failures, and runtime failures. And this is predefined. Remember, we talked about curated knowledge. Someone who is familiar with Kubernetes knows how startup failures can happen, deployment failures, runtime. And if you look at this, this says it is constantly in translation and it's saying container not ready. We are collecting this from the Kubernetes state events. There is a backup restart. Why is there a backup restart? Because the image is not loading.  What happens further down? The service that calls it does not respond. There is one more thing that's detected automatically dynamically. There is an image name problem that is incorrect. And therefore, because it's invalid image name, not surprising, it's continuously trying to pull it off and never getting ready, which means as a result, if you go back to the problem and you look at this, you can now see what has happened. Image did not load correctly. Kubernetes issue, pod did not come up. Card server has no pods. This starts seeing network errors and errors. That increases the response. And as a dominant effect, all the back goes back to the engine next. So I know we'll run out of time, but what I wanted to point out is, unless you're able to pull together all of this information in context, eliminate the pads, look at the dependencies, analyze each of them, you're really looking at, if you notice, flow metrics to understand why this was slow, understand all the way down to card ML metrics to understand why that problem happened, looking at Kubernetes state metrics to figure out why and analyze it to say, hey, if you fix that image, pod will come up, pod comes up, card server can respond, then all of these will be straightened out. There's an old saying, I think you've heard of, for want of a nail, a battle was lost. The nail was a bad image name that's propagated all the way up to create an SLO case. So I just wanna give that because I know I wanted to show a couple more examples, but having specific contextual knowledge on the system, analyzing things in sequence, all of that, bringing all that together is the key here for us to be able to understand how this problem works, right? So I'm gonna actually go back and bring this other deck here. So the whole idea is in order to automate the causal analysis, you have to really leverage knowledge about the stack, right?  Once you have that, when you see a specific kind of problem, you can narrow it down and then eliminate all the scales as the examples that I showed you, eliminating all the other paths to get to that point. So that's kind of the way we can automate this causal analysis. As I said, it can never be perfect because you don't have 100% information at 100% granularity of time, but you can use all of this information to solve the problem, right? In this case, we were not even using traces. So just to kind of summarize before we go into Q&A and throw it up for discussion, as you're probably well aware of if you're using Kubernetes, there are multiple issues that impact the application performance. And that makes just the cardinality and the complexity of space and time and dynamism makes this RCA quite challenging. We can't solve the problem not by blind correlations. That's not gonna help because the number of possible correlations is gonna be very large. We have a cardinality problem there as well. But if you go look at, understand how to resolve this, how we do this very well, it's really, as I said, follow the breadcrumbs and eliminate things that are not relevant. And that's where a decision system is needed. That has to be runtime and automatically doing that for you so you're not spending time. Leveraging curated knowledge is absolutely important. There is no such thing as blind correlations and blind ML that does not work. It'll just lead you to false paths. And that means understanding the full telemetry, the configuration is changing. And finally, the message for all of you who are following the CNCF, you can leverage all the open source CNCF. So we've just pulled that information together to do that. So I'm gonna throw it for Q&A and switch back. Are there any questions that we can address? Go ahead and leave your questions in the chat if you're looking to ask something. Please post your question. Oh, look, there was a question from Oliver. Does offscrews show a metrics for services in the cloud provider that hosts the Kubernetes cluster? Exactly. For example, AWS SQS and things like that. Yes, we do. Let me share this step. Yes, of course you can because you can collect the data. So what we do then is use that data that we're getting on the infrastructure. So I think the example I'm showing here, Oliver, is what we collect. For example, there's load balancer that you have to specify to the cloud, whether it's one or the other. So here's an example of collecting data from AWS on the load balancer that we showed. It's also visible, Oliver, at the app map when we were talking about, so for example, this is postgres, oops, sorry, postgres metrics, and you can get the queue depth, et cetera, from the cloud vendor itself. You know, obviously it's not Prometheus open source, so we have to kind of be able to pull that to do that. Hope that answers your question, Oliver. Anyone else have questions? Great, I'll give everybody just a minute. I'll just, if there are questions, I'll just post this if they want folks want to follow up later, want to talk to us or general question on CNCF, metrics, monitoring, whatever. Any questions for Nick? You can put in the handles or select, okay. That's kind of one thing I do want to mention. Offscrews is free, by the way. Yeah, I'll just say offscrews is free to use if you want to sign up. This is a freemium case. So I'll just put in there. I know some people are concerned this is just a product, but it is free. So we are, again, built totally on open source. You can do what we are talking about. And if you don't, you can obviously get the free version as well. All right, thanks guys. All right, does anyone else have a question? Yeah. I think that's it. Thank you, Alok and Nick so much. Thank you everyone for joining us. Thanks everyone for joining. Thank you very much for you to reach out. All right. All right, and we will see you next time. Thank you so much. Thanks Libby. Bye. Thanks Nick. I will sign off.


Hello everyone, this is Amit Dan. I went to Walmart yesterday to buy the Flip HD Ultra Video Cam, which is about $270. But the associate showed me this movie pixel video cam, which cost about half of what you pay for the Flip. And he insisted that the picture quality and the features are a lot better. So I wanted to give it a try and I'm doing this video clip for the first time with this HD video cam. And I'll be posting it on my website very shortly. You'll be able to compare whether this gives you a crisp player picture quality because I have another video clip from the same location showing the same stuff, which is a few from my balcony. But it was done with a regular video camera, it was not an HD. So once I post this, I'll give you a link to compare with the other one and hopefully you'll be able to see. So this is a view from my balcony. Right now I'm on 18th floor. I live in a condo and there's a shopping complex in front of the building. So I'll be just showing you a bit of everything. That will be all for today. I hope to see you shortly. Thanks for watching this video clip at www.adangernal.com. Thank you.


Russia uses Putin's air fleet to return soldiers killed in Ukraine. Russia's presidential air fleet is quietly repatriating the bodies of soldiers killed in Ukraine and sending them to local morgues in small numbers to prevent public backlash. The Moscow Times Russian Service reported citing flight-tracking data, morgues workers and the soldiers' relatives. Russia Airlines, a subsidiary of the state-owned Aeroflot Group, operates 68 jets and helicopters for the Kremlin that include Russia's equivalent of Air Force One. State media in the Central Russian Republic of Tatarstan filmed one of the jets a TU-134A3 airliner with the registration number RA65911 returning the region's first dead soldier on March 2, 2022. President Vladimir Putin had flown on a plane with the same RA65911 tail number to a neighbouring Russian region in 2016 according to a local newspaper that photographed its landing. Flight-tracking tools have spotted several of the Russia aircraft, which form the airline's special flight detachment crossing to and from the Russian regions between January and early February 2023. The Moscow Times has found a correlation between these flights' arrivals and local news coverage of soldiers' funerals that followed. A Russia jet with the RA61720 number landed in the central city of Yekaterinburg on January 23rd followed by reports of a serviceman and the Wagner mercenary being buried there in the next 24 hours. Flight radar then registered RA61720 flying to Yufa, the capital of the neighbouring Republic of Bashkortostan, on January 13th the same day that Putin travelled there. An RA87972 tail number jet landed in the Tatarstan city of Niznekamsk on January 25th with the same AA Yangebeev seen on one of 11 zinc coffins caught on video. A week later Tatar residents posted news of Yangebeev's funeral on social media. The same pattern of presidential aircraft flights followed by soldiers' funerals were observed on January 26th in the northwestern regions of Arkhangelsk and Vologda and between February 8th to 12th in the Orenburg and Chelyabinsk regions, while many Russia aircraft depart from Moscow. Others turn their transponders on over the Lipetsk region which is equidistant from Moscow and the Ukrainian border and the neighbouring Tambov region which is among a handful of Russian territories where wartime flight bands are in effect. It is impossible to establish the exact location where the soldiers' bodies were picked up.


Live from Las Vegas, Nevada. It's theCUBE, covering EMC World 2015. Brought to you by EMC, Brocade, and VCE. Hi, everybody, we're back. This is EMC World 2015. I'm Dave Vellante. John Furrier is here. Kumar, this one up then is here. He's the chief technologist at Verizon Labs. He's up on the keynote today, Emerging Technologies Day as one of the customers. Kumar, welcome. Thank you, Dave. Thank you, John. So what is Verizon Labs? Tell us about that. Sure. Verizon Labs is part of the overall product organization within Verizon. Our goal is to look at technologies that we can build as well as technologies we can buy from our vendors that can create a common platform across all our products for us to launch products a lot quicker than what we've been used to. We've traditionally been a fairly vendor solution dependent model and that has its challenges. So we are actually going away from that model and trying to own a more common architecture that will let us launch new services and products fairly quickly. So that's what the organization was set up for and the key talent in there is software development talent that we brought in as well as software architecture and the DevOps skillset that we can bring into launch and run applications and that more. So that is really what Verizon Labs was set up to do under the product organization within Verizon. So talk about your business a little bit. The big four, cloud, mobile, social, big data. You're certainly taking three of those four boxes at least that we can talk about social, I guess, but you're at the heart of the mobile activity, very much data driven, service provider, you are cloud. But things are changing in that business. How are those changes affecting your business? What are the challenges and what are the opportunities for you? Sure, so you're absolutely right. We're in all of those businesses. We're driving a lot of the change in that business but you're also aware of a lot of the macroeconomic pressures within those businesses. There's a lot of change that is happening in both in regulation as well as competition that always continues to threaten that base. Our goal is always to look ahead to see what we can do from a technology point of view for us to be more cost effective and where we can leapfrog certain technologies to get to the next generation of technologies that will help us offer services quicker, faster, much cheaper. And that's been always a challenge because when you see the technology landscape, even if you take EMC world, the number of different options you always see is so many. So my group constantly goes through trying to figure out what are the areas we need to play and what are the areas we need to own? How do we go about creating new business from some of these? How do we optimize existing work? So yes, we participate in it but we're always constantly on the lookout to say, what should we do to be different next year? What do we need to do to lay the foundation for a completely new rise? It's a big challenge. You've got an industry that has huge capital investments. You've got to get return on those assets so you have to keep those lights going. Like you said, cut costs, everybody's trying to do more with less. So what's your innovation focus? Where is that innovation coming from? So let's take specifically the data center. So if you look at my product, all of the products and how they currently run in the data center, you would, I mean, it is typically what you would expect from a data center that was built, you know, more than a decade ago. You would have silos. Every application or service has its own silo. And I mean silo, I mean the application, the operating system all the way down to the hardware. You are completely siloed to a point where if you look at, I mean, our way to achieve fault tolerance was redundancy. What I mean by that is you have completely replicate copies of certain things running for you to be able to achieve the scale you want to achieve. But that is a downside to it. The downside to it is that I have stranded hardware. At the lowest layer, if you look at it, I have stranded hardware. My power usage efficiency is incredibly inefficient because I have two CPUs doing the job of one or maybe a fraction of it. And my operating costs and trying to keep up that many boxes, not just the people, it is the power, the cooling, everything is high. And now the headache of maintaining so many different operating systems. So I mean, I have so many flavors of Linux that are running in my operating system. Each vendor brings his or her own flavor of Linux. Now the problem is if I have a bug, like a heart bleed bug or pick any other bug, the number of different patches I need to install and how I can roll out those patches, it's not instantaneous. It's over a few weeks or months. I can never get anything down quick. And the key I want to leave with you is that if I transition that to a more homogeneous environment. So homogeneity would give me the operational efficiency I need by saying that actually you don't have that many options on hardware. I mean, it's commodity hardware. You have three or four flavors of boxes you can buy, typical server type boxes, some with storage, different types of storage, maybe some with flash storage, some with disk storage, but everything else I'm migrating to the software. So it's very easy to do on some of the technologies are more mature than the others. I can talk about, if you look at on the compute side, we want all our software, not just the software Verizon rights, but even the software our vendors bring in to completely transition to a model where it's micro software architecture components that you can bring in. They run as loosely loose components that are coupled together with well-defined APIs, if you will. So I can now take these containers as we, early on we invested heavily in Linux containers and said, you know, that is the way of the future because it provides a clean name for us to package all the way to production, how we take the software. So that model is solved. Networking, we have talked enough about software-defined networking. There's plenty of solutions for us to run very scalable networking within the data center. So applications can request access to outside networks disparate external networks fairly easily and how they get granted, what they do with it, very easy to achieve. The big challenge was storage. And the reason I'm here, the reason I was talking to EMC is how do we solve the storage challenge? Quite honestly, I think when you listen to CJ talk about in the keynote, some of the technology, the emerging technology from EMC was hard to find. We couldn't actually find SCALIO because when people thought about EMC, they were thinking about big boxes that took up the entire rack and had lots of lights and lots of colors. And that's really the vision of EMC. But when we started engaging with the emerging technology division under CJ, actually CJ made the personal pitch and the Kudos to the Verizon accounting for pushing hard to get CJ to come talk to us. But once we got that engagement going and we got a vision into SCALIO and ECS, there was almost no going back because it kind of opened a whole world of possibilities that we had not considered before because we did not have to come out of storage. So you can take some typical scenarios. For example, what happens when disks fail? Your applications still need to get running. What happens when you have networks going up and down? Still need to get running. What happens if your entire data center is isolated? I still need to get my apps running. If you look at some of the guts behind how ECS is built, a lot of the geocaching, georadiance that they build in adds tremendous value to how I can now leapfrog onto my next generation of the cloud where I have all this compute, the networking and the storage completely in software. So just imagine a container with some software running on it that requires storage. Container is running, it's running happily, it's processing its load. Whatever happens, you have to now migrate the container from one machine to another machine either on the same rack, completely different rack, a completely new data center. I can have the storage following. So the application wouldn't even know that it lost the storage because ECS would let me follow it. Same is true for Scalaio. I can just pick it up and follow it so that I have the renunciate need. So back to your original question, if I now get to that homogeneous environment, I drive down my operating costs, I drive down my complexity within the network. And the most important thing for me is I can launch applications a lot quicker because nobody needs to worry about the whole silo. My point is just bring in your components, plop it in, get up and running. And that's your scale model. That's my scale model. So you have the DevOps philosophy in the labs, silos are kind of like really not compatible with this new platform three they call it.  How are you guys enabling those apps? I mean, APIs, new CMSs are coming in, get node, Angular, new technologies. Real time is big. How are you guys dealing with that new real time piece in the static silos, firewall rules and whatnot? How do you guys deal with that? Is it software? So in the static silos, actually we have, I mean, when I talk about DevOps, I'm really talking about DevOps only in the new environment. Our existing environment, the siloed environment is actually follows a traditional vendor operating model. But on the DevOps side, if you look at what really is the key, I mean, it's true for everybody who does DevOps is automation. And the key to automation is API. So whatever APIs we can leverage. So for example, it's not that we know that a disk has failed. Before it failed, we would know how many read write errors it has to predict the disk can fail. So for us to transfer the workload. So everything about the DevOps model on the new cloud is all about what I can automate. And to be quite frank, there is no one magic bullet that solves the whole thing. A lot of that are stuff that we are building, we are growing, we are learning as we go and we are writing it. But about 70 to 80% of any DevOps is very application centric. So you need to know what your application is doing to do it. There's no turnkey boilerplate product. I mean, there's some tooling that you could use, but. Absolutely. There's plenty of tooling we can leverage, but ultimately at the bottom line, it's all about how we can get a lot more automation. So the key about automation is once I have more automation, I don't have any procedures or mobs. I eliminate all the people who have to now take it from development to operations. So when I eliminate all that, I create the seamless cluster operating system, if you will, for me to launch new applications. So Kumar, talk about the personnel changes. Obviously with automation, orchestration, this is all goodness in the cloud, and it's evolving, it's early days. But roles are changing, right? I mean, the old storage guy, provisioning the silo and statics on the infrastructure, with DevOps, there's personnel changes. How are you guys hiring, share some insight into your thoughts, your vision, and then some examples of the kind of role changes happening within the organization and to position yourself for those apps. Absolutely, and you hit the nail on the head. I think the skill set we are looking for is grammatically different. We are not looking for an operations skill set anymore. DevOps is all about being able to write tools, write scripts, write technologies that can help you drive the automation. So it is certainly new skill set. And in Verizon, we're consciously working to attract the talent that we need, and it's not easy because when you go to a college campus and try to recruit as Verizon, the first impression is it's a cell phone company, there's a lot of technology behind it that we are trying to drive. And that has been a challenge that we have been working towards. We're trying to create innovative ways, but the personal change will happen. It has to happen for us to be able to take advantage of new technologies. Because frankly, the days where you could create those silos are behind us. So there's no way we can go back. But for us to take advantage of it, we do need to get new technology. We are talking off camera. I asked you about network function virtualization. You were too sanguine about it. What's your take on NFV? So when I was talking, what I was trying to explain about that is, there is multiple ways. So the network function virtualization really came about where you took the same existing functions and just ran the network functions in a virtualized environment in the cloud. Now, what is the problem that it really solved? All it did was it let you optimize on the hardware specific to the hardware purchasing, if you will. So instead of me buying three flavors of x86 servers, one purple, one green, one blue, I'm just buying one flavor of a server and I'm running VMs and running all of that inside it. But I submit to you that the fundamental problem that we face is a software management problem and not a hardware management problem. So if you now peel back NFV and look at the VMs, the VMs are probably running operating systems that were deprecated before I was bombed. Now, how do you support it? Because it is easy for a vendor to just pick the whole software, port it, and run the entire virtual machine and say I have a virtual machine up and running, I'm NFV compliant, but it doesn't solve my problem because my software maintenance on that still is going to hurt me. So when I told you I was like this on NFV, my point is that when we need to transform a network, I need to look at the holistic picture. I need to look at how I'm solving the software problem, how I'm solving that. The hardware problem I submit is a more trivial problem to solve and that's why when we looked at the cloud, we did not just use standard virtual machines and say bring your own OS, run whatever you want. I want homogeneity and then I can drive automation and software management on top. That was why. And that's your scale strategy. All right, Kumar, I'm sorry, John, we got to go. We're out of time, got to leave it there. So thanks very much for coming to theCUBE. John, I will be right back after this brief word. We're live, this is theCUBE. We're ADMC World 2015, we'll be right back.