Thank you everybody.  You start worrying about which of your slides you have to throw out or modify or correct. And turn on the clicker. There we go. Hi, I'm Casey Schaufler. I've been doing kernel development since 1978. So I've seen all kinds of interesting things go into the code base. Why don't we think the kernel is hard? Anybody here a kernel developer who does not work in security? Let's see. Okay, good. So this won't be completely wasted. First thing, it's too easy to cause damage. We all know, we all know about buffer overflows, stack these string functions, and it's just there's so many different ways that you can cause the kernel to do things that that you don't want to do it, either for bugs or malicious. It's just it's still too easy. And at this point we've got a bunch of people who want to do damage who are really clever. And not only are they clever, they're motivated. They're making money off of it. Exploits, stealing credit cards, all kinds of good things. Even we have a title which has evolved over the past few years of security researcher. Used to be called hackers. Now you've got a differentiate between the good guys and bad. So now they're security researchers. If they've come out in the public and done something that they're getting paid for as part of their job, they get called a security researcher. Otherwise they're a miscreant of some sort. But that's not new. It's like we've known about this for some time. So what's kind of the base of the problem? How old is this problem really? And really it's as old as the C compiler. If you've ever seen the original Kernhahn-Ritchie C book, that's about Ye Big and it's in a small format. And it's really easy to read. I used it as part of my educational process in learning how to do programming, which explains a lot about my code if you've ever seen it. But the C language is written in a time where operating systems were written in assembler. In fact, text formatters were written in assembler. In fact, virtually everything was written in assembler. And for systems programming, C is a really, really, really good language because it allows you to do the things that you can do in assembler while actually giving you some rational structure to your program. So you can organize your memory with data structures and you can, your control flow, you can follow your control flow because of the way the constructs in the language provide for. But it's not strongly typed. It's more a suggestion or a guideline than an actual rule set. This is efficient and convenient. You can do a lot of really interesting things efficiently. For example, my favorite one is the one where you have a data structure which defines a header and then it defines a one element array. And you can use this. You can allocate memory for your data structure and say, hey, this thing's actually got 74 things in it. So you allocate enough memory for 74 things plus the header. And you're good. You don't have to worry about them. You can dereference it appropriately. So you can understand what you're doing, even though you haven't constrained yourself by the data type. You can be clever and you can be precise. The way you have to do sometimes. For example, if you have an internet protocol header, you don't know at the beginning whether it's an IPv6 or an IPv4 and the structure is different depending on that. So you can actually go look. You can have a data structure that defines what this is and if you say, oh, that's the wrong one, after you go look, you can go switch it. That's pretty convenient. You can't do that in a strongly typed language. So with all that good stuff to say about it, why would I want to give it up? And the answer is you probably don't. We all know that strong typing is for weak minds. And strongly typed languages have their own issues. For example, if you have a piece of data which legitimately might be an IPv6 header or an IPv4 header, how do you declare that in a strongly typed language? The answer is you can't. You need to have some way to circumvent the strong typing in order to deal with that kind of data. Now, you can talk about object-oriented programming. That's always one of my favorites. Let's do garbage collection during an interrupt handler. Real-time impacts are awfully good on that. And the other thing is every now and then I'll hear somebody say, hey, let's rewrite the kernel in Rust or a language du jour. And I say, great. How are you going to do that? Well, we'll automatically convert it. We'll write some Ock scripts. We don't use Ock anymore. Do we? We're past that. Pearl, no, wait a minute. We're past that. We'll use some interesting language and we're going to just do an auto-convert. And that will work for about 90% of the system, at which point we hit the 90-90 rule, which is that 90% of the code can be converted in 90% of the time. And the last 10% will take the other 90% of the time. But there are things we can do. We can use the typing that's available. That makes things a little bit easier. Now, we can fix what we know is dangerous. He's talked about, in case, sorry, no name for years, and I still get that wrong. And only in public. We can fix things that we know are dangerous. And we can prepare for failure because we know it's going to happen anyway. So typing, how does that help? Let's say, well, our good example here is, in this case, referenced earlier, the ref count T. Now, a reference count is a very, very specific kind of behavior. It really is an integer. But you are using it in a very specific way. It's got a couple of properties that make it very interesting. One of which is that it should never be zero. And that you should never assign to it. You should only ever increment it or decrement it or look at it in the case of freeing it. So if we use a ref count T in a place where it's appropriate, we can control the behavior of that. And we can put checks into the handling of ref count T so that when we find a problem, we can assume that something bad has gone. Bad has happened. Either it's been attacked or there's a bug. At that point, we don't know and we don't really care. We just can just say, oh, bad thing happened in a ref count T. And that'll find a lot of bugs, prevent a lot of attacks. Good thing. So what do we know can be dangerous? Well, string functions. Anybody notice what's wrong with the stern copy example here? Somebody must be able to figure that out. Actually, it shouldn't be a sterling of anything. Yeah, if it were the sterling of desks, that would be whatever's already there. Of the source means it's coming. You're saying, oh, yeah, we'll just copy everything. And actually, there are two bugs in the second one. Because if it's actually the exact length of desks, you're going to lose the terminating null. So say there are two bugs in that. So the thing with string functions is that if you use them correctly, it's okay. The problem is that we've got a lot of people who've never learned to use them right. Oh, well, maybe they shouldn't be programming in the current. We've got no JS these days. Okay. Okay, sorry. So and then automatic arrays. Case was talking about this earlier. Why is this a problem? Well, because you've got a function where you're going to say, I need an array here and needs to be big enough for whatever the caller is using. So he's going to tell me how big it is. Well, we got two two choices here. If we want to actually make things safe, either we can check in the in my function here to make sure that that number is appropriate, or I can check all the places that call it and make sure that that number is appropriate. But both of those those approaches have have their problems. When you know what's appropriate, when we're doing casts, you can cast you can say I want to treat, you know, I want to point her to this object over here. And I want to treat it like a cred structure. Okay, but if I as an integer, I've got problems because it's not clearly not enough space for the credential. On the other hand, if it's something that's bigger than the credential, I may not be fixing setting all the information that I need, because the credit is only so big. And if I really have got a credit and then some stuff after it, I'm not setting that. I may have trouble there. So casts are kind of dangerous in this case. And then my second example here. I'll leave it as an example to you to figure out what that is supposed to do. But really what should have happened here is that temp should have been defined as an unsigned int and then there wouldn't be any problems at all. By the way, 80% of casts and I'm making the statistic up on the fly. 80% of casts are incorrect or unnecessary. Now it's not that these things can't be used safely, they can be. But checking that you that doing checks on your parameters can be expensive. And sometimes you don't and if you if you make the check at the at the end, you may be making a whole lot of checks you don't need to. And if you make it from from all the places that are calling it, well, what about out of three modules that you can't see? Are they doing it correctly? So we got a we got an issue here. It's like, you got to figure out which is more important. Do you find all the places that call it? Are you doing check there? Are you going to put the check here? And everybody's going to go through it, even though all but one of your callers isn't isn't doing it incorrectly. It can be a balancing act. Stacks. Why can't we get rid of stacks? They're they're machines that didn't used to have them. But anyway, here's a picture of the guy who invented them, you can blame him. They're convenient to push stuff on them, you pull your pop stuff off of it can be hardware accelerated, they're really great. But they're also convenient for mucking up. Why is that? Well, if you're in a function, you know that that the stuff, the parameters for the functions that are in this in the trace that got got you there are on the stack, you could go look at them. No, no, no need to actually think about passing everything.  And you know that anybody you called the last function you called is still on the stack below you. So you can just go look there, right? If you make a call and then a while ago, you know, he had an intermediate result that I could use. So you just just you know where it is on the stack, you've read the code. So you just go look for it. Or if heaven forbid, you're an attacker, you can use that same mindset, we'll find information you probably shouldn't have. There are some intermediate states from functions you're calling you probably shouldn't be seeing. Now you can make it harder to get to do that. You can put gaps between the stack, you know, between the stack elements that stack pages that works that that's a big help. You can if you know what what the gaps are, you can still work your way around that. Or if you're cleverer than I am, you can come up with other ways. You can erase what what's no longer needed case was talking about this earlier. So you come out of a function you erase the you erase the stack that solves that half of the problem right there. Now I just had a random thought. Let's randomize everything so that nobody knows where it is that'll take those hackers and just walk them up the head because they won't be able to find anything anymore. Well attackers and developers hate randomization and they both hate it for the same reason. That is it makes it really hard to find where things are when things go when things aren't working the way the code is supposed to. Sometimes you really need the real address of something or if you have the real address it's much easier to exploit. If you're looking at logs trying to figure out where your bug is or you're looking to log to figure out where where data is that you want to get at. And you don't have the address in the log. It's really tough to do that. You have to start working. And of course anytime you've written the debugger. I was like you've written the debugger is out released. Yeah it's I haven't supported it for two years and all of a sudden bam we've got hashed addresses. Well what am I going to do I'm going to rewrite my tool to deal with the fact that the address isn't really the address it's an arbitrary representation of the address which may or may not be accurate relative to the other addresses. Things yeah my tools get buggered. We can randomize data structures this is always fun. Someone has spent months getting this data structure in the right order so that on that arm arm 64 box that they've got it the caches never miss ever ever. And so now you put a little thing in into the compiler that says oh just whoop the things around into different orders. Now not only you're missing your cash lines but the structure actually got smaller because their ordering is you know the random order happens to be better from a size what size viewpoint than your careful cash line thingy. And so now your system actually runs slower and this is all bad and it's in the networking networking stack and everybody knows that nothing is more critical than cash lines in the networking stack. So there are we have this in now you can say you want you want to randomize your layout or not randomize your layout or let the system decide for you. It's OK. It's really one of the one of the big benefits of this makes it hard for somebody who's running a program to find a particular data structure in memory to do so because if they don't know what the order things are in they can't finger predict. Stack pages. Stack pages are just pages right. There's no reason we can't just shuffle them around and put them wherever we want. No reason they have to be in any particular place. We'll leave this as an exercise to the reader. Functions. Yeah. There's no reason that functions in a in the kernel should be in any particular order. There are optimum optimizations that that are a little bit trickier if you're going to randomize the order of functions but not that much right. And again this makes it a lot harder for somebody who is looking at the system to find thing find where things are so that they can go and find how they're going about it. How they're going to go about exploiting it. Thank you. This is a lot harder if you want to do it on every boot than it is if you want to do it at build time. But again there are a lot of clever people out there who know compilers which I fortunately don't. Now do I have to worry about performance in all this when I'm doing my looking at making the kernel harder or just doing things in general and it's like does the sunset in the West. Yeah actually it does why do I have to be particularly concerned about performance. Well true story. What's happened to me. All right. Hey we want to put this security code into the networking stack. No. Well why not. We can't measure any performance. Well then your benchmarks are good enough. Oh OK work work work work work work. Hey we've got benchmarks now up up new up they still can't find it still can't find and still can't find anything. Finally got a new piece of networking hardware in that was 10 times the performance of anything we'd had previously. Yes in this case in this error case under these circumstances we have a 2 percent performance degradation. OK great. You can't check that in because it has a performance impact. OK. Well fix fix fix fix get clever get clever. Hey look we fixed that performance impact. So it now is no performance impact. Can we check it in. No your benchmarks aren't good enough. OK. So performance is always going to be critical. And there's a reason for this. Damn. I hate it when my slides show up before they're supposed to. OK. Performance trumps security more often than not. Unless you're in an environment that is explicitly made the decision that they are going to take security over performance performance will win in an argument. And the reason for this is really pretty simple. Performance is objective and quantitative. You can come up with a performance number. You can say 2 percent degradation or 50 percent degradation or I ran this benchmark and I got a 17 instead of a 32. You can come. If you're a performance guy you can come in with a number. Numbers are really easy to use when you're making an argument. Where security is is quantum. If you don't know of any problems any security problems with some code. Well nobody cares about security in that. If you've identified a possible vulnerability. Well there are a few people who are interested but really in general nobody cares. You're not going to win an argument with that. If you've demonstrated a vulnerability people start to notice it but until it's exploited and has a name nobody gives a rats anatomy. But again once it's exploited people until it's exploited you can't go into an argument with a performance guy because a performance guy is going to come in with a number. So at the end is it worth the bother. It's like some of the stuff that we're doing in kernel hardening is really kind of minutiae or really looks like it. We introduce code churn. Ref count T 180 files with Ref count T's in them. 500 instances of Ref counts T's in those files and there's still more to do. The variable length arrays. It's like we're churning a bunch of code here. We're turning code that hasn't been touched in years. In order to do this hypothetical security possible maybe some day somebody's going to have a problem with that stuff. And we're introducing runtime overhead when we do it. Not always. I was like some of the the removals of the variable length arrays have actually sped things up. Harden user copy on the other hand has real performance impact. There are cases where people were copying information from inappropriate places in the user space and vice versa. You certainly don't want to copy directly from user space into a DMA area for example. That's usually a bad thing to do. And we introduce you introduce a lot of checks in places. When audit went in for example there was a lot of concern about how much impact it would have on every system call. Well and these are completely legitimate concerns. And the other thing you have to worry about is the developer experience. Okay now there's the user experience, there's the developer experience. The Linux community has been very big on the developer experience. Linus Torvalds is very big on the developer experience because if you don't have a good developer experience you don't have developers. Developers are really pretty important especially in a situation where not everybody who's working on the code is getting paid to do it. Now some of the things that we can that we do are as simple as check patch. All right? You put something in check patch that says hey you know what you're using this this interface but it's that's deprecated you shouldn't be using that. Or you're using this this function in a way that's really not appropriate. That okay well I haven't checked the code in yet. Yeah I can fix that before I do it. It can also be on the other hand it can be pretty picky like the the whole thing with %p. Where %p has that's how you're printing a pointer. There was a lot of debate on that about how that should be handled. Eventually it ended up being the simplest way but before that there were a lot of proposals like well you only you report %p and you weren't about %p and you only allow people to use it under these circumstances with these modifiers. Finally just dealing with it took care of it but that was a choice you know that was a choice in the developer experience of you know Linus and we see a lot of compiler warnings. How many how many people remember lint? Okay for for for those of you who are under 35 lint was w-w before there was a dash w when the compiler just said yeah I can do that. The more compiler warnings we generate the slower things can be. Compiler warnings about casts and about data structure usage really should be paid attention to as I think and that's one of the big reasons why we have that policy in the links from that we don't introduce warnings and that we get rid of warnings because we put them there for a reason. Because the stuff that that you're being warned about really can be dangerous. Finally a harder is subjective you know is it is the kernel actually harder when I make these changes. Does it really make it harder for people to develop the kernel if I make these changes. Yes sometimes yes it is yes we're making it harder here but are we making it more harder over here. So the answer to the original question yes it is harder. We are making it harder to develop the kernel. But the communities buying into it. I think before yeah yeah ten years ago they wouldn't have. Now why is this well part of it is that we're working in the open. Previous efforts that I won't mention by name that were done off on the side in their own patch stream for for direct commercial exploitation weren't going in and the reason for it you know pretty clear here is that that's not a community effort. Community effort makes a huge difference whether you're talking about performance security functionality support for obscure and bizarre and unnatural hardware it doesn't matter the kernel being involved in the community getting the feedback giving feedback working with everybody to make everything better gets you some slack. The amount of help we've been getting this has been awesome. Every now and then you think oh god I'm working so hard on it and then you look at the people who have contributed to the work you're doing and you say oh I don't have no idea who this person is. They popped out of the woodwork and made this little comment here and all of a sudden my code runs in half the time. Thank you yeah or I was going to do it this way we looked at you know looked at the 45 different interfaces that involved and somebody said dude why don't you just do this and you say oh duh or even better it's like oh yeah somebody else had a patch for that but then they got sick maybe you should look at that cool okay so again the amount of help we're getting just incredible the community really is buying into this but at the same time we're still learning where the bounds are as case was saying earlier when he has commented earlier yeah when you do your pull request you pull on your asbestos underwear and hold your breath until you get a response sometimes it goes goes in when you don't expect it to sometimes you get feedback but on the whole we're making the current making it a little bit harder for people to develop in general and we're making it easier for people to maintain because with fewer bugs fewer exploits people can go work on new stuff instead thank you so questions questions are any questions just a small comment I wouldn't completely discard rust as a potential language to use in Linux and and and the reasons are that that it's it's probably works in an environment where where you have both see code and rust code I mean I mean it's used in the Firefox that way so that so that portions are written with rust well and and generally speaking all the most of the kernel developers that I know love so so it has much stronger support in the company that well hang on hey for example see plus plus okay quick poll how many kernel developers here no rust how many kernel developers do not know rust it looks like about three to one against our questions next question I'll and I'll be nicer to you than I was to Yarko that's alright I'm gurney hunt your your comment about performance versus security is certainly true historically but at least on the hardware side of the house for which the kernel is the sibling of the hardware that's shifting because people are beginning to realize they've got to think about the security implications of their micro architectures or they get screwed in the long run through exploitation so don't you have to rethink or we ought to figure out how to balance this thing between performance and security rather than say one always wins and the other one always loses I'm trying to balance this off but I think that I am safer career wise not say anything about hardware development processes I just within the kernel I think I really think that we're still going to see until somebody makes a big stink about well okay first off we're as I said we're getting better about being able to do hardening stuff proactive things even if there is some impact but we still have to come in and say here is the kind of exploit that we expect to have a have this as a problem regardless of what the hardware goes okay and we are seeing a change in new hardware features for security that make more sense than what we've seen in the past so in the past we've seen you know whizzy hardware features intended to help security get introduced and people look at them and say gee how are we supposed to use that and it's like ring architectures are a perfect example you have a machine with 17 rings for security architecture and everybody looks and says huh cool ring zero ring ring something else and the other the other example and Tom Lyon had a paper on this in the 1970s was about a chip that he got for his Unix machine that they were building in Unix machine and it was a time a time chip and it would give you daylight savings time you know time in three different times it give all kinds of wonderful things and and they said well great but how do I make it give me seconds you know seconds since January 1st 1970 because that's the only thing I care about so spiffy hardware anything about hardware I don't think we can count on that in the performance security argument I think in speaking to the balance between performance and security where there is a trade off on that because there isn't always you can get security features in where there's no problem and everyone says that sounds great let's do it those are the easy wins but for the ones that are harder I've seen a shift in the culture in the kernel it used to be that nothing would win against performance and as you start demonstrating this long history of attacks against the kernel that has begun to move and I would say that there is a bigger shift in getting rid of bug classes like accepting a performance hit for killing a bug class is easier for maintainers to accept than killing exploitation classes because it's too far away from what people are sort of thinking about the old style was why do we need to kill the bug class let's just fix the bugs but that's not the right approach and I think the needle has moved enough that we say okay I guess we can get rid of bug classes but why kill exploitation methods they need a whole series of bugs to make that work so I'm hoping there will be some balance point that we reach but I'm still pushing to get as far towards killing everything as we can just my thought on the balance and how we'll reach it. Our proposal to continue discussion over the break which is now so let's break for a break and just to note so there is now a board here for the first session proposal so I know that there is already one plant on TPM and this TPM in the afternoon so please if you have more put it just on the right side of the writing device very so and let's get back at 20 past.


of pumping and dumping. Your pains are not to learn from us so that you don't repeat the same issues, the same errors, the same problems that Alex and I had to go through. We lost millions of dollars, dude.  You know, I was like, why do you need to go through that path? We are telling you what to do. What to avoid. ["I'm Not a Man"]


My fellow Americans, like so many of your homes, the White House is brimming with greens, colorful decorations, and a tree trimmed and ready for Christmas Day. And when Nancy and I look out from our upstairs windows, we can see the national Christmas tree standing in majestic beauty. Its lights fill the air with a spirit of love, hope, and joy from the heart of America. I shared that spirit recently when a young girl named Amy Benham helped me light our national tree. Amy had said that the tree that lights up our country must be seen all the way to heaven, and she said that her wish was to help me turn on its lights. Well, Amy's wish came true, but the greatest gift was mine, because I saw her eyes light up with hope and joy just as brightly as the lights on our national tree. And I'm sure they were both seen all the way to heaven, and they made the angels sing. Christmas is a time for children, and rightly so. We celebrate the birthday of the Prince of Peace who came as a babe and a manger. Some celebrate Christmas as the birthday of a great teacher and philosopher, but to other millions of us, Jesus is much more. He is divine, living assurance that God so loved the world, He gave us His only begotten Son so that by believing in Him and learning to love each other, we could one day be together in paradise. It's been said that all the kings who ever reigned and all the parliaments that ever sat have not done as much to advance the cause of peace on earth and goodwill to men as the man from Galilee, Jesus of Nazareth. Christmas is also a time to remember the treasures of our own history. We remember one Christmas in particular, 1776, our first year as a nation. The Revolutionary War had been going badly, but George Washington's faith, courage and leadership would turn the tide of history our way. On Christmas night, he led a band of ragged soldiers across the Delaware River through driving snow to a victory that saved the cause of independence. It's said that their root of march was stained by bloody footprints, but their spirit never faltered and their will could not be crushed. The image of George Washington kneeling in prayer in the snow is one of the most famous in American history. He personified a people who knew it was not enough to depend on their own courage and goodness. They must also seek help from God, their Father and Preserver. In a few hours, families and friends across America will join together in caroling parties and Christmas Eve services. Together we'll renew that spirit of faith, peace and giving, which has always marked the character of our people. In our moments of quiet reflection, I know we will remember our fellow citizens who may be lonely and in need tonight. Is the Christmas spirit still alive, some ask? You bet it is. Being Americans, we open our hearts to neighbors less fortunate. We try to protect them from hunger and cold and we reach out in so many ways, from toys for tots drives across the country, to goodwill by the Salvation Army, to American Red Cross efforts which provide food, shelter and Christmas cheer from Atlanta to Seattle. Churches are so generous it's impossible to keep track. One example, Reverend Bill Sengel's Presbyterian Meeting House in nearby Alexandria, Virginia, is simultaneously sponsoring hot meals on wheels programs, making and delivering hundreds of sandwiches and box loads of clothes, while visiting local hospitals and sending postcards to shut-ins and religious dissidents abroad. Let us remember the families who maintain a watch for their missing in action. And yes, let us remember all those who were persecuted inside the Soviet block.  And because faith for us is not an empty word, we invoke the power of prayer to spread the spirit of peace. We ask protection for our soldiers who are guarding peace tonight, from frigid outposts in Alaska and the Korean demilitarized zone to the shores of Lebanon.  Now, she said because of our presence there, her daughter can live a normal life. With patience and firmness, we can help bring peace to that strife-torn region and make our own lives more secure. The Christmas spirit of peace, hope, and love is a spirit Americans carry with them all year round everywhere we go. As long as we do, we need never be afraid, because trusting in God is the one sure answer to all the problems we face. Until next week, thanks for listening. God bless you and Merry Christmas.


In this module, we will discuss the isomerism with respect to alpha and beta configuration, which is the formation of a ring structure from a straight chain structure of a monosaccharide. In aqueous solution, aldotetrosis, aldotetrosis kya hai, tetra jis me char carbon hai, aldehyde group attach hai, and all monosaccharides with five or more carbon atoms in the backbone occur predominantly as cyclic or ring structure. In the ring structure, the carbonyl group has formed a covalent bond with the oxygen of a hydroxyl group of a long chain of the carbon. Therefore, forming an ether linkage, ether linkage kya hai jis me ek oxygen hai aur dono side pe r groups attached hai. So, ek carbon atom number, first or carbon atom number five ke dirmyaan ek cyclic ring banshata hai, which is shown in this slide. So, this is the carbon atom number one having the carbonyl group. Aur carbon atom number five ke dirmyaan, in dono ke dirmyaan, ye five hai, ye nazar aar hai. In dono ke dirmyaan ek ring form banshata hai, jo ek hi linear structure ko convert kar deta hai, in the structure of a ring. Is ring banshata hai, and other asymmetric carbon atom is formed. So, forming an other type of isomerism, which is called alpha and beta isomers. As the two reacting groups, the aldehyde and the alkoholic group belong to the same molecule. Therefore, a cyclic structure is formed. Eki molecule ke dono part hai. Isliye straight chain convert hojai gi ring structure mein, jo usi ka part hoega. The formation of these ring structures is the result of a general reaction between alcohols and aldehydes or ketones. Jab aldehyde ke saath alkohol react kerega to wo hemi acetyl binayega. Jab keto group ke saath alkohol react kerega to wo hemi ketol binayega. Ye hemi acetyl aur hemi kya? Hemi means half. Is ka matlab hai ke half ye apas mein inter convertible hai. Alkohol group aur aldehyde group in case of hemi acetyl and alkoho group and keto group in case of hemi ketols. Ye apas mein inter convertible hai. Agar ek alkohol group attach hoata hai, uske saath, aldehyde ke group ke saath, to it will form aldehyde aur hemi acetyl and hemi ketols. Which contains and is se kya? We have already defined kya hoega? An additional or additional asymmetric carbon atom. Thus it will form an other type of isomerism of having an alpha beta to isomeric forms. This is the general reaction of formation of aldehyde and alcohol. If this aldehyde reacts with a molecule of alcohol, this hemi acetyl is formed, which is having this alkohol group and this acetyl group. So these two groups are inter convertible to in each other. Iske waira se hemi acetyl aur hemi ketols, if ketone group is present here, it's reaction will be with alcohol, it will form a hemi ketols. We have already said that this is inter convertible. Half is in the form of alcoholic forms and half is in the keto form. keto form. When an other alcoholic group is attached to this amyacetol and amyketol in this reaction, it is converted in a complete acetol and in case of ketone, it is converted in a ketol group. We discussed this earlier. When glycosic linkage is formed, this amyketol and amyacetol are converted in a complete amyketol and in acetol groups where the interconversion ends. The glycosic linkage we have discussed in the previous modules that that particular linkage is formed between two different monosecrite units. Whereas this amyacetol and amyketol are formed in one molecule forming an other asymmetric carbon atom.


OK. Good afternoon, everyone. My name is Trini Alexander. I'll be talking about powering up eBay OpenStack Cloud and what goes on under the hood. So I've been with eBay for about seven months, and there's a lot of learning and the massive scale of eBay. It's absolutely incredible. So let me quickly walk you through the agenda. So I want to talk about cloud architecture. How do you build for scale? What are the things that goes under the cloud lifecycle management? How do we bootstrap a cloud? How do we add capacity? And also, we want to talk about some of the future directions that we are thinking. So before I begin, I would want to talk about eBay PayPal, the business and the demanding business. So if you look at eBay, we have about 157 million active users globally. PayPal, it's presented in 203 markets worldwide. So both have a lot of transactions happening, which requires a highly scalable dynamic cloud. So the eBay OpenStack, eBay PayPal OpenStack Cloud, roughly looks like this. So in its entirety, we have over 12,000 hypervisors, and that keeps growing on a daily basis. We have 300,000 virtual cores, 70,000 VMs, over 1.6 petabytes of provision storage. We have more than 10 AZs, 15 VPCs. And we are currently on version Havana, but we're in the process of upgrading to Juno. All right, so I wanted to talk about the cloud architecture a little bit, some of the fundamental design principles. So the cloud in eBay, we have different regions. Each region has different AZs. So we plan for failure. We want to distribute the workloads. It's not all in one region. So if you want a horizontal scale, for example, we want to add more regions, more AZs. So that's the fundamental principle. And the key thing is around planning for failure. Things could go wrong, either at the AZ level or at the region level. Or between regions, there could be loss of connectivity or between AZs. So we have plan for failure and the shared nothing model. So most of these services are in the shared nothing, but there are very select few services. Like Keystone, for example, we have a shared model. All right, so how do we look at the cloud lifecycle management? So there's a lot of moving pieces under the hood. So fundamentally, we have grouped it into these four big functions. So we have provision, deployment, monitor, and remediate. So provision is basically doing a bootstrap of the cloud, setting up your region's AZs, deploying all the open stack services. And on top of that, the tenant's workloads. The deployment is basically upgrading your control plane, doing the software rollouts, config changes, et cetera. And of course, if you have a cloud that's running at scale, you want to monitor it and listen for help. And if it's deteriorating, we want to do the remediation almost in a real time. So if I expand these four bubbles, like I said, the first one is the cloud buildout. The second one is doing the ongoing software deployments, monitoring, and remediation. So what I'd like to do is go through the pre-step of building a cloud. How do you bootstrap a cloud? So we have a process called rack onboarding. So there's a lot of pre-work that goes into building a cloud, so there's a lot of plans in terms of what data center, what AZ, what network bubble, what rack location, et cetera. These are all planned out. And the gear lands into the on-premise, gets wired up and powered on. At that point, there's a whole bunch of automation that happens to get that rack onboarded nicely into the system.  So there's a bootstrap ticket that gets filed with all the rack profile, the assets, skew information, et cetera. And when the rack is powered on, all the information is uploaded. There's a pixie boot that happens. And there's a custom image that we download onto all the assets, which does a scan of the system, collects all the infrastructure, the asset level information, uploads it into a central repository.  So that's basically what it really has. And in the bootstrap ticket, we have all the rack profile, assets, skew information. So we do a cross-check with that to make sure it is the right skew that landed on-premise. And we onboard it correctly. And also, we do a whole bunch of checks, like we do a LOM check to make sure that the BMC is accessible. You can do out of bad management. We set up the network interfaces, the relationships between VLAN subnet, the bubble, et cetera. And also, we try to calculate the asset fall domain. And once that's done, we go and update our internal CMDB with all the asset rack model information. So at that point, we're pretty much done with the rack onboarding. But we do one final step of spinning up a test compute and making sure these are really functional. So that's the last step. And another thing I wanted to call out is we do a, we only qualify a rack as onboarded if more than 90% of the assets are working and functioning. So if it's not, we would automatically, by the way, any of these steps, but it fails, we'll file a ticket which gets assigned to a team which works with the vendor and fixes the faulty hardware. Sometimes we get faulty hardware assets trying to get onboarded. They are stuck in the queue. But the key thing is we catch it early and detect and remediate quickly. So this is a process that's completely automated once the rack is powered. It's a fully automated process. This used to take weeks to get done. Now, with the automation, we have a time savings of, you know, from weeks going to a few hours. So that's an incredible savings. The next thing I want to talk about is once you onboard the rack, all these assets, they go through a certain lifecycle. So the lifecycle, there's all the way from ordered. Ordered basically is you've ordered the asset working with the vendor to get that on premise. Once it lands on premise, it goes into something called cold cache. And once it's onboarded through the previous process, it would be in the warm cache. So it's basically ready to be consumed. And once the assets are allocated, they go, obviously go into the allocated queue. And last but not the least, there are assets that end up live. So we want to do the tech refresh. So that goes into the DCOM. So there's another state I haven't called out here. It's just called faulty. So between warm cache and DCOM, it could go into a faulty state based on multiple reasons. So we have that state as well that I haven't really called out in the workflow. So what does this get us? We could look at our entire cloud and figure out what are the assets in which state. So it gives us a view into the system. So this is a sample data I cut out. So we have all these caches between ordered cold cache, warm cache, allocated. And I have faulty here as well. So the key thing here is this gives us the insight to operate our cloud at a much more efficient level in terms of the asset utilization. If things are stuck in cold cache, then it's indicative of a problem that in our onboarding processes not efficient. If things are in the faulty state and that's expanding, that shows us the insights into you have a lot of infrastructure, but you're not efficiently utilizing it. And same thing with warm cache. If the capacity gets depleted, it's a trigger to the capacity team to onboard new capacity into the warm cache or from cold cache to warm cache. So this basically provides the operational efficiency operating cloud at a cost optimal way and reacting to faulty assets very quickly. And this gives us the data transparency and the data confidence to operate our cloud better. Next thing I want to talk about is the capacity add process. So we have racks onboarded. And we want to add that into the respective open stack services. So we have a internal tooling, again fully automated, which would do a provision of the hypervisor to adding it to the various subsystems, like registering with Foreman, NOVA, doing the network onboarding, and doing a test of the newly provisioned hypervisor, basically by doing a test VM provision and also trying to check for IP pings and reachability, et cetera. So once all those various steps are passed, it's when we would do the enable hypervisor. So the key thing there is we don't want to enable it until it's fully onboarded. Because otherwise you will have a half onboarded hypervisor which would take workloads and it gets into a messy state. So this, again, is a fully automated process, which we used to take days to do this. Now it's within a few minutes. So there's a reverse process as well, off-boarding. So sometimes assets, hypervisors, drift. And they get into a state where we want to remove them from the system. So the previous process, if you kind of reverse that, you get into the off-boarding process. The same thing, fully automated process operates within a few minutes. So that's a bit of the cloud bootstrapping. So I want to jump into things that we do on an ongoing basis, which is capacity management. So we have a lot of VMs that users create, spin up. So any person with an eBay who has a badge basically can spin up a VM and do their stuff. And the key thing for us is we want to make sure it's not regulated from probably the wrong word. But we want to make sure it is effectively utilized all the VMs. So what we do is we look at all the VMs utilization. And if it's underutilized based on certain triggers, we would create some leads. And with the leads generated, we would work with the VM owners to say, hey, this is underutilized. Would you like to send them an email? Would you like to delete it? Or extend the tenure? Or do nothing? If it's a do nothing or delete, of course, the delete will delete the VM. But if they do nothing within 14 days, we would shut down the VM and give a gray spirit about seven more days before we delete the VM in case some important work is lost because of the employee's on vacation or whatever. And by the way, when we send the email, we send it to the VM owner as well as the direct supervisor of the VM owner just in case if folks are out. So this is an ongoing process, fully automated. You can see that that's pretty much a theme here. And till date, we have reclaimed about 15,000 VMs. That's a huge deal. So this is basically looking at your usage and you're trying to run your cloud at optimal capacity and also the best efficiency. All right, so the next thing I want to talk about is how do we monitor cloud? So if you look at the four bubbles said earlier, one of the things is monitoring. So we use Zabix internally to monitor our OpenStack services. So we monitor hypervisors, OpenStack services, and also Zabix per AZ. So we want to make sure if the Zabix service on one AZ goes off, it is detected and remediated. So there's a cross AZ Zabix monitoring as well. And once the monitoring detects an issue or an incident, there's a capability to alert. So we have various levels of alert. There's a page of duty for high-level, high-critical alerts, basically what wakes up somebody in the cloud upstream, or email if it's, say, not critical but high alert, and if it's not a crucial alert or a medium size alert, it files a gearticket. So why do we need alerting? It's pretty obvious. We want to monitor the health of the cloud and be humming along very nicely. If there are issues, we want to detect it very quickly and react to it fast. And one of the things that I want to call out is more of a, from our experience, we went a little crazy with creating all these alerts. And a lot of them were not really actionable. So you want to be very careful when you build out alerts. You want to make sure each alert has it's actionable and meaningful. And if you have an alert, you should have an equivalent runbook that lists all the things that you do when that alert happens. And if you don't really have a runbook, you really have to question what is the purpose of that alert. And one of the things that we did around the geartickets is all the lower-medium alerts, we just file a gearticket. It doesn't even surface on email or doesn't surface on anybody's page or duty. So it's containing the volume very nicely. Zabix Architectural, I'm sure it's a ton of material out there on the Zabix site. But this is just want to call out, there's Zabix Agents, Zabix Server, Zabix Database, and Zabix Frontend. So agents are installed on all the hypervisors that collects information from various hosts and feeds it into the Zabix Database. Zabix Database monitors the operating range. If the signal falls off that operating range, it'll go from OK to problem state or problem state to OK state based on the good or bad. And there's an alert or service that would do the necessary action. It could be a gearticket, it could be an email, it could be sending out a page or duty alert. All right, so moving on. Another thing we do is we collect, we have a lot of information across our AZs, across various systems. So we have Zabix, for example, that collects the metrics from various hypervisors. We have all the Nova, Cinder, Keystone databases. We have our own internal CMDB and the user databases that we have. And we could pull all the information from that and create analytics on top of it. So what we've done is we think of it as an ELK cluster. So we pull all the information into an elastic search cluster from all the various AZs into one central repo. And from there, we can cut different dashboards for investigating things, or we have dashboards that we watch on a regular basis. Like for example, there's an alert dashboard or a compute provision dashboard. So we have all these things, capabilities. And the power of this is you can actually look at an alert incident and pretty much drill down to, sometimes you have a stream of alerts coming in, you go to the alert dashboard, you can actually find out which region it's coming from, which AZ, which hypervisor to the VM level. So that's pretty powerful in terms of quickly reacting to some incident and getting to the bottom of it. Then the next bit I want to cover is the cloud infrastructure remediation. So this is the bit where infrastructure goes bad, like assets, hypervisors. So how do you react to it and how do you handle that? So with eBay Scalettes, you have to have automation in place to manage this effectively. So what we've put together is a remediation software that basically has a whole bunch of automated remediation jobs that run on alerts. And there's a ops console where the cloud operations team can go in and do all the things that they can do directly with the hypervisor. They can do it from the tooling itself. For example, they can see the asset based on the asset ID or the FQDN. They can click on a cog that does reboot hypervisor. All those various actions can be performed. And also, there's reports on all the jobs that are being performed. There's a lot of nightly or daily jobs that get executed. So you can see the status of that in a nice report console. And there's another thing that we recently added. So oftentimes, we get into the situation of migrating workloads. So migrating workloads could be because your hypervisor is running too hot, and you want to move certain workloads off of that, or you're dealing with Tech Refresh, or there are some specific use cases that we had to recently deal with within eBay. So we've been doing this a lot. So what we did is we automated that whole process of migrating workloads. So basically, flexing up new VMs onto new hypervisors and flexing down the ones that are running on the source hypervisor. So that helps. Like, for example, if you're running a hot hypervisor, when you migrate workloads, it cools down the running hot hypervisors. And we do that primarily because we want to be DR-ready. So we want all the hypervisors to be operating at a certain range and not above that. So hypervisors go down. How do you handle that? So we want to do a near real-time detection. And we want to do a near real-time remediation when hypervisors go down. So we have the monitoring system. We check against signals. And if the hypervisor is not reachable, we need to flag the remediation engine to go remediate the hypervisor down issue. But we want to be careful here because if there's a site-wide incident like a network glitch or something, you don't want to react to these signals. So we have a quiet period before we alert the remediation engine to go do the hypervisor down remediation. So what do we do as part of the remediation? The first thing we want to do is find out all the VMs that's running on the bad hypervisor and notify the owners of the VM saying that your VM went down because of the hypervisor down issue. Here's the best practice in terms of setting up your high availability VM cluster. And if it's production applications, we already have automation which would spin up VMs without even the application owner knowing it. So basically, they will see a dip in the capacity, but before they know it, it's replenished. And there are some things that we use to collect diagnostics from all the hypervisor down cases. We want to collect the diagnostics such that we can learn from patterns. And later on, we can do a predictive eviction of VMs from those hypervisors. So if you see a certain pattern that is leading to a hypervisor downstate, we can look at that and say, let's notify the VM owners and ask them to evict before the hypervisors have issues. So this is another fully automated process. The key KPI here is to reduce the time to detect and time to remediate. So we were doing this over days. We would take about three or so days to detect a hypervisor down issue and remediate. Now it's in real time. And it could be real time if we didn't have the cooling period and we opted to have that sort of the quiet periods, so that we don't accidentally react to false alarms. So the next thing I want to talk about is the remediation workflows. So all the automated jobs in our remediation engine, on the left side, you have sort of the various life cycles of various stages of the hypervisor. So it's getting the assets bootstrapped to provisioned to operating. And you're doing an ongoing ping. You're checking the hardware, monitoring. You're doing decom, or sometimes even the operator looks at it and manually flips it to a bad state. So the thumbs down basically shows the hypervisor that's in a bad state or asset is in a bad state. So that goes through a auto remediation. So we have various pools in terms of there are some assets that we fully operate and remediate. There are certain pools which are exclusive to a specific group. So they want to manage their own assets. So for them, we act as a pass through and notify the folks. And the idea is if there's a faulty asset, we would do the VM migration. We call it the faulty migrate. And get all the VMs out of the hypervisor. And get it to a healthy state. So the thumbs up basically brings it to the job, the remediation job succeeded. And there are times where not always all the hypervisor issues are remediatable. So there are times where we would have to work with the vendor and file an automated and get that rectified and taking care of. So this whole thing, again, it's a lot of steps. It's a lot of passing the bug from one team to another. So automating this has definitely helped in terms of getting the faulty assets under 1.5%. So we used to run at a greater than 5% faulty assets in the system. And getting that down to 1.5% or less is actually much lesser than that today. It's gotten to a much better utilization of your assets and huge cost savings. The next thing I wanted to touch upon is, we have all these tooling automation, but are we done? What's our future direction? So we want to look at, we have a ton more work to do. We have automated a lot, but there's a lot more work that needs to be done. So we want to get to the Cloud Lifecycle Management almost zero touch. So we should be able to run the Cloud fully automated, a self-healing Cloud, if there's an issue with the control plane, sends a signal to the remediation engine, the remediation engine kicks in, repairs it, and brings it back to the normal state. So all those things need to be built. So there's a lot of bits that we are still missing. And another thing we're looking at is containerizing the open stack services and looking at not just containerizing it, but also managing the containers, clusters, very evaluating Kubernetes at this point. And we're looking forward to building the next generation Cloud. So there's a lot of work. There's a lot of challenges. You're working with the cutting edge technologies like Kubernetes and all. You're looking at a fully automated Cloud operation. So we're definitely excited in this journey. And there's a lot more things that need to be built. So if there's a builder in you who's excited about the scale, feel free to reach out to us. We're looking for great people. Thank you very much. I'll take questions. So there's a mic up there. If you don't mind walking up to the mic. I have two questions. So by looking at this model, what are the capabilities that your tenants actually get out of the platform? Because as a service provider, you are actually taking all the actions when the failure happens. What kind of capabilities are tenants that are just hosting their workloads on this platform? They're getting from this platform. What is the company and drivers for them to migrate to this platform? So it's the eBay Cloud. We have Elasticity. That's the key thing with the Cloud model. And Elasticity provided at this provider level or at the tenant level. So we provide, you can spin up a compute. You can spin up a VM. And you can do your test project. And you can fold them once you're done. So with the Cloud model, it enables you that. So the value to the tenant would be to easily, without going through 10 levels of infrastructure procurement, you can directly go and spin up an instance, do your testing. Do they have to read their code to do that? That's what I asked. So the major problem that I see from my perspective is, if I have to migrate my listing applications onto the OpenStack to use this capabilities or APIs to be very intelligent there, so they may have to read their codes to call those APIs to do that. Or I can provide it from the service provider level. So like I said, you're running in a Cloud. You can do whatever is needed for your application. That's all on the tenant. What we do is we enable the Cloud operations. So if you want to spin up a VM, so we use OpenStack clearly. And we use that to use no sender as the block storage and give you all the services, the front end of the front office services to get you enabled. My second question is about what kind of capabilities that you provide to your tenants so that they know their health of their infrastructure. At the same time, health of their lifecycle, health of their security, do you guys provide any kind of capabilities from the Cloud for your tenants? So we're looking at building services that's in the works. So we're looking at creating a telemetry equivalent. So we can look at all the stats from the VM and we can subscribe to certain things and you can get alerted based on. Again, that's on the data plane side. The monitoring that I described here was on the control plane, the infrastructure, sort of the layer between the hardware and the OpenStack services. Thank you. You're welcome. Thank you. Could you talk about the automation tools you use?  That sounds like amazingly large. And what those indexers are? So it is a single elastic search cluster. So a lot of it is homegrown. And that's where I think we can partner with the community and make that better. I'm sure a lot of folks are running through the same issues. So let's solve these problems together. So I would love to have more participation there. Hi, two questions. First one's hopefully easy.  Do you have the means to go in and stop that from happening if the problems escalated beyond? Yes. Yes, I forgot to mention that. Thanks for asking. So what we have is even if it's not a site-wide incident and it's a lot of hypervisor going down, we rarely have that issue. So if we basically have a threshold, if it crosses more than X number, then we would flag it and stop that hypervisor down remediation process. And we want to be extra cautious because you react to a false alarm, it could. Cascade. Second question is, I know eBay spends a lot of time engineering resources on heat and power management and conservation of both of those. Do you do any work with that team to be able to basically spin up and spin down servers or to better consolidate services on to hypervisors that are underutilized? Have you gone to the full step of automating the system towards those energy and heat conservation goals? Not right now, but that's a great opportunity that we can look into. OK, thank you. Just one question. Could you share a bit of detail on how you made your decisions on storage-centric areas, like what kind of storage you used, and you mentioned Cinder, any of the services you use, and so on? That's slightly out of the topic. But maybe we can chat offline? Sure. Hi. You mentioned in the beginning of the presentation about you talk about regions and AZs. So did you mean OpenStack region and OpenStack AZs or something else? So let's say it's a cloud concept. So you want to have regions. You want to have multiple regions. Regions could be like US West, US East. And within each region, you could have multiple AZs which are having separate. If one of the ACs gets back up, if one of the ACs get impacted, your workload's running on another AZ. So it functions because it has a separate network backhaul, separate power supply, everything. So that's the AZ region model. So that's what we're trying to describe. OpenStack has a concept of region and concept of AZ as well. The mic is not on. Amazon's Google-style model where AZ is a fairly large control plane includes computers to the network. So I understood that what you presented is not the OpenStack concept of region. Exactly. Right. Exactly. All right. Any other questions? Thank you.