Welcome to my talk, it will be about confidential computing and the role of SUSE in this area. I hope that I will be over the course of the talk, make you excited about this topic and actually make you want it for your future deployments. So just briefly about myself, my name is Wojtek Pavlik, I am the general manager of SUSE's business critical Linux group, so heading much of SUSE's Linux engineering and business. And as such as being a general manager it means that I'll be mostly presenting the work of others, but I'm glad that I can because it's pretty awesome. And to introduce you to confidential computing, right, so we did a little bit of a research asking people, so would you actually be worried in case a cloud provider for your instances that you are running in the cloud would have access to your data? And the overwhelming answer like what is it 82% was yes we would be worried. And well this is kind of surprising because you know the cloud provider actually does have the access so people should be worried if they feel that they need to be worried in such a case. The cloud provider obviously does not have any reasons why to access the data and actually most of them are building significant infrastructure internally and regulations to prevent unauthorized access to customer data, but still technically there is the possibility. And this is one of the things that confidential computing is solving. Also 94% of customers are willing to invest into security of their data in the cloud and elsewhere beyond what the regulations require which is great because again this is clearly indicating that our customers are willing to look into new technologies for security and confidential computing is one of those. Obviously when we come to that the willingness to invest is there but there is also an expectation that well this will be easy and so that's what where we are going with confidential computing we are trying to make it really easy to consume. So what it is about it is all about data in use. When you are running a system most people today would not give a second thought about encrypting data on the network and then early days of the internet everything was going on plain text and actually almost human readable even like major protocols like SMTP or HTTP were designed to be human readable so people would just send packets open over the network. Nobody does that today and even just suggesting that somebody should use HTTP instead of HTTPS is considered almost insane so everybody is just encrypting data over the network. The same thing is happening with data in storage so if you look at your own phone then you will probably know that some years ago the internal flash of the phones was also unencrypted but today every single phone manufacturer is encrypting the data in the phone by default so that when it's lost it's not easily retrievable by somebody that has phone to phone. So the remaining state of data that actually we have not looked at is data in use so data that is residing in the computers or phones or anything operating memory the RAM the DIMMs themselves and that is still a potential avenue to attack a vector where contents of the RAM in clear text are accessible either by snooping the bus or by dumping the computer in liquid nitrogen while that's a little bit more advanced technique and then pulling the memory out because when it's so cold the RAM the dynamic RAM does not lose the contents and you can then read it off so that's what we'll be talking about figuring out how to encrypt the data in memory and that is not an easy task because obviously you can't operate on an encrypted data because it's encrypted there are some theoretical studies on how to do that but they are more on the crazy fringes of mathematics so there are two implementations that are interesting for us in this respect and that is AMD SEV and Intel TDX so how does it work it is actually pretty simple there is an encryption engine that sits in between the memory access module on the CPU and the actual physical memory that is using AES the industry standard block cipher to actually encrypt the data as it goes between the memory and the CPU so the CPU caches are not encrypted but the memory is and you can already see that this means that inside of the CPU and the CPU data is unencrypted and but at the moment at least it tries to leave the CPU it gets encrypted in the memory that certainly prevents any attack that would be using like physically monitoring the bus or physically getting at the memory contents there is also quite obviously a performance hit because every time the data goes from or to the main system memory there is an additional latency of a couple nanoseconds for the encryption and decryption but fortunately that is not a huge system impact because compared to the CPU the memory is already very slow and so most of the immediate data interactions happen in the caches chips are available today from Intel ARM AMD and even IBM but not all of them are in the mass market at the moment AMD has been there for quite a while Intel is coming to the mass market this fall with the Emerald Rapids chipsets now people have been talking about confidential computing for ages and that is also why a lot of companies are kind of tired of it and do not pay a lot of interest the first iteration actually was in 2008 where Intel did introduce loadable applets in the CSME in the management engine of the CPU it's a small processor that actually is embedded in the CPU that is handling the management and has been a center of a couple controversies as you have been you may have seen in the news so Intel actually implemented a technology where you could run your security critical code on the management CPU after being verified by a proprietary algorithm which was pretty secure but also not very useful because only very small snippets of code could run maybe something for a banking application there was a complementary technology called Intel secure display that would then allow a secure way to put the content of that on the screen like for pin n3 or similar but given that the use cases were so limited this never really caught on so later we see Intel trying again with SGX in 2015 and that is a lot cooler it actually does encrypt memory it does allow for a large application even a possibly a complete container to run encrypted but it has significant limitations so again useful for um very critical data processing that must not be seen by other processes or other applications or other containers but the main limitation is that although two applications would not be able to see each other memory even if they bypass some of the restrictions that the CPU is putting on not allowing that access um they still share a common kernel and that kernel is what is trying to separate them what is trying to enforce the separation so if one application manages to somehow get control of the kernel because it's actually a hacker running and exploit well then there is no protection from the encryption to the other application because once you go to the kernel you get everything and that's where the latest generation of technologies AMD, ACV, S&P, Intel, TDX or ARM CCA actually differ they implement the separation at the virtual machine level and the main difference is that this separation is enforced fully by the CPU it does not require the hypervisor to actually do it so the hypervisor itself and that is the beauty of it is outside of trust zone so the hypervisor even when compromised is actually not able to access the contents of the VMs and that gets us to the analysis of possible hardware attacks or possible attacks on on AVM that this is designed to protect against so the first one that we see here is actually a physical attack so somebody gets their hands on a server that is running a critical workload in a virtual machine and are trying to get the secrets that are inside so they employ a I don't know a bus analyzer a high speed device that usually they are those are quite expensive and is sampling what is on the bus or actually then using the liquid nitrogen method that I explained earlier to get the contents of the memory that is certainly protecting against that so one of the things that that comes here into play is that we are trying to get the data are limited to just the CPU die right because the die the silicon itself is really hard to examine first on modern CPUs it's actually flipped upside down so that all the actual active content is glued onto the substrate so really getting at it requires well disassembling the whole CPU and just extracting the die and then connecting it which usually just damages it beyond repair but also you would need something like an atomic force microscope to actually observe the data that is that is live on the CPU so that would require a significant research lab to just implement that kind of attack so we are trying to limit the the the plain text data to stay just in the die this technology helps us to encrypt it when it goes to memory but data actually needs to get off the chip also to the disk and to the network and that encryption happens in software on the CPU itself right so through the software encryption and this hardware excited encryption we can make sure that any data that exists in clear text exists only on the CPU so that is sorted the typical application for this would be when really our machines are in a hostile environment where something like this can be expected there is one way how to get through however and that's been documented in a few research papers but it's not really quite that practical and new CPUs are actually fixing that so the new silicon revisions are taking this attack into account and it's voltage glitching so most processors most electronic components do not react very happily to disturbances in their power right so if if a CPU is executing an instruction and suddenly the voltage that is powering the CPU drops then it actually may interpret a zero as a one or vice versa in that calculation simply because it's comparing the voltages in its internal memory to the voltage of the power supply and suddenly if they flip and they are the other way around well then it actually is executes instructions wrong and this is an attack that can be applied to the internal security engine inside of a modern processor which then allows to extract and reveal the internal keys and basically the nut is cracked the the secrets are spilled there is nothing more to do the the the fix is actually pretty easy most embedded CPUs have something that's called brownout detection so they have a circuit that is monitoring the voltage at very high speed and as soon as it sees a glitch the whole CPU is reset into default state so we're like rebooted and that means that that cannot progress because a CPU requires a stable voltage as soon as the voltage is unstable it really should just give up that is one way the other way is like embedding included or including embedded capacitors and diodes that make sure that any energy that went into the CPU can can get over such instructions and not be siphoned out anymore so this attack is currently possible with some of the existing CPUs but mainly desktop CPUs it will be rectified in the future and even then it is actually this on the picture on the on the left is a picture of how to do it right it requires opening up the the the machine and then spending several hours tuning that attack such that the voltage glitch happens exactly at the right instruction in the security processes firmware such that it hits exactly the condition that then skips the security checks so not very practical for um let's say attacking um something very quickly or remotely you need to have full physical access and it will be rectified so but there are many more interesting uses that do not require physical access and the number one is an attack against a compromise hypervisor so because a hypervisor can be compromised remotely this is a something that is much more likely to happen when a hypervisor has a vulnerability where an input from a vm can actually allow to take it over then this technology will prevent from the hypervisor from going back down into the other vm because the hypervisor does not have access to its memory it could overwrite it for sure right it could just write into their memory because the hypervisor has full access to the machine but then because the the vm expects the memory to be encrypted a block of zeros or whatever data that the hypervisor would put in would be encrypted into garbage and it would just crash so it's not a protection of the vm against crashing but it's certainly a protection against infiltrating or exfiltrating any kind of data from the vm so one interesting part is that caches cannot be encrypted because it would be a too massive slowdown our caches needs to be accessed within a few cycles of the cpu but they are each tagged with the encryption id so we can have multiple vm's running on a system and each has its own encryption key for the memory and so the cache lines in the cpu are tagged with that id such that access to that cache line from a vm that has not caused that cache line to be filled is resulting in a cache miss such that the the vm does not see that cache line the same for hypervisor if it tries to read something that is cached by the wrong id it just sees a cache miss and it tries to fetch it from the main memory gets it encrypted doesn't see the content so this way we can actually ensure that that the hypervisor cannot go into our vm we may still lose its contents but we will know that the contents will not be tampered with or that the attacker will not get what is inside. Now the even more interesting use case is actually a protection against a rogue hypervisor administrator so you may be running your workloads in the cloud or at an msp or at a data center where there is a 24 by 7 service monitoring your machines and a administrator may try I don't know be bribed or something to access your data and this is the same case as the last one there's really not much difference except that this is always an option right somebody that has rights to the hypervisor today has all rights to all the vms that are running under that hypervisor so the technology that prevents a compromised hypervisor from accessing the data in the vm as well protects against the administrator of such a hypervisor and this is actually quite beautiful because that means that you can put your data into the cloud and trust that nobody can see it now just encrypting the data is not enough because you could potentially upload your vm image into the hypervisor sorry into the into the cloud ask the cloud to run it but the cloud could either modify your image before it puts it into this secured confidential encrypted state or just emulate the cpu if you have a emulated fake cpu that does not really implement the encryption but just tells you that it's encrypting well then you all your protections would be again in vain and so we need a proof right we need a proof from the cpu that the image is running encrypted in the confidential computing mode and that it is running unmodified and the question is how do we do that and the answer is a remote attestation how does remote attestation work and this is really the magic of the whole technology you upload an image um the hypervisor puts allocates memory for it puts the image in the memory and then the image starts and the image asks the security processor inside of the cpu the your security processor please attest me and what the security processor does it looks at the memory contents and calculates a hash a cryptographically secure hash of the memory it looks at the register contents and calculates a hash of the register contents and then signs those hashes with its own embedded key that has been put in by the manufacturer of the cpu again a asymmetrical signature that is that is then put there this signed hashes this is called the attestation report and that then goes and is sent to you to the owner of the machine to the administrator of the vm that wants to run that vm and you can then check well this hash matches my image this register hash matches the expected state of registers at the start of the image or at the actual actual moment that it's asking for that attestation and because it signed and that signature or that key that signed it is actually signed by amd or intel or um whatever manufacturer of the cpu this is telling me that it's exactly the image that should be running and that is running encrypted and that the hypervisor did not modify the image did properly enable the encryption encrypted mode and could not have access to the contents and from now on the vm i know is the right vm and is running in a mode where the hypervisor no longer has access that is cool but it's also somewhat impractical and so that gives us to the next slide current software is not supporting this mode um using a new hash using uh while calculating the hash for the whole vm uh and the vm may actually be changing over time if it is stateful if it actually has some configuration the hash will maybe always different every time it started uh that's impractical so the easiest way how to take it from here to an actual practical state is using trusted boot trusted boot has been around for quite a while it is using a tpm to again measure individual stages of boot and so if we just could have a virtual tpm that is running inside of our vm that is providing these services like the trusted boot needs well then we would have a solution to the problem we would have something that is connecting the confidential computing the s e v s n p and the tdx technologies to our own uh old known uh trusted boot and so if we can get that piece of glue uh then we have a full solution that works with existing software and provides what we were asking for and that is the confidential computing there is such a piece of software and actually susan has implemented it it's called coco not coco for confidential computing svsm is for um secure vm services module something like that um secure vm services yeah second vm service module it's it's it was written by ur gretel big thanks to him um he is the reason why i can have this talk uh because he has not just implemented this but also a number of other components for confidential computing in the next kernel um it is now the default and standard part of the solution um taking over even other implementations and what it does it attests it's it's written in rust it attests itself using the service processor um and then creates a virtual tpm process platform module using the own chip uh ftpm or firmware tpm and then provides those services in uh to to to the operating system it is running as a browser so it actually running in vm pl zero so the privilege level zero and runs the operating system then in vm pl one and so it's isolated operating system from itself so the trust part is limited to a fairly small amount of very carefully tested and reviewed code and then the operating system even if that that gets compromised cannot compromise our coco not so it's just another level of of security in there um one thing one more component that we need to complete the whole picture is the remote attestation server so a machine that can talk to coconut and ask okay who you are give me your attestation report i will validate it and that is key lime it's a solution that was developed primarily for the standard trusted boot but can be just as well applied in this case because again we have converted the case of confidential computing into a trusted boot scenario uh using using coconut so it is actually more extensive than just the validation it is then using also runtime monitoring using ima um of course we would be using encrypted disk uh full disk encryption via tpm which is also something that is now available in linux product in susus products and has a full revocation framework for keys and what else so this is what completes the recipe with this we can have a fully attested fully confidential machine uh running anywhere and um this is then what i call a on-premise equivalent privacy right so your vm is as secure as if it would be running well in your basement if you want to try any of this uh susa is uh releasing regularly every three months a release of our next generation linux product um it is both available in its open source variant open source open susa variant community variant and uh currently also freely as the enterprise version the latest prototype that was released in march and we will be releasing another one soonish uh is called piss bernina and includes all these technologies in it so if you are looking at testing them yourself download piss bernina and feel free to play with it again susa will continue dropping more code and more complete implementations of this um and we will be very happy for any feedback now there is much to do this is not a complete stack yet so number one the cpu's are still missing right so md has been on it for a while since 2021 there are cpu's on the market from amd that support this but they are still not all that widely deployed when it comes to intel um the current uh sephir rapids cpu's do have an experimental microcode that enables this functionality but i doubt that will become available widely uh only the emerald rapids coming this fall will have it so we are talking about really um software implementation that is ahead of the actual availability of the hardware nevertheless even on software end there is a number of things that still need to get done and one of them is that um kvm is still missing support for a cvsnp so we do support a cvsnp as a guest uh but not as a host yet in the linux kernel um and that is depending on specific patches that um enable and i will not go into the details here enable specific aspects of member management to actually enable that and this is also what is currently blocking intel tdx support in the linux kernel but i'm quite sure that this will be resolved rather sooner than later and that both those technologies will get uh fully implemented into mainline currently they exist on this out of three patches um there is also uh somewhat missing support uh by some of the major cloud providers um aws and google cloud are using kvm or sort of kvm because what is aws is using isn't today called micro sorry nitro uh but it is still originally based on kvm and so they are hit by this missing support um microsoft azure actually is using their own little para visor and that is currently having full support for a cvsnp so that would be the place to test this technology um if you want to test in aws there are reserved instances with the md epic processors or somewhat more expensive than phc go instances but at least um the the the amazon nitro hub provider is not blocking anybody there from from trying this so we will of course as soon as i continue working with all those cloud providers making sure that the technology is fully available and enabled in our products such that this can be properly used uh and looking towards like end of the year time frame to put all the pieces together now i was talking quite a bit about how this protects the user from the from the cloud provider right so as if the cloud providers were evil of course they are not and so the question is should a cloud provider care about computational computing well yes because um when i'm saying that the technology protects a cloud provider also a user from the cloud provider that also means that this is a benefit for the cloud provider they have an additional technology that protects their customers from anything that can go wrong on their end right a again rogue employee or um even a even legally they are actually giving or they're actually being relieved of the liability of what is running on the customer system if they have no access right so uh that is actually quite a big win that the um cloud provider no longer has to care what the customer has on the system because they will never see it um what also is important is that this brings potential additional customers to the client provider or to the msp customers like um banks any enterprise that is actually um you or that that that is managing personal data under european gdpi directives or equivalents um it is not possible to transfer the data to other countries other legislations and so limits the ability to process them in the cloud banking industries uh trade secrets uh car processing what else uh any regulated market actually has trouble moving into the cloud and this enables that uh even a customer like susa currently uh would uh be enabled by this to move to the cloud susa cannot currently use process our code when we are compiling the distribution uh in the cloud because we have a certification called con criteria el four plus um that prevents us from using that because somebody on the way could modify the code introduce back doors or what else find out about vulnerabilities and so we have to process on premise again this technology would given that we will be able to prove that at any point in time uh the code was always encrypted and not like accessible by any third party would allow us to do that so could be a major business enabler what it means for the edge right um that's back from the hypervisor to um to the physical security if you are if you if you have your service if you are a telco uh and have your service at the curb of a street streaming videos to uh the houses in the neighborhood um or you are a restaurant chain where you have small communities cluster in every restaurant uh those places can be considered hostile environments where somebody may be interested in actually tampering with your system making sure that they get all the streams of the video for free or whatever and so this is actually a very nice way how to do that because you can take off the shelf hardware and actually run something on it that you know that it's exactly what you want to run on tempered with and um even though there is no specific modification hardware to enable extra security now given that we are at the open source summit uh i need to ask myself the question of devoyization so as probably everybody knows gplv3 was created exactly to combat a situation where somebody using cryptography or additional security measures prevents um modifications to the software that uh that is running on a device that is based on open source but still completely locked down um because like the vendor does not want that device to be tampered with uh a typical example or the first example that actually cost this to be called devoyization was the tivo uh video recorder that uh would not allow users although its firmware was more or was fully open source based modified that firmware and potentially fix bugs or improve it so simply that would mean that if you would modify the firmware it would just break itself the device would be a paperweight uh the nice thing about solving this exact problem so i am actually delivering a service and i absolutely need to make sure that that service is being delivered um by my uh business uh without somebody being able to temper with the service it could be uh maybe a device in a car that actually needs to pass all the certifications such that it is safe and does not kill people um or something similar so i need to deliver that service um and i need this level of protection and confidential computing will give it to me but at the same time it will allow a user to actually replace my VM with something completely different um and the device would still continue functioning so given that the protection is only given to an ephemeral temporary VM in the system i can use the hardware for anything else once my serve once is no longer being used for my service so from a operating uh from a open source kind of ethos this is actually a much much better solution than locking down the bootloader and the firmware so i hope that you have enjoyed the talk i think i'm mostly on time uh and i will be just closing with the question so why would we be using this right and the answer is really why you wouldn't because the same way that everybody is encrypting um network today everybody's encrypting drive today uh well eventually it is quite likely that everybody will be encrypting the memory just because it is giving an additional layer of protection without almost any drawbacks so you will want to have your memory encrypted you will want to have uh well many people will want to have compliance uh with uh with regulations we are striving to make it really zero effort such that the operating system potentially may be even completely agnostic of of this as long as supports are trusted boot um so i hope that um my talk made it an interesting um sorry my talk made made you actually be interested in this technology and consider it in the future and thanks please right so it would be multiplicative effect right so if you lose a certain percentage because of ima and certain percentage of this extra not at the moment plus you actually do not require ima to run this right so it i mean if if you run ima it gives you an additional layer of protection for runtime but um but confidential computing is useful even without ima right so so there is there is one more source of potential uh performance impact um that i did not mention so far and that is that um given that you want to have all your disk encryption and all your network encryption happen on the cpu die you can't upload it to a network card or to an io card because then you would have the data going over the pc express bus are completely unencrypted so we are watching the space but uh but of course um well not majorly involved in coding up anything yet right um there is another that there are like two avenues to solve this one is to actually have encryption on the pc express bus and to the card the other is having uh the encryption acceleration on the cpu and both are being developed in the hardware industry right so um i guess eventually we will support both but yes yes because because um if you are actually if you would be today combining confidential computing with that you would not have a solution true true but still even many of the models are hugely benefiting from running money on a gpu but so so then you would actually be sending plain text data over over the pc express bus which is not great here certainly not for this model over this threat model so we have key lime as a part of the distribution and we i believe made some contributions already but not a huge amount yet because from our point of view that is one of the more functionally complete parts of this whole puzzle uh and some of the other parts needed a lot more attention you want to answer well actually i was talking to uh to aws just last week in berlin and nitro actually does not really uh get in the way of confidential computing let's say that way right so they will be able to enable it that's the point right uh when you are doing the remote attestation it's the vm asking actually it's it's it's the coconut component in the vm that is running at vm pl0 asking through the hypervisor to the security processor please attest me yes yes that is the idea and then of course the system can use that to talk to the attestation server validate itself establish htds communication ssh keys and so on that are based on this this this exchange and then be security securely identified by anybody connecting to it that it is exactly what we wanted to have it is provided the keys to unlock the storage as well during the boot process that it doesn't have to contain them uh and so on fine i'm cool again thank you all


We got a cache! We got weapon cache! We got weapon cache! Get everybody inside! Sorry, Rupert. Landmines. We got RPGs, AKs. Landmines. We got landmines down there as well. We got grenades. We're on the Avenue of Approach. Coming up to our house. No one can come up on it. Hey, when I call for a gun one. I carried it. Incoming! Get over him! Get over him! Airways intact. He's got minor lacerations to the feet. Hold on, I got to write the front and back. Chestplate is applied. Corbin is treating at this time. Golf Romeo. Yeah. 7, 8, 2, 5. GSW. Right leg. What do you need guys? Sucking chest wound. Hold up, hold up. Hold up! We're sending machine. Hold up!


Welcome to foster them again. I hope you had a great day yesterday already enjoyed the talk so far My name is Lane Scurma. I belong to Sun Microsystem I joined the company years last year through the acquisition of my SQL. I've been with my SQL since April 2002 so One of the people that have been around for a while in the company Talk will be about my high availability solutions and I'd like to make a service announcement Right away, and we do have a deaf room in room in building AW Today, so we're having a number of my skill related talks and sessions in there and due to some unfortunate timing We have a bit of a timing problem right now There is a talk about my skill cluster in particular which has just started So if you're deeply interested in learning more about my skill cluster in particular I encourage you to to move over to this talk right away because I would you just scratching cluster on the surface and Not going too deeply into the technical details and in addition to the Schedule that is printed in the foster brochure and the schedules that have been printed out here We had to make some changes to our schedule and I tried to update the printed schedule in some places But it's probably best if you check the wiki page for the latest Changes that we had to make we had one speaker who had to return home due to some family emergency and Especially due to the recent happenings in the my school world last week We decided to have a Q&A session where Kayana is going to talk about the things that happened at my school last week And what they mean and we want to get into a discussion with you about Well, what do you think of what son should do with my school? We are going to explain what the changes at the happenings last week me to my school and how we plan to go forward So I if you're interested in my school as the project. I encourage you to be in our deff room at a quarter past one today Okay, that's it for the announcements Let's get going So high availability solutions for my square what I'd like to talk about is giving you a general overview About what the typical configurations are how people that we're working with make their my school installations highly available I'll be starting with a few basics just to give in and An introduction to high availability what the concepts behind it are some Practical advice and then I'm going to start to explain the various features that my school provides be itself and The additional tools and technology that is available in open source to make them single my school server Highly available so you can really rely on it in case one of your system goes down I'll be talking about a bit more in detail about DRBD, which is a tool that is Frequently used in these configurations. I'll be mentioning my school cluster What it does and how it differs from the regular my school server and I'll be going over some other tools and applications that are useful in this in this context Okay, why does high availability matter at all? I mean I Learned it at the hardware last week. I have one single PC that serves my personal home page blog and everything It died so I'm pretty much offline right now And I I don't have any high availability solution at home So I should have listened to my own advice, but of course, it's also a financial issue the thing is in IT something can break and usually it will at some point in time there are various components in today's computers that do wear out over time and Accidents can happen Disasters there are various things that can really disrupt your service operations or Simply put from time to time you may even want to update a running system to a newer version because there's a security flaw The server that you're running is known to be vulnerable for a buffer overflow How can you update the server without disrupting the ongoing services? So even in for those cases it may make sense to consider having some kind of an HA configuration That allows you to perform such tasks downtime is expensive. Yeah Especially in today's world everything is connected websites serve to Customers want to take their orders once the site is done and people can't deliver or dispatch their orders They look elsewhere. There's plenty of alternatives and well your boss won't be happy about that so if you are in charge of Maintaining some kind of infrastructure that serves a public service make sure that high availability high availability is high on your agenda and Yeah, visitors won't come back and Plan about this in advance Really if you start configuring and designing your structure or infrastructure Make sure that the high availability aspect is being considered from the beginning Because once your system is up and running it will be much harder to implement such changes and making it highly available without Disrupting service again to make those changes in particular we will be talking about High availability clustering which means we are providing redundant components that make sure that in case of one of the Components of services goes down There's another one that can take over the same services immediately and can make sure that The user or the application does not really notice what's going on in the background the typical scenario is that you have multiple boxes and You have them configured that they are able to switch IP addresses quickly So a service that is used to point to the IP address of machine R can simply be redirected to machine B or machine C and get the information from there if needed and Also, it's possible not only to switch the IP address But you can take down an application on one machine and start it up on the other with taking over the data That just has been accessed from the from the other system before There are a few terms that you should be aware of fail over versus fail back for example fail over is What's happens when one machine goes down and the services fail over to the backup system Fail back means that once the system that has crashed has been up again that services move back to the original system before in some HA scenarios the The backup machine might be a bit lower in performance and is just there for the eventual case That something goes down that the service doesn't Get disrupted completely but can still be served even maybe on a lower performance level And once the original and the main system is back up you fail back to this one. So you have full performance So it's just helping out in case it's needed and The term switch always usually used in cases where the administrator Knowingly and willingly switches the services over to another machine for example because he has to take it down for maintenance of any kind Clustering is also a term that is being used in high-performance computing where you put Thousands of boxes in a rack and they all crunch on the same problem to solve a problem in parallel And this has nothing to do with high availability. It's it's really about scaling a job across many nodes and High availability clustering is not in particularly designed or aimed at providing this kind of service And it's also not provided or designed to add you to give you additional throughput So even though it is possible in some scenarios to use an HA cluster also as a load balancing cluster You should really consider Having these in two separate functions because high availability and high performance are really two different things and if you If you are depending on your HA cluster to deliver the services Because all machines are active all the sudden one of the systems goes down and you can't deliver The data in the performance that you still need it There's really no point in in misusing your HA cluster or your high performance cluster for high availability So there are a few things you should keep in mind about that This is a very popular slide it shows what high availability means in terms of downtime Usually people want to get as much nines as possible people talking about yeah We need high availability with five nines, which would mean that you have five minutes of downtime in a year So the service is running nonstop all the time except for Accumulated time of five minutes You can get there, but it's getting very very expensive. So you really have to consider How much downtime can you afford and how much availability do you want to achieve? And yeah, this slide gives you a few examples on What people usually aim at? So data centers can live with eight hours of downtime within a year Banks usually have harder requirements and telecommunication and military Wanted even more higher available SPOF the single point of failure is the single component in your configuration That Causes the entire system to go down if it fails So what you want to make sure is that you don't have any single point of failure Which would cause the entire system to Disrupt So you go through your systems you make an analysis And some of the components that usually Should be on your list are hard disks Since they are mechanical devices they can wear out vibration can kill them But also you should consider what other components of my systems may be vulnerable to to break down network cables Maybe someone tripped over the cable connection is down the system isn't reachable anymore If you have an application that has a memory leak it starts to Crunch up all the memory your system comes to a grinding halt because it starts swapping is mad In that case you might want to consider switching to the other system until you have resolved the problem power supplies are also pretty Common for blowing up at some point and usually if you if you buy service nowadays They have redundant power supplies that can be swapped on the fly And the same is true for even more and more components in two days pc hardware so make sure that your systems Provide the redundancy that you require and all the components that you depend on are available at least in two if not more Versions or copies of the same thing The software that provides high availability usually consists of a few components that take care of of certain tasks Usually what you have is some kind of an heartbeat application This tool basically makes sure that all the nodes of your cluster Or of your high availability system are alive it makes sure that the applications are up and running Giving you the correct answers and the system is running properly And the nodes check each other if the other partner in his configuration is alive as well And depending on the outcome and the result of what the heartbeat application Identifies you usually have one controlling application, which is called the h.a monitor that then performs particular tasks or actions depending on on what the heartbeat has discovered So if if heartbeat Realizes that one of the nodes is down The h.a monitor then performs the steps required to migrate the application and the ip address To the remaining nodes and make sure that everything's under control again In most h.a scenarios You usually have a shared storage device like a storage area network where all your data is being stored So the the application servers in front are redundant But they all connect to a central storage where they access the data that has just been operated from the previous node An alternative to that is to add even more redundancies that you use a replication In any way that makes sure that the data is being copied instantaneously to another storage subsystem So in case that your central storage goes down you still have a replicated copy somewhere else And I'm going to Discuss this a bit more A shared storage resource like a sand can of course also become a single point of failure. Usually these these Sand devices provide lots of High availability functionality by themselves But in case your whole data center runs out of power something like that. This doesn't help you very much so depending on On your requirement you may want to consider having Backup storage somewhere else that really makes sure that you're not getting into trouble if your primary storage fed system fails Um split brain is also a term that I'm going to talk about it in a moment a bit more This is a situation that can be dangerous in case you have a centralized storage device because all of a sudden it may happen that Several applications try to take or access your data at the same time and Depending on how your application is written. They it may not be able to cope with that and of course Significant trouble on on the central storage system, but there are techniques that Help to avoid such situations, but it's something that you should keep in mind Split brain is something that can happen in certain configurations And your system should be able to cope with it And of course if if your storage is somewhere on a network you have some overhead when it comes to Reading and sending data over there versus having it on a on local disk drives on your on your note um So what you can do is you replicate your data and copy it to a separate separate storage subsystem somewhere else and Here if you have to decide if you want to make this Asynchronous operation that means that every change it has or every file that has been stored on that storage subsystem Is synchronously copied over to the other side and you really know that the copy is Complete and ready before you continue with your operations the other hand You can use asynchronous replication Which means that the data is being copied in the background and you don't really have Confirmation that it's the operation has already completed while your while the application proceeds this is a bit Easier to handle and provides lower overhead and Usually gives you higher performance because you don't have to wait for the synchronization to happen But in case of a crash of your primary system You can't really be sure that every data that has been modified has reached the other side yet before the system crashed So you may run into or you may face some data loss depending on how fast the replication system works Some more about split-brain that i just mentioned A classical scenario is you have a high availability configuration with two systems They are able to heartbeat each other So they are both alive and happy And think everything is good and all of a sudden the connection that is used for the heart beating dies So both are still alive, but they can't reach the other one anymore and They might be considering. Okay. The other one is not reachable anymore. I suppose it's not my task to take over All the jobs and they start turning over the the jobs and Well, it starts to assume it is now the The only remaining node in the cluster and starts to use activate the applications And if you have shared storage the node on the other side may Want to try the same thing at the same time. So this can really lead to bad things And what you need to do is make sure that you have some kind of a so-called fencing System or you have a third system that acts as a moderator and can really tell the two nodes Hey guys, even though you do both two can't talk to each other anymore I know you're both alive.  It simply Turns off the other system so that it really can't Perform any badass anymore until the administrator has resolved the problem Some general rules Well prepare for failure We simply have to assume that it's going to happen at some point So you better make sure that you know how to handle it And these preparations can start with Simply rules for the administrators of what switches to use or what cables to put somewhere else if you have a manual failover system Or how to define your automated higher availability solution And of course consider what data is important to you What part of your system should be highly available? What Is it necessary that you have a redundant copy of the operating system that is always in sync? Is it enough to just synchronize the data directories that are changing from constantly? So there are a few aspects that you should keep in mind and consider here And very important don't over engineer this whole thing. Keep it simple. So it's it's easy to grasp And and to see what it does Stick to established components I know it's tempting to start writing your own ha system creating cool scripts that do these things Um Not sure if it makes sense if there are so many existing solutions available So try to lower complexity as much as you can try to automate it and of course test it don't wait for the Disaster to happen before you realize that your failover solution is not performing the task as it should be so every change should occur With a repeated test afterwards to make sure that the modifications don't cause any mayhem And if your ha system is properly designed, this usually shouldn't cause any service disruption or at least only very little Compared to what would happen if you don't test your changes and the system doesn't fail over in case a crash occurs Okay, let's get started with mysql related things here Mysql provides an inbuilt mechanism that allows you to replicate data from one mysql server to another It's called mysql replication. It's a one way replication system So if you have one server and another one is able to get all the changes From this one and and apply them locally So you have one master, but it's possible to distribute these changes to Many slave servers in parallel. So you're not limited to a one-to-one system here We can use what we also call fan out replication So you have one single system that fends out the changes to many others The whole system is asynchronous which means that You can't be sure that all updates deletes And so on with which happened on the mysql master server are also replicated on the slave at the same time Depending on your network connection and depending on the amount of changes that are being replicated The slave can lag behind and that's something that you should keep in mind Um The master maintains all the changes that are happening All the sql statements that modify data in any way are being stored in a binary log file, which is A special format file, which includes usually all the sql statements That's what we usually call this whole statement based replication So the mysql server really records the sql statements that were issued to modify data and puts it in his log file The slave takes these statements and then simply repeats the commands that have been performed on the master to update Its local copy of the data accordingly Um starting with mysql 5.1. We have also introduced something that we call row based replication In this case It's not the statements that are being replicated But rather the data that has been changed to the actual rows that have changed in the tables are being copied over to the slave This has some advantages and some disadvantages that i'm going to talk about in a in a future slide The cool thing about mysql replication that it's really Very easy to set up. We have It documented in the manual Um, you just need to have a second mysql slave server Configure it point it to the master and tell it okay. This is where the master's data is at right now start replicating now And this usually works pretty reliable The thing that you should keep in mind is that the Replication slave performs all the updates using a single threat so In comparison to the master which can process many queries in parallel The replication is replicating these changes in a serialized fashion So the slave takes more time to process all the events that are happening on the master So as funny as it may sound in some cases it may make sense that your slave server is a bit More performant than your master server because of of this particular problem However, all mysql server provides really is the replication We don't take care of automating the failover There is not really much of a high availability solution built into the mysql server It's just replicating assuming that the master is alive So in case that you want to use this system for a high availability configuration You have to perform some additional steps And make use of some additional tools that i'm going to cover now This is just a graphical overview of how the replication system works On the left hand side you have the mysql master which is the primary server receiving all The data modifying statements like update insert and delete It applies these changes to its local database And also stores all the changes in its binary log file On the right hand side the mysql slave simply connects to the master Pulse it for all the changes that has happened since the last time it connected And stores all these changes in its local relay log file before it starts applying them using This the sql thread And as you can see here the mysql slave itself can also store these changes in its local binary log again So this allows you to daisy chain them for example Some more words about statement based replication This is the system that has been included with mysql for a very long time I think yeah, it was mysql 3.23 or even earlier The rock files are quite small since all the only thing that we're logging is the sql statement itself Which also allows me to use this binary log file for other purposes like auditing to make sure that the application isn't performing queries Queries that it's not allowed to or if you want to check on your users what kind of data modifying statements they have issued There's a tool that is part of the mysql distribution called mysql bin log That converts the binary encoded log file forward into plain text sql again, so you can look at it And the replicated tables don't really have any requirement when it comes to having primary keys or something like that It just replicates the data as it is Unfortunately, this approach has a few downsides to it most importantly If you use sql functions that are non-deterministic or if you use user defined functions You may have to watch carefully if these are being replicated properly For example, uuid which creates a unique id which is unique to that particular sql node Can't be replicated safely the uid would be different on the replicated side And there are a few other functions. We have a whole section in the menu that really describes Which functions are replication safe and in which you should handle with care but in general Most users still prefer using statement replication because it's proven and Easier or more transparent to to to work with So on the other hand, we have row-based replication Which means that the master server performs the sql statements and then Locks all the rows that have been affected every row that has been changed are added as being included in the binary log file Which is a technology that most other database management systems also use This makes it easier on the slave side to insert the data Because it can just take the rows as they are and pipe them into its local tables without much consideration or much processing work That of course means that the binary log file grows significantly depending on how much rows are changing If you have a statement that just changes a single row, this doesn't really affect you much But if you have massive updates that update your entire tables Every single row needs to be replicated. So in some cases this really Increases the size of the log files And that is something that you should keep in mind if you use binary log files for performing backups And want to restore from the log file. You have to consider the space that is being needed And there is a restriction that row-based Replication tables need to have a primary key, but depending on your database design, this might actually not be a much of a problem And the replication slave may provide different results of queries. So if you have An ongoing row based replication that really updates a lot of rows on the slave and you run the same select statement Since the slave hasn't fully caught up with all the rows It may return just a subset of the results that would be returned on the master server. So In that case, this is something that you should keep in mind if you're using replication To scale reads, for example, if you have many MySQL servers that are serving your application with read statements The slave may be behind so the results it may differ if you're using row-based replication And it hasn't fully caught up yet This is an overview about common Topologies on how to set up master slave replication On the top left you have one master one slave. That's the standard configuration On the top right you see one master that replicates the two slaves in parallel And these slaves are really independently from each other if they are made out of different components and have different performance Values one slave may be more behind than the other or So They they don't really care about synchronization between themselves As I said, you can daisy chain them. You can have one master that replicates to one slave Which in turn then replicates to two other slaves Something that isn't possible. Unfortunately is having two masters that replicate to a single slave a fan in configuration It's currently not supported by MySQL replication It's also possible to do a circular replication So that you have two MySQL servers that are both being a master and a slave of the other one So in theory you can perform inserts and updates on both sides and they're being replicated to the other side But this configuration has a few things that you have to keep in mind. So That's a I think I have a slide about this later on And you can also configure them in a ring configuration that really Statements are being replicated in round robin fashion, but that too has the same Caviets that master master replication has Okay, more about master master. So as I said, you have two MySQL servers Both a master and slave of each other at the same time Which is a very popular configuration for MySQL high availability systems Something that you should keep in mind here is that this system does not really help you in sharing your load even though it is possible to Issue writes on the same side Since they have to replicate the changes. They are performing both jobs So every update that is happening on the one side Plus the updates happening on the other side have to be processed. So this doesn't really give you Any improvement in in distributing the load So in theory, you usually have them configured in that way that both could accept writes, but you the application only Performs write and data modification operations on one of the two nodes And in case of an emergency, you could fail over to the other one Because if you start sending writes to both systems, it's really hard to tell which of them is the authority Especially if they are lagging behind and replication hasn't finished yet And one of the nodes goes down you may end up with data that is in an inconsistent state So if you're using this configuration, write updates should only happen on one node all the time If you want to use MySQL or if you want to distribute write load What you should look at is what we call sharding where you have several MySQL servers which serve different parts of your data and you have either Using either the MySQL proxy or some logic in your application To distribute the load depending on maybe a key or a hash on on the different MySQL servers Okay, what happens if the master fails? Nothing the slave simply stops accepting Updates from it. It tries to connect and if it doesn't connect it just waits and sits But it's happily answering queries at the same time. So The replication is just Taking down if the slave fails again, nothing will really happen. The master will happily continue updating its binary lock It's just that the slave isn't following it anymore And when the slave comes back up again, it's a later point in time It will simply have to replicate all the changes that have happened on the master server in the meanwhile And yes, this is not an ha solution yet simply because we're just doing replication. We're not taking care of Failing over automatically or anything like that So what you want to do is combine MySQL replication with some other solution and usually In the linux world you would use heartbeat, which is part of the linux ha project Which allows you to move ip addresses between systems and also Allows you to start or stop MySQL server using scripts that are being executed in the case of a failover situation Load balancing here in that respect means that you can use the The replicated slave for distributing read loads. So if you're having lots of selects The slave can happily answer these as well And a replication slave is also very useful if you want to perform backups Without disrupting or disturbing the master's operation Depending on what kind of backup technology you use it can affect the performance of the MySQL server MySQL dump for example, if you're using my isam tables will lock the tables for a while until the backup has finished So your application may not be able to proceed until the backup is done If you distribute that on a replication slave it it can be even shut down Through performing the backups and it will then just catch up with the master server once it's back up and running again Tricky to fail back Yeah, you have to make sure that your tables Are in sync once you have you've had a failover situation and you're ready to fail back to the now restored master server Take a look at if your tables are consistent and intact so We don't have anything in here that allows you to resolve conflicts that may have occurred in your data after the failover solution There are lots of scripts That take care of that and i'm going to give you a few links about this Some more about linux ha also called heartbeat. Oh, there's a typo um Here common configuration is two more nodes, but you can also configure to set up even more nodes, I think The linux ha project Refers to some configurations where they have 16 nodes in an ha cluster that is being monitored The fencing mechanism used by heartbeat is called stoneth Shoot the other node in the head Which basically means that you have a device that can be controlled by the cluster to switch off the power from a certain machine So if if one of the nodes is acting up it's simply being switched off um linux heartbeat allows you to create policies that depending on on your current situation May result in in different actions that are going to happen These can also be dependent on the time. So for example A failover mechanism may look differently during the weekend or During daytime when the system is very busy. All this can be configured Um, it ships with quite a lot of applications including support for mysql um, it has a GUI tool and Very low dependencies on other external applications libraries And it can perform failovers in a very fast way and this is how It usually looks like You have a mysql master so that replicates to mysql slave You may have additional slaves that are also taken part in the replication and heartbeat Make sure that the applications Only the access one of the systems with the virtual ip address that you are moving over in case of a failover case Another system that is commonly used to make mysql more highly available is drbd Distributed replicated block device. This basically says it all um Other refer to it as raid one over the network. So you have assist two systems Which both have local hard disks and one of the system is the primary server. The other one is configured as the secondary Every block that changes on the primary server has been Or every change in the blocks of the disk is being replicated to the secondary system So you always have a one-to-one Copy of your hard disks that is in the primary system so to say Um Recent versions of drbd can be configured to do this replication either In a synchronous way. So once you commit and save your data You can be sure that the copy on the secondary has been updated as well or if you have lower requirements on on your system you can Configure to be in an asynchronous Method as well So your local application doesn't have to wait for the commit to happen and can proceed but then of course you again have to make Sure that You can live with some data loss in case the replication hasn't finished If one of the nodes goes down it has been reconfigured It automatically resyncs with all the changes that have happened on the other side in the meanwhile Since it's performing on the block level of your application of your of your operating system It doesn't care about the application on top of it. The application writes its data into the file system drbd takes care of just replicating the blocks that have been changed regardless of what the application is doing Um, it can also mask local IO errors in case that the local hard disk on the primary fails It is still capable of getting those these blocks from the secondary if needed So it's not like you have to fail over immediately in the case of a disk corruption And the the common configuration is you have one primary That has mounted the file systems where all the changes are being happened and the secondary is a so-called hot standby That just performs the replication and does nothing else in the meanwhile Because the problem is since the changes are being made on on the block level of the disk You can't really mount that file system where the the blocks constantly changes underneath the file system There are only a very few file systems like gfs or the oracle cluster file system that are capable of handling This kind of situation your usual file system like xd3 or Xfs would become severely corrupted if you would try to mount The secondary file system while it's still being replicated drbd is open source. It's part of the linux kernel. You can get it from the link down there And in the mysql scenario what you would do is to configure the mysql server to put its data directory on A device that is being replicated using drbd. So mysql updates its local tables Stores all the data in its file system drbd takes care of synchronizing every changes to the to the passive node as you can see over here In the case of a failover The first thing that drbd has to do is mount the file system perform a file system check And then mysql d can be started and access the data on the replicated copy and continue as normal So the mysql server usually is only started in the case of a failover You you don't use the mysql replication in that case One downside to drbd is that it's available on linux only There are other solutions on other operating systems that perform similar tasks Okay, mysql cluster is a completely different beast I have to speed up a bit. I'm running out of time again um mysql cluster Has been added to the mysql server. I think two or three years ago um We acquired a company that was a spin-off by ericsson who created this cluster system for telecommunications Solution so mysql cluster isn't really an integral part of the mysql server um What it does is it it creates so-called data nodes Which are processes that run on on separated systems and they take care of distributing data Equally between the nodes and they take care of keeping the replicas Consistent and making sure that you have redundant copies in in the depth in the various nodes Um, they synchronize those changes in um synchronously between the data nodes And the mysql cluster nodes take care of the failover situation by themselves. So that's already built into the system Once a node has gone down and you bring it up again It automatically integrates into the cluster again and synchronizes with the nodes So the redundancy is being restored as fast as possible Um, it supports transactions and there are various ways on how to connect to it Some more about cluster here um Up until recently mysql cluster had one severe limitation in the sense that all the data that you wanted to store in the cluster needed to fit in the main memory of Your systems that wasn't capable of storing data on disk This has been fixed in the meanwhile With the only requirement remaining that your indexes have to be Stored in memory, but every other data of your tables can be stored on disk It's not suitable for all applications So you you if you are designing a system using mysql cluster you have to Look at what kind of queries that you are issuing Since all the data nodes have to constantly communicate with each other There are a few query patterns that aren't really suitable for mysql cluster Especially if you have complex joints that that involve Fetching data from lots of nodes or if you have table scans or range scans that scan An a large amount of your tables It doesn't support foreign keys yet And yep It depends on your data It's it's a very specialized solution and you should consult with us if Mysql cluster would be an option for you But it can also be combined with for example the mysql replication system to make it even more redundant Just to give you an overview the mysql server itself Connects to the mysql cluster and uses this at a storage engine You may know that mysql is capable of storing tables in different storage engines The most two common use ones are myisam or inudb Which store tables locally on disk In this scenario the mysql server simply hands the the tables over to mysql cluster processes Which run on on separate machines and they take care of distributing and storing the data The interesting thing here is that you can have several mysql servers all accessing The same cluster nodes and all seeing a consistent picture of the data So if you're performing updates on the server on the left and you perform a select on the mysql server on the right You will immediately see the changes that have happened in the meanwhile In addition to using mysql server The mysql cluster also provides an API to talk with the cluster directly So you don't have to use SQL or the mysql client server protocol You can talk with the cluster with your application directly using our API I'm going to skip that one MMM is also a tool that is commonly used It's particularly suited if you have two mysql servers configured in a master master configuration So it's an alternative to heartbeat a bit more simpler. It simply changes the IP address And and updates a few changes on the mysql server. So it's it's really focused on on this particular configuration But it's pretty widely used and for people that want to sit up linux heartbeat. It's a good alternative Flippa is also a little script that performs a similar task But it's not doing anything in an automatic fashion. This is really lo-fi so to say Flippa just automates the process of switching The roles between a master master configuration and you have to do this Consciously by issuing a certain command, but it's still quite helpful if you're an administrator And you just want to switch your application to a different mysql server Sorry Flippa may come in handy Right at cluster suite also very Popular in the meanwhile many people use it for their applications Has similar features to linux ha but can do a bit more and also support some load balancing functionality Also has good support for mysql server. So that might be something if you're using sent os or reddit enterprise linux It's very well integrated into it Solaris cluster open ha cluster is a solution made by sun primarily for the solaris operating system We recently open sourced the entire suite and are Working on improving it. It also has excellent support for mysql And also provides replication to remote facilities And using it in combination with storage air networks for example So if you are running on solaris The solaris cluster suite is something that you should investigate closely And some more links. I think I mentioned many of these before You can take them from the slides Yep, we got these before Oh, yeah mudkit is also a very popular Collection of scripts and tools that you as a mysql administrator should be aware of they really Simplify a lot of the processes of working with mysql And continuant has just recently open sourced their replicator Which is Another way of how to replicate data in between databases I haven't really looked deeply into it yet, but it sounds kind of promising. Okay Five more minutes left for questions Any questions? Yes So the question was which of the limitations of replication are Just a problem of Of the the system itself and how it's configured and which are limitations of the mysql server and could be improved by fixing the mysql server And I am honestly can't tell you out of my my head right now There may be some limitations That we're working on fixing and some simply are inherent to the system and how it works If you have a particular one that you're interested about I can try to find out but right out of my head I don't really have an answer for you. Sorry Yes Right. So the question was mysql master master replication has a few Issues that you should be aware of and one of them for example is if you issue updates to both servers at the same time You have a consistency problem that since it's asynchronous replication You can't be a hundred percent sure that both servers are always in the same consistent state at the same time So if your application distributors the rights to both nodes and one of them goes down It will be very difficult for you to find out which of them has the um The real data Yes, as long as you write to one, it's pretty safe Considering that you know, this is the main master server It it has the authoritative information and the replica may be behind but that's something you can check from the logs No, it does not the question was if it supports synchronous replication and no unfortunately mysql replication is always asynchronous If you need synchronous replication, you should take a look at the rbd and do it on the file system ever Any more questions? Yes The practical use case for replicating from one master to multiple slaves or right um What is the practical use case for replicating from one master to one slave and then to several slaves chris Scaling read only scaling right, but Yeah, so really if you have more slaves You can have an application that distributes the selects to all the various slaves instead of just putting it on the master so um It's hard to explain I'm really missing the the key point why you want to do it master to slave and then from there to in several more slaves if you have Multiple data centers. Yeah, that would be one example If if you have a distributed system where you have a company with several branches that need copies of the data You can then distribute these into separate locations for example All right, then no more questions I hope to see you at the mysql death room then for some more additional mysql related information and Thank you very much


As you know freedom offers a ton of music for you to use for free in your videos and live streams. The three main catalogs however have some restrictions and some of these will be new to you so pay close attention. We have EpidemicSound.com this is what it looks like when you sign in on that website. We have AudioMicro.com which has a ton of music and sound effects and we have PositionMusic.com which has a lot of professional music used in movies, video games and TV shows. The first restriction that many of you don't know has to do with AudioMicro. You cannot use Loops Packs. Loops Packs are something you'd have to pay for but you have access to everything else sound effects and music. In fact there are only three kinds of music on AudioMicro. Music sound effects and Loops Packs. You just don't have access to Loops Packs which is not too bad considering how much you would have to pay to get AudioMicro if you were not partnered with freedom. Another restriction comes from EpidemicSound. You have to use the music as background music. You can't upload the full track using a music video or a music visualizer. That is a restriction to the EpidemicSound catalog. For example, take a look at this channel. It has videos that are essentially music videos. Let me play a clip. This entire video is a music video that contains a full length EpidemicSound track. The music is in the foreground. That's not allowed. The music has to be in the background like you hear behind me as I'm speaking. That's an appropriate use of EpidemicSound. Just don't use the full tracks on stand alone videos. Now that restriction does not apply to position music or AudioMicro. You can use stand alone tracks in music visualizers or music videos for these libraries just not EpidemicSound. Do you think these restrictions are fair? Tell me in the comments below how you're using these music libraries. Send us links to your videos featuring music from position, from Epidemic, from AudioMicro. We would love to see them and help each other grow together as a family. Because that is the spirit of freedom. We all help each other grow together as a family. So click that I to partner with freedom if you haven't already so we can all grow together. Now let's read some of your deserving comments from the video talking about the latest feature, YouTube end screens. I created a video that showed you end screens for that showed you how the new end screens look. Are you using the new end screens available in end screen and annotations? The new tab it rolled out to everybody a few days ago. And if you skip ahead in this video you can see exactly how it looks. This is the end screen subscribe button and these are the end screen links to videos that you can position anywhere you like in the last 20 seconds of your video. YouTube also launched new analytics for tracking the performance of end screens that you can see right here in your shiny YouTube dashboard. And as you can see for us we have a big jump because we have only one day worth of data but you can see exactly who is clicking on your end screen links, what is the click-through rate and you should use this to optimize your end screens to get the exact right layout that's best for your audience. All of this works on mobile. That's the big difference between this and annotations which do not work on mobile. Are you using them? Send us links down there in the comments below to your examples and include the time code and t equals 34 seconds so it jumps right to your end screen now on to your comments. We had over 300 actually exactly 300 comments and a few of you said may not be very long but useful nonetheless that was twilight prince must-cash-full said hey george so can you reply because one person is telling me that you should do more uploading your videos and stuff that's somewhat that some of us want sometimes i would love to upload more i upload as much as i can but because we have this new 60 million dollar cash investment click that i in case you didn't know that that's keeping me very very busy i'd love to get to making two videos of the george show a day which is what many of you told me you want when time permits twilight prince also made a good point to add end screens to all your old content because even though it may not get many views you never know what might happen he wrote that on some of his older videos that get views now that they have end cards there's a slight chance the video can come back alive and dem keys even replied my man i didn't know that that's an excellent example of the freedom family helping each other grow together share your tips your best practices down there in the comments below so we can all learn from each other here is how i want you to link your end screens in the comments below do you see this and t equals 34 seconds that's what jumps to 34 seconds when i click this link in that video and if it's more than a minute you just use the m so like two m 34 s would jump to two minutes and 34 seconds and i imagine if your video is over an hour long then there's an h like two h two m 34 s and would jump to two hours two minutes and 34 seconds i don't have any two hour videos to test that on did i get that right tell me in the comments below dr dunke said this is really useful because phones also use it yes these end screens work on mobile phones which is definitely one of the biggest reasons to use them over annotations ibi also said glad to see people are picking up the trend and you're early again you beat me again again are you gonna beat ibi this time in this video let's see ashley silver dust wrote end screens yes finally this is an extremely useful feature i agree i'm hugo leandro said finally was waiting for this update like two or three months ago me too the tokin wrote finally a useful video from you i'm curious is this video useful as well of course i want to make useful videos i want to bore you and sometimes when i do updates i can imagine it's not interesting to you sorry about that but i try my best what topics do you think i should cover the tokin that would interest you even more tell me in the comments below i read every single comment so you can be sure that your suggestion will get read kimofai said they launch it right after i upload a video without an outro yeah they did that on purpose kasanova kurai wrote i love how your video cut off mid-sentence yeah i ran out of time i'll do a better job at the end of this video i actually wrote i felt like this was a presidential debate with a 20 second response timer time's up stop talking dark lord vids asked saw this can you save them like a custom template then you could add the same on all videos and ashley silver dust answered while you're in the end screen editor there's an import from video button click that and it'll let you choose from any other videos you've previously used end screens on this will apply the same elements just like a template as the chosen video to the current video hope this helps and who am i i am george sunpai papa george many other names you the beautiful freedom family has given me i am george co and founder of freedom or freedom and the freedom family and if you like this video click the like button down there and now for the end screen to get more george click that big f that will subscribe you to freedom central home of the george show click the freedom family plus one video to see i'm having a new baby number four and the 60 million dollar video to see the 60 million dollars coming to freedom to help you grow


Today I'd like to talk to you about the C word. It's a word that over 14 million people worldwide here annually and it's a cause of death of over 8 million people each year. The C word to which I'm referring to is cancer and it's a devastating disease and diagnosis for anyone. Now, cancer does not discriminate. It affects the young, the old, the rich, the poor, friends and families. You can clearly see this as I have in the waiting room of Moral Sloan Kettering in New York City. This is a hospital that's larger than a city block and is completely devoted to treating and caring cancer patients. Now, through my family's journey with cancer, we have learned that the disease will spread through cells that have shed from the primary tumor and enter into the bloodstream. Some cells unfortunately will bind to a distant organ thereby producing new tumors, new metastases and complicating the disease for the patient. Given this, I've often wondered, why can't there be a simple blood test that can detect these rogue cells in a blood of a patient? And could this be used as a liquid biopsy, perhaps replacing or supporting invasive biopsies that are traditionally used today? Now, given this, unfortunately, this is a needle in a haystack problem where there are only 1 to 10 cells in a single draw blood that already contains billions of other cells. If we were successful, however, we would be able to monitor cancer patients' cancer more rapidly, and in addition, we would also be able to molecularly characterize these cells. Now, this may seem like an impossible task, but in fact it really isn't. We just need to remember that it wasn't too long ago where we thought that sequencing the human genome was impossible, and yet today we're able to sequence anyone's genome for less than $1,000. So by pushing the boundaries, we're able to achieve the impossible. What are some of the approaches that one can use to isolate these cancer cells? Well, simple filtration is one idea because these cells are much larger than red blood cells and larger than the majority of white blood cells. Unfortunately, we found that these cells stick to the filters. They also get damaged during the process, and purity is an issue. Another possibility is to use antibodies to attach themselves and identify specific surface markers on the surface of cells as you've heard from Amy. Now, this is great, but unfortunately, some of these cells do get changed during the process, and you do lose some of the cells during preparation. More importantly, just as our fingerprints are all unique to ourselves, so too are our cancers. As well, down to the single cell level in tumors, those cells are also unique, and they might not share the same surface markers, thereby complicating our challenge further. At UC Berkeley, we're creating microfluid platforms that can take the blood of a patient, isolate the cancer cells, and detect specific surface markers and identify them. The way we do this is we use basic physics principles. The very first principle that we use is inertial force. This is the force that you feel as you go around the roller coaster at very high speeds. It's the force that keeps you in your seat. It's the force that keeps the roller coaster on that track. If we didn't have inertial force, the consequences would be obviously very devastating. Now, at the same time, at the microfluid range, if we employ inertial forces, we can actually use it to separate cells based upon size. We can separate out the red blood cells, platelets, and we can also separate the white blood cells and the cancer cells themselves. The cells naturally just separate from one another. They're not damaged during the process, and we can isolate them and further examine them downstream. Now, the way we do that is we use velocity. What we do is we have a microfluid channel. It's coated with all different antibodies, and the surface markers on the surface of the cell, some of them may recognize a particular antibody and will slow down. Others will just pass through very quickly, and this way we're able to identify these markers. Our entire microfluid platform is low-cost, and it also sits on top of a PC board that's very compact, and that allows a doctor to use this technology in his office. This means that a patient wouldn't have to travel very far to a distant hospital to a lab in order to monitor his or her cancer. Of course, with any new technology, there are barriers to adoption. These barriers can include costs, and they could also include regulatory issues. I would argue that another barrier is actually the medical community itself, doctors who are very conservative and not open to using new breakthrough technologies or try them out. Now, there is reason. Obviously, with any new technology, you need to extensively prove it and to test it and validate it. But even so, I would argue that we still need a community of people to actually light this fire of adoption of breakthrough technologies within the medical community. Now, in conclusion, I'd like to say that every cancer patient is different, yet they all share, and I think we as a community all share the same desire. We have a technology that can allow us to monitor cancer more quickly and often, and furthermore, we'd like to have a technology that can actually, in the end, detect these unique differences in cells. I finally posed these two questions. What are the pieces of the puzzle that will enable a faster adoption of the cancer? And in addition, who are the players that will help us encourage the adoption of new technologies by the medical community? Thank you.


I know you have very interesting views on it, greed, greed, I'm talking of in the corporate context and we'll explore many hues as we go along of greed, but I'm sure, you know, we would all... very curious to hear your views. If you use the word greed, what it means is you're operating market economy with communism going around in your head.  If your market economy don't talk about greed, talk about better loss. If somebody is making a billion dollars per day, let's tax him. Nine hundred million, you can tax him and that will be distributed anyway. But enough to give him incentive to go on, not to take away his spirit. So this is how capitalism is managed. Here you don't talk about this whole corporate social responsibility, I think it's funny. It is just an eye wash. Instead of that, the government, if the administration, if it has the necessary sense, you can bring the right kind of policies where businesses can grow unbridled, but it will naturally benefit the rest of the society. This can be done with simple laws, taxation and other kinds of laws, business laws can do this. There's no need to talk about service, there's no need to talk about greed. Everybody can live with dignity because if you are the receiving end of the service, you lose your dignity. If you are at the giving end of the service, you go to heaven, both are wrong. What I hear Sadhguru saying is profit is a good motive. And if you call it greed, that greed is good. And I think that is something that, you know, profit is something that... If you look at the definition of greed, what is greed? See, for somebody who is walking to his workplace or his home, somebody who rides a bicycle and refuses to take him on the bicycle is greedy man. Somebody who is riding a bicycle sees a man driving a Maruti car and he thinks he's greedy. Just single man riding a Maruti car, he could take me, his family and my family in the Maruti car because Indians are able to squeeze, you know. A man who's driving a Maruti car, when a Mercedes or a BMW whizzes past him, he says vulgarity of greed. A man who's driving BMW looks at a man who's driving a Bugatti and he thinks that is greed. So who is to set this definition for what is greed? I'm saying once you have chosen market economy, there is no room for that vocabulary in your life. You understand, greed is the driving force. Now, it is for the loss to manage the greed so that your greed benefits everybody else. There is a... in African lore, there is a saying, when the lion feeds, everybody else gets fed.  When you feed, it is the loss which should make it in such a way, if you make profit, everybody should live well. Instead of talking about greed and service and corporate social responsibility, I don't know why they're deviating themselves into all this... So stay focused on maximizing profitability, tax appropriately and let everybody grow.