Hi, my name is Ivan and I'm director of engineering in ring central. So I've worked in different roles in an essential I work as software developers or for architect team lead Product manager like every everything can manageable and right now I'm working in ring central and advocating can implement in github's processes across the whole company So it will be my pleasure to share with you our experience how we approach github's in the large enterprise company thousands upon thousands of users Huge work laws and what challenges we faced Excellent, and my name is Toma. Oh Nakahara. I run the developer experience team at we've works the company that created flux and flagger And coined the term github's so we're really excited to share our stories here for you to learn from So please take a picture of this QR code It leads to a site where we have all the flux things for this week whether it's our flux booth and all these talks And Activities that we have and for those of you watching the recording We'll still have the site up so that when the recordings are up. Hopefully you can see it It also has ways for you to follow up with questions. So please do chat with us I'll show this at the end as well if you miss it, but thanks for taking the picture So ring central will talk about what they do and what their drivers are and they're kind of in four key buckets Security and compliance scale velocity and reliability and we'll share how flux and flagger help with all of these in really important ways So first of all security and compliance so Ring central is a company that absolutely needs security and compliance and get ups helps with that First of all, if you don't know they have multiple products. There's Telephone and messaging and video I'm sure many of you have worked at companies where you actually have those physical phones As well as other capabilities. So imagine it's a company that has a chief compliance officer It has auditors both inside the company and outside and so auditability and observability are absolutely required Needs that they have they both out. They also have legacy and modern systems which brings with it dependencies and Requirements and configs and secrets management stuff which brings certain levels of complexity that they need to take care of and Similarly because of that they have multiple solutions and products that bring those same challenges so that means they absolutely need to have a reliable get-ups system and Tools like flux help with that and to be able to give them the security compliance Auditability and policy management that they need that will cover in this talk Especially when you have human beings involved It's just not gonna everything that you see on the left here The things that they need to do is not going to happen if they're chasing after human beings are like who made what change Under what context when you know all that is too manual and that would not be able to be possible So flux helps with that So quickly if has anybody not used flux yet Right, so great if you're new to flux will kind of segue and share little bits about it So flux is a project in the CNCF. We're incubating and actually right after cube con will be finishing the final steps We hope to get to graduation. So it's been a wonderful journey here in the CNCF and flux Like I said, it comes from we works and the company that coined get-ups so it's been providing get-ups both for apps and infrastructure and One thing that it does is it listens to the repo and if there's a change in that repo as a single source of truth It talks with kubernetes and implements that change So it kind of created that that idea of having that single source of truth in git And because you already have a version controlling system that means you already have an audit trail That's part of that. So there we go about the need for audit ability And also provides added security because you're not exposing your API to the outside world And you're also not Putting your secrets in CI and if people have been following the news, you know More more hacking is starting to move in the CI space because they're seeing vulnerabilities there and being able to access a lot of data So flux is already, you know steps ahead of this and always making sure that we're evolving so that we meet those security needs That includes supporting OCI the open container initiative and cosine, which means that you have added capabilities for verification Okay Yeah, and like Tomas said we will start firstly with security in mind because if you're working in a large enterprise and if you're a large enterprise when you're trying to implement some delivery tools or development tools which tightly coupled with your production system I think the first question everyone answers is What you will do with compliance and security and I would not talk only about just the compliance I will talk about actual security because ring central is a As you can see from this map every country market in orange We actively operating in you can buy all services and most likely every one of you in your life encountered or dial it to the phone line or contact center line which Operated or hosted by the ring central So we like biggest telephony company in the United States at least on unified communication as a market space So when we Yeah, so when we're trying to tackle such a diverse Geography and tackle different types of compliance. So we are creating government and health care So it's a hippo compliance different sock compliance. We are public company. So you can imagine it's Like audit upon audit and we have it constantly during our years. We also Always try to target it by the hackers we have different types of attempted breaks in our system and we won't make sure all our clients are safe and Every data is protected. So the first first and most important question is a security Not the scale not any other ones. But if you fail at security, you basically do not need to use this to or If you cannot answer this question so When we talk before we talk about the security itself and how flagrant flags approach it I want just to segue into one particular Important part we use and we use it. I think it's very tied to our success of using githubs in the company And we use it everywhere. It's a dramatic build So using if anyone views familiar with build systems such as Basel or Nick sauce or just a term of hermetic build It means regardless of the environment regardless of the computer you run your build on or build your container on Daytime Different libraries you get beat to beat exact copy of the container if you build from the same source So what does it mean the same source source is a source of truth? It's our source code. So if we have everything which needs to be so there's no dynamic variables dynamic dependencies Everything can spin down and if we have it everything pinned down and inside the repository We can build a dramatic container And we can argue it's Basically, we can build on ten different machines It should be beat to beat exact replica if something goes wrong and the replica is not exact It means the machine could be compromised as a process is compromised and As you can and it's a development practice yet It's not have nothing to do with githubs yet But you can see we how we go from here to githubs because in our development life Git repository is a source of truth as this is source of our source code in all the dependencies So we wanted to do the same in our infrastructure So how we can approach it in our infrastructure and here is DevOps or githubs Helps us and answered question. It's a little bit complicated image. I think we straight slides will share I will not go detail it through it but We definitely divide our domain into runtime and think configuration domain and configuration means everything which sits in gith so we Apply it everywhere we use githubs and it's very important. We use gith centric approach or configuration as a code infrastructure as a code as I would say it in the keynote's presentation we're trying to See what everything we do can be outlined as a YAML file or terraform configuration file Or any other file inside the git repository which can we act upon and act in a mutable manner same as for example with our Distralis and our grammatic images So going from here we can definitely see From the security point of view. We also while we're using flux We implement in different security practices such as static scan for the CV vulnerabilities as CV lists So if we have every dependency pinned down, we can easily do the reverse scan which helps the security a lot So what does it mean? It means if you have zero days as vulnerability in your in your system You can and you know every desired in your desired state of our cluster and the desired state diverges It's an incident. So we manage and with basically Always monitor the version of the state But if we have the desired state in the repository we can Go and see on the release of zero-day vulnerability We know exactly whether this vulnerability is existing thousands of clusters So we do not need to time to scan because if we scan every system for vulnerability After we receive the CV vulnerability it can take weeks in a large company such as ours So it definitely helps us to with detecting the CV vulnerabilities zero-day vulnerabilities We also employ flux integration with Mozilla sops So if you don't know it's really great tool to keep your secrets inside the git repository sounds counterintuitive But it's very nice thing very nice Solution to keep things atomic so it changes. I'm not distributed. I'm on different repository like a secret wall And for example your git repository. It's all in one place in atomic state and a git commit. So we do not wait with Slowly Adhere to the practice of storing everything everything can get it It's helped us and helped security compliance a security audit in our company It made it a lot easier all the trails all the changes in git Merge requests is it's change requests So if you for example enterprise, you know this change management system Etc so merge request is completely a maps one-to-one and we managed to convince all our Auditors such as KPMG price waters house coopers and different other auditors and our internal security teams what it really does Adhere to our existing security practices. It was a big win because we did not need to re-certificate So we do not need to do additional certification in terms of security and compliance here So yeah, this is how we approach the security Come on All right So hopefully company it's like yours that have regulations as such. That's something that's relatable I'm sure many of you have issues with scale So ring central's company as well as we talked about Their global company, but they so they have millions of users hundreds of thousands of businesses 200 Kubernetes clusters and growing And thousands of deployments and workloads. I think tens of thousands perhaps and as we showed on the map multiple regions around the world They've got hybrid clouds got their own data centers as well as GK and EKS and again As we mentioned multiple solutions as well as different development practices within house So there's a lot that needs to get managed and internal customers that need to be happy at scale So how does flux help with scale? so the flux project is actually a Microservices architecture and then a set it's a set of controllers that do things like creation deletion and updates to both your App deployments and to changes that you make for your infrastructure The source controller is one of the key controllers and one of the other ways that it helps with Scalability is that it has caching features that kind of unblock any issues that happen And it's important to mention here that our controllers support natively both customize and helm So that means like if you've seen the story in the CNCF the Department of Defense Absolutely wanted to use flux because they needed to use their entire helm ecosystem and flux Allows that to happen because it has native support And you have many get-offs options as you hopefully heard in the keynotes You know, there's the get-offs principles by the get-offs working group Flux completely complies with them and within those options you have the version that we talked about where you have Get repo that has a single source of truth But also with OCI now you have the availability to have flux consume artifacts OCI artifacts and to implement get-offs in that way So some of our companies enterprise companies who I can't name who Major major scale issues, you know, they're at the forefront and their major uses of flux So we've made sure that we've met those needs so they can not have Scale issues that come with git itself. So git is still there. It's just moved to the left so that we can meet their scaling needs So wherever you are in your journey just starting or wherever, you know that you can start with a project That's already built for scale So a little bit about your repo structure Mono repos are one methodology that we Kind of recommend and I know that Eva is very passionate about Mono repos Yeah, so the second need is Scale so if you figure it out your compliance it in the security need you may need to make sure you can run it at a large scale So meaning meaning yes, you can definitely try and many projects starts like this as a toy project inside the company But usually it does not make much sense to have many different processes Many different ways to deploy and manage your infrastructure in a large enterprise You want basically to make sure the things you build the things you're trying to adopt it can be adopted in the entire company and When we're talking about the large deployments and like Tomo said we have thousands of 10,000 of clusters different all around the world in Europe and Southeast Asia multiple clusters and multiple servers in the United States also we present on the cloud servers and a Google and an Amazon so it's a huge infrastructure and If we cannot answer answer how do we scale it infrastructure efficiently? We will most likely fail so it will not be adopted widely by the company It will be adopted by the some departments, but make me it will not work will not work So how do we approach it already on this slide? So we approach it on scaling on Mono repositories of what does it mean? So why Mono repository is so important so many of you may be already heard worked with Mono repositories They've just your software development for example Google has a really nice way of managing this Mono repository system Which we also use called Basel now for build process So we have quite extensive for history and working with Mono repositories and we also have history of Kubernetes So we started to use Kubernetes back in 2016 and quickly found out Two things first thing is ain't easy So it was quite complicated to manage multiple clusters and the second thing is we most likely it's really hard to manage large Kubernetes clusters and We started to think about how we can manage and create multiple smaller Kubernetes clusters and Dissect them on the networking level and trying to configure in a way Different types of workloads work on different types of Kubernetes clusters in the latest systems We also have huge lab environments, which have spun so pound like the 200 Workloads Kubernetes workloads already old number around head 500 Kubernetes cluster up and running right now and half of them in the lab and the staging environment And yeah, so you have multiple clusters So how do you manage this multiple clusters efficiently and here's a Mono Rapa? I think is a great answer to that because you could have in the single repository Information and configuration of all your clusters in the same place You can have multiple Mono repositories if you can distinct them by some common Think like for example in the central we have multiple product lines Which are more or less separate like video and phone a messaging Contact center and voice over IP in telephony solutions. So we have Mono repositories for issues of slides, but They still are huge product lines. So what we do we basically do very simple Thing and it's worked for us great So first of all one thing I want to mention try to keep it simple, especially in large enterprises So what we do we just using customized with flux and we have our main branch Acted as a as a basis So we keep our basis all obeys and I come on things in the main branch and we have multiple branches and Folders inside this repository for different environments which top builds on this top Main branch, but what it gives us it gives us ability to infer the state of all our infrastructure and Will it will come a lot more important later when we will talk about how we keep it reliable But Essentially as it's very simple distinction and flux Helps you to basically as long as you get repositories performant enough and you can pitch changes and you can fetch State of the git repository fast enough from the multiple clusters. You're good. We're using custom github deployment our self-hosted one enterprise solutions, so We have multiple caches and set up and making sure our git repository is always available and performant But yes, this is important part of it but if you again a large company you really want to have your git repository and your Workplace with every developer is relying on to be always available Performant and well behaved. So if you already have it, I think it's a great way to scale Flux and only any github the repository simply without any complicated solution to Basically multiple installation of multiple clusters Excellent. All right Third point third need velocity. I'm sure that's important for all of you So imagine at ring central, right? They've got various teams. They got to move quickly. They can't Have slowdowns from miscommunications or people not knowing what the desired state is and human error has come up a lot in our topics of Conversation so git ops with flux provides this ability to have a single pane of glass for the desired state And that really helps with these issues of miscommunication and that helps them really reach a state of Being able to implement dev ops in their company, which means a shared ownership and end-to-end ownership So their teams even talk about the own design creation maintenance testing to even support tickets and That means there's no separate delivery team. They're not detached from the production process So how does flux help with that flux has a lot of capabilities with observability and notifications on through Its own use and and integrations that our community has worked on I love the quote that even was thinking about that flux makes Kubernetes easier. So if you're You know having certain challenges will talk about how flux can help with that So one of the basic things that flux can do is it has events And so you can get notifications through the platforms that you use such as slack or discord or others that we support And we're also increasingly working across various integrations. So flux is moving into your IDE So if your app developers are doing deployments, they can do less context switching and they can just be within one place So we've started with visual studio code. How many people use visual studio code or people in there? Excellent. So you should definitely check out. It's our get ops extension Kingdoms here. Who's working on it? It's really fantastic. So you can literally Manage flux and do your deployments from within an IDE and not have to context switch If you like you eyes, there are options There's a free and open source. We've get ops UI that we have out there You can just check it out and I think it's really important to mention here that you can also use tools like arc Kubernetes EKS anywhere and D2 IQs platform and so Microsoft Amazon and D2 IQ have chosen flux and they trust flux as The tool that delivers get ops to their end customers So if you're already using those clouds, then you have those options as well So that the cure code will have links for you to check them out So I think that's it Okay, so I'll try to keep it very short because we have not much time and I want to leave it for then so Very important flight site against sigway into the velocity is a DevOps. So as it develops So I'm very heavy advocated DevOps and DevOps not I think it's misunderstood term in many terms I especially in many ways especially in enterprise because right now, but not right now But before we had so cold in so cold. We still have DevOps engineers. So the people who Try to build infrastructure and try to support the infrastructure for our development teams and we have around 3,000 developers since 3,000 engineers and about half of them was on the supportive role so it was not not not way were not software developers and Like big bunch of them was developed the ops engineers It outlines a big problem because if you need so many engineers specific engineers to maintain you infrastructure something is wrong and For me DevOps was always been a shared responsibility within the team and shared responsibility for the product So the DevOps team as a team who owns the product who makes the changes to this product for maintain it and deploy it So when we're talking about github and the flux and all other solutions We bringing this infrastructure closer to the development So we were able to simplify infrastructure management We were able to make it in in a way It's easy for our developers to work it because it's a source code it git repair monorepositories all the familiar way and It makes So basically I think it make the DevOps it like github's and flux the main DevOps What it's meant to be and then what's it evolved right now, which is like kind of some kind of abomination from my point of view so And it of course it increased sorry about it increased our velocity Sometimes ten falls when we use like other practices when you have this waterfall process or different agile processes and Pipelines so we do not have any pipelines in the company right now when we use github's and it made our delivery Like with 10 times or even 20 times faster. So it's it's a great tool and DevOps is a key Practice to utilize when you're working with github. So you always should think about Awesome. All right, and the fourth need reliability, which I'm sure is important for all of you as well So here again, we'll talk a little bit about flagger, which is a sub project of flux And this is really important at ring central because as we talked about millions of users hundreds of thousands of businesses These are business critical needs. I mean their telephones have to work, right? So these are things where they need reliability Built into the infrastructure They also need deployment processes that are verifiable testable and safe And we'll talk about how exciting it is that they can create policy-led environments for your developers to Be able to do the things that they need and know that they have the guard rolls up And so the human error won't come into play So flagger the sub project that provides progressive delivery Which I really love about this story. It helps users if all of you are kind of on your get-ups journey We're aware that there's a lot of changes that you have to make or your teams have to make even in the way that you're thinking Like things can feel new and unsafe So flagger really helps with that and it's really great to hear people out in the wild feeling that safety net for errors because Flagger provides so this term is fairly new progressive delivery. It's an umbrella term for things like canary deployments blue green AV testing etc And what it does is it decouples the app deployment process from the release process so basically if you have a service mesh or ingress controller flagger can use that to Decide to route traffic from your first source and then slowly dole it out to the new one based on metrics of success or failure and Roll back if it doesn't meet that threshold So basically flux will notice the change and start the process the get-ups process But before Kubernetes can take over flagger will come in and say okay the user set thresholds for me I'm going to take metrics for example from Prometheus or datadog or whatever and based on those metrics I'll keep checking, you know every 10 seconds, you know I'll route a little bit of traffic are there errors. Am I within the threshold? Okay, I'll route a little bit more and a little bit more until you've completely moved over or if it says no I haven't met the threshold it will roll back And so what's really great about this as Stefan has said is like you can deploy on a Friday and know that if anything goes wrong You know, it's not going to Have you be on your pagers on the Saturday? So you'll talk about how that helps with you. Yeah, so very important thing about the flux So if you're just using flux and just using get-ups. It's very important to understand It's a very powerful tool and if you approach it blindly It can make things worse because you can make a change we can make changes which bring your whole production down especially if you use monorepus, which is counterintuitive, but still so What you want to do is to build a Like set on the forcement of rules which you enforce upon the developers and in enforce upon the process in a way We cannot avoid it. So what I'm talking about. So I have I had not much time But I will bring one simple example of enforcement is a high availability and chaos engineering So when you build your system, you have this non-functional requirement of high availability Every like everyone needs to be high available cool But if you just write it a non-functional requirement and do not test it and do not run it an environment Which is inherently unstable you will miss it You'll most likely it will be just high availability on paper which will not work when Something keeps the fun. So what you do you implement chaos engineering.  So you run house engineering from production Sounds crazy sometimes what it is You make it inherently unstable and developers have no way to avoid this non-functional requirements the same way it goes with everything else and when we build our systems and when we're trying to Like basically make it universal across a large enterprise We want to have this enforcement rule these best practices to be shared and to be enforced it upon everyone. So this is I Think it's the most important part of success if you do not do it. You most likely will either break Your management will stop it. Something goes wrong People will not understand that we'll have too many ways to do stuff and it will get a break So again, we as I said we have runtime and configuration domain for configuration domain We run GitLab State like it's not pipelines just a single verification with GitLab runner But we have custom because we have more than repository like I said before we can run static analysis on this money repository We can detect drifts in Configuration for example if you have different FQDNs with different services which aims each other You can statically check and analyze what the QDN is the correct one Or it leads to some another resource which will will not exist in the class You can run policy management and policy enforcement using cavernet for specific resources exist And we also have runtime so configuration is not enough because during runtime things can have go wrong people make mistakes Yes developers may especially makes a lot of mistakes So we use flagger for that purposeful but with question asked about red green blue We have canary deployment. We have progressive deployment different types of deployments by using flagger Which flagger support and we tied it with test cube So it's run test cube checks in all our environments in lab and stage and production is very important And it makes sure what when we deploy stuff and the developers deploy stuff We are not afraid so we build trust into the system and when you build trust into the system when we started basically Trusting what we can really deploy on a Friday We start to deploy stuff on a Friday because we are not afraid to break stuff and then and I last finger in central Have five nines SLA for availability. It's a six minute lesson six minutes of downtime per year So when I'm talking about the stuff, I mean like we're really really serious about it So, thank you so much. I think we are Yes, no problem. So just a summary. So security compliance scale velocity and reliability Again, if you didn't take this QR code, we've got plenty of talks throughout the week. So We also have the slack channel. So we'll be outside if anybody has follow-up questions Or if you want to follow up questions in slack, please do that We also have our email address And you can email us if you have questions So, thank you so much, and I hope you have success with flux and flagger Yeah, you can ask questions me directly later. I valuable I whole day. I will be holding Thank you


Welcome back to 353. So today virtual machines. We actually know enough to kind of explain what was going on when we installed our virtual machine back all the way at the beginning of the course. So what do virtual machines do? Well they abstract the entire machine. So the goal with virtual machines is to run multiple operating systems on the same machine and kind of like processes think they're the only thing using CPU and memory. Virtual machines believe they're the only thing using the hardware, using the entire computer. So each OS believes they're the only thing running. We can't see any other virtual machine that is running. So some terminology. So there is the host and the host is the thing that has direct control over the hardware. So sometimes it'll be referred to as the hypervisor or virtual machine manager. They both mean the same thing. And that is the piece of software that controls the virtual machine. So they control when we create virtual machines, how to manage them, scheduling them, which should run, isolation. So which hardware they have access to. And kind of like we have the kernel that manages our processes. Well the hypervisor is the thing that manages virtual machines. So there are two kinds of hypervisors that have to do with how they are implemented. There are type one and type two. A type one is called a bare metal hypervisor runs directly on the host hardware and it requires special hardware support because the virtual machine itself while the kernel is actually going to run in kernel mode, which is if we recall that special CPU mode that lets you access instructions that can directly talk to the hardware. So type one, the virtualized operating system is actually running physically in kernel mode. So there needs to be some special hardware support that's more privileged than kernel mode that can go ahead and make sure and monitor our virtual machines. Type two is called a hosted hypervisor and it just simulates a hypervisor. It runs as a normal application, runs as a normal process. So it has to kind of create the illusion that your kernel is actually running in kernel mode even though in type two your virtual machine is just running as a normal process. So it's actually 100% running in user mode. So it's going to be a bit slower. We'll go over how you would actually implement that today but going to be slower but you don't have to have any special hardware. Even though nowadays you probably have special hardware that makes us go way faster. And another terminology, a guest, a guest is just another name for the virtual machine. It sees a copy of a virtual copy of the host which may or may not be complete. So it might not have access to all the same hardware or it might just make up fake hardware in order that you can monitor it or something like that. So here's review over on the left hand side so that this is the world we currently know where we have the hardware and then above that the software that is running and controlling that hardware is called the kernel and one example of it is the Linux kernel and we went over, I guess this is tearing into somewhat review. There's different kernel architectures, right? There's micro kernels, there's monolithic kernels which is what Linux is which change what code is actually running in kernel mode which is that privileged mode that can directly access the hardware. And then no matter what curl design you have you're going to have this programming interface between the kernel and normal user mode processes. All right, review time, that programming interface, what's it called? To go from kernel mode to user mode. Anyone remember the name? Uh-oh. Yeah, syscalls, yeah system call interface, whatever you want to call it. So that's this layer that's that programming interface if we want to go ahead and monitor it that application or that process program sorry that makes it easy to monitor things, right? strace, we can go ahead see all the system calls everything makes and figure out why JavaScript is actually so bad. So that's the world we kind of know right now, right? We have processes they use the system call interface they might use libraries so they might use libc, they might use I don't know if you use graphic stuff they might use other libraries like that and then we have the system call interface that talks to the kernel, kernel talks to the hardware. So if we have uh if we have virtual machines we have the picture on the right. So what controls the hardware is the virtual machine manager or the hardware or sorry or the hypervisor so that what that's what has direct control over the hardware and that manages the virtual machine so in this case we have three virtual machines one could be running Linux, one could be running Windows, one could be running macOS they all just see a virtualized copy of that hardware and then each virtual machine is running its own kernel again like I said they could be completely different and then each kernel is just managing their own processes so this part of the software stack from the kernel upwards is going to be exactly the same um and it's independent we can go ahead we can run as many virtual machines as we want on the same hardware so any questions about this so some and this is good you know makes us protected because if we wanted to we could I don't know install anyone know the official offering system of North Korea yeah red star os yeah so that is the official offering system of North Korea I would not recommend installing it on your laptop but if you wanted to see what is going on in North Korea you could go ahead you could create a virtual machine probably a good idea to not give that virtual machine internet um probably a very good idea you can install red star os on it and you can see what's going on in North Korea and that's fun if you wanted to hopefully your virtual machine your hypervisor implementation doesn't have any bugs in it so you can't actually access the hardware but if you wanted to you could install virtual machine not give it internet access try and download as many viruses as you can put it all on there and let the viruses duke it out and see who wins so you could go ahead you could do all sorts of fun stuff without ruining your hardware and ruining you know your actual money and you can go ahead and you can have fun with it so virtual machines typically not emulation that's kind of a different idea so emulation is typically used to translate one isa or instruction set architecture to another and for review I guess you did take an assembly course so an isa is just the machine code that the actual physical cpu understands so for instance you might have to do some emulation if there is a mismatch between what your offering system is using and what your machine physically is so you might have to do something like x86 to arm or something like that so that's why in lab zero if you have the newer mac laptops that run arm that's why you had to download a different virtual machine or sorry a different uh a different installation of debian than someone using you know a windows machine because there are different instructions so dark detectors and we don't want to have to waste time translating one to the other when you're trying to run your virtual machine so generally you want it so that the guest operating system is executing the instructions directly using the same isa otherwise you have to translate the instructions and it's typically pretty slow unless you have uh apple laptops so they have like special hardware to convert arm to x86 but otherwise typically slow and you want to avoid it at all costs it's okay for some uses like if you try and write an emulator for a nintendo entertainment system or nes you can go ahead do that because it's an old eight bit cpu really easy to do and it's a fun summer project you can do while you're trying to avoid working on your thesis not that i would know or anything like that um so could go ahead and do that um virtual machine could use emulation to run a virtual machine for a different architecture but generally the performance suffers and you want to avoid it if at all possible all right other cool things virtual machines can do is they can enable pause and play so much like the kernel can go ahead stop a process from running and then you know run another process and run that for a while then switch go back to the original process pause it switch go back to the original process hypervisor can do the same thing but instead of pausing a process it pauses the complete operating system so the hypervisor would have to context switch between virtual machines much like the kernel has to context switch between processes same idea it just it'll save the current state so it can restore it later but you might imagine a whole machine probably has more state than just a single process so there's some more involved things here but the idea is exactly the same just save the current state restore it later run another virtual machine then we would have scheduling between virtual machines go ahead run another one later and the cool thing around about this is we could actually move it around as well so we could pause it and then physically move it to another machine if we really wanted to and then unpause it on a new machine so you can go ahead one second your virtual machines running on machine a then whenever and then without you noticing it's now running on machine b and your switch machines and you have no idea about it it's just a seamless process and that's why like all cloud providers will probably just give you a virtual machine because they'll physically move it around servers you know maybe they bring one physical server down for maintenance maybe one gets overloaded because someone decided to I don't know run something super intensive on it so they want to move your virtual machine to an unloaded machine and try and like load balance things across their vast data center so other things they give they give you some protection through isolation so like processes are isolated from each other again you this is pretty much the same idea as processes but just applied to the entire machine so guests are isolated from each other just like processes are isolated from each other and hopefully isolated from the host and because of this the hypervisor is the thing with ultimate control over the hardware so it can set limits on cpu time so if one I don't know fork bombs itself or does something stupid you can go ahead you can limit it and say hey virtual machine you can only use a third of the physical cpu or just set a time limit or whatever you want to do you could set limits on memory network bandwidth so or no network at all etc you can limit how much hard drive space it gets you can limit whatever you want on it because ultimately you are the thing controlling access to the hardware and the guest even if you install red star red star linux on it then doesn't matter just has access to its own virtualized hardware might ruin its hard drive or whatever but you can go go ahead just delete it after the factor do whatever and if you actually were managing a whole bunch of machines and something screwed up hopefully you have backups and then you can just roll it back to a previous state or just remove it completely and just get rid of it so yeah yeah so the question is how is this guest different than the guest user on your laptop so the guest user on your laptop is typically the user for that operating system and typically it's just another user account that doesn't have access to like your account's data or anything like that so it's no no no it'd just be a different user that can run processes and generally for like they're all running in that same operating system generally the only difference is like what files you can access okay and that's pretty much it so like if I use your computer as a guest user I can't like access your documents folder or anything like that yeah yeah so guest in this context is like guest operating systems like virtual machines that running on it so for you uh your Debian virtual machine that would be your guest and your host would be whatever you're using mac or windows or whatever you're using yeah and all right so other things virtual machines can help you with is consolidation so data centers or even when you're like developing uh web applications or whatever there's a lots and lots and lots and lots of servers running often not making use of all the hardware so sometimes services with different purposes could be like using the same hardware or sharing the same hardware so maybe I don't know I have a really powerful computer but my application just one application uses a lot of cpu and one application uses a lot of memory well oh that is a mouse okay great down there all right cool we have a mouse in here they should they should probably do something about that who the hell they go all right that's not distracting it whoops I don't know where it went that's not good all right well people I thought it was a leaf or something before but turns out to be an actual physical mouse like the animal kind not like our kind huh all right that's not distracting at all all right sorry about that um what was I talking about oh yeah so you could go ahead you could virtualize both those application one that uses a lot of cpu one that uses a lot of memory and they can both run on the same physical machine share hardware and do everything like that so instead of just having one system that's not using a lot of memory and one that's not using a lot of cpu make them both virtual machines put both virtual machines on it and then suddenly way better hardware utilization which is like the main concern of data centers and like google will pay you a lot of money if you work on this to like bring down or increase their server utilization by like one percent since they spend billions and billions of dollars of it turns out to actually be lots of money so when we're actually implementing virtual machines a virtual cpu is like the key abstraction so like for processes well like we use the process control block that kind of acted as a virtual cpu it doesn't virtualize all parts of the cpu just enough for user mode so like saved all of our registers and then we are page tables our open page or open file tables things like that the virtual cpu structure tries to store the entire state of the cpu so like what mode is the cpu currently in um anything that has to do with the hardware any special flags it has to set like any special things you normally can't access in user mode it has to save because we're trying to emulate the intact well not emulate we're trying to go ahead and abstract the entire cpu not just enough for a user process so when the virtual machine resumes like the pcb or low data and resume just like we had for processes so the guest in this case basically a virtual machine still uses user mode and kernel mode so like there's no changes to the guest operating system it's still the linux kernel in case of the devian operating system so you could install it directly on your hardware or you could run it in the virtual machine but it is the exact same code so if we are on x86 we have like the your user code would be if we remember the stupid numbers user mode code is running in something called ring three kernel mode is in ring zero which is the most privileged and then if you actually look and see if your cpu has hypervisor support so it can support like a type one virtual machine the the special instructions it has access to are officially in ring negative one so you can tell that that did not exist when they first made up that cpu because ring negative one doesn't really make a lot of sense but kind of does because it's supposed to be more privileged than more privileged than ring zero so when they go ahead and they designed it initially the virtual machines were not a thing because x86 dates back to like 1978 or something like that so that it has to support everything has to stay the same same numbers same everything so whenever they created virtual machines and hardware support for it i guess around the time you were born then they created a ring negative one so that is where all of the hypervisor instructions are if you have a type two hypervisor well then the host has to create like a virtual virtual kernel and user mode and try to emulate what is going on and just try and simulate what's actually going on so if you had to implement a type two hypervisor where the guest is running on the host in user mode at all time if you try to run any actual privilege instruction that can only run in kernel mode then it's going to generate a trap which would say wrong mode or illegal instruction or you don't have access to run this instruction so what happens is whenever you whenever your virtual machine tries to execute one of those instructions your hypervisor which is just the program that's kind of monitoring your virtual machine should go ahead recognize that hey you tried to run an instruction which you don't have access to so it should go ahead and write essentially a signal handler and then simulate the operation that actually is happening on the guest and then resume it so it will significantly slow things down because it's trying to execute just whatever your instructions are but if it turns out to execute an instruction that is only available in kernel mode you will go ahead catch the error and try and simulate that kernel mode instruction and you would have to like simulate the entire rest of the cpu so you'd have to look at all the documentation and everything like that so it's a fairly boring job so how it would look visually is well this is all running as a user process we have our guest above this line so it has user processes and then privilege instructions which are running in kernel mode if it tries to actually execute one of the privilege instructions it would generate a trap or a signal or an interrupt whatever you want to call it and then our hypervisor code would have to emulate whatever that instruction is supposed to do update that virtual cpu that is supposed to represent the entire state of the cpu including whatever it's doing to the hardware and then we return back to the guest operating system continue running their code just like nothing happened and we just keep on doing this over and over again so it'll be really slow so instead of executing instruction directly we have to go ahead simulate it which will will be slow but hopefully you know most of the time our code generally isn't in kernel mode doing stuff it's actually running our user processes so may or may not actually work and not be that slow although sometimes doesn't always work on some cpus like especially x86 if there were clear instructions that were like this instruction only runs in kernel mode and this instruction only runs in user mode it would be easy but on x86 especially it is not clear some instructions do double duty because on x86 virtual machines did not exist when they first made up that instruction set in the 70 in like the 1970s which is old even for me um so that didn't exist it wasn't a consideration so they took some shortcuts to go ahead save space and make things easier to implement and make things do do two duties so for instance there is this instruction called pop f so it loads a like flag register from the values on the stack and which flags register it actually affects depends what mode you're currently in so if you are in user mode it updates the user flags and if the CPU is currently in kernel mode it goes ahead and updates all the kernel flags so the same instruction updates two different things and the only way you know which things it updates is what mode is it currently in so if we're trying to if we actually run this instruction and like we actually run this instruction on a type two hypervisor it will always update the user flags but if our guest operating system actually thinks it's in kernel mode and we run this instruction it will update the wrong set of flags and then your kernel will probably freak out because well you didn't actually update the kernel flags like it thought you were going to so this also wouldn't generate a trap you can't trap and emulate so instructions like pop f would be something considered call or something considered a special instruction and we need another approach to be able to simulate what happens with that so what they need is something called binary translation so if the virtual cpu is currently in user mode and i'm running user mode code in my guest well they both agree so they both think we are in user mode so i don't have to handle anything specially i can go ahead i can just execute it doesn't really matter if one if the guest thinks it is actually in kernel mode then the hypervisor is actually going to need to inspect every instruction before it executes so it will inspect every instruction before it executes if it's one of these special instructions well it needs to be translated to instructions with the same effect and we go ahead we have to simulate like we did before and update that virtual cpu so so the kernel it's just going to use like for instance we're going to be able to do this because the kernel well if we want to switch from user mode to kernel mode luckily that is an instruction that we can do trap and emulate so we can go ahead see when the guest switches from user mode to kernel mode and then when it switches from user mode to kernel mode we can go ahead we can update our virtual cpu to note that okay now this guest thinks we are actually in kernel mode i should turn on binary translation and i should just inspect every instruction before it executes to see if i need to simulate what's going on so overall the performance is going to suffer a lot because it's inspecting every single instruction but generally it works more or less okay um and performance you can go ahead you can actually run something like that with modern hardware so what that looks like looks like exactly the same thing but now instead of like actually just generating a trap the hypervisor or the virtual machine manager is going to go ahead inspect the special instructions if we believe we are in kernel mode and then it would translate it or simulate what it needs to do then update the virtual cpu that would have all of the hardware registers and then return keep on executing things and let them go and binary translation is spoiler alert also how valgrind works so valgrind will go ahead and inspect any instructions that use memory to check you know whether or not you have access something that is actually in bounds it would go ahead and translate all your calls to malloc so that it can go ahead monitor that monitor them and know what memory you actually have valid access to but if you go ahead and run valgrind or you know you actually care about performance valgrind is like 30 times slower or like 300 times slower i forget the number then actually executing your program so yeah there's a big performance penalty but generally computers are pretty fast now so for some tasks you can put up with it all right so that's for type two we use type one all the time so this is wait 2005 is this this is newer than you right you are born okay in the year of your birth they they decided to no not this close oh three okay so this is this is brand new all right so when you were one or two then they introduced virtualization as like a standard in cpus um intel introduced it as vtx amd did it in 2006 as amd-v but if you go ahead and you try and enable this if it's not enabled for some whatever reason in your bios they had different names for it so intel called it like vanderpool and then like publish it internally as virtual machine extensions amd their codename was pacifica they published it as secure virtual machine or svm so all of those names generally they all mean the same thing it just depends the internal term the marketing term the the official boring documentation term um they're not consistent about it whatsoever but they all mean the same thing they all mean they support that hypervisor mode or that ring negative one so all of them do they added the concept of hypervisor mode and then what usually happens nowadays is the host so whatever operating system is actually installed on your computer whatever you boot it up so you boot up windows then windows is going to claim the hypervisor and then windows will actually also serve as a hypervisor for your virtual machine and your virtual machines can go nice and fast so so it's the only one able to access it windows goes ahead it can set isolation for all your guest virtual machines what hardware can access and everything like that so technically windows would be your hypervisor as well as uh running in kernel mode although windows does like weird things now where it virtualizes parts of itself and um actually supported or actually needed for like game anti-cheat stuff oddly enough because that stuff will run in kernel mode and then windows needs ultimate security so windows needs to be the hypervisor so lots of fun stuff so we can talk a little bit briefly about like what the hypervisor has to do so it has to do some scheduling so if there's only like one CPU on the physical machine guest doesn't know it you could lie to the guest you could say I have a hundred CPUs like you could present as many virtual CPUs as you want for the guest and then in your hypervisor you can map the virtual CPUs to physical CPUs or schedule them like processes and like a normal kernel right there's also hypervisor threads so maybe your hyper your hypervisor would be running in the background and your hypervisor would be scheduling all of your virtual machines what to run and what CPU they go on to so one approach the hypervisor might take is like just assigning virtual machines to CPUs directly so if there's more physical cores on your machine then all the virtual machine or all the virtual machine virtual CPUs put together you can just map them one to one and then the host can just use any spare physical cores if you have to share that's where things get more complicated they call it over committing for some reason so at equal numbers you can still map one to one and because the hypervisor threads like the kernel don't really run that often but otherwise in general you would have to schedule them just like processes it would use all the same scheduling algorithms the only difference is we are context switching between virtual machines instead of processes so as you might imagine context switching between virtual machines probably takes a bit longer than context switching between processes so maybe I want to let them run for a little bit more before context switching but otherwise I mean everything is pretty much similar so some other problems that CPU over commitment could have is that like guest operating systems really run way too unpredictably for soft real-time tasks because you could context out a whole virtual machine even when the user process says not to so on your virtual machine if you really cared about something you could you know set yourself as a real-time process and try and have like the highest priority so that you're assured that you do not get context switched out so let's say the round robin time slice is like 10 milliseconds and the guest like if you ran it on physical hardware then you're guaranteed that you know if there's four other processes around you'll have to wait at most 30 milliseconds before you get to run again but if you are virtualized well that guarantee is now thrown out the window because even though that's true in the virtual machine that virtual machine might be context switched out it might not be running so it would miss deadlines it wouldn't have if it was actually running on the physical host and in this case virtualization fails a little bit because we can actually observe different behavior now we can no longer hit all of our deadlines but this is like a very special case for very very performance sensitive code so also as you might imagine with virtual machine or yeah with virtual machines virtual memory is a lot more complex too because virtual memory right each process thinks it has the entire address space but now the kernel also thinks it is managing the entire physical memory but the virtual machine well it doesn't have access to physical memory anymore so it has to virtualize it too so the hypervisor would have virtual memory for the guest operating system or the virtual machine and then the virtual machine has is managing virtual addresses and virtual memory for each individual process so we have like two so if you thought just multi-level page tables were wrong now we have multi multi-level page tables so we would have two levels of page tables and that would be great so technically you have like a nested page table so like I said the guest thinks it controls physical memory does its own page table management but the hypervisor actually controls physical memory in this case so it would have a page table as well that is valid mappings for that guest or that virtual machine so in order to translate a virtual address or to translate a virtual address on the virtual machine well it's going to have to translate that processes virtual address to what the kernel of the virtual machine thinks is a physical address and then the hypervisor is going to translate that physical address which isn't really a physical address it's a virtual address for that virtual machine and then go ahead and it would translate that to an actual physical address so we would have in this case kind of like a nested page table so that is one of the hardware support things that hypervisor mode actually does it allows you to have nested page tables and tell essentially tell the tell the mmu or memory management unit about this so you don't have to do two steps in order to translate an address it will go ahead you tell it about the nested page tables and it will actually use those nested page tables to translate an address for you directly all in one shot instead of having to do it like one step and then another step and that is generally where most of the speed up from hypervisor mode comes in is telling the guest or while telling the hardware about nested page tables so if and then you could imagine this gets even like way worse so if you have if you over commit on memory so you're using more memory than your virtual machine can actually use well maybe that virtual machine has like a bunch of swap space and then swap stuff to what it thinks is a physical disc but now the hypervisor could also do the same trick right it could go ahead and have its own swap space and lie about how much memory it has essentially so you might have two different page replacement algorithms one would be running on the hypervisor another would be running in the kernel on the virtual machine and you don't really it's really unpredictable to know which one runs for what and generally for this you don't do page replacement in the hypervisor you just let your guests go ahead and worry about it so you don't have this weird thing where there are two things trying to do page replacement other fun things we did copy on right right we did copy on right for processes to save mep to save memory so guests could also do the same trick because they could share pages if they are complete duplicates but unlike copy on right that while we start copy on right because we fork we know two things are supposed to be exactly the same we don't exactly know that for virtual machines so what they do is the hypervisor tries to detect if two virtual machines can actually share memory so it will actually try to detect duplicate pages between virtual machines and how it does that is basically they hash the contents of all the pages if two pages hash to the same value that means their content could be the same or but yeah it could be the same doesn't mean they have to be the same they could be so if their hashes are the same then it will go ahead and it will check it bite for bite to see if they're actually the same bite for bite if they're the same bite for bite we're going to head go ahead and share them so we would point both entries to the same physical page and then we would do the same copy on right as before we can share them as long as both are only reading to them as soon as one writes to them then we can go ahead we can do our copy on right thing again who other things the hypervisor can do is it can provide like virtualized IO devices so for like your network card or something like that it can multiplex one device to multiple virtual machines so for instance it could share your network card between all the virtual machines they all share it it could also emulate devices that don't exist so you could go ahead you could make a fake network card give it to Red Star OS and then monitor whatever it writes to it and whatever it's trying to do but as far as Red Star OS is concerned it actually has access to a network card and it can go ahead and try and do whatever it wants other things you could do you could just map a physical device to one virtual machine so that that virtual machine has exclusive access to the device but the hypervisor is still kind of in the middle and still doing some translation which might slow things down especially if I don't know you're on Linux and you're trying to pass a whole GPU to windows that can actually use it better so you can play your games or whatever you want to do so that is slow and there is a newer hardware solution to remove the hypervisor during runtime and that is called the IOMMU so what that does is let the hypervisor map the devices to virtual memory and then exclusively give that to the guest so that that virtual machine has exclusive control over the device and then whenever you install drivers on that virtual machine they actually are interacting directly with the hardware there's no hypervisor that gets in the way and it allows you to run things like GPUs at pretty much native speeds on your virtual machine so you can go ahead you can use Linux or whatever as your hypervisor install windows virtual machine give windows your GPU and then your GPU which is most of what you care about in terms of your performance runs at full speed you can run your games you can do whatever you want with it other things virtual machine well the only thing a virtual machine actually needs is like your computer all you need is an operating system on your hard drive so virtual machines work the same way they boot from a virtualized disk so that is what you set up when you set up your virtual machine the main thing that you set up was you installed Debian to a virtualized disk also called a disk image that has all the contents of a physical disk it would contain like partitions each partition would have a file system on it and in lab six well you're making a file system so it would look exactly like that usually that big disk image is one big file some formats let you split it up some let you do copy and write things but they're all the same idea the guest kernel just sees it as a normal disk that it has full control over and that has all you need for your virtual machine so it makes it real easy to move has your kernel on it all your programs all your settings all your files everything like that so if you want to go ahead and say your laptop well you got a new laptop or something like that you don't have to redo all the steps in lab zero and set up the virtual machine again you just take that virtual disk image and just do ye old copy paste onto the new machine and you can go ahead you can run it and resume your virtual machine on a new computer um yeah some some things will have you like download a file for the whole virtual machine but the majority of it is going to be the disk image and then settings for that particular software that tells you oh the default number of cpus it should have memory and like just settings like that so if you used at one point we used it if you used uh what's it called virtual box they're like ova files and it's basically a disk image and then just a whole bunch of settings all right other things virtual machine could be due that's more relevant to you is they can be used to isolate an application so let's go back to lecture three so if you use a dynamic library and we discussed that if we change the a bi at all even in subtle ways like changing uh the order of fields in the structure round then we could break our applications they could no longer work or change the behavior and they would suddenly now be wrong now so what you may want to do is like freeze your dependencies in production so that even if you update the operating system running on it you're sure that won't affect your application and you won't get any weird dependency errors so what most people more or less do is create a virtual machine for your application with all the libraries it needs and then you just go ahead and run that virtual machine in production so you and you might also use it to separate out your application so like if you were running a web server well maybe you you run i don't know the maybe you run the back end in a virtual machine the front end in another virtual machine in the database in another virtual machine and now they're all separated you can talk to them like they were just like actual machines through networking and then you also have the nice benefit of well if you're starting off small all the virtual machines could be running on the same physical computer and then when you grow a little bit and that machine's overloaded makes things really easy you can just buy another physical machine and then move a virtual machine over and you don't really have to change anything suddenly wow you don't have to reset up anything you don't have to reinstall anything you just move the overloaded part onto a new machine and you're off to the races and you could do this as many times as you want separated out give it give each virtual machine its own computer eventually and then when you really grow well then you might have to have multiple copies of the virtual machines and then you'd go ahead and try and like load balance between them and then suddenly and that would be like you're operating at the scale of google or amazon or something like that so have we heard of containers before docker or anything like that all right so that's kind of that idea right so there are things called containers so docker is an example of it so they aim to have that same idea of like using a virtual machine for your application with all of its libraries but if you're give like an application as a virtual machine that's kind of dumb because while your application's probably not that big your libraries probably aren't that big and then most of your virtual machine is the kernel so let's say again you had I don't know three different virtual machines and they're all running the same kernel so all that is the same between all of them and then the only thing that's different is the application and the libraries well it's essentially what docker aims to solve so like the hypervisor can set limits on cpu time memory network bandwidth and things like that so that's another reason you would use virtual machines but nowadays the linux kernel supports things like that directly without virtualization so linux has something called control groups so that's part of the kernel api or the system call interface that supports hypervisor like limits but for processes so you can isolate a process just like yeah you can isolate a process kind of like you can isolate a virtual machine except instead of each virtual machine having its own kernel all of the containers are sharing the same kernel so you can set other resources for namespace like mount points like what directories it can access what ipc it can access things like that so containers are basically just like a lighter weight virtual machine because they all share the normal kernel if you've used docker on mac or windows well how that works is because docker needs linux to operate because it's all built on cgroups windows or macOS will actually install a linux virtual machine and then all of the docker containers will use that one virtual machine that it is managing but under the hood it just is managing a virtual machine so virtual machines they virtualize a physical machine they allow multiple operating systems to share the same hardware provide isolation and the hypervisor which is the thing controlling the hardware can go ahead allocate resources to each individual virtual machine a type 2 hypervisor is implemented in user mode it's slower it has to do trap and emulate binary translation type 1 fast support by hardware ring negative one hypervisors might over commit on resources physically move a virtual machine but that is possible just kind of is a bit difficult and containers aim to have the benefits of virtual machines without the overhead aka they share the same kernel so with that just remember pulling for you we're all in this together


Hi everyone, this is Alice Gao. In this video, I will continue talking about the policy iteration algorithm. The main idea of the policy iteration algorithm is that we are going to alternate between two steps. The first step is that we'll take a policy and evaluate it to update the utility values. This is called the policy evaluation step. And the second step is that we'll take the updated utility values and derive a best policy given that. This is called the policy improvement step. In the previous video, I also showed you how to perform policy improvement on policy evaluation. Turns out policy improvement is straightforward. We already know how to calculate the best policy given some estimates of the utility values. But performing policy evaluation is a bit more complicated because it appears that we have to solve a system of equations which look very similar to the Bellman equations. But turns out this system of equations is much simpler to solve than the Bellman equations. Let's understand why. My main goal of this slide is to compare the equations for policy evaluation and the equations for the Bellman equations. So first of all, let me highlight the parts of these two equations that are the same. So we have the utility values, we have the reward for entering a state, the discount factor. So these are the parts of the equations that are the same. So the remaining part are what's different about these two. So what is the main difference between the two? Well, for policy evaluation, we are restricted to one particular policy. Because we're given the policy, so the policy specifies what do we need to do in each state. It specifies what action we should take in each state. That is why we have the pi of xs here, which is what is the action specified by the policy for this particular state. Given the action specified by the policy, we do not have to consider other actions. That is why in the policy evaluation step, you do not see a maximal over all possible actions. Whereas in the Bellman equations, you do see this maximum across all possible actions, because we don't know which action is the best. So mathematically in terms of the expressions, this is the main difference between the two. Now let's turn both of these into concrete equations that will help us to understand why solving the equations for policy evaluation is much easier than solving the Bellman equations. Let's write down both equations when the state we are considering is the starting state. So s is equal to s11. And let's assume that the policy pi says that we should go downward when we're in state s11. Now take some time, write down both equations yourself, and then keep watching for the answer. Here are the answers. For policy evaluation, I've written down the equation, whereas for the Bellman equations, I've done this in previous lectures, so I've just copy and pasted the equation here. So again, you can see the main difference. For policy evaluation, we are already fixing the policy that we're using, which means we're fixing the action we're taking. Since we know the action we're taking is going downward, so we only have to consider one of the four possible actions. So all we have is this one term for the action of going downward, which means with 80% chance we'll actually go down and reach s21, with 10% chance we'll go left or right, and going left will bump into the wall, and going right we will reach s12. So we have this one term without the maximization anywhere in it, whereas for Bellman equations, we do have the maximization over the four different actions, four terms for the four different actions. The last one is corresponding to the action of going downward. Now that I have carefully compared and contrasted these two cases, you should have enough information to figure out why policy evaluation is much easier to do, to perform than solving the Bellman equations. So take some time, think about this yourself. So think about why is it easier to solve the equations for policy evaluations than solving the Bellman equations. Then keep watching for the answer. Here's the answer. Policy evaluation is much easier to perform because the equations are linear. We don't have the maximization anywhere in the equations. Solving a system of linear equations, we have many standard linear algebra techniques for doing it, whereas the Bellman equations are not linear and there's no general algorithm, efficient algorithm we can use to solve a system of non-linear equations. This is why performing policy evaluation is much simpler than solving the Bellman equations. Now given this, we can solve the policy evaluation equations in two ways. Let's look at them. So first of all, we can perform policy evaluation exactly, solving these equations exactly using standard linear algebra techniques. I'm not going to go over details, this really belongs in another course on linear algebra or linear programming. But in general, if we have n states, then a standard technique will take n cube time to solve this system of equations. When our state space is small, when we don't have a lot of states, this is quite a reasonable time complexity. But when the state space is large, this might still be too much. A cubic time might still be too much, although it is polynomial. So in the case when the state space is large, we might still want to solve this approximately instead of exactly. If we're okay with solving policy evaluation approximately, then we can use something that's very similar to the idea of value iteration. Because after all, we have a set of equations and they look very much like the Bellman equations. So we can do something very similar to the Bellman updates that we did. So take these policy evaluation equations and convert them into some sort of iterative update rule. So same as before, we can take the equality and turn it into an update rule where we plug in the estimates of the v's on the right hand side. And then calculating the right hand side will give us the new estimates of the v values. So the idea here is that we can perform something very similar as value iteration. So take these update rules, update the utility estimates in using a few steps until it's somewhat converges. So the number of steps really depends on us. It depends on how accurate we want these estimates to be. So using this approximation method, we can derive reasonably good estimates of the utility values without spending a lot of time. This concludes our discussion of the policy iteration algorithm. Let me come back to slide six and summarize the key ideas. The key idea of policy iteration is that we are going to iteratively improve both the policy and the estimates of the utility values. We'll do this via two types of steps. One is policy evaluation where we go from a policy, use a current policy to estimate the utility values. And then after that we will do policy improvement where we use the updated utility values to derive a new best policy. The policy improvement step turns out to be straightforward as we've done similar things before. Now for the policy evaluation part, we need to solve a system of equations. But fortunately, this system of equations is linear. So there are two ways of solving this system of equations. We can solve it in cubic time using standard linear algebra techniques. That is if we want to solve it exactly. Or if we're okay with solving them approximately, then we can use an iterative approach very similar to the value iteration algorithm. So during a few iterations, we should be able to derive reasonably good estimates of the utility values. And we might be happy with that. Thank you very much for watching. In the next video, I will start talking about passive reinforcement learning algorithms. Bye for now.


How do you detect mental illness in a friend, spouse, relative or colleague at work? When these people stop being their usual selves, how do you identify that they are having a bad day or there is much more serious issue than it? When your friend or loved one begins to drink a lot, party a lot, cry a lot or exhibit any other significant changes in behavior, thoughts or their feelings that seem unusual, you may start to recognize that they are showing signs of mental illness. You can identify the tangible signs that somebody you know or love is dealing with a mental health issue. This way you can keep them in yourself safe. Usually, when there is a mental health issue, it doesn't only exhibit itself in a single change but in a combination of changes. We all go through changes in our emotions, thoughts and behaviors from time to time. However, when these changes make you less able to function day to day, there may be signs of a mental health illness. Mental illness takes many forms, including depression, anxiety disorders, bipolar disorder and schizophrenia. It's common and there are no simple tests to tell you whether you or someone you know has a mental illness. Often, you or people around you will notice small changes in how you act long before the symptoms seriously affect you. In this video, I will be sharing with you some signs that if you notice in yourself or anyone you know may indicate mental illness. 1. Sleep problems We all generally need 7 to 9 hours of sleep each night. When there is a persisting change in your sleep patterns, it could indicate a symptom of a mental illness. For example, insomnia could be a sign of anxiety or substance abuse. Sleeping too much or too little could indicate depression or sleeping disorder. Experts agree that you are emotionally distressed if you are sleeping more than usual or less than usual if you can't fall asleep or wake up after only a few hours and can't go back to sleep. They also admit that if you have recurring sleep disturbances more than once or twice a week, where your doctor cannot identify any physical reasons for it, your problem may be connected to anxiety or depression, both of which are basically psychological problems. 2. Germanic weight fluctuations or changes in eating patterns A lot of people would love to lose weight, however, when your weight continues to fluctuate or you experience rapid weight loss, it could be one of the warning signs of a mental illness such as depression or an eating disorder. Of course, other mental health issues can impact appetite or weight as well. If you find yourself gaining weight or losing a significant amount of weight without any changes in your diet or exercise regime, it may indicate a mental health issue. If you find yourself always thinking about food or repulsed by the thought of eating, some experts have admitted that it is an indication of emotional distress when you are always concerned with food, weight, and your body image, it is a sign that a certain eating disorder is cutting off energy from other areas of your life. One of the signs of depression can also be a lack of appetite. When women or young girls lose weight during menstruation or experience changes in appetite, this can also be a sign of mental health issues. 3. Unexplained physical symptoms You may find that despite your regularity with workout and consistency in visiting your doctor, you find no particular reason behind your physical issues. This may be a sign of some mental health issues. Your body is trying to let you know that your mind is going through distress. When you catch yourself experiencing constant headaches, a rumbling stomach, diarrhea, constipation and back aches, this may be connected to emotional distress and mental health issues. 4. Difficulty managing anger or controlling your temper We all have different moods. However, when there are sudden changes in mood, such as extreme distress or anger, this can be a symptom of mental illness. Mental illness can start with less obvious changes to a person's feelings, thinking and behavior. When the changes become significant or noticeable, it could be a sign that they have or are developing a mental health issue. You may seem to be comfortable being alone, but find you are suddenly prone to frequent outburst of anger by your spouse, children, friends or coworkers. By this, you can know that you may be overly stressed. Such a situation is dangerous to your physical and mental health and unhealthy for those around you. You may not observe the signs in yourself, but it is worth seeing a counselor if friends and loved ones continually tell you to calm down and cool your temper. According to Annie Kolei-Chen, EDD, Aaron, a young professor of psychology at Fortham University in New York City, not being able to control your anger is a sign of inability to manage feelings and this is the one symptom that has the biggest impact on other people. Depression and women especially are affected. People who have anger management problems do not recognize the symptoms because they feel fine when they are by themselves. This is something that only comes into play in relation to another person, so it's easy to blame the other person for what is really your symptom. 5. Compulsive or obsessive behaviors According to experts, obsessions are repetitive thoughts which resemble worry and are accompanied by anxiety. Compulsions are behavioral acts designed to eliminate the obsessions. If you feel a constant compulsion to do something, like clean up a table or you're overly obsessed about washing your feet or hands, for no serious reasons, it may be a sign of mental health issues, you're emotionally distressed. If you catch yourself constantly worrying that something terrible is going to happen, that's another sign to watch out for. If you're constantly obsessing over things being in perfect order before you leave your home, such that it takes you longer hours to leave home because you're arranging things for no logical reason. You might be dealing with anxiety in your life. If your mind is overwhelmed with obsessions and your day is overloaded with compulsions, your life will be hijacked by anxiety and activities that are not productive. 6. Chronic fatigue, tiredness and lack of energy Your body cannot always work magic. When it is overloaded emotionally and it cannot seem to take it any longer, it starts to turn itself off until it shuts down. This will show itself in fatigue, tiredness and lack of energy. Also, when you find yourself feeling too tired or fatigued and lacking the energy to do even the things you used to enjoy, however, a visit to the doctor shows you perfectly alright. It may be a sign of emotional distress and depression. 7. Memory problems You may be experiencing mental health issues if you find that it's harder for you to concentrate, recall information, analyze problems or even speak your own thoughts. Most people who suffer mental health issues show signs of impaired cognitive functioning. This can happen when you have so much to deal with, mentally, physically or emotionally, and you are not allowing yourself the time required for you to rest your mind and do things you enjoy. You will experience what is popularly known as brain fog. Lots of things can temporarily interfere with your memory, from the hormonal changes of temperature to a preoccupation with a work problem to a lack of sleep. But it can also be caused by stress, a reaction to a traumatic event or sometimes an illness such as Alzheimer's disease. How do you know the difference? You need a physical examination first and foremost, says Collagen. If everything checks out okay, she says, then anxiety, depression or sometimes an unrecognized reaction to a traumatic event you have yet to deal with may be behind your forgetfulness. 8. Shunning social activity Did you love to go to the movies with friends and now you don't? Do you seem fine at work, but the minute you're at home you jump into bed and just veg out? Are you turning down invitations because you simply feel better when you stay at home? Experts say all can be signs that your emotions can be getting the best of you. Any significant change in social behavior for a significant amount of time could indicate a stress overload or other emotional issues at work, says Collagen. Aaron awaits, adds that, if phobias or fears of certain places or events are keeping you from doing what you want, then anxiety may be looming large in your life. 9. Sex is no longer fun Are you going through the emotions and not feeling the pleasure that sex once brought to your life? Do you love your partner, but just don't want to make love? If a physical checkup reveals everything is fine, then Collagen says, an underlying depression or an anxiety disorder may be behind your slump.  She says, while that stress may be linked to your relationship with your partner, experts say it just as often. It could be linked to anxiety stemming from a totally different area of your life. 10. Mood swings and erratic behavior noticed by more than one person while life may seem like business as usual to you. If friends or family members are commenting on your moody behavior, experts say pay attention. You have to listen to not only your own inner voice, but also listen to what you hear from your best friend, your neighbors, your spouse, your family. Others can have an observation of you that you cannot see, says Collagen. The greater number of people telling you that something is wrong, the more you need to pay attention. If this video inspired you, like the video. We love you.


Hi, I'm Heidi Joy-Trathaway with the OpenStack Foundation and today I get to speak with Dan Cohn from the Cloud Native Computing Foundation. We're talking about all sorts of cloudy things today, including Kubernetes. Absolutely. Well, I'm thrilled to... Yeah, would you please tell us a little bit about yourself and the Cloud Native Computing Foundation? Sure. So the Cloud Native Computing Foundation is just a little more than 18 months old. We're part of the Linux Foundation and we host a set of technologies that have been of a lot of interest to OpenStack members and users and are very complimentary in a lot of ways. Okay. And, you know, we did our user survey recently where we talked to folks about what kinds of technology they're using and we saw in the latest user survey 47% of folks indicating Kubernetes is kind of the number one container or POS tool that they're using. Well, that makes me very happy to hear. We've been working hard with that community to raise awareness for it and try and help with its growth. It's actually one of the very highest velocity projects in the history of open source. So when you look at the number of commits per day that it's getting over one per hour and over 1,300 authors that are contributing from dozens and really now hundreds of different companies and the number of pull requests and issues, it's a pretty amazing project. It has some downsides that it's also a little bursting at the seams of all these people who want to participate and take part but it's really been a lot of fun to get to work with them. Yeah. It really is exciting to watch. And what we've seen over the last four user surveys, for example, is 75% or more folks are telling us that containers are the number one emerging technology that they're interested in. But I know that Kubernetes is not the only thing that the CloudNative Foundation does. Can you tell me a little more about that? Sure. So we're now hosting nine different projects of which Kubernetes is by far the biggest and best known but we have a number of complementary projects to that that are used with both with Kubernetes and other container orchestrators. So in the monitoring and logging and tracing which are all kind of complementary services we have Prometheus and Fluent D and Open Tracing. And each of those projects is building up a good amount of momentum and excitement behind it. Prometheus, this was already a really extremely popular project and we think we've just been able to help that along. We're also pleased now to have two of the best known container runtimes, Container D and Rocket, were just donated to us a month ago. And our newest one along with those is maybe not as well known but quite exciting is called GRPC and it's an alternative remote procedure call platform for when JSON REST doesn't give you the performance you need for an API or internally and you can swap in GRPC and it is being used by Docker, by Kubernetes, by a ton of other projects out there. Wow. That's fantastic. So one of the reasons that you came to this OpenStack Summit is because it's not just an OpenStack Summit but we're working with a lot of different open source communities, in fact 11 different open source communities having their own kind of mini conference within this summit. Tell us a little bit about Kubernetes Day from yesterday. Yeah, so we were really pleased to be able to work with a lot of our in fact common members who are members of both OpenStack and CNCF and sponsored this Kubernetes Day at OpenStack and we got a good turnout for it and a lot of interest and just a ton of folks who are running them together. There's a little bit of debate out there where some people are saying, oh, how can I run Kubernetes on top of OpenStack and then there's a whole set of other folks who are saying, oh, here's how you can run OpenStack as an application on top of Kubernetes but we got a good set of discussions around that. So I'm hearing so much about how containers might be running beneath all the stacks or on top of the stacks. What is your vision for the future of what Kubernetes can do? We definitely see it in a lot of different places. So I think probably the most exciting area and the folks I'm talking to are Kubernetes on bare metal and then using, we have a package manager that's part of Kubernetes called Helm and it supports pretty much automatically deploying OpenStack on top of that. So I think that's the area that's getting the most traction but they're all open source projects. There's a lot of folks who are just trying out different things. We're very supportive of different collaboration, different experimentation. Yeah, and I think I understand you have another event in the works to bring people together and keep working on that. Yeah, thanks for giving the opportunity. So it's CloudNativeConCubeCon is our big annual summit and for North America, it'll be in Austin on December 6th through 8th. And so for people in the OpenStack community that want to learn more about Kubernetes, get to interact with a lot of the core developers and folks in the community. We'd love to see them in Austin. Great. Well, thank you very much, Dan Cohn, for joining me. Oh, thank you. I look forward to seeing you in another event.


You should already know how to carry out titrations, but if you've forgotten, watch part 1 again and refresh your memory. In part 2, we will see how to do titration calculations. We will look at two different examples together, and then one for you to try out yourself. These calculations enable us to work out the actual concentration of an unknown solution. So let's look at an example. We have carried out a titration. We started with 25 cubic centimeters of an unknown solution in the conical flask. We then added 20 cubic centimeters of the known solution of concentration 0.5 mole per cubic decimeter to complete the reaction. What is the concentration of the unknown solution? First we need to convert volumes to cubic decimeters. We do this by dividing by 1000. So 25 cubic centimeters divided by 1000 equals 0.025 cubic decimeters. And 20 cubic centimeters divided by 1000 equals 0.02 cubic decimeters. Now you need to work out the number of moles for the known solution, because we already know the concentration and volume for this. The number of moles equals concentration multiplied by volume. So in this case, for the known solution, the number of moles equals 0.5 multiplied by 0.02, which equals 0.01 mole. Using this, we can now calculate the unknown concentration. The concentration of the unknown solution equals its moles divided by volume, which equals 0.01 divided by 0.025, which equals 0.04 moles per cubic decimeter. So the unknown solution was at a concentration of 0.04 mole per cubic decimeter. It is much easier to take the time to understand what is happening, but if you prefer to just learn an equation, this is all you will need to know for titration calculations. Using the values we've just determined, we can make use of the following formula. The concentration of the unknown solution equals the concentration of the known solution multiplied by the volume of the known solution divided by the volume of the unknown solution. So the unknown concentration equals 0.5 multiplied by 20 over 25, which equals 0.4. But you may not pick up all the marks in the test or an exam using this formula, because it doesn't show your understanding. Sometimes you need to work out the proportions involved in the reaction by looking at the equation for the reaction. So if we had this question, the balanced equation would look like this. So two moles of sodium hydroxide reacts with one mole of sulfuric acid. We are now ready to do the calculations. Convert the volumes to cubic decimeters. 40 cubic centimeters divided by 1000 equals 0.02 cubic decimeters. 40 cubic centimeters divided by 1000 equals 0.04 cubic decimeters. Now calculate the moles for the known solution, sodium hydroxide. The number of moles for sodium hydroxide equals 0.5 multiplied by 0.02, which equals 0.01 mole. Now calculate the concentration of the unknown solution. The concentration of the unknown solution, sulfuric acid, equals 0.01 divided by 0.04, which equals 0.25 mole per cubic decimeter. But from the equation, we know that two moles of sodium hydroxide reacts with one mole of sulfuric acid. So 0.25 is for two moles, because we have so far assumed a one to one ratio. So we need to divide this by two to get one mole of sulfuric acid. So 0.25 divided by two is 0.125. Now here's a question you can try out on your own. Pause the video, work out the answer and click play when you are ready to check. You get 0.3 mole per cubic decimeter. Here's how you work it out. So using these titration calculations, we can work out the exact concentration of an unknown solution. We have to carry out the titration first, so that we know the exact volume and concentration of a known solution. And then we also need to know the volume of the unknown solution. Armed with this knowledge, we can then work out the exact concentration of that unknown solution.