It might help if I would turn my camera around. That looks a lot better, don't y'all? Okay, this is the Star of Bethlehem. I was just reading about it online. This is in our back door at our neighbor's house. They live on the marsh out here on the river and you can see the star up to the left if you look direct northwest of the flagpole. You can see the sun setting in the background and that is the star. Now, I will zoom in on the star and let you see. They will pass each other and get really close to each other and you can see it pretty good right now. I'm going to pull this camera up a little bit over this tree and let you see the moon shining bright in the sky and Chris is out here with me, my sweet man. I don't know if you can see him or not. It's pretty dark out here, actually. But it's so pretty out here over the water. You can see that sun setting and how the sky looks. I'm going to try to zoom in just a little bit and show y'all the star. And that's how it looks right now with your phone. But if you're in, there it goes. So you can kind of see that there's two lights. One is Jupiter, one is Saturn. And I do believe the brightest star is the Jupiter star. If you had a telescope, they're saying that you could actually, you might could actually see the rings around was it Jupiter that has the, not the rings that they call them. Jupiter and Saturn? Yeah, it's Jupiter and Saturn. Yeah. So there they are. Isn't it beautiful? Y'all, they said this happened 400 years ago, but it wasn't at night. So they said it has been 800 years. Since these two planets lined up and the star was going to be this bright. That's a long time, 800 years since this has happened at night. So this is a sight to be held or to be seen. And I wanted you guys that are stuck in the house or maybe in an area that doesn't have a lot of darkness, like a big city or something. You might not can see it as well. But look at that star up to the northwest of that flag. Isn't that beautiful? And isn't the flag beautiful? These is our neighbor's yard. And what is this out here, Chris, called? The Marsh. This is North River, the North River Marsh. This is the North River Marsh in St. Mary's, Georgia. Live showing y'all the Bethlehem star. Isn't it gorgeous? I'm gonna zoom back in to those who haven't been on here in a while, haven't been on here a long time and show you the star up close. It takes the camera just a minute to focus. Uh-oh, I must have flipped it back around on me. Sorry. How did that happen? There it goes. I think if I do this a certain way, it actually turns the bone around. I don't think that star is moving. It's me moving. Okay. I'll see the two planets. There they are. You can see both of them. You think you could hold that more steady or is it just me? I don't know. That's really weird. I mean, when you zoom in that much, if you move at all. Right. Because I'm so far zoomed in, it makes it look like I'm moving a lot, but I promise y'all I'm trying to be as still as I can be. But that is Jupiter and Saturn lining up for the Bethlehem star. I just want everybody to be able to see it and it looks like it may be an hour or so before they actually come together. I think they said it would be about an hour after sunset and really would you call that sunset? Do you think that sun has said it, Chris? You need to zoom back out even though I could have said it. I'm sorry y'all. I'll show y'all the sunset. The sunset's at like five thirty. Okay. Or five something. Now you can see the star direct northwest from the flagpole. There is our beautiful Bethlehem star that we have not seen. In eight hundred years. Eight hundred years. A long time. So I hope y'all have enjoyed us coming in here and showing y'all a clip of the star. I think this is a great moment to think about our Lord and Savior Jesus Christ who is the whole reason for the season. And you want to see anything about that, Chris? No? Okay. It takes me a minute. Okay there it is again. So these will actually line right up together and it'll shine its brightest. Probably within about forty minutes or so. It looks really good. So I hope y'all have enjoyed seeing our gorgeous Bethlehem star today. We're going to zoom back out. Show y'all the sunset. And just seeing that flag in there even makes it better don't it y'all? This has been a wild ride, a wild 2020. And even as it has been crazy, God has blessed us with something that we haven't seen in eight hundred years. And he's blessed us at the end of the year with this gorgeous Bethlehem star and we thank him for that don't we? I hope y'all have a blessed and merry, merry Christmas. I hope you've enjoyed seeing this bright light in the sky because Christ is the light of the world. And I just want to tell y'all how much I love him, how much I love y'all. And I thank you very much for watching our channels and supporting us and always being here. And we thank the Lord for this gorgeous sky he gives us to. I'm going to zoom in one more time and let y'all see them both together. And it's way brighter than any of the other stars if you look around. Yeah, even if you're in a city they're saying that you should be able to see it because it is so bright. I'm going to zoom out and just kind of show y'all the rest of the sky. And then we're going to conclude our live showing. So about this way, well there's a bright light in the air. There's a lot of stars but now they're not going to be able to see all those stars. It's it's bright enough that y'all can't see the other stars in the sky. Look at the difference you can see. There are stars everywhere. Wait, you can see that star but it's hard to see the other stars when I zoom into the other parts of the sky for y'all or even just not even zoom. But y'all can of course see the moon. There's the moon. Y'all have a wonderful night. And I'm gonna try to see if y'all can kind of see me out here. Y'all have a wonderful night and thanks for watching.  Bye y'all. Thanks for watching. The Bethlehem Star. The wind has died down just since we started and you can tell by looking at that flag. You could rewind the video and and look at that flag just 10 minutes ago and it's blowing in the wind and now it's starting to die down and lay down on the pole.


Hi everyone, welcome to our session on groupless auto-skilling with Carpenter. My name is Prithee Gogia and I'm here with Alistan. We both work in the EKS scalability team at AWS. We have been working on the Carpenter project for the last few months and today we will be talking about the current auto-skilling landscape in Kubernetes, how Carpenter works and a quick short demo. There are multiple ways a pod can be created in a cluster. Users can use a CLI tool like Qtctl or there could be a crown job running or there could be an auto-skiller tool such as HPA, KDA, Knative which could add these pod into the cluster. Once these pods are added, they need resources to be able to run. These resources can either be added by a manual effort or by an auto-skiller. Capacity planning is hard and manual scaling can be slow, error-prone and cost significant delays as compared to an auto-skiller. Auto-skiller has benefits over manual and today cluster auto-skiller has become the de facto for auto-skilling in Kubernetes. So, what are some of the benefits of cluster auto-skiller? As pod scale up, load increases in the cluster. Cluster auto-skiller can request more capacity from the underlying provider and when the load reduces, the cluster auto-skiller can also request to terminate these instances from the provider which in turn helps user reduce their infrastructure cost and remove the burden of capacity planning. There are other advantages such as cluster auto-skiller is vendor-neutral so has support for all the major crop providers. The approach is battle tested and widely adopted and it works great for cluster sizes above 1000 nodes. So, let's take a look at how cluster auto-skiller works. Cluster auto-skiller works around this concept of node groups which is an auto-skilling group or an ASG in case of AWS. It looks for pending pods, sets the desired count of instances based on the number of pending pods on a node group. Once the underlying provider provisions the capacity based on the desired count, kubes scheduler places these pods onto these new nodes being created. Node group here plays a critical role by maintaining the desired number of nodes which are requested by the auto-skiller. So, let's try to understand what node group is. Node group is a logical grouping of nodes which can be auto-skilled and managed together. Cluster admins can create these multiple node groups specifying instance type options and some other configurations and also set a minimum and a max node count. Node groups are not auto-selfilling. It's done by an auto-skiller by setting the desired maximum. There are some challenges with this approach though. Let's see what are those challenges. Cluster auto-skiller assumes that the instance types are all identical in a given group. And if you want to use multiple instance types, you need to create separate groups. And furthermore, if you need to spread pods across AZs, multiply the number of groups by each AZ. Some of the other challenges here are if there is an error creating a node in a node group, cluster auto-skiller will not know until it times out and this causes delays for an application when a pod stays pending. There are users creating clusters larger than 1000 nodes today. But there have been some efforts from the community, but unfortunately either they are deprecated or not maintained. And one of such efforts is from Cerebral which is Cerebral which was created because cluster auto-skiller was slow. Another one is Escalator which is a batch workload specific and works on a selected set of groups in tandem with the cluster auto-skiller to fix some of the challenges we just discussed. Another interesting approach is from Zalando which tries to improve and build upon cluster auto-skiller. They improved how cluster auto-skiller generates template nodes for groups to decide what kind of node to expect when scaling up. They added reliable pack of logic in case of node to phase to scale up nodes. And they made some more additional changes and some of their efforts which have been upstreamed to cluster auto-skiller which makes it more robust and productionary. But one thing to observe here is all these different approaches are trying to solve around node groups. Keeping these challenges in mind, now I would like to hand it over to Alice to talk about if we can do any better in terms of auto-skilling. Thanks Pateek. So we spent a fair amount of time in this node auto-scaling space and we're trying to understand what was exactly the root cause of all these issues. Some of them were just code bugs in our cloud provider implementation, things that we could fix, you know, things that we did fix. Sometimes it was just customer configuration since these things are really hard to get right. It's really easy to misconfigure node auto-scaling. It's one of the things that we decided we really wanted to focus on to make easier. And finally part of it was just architectural. By nature of having node groups, you take on some assumptions and that was where we decided that a lot of the inefficiencies were coming from. So we really started to play with this idea of what would happen if you remove the group? What if we provisioned capacity directly? The idea is that when some pending pods come in, you can compute the resource requirements of all those pods just like the cluster auto-scaler might. But instead of simulating from an existing set of static templates of a set of nodes that could be created, you just ask EC2 for exactly the instance that you want. So we know all of the instance types. We can simulate those pods on, you know, we can pick an M5 large or any instance that makes sense for those pods, whether or not it's been seen before. And additionally, the other node properties like the availability zone and the operating system, all of these things we can compute dynamically. So we've shifted from this static template world to this dynamically generated template world, which reduces the configuration complexity pretty significantly. So once we've decided what instance we want, we just tell our cloud provider, please give us this instance. Of course, one of the value props of groups is they track nodes so that you can, you know, you can roll out a change or you can delete them all at once. So we decided to track nodes using Kubernetes labels, which we thought was a little bit more Kubernetes friendly and native to the other tooling that you're using. So the result of all this is you have pretty significantly reduced configuration burden. We've seen customers deploying over 50 ASGs in production, which is the cross product of instance type capacity type availability zone and scheduling properties, which can blow up really quickly. So you just don't need any of those anymore. It's all done dynamically. Also, the API load is pretty significantly reduced since for every single one of those ASGs, you have to ask the cloud provider, you know, what is what is the current status of this thing? What is the what is the value? And those API requests really, they really stack up. And you can run into API limits and the whole system grinds to a halt. So instead, we just make calls when we're creating and deleting capacity. And otherwise, we rely on the Kubernetes API server, which can support much higher throughput. It also reduces the simulation complexity, we no longer have to to look at such a large number of templates, we can collapse the simulation space to just instance types because we can prescribe the availability zone and scheduling properties, which really simplifies things. For example, instead of looking at all the possible groups and saying which one has this zone, we can just tell our cloud provider run an instance in this zone. So we ended up with this architecture, which should seem fairly familiar if you're familiar with the cluster auto scaler. We start up at the top left where you have pods that were created and haven't been scheduled yet. And the coop scheduler gets the first crack at it. So it looks at the existing capacity and the and the pending pods and it tries to find places to schedule those pods. And if for some reason, the resources aren't available to schedule them, the coop scheduler sets a status condition in the pod that the pod cannot be schedulable. And that's when we kick in. So we have a process called the allocator that looks at pods that have that unschedulable status condition. And then we attempt to create new capacity that can fit those pods. We try to make sure that capacity is as efficiently packed as possible, and that we can bring it up as quickly as possible. And then that capacity joins the cluster. So your existing capacity and your provision capacity, just now are your new cluster state. Over time, you can get you can get holes in the capacity you can get inefficient utilization. So we have another controller that looks globally at the cluster and attempts to optimize it over time. So where we end up with is two controllers. One is a fact fast acting latency sensitive controller, the allocator, and the other is a slow acting cost sensitive controller called the reallocator. And together, these work to both schedule pods as quickly as possible, and optimize the resource utilization of your cluster. So there's a couple ways to launch capacity directly. Well, there's many ways to launch capacity and easy to at least. So previously, we were relying on ASG, we gave it a template and we gave it an instance count, and it was responsible for launching all of those instances. Now we've removed ASG. And so we want to use EC to direct capacity launch API's. There's two of these, one is called fleet, and the other is called run instances. They're roughly the same, we could actually use either of them. And I imagine that other cloud providers, they might have multiple options as well. We know that every cloud provider does have the ability to run instances in general. But we for our use case and for our cloud provider, we selected the fleet API, because we think that it gives us the most flexibility. So for example, when when deciding what node we want to create for a given set of pods, we don't actually limit ourselves to one availability zone and one instance type. And this is constrained to our cloud providers implementation, you could have another implementation that selected the specific instance type in the specific availability zone. But we actually can get it some additional value by being flexible. So we tell EC to these pods will fit in any of these instance types. And given their topology constraints, it'll run in any of these availability zones. And by giving it that flexibility, EC to can actually pick the cheapest instances given the constraints, the more flexibility you give easy to the better of a job that can do to pick the right instance type given your constraints. So it's almost like it's almost like a run instances request that's been expanded and made more flexible. But at the end of the day, you are still getting one node that those pods get packed onto. So we get pretty massively increased node flexibility here, we get this really nice failover behavior of a particular instance type is out of stock. It also pretty significantly reduces the node provisioning latency. So systems, systems like auto scaling groups and other group systems, there's there's a little bit of overhead to them. So there's some reconciliation loop that's happening in the implementation of that thing. And, you know, maybe it's really fast, or maybe it's not very fast. But we found that with ASG, it was adding about 30 seconds from the time that you increment the node count to the time that the node is brought up. So by removing that and asking EC to to create fleet directly, we get instances 30 seconds faster, which is a pretty big deal. Scheduling is also critical to the provisioning workflow. So like I mentioned earlier, we work in tandem with the coob scheduler, we give the coob scheduler the first shot. And when it fails, we kick in afterwards. But one of the corollaries to this is that is the realization that provisioning decisions are scheduling decisions. By nature of picking a new node for pods to schedule on to you are making a scheduling decision whether or not you like it, or whether you like it or not, it's it's it's unavoidable. And so one of the differences that we were able to do because we're creating the capacity directly is we can actually create the node objects. So rather, we make the the create instance request, we get an instance ID back, we're able to create the node object immediately, and then bind the pods immediately. And so we've enforced a scheduling decision kind of outside of the loop of the coob scheduler. And that's a fairly interesting pivot from the way that the cluster autoscaler works, which is it brings up the new capacity and then waits for the coob scheduler to do the scheduling. But we decided that if we're already making a scheduling decision, we might as well enforce it. And there are some benefits to that. So for example, because we've already bound the pod, the second that the node comes online, you can start pulling the image because you already know the binding decision, you don't have to wait for the node to become ready, you don't have to wait for the coob schedule scheduler to schedule it. So that was just kind of a nice dual optimization we did. So one is we enforce that the decision that we made is actually is actually it actually happens in the cluster. And we also get some, you know, about five seconds of latency improvement, which is which is quite nice. The other thing that we realized is that the cluster autoscaler is limited in its version compatibility. So they copy the coob scheduler directly into their code base, and use it. And so because of this, there's a recommendation that the coob scheduler must be the same in the cluster autoscaler's code base as as you have running upstream, or as you have running in your masters rather. And so this this creates a tight version coupling between those two things, because they're not enforcing the the binding decision. And so by doing this binding binding decision, we now free ourselves from this version compatibility, because the the coob scheduler is its own system that has its own set of responsibilities. And the carpenter allocator scheduler is is the same is it also has its own responsibilities. And so you no longer have this type coupling and versions of carpenter can work across many versions of Kubernetes. Another corollary we have here is the scheduling API conformance. And so by nature of writing a scheduler or writing logic that decides new nose to provision, we have to support all the common things like resource requests and selectors and affinity and topology. And so this is this is just kind of a burden that we take on as part of trying to play in the salt in this problem space. And specifically, the resource requests need to get a little bit complicated. So when you have, I'm sure you guys are familiar with running GPUs or running potentially other custom instance, or custom resource types, your cloud provider effectively needs to have knowledge of how to launch instances with those particular instance types or resource types. Sorry. And so there's kind of an ongoing burden here of the cloud providers to ensure that the the particular properties of the instances that the pods are requesting are actually able to be launched. And we built a bunch of machinery that's that's vendor neutral that supports the node selectors and all of the scheduling that the non then non vendor specific scheduling constraints but there is there is somewhat of an ongoing maintenance burden as as new types of resources enter the enter the scene. We also operate based off of this concept of well known labels. So for example, this is how we communicate that a pod needs to run in a particular particular zone or in a particular instance type. There's this concept that if in your pods node selector, you use for example, topology dot Kubernetes dot io slash zone and you state that zone, that's a signal to carpenter that the the node that you need must be constrained that then it must have a label of that zone. So we use these well known labels to apply additional constraints. There's also a set of defaults so that if you don't, you don't define these or you don't have to define these in all cases. But if you do need to do apply additional constraints, you would do that via node selectors and well known labels. So once we've sorted out our scheduling, you end up with kind of groups of pods. Imagine you have 100 pods that need to be scheduled. And some of them have once another node selector and some of them have another node selector. You end up with these what we're calling constraint groups. And these are pods that that can be scheduled together according to their scheduling constraints. But we still we have another problem here, which is how do we pick the instance that can fit all of those pods. So we sum up all of the resource requests of the pods. And then we do what's called bin packing. This is a classical computer science NP complete problem. There's a couple of variants. The one that this is is called online bin packing, which means we're looking at the subset of the overall pods in the cluster. And we're attempting to pack them into the existing capacity. There's a bunch of research in the space and we're actually super excited for all the innovation we can do here. But the algorithm that we've picked right now is called first fit descending. And what that basically means is we sort the pods by size. And we try to pack the pods biggest to smallest. And what this does is it kind of has this nice effect. It's called first fit descending has this nice effect where you attempt to pack pods.  And so as you get to the smaller pods, they tend to fit in the gaps that you've created by starting with the largest one. And this is actually, there are some great analogues here to actual container shipping and and how containers are packed in that world. But the main idea is that you take a set of pods that can all be scheduled on a set of nodes together, they have the same scheduling constraints. And the result is you end up with a set of nodes that can fit those pods. You might ask, well, how do you order the pods? Because it's not a single dimensional problem. There's a couple approaches to this, you can either prioritize CPU or memory or do the Euclidean, which is the sum of the root of the sum of the squares. And so we found that the Euclidean is kind of a best effort simulation of kind of waiting memory and CPU equally when sorting them. And that it tends to work fairly well for bin packing. But like I said earlier, we're really excited to innovate in the space and figure out better algorithms. We've talked a little bit about genetic algorithms, which could be really interesting. And there's a whole field of research here. We're not just responsible for bringing up nodes, we're also responsible for terminating them. It's, you know, what goes up must come down. And so we have we have another controller as part of the rather it's part of the reallocator controller. We look at nodes that don't have any pod schedule to them anymore, and we apply a TTL. And if those pods if those nodes don't get any pod scheduled to them for a period of time, which we default to five minutes, those nodes are eventually scaled down. We also respond to other events that might cause instances to be terminated. So for example, when EC2 notices that your virtual machine is unhealthy for some reason, it'll send an event. And so we respond to those and we we drain the node as quickly as possible and try to try to heal the situation. Similarly, there's a couple of other signals called spot rebalance and spot termination. Rebalance is kind of a heads up from spot that your node is going to be preempted. And so you can do your best to drain and reschedule everything. Termination happens after the rebalance. So you're not guaranteed to get a rebalance request, but you typically do. You are guaranteed to get a termination signal. And that termination signal means you have two minutes to drain. So ideally, we've we've managed to terminate the nodes and reschedule everything by the rebalance. But if for some reason it happened so quickly, or some things getting gummed up, then it has to be done before that that node termination. The other thing that we've realized is that you can apply a node TTL of a long timeframe. So for example, there's there's some value in saying I want this node to live for 90 days. And this is super useful in the upgrade case. As long as our allocator is creating nodes on the latest Kubernetes version. And as long as the nodes only live for 90 days, your cluster is basically always up to date, given the Kubernetes three month release cycle. There's some other compliance use cases where you might care about how long the node lives. Maybe you want to roll your fleet every month or so. But this is another feature that we built in that that we think will be super useful for upgrade, you know, potentially weird memory leaks, potentially compliance cases. Of course, you can not use the TTL and your nodes will live forever. So previously, I mentioned that we will only drain nodes that are empty that don't have any pods on them. But you could also imagine that over time, most of the pods get deleted, but not all of them. And this can end up with what we call defragmentation, where you have nodes that are really poorly utilized. And this is where this this reallocator process kicks in. In addition to the really easy case where we just scaled down an empty node, we can look at poorly utilized nodes, and ask the question, is there a better selection of nodes within the cluster that we can replace our current nodes with that will more efficiently pack these pods? This is actually a really challenging problem. Because pods are immutable, they can't be moved, you have to be really careful about deleting a pod and waiting for the new one to be created, and then enforcing your repacking. So this is still very much in the design phase, we're still trying to figure out how to do this, but we're committed to the idea that you shouldn't be stuck with the the initial state that things were provisioned in, you should have another controller that's always monitoring always trying to improve the situation. So we're exploring different algorithms here. Also, one of the most important pieces of this problem is one that doesn't really come up very often, but we realized is absolutely critical. It's how quickly are these nodes launched? There's a whole bunch of steps, they're all listed here in the critical path of a node coming online. And right now in AWS, that's about two minutes, in different clouds, I think it's actually fairly common for it to be about two minutes. And when it's this slow, you end up with this use case where you actually want to over provision, you can't wait those two minutes. So you'd like extra nodes to be in the cluster so that the pods can be scheduled more rapidly. And so we've decided to really attack this problem, we think we can get it down to maybe 50 to 30 or 15 seconds if we do a ton of work. And so we're really excited to to attack this this latency problem that which we think will mitigate the need to do over provisioning. Of course, you know, there are going to be use cases where you need to over provision, which we recommend doing at the pod level. And in the very worst case scenario, if you can't over provision at the pod level, and you can't tolerate the latency at whatever we've optimized for the point in time, we recommend that you just statically provision your capacity. Of course, we're always open to new ideas. And we've actually had some discussions about some sort of feature about leaving headroom. But that's not fully baked yet. I also want to mention quickly that we do this using Kubernetes custom resources. This is kind of the modern standard way to write controllers in Kubernetes. We found that using command line arguments and a single global operator is kind of untenable. So all of these configuration parameters, the taints, the labels, the cluster that the new nodes connect to, all of these things are encoded in a custom resource. This project is, of course, open source and vendor neutral. It's currently incubating in the AWS labs GitHub repo. And we're excited to figure out where it's going to go after that. We have a roadmap that you should come check out to see our 0.2, 0.3 and 0.4 releases that we have planned. We're planning on executing about the rate that Kubernetes does so releasing as Kubernetes releases. And the other thing is we have a biweekly working group where people come from Segato scaling or from our customer base or really just people who are curious and we talk about the project and talk about things that we could do to make it better or prioritize features or design reviews. So if you're interested in that, we'd love to see you drop by. There's a QR code here if you're interested. And now I'll hand it off to Patik who's got a demo for us. Thanks, Alice. So let's go ahead and see some of the concepts that Alice has explained in action. I have already provisioned an EKS cluster. It's running a single node. And I have also deployed carpenter onto this on this one node. So at the moment, we have carpenter running some of the components which carpenter needs and some of the system parts. So as a next step, let's go ahead and keep a watch on all the nodes on this cluster. At this point, we are ready to configure carpenter to be able to talk to the API server. So these are the configurations that we need to provide to carpenter to be able to talk to the API server. And the other configurations I have here are optional. We provide a list of instance types to limit the the instance types carpenter can use when requesting for the capacity. We are also limiting the zones in which carpenter can create the capacity and detailed second still carpenter for how long to wait when it detects an underutilized nodes before it terminates the node. So let's go ahead and create the CRD object which carpenter can watch. And using all this information, carpenter will be able to talk to the API server. So at this point, we are ready to add some workload into the cluster. But since we have this node tainted and it's only running system critical components, no other workload can be scheduled on this one node, we will need additional capacity added to the cluster. So let's go ahead and add a deployment into the cluster. This deployment contains a single container and replicas right now are zero and reduces a pause image with CPU request for one core. And before we start to scale up, let's keep a watch on the pods which are getting created as part of this. Let's also go ahead and tail the carpenter logs to see the actions carpenter is taking. At this point, we are ready to scale up our workloads and let's start with a single replica. So once we create the replica, what's going to happen is this particular pod is going to go in the pending state because there is not enough capacity in the cluster. And carpenter will request for more capacity from the underlying provider, which is AWS in this case. So as we scaled up, the pod went into the pending state and carpenter detected that there is one pod which is pending and it requested a node from EC2 and it binded this one pod on this particular node. So let's give it a few seconds. And this is the node which is coming up. And in another few seconds, we should be able to see the version instance type zone and architecture. So here it is. We are running on an X large in US East to see an AMD 64. And as soon as the node comes up, the container will start getting deployed and will be running in few seconds. So in about 49 seconds, we had the container running on this node, which was requested by carpenter because there was a pending part. Let's try to add two more parts of this in the same deployment and see what happens. So once we add two additional parts, what happened was because this extra large node had sufficient capacity, these parts went into running state and they were never detected by carpenter. Let's take it a step further and try to deploy some additional parts. So once we add four more parts and carpenter detected all four parts and requested another instance from EC2 to add these four parts.  Similarly, in the next few seconds, as Kubelet comes up, we should be seeing the version instance type zone and architecture. And these parts should go in the container creating mode. So at this point, the image for these parts is being pulled. And once the image is there on the node, they should go into the running state. And the difference to note here is carpenter distance selected at 2x large because we needed four cores for the four parts we were creating. And then the zone it selected was 2b. So at this point, we have all the seven parts running. Let's also go ahead and check how bin packing works in carpenter. So this time I'm going to add 100 parts in the same deployment. And once these parts go in pending state, carpenter detected that there are 55 pending parts and it tried to bind all these parts on the same node. And then it detected there are 35 parts on the second and tried to bind these 35 parts on the second node it requested. So the way this works is carpenter runs in a reconciler loop every 10 seconds checking for the pending parts. And in the first reconcilation loop, it detected 55 parts. In the second one, it found out 35 parts. 55 parts it requested a larger instance type and for the 35 is requested at 12x large. We can verify this. So a 97 is a 12x large and 174 139 is a 24x large. So as these instances are now coming up, we can see these parts are getting into container creating. Some of the parts are already going into the running state here. So let's give it a few seconds and all these parts should just go into running state. At this point, we can quickly check how many are in the running state out of 100. So we have 95. Give it a few more seconds and see once we reach 100. So we have 100 parts and we can go ahead and watch the parts again. So it took about two minutes for all these parts to be created across two nodes which are 12x large and 24x large. Next, let's see how scale down works. So let's say the load has decreased and the replica count is being set to zero now. At this point, all the parts are being terminated across all the four instances that we just created. And once all the parts are being completely removed, Carpenter will detect that these nodes are underutilized and Carpenter will wait for 10 seconds. That was the TTL seconds we configured earlier and remove these nodes by draining these nodes and then and then terminating these nodes as a result. So we started terminating one by one and two of the nodes are being removed. So we are left with two more nodes which are 12x large and 24x large. So at this point, all the parts are being terminated and Carpenter has already removed three nodes and we are just waiting for the last node to be removed from the cluster. And these nodes are also being deleted at the EC2 level. So they are being terminated in AWS also. So now we are back to where we started with a single node and no workload at all. Let's say as a use case, you want to schedule some of your parts on a different architecture for a node. Let's say you want to use an ARM64. So let's repurpose our deployment and try to run the parts on in this deployment now on an ARM64. Let's keep in mind we never created any auto scaling group or a node group for auto scaling based on particular architecture or instance type. And let's go ahead and create two replicas for these parts and they should come up on an ARM64 type architecture node. So they both went into the pending as expected. Carpenter detected two provisional parts which need capacity and it requested a node from EC2 and binded two parts on this particular node. And in this case, the architecture it requested was an ARM64. So in another few seconds, once Cubelet is up, we should be able to see the version instance type and the zone again in which this instance is being created, which is an ARM based node. And these two parts should get created on that particular node. So it took about 56 seconds and the instance type it selected was a T4G extra large and the zone it picked was a US EC2C. And we can see that the container are in the container creating more and the image is being pulled in the background. So they both are in the running states on an ARM architecture. So yeah, this is all I had for the demo. Thanks for joining our session.


OK, so I'm extremely honored to be here with you and meet with you. I have an amazing respect for the Next Cloud community and what the Next Cloud community does. So it really is an honor to be here. And you will see why I have such an honor. I will explain that to you during this presentation this morning. So three things to summarize who I am and what I have done. First, the little furry logo on the left side is Firefox. I have spent 17 years of my life devoted to the Mozilla Project. It started when it started. I was lucky enough to be an employee of an internet startup that was called Netscape back in the days. Well, I started back in 97. And in 98, I had the honor to introduce the Mozilla Project to the world because I was in charge of PR in Europe and nobody at the time understood what Open Source was about. And they just thought it was like software you didn't have to pay for, and that was all. And so it happened that I already knew Richard Stallman from before. And so I've owned here to be the guy to help with the Mozilla Project to be known in Europe. And after that, I founded Mozilla Europe. I was the president of Mozilla Europe for nine years. And after 17 years in the same project with high and lows and happiness and less happy times, I decided to do something else. So I decided to leave Mozilla. I still call myself a Mozzillion. I am still involved with Mozilla as a volunteer. But I'm not and paid employee anymore. And so I decided to write a book which is called Surveillance. And pretty much this is what we are going to discuss today this morning, this book, Surveillance. To my surprise, it's selling really well. My publisher announced me earlier in the months that they would do another reprint of the book. So it's the third time it's being printed, which I can't understand. But well, it's a good surprise because in my opinion, it's a very important topic. And finally, I have joined Cozy Cloud, which is an open source project that is similar in some ways to NextCloud, which is based in Paris, France. And I will explain you why I'm interested in all of this because I think it really matters. So we'll start with some history. This is one of the first machines I owned back in the days. And so this is an Apple II, which is, when compared to an iPhone, is extremely different because it's open first. You see, you could open it. You have some Velcro stuff to open it and close it. No tools needed to open, which with the notion of an Apple device is extremely strange nowadays. But at the time, it was really open. It was hackable. It's different also from such devices because all the data that you wanted to put in, you had to type it with a keyboard. Like now, it's different because data is generated by this thing. Just moving the thing is generating data because you have sensors now in smartphones. But at the time, if you wanted data to be in the computer, you had to type it with your hands. And also, you had to write the software because there is no software available. So that was really the early beginning of personal computing. And do you know the difference between these two? I don't know how to call them. Not vultures, but the one on the left is a vulture. And it's from California. It's a California condor. And on the right, it's a California not vulture. No, that's not right. He's dead anyway. So that's Steve Jobs. And actually, Steve Jobs, and so it's barely readable. But it was a very interesting thing that Steve Jobs said. That the human are different from the big, is it, primates? Yes, is that we build, as humans, we build tools. And they made a comparison on what is the most efficient animal to move for one kilometer. And the animal, which is the best at the game, is the condor. When it flies, it saves energy a lot. So it takes little energy to move to one kilometer. And the human is not really good at that, unless you put the human on a bicycle. And then it changes everything. And finally, thanks to the tool, the human becomes the best at the game. And Steve Jobs explained that the computer is a bicycle for the mind. It's the tool that expands the mind and does more. So that's the key thing. It's that the computer is the bicycle for the mind. And sorry for stating the obvious, but the computer is something which is totally useless unless you have software. And in 1999, I'm keeping with the history thing, this is Larry Lessig. He wrote that code is low, because the person who writes the code is the person who decides what the user can do and cannot do with the computer or with that mind bicycle. And if you don't have open source, free software, and access to the code, and the right to change, and edit, and all the code, then you don't have freedom in the digital world. Means that someone else is writing the code for you and that someone else is deciding what you can do or cannot do. So this is extremely important to remember. I'm sure you all do because you write open source code, but it needs to be reminded constantly to ordinary people who forget. They get devices, and they don't realize that the code that is important is not always open. In the years 2000, something very significant happened is what people call cloud computing, because, well, that's not the best way to describe it. I would rather say software as a service. And they all use software as a service with the Dropbox and Google Maps, Gmail, Office 365, Facebook, Evernote, anything. It really is very interesting. It runs on a server. You need to have a web browser, and that's all you need, just a web browser. And you can use the software, and you don't have to install anything apart from the web browser that anyway bundles with your computer. That's really nice from a user's standpoint. Now there is a significant issue with that is, with this, you don't have control over the software that you use. You do not have, generally, access to the server. And you can't change the software on the server. Can you change Google Maps? No. Can you change Gmail? No. Can you prevent it from reading your email? No. That's the way it is, because you do not have control over the software that runs on the server. So that's a very significant issue. In 2007, something interesting happened that was the birth of the smartphone. And I'm sorry, it looks like I'm a big fan of Deep Jobs, which is not the case, but anyway. So the smartphone is always connected. It's still a computer, but it's always connected unless you really need it. When you really need it, I don't know if you've seen that, but it always failed. But I don't know, maybe it's just me. The difference with the initial Apple II, apart from that, is it's full of sensors. This thing generates data like an Apple II, much more than an Apple II, because you can take picture with it. It has tons of sensors, a camera, microphone, accelerometer, compass, all of that, and it generates data and sends it to the internet. And you always have it with you, and it's always on. And these things sell more than PCs for four years now. Since 2013, smartphones are more popular than PCs. The PC is slowly dying. The smartphone is still growing. It's growing not as fast as before, but still growing. And now today, people use three kinds of computer. The PC, and not all people use a PC now, smartphones, and clouds. That's a fact of life. We all have three different kinds of computers. But there are a few things about the cloud that is not being said, and I think we need to say it to the world. The first one is actually, it's a German slogan from Free Software Europe. There is no cloud, it's just someone else's computer. Remember that. If you use the cloud, it's not your computer. You don't have control over it. And so someone decides what you can do or cannot do. I mean, Gmail would be totally fine if Google didn't read my email. Google Maps would be totally fine. It's a great piece of software. If I could change it. Another thing is that many people think that data is the oil of the 21st century. That's what they have in mind. And so they do stuff. They come and pump the data, pump that digital oil directly from our pockets. So I'm sorry, the screenshot on the right is in French. It was on my Android phone back in the days when I used Android. And I wanted to use Facebook. And so I started installing Facebook. And this is the list of things this piece of software wants. And it's amazing. So let's review this history of apps and device identity. My calendar.  I mean, seriously, people. Why? And contacts. So this I know because it takes my contacts and uploads it immediately to Facebook in order to create shadow accounts for future users. Position, GPS position. Why? But why does it want to know where I am? He wants to read my text messages and know who I have called and access to all my files, photos, and movies. And then there's this green button that says Accept. It's green like a green light. Like green like hope. Yes. And no. I mean, no. I cannot accept to do that. I just did not click on that thing. I stopped the application and never run it again. I am not willing to surrender all my data for what? Kittens. I mean, I love kittens, but I don't need to surrender that data to kittens and CD things. And talk to my friends, right? And say, you people say, yeah, but you know that Tristanidogai, he doesn't like Facebook. True. But Google either. So this is Google Maps. Well, actually, this is data on Google Server because I use Google Maps. So that was on the 2nd of December 2014, was it? I don't know, yes. That's the day I decided to write my book. And so this is Paris. And this is all the data points of where I was that on that day in Paris. There are moments where there are lines. It's because it cannot track me in the metro yet. But they're working on it, right? And this is on Google's server for every day of my life because I had an Android phone at the time. And so Google yet is tracking me on everything that I do. And there's that question. Are we really the clients, the customers, of these internet giants? And so I will show you a photo of two of my friends. These are Alice and Bob. Alice is on the left. Bob is on the right. And they are really cool pigs. And Alice says to Bob, you know, Bob, I love this place. I love this place because the food is free. And Bob said, yeah, yeah, Alice, you're totally right. The food is free. The place is amazing. And you know what? The lodging is free. We can sleep here for free. This is so amazing until you realize that this is not a gift that the farmer is doing to pigs. It is not a gift. The pigs are going to end up in French or German sausage. And this is exactly what happens, except that normally Google will not slice into pieces, neither Facebook, as far as I know. But the truth is, they are like farmers to us. We are not the customers. The pigs are not the clients, the customers of the farmer. We all know that. And some people stop eating food or at least eating meat and pork for that very reason. And in the cyber world, if you will, the users of Google and Facebook are not the clients of users. It's not the clients of Google and Facebook. We are the thing that they sell to the actual clients. And the actual clients are the advertisers. So this is something we need to tell to the world because people don't get it. They do get it about the pigs and the cows and the chickens and all of that, but about Google and Facebook, they don't get it. And the worst is that we get a terrible deal, right? I did the math. And if you go to public information about Facebook and you see the amount of money they spend every year and the number of users that they have, you do the math, it's easy to do. And you see that for slightly more than 3 euros a year, they can provide. It's the cost of providing the Facebook service for one person. And of course, they make a lot more money than that from each of us. So all your data that you give up, it's only for 3.33 euros a year. That's a terrible deal. So this is Edward Snowden, a whistleblower who blew the whistle in June 2013. He was a CIA employee and then he was an NSA contractor. NSA stands for National Security Agency in the US, which goal is basically to do mass surveillance on the world. And here are a few slides that you can find on Wikipedia that were revealed by Edward Snowden. So on the right is just an example. It's called Dropout Jeep. So basically, Edward Snowden was an American patriot and was extremely concerned to see that the NSA, his employer, was doing things that were illegal in the US. And as a citizen and a patriot, he could not stand, he could not accept that, he could not work, he couldn't see his employer, couldn't accept his employer, would violate the American Constitution. And so he decided to give tons of documents he had found on servers because he was a citizen of the servers. And so he took a lot of PowerPoint documents and gave it to reporters in June 2013. And so Dropout Jeep on the right is basically, it's malware, you can be installed on iPhones and pretty much collect all data and send it back encrypted to the NSA. On the top left is a magnificent post-it note where they explain how they manage to hack the Google servers and access data so that they can have it, gather information that pretty much every Google user. On the bottom left is actually, and I'm sorry, you can't see it better, this is how they collect data from cables that run under the ocean. And if you could zoom on it, you would see that in France, in the south of France, in Marseille, there is spot where there's a cable starts and heads to Singapore. And it goes, of course, through all the Middle East countries where they connect to the internet and they listen to the traffic on that cable and sort it and copy it and analyze it. So there were thousands of documents like that in the Snowden revelations. And we're still discovering new stuff and you see new articles about it. And it basically, it comes down to one thing.  It would be extremely costly to do it, but what we do is really crazy because we put all our data in the hands of a handful of companies, Google, Amazon, Facebook, Apple and Microsoft. Just like these huge silos here, we put data in a handful of places. And so we're making the NSA's work a lot easier by centralizing this. So in short, the Snowden revelation explained us that centralizing data is what makes mass surveillance economically possible. If we had our clouds, personal clouds, each of us would be extremely complicated for the NSA to create mass surveillance. It would be too expensive. They wouldn't do it. But because we centralized data, it's possible. And yet, usually, I hear always the same answer, yeah, but I have nothing to hide. So why do I care? Think about it. Does it matter? I think it does. Mass surveillance is toxic for democracy and freedom. And I'm going to explain this to you why it is. Because when we are under surveillance, we tend to change our behavior. We tend to self-censor. And in many cases, we refuse to acknowledge this. But we actually, we do. There was a very interesting study that shows that if you ask people if they change their behavior when they're under surveillance, and the more the people say, no, I do not, I really do not change my behavior, the more they do when you put them under surveillance. That's really mind-boggling. So here is a concept called the panopticon that was studied by a French, well, a couple of French philosophers, Foucault, in a great book, explained how when under surveillance, we change our behavior. And this is a prison in Cuba. It's an old prison. So imagine it's a circle. And cells are in the around this. And in the center, there is a tower. In the tower, there is one guard, or a couple of guards. And they are looking at the prisoners. And the thing is, because the windows are really small, you never know if you're being surveyed or not at one time. You know there is someone, he may be looking at you, maybe not, but you stay quiet. Because otherwise, you get into trouble if you do noise or stuff. And the beauty is, the guard can go and, I don't know, go to the restroom or get a sandwich. And you still behave correctly, because you think you're under surveillance, even though you're not. So this is explained by a French philosopher, Foucault, and also Deleuze, who basically explained that this notion of panopticon, the way that you can survey everyone from a single place, which has been applied by, it was invented actually by English philosophers back in the 19th century. They invented it for prisons, and they also invented factories so they could survey workers, make sure they didn't steal stuff. And basically, this panopticon thing, it's not to see without being seen. It is to impose specific behavior to a human group.  And there is an example, a real case study about that. This is the number of page views for Wikipedia US pages that are related to terrorism, which is a very important topic. We talk so much about terrorism. And you hear about it in the news, and you especially in the cable news, always the same thing, you see pretty much nothing. You see cops. You see fire. You see dead bodies over and over. Little, very little room for thinking and explanation, because they know nothing to just reporting as it happens. And so people go to Wikipedia to read about terrorism. And it's good. I mean, citizens could go and fetch information from Wikipedia about that. It is absolutely legal to go to Wikipedia. And I think it would be recommended to learn more about terrorism, to understand the issue, because it is so important in our society today. And so as you can see, over time, people go more and more and visit more and more these pages in Wikipedia about terrorism. And it's good. And it increases, as you see the black bar, it's getting more and more. And then something happens mid-June 2013, the Snowden revelations. And Snowden revelations say, yes, we are under surveillance. And suddenly people go a lot more, a lot less, sorry, a lot less to Wikipedia to learn about terrorism. It is still 100% legal to go and read Wikipedia articles about terrorism. But people change their behavior. And they go less and less. And over time, as it decreases, they don't go to Wikipedia anymore to read about terrorism. They're concerned. Maybe I should not read about that. Maybe it's bad. Maybe I'll get into trouble. So this is exactly why when you are under surveillance, you change your behavior. And in most cases, you don't even realize. You think about going to see the Hamas or Al Qaeda or whatever page on Wikipedia. And then, ah, no, I'll go see some kittens instead. You see? It really is just a tiny moment when you change your mind. And so the question is, should we get rid of these wonderful bicycles for the mind? And my answer is no. My answer is no. So what do we do instead? So in my book, I explain seven principles to build systems that allow people to regain control of their future, of their digital future. And I'll go through these. First one, do not rely on targeted advertising as a business model. Otherwise, it won't work. Because that's the whole thing. Google and Facebook, their businesses, are just about doing targeted advertising. This is why they gather so much data. They want to know everything about you so that you always stay with them. And they can sell targeted advertising to advertisers about you. So do not use targeted advertising as a business model if you're building something. Second is rely on hardware you can control. Either you self-host your computing and your cloud computing, or you delegate this to a third party you can trust and that you can pay. Because the third party won't do it for free, and they won't be able to rely on targeted advertising, principle number one. So you need to give hard cash to the people who run the hardware, buy the hardware and run the hardware. And also it's decentralized. It has to be decentralized. Otherwise, we're getting into the same issue as Google and Facebook. Number three, use free and open source software. Why? Because you have access to the source code. So you can audit it or have someone audit the code for you. It's transparent. You know what the code is doing, and it's adaptable. You can change it to fit your needs. You're not at the mercy of someone who decides for you what to do with the code. Number four is use encryption so that the data is safe when it travels from one point to the other and where it is stored. Number five, have a great user experience when you build something. And I know you're working on that hard at NextCloud, and you have a special designer that helps you building the apps. And that's really, really important. Having a great user experience. People will not leave Google Drive for a terribly looking product. They want a nice experience. They want something that is nice and cool and pleasant. Number six, interoperability thanks to standards. If we start building new systems with diversity, we need to be interoperable. Otherwise, people will stick with Google Drive because pretty much everyone needs to interface with Google Drive when they build a product. So you need to make sure you're being interoperable. And finally, a killer feature, something that makes the product so sexy, so cool, so innovative that people will look at Google Drive and say, Jesus, you're still using Google Drive? That's terrible. You can't do this or that. Now, it's tricky. It's easier to say than do, right? But we need to find something. And there are many different solutions. And this is something I'm going to discuss now. So this is a slide I have in pretty much all my presentation. It's not something I did today to please you guys. And actually, NextCloud is always the top of my list. And Cozy is the last one, so they remember Cozy better. And so I list different projects that do something which is in these seven principles. NextCloud being number one, Solid, which is led by no one else than Tim Berners-Lee, the inventor of the World One Web. It's an MIT project, very promising. There is a French-Belgian project called YUNA host, which is also free software. It's a real self-hosting solution.  I will give you a few words about CozyCloud so that you understand better what it is. Just like NextCloud, it enables you to sync all your devices. But what makes it very different, that we put a lot of focus on having connectors that allow you to import data from the various sources, all the brands that have information about you, that your energy provider, is it your bank, is it your shopping service, is it LinkedIn, Amazon, whatever, and import that data on your personal cloud, data that can be accessed from all your devices. And then we put apps that are data-oriented apps. Now that you have data in a single point, which is yours, under your control, what are you going to do? And this is where we need to be creative. What we do here is we enable brands to create apps that can interact with data locally without getting a copy. We do not allow them to get a copy of the data. But they can offer service that are tailored, customized for the user. So the difference is we are both open source and decentralized. One major difference is that COSY is absolutely a single user. So in this, we're not competing at all. It's a very strong differentiator. COSY is not for groups. It's for individuals. You can have several individuals next to each other on the same machine. You can share files if you want between them. But basically, you will see it's a single user. Just like a smartphone is a single user device. And we also have the ability to run third party apps in COSY. I'll do a very, very quick demo. I don't remember how much time I have, I think. Not much? OK, because we started really late. I'll go very quick for the demo. So as I said, single user, you go to a specific URL, which is yours and only yours. And so we don't ask you for a username. We just ask you for a password. If you have the password, you are the person.  Because you have the device and you have the code. Then you have the URL and you have the password until you can log in. And so this is a set of screenshots that I did yesterday. It's the current product. But it's not a live demo because I don't trust Wi-Fi in places like that. So it's a drive. Just like you imagine, you have folders, you have files. And you have a button on top right, which is apps, which is a list of apps that we are working on. And so I'm going to COSY photos. You have the photos. You have an album feature. And once you go into an album, you can also share it. I'm going to another app now, which is the COSY collect app, which is the connectors application. So I have different categories of connectors. This is the list of connectors to connect, to collect my phone bills. So I can pick if the friend, this is with French operators. So I could import data from these mobile operators. I need to give to COSY username and password. And as you may see here, it says import your bills. It just imports your bills. But we'll see. This is the French electricity provider. And it's different here. You can import a lot more than just your bills. You can import your bills, your contracts, your consumption, your client profile, and your home information. And all the information, the files here, the bills are collected and put into a specific folder. Of your drive. So it's a lot easier to access your bills from your mobile operator, electricity provider, and such. And now we will go to the banks application, where you can see, sorry, this one is not localized into English yet. You can see how much money you have in your bank. It can connect to several banks, different banks, and do a global view also, which is very interesting if you have multiple bank accounts. And I can see I have a bill here, which is about a phone. It's a phone bill. And I have a link here. And if I click on the link, it takes me to the PDF that is automatically downloaded by the connector. So I can really access my data very quickly from my bank account to my bills. Another example, and this is very French-like, if I click on the health expense, it will take me to this application. And the French system for health reimbursement is very complicated. You have a state insurance that gives you back money. When you spend money, you go to a doctor. You spend the money. And then the state insurance, social insurance, will give you back some money, but generally not 100% of it. It will give you a share, just a share. And you get the money on your bank account. And you can track whether you've been reimbursed by the state insurance or not. And then on top of that, you have a secondary insurance that will add another sum to that. And it really makes it complicated to track. I went to that doctor on that day. I have been compensated by the state insurance or not. And so have I been compensated by the secondary insurance or not. And then you can track. There is this application that enables you. So you can see you've been two sources here of reimbursement. And you can see what has been reimbursed and what has not yet been reimbursed. This is pretty much it. Well, thank you very much. And I hope we have a couple of minutes for questions. Thank you very much. Questions? Frank? Thanks for the great talk, first of all. Really, really impressive. So I think most of the people here in the room agree with you that privacy is really important. But it seems that a lot of people out there don't care that much about it. I mean, there's no revelations out for a while now. And there are some people who think about it. But the majority not. So what can we do to basically push this forward or to tell people that I should care, that I really should think about alternatives?  I mean, I'm just fine with the situation as it is today. Thank you. That's a question. What can we do if people don't care about privacy? Well, first, I've done my share. I wrote that book, which explains what I explained to you this morning. And by the way, if you know of a good publisher in Germany, maybe we can translate it into German and sell it here. By the way, I give all my money to a non-profit that promotes privacy in France. All my money I made with the book. So I'm not doing it for the money. But I think it does. The topic is really important. So explaining, teaching, I give talks. I get invited on radios and TVs. And I do all of that. I do a lot of evangelization on this topic. And I think I preach to the dessert more or less. Well, there is some interest. The book is selling pretty well. I wish I could do more. The second thing, and by writing the book, it's telling people, but also explaining to people that already understand is giving them arguments about this, how to explain it to people that privacy matters. So you know it. You feel. You feel like privacy is important. But if you read about it, then you'll be better at explaining why it's important. So what I am doing, it's basically what I did with Firefox. I'm explaining Firefox to. I used to explain Firefox is really cool.  And I would explain that to people. And they would explain that to others. And they would install Firefox on somebody else's machine and stuff like that. The second thing we need to do is build great products. It's build alternative products that are really cool. And this is exactly what you guys are doing. Build a very good next-cloud experience so that people will say, yeah, I love it. Data is on a server. Google's not reading our data and all of that. So building a great product is the second way of doing that. Because it's true. Not so many people care about privacy, but they do care about good user experience. That it really works. You may remember how Google won over Yahoo and Alta Vista at the time. It was just better. And people didn't know about Google until someone came and said, oh, you should use Google because Yahoo and Alta Vista, they are such craps compared to Google. And so they started using Google Search. And even before that, I vividly remember, what? You're still using Mosaic as a web browser. You should totally use Netscape. It's so much better. And then it was Internet Explorer, so much better. And then it went Firefox, so much better. And now Chrome is so much better. No, I'm just kidding. OK, so now building better products is the way to do. And by the way, Firefox is better. But you know that. You should try, otherwise. Did I respond to your question? Yeah. I wish I had a magic button we could press, but I don't. Is there another question? Yes, I'm listening. I'll repeat the question. That's another very good question. And I don't think you would like the answer. So and I'll repeat the question first, because I think people that are watching us on the internet, how do we balance the need of police forces and government to prevent terror attacks? Is that properly summarized? So first, I have nothing against police forces. I am not someone who dreams of being without police forces or anything. I don't hate them. I have even people like that in my family, so I'm totally fine. And I still talk to them. So really, no worries. Now there is one thing, major thing, and it's hard to understand, because we do not think, but we feel about terror. We feel, but we don't think. Let's just remember that the chances of being killed or harmed by a terror attack is less frequent than being killed by a light storm. It really statistically, it never happens. But when it does happen, we talk about it a lot. So it puts a big impression of us.  I think it's unacceptable. But the risk and the perception that we have are extremely different. The risk is very, very, very low. But this is what we talk about all day long in the news. And this is putting our society in a difficult situation. And I would say it's the beauty of terrorism. It works because it's scary. And it's more scary than effective. And actually, it's effective because it's scary, but it's not effective at killing people. Right? This is something which is not a very popular view, but it needs to be said. And then putting mass surveillance systems on society is actually helping terrorism winning.  They hate what we do, the way we live. They hate our freedoms. And if we start reducing our freedoms because of them, they have been succeeding. And it's not something that we want. So I am totally OK with police forces having targeted surveillance about suspects. I don't think having mass surveillance is what is going to make the Western society a good society. So let's sure not to, we have an expression in French, let's not throw away the baby with the bath water. We need to address terrorism issues, but we do not need, we should not do mass surveillance because it's going to make all of us very unhappy. You have to remember that because we tend to self-censor under mass surveillance, means that the creative people, the people that want to change the world, they will not emerge and the world will stay as it is. And I don't think the world is really great right now. I don't think the inequalities between people in society is good. And if we put mass surveillance in effect, dissidents will disappear. And dissidents is progress. That's it. Do we have a third question? Or was I too scary? Yes. Thank you. Thank you for doing stuff for privacy. So your question is, have we lost? Have we lost the privacy battle because, so let me maybe summarize your question. Have we lost the privacy battle because anyway, websites use Google Analytics and Google AdSense and AdWords and stuff like that. Now maybe. Maybe we have. Maybe we don't know we've lost. And maybe we haven't lost. Maybe it's hard. It certainly is hard. But maybe it's just a difficult moment of this. I don't know. But I know that we should not have the perfect solution to start using it. If there is one thing I have learned during my Mozilla years is that in Mozilla, things in 98, in 2003, the early years of Mozilla and Firefox, it was terrible. My job at the time was being an evangelist. I was trying to promote web standards to websites who told me, yeah, I use the standards. I use Internet Explorer, which is a standard. And they were building websites for Internet Explorer with all the bugs of Internet Explorer. And they were happy with the bugs because they were used to the bugs. And they know these features, you shouldn't use it because it's buggy and all that. And we had like 0% market share. And there was terribly distressing to do that. But in the end, we've won. That's the one thing we've learned is that you can build a future you want. You are, I think, a developer, or at least a fan. You understand technology. You can change the world because you can write free software which is distributed for free on the Internet. It's amazing. You're building tools and they can spread. You have a lot of power. It's not easy. And you should use an ad blocker, by the way, because really, this is easy. And you should tell everyone around you to use an ad blocker. I used to say not to use an ad blocker, but I think we need to resist the advertising business now because it's like cancer to privacy. And keep doing what you do. Do we have time for another question or yes? Last one, then. I don't know. I used Firefox, which last time I checked was not owned by Google. I use Ublock Origin, which is free software.  You can try Privacy Badger. You can try Disconnect.me and stuff like that. And you should totally not use Chrome. That's for sure. Maybe you don't know, but Chrome takes all your browser history and sends it back to Google. So what's the point in using an ad blocker if your browsing history is sent to Google? So stop using Chrome.  And use Ublock Origin with it. And you'll be like, you've done 90% of the job just by doing that. And it's easy, really. So that was the last question. Thank you very much.


Dear Feminists, would you please stop ruining our society? Good men matter. Good men have a valuable and necessary role in any society, including ours. Good men provide safety and security for their families and their communities. They give their male children an example to model. And they provide their female children with standards that they're going to need as they mature and choose spouses of their own. They are excellent at problem-solving and innovation. They are skilled, engineered, and physically endowed to build the infrastructures that we all depend on. They put themselves in harm's way to ensure the safety of our nation and our communities. I could go on. A society without good men, without strong men, cannot survive. And yes, there are men out there who are predators. But guess what? The best defense against a predator is a good, strong man. But your constant mail-bashing has disincentivized good men from being good. This serves no one. It's gotten to the point that good men are afraid of opening a door for a woman, of standing up so that a woman can take a seat, or of doing anything nice that good men usually do. This literally is ruining society for the rest of us. Stop. If you feel the need to continue to mail-bash, if you feel the need to rile against men when they do kind and courteous things, why don't you just wear a pin or a t-shirt to let these men know who you are so that they know who we are? We appreciate good men. Stop ruining it for the rest of us. I think it's kind of funny how a lot of pro-choice, pro-abortion people will kind of flatter themselves and say things like, don't you dare try to control my body? Keep your laws and your opinions away from my uterus. Like, Karen, let's make a couple things very clear. Nobody cares what you do with your body. Nobody wants control over your body. And nobody wants to be anywhere near your uterus, believe me. Truly all we care about is a separate human being. A separate human being who has already been brought into existence that you want control over, that you want to dismember and ultimately kill. It's quiet ain't no bad talk. Navigating the relationship world for men is challenging. Here's why. With the feminism movement, we gave women a voice which has now evolved into ever-expanding allowances on their behaviors, actions and words. Men, however, have been left behind somewhat. And in many ways, it's been turned against them with ever-increasing limits on what they can say and do. I'll give you some examples. When a woman opens up to a man about his struggles, he's expected to support her. When a man opens up, he's considered weak. When a woman punches a man, she feels it's somehow okay. When a man punches a woman, yeah, that's abuse. Obviously. When a woman tells a guy he needs to change his body, she considers it in his best interests. When a man does the same, that's body shaming. It's bullying. When a woman brings up women's issues, it's accepted and applauded. When a man brings up men's issues, he's a mock. In three words or less, trigger an entire fan base. I'll go first. There's two genders.  And that's the only thing I care about. I've dedicated the last four years of my life to researching neurology.  So I don't know the difference. The way you feel is not a science. That's not fact. So if you can't prove me wrong with research with facts, go ahead by all means. That's how science works. That's how we progress. But you guys literally use your feelings as facts and that is not a source.  I do not owe you anything. You do not owe me anything. I do not need to validate you. You have every right to call yourself whatever you want. You do not have the right to force those beliefs on me. Again, I don't need to validate you.  I want my try. Oh, I freshman guy. What do you think? Oh, now you know I ain't allowed to think. So Cosmopolitan put out magazine covers of two women who are noticeably overweight titling it, this is healthy when it's not. I don't mind them being on the cover at all. It's the message I don't agree with. You could feel good about yourself at whatever size you are. But feeling good about your body doesn't make you healthy. Now, size has no impact on your value as a person, but it certainly has an impact on your health and these leptis magazines need to stop conflating the two. A lie repeated enough times would be believed and thereby repeated more. It's a vicious cycle. And the same people screaming, believe in science. It's about a multitude of things, but not this because feelings take precedence over fact if it weakens the need for personal responsibility. Now, wellness isn't a one size fits all, but it certainly isn't all sizes and obesity is unhealthy. Just like anorexia is unhealthy. We don't get to just redefine words because we feel like it. Now, we shouldn't shame people and their health, but instead encourage each other to a fit and healthy lifestyle. Not a complacency and gluttony. Stop glorifying unhealthy lifestyles in order to stay woke. I love all women. I'm a feminist. That's so cool. You must have been so excited about Amy Coney Barrett going into the Supreme Court. No. I thought you said you love all women, not when they want to take away my rights. Which rights has she taken away from you? She wants to ban abortions and killing a baby is a right because... because it's my body, my choice. Okay, so you're okay with me not wearing a mask.  Remember? But by not wearing a mask, you're literally harming innocent lives. Oh my gosh. It's almost like that's exactly what you do with abortion. I just don't think men should really have a say in the whole abortion discussion. Oh, that's really interesting. How come? They just realistically don't have to carry the same burden that women do when they're pregnant. Burden's a heavy word, but okay, you do have to agree that they are exactly half of the reason that the woman is pregnant, while it's half of the DNA that was created at fertilization, right? Yeah, sure, but they're still not as responsible for carrying the pregnancy. Okay, well with that point of view, let's say the woman does want to keep the baby and the father doesn't. Don't you think that just provides the perfect narrative for the father to leave because he's realistically not as responsible or doesn't have as much of a say anyways? Well, I will. Just provides the perfect scapegoat for men that don't want to take responsibility for the pregnancies of women. I guess maybe. I feel like this is the root of the issue with the lack of fathers in America because as soon as you give them the idea that they're not as responsible for a pregnancy or don't have as much say in the pregnancy, then they can leave and just not care about the whole rest of the life because they didn't have a say anyways. 7% of single mothers work more than 44 hours weekly. 24.5% of single fathers work more than 44 hours weekly. 7 compared to 24.5. 46.2% of single custodial mothers receive public assistance. 20.8% of single custodial fathers receive public assistance. 46 compared to 20. 40% of mothers reported that they had interfered with the father's visitation to punish their ex-spouse. They have literally like this kills me. Women literally try to punish their exes by withholding affections from their kids and visitation from their kids that they think that punishing the dad doesn't hurt the child. What's your biggest clue that the American college system has failed? I'll start.  That's amazing. Women be like, you can't depend on no man. No sis. You can't depend on your man. Man always comes through.


See, it can become powerful medium. I'm saying, at some point you can… if it is properly accounted for, you could vote through Twitter. You don't have to go to an election booth. If the platform is running in a very responsible way, look at this cost of the election. Look at the mess, look at the influence that's happening.  You could run democracy on Twitter. I am… I see a possibility. If it becomes a responsible structure rather than just a gossipy nonsense, there is tremendous power in it. One thing thing is, numbers will come down. So what? You're anyway not charging for the account. So what does it matter whether you have fifteen billion people doing Twitter who don't exist, only three billion people who actually exist do it? Isn't it better? You can become a powerful force in the making of a nation, in the making of world's democracies. Not… I'm not just saying outright voting itself in so many ways. Government wants to know something. We're going to do this, are you happy or not? Tomorrow morning everybody can tweet and say, I'm happy, I'm not happy about something. Various issues, policy issues, everything, if it comes to this, then you will become a powerful medium. Right now it's like page three stuff, because ninety percent of the people are talking. Some rubbish, because their names are not out there. I feel this anonymous social media must go. If somebody wants to say something, he must have the courage to say, I'm saying this. Hiding, putting your name in the front and I am saying something, I don't think this is right. All these people have five-ten accounts and they're saying their own rotten stuff because there's no accountability. I think this anonymous social media has to be curved. Everybody can say what they want, as long as they stand up and say it. I think that aspect should be brought about which would make the social media much more powerful and responsible instrument. Maybe the numbers will come down, but it will become far more powerful and responsible force in the world. Right now, slowly it's coming to a point where you think you have to dismiss what is said on social media. It's not true, it's all just something, because those numbers are increasing more than responsible voices. You must see the possibility of this tremendous tool of communication. And I'm telling you, only if there is responsible usage will it become a powerful medium. If you take away the anonymous factor, if responsible tweeting begins to happen, then it will become imperative that everybody must have a Twitter account because this will become your way of communicating with each other, communicating with the government, communicating with various agencies. You want something to happen, you don't have to go and protest on the street. If a million people tweak and say, this is it, people know this is what needs to be done.


The best theory is that the Chinese created a virus, COVID, in order to try to create a vaccine to oppose it, to try to see if they could create a vaccine that would work for all coronaviruses. The person who was working on these vaccines, we know his name, General Zhou Yusun. And he's developing this vaccine sometime in 2019, but he has to have somebody develop a mutant coronavirus that actually infects humans well. We think that's what COVID was, was developed in the lab to create the vaccine. We also know that Zhou Yusun got a vaccine and he has it created by February of 2020. And most people think there's no way you could have gotten it that quickly unless you'd been working on it for some time. We also know that this general dies mysteriously two months later. So there's a lot to be said here that what was going on is a creation of a virus, creating a gain of function coronavirus that would infect humans easily, and then creating a vaccine from that. And what happened is that it accidentally got out of the lab. We also actually know the names of the three scientists who got sick in November of 2019. We actually know the names. Some people say these are the patient zero and they got sick in 2019, November. And yet the Chinese didn't admit to anything going on until January. And even in January, second week of January, the Chinese are still saying, Oh, nothing to see here. We don't think it's transmittable between humans. Meanwhile, it's been cooking and growing for three months and they knew that wasn't true. How should science funding in this country change given the facts we've learned through this discovery? You know, this debate's been going on a long time. Some scientists engineered avian flu and made it transmittable through the air among mammals. And people were very concerned because avian flu is very deadly in humans, but fortunately doesn't infect humans very easily. So they had a big debate and they shut down gain of function research between 2014 and 2016. The problem was Anthony Fauci kept giving exemptions to all the people that were doing this research. And then in 2017, the pause on gain of function funding expires and they set up a new committee, a pandemic pathogen committee that's supposed to review things for safety. Well, all those things sound good, but then none of the Wuhan research ever went before the committee. In fact, the committee, the chair of the committee told me they only looked at three projects out of probably possibly hundreds of these gain of function grants that they were looking at. So what you end up having to have is I think you probably need a committee of scientists as well as people in the National Security Theater who are aware of what kind of weapons can be made from these viruses. You need them looking at it and trying to decide whether the taxpayers should fund this. I think rather than a blanket ban of just saying no gain of function, the problem is if you say no gain of function, Anthony Fauci already says all of these experiments which were obviously gain of function were not. So he just defines himself out of the rules. And I think they'll continue to do that. But I think you need a committee. And the other thing about the committee is it has to be independent. They have to be able to review any grant throughout government. And they also can't be the same people getting the money. So for example, this Christian Anderson who does Fauci's bidding to write a paper saying nothing to see here, he got a $9 million grant the next month. So you can't have the people involved with determining whether something's safe also be getting grants from the people that are judging the safety of. That's too incestuous to be a real oversight condition. But we think it can be done. Our hope still is that we can get a bipartisan bill out of this. We continue to try to work with Democrats every day to get them interested in the issue. The hardest part is just the lack of curiosity. The Democrats haven't really cared much about this. A million Americans died and Democrats seem to be blasé about doing anything about it. You also suggest in the book some sorts of controls over the actual materials that are used in labs to manipulate viruses or bacterias or other life forms. And we're in the synthetic biology age now and in some sense the gatekeeping is gone. I just wonder how do you think about the trade-offs there in terms of not stifling scientific progress while at the same time making sure we don't have rogue groups creating super viruses that cause a global pandemic? Most research goes to universities and through grant agencies giving to the universities. And so I think that can be policed through some sort of safety committee application of judging what is safe and what is not safe. But we live in an era you're right, an era of synthetic biology. You can order on the internet the RNA to create the polio virus. You can literally get on the internet right now and order the bits of it. If you know how to put it together you can create the polio virus from nothing. It's almost as if you're creating life. Now a virus may not really be life because it has to live in a living cell, but essentially you're creating something that you can put into and basically bring alive from nothing. So I think there should be some rules on ordering this. We don't let you buy centrifuges online. We don't let you enrich uranium in your basement. This is I think equivalent to nuclear weapons and how many people can die from this. And so even being a libertarian and not believing in many government rules this is one of the exceptions that I think government could participate in. The other thing is is since government funds so much of this, even as a libertarian there's no real restriction on how much you want to regulate government or regulate the expenditure of government funds. So I think there needs to be a great deal more done to this and probably 95 to 99% of it will be government funds that need to be regulated more strictly. Hey thanks for watching that clip from my conversation with Rand Paul. You can watch another clip right here or the full conversation over here.