Hi, everyone. My name is Anton Smith, I'm Director of Product at SpectraCloud. I'm going to talk to you today about something I find extremely interesting, and I hope that you're going to enjoy it too. It's going to be all about on-prem costs and how to slash them and unleash the beast. So let's jump straight into it. A little bit more about me. I'm a networking nerd. Basically, all my career, I've spent working with networking and other infrastructure-related technologies. Last three years or so, I've been working on Kubernetes and OSS. I spent some time at Canonical where I was actually the product manager for MAS. That's why I love talking about MAS or Metal as a Service. I love Stone Temple Pilots, and I'm currently the Director of Product Management at SpectraCloud. Okay, so looking at the agenda, what I've got for you today is, first up, let's actually explore the control plan conundrum. Why do we have this problem with bare metal as opposed to what we have in the cloud? After that, I've got a solution section where I want to talk about some of the different ways that you can solve that, and then I'm going to show you a demo of one way to solve it. So in order for this discussion to be more meaningful, we should also just revisit why are we here? Why are people doing on-prem and why are we talking about on-prem and bare metal? I won't go through all of these. I just want to touch on a few reasons, some of the biggest ones. I think the biggest one is probably around data sovereignty, data security, and compliance. This is simply where some companies or organizations don't have any choice. They actually need to have it on-premises. Take, for example, COVID data in Iceland. They don't have any data centers from Google Cloud or AWS in Iceland, so they actually had to process it on-premises. They were not allowed to take it out of Iceland. There's performance and latency as well, processing things near to where the data is generated, intellectual property protection. It's another big one. And then finally, organization, culture, and skill set. In short, there's many reasons why an organization might choose to do on-prem. Now, the trick is when you do on-prem, how do you do it well and get all the benefits that we get from the cloud? Now, just taking that one step further, we should also talk about why bare metal Kubernetes. There are quite a lot of reasons and performance is not the main one. There are actually several other reasons that are more important to do or why it's better to do bare metal Kubernetes. Now, the first one is that resource efficiency is better. You're not slicing up hardware into VMs and then putting them together again and making a cluster out of it. You should be able to run around 10% to 20% more workloads using bare metal instead of VMs. The total complexity is a bit lower because you don't have multiple orchestrators at work and in some cases, perhaps competing with each other. So take, for example, VMotion trying to do things at the same time as your Kubernetes cluster is trying to organize resources. Every host is guaranteed to be a physical machine and that's good because you have more direct access to the hardware. The cluster is able to read data more meaningfully. There's no sort of shield between it and the actual hardware, meaning that when you're on-prem and you have to manage all of that hardware, it becomes easier to get visibility on it. You get very fine-grained control of resources, so you can use the Kubernetes resource management capabilities as they're intended without something under the hood, perhaps CPU oversubscription or noisy neighbor where you have VMs next to each other that are actually competing for resources and you don't have to do things like CPU pinning in order to deal with that problem. So with that said, I think that there are many reasons to do bare metal Kubernetes. It doesn't mean that there's a right and a wrong and that you have to pick one or the other, but in some cases, people will choose to use bare metal Kubernetes. Now finally, there's another one there, Accelerators as well. It's also easier to pass things through like GPUs and stuff or just use them directly in Kubernetes rather than having to fiddle around with the VM layer. Okay, so there are definitely reasons to do bare metal Kubernetes or bare metal in general, but there are also some big challenges and what I'm going to talk to you about now is one that I call the bare metal control plane conundrum. And if we look at that, that arises because of the way Kubernetes is built. By default, it uses something called etcd or etcd, which is the key value store for Kubernetes and this is found on the control plane nodes in the cluster. These are the ones that decide where all the workloads in the entire cluster are going to go. Now you have two options here. You have a single control plane cluster where you have only a single box. You also have high availability clusters which most people want to have in production and you always need at a minimum three nodes for that high availability control plane. So what does that mean? So let's talk a little bit about what that means in different types of sites. At edge, it doesn't really mean much. You're going to have to run your control plane in your workloads side by side, especially if you only have one or three nodes, for example. However, for medium and large, it's a bit more meaningful and a bit more nuanced because there are different things we can do.  Okay, so let's take a look at large sites first. Now, in the large site, I'm just going to go with a contrived scenario of 900 machines and 100 clusters. Now, let's say we go there, we create 100 clusters. We want them all to be highly available, so there's 100 highly available control planes. And that means that in total that 300 of those are going to be control plane nodes. So out of the 900, we lost 1 third to the control plane, and only 2 thirds is available for the workers. And if we dive in a little bit more detail there, then what we're going to see is actually a lot of resource wastage. So just to hit that home, assuming that these servers each have 128 cores and 512 gigabytes of RAM, then that would mean that there's 38,400 cores allocated just to control plane and 150 terabytes of RAM. Now, clearly this is not the best way to use these resources, right? We want as much as possible of these machines to be focused on actual workloads that are related to the business. So what about medium sites? Here it's even more difficult, I would say, because medium sites could be somewhere around 9 to 12 hosts, and I'm going to use 9 here just to really illustrate the point. So at a minimum, 3 of these machines have been lost for the control plane, and we have very limited flexibility and as a trick question, how many HA clusters could you actually build here? Now I'll just show you. So if we take one control plane here with 3 nodes and we take another one, there's only 3 nodes left and we can't actually build any more highly available control planes. So we'd only end up with 2 clusters and only 3 out of the 9 machines would be doing workloads. There's actually a much better way to do this, and I'm going to show you that now. So what we'd like to do is move to control plane pools. So instead of all those red nodes being taken fully one third, we do this. We end up with approximately 5% of the nodes being allocated for control planes. But how can we do that while also observing things like availability zones? Because in a data center of this size you most likely want to have those. So the wanted position in an abstract way looks like this at the end. About 5% of the nodes we want to hold back for control plane and that's entirely up to the organization. I've just picked 5% for today. It could actually be less when the machines are so powerful. And you want the majority of it to be available for worker nodes. And so what you'd end up with here is this slice of the machines would host all of the control planes and this is what we call control plane pooling. So I could put them into a resource pool and then all the rest of the machines will stay as bare metal worker nodes. Okay, but how can we do this? I'm going to talk about three different types of solutions today. One is virtualization, the other is heterogeneous hardware and the other is allowing workloads on control plane nodes. Now with virtualization there are pros and cons to it. So the pros are that you have full freedom to allocate resources flexibly and it's easy to move the control planes around. The cons are that you need a virtualization solution and there's no bare metal management. Most virtualization solutions don't give you a bare metal fleet management system and of course you have licensing and other types of costs associated with the virtualization solution. Now heterogeneous hardware, here you can write size to hardware. So what you're saying, today is I'm going to actually allocate or buy separate machines like smaller boxes which are right sized for the control plane function. They're not big 128 core things. The problem with that is that it requires managing different types of hardware which is going to mess up your logistics and your spare parts management for example as well as being inflexible. So if on day one I said I want to have 100 clusters and then day two I say I want to have 200 clusters I can easily go and just deploy another set of control plane nodes. And then finally allowing workloads on control plane nodes which is a common technique and is absolutely necessary at the edge where you don't really have any choice. It does give you more flexible usage of resources but at scale in a large data center this is actually untenable. It will create very messy operations and it can create resource contention requiring you constantly to be performing very careful management. So what I'm going to do now is dive a little bit more into virtualization. That's a bit the topic of the day today. How can we do that together with bare metal management? So imagine that I told you that you can just have this sprinkling of virtualization plus full bare metal lifecycle management for all of the machines and without the costs and the complexity. And here's the solution that I'm going to talk about today in a demo. So it's called MAS Metal as a Service. Now the thing that a lot of people don't know is that MAS can actually do VM management as well in a lightweight way. That lightweight way is using LexD another canonical project that allows you to manage VMs on machines on Linux boxes. Now the setup then that we're going to end up is we're going to build a control plane pool which has VMs that are managed by MAS together with LexD and we're going to have another pool where we keep bare metal workers. So this is going to give us the control plane pooling and avoiding the overutilization of hardware for control plane. Now MAS also has an API, a very rich API. It has Terraform. It has a Cappy provider that was built by SpectroCloud and is open sourced as well. And it also has a beautiful UI that you're going to see later on. So when people normally ask me what MAS does exactly which was interesting party conversations I said it does this. It puts operating systems on to and debopses. So imagine I gave you a USB stick and it's your first day on the job at a data center and I asked you to take that USB stick and put Ubuntu on all 900 of those machines. Well there is a better way and that's basically what MAS does. MAS works with a wide range of different hardware it's open source and you can build and provide your own custom images or you can take some of these operating systems where they publish nice images or ways to build images for you. But bare metal is not only about provisioning. Day zero and day one matters but as we know from Kubernetes it's not enough. Day two is actually where it gets really hard it's kind of like running a marathon. So MAS doesn't just provision operating systems onto boxes it does a really wide range of other functions that are all about day two. And the thing is with hardware you have to manage the hardware it has physicalities to it obviously but there are things like hard drives failing or new boxes coming that need to be set up or boxes replaced. So MAS offers all the day two capabilities that are needed to manage bare metal. So the first is obviously that it has automation it has a number of APIs or it provides an API upstream but it also works with a number of other APIs to work with devices so it can manage say the power or do network booting and provide network services with IPAM, DHCP and DNS. It's also very fast it only takes a couple of minutes to completely provision a new box it has inventory management so that you're able to see every single device that is in the data center every single graphics card every single network card the serial numbers, the drives etc and this is very important when you're dealing with say hard drive failures it supports storage layouts and it importantly also supports hardware testing which means that you can test boxes before you declare them ready to become part of the data center. Now it also supports bare metal DevOps so as I mentioned it has Terraform but it also has Ansible, Chef and Juju support which is also from Canonical it has a Python binding which is very nice to work with it provides network monitoring it provides LDAP integration it also has support for composable systems like Cisco UCS, HP Moonshot and more and it also supports cloud metadata or cloud init which is actually some of the secret source for supporting CAPI with Kubernetes. Now lastly it also supports this KVM micro cloud integration which is LexD. Now that KVM management is provided by LexD so there's an integration out of the box when you deploy boxes with MAS you can actually choose to simply enable LexD and so this gives you virtual machine and system container management it's Linux based it has itself a very rich API which is what MAS uses to integrate with it it has very flexible storage and network configurability it's easily deployable by MAS and it's very easily managed by MAS which I'm going to show you. Alright time for the fun stuff demo time so what I'm going to show you today is based in my home lab so I'm going to be keeping it small unfortunately I don't have 900 servers under my desk I have six. What I'm going to do is I'm going to create a resource pool bare one, bare two and bare three these are three physical machines that are going to be added into the control plane pool and on each one of those I'm going to create three VMs and I will use these VMs to draw or create a control plane three times there will be three control planes one, two, three and I will take one machine from this resource pool down here called WP or worker pool M1 through M3 and I will allocate one of those machines as a worker to each one of those clusters so at the end of it what you're going to see is three clusters each one will have a highly available virtualized control plane and each one will have one worker node in it now I will also be using CAPI together with MAS so the CAPI Open Source CAPI project by Spectro Cloud I'll be calling that using our palette tool and this will integrate with MAS API and do all the magic and so at the end you're going to end up seeing this you'll see as I described one cluster here drawn across the availability zone so it has one virtualized worker control plane in each of the three availability zones you'll replicate that three times and they'll get one worker node each okay so the first thing we're going to do is take a look at the MAS interface I'm going to give you a really quick tour what you're looking at right now is the machines page so this is showing all the machines that we have available to us bare one, bare two, bare three as I said I've prepared them I also have M1 through M3 available now all of them are in this status called ready which means that they're actually available to be used and provisioned we can see things like how many cores and how much RAM they have available on them as I mentioned this is my homelab so they're not gigantic boxes but they're big enough for today we can see the KVM section over here which is where we would be able to manage the hosts and create the M's if we want to and if you click on one of these boxes let's take a look at bare one dot MAS we have some information about storage memory how many CPUs it has we can initiate tests on this we can see how the power because we need to power cycle the boxes we can see how that's managed in this case it's done through webhook there's networking information available here and also other system information here now of course we have more detail if we want to go into storage we can see as I mentioned the inventory, all the PCIe devices we can see USB interfaces we can see the results of what we call commissioning in MAS these are a bunch of tests you can also define your own tests but even things like capturing LLDP information from the network to see what else is there is saved here and so all you would need to do is go and look at the details of this test or the outputs and you can see everything that's going on now in this case there wasn't so much because I don't have anything on my network so as well as logs for this data of the machine as you can see I've been preparing this demo so this machine has actually been used and has been recently released back to the pool of available machines so yeah this is the MAS interface and so now I want to start showing you how to actually provision some of these machines so the first thing I want to do is also show you how to filter things so I'm using the tag today demo so I can easily filter on a whole bunch of information here which is super nice these are the six machines as I mentioned now let's grab all of them one two three bear one through bear three and we simply say I want to deploy those machines so remember I need to deploy Ubuntu on them to like bootstrap lexd so we can then create the virtualized machines so all I have to do actually is select register as MAS KVM host with lexd just like that and I'm totally fine with Ubuntu 20204 for today I could also select other kernels if I wanted to do something special or other types of operating systems including custom images I'm not going to do that right now if I did want to provide some cloud init data I can do that here but right now I'm just going to start these machines deploying and straight away what you see is that it's starting to power the machines and as part of this boot process because MAS is managing the network it's actually going to intercept their DHCP request and it's going to initiate pixie boot now all of these machines are configured to use pixie when they boot up and that's how they're going to get their operating system installed okay so after a while about five minutes all these machines have been deployed with Ubuntu which is really cool they're all on and they're all available now they've also all had lexd installed on them automatically for us so we can go straight to the KVM tab here and if we click that we actually see each one of those hosts is available as a lexd host and we can see that they have some resources available that we can use to create VMs so let's check out bare one and go to KVM settings the other thing I'm going to do is I'm going to over commit the cpu a little bit just so I don't have any problems with number of cores I probably wouldn't do this in production but you could if you wanted to I just have to do it because of my lamp now that we're all set up with that I can go to virtual machines and here I can actually start to create machines so I'm going to add a VM and I'm going to call this bare one cp1 I'm going to take two cores, two gigs of RAM and I think 30 gigs of space should be more than enough and I'll compose this machine and then what's going to happen is that mas is actually going to automatically commission it and we will see a new machine pop up in the machine listing page that says it's ready for usage okay so back on the machine listing page we can see that machine that we just created that virtual machine and here you also see very nicely that it's lexd because of the power driver is currently commissioning so mas still does this process that it calls commissioning which is discovering what it needs to log out the machine even though it did actually created itself it was actually created by lexd so mas still needs to go through the same process that it treats physical machines with lexd machines now I'm going to run away and configure all the other machines and then we're going to come back and we're going to see them all in this list when they're ready so here you see I'm looking at the three control plane nodes that we're creating on bare one two of them are currently in progress now if you want to allocate those to a resource pool you simply click here and you can allocate it now also as zones I'm grouping by zone right now so all of these are going to be in az1 so as I allocate those oops that was not the right one to allocate there I want each one of these control plane nodes so I take this and I put that into az1 and then I also take this and put that into az1 now we're going to use the zones later to do placement of these and ensure that the clusters are built across the availability zones the actual hosts themselves I'm not going to put into a resource pool or a zone because we're not going to be using them directly now as for the worker machines that's m1 through m3 here I am going to allocate those so let's put all of them into the worker pool like this like so and they're not in an az yet so they don't move around but I will so let's put them into availability zones as well so m1 let's put that into zone 1 m2 let's put that into zone 2 and m3 let's put that into zone 3 and so now you see these availability zones are starting to populate and so here we actually have everything in zone 1 ready to be used to be part of a cluster now I need to go and complete the rest of this for the other availability zones and I'll be right back okay I've completed that and you can see all the machines currently going through commissioning and testing phase they've all been created and added to the right resource pool and the right az so in a little bit these are gonna be ready and we can start building clusters okay so that's finished they are all ready right now and they're all turned off and just ready and waiting they're in the right control plane in the right pools and they're in the right zones now just a quick flashback to the slide so you can take a look at that what we've done is we've prepared our bare one two three we've got three VMs on each one of those and we've got our worker machines our bare metal machines ready so it's time for us to look at the cappy part of this and start creating some clusters alright so I'm going to use pallet the tool built by SpectroCloud for managing Kubernetes clusters this is the console you can see my testing from the previous couple of days a bit of historical information there about my clusters really quickly here we have the concept of cluster templates cluster profiles we call them I've prepared one in advance called mass bare demo this is going to deploy Ubuntu using mass it's going to deploy Kubernetes 1.28.3 it's going to deploy Calico and I'm going to use Longhorn for the storage so a fairly basic cluster profile here so in order to create a cluster we have to come in here and we say add new clusters add new cluster now in my case I'm going to select a mass type we also have others like AWS etc. but we are focusing on mass today and I have also created something called a gateway but we don't need to talk about that right now and let's give this cluster a name so let's give it the name bare demo cluster one and we go to next now we select the profile which is mass bare demo and we get an overview I have a chance to override some of the settings in here if I want to and now the important part we come to the cluster config so right here we have the master pool and the worker pool now I want three nodes remember we want a HA cluster we don't want the worker capability on there because we're using VMs behind the scenes and here I can actually specify the resource pool and the azs and you see it's picked these up from mass which is great so take the cp resource pool I will enable each one of these because I do want to have it spread across those azs and I'll change the cpu in the memory down to match the resources that I created earlier now in terms of the worker pool configuration here I just want one node so I take the resource pool wp and I'm going to take az1 because remember we had one of our bare manual machines in there and this looks good to me so I can just continue next there's some other options in here like keeping the operating system up to date enabling scans scheduling backups or back control I actually do want something here so I want to make sure that I will be a cluster admin on this cluster once it's finished so I'll add that there and now we'll validate this and what's going to happen is it says it looks good and now I say finish the configuration and what's going to happen now is that it's going to start working with mass the mass that has managed all of these is managing all of these machines and it's going to start creating a cluster okay great so after all of that and you've seen all the little spinning dials and things working while we were watching we'll end up with this which is three clusters all running quite happily bare demo cluster one down to bare demo cluster number three they've got four healthy nodes each which is the control plane three control plane nodes each and one worker node and if we drill into one of those we just take a look at say number three here we can see some more information we can see that it's healthy there's some things going on in the reconciliation because it works be a cappy if you recall we can see the stack here that we deployed we can see CPU and memory usage in total and if we click on nodes we can actually see all three of these here and the host names here bare three cp2 bare two cp1 bare one cp2 and m3 here now the reason why some of these are not matching exactly is because we only said that they're in different az's there's no guarantee that all of the control plane nodes are going to be numbered exactly the same they're in a pool within an az so they can be taken dynamically and so that means that these clusters are nice and happy right now and ready to start taking workloads and we'll take a final look here at mass and we can actually see that all of the machines here in the different az's have been allocated and deployed they're all running a custom image that we've created and this satisfies all of the needs of cappy these are kube8m clusters under the hood and we can also see the memory allocation storage and so forth as well as the resource pools worker pool and control plane so that's a very successful setup of three healthy clusters okay and so with that what we have seen the clusters being built we've created a control plane pool using virtual machines without any commercial software whatsoever extremely simple with all the best of bare metal management as well with complete flexibility to divide things between a control plane pool with virtualization availability zones resource pooling as well as having worker nodes running on bare metal so pretty much what I described to you earlier about a data center where you wanted to divide things flexibly and easily and even in medium sites as well I hope that you've seen in this demo that that was really nice and easy I will also add that we at spectro cloud use this technique ourselves we use it to get better utilization of our hardware and our labs during development and testing and we have a lot of interesting discussions with different customers who are also investigating the same way of approaching things without getting into purchasing expensive licenses for commercial software so again they want the best of both worlds to mix and match virtual machines and bare metal flexibly without the complexity of multiple different solutions so with that I will just pull up this next slide with a couple of QQR codes I would encourage you to try a MAS so go to MAS.io they have a nice tutorial that will let you try it just on your laptop if you're running Linux follow us on LinkedIn we're always posting a ton of articles interesting things not just about bare metal about almost anything Kubernetes and then finally you can find our resource center with the bear icon there and in the resource center all you have to do is tick bare metal and you will find all of the articles that we've written about bare metal which is quite substantial as well as all the other materials that we've created now one final tip add me on LinkedIn as well and send me a message I'd love to hear what you thought about this presentation as I'm always looking to improve as well as any thoughts and general ideas that you had around it and the very final final tip I get my hardware in case you're wondering how I have six servers under my desk I find it on ebay I just find some old small enterprise machines that the type that strap on the back of people's monitors and I just buy those and add a bit of extra memory and storage and then I'm able to manage them with mass which is really nice to have your own lab so with that I'm going to say adieu and I hope that you have a fantastic day and that you really enjoyed this presentation


So there is a double angle identity for tangent. It's given as tangent of two A is equal to two tangent of A over one minus tangent squared, all right? And you could prove that easy enough using the angle sum identities for tangent if you know that one. That is the left-hand side would equal tangent of two A which would then equal, you're gonna treat this as tangent of A plus A like so for which then you're gonna get tangent of A plus tangent of A. Usually it's tangent of A plus tangent of B but both angles are the same here. Then in the bottom you get one minus tangent of A, tangent of A. Again, the more general setting in the denominator looks like one minus tangent A, tangent A. And this then simplifies to be two tangent of A over one minus tangent squared of A which is then the right-hand side. So you get that easy enough but mentally we're not gonna use tangent of two A very often, right? Because really the identity you really wanna be thinking of tangent of A is equal to, of course, sine of two A, excuse me, two A and then cosine of two A. If you can compute tangent of, excuse me, if you compute sine of two A and cosine of two A, you can get all the other identities as well, secant of two A, cosecant of two A, cotangent of two A. That's gonna be the basic idea for which you could expand the numerators here.  Now it's not as obvious that these things are equal to each other. How do you prove something like that? Well again, it's not the worst thing in the world. We could times the top and bottom by various things. We could prove it. Basically what we have to do is we have to times the top by one over cosine squared and one over cosine squared. If we do that, in the numerator, you see that there's gonna be a cosine that cancels out. Then you're left with a cosine in the numerator. So you end up with two sine A over cosine A. That's great, cause that's a tangent. Then in the denominator, you'll distribute this thing. Cosine squared over cosine squared is gonna be a one. And then you get sine squared over cosine squared. Whoops, my handwriting's getting a little sloppy there. Sorry about that, cosine squared A, for which then that gives you two tangent A over one minus the tangent squared A. So we guess what I'm trying to say here is that when it comes to double angle identities of tangent or other of the trigonometric functions, you're probably good enough with just using the double angles for sine and cosine because any identity you could come up with involving them is really not that far afield from just using the identities of sine and cosine as well. So when it comes to proving trigonometric identities involving double angles, you're probably good enough with just double angle sine, double angle cosine. So let's prove a trigonometric identity exactly in that format. Notice we have a cosine of two A, we have a sine of two A, and then we just have a tangent, excuse me, cosine and sine of two theta, just a tangent theta on the left hand side. So I'm gonna start with the right hand side cause it looks a whole lot more complicated in the left hand side. So let's give this one minus cosine of two theta over sine of two theta. So you'll notice that the right hand side involves double angles, but the left hand side does not. So we have to use the double angle identity to convert from the angle two theta to the angle theta. Now, when it comes to cosine, there's a couple options. So I'm gonna hesitate to do that one. For sine of two theta is basically one option.  And so for the top, you're gonna give one minus cosine of two theta. What are your options again? Remember, this is both a blessing and a curse when it comes to double angle cosine. So you have cosine squared theta minus sine squared theta. You get two cosine squared theta minus one and you get one minus two sine squared theta, like so. So how are we gonna end up with a tangent here in the end? Well, if you're not sure, it might make sense to kind of work backward on this one. So that is if we start with the left hand side, this is equal to tangent. And recognizing that, we started off with a quotient here, which has some stuff. We might write cosine, excuse me, tangent as a quotient. So we end up with a sine theta with a cosine theta on the bottom. So we want sine on the top. We want cosine on the bottom, which we do have a cosine on the bottom already. Once you have a cosine theta right there, there is a sine on the bottom, that's okay. But in the numerator, we need to have some type of sine. And that kind of leads me to think that for the cosine of two theta, I wanna use one minus two sine squared. Notice that involves sines, right? So does this one. This one has a sine and a cosine, but how do you get rid of the cosine? E, it's a little bit scary. One minus two sine squared only has a sine. Also, notice I need to end up with just a sine. I don't need a difference of anything, which since there's a one minus here, that might cancel with the one that's right here. And so that's the form of two, or cosine of two theta we wanna use. We wanna use one minus two, sine squared theta. That's gonna be the one that seems most productive here. So then you get the one, well, let's distribute this negative sine next. So we end up with one minus one, plus two sine squared theta over two sine theta. And now we can start to see how things are gonna unravel here. You get one minus one, they cancel each other out.  In which case then the twos cancel, one of the sines cancel, and we're left with sine over cosine, thus proving the trigonometric identity we were looking for. And so then the moral of the story here is when you're trying to prove trigonometric identities with double angles, mostly just focus on sine and cosine. That'll probably be good enough. And when it comes to cosine, you have some options. And so you wanna use the option that seems to best serve you. Sometimes it can be a little bit confusing on which option to use because there's multiple ones, but look at your goal, look at where you're trying to get to, and then use that to help you decide what you're gonna do, and then you can prove these trigonometric identities.


Why are narcissists so jealous? Narcissists are very jealous. They experience feelings of discontentment, misery and misfortune triggered by a person's possession, qualities or luck. They experience feelings of bitterness and resentment. They experience anger and disappointment as though they have been treated unfairly. When the narcissists first met you, they put in a lot of time and effort to secure you as their next source of supply. They wanted your attention. They wanted you to take notice of them and regard them as something interesting or important. They wanted you to desire them. The more time you spent around the narcissist, the more they began to consider you as their personal possession. They began to consider you as something that belonged to them. They believed that they had the right to control you. They tricked you into giving your power to them so that they then had the power to control, to influence or direct your behavior or the course of events. They used and manipulated you for their own selfish needs. They demanded and persisted to always have their way in every interaction, every situation. The long you stayed with them. The more they began to see you as their personal property. They began to isolate you, hide you from the world and other people. You stopped doing the things you once loved to do.  You stopped going to the places that you once loved to go to. You did all of this to serve the narcissist in an attempt to make them happy. They didn't like it when you had friends or your own hobbies and interests. They wanted to keep you alone to yourself. They did this to control you. If the narcissist believes that someone else is interested in you, whether as a friend or romantic interest, it could mean the end of their ownership over you. If you have an interest in a hobby or activity outside of the narcissist, it poses a threat to the narcissist control. Any outside source of influence, whether it is a person, hobby, interest, movie or even an inanimate object, can be a potential threat to the narcissist's influence and control over you. These are all potential threats to the narcissist's influence and control over you because you are given your attention to something outside of them. Or something they have no control over. The narcissist wants you to be focused on their needs and nothing else. When you give your time, energy and attention to something outside of them, in the narcissist's mind, that means less time, energy and attention for them. They also want to isolate you to take away any potential sources of help or support that might assist you in establishing your own independence. Which would be another threat to the narcissist control. The narcissist wants to keep you in a false reality where they are the only thing of significance or importance to you. They want you to see it as though there is nothing outside of them. There is nothing greater than them. This is why they have such an obsessive need to control and isolate you. They also fear that if you were to experience something outside of them, you would then realize that they're not as great as they may themselves out to be. While you were in the narcissist world, they might have tricked you into believing that they were this all-powerful, all-knowing, attractive or intelligent person. But once you begin to leave the narcissist world, they then seem so small. They begin to seem so insignificant in comparison to everything outside of the world that they are designed to accommodate you. When you make an attempt to leave the narcissist world, they feel threatened because it makes them feel small. Anytime that you try to contact someone outside of their world or start something new, it is a reminder to the narcissist that maybe they are not as significant or important as they believe. It reminds them that maybe there are people outside of their world who are more powerful, more knowledgeable or attractive. Maybe there is something outside of the narcissist world that interests you far more than anything within their world. These are all threats to the narcissist influence and control over you. Just the thought or idea of you desiring to partake in any of this triggers the narcissist to experience feelings of bitterness and resentment, anger and disappointment. The narcissist acquired you as a source of supply to provide them with stability, to provide them with safety and security. They understand that they have no control over the quantity and the quality of the narcissistic supply that they receive. They understand that the amount of narcissistic supply that they receive can change. Their confidence, self-worth and self-esteem are dependent on the quantity and the quality of the narcissistic supply that they receive. To ensure that they have a steady source of supply, they must secure a reliable primary source to give them that supply. They become dependent on this primary source to compensate for their instability. But emotional intimacy and vulnerability is a potential threat to the narcissist. They fear and love intimacy. Intimacy to the narcissist means danger, pain or harm. So they feel intense dislike or disgust. For anything that involves intimacy or being emotionally vulnerable. Because of this, the narcissist will distance themselves from their primary source of supply. They will remain detached or separate. Having no particular interest or concern for their primary source. Although the narcissist may cause emotional pain and suffering to the primary source, the primary source is also seen as an escape. Something that is meant to support and maintain the narcissist. Something that is meant to empower them. Make them feel stronger and more confident in controlling their life. The narcissist represses the thought or idea that the primary source might have needs of their own. The narcissist is locked into a cycle of behavior known as approach, avoidance, repetition, complex. They initially crave and feel desire to have some form of intimacy, stability and security with you. But this is then followed by an avoidance of commitment and abandonment and neglect. As though they never had any interest or concern for you. The narcissist has to behave unpredictably and abandon you to maintain control. They cannot be intimate or emotionally vulnerable with you because that would involve giving up their control. Intimacy to the narcissist means danger, pain or harm. They fear and feel intense dislike or disgust for intimacy, stability and security. Yet this is also what they deeply crave and desire. This results in them approaching and then avoiding those who are significant to them in a process of incompatible cycles which lack any connection.  They cannot get the same fulfillment from the experience of intimacy, stability and security as you do. They cannot experience the achievement of something desired, promised or expected. And this is why they hate. This is why they get so envious and jealous when you engage with someone outside of their world. When you participate in a new hobby or interest. When you have been intimate with someone else. The reason why it preoccupies their mind to such a troubling extent. The reason why it dominates and engrosses their mind to the exclusion of other thoughts and absorbs all of their attention and interest to the point where they develop this obsessive and desperate need to restrict and control you. Where you cannot experience anything outside of them. Where they have this need to isolate you and keep you alone to yourself. The reason for this is because they cannot get the same fulfillment from the experience of intimacy, stability and security as you can. This is why they hate. This is why they get so envious and jealous. So bitter and resentful. So angry. It's because they cannot get the same fulfillment. They cannot experience it in the way that you do. They crave and desire those things. But upon consumption it produces nothing but intense dislike and disgust within them. This is why they are so jealous. This is why they are so hateful and envious. This is why they are so angry. And it is also the reason why they abandon or avoid you. Because they already know that sharing any form of intimacy, stability or security with you is not going to benefit them in any way. It is not going to fulfill them in any way. This is also the reason why they isolate you and try to sabotage anything that you've got going on. Because anytime that they have to witness you experiencing those things it reminds them that they cannot partake and get the same fulfillment. So by default all they can do is hate or take those things away from you. So they don't have to watch you experience all of the things that they cannot partake in because they already know it's no good for them. This is why narcissists are so jealous. Thank you for watching. I hope this video resonated with you. Please like, comment, share and subscribe. Click the bell icon to receive notifications for my future videos. If you would like to donate, my PayPal link is in the video description. Coaching inquiries you can email me at NargisFeverCoaching.com Check out the immersion dice in the Nargis Fever store where you can purchase your own Nargis Fever t-shirt, tank top or mug. The link is in the video description. Thank you for watching and I'll talk to you soon.


I'd often like to think of myself as solid, as strong, as faithful. I've even crafted a good story, my story, the story which I operate under. I've been told to see myself as a protagonist as if everything revolves around me, which works really well for me. But what happens when this narrative fails? What happens when people start to see my inconsistencies? What happens when the bottom really falls out? Am I still the protagonist? Is the story still about me? Am I still faithful? Am I actually steadfast?


Hello. So I think it's about the time to start. So hello everyone. So my name is Li Ma and I'm from AW Cloud. So I think most of you don't know AW Cloud. So we are a pure OpenStack player in China from 2012. And we help our enterprise customers to design, build and operate OpenStack clouds. So I'm the architect in this company and I will study with the focus on software defined networking and distributed systems. So today I'm going to talk about messaging and the RPC messaging patterns in OpenStack and why not RabbitMQ are occupied for us. And I will introduce an asynchronous messaging library called ZRMQ and also the distributed messaging driver in OpenStack, which is developed based upon ZRMQ. So finally, I will introduce some real world deployments that we have been done. So RPC messaging. As most of you guys know, it is a very critical component in any single OpenStack installation. So when your messaging system has problems and when it doesn't work with you and it doesn't scale with you as you continue to grow, so you may run into major issues with the rest of your OpenStack installation. So the other reason why is because most of the OpenStack systems depends on messaging to be reliable and to work with you all the time. So if your messaging system has problems, for example, like compute and conducts and all neutral agents if your messaging system has problems, all these systems is to work properly for us. So you can see that RPC messaging is very critical for us. So when we set out to deploy OpenStack in 2012, we chose RabbitMQ as it is official reference. But however, we found numerous problems with reliability, resiliency, and speed. So I know it's not just us have RPC issues. There are various topics that have been presented in every submit, so you can review them from YouTube and I'm not going to discuss with them in detail. So in conclusion, we really faced these problems. So when we deployed OpenStack in 2012, and our OpenStack distribution is based on PySX, and we deployed a single RabbitMQ instance at the early stage, and we relied on monitoring and operations to make it stable. And as time went by, we switched to cloud stream technology. But however, problems were still there because the deployment became much more complicated for us. So for example, RabbitMQ itself is a black box for us, really. And moreover, when you deploy in a production system, you need to deploy HAProxy as an external heartbeat method along with SpaceMaker, which makes everything highly available. And you also need to identify the ratio of the RAM nodes and disk nodes without any best practice guide. So moreover, we also kept pace with the community to continue to upgrade our RPC libraries to get rid of these bugs, which is reported in Launchpad. And finally, the monitoring and operations are still needed to check if the whole cluster is overloaded or malfunctioned. So the bottom line is here. We found that scouting a broker is not practical for us because we have a lot of customers and partners in the ITC industry. So they need to start small and scale online. So scouting a broker is really difficult. So we need to find some other solutions available for us. So let's review our requirements for RPC messaging. So firstly, there should be no single partner failure. And it should be something that be horizontally scouting. And it also should be reliable. So finally, we found that the RAM QDriver is the answer and it met all of our requirements. So in 2014, we launched our OpenStack distribution based upon S-House. And we totally switched to the RAM QDriver in OpenStack. And it has a distributed message architecture. It is horizontally scouting. And it has fast delivery. But unfortunately, in S-House, some of the objects don't switch to Oslo.messaging. So we need to develop it for each project. And also in this year, we stick on the RAM QDriver. And also, we have some improvements on performance and stability. And also, the RPC library totally switched to Oslo.messaging. So the RAM QDriver means zero broker, zero latency, and zero cost. So the RAM QDriver is a high-performance asynchronous messaging library and not used in scalable distributed or concurrent applications. So it really provides a messaging queue, but without a dedicated message broker. So the RAM Q library is considered as an intelligent socket library, which gives you sockets and that carries atomic messages through various transports like in-process, inter-process, TCP, multicast group communication. So it doesn't have these rigid defined structures like MQP does, like queues and exchangers. Instead, it has a single interface. A single socket interface is like bind, connect, send, and receive. So it is clear that anyone who is familiar with socket programming is likely to work with Zero Pro very well. And the most powerful stuff is the patents. So Zero MQ patents is implemented through a pair of sockets that has matching types like request and reply. It is a classical request and reply patents. It is synchronous communication because one client sends a request, it is blocked immediately, and until it receives the reply from the server. So in Zero MQ, it doesn't rely on the sequence of service that, and it is usually used in RPC and CSCon applications. So and the next pattern is pub and sub. So it is a published and a subscribe model and it is asynchronous. When pub socket sends a message, it returns immediately and the IO threads at the background does the rest of the things. And the pub socket cannot receive and sub socket cannot send. And the next pattern is push and pull. So it is a pipeline pattern and it is also asynchronous. So it implements a fair queueing to sort the request from the upstream. So and the Zero MQ also has some advanced socket patterns like dealer and router, x pub, x sub and you can also define your own. So I'm not to expand this part and you can review Zero MQ guide in this official upside. So here, what has Zero MQ brought to us? So firstly, Zero MQ is a messaging library. It is not a messaging system. So it allows us to define our own messaging models. So finally, we found that Zero MQ library can bring brokerage architecture. So it becomes possible to have a messaging layer for OpenStack without an essential broker. So let's see the Zero MQ driver in OpenStack. So it was introduced in Fossum. So really it has been there for years. But I know some of you don't know Zero MQ driver because it lacks of test coverage. So it is mostly unmountable in upstream for years. So there is a summit topic taken in Hong Kong summit going broccoli as the transition from QPL to Zero MQ which is presented by Bluehost. It is a huge service provider in United States. And actually I was on spot in Hong Kong. So I had a conversation with engineers in Bluehost after the, in the summit. So and at that time I had a feeling that Zero MQ driver is really worth trying. So when I returned from Hong Kong and we started to experiment and do some evaluations and it really great. So finally in upstream it has been switched to Oslo.messaging. So actually theoretically all the OpenStack projects can take advantage of this driver. So as I stated before, Zero MQ driver implements blockless architecture for OpenStack RPC. So let's see how it works. Firstly I will introduce messaging topology. Zero MQ driver has a distinct message topology with traditional messaging systems. So let's see the left part. It is a start topology which brought by a broker solution. So all the messengers needs to went through the central broker when some OpenStack components needs to communicate with others. And the left side is Zero MQ. Zero MQ is considered as a peer to peer distributed system. It forms a partially connected match. So you can see that any OpenStack components can directly send a message to the final destinations. So there is no central broker. So theoretically it can scale out very well. So let's see some components in the Zero MQ driver first RPC client. So here are five OpenStack services which are running in five hosts. So each OpenStack service can consider it as an RPC client. So it sends messengers through a message queue to the others. So the next component is Zero MQ receiver. It is a new component which brought by Zero MQ driver. So it is a distributed broker running on each host. So let's see how it works. So here's still five OpenStack components and each with Zero MQ receiver. So there's no single broker and the messengers directly send from OpenStack services to another. So here Zero MQ receiver receives the message from RPC client via TCP and then it redistributes messengers to local processes via IPC. So that each receiver managers all the messengers within the local host. So you can see that here are the five hosts and each have a Zero MQ receiver. So when the message goes through from OpenStack service to another from OpenStack service to the destination Zero MQ receiver, then there will be the redistributing message to the destination OpenStack service. So let's see in a single host. Here is the Zero MQ receiver. When it receives the message, then it manages the set of IPC channels. Then the messengers, the messengers topic is scheduler. So it is going through the IPC channels to the Nova scheduler. So it is inside a single host. So that's why I say that it is a distributed broker running on each host. And the final component is matchmaker. It implements the out-to-host discovery. It is designed for bare topic. So in OpenStack, there are two, generally there are two types of message topic. The first one is direct topic. So you can see it is the topic type. Here is the compute means that this message will send to the Nova compute host. And there's a dot, here is the host name, host one. It means so I know that I will send this message to the Nova computer, which will set on the host one. So it is a direct topic. And then the next is the bare topic. It's like you can in OpenStack, there are the find out messengers. So when you send a find out messengers, it doesn't specify any destinations because so I need to broadcast this message to all the Nova computes. So the matchmaker in Xerium Q implements was implemented to support bare topic. So let's see. So here is a Neutron server. And here we have two Neutron agents on each host. So we have each has a Xerium Q receiver. So here we have a matchmaker server. So when Neutron agents agent restart, it automatically adjusts its host name to the matchmaker. So it has the host name ALB agent one, has the host name ALB agent two. So when Neutron server need to find out a message with this topic, and it initially it goes to matchmaker to ask. Here I need to send a message. So where should I send to? So the matchmaker returns a list of host names of the Neutron agents. So then the Neutron server send the messengers one by one. So it simulates broadcast. So the matchmaker in Xerium Q driver is implemented based upon a radius server. So in radius, the key value is topic and host name. So you can see that I send a topic and Neutron server can find where I should go, where the messengers should go. So here is a typical deployment scenario. So we have a controller which runs Nova Neutron Cilometer Keystone Glance Horizon and Database. So we need to start a new process called Oslo Xerium Q receiver. And it manages all the messengers related with this process. So I need to run a radius server in our network. And also on computer node, there's Nova, maybe Nova compute, and Neutron L2 agents and a Cilometer computer agents. And you also need to run Xerium Q receiver to manage the messengers inside the host. And also on network node, there are Neutron agents so you also need to run Xerium Q receiver. So there's no any central brokers like RabbitMQ resizing this network. So all the local brokers are Xerium Q receiver. So Xerium Q driver in Oslo.messaging, which is a decentralized PC solution. So it implements multi-host mode for messaging broker. And really it is supported in Oslo messaging kilo, so we can use this. And also it is supported in DevStack in kilo. So if someone need to have a try, you can run DevStack. And here is a simple configuration for Xerium Q, just disable RabbitMQ, disable QPID, and enable Xerium Q. And also you can specify Xerium Matchmaker to radius. So here it is only supported all in one deployment in DevStack. So here some future directions in upstream we are discussing in the mailing list to improve the Xerium Q driver. So first is outbound socket pull. Actually this function has been implemented in Liberty and by Chronicle. So outbound socket pull is very important for Xerium Q. So in kilo Xerium Q driver, when some OpenStack service needs to send a message, you then need to create a new Xerium Q socket to send it and close it. So every time you need to send a message you need to create a new Xerium Q and close it. So it is not very efficient. So here we implement socket pull to reuse and recycle Xerium Q sockets in the lifecycle. So it is very important. And another part is some, we need to do some reflection work. So this reflection work are proposed by Marantis. So firstly, currently all the logic was implemented in implicit Xerium Q. That PYT is only one Python source file. So when some newcomer needs to learn Xerium Q driver in detail and you need to read the code and maybe find some problems because everything is in one Python file so we need to separate it. And another stuff is to improve its socket pipeline by using some advanced patterns which I stated before like DLR router. So it is more technical detail. And another important part is supporting the multi backend. So currently Xerium Q can run with most other open stack projects. But unfortunately when Cinder implements multi backend feature it changes the directed topic format. So as I stated before the directed topic usually forms as a topic name dot house name. But however in Cinder multi, when you enable Cinder multi backend it follows by a backend name. So usually the host in Xerium Q driver, the host one at staff is considered as a real host name. So actually usually the host string is not resolved. So we need to fix it. And also the very important stuff is that if you are really interested in Xerium Q try to use it and give feedbacks to the upstream. So Xerium Q driver really needs more and more real world deployment cases. Actually we have some real world deployment cases but when I discuss with the community I find it is really like, we really like some other deployment cases. So if you are interested in it try it and give feedback. So we can get this message and can discuss with other developers in Xerium Q driver to see how to move on. So here are some real world deployment cases from AW Cloud which we have done so far. So actually during the last year we deployed more than 20 open stack in production. So all the cases are using Xerium Q. So first one is Beijing Telechrom Public Cloud and the total physical machine has more than 300 and the running instance more than 8,000 and this project we started from 2013. This is a public cloud. And another one is China Education Television. So it has more than 5,000, 500 physical machines which run in open stack and this project was started from 2014. So and here is another public cloud which is in development stages. It is at the early stage and it had more than 1,000 physical machines inside the data center and I'm not sure how it is because this project is at the initial stage. But actually I have a feeling that there is no problem in RPC plans because Xerium Q is a fully distributed solution and it can scale out. So we don't need to care much about it. So I think I will spend more on database part. So here are some deployment tips when you try Xerium Q. The first one is a host name should be resolvable. So either in static part or static file or you can run a DNS server in the cluster. And the next one is sometimes Xerium Q when using Xerium Q message may be disappeared without any exceptions. This problem is discovered in our production system but usually happens in larger scale deployments and the solution is very simple to increase and receive socket buff of kernel. Because Xerium Q is a low level socket library so its performance mostly rely on the operating system, the TCP IP stack inside the kernel. So maybe in some situation you need to optimize kernel stack. And here are two references maybe you are interested in. The first one is Xerium Q deployment guide which I was, I written it and put it in the of doc-style-openstair.com. So if you want to deploy Xerium Q manually and maybe this is an important reference. And another is upstream status which I will maintain in this wiki page and all the Xerium Q, all the bugs which is found in Xerium Q driver and its review status and all the blueprints and all the references and I will put them on this wiki page. So you can check it. So thank you very much and if you have any problems you can send an email to me so I'm willing to help. So any questions? I have a question. Okay. Since you're going from a start topology to a peer-to-peer mesh, does it change the way you want to deploy your networking on your management layer? Do you want to use a different way? Do you need 10 gig on your management layer or is it going to change the way you would do that? Oh, it is okay. In all deployments, we always put Xerium Q network within the management network but you can, let me see. So okay, you can read this deployment guide and it will help you to, if you have a dedicated network, just running for the RPC, you can set it up. So it is easy to just specify the network in the Xerium Q options in the configuration file. Hey, it was good to see the real world use cases. Thanks for that. So the question I have is like, do you guys care about like, I know at least from when I looked at Xerium Q that it didn't have any security support, so like you could not use SSL or things like that. So does that matter or for now you guys are just not tackling that problem? You mean the security, the encryption or decryption of the Xerium Q message? Currently, I finally know that currently Xerium Q library supports the message, message encryption and decryption but the driver in upstream doesn't implement it yet. So we will find some time to discuss how to implement it in the upstream. The message disappears in a large scale deployment. Is there any retirement mechanism without, because the risk disappears without any errors or exceptions, right? Uh-huh. Are there any try mechanisms built in? How does the high-level application work? Currently, when I discovered this problem in our production system, it happened for the Neutron Security Group because Neutron Security Group has a very large payload of the message and when Xerium Q send this message and the message is discarded by the operating system. So we find out that we need to increase the socket buffer of the kernel to make it work. Do you have any numbers to show us the difference in performance between the rabbit and Xerium Q? Oh, let me see, I just, let me, let me, okay. Check, so, so I just mentioned that there is, there is some of the topic taken in Hong Kong and maybe, if I don't know, there is, let me check. Oh, so here it is, so this topic, this topic and Bluehost provides some data for performance. So you can check this, this presentation. Right, so I have probably some basic question. So how do clients talk to, can actually identify themselves? Well, find this matchmaker, right? So they're- Oh, the matchmaker is a radio server, so you need to specify the radio server's IP and the port and username and password in the configuration file. So in terms of high availability, you still need to have this radio server somehow make high available itself, right? And for example, the every new client receiver, how do you call this? In the first place is to auto discover itself, auto discover all the cues and topics using the radio server. When the, okay, okay, I got it. Sorry, let me see. Oh, sorry, so Xerium Q receiver. Okay, so when you enable Xerium Q in your open cell cluster and all the open stack services, when it started itself and it will dynamically adjust the topic and its host name into the matchmaker and we implemented it. So any like an implementation guides how to make this matchmaker HAA basically because all these clients are very nice because they're distributed, okay? And independent, you don't have the central point by this matchmaker server seems to be kind of again, much make, well, it's like a single point of failure for me. It is a radio server, so you can make it highly available according to the some radius guide. So it is not a single failure point, single point of failure. Thank you. Yeah. Hi, I came in a little late, so I may have missed this. Hi. So this architecture doesn't involve queuing and eliminates queuing completely. Have you had any problems with notification-based services? If there are no listeners for a notification, I imagine it would get dropped, or? There is a listener inside the Xerium Q driver, so you mean a centimeter? Yeah, I'm thinking about accounting, for example. RPC, it makes sense not to go queue with a queue, in my opinion. There's nobody listening, the RPC will fail. But if you're setting notifications, that can't get lost. If there's, say, a reset of the system and the servers are listening, it may drop notifications. I'm just wondering if you've seen any application misbehavior or something like that, or is it actually working quite well? Okay, currently Xerium Q driver just supports the RPC notification. Oh, just, okay. Yeah. All right, thank you. The salamander event data doesn't go over rabbit and queue the first place. Isn't that true? Mm-hmm. Use the moment. Isn't it true that the salamander event data doesn't go over rabbit and queue in the first place, right? It's on event database, is that correct? So this wouldn't apply in terms of salamander event data. Or does the salamander event data go how does this affect, I guess I understand his question now, how does this affect the way that salamander would communicate with this event data that each service sends to salamander? Each server can directly send a salamander, currently salamander also can run with Xerium Q. So currently the service will send, directly send the notifications to the salamander host. Yeah. Yeah, it works. Yeah, you can, you can, okay. I, oh, okay, okay. I just, in my, in my, let me, let me see this one. Okay, in this deployment guide, and you can specify which network you use for the salamander queue. Yeah. You can specify the salamander queue as the listening address actually. So usually we deploy Zerium Q, Zerium Q network within the management network, but you can, if you have a redundant NIC for just for RPC, so you can do that. So, okay, thank you very much.


we could talk about with that gameplay and I have a lot of thoughts but one thing that's interesting whenever you go through the gameplay are the things Nintendo didn't announce but are present and make you wonder and it does appear that a discovery has been made well several discoveries are being made from this footage that hasn't been confirmed by Nintendo but certainly looks like it's a thing what are we talking about well before I tell you I want to remind you we're on a road to 100,000 subscribers if we get to 100,000 subscribers by the time Tears of the Kingdom comes out we'll give away a collector's edition a pin oh yes a pin from PAX East those exclusive Tears of the Kingdom pins and we're actually on our way in a moment to pre-order the Tears of the Kingdom OLED to give away as well so uh oh boy we're pretty excited about this now what are we talking about when we were looking at this game what feature have people been wondering about for a long time well that feature is shrines we have been wondering if shrines were going to come back in this game in some form they were a massive part of the original game but we haven't seen anything to confirm it and when you watch through the gameplay a few times it's pretty easy to miss that it does appear shrines are coming back now it's only doing one little part of the video and it's on the minimap but a symbol appears that looks identical to the symbol used for shrines in Breath of the Wild heavily suggesting that while we have yet to actually see a shrine shrines in some form are in this game now shrines were teleportation points as well and people have speculated oh maybe that uh that that symbol actually is just a teleportation point and it's one of those green spirals while true I do think if that was the case they could have come up with a different symbol the fact that they're using the exact same symbol they use for shrines really makes me feel like shrines are back now what's interesting is shrines in the past were accessed with the Sheikah Slate which we have yet to see Link have but that doesn't mean that we can't use it at some point that also doesn't mean that these shrines necessarily need that now while we did get the zonai confirmed and you're gonna see a whole lot of theory videos popping out now about the zonai as if we haven't had those this whole time but ever since the zonai technology was confirmed people haven't kind of leaning away from Sheikah technology and it's true that we have seen less and less of it but we have seen some Sheikah technology already in the works in prior trailers so it's not impossible for shrines to come back and I'm actually curious on the dynamic between the zonai technology and the Sheikah technology but it does appear shrines are coming back we don't know what form they're gonna take we don't know if they're in the world if they're underground like the prior shrines if they completely re-thunk the way shrines work but they do appear to be in the game we don't know how many now you could go well with how big this game is it has to be more than Breath of the Wild maybe or there's going to be a lot less of them but they're going to be much bigger maybe shrines are actually dungeons this time and we could just teleport around to the different dungeons that's possible as well as Nintendo did sort of infer that shrines were kind of a replacement for dungeons in a way but also a way to give you something to discover so I am really glad to see them come back I do think that there are maybe less of them than we had in the first game but I also think they're going to be more important that's sort of the way I've been leaning this whole time if shrines did return is we can't just do what Breath of the Wild did that would feel a little lazy I think if you're going to have shrines come back they need to have a bigger better purpose in the world and I think that's going to happen this time but again for now you could say it's a theory it's not technically confirmed by Nintendo like they haven't come right out and said oh yeah shrines are back of course they haven't said a whole lot of things about the stuff we saw in the footage as an example Koroks are back and you didn't even see the A.G. Anuma like even reference or you know say that Koroks are back and yet we factually see a Korok with a giant backpack in the footage so again it's very clear that we see the shrine symbol on the minimap and until further notice until we have more information that looks like shrines are back I don't know what else you could call it it is the identical symbol from Breath of the Wild so you let me know we think about this down in the comments below I'll put a link to a Reddit thread that I originally found this on just to make sure we give proper credit and sourcing on all of this because I've been scouring through the footage I sort of missed it because I wasn't staring at the minimap enough so now I got to start looking at the minimap but I wanted to put that out there for people that are looking for more information on Tears of the Kingdom there should be many many more videos coming out in the coming days but you guys are awesome and amazing thank you so much for being here and I'll catch you in the next video