Happy 410 p.m. Slot y'all are the The real troopers getting to the end of the conference. So thank you My name is Isaac wreath. I lead the no sequel infrastructure team at Bloomberg and with me today is Lindsay Serov Jack She's an engineer with me on the no sequel infrastructure team It's been almost seven years since Cassandra as a service came to Bloomberg over that time We've learned a lot about the many obscure ways that Cassandra can break and this talk will present a few of the problems We've seen over the years some of the warning signs that Cassandra has for identifying these issues What you can do to both fix the issue in the immediate term as well as prevent it from happening in the future Before I talk about all of that I'm going to talk a little bit about Bloomberg in case you aren't aware of what Bloomberg is and what we do So Bloomberg is a technology company that provides financial data primarily through software knows the Bloomberg terminal The Bloomberg terminal has more than 350,000 subscribers and offers different financial analytics tools Which we call functions to finance professionals around the globe who use the data and news we provide to do their jobs every day We have over 21,000 employees in 70 countries more than 8,000 of whom are engineers our engineers are active members of the open source community Both as users and contributors if you've merged my PRs Stefan's in the audience somewhere. Thank you And we even have people who serve on as PMC members as well When we transitioned over the last two decades to becoming an open source first company use hundreds of open source projects to power our products and infrastructure A little bit about the no sequel infrastructure team We offer Cassandra as a managed service to engineers at Bloomberg Our goal is to make it easy for engineers of Bloomberg to use Cassandra to power the applications without needing to understand the complexities of managing large-scale distributed database We started in 2017 and we've been growing ever since The scale at which we now operate is around 2.5 petabytes of data served by 3,000 Cassandra nodes broken into 250 clusters Operating at this scale is presented a number of interesting challenges And we'll dive into those next through a bit of a role-playing exercise Lindsay and I will both take the role of engineers doing an on-call shift for the no sequel infrastructure team We'll talk a little bit about the problems we see and how we fix them But as with all based on real-life media will preface our story with a disclaimer The events you're about to witness are fictitious and have been condensed into a single on-call shift for dramatization purposes No engineers were woken up in the middle of the night in the making of this presentation So I start my own call shift and I get my first issue I log in to figure out what the issue is and assess the impact I see that disk usage is getting pretty high in a number of our Cassandra nodes It's not yet impacting our users, but we need to figure out what's going on before the disks fill up Looking further at some of the Cassandra metrics for the cluster running on these machines I see that there's a table with a large number of data files in this cluster When I look at the droppable tombstone ratio which defines the ratio of data which can be cleaned up by Cassandra to live data It's very high. So I need to ask myself two questions One, what is deleting the data and thus generating the tombstones and two, why isn't compaction deleting the tombstones off of disk? So I take a look at the table schema and I can see that TTL is very clearly was generating the tombstones And the table here is compacting data using time window compaction strategy on the surface. Nothing seems to be too strange here Then I asked myself why isn't time window compaction strategy cleaning things up? Luckily Cassandra offers a tool called SS table expired blockers to be able to figure that out So using that tool I discover the culprit and I use the SS table metadata tool to actually inspect the data One thing that I see on this SS table is that it's mostly expired data However, the max expiry time the meta expiry time of the data in this SS table is different This is what's preventing the data file from being dropped and in turn letting other tombstones build up So what's actually causing the expiry times to be different? I look at some application code and I see the culprit. There are some code paths which have a using TTL clause Adding to the insert statements in this case It's using a TTL of two minutes However, if you remember on the last slide our default TTL was set to six months So that means that some of our data is expiring in two minutes. The other data is expiring at six months And so to explain why this is a problem I'll go into a small example of how time window compaction strategy works and we'll be able to illustrate the issue But before we dive into an example give an overview of time window compaction strategy for those who aren't aware of it So time window compaction strategy is a compaction strategy whose objective is create a single data file for it given time window Data in a given time window will generally be queried together in a time series use case So having a single data file where all the data for that time window is in the one data file as well as expires at the same Time is a really nice optimization we can have However, once a single SS table is created for a given time window TVCS won't actually merge it with any other SS tables in that time window And it will instead rely on dropping fully expired SS tables as well as tombstone Compactions to actually clean up the tombstones So now that we've talked about that we'll go to an example So in our very simplistic example, let's say that we have a time window of one minute And we start with no data on disk So we first starts writing some data and we generate our first SS table if you call SS table one And this SS table what you'll see is some numbers those represent partition keys Those become important in a minute We continue writing day and we flush another SS table SS table 2 and then compaction runs and merges them together And an SS table called SS table 3 So now one minute has passed and we have like a new time window that started all the SS tables for the old time window Have been merged together We see SS table 3. It's actually pretty nice SS table all the data in this SS table is Expiring within two minutes. They all expire at the same time So we'll keep writing more data and we'll create a new SS table SS table 4 And SS table 5 this is where we start to run into problems This data is not nearly as nicely organized some of the data in each of these SS tables expires in two minutes But there is some data that also expires within six months So compaction will run again and merge them together to a new SS table. We call SS table 6 So now that two minutes have passed since the original SS table was created you'll see that it's fully expired It would be awesome if we could clean up this data. We don't need it anymore. We know we have a TTL You know expiry immutable data type use case However recall that Cassandra will no longer run a compaction to merge it with SS table 6 because they're part of different time windows And so instead we need to rely on purging the fully expired SS table However to do that Cassandra will run a check first to see if there are any partitions Which overlap with live data before dropping a SS table and if it does that SS table won't actually be dropped And so what you'll see in this case is among other partitions Partitions zero and partition one overlap between SS table 3 and SS table 6 Even worse partition one will expire in six months, which means SS table 3 is going to be sitting around for a while before we're actually able to clean it up So this is going to be our first issue and we'll talk about how you can fix this in a minute But continuing on will flush two new SS tables SS table 7 and SS table 8 Which get compacted again into a new SS table SS table 9 So now more time has passed since the two-minute TTL data was written in the SS table 6 and that's now expired as well So SS table 6 is not fully expired But it would be really nice if we could run a tombstone compaction to clean things up here So why isn't Cassandra running that? Well SS table 6 will only have a tombstone compaction run on it If again, there are no overlaps between other live SS tables. Otherwise the tombstone compaction by default won't run So what can we do to actually get Cassandra to clean things up here? Well, the first thing that we can try is by setting the unchecked tombstone compaction parameter in your compaction strategy This will let Cassandra run a tombstone compaction even if the SS table in question that you want to compact has overlaps with other live SS tables In this case only the tombstones which do not overlap will be purged and this could have helped clean up that data in SS table 6 However, if unchecked tombstone compaction isn't good enough to clean things up Then we can also set allow unsafe aggressive SS table expiration This name sounds really scary And I think it's supposed to be kind of scary because it can End up deleting data that you don't want to be deleted if you don't have a 100% appended only TTL use case So be very careful when setting this this argument You need to set both a JVM option and alter your tables compaction strategy to set it And this would have allowed us to drop the SS table SS table 3 for example So this would help clean things up in the immediate term But what could we do maybe longer term to fix the problem? Thinking longer term if you can change your data model to work better with time window compaction strategy By bucketing different XBerry times into their own tables This can help prevent the issue in the first place But this is only really possible if you know up front all the different TTLs that you need We understand that sometimes isn't always feasible to do sometimes you really do need fully dynamic row level TTL to solve whatever problem you need to solve And so in that case if you can't break apart your TTLs don't use time window compaction strategy We'd probably suggest using leveled compaction strategy It's more aggressive. It'll help purge tombstones more aggressively out of these types of use cases one thing really excited for in Cassandra 5 is unified compaction strategy. So hopefully we can spend a lot less time thinking about compaction So after getting the tombstone issue out of control. It's a really busy shift for me. I have another problem I got to solve So all of a sudden we're starting to see right timeouts acute occur on a few nodes in one data center We're not really sure why this is happening but these rights seem to always be timing out in the same data center and They're indicating that three responses have been received in that data center And so what this is indicating to us is really that a cross data center handoff is failing in Cassandra Looking at the remote data center to where we see these timeouts. We see that one node had an out-of-memory error But this is really strange, right? It shouldn't the node be down like why is it still getting requests? Why is it still participating in anything? Well, if you use the default JVM option that ship with Cassandra There's a flag exit on out-of-memory error that will be set that causes Cassandra exit when an out-of-memory occurs However, this flag will only work when an out-of-memory is a result of a heap space or metaspace exhaustion And for a native thread out-of-memory the process will sit there still running So you might then be asking yourself, okay, well if that process is still running Why is it still reporting us up to the rest of the cluster and this has to do with gossip? So if the gossip threads are still running it will still be responding to the pings that the other nodes send to it And so the cluster will actually think that this node is healthy when it actually can't create any new threads And so why is this causing a timeout? Well when a cross data center coordinators chosen for the right The right request will just most of the time be pending and it won't actually be processed Because that right request isn't processed the cross data center coordinator can't forward on to the replicas in that data center then from the Query coordinator perspective. You're just going to see a timeout So what can we do in this situation to make sure that we don't have to deal with this problem? Well first and foremost you should Identify that native thread out-of-memory has occurred and when this happens you need to jump into action right away I'm sure the node that is killed or JVM agents like JVM quake, which can be really good at helping to To automate this problem, but at the very least you need to make sure the node comes down So this will help mitigate the issue in the immediate term But in the longer term you need to figure out why you're not able to create new threads on a given machine a lot of the times at Bloomberg when we've seen this it's because there's some sort of Configuration on the machine that's wrong like and proc isn't set properly or maybe you're running with system D And you forgot to set the task max parameter And so one thing to note with this is even though that we found this in Cassandra It's really a problem that plagues all JVM software So we definitely recommend taking the learnings that we had from this problem and applying them to kind of all your JVM based applications So this has been a pretty busy on call shift for me and Lindsay can see that I'm getting pretty tired So she's gonna take it over from here, and I'm gonna go get some rest Enjoy your sleep Isaac So I've barely been on call, and I'm already getting paged. Let's see what's happening All of a sudden an application is seeing sporadic authorization errors But we haven't made any updates to the authorization policies for this cluster on the server side We see that we're under a lot of load, and we see this specific error message about a failure to authorize a user What's happening into the hood? So in order to figure that out we have to see how Cassandra does authentication and authorization Cassandra maintains internal caches of authorization policies roles and credentials on each note and on a cache hit the request is authorized However, if a cache entry has reached its expiry or if the policy isn't present in the cache Then Cassandra will perform an internal query to system off roles and role permissions tables in order to refresh it So why is the failure happening these authorization queries get added to the queue with other queries and therefore when a note is already Overloaded authorization queries can start to time out as well So if this auth request fails there really isn't an exception which can be thrown other than unauthorized exception or Authentication exception depending on which part of the pipeline actually fails And so when this error gets sent back to the client it gets sent back as an authorization failure Instead of being categorized as a query timeout, which applications will typically retry So what can we do about this? Well, these types of except exceptions will indicate when the exception is a result of a failure to authenticate or authorize Including when the system off table queries timeout So we can know that these failures should be retried So we highly recommend adding retries and error handling to your application for this case if you haven't already Now this error isn't always intuitive So the error message itself is going to have details around the failure so that you know you're hitting this case And if you're curious about this and you want to understand it more thoroughly Cassandra 15.041 has more information And it may be a medium term solution increasing the validity of the caches with configs such as roles validity in milliseconds permissions validity in milliseconds and credentials validity in milliseconds is going to result in fewer cache refreshes Which can further help reduce these failures and In more of a long-term solution if you're not on Cassandra 4.1 you want to upgrade so you can use this last feature Configure Cassandra to asynchronously refresh this cache With their your roles cache active update permissions cache active update and credentials cache active update Which you guys should have these slides so you can see this later To emphasize how these last two solutions can help. Let's revisit this diagram So when we set the async refresh these additional system off queries aren't going to happen at query time And while the validity and the refresh configs actually both default to the same value of two seconds a Strategy could be to set the async refresh period to a value. That's less than the cache validity So you know you always hit the cache and also if you were at mix talk earlier I believe he said you can up it to like 30 seconds. So There's that too It's also worth noting that while using async refresh will help avoid this type of error It doesn't necessarily reduce load on your cluster So in times of high load you may still experience the load shedding that causes this problem and the queries to system Off-tables can still time out, but the difference is is that this error isn't going to get sent back to the client and cause this confusion And additionally you may now be reading from these tables more frequently depending on the refresh rate that you set So be conscious about this as well We get yet another call. It looks like some nodes are under higher load than others, which is causing timeouts Let's see what's happening In addition to seeing timeouts We're also getting alerts that the disk is filling up on some of these nodes within the cluster And we look at node tool status to help us understand a bit more about the topology We see that this cluster is very imbalanced. There are some nodes which are going to be like twice the size of others, right? What what's causing this? So to figure that out we need to look at how Cassandra handles data distribution Cassandra distributes data around the cluster and determines which which nodes own what data when nodes are initially added to the cluster So assume these six nodes are in one data center here This makes up what's called the token ring each node is responsible for a subset of data determined by this ring The partition key which is defined in your data model is hashed to determine the first replica where the data is stored So for example in the query we see here The partition key is the name John Doe the value is hashed to equal the token seven Now Cassandra uses consistent hashing so no matter how many nodes we add to this cluster The partition key is always going to hash to the same token value We see that node one is the node responsible for this data because it owns token zero through ten If we assume that we have a replication factor of three we have two more replicas that are going to hold this data Now the replicas are determined by moving clockwise around the ring and selecting the first node That's in a different rack than previously chosen replicas until the replication factors met So we see in this example We now have node two which is on rack two and node four which is on rack three have been selected as replicas Now imagine there's a rack with only one machine this one node is going to be supporting data that other racks would split between several nodes So this is going to become a problem The issue is even clearer when we see the physical view here rack one has only one node in it and no more inventory to add Into the cluster So we'll need to get rack one out of the cluster entirely and add in a few more nodes from a new rack such as rack zero here Which has more inventory? Now in this cluster example, we have four racks of nodes to choose from but a replication factor of three So how can the relationship between replication factor and rack count help us to prevent this in the future? So having a replication factor that's less than the rack count is going to make it more difficult to tell which nodes are replicas for What data as Cassandra can choose many combinations of racks depending on the token value of the partition key and the token ownership Of the nodes in the cluster as we saw before So one partition might be replicated across nodes in racks one two and three But another partition may be on nodes racks in nodes of racks one three and four So this is going to result in some imbalance and some racks may end up being responsible for more data than others Depending on the partition strategy because remember the partition hash determines where we start in the ring On the flip side having a replication factor greater than the rack count will result in placing replicas within the same rack Which Cassandra tries to avoid this makes your cluster more susceptible to outages as nodes within the same rack Could all come down at the same time due to network or cooling issues and that's going to result in loss of quorum Which obviously we don't want So the solution here is to set the number of racks equal to your replication factor by creating a logical grouping of machines This results in Cassandra placing a replica in each rack and spreading data between the racks So let's visualize this We see here now with a replication factor of three and a rack count of three We know that there's a replica in each rack and the machines within the rack split the ownership of all the data and Since all machines are now part of one of three racks the same set of machines now results in more usable inventory available for the cluster Now even though we're no longer bound by the physical location of the machine when adding machines into the cluster It's still important to remember the physical locations when creating these logical groups, right? When taking this approach you should ensure that the machines on the same physical rack are part of the same Replication group so that no two replicas are placed are placed on the same physical rack And if you're an AWS using the EC2 snitch or the EC2 multi region snitch It's important to align the number of availability zones that you deploy in with the replication factor Since these snitches are going to use the availability zone as rack information So now when all the nodes in the cluster begin to fill up we can spread the data out more easily by adding more nodes in each rack It's important to note that the strategy really only works when you enable the config allocate tokens for key space or Allocate tokens for replication factor depending on which Cassandra version you're running These configs trigger the distribution of tokens more evenly around the cluster taking into account your replication factor here So while the strategy as a whole is going to help you with controlling data distribution around your cluster It's not going to solve large partitions It's not going to solve the imbalance of data relating to a bad partition strategy, so still be conscious about that. This is not an out We made it to the end of our own concept, so what do we learn? In addition to the solutions of the individual problems that we saw there are some themes across all of these problems that we need to Take note of going forward first observability is immensely helpful in identifying issues in Cassandra There are a lot of metrics available as well as very thorough logging So make sure to use these and understanding what's going wrong here in addition to metrics and logs Cassandra offers great tools such as node tool SS table metadata and SS table expired blockers that help diagnose these cluster issues Second configuration is very important. It's really tempting to use the default configs But there are a lot of opportunities to enhance performance and prevent the types of issues that we talked about if you're willing to dig a bit deeper In addition to the official documentation the asf Jira is also very helpful in understanding what these configurations do And documents some of the more esoteric configurations that Cassandra offers And finally I cannot stress this enough be proactive when monitoring your systems learn the early warning signs Such as an increasing dropable tombstone ratio and act when you see them to prevent the fall of your application Okay, so we're gonna take questions, but while we do that We're hiring So if anything that we said today sounds interesting, I promise that's not a typical on-call shift Come talk to us. Feel free to scan the QR code Any questions?  We'll take it Yeah, I'd say in like the extreme So that's like a very real example that we had where we had a single Like rack node in a single rack like that those examples It didn't really help a whole lot and we did need to be able to add inventory in there Just spread out the load Before we went with the strategy of just racks equals replication factor just setting that parameter did help a lot with rebalancing your data But for an example like that it you're still need to be able to spread out that load a little bit more All right. Well There's no other questions. I mean, we'll be right over here so you can come ask those questions there You know where to find us Thank you all for making it this far into the conference really appreciate you all coming to the talk and hope you enjoy the rest of your day


The study found that the cost of producing ethanol from tall fescue using different pretreatment methods varied significantly. The dilute acid method had the lowest cost at bin-share 83 per litre while the steam explosion method had the highest cost at 0.05 per litre. Additionally, the amount of water used in the dilute acid method was the least at 5.96 kilograms per litre of ethanol produced while the steam explosion method used the most water at 6.07 kilograms per litre of ethanol produced. This article was authored by Muffy Ganties and Kuma Deepak.


Our sympathy goes out to the families of the four Marines and Sailor, killed in Thursday's domestic terrorism attack in Tennessee. Service members are reminded to stay vigilant whether at home or overseas. Sailors assigned to Commander Submarine Group 7 recently received care packages donated by Operation Shoebox. Operation Shoebox is a nonprofit organization founded in 2003 and has sent over one million care packages to service members around the globe. And finally, USNS Mercy participated in the first disaster response exercise of Pacific Partnership 2015. Simulating a volcanic eruption in Papua New Guinea as a training scenario, personnel from the Navy, Royal Australian Navy, and disaster experts from Papua New Guinea work together to prepare a disaster management plan for the region. For these stories and more, visit Navy.mil.


Hello. Hey, Peter, can you hear me? Hey, yeah. Can you hear me? Yes. Okay. That's Jake Sandeep. Okay. He's joining. So, uh, you had to, you were trying to do some experiments on the package publishing, right? Yes, or Sandeep showed me something you had in, uh, in a pinch. Yeah. Unfortunately, I have not had the time to do more on it since then, but I'll definitely get something done this week. Okay. So, from my side, not too many updates just yet, but I've been working on it. Sure. Yeah. I'm also been waiting for the packages to get published so we can start experimenting with them. Uh, so the API, uh, I think Sandeep had, uh, so we have one PR and Sandeep has another now. Can you talk about them? Okay. Yeah. So the PR that got merged, that I updated all the names and the targets that it should be published to CACTA. And now the second PR, I am adding that will add the workflows that will publish the packages in the CACTA. Sounds good to me. Yeah. Maybe Peter, can you review this? So I added the logic where on the tag release, it will take the version from that and make a release of the package. I tried to use something similar to what CACTA already has. CACTA's already has some published workflows. Oh, sorry if I didn't look at it yet. Yeah. I mean, it's not hurry. I just, I opened it yesterday only. So. Yeah. Honestly, I, I want to, my aim is to review pull requests either the same day or the next day, the latest. Uh, it's not easy, but we should, I think, strive for that because especially if the CI was faster, we would, we should be able to merge much more pull requests. Much faster than what we're currently at. So I definitely want to improve that significantly. I mean, I want to speed it up 10 times easy. Yeah, I'm guilty of being slow at reviewing myself. It's something better to remind me, but I will, I'll also try to be faster at it. Thank you. And we have, I mean, the final thing we've been discussing the, in the context of project perspective is to write. Yeah. Okay. I proved it. I was just looking at it right now. It looks good to me. You know, it's, I didn't look at it in too much detail, but I did take glance and if there's something in there that I very much disagree with for whatever reason that I didn't realize we can always talk more about it later and then just change it maybe if it ends up being like that. Okay. How many does it appear? It's two votes. It's two companies. But since I just approved it from Accenture, you Rama could approve it from IBM and then that's it. Yeah, I'll do that. The only thing is that if at least two organizations agree that something should happen then it should happen because that's currently the majority. That's a 66% majority. Right. And I'm, I have this other idea to write a GitHub action based bot that would make it possible for us to actually enforce this logic. Automatically. So that it would have, it would parse the maintainers file. And then it would have a configuration that says not just that it's any two reviewers, but two reviewers from two separate organizations. And then it would block the merge until at least two different organizations approved it. And then with that, we could enable auto merge so that the moment the necessary approvals are there, it just gets merged immediately. And then we are on to the next poll request. That's cool. So right now, it's just the block is only for two words, any two words, right? Yeah, any two organizations out of the three. No, I mean, currently, so say, if you and Jacpreet voted, would the PR go through? Oh yeah, yeah. So GitHub's logic is not aware of the organizations. So yeah, that's the problem. That's why we cannot enable auto merge right now. Yeah, because then me and Jacpreet could accidentally get things merged just because both of us approved it and you didn't even have a chance. For example. I understand. Yeah, I think among us. All the lead maintenance, we know to it for other companies. Yeah, they're right now we're all just complying with the rules that we know of. Yeah. But it would be more efficient if more automation could be in it. Yep. So Sunday back with PR. This for this is the last when this will publish all the packages we need right for the previous. So you can come again. The PR that you currently have that will publish the packages we need for the previous version. Yes, whenever we'll have to release a tag for that and then it will publish. Okay. And the go packages for go packages it's not there right now so we I still couldn't figure out how to do that in one single release. Yeah, that's on me. I said I would try it as well but I have not yet time to finish my investigation. I'll put some more effort in this week. No problem. But in the meantime, how do we probably to go packages. Ah, it was per package isn't it. Yeah. Sorry that's just super inefficient. I was trying to create a script that would generate all the check go checksms in one commit. But I'm not sure whether it will be 100% for proof. Yeah. Yeah, not easy. If you can share that script. I'm also happy to take a look at that maybe, you know, the two of us together, figure something out faster. Sure. If you want you can also commit it to that. I can invite you to be a collaborator on on my sandbox repo you know the one that I showed last time. Yeah, yeah, sure. Yeah. Let me invite you give me one second operators. All right, give me give me two more seconds because it just hit me with the two factor authentication the phone is in the other room. Give me a sec. Okay, it says that the invite has been sent. Okay, let me check. Can you share the link to the local. Yes, one second. Read on the zoom chat. Okay. So, I'll add the script here to test. Okay, then I can also take a look and also feel free to use the repo as well if if there's something that you want to test that is also difficult to test in in the actual official repo. This is a good lab environment and you know, nothing bad is going to happen if you put commits in there. I don't care about it. It's just a test repo. Okay. Yeah, I don't do anything else right now. I had asked something on the discord so it's about the node modules. There are some conflicts in the node modules, because there's there's a global node modules folder in the root of the cacti right when we clone it. And that's causing some conflicts with the viewer packages. I don't know, like how cactus have avoided that but just wanted to know how to fix that. So that's the mono repo. Both yarn and npm supports it and they are called a sort of a mono repo they're called workspaces. So if you want to read up on it, then there's here's a link I put on the chat. So what I'm thinking is that probably your package JSON is just not yet set up to be part of the workspaces. Okay. And yarn is getting confused about whether it's a workspace base repository or a non workspace base repository. That's the best thing I can imagine. And probably the solution is to tell it that the viewer package is the node JS viewer packages are part of the list of packages that are in the project. And you can do that with globs in the learner that Jason file. The packages and tree that has globs. So I would try to add it there. I'm trying to add here's the logs image. Yes, I put that also in the group chat on discord. Okay. I'm happy to fix it, but if not, I'm happy to take a look myself. Okay, I'll try this. If nothing else, then I'm sorry for joining. I'll talk to you next time. Thank you. Thanks. Bye.


to my channel and welcome back to my kitchen. So this week's video, I am showing you guys easy chicken dinners. And I feel like a lot of us make chicken about once a week and you wanna find new ways to make chicken taste good. Actually, I heard a speaker say that one time that her life was full of taking care of her boys and figuring out how to make chicken taste different, which could be slightly true. But I promise that you guys are gonna love these recipes. It's been a little bit of a dreary week with a lot of different weather. So since these meals were filmed at different times, if the lighting is a little different and stuff like that. But I promise all the recipes will be super simple, like my recipes generally are, and I hope that it gives you some inspiration. Also, don't forget that you can jump to any of these recipes in the video in the description box by clicking on the timestamp that's there with that recipe. All of the recipe ingredients and instructions are written out in the description box as well as the link if you wanna go and pin that specific recipe, which is really nice and handy, and it's an easy way to keep your recipes all organized. Also, don't forget to follow me on Pinterest as well for more recipes. Okay, I'm gonna quit rambling, let's get cooking. Okay, the first meal that I have for you is a crunchy Parmesan chicken with a side of broccoli salad. So I personally like to have my broccoli salad made even a day before because I feel like it does the best if it can sit in the refrigerator overnight and kind of absorb all the flavors into the broccoli salad. So the first step you'll need to do is fry up four or five pieces of bacon, so you've got some good crunchy bacon bits, and then you're gonna mix up your sauce. So you wanna add in the mayo and the Worcestershire sauce and just a little bit of stevia or sugar, depending on what you would like to put in it. Next, you're gonna shred up some cheddar cheese. You could also get pre-shredded. I just prefer to shred my own. You're gonna want about four ounces of that. And then you could tweak around some of the amounts of this, but this day I used about a pound of broccoli florets. And a little trick with the broccoli is it actually makes less of a mess if you pull apart the florets with your hands instead of chopping them. In my last video, I put all of the ingredients on the screen. This video I decided not to either way. They're always in the description box, but let me know which way you prefer. Is it distracting to have the ingredients on the screen or do you think that it's helpful? Let me know in the comments. Then to add a nice little texture and crunch, I like to put in some slivered almonds and mix it up and you got your broccoli salad. To go along with the broccoli salad, I have a Parmesan Crunch Chicken.  It is one that you could even make ahead of time and then warm up in your air fryer and it would still be crunchy. So you wanna take your three chicken breasts and cut them thin and then I tenderized them with a fork.  You could also add a little bit of water to this if you wanna thin it down just depending on how you like your egg bath to be. And then you'll also wanna add in your spices to the egg mixture as well. In a separate bowl, you can dump out your Parmesan. You could also buy the block and shred it yourself. And then in your frying pan, you'll wanna heat some olive oil and some butter. You just wanna get it nice and oily in there so that your cheese does not stick to the pan. And I also use blue diamond pans. I really like them. They are a non-toxic, non-stick pan that's really affordable and I'll leave the link below for them. And there you have it, a super simple meal that could be for dinner or lunch. The next meal I have for you is some orange chicken. It's Asian-inspired and then I also have some fried cauliflower rice. So you'll wanna start out by cutting your chicken thighs into about one inch pieces and putting them onto a plate.  You can use a rolling pin or a cup, but you just want to smash it into a powder so that you can use it to coat the chicken. Add either breadcrumbs or almond flour, depending on if you need it to be gluten-free, with the pork rinds and a little bit of salt and pepper to taste. Stir it all together and then the fun begins. You want to dunk your chicken into the egg bath part first and then you'll put it through the pork rind mixture. Once you have all of your chicken ready, you'll want to put a good amount of the avocado oil into the pan. I would say maybe a good half inch, just to make sure you've got a lot of oil in there and it fries the chicken up crispy. You'll fry the chicken for about four minutes on each side until it's cooked through and then you'll set it aside while you make the sauce. The sauce starts out with a cup of water and then you can use sugar or a sugar replacement. I always like to give you all the options in case you don't have special ingredients on hand but you can also opt for a little bit of a healthier ingredient if you want to. Next, you'll add in your orange juice, your liquid aminos, which is kind of like soy sauce. It tastes pretty much the same. You can definitely use soy sauce and then I just used a little cutie orange to get my orange zest and you could use about half of a larger orange if that's what you have on hand. You'll add the other ingredients, the sesame oil, the white wine vinegar. You could also use rice vinegar if you wanted to as well and some ketchup. I do use a sugar-free style ketchup but you could use regular if you want to. Whisk it all together and then the last ingredient you will use is some xanthan gum and if you don't have that on hand you could also use cornstarch but it won't be keto friendly if you're looking for something that's keto friendly but you just want something to kind of thicken up the sauce a little bit so you want to whisk as you go. Once the sauce is thickened you can add in your chicken and stir it all together and keep it warm on the stove top while you make your fried rice. All right, to make the fried rice I put a little bit of avocado oil in the pan and then just a little bit of sesame oil. So what I'm going to do is actually leave all of the ingredients that I use for the fried rice in the description box and then you guys can use your taste and decide how much of each thing you want to add in. It's not necessarily something I measure out. I just kind of pour and go and in the end it tastes delicious. Thanks so much for watching today you guys. Don't forget you can pin these recipes. All of the links are below to pin them on your Pinterest board. Share this video if you enjoyed it. Don't forget to subscribe if you're new. Give this video a like and I'll see you guys in my next video.


Hey there friends, thanks for checking in. Today we're looking at five subcompact 9mm handguns. These manufacturers are able to produce magazines that carry 10 plus rounds and a lot of people really enjoy the additional capacity. We're going to do an overview as well as some comparisons. Here we've got the Springfield Armory Hellcat, the Sig P365, Torus GX4, Ruger Max 9 and the MMP Shield Plus. We'll start in which the order they were released and that is the P365, so let's get to it. In January of 2018, Sig Sauer released the P365. People fell in love with it. They loved the capacity. They designed the gun around the magazine which held 10 rounds with a flat base plate. They've expanded that to a 12 round mag and 15 round mags are available. They kept it subcompact with a 3.1 inch barrel and a very lightweight subcompact 9mm handgun that many people carry. It has front serrations, a pick rail for Sig accessories and 3 dot day night sights that Sig calls X-ray 3. That is a night touch with an MSRP of 499. Very nice grip texturing, comfortable gun to carry. This has a X trigger here. They go for about 50 bucks. Many of the models come with the X trigger. Most come with a rounded trigger. That's certainly a preference thing. I am measuring the trigger weight right at five and a quarter pounds with the reset right there. They had some problems initially, but Sig cleared them up rather quickly and people are carrying this with confidence and trusting their lives with it. In my Limpris test, the Sig Sauer P365 passed with flying colors. No problem. So whatever initial problems that they had, Sig did a great job. And that was a while ago. People love this gun. They love the P365, the weight, the size, the accuracy. It all just measures up to one fine CCW gun and Sig led the way. And I stand by that with making a real subcompact very nice 9mm pistol. In September of 2019, Springfield Armory released the Hellcat. And many people fell in love with it. They took the same subcompact handgun and produced magazines of double stack that carry 11 and 13 rounds, which is the most of any handgun in the industry of this size. They now make 15 round mags, so they've been expanding with that. But they came out with two models. This is a standard Hellcat with serrations that go throughout the slide. It also has front serrations, but they have an OSP model. Optical sight pistol that has a slide cut to fit a red dot. And they also came out with Flat Dark Earth. So they've been expanding the Hellcat line, much like Sig has expanded the P365 line. But they came out with a great pistol. They have adaptive grip texturing. Feels great. I like the 13 round mag for three finger contact. Does have a tradium front dot, a U in the rear, and a very nice soft shooting pistol. The trigger I'm measuring right at five pounds. It's a nice trigger with a very short reset. Right there. MSRP for the standard Hellcat 569. And then, of course, a little bit more 599 with the OSP model. But I've got a lot of rounds through this. It's performed great. I really do enjoy it. It passed the Limpris test as well. There's no problem with the Hellcat. I haven't had any issues whatsoever with this handgun since I've owned it. It does have a rail there. Breaks down nice and easy. And the sights align really well. Springfield Armory, they hit it out of the park with the Hellcat. They've been expanding the line. And many people love this as well. They're calling this their carry gun, and they're protecting their lives with confidence using the Hellcat. We knew that Ruger would get in on this one. And they did with the Ruger Max 9. They did a very nice job. It was released in late spring of 2021. It ships with a 12 and a 10 round mag for those who want the shortest configuration. I like the 12 round mag for 3 finger contact. But when you grip this, it just has that Ruger feel, has the Ruger reliability, and many Ruger fans out there excited with the Max 9. Very thin and lightweight. It is optic ready with a slide cut to fit 14 different red dots. So if you want a red dot, the Ruger Max 9s have that. This one does not have the thumb safety, but they also offer a thumb safety model.  MSRP 499. They did a very nice job with the Max 9 front serrations as well. And a trigger that I'm measuring at five pounds, it does have a trigger bar safety in there, but a five pound trigger with a reset right there for the follow up shots. I've enjoyed the Max 9 since I've owned it. It's been a great gun. It also passed the limp wrist test, which I use for measuring reliability. And some guns pass it, some guns don't. And the Ruger Max 9 did. It's a good looking gun. It is optic ready out of the gate. No special orders there. It's very thin. It's very lightweight. It feels and acts like a Ruger, which is exactly what we want. And we have it all right here with the Ruger Max 9. Here we have the MMP Shield Plus. This was released the same day as the Ruger Max 9. They have the 2.0 grip texturing right there. It feels good. A little more heavier texture than the original. 13 and 10 round mags are available with this and a 3.1 inch barrel. All right. Does have the three dot and metal sights. But what they changed most outside of the capacity is the trigger. I'm measuring it right at five and a quarter pounds. It does have a trigger bar safety that shifts away from the hinged trigger that many of the MMP owners currently own with a hinged trigger. This is a flat face trigger five and a quarter pounds right there. And the reset is right there. You're looking at 553 MSRP for this. Now they also have a performance center model with a four inch barrel. It has fiber optic sights on there. It also is cut to fit an optic. This is just your standard Shield Plus right here. Probably the lowest grade model that you can get. It also comes with a thumb safety. The Shield Plus did not pass the limp wrist test. When I took this to the range it hung up several times. It could be that it's a new gun. It could be that it's not broken in yet. But for some reason it did not pass the limp wrist test. But it did shoot accurately. I can say that. I shot this very straight. It felt just like a shield with more capacity. Which is what I think Smith and Wesson wanted to achieve. Just last May 2021 Taurus released the GX4. They made it much smaller and lighter than the very popular G2C G3C that so many people love. They made this lighter smaller and have mags that carry 11 rounds. Comes with two mags and additional back strap for larger hands and an MSRP of 392. So it looks to me to be amongst the smallest of the group and the least expensive. It does have a three inch barrel front serrations.  They stepped away from the G2C G3C trigger and they added a flat trigger breaking right at five and a quarter pound. It does use a trigger bar safety in there. But you're looking at a reset right there. So they priced it right. They sized it right. 13 round mags are available on their site. They already have additional accessories on their site for the GX4 and it also passed the limperess test. It did. It passed the limperess test. No problem. You know it measures reliability and right out of the gate the GX4 passed. So Taurus fans out there certainly happy to see the GX4 enter the double stack subcompact 9 millimeter handgun smaller than their popular other models. They did it with the GX4 and a lot of people are certainly enjoying this pistol. Let's go ahead and weight each of these handguns with 12 rounds. Now you can see right there 12 rounds weighs four and seven eighths ounces approximately five ounces with the ammo alone. The P365 weighing in at 23 and seven eighths ounces Springfield Armory Hellcat with 12 rounds 23 and three eighths ounces. So a little bit lighter. Let's go with the Ruger Max 9 23 and three quarters ounces. This is why a lot of people are carrying these guns. Here we have the shield plus 25 and seven eighths ounces. So that'll be the heaviest the GX4 weighing in at 23 and three quarters ounces. So right around 23 ounces. The shield plus is a bit heavier. The shield plus is also the largest handgun. So if we take the shield plus and we look at the P365 we can see that it is much larger. The P365 pretty much is the average I would say amongst all these.  They're very very similar in size and weight across the board. Even the GX4 if we look at the GX4 here. All right we'll see the size and weight. Like I said earlier that this is probably the smallest. It certainly feels the smallest. It may be because it has an 11 round magazine without an extension there. But we can look at the the Ruger Max 9 and the Sig P365. We're looking at not a great difference in slide length or grip length right there. Both with 12 round magazines currently inserted. So we're looking at you know price differences. We're looking at size and weight differences. We're looking at range experience differences. You know they all perform really well. Some better than better than others. I'm going to go ahead and give you what I think. First off I will mention you cannot do wrong or make a bad choice with any of these models. Each have been proven in their own way. As well as they could for for many new releases. But the one that I feel most comfortable with is right here at the Springfield Armory Hellcat. I just find this gun fits me. It fits my personality. It fits my shooting style. It carries well. It's lightweight. It's been reliable. It's fired everything that I've run through it and it's been a phenomenal handgun and I just I think it's great. But my second choice I have to go right here with the Ruger Max 9 to make an optic ready bottle out of the gate with an MSRP of $499, 12 and 10 round mags, tritium front dot. I'm going to actually give that a tie with the Sig P365. I think either of these would be a great choice for anybody. So that's going to be my second place. I know your opinions may vary. I love the GX-4. I think what Taurus did and the changes they made compared to their other handguns were all positive. You got the flat trigger there. You get one of those 13 round mags. You have more capacity. It's a great feeling gun. It shoots really nice. And then you know for these being subcompact this one just isn't as much. It's a great gun. The Shield is a proven platform. They didn't create a new pistol. They took a familiar pistol, a common pistol from back in 2012 and made higher capacity. Is that a good thing? Absolutely. You know, it does share the same holster size. However, this one just doesn't excite me as much as these do. That's just my thoughts guys. I know everybody's going to have a different opinion and I would love to know yours. I certainly hope you enjoyed this video. If you like videos like this, please subscribe and share. I always appreciate thumbs up button. Thanks for watching and you guys be safe.


Do this question. This is 1 kg, 2 kg, 3 kg, 4 kg, 5 kg, and they are placed on the sides of an imaginary square. The sides of the square is 1 unit. You need to find out the center of mass of this entire system. The system has 5 particles. First thing what you should do, first thing is place your x, y coordinate then only you will have y coordinate of the particles. Your x and y coordinate, what coordinate x is you take should be center of mass somewhere between 4 and 5 because most amount of mass is around that. So, let me take this x axis. What is the coordinate 1 comma 0? This is and this 0.5 comma 0.5 simple. So, x center of mass how do you find that? 1 into 0 plus 2 into 1 plus 3 into 1 plus 4 into 0 plus 5 into 0.5 divided by 15 sum of all the masses and similarly y c n. Y c will be what? 1 into 1, 2 into 1 plus 3 into 0, 4 into 0, 5 into 0.5 divided by 15. Is it simple calculation? Yes or no? What is your coordinate related to this coordinate? But if you take coordinate axis which passes through this point, originally this point center of mass becomes 0 comma 0. So, coordinate depends on what system of particles like this. Need not be there are particles placed at different locations. We have not location. The assumption is particles point masses. Are you getting it? What if the 5 kg mass is a big circle like this which is not located exactly at 0.5 comma 0.5 it is spread across. You cannot say mass of 5 into 0.5 because 0.5 is not a while applying this formula. Let us talk about that.  Then I will say that this m 1 masses I will divide this m 1 into different point masses starting from small m 1 to small m n. Because the formula I have is for the point mass. I cannot have small small point masses. So, if that is the case then x will be equal to the small point mass inside m 1 into location of this small m 1. That is it. Location of that is x 1. If I divide it into small point masses then I take 1 x 1 plus m 2 x 2 and so on up to m n x n. This will give me summation of entire capital M 1. Then summation of all this will be equal to capital M 1. This plus the second masses of point mass m 2 let us say capital mass m 2. Now see what I am doing. I am multiplying divided by m 1 in the numerator.   This will be total center of mass. So, if there is a bigger mass you multiply it is center of mass plus m 2 into its center of the system. If mass is uniformly distributed then the center of mass location of center of mass is location of another center. How you can see that x is equal to m 1 x 1 and so on. If mass is uniformly distributed all the points and this is a uniformly distributed to the distribution of mass then the center of mass will lie center of mass because it should lie of both the mass. For example center of mass which is not regular shape and size. This one because there will be infinite particles and also there will not be any line of symmetry to help me out. Then how to find center of mass will get transformed into integral type and this is how it will happen. So, x center of mass is summation of m i. Now I am saying that suppose there is a big mass.  The location of dm is x. So, the numerator will become what integral of x dm divided by summation we get transformed into integral integration is now can you integrate that I cannot integrate this. So, first of all second there are two variables x and dm. Similarly here this is a formula which you have to solve for different different objects. Any doubts? No.


Next question is from Brandon Lee X. What's your main health concern for the next generation? Boy, you know, besides the one that is just on everybody's minds that we talk about all the time, the obesity epidemic and all the issues that that causes, you know, the fertility issues are starting to become really alarming. I don't remember what the statistic was, but young men today have the same fertility as old men did, you know, decades ago, like a few decades ago. Women today are losing their fertility at record rates, even young women. This is a bit alarming because it's like the canary in the coal mine. Like, when you're not healthy, one of the first things that starts to go away is your ability to reproduce. So for men, sperm motility or sperm number starts to go down. Of course, hormones associated with those things start to, you know, reflect that. With women, they start to have much more trouble conceiving and then trouble holding on to a healthy, you know, birth. This is scary because this has profound potential, you know, effects in the future. Now, in my opinion and my strong opinion, this is connected to the health epidemics that we've already been watching, you know, the fact that people are becoming more and more obese, the fact that people are becoming weaker and weaker. This is, by the way, one that we haven't talked about a lot, which is, we all know about the obesity epidemic, but we're just starting to realize that what's going along with that hand-in-hand is this weakness epidemic. So like, for example, there was a study that was done where they tested young men, like college men's grip strength, and these young men's grip strength was like what you see a 60-year-old man in the 1980s, which is, that's released, that's not very good, right? And with women, we also saw, you know, a weakening of their strength as well. This is not a good thing. All of this is not a good thing. So I think what's happening is we're just, this is a reflection of how much we've changed our environment. We've made life very easy. Food is very easily accessible. It's not very healthy, but it tastes really damn good. And our microbiomes are being affected by antibiotics and by chemicals that we're constantly exposed to. And so you've got all these different things coming together, and now the scary thing is happening, which is, holy shit, if we stay down this path, I think the last time I read about this, they said something like by 2040 or something, like not that far from now, like a couple of decades, we would lose our ability to procreate, which is, that's not cool. Yeah, I think just along those lines, the weakness part of it is something that I've really started to see more than venturing outside of our bubble. We talk a lot to people that are in the gym and that are pretty focused on improvement and weight rift, weight rifting, weight rifting, weight rifting. And that's something that they're into. And just getting back into it and realizing that the general public, they're fascinated by working out, like they've never been taught a lot of the skill of working out and being in the gym and lifting weights. And so there's just, to me, it's pretty alarming. I felt like growing up, that was a big part of how you played sports. You had to get better in the gym in order to play better in sports. And I just don't see the same drive and the same appeal that sports used to have, which was a massive outlet for this country and I know elsewhere to display your physical abilities. And I just feel like we've kind of shifted a lot more to video games and like professional video games even. They're making more than professional athletes now. And so it's just interesting to see how the culture shift with all these types of things that have impacted just your average person.  I don't think we're going to see anything. I think we're going to be able to reproduce. I don't think that's going to happen.  And I also think that we tend to go one way and get real extreme and then we all kind of wake up and come the other direction and over correct. But the thing that I notice even about myself, the bad habits or behaviors or the things that I've seen myself start to pick up and do is that we have evolved to a place where very little movement is required to do anything. And it's taking more and more effort to get out and just move, period. You have to plan it. Yeah. I would like to say exercise and strength. That's been a problem forever. I mean, most people didn't strengthen an exercise forever. But what to me is more dangerous or scarier about the future is this how in mobile, I mean, the way the body works is if you don't use it, you lose it. And so I'm worried about the generation coming up. Just we keep making everything so convenient and to your point about video games and your point about things being delivered to you and stuff like that. I mean, we're heading in this direction where you're just not going to move. I mean, not just... There's no real incentive to do it. Right, exactly. There's no incentive to really do it other than hopefully that you would still be able to do it, which I don't know if that's enough motivation for the generation coming up. And I also, again, back to my point, what I see with myself, like we're fitness enthusiasts and I still see how much my behaviors have changed. I mean, I love DoorDash. I use things like that all the time. I have streaming everything now too.  Everything is brought to you. I mean, I don't even go to the grocery store anymore since COVID and got used to using delivery service. So I just... And you just start doing these things and you go, oh, this is so cool and this is so great.  And in those 50 times, I take X amount of steps, which is X amount of calories. And nobody is factoring that in.  And that going forward. And we just keep doing that. So I'm most concerned just about that the generation coming up just flat out moving. Yeah, along those lines, you know that there's a huge increase in kids with back pain and neck pain going to the doctor. And they were blaming it at first on backpacks. In fact, some schools were telling kids, wear in the front and stuff. You know, wear in the front or to have those rolling bags because they think that the problem was, it's not the backpacks. It's that kids are not strong. And they're not moving. And you're seeing forward head. You're seeing forward shoulder. You're seeing really bad posture, back pain. You know, I remember when I first got certified as a trainer, type 2 diabetes wasn't called type 2 diabetes. It was called adult onset diabetes. And that's because only adults got it. So it's something that you developed as an adult through your eating habits and your lifestyle. Well, sometime in the 90s, early 90s, I believe, they changed the name to type 2 diabetes because kids started getting it. Well, back pain was, that's non-existent for kids, for a long time. Like when does a kid, unless a kid hurts themselves, falls off something. But a kid never went to the doctor for chronic back pain. You're now seeing this chronic back pain and chronic neck pain. Well, you see schools eliminating a lot of their physical education because of the funding or whatever the case is. But there's been a lot less emphasis on physical education and outlets, after-school programs and things that they could be a part of. And I really feel passionate about that. That needs to be a massive priority. Total that we shift.  And they said, this is not as important as science and math. They did the same thing to music, by the way. Everything has to be cognitive. Yes. And what we're finding now is music and activity makes you better at everything else. And they're all, they're very, very important. It's all intertwined, yeah. And number two, they've systematically taken out some of the most valuable activities and games out of the, out of recess and out of PE because they're too competitive or because too- Because somebody feels left out. Yes.  Like the stuff that you learn, what you learn playing games like that are their important lessons. Is their chance to get hurt? These are formative years where you're going to have to learn these challenges. It's better to learn them early than to learn them way later in the workforce and you don't know how to deal with it. Mark my words. The next thing for them to start to remove from schools are test scores. Mark my words. Why? Because some kids get bad scores and they feel bad and you can't have kids feeling bad because everybody needs to feel the same. Mark my words. It's going to fall the same path that we saw with activity, which is totally ridiculous. And I think that'll be the death of public education because parents are going to be like, that's enough. I'm taking my kids out. So anyway.