Cheswch eich defnyddio, yw wedi bod yn gyffredegllu y ni hefyd yn gondol yng ngwyloeddmagau ar gyfer gwirioneddol. Efallai gydag pam allu eu brifyddoedd genniololion, gyda'r pethau a alliau lleolodau, yn sgwyloedd diwrnod, ac yna ymdweud i'n rhai cyfuates yma ar y cŵr. Fyddwch i gael, rwy'n cymorth. Ymdweud eu rhan oedd yn iawn. Rwy'n yr ysgwp wedi'i meddwl yn CNCF Argo Slack, felly yw'r ysgwp wedi'r pan fathau ar gyfer gwirioneddol? Rwyf wedi bod yn dweud i mi ar y dda 4 yw'r ysgol. Rwyf wedi gweithio'r cyffredinol o'r ysgol. Yn hyn, Alec, rwyf wedi'i gweithio ar y FFETCH anelidic, a rwyf wedi'i gweithio ar yr agrawerc, ar y llos, ar ddwy o'r 6 ymlaen. Rwyf wedi'u gweithio ar y Llyfrgell, a'r gweithio'r ddeiligion, ddeiligion, ddeiligion, a'r ddeiligion, a'r ddeiligion arall y ddeiligion. Rwyf wedi gweithio ar y ddeiligion, I can arrive as I am defending it is my time. This is the time. Very briefly about Puppe kit. We offer two main services, we offer services on Argo workflow so if you need support on Argo or structure and things like that, we're more than happy to help you. We also offer a product that sits on top of Argo. It just helps you with scaling. It does cool things like multi-cluster making Arback just less sucky and all that kind of stuff. We're at fetch analytics bydd yw y pwyllteideg ond y cyhoedd prosiect-ŷlchedd a'r cyfer sydd yn s���dd a chael gymunedol i'r cyfnodd, sy'n gyntaf o'r nifer o gweithio cyfnod. The one problem I like is trying to work out where is the best place to place a coffee shop in a given area, but we've also got transport use cases. The people interested in their clientele, people planning high streets, managing historic monuments, etc. we active in these five different areas, and yep, start up, so we're scaling. So, yeah, how does our go fit in with our business, what's the core of our company is this pipeline that runs twice a week. It's got about more than 500 billion records, and essentially we've got these algorithms that we've written that help derive the morbidity patterns to surface this information. Yeah, if you rewind six months ago, they were kind of reliable, unreliable, we had this feeling that there could be potentially better or faster, and then like the image on the right, it was kind of a bit not transparent as to what was actually going on under the hood. So, yeah, then we kind of embarked on this last couple of months of improving and want to share some of those insights that we've had along the way. So, I'm a data scientist by trade, so I guess the official term would be like the parameter grid search for optimizing our workflow. We ran about 66 different experiments, it was super costly, that kind of came down to about 15 different regressors or things that we're tuning and changing and experimenting with. Yeah, some of the important ones were looking at thread usage, looking at how we tuned the pods to the different nodes, types of storage options, how we gave data to each pod and how many chunks of data we give to each pod for a specific run type, and then machine specs. Essentially, we're monitoring pipeline run time as our main metric, also looking at improving reliability, reducing cost, and then I guess the headline results were that we've got much more reliable, much more consistent deterministic pipeline, we know what we're going to get when we run it. The test pipeline that we were using, we got down from four hours to one hour, which is really handy, and our CTO said that he can sleep in on Sundays, so I think that's a good result. So, before we look at how to scale our workflows, it's probably worth just briefly explaining what happens in a basic lifecycle of a workflow.  You're going via the Kubernetes API to get that manifest into LCD. The workflow controller kind of wakes up at that point, realises you've popped something in there and starts doing workflow controller stuff. So, it starts asking the Kubernetes API to create your pods and starts kind of just monitoring all that, and all of that's going through the Kubernetes API with lots of back and forth, lots of calls. Which at scale gets very noisy and very scary quite quickly. On top of that, the controller is writing back the state of all your steps in your workflow, back to the manifest in LCD. And of course, if you're a user using the UI to look at your workflow, that's also going through the Kubernetes API, looking at the manifest, basically rendering the manifest into nice pretty blobs on the screen for you. So, there's a lot of chatter that goes on just to create even quite a simple workflow. So, in terms of stuff that could break, in no particular order, at a high level, your Kubernetes control pane might explode. Your workflows manifest may get too big for LCD and you get all sorts of scary errors. Your workflow controller and server can't handle what's being thrown at them, so you run out of resources, they keep out of memory and that kind of stuff. And lastly, your cluster itself could just not have enough nodes to do what you're asking of it and things get quite sad quite fast. So, if we take them one by one, if we look at the Kubernetes API, by default that's quite a black box, especially if you're in a cloud environment. The Kubernetes API is over there somewhere and the workflow is over here and you don't really see anything. You need to get some sort of observability in there to be able to see what's going on, which is shown in the graphs on the right-hand side there. Those are the graphs that come with the Prometheus stack. I'm sure there's others out there. The equally as pretty in Graphi. If you don't have observability, you can look at the workflow controller logs and you'll see stuff like... I tried to ask the Kubernetes API to do a thing and it said no and I'll try again in 30 seconds. Those kind of log messages are a good indicator that your workflow controller is basically asking too much of the Kubernetes API. And this is where the big lists start to come in, where I just try and give you some things to look at. You're going to have to go away and actually Google some of these things to work out what's best for you. But essentially you can ask the workflow controller to back off a little bit on how often it talks to the Kubernetes API using the ReqTime. You can look into limiting how many workflows you're running at once using workflow parallelism. There's an option to limit how many pods the workflow controller creates at each time it tries to ask for something to be created through the resource rate limit. If you're on an older version of Kubernetes, you will see quite a significant advantage in going to 127 or above. I think we found a three-fold increase in Kubernetes API performance just by upgrading. You didn't have to do anything else other than just upgrade. If the worst comes to the worst, you can throw more clusters at the problem. I know a great product that will handle that for you. And you also need to consider what else you're using that cluster for. It's very rare that we see people only using a Kubernetes cluster for workflows. They usually got something else going on on that cluster as well. That something else will also be talking to the Kubernetes API and it will have its own demands. And so you need to kind of balance everything in your cluster to make sure that you're not going to break things. Something we ran into was this error at the bottom that your request entity is too high. This is when you're exceeding the ETC limit, which can happen if you're trying to push how many pods you've got in your workflow. It can also happen if you're trying to put data into your parameters. So you can consider using rewrite many disks to actually write those out rather than storing them in the manifest. You might have a really big workflow, and in that case, splitting them into smaller workflows is a good option. In our case, we wanted to have a lot of parallelism. We wanted to have a lot of pods. You've got quite big workflows. So then we went with this option of persistence where you offload that manifest to a database. I'll check in at the workflow controller. If you install the controller a sensible way through the helm chart or through the official release manifest, there's no resource limits or requests set on that controller out of the box. So if you've just installed it and walked away, you're in a quite a scary position. You need to go ahead and set some kind of resource requests on that controller. Unfortunately, I can't stand here and give you a magic number because everyone's workflows are different. Everyone's requirements are different. So you're going to have to do a bit of experimentation to find out what your magic number is. And then possibly slightly, controversially, I would suggest not running the controller in HA. So in the example here, we've got three controllers. Only one of them will actually be running. The other two will be sat there dormant waiting to win a leader election. So if you, for example, request 10 CPUs for each controller, you're burning 30 CPU only to ever use 10 at a time. You might be better off burning 15 CPU on one controller using priority classes to make sure that controller comes back healthy if it ever dies. The other thing to remember is if you just run one controller, it will still try and do a leader election even though it will always win. And you could turn that off using the environment variable there.  And then likewise with the server, if you're using the server, I recommend spending some time improving that experience, get SSO up, get your TLS sorted. If you have a lot of workflows on the server, you're going to find it slowing down. Again, there's no magic number depending on the size. There's a delete command from Argo. It's quite useful. You can also use this TTL strategy on the manifest that will delete them for you.  So having that link out and keeping your logs, no matter when you're going back to your workloads, super handy. You can directly, obviously, skip the whole UI, which is also another option, just use kubectl commands. And then likewise with the server, there's no default set by the official manifest. So if you've got users using the UI, you might want to set those. Here again. So we went on this journey of tuning the nodes and the resources, which is where we got quite a lot of performance out of. First thing you're going to need is obviously monitoring to know what the performance is like. The top left image is basically what you want to sort of aim for, where you're using 80% of your resource utilization. You're going over that with memory or ephemeral store. It's going to break. And if you're going over that with CPU, things will slow down. Bottom right image, you can see we've got two steps that have got quite different resource requirements. So in that case, it made sense for us to split those up, put them on different nodes, tune them accordingly. So save some money and give them exactly what they need. And then you can see there's one node that's going wild there. That was our workflow controller, which wasn't tuned. So also, as Tim was saying, tune that, and that'll also have some performance impacts on your workloads. And then lastly, just to look at what else is running, your other metrics on the pods and any sidecars or drivers that might also be draining resources and causing things to slow down. So once your Kubernetes nodes, hopefully you're autoscaling. If you're in the cloud, please autoscale. In case you're not aware, if there's a node attached to your cluster, you're paying for it, whether it's being used 100% or being used 0%, you're paying for all of it. So the important thing here is really to turn them off when you're not using them, rather than anything else. We scale down to zero at night on the weekends. It's definitely doable. In terms of pods pending, a good indicator is the image that I mocked up on the right-hand side there. If you've got your pods pending for a long time, there's something a bit iffy. Maybe you're not scaling fast enough, but maybe your images are big. Maybe you're not caching your images well. There's a whole rabbit hole to go down there. There's a whole talk in that. Basically, try and avoid the yellow blobs. Do whatever you can to make the yellow blobs go away. Slightly on a tangent, just in terms of cross-saving, do consider using ARM over AMD64. We use 100% ARM in our cluster. Argo workflows works well with ARM. Obviously, depending on what you're running in your workflows, that's where the things might get a bit trickier. But you should see a significant cost saving if you're in the cloud using ARM over AMD64. Hopefully, most people know about spot instances by now, but spots is significantly cheaper at the risk of your node being pulled out from under your feet within two minutes. You obviously need to write your workflows in a way that they can handle the surprise eviction event. But it's definitely doable. We do the same. We're 100% ARM. It definitely can be done. Something to draw your attention to and think about is how you structure algorithms in a way that you can scale horizontally and quite easily. Fundamentally, you've got thread level is your lowest level of parallelism that you've got to work with. But you want to start thinking about how practically you can make that easy for yourself to just go horizontal with more and more pods, given the right resources and data per process that you're running. Yeah, so that can inform quite a lot of other decisions which nodes you choose. How long running do you want your pods to be? Ideally, you don't want them to die after a very long time and then you lose all of that progress.  That was fine. Hopefully, by now, we've got the point across that you really need observability in your cluster to have any chance of scaling workflows. Hopefully, we have a few things here that should help you on that journey. There has been a long-standing Grafana dashboard for Argo. It's pretty long in the tooth. It doesn't work with the latest version of Grafana. It doesn't really give you all the metrics you need. We just spent a bit of time polishing it up. The exact same dashboard, as it was before, but it works, which is kind of important. Feel free to download it. If you've got observability stack already, you can just plug it in, get running with it, obviously tweak it to your own needs. It's there. Just use it, basically. In terms of the future, my colleagues at PipeKit have started work already on enhancing the metrics coming in Argo workflows. We hope for 3.6, but I'm not going to promise anything. We have basically ripped out the old ones and put in some new ones. There's some questionable things in the old ones, which we don't have a lot of time to go into today, but hopefully the new ones will give you the answers to the questions that you really need answered. As you can see on the screen now, I won't repeat them, but the bottom graph is showing how long a workflow template takes to run across multiple workflows. It gives you an average of how long your template takes. The top corner, the reasons why your pods are stuck in pending, so you can start to see, oh, okay, I've got an image pool problem or I've got a scaling problem, or it just starts to hopefully give you the better picture straight out of the workflow's controller. The proposal link is there. We'd love a thumb up, we'd love some comments, all that kind of stuff. As I say, it's in the works. Hopefully we'll get it out soon. If you don't have observability, I do appreciate that's quite a scary thing to take on. It's not a quick thing to just roll out. The quote at the top is from a recent Grafana study that they did, you should have observability. Do invest in it. Hopefully we can help you on that journey. We have written this free plugin that you just plug into your cluster. It covers exactly the metrics you need for your other workflows and you can just see them on a dashboard and you can start to see some of the answers to the things that we just went through today. I'll show it to you really briefly. First, what I've done is loaded it in separate tabs and I'm not going to hit refresh. This is a natural cluster. I've been desperately running some random jobs just to try and get some spikes and some graphs. This first page shows you everything about the workflow's controller itself. It's showing successful workflows and the failed or erode workflows. As well as what the inside of the controller looks like so how many things are being added to the controller's queue. The depth of things in the queue. If we've got a problem we'll start to see those lines go up and not come down. This cluster looks fine. The Kubernetes control plane will be in here as well so you can start to again see if your control plane is crying it will really cry on this graph and you'll see it at a glance. The same you can see of the queues and things like that within the API itself. You can then see your workflows pods. You can see pending versus running pods. If you see a large spike of pending and it takes a long time for them to turn into running then that's when you can start to see that you've got some kind of scaling issue. In fact this large spike here is when I made the screenshot a few slides back showing all the pending pods. Lastly looking at Kubernetes nodes. How many nodes have you got? Are they scaling up and down? This particular cluster scales down to four when it's doing nothing and spikes all the way up to 30-ish when it's doing some work. Really you want to aim to use as much of your nodes as possible because you're paying for all of them anyway. In this particular cluster we're doing a pretty bad job. We're not using much of the CPU at all and the memory is a little bit low. So we've got some work here. We can now see at a glance that we need to do something in our cluster to try and get the best economy out of them. We hope to have this out very soon. Please just register on the quick link there. We'll get in touch with you in a couple of weeks. As I say it's completely free. The collector itself is open source. You can see all the code. You can see all the data that's being collected. Everything we collect, we show back to you. There's nothing hidden there. We just want to help people just get the best out of workflows. That was a key summary of our findings. Definitely key theme of getting observability up was pivotal. Having Prometheus, Cube State metrics, Argo, Custom Metrics, all super useful. Some other kind of tips were using NFS, a network file store. Tim's got a great talk on that. Worth checking out if you haven't seen it. Getting rid of life cycle hooks was quite important because we'd often have a screen like you saw earlier with a bunch of green where things would actually be failing under the hood. Getting rid of those is worth it. And then also just investing in some tooling that we hadn't come across. Making things more dry with customise, also controlling cost with lower environments using cheaper storage versus production using the high end NFS. And canines is super useful if you haven't come across it. And latest version of Argo is also obviously good. Yeah, so those optimisations are what kind of got us much more performance at various levels, a part on node level, a quarter of the time, and knowing what we're going to get each time is super valuable. We just collected all the stuff we've mentioned, basically onto one slide just to make it a little bit easier. I won't go through them all again. These slides will be available on the top link. There are extra slides in here that we didn't have time to talk about today. So if you want to go into a little bit more depth, there's some things to look at. In terms of pipe kit, we're more than happy to listen and help wherever we can. We're on booth E34. We're happy to hear your open source questions and try and answer them. We do regular office hours to do exactly that. Just sign up, sign up and throw your questions at us and watch us cry sometimes when we're not quite sure what to do. But yeah, do just turn up and get help. That's what we're here for. Yeah, and that guys, if you're interested in this data and the story of human movement, come reach out to us because that's our, that's what we do. Thank you.


Good morning, everybody. Welcome back to another stream. Hope you guys are doing great Without further ado, let's get started. I always get started with yesterday's sort of review and move on to what's happening now Actually, none dessert. Let's do it now quickly. I mean you can already see prices Sort of hit out into these sort of three highs after sort of coming down yesterday So the main thing you're all looking for right now is you see this bid hit now You're seeing this sort of climb up now. We've got a collection of shorts that were re-accumulated here and here So you see in these shorts getting re-accumulated so you're expecting at least in the sort of short term or the in the current sort of Climate for it to sort of hit up into sort of higher territories at least to liquidate some of these shorts from the market And we've got a newer level shorts here that still haven't expired So that's something that we need to look for at least if we're looking for a downside because we do have a slight sort of Disparity in shorts and longs and like prior these longs here and These longs here are pretty much liquidation longs. So they were added back as in they don't any There's no real trade volume there more. So there's just emphasis on people being liquidated and maybe some trade so We know that there's no real trapped liquidity. It's more that people have been liquidated into sort of static Short bids then now push back down Now we want to try and get I think on the third one if they we in fact get a third sort of attempt at these highs An actual sort of input from traders to buy and then maybe we can look for some downwards movement So that's pretty much the short term covered. Let's go over Yesterday do yesterday's review as yesterday was pretty nice day in terms of having it as a reference point Let's sort of zoom in and just look focus on this section that we had and were privy to yesterday We talked about how we're probably expecting a third strike yesterday coming in Then after if we get a third strike Before going down will look for a price to go down and they ended up going down. So pretty much No mess no fuss. It's a clear textbook example of three hits the high Very controlled just a free added back to the same territory. It did before This is still not focused correctly. So let's There you go, it's not you even just do this Yeah, there we go as you can see removing that section there we have The section you'd be privy to when you were trading potentially trading this so This is information we had we were Already here. So if we were to go in to this bed wall here, so this cluster of longs without sort of Releasing too many longs or without sort of Going too far above. I mean we could have gone all the way to 6800. We talked about that as a possibility But and we still do I mean we did have a massive sort of bed there In terms of we would take out these highs if we didn't get the volume of actual buyers coming in Because these moves here again, like I mentioned were liquidations one Liquidations just good cascade of liquidations here You can see the shorts that were stacked on these moves here were liquidated there as you can see That was created that cushion here But again, we had more shorts stacked so We need to clear it out. So clearing it out and we got an addition of longs Just coming in and just saying hello and you can even see a small micro Double top Nice little entry that you could have made for yourself Either on the first leg or the second leg again. I always recommend people Waiting for the second leg, especially because this take data. So it's it's a matter of seconds. You're waiting Not in or even minutes at most So that's something that you can use to advantage because In most cases, you'll probably be stuck on Waiting for minutes to an hour For a second leg, whereas this is just instant. I mean we can look at it on a traditional chart I mean there's your two candles 15-minute candles if you so break it down to one minute There you go. There's your little M here coming in and just slams it down That was sort of pre Precession so you could have avoided taking the show if you really Were not too confident in doing so because there's your New York range and they did all that in the first range So it went down and we typically say if it's going down in range Then developing a sort of these top hunts outside You're looking you're looking for it to at least try come back in as you can see here and Well, this one did went high and then went down this one again Retested the range here Then continue to go down and the there is a possibility, but this one controlled itself just collected within here collecting or collected orders Hey, hey, and you can see where they collected orders and If you were to use this in an extra entry, so we know that our session Basically finished setting up here. So everything passed here was outside of a Range so was to basically stop on so we could have used this to a disadvantage saying, okay You can see that first strike. It came back in just grabbing longs Then it just took those longs out the market hit into that bid and just re-fortified that after the fact just say if we do end up Sliding a bit or people were panic-selling It would be caught. Hey, hey Can you check crude oil? Yeah, sure. Let me just finish up with Bitcoin then we can go on crude oil Yeah, that anyone just leave a request. I don't have any preferences for What you want me to chart up for you? Everything's basically the same So we looked at that Get out into that then pushed up gain You could have exited either here and in steps here here then finally here And watch that take here then it re-entered much later here if you wanted to or waited for that But now we're sort of into this bit now We want to see some more thoughts collected If we end up seeing that we'll expect price to come high only sort of issue about that high coming in is That potential Wall there was was weakened. So That's kind of sort of not always that's less than ideal if this was to remain fortified we could have used this as a potential reference point to say that price may have Come here to say hello But as of yet, not no word In terms of timing again, we talked about yesterday and the day before 10 p.m Typical time for us and 2 p.m. Which is a usually a two-hour offset from the typical 12 12 p.m. It's FOMC. So it's going to be probably a Lot of build-up prior so you get Most likely a telegraph move and the move again Would always be the same is just do the opposite of what the market does So if the market at 10 o'clock spikes down stop hunts just stays or just continues to go down two o'clock hits You're just gonna do the opposite Then that'll be your first leg you sit wait to see if you can get a whip for motion Like So Yeah, something like that. I know it's done to not in too many prongs. Yeah, something like this where maybe you expect high Push back down and it starts to correct against self but Just know that you're probably again build up to do the opposite and This that's your move and they sort of corrects itself and Just hits back into this collection of bits here then pushes back down and continues in the direction. They want to do it initially go So, I mean that's pretty much today sentiment We're already at range weekly range high as you can see here. So this is your Monday and Tuesday range Basically, just ended up being Monday as Tuesday basically traded within Monday and I'll cut off for expanding the ranges on Wednesdays start so other than that This is There hasn't been an expansion of that range. So now we're outside the range We retested above could be again something we slide just a bit under where we have those bids visible Then just say it's a retest who knows pushes back up Ultimately It's it's not in too bad of a shape as long as it sort of stays Around here.  That's what you're looking for Where's London making? potential the potential high of the of the day it could be something that we will look for a Basically a down day today potentially and So that's something that you want to consider So someone on your crude oil, I mean We'll quickly do crude oil. Yes Okay, it's been a while There we go. We're looking at our sort of starting from daily Actually, I don't see if it's been a while to see if there's any weekly levels we want to be considerate of. Oh, yeah I forget. No, why do I always forget? No undue undue undue undue I Remember why there's no weekly levels on crude oil because it's broken on this been broken forever fix these fix it and If you guys are transmitting you check if it's broken for you as well because if it's just me and I just constantly break stuff then I mean sounds like a me issue and These are too far out to want to be considered marked. Anyway, I Shall just mark them for so you guys can have them in the future There you go That's center. Yeah, it's center There you go These are your levels that you want to be considerate of and I need to refresh this So it shows the numbers. I just nothing work Bloody hell, man No, it's not working. Okay Yeah, it says show price. I mean, okay Yeah, I just go run them down again just hover over that that's Again one two one point two five just round it up round it down. This is one one four point 115 just to say 117 109 10 basically Next one down is 108 15 let's just say 15 10 up to you to where you put it Again, you have a lot of sort of leeway in terms of allowing you want to allow yourself some margin of error with these anyway Because it's just a buffer zone There's again this whole area here is the potential candle that you come revisit to I just find more success in getting the middle thing and After the fact you want to look at the sort of shape of that of that candle itself that teardrop Raindrop however you wanted to call it drop top. I don't know but This is one or two 94 90 you guys can read this off yourself 77 and seventy one point two two It's the broken Okay, I mean those are the day levels. I mean you can see that you can see how it came here came back up Retest gave you a nice little triple top them double top action We can mark out these four hours. Okay, look it works on the earth. It's just Why is we're all broken? I don't know Questions asked myself Yeah, you can see how this four hour comes in play here and also here's a sort of an area point and It's more so this one being More reactive than this with the prior one, but I take the most Pre The oldest one you can just see how they react to Alongside it and it just becomes looking at a volume after that within this area as you can see these two Trapped the volumes here. See where it comes soon says, okay, we've got longs previous longs here Probably still not out of the market. Let's collect more just under it not allowing these guys to be free And they even tried to fish for just above this So they allowed some momentary hope And even retested daily level here. So you could have retested that and just pushed off but it's sort of Cut itself early here Right now we're sort of developing sort of heavy short here. You can see that shortfall Into these points here. So it's finding some level of support Which is which is nice And you know how you're trading there's all coming into an hour now you can see that very large slide here That's your first candle where you see arcade Collection of shorts push back up now we're getting those shorts liquidated. So you probably might see that Continuation down downwards to probably target stops I mean, it's already hit out this stop here That's what the that area was so instead of being collection of new shorts. It was a liquidation of Longs released the market Then an actual sort of addition of and more longs piled on top here Just because they thought they found support Nice it'll stop on but it's just moving into sort of shallow resistance So maybe looking for that to sort of taper downwards for this session Terms of points where you might want to capture it I mean if it runs out below, I mean you've already got these two areas So you look I mean these three stops this one coming just a bit higher because we have that collection of volume here So you want to see sort of additions if we just take out these two areas just collect Basically collect we were looking to collect shorts at these lows Once you get that sort of short volume sort of identified You're probably better entering into a long Right now. It doesn't look very sort of happy to be in a long right now I mean even though it did slide a lot From what yesterday to today it's been dropping so It's still sort of up in the air as far as I Mean if it was me, it's it has the potential to keep dropping at least Maybe look for an extra bar one more slide No, excuse me But yeah, I mean we've got this sort of you can mark this out See one for our you Got another sort of four-hour points. I mean best case or worst case you can get a short You can get price coming back all the way up to here So if it drops comes all the back to here just closes this Off which is back down that could be your opportunity to enter short at a higher price And and continue down again. You're giving that some leeway with with this area here Just because that's where you have a larger sort of settlement of volume for those for those longs. So You ideally don't want to just let them be in profit. I mean they had a chance to take profit But you know trade agreed and they even re added back at the same position. So If we do end up climbing higher, you're looking at this is resistance If you're in a long and in that climb high, you're probably likely to sell at these points here Again, you're sort of taking partial profits moving stops.  I can just tell you why you see Is that's all the fun in trading managing your risk all the that's where the fun stuff is I Mean other than that. I mean looks decent nothing sort of too overtly Garrish But Yeah, you you have this Any questions on on a crude oil And why it's broken like always Yes, coming back down. I mean you got basically two collection of sort of blue candles Which is might show that there's some pivot in this area. I mean we already know there's a daily pivot area. I Mean, this is a daily pivot area been consolidating. I mean so Bitcoin is moving a bit of a bit ahead of the steamboat here It was bookmapper. Oh, I just I don't have that. You know, I need to ask them for that Like my book map like the general me broke yesterday. I still have an answer. I only have No, I should have it should I know Let me see if I have it. That's a thing like Everything keeps breaking for me Like I have the worst luck I think in the world Oh Bear with me a second I can that's the first thing Yeah, it's just gonna be like I broke it. Yeah, I think I broke it Okay, yeah, it's it's because I don't I don't bother fixing things because I'm at a stage where I can I can basically see where maps are but that's that's because I'm Online retarded I can't see that oops No, I don't know. I yeah, I'll try to get Probably tomorrow the day after but it's something that I just Just didn't didn't really need because it's not something I trade. Oh, yeah, we can look at oil. I mean gold, I guess I mean, we talked about the potential for it to hit that weekly I mean, it's it's finally made its way all the way to that weekly now It's at a pivot point. We want to look to see what it does. I mean this day has already been pretty hammered in It's been collecting we see we want to see how they play with this how far they want to go above it But generally we're looking to try collect some loans here to just pass that off as What's going yeah, pass that off as a pretty short. I mean it's showing similar If it's somewhat similar things, I mean it's mid-week. We still have three days left for the week So we still could just spike all the way into this into the day daily resistance up much higher Before coming down So it'll be a nice little buffer that we have it just to say, okay, if it plays with inside this range We're probably gonna short this pretty sort of aggressively back down Then that's probably you what you want to see And because if this comes down, we're probably looking better for equities. I mean a lot people will be more risk-on But generally yeah, it's playing nice in the full way at What of all these areas here? So I think you want to see I mean you cost so many thin bars But it's all about what that FOMC does to this rather than what we can do to it Just do I mean we already sort of undercut some longs aggressively here We'll see that on the hour as you can see just hits that flash flash It's a little railroad. We'll see how far they take that up. Yeah, I mean Realize that I mean we want to look at lower timeframes Yeah, basically just took out those stops pushed down aggressively Give you a nice little doji's here push back up not just trapping volume You want to see if they just leaves that to a double top or if that's something that just gets pushed much higher The hours probably suggesting it's got some leg room, I guess We'll see if it just extends the high if it short throws It's still sure they're probably gonna short it short it down again. Look at lower timeframes for confirmation You're probably looking for at least what 18? 1845 I mean best get best best case you can get it's probably all the way back down to 18, but How we can get that? Yeah, nothing really too much to say now It's just pretty much a waiting game you guys have to wait for that 10 o'clock. I mean I'll have to wait for it But it's better to wait for it because we know you're getting signs that it wants to sort of try bounce Off of this area here. I mean yes is looking decently strong. I mean you're already getting this consolidating But pressure piled up on the above wall here So maybe we get that just one aggressive push down before hitting it up nice little whips or leading to that 10 o'clock sort of news news post But I mean Considering it is sort of what they expect those to increase Yeah, I think I would expect better numbers I think out of coming out of today I mean we set up like Bitcoin set up and moved up much higher Which is the only sort of contention to wanting to go higher for today But considering we're sort of stagnating around here If we sort of push it down We're not probably looking for it gain to retest Was like some this is some Level of strength at these areas All the way down to here. I mean this is where you stop considering a retest and start considering Okay, let's look for if it goes up a bit higher look for it to go continue to go back down back into the range If it bounces off the range here looking again, just take out the highs and probably Last week's highs it as well Yeah, nothing. Yeah, nothing too interesting basically to say today. We'll change them also similar to using canister patterns on a block With yours, I would say market timing. That's something that is really important to understand prior to anything else You can see these pink boxes. These are basically just session times. They tell you basically Where when the setup occurs within the market session Then from there you start using whatever you really want to use it. So it's then I mean, for instance this here just comes back in to say close off this candle here which actually Was great succession one. I'm looking for a line tool. Where is that? Close that off. I was small gaps, but Asked that we already know that was there was beds here anyway so we could have entered that without using that as a knowledge base and Because if two or three things are saying the same thing, you're more likely to enter anyway Because you're like, oh, yeah, cool Both ideas line up. It's when you have ideas that are posing And that's when you want to really sort of look at things like okay. It's the things, right? I don't really use too many sort of patterns The only really patterns I use is just identifying M's and W's I thought like I think I did a whole yeah, I did a whole video on that like this short program There's your bigger Big M doesn't matter about magnitude. All this is a stop pump. All this is is just as Ablution back lowest this trapped liquidity that they don't want to release here and then They just read longs at these legs here because they went down for a extended period of time or Period of time that collected too many shorts. So they need to to liquidate those shorts and add back Without releasing too many longs because they really want to go back down Then as soon as they complete that M you're looking for a W to come back in And I think I've talked about this before as Yeah, I did an entire video on on my channel About sort of M's and W's just There we go. This is you Yeah, this is a nice little blog just again Arthur Merrill You can even look at the MQ5 articles again This is clearly depicts what you're trying to look for and they basically cause all your sort of Triangles or whatnot don't really look for those Head and shoulders triangle flag patterns unless I want to do the opposite of what they do You want to look for them? So for instance, this was your weekly range This could have been construed as like a flag or a pendant or whatever they want to say is like Oh, yeah, this this continues to go down or whatnot because Retail patterns but understanding that the context of where the pattern is being printed. Okay. We're already into sort of Discounted territories where market will be printing out bearish patterns to trap shorts Where stop hunting our sort of love our New York range anyway, we're just closing off this area Just taking out stops here. We're hitting into sort of bid. So shorts are getting eaten and then off the rip There's a cell candlesticks Like I've mentioned was like using I mean like you've seen like I use Transpilers raindrop. It's like super nice. Basically. All it is is just VWAP So, let's say this is a four-hour candle first half second half view app Just does that gives you a nice little blue when they both at parity for those sessions and then that gives you sort of nice little Area to look for support and resistance that Works both ways because it's just basically like what I've discovered Oh What I've used and what because I listen there's no literally on using these except for what I've said because I it's just Not many people know about it Or if they do they've kept really quiet about it Because I even said it transpired and they like oh wow that's that's that's new. I mean you didn't think about Maybe it's just I think I think weird that's why But those would be like the most effective strategy like just Understanding that there's a time element to the market. So you don't want to Do things and you want to ignore certain things you want to filter out certain things based on the time of day So and then a really good person to if you can get the resources I would be looking at Ah Steve Steve Morrow think yeah Steve Morrow Beat the market maker series is on YouTube Yeah, you can find on YouTube Like it's it's like usually I think you pay through I'm saying for not 50 or five grand for it But like what it is it just teaches you this this timing to the market and And and from there, okay, you can understand and you can adopt that That things happen because Because of time rather than of a set pattern and Because I mean this time and volume is what basic candle is So if you understand basically you just instead of reading What's printed out and the output you just reading the source code if you're just understanding that okay at this time This will most likely occur because you're gonna get more volume What's the volume saying? Oh, yeah, let's just go over here. Oh, yeah It's saying that we've got a lot of volume here And it's it's pretty sort of skewed to being long. Oh, there's this people catching beds here. Are they remove those? Okay, you can understand how these traders are Feeling about about training this point. They want to move it need a little bit higher But it's still keep it Controlled till they get the beds. They want Yeah, all the books are nice again. It's just you need to filter out the ones that are useless And that's just cons with experience And understanding this time element can help as well again. You understand that. Oh, yeah, stop haunted strange What are you looking for now? I just closing off this. Okay. Yeah, can I take this? Yes? I can so and so on Okay, it's climbing up And we can see where the bids where it stopped up here The high they remove those next the next level here Yeah, then finally it stopped at here But then came back up Give you a little marker remove that marker. So now you're no longer And she expecting that high to get hit was a hundred percent certainty. There's now some What's the word? But you're expecting Expecting some points say, oh, yeah, it won't really go up there anymore. It's not a hundred percent Assured anymore as it would have been if that stayed where it is so When I said I don't use patterns, I don't use typical patterns, I guess and most of training is it is pattern recognition but Patterns are like one of the most easily exploitable things so instead of doing pattern recognition in the traditional sense do a sort of Recognition in the fact that okay, this is how traders react and this is the behaviors of traders rather than looking at oh Chart is printing this I must do this. It's okay traders are doing this I'm just gonna go on the other side because They're free real estate at this point Hope that was Hope that was helpful Kind of rambly, but yeah, if it made a point it made a point Yeah, definitely look into Just if you're using patterns these are really nice because Again, I can sort of break these down as basic simple things where you don't really care what this does all the services Okay, this is short throwing. This is short throwing. This is just short throwing on this leg So let's say this leg came down like this because you will have the corresponding W pattern that will do this. So they just trapping beds here. So you and since we have beds here We want to be below those beds We have trapped beds here We want to be below those but we don't we want to but we were going down for too long So we collected short short short liquidate Don't release those trap long push back down same story here And again these correspond to the opposite pattern here Use line tool once you sort of practice these for one or two days you get you can easily spot them and Like we talked about these is just okay We went down. Oh now we need to there's With this beds up here that we can collect because we can see the order book. We know that this This is either liquidations or beds here.  That's why you guys don't need that if you just I understand when Do you trade them and when they'll be tradable? Let's say let's say you see this flag pattern But it's a much higher is it's here rather than being here. You'll probably expect it to actually come to fruition depending on a Composition of what's on the left and be based on the fact that we're pretty high up Relative to the weekly range. So we most likely need to backtrack a bit So, I mean patterns are not bad if you if you know how to trade them if you know how to be selective But thing is most people can't because they don't have these colorful boxes. They can just do this and just like, okay Cool, I don't know what time it is. I see a pattern. Let's take that pattern filter filter out And you sort of okay cool I only want to trade past this line and I see something tradable. I just counteract whatever retail's doing here And I'm in good standings I mean this you see a nice little triple top is then slams back down again I mean if you see in the context of look, there you go. There's that m pattern We saw this extended much further than Rest most people would have entered I think on the break here, but there was no entry to be had So there would be There's just one candle they would have waited with clothes and short here It's just as soon as they enter short they get liquidated on this swing here Then it goes back down and think are we are what I accept my short Let's enter into here because it broke that trend line But then it just continues up leaving them holding the bag. They get liquidated on this sort of swing high This understanding And that's the main thing with the markets just understanding what reactions why reactions will happen And If I'm if it's all the complicated and it sounds sort of abstract just tell me because I'm at a stage where This maybe maybe I'm on spectrum. That's why I can understand this like Without without it being too hard But yeah, it's it's it's simple to me If it's if it's not simple to you just tell me and I can I can try make it simpler Yeah, and I was just coming back up. You see it trapped in this sort of long ledge here my expected to be Pretty level of down and high So we'll probably expect first down hit out into the is that you're already getting this set up getting that pressure We're just stagnating So we're not really getting too many sort of long entries. We're hitting into sort of these bits here Which which just shows that we might just release release some of that Some of those longs here before pushing up in everything's just a game of probabilities We're just waiting on I think just news release to say boom. Let's drop that You need to learn them one Yeah, that's I mean just wallet down just don't don't read books Anything that's the best seller is is is wrong Because by the time he becomes best seller everyone's read it. So everyone tries to incorporate it the order flow So the patterns will start changing the order flow will start changing and so being 100% accurate As the book might sell you This was changed is make it Less accurate and you get to see when it will work and when it won't As well people let's say if this didn't like people buying a break out. So, yeah, look at this It's a breakout. They also bought the breakout here and what happened. Oh, well, they don't know So they say, oh, yeah breakouts always work. Look at this. Let's just buy the breakout and they invented new thing I just would buy it by the break out They always buy the break out and it's just like I'll look by the breakout here It is just it's just it's daft at some point. So you just got to cut through that BS and just understand Oh, yeah, you don't really want to buy this here because you're expecting Well, you're expecting these lows here anyway Because you're in that time where you're setting up and as soon as traders come online 930 They just built to fall And it's just this candle here. So 930. This is why you want to trade 10 Those traders come here. They see the markets. Oh, yeah, let's let's buy the breakout.  I tell you It's simple, but also isn't Yeah, that makes sense It's simple, but no, I would recommend if you're finding this hard just pick up like high school and No, I school secondary school Jesus and college College or whatever mechanics and and not mechanics pure mass do Yeah, I know it sounds weird like oh, yeah, do math, but like you start with picking up Some Well-needed dots I think you can apply to real life Especially with pure mathematics Mechanics is just for the fun of it. I mean It's always fun. I mean you could do with some statistics But I think understanding pure mathematics is a lot better because you learn more about geometry and you start thinking more abstract Rather than being so obtuse to about oh, yeah, it needs to do this.  So maybe we fill those out. I mean we already had Something here now. We're just filling these out. I've got a bunch of lungs here anyway We'll see if they retest the top of this. They close this off.  It'll probably do something this push back come back in Give you a nice little double bit out into that Might be something similar to what we had yesterday If you want to remain bullish, that's what you're just the only thing you're looking for Other than that probably look AP dot if you see it drop a bit lower You're looking for a leg higher to try and short throw this all you're looking for to 2pm to see if it continues downwards and 2pm reverses it so God is aware it's that you can almost just blind Just like it's it's it's just do the opposite and this like it's because people just trade They see it's going up. So yeah, so then just press this buy button. You see bad numbers. They sell so so so so But the reason why price will go down before Before a date is because the numbers are typically either Sort of good or it doesn't matter if it's good or bad It's just they want you want it to be down So in case it's good you spike a high let people buy and they can drop it back down and continue into the direction They really want to go Let's say it goes up like oh wait. We've got really bad numbers. Ah, but no we want to do this It's not always the case sometimes those bad numbers are really bad to the point where they build it up to Price in really high numbers and what they typically price in is the is the forecasted so as long as we're if we're So for this we want to be the same this we want to be slightly Low okay So this is already saying this is bearish the expecting bearishness. So price went up So if this goes down if the expecting price bearishness of pricing in this The pricing in these numbers Awesome minus Unless job kits cool, that's nice But considering the pricing in these numbers You want to look at It's like what the actual numbers being I mean you don't really care to be honest You just know that's the time you want to do the opposite of what price is doing And you're expecting whips or so if you miss the first leg wait for it It'll backtrack the other way so don't enter in on like a high Just finding when that will stop better retracing and now it's sort of attacking that a little bit here. So Probably cut through here. I want to see it relax at least at least here But I mean it could cut in midway push up Come back in push back down. So now we're getting into those aggressive territories as soon as we fall below these guys Like I said, it becomes Poignant to say that okay now these guys are underwater. Is it something that we want to be partied with? There we go. Nice little fall So we're gonna see how far it falls and whatnot But that's pretty much all I got for today I can leave you guys without knowing that you should be able to trade today Because my time is up anyway and We can look at this and reflect on this tomorrow Okay, thank you guys so much for coming. Hopefully you guys enjoyed today's stream found educational Any questions refill to ask in discord or get to them when I get some Other than that, I hope you guys have a lovely day and trade safe. I will see you guys tomorrow Peace


Oh, what's up everybody? Once again, it's brand man, Sean, and I'm excited to give you this snippet from an upcoming interview with a guy named Peter Prue. He is the founder of e-commerce empire builders. And when I say empire builders, I'm talking about he's helped people build million dollar revenue businesses online. He's built multiple himself. And why am I doing this video on this channel? Well, first of all, artists today are building empires online. That's exactly what you are doing, whether it's from your music or whether it's from merch. So learning about the digital space from different aspects outside of just typical music is something I'm going to be bringing more and more of to the channel. But this particular snippet right here is something you should all think about because we've heard of this issue happening before, this threat about your audience and losing access to your audience. But he had a real business making hundreds of thousands of dollars and lost access to all of his audience data. Exactly. You're exactly right. And that's exactly with Amazon. That's how we got like screwed over with Amazon, right? Because we had some false claims made against what we were selling on Amazon, completely fake. But Amazon just banned us, right? Took us a year to get the account back, you know, account back from that point. And think about it guys, like your fans, right, they're your fans, like those customers on Amazon. Those are my customers, but I never had that data. Right? Like I wasn't able to do it. So when you're building your business, make sure and the strategies where we talk about is going to like do this for you, but you need to pull them off of YouTube. Pull these people off of your podcast, pull them off of Instagram, right? Create an asset for yourself.  Like what's like, how do you, what do you mean pull it off? Well, like for one, like getting their email address, getting their contact information. A lot of people say that like email marketing is dead. Nobody checks their emails anymore. But I can tell you, like every business that I've ever had, right?  That email list is your, your most valuable possession, right? It doesn't matter guys, like the reason, like, for example, like, well, like, so look at your, like your best friend, let's say he emails you, right? You're going to open it, right? Yep. Right? You're going to open it. Same thing goes for us. It's the relationship you have with the people on your email list. They should be excited to open your email, right? Like, and you should be providing like valuable content for them or new music or whatever it is that you're, you're putting out for them, you know? A part of that. So that's a great point because one, I even did a video on this probably a couple of weeks ago, just the fact that no, email lists are not dead. Like, even if you look at the percentages of my traditional email openings or people I've worked with, it's on par or even greater. A lot of times then how many people see your Instagram posts, right? So that's one thing. But the second part is you said they should be excited to open your emails. That's up to you. Right? Like, you shouldn't be spamming your audience.  There's a lot of emails that I, I scroll right past, but there's some people who email me and I'm like, let me check this. Yeah, exactly. And the thing is like, I actually just had this conversation with somebody earlier today is like, think about, okay, so let's say email, you know, open rates, they're, they're all low, right? A lot of people don't open their emails anymore. It is, it's fine, right?  Like think about different people, how they consume things. Like for instance, like I have a YouTube channel, right? It's not huge, but I have a YouTube channel, right? I also have a podcast, right? I have people that listen to my podcast that never ever, ever been on my YouTube channel, right? I have people that are on my Instagram that have never been on my podcast on my YouTube. I have people on my email list. I've never been on any of those things. People consume content differently.  You get 20% on the email, you know, another 4% on Instagram, you know, another 5% on YouTube, right? You, you start getting them from other place. So that, that number, it goes, it goes drastically higher. Right? So it's worth having that asset because if you don't have that, like an email list or something that you control, because all of these platforms, like, like they can shut you down, like for no reason, guys. Like I felt it really bad because it impacted my blood, literally my revenue. But like, even like YouTube, even with my YouTube channel and like my podcast, I'm like, okay, how can I get people off this? Cause you never know what might happen. They might not rank your video for whatever reason they decide, right? All right. So that's a snippet, be looking out for the full interview, the link in the description below. We'll be active the moment the actual full interview drops, but there's something that he said right there. One thing that I really want to clarify when it talks about controlling your audience is not just about being able to control your audience. It's about being able to own your audience because there's two types of traffic online, right? There's a traffic that you have no control over, right? You might have a website, you might have a streaming page. They just come and you don't know how they got there, but they came, they're there good. And then there's control traffic and that's traffic where you're running an ad, right? You might have a Facebook ad, YouTube ad, whatever type of ad that you're running and you control that traffic because you picked demographic settings, right? And you might have even some sort of retargeting mechanism through a pixel or something like that. So you have a sense of control of that audience or that traffic and how it comes to you. But owning the traffic, owning the audience is when they come to the website, they come to this streaming page, whether they were controlled or not. When they came there, you figured out how to turn that into a ownership situation, whether that meant you were selling merch or something like that. So you collected their information or whether you just had some kind of free offering where they gave you their email to get it, whatever that is. But you now have this information where you own it. And that's something that you can take anywhere, any business. If Facebook dies today, if Instagram goes down, there's always going to be a new platform that comes up that you'll be able to market on using people's emails and, you know, as scary as it might seem, you can actually just send the people an email. But anyway, that full interview will be dropping soon. I'm excited for you guys to see it. Definitely stay tuned for it. If you liked this video, go ahead and like button. If you like it, you might as well share it. And if you're not subscribed, you know what to do. Hit that subscribe.


Hi everyone, my name is Asha Levin and I'm one of the maintainers of the Staple Kernel Tree and I wanted to talk about the various safeguards we have in the process to make sure that we do the right thing with regards to the kernel tree. So starting with the purpose behind the kernel tree, we have two main goals we want to achieve with the kernel tree. We want our customers to be happy and we want to do that by making sure that first we do not introduce regressions into the kernel tree and just as important as that we want to make sure that we do not miss any fixes that should be in the kernel tree. Now these two goals are a bit competing because we want to take as many fixes as we can in the kernel tree but we have to be very careful that we're not taking regressions and that's very tricky and we'll go over the stages in which we the stages which a patch goes from being written all the way to being released in the stable kernel tree and each of them we will talk a little bit about the various safeguards and ways we have to make sure that we're doing the right thing with regard to those patches. So the first one is the rules we have for the stable kernel tree and this is the most basic way we have to guarantee that we're both not taking any potentially buggy patches and that we're taking all the fixes. So the first rule is that the patches must be small straightforward and correct. It means that no complex new mechanisms, no new features, no new big piles of code and usually fixes are pretty small straightforward and correct right usually something that will fix say you're simply free or something that would fix a double free for example tends to be a one or two liner some of the fixes might be a bit more complex than that but really fixes should be simple just because they are fixes and what this rule helps us is to prevent from commits having two things only a mix of things we're making sure that the fix is only fixed but it's not also something that optimize a bit different part of the code. We also want our fixes to be already upstream it's important for us the fixes live they already existed in this string and it's important for us that those patches have passed this bar of being in this string. We do not want to fork the stable kernel tree but we also and we also want to make sure that anything that would take us past that minimal bar of being accepted into the string. When we backport commits we very rarely modify them because we would rather because of the rule we mentioned above which patches to be already upstream and if you modify a patch it means that it's not upstream anymore and it's just our version of the patch but it's also important for us not to do that because we want to rely on the testing being done upstream and if we modify a patch we don't really have a good way to test it again so we would often rather take series of patches that are already upstream rather than to modify a patch just to make sure it applies. It also means that in the future it's easier for us to keep taking patches because we won't be seeing conflicts as a result of our modified backport but rather patches we just apply as they would on the linsa string. If we do end up with a case where a patch has to be modified or a patch is really big and scary sort of like what we saw with all those speculative execution vulnerabilities last year and two years ago we require really good documentation and really solid testing as to why these patches needed and how the backport to older stable branches is different than upstream. We usually want the maintainer of the subsystem to sign off and we want to see which tests were done exactly again to make sure that we're doing the right thing with those patches because they haven't passed upstream testing we want to make sure that we're taking something good into our stable kernel tricks. So the first part begins as one is still writing this patch you are writing your patch and you might send the first or second version of that to the mailing list and when that happens with patches that have a stable tag in them usually maintainers and reviewers will pay more attention to them they know that those patches are more important because they'll end up quickly within users hands in their environments and running their workloads and it's important for maintainers to make sure that those patches indeed do the right thing and they indeed fix the bugging question and didn't not introduce any new regressions. Similarly if you sent a patch that might fix something but it doesn't have a stable tag or a fixes tag we might ask you or the maintainer might ask you to add something like that to indicate that this patch indeed should go to stable trees and to help us make sure that we're both back ported into the stable branches where it's relevant and to make sure that we're not trying to backport it into stable branches where it should exist. This is where stable tags with version number or fixes tag indicates which commit broke what you're trying to fix is very important. We also have a simple bot that fetches stable tag patches for mailing lists and attempts to apply them on our various stable branches where the stable tag and the fixes tag indicated there should be and that bot will send you an alert if something goes wrong. So for example if you indicated that the patch should be applied to the 414 stable branch and we did it on the bot couldn't apply to the 414 stable branch you will receive a mail indicated what happened and it might offer suggestions and to which patches are missing to allow it to apply or build cleanly and the purpose of that is to make sure that that we get responses what we found that if we sent these mails only one we actually fail applying the patches it's usually two or three weeks after folks have written a patch and they kind of moved on to working on something else and it's hard to drag them back into looking at the patch and trying to apply it on stable branches but if we send this alert only on early on when the patch is still in development we find that we get more responses because people are still working on the patch and are encouraged to go back and figure out what's going on with stable branches. The last point is something that is really a property of Linux's stream and it's that you can always send a patch in there's they could always send a fix in there's never a bad time to send a fix you can you don't have to wait for the merge window you don't have to wait for release candidate cycles there's never a bad time to send a fix upstream and you really shouldn't sit on the fix if you if you think it's good to go there's no reason to wait for anything if you have a fix send it upstream and we'll be more than happy to review it so once your patch was reviewed and it was accepted into a maintainer stream it will also usually end up in the Linux next tree where it will be hit by a battery of usually bots that test those trees other systems that kernel CI also built individual maintainers trees and run tests on them so really once the patches in the maintainer streets exposed to a battery of bots testing it which is a good first bar to make sure that we clear all the stuff that automated testing can find and once we're once the patch makes it into a Linux next we also have a derivative tree named stable next which is just the stable text commits that they could live in linus in Linux next but don't live in the system and the idea there is that you want to make failures in the testing of stable tech commits very obvious we don't want those failures to be swollen by other failures in Linux next so the stable next tree is a very good way for us to make sure that we're seen very clearly failures caused by stable tech commits and we could address them properly before they're being pulled into the stable trees to stable cure actually in the release of the stable tree now at that point if the patch has a fixed stack that points to a different patch which is a fix we might avoid taking that other patch into the stable tree because we know that it's buggy and there's a fix waiting for it in the maintainers tree we can't take both patches at this point because this new patch is still isn't upstream and that's one of the rules of the stable kernel tree but what we can do is delay the inclusion of the other patch until this patch is ready to go and until any discussion around this patch has been settled so once the patch has passed the maintainers tree and it's exposed to linus's tree it's now getting even more testing and now the testing they're becoming there's more humans running this test now it's actually a lot of developers who are taking this kernel tree building it running through the laptops running it on their servers so there's a lot more exposure to different workloads and a lot more exposure to different hardware and this is sort of the bar we're trying to achieve when we're taking patches for stable we want to make sure that the patch was at least tested and reviewed enough to make sure that it got into linus's tree and we really appreciate the work people are doing testing in this tree and it's important to us that anything that goes into stable was tested by those people and really appreciate the work being done there if the patch doesn't have a stable tag but we suspect it might be a fix we will run it to our autosale bot so autosale is a neural network which identifies patches that might be fixes by looking at the commit message it's looking at the author of the commit people who signed off on the commit it's looking at which files were changed and it's looking at the various code constructs and it tries to guess whether a certain patch is a fix even though it doesn't have a stable tag in it and I will also talk about how it works more next slide at this point also if the commit has a fixes tag which points to a different commit that didn't make it into the stable trees yet we will sort of work with those two or three or as many patches as it takes as one unit and usually one we will merge we will either merge all of them together or we won't merge any of them just to make sure that whatever we merge is completely fixed and tested it's not always the case sometimes we have scenarios such as a fix that is the fixes white spaces or indentation where it's not interesting for stable trees but usually the cases that we would take all commits based on their fixes tag together to make sure that we never introduce regressions into the stable kernel tree so now that the patch is upstream and in Linsus tree usually most authors are happy at this point and this is their mission accomplished they did their work but for us really this is where the real the real work starts so for every patch that we consider for stable the first thing that happens to it is that it's being carefully reviewed by one of the stable maintainers we look at every patch manually no bots and we make what we try and make sure that the writing is being done there and the patches really appropriate for stable for patches that went through the autosale process or the neural network suggested that they're relevant for stable we will kick them off for another round of reviews that where we allow at least another week for folks to object and to comment on those patches just to make sure that we're doing the right thing with autosale patches when the patch has been queued you also get an explicit mail you want to make sure that people are aware that their patches point through the stable kernel subsystem or stable kernel workflow and we want folks to comment on whether we might have missed a patch whether the patch isn't appropriate for the branch it was queued on we're basically trying to make this whole process very explicit for users another thing that would happen is the dependency chain analysis process so when we try to apply a patch and to order a kernel version it might not always apply cleanly and we do not want to modify this patch so instead we would look at the dependency chain of patches that are required for that patch to be applied on and you can take a look at the data tree basically a library of dependency chains and see how it works the idea there is that we would rather take a few more patches to make a certain patch apply and build and test cleanly rather than modifying a patch but another benefit from this process is that it helps us find other fixes that were lost usually looking at the dependency chain we can see that we missed a different patch that fixes an issue in that area we also should take it so really it's also a way for us to make sure that we're missing less fixes on other kernels so now that your patch has made it into all the way into the stable queue this place is very similar to linux next where it will be tested by a lot of bots that try and build this queue it receives the same quality of testing that linux text receives where this is a tree that gets generated very frequently and usually bots run it and it's a way for us to detect issues that come through the issues that patches that were just introduced into their queue rather than been there for too long it also give the scale of this queue is very small relatively compared to linux next so it's actually much easier for users to see if their patches which were missed or is the patch that wasn't backward correctly it is just an easy way for folks to do sanity check on our work and make sure that we're doing the right thing next we have the release candidates so this process happens about once or twice a week or when we try to do a stable release the first the most important thing that happens with release candidates is that you will receive yet another mail saying that the patch made it into a stable release candidate and this is again to make sure that folks are aware of what's happening and that they can object or they can comment on their patch being on the patch going into the stable tree and another important thing that happens here is that these stable release candidates are being tested on real workloads so rather than just developers or bots it's actually going to real users of the stable tree it's gonna run real workloads it's gonna run on real data centers it's gonna make sure that we don't regress real users with real workloads this is a very important step and it's very different from what we saw in linux tree or what we saw in linux next we're here it's exposed to the actual end users of the patch so the tests end up being much more comprehensive they're very different from tests that were done on developer laptops testing linux linux's release candidate cycles here it's real live it's real workloads and a question that keeps coming up often is that how can I make sure that my workload isn't being regressed by new releases of the stable kernel and an easy answer here is just replied to the release candidate mail that we're sending out feel free to test the stable kernel with your workload when you receive the stable rc email and if you see any issues just report them back to us and we will make sure that we're addressed all these issues we will never release a kernel when we're we know it has a regression so really you have enough time to test your test the new stable kernel with your workload and just come back to us report that everything okay or if you see an issue that the issue exists and there's no standard template for this and it's very easy to do you can just either reply to the mail you can reply to us privately if there's an issue with talking publicly about your work it will make sure to address it it's a very important goal for us to make sure we do not regress a stable kernel tree and this is sort of the last step we can do to make sure that it doesn't happen before we release the kernel so please if you have workload that's sensitive to changes in the stable kernel tree just test it during the release candidate cycles and report back whether we broke something or not and it will help us make sure that we're releasing a kernel that doesn't have bugs and it doesn't have new regressions in it so now after we've released this stable kernel it doesn't mean that our work is done we still keep monitoring the both upstream and we monitor the mailing list for new patches that might fix patch or fix a commit that would have an unstable kernel tree we look at bug reports that show up on a stable mailing list both for issues with the patch or issues with the back port process of the patch and we try to address it very quickly to make sure that regressions don't live for long in a stable kernel we really keep monitoring the work we did in the past just to make sure that any new issues that come out of it are addressed very quickly to make sure that there's no regressions in the kernel tree and beyond the work that we're doing directly with with these commits and this workflow an important goal for us is also to make sure that we improve the kernel's testing validation story that there's a lot of arguments that the stable tree should be should be seeing very minimal changes as it's stable and it shouldn't be receiving a big amount of patches but that also means that we're missing important fixes that go to the stable tree and I think that the way to address that is not by taking less fixes is by beefing up the kernel's testing story and we do that by working on projects such as kernel CI and zero data make sure that they're healthy and the work is being done there is solid and good enough to make sure that it helps the kernel's validation story we also review downstream trees so we often look at we put this kernel tree we look at fedora's kernel tree and other kernel vendors just to make sure that if they take fixes which did not exist in the kernel tree we should probably consider taking those fixes as well so we often review downstream kernel trees to make sure that we haven't fixed any that we haven't missed any fixes that those vendors took in and similarly for backtrackers we follow both the kernels bugzilla as well as vendors backtrackers to see if users report real issues with the stable kernel tree and those vendors take fixes for those issues we want to make sure that those issues are being taken into the upstream stable kernel tree as well and another thing we've been doing more recently is reviewing all the commit ranges for older stable branches just to make sure that we haven't fixed any that we haven't missed any fixes going into those older kernel branches and this is sort of an ongoing work so if you see a lot of patches for older kernels showing up this is why and please comment on them if you think that we either missed any fixes any like older fixes or if we've taken a patch that we shouldn't have into those older branches so maybe to summarize here the process the stable kernel takes has a lot of safeguards in place just to make sure that we don't mess anything up we're really careful about not to induce any regressions and i think that the safeguards have listed are are more than enough a really good measure to make sure that we don't have any any regressions in the stable kernel tree the end result of the stable branches is that they're way better tested than lunacy's tree right because they're seeing actual real workloads rather than just developers trying it on their laptops based on historic data of regressions in the stable kernel tree we can see that the regression rate is very low compared to lunacy's tree so there really shouldn't be an issue being afraid of waiting to a newer version of the stable kernel tree and really usually should be upgrade upgrading very frequently and not fearing of of us introducing new regressions it's also the case that when moving forward between major versions of the stable kernel tree so for example moving between 419 to 54 we guarantee that there won't be any new regressions introduced by us because we never backport a patch to an older version if it doesn't exist in the newer version so for example if we have a fix we will never apply it to 419 but not to 54 and this is for this reason if users upgrade from 419 to 54 we make sure that they won't be seeing any new regressions that were fixed in the solder kernel so really upgrading to newer stable kernel tree should be a very easy process and users shouldn't be worried worried about doing this if you do encounter regressions please do report them to us and we'll be more than happy to take care to take a look and try and address them with regards to missing fixes we also have a lot of mechanisms to make sure that we're not missing important fixes that go into the stable tree we have a few safeguards in place to audit the all the commits that go into linux's tree and to make sure that anything that might be a fix gets reviewed by us and if it's indeed a fix that's relevant to the stable branches we'll be taking it in the autosol process which i've mentioned is an important process that found a lot of fixes that doesn't have a stable tag and it's really improved the the amount of fixes and decreased the amount of fixes we miss that go into the linux's tree but not into the stable tree for patches that have a fixes tag even without the stable tag we will still review them just just because of that fixes tag to make sure that if the fixes tag points to a commit that's in the stable branch we will actually look at that other commit to validate that whether it needs or it doesn't need to be in the stable tree as well and similar story for downstream trees downstream vendor trees we really audit those trees very often and we really do our best to make sure that there aren't any there aren't any outstanding fixes in the stable back in the state sorry there aren't any outstanding fixes in the in those vendor trees that aren't that don't exist in our stable branch and that's it i'll happily answer any questions or comments that folks have thank you everyone thank you for listening and i'll be happy to answer questions um self sarang is asking with regards to fixes and testing the stable trees run specific test cases to confirm a patch correctly fixes the problem and b there is no other regression introduced so there are no test cases on the patch by patch basis but we do run systematic tests on the stable trees before they are released and users are more than welcome to plug into those tests and basically gait releases of stable kernel trees if we cause regressions so if you fixed an issue that was a regression before and you provided us the fix and we took it in the stable trees you can easily add the test case either in an existing system such as kernel ci or ltp or so on or test them yourself if it's a use case that matters only or mostly to you and we will make sure to look at those results before we do stable kernel releases to make sure that we haven't regressed the fix if there's no other questions i'm sorry there we go florian is asking where is the whole process documented so the stable tree process has some docs in the commutation folder in the kernel tree it's not completely it's not fully complete with regards to the process it's more around the rules we have for the stable kernel tree i don't think there is a full documentation as to just the general flow of everything all right so ph is asking i'm new to kernel development and create the patch and send it out who gets the patch first the maintainer or developer so maintainers are also developers so usually what happens is when you send a patch out it goes to a few recipients who are who are listed in the maintainers file and those folks are both the maintainers for the subsystem as well as reviewers who help maintainers so ideally your patch would get out to all the people listed in the maintainers file basically it doesn't matter if they're the maintainers or if they're reviewers for a certain subsystem so before you send the patch out make sure you send it to the right people rob is asking there seem to be a lot of lf projects like cip that are trying to use lts kernels is there any coordination between teams yeah definitely projects like cip contribute back and they base their work on the lts kernel trees there's a lot of cooperation we work both on making sure that we have all the patches we need to integrate into stable kernel trees as well as we both work on initiatives to help testing of these kernels for example the kernel ci initiative where both cip and the upstream process kind of come together to work on making on improving the testing for the stable kernel trees as well as approving testing for scenarios that are answers to cip and other members of kernel ci i'm sorry may questions are a bit off there you go Alexander is asking how about using sysbot 3 pros as regression tests so that's something we're looking at i think that the plan was to integrate them into the case health tests right i think at the very least we want to be running them maybe in the context of a different test framework on zero day or kernel ci we talked about this for a few years i think there are some snags here because i don't remember what happened but it's definitely a test we want to have it's worth following up on why it's not there yet i think the meter will know better the meter view of the is behind all sysbot work i think that we definitely want to have that as a regression test now adam Ford is asking when they have the fixest tag added to a patch i've seen them apply to stable kernels without seeing stable some cc stable what is the rule and went to and not to cc stable so the rule is really if you want your patch in the stable kernel please cc stable explicitly we try and review patches that only have a fixest tag as well patches that don't have any tag at all to make sure that they're not missing the stable tag so we may take them in manually but really the process we should be doing for having patches included in the stable kernel trees to cc stable explicitly and you can have both right you can have a stable tag and a fixes tag in the same patch and that's sort of the preferred way to do it and that way we know how far we should backport your patch based on the fixes tag but really the the cc stable tag is really important and you really should use it to make sure that your patch ends up in stable trees so rand is asking could you share a link to know more about stable tree testing i don't have a single resource but maybe let's chat on slack after this and i'll try to collect a few links that can help i'm not aware of a single resource for that pavel is asking is there a way to mark patches is not for stable that's a great question i don't think that there's a standard way right now but we can definitely add that in usually we find that having a comment inside the patch like in a commit message enough if you just write don't include this page in stable it's nice if it explains why it's not stable material otherwise i don't think there's like a standard tag for that all right i'm not seeing any more questions so i guess i'll close it up thanks everyone for coming and listening to the talk and i'm available on slack or email if you have any more questions comments or if there's anything i can help with please let me know thank you very much


I remind members of the Covid-related measures that are in place, and that face coverings should be worn when moving around the chamber and across the Holyrood campus. The first item of business this afternoon is time for reflection. Our time for reflection leader today is Professor Sir Jeff Palmer, Chancellor of Herriot-Watt University. Bear with us members. We'll just suspend shortly. Thank you. It is a great honour to speak to this Parliament today. Scotland has historical links with slavery in Jamaica, but today our flags have different colours but are the same. We are different but the same. My late dear mother migrated from Jamaica to London in 1951 and saved £86 to pay my fair to join her in 1955. She got me a job in a shop but was told by the authorities that I had to go to school until I was 15 years old. I was 14 years and 11 months old. One school rejected me. They said I was educationally subnormal. A secondary modern school took me and later in 1955 I was transferred to a local grammar school because I was good at cricket. I left school in 1958 and worked as a junior laboratory technician at a college in London. My boss, Professor Chapman, allowed me to improve my qualifications but had to help me to enter Leicester University in 1961. I left Leicester University with an honest degree in botany in 1964. The only job available was peeling potatoes in a restaurant. My potato peeling ended when Professor Anna McLeod at the Herotwatt College offered me a PhD place in 1965 and in 1967 I gained a joint PhD degree from Herotwatt College and Edinburgh University. I worked in a research institute from 1968 to 1977 and returned to the Herotwatt University as a lecturer in 1977. It has been a privilege to have contributed to the science and technology of serial grains, taught many gifted students and helped in 1989 to set up the 1.3 million international centre for brewing and distilling at the Herotwatt University. I started my charitable work in Scotland in 1965. Recently I contributed to a video made for this parliament called We Are Scotland. We are Scotland indeed. My DNA says I'm in part a Viking from Shetland. Sadly, the past has given us racism and I'm honoured to be chairing various groups which are examining Scotland links with race, slavery and colonialism. Although we cannot change the past, we can change consequences such as racism for the better using education. Finally, it is a great honour to be Jamaica's first honorary council in Scotland and I must thank my mother for investing her 86 pounds in my education. Education works.