All right, well, folks, welcome to our presentation on power leveling over 9,000, improving application performance with some chaos engineering. Hello, everyone, I'm Saiyam Pathuk, Director of Technical Evangelism at CUO, building the next in simplified QBinary service on CUO. I'm a CNCF ambassador and influx ace. I'm author of Learn CKS Scenarios, which is published on Gumbroad based on practicing Kubernetes CKS certification. Yes, I am CKA, CKAD and CKS certified and I organize a few of the meetups at Bangalore. I run a YouTube channel where I talk about all things cloud native and you can reach me on Twitter anytime at the rate Saiyam Pathuk. And I'm Karthik Gekwad. I'm the head of cloud native engineering over here at Verica. And lately in the last probably eight months or so, I've been doing a lot of work in the chaos engineering space, building chaos engineering verifications. I've written the learning Kubernetes and other cloud native courses on LinkedIn learning. I used to manage the Oracle Kubernetes engine over at Oracle Cloud and also do a bunch of things in the DevOps and cloud space here in Austin, Texas. You can find me on Twitter at Erasion One. So let's talk about the storyline today. So we'll do a quick overview of the state of the cloud native universe, figure out where chaos engineering fits in and then kind of talk about what we mean by chaos engineering, it's intersection with cloud native and then the most important part, the demos of course. And finally we'll go into some speaker recommendations at the end. So state of the universe. So I'm a big fan and I'm kind of a nerd when it comes to looking at reports and things like that. So the state of our world is we're kind of going to a multi-cloud world. We've known this for a while, but in the research report from 451 research, this is kind of, it's a little bit older, came out in 2018, 2019, that timeframe, but it's at 67% of enterprises will have a multi-cloud or hybrid IT environments by 2019. But with greater choice brings more complexity. We saw this again in the CNCF report that came out last year, where 26% of usage is actually multi-cloud. So that's grown a lot from years past, which if you notice in the graph, there was nothing in 2019, but 2020, that's a brand new thing. There's also increased container usage. So this is no surprise to anybody. The use of containers in production has become the norm today for the survey. And so there's steady increase in use of containers in production today, as well as everybody and their parents and sisters and brothers and coworkers are using Kubernetes. It's kind of become a ubiquitous thing. So that's kind of where we stand from a cloud native standpoint. But also in the 2020 CNCF survey, they talked about cultural challenges, where cultural challenges and complexity were the biggest challenges with respect to using containers. Actually, let's talk about complexity for just one second. So on the complexity scale, you have the simple systems and then you have the complex systems. So simple systems are defined where the process flow is very linear, you have predictable output and everything is comprehensible versus more complex systems are, everything doesn't really happen in a linear fashion. So things are non-linear. You have a lot of unpredictable behavior, so you don't really know what might happen.  One of the really interesting things that I found from this was, in the past 10, 15 years ago, you'd have one architect that was able to understand the whole system. So when something went wrong, you could go to that one person to ask him or her, hey, what's going on with the system? And you might get a response of like, oh, there's maybe the messaging service or something like that might be offline. But in current world today, when everything's a little bit more distributed, it's hard for one single person to have like a complete mental model of the entire system. So it's become a lot more complex. And testing is hard. So this wouldn't be a conference. I'm guessing this slide is gonna be in every presentation, but you have your productionized, distributed application on the right, large container ship, and then you're trying to test this. And we're all kind of like using, probably using JMeter to like load test our applications. So you have the little like forklift on the, on the left-hand side of this image. So, you know, testing is just a hard problem in a distributed world today. So enter chaos engineering. How does this help us? So chaos engineering, it's defined as a discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production. This definition comes from the principles of kiosk.org. So go check out that site. So let's talk about testing versus experimentation real quick. Testing is the idea of making an assertion on a property of a system based on existing knowledge and then validating that specific property. Experimentation on the other hand is proposing a hypothesis which can either be proven or disproven. So as long as the hypothesis is not disproven, then the confidence grows in the actual hypothesis. If it is disproven, then we know something's wrong and we can figure out why that hypothesis was wrong in the first place. Okay, so you were trying to practice chaos engineering in the real world. So how do we actually make this happen? So for the principles of chaos, you're first trying to define a steady state. So you know what normal behavior might look like in normal conditions. And then you take your hypothesis and you hypothesize on the steady state which will continue for both your control group and your experimental group. Now that you have two groups, you start introducing variables to reflect real world events in the system. What does this mean? So you might like in the Kubernetes space, you might start to delete pods or take a note away, et cetera on the control and your experimental group. And you start to inject real world chaos to it. And then you try and disprove the hypothesis by either looking at the difference in the steady state between the control and the experimental group. So thinking about this from an advanced principle perspective, you really want to build your hypothesis around steady state behavior. So you want to think of this from an outside in approach. So you want to look at your measurable outputs versus internal attributes. What does this mean? So you want to look at, if you have an application running, you want to look at things from the outside like traffic coming in, response rates and things like that versus looking at something inside of your application maybe like CPU or memory for example. You really want to vary real world events.  So once again from an outside in perspective, but you want to change things that might happen in the real world.  Also blast radius. So I just said that you are running experiments in production. So you want to remember that you're doing these things in production. So you want to minimize your blast radius. So in order to not take everything down. And last but not least, continuous. We'll talk about this as we go along. But really what I mean by this is you want to keep doing this from a continuous kind of perspective. So you don't run your experiments maybe once a release but you're really doing this on a day to day basis hour to hour minute to minute even potentially. So what's the intersection with this? The theory I explained with chaos engineering and cloud native. The coolest thing about cloud native is that we are in a rich ecosystem and there's a lot of great open source tools in the landscape. Some of them are, you know, Kraken, Litmus and Chaos Mesh and there's a whole bunch of others as well including enterprise tooling. So for the purposes of this presentation we'll focus on the first three because realistically we only have 25 minutes probably, you know, 22 by the time I'm like done talking about this stuff. So we'll cover the first three but definitely take a look at all the others as well. So first up Kraken. So what is Kraken? Kraken helps us inject failure into either open shift or Kubernetes. It's kind of built from two components. One of them is powerful seal that injects failures onto Kubernetes clusters. So powerful seal is the actual thing that, you know, injects failures or does something to your actual cluster. And then there's cerebrus that watches and reports on cluster components. So cerebrus is actually the watcher component. So when powerful seal goes and maybe kills a pod or, you know, slows down on a road, for example cerebrus actually sits there and watches board changes in your infrastructure. So it'll tell you what's actually going on. So Kraken is actually used for a bunch of scenarios. You can use it for pod chaos.  Same thing with API server pods, adding some node chaos where, you know you might simulate a crash node and see how your application or your cluster recovers from that. And then time chaos, for example where you screw, where you're skewing your clock time to see how your cluster or your applications might respond to that. Here's a quick architecture diagram. So Kraken, you have, we have it on the left hand side here. This injects failures and open shift. These are the different components that it can kind of work with. It also works with, you know, different cloud APIs. So if you want to use the AWS Azure, for example it can, you know, call out to those as well. On the left hand side, we have cerebrus that actually watches all of these different components as well. But Kraken itself, you know behind the scenes it uses powerful seal that, you know, actually goes and does the actual experiment parts with your workers or your masters, for example. Thank you Karthik for the awesome introduction to chaos engineering and why it is needed. Next up on the tooling is chaos mesh. So chaos mesh is a powerful chaos engineering platform for Kubernetes. It is extremely simple to use can be installed on any Kubernetes cluster and it is designed for Kubernetes. It has a interactive dashboards which you can use for creating the chaos experiments and also can be used for analytics as well, like which all points your chaos actually ran. Some of the disruptions that chaos mesh causes are getting the pod, latency, network, system IO chaos. So, and there are others as well. So all these chaos can be introduced within your, within your Kubernetes cluster. Chaos mesh architecture. So these are the set of experiments that are there with, which you can create as a CR and submit it to the Kubernetes cluster. So you'll be writing a YAML file, which is a custom resource and then you'll be submitting to the Kubernetes cluster. So after the control manager takes up the CR, it'll then hand it over to the demon and the demon is actually, the chaos demon is actually running on all the nodes which will internally run your chaos experiments. And these experiments then are based on certain selectors that selects the particular workload to which this particular chaos experiment has to be run. So that's how in a short summary, chaos mesh works. Next up on the tooling is litmus chaos. So litmus is again an open source tool. It's also a CNCF project and it is basically chaos workflows for Kubernetes. So litmus features are team collaboration. So you can integrate it with your GitHub and you can add your team members. You can add public private chaos hubs which will contain the chaos experiments. So you can enable chaos GitOps as well, which can act as a single source of truth. So whatever workflow you create in the litmus will be stored in your GitHub repository, whichever is connected. And then you can have the chaos observability as well. You can view the duration of the experiment when it ran on in the Grafana dashboard and you can also have the Prometheus as well. Chaos APIs that can be used and chaos workflows for creating the complete workflow of the chaos experiment that has to be run on the cluster. Litmus architecture. So this is litmus 2.0 architecture. It has a portal and portal comes with a web UI. So web UI is very, very great component of litmus 2.0 with which you can create the chaos workflows from the UI itself and you don't have to write any of the YAML files. And then you have the litmus server and you also have a litmus DB which stores internally all the workflows. And if you have enabled GitOps then also your workflows are stored in your Git repository. So you have chaos agents on your doors and chaos agents are set of the operator in the CRTs, the chaos probes, the exporters exporting the metrics and the subscribers to which the applications can subscribe. And here you have the chaos experiments and the chaos workflow.yaml. So you can also edit the YAML when you are creating them and you can enable, disable some of the feeds. So this is a complete runtime workflow for litmus. So you can see here the CRTs. So there are three CRTs, the chaos engine. So we write the chaos engine file, the chaos experiments, the chaos experiments are installed onto the cluster and then the chaos results. So chaos result will tell the success, failures and what are the metrics for that. So you have the chaos operator installed onto the cluster, you write the engine.yaml, chaos engine.yaml file. And so user creates a chaos engine CR and maps it to an application to which the chaos experiment has to be run. So then the chaos operator will read that file and create a chaos runner pod, which is actually reading all the configuration, mapping it all to the workload which it has to run the chaos experiment on. And after that it triggers a chaos job. Now chaos job is actual experiment that runs and performs the chaos checks and do the experiment and then hands over the result. Chaos hub is the place where you have all your experiments stored. So you can use the public chaos hub or you can create your own chaos hub privately and link that to the litmus portal. You can also attach multiple target clusters to litmus. So it's not only limited to single cluster, you can have multiple clusters and select targets for different workflows. So this is how all in all litmus works. Demo time. So this is what we are going to do in today's demo. So I'll be launching a CEO Kubernetes cluster, which is K3 is based. We'll install litmus 2.0 and we'll create a pod network loss experiment. And we'll also enable GitOps with the repository and we'll install flux. And again, it will be watching the path, a specific path in the GitHub. And we have the GitHub actions which will be used to build and push the Docker image and also commit the deployment file to the same Git repository to which the flux will read it and deploy it onto the cluster. And after that, the application would have been subscribed to a particular pod network loss workflow. And in the black and using the black box exporter and from ethios will be able to visualize everything in. So for this demo, we'll be using CEO Kubernetes and we launch a cluster. Let's name it cube con. We'll select a large size and we'll just create the cluster. So the cluster is ready under two minutes and I have already downloaded the cube config file and exported it as a cube config variable. Let's set cube CDL, get nodes and our cluster is up and running. So I have already cloned the repository and let's not deploy the components required for this demo. So the first one is the deployment of litmus. So we'll create cube CDL, create namespace litmus and we'll deploy litmus 2.0 beta. So it has created all the components. So let's check what is happening in the litmus namespace. So cube CDL, get pods, iPhone and litmus. So we have our database, frontend server all the pods running for litmus. Now we deploy the application. So cube CDL apply, lower application it will create the deployment and the service and we'll deploy some monitoring components. So cube CDL apply, I can have monitoring Prometheus. So it is creating all the Prometheus components and we also need black box exporter approving the HTTP endpoint and we also need Grafana. So it creates a deployment and service. It has also a JSON dashboard that will import later which will not be applied to the cluster, obviously. Let's check the application is running or not. Our application is up and running and let's access the application. So our application is up and running and here it is. So now let's access the litmus dashboard. So this is running on the node port 30406. So this is the litmus dashboard and the default username and password is admin and litmus. We'll change it. We have to do a project name, keep on. So this is the litmus portal when we log in. So it has workflows, it has the hubs and the target clusters. So first let's enable GitOps for our repository. So we'll give the Git URL, we'll give the branch as main and we have to give the access token and we have successfully connected to GitOps. So all our workflows will now be stored in Git repository. So let's create a workflow. So we schedule a workflow, we select target cluster which is self cluster and we create our own workflow. We give it a name, network loss and we select the experiment from Kiosk Hub or network loss, we go next. So we select the application based on the app label which in this case is Hello Service. We'll run this experiment for 120 seconds and since we are running CO-Cubinities which is container debased. So container B, the socket path will be slash run K3S container D, container D.soc. So let's add this experiment, finishing editing and construct the workflow. Now this is the YAML file that you can edit as well. So we edit the monitoring section. So we want to enable the monitoring so that it can flow and tell us what the exact Kiosk duration was when the experiment ran and let's schedule the workflow. So the new workflow network loss has been successfully created on our Kubernetes cluster. So let's go back to our Kubernetes cluster to see what is happening. Let's check the written name space. We can see the network loss pod is initializing. After that it will create the runner pod and actually run the network Kiosk. So our workflow has ran successfully and all the network loss pods have done their jobs and we can see the workflow succeeded. Now how to visualize this? Now let's go to our Grafana dashboard. So I have already imported Grafana and connected it to Prometheus and loaded the JSON. So this is how it is showing the probe duration seconds and you can see the graph which is having the network loss. So the red area that we are seeing is coming because we said the monitoring true in the YAML file. So monitoring true in the YAML file will give the metrics the proper chaos duration where actually the chaos ran for the particular workload. So this is that particular duration for which the chaos ran and it helps to visualize when you have a lot of chaos running in your cluster. So that's how you can run chaos workflow and visualize it. Now let's do a CI CD for that. Now in the repository, I already have a workflow section which is the build and push image. So this is the GitHub action which on the push commit to the repository will build the image, push the image with a tag using a Jinjar template. It will edit this particular YAML file as well. So there's a template, which is in the TMPL section. So this template, it will update with the deployed tag and again commit back to the same location. Now for the CD part, we'll be deploying flux. So flux is another open source tool which is very good for continuous delivery. So let's deploy flux to, let's bootstrap flux and with this particular repository, it will create the flux system namespace and deploy all the components. So we can see in the KubeCon repository, we have the litmus folder created. So when we did the workflow and we did the GitOps for litmus, so it gave the workflow, it created the workflow inside the repository, it pushed it and we have the workflow ID and the cluster ID. So we need to have these two so that we can subscribe the deployment for this particular workflow. So we'll copy the workflow ID and we'll edit our template and we add the workflow ID over here. So now when there'll be a Git push from the GitHub actions, it will actually be keeping the workflow ID same so that it runs the workflow, the same workflow whenever the new version of the image gets deployed by flux. So the bootstrapping of flux is finished and we can see everything is there. The flux has been bootstrapped, KubeCVL get pods, hyphen and flux system and we can see all the pods are running. So what we have done is we have bootstrapped flux with some of the components on the repository KubeCon branch main and the path deploy. So the path deploy will be taken and anything that gets deployed or edited for the deploy folder, it will apply the new version of the application onto the cluster and in our application, we have added few annotations. So in the annotations, we have GitOps as true and the workflow ID we just pushed to GitHub. So if I do a Git pull, it will give me all the latest details. So for the deployment, we have added some annotations which is the GitOps enabled and subscribed to this particular workflow. So anytime the new version of this application deploys, it will trigger this workflow which is our network loss workflow. But we can see that the previous pod has been removed and a new pod based on the commits that we have done is already deployed onto the cluster. And if you go back to a workflow, we can see a new workflow has just started trigger. And if we again go back to the Grafana dashboard, we can see it live in action. So we can see this red bar has already appeared and the chaos workflow is starting again. So that's how you can use litmus chaos using your GitHub actions and continuous deployment, continuous integration. So all these things will be helpful when you are running for a production environment. So you will be having on every commit or on a specific commit or in specific action, you can run various different workflows and your GitHub repository acts as a single source of truth which has all the workflows that you created.  And if you edit anything from this workflow, the other workflow will automatically be updated. So that's it for this demo. Hope you like it. Please try it and let us know how you work with this particular demo. Simon, that was awesome. It did look like we went through a lot of things in that demo. So let me see if I can break it down and see if I can get this right. So we had an application that we deployed and then we ran litmus 2.0 on the cluster and we decided to run the pod network class experiment. So we ran that a couple of times just to make sure it was running. We saw the output of that in Grafana that was using Prometheus behind the scenes to go and pull the metrics from the cluster. And then when we wanted to run that continuously, basically meaning whenever we did a push to the application that was using a GitHub action, we pushed the image to Docker Hub, et cetera. And then Flux was basically configured to go and pull the commit hooks from the deployment from the file and git. And then Flux would rerun the actual litmus experiment again. So anytime you pushed your application, once it got pushed into Docker Hub, Flux would rerun the experiment again, correct? Yes, we have used the annotations so that the subscribed workflow runs. So bang on. That's awesome. Yeah, and I like it a lot because it allows us to basically tie the continuous aspect of running a chaos experiment or chaos verification over and over again. So litmus is the actual runner that does the chaos in your cluster or chaos free application. But then we're also using Flux as well to do that in a continuous format. That's great. All right, well, let's wrap up. We talked through a bunch of things in this specific talk. So let's give our recommendations for just chaos engineering in general. So I'll start by saying that the whole idea of chaos engineering is pretty nascent. It's still pretty new. It's kind of like where DevOps was maybe in like 2009, 2010, that timeframe where we're still trying to learn and come up with ideas for how a lot of this is gonna be implemented. It's really important to get the basics and fundamentals right of chaos engineering. So learn the theory behind it versus running to implement something specific upfront. This is like the best time to talk to the maintainers and the practitioners in the space that have the ideas and are actually doing a lot of these things in the space. So come talk to us, talk to the folks, the litmus folks are all here at QCon. It would be great to kind of understand their thinking behind the scenes. Probably the my most important recommendation right now would be to communicate your experiments because a lot of this is, it's recommended to run these tests in production, but it's also important from an organizational perspective, from a team perspective to be able to communicate that, hey, we are running chaos engineering tests in your cluster on production, for example, so you don't get into a weird corporate or a weird organizational kind of trouble or whatever. So that's my big recommendation. There's a bunch of like community stuff and community groups and meetups that you can join. Put a couple of links in here. And also I talked about theory. If you want to get the chaos engineering group book from Casey Rosenthal, you can get that at verica.io slash books or slash book. So just to add on to Karthik's recommendation, I have some of the things like chaos engineering is expected to grow even more. So it is new, but it is growing rapidly with the number of tools with everything that is growing in the cloud network ecosystem. And we should work towards continuous chaos. As in the demo, we have shown with the power of continuous integration, continuous delivery and chaos engineering implemented in that. So obviously work towards that so that you can have a complete life cycle integrated with chaos engineering. So keep an eye on the new tools and get involved in the community. There are a lot of new tools coming up for chaos engineering and the current tools are maturing enough to mask the needs of the production-grade clusters. Check out for the integrations that will help you in your developer workflows and CI CD workflows. Like Captain is a very good project and Captain plus Litmus works very well. Octito plus Litmus works very well. So you can try these out. They have a lot of blogs, a lot of meetups talks that they have done and others as well. Awesome. And with that, that is the end of our talk. You can reach me on Twitter at iteration one and then you can reach Siam on Twitter at Siam Patek as well. Any last words Siam? Yeah, definitely the try out the repository. It's public and you can try out the demo and you can try out on any Kubernetes cluster as well. So try it out. Let us know what you feel about chaos engineering. Any questions, we are happy to help and we are there in the Slack channels, Kubernetes, CNCF, you can reach us out anytime. All right. Thank you, folks. Thank you all. Bye.


This is covering the spread part of the Fandall podcast network It feels like we've had a lot of low scoring primetime games so far this year in 2023 But on Sunday or Monday night tonight between the Broncos and the Bills a pretty high total over a Fandall sports But 47 and a half for that game and will we actually see an over potentially? We'll talk about that today in previewing Monday night football Ryan Williams is with us for today to break down that game talks and play our props at Fandall sports book We'll break down the spread the total and much more to get you ready to close out your week on a high note This is covering the spread right here on the Fandall podcast network and Fandall research My name is Jim Sonis I am a managing editor of digital media for Fandall research joined here as I am every Monday by Ryan Williams Check him out on Twitter at Ryan Alexander underscore W Ryan fresh off a fantastic week in DFS So welcome to the show. How you doing today? Oh We're doing great Jim. We're doing great. I guess I'm wearing a stealer shirt should be wearing a cowboy shirt Shout out to vintage branding cooks coming through for the squad yesterday just unbelievable And you know, I talked about or I didn't talk about because we didn't talk about DFS last week But I was talking with other people about how I didn't know what to do yesterday like yesterday just felt like a day where Anything could happen and just the good teams were in great spots and you know, I thought about going to the Ravens again But but yeah, you know, shout out to shout out to Dallas taking the Giants to the woodshed woodshed once again And and making it a profitable day for the Williams household I'm happy to hear it. Love to hear it You mentioned Brandon cooks and like there were a couple times yesterday where I confused Brandon cooks for Kovante Turpin Kovante Turpin is like the burstiest Wildest guy in the league as far as like speed. If I'm confusing Brandon cooks for him in 2023 That was well for the Cowboys because if Brandon cooks can do what he's doing Well, CD lamb looks like a top 3 receiver in football now top 5 receiver football Like that's gonna probably bode well for them least against, you know, if they could play the Giants Right, exactly. I mean, that's what would be favorable for them for McCarthy and Jerry Jones, most likely Absolutely. It was fun to watch. I'm glad that you were able to benefit from it And hopefully we can close out your weekend on high note too with this Broncos at Bill's game We're gonna break down Broncos and bills here in just one second to get you ready for some Monday night football But first a reminder to make sure you're subscribed to the covering the spread podcast feed wherever you get your podcast We were here every weekday Monday through Friday in addition to some one-off shows from top Vecchio to previous Thursday and Sunday night football all those shows are right here in the covering of the spread of podcast feed If you like what you hear leave us a five start rating on Apple podcasts or over on Spotify as well You can find the daily shows here on a faddle TV plus and the faddle YouTube page Score early this NFL season with faddle America's number one sportsbook right now new customers get $150 and bonus bet any winning $5 moneyline bet That's a hundred and fifty bucks if your team wins if you've been thinking about joining faddle There is no better time to get in on the action. The app is so easy to use There is a wide range of betting options clean spreads player props totals and more So visit faddle a kickoff the NFL season faddle official partner of the NFL must be 21 plus and president select states Faddle is offering online sports waging in Kansas under an agreement with Kansas star casino LLC first online real money wager only five dollar pregame and money line which required $10 first deposit required Bonus issue is now a travel bonus bets that expire seven days after a seat C terms at sportsbook.faddle.com gambling problem call 1 800 gambler or visit faddle.com slash rg in colorado Iowa, Michigan, New Jersey, Ohio, Pennsylvania, Illinois, Kentucky, Tennessee and Virginia Call 1 800 next step or text next step to 5 3 3 4 2 in arizona 1 8 8 7 8 7 8 9 7 7 7 7 Or visit ccpg.org slash chat connecticut 1 809 and with it in indiana 1<PHONE_NUMBER> or visit ksgamblinghealth.com in kansas 1 877 770 stop in louisiana visit md gambling health at organ marillons 1 800 yam work dot net in west virginia 1 800 5 2 2 4700 in wyoming hope is here visit gambling helpline ma.org or call you 103 2 7 50 50 for 24 7 supported massachusetts or call 1 877 8 open y or text open lie in new york Let's dig in now to the broncos at the bills right now at faddle sportsbook The bills are seven point favorites total in this game is 47 and a half and ryan this spread was seven and a half Earlier all this week, but a sense tightened by a half point. It's now seven Do you think that movement in denver's favor is the right direction here? Yeah, i'm i mean Does it move in denver's favor or does it move in buffalo's favor jim? you know because it's like The winds of buffalo and i know the past two Winds have not hit this uh hit this spread of seven But when we're talking about traditionally like with josh allen and the buffalo bills winning like They go out and they win by seven points or more like that's what they were doing at the beginning of the season That's what they've done historically And so like the seven and a half really did not feel great for me to want to take buffalo at home here But now that it's at seven i'm like, yeah, I could probably I could probably ride with that number Denver is coming off of there by so, you know, you kind of think about what what kind of changes that they have made But this is a hostile environment to have to play in you know You're looking at the buffalo bills there and we talked about it with the afc like the chiefs weren't playing You know, uh yesterday the the ravens lose a tough one to the browns the bingles, you know We're you know reeling After they won four straight and so the bills have a chance to kind of you know Put themselves back into the conversation here at home And you know, I think that they'd be able to do so against Sean patin rustle Wilson and company and so while you know the hook might lend in denver's favor And we have a lot of you know the public money Coming on to denver um at that seven number And we'll see if we could trust rosa wilson, but I just have not been able to uh to trust the broncos in some time And you talk about you know playing a team consistently like We're Holding the chiefs to 27 points um across two games and I just you know The the chiefs are the are the chiefs, but they also haven't looked like themselves this year as well too So are we putting more stock into those games than we should be? And let's see what buffalo has in the hopper But I feel like they're they're gonna be reeling and want to showcase uh that they still belong in the conversation on monday night Yeah, that's where I'm at too. I think this spread is a little bit light right now I I did lay at seven and a half personally when it was there So it doesn't feel great that it moved back to seven But the reason that I was willing to lay that was because a my model does show a lot of value there I've got the bills by double digits so getting a seven and a half even was kind of you know It's not a number you love, but I do think that It's there for a reason and that reason is that this bill's offense is still playing very good football the defense They're struggling and I think that's sustainable. I think they will continue to struggle given that all the key players They've lost defensively. I don't expect them to suddenly turn things around there, but do we trust? Russel Wilson, Sean Payton to be the team That like punishes them for their middling level defense And I personally don't like we've had a lot of focus on the denver Defense like you said, maybe we're waiting putting too much weight on that kins city game But also like this offense hasn't been that good. They're a bit better ones who adjust for opponents so Like they're not bad, but if I look at just my 2023 numbers on offense They are currently my 18th ranked offense By my numbers again, that's just for 2023. I had a decent prior them coming into this year You know showing faith and also Wilson faith in Sean Payton. So they're about the same if you include the prior, but I don't know. I just I just don't have a lot of faith that they will be the team that really Exposes the flaws in this bill's team the bills Like you said keep their foot on the gas even when they're up because probably because they can't run the football So they kind of have to keep throwing but like they do have a tendency to throttle team So I did lay the seven and a half when it was there So don't feel great about it moving against me with it going back to seven But that's why I was there is that this this bill's team this offense is still very good I mentioned denver is 18th in offense for me Buffalo second with no prior involved. They're behind san francisco and above miami. That's where they sit right now for me So that's why I'm there and also ryan when you combine The bill's offense with still skepticism around the the bronca's defense with the large spread I actually did take the over one as a 47 now 47 and a half I can't get there anymore Because I've got a 49.39. So there's still a bit of value on my side With regards to the total but like maybe not as much there 47 is a key number So what's to read on this total with it now up to 47 and a half? Yeah, 47 and a half We we need buffalo putting up 30 points like that's just how I feel with with this with this spread Can can we get you know russia wilson to to dig back into the time chamber and and really come out here and You know put up 21 points for For for denver sure But yeah, I mean we want four scores here our score in every quarter from from buffalo at minimum To get to the 28 30 threshold in order to get this going now I will say you know part of what buffalo part of what buffalo is a killy hill has been is they have not been able to start Games out early in the way that they would want to I think you know over the course of their five games They've only had four the past five games They've only had four first half touchdowns and that's just not what we know of the buffalo bills Like they have to come out here They have to come out firing if we get the buffalo bills putting up like 14 points or something like that in the first quarter Like 21 points in the first half. I think we're feeling good about that 47 and a half number But that's kind of what we're that's what we're gonna go for I'm I'm willing to kind of take that shot because I actually like both passing quarterback props in this match up So i'm going to take the 47 and a half and go off of the reservation of what we normally been doing With the undersitting and insane rate. I believe we only have one monday night football game this year That's hit on the over everyone else has been on the under and prime time unders are just hitting at an insane rate this year So, uh, but what's you know, what's what's not to like there? I mean it is buffalo at home and you know the type of things that you know Leading us into week 10. It was just like again You got to get you got to get some of these like narratives and some of the things that have happened Like all season just kind of out of your out of your brain out of your memory Who we're going into new weeks and like this would not be a game where you know We would be surprised if buffalo put up like 35 points At home in a favorable matchup against the Denver team that we have not been able to trust since russell wilson has been there as the quarterback So yeah, i'm riding with you jim. Uh, I like the 47 and a half but in that same vein I'm trying to get every type of you know, bonanza number that I can get Um for the bills to be able to put up points in this in this manner because I feel like they're gonna they're gonna carry that The time well, let's talk about it them Let's talk about that bill's offense break down some player props on their side of things Dalton kinkade clio Shakir both got a pretty big boost recently With doson ox on ir so some of the bills side of things ryan given where things are right now Where do you see value for bills props for tonight? Yeah, well, I think it's it's on the two guys that you mentioned uh with doson ox being out You know pretty favorable matchup for for dalton kinkade here. Uh, this guy has been absolutely incredible I I don't know what the number off the top of my head But his catch rate has to be like 80 or above when being targeted. This dude just does not They're they're linking up every time that the ball comes his way. It's it's been incredible I love his reception number At five and a half on the over plus money there I just feel like this is a matchup where he could get involved against dimper Who's kind of you know, they're not one of the worst teams against tenants But they do allow, you know a little bit of production there To the tight end group and You know year of the rookie tight end. I know I think you so you sent a tweet out last night I'm about the rookie tight ends and it's just incredible and I don't even really think of dalton kinkade As a tight end. He's he's a wide receiver. He's getting in line out wide So love the dalton kinkade props. I am going to be interested In the other guys not stephane digs too from the wide receiver standpoint petrick Patrick sertan Probably going to be on stephane digs there I would think I mean coming out of the buy like what do you want to do you want to stop the number one receiver? So they'll do their best But you know stephane digs to stephane digs, but gabriel davis and khalil shekir's Numbers just aren't high enough like especially if i'm expecting points if i'm expecting You know buffalo to kind of come out here and be able to move the ball Then i'm liking shekir's number i'm liking gave davis number gave davis's air yards like are just really insane So he could hit this, you know 39 and a half number on two catches like that's just the type of of player that he is Khalil shekir has just been getting more involved without dos and knocks being around he has four receptions I believe in three straight games and you know his reception number again comes in at three and a half At plus money. I like shekir To hit that too plus 130 32 on that number as well. So yeah, I mean it's pretty much all action like We can talk about it jim. I just have I don't know the last time that I've put hard earned money On a buffalo bills running back to do anything Of the nature outside of maybe score a touchdown with like latavius murray But like I just the volume is just it's just not there Like maybe you could take some shots on james cook from his receiving number But I just I just never can get behind them like this team wants to pass They want to pass three out of four times when you give them the chance like passing is just as good as rushing To this offense and so for me it's getting all the props that I can on their pass catchers And I think you're taking the right props with them because of the archetype of players that those are So like you mentioned the concave reception prop You know five and a half perceptions is a lot and you mentioned his catch rate And a lot of times when a guy has a high catch rate you expect regression But concave does not get deep targets. They are all Near the line of scrimmage and those are high percentage throws. So You know, it's not six targets mean six receptions But it's kind of a long that vein where he's going to have a higher catch rate than usual As a result of where he gets His targets at so I think that's a good thing for him Shakira exact same thing where he's a slot guy Gets a lot of stuff towards the line of scrimmage. So I want the reception props in those guys versus the yardage props So Shakira over three and a half plus 132 can kate over five and a half plus 114 Talking about gape davis, uh, the baseline number you mentioned 39 and a half I think if I want to go to gape, that's where I go towards the alt numbers because Davis is volatile And that's fine. I don't mind volatility if I can benefit from it and with davis I think that's where I think you can, you know, looking at the alt numbers fadwell knows this like they're not stupid You're not getting a ton of extra leeway by going, you know, 60 plus yards is plus two tan That's not a huge number by any means when his baseline number 39 and a half So they understand the distribution of the way he gets his his targets the way that he gets his yardage It's more so in chunks versus in small gains So but I do think that even though fadwell is aware of this and they are accounting for that in the way they Mark them out. I still think I'd want to check out davis and the alternate markets to rack up some yardage Maybe that means a touchdown bet instead. I think he's plus two 20 or somewhere around there to score a touchdown but I think that the the upside markets are where you want to go with davis So I think that you're playing it the right way ryan in checking out the reception numbers for concade and for Shakir But then looking at the receiving yardage numbers and the upside markets for davis I think that's the way I'd want to play things here personally Yeah, definitely and if we're looking at gave davis longest reception of the game He comes in at six and a half to one there plus 650 Which I think is is pretty enticing to to Cash in on that Yeah, the fact that he's behind Marvin men's and jerry judy is weird It's very like marvin men's might not have a catcher. They've talked about getting his roll up like They might let's talk about the broncos side of things because ryan. They have been the worst this year They have been so frustrating.  You know, we always talk about the rundown Going into going into the shows and I was looking at it Actually this morning. I was late to it But uh, and you talked about the the broncos spread the ball around and I was like jim made a mistake here Like he's talking about the bills You know going back through the fats and it's like no actually The russell wilson and the broncos team are just you know getting so many of these of these guys involved And I just don't know why You know at some point, I guess it may be made a little bit of sense When jerry judy was thought to be on the trading block and and what have you and trying to you know switch up the teams here But you know, they they have two Two players that I think should really be touching the ball, you know every chance that they get it's courtland Sutton It's jerry judy And they need to be able to find these guys in space against buffalo who's been giving up, you know big plays As as recently, you know to the wide receivers courtland sunburn courtland Sutton's dumper is just wrong to me Like this dude has been an absolute catalyst for this offense. Absolutely insane like catching a touchdown I believe he has touched on in six out of eight games this year Really has been hit in a stride with russell wilson like as that security blanket Especially with the tight end situation that they have dulcich has been on ir they have adam trotman there You know another sean patin guy But courtland Sutton at 45 and a half just feels wrong I mean jerry judy, you know in the sense of like if buffalo is putting up, you know 28 30 points They have to pass. Um jerry judy at 52 and a half, you know makes a lot of sense as well, too I just you know, it's just the matter of the completion rate Like is russell wilson going to be able to find these guys or are they just going to have a lot of drives that Stole out but in even in that sense if like buffalo struck or a buffalo Denver struggling Um, I still like court and some Sutton's number Um anywhere that I can get it the one guy who I just I never know what to do with is javante Williams like And you know shout out to javante Williams coming out from coming back from the injury and and all that he's done to Kind of get himself back, uh to to the mold and it's been great.  Uh And watch them run I think I think jaleel's number at rushing and receiving what you have up here at 23 and a half Like if this game kind of gets out of hand and out of reach like do they give mcglockland more reps? They're coming out of the bi-week. I think we're going to see some different stuff from Denver I wonder what that is. Maybe it's using two backs. And so Getting some action on on jaleel here is something that I am kind of interested in Yeah, mcglockland as you mentioned his number is 23 and a half for rushing plus receiving over as minus 114 Javante will answer 74 and a half is minus 114 I have interest in that number because they did using quite a bit right before their bi-week And now it's had another week to rest up and get that knee healthy from where he was Coming off the entry. I just really worry about game script given the The way that my numbers show this game playing out So I think that's why mcglockland interests on your behalf is probably smart Because it might not be a great situation to get to javante Williams As far as the past catchers go Give a preference between Sutton and Judy Sutton's number is 45 and a half Judy 52 and a half preference for you between those two If you were forced to pick one I would take Sutton. Okay, cool. Yeah, I think that's that's a lower number He's been like he looked has looked like typical courtland Sutton like where he's just like Mossing dudes like left and right like that's just kind of what he does So I think that that does make a lot of sense. Let's talk some touchdown props Ryan Any touchdown props you are digging over at vandal sportsbook for this week Yeah, uh, I I think we're looking at uh, gabe davis anytime touchdown plus 220 I think he he kind of gets back into the bold This week is like you're saying he's boomer bust, but you know on the on the boom games that we get You those dudes tend to pay off. Um, so I find that very interesting I mean delton can cade to get the first touchdown of the game At 11 to 1 like what's not to like there the the bills actually have four of the top props in in this market, but uh, you know, if I was looking at the denver side I'd possibly be looking at courtland Sutton there at 16 to 1 He's just been the red zone red zone target red zone favorite But uh, but yeah, I mean outside of that, you know, we're taking the usual suspects Um across the board. I do like josh allen Alt touchdown market. Um, if we're looking at him to throw three touchdowns It's plus 240 him to throw four touchdowns is plus 870 like You know, uh, again, it's To get to the 47 and a half number like what does buffalo need to do they need to put up points How are they doing that? I likely think through the air You know, there is some merit there where you're thinking, okay Does josh allen rush one in or you know, do they get kind of you know They get wonky with who they've given the ball to but even the shovel pass that they use like five yards out From the end zone, you know, three touchdowns I feel like it is great a great number to be on at plus 240 And then if you're really feeling if you're really feeling it then go for that four touchdown there plus 870 Okay, I love that. So four plus touchdowns for josh four plus passing touchdowns for josh allen Plus 870 a fan dual sports book. That is a very fun number As well any other props you want to talk about for tonight ryan before we close up shop No, I think I think those are those are the main ones if you if you do If you are taking the buffalo side of things, I will say try and get some action on the first half Spread there four and a half For buffalo minus 105 try and get their first half, you know Points if you can where you can I just I just really feel like if buffalo is going to Get this done tonight if they really have to come out early and showcase Showcase what they can do All right that first half spread a buffalo minus four and a half is minus 105 at fan dual sports book for tonight That is all we have here for today But do not worry because ryan is back with us once again for tomorrow We'll talk about some futures coming out of the All the craziness that went down in week number 10 breaking down thoughts on that in the future We want to grab heading into week 11. So ryan is back with us for tomorrow But ryan for today. Thank you once again as always congratulations on your very nice weekend in dfs And hopefully we can finish up with another good night for tonight. Thank you. Yeah, let's finish week 10 strong Have a good one everybody That is ryan williams checking out on twitter at ryan alexander underscore w i am on twitter at jim sonnis j i m s a n n e s you can also find me on threads at jim dot sonnis and find fendall research on twitter At fendall research want to thank you all for tuning in for today Good luck to you with your bets across monday night football We'll talk to you once again tomorrow to dig in the futures market heading into week number 11 This has been covering the spread right here on the fan dual podcast network