Hi guys, my name is Amol Chaubey, I work for Ericsson and here is my colleague Sudeel. My name is Sudeel Ketamaka, we both work for Ericsson specifically for cloud management solution and Amol is the infrastructure architect and I lead one of the development teams. It's going to be Amol's show today, I'll be helping him out but if you have any easy questions, I can help with any tough questions, I'll ask Amol to help out. So we are going to keep it simple, feel free, free to interrupt me and ask me any questions. So the way we are dividing this particular session into a short presentation about the good practices on the logs and monitoring aspect of it and then there will be a lab introduction, in fact everyone, the prerequest for this lab is to have a virtual box installed and yesterday I believe the email was sent out for everyone to download the OVA but if you have not downloaded it, we have the USB's in the room so just ask for folks and start copying the files because it will take quite a while to copy all the files. So what we'll do is we'll start with the short presentation and then we will go into the lab introduction where we'll set up your DevStack, it will hardly take 5 minutes to set up the DevStack if virtual box is installed. So as you mentioned, we both work for Ericsson and we have been working on OpenStack since Qualcomm release and since then like every release, all the new features are coming out and we have been playing with it. So when we started with OpenStack, at that time, especially me and my team, we struggled a lot because at that time the installation was not that mature, now we have a lot of vendors around who had like literally one click installation that we have for OpenStack and all that. But installation is one aspect of it, once we install, it's done, then what we need to worry about is maintaining that OpenStack, troubleshooting that particular OpenStack. So this particular session I would consider is more for the beginner to advance that level and that category falls down. In a sense like you got to be little familiar with OpenStack components and basic architecture and the labs are pretty simple for the advanced users but if you are a beginner, the labs will be kind of challenging for you. So OpenStack, like in the architecture we have a lot of moving parts, so if you look at it, if you want to create a VM, while creating a VM we got to go to Keystone for authentication, once you get authentication you go to a glance to download the image, then it is pushed to no compute and the VM is created. So there are a lot of moving parts and because of that troubleshooting become quite a challenging aspect because each component has their own logs and we have to then dig down to those logs and try to figure out where could be the problem. So like at high level, I just put a basic building block there with the core component where you have a controller in the center and then you have like a NOAA, Neutron, Keystone, Glance, Swift and the Cinder components surrounding it but apart from that you have a basic set of hardware, on top of that you have an operating system and then the hypervisor. So when we are troubleshooting OpenStack, at that time we got to worry about all these individual components. So what we are going to do is we will try to look at the basic aspect of how to troubleshoot. I am not going to go to each individual component troubleshooting but at least a high level where we should be looking at and what are the typical pain points for everyone. So starting with the very basic stuff like logs, in the OpenStack the logs are like for each component there are different log directory and you have to dig through it. So on the left hand side I have listed out for a typical distribution of OpenStack, what are the directories, not necessary all the distribution will have same sort of log location but the lab that we are providing you is DevStack environment. So in that environment the log is located into the OpStack log directory. So that is one thing I wanted to point out because if you try to look at the directories which are listed on the left hand side you may not find it on the DevStack VM that we gave you. We have been using, for the commercial version we have been using Mirantis OpenStack in Erikson. So those are the typical path for Mirantis and perhaps it is the same one for Aldio. I am not that familiar with that one but I think the path is still the same. So now the basic component that we have got to worry about here is the monitoring aspect of the OpenStack. So what we should do like I am giving an example of FCAP model which is ISO model basically. We define the network management task. So like FCAP as it is listed out stands for fault configuration, accounting, performance and security. So not necessary like one tool will be able to monitor all of these things. You have to figure out what is the best tool that will monitor all these components. So and we need to worry about checking at multiple levels right. We got to worry about at the host operating system level, at the hardware level, at individual OpenStack component level and also at the VM level which you have provisioned through the OpenStack. Typically the VM level could be the responsibility of the application side for which you have created the VM but these all monitoring we have to do that in order to ensure our OpenStack is working sound and good. Now in the best practices right there in last couple of days there were lot of sessions on Elk right which is elastic search, lockstack and Kibana and I am not sure how many of you got a chance to attend those but those were the good sessions and in the market there are lot of commercial tools available like Splunk or Stackify which are commercial tool but if you do not have those tools right from the open source point of view Elk is the best model I can think of where which will collect all these different components log in one dashboard okay. So in nutshell what Elk is Elk is elastic search, each stand for elastic search for deep search and data analytics, lockstack is for the centralized logging and you pass those log as per your need and the Kibana is the GUI which is UI interface to it. So if you have not look into Elk model I would strongly suggest look at that and as I said there was one session this morning yesterday also there was one session on that one go through those videos and try to do it yourself it will help you in the long run. Then there is something called intelligent alerting like using a tool like Nagios or something like you can periodically check and get the alerts like if something goes wrong or something is not right Nagios will keep on sending you alert and that is the reactive state you will react to a problem. Then there could be something called proactive state where through trending you can try to find what is going on right. So what is trending? Trending can give you a great insight into how cloud is performing day to day right and you can learn like for example right on a busy day is it like if there are lot of activities on NOAA happening there is some slowness is it a kind of a rare occurrence or is it a time for you to react to it and start adding more compute nodes so that you can ease out that performance issue. So trending overall is like little different than alerting model right so while alerting is based on the binary result 0 or 1 whether it is working or it is not working but trending it kind of is record the current state of something at a certain point and that will help you to analyze how your open stack is performing. Then there is something called guru meditation report that gives you so NOAA has done a good job of like integrating with that one and if you issue there is a command like kill-user1 if you issue that command kill-usr on the PID of the NOAA API the guru meditation report gets generated and it gives you the current state of what is going on in the NOAA it gives you the thread count it gives you what configuration your NOAA is running and all that so we do have a lab on that one at that time we'll discuss in more detail about that. Any questions so far okay I'll move on so now in troubleshooting right where to start we have like operating system level tools which if you are from the sysadmin background you are quite familiar with it what are the operating system level tools or there are some open stack specific tools right so using the open stack CLI you can troubleshoot then there are some open stack specific tools for administration aspect of it so if you want to create a user or if the VM is if you want to set up reset the state of the VM the so NOAA CLI has a very specific tool for the administrator so in terms of the troubleshooting tools right the tools used to investigate or fix a problem within your stack right so that will help you to narrow down the issue where the issue is okay and also these tools need to span virtualization networking and the normal system at administration operation so that also you need to touch base all the domain in order to figure out or troubleshoot the open stack problem something that we did in our lab which is like we wrote our own canary script like which will every five minutes it will run it will create a VM create a volume and delete it every five minutes it's gonna run and if the run is not successful we'll get alert message the advantage of that is I'm creating a VM right that means my keystone glance NOAA neutral on all these components are working fine if VM creation is successful if I'm creating a volume that means my cylinder component is working fine so this gives me like at least a assurance that this is working fine if you have a third-party commercial products like in Agios and other then you don't need it but if you don't have those start with this at least it will help you to get alert in this aspect so canary script again it is specific to your lab we are more like in my particular group we are more focused on these these core projects but it's really you have to decide for yourself typical commands I believe most of you will know like how to check the statuses of like components of neutron or up or components of NOAA is up and running so for each component there are different set of script so try to look for the NOAA manager sorry hyphen manage whatever you are using cylinder manage or you neutron manage a neutron doesn't have managed they do it through the agent list but most of these core projects right they have like a hyphen manage extension and that's where you can like monitor the current status of it one thing I would like to bring up here like NTP is very important here okay so when you are across the different nodes which is running different components of open stack if your time is not in sync right open stack things that it is down should ensure that you have NTP server setup for all these operations now troubleshooting hypervisor so again it varies by hypervisor but most of the common hypervisor use is the live world or KVM hypervisor okay so each one has their own tooling so when you try to troubleshoot a hypervisor right we need to kind of map a virtual machine to a hypervisor using a NOAA so command and then investigate a hypervisor through the worst tool so like typical example is I'm not sure how much clear you can read on the screen but you create a VM right doesn't matter whether you create it through the heat stack or normal horizon or NOAA command but you create a virtual machine then what you can do is you can do a NOAA show on that one and there is something called OS extension server attribute for host and hypervisor it will tell you where is your hypervisor running on what host it is running so what you can do is you can assess to that host and try like a KVM has a worse something like a worse command line utility you can do a worse list and try to find out whether my VM is running or not there now here keep it in mind DevStack that I gave is all in one DevStack that means it has the operating system hypervisor and all the components are running on the steam virtual machine but in reality you will not have this scenario reality it will be pan across different host so now you figure out right you are on the like you can go on NOAA compute host and go to instances folder so the VM that you have created right it is it has a UUID so NOAA compute host will have a directory called NOAA instances now in our case in case of DevStack it is an off stack data NOAA instances but typically there is a separate file system for NOAA on the compute and that's where this folder will be what this folder tells you about it has some files like console log disk libworld XML these are very important files because like I'll give you example of like libworld XML libworld XML contains the information about the network that you have created right so neutron create a tab device for that one so it has the name of the tab device you can actually go in there and try to figure out what is the tab device as allocated by a neutron for that so in one of the lab we are going to look at that the VM that you create you're going to look at the what is the tab device mapped to it so this help you a lot in terms of troubleshooting when you are kind of clueless what's really going on apart from NOAA API and scheduler log also what you can do is you can use the chemo tool and get a info on a disk which kind of tell you the what is the file format what is the virtual size of the disk and all that information so disk comes in very handy while troubleshooting now troubleshooting network now this is the network problem is like for the beginner or even for the some advanced user it's freaked out people once it's a network issue people are clueless and try to give up like I'm not sure what's really happening so again with experience it will come at the more you do a hands-on on the open stack it will come but the majorly right there are three areas that we got a look in troubleshooting networking one is at the Linux level one is at the open we switch level and then there are some open stack tools for it so for example right the open stack tools will show you the logical configuration of your network so there are something called neutron port list command what it does it it lifts out all the interfaces and the port allocation and then it isolate to a particular user or the netlist it shows you a network created by the cloud operator or a provider network or a tenant then router router shows a tenant or provider any router that has been created again it's only a logical view from open stack but then if that doesn't help you right you can narrow it down to open we switch level where there are some command like OVS VSTL and DPCTL what it does is it will kind of map the internal and the external bridges okay so there you can figure it out like what is the tab device what are the bridges created for my particular VM so we do have a lab exercise number three where you gonna run these commands and try to map it to the virtual machine that you have created and then there are very specific Linux tools that is very common but one is like we need to look inside the VLAN or Linux name spaces because Neutron heavily uses this for if you are using a GRE type network or VLAN or VXLAN type network so this IP net NS is your friend it will help you to narrow down the issues there okay so we have a lab and at that time we can talk more about this particular aspect so in nutshell right this if you do IP net NS so at that time it will look at the name space okay and the DHCP get created so whenever you create a network right you need to know all the flows it goes through so now sometimes what happened is you create a network right and when you boot up the VM you see that there is no IP address assigned again you get clueless like why there is no IP address assigned on open stack horizon or command line it shows there is IP address assigned but in reality when you started troubleshooting you realize that oh my IP net NS so inside a DHCP I'm not seeing any tap device for my particular VM so then the most common problem is there is something called DNS mask queue which is not up and running and you restart it easiest way if you are in the test environment is just to drop that port and recreate it that will solve the problem but again you should be able to narrow it down where the problem is okay so we do have a lab which is going to cover that one now then again we go to the operating system monitoring right so again at operating system level you got a monitor CPU memory the availability of the OS itself the space disk space couple of times I have seen issue where the host OS then you have VMs running sometime the host OS run out of the disk space and because of that VM is not up and running because VM try to read or write some bytes to that disk space but since there is a hundred percent disk space utilization you cannot really write to it so always never ignore the host OS monitoring because sometimes the problem is at that level and you will not able to figure it out from the open stack logs then process level information so all the open stack processes like we talked about the core component but then there are cues rabbit mq is very common there is a database post grace or my scale or if you are using kilometer MongoDB is very common for that so you need to have the process level monitoring as well for these processes okay and of course all the open stack component related processes okay and also you should depend on the host to send out a notification to monitoring server what do I mean by that is when in the cloud world right you will have n number of compute host so you should prepare a model where host is notifying back if something goes wrong host is notifying back through SNMP that this is wrong rather than a monitoring server is constantly probing the whole this is the right practice and the last part which I wanted to touch base is the debugger this is little bit for the advanced user where you are trying out everything okay and issue could be at the open stack itself at the API level itself and at that time what you can do is open stack for all the component except horizon is written in Java in Python so you can use a Python debugger okay so this will help you at least to figure out where the issue is but now here the challenging part is you have to have some knowledge of Python language and how the API has been implemented so it is going into a little bit advanced user category but I will we have a lab actually that will show you a very simple use case and you can create like set up the Python trace and figure it out where the issue is so this will help you once you get advanced with all the troubleshooting tool kits and all that debugger component will also help you to narrow down the issue and if you can do that you can even contribute back to community if you figure out this is the bug just you can contribute it back default ports as a open stack admin or open stack user you should get familiar with all the default ports because on all the logs you see this default ports information and you should be aware of what component is running on what port you should be aware of that no matter what so I have listed this out what I am going to do is after this session is over I am going to upload it on my Google Drive for which there is a link I will show you right now so these are again the ports so in means I am just concluding the presentation aspect of it so it is a complex topic open stack troubleshooting okay but it is very easy to navigate and troubleshoot if you have proper tools and proper mechanism to monitor it okay you definitely need to have a proper understanding of open stack architecture that is going to help you a lot at least how the flows are how when you create a virtual machine or when the neutron goes and create a port what really have happened in the background at least have a basic knowledge of that aspect and that will really help you to troubleshoot now hands-on is the key there is no alternative to hands-on you have to try it out play with it and for that day stack is the best tool like install it on your laptop and start playing with it virtual box has a feature called snapshot so if you have clean image take a snapshot of it and then play with it if it goes something goes wrong reward back to a snapshot you can but hands-on is the key here so let's start with the lab now so I am assuming that everyone has copied the files that we have provided for the lab anybody who still need to copy we have usb here yes I have slide for that yeah any question so far a more question here yeah so the question is if the image has stopped working that he is not able to import the image well it should be because I did it like yeah it could be because we had tried it out yeah yeah let's let me go to that screen any other question will go to lab yes it is possible to assess it to the image so I'll show you that so what you need to do just to make sure that you are using this 192 168 64 network if you want to automatically start up the image ensure that you are using that network how can you do that so in the open stack right sorry in virtual box go to file preferences and network so I can show on my screen it is before actually did you know imported good before you started so import is good if you import it you are good but then what you do is go to this file then go to preferences then go to network and ensure that you have a NAT network setup okay and in here in the host only network at least I have three here but if you go to to ensure that you have 192 168 64.1 define that because what happened is the image that we have provided you has 192 168 64.21 as a static IP address so if your virtual box has this network defined then when you boot up the Dev Stack and you start up the Dev Stack at that time everything will be up and running so ensure that you are doing that let me check so 4.3.26 and higher so I tried with 26 and so maybe that could be the reason that virtual box so in the USB we have provided a binary for virtual box as well yes there is yeah okay but just to assure you the OVA is good that good so when you are able to finish that you can try it out yeah yeah just just set up like that and that will take care of it what kind of IP address dot 21 yeah yeah and the credentials are Dev Stack Dev Stack one two three so if you are not able to assess that means at the virtual box level something must be wrong so yeah so this is the IP address that will be so how many of you are able to at least set up the Dev Stack so far okay okay so I what so you know what what we can do is I'll just run over the flow one more time of this one so okay okay let's let's go over the flow okay so see I'm gonna be I'm gonna stop mine and I'll go to over the flow so sorry let me duplicate this I cannot see how to duplicate it okay so I'm gonna stop my I'm gonna power off my Dev Stack so let assume you are at a state where you install the virtual box right and you imported the image that's all you did so far okay so virtual box is up and running and you imported the image which is in this state okay now first is go to the file and preferences okay and go to a network so file preferences network ensure that you have a NAT network defined there if not just click on this add button that's all you need to do there okay then you will go on host only network you may not have anything if you installed this virtual box from scratch so what you will do is you will go and do a plus here okay it will say yes yeah so it will create a new network interface for you but what you need to do is go and edit there and ensure that that you are using 192 168 64 dot 1 rain there that is the first thing that you need to make sure you do that okay and then you click okay what will happen is virtual box will create that interface for you you okay you okay you okay you guys okay here okay you guys okay good you guys okay so what I'm trying to say I tried it on Mac to follow so exactly go to network right host only network so now hold on no no no this was already defined for you just to let you know because so just to let you know this is the page link to the Google Drive you can do a QR scan as well that's where all the OVA and the labs are posted as well as only thing is remaining that I need to post there is a this presentation which I'll upload it but you can download it from there one note for the people who are starting the first lab real quick is what I noticed that the curtain paste doesn't work when you try to do the first lab the heat template heat template is YAML and it is quite sensitive about the format so this PDF the curtain paste is not it is messing up with the formatting of YAML so I actually added a line there in the lab I just updated it an hour ago on like a W gate command so if your VM is connected to internet you can directly download this file rather than trying to curtain paste because you will get a error if you try to cut and paste unless you type it by hand but I would suggest do not do that on my page here I have also put the YAML file you can directly download it from there and FTP so whatever you are comfortable with W gate or FTP do it from there yeah you can actually easy part is go to that page and download the YAML or you can do the W gate still not good good you got lucky or no okay and and and just oh no okay got it got it so actually I should have tell because the start DevStack script right you have to run that so if you are not able to connect to a network run the start DevStack script what it is doing it is bringing up the network I forgot about that yeah no you don't need to run stack.sys at all stack.sys will reinstall the DevStack for you you don't do that so if it failed perhaps I would suggest re-import OVA because you are you probably might be in the different stage did it try to do one field yeah so okay do an LS here do an LS here and there is a VIS start go go go one directory up go one directory up start DevStack no no so now it might fail for you because it kind of in in the stack.sys might have tried to recreate everything I doubt it you might have to re-import OVA one more time but try it out dot slash start DevStack.sys so this is starting the services and that is one of the reason we are sticking with that 10 sorry 192.168.64 network because it is pre-configured you just start the DevStack and start using it stack.sys what it will do it will try to redo the installation of DevStack so that's a different so let's see what I would suggest it try to once it come up try to go to the horizon go to browser go to your browser type 192.168.64 21 21 that's it that's it let's see if it it's you might have to re-import let's see so just a note this is pre-configured DevStack don't run stack.sys or something just start up the DevStack there is a start DevStack script just use that to start it up if they go to that Google Drive they can download or you can open another session or you can get out of screen yeah controller 2D so that this network you define right you log into DevStack okay and ensure that you are running the start underscore DevStack.sh that is going to start your services I should have put that in the lab page now now all the services will come up yeah that's good now leave the screen open or you can do it is still running it's on the screen so go to the lab folder and download the MyStack.yaml file either delete the old stack or create the new one with some hyphen one or some different name so what happened is just a background NOAA right when you create a virtual machine it really doesn't care it allow you to create with the same name again I can create 10 virtual machines with the name something but heat they have a unique constraint for the stack name so that what the error is NOAA completely work using a UUID which is unique for each name they all their call goes to the UUID so you change the flavor to M1 tiny I'm assuming no no no no no no but the file the ML file what did you update the flavor to this will fail now this will fail this is supposed to fail good hi there is a solution towards the end of this lab but what I expected if you notice you created you try to create a stack with the flavor of M1 extra large extra large flavor need 16 gig of RAM your day stack is running on 4 gig of RAM so you are going to bomb out saying that host not found from hits that host not found means what happened is call is go goes to NOAA scheduler and NOAA scheduler try to find out appropriate compute host which has that amount of memory and since we do not have that amount of memory it tells that expected RAM and what is what you are trying to utilize is not there so the work around for this lab is just change the flavor to M1 tiny which is M1 tiny is of 512 MB RAM so just try to change it and redo the stack the people who are moved to lap 2 just to note in the network try to use a predefined network which is a part of the image to create it and it is going to throw out the error and try to troubleshoot it according to a locks or anything is always run if it failed run NOAA show and the name of the VM and exactly exactly exactly look at the so I know for sure but so look at RAM I know okay look for the RAM try to at least cover the high level what are these lab for and what you need to watch out for in the in those lab so lab number two right we created we have given you the predefined subnet and we asked you to create a virtual machine with that subnet if you notice that subnet has a CIDR of slash 30 that means it doesn't have IP address to assign to your VM it's gonna bomb out so alternative is create a new subnet and then recreate your VM you'll able to create a VM so on purpose I gave you a subnet which really does not have any IP addresses and it tried to assign that IP address to a virtual machine it's gonna bomb out which so the first thing that you can do always is do a NOAA show okay NOAA show and your VM name that itself if you look so so NOAA show VM will show you this error basically fail to allocate a network with the error no fixed IP available dadada okay but now which log file you will go at okay now since you are running a NOAA command right so the first request is gonna go to NOAA API because you are running a NOAA command so look at the NOAA and API log NOAA API log first so most of the distribution that we have right now they sync up everything at the controller node so if you have five compute host right most of the distribution available kind of sync all the locks to the controller and you can view it from there you don't have to navigate down to the compute host you have a centralized view so I would start though troubleshooting wise the way I will start is I'll go with the NOAA show try to figure out what's wrong mostly the common error is no host found but now in this case it is very specific about it I can certainly say that based on my prior experience is this is the change in Juno sorry this is the change in kilo you are running a kilo version of Dev Stack right now till Juno it was always no no host find no valid host find but this is much improvement in that one so you start with NOAA show right if you don't find any information then go to the NOAA API log and navigate through that one you don't bother with neutron log at this point because NOAA make a call and that away that's where it is telling you okay so that was the problem and then basically the solution is you will create a subnet with the class 24 range and that's going to solve your problem that is lab 2 so you'll create a two VMs okay so at the end of the lab what's going to happen is you have certain something like this view the two VM connected on the same subnet okay with unique IP address okay now the third lab is the networking lab this lab is it is nothing destructive in this lab the commands are there but this will help you to narrow down to find the UUID so you use a neutron netlist which give you a logical view then you basically narrow down like something calls IP net NS I'm doing a pseudo here keep it in mind it is only root can run this command and you figure out your DHCP so if you notice this is the UUID of my network and this is the DHCP UUID it is matching so again in your case it might be different as you create a different UU networks and all that thing but just to highlight the point okay then what you can do is you can go into a net NS and do a if config on that net NS to find your DHCP server okay now whatever VM you created right you should be able to ping to those VM using the net NS so this is just showing example so you should be able to ping both the VM okay now you should try this is kind of a tricky part where you will try to do SHH to that particular network and it's gonna fail it's gonna fail because the firewall rules are not there so you will basically go ahead and add a security rule for port 28 which is a port for SSH and then here I actually covered like how to find out the tab device associated with your VM that you created and this is a subset of the picture but if you look at the big picture here is the big picture like how the neutron network works and it basically you have a VM it create a tab device then a bridge and then that get hooked up to the physical layer of this stuff so these commands it just gonna help you to narrow down if you want to investigate what are my each layers are okay nothing destructive in this lab then the lab is the fourth lab is about guru meditation report which I was telling with tell you the current state of the NOAA environment so it is just like you just gonna run kill minus USR you're gonna find a PID I'm looking for 8774 that is NOAA API so I'm going to do a kill minus USR on that and in the NOAA API it is going to generate this GMR report which is the current state of the NOAA so then in the exercise you're gonna do a trending exercise where you're just going to go and grip for space 200 in the NOAA API log what it is going to do is it is going to give you account that how many requests was fulfilled on that in that given log file this is helpful because if you are generating a log file on the daily basis or something you can at least see the count of the successful request and you can keep a track of it like if the pattern is you are creating 50 VMs a day or 30 VMs a day you know and suddenly you see today you created 200 VM that can help you to like proactively figure out what's really going on do I need a more compute host or not same thing for info how many calls NOAA is getting you can get a count on that one then NOAA users list that is the command in NOAA which said for each tenant how much servers are there how many ramps are there like how many RAM is used for how long RAM hours it is in RAM per MB in hours that's how it's called but it helps you to figure out who is my tenant and how much resources that particular tenant is using you can alternative go to your MySQL database of NOAA and do a quota uses so if your quota is defined how much you are using or utilizing out of that when you are managing multiple tenants this is very helpful for you to figure out what is going on there okay so this again it is a very informative lab in that sense and the debugger lab debugger lab it little bit tricky but the what we are doing essentially is here is we are trying to add a new key pair in the stuff and we are giving a name like using these characters I know that this doesn't support these characters okay so what we are doing is I'm showing an example like take a backup of your API.py which is a Python file and then you're gonna add go to the line number 365 it set this trace up and restart the NOAA API okay once you restart the NOAA API and if you rerun the same command again your screen which is a DevStack screen it's gonna stop at one point and you're gonna go to that screen and type this safe car which basically read the code and tell me that these are the only allow values for this particular component so and then you get out and then you restart the API so this is just to troubleshoot it but get there are so if you Google right you will get tons of stuff so again in OpenStack there is something called Tempest files okay and there is a project going on right now where they have implemented a different set of debugger okay this is very basic but if you look into there are some white papers which I'll put it in my link when I upload so those white paper will help you if you are interested in that aspect of it there are tons of tools available in OpenStack for debugging this is the most simplest one for the NOAA's but there are very good and powerful tools available I think only one minute is remaining any questions and I will also send my email address my own Sudeer email address if you have any suggestion for future lab please do so yeah not just that I mean if you for a few days will will support any questions that you have I'm sure at leisure if you try this thing in a much more relaxed environment you should be able to figure out all these labs they're not really yes yeah virtual box virtual box okay sure sure we'll do that okay yes so that's the one I was telling you about that try to use this elk okay and if you cannot do elk there are tools available like Splunk or Stackify that those are commercial tools again it's expensive but at least if you have a good Java developer in your company you can take his help and it's just a matter of in a day or two you can set this elk thing yeah you can set up it on the controller node because that's where you have direct access you can even set up it remotely but then you have to make a call to the controller node either way yes exactly yeah yeah so what it does is elastic search it has your own database actually it pull all the lock parse it and keep it you know no no it is it is very good if the person you tell and you can narrow down exactly where is the issue and you can even map the no longer neutron lock side by side you don't it is pretty cool it is very powerful so there were a couple of sessions yesterday and today about that so I would strongly suggest what the video of those no and there was one hands-on lab I couldn't attend it but if you can get a hold of that material that will be good he'll tell you I think we have to get out alright thank you guys thank you for coming


Everybody forced here ripping 2022 tops from f1 sapphire hobby a light four box makes your number two random team style There are The results check them out anytime Elaine sports cards calm. There's a link and chat for you. Let's get started y'all To light one hobby one sapphire. Let's do it. So to light up first. Yeah, I mean he got to come into league at 18 That's just a one-year advantage to what players have now But it's not that's not why he broke the scoring record broke scoring record because he's never not been great in the NBA He's always literally been the best player on his team He wasn't like a lot of other guys like Vince Carter or something like that or you know Even Kobe had those injuries like LeBron has just stayed healthy and just kept it going kept it going man Future stars black and white Ray wave Philippe a drug of itch for MP Yeah, it does change soon too. That's the brawny rule refractor Grand Prix winner Brazil Lewis Hamilton For Mercedes, he's like the Tom Brady of a basketball take care of himself. Just keep going keep going keep going keep going and That sense anyway, I'm not trying to compare them as like championships or whatever, you know, that's what it is George Russell refractor Mercedes there we go on the George purple Come on Nova lock for MP to 399. I don't know man I'm 30 my back hurts every day I Can I imagine LeBron? He's six eight or two. Yeah, six eight got the vets old black and white black and white Ray wave driver of the day for Aston Martin Gonna play game against 20 year olds 25 carries Shockwave your whole body Refractor formula formula one freshest Joe Guanyu rookie Afro Mayo You know, you know I'm gonna feel like that after breaking the simple mixer on on Saturday Ray wave lanche stroll for Aston Martin black and white Yes, Sam and Kenzie are like And the three 310 a four-ish so they just want to swap off doing the top one for each other. Yeah pink refractor Fernando Alonso number 28 of 75 Rich and Ortiz Dream team for real Fractor I'll you move Yeah, I'll you move a wasa I Said his name before I just Thought I said it wrong for Dom's That is gonna be fun It's gonna be exhausting. It's gonna be fun purple green max for stopping For Oracle for our red Red Bull Oracle Red Bull Max for stopping Extractor Roy Nassani for Dom's, but yeah, how long until Cameron Thomas breaks LeBron's record? like a year This is that what Walter said earlier one year Future stars black and white Ray wave tail per shower for ART Probably before he's 30 I would think Realistically pure ghastly refractor for Alpha Tari a mini diamond Dennis Holger Or Prima 199 to 299 All right, another light box coming up How we look it on top scrum Seven to go still nice guys seven to go pass halfway full on top scrum champions League Coming up soon Black and white Ray wave Logan Sargent Karlin future stars and do appreciate starting with two sold-out breaks though That's awesome guys, especially at a 10 a.m. Start Pink Ray wave Oscar Piazzari F2 award winners for Prima 67 to 75 nice Oscar. It's a Dave Chappelle Oscar.  Trash can Obviously, there's an effort in there. So yeah, it's Hamilton refractor from Mercedes There's there's a lot of there's actually a lot of bad words in that and that uh, I forget what I forget which sketch I was watching have you rolling though? I saw it was on tiktok. I'll scroll through rookie checkered flag Joe Guanyu for Alfa Romeo It's one of the ones I forgot about Here's Sean's boy Mike crack. Yeah black and white Ray wave Aston Martin. That's my dude right there Mike crack Yeah, I forget what it was. I don't want to like talk about it a whole lot, but yeah, what was the Is is pimping easy, you know that whole one or he's like I don't answer the questions that is correct Yeah, dude that one's so good Sebastian Vettel Aston Martin refractor not a whole lot to talk about there though Or the Racial drafter whatever Yeah, that's classic. I don't want to talk about it. I don't want to come off one You know wrong, but it's funny. Haas purple 248 to 399 for make Schumacher there. So good Gold purple Sebastian Vettel Aston Martin Nice Vettel.  So don't you worry guys tonight's gonna be smooth Got Sebastian Vettel black-and-white ray wave for Aston Martin refractor rookie Joe Guanyou alpha Romeo Let's see. Let's see nice SP authentic down to 10 tops chrome down to six Six to go on that tops chrome now SP authentic the first one's down to 10 That's awesome, man. Thank you everybody for grabbing some spots today Hoops is down to 10 as well. I feel like it for some cheaper basketball Or to grab some newer rookies Speed demons Alexander album. So, you know how the NBA does like the rising stars game for the um, like first and second-year players At all-star weekend the first two picks were Palo Ben caro and Franz Wagner Were they really yep future stars black-and-white ray wave Liam Lawson for Carlin. They know what's up. They know Cats out of the bag I'm working on some Hallie and I are working on some good content to put together It's gonna be good. It's gonna be good. Can't tell nobody nothing right now though. Ricky refractor Joe Guanyou for alpha Romeo Six of two ninety nine mini diamond feature stars Marcus Armstrong high tech You'll be in on it. You'll be gonna be good. I'm telling you refractor Max Verstappen Grand Prix winner Belgium Red Bull and Orlando speed demons You know, Orlando speed demons was like Orlando speed demons, which is apparently everybody that drives in Orlando Hobby box here one hobby box. I'm a wizard. Dude. I can't wait to get it's still do the Fed I did it ship from Texas my game ship from Texas yesterday. What'd you get two days ago Hogwarts legacy? I Never pre-ordered from GameStop online you guys just don't do it Just get the get the digital Or just get it in person Because I got the deluxe which released yesterday. I won't get it until today at the earliest Stupid I don't have I don't have much time to play it anyway. So it doesn't matter that much like yesterday I was busy all day and Today I'm working. I got planes at night. So it's whatever. But still I'll look Claire on the refractor driver of the day Great Britain For Ferrari mini diamond Marino sato virtuosi 187 of 299 Yeah, it's not even FedEx. It shipped Monday from Texas It is FedEx delivering it but like and you can talk whatever about FedEx, but like it's not even them It ships a Monday It should have been probably shipped like over the weekend or something to get here by release day That's what normally happens.  Refractor Carlos signs for Ferrari Gold wave. Oh nice constructor coalition for Williams. Let's see if he an albone. I don't think I've ever seen a numbered parallel of constructor coalition 27 of 50 That's pretty sweet looks good. Oh nice an auto again Ralph Bosch on compost. That is That is some signature Ralph Bosch on compost. Oh boy. Oh No Ralph That's not even like initials Not even that. Oh boy. Oh Ralph Nice auto there for for that spot, but Ralph come on. Oh Boy, I look up on my TV. I see Carrie ever got the microphone in this hand. I got to see what he's saying For better but normally for worse Grand Prix winner refractor Abu Dhabi Max Verstappen Red Bull He's a big what if player and like hey if you just shut up and play basketball Like truly like the sky would be the absolute really good. Look Claire at checker flag extractor for Ferrari there Ralph Alexander album refactor Williams and Ralph isn't a name you see a lot So I feel like you it could be a pretty unique autograph if you made it Right like a unique signature Ralph E B Even like Joe Burrow like just signs Joey B, but it looks good. So it's like dude. That's cool We're not saying have the nicest script and like all this but you know that and Make it so like you see it. You're like that's a unit. You know who it is You see Brady's auto, you know who it is right away Not the best signature, but you know You know guy I work here refractor formula one fratches Joe Guan you Yeah, I'm not saying that Adam 104 of 299 mini diamond Max Verstappen for Red Bull I'm definitely not saying shut up and dribble Especially in the context that I was first used.  That was his name and then He signed like that. It's cool He got lucky though Jordan love the Jay Hart signature gold purple gold Fernando Alonso for Alpine Grand Prix winner of the day guitar. We'll see in all LeBron's family there, too at the game refractor feature stars JQ's refractor Ben Emmer sports and look at that Both breaks had a our due Grand Prix Barcelona specifically to That is wild so in the previous break. It was the lightbox that this was in it's the hobby box this time So just got some case hits out of f1 for both breaks there seems the same exact card to that's very strange Sometimes it happens like that Again, that's random to everybody in just a minute here at the end of the break. No points. All right sapphire time Max Verstappen Green rookie Joe Guan you 45 is 75 very very nice Vettel and Lewis Hamilton Sebastian Vettel Sergio Perez Frederick Vesti and Daniel Ricardo Dude wreck it Ralph would be cool Devon. You're right. That's an auto. Although. I guess that's what you don't want to do when you're a driver Joe rookie again feature stars Liam Lawson For Carlin on these chartreuse two of 199 Kevin Magnuson Lando Norris Logan Sargent Charles Leclerre Yuki Sinoda Max Verstappen Max Verstappen Landau Norris we've got an aqua now Carlos signs Ferrari 24 of 99 Lando and Marcus Armstrong Orlando speed Daniel Ricardo Come on novel lock Lewis Hamilton and yos Capito Max Verstappen and a gold Gold sapphire Lando Norris McLaren There you go Kyle 28 to 50 on the gold sapphire Lando Alonso Okon it was Hamilton Ayuma Iwasu Sergio Perez and Yuki Sinoda Good stuff. Let's do this random and then we got a little recap as well. Art de Grand Prix Barcelona Case hit random to everybody here Very nice to get one of those in a four box mixer two and three five times Remember some top gets it one two three four and five red bull Red bull gives you wings Here's our recap. So we had a purple mixed shoe mokker mini diamond to Max Verstappen Marino Sato Marcus Armstrong and Dennis Holger Purple Clemant novel lock chartreuse Liam Lawson aqua sapphire Carlos signs pink Fernando Alonso pink Raywave Oscar Piastri Green rookie sapphire Joe Guan Yu gold wave constructors coalition Latifi an album Gold sapphire Lando Norris and Otto Ralph Boscham That will do it on the soccer now for a bit. Thank you everybody


So then I started to think about this card, which I was spending $4,000 a month on and I started putting everything on to this card So somebody said it's so funny because your red card you stop and your green card now everything is go So now I'm paying $4,000 a month and bills that I have to pay anyway And I'm getting 8,000 Z bucks because it's double Z bucks So when I started thinking everybody in your life swipes a debit or credit card You don't have to train people how to swipe a credit card But also they're swiping that credit card 40 50 60 times a month


So thank you for coming here.  So today, we talk about how the Bank does observability with multi-tenants using pro-miches and tenors. First, let me introduce myself. My name is William Costa. I work at Ital Bank as a SRE. It's my head facing in Star Wars in Rise of Resistance. I'm a huge fan about those Star Wars. It's an amazing place for a huge fan. You can follow me just scan this QR code and add me in my social in the community. And Rodrigo Serra is a speaker too, but he can't to be here for a good reason, OK? The Rodrigo is a father now. So it's a beautiful moment for him. Thank you, David. So what is our mission inside of the bank? Today, we help the teams along their observability journey. And we assist all the squads that support the platform of metrics, logs, and traces inside of the bank. So let's see our agendas for these sessions, OK? I will give a brief introduction about the Ital Bank, OK? So we see how we work inside of the bank about my team. The challenge of metrics is inside of the bank and how we took to build it, this big platform. Let's see the components and architecture about the tunnels and culvernets. We have big numbers, some big numbers about the some components, OK? And who monitors the monitors, how we made the monitor of this big platform. And let's see some panels about inside of Grafana that we have and more users use in the bank. And next steps along this year. So the Ital Bank is a huge bank in Brazil. We have 98 years of history, OK? We are a full-service bank in Brazil. We deliver many kinds of financial services, sorry. And for example, insurance, investment, credit card, and check accounts, and others, many services. We are present in 18 countries, OK? The main countries is Brazil, Chile, Argentina, Paraguay, and Uruguay. So who you are inside of the bank? Data will have some strategy by community, OK? And my community is really ability community. And really ability community have some strategy about the reliability and one of them is observability. So we support both system and environment of the bank, the truth observability. Metrics, blogs, and traces are inside of our community, OK? We deliver journeys through in their search and we love that all users practice this support. And we are open source first. We support the wall users and the wall teams make best solution using open source projects. So let's talk about the challenge of metrics inside of the bank. Today we have two main environments inside of the bank. We have the cloud environments and on-premise environments in our data centers. So build a big platform of the metrics that offer some points, it's very important. So today we have about 5,000 accounts and WS accounts among development, allocation, production environments. And we have a lot of workloads running inside of each account, OK? We have 13,000 hosts and heat applications in our on-premise environments. And it's very important to offer one single panel for visualization in these both environments, OK? So we need to make alerts inside of this platform. Then we offer to the both environments. We work at 24-7 operations, the bank never stops, I think. And we need the scalable and resilient platform. The name inside of the bank, the name of this platform is Cloud Metrics. So how we touch to build it, how we think when they made this big platform. Availability and scalability and resilience for us is very important here, because monitor both systems and environments of the bank. So the keyword for us was isolation. So for us, the most important was to isolate the max components possible, including your infrastructure and application. So each account has their tenant, and each tenant needed to be isolated, OK? So it's very important, because here, we try to reduce the noise between tenants, OK? So we're talking more about the next slide. So let's see what is the confidence we choose for running in Cloud Metrics. For one, today, we have the Kubernetes for running. We have three main clusters, each cluster by environment, OK? For collect, we have some journeys available for wall users that can deploy in your account. For us, the main core, the main application here inside of the Cloud Metrics. For visualization, we have the Grafana. Wall users love Grafana for visualization. And for Alert, we have the Alert Manager for integration with some channels inside of the bank. OK, let's see how we made the infrastructure of the Kubernetes of this platform. Today, we use the AWS EKS, OK? We hosted in São Paulo, in the region of São Paulo, Brazil. Bond nodes are deployed in three AZs, OK? And we have the main conception here, where each node group have running on one specific workload, OK? So for example, we have one load group for network, monitoring. Here, we talk about the tunnels. So we have one load groups for specific for tunnels received, tunnels query, and other many workers that run inside of their clusters. Let's talk about the application. Tunnels is the main application here, OK? So here, we separated in three main layers about the tunnels components, OK? So the first layer here, we have Tunnel Squared Layer, OK? And the Tunnel Squared Layer can get one metric inside of your specific tenant, OK? The second layer, we have the Tunnel Receive Soft Layer. The Tunnel Receive Soft Layer receive one metric that arrive inside of the cluster, OK? And the middle layer, we have each tenant with some components. Inside of each tenant, we have one-tone store, one-tone receive hard, and one-tone rule, OK? So here, we have two main drafts about how we implemented the Kubernetes infrastructure and the Tunnels application. So let's see how our users can see your metrics. I think that here, it's common for all of us. We have the Grafana as the main tool for visualization, OK? Grafana send wall queries to the Tunnel Squared Layer. And Tunnel Squared Layer get wall metrics inside of your specific tenant, OK? For a store, remember that we have accounts by tenant, OK? So each tenant have your account. So inside of each account, we have one bucket, we have three buckets, OK? And the tenant access this bucket by a cross-account policy and can store and read wall metrics that have inside of this bucket. For scrappy wall metrics inside of each account, we have the Promeaches. Today, we have some strutters about how our users can deploy Promeaches. We have Helm charts for Kubernetes. We have the CloudFormation and Terraform for ECS and others, ways that we have inside of the bank. So when Promeaches scrap your metrics inside of your account, send by remote write. The remote write is a proxies that you have inside of the Promeaches. And when wall metrics arrive in Tunnels, receive salt layer. The Tunnels receive salt layer, identify your tenant, and direct your metrics in your specific tenant, OK? One point here, an important point. Sorry. Here we have an important point about how works Tunnels receive salt here. The Tunnels receive salt layer works like a proxie, OK? So the Tunnels receive salt layer met your tenant ID and always forward in your specific tenant's receive hard. So here we have the simple draft about how we made alerts. Remember that I told you that each tenant have some components, OK? And Tunnels really start wall rules and record rules that you have in your tenant. When you have one file about the one rule, the Tunnels really trigger alert manager and alert manager trigger your channel. Here we have, inside of the bank, we have some channels available, OK? These channels are ServiceNow, Microsoft Chains, and MAU. So who monitors the monitor? How you made the monitoring this big platform? Today we have the same structure and concept and separated environment for monitoring wall components of the cloud metrics. Here we have one example about the numbers and graphs about the health environments of the cloud metrics. Let's see some big numbers about some components here, OK? So about the components today, we have three clusters. And for each cluster, we have 200 nodes. It's a big cluster. We receive 6,000 requests per second in more English, OK? 7,000 containers running inside each cluster. About the graphana, we have 5,000 dashboards, OK? And 4,000 active users. About the Tunnels, we have 3,000 Tunnels tenants. We generate 1.2 terabytes storage by day and 1.5 billion samples per day. So here we have one example about the dashboard inside of graphana, OK? And this dashboard is we have traffic lights about the resources in AWS. And when you click in one specific traffic light, you can see graphs and numbers about the resource. So it's very important for us because the bank have a big strategy that move wall workloads to the cloud. We support that wall users make business monitoring with using metrics for following your nearer time, OK? And here we have one example about this practice. About the SLISLO, we support the wall teams make SLISLO. It's very important for us. And here we have one example about this solution. For next steps along this year, we need to deliver a single pane of glass with graphana. At the moment, we work for deliver this. And we would like to implement Keda and Carpenter in wall clusters, wall clusters that we have, and deliver a single journey for alert inside of graphana, OK? So I think that's it, guys. Thank you for coming. Thank you for KobyCon staff and any questions? I'll stay here for a sec.


What's going on guys got Kyle here relating sports cards, and we are ripping 2023 legacy football six bucks number 13 random team style And show you guys results here real quick. We did a random before the break got dials P with the bucks down to Andrew s with the Jets Alrighty good luck here guys. Let's do it legacy football You said that base was Regular regular base cool a regular base. Yeah, so like the legends The paper on the back. It's yeah, okay like the vets will be like the legend cards and then Or like the legends will be like the base and then the rookies will be like that crowning yellow Valley the base for the rookies Got Tony Pollard Vets got a silver of a cam Smith rookie. So the legends go with the Regular vets and then the rookies separate So those paper though those are considered rulers they'll go here than the crumble though Sharpen a camp Smith will be sorted. Cool. All right, you guys good luck on this one. Let's see a dare to tear Let's see some fire. Let's keep the fire going from the last one man. My goodness got a orange to 199 Echler Got a prize prospects rookie Quinton Johnston rookie Bryce young and a rookie Andre Carter, dude, I don't know anyone's seen the I Don't know who's on chargers Twitter or whatever, but Quinton Johnston looks good He was snagging balls like left and right. Let's go. Nice auto here. We got under the lights for the Vikings J. Jetta Good start there Vikings on the Justin Jefferson under lights auto. Thanks. That's receiver in the league right there not numbered Got a DeMarcus where for the ages Um Thurman Thomas In jigba and Michael Meyer. This one is decade of dominance Joe Thysman Brian Branch We've got a green this one is decade of dominance Ronnie lot to 100 Niners Prize prospects Jaylen Carter for the birds There's a mini silver it's Brock Purdy under the lights Mini Brock thought that was like a player from the 20s. It's Gronk That black and white got me Like oh must be Gronk's grandpa. There's a nice one under the lights CJ Strout on the chrome Nice one there Texans Zollez rookie prize prospects of 249. We got a Darnell Washington rookie There's my homes on the for the ages Will Anderson Zach Evans This one is the 149 Zach Evans on the auto for the Rams Under the lights, it does actually say what the team they're on under the lights will have the Go rams with the Zach Evans auto not to 149. Can I get some more small shorts? Yeah Got my homes in Montana generations Hutchinson The orange Dalton Kincaid bills rookie at a Utah to 149 Yellow almond raw to 150 Lions Got prize prospects bootay for the Patriots Mini Trying to sneak away. We got a Brock Purdy mini again. This one's the 75 bronze doubled up on the mini purties Got a silver rookie will levis go into the Titans in the base Alrighty guys box two. Good luck here guys Let's see some fire Trevor Lawrence under the lights chrome Rishi rice working to 100 Tony Pollard Cowboys prize prospects Rishi rice Chiefs rookie Charbonnet to 25 we got for the Raiders Trey Tucker on the nice one there. I got a little uh Logo man Little NFL shield that is to 25 for the Raiders futures RPA Nice one there for the Raiders Bryce young There's orange rookie Rakeem Jarrett to 149 Jackson Smith I'm near left in the select There's Zach Evans to 299 second of dominance young blood Still four to go guys on select team all random division case Coming up here pretty soon Or to go halfway Got some Star Wars with spots still and then we're finishing the night with that big uh antique dual case mega dual case There is lions Hutchinson number 210 nice one there 10 of 10 for the Lions On the Hutch second year Many under the lights Dan Jones silver under the lights Jalen Carter for the birds Yeah, I think so sir. I think it'll be pretty soon Um lane was definitely doing some buying at the national probably with that in mind. So I think it'll be pretty soon Uh, it's a 199 war and moon legends. They literally just got back So I don't know when he'll start working on them, but I would imagine uh, that is in the works pretty soon here Next couple weeks or so maybe nice. This one is in the lions gear. It's futures auto of hendon hooker Nice one there in his lions gear number to 249. I set their lions QB out of tennessee Got a elie So I guess that's it's the only one with the Yeah, these are the these are the only ones pro you need a sweep. Um, what's this? It would be a name The heck is that what? That is a rookie.  I suppose shawntucker We don't know exactly what parallel it is because the checklist hasn't been out Interesting this one's going to the bucks shawntucker on the rookie sparkle That's a sweet-looking car, man. I hadn't seen one of these This one bucks got the watts Um, roll dice for this one guys. Um, one two three top. We'll go to texans. Uh, four five six bottom We'll go to the stealers table. Just usually like whatever he rolls here. I'll just do it for the whole break Unless it's like a numbered. Yeah, we'll do it for the whole break for a top or bottom We'll do it for the whole break top will be a one two three bottom four five six Two we'll go to the top. We'll go to the top on these dual, um cards JJ watt for that one In base Red to 2.99 got damien pierce texans Got anthony richardson prize prospects in base Checklist dropped today Oh, thank you, joe Thank you, joe. All right. That's good to know that is very good to know. I appreciate that Got a mini silver. This one is uh, remandre stevens and patriots Got will levis on the prize prospects. Oh, righty guys four boxes left. Good luck Some legacy brian robinson under the lights Or bijan robinson What'd you call brian robinson different person? Still running back Yellow rogers in his jet skier there to 150 that's weird rogers yellow prize prospects murphy So, yeah, the black and whites are just literally black and whites. Oh Makes sense. Yeah This is auto to 35 nice one here's ze flowers for the ravens Nice sit on the auto to 35. There we go ravens out of boston college Nice hit there For the ages chuck woodson Decade of dominance bailey base Yeah, I think he's gonna be good chris for I think he's gonna be good There's some good receivers in this class for sure jaylon hurts to 199 jackson smith looks really good Darnell washington prize prospects the uh seahawks receiver corps in general is uh quite good There's a mini this one is john michi orange mini for the texans to 125 orange mini got colch dick cut Duh bears Duh bears Volunteers hendon hooker there lions on the hendon some base Thank you tom get an order in a personal order John t and chris all grabbing personals guys. Um, I will not have time to do those tonight guys if you did just order Um to one two ninety nine eckler. We'll have to get those done either tomorrow or wednesday guys. We're a little bit behind Um, i'm scheduled pretty late today, so I won't have a lot of time for personal tonight If you guys did order Yeah, i'm excited to see michi play man. I'm excited To 149 we got josh downs under the lights auto for the colts Josh downs to 149. There's a top gonna go to the top there. Um lt. That's the same team actually chargers I'll say orange rookies a flowers go into the ravens again to 149 Richardson base There's orange prize prospects jalen carter at 249 Prize prospects jalen hyatt base Um should be monday or wednesday morning, uh nick should be wednesday morning with forest Tomorrow night will be our last night for the promo and then uh wednesday morning. He should run it There's a rookie miles murphy mini and nice one. Here's silver under the lights brice young ego panthers Alrighty guys three more This is birds on the auto. It's gonna be tanner mckay under the lights auto eagles And auto is shawntucker double auto there shawntucker Just look that one up. It's bucks Bucks on the auto do two autos in that pack Got dan jones under the lights Jackson smith Yeah, I know jg. It's uh forest and uh forest just got back so that'll help for sure man. We'll get another breaker We'll get caught up here for sure Uh for the ages mahomi Yeah, uh shadey you should watch that um donriss optic choice break. We just did at least watch the recap Um to 150 we've got a roger stallback if you want to see some uh some crazy stuff There's a chief's dual card Silver is pickens for the stealers uh butte prize prospects mean joe green This one is jesson jefferson on the main attractions Nice, that's a sixth short print there for the uh vikings. Got a jj to auto now. We got a short print You know, it's weird last time. I think me and sam or ripped this We we got the same the under lights auto jj and then that jj too. Really? Yeah, interesting. That's weird There's marvin mam's number two 149 broncos very interesting cj stroud Blue is to 100 you got eric dickerson Yeah, it was pretty it was a pretty good one. It was a pretty good one, man The state farm guy. Yeah, I forgot about that Mahomes with the state farm commercials Hey, there's darnel washington gonna be under the lights rookie for the stealers. He's a bath bomb guy Steelers bath bomb guy State farm commercial You never seen that one. I'm already like you flood this house No, he's only the trading room. He's like just throwing like bath bombs in the tub I don't remember that one. I remember the guy trying to change his jersey because he had like the lucky jersey I don't know if that was even state farm There's a damn arena dolphins. I don't remember the that one doesn't that one's not really popular You need to check it out It's not it's nothing exciting Just Mahomes is saying he's a bath bomb guy There's miles murphy that is to 199 rookie at a Clemson Nolan smith rookie See a bird Murphy Nolan smith. Oh, yeah Another one of the philadelphia bulldogs Yep, there's a bronze many josh alan that one's numbered to 75 bills bronze many Decade of excellence Rookie base under the lights silver is aj brown for the birds Red to 299 got von miller bills Uh blast from the past is jerry rice Yeah, no, typically not, um tawn Righty guys two more legacy Good luck. Good luck. Well, herbert banger. I'm on it. I'm on it Good luck mike Hey, I appreciate that Appreciate that blue is calvin ridley number to 50. He's gonna have a great year for the jags Calvin ridley Ed reed let me try to grab him in fantasy for sure Jackson smith The sherbert sounds pretty good right now honestly Wouldn't be against that to uh 100. We got rishi rice. I believe he's a chief's rookie Uh rice. Yeah. Yeah Yep chiefs And charbonnet rookie for the seahawks. That's gonna be bronze as well number to 100 decade of dominance john riggins Base Zach's a patriot, right? Oh, he's a seahawk seahawk. I'm thinking of the boob guy. Yeah to 25. We got prize prospects, uh, jordan adison booty booty We'll go, uh Decade of dominance there. I thought I'd tell you from this guy's last name like the bonnet and those for some reason I thought he was solid. I don't know I don't know It is silent Silver rookie, uh, jackson smith. It's charbonnet Got a will anderson rookie prize prospects. That one's going to houston. Donny just woke up from yesterday. Yep He slept it all under the lights travis kelsey base This one is rishi rice at 299 prize prospects rikin garret or jerry Not enough carl.  Is that the qb from, uh, ucla? No shady. I'm a friend erin rogers I think it was dorian thompson. Maybe I think so It was toke. I got you. Yeah, I watched a few of those games last year. UCLA had some good games You see, uh, the usc game was good It was actually insane. Decade of dominance base Shady's thinking of shawn Shaw didn't compare me to shawn. I don't like it shady To 199 we got a rookie, uh, dauntavius wicks Got brian breece on the rookie base There's red. This one is tractor cedo to 50 under the lights mini titans joey porter prize prospects base There's a silver will anderson. Nice one there texans one more box guys. Good luck legacy box number six Appreciate you john and tom grabbing some, uh, personals here Again guys not gonna have a lot of time for personals tonight. We're scheduled pretty late Um, we'll have sam with some more personals on uh tomorrow night after group breaks and then wednesday I'll be back at five or seven and we'll go um all night on those We'll go all night on personals on wednesday josh downs rookie base This is two 100, um archie manning for the vikings never seen that I always think of saints for archie manning Yeah, same can't get over it either. Really? I don't like it. I don't like it. He's a warm moon on the sea hawk. He's like, uh, oh Well, I don't want to see that You're gonna see it though Here's a nice one another auto under the lights hendon hooker. I'll be the second hendon there for the alliance Hey, what's up metro? Um after this I got some star wars. Um, we got some select team all Some good stuff tonight, man That could have dominance read Got a mega dual case it in the night. Um still got some work to do on that one prize prospects tanner mckay mojo brice young andre quarter There's barry sanders to 299 rookie For the ages anthymunios sapphire break. Oh, uh made from yesterday Um, we didn't have one today. Um, at least not tonight There's a sake one under the lights Decade of dominance crouse You can always check our wet uh on the channel here though I got you metro. Yeah, you can check on the channel here if you want just go back on the videos a little bit You can get to it under the lights bosa All right, they have those in um What's the alan and ginter they have the little mini autos? They were framed though, I think so they were like a full-sized carpet the mini was inside like a frame It's kind of weird. There's a yellow 150 joe burrow Richardson base It would be cool though. I like the I like the idea There's clayton tune on the uh rookie auto base Clayton tune tune I believe is cardinals cardinals There's a I forget what the viking's going back is. Um, there's jonathan mingo rookie 250 jonathan mingo base two packs left orange to 199 cow pits God de jon robinson prize prospects. That is that is the cubie It is kirk Someone was talking about trades for kirk trade lance or something earlier. It'd be crazy Yeah, I did like kirk on that show. He was really good. Nice one there to 125 We got under the lights mini for houston will anderson silver is butte patriots And base on the back all righty guys that'll do it for the legacy six boxer do a quick auto recap We had clayton tune on the rookie auto for the cardinals Hinden hooker lions. We had dorian thompson for the browns Rishi rice chief's auto Darrell washington for the um seahawks. We had shon tucker or sorry darno washington for the um Steelers shon tucker bucks. We have tanner mckay eagles auto Colts josh down We have ze flowers the 35 ravens rookie auto. We had hinden hooker lions on the rpa. Nice one there We had tray tucker for the raiders the 35 double rpa Zach evans auto And justin jefferson Alrighty there guys that'll do it for legacy six bucks We will get um Get some donner's optic retail going next


Missing data is very common in empirical research. In fact, almost every project that I've done has had some missing data. Whether missing data is a problem depends on a few things, but fortunately there are also tools that we can apply to deal with missing data. Another related concept is sample selection. In this video I'll give a quick overview of what missing data is and what sample selection is. Let's start with an example. This is a dataset that I like to use when I teach rigorous analysis. We have a bit more than 100 observations of data and the units that we observe are occupations from the census of Canada in the 1970s. Our task is to explain income with the other variables. So we have education, share of women and a prestige score for the occupation and we want to explain income. The data are complete, so we have every value for every variable for every case. There are no holes here, no missing data. So this is easy, but what if, for example, we don't observe the prestige score for the government administrators? What do we do about it and what's the consequence? If we use this data to run a rigorous analysis, then by default our statistical software regardless of which software we apply simply will drop government administrators from the data and run the rigorous model using the rest of the cases. Is that a big problem? We will still have a bit more than 100 observations, so maybe dropping one out of a bit more than 100 is okay. Okay, so what if that's not the only missing value? What if we have missingness also here? So we have general managers and share of women is missing. Two observations out of a bit more than 100 probably not a big deal. We can just run a rigorous analysis. Okay, what if there is more missingness? If we have a missing value for one variable in every case, if we run this rigorous analysis with this data, then the default regression command will just report that there is no data to run because by default the action is drop all the cases that have any missing data. So are these data useless for estimating the effects of education, share of women and prestige on income? The answer is no. There is lots of information. We still have 75% of the available information because we have four variables and only one fourth of the values are missing. It just happens to be that that missingness is spread out through all the observations. We can estimate the effects of education, women and prestige on income, but to do so we need to apply modern missing data techniques. Simply eliminating observations will not cut. This is a very common scenario. So quite often researchers face with a limited number of missing cases or missing values. For example, in Nyli Rengös article that I used as an example measurement, they report that they had 180 final cases after dropping 15 observations. The amount of data that they drop is about 8%. Is it okay to throw 8% of your data away without analyzing what is the impact of doing so? Probably not always. Sometimes throwing 8% of data away might be okay. In other cases it might not be okay. Depending on why the data are missing and what is the missing data pattern and what's the mechanism for creating the missing data. This is something that the paper doesn't address. So it's quite common that as long as the data are missing, the amount of data that is missing is less than 10% and people would just drop the cases and be done with it without explaining. If I read a paper, if a paper drops more than 5% of observations, I'll ask the authors to justify the assumptions that are required for unbiased estimation if you drop the observations. In practice I'll ask about the missing data mechanism. I'll talk about the mechanisms in a different video. So what is missing data and what is selection? Missing data refers to a typical scenario where we have some cases and some variables that don't have data for some of the cases. There are a couple of techniques that people commonly apply that are bad. First, deleting cases, unless the missing is random and there is not a lot of missing, it's not a good idea. Even worse, people use something called mean substitution. So if you have a variable that has a missing value for a case, then you take the average of all other cases and you use that average in place of the missing value. This is a common strategy and it is always a bad idea. Dropping cases is a lot better idea than inventing data based on mean substitutes. There are better techniques for dealing with missing data. You can use maximum likelihood estimation that takes care of the missing data problem by feeding the model to the data that you have and ignoring the missingness. This is called full information maximum likelihood because it takes also the information from those cases that are partially observed. FIML is the acronym. Then we have multiple implementation which basically it comes up with guesses for the missing data, missing values in the same way that mean substitution does but it is just a lot more refined approach for doing so and it has been proven to work well. So there are ways of guessing the data that does not cause bias. In fact, multiple implementation corrects for certain kinds of bias caused by missing data. Then we have a related issue of sample selection. Sample selection can mean one of two things. It can mean either that some cases are observed and others are not depending on the phenomena that you study. A classic example is Hekman study of wage offers for women and whether a woman gets high wage offer or not of course determines whether they go to work or not. So if the job market will give you a low wage offer then as a woman you choose to stay home. This is a study from the 1970s. So the things might be a bit different now. The problem here is that because you don't actually observe the wages that the women were offered rather you observe the wages that they got then there is systematic selection because the cases that you observe depend on the value of the dependent variable and this is a problem. You can deal with this problem using a selection model. Another problem is sample selection or selection to the dependent variable. A classic example is a medical trial. If you allow people to select themselves in the treatment or condition only those people who are sick go to the treatment and those people who are not as sick go to the control group then if you compare after receiving the medication the treatment of the control the difference between the treatment and the control group is not only because of the medications effects but it's because the sick people decided to go to the treatment the people who are not as sick decided to go to the control. This is a difference in levels but there is also another kind of sample selection which is a more tricky thing to deal with and it is if the effect depends or varies between people. For example in management one of the key principles or key things that we believe in is that managers make informed decisions. So if we study for example which companies invest in factories and what is the effect of investing in a factory only those companies who think that they will benefit the most from the factory investment will invest in the factory. So the effect is not the same and managers make decisions based on the expected effect. If the managers are any good they can actually estimate what is the expected effect. And this kind of selection on the expected effect causes some issues that we need to deal with. The techniques that we apply for that case if we have these different effects we actually apply the same kind of selection model that we apply to the missing data problem. So in this way selection model or missing data and sample selection are related because the same tools and principles apply although this might seem as unrelated issues. I'll explain this concept in more detail in other videos but for now just need to understand the missing data. Some observations are missing that we would like to have. Sample selection can mean a systematic pattern of missingness. Either full case is missing or the dependent variable is missing or one of the values of the explanatory variables is chosen based on the expected gains from that variable for that case. This is a large and developing literature on missing data analysis. Unfortunately for applied researcher these literatures are technical but once you understand some of the basic principles then it's not that complex to apply these techniques.


what's going on guys it's your boy dpj here today with another destiny video and in today's destiny video I'm gonna run you guys through my night for lute results for the past week but before we get into the video tell me down below in that comment section what you got from the night for this past week and if you do enjoy this video if you could leave a like I would appreciate the support we're getting straight into the video starting with my titan I've got the peregrine greaves which are exotic leg armors muzzanese bad boys are came all ammo for heavy weapons and shoulder charge deals bonus damage when activated in mid-air now these were one of the three remain the exotics I needed before they're brought them a few weeks back as you can see S.O.F. kill switch got an aphorite light and easy now got the vanguard scout rifle the one slash one sinus feet or sinus feet mugs on this were hidden hand this weapon gains better target acquisition and full auto this weapon can be fired in full auto mode actually not a bad scout rifle whatsoever moving on to my warlock and I got the pest control matrix which is the vanguard auto rifle mugs on this where this weapon has increased stability and this weapons magazine will be refilled whenever a super is cast as you can see easy now again got the one slash one vanguard scout mugs on this one though were causing damage with this weapon increases its stability and improved accuracy while airborne aquatic arts got the plan C exotic fusion rifle moving on to my hunter which actually soloed and recorded live for you guys and if you missed that video and do you want to watch it I will link it in the video description I got a heavy legendary engram when I decrypted this bad boy I got the calling which is a future walkout machine gun mugs on this were kills this weapon increase reload speed for a short time and this weapon gains better target acquisition so I still need a few exotics I need a last word because I dismantled my because I lost it in a challenge for PVP and they also need the bones of EAL their hunter leg armors and then I'm all done then I'm sorry then I've got everything but guys let me know down below in that comment section what you got from nightfall this past week did you get anything you needed or wanted let me know thanks for stopping by as always hope you enjoyed the video and I'll catch you on the next one please out until next time peace


And I'm grateful for the privilege to play and do what I love. I'm grateful for my teammates. I see you guys snuck in there. My coaches, our front office, Clara and Josai, Wasserman, the fans, Topuma, Cliff, Darabadi to my family, my grandparents, my dad, obviously wifey, and to the people who voted for me. And I'm also grateful for the people who didn't or who are mad at the people who did because the buzz around this race is too good for our league. Robbery is good. And the business doesn't go on without the fans. So thank you. To all the players, we make each other great by showing up to compete every night and by standing together through adversity. We flip the Senate. We help bring BG home. We can do things beyond many's imagination. And I'm really proud of the work that got me here now. And now, being a two-time MVP of this league and the first to bring it, bring the award home to the liberty of franchise whose legacy is built by players whose shoulders I stand on. This one is for all of us. And we are not done. Thank you for everybody being here.


Yanoski off the inside game, look at Lindgren, he else is going. And it's been easy set for another Grand Prix win here. Yanoski second, Manson third. And the Guterres in the back, but look at Lindgren, go Cal. Sly, the semi-final. Rodin is right once again, but Lindgren's out in front. Got just over a lap to go, can he hang on? Well, Manson's really back, and just the right spot. It's got a nice, it's got a... He can hardly walk, he can hardly lens to avoid Patrick Dudeck. Oh, winning the Scandinavian Grand Prix. Hats off to first, Freddie Lindgren. Hancock, gate two, will he do it? Here we go, we'll try again, he's the off one. Jason Doyle has got there in the first turn. Pavlicki's going to try to cut up the inside. Holders still in the mix and wheel to wheel down the back straight. It's Jason Doyle who has the lead here. Hancock is going to try the outside run. Doyle's got to be careful because Hancock has got plenty of speed here. Going into lap two and I'm not sure the inside run is the right way to go. It's working at a moment, but there's no doubt that Hancock is giving it an absolutely everything right round the outside but the track has gone fairly slick. Doyle is just about hanging on there, but look at Hancock. Hancock on the outside, he's neck and neck, Doyle just about hanging on. Doyle moves out, Hancock now switches back to the inside of him. Yes, Hancock's there, he's going to win. What a move, he's going to have a 12 point lead in the title race and he's done it the hard way. The 46 year old from the USA, Greg Herby Hancock. A remarkable man. He is a remarkable man and it's a fabulous victory. You know, it's a fairytale. It's a super story that he finally wins in Sweden. He's adopted home with his Swedish family. The final here, Han falls on his way through. Holders super. Here goes Pedersen up the inside. Oh, he's all kicking off there. Zagar furious with Nicky Pedersen. Pedersen not impressed by Zagar's response. It's all kicking off, huge drama here in Malala. Well, Zagar made his point. Down in the pits area now, I'm quite sure that one or two words are going to be said. Intention, adding motions down in the pits area. A little discussion here now and it's still very, very tense. Krista Gardell trying to get involved there to separate them. Zagar quite clearly fuming here down in the pits area. Nicky bit of a wind-up merchant, egging him on almost. Big smile on Nicky's face. Well, Zagar's incensed and quite clearly what Pedersen was saying to him just fired him up even more. I find him. It's, um... He just said find him. Yeah, he says find him and could well be in for a find with blashing out. Wait for the star to go and Yanovsky's made a pg off gate number three but Zoas lit back off the inside. Now Smarshlik coming through in yellow and it's a one-two for the Swedes in a moment but Smarshlik's coming on strong. Can you believe this from the outside gate? Antonio Lindback has the lead and now Lindgren coming through again. They're all bunched up. Which way is this going to go? Smarshlik now roll. Smarshlik comes on strong. Lindback is still in front, Lindgren's there. Smarshlik is riding out of his skin in second place but not for much longer. Lindgren now through in the second place. Smarshlik neck and neck down the back straight. Lindback in front. Oh my goodness, this is going to get tight. Yes, going into the last lap we are. Is he going to be a Swedish winner or will Smarshlik pick the pocket of Antonio Lindback? Here it goes. There we go with one Lindback staying on the inside. Right gracious me. Smarshlik stuns and silences the Swedes with a stunning move. Big better and better. Well, well, well. Let's hope the boys are going to be okay here because that was a big one. Freddie Lindgren trying desperately to get out of the way. Let's have a look at this. Is that too hard? Is that too hard for Mikkel Mikkelsen? I'll tell you what, Freddie Lindgren. Oh, frick. Nasty crash. I'll tell you what, that is a minor miracle that they've all got up. Mikkel Mikkelsen's hard on Dodeg there. He is very hard on him. Wow, bikes and bodies everywhere. What a spectacular crash. Max Frick is still being attended to. I think he's holding his wrist, Max Frick. This is a big decision for the referee. It is now, yeah. Mikkel Mikkelsen moves over there. Just there. And basically leaves him with nowhere to go. I'm not quite sure which way the ref's going to go there. I don't like that move there. It just gives him a nudge there when he's alongside him. He knows he's there. He must have had an idea he's there. Freddie Lindgren picks the front wheel. Well, Dodeg is excluded, we're told. I can't see an exclusion light, but that's just information that is being fed through to me right now. But I'll tell you what, that could have been such an even worse outcome. We hope that Max Frick's going to be okay. He's still sat down getting medical attention. Holding his wrist. Well, it was a dreadful, dreadful crash. Now, let's hear from the most spectacular stuntman in Sweden. Evil can evil. No, it's not, it's Freddie Lindgren. And he is with Kiri. Well, amazing reactions for you. How did you react so quickly? How did it play out for you? Yeah, so Dodeg came down, and it was in a really awkward position. We don't have any brakes on our bikes. It was just instinct really. I tried to avoid him as much as I could. I tried to lift my bike up and hit him as little as possible. What a hero. That's exactly what he achieved. It was a split second decision he had to make. And great to see, great to get the reaction, of course, because it was quite an incredible moment for him.