Okay, so my name is Tito from University of Oslo. I'm based in Nairobi and today we want to talk about things that people do after the installation. When you have a fresh install or do a VHS2 instance, maybe you are using tools or you're just following a guide doing your install. There are things that you would do to make sure that your resources are optimized to make sure that whatever you have on your post are being utilized by the apps that you have, the application components which includes Postgres and your web application. So I prepared a few slides just to walk us through what we have today and then after that we shall go to use one of the problems that I just created for demonstration purposes and a few problems that you might face and you've completed your installation but it's nothing is working. You're not able to access whatever you've deployed from your browser. So these are the slides that I prepared and it's going to check us through the tips and tricks, troubleshooting tips and tricks and post install things that you need to do. So I'm going to start the slideshow. Are you able to see my screen? You can see the screen, it's okay. Okay, okay, okay. Thank you. So what's the problem? What's the reason why we are having this conversation today? And it's because normal installations, maybe we're using tools we have at our disposal. Maybe you're just following the installation guide and you're building those components one by one. And by default, the resources that you have are not or the apps that you deploy are not optimized to use. The defaults are not like stretching to the resources that you have on your system. And once you've completed your installation, you need to tweak configuration files that allows your app components to use whatever application or other server resources that you have. And then another section is you've done your installation but nothing works.  You're getting errors instead. So what are the tips and guidelines, troubleshooting guidelines that you need to deploy? Yes. So what are the categories into two that you need to do to tweak your app and optimize your apps and then troubleshooting now tips. Okay. So we have components that when you get your installation with the tools or if you're doing manually, you will not miss to have a database and Proxy and Tomcat. Proxy is going to be really your choice. You might choose to go with InginX or Apache 2. And then Tomcat is normally our standard install comes with Tomcat 9 instance. And then the database, which is of course Postgres Equal 13, as of this recording. So in our standard installation, we want to do an installation where it has all these components packaged. Of course, there's another component here that is not included on this slide, which is monitoring because then you want to again be able to get metrics about how your database is performing your Proxy and Tomcat. So that one is also included here. So when you get a server, which is normally an Ubuntu 1820 or 2204, you have a host resources that is CPU and memory. So you want to budget your total memory so that you give Postgres fair amount of your memory and then web application. And of course, you don't allocate applications all the memory and you get your host with nothing. So you budget your memory footprint so that the apps have fair amount and also the host. So we have Postgres and the web applications. Those are the main applications that we will talk about today and then the host. So normally, when you have say about an estimate of around 64 GB of memory, you want to and say you have two instances of Tomcat. So that means you have how many applications here to budget your memory. You have two web applications and one Postgres. So by convention, it's good to give your database even a larger share and then the remaining share you divide between your two web applications. So if you have, for instance, 64 GB of RAM, then you want to ensure that your Postgres has about 32 GB and then your other two web application is say HB, HB, H or 1010 and then you want to leave some amount of memory to your host also. A normal operation of your operating system needs some memory. So you want to leave some amount. So next slide is really talking about Postgres SQL database, you know, that you have now given your Postgres some storage. Do you have a question? Okay. So you have allocated your Postgres SQL database some storage. So you need also to tweak Postgres SQL configuration. And these are the settings that you need to set generally because today I'm not talking about, you know, Postgres SQL configuration extensively because that needs a whole session on its own. It has a lot of things that you can play around with. But then I want to talk about four main components that are even mentioned in our install scripts guideline. And one of them is the shared buffers.  And then if you have, say, 32 GB, then that is going to be about, I think, 88 GB storage. And then work memory is normally calculated total amount of work memory that you want to give your configure in your Postgres SQL. It's going to be a factor of connections that you have opened on your Postgres SQL configurations. So if you want to calculate total amount of work memory that it's going to be at the end of the day is by multiplying work memory value plus maximum number of connections. And if 42 GB you can do 10 MB and then you can multiply that by factor of maximum number of connections that your system will be supporting. And then there is maintenance work memory.  And then there is effective cashier, which is now going to be utilized for caching, Postgres SQL caching purposes. Any question up to that point? I don't think that it should be a question. You are basically the example that you've given already, you said 64 and you're giving 32. So basically the calculation that we're doing here for the Postgres, is it going to be based on the 32, not on the 64 GB? So this 64 GB is assuming that it's really for the Postgres. But then before that there's the budget first, which is going to, with a budget you will know which amount of memory will you allocate to Postgres. It's not necessarily, you know, this is really not the system storage, sorry, not system memory, but only what you've budgeted for Postgres. So if you're going to budget 64 GB for Postgres in your case, then that means your system has a lot more memory, even 128 or so. Okay? Yeah. Yeah, so, so um, yeah, I had a comment, I nearly forgot. Yeah, in terms of Postgres in particular, you know, with Tomcat you can kind of restrict how much memory it's going to use, because mostly it uses its heap, and you're probably going to show us this later, so you can set the maximum heap size. With Postgres, Postgres is greedy, right? And all of these settings that you tell it, is not going to prevent it from using as much memory as it can find. That's the way that it works. And particularly that effective cache sizing, it will just try to use however much memory it can see. So the really important thing with the Postgres, if you're using containers, and I know you haven't spoken about containers, but if you're using containers for your database, and it doesn't matter if it's LXD or it's Docker, you have to constrain that container. So you have to, when you create the container, you have to configure it to say, this container can only use 32 GB RAM. Then after that, you can do all of these settings. One of the things we've seen happen, if you don't constrain the container, Postgres can use, it'll use as much as it can, and sometimes then you try to start Tomcat. Tomcat will fail to start, because Postgres has already chewed all the memory. I was actually going to demonstrate that. And here we have, which has to deploy it within LXD containers. If we issue LXD list, we're going to see that we have Postgres here. And the Postgres here is, I guess, seeing all the available memory. If we check the memory that we have here, we have 64 GB. So let's execute into the container, Postgres container, and check the memory that we are able to see. So the memory that Postgres here is exposed is about 64 GB. And that means, as Bob mentioned, that it will be created and try to use all this memory that is available on its disposal. It will try to use all this available memory, which is much going to even use all the available system resources. So we want to limit this Postgres container, so that it only sees memory that is budgeted to it. So that is done by you issuing LXD limit command, which is LXD, config, and then container config set. That container name, which is in our case Postgres, and then memory, limit.memory. And then the amount of memory that you want to give that container. In this case, it's 32 GB. So that is going to limit your container, Postgres container in this case, so that it only sees this amount of memory. When we execute back into the container and check the available memory, we see that it only sees 32 GB right now. So even though it wants to be very greedy and it wants to use all the available memory, then it's only able to see up to 32 GB. That is what Bob was talking about. Is that clear? Yeah, that's clear. So next, what we want to talk about is also Tomcat. So when you've completed your installation, yeah, you want also to make sure that your Tomcat is able to use total amount of memory that is allocated to it. And that configuration parameter is in this file. Yeah, sorry, is in this file default Tomcat 9. So let's just quickly get to our install here and list containers. And here we have around six containers. The first two containers are posting our Tomcat instances. Let's just get the first one. Let's see, exec, the showers, and then bash. And then we just view that file, edit that file and see its content, which is its default Tomcat 9. So this is the file. And the file has configuration, java configuration parameters. And I guess line number what, sorry? Line number five is the line that you want to uncomment so that you can tweak your memory that your application is going to be using. So get to that line and then out of the available memory, the budgets that you have set for your Tomcat instance, then you can change here. You can change these parameters to the amount of memory that you want to set, say, HGB. And then after that, you want to reload your restart your Tomcat instance with system ctl, system ctl, comma. I don't know if restart, reload will apply configuration, but you can restart the Tomcat instance. So that will apply your changes so that if you say, check that process, you will see your configuration changes. That is HGB here. And so your running instance is using that amount of memory that you wanted to see. So the other parameters here that I can talk about, like the last one here, about line number 26, this is for application monitoring. It's a chloride plug-in that when you want to enable your chloride monitoring, you will uncomment this line. Of course, you will need to have your chloride package extracted into this directly so that it will now be monitoring your instance. Of course, there are the configuration parameters here that I guess we are not going to talk about all of them today. These are called on its own. And then there's another file, which is opt DHS2, which is instance configuration file. This is just an environment configuration file for the Tomcat. But then there are configuration files that are specific to the DHS2 instance. And they are all on this file by default. So we want to see how that file looks like. It's seen opt DHS2, DHS.com file. So this is the file. And on the top, we have these lines that are uncommented. This is the standard install. This is the default that comes when you do your installation. However, there are things that you want to change later on to suit your environment. Like for instance here, we have connection pool maximum size. Sometimes you might want to, in a very, very busy system, you want to increase this number. And this number needs to be really in compliance with the number, with the maximum connection allowed on the post-recipement database. Yeah. And even much more configuration options that you can tweak and enable other features that you want to see your installation to be supporting. Yeah. So the standard install will come with these four uncommented or enable settings. Of course, there's database password. This is just a demo instance and it's not production. And these are going to be on this. Hello, Tigo. Yes. Yeah. I think this is very good. Yeah. I have one question. It's Lamin from Gambia. Will it be possible like some of these things, like this comment, it may be like those things which are needed, like to put them in your presentation so that if anybody do the installation, you know that these are the things you really need to comment. And these are the things you need to do as part of your presentation. Yeah. So this comes with the standard install of the H2 with the tools. And normally, these configurations that you see and commented here are sufficient enough. You don't need to really touch anything else here. And this, if you're using automated install, you don't touch this file. It's going to happen on your behalf. Otherwise, really what you need to change is the first defaults that I showed you that you need to tweak these to suit your cyber memory availability. Otherwise, the other details to configuration file, you don't need to really touch that unless you have a special need for your installation. Yeah. It might be worth, we have a two hour session planned in the upcoming server academy just on the dhs.com file, all of the options which are in there. So I mean, we try to make sure that one gets recorded, I think. But yeah, as Tito says, it's those first couple of lines which are the essentials that you need mostly around your database connection. Yeah. So those are the tunings that you need to tweak after the installation is completed, at least on the side of Tomcat. And then we go to proxy. Sometimes you have your application deployed on slash dot application name directory. And even general last time when he was testing the tools wanted to really access the app from the root without appending the application name. So that is something that is come most of the times and the tools do set up the install, but it doesn't do redirects by default. So he wants to come back later and gets to the proxy configuration, which can be in the next or a party to the line is there, but not at least not enabled. It's commented out. So you need to get to them. The proxy configuration gets to the line that you need to uncomment. And I will demonstrate that on this call, at least for for the next proxy that we have running right now. So the site that we have is BHIRs.com and it's returning empty response. If you just get to the root direct, you don't append application name. It's appending. Like you don't put a font slash application name. It's empty response. The reason why it's returning empty response. Let me just get to them. The root cause of that is that let's get to the proxy. This is the main configuration file. And the reason we are getting empty response is this, that anything that is not matching, it's not matching the application name that you want to access is it was going to return a 444. So the 444 that is there is this really, this is its empty response 444. However, you could instead of returning an empty response, I could maybe change the tool so that we get the default engine next, the default engine next site static site that normally you get with the installation of engine next or static site that you normally get with the Apache to install. But then the line that I wanted to talk about is this rewrite, rewrite line. So you want to recommend this. That means whenever you access the root domain, it will be rewritten and redirected to HMI. Of course, you need to have, this needs to match with the app that you have in your system. Yeah, so pseudo or let's say system CTL reload the next. It's 15. So that means whenever you access this, it's going to be redirected to default application like you've seen right now. Yeah, so that is another thing that you can tweak on the proxy after the installation. Yeah, okay. Yeah, I have a question. So that was the what file name, which you have recently uncommented? It's the main file for engine next. It's in the next directory, you see? No, yeah, I can see it. But in the configuration file, you have uncommented rewrite line there and slash HMIS. So HMIS is our bar file name, right? It's the application name. We will list them. It's the Tomcat application name. Let's get out of this container and you see we have HMIS and we have DHS. So that means every request that comes to this server are redirected to HMIS. However, if you want to access our DHS, that means you need to append on your browser DHS, you see? For you to be able to Okay, fine, fine, fine, fine. Got it. Yeah, something like that. Understood? Yeah. Okay, so that's one. And then number two is monitoring tools, one of which is Muneen. Muneen is we use to monitor our instances, which can be servers, depending on the install that approach that you took or containers or the host, you know? And normally default install is not is leaving Muneen exposed. Empty response, Muneen spelling errors. There you see. So that means you're not supplying username and passwords for you to get to this Muneen. So what we've done for that, we are using basic authentication on the on the on the engine next level, or rather the proxy level. And on this call, I'm going to demonstrate how you're going to enable at least basic authentication for them for them for the Muneen. And these are the steps. Of course, you're going to need a patch to your tools, and then generate the password, and then edit Muneen location configuration, and then enable that basic authentication. We're going to run through that quickly. And on the same server, we have proxy, that's why you're going to do your, you're going to enable your basic authentication. So you need to execute into that proxy, and let's see except proxy, then bash. And then you need to install this package, which is Apache to your tools. App install, Apache to your tools. Yeah, they're already installed. I have installed. And then you want to generate the password with this, this line here. Just find the command industry. Yeah, with this. So this is going to be a password, and then where you're going to start to store your password, and then which user are you going to generate, going to generate password for. You can choose username that you want. And for this call, we're going to go for admin, and then it's going to ask you for the password for that user. I'm going to just put admin. So that's, it's generated the password for user admin and the password is the same. So and then after that, you want to edit location configuration for muning, because it's muning that it's actually exposed to the it's without password. So normally with the standard installed, we have all in the next configuration file within, within a conflict directory. And here we have main configuration file and then upstream configuration files. Let's get into the upstream and see what we have there. We have the HH2 configuration. And let's just see what we have here. Here we have the HH2 for two different instances that we have here. And then the file that is of our interest here is muning. And at the very end here, you add the two lines. You add the two lines, which is all basic. And then you want to get, to add some lines here, say basic.  User and that file is in our case, et cetera. HH2 password. HH2, HH2 password. Yeah. And then you want to add those and then check your engineers configuration if they are valid. And then you load pseudo service or system. And now let's try accessing our muning on the new InvoVnito window. Now it's going to request for username and password. Those are the ones that we just configured. And if we supply admin, default admin, then it's going to take us through the muning. But at least now, you will not get to access this site without credentials. Please, we're monitoring endpoint without credentials. Question? I have a question. Yes. What is the effort of having password to muning? Is it a security trade or what? Like, what is the main reason of having password to muning? Like you can see right now here, your resources, resources that you have on your infrastructure, that is Postgres, Apache, everything is just exposed to the internet. Whoever has this link can log in and they can, they can like, they can see what you have already. They can get much, much more information about how your system is set up. And it's not good for security. You need to hide your stuff. You just don't leave them to the public so that they can in a snapshot know what you have, which one can be their next, they can, you know, they have a lot of information that they are not required to have, I think. So you need to at least, and you don't want to open this to the public. This is private to your infrastructure. You don't want everybody to be able to access these endpoints. It needs to be really secured. Okay, this is good. What for the night? It means that we have installed muning in our systems, which means that we need to do this, but it will be also important maybe like to have a demo, like someone who has muning, how you can just enter into his system without no password. Maybe like next Tuesday or after that we can try to look into that. Okay, okay. That would be interesting. Yeah. And also most of our installation is being done using Apache. I see you're using Internet. I know basically the procedure will not be the same. Probably before sending the slide, you might just need to add the same for Apache and then you send the slide so that we can protect it. I never knew that the muning exposes the information, but I don't know that someone can take this information to hot the system. So we need to actually protect our systems now. Yeah. Yeah. So when you've actually mentioned about Apache 2, last time when I demonstrated the tools, I had not developed support for Apache 2, but right now with the latest push, you can pull latest in source scripts and it supports Apache 2. So you would need to really change one configuration directive from engine X and put it to change it to Apache 2, like you had done general before, but it was not working. Right now it's working. Yeah. So yeah, next is really backup. So backup plan is not a cost-installed thing. You need to really plan for your backup even prior to starting your installation. You need to know a few things like which backup policy are you going to, retention policy are you going to use, which upside backup are you going to push your dumps to. And also for instance, yeah, those things, the details about backup and scripts and all those kind of stuff, it's something that you need to really plan even before you start your installation. Something that you just do as a post is maybe testing your backups. You've had your system up and running and you want to test and see if your backup script is doing its work. And number two, can you kind of restore your backup in a fresh environment and is it working? Something like that. That's something that you do after the installation, just to make sure that your backups are working and test that your restore are working. Otherwise, planning for the backup is something that you do before you start installation. And all the bar scripts that we have before the DHS2 tools have backup scripts, which I'm still on process to put into the Ansible scripts that we have. Question? I have one question. Okay, Gerard, you can come first. Okay, so my interests have been always with this because though we are saying we're using open source, but we have been at a greater disadvantage when it comes to cloud hosting. And so backup is very necessary. So one of the things that I am peculiar about is actually the backup at off-site. But then there was this project where I always have in mind, where once you backup the off-site, how do we like replicate the system? It just automatically replicates itself on the system. Maybe there is a script that does drop the database and deploy the new backup that you've already done, which is the backup testing. And then we start the local instance, and then it works. So that was one thing that I've already, I always take to the system administration training. But it's something that I think all of us should collaborate in order for us to have a one script that can do all of this process, do the backup from online for the cloud hosting solution to a local instance. And then on the local instance, you can drop the database and rewrite the new or upload the new backup. And then we start that instance. Because sometimes we lose data because we don't have resources to make those payments and all the rest of it. So it had been a challenge actually when it comes to this part of Africa. Maybe I can comment on that. Broadly, I would say that backup is also something that is, you know, we can broadly classify into two main steps. One is making the backup on your host. Just making a dump of your running instance database. And then number two is now storing that backup in a place that is safe, which in this case is remote site. So the script that we're talking about needs to be able to do these two components.  So we have scripts that does backup. The scripts that were developed by Bob was doing backup and pushing using tools like Arsync to another remote site that you want to push your backups to. And we also, we could also, you know, push to S3 endpoints. And S3 is on cloud environments of your choice. And normally we normally go for, I think, Linux S3, but major cloud providers like AWS and Google Cloud have S3 endpoints, which you could also push your backups to. But that everything that I'm just talking about needs to be automated in a way. Of course, backup policy is retention policy is how many copies do you want to retain, daily copies do you want to retain, and how many weekly copies do you want to retain or even monthly copies?  Next question. Hello. Yes. Yes, I also wanted to ask, like Gerard said, these backup things, because for us, even here, we are doing backup in the same systems, in the same Linux. So we're trying to find a way whereby we can do backup off site so that in case that there's something wrong with our primary instances, then the other one will just pick up automatically. And it will be like a replicate of the same backup things that works. Because we are doing backup in the same instance. Yeah, because if you have a backup on the same system, assume that same system is compromised. Yes. Then your backup is not useful anymore. It's not useful. Yeah. It's like shooting yourself in your leg. No, we need a suit. Let's suit. So where we could talk about off site backup approaches that you could employ. And I've talked about S3. And you could also have another server sitting somewhere else, just procure a server sitting on another data center or another cloud environment. Then you just, you have a set station because Arsinki is using a set behind the scenes and push your backups to that instance. Yeah. Okay. We've done the presentations, but then I just had something for us today, which is having this site, which I've actually, I deployed DHS2 on this endpoint, but it's not accessible. I just brought it deliberately for us to have a discussion about. And when I'm doing table shooting normally, the approach that I use is follow the packet from my clients, from my Chrome browser or Safari or Firefox. Your traffic goes through the internet, through the proxy that you're using, which can be part two or the next. And then from there, it's routed to the backend application that serves your request. And depending on what you're doing, if you're retrieving data, then it reads data from the database and it gets back to you. Or if you're posting something, it depends on what you're doing really. So this is also a guideline for the table shooting that you can follow that I have an installation. So you can just segment into steps starting from your browser and then get to the network, which is the internet, and then to the proxy from the proxy back to the your app and then finally to the database. So this site that you see here, which is this domain, is not accessible. And as you can see here, we put it's getty.com-load. Just put m, it has m. Yeah, you see, this is saying that the site can be reached. It's not accessible. It's different from the one that I had before.  This is DNS BROC. DNS BROC possible. That means for this one, even this domain does not exist. It's a DNS issue. So that gets back to post-installed troubleshooting guide that you need to have a domain that dissolves to your server's public IP address. So because you can see even from the errors that we get from the browser here that this is giving us a different kind of feedback that DNS BROC possible. It's a problem of DNS. This is not dissolving to any public IP address or rather any server's IP address. And you can also test on the terminal using tools that are available like nslookup puts the DNS here. And as you can see, it is not finding IP address for that. However, this other sites that I just broke is really resolving to public IP address, as you can see. But again, it's not accessible. Even if I put here m, just to complete the thing is that it's not accessible. So that means, first of all, you've seen that it's resolving to public IP address. That is checked. But then are you able to reach that server? So you could use tools like Ping. If your server is exposing ICMP packets, it's enabling ICMP packets. However, this is not to say that if you don't receive any equal replies that the server is down now. Sometimes even on the firewall level, ICMB is disabled. But for this, for our case, it's enabled and we're able to even think that server. And we've seen that we are able to get to the DNS is resolving to that server's IP address. But we are not able to access on the browser. So we've checked the first part that our browser is okay. The internet is okay. We are able to ping our server. Now, the problem could be lying on the proxy that we have running on that server, which is in the next. So you could access that server. And just to make things easy, I had access into the server and we can list the containers that we have here. And we have proxy. We have proxy here. So, of course, the proxy is up. The container is deployed, but we are getting nothing. So one of the things that you can do is try connecting to proxy. Our proxy normally is exposing two ports, 80 and 443. Try connecting to those ports. And you could use tools like TenNet. TenNet. Your servers are public IP address and then the ports that normally we are exposing on the proxy. And as you can see, we are unable to connect. Even 443, we are unable to connect. So that means our proxy is having a problem. It could be the host firewall or the proxy itself not even listening or the service is not up. So we've gotten to the server, which is this one. And we're seeing the proxies here. So you can execute into the proxy with LXC accept. And you want to see, you want to check if your proxy is listening or the service. Proxy service is listening on the network. You could use tools like SS, PANMP. And here you see that we have nothing listening on port, 80 or 443. We have nothing completely. So that means our proxy service here is not listening. And you could check even the firewall here. That's another thing that you need to always check. Check UFW status. And when we check the firewall, we see that it's listening on port 80 and then the ports traffic are not filtered on those two ports. They are just open. But then we have the real issue here is that we don't have service here. We don't have service listening on that port 80 or 443. So you could say service systems tell, depending on your proxy of choice status in the next. We see that in the next service is not running. Here I have used in the next, but it's not running. So it's not running. And let's try starting it. System CTL. But we are getting errors. It's not coming up. We are getting errors. And they are actually here we have an error. And normally most of in the next errors are related to the configurations that you have. And you can check configuration syntax within the next dash t. That is going to give you where the problem is normally. And in this case, it's on line five. So you need to edit this file and see where the problem is. And in our case, I think it's not terminated only. So you can edit this file with the editor of your choice and then line five. There's a line here that is not line five. Yeah, here it is. It's here. It needs to be terminated with semicolon. Yeah. And that was the problem. Why our next was not, for example, running. So after that, you need to start your service service or system CTL. Or you could check even directly if it's passing fast, configuration with it's passing. And now in the next minus t, you see that it's okay. It's not, it's no longer giving us this error. And you can now start your in the next system, in the next, and it is started. So let's check if now we can see port 80 and 443 on SS command. And for sure, we are seeing port 80 and 443, they are listening on the network. Even when you get back to your clients, whatever you are accessing from and turn left 443, you're seeing that we are now able to connect. This is now going to tell us that when our proxy is okay, it's accepting connections on 443 or even port 80. You can test the two and you see we are able to connect to the two comfortably. So that means we've checked this part that our proxy is now okay. It's listening for port 80 and 443 over the internet. So are we now able to access our site? Let's reload. No, we are getting M2 response. So this is now the 443, remember the 444 error that we talked about when you do not redirect your traffic to, you know, this is just root. So, but it's not where your app is listening on. If you issue LXC list here on the host, your app is listening on DHIS and, you know, endpoints. So this is the error that we just talked about before. It's 444 error that when you had not, you have not redirected your traffic to return to minimize your whatever your application endpoint is, you get an empty response. But let's put the name of our app there, which is DHIS. You are getting bad gateway. This is another error now. It means that our next proxy is accessible. It's trying to pass our request to the backend application, but, you know, it's not, it's bad gateway. Our app is not responding. So let's get back to our install again. LXC, but then the app is DHIS. It's here, but proxy is not able to access that application. So we might say, let's execute into the proxy and try pinging, try pinging our application, which is DHIS, just to make sure that the network is okay and we're able to ping. You see, we are able to ping. But then what port is our application listening on? Normally it's port 8080. You know, you could even turn it here. Standard Ubuntu container install comes with a 10 net client. You could even turn it, your servers, your apps IP address on port 8080 and see 10 net spelling. I'm sorry. It's not able to connect. So that tells you that you have installation here, but then it's not listening also the same case. It's not listening on the network. And then one of the reasons why it's not listening is that maybe your Tomcat service is not running or number two, your app didn't complete startup process and many, many other reasons. Yeah. So you need to execute into the container, which is in our case proxy, I'm sorry, it's in our case is DHIS and see what's happening there. Normally you can check the logs, the Tomcat logs, but first thing that you could check is firewall. Do we have firewall running here? Yes, we do. But then our firewall is allowing connection from our proxy. So that is not the issue really, because our proxy is 2.2 and it's open. This entry here is open. Yeah. Any question up to that point? Hello? No, I don't think there is a question now you can continue. Yeah. So that means our requests are getting to the proxy, which is this container here, but from this container, they are not getting to the backend application. And we've seen firewall is okay, but do we have anything listening? Let's check what's listening on the network, because you could have firewall exposed opening that port of ours, 8080, but nothing is listening on that endpoint. So we could use SSTanel PNC. Indeed, there's no service listening on port, port 8080. So that means our Tomcat service is not is not running. So you could say PS out and then you grab Tomcat for that matter. And of course here, there's nothing listening on that endpoint. So what could make our Tomcat note? Do we do? Sorry. I'm going to have to leave you. I know it's, but if you are happy and if there's still people happy to carry on, I think feel free. Okay. Okay. Hello. Sorry. But before you leave, hello. Yes, let me. I have one suggestion, because this is very good and it's really helpful. Like maybe if we can have like every Tuesday, if you can have like 10 minutes or 15 minutes added on top of this, so that the issues people are addressing in the telegram group to see how best we can resolve those one, because many people will report their problems. And it's another way why we can try to see if this is the solution to their problem, then we can have a documentation to avoid such problems again. Okay. That's my suggestion. To your suggestion, we have, we have 10 minutes allocated for this call. It's not a particular user's questions. Corrections, yeah. They normally send to the telegram group and to find a solution to that. For the benefit of orders in case it happens again. Okay. Yeah. No, we can look into doing that. Yeah. All right. Thank you. Yeah. I'm just a bit reluctant. I mean, to just restrict it to the telegram group, because not everybody is on that. But for sure, yeah, we can take questions from there. Any other questions if people have a particular question, they can send it directly to, or we might also look at the community of practice data, the community. So what new posts have there been this last week, for example? And see if we can make any comment on those. No, good idea. All right. I'm going to have to love you and leave you. But I'm sure Tito, you can carry on. Yeah. I'm going to make it summarized in a very few minutes, so that we can have this decide at least accessible quickly. So what you normally do here is that you want to start your Tomcat instant and see what happens. Maybe you can follow logs and see what really happens with a system. It's bundled into simple system B service. So you do system CTL, start Tomcat 9, and then you might want to follow the logs with general CTL minus follow unit Tomcat 9. Yeah. Yes. So this is going to start your Tomcat. And at the same time, you're going to see on the login what's really happening. Normally, if you have connection errors to the database, you just have to see here why your Tomcat instance was not coming, you will get to know where the problem lies. But to shorten this demonstration, there was no really a problem why this Tomcat was was not running. It was just because I had shut down the instance. But then normally, it could be related to instance connection to the database, un-upgrade, gone wrong, things like those ones. So you will see on the logs that runs through here where the problem is exactly. So yeah. So once this instance is started, we will be able to again access this site from the internet. But then it will have checked at least four of these diesel components that we make make sure that our client is OK. The network is good. DNS is resolving. And then number two, the proxy is not the problem. And the web application is not the problem. But the problem would also even be lying on the database about the firewall blocking access from the instance about the database configuration files that has entries for BGHB configurations. Because on that file, it's where you configure maybe application user and the password and where it's connecting from. It could be the reason why you're not able to access this because of that file. So we will go up to this point. And I guess after that, we will have our service up and running. Any question after that when the app is coming up? Yes. Thank you so much for this meeting. It was a great meet. Is there any way to download these recorded video I can see later? Well, yeah. Normally when we have completed this presentation and then it's pre-recorded. And we normally upload this to YouTube channel. And if you want to follow even all previous recording you will find in that channel. So after a day or two, we will have all these recording uploaded to the YouTube channel. Okay. So can you please send me the link of the YouTube channel so that I can see? Okay. And the channel has a lot of other things that you might even be interested in following. So let's see after this application comes up, we should be able to access this site. It would not give us five or two bad gateway anymore. It will be accessible, but takes a while for the app to come up. Yeah. I guess that wraps up what we had today. But we can just wait and see what we can get after the app comes up. Just to mention also is that all these apps that we are demonstrating with right now are actually installed with the automated Ansible tools. DHS to server tools. So yeah, mostly you would not, you see that the app is now accessible. So the problem was there. But then as I mentioned, it could be somewhere else. It could be on the database. Maybe if we have a call session that is really dedicated to troubleshooting, we might explore all the possible problems that we might encounter. But for the automated install, normally you would not get these problems. They will be fixed. However, in some situations where people are not using the tools, maybe it's not suiting their deploy architecture. And they also deploy DHS to each and every component separately. Then that means this is going to be helpful for them. Yeah. So right now the app is started and it's pretty much accessible from the internet. Next slide is just for questions if there's any question. Otherwise we are actually on top of the hour. We are even past the hour. Yeah. Do we have any questions? I don't want to say it. I have a question. I will just say thank you. I think with reference to the backup, I think we've already put a hold to that. Probably it might be part of the next session that we're going to have. And pretty much what we're going to do in Rwanda. So I was to put that as a pending action for me. But then this is good and it creates a lot of awareness. It's something that you have to have the love in order for you to go through the processes. And for me, I spend most of my time doing what you're currently doing, fixing problems, identifying problems. And that is how I learned as fast as I could. So this is just an addition to the package and I appreciate it more. And it's something that I want to keep moving forward. So I wanted to say that things that I just talked about right now, we have not gone really deep into, because each and every component here could be its own topic. When you talk about Postgres tuning, it can be its own session. When you talk about in GNEX or whatever, it can be a whole complete two-hour session. So these topics are going to be deep-dived in Rwanda. You're going to have each and every session talking about one topic, extensive. Otherwise, we can just finish at that point for today's call. And thank you, everyone, for joining. Yeah, thank you so much.


This is the OTP presented by Farm Bureau Health Plans, plan on paying less for the coverage you need with Farm Bureau Health Plans. Get a quote today at FBHP.com. I'm Mike Keith joined by Coach Dave McGinnis and Rhett Bryan from Titans Radio here in the Bette MGM Studio. Gentlemen, great to have you with us. It is good to be here. Great to be here. Week 17. Here we go. Here we are. Who of us wants to play? Dave McGinnis, what is the balance between being excited that your young quarterback wants to play and trying to play with everything he's doing and then protecting him from himself? Well, they'll have to determine whether he can protect himself with movement because, you know, after one day of practice, he participated fully on the first day, but it wasn't a full speed day, which, you know, they're trying to get everybody up to speed, as you said, at the 17th week of the season. But as the week unfolds, seeing if he can protect himself movement-wise, you know, throwing wise and everything of that nature, he's fine. But it's just, you know, with a lower extremity injury, you want to be sure your quarterback can get himself out of harm's way because, you know, we all know that sooner or later, any quarterback in the league is going to have to maneuver the cylinder. So that's what they'll look at. Fans say he's been involved with a team that has given up 56 sacks this year. He hasn't taken all of them, but they have given up 56 sacks.  How do you answer that? No, you've got to, you know, you're competing. You cannot play, coach this game or play this game with fear of something bad might happen. You've got to go out there and make something good happen. That's just not how you approach it at all. But they will consider his health as far as to what he can do to protect himself. And then, I mean, my answer to that is block him. Right. What is the value in him playing the last two games? He still needs experience. This is going to be the quarterback. He's the quarterback of the future and he still needs experience. He's had eight games and that's a small microcosm of what he needs to develop being a starting quarterback in the National Football League and playing two division games. Coming up, those take on a different nature unto themselves. And so any experience that he gets right now, you can do all the whiteboard stuff you want. You can do all the walkthroughs you want. You can do, you can do all the film study you want. But until you experience it full and in live and when everything is rolling, that experience doesn't get embedded in your DNA. That's what he needs. He watched the first six games of the year, Red. Then he started the next eight and then he watched last Sunday. I thought in his press availability this week, Will Levis was very insightful about how different it was to watch last week than it was in September and October. I think that's one of the reasons why outside of the obvious one that Coach Mack pointed out about competing, I think you want to see what he does when he gets the next opportunity after he's had a different vantage point. First-hand full of weeks there of the regular season, he was doing it, watching him from the sidelines as the emergency quarterback and now he has a different perspective. But he did. He told all of us how much it was different in how much more quickly the game will come to him.  What's the most impressive thing you've seen Will Levis do as a rookie Coach Mack? He's fearless in the pocket. He is fearless in the pocket and then you can watch somebody throw the ball when there's not live things coming at him and making quick decisions. But his ability to stand in until the last instant and then flick it and get it downfield and get it to where it needs to go, that's very impressive. But the first thing is, he doesn't watch the rush. He doesn't watch the rush. Most young quarterbacks sooner or later, you mentioned the number of sacks. He hasn't taken all of those but he's taken too many of them. But it still has not deterred him from looking downfield and letting routes develop. He's not looking at the rush.  And he doesn't flinch. That's what impresses me. Reference to a hunting dog here early in the OTP. Sounds like a franchise hunting dog. I've got a lot of references in my life. I love the references. I wish we were winning more because I think the references come out more on Titan's radio when we're winning. They do. Because they're just less opportunity. You know, you've heard that old axiom, that dog will hunt. Yes.  You don't want that one. Yes. Will Leviss is the first group of dogs. Will Leviss is the first one out of the tailgate. Buddy has pocket awareness. He really does. Because some guys, I mean, like David Carr, for example, was a quarterback who took 72 sacks in a year, and it was like he just never knew it was coming.  And I sense that while Leviss has the toughness to not look at the rush, he also does have the clock in his head. Well, he does. And David Carr is a great example. Of course, you know, he was with a brand new team that had put a lot of brand new guys together. I had David Carr in the senior bowl when I was a head coach down there and he had all the physical attributes you want. And he was a smart guy, but you could see him start looking at the rush about halfway through his rookie year because, as you said, it wasn't only the awareness. He didn't know where it was coming from. And that and he never got out of that. And that was a shame. He did win a Super Bowl with the Giants. He was the backup quarterback. Now he's on TV. And you know what? And doing a good job with that, too. He does a nice job on television. You know, the other thing that impresses me about Will Leviss in the eight games that he started, it'd be easy for a guy like him who's afforded a great weapon in Deandre Hopkins just to go there a bunch. He found all how many of those games in postgame we're talking about. He found six and seven different receivers and deals the ball to, you know, he didn't go into the well at one particular well. He's distributing the ball like you would want to see a pro quarterback do it. Well, I mean, I think he's been coached to do that. And that's a good point that Rhett makes because the first time he went out there, he was forcing the ball to hop. He was forcing the ball to hop. And I think I wasn't privy to their conversations, but they talk a lot. And I'm sure, you know, that Deandre Hopkins has helped him a lot. And I know Deandre Hopkins is not telling him just throw it to me because Deandre Hopkins is way savvy in the way that this works. And to Rhett's point, you have seen him grow in that instance. Now, is he there yet? No, he's not there yet. That's why, again, back to your original question, why it's important or why it would be helpful if he could be able to play these next two weeks, those types of things. I'd like to see Hopkins go back to Houston and have a good game because he had a less than stellar game against Houston when the two teams played up here. And it would be nice to get a chance to go over a thousand yards back in the familiar stomping grounds. Derek Henry, very close to his fifth thousand yard season. Like to see him be able to accomplish that. I mean, there are individual marks that are out there. We talk about Hopkins with a thousand yards. We talk about Henry with a thousand yards. Landry is a half sack away from double digits. He's got nine and a half. And we've talked about the fact that Danico Autry is four sacks away from setting the franchise record for sacks in a season. That's hard to believe. And career numbers for him. Career numbers at age 33. 12 quarterback sacks. Well, and it just shows with this season with seven one score losses.  And of course, any of these guys would trade any of those individuals for wins. I mean, you ask any one of them they would, but it's an important part of it because and they are still performing at a high level.  They're performing at a high level regardless of the disappointment of not. We've talked about this before the rewards in this league are not money. The rewards are winning games. And but they're still going at it even without that reward at the end that has been, you know, happened seven times with one score losses. Feel like we're going to get a healthier football team this weekend. It sure seems like it. And I think that was based off of just the initial look at the injury report that came out Wednesday afternoon in the fact that I know they probably didn't do a ton of things, but there was no one who didn't participate. Right. That was my just looking at things going, OK, where's the silver lining here? OK, that's good. So, you know, I think healthy relative to where we are now. Well, healthier than last year, healthier than last week. We declared what eight players out on Friday? Coach Frable did. Right. And then had to put some on I.R. Because they just weren't going to be ready to get back for the next week. It's it's not that guys would necessarily be out for the year. In some of the cases, it's that you need a healthy body now. There you go. It doesn't feel like that's going to be the case at the end of this week. And so that's a good thing is that you, you know, potentially could take a team down there that has more available bodies and some more experience. As part of it, I mean, you looked at last week, if you took out the three specialists, Josh Wiley wasn't going to play. He was dressed, but he was not going to play. And then, you know, a couple of other guys were kind of dinged up, too. I mean, they had maybe 41, 42 guys who could really who could really help. Yeah, if you're being honest, that's exactly that's that's the number. That's the number they, you know, they had guys dressed. It's a great point that you might, but they they didn't have a full complement of players that could help in the game. And Houston and Houston was not good against Cleveland. Well, first of all, we've mentioned it on the OTP before that Cleveland defense real deal. But yeah, they made life difficult for them. I mean, Case Kingdom throws two interceptions pretty early on. They end up putting Davis Mills in there. And offensively, they had no answer for Amari Cooper.  Do you have him on your fantasy team? I do not. I wish that I did. I had Justin were you playing the guy who had or the person? No, no, and I did lose in this round, but not by a ton of points because if the the guy played had him, it would have been a boat race. So let me let me ask you this question. Who who are the best players in 2023 in fantasy? Christian McCaffrey has to be. Mm hmm. Absolutely. Who else? Um, I'll tell you one who has been really good as a rookie to that have been good as rookies. Jameer Gibbs, the running back for the Detroit Lions. OK, Sam LaPorta, the rookie tight end for the Detroit Lions. I had both of them. OK. And I'll tell you one that's a solid person who just gets you double digit digit points every week is Rishi Rice with the Kansas City Chiefs because they don't have a lot of the weapons they've been used to having. I've never played fantasy. So I don't I don't know how this works. And the reason I don't play fantasy is because I don't want to pull for a player against the Titans. Because I'm a competitive person.  And so but I don't want to pull against anybody against the Titans because I don't want one of my players on my fantasy team to score a touchdown against the Titans. And I get just as excited about them as I do a Titan scoring a touchdown, which could happen because I would lose it. And then that would be bad. There's a lot of times when I construct my roster, I have that in mind going, oh, they're not playing them this year. I'd like to take this. Yeah, I see. I could I'm too. I can't do it. Sure. I did. I never have never. Well, you did real football.  Well, there's 31 years of you. You got to pick. Help pick. Realty, there's other reasons I like hanging with both of those like y'all. Well, there's other reasons. Fantasy football is way down the list. I've played fantasy football off and on for over 20 years. And I really I enjoy it. It's a good thing to do with your friends or coworkers, buddies, whatever. You make money doing it. When I used to play in a league for those things. Yes, this league is just for pride. There's nothing invested.  But I made it to the semifinals in the league that I'm in this year, which I thought that's pretty good because I had Justin Jefferson and the quarterback with the Vikings who threw Kirk Cousins Kirk Cousins and both, you know, injured off throughout the year. And that kind of so I felt fortunate that I made it that far in the thing. But it's fun. That's the thing about it. It's fun. But I was the first pick in your draft. I'm interested to know who who is I don't know. I think it was Christian McCaffrey, Christian McCaffrey, because I took I took Justin Jefferson to overall. OK.  What was it? It was it was it was real. It was a snake draft. And I was I was fascinated by it.  I love that. But I stayed for a long time just watching it. I mean, it's fascinating the details that go into it. Well, what's funny is when fantasy became a thing. We could not play as NFL employees. Correct. We were not allowed to play. Correct. Was gambling. And then the NFL realized how many people were playing fantasy. Oh, yeah. And so then all of a sudden, fantasy was great. So then the league was sponsoring fans. They embraced it. Well, it's like what's happened with gambling. Sure. It's it's like. So when when the realization hits, that there's a way to do it to, you know, up your product. Profit is not a dirty word. Profit is not a dirty word, but it's also at the time. And I think it's probably even higher than this. Now, one in 10 Americans was playing fantasy football. And to me, I believe it. That's crazy. And it's great for the game. Sure. I had at the at the heart of the way that fantasy football has gone for, I guess, it's been around 30 years, maybe 35, whatever, whatever it is. I have a cousin, Greg, who lives in Memphis, and he literally has been playing fantasy football with his same friends from college for 31 years. I believe they all they all it's how they stay together, stay in touch with each other. Right. Life things happens. They go play golf together. But all the time he's the one that kind of got me turned on to it years ago. And he's a great cousin, Greg. Cousin Greg's big from Memphis. Yeah. Cousin Greg from Memphis. Yeah, works for FedEx and has for his whole career. And but he loves fantasy football. And it's a way he and his college buddies have stayed connected. Oh, I really I like I like it for what it does. I really do. I really like it for what it does. Well, that's why the NFL whips everybody else in terms of the ratings is because they have so many different ways that they pull you in. And they're always looking to pull at any thread to add audience. It's like just what they did on Christmas. You know, the NBA was always supposedly the owners of Christmas in terms of television. Well, guess what? Guess what? Not this time. Not this time. And, you know, there there's some talk is when Christmas is on different days, does the NFL play a Wednesday game? That's that's what it will be next year. Right. It's a leap year. And so, yeah, do they start looking to do that? But fantasy football was one of the first threads that they pulled. They understood the power of the draft and we understand it.  Everything we do will be available via one of our platforms. Yes. Because people love the draft and then the gambling aspect, you know, them talking about gambling now on ESPN and and other things about here's the over and under and here's the line and here here's. I've got some interesting prop bets for you. Well, but, you know, that used to be an outlier and the thing was you didn't, you know, like I couldn't have when I did a radio show. I could not have a gambling guy on because I was in a college town and I was on the flagship station for the for the University of Tennessee. And so I could not have a gambling guy on. Well, now gambling guys are they're part of every segment of of different shows everywhere. So you have all of these different things that pull people in and it's it's like the Taylor Swift thing. The league is embracing the Taylor Swift thing because it pulls more people in. It brings in a little bit different audience of people who who want to be interested. Well, because this is, you know, professional football is professional sports, but it's also the entertainment. It is entertaining. It's entertainment business and it's all of that stuff is great. I hope the Titans invite me back for their fantasy draft next year. I loved it. I mean, I thought I'd say you'll get an invitation. I would say so. Well, I'm begging for one here on the OTC. You kind of are. What is it? What is it about the draft process of that that interests you? The fact that first of all, people are they are, I love, of course, you know, the real draft, how much I love that we all done that in my whole life. You know, I've got notebooks full of draft stuff. But I was amazed at how knowledgeable everybody was and it was all different types of people on the team, but they all knew. And as I said, they all had their computers there.  And when I opened that can, when I opened that can up, they could start telling me, start pulling and seeing and here's where this, I was going, wow, these people are invested in this. But that's what I like.  Well, I've enjoyed, I've enjoyed, and it's a great point. I enjoyed now being able to interact with the fans. Because as a coach, you don't get that chance, you know, maybe once or twice a year at a, you know, at a big event that you're asked to go to, you can do that. But now I can be, I can be with the fans and yes, I love that. And fans of the National Football League, for whatever reason, they're fans of the National Football League, they are fervent about it, fervent about it. And I love that. Well, and that's a mistake I think a lot of people in coaching, no matter what level it is, make is they assume that people, all fans, all fans are dopey. That's the assumption that some coaches make. Yeah, and that's completely wrong. It is completely wrong. And then there are other coaches who don't. We have a coach here who does not think all fans are dopey. Not at all. Mike Rabel doesn't feel that way. But when you, you know, if you open yourself up to it, and it's like some guys too, and this is interesting, Rhett, because how many guys get out of coaching, go into media, and then in some cases go back into coaching, and they have a different advantage point from where they've been dealing with fans, dealing with people behind the scenes at ESPN or CBS or wherever they are, and it makes them better. It helps them. They're more amenable. They are. Sure. Because more respectful. They are. More understanding of what is really going on. They've walked in some different shoes. Right. They really have. In a small way, it's what we've just been talking about with Will Levis. I'm interested to see what he does if he plays in these last two games with that one game of watching from the sidelines with a different perspective now that he's played and started eight games prior. That's a good comparison. That's a good point. Good point. And then some coaches don't care at all. Zero. Well, I mean, there's a certain level you can't care what everybody thinks and what everybody's saying. No, that's different though. That's different, worrying about what people think as far as respecting people's fervor for the game. And that's a big thing to me. And I enjoy it.  I mean, it's great. I mean, there's a lot of things. But some of the most fun is after a ball game, especially at home, when the Titans win, being able to hang out of that window and see the fans. It's waving like the Pope up there. We have to hold his legs as he hangs out there. It's a little dangerous. Yeah, I don't want him to fall out. That would not be good. I just love that because everybody can, because that is the reward for this whole thing. And the fans feel it.  They feel it. I respect that. Seat Geek is now the official ticketing partner of the Tennessee Titans. Whether you're buying or selling tickets to Titans Games or any live event in Nashville, Seat Geek is the place to do it. Seat Geek, the new official ticketing partner of the Titans. So... The Titans fans can fan. Nicely done, Red. And they can wave out the window. They can wave out the window. Coach Mack. So I want to go into an area of expertise that Coach Mack has certainly shown on the OTP lately. You are the great defender of officials among us. You started this two weeks ago defending them for calling off sides in a late game situation in Kansas City. They call it again. I know. I know. And then they started calling it. They call it again. Miraculously. Yes, during the holiday season, we had another miraculous happen. Was it Valdez Scanley? I don't know. Yes, it was. Anyway. What's the deal not signaling anything after the DK Metcalf play in the end zone last Sunday at Nissan Stadium? He catches or doesn't catch a pass.  Nobody signals anything. Explain that to me, Coach. What teaching point is that in the officials? In the officials? I think you go to page three and go page three and say, not good, professional malpractice. When in doubt, don't do anything? When in doubt, just fade away. It was awful. So he catches the ball. But the question is, did he possess it and did he get two feet down? Mike, that's weird. There's an official standing right there. That's way too many questions. You can't expect that. That's way too many questions. But the answer is if he's not sure, signal touchdown. Because then it is automatically reviewed. Because they are anyway. The answer is it's horrible. But it's professional malpractice, Mike.  But I think on Titan's radio at that point, I've gone crazy. Because I'm looking out there. You see nothing. They see nothing, nothing, nothing. And at that moment, that's the biggest fear in what we do, is that because radio is not a visual medium. So people can't see it. We're trying to help them see it through the radio, so to speak. And the guy gives no signal. Nobody gives any signal. Why would he? Nobody anywhere gives any signal. Let me ask you something. They don't throw the arms up. They don't wave in complete. It's like the play never happened. It's like it never happened. Can I ask both of you something?  I mean, they're only officials. I mean, why would they give us?  Well, then they come back and mark the ball. It's awful. So I look at Brad Willis who spots. And by the way, if they're present company excluded, because you were a very great spotter, Rhett, when you did that job, Brad Willis is a fabulous spot. Tremendous. I mean, I could not look at the field and just look at Brad. And he could show me the game. I could do the game just looking at him. Because he could point, he's got the ball. Here's where the ball went. I mean, he's amazing. I look at him thinking, OK, this is the point where I've lost my mind. I knew it would happen someday. But this is the point. You just thought it was December 17, 2020 later. Listen, I thought at that moment of time, I thought I have lost my mind. 24, I should say. And they've got that camera right there in the booth on us. And so you could see the look of horror on my face. Because I don't know what's happened. Because he's going away from us. He's at the far end of the field to us. He's on the far end of the far end of the end zone on the far end of the field. Far pile on. Yes. We can't see it. It's obvious he's caught the ball. But we can't tell. And no signal. So I look at Brad. And Brad looks at me like, I got nothing. I got absolutely nothing. I mean, you can't do that. Do the total deer in the headlights. Total. Mike, it just it just points out. And again, I know how hard it is. I live I live it on the sideline for 31 years. I know how hard it is. But it's not hard to do your job. And when you're that indecisive, all the instant replay has paralyzed these guys. It has paralyzed these guys to the point as. And again, I keep talking about they fast track people into this league so fast that it's just it's it's it's too much for it's like the screen went blank on that dude. And I'm just sitting there looking going, this is amazing because the no call and it took them a while to sort it out. Now I want to ask you, yeah, there's been talk about. Will they ever be be made full time official? I don't think that will matter. OK, that's my question. Well, what would they my question to that is I've heard the advocates for full time official. What would they do all day? Well, yeah, but here's the thing. Hang out, touchdown. Play fantasy football. Are they so they'll watch every play that ever happened. And I think they're just paralyzed right now. And I think this is a great example. And it goes back to the offside thing. And it's like, are we. Are we going to call everything?  So they called illegal contact on a play that if it was beyond five yards down the field, it was five and a half. And it was on the other side of the field. Oh, my just wow. And you're just like, OK, so we're. I understand you're seeing this and I understand it's your job to call certain things. But doesn't there have to be a bit of common sense in it? Well, there is no common sense now. And here's the thing. There used to be common sense because there was no replay. And a call was a call and it was over. But Pereira had it when he was running the official. Pereira had it because but here's what he did. I mean. Look at the number of years that you had to be in the National Football League when I first started. Right. Before you could be a referee, a referee in a white hat. Now they're fast tracking those guys so fast. What does that tell you about the roster that's behind them? It did. They don't have enough experience in the big time doing it. And also all of this, all of all of the scrutiny that they are under now. They're choking on it. They're just choking on it. My question for you is how much of the league tweaking rules every offseason has accelerated this? A lot. It's a significant part of the blame pile. It is. But to Mike's point, and I think they need to go back. I mean, I've been in competition meetings. They need to go back and say, hey, how about all of these holding calls away from the play that have nothing to do with the play downfield that all of a sudden alter the whole sense of what's going on in that series? Can we come to an agreement that maybe this isn't part of it? And then the other thing is, is start making them make definitive calls. Right. Make a definitive, make a call. If you're wrong, you're wrong. But by not making a call, then you're never right. Because just as you said, we're sitting up there. And I feel it now more than ever. When you're on the sideline, on the sideline is the worst vantage point to watch a game. That's why there's people in the booth that tell a head coach when to challenge. There's a lot that goes on communication-wise because you can see. But we can see it. And when we see something like that, all we need is a little indication as to where they're initially going to go. That just happened in this last ball game was frightening to me because there was nothing, nothing called. And then it took him forever to figure out nothing. What? It was unfair to Seattle. Yes. Because Pete Carroll had to challenge that. Yes. And he used his second challenge to do it. So he had over 10 minutes remaining in the fourth quarter where he didn't have a challenge left. Yeah, and that's not fair. That's not fair to Seattle. That's not fair to Pete Carroll. The other part of it, too, is we're talking about fantasy football and we're talking about the different things that go on with the draft and Taylor Swift and gambling and how many different things people choose to follow because they love the NFL. I mean, this is entertainment. And if you're calling something 45 yards away from the play that doesn't affect the play, is that good for the business? No. It's not. It's not. And so they're going to come back and say, well, we're calling it by the letter of the law. No, you're not. No, you're not. Now what you're doing is. So you now agree they shouldn't have called that off sides. No, he was off sides. It all comes back to that. He was off sides. And then he was off sides. And they called it again. But did it affect, should you have called that in that moment based on what I'm, and that's what I'm talking about with some of these other things. But some of these things, I think it's the common sense element to me that I mean, the call on the illegal contact was it on Al Shire? I believe so. I mean, I was looking at it. I mean, you have five yards. He didn't punch the guy in the neck. He just sort of he bumped him pretty good. But if if it was beyond five yards, it was six. You're right. Come on. I mean, really? Well, and the other thing that just wears me out to is, of course, I agree with protecting the quarterback. I agree. But when you got somebody and we had a call on us and then I mean, I saw it twice last week where they're hitting hitting a quarterback in the midsection at the step and a half. And then it right. So it's way out of whack, Mike. It's way out of whack. And I'm not saying this because I am I'm trying to denigrate the officials. I'm just I just know that the groundswell that's coming, even the uninitiated can watch it and go, is this right? Yeah. Yeah. Well, you know, I think a lot of where the problem starts red, the state high school associations all over the country are having a brutal time finding officials. They are they are begging. They're advertising for officials for Friday nights. And there has to be carryover to college from that. And if there's carryover from that, it's got to work its way up the pool from which you start. Because I mean, you don't start in the NFL. No, you start in high school. And that's the daisy chain of it. Sure. You're 100 percent right. And there's not enough aren't they changing when high schools have to play games the day of the week? Because in Tennessee, every high school is required to play at least one Thursday so they can make sure and have enough crews for every weekend. So so a crew can work a Thursday and a Friday. I mean, it is it's it's it's a problem that I don't think is. I think it's one of those things that that. You try to do people try to dismiss and think it'll just go away. It's not going away. No. You heard Mike Vrable address it at his press conference on Tuesday. And the frustration you hear in his voice is he said, all I want is consistency and stop the indecisiveness. Now, he mentioned his son, Carter, who plays baseball now at what Tennessee Tech, right? He's like, you got a ball, you got a strike, you got an out.  Then there's replay elements to help, you know, figure out exactly what things are, if it's something in question. He said that, you know, the indecision is is a part of the problem. And I thought how he talked about the particular play we were discussing. He sounded much more like a member of the competition committee. Absolutely. Correct. Then he did the head coach of the Titans, because like I said, it wasn't necessarily unfair to the Titans as much as it was unfair to Seattle in that instance. I mean, the official to me has to know if I'm not sure I've got a call touchdown because then it automatically goes to replay. Well, and that's and you're right. And Mike Vrable wasn't talking as the head coach of the Titans. No, he was talking for the good of the league. Sure. But he's what this is affecting. Right. He's not the only one talking about this. And we're not just talking about, you know, calls against the Titans. That's not what we're talking about. We're talking about administration of the game. That's what we're talking about. You know what would really help the officials? Duncan Duncan. They need to if they had more Duncan on game day, this wouldn't go on. This wouldn't go on. Yes. Officiating runs on Duncan. Hey Titans fans and future officials. It's always game on with Duncan. So grab a coffee and kick off the action with that drinking a cup of coffee on your way to the game or grabbing one to go before watching the game at home or officiating a game. Duncan is always there to help you get your game on, just like the pros. We need to be at our best come game time, which is why Duncan is the most important part of your game day ritual because it's always the best call for football. America runs on Duncan. Ready to go to Houston? Let's go. Yes, I'm ready to go. Let's go to Houston. Let's go to Houston. That's what a lot of folks. They ask, you know, how do we keep going when the season's not going well? It's like, we get to go watch NFL football. It every week you build up, you talk about we've all been doing this a long time. And but it's what you do when you when you when you want the competition part of it, you build up to it. Mm hmm. Forget it. I mean, we're talking about game day in the NFL. It's the best. Wooden trade places. It's it's the absolute best. I mean, it you have those moments, Mike, we look and go, how are we? We're doing to do this. This is crazy. It is crazy. Well, because because because we're a couple of goofs. Y'all are really good at it. And when it's called us, I'm just I'm not disputing. I'm an agreement. Well, he's been there the whole time. So he know full time goof, full time. All fans are not dopey. That's correct. No, that ought to be on a t-shirt. The other thing about going to Houston and, of course, it's just it's a sidebar to all this is you got the original Tex-Mex restaurant down there. Oh, we might go by there. Oh, no, we're going. OK. We must we're going in that. And that and I as long as I've been with with Titans radio, I've decided there's two things that as long as I'm with Titan radio, I'm going to do and it's going to be Coach Max centered. We're going to go to Mama Nymphas and we're going to St. Elmo's. Well, we're going to eat. Yeah, but we're going to I mean, we're going to get that's my one thing. Yeah. And the other thing is, is you're not you're not those two. Those two, like I'm just saying this for the OTP people. You're not paying for those two. So just get ready for that. Well, I appreciate that. We'll see. You're welcome. No, we're not seeing nothing. But but the thing is we're going to eat. We're going to eat some spots. You mean like the officials? You're not seeing nothing. That's like not seeing nothing. Didn't you like how I I came up with the follow me through Tennessee feature so I could go eat good places in Tennessee. That's genius. First of all, it is genius. I like it is. It's my it's my highlight of 2023 genius. I like the feature to begin with because I'm always interested in places in in our great state of Tennessee. The excuse to eat just makes it. Oh, it's so great. The one we have this week where we took Brett Kern to Jiffy Burger in Manchester. OK. I don't want to say it's my favorite. It's right up there because of Kern. Because he I mean, he had just retired. He had he had just retired. And I had just had my shoulder replaced two weeks earlier. So we had an opportunity to go and he says, listen, if we're going to do it before I start traveling with my family this summer, we've got to go now. So I didn't know if I could drive. Yeah, because you're all slinging. Oh, I was totally slinged up and I might have spent part of that day without the sling on. I guess doctors.  I did. Yeah, I did. But the Bonnaroo Burger was worth it. If you go to Manchester, go to Jiffy Burger. First of all, have your picture taken with the Titans helmet that we have there. They have a game warm Titans helmet now that they're very proud of. So ask them if you can be photographed with the Titans helmet and send it to us. We'd love to see it. But the Jiffy Burger is crazy with this Bonnaroo Burger. They have a lot of good stuff. Apparently they have a great breakfast, too. What is on the Bonnaroo Burger? It's an egg. Oh, come on. It's here. We're starting off good. Cheese, two patties, an onion ring, mayonnaise. While we're talking about on the road with Mike Keith, let's where's the place that you stop when we go to Mobile? Durban Farms. Oh, Durban Farms. Unbelievable. It's pretty great. It is unbelievable. Clanton, Alabama, exit 205. I mean, we went the ice cream is unbelievable. No, the whole setup is great. Well, so it's a farmer's market. It is that and so you can get all kinds of produce and good preserves and pecans and peanuts and I mean, anything you want. And then you go inside and then they do like they've got a kitchen and so they do sandwiches and so on and so forth. So it's a good lunch. They kind of have a little bakery dealio, but they they homemade. The ice cream is all homemade. It is. It's the greatest cookies and cream ice cream that I've ever had. The whole thing was I was just fascinated. I mean, I loved it. I loved it. I love those kind of places that you find. I loved it. And then you go to that farmer's market. I mean, I walked around in there with you and Rhett for a long time. Took us longer to get to Mobile, but it's worth the stop. We hit it on the way back and I brought home all kinds of produce and next time you have to pay for it, though.  I don't even know if I could go if you're telling me nearly. That was a good move, though. It was cheaper that way. It was a good move. It just made a dash to the car. Give me the keys, Mike. But against the line. I kind of wondered why you were in such a hurry. You were sweating. Mike, give me the keys. But Durban Farms, if you go there, tell them we said, hey, people who are on the way to 30A will stop at Durban Farms. What a great place. On the road with Mike Keith, there are some places that are really cool. Well, I love that stuff.  Amy, when she comes back, you know, she started doing the follow me through Tennessee because she made fun of me on the OTP.  And I don't know a spot in every county, but I know enough. I bet it's close. I know enough. And then I know enough people who can tell me. Make recommendations. Places to go. Yes. Yep. And it's like we went, you know, we went to the pie shop. That's so great. And just outside of Cookville. That is so great. The Rose Garden. The Rose Garden. Off exit 273. And you know, and the first time I was ever exposed to it is going to Cookville to record something with Larry Stone and Doug Matthews just whips off the road and starts driving. So I said, where are we going? He said, you'll find out. If you're going east, just get off at 273. Take a right and you go a quarter of a mile and it sits there and it just says restaurant. It says restaurant. And you know a place is good when it just says restaurant. Because that's really all you need to say. It's all you need to say. Food. Here. And the food is fantastic. But the pie. Unbelievable. Unbelievable. And I'm not even the world's biggest pie guy. I would be more of an ice cream guy than a pie guy. The thing is cool about that place though. And it's just like all these kind of places we're talking about. I'm sure throughout Tennessee and Texas. Anywhere you one of these awful.  That's right. The same people will be sitting there. I love that stuff. Yeah, it's it's been a lot of fun. But this one this weekend, Jiffy Berger in Manchester. Bonnaroo Berger. Sweet people too. Kylie. David and his daughter. Well, they're excited to when the voice of the Titans comes walking in the door. And he comes bearing gifts when the game you tell them. He comes bearing gifts. He could lift it, but I had one. At that point, yeah. I was like, okay.  I mean, Mike. There was some trick photography involved with that shoot. Because if you watch it. Is there some Titans CGI involved here? It's not really CGI, but they they had to like, because I couldn't bump into anybody. When you have that kind of surgery, you know, the big thing is you can't take a hit of any sort. So they hit, they let me get to the booth. And then I was back in the corner of the booth with my shoulder just up against the wall. So nobody would, would bump into me. And then I'm just sitting there. It's, it was kind of like, oh, he's really lazy. Because Brett Kern went in the kitchen and flip burgers. And I would normally love to do that. That would be a lot of fun, but I couldn't get near anybody at that point. See that's a little scene behind the scenes. So if you watch it on Titans, all access or you watch it on the Titans YouTube channel, keep an eye out for the fact that, and I mean, I've got like a wind jacket on with a t-shirt because that was all I could pull over my shoulder. I couldn't do a button-up surgery. In other words, if you see this feature this weekend, you're going to go, what the devil's wrong with Mike? Oh, at the shoulder surgery. Yeah, you've got some Easter eggs. There you go. That's right. But Brett Kern is just the best. Well, of course. Of course he is. He was hilarious. He, because he just really didn't care anymore. So he told like every story in the truck going down. And yeah, that's going to be the next OTP as a matter of fact. That's awesome. Outstanding. Well, that's what we do. We attempt to do outstanding. Subscribe to the OTP. It's worth it. Please, what do they say? Subscribe, rate and review. All right. Well, thanks for rating and reviewing and watching and good to be headed to Houston with you boys. Final road trip. Happy New Year. Happy New Year. Happy New Year, Mike. All right. For Brett Bryan and Coach Dave McGinnis, I'm Mike Keith. Thanks for joining us for the OTP.


Okay, so this is a hand We can see the wrist bones here. So there's multiples of those Collectively, we'll call those the carpals the long bones here In the middle are your meta carpals and then each of the finger bones are collectively called phalanges, but we have Medial phalange distal phalange and proximal phalange the exception of the thumb which doesn't have a medial phalange It only has a distal and a proximal