This is where we are in the media architecture. Remember, we talked initially about Express Database Server. We spent a good bit of time yesterday talking about Tomcat that's running the DHS2 instance, the way that it's set up, and how to do basic operations with it. So this morning, we're looking at the front end of the system, really, the proxy or reverse proxy server. And as you can see in my diagram there, I've set Apache 2 slash Engine X. There are actually quite a number of different options one can use for the proxy, not even just those two. But we'll talk a bit about that as well. So what is a reverse proxy? OK, first of all, it's interesting. I tend to use, and people often do, tend to use the terms interchangeably, a proxy and a reverse proxy. Technically, they're slightly different. Reverse proxy, I guess, this is my own definition, so it might not be authoritative. But it's basically a web server, front end, which accepts requests from client software out there in the world, people's browsers, people's Android tablets, whatever it might be. And it routes them to back end application service, which in our case is primarily these Tomcat servers running DHS2, but there can be little Node.js servers for any kind of web application server that's running at the back end. So your reverse proxies function is to take those incoming requests and route them to application servers at the back end. As a technically, that's a little bit different to what a web proxy is rather than a reverse proxy. Web proxy traditionally is something which does the opposite, in fact. It's the kind of thing that you find in a lot of particularly large organizations, enterprises. It's a server which accepts requests from the inside of the network. So maybe in the university, for example, when you're accessing the outside world from inside the land in the university, you most likely don't access directly, but you go through a web proxy. And a web proxy is used for caching. It's also used sometimes for censoring what websites you're allowed to go to, things like that.  In fact, it's a reverse proxy, right? It's slightly different to what's typically described as a proxy. But essentially, it's something that sits in between, right? Typically, these things are written in C, often very widely used. Everybody uses proxies for all kinds of things. So battle tested, highly optimized for serving very, very massive loads. Generally speaking, when it comes to performance, when you're looking at performance programs that you might find it with DHIS too, the problem is almost never going to be the proxy, right? I mean, you get big arguments about this proxy can serve a million requests per second, and this one can only serve 50,000 requests per second or whatever it might be. But generally, those figures are much higher than anything that we see. I mean, sometimes we get some pretty big servers out there. I know Bangladesh. I've probably seen some of the biggest loads that I've seen. But even there, it's not Netflix, right? So the kind of software that we use for proxies are generally more than adequate to deal with the kind of loads that we have. So why do we use it? Why do we have a reverse proxy? If we go back to the architecture diagram, we know that Tomcat is running like a kind of web service here. Why don't we allow the clients to go with it and just go straight to the Tomcat? Why put this thing there at all? Well, there's a number of reasons why we do it. Yeah, technically, it's not strictly necessary.  But you can configure your Tomcat also to listen on port 80 and port 443 and just go directly to it in your browser. There's quite a few reasons why generally we don't do that. I've just listed out a few here. There's probably more. One is kind of flexibility of routing.  Then you can change out your application server, move it from one machine to another, whatever it might be, and do an upgrade on one container and then simply flip over the connection. There's a lot of flexibility you get through having a man in the middle. Related to that, I guess, is load balancing. What load balancing is where basically your proxy runs one of a number of algorithms to decide incoming requests, which back-end worker to send service that request. So instead of having one Tomcat, you might have four or eight, whatever it might be, configured as a team of workers. And this way, you can horizontally scale. People often ask me about load balancing and why we don't do it in these tools. It's primarily because it does add some complexity because you've got to manage the state between your different Tomcat workers. And there's no benefit in doing it unless your Tomcat back-ends each are bringing more resources to the table. So if you have, like what we typically have here, a single virtual machine, then no matter how many different containers we make inside of it, we still have the same amount of RAM and same amount of CPU that we started with. So it only makes sense to load balance when each of your Tomcat workers that you're creating effectively are coming with their own CPU and their own RAM. And in an ideal case, you can scale them up and downs of periods of heavy load. You might have more workers and periods of less load. You might have less workers. That can be important in scenarios like things like AWS where you're billing by the second. You pay more under periods of high load and you can pay less under periods of low load. Anyway, as I say, I generally don't advise people to jump straight into load balancing just because it is described in the manual and it is supported. It's something that you should jump into after you've explored all of the other performance optimization possibilities and found that on the basis of the evidence of what you've measured and what you've seen, the only solution to your problem is going to be to start load balancing because it's a little bit less trivial to set up and manage. The other thing your reverse proxy typically does is we call SSL termination. When you come with your browser to HTTPS, whatever the URL of your site is, typically we install the SSL certificate not on the Tomcat, but rather on the proxy. So the SSL connection is effectively dealt with by the proxy, which is generally quite fast. It's C code. It's code that's used in hundreds of thousands, millions of other sites, so it's well optimized. And then typically the connection to the Tomcat is just plain HTTP. Again, if you're in a cloud environment where you've separated these functions onto different machines and you're going through your provider's network, you might have to have SSL connections between your proxy and your Tomcat, as well, to make sure you don't have plain text traffic going through the foreign network. Within our internal world, within our containers, and our LXD setup, there's no need to SSL between the proxy and the Tomcat. So SSL gets terminated by the proxy. Proxies, again, depending on which one you use, bring a whole lot of additional functionality. Other advantages you have by having something in the middle. Picking related to security, there's quite a lot of different security features and possibilities you get, which would be more configurable on your Apache or on your Engine X than just talking to your bare Tomcat. The other thing is related, of course, to if your Tomcat was listening on port 80, it would have to be running with root privileges. We talked about yesterday how your Tomcat basically is a bit vulnerable, and if anybody does succeed in attacking your Tomcat and getting access to the server as the user that's running Tomcat, if Tomcat is running as root, which you would have to do to be listening on port 80 or port 443, then the extent of damaging and cause would be much more. Yeah, there's a whole bundle of security related things that's using the proxy helps you with it. You can configure web application firewalls at the proxy level. Yeah, there's other things. URL blocking. URL blocking is quite a common thing. It's not so common, but it's something that we do. It's a useful thing to have in your toolbox to be able to block access to certain URLs or perhaps blocks it and types of clients as a way of dealing with a particular problem that your server might be having. The other thing on the pro, you can actually rewrite the URL in the request or other aspects of the request. Again, it's something that's sometimes useful to do. So you can actually modify what's coming in before you send it off to the back end. Rate limiting, something we're going to talk about a little bit. If you're Tomcat's getting overwhelmed or swamped by basically too many requests coming from the outside world, a lot of proxy software have the ability to perform various types of rate limiting or request limiting. We'll talk a little bit about that. The type of software that you use often determine how much of these different types of functionality that you're going to get. If you view all of these sort of things together, sometimes they're described as virtual patching. One of the things that your reverse proxy is very handy for is reverse is virtual patching. Virtual patching basically is you're compensating for some problem with the application perhaps, which maybe has come to the attention of developers. Developers say they're going to fix it and maybe it's going to take them two weeks to fix it. Maybe it's going to take them two months to fix it. But in the meantime, your server is falling flat for one reason or another. So you need to take some kind of emergency measure. You've got to patch it somehow while you're waiting for the real fix. And your proxy frequently is your way out there. The other thing your proxy can do is caching, particularly things like static content. We've got a few issues with caching at the moment, DHS2 I need to figure out before I turn on the caching. But yeah, particularly for things like static files, it's kind of a waste of Tomcat resources to be going to the Tomcat all the time, to be picking up JavaScript files and CSS files and things like that. Having got them once, they can be cached at the proxy and that relieves load on the Tomcat for subsequent requests. So yeah, lots of reasons to use a reverse proxy. There's probably more. These are just the prominent ones that I could think of as I was putting the slide together. Well, what kind of reverse proxies are out there? Well, quite a few, but there are probably three names that come up the most in this regard. NGNX, Apache 2 and HA proxy, each of which have got different strengths, different weaknesses. A lot of the, I mean, as people would know, I've used Apache 2 quite a lot. It's still the default in the DHS tools setup. I've been swimming a bit against the tide on it. I know there's a huge demand. People want to use NGNX. But most of the, if you go googling around on the internet about NGNX versus Apache 2, you'll read a lot of fairly unscientific, poorly researched stuff. From a performance perspective, there's actually not very much difference between them currently. I mean, there used to be years ago. But I think it's less about the performance. Probably it's more about the familiarity with the kind of configuration, formats. To find NGNX is just more, a little bit more modern, a little bit easier to grasp, perhaps. Apache 2 configuration is a little bit odd to say the least. It's been around for a long time, and maybe it shows it. But yeah, so a current implementation is based on Apache 2. Tuzo, I don't know if Tuzo is on the call. Tuzo from his Tanzania. He actually made an NGNX version. You still see the remnants of it is in there. Unfortunately, the first attempt at that, it did kind of break everything else. So we kind of retired it temporarily. I did tell Tuzo I was going to go back and fix it, and that's now six months ago. So yeah, I've got more urgent reasons to move now. We'll talk about it a little bit later on. So yeah, we will make an NGNX version. It's quite possible to set up a container.  I mean, creating containers is relatively straightforward. And when you make a container, you basically just have the equivalent of a running a Ubuntu machine or whatever other system you want to put in it. And you can install whatever you want on it. And so in practice, I've actually put together quite a number of NGNX proxies. So using the same setup, but just creating the NGNX server manually. What I don't have is an automated script to do it yet. So I didn't want to show you that today because I've been tripped up a few times yesterday, being one of them, trying to do last minute fixes just for the presentation. So I'm not going to do it for this. But within the next two weeks, I'll sort it out. As we go through this presentation, you'll see what's involved is not really very much. So yeah, the Apache 2 works fine. And big advantage is that it's all managed automatically at the moment, which is certainly convenient. And it's well tested. But I do understand people also want to, people have preferences. And also there are some benefits currently around the rate limiting for using NGNX. So yeah, as I said, the Apache 2 installation, it's been very well tested. It's fairly secure. I think I've done quite a lot of due diligence on it. I would want to do the same with an NGNX version before we released it. And it performs really very well. As I said, the rate limiting module on NGNX is a little bit better. It's still got problems. As I'm going to show you, it's far from ideal, but we'll talk a bit about that later on. Okay, so how does the Apache 2 get deployed?  The container just comes up and what needs, when it comes up, it's running the, all you see when you log into it is the kind of Apache default landing page on HTTP. At that point, you need to install your SSL certificate on it before you can enable the DHS2 configuration. By convention, I guess, I always have it configured with IP address 192.16802. 192.16801, that's the IP address of the host machine. And the proxy is always on two. Now it shouldn't have to be like that. Could be any IP address at all. But I've got a number of scripts where I've taken a few shortcuts and just assumed that hard coded IP. I need to generalize them a little bit. So yeah, for the moment, don't try changing the IP address of the proxy, just assume the proxy is always on zero two, then everything else will work with it. If you want to put it somewhere else, you can, but you're going to have to make some modifications somewhere else. The way that requests are forwarded from the outside world into that proxy container, we, I've taken a very kind of brute force, simple approach and use the IP tables, which is the kind of built-in firewall within the Linux kernel. Again, it doesn't have to be done that way. There are other ways to do it. You can do it using LXD. LXD has a proxy device you can use. That's more similar to the way they do with Docker as well. I prefer in some ways just doing it with the IP tables. And essentially what it means is these couple of cryptic-looking lines here, what is it, six or seven lines, six lines. As part of the install script, when you do run your create containers, those six lines get appended to the top of your before.rules on UFW. And that's one of the reasons why, you know, a few people have found this already. You try to run create containers and if you're not running UFW, it's gonna fail you. That's partly because, you know, UFW running for these before.rules to kick in, as I thought your proxy wouldn't work. You'd have to manually create this. The rules themselves, it's not too important. It's good for CIS admins to learn a little bit about the way that IP tables rules look like. But you don't actually have to do anything unless you're gonna change the IP address of your proxy. Essentially the important bits are here is to, these two lines here basically say everything coming in on ETH0, which is using protocol TCP, which is trying to get port 80. We're basically gonna push that instead of handling it on the host, we're gonna take those packets and forward them through to 192.16802.80. So if I go with my browser to port 80 on my host machine, effectively what's gonna happen is that browser request can be routed straight into the proxy. And we do the same thing with the HTTPS traffic, which is coming on port 443. Okay, so that's the way traffic gets routed into your proxy. So you're looking at it more broadly from the flip back up again. If you look at this whole machine here from the outside, the only three ports that should be open at all ever. So if you have an external firewall and you're requesting various ports to be opened, all you really need access to is port 80, port 443 and whatever port it is we're using, you're using the SSH port 22 or whatever it is if you've changed it. You shouldn't have any other ports open at all. Everything else should be internal to the network. So the SSH port we know about and anything of port 80 and port 443 is gonna be handled by your reverse proxy. Sorry, back to the right slide. Okay, so this is just really a bit of explanation of what's happening under the hood. There's not anything that you need to do about this. It's worth noting that if you're using a proxy other than Apache, whether it be engine X or HA proxy or something else, exactly the same thing would apply. We'll set up on 192.16802 and these same forwarding rules would be used to send the request through. Okay, from how is the Apache configured? Well, so by convention again on Ubuntu-based systems, it is possible just to have a single configuration file, by default the configurations that have broken up into sites. And so when you install Apache through the package manager, you find that you've got a list of site configurations sitting under et cetera, Apache two sites available. Initially all you've got is this two default sites as a default site for HTTP and there's another default site if you have SSL enabled. And the way that you enable sites, turn them on, turn them off is effectively creating a, what are called in the Unix world symbolic links. You create a symbolic link from the sites in site available into the site in sites enabled. That's something like, I guess for Windows people, that's like a shortcut, right? Make a shortcut, basically a link to the other file. Again, if you're doing this with engine X, it's exactly the same except you've got et cetera, engine X sites enabled and et cetera, engine X sites available. Apache two has got two handy little shortcut commands, which I think engine X doesn't have, which is a bit peculiar, but I find them useful anyway. When you want to turn sites on and turn sites off, two little command, Apache two enable site and Apache two disable sites. I think I've got too many S's in that. I think it's actually a single S, A2EN, S-I-T-E. That's basically what we use to create the symbolic links between sites available and sites enabled. I'll show you a little bit how this script does this. So when you just install Apache two with a package manager and you don't fiddle around with any of the sites, then the default site is there by default, right? And that just gives you Apache two landing page. We looked at this when we did the installation last week, right, when you very first install and then you can go with your browser because the firewall rules are in place and you go with your browser, you end up at the default site. And that was the thing that I had said was important. You want to test that, make sure you can get to the default site before you attempt to try to set up your SSL. Because if you can get to the default site, it means your networking is basically set up, is working, it means your firewall rule is working, it means your DNS is working. Those are all things you want to test before you want to try to get your SSL sorted. After you've tested all of that, again, this is what we did. At that point, you can install an SSL certificate. Easiest way to do it is to get one from Let's Encrypt, but there are other ways to get an SSL certificate, which you might pay for, you might some places even have a government's CA, which won't issue them. And wherever you get your certificate from, after you've installed it, you can then disable the default site and enable your DHS to specific sites. And we saw the other week how the little script SSL set up, that does that automatically. Might be a good time at this point to just quickly review that script. So in light of the explanation I've just given, so you can see again what it does. Why it's still logged in here sometimes, SSL timeout. It's not in a very clever case, I just put it here and it probably moved it somewhere more sensible. This is basically the script that you need to run after you've installed Proxy for the first time. You can see I've got this bold warning up here telling you to check out now if you haven't tested yet. Okay, this thing parses the configuration file. This is just to set a couple of variables. Probably vaguely remember this at the start, you define things like your domain name and your email address and your time zone and kind of thing. They get parsed up there. You get the choice to back out if you haven't checked everything you need to check and then off it goes. And it's basically gonna do run these commands on the proxy. You see lots of LXE exec parts. Stop the Apache server. We'll get a certificate using cert parts. Disable default site. This is the default landing page you were getting before and then enable the configuration which is called Apache-DHIS2. I'm gonna show you what's inside there shortly. And then restart your G7. So this is really all that happens when you install your SSL certificate. We just bring down the default sites, load up the certificates and then enable the DHS2 specific sites. I generally do these steps manually because I'm very familiar with them but I made the little script really for your benefit rather than mine. The other thing that we do is the front job that comes with the cert bot package. If you just do a apt install cert bot which is basically all done, doesn't really work very well. I've installed a custom cron here that just make sure that your certificate is up to date. The one downside of these let's encrypt certificates that we get with cert bots is, I mean, the great thing is that they're free. The other great thing is you don't have to go through any complicated application process. You don't have to provide a credit card or anything like that. You just need to be able to prove that you own the domain or that you have access to the domain. But the downside is they only last for three months. I typically, when you buy one, you buy it for a year or for two years or whatever it might be. These free ones, they expire every three months so you've got to keep checking whether you're getting close to expiring time or when you get close to expiring time you need to renew. And that's all this little job because I'm basically substituting the job that was already in there with a new job starting from here, which says at zero minutes past three, so at 3 a.m. every morning, right? Star, star, star, we just mean every morning at 3 a.m. As the user routes, I'm going to run this command. I'm going to say cert bot renew. Minus minus standalone means that rather than using Apache 2 or Nginx to service the requests, I don't know, I just felt more reliable to just shut down the Apache server, get the certificate and then start up the Apache server again. We do the same with the Nginx. Now you might think this is an intolerable service disruption. It's not actually as bad as it sounds because what happens with the renew command is that it's going to check, right? It's going to check every morning at 3 o'clock whether your certificate is within three weeks of expiring. If your certificate is within three weeks of expiring and it needs to be renewed, only then, right?  It'll refresh your certificate and then start your Apache. Typically that whole process takes about 30 seconds. So it means that your once every three months at three o'clock in the morning, your Apache server will go down for 30 seconds. Generally speaking, I think that's adequate and it saves lots of other configuration fiddling that you might need to do. Again, if you're running an Nginx proxy rather than Apache proxy, this little cron job is exactly the same as it's going to serve as Nginx stop and serve as Nginx start when we start up again. Okay, it's not the only way to do it. It's the way that I do it. It's quite important to have that cron job there because otherwise you're going to get complaints after three months so your users suddenly can't get in because the certificate's expired. If you've configured your email address properly, I'll get out of here, I'll get to my windows. I'll do it here. If you configured your email properly at the point where you're applying for your certificate, then Let's Encrypt will send you an email reminder to say your certificate's about to expire and then you can go in and manually renew it. But it's much handier. Once you have the cron job working you can just forget about it. It will simply check every morning at three o'clock for you and once every three months, it'll just renew it and take care of itself. Okay, so configuration wise, that's basically what happens behind the scenes. There's not really much you need to do unless you're going to do this manually. When you run create containers, the Apache server gets set up like that. It's got the sites available for use and you need to just set up your SSL before you can enable the site. And that's what the little script does. Okay, so what's in that Apache minus DHS2 configuration? Well, we'll have a look at it after we've gone through the slide. Basically from a basic user's perspective, right? Unless you have a little bit of expertise and experience for these things. Basic perspective, you don't need to do much, right? The configuration has been done for the most part. We did spend quite a bit of effort. Again, like with the Tomcat going through the CIS security benchmarks, kind of like a long checklist making sure we could tick as many boxes as was sensible or possible. So it's reasonably secure as I tested them on SSL labs and just checked again last night, still getting an A plus on it. One thing about SSL configuration though is that what is secure today isn't necessarily secure tomorrow, right? The world keeps changing. So you can't set up something like this and then just leave it running for five years and without really reviewing it because protocols that are considered good now, encryption ciphers that are considered good now are not necessarily still gonna be good in a year's time. Vulnerabilities get discovered, things get modernized. So you do have to review this from time to time. I'm due to give it another review as well. It's probably quite a few things we can still fix up and it's always great to have other people having a look at these things and coming back and making suggestions, things that may have forgotten, things which we could do better, et cetera. So yeah, the main reason you'd ever want to manually edit the configuration file would be to implement some kind of virtual patching, right? You've got some things you need to block or limit or whatever it might be or because you need to update the SSL configuration for some reason or another. The actual proxied locations, right? The things that's the URLs that you are routing through to the backend applications. Again, if you're using the deployment scripts and those things are created automatically when you create an instance, right? You saw us run a few times yesterday, DHS to create instance does three things, right? It makes the database for you, it makes a Tomcat container for you. And the third thing that it does is it goes and creates a little snippet in the proxy to tell it how to route it through to the Tomcat container. A kind of ease of maintenance, each one of those little snippets end up in their own file. So I've created a little directory there called, that's not called uploads at all. It's called upstream, et cetera, but I should fix that now before I forget. I need to get out. I do want to get out a presentation mode. We're going to look at it in a minute anyway. So every time you create a new container, you make a new little snippet and inside that little snippet, it describes the way that request get root to your Tomcat. Just as kind of a little aside, the Moonin node has a plugin for Apache 2, also has a plugin for engine X, and that also is configured by default. So you can see basic Apache 2 performance graphs on your Moonin. What does that snippet look like? Yeah, I've got one here. So for example, we create an instance called HMIS, which was running on 191.680.10. This is the way a location block looks in Apache 2, which will take requests for HMIS and forward those requests to Tomcat server, which is running on 1680.10. Every time you create a new instance, you create a new upstream snippet like this. The engine X ones will be fairly similar. The syntax is different, but the process would be the same. Okay, I've made a few little notes here on what's actually involved of if you're doing an engine X equivalent of this. It requires making a standardized configuration file. I'm going to show you my Apache one shortly. We need to make sure that we have a standard Apache one shortly. We need to test it and review it pretty heavily, make sure that it's good, but I think I've done so many of them manually now that I'm pretty much sure how to do that. We would need to modify DHS to create instance because one of the things DHS to create instance does is it creates that little proxy snippet. So what we would have to do is to make it a little bit smarter. And before it creates the snippet for the proxy, we just need to check what kind of proxy is it dealing with. If it's dealing with an Apache proxy, create one like it just did. If it's dealing with an engine X proxy, just make the location block using engine X syntax. The other thing that would be slightly different, but really very minor instead of installing the Apache plugin for your monitor, you'd install the engine X plugin. I think it's really very easy to do it. It's not rocket science and probably take a couple of hours to get it all down patch. I've been saying this for months, but I still haven't done it. Unfortunately doing too many other things, but I'm running out of excuses. So yeah, we'll get this engine X proxy up and running shortly. Okay, before we do that, let's have a look at what's in the Apache proxy. I don't want to discourage people from using yet proxy because it actually works very well. Once you get used to the Apache syntax, it's actually quite powerful. There's very little that you can't do with it. I prefer it in many ways to the engine X configuration syntax. And the other thing is it's fully open source. It's not like engine X, where you have a free version and a pay version. Mind you, the difference between the free version and the pay version is a little bit less than it used to be even. So I think the status information that you're getting from the free engine X is a little bit limited compared to what you get on the free version. Apache is kind of fully free, so you just get everything. Okay, let's just see, please. Those are the three files that are in your... Hitting the wrong buttons. I need more coffee. This is the file that's sitting inside site enabled. Apache minus DHS 2.0, what you can see in there is that that file, it's actually, it's not a real file at all. It's actually a symbolic link to the file in sites available for Apache minus DHS 2.0. So the real file is sitting in sites available. We can edit it or view it from either place. Let's go have a quick run through what's inside it so that you can clear away any mysteries. Just check how we're doing on time before I do that. Five to 10. Okay, this is the last thing we do before we take a little break. Maybe we won't even do all of it. Going up to the top of the file is what it looks like.  It's kind of basic security stuff. You're basically trying to hide the fact that what kind of, what version of Apache you're using or operating system it's running on things like that. SSL stapling cache. This is a slightly weird feature of SSL that's supported by some browsers, not all. I'm not gonna go into the detail of it here. I'd recommend you Google it, read on the web page what it does, got it in there because the CLS benchmark recommends that it's there. I don't think it actually does very much. And I think if I'm not wrong Google Chrome kind of avoid it, ignores it anyway. Okay, one of the things that we wanna do is we want to, first of all, we enable the rewrite engine. You'll see quite a few places in this file we have to make use of something in Apache called mod rewrite, mod rewrite is probably one of the most powerful features in Apache. Allows you to basically rewrite URLs and here's a good example of one here. If we get some ancient browsers trying to connect to us using HTTP protocol version 1.0 or that we simply rewrite the request and make sure that, I don't know, we don't rewrite the request. What we do, we sense correctly, it's gonna look at the request and say this means if the request is not HTTP 1.1 then the rewrite rule that gets fired effectively is F, which is forbidden. So you could probably try this with Curl, right? If you try to get to this website using HTTP version 1.0 you just get a forbidden, right? It won't let you go any further. It's HTTP 1.1 or nothing. This, we can ignore for a moment, that's just something that enables the status page so that the monitor can pick up some stats. The virtual host that's running on port 80 is HTTP. The most important thing that this thing does is it just takes everything unless they are attempt by let's encrypt to get to us, takes everything and redirects it to HTTPS. This business, all of this stuff here and this thing here, I can actually get rid of I don't use it anymore anyway, we just get our let's encrypt certificates using a certain standalone but the important rewrite rule is this one. So basically all requests coming in on port 80 effectively we're gonna redirect to HTTPS. So this is where the main action starts here with our virtual host 443. Many different ways of setting up virtual hosts you can use name-based, IP-based ways of breaking them down. This is kind of our fairly simple case that we have. Server name, server admin, and self-explanatory. One of the things that is quite useful to hear I'm checking, I'm checking that the browser has accessed this website using my fully quantified domain name. If they've managed to access it through any other way, for example, by just typing in the IP address of the host, then I'm just gonna forbid them again, right, I'm just gonna throw them out. There's a good reason for doing that, particularly you want to block people accessing your website using the IP. Taking with cloud service, if you go with a big cloud provider like AWS or LinNode or Google or Digital Ocean, whatever it might be, those IP addresses that are owned by those big providers, those are all well known. I mean, the public information you can find out which blocks of IP addresses are used by LinNode, for example. And so one of the things that potential hackers are gonna be doing all the time, and they do it all the time, is to be perpetually trawling, going through those IP addresses looking for vulnerable service, right? So they might not know the DNS names of everything, but they know all the IP addresses and so they can just scan them. So it's a good idea for your server to respond to anybody trying to access your website, it's IP address by just throwing it out. You'll find quite a lot of websites will do that. If I just try our one, you'll see. And so it's Li417, if I were with using this LinNode assigned URL, it's hard to remember it. I have it here. So this is my server here. I'm getting forbidden for some reason, I don't know why that is. And I've done something. I go back and see what I've done on that. I think I was playing around with the blocking rules. Yeah, I blocked that one. Okay, so here I can access my website legitimately, right? I'm using its full name. If I try to access this website, I'm using its IP address, I would just get thrown out. I don't know it's IP address off the top of my head. Quickly look it up. What's the IP address? It's IP address, it's this one. Yeah, if I try to go to my website, so imagine I'm some kind of malicious actor, I've got a list of all the IP addresses that are owned by LinNode and I'm now going to check, what is everybody running on all of their IP addresses? See if I can find something vulnerable. So if I try to go to that IP address. Okay, it's throwing me out already because it's throwing me to HTTPS, but even if I accept that, if I proceed to it, it's gonna forbid me from getting in anyway. So again, that's a useful thing to do. You should be the same regardless of whether you're running Apache Engine X or any other kind of web server, if that matter. There is a legitimate way that we want people to be able to access us, so that's via HTTPS and it's using the fully qualified domain name. It's trying to see in by any other way. And we're not interested, we just keep them out. Stuff around, keep alive. And here's a whole number of parameters that's kind of built up over the years. The request timeout here. By default, we got this set as 300 seconds, which is kind of long, right? That's five minutes. Most security benchmarks will tell you that you don't want to have that more than 10 seconds, right? Otherwise, you're going to be vulnerable to denial of service attacks, because you might have. If you allow many clients to connect and remain connected for a long time, you very soon saturate all the available connections that you have. Unfortunately, we don't have too much time. We've got quite a lot of requests which often take more than 10 seconds. So, I've kind of set this timeout as 300. I'd like to bring it down more. I think as we make continued improvements in performance, I hope that we can eventually bring that lower. This is a little attack you can read about at the URL there. It's basically, I recall correctly, this slow wars attack. I think it's about a client making a connection and then just not proceeding with it, just leaving the connection open. And if you have clients doing enough of that, again, it can cause a bit of trouble with your salary. This is a useful header to set. Basically, you're telling search engines not to index and not to follow links to you. Okay, hiding in the days of the internet is not necessarily an effective strategy, but it's not a bad thing either. Don't talk about security by obscurity as a kind of a bit of a joke, but sometimes security by obscurity has some benefits. If it's not to help you against a determined attacker, but it can keep you out of the firing line of the many, many thousands of casual attackers who are just scanning, looking for targets. I mean, if we want to find targets, they should do this and it's sometimes a bit scary. Looking for DHS servers around the world to try and find one to attack is a nice string to look for there, right? If you go to Google, let's go to Google. And so go find me. I don't want to show you how to do this really, how to find servers to attack. So mainly trying to show you, there's a string. Let's go search for that string. And we find loads of them. Ooh, here's the HMIS in Rwanda. Coming up, top search string. Bamsa, I don't know if you want to call, need to fix that. That's because you're running an NGINX server there. You're not running my Apache 2 server, right? And the NGINX server is not setting that header. Some crowd in Uganda. Same BHIS in Tanzania, Tuzo, Beto, you're running NGINX on that one as well. Ah, the DHS2 play server, right? So yeah, basically that string, that string has been indexed, right? On Google, the search engine. And so if you search for it, all these sites appear. What I'm doing there is making sure that you're going to find all of these sites before you find mine, right? I kind of prefer it that way. I'd much rather all of these things get attacked before you attack my thing. So trying to keep yourself out of the way of search engines, it's not necessarily a bad thing. Only people that really you need to have accessing your health information system is meant to be your health workers and partners. That's what that does. I'm only going to do a few more lines of this and then we take a quick break. This one is actually very important. That's to do with offsite request forgery. There are some places in DHS2 code. Remember some of this code is quite old, right? I think the first edition of DHS2 appeared, went live around about 2007, maybe earlier than that, I think the state of Kerala in India. Some of the codes been around then for, yeah, that's 13, 14 years coming up for 15 years. There are some vulnerabilities which now we've got a security group within the DHS2 community that work hard every day trying to knock out vulnerabilities and make it more and more secure. But one of the issues we have is around cross-site request forgery. Cross-site request forgery is the kind of attack, it's a social engineering attack. You've probably heard about these things we've seen. People send you dodgy emails and I ask you to click on something, right? And then you click on the thing, thinking that it's going to take you to a picture of bicycles or something like that. But in fact, what it's going to do, is it's going to make an API request to DHS2. And if you happen to be logged in to DHS2 at the time on your browser, then the cookie will be used and will you just innocently clicking on something in an email, in fact, what you've done is you've triggered some attack on your server. One mitigation we can do against that attack is that we can set the same sites same site flag, which your browser, if your browser sees that, then basically it won't follow links to your site unless it's come from the same site that it's got it from. The way it does that is it, wherever it sees set cookie, so this is where your session cookie gets set, it gets set, it's just going to add to that. It's basically a rewrite, follow one means take same line of set cookie and add the end of it, same site, it goes straight. Again, you can take a look at that on, you can look that up, same site on Wikipedia or somewhere, they give you a good explanation what it does. STS, this is quite important to set. Basically, this is telling your browser, if it's seen this site before as HTTPS, then if you try to go to it subsequently using HTTP, then your browser should just refuse. Say no, this site is an SSL site. Again, this is to protect against kind of mad in the middle attacks. And this is a basic tracking production, your X-ray model. That was a few basic security settings which have been built up over the years. It's not completely adequate. There's a few more things that probably we should add. It's something that needs to be reviewed fairly. I can review it probably every six months, which is not often enough just to make sure these settings are up-to-date. And again, if you have a server that's running in the field, you can't just leave it there forever and assume that it's gonna be, its configuration is fine indefinitely, right?  Again, this is a great advantage of having a reverse proxy there.  Proxies have a whole lot of features built in which allow you, as you can see here, to set custom headers, to modify headers, to do things with the content. This is not Apache 2 specific. You can do exactly the same stuff using Nginx. One thing Nginx is not so good at is editing cookies. You've gotta do a bit of a hack with Nginx to do that, but anyway, you can achieve the same effect. Inlets, I think that's, let's analyze this. I think let's take a break for a couple of minutes. Maybe you guys can accumulate a couple of questions from me if you have on the content. And over time, some updates get made to the app, right, things get improved. They could be related to security, they could be related to performance, could be related to any kind of thing. And the trouble that you have is all the old ones are still out there. And if the reason for doing the updates were related to security or related to performance, then it's quite important that you get rid of those old ones and make sure the user's upgrade. Right, if you're using mobile device management, you could do that automatically and that's really the proper way to do it. But sometimes it's just too expensive. Hasn't been properly budgeted for, it's not practical. And so again, we come back to the poor old proxy as the weapon of last resort, right, to deal with the problem. And I got a simple way of dealing with this one is to say, well, if you know that certain versions of an app are undesirable for one reason or another, we can just simply recognize them and block them. And fortunately, after some discussion I had with Android team year or two back, Android team are very diligent about identifying the version of the app that they're using in what's called the user agent header in the request. I think I've listed a few out here. This is a old list that was sent to me by Marta year or two back. This is the way that Android app identifies itself as a user agent. All right, this is a string. You'll see this in the log file. You'll see the user agent strings that are being used. Old versions used to use a string that started with HHS2 like that slash version, oops, version 1.1, et cetera. And over time that string has changed. More recently it's the current strings that we look at look like com.phs2 slash. And then the version of the app here. This is quite an old list. I think the latest version of the app is 2.3.1. I just saw that on the website. Any of the Android team are on here, they can correct me. So what you often find, and I was actually having a look, looking at Gala just yesterday. A lot of Android apps that were deployed years ago are still out there and they're still trying to access the DHS2 server. And all kinds of good reasons, we might not want that, right? We want to make sure those old apps are effectively banned. And yeah, this is the kind of thing that you reverse proxies to be very good at. Is examining requests and deciding whether to route them forward or not. So imagine in this scenario that we have here, we want to make sure that anybody who's using the app version two or above, right? We'll accept anything that's 2.1 or 2.0 or one of these old strings here, we want to effectively ban. We can do that on our proxy, quite simply like. And using mod rewrite is the neatest way to do it. There's other ways to do it in fact, but this is the way I do it. The way mod rewrite works is you lay out a list of conditions, rewrite condition, rewrite condition. Right at the end of that list of conditions, you have a rewrite rule that say, well, if any of these conditions match, then you trigger the rewrite rule. Rewrite rule is similar to previous one. Effectively the F is what's important here. It's not important. We're not actually rewriting the URL in this case. We're simply sending an F for forbidden. Let's have any of those conditions match with forbidden request. And you can see this is, these are effectively regular expression matches. Anything that starts with hs2 slash zero, it will match all those early strings, should be forbidden. Anything that's com.hs2 slash one or com.hs2 slash zero or com.hs2 slash two point zero. That one's not right. I need to escape the two point zero to the two point one. Made a mistake in my regular expression there. Fix that on the slide. Effectively, you set the rules here and if anything matches any one of those rules, then you simply forbidden to go further. You can test that quickly. I've set up this very rules on um, HMIS instance on the node. Do I have them in my history? Let me just check. I don't want to copy them backwards and forwards off the slide. I just want to show you these things in action. Before I do that, let me see where I've done it. Got one upstream. Here's my HMIS snippet. You see inside the snippet, I've added stuff. All right. It's because it's a regular expression that has a special meaning. I can't just put it to play. Let's just fix it. And so any user agent that looks, that matches any one of those streams, we won't let them in. Just reload. Let's test it. Easiest way to test it is going to be to problem going in full screen mode. We get after full screen mode. That's what's causing me trouble. Okay. So, if I, this is first of all, check the website is working at all. So copy that stream. I'm just testing with curl here. I'll go back to my laptop. Do it from here. Oh yeah, I was doing them here the other day. I was doing them, I was testing this last night anyway. So let's just check, we can access our sites. Oh, we're getting forbidden. We don't want to get forbidden. Why are we forbidding? Don't have permission to access this resource. What did I change from what I did last night? This is the only things that are for thing. It was the last one. This one is not an all. Or, or, or, and that's the last one. I'm sure that's the one I did there. It was another string I had, which let's just test again. That works if it's just not following. So if I access the website, it'll work and that's the login page that's getting returned there, right? So I can, I can access the website normally. What if I was an old Android? I need minus L on it. What if I was an old Android device? Let's get the history up here. Let's see, take this one. Now, one of the things you can do with KEL is you can, you can pretend you're any kind of user agent that you want. So this is the user agent string. That header would be set. Okay, so I don't know at what point this thing fell apart. Basically, if I use KEL, can access this to my application there. What I've done with those rewrite rules is we've blocked, if I try to access it like, this is an old Android, this is what an old Android app would look like. I had a very old one, and I was like, I said that before, that's actually a new one. These are the old ones here. So something like this, if I try to access it now using an old Android, ooh, let me in again. That should be blocking me. It's not blocking me.  Test everything last night. He's working fine. I fiddle with a few things. Then I break it again. Let's just check my rules here. Anything with DHS2 slash zero is gonna get blocked. And what have I tried? DHS2 slash one. In fact, anything with DHS2 we can block. We don't need to say anything that's got a user agent that begins with DHS2. That's an old style Android. So we'll just block it like that. And I've quite, when you test these rules quite carefully or you inadvertently end up blocking things that you don't want to block. That's also something that we've seen quite recently. Try that again. If I try from using any user agent that begins with DHS2, it won't let me in. That's what we want. Similarly, any user agent that looks like com.dhs2 slash let's have a look at what those strings look like. My previous slide. So, this old one here, 1.1.1. If we've got anything with com.dhs<IP_ADDRESS>, we wanna block that. We basically only want to allow through any requests that has 2.2 or above. 2.2 is fine. Hopefully 2.3 is fine. Yeah. So we're blocking anything that's below that. So that's a very useful thing to be able to do because otherwise it's actually a real problem. Right? There's a lot of projects. You know, they they deploy loads of thousands of Android apps, but they've no real long-term plan or what they're gonna do about it as these apps start to get old. Right? As the world changes. And yeah, your proxy is a very useful tool for doing that. The rules I've got to do when you've, I've modified the rules slightly from what's on this slide. Let's just go back and make sure that it's fixed up the slide if there's a mistake on it. Yeah, there is. It's because I had an additional one I was blocking. So it's all, all of them except for the last one and these dots, these dots need to be escaped just to make sure you have it correct. So the syntax isn't hard to understand. It is, you don't have to make loads of these things all together like this. I mean, if you want to be really smart, you could create much smarter regular expressions that we could probably create a single regular expression to match all of the ones that you want to block. Sometimes, even though it's slightly less efficient to do it like this, it's a bit easier to maintain. You can see them all broken down one line at a time. But there are many cases where you might want to block certain things. There might be particular, there was another case actually quite recently, a big aggregate server which also had a bit of events on it and they'd upgraded prematurely to version 235. And they found that the very few people who were using the events, but they were actually bringing down the server. And the only way to keep it up short term while developers were fixing it was to simply block all access to API events. And they kept that in place for a week and that kept the rest of the server up and running while the events API stuff got fixed. So lots of reasons to block. This kind of Android cases is a pretty interesting one because I think it's probably going to be quite common where deployments are made and those deployments eventually get old and you need to have ways of dealing with them. This is using mod rewrite on Apache. I really love mod rewrite. As I say, the syntax is sometimes a little bit odd, but it's really very, very powerful. You can do lots of things with it. If you want a little tutorial on how to use mod rewrite, link there, it's quite useful. You can of course do the same thing. This is some of the tests that we've done. This is a good example of what I was referring to earlier as virtual patching. We're basically using our proxy to compensate for issues people didn't think about. You can do the same thing on engine X. They're kind of blocking rules, would look something a bit more similar to this. We'll get to full screen mode again. I think Jaime contributed this. This is basically a kind of list of ifs. I'm very similar to my rewrite condition statements in Apache. If you use region, look like that. Again, this is a regular expression. Return 403, if it looks like that, return 403, et cetera. You can see how by using different variables and different combinations, you can put together quite a sophisticated set of blocking rules. Okay, so that's one useful bit of skill you need to have in your toolbox. If you have your proxy server, which you will have, whether it's Apache or engine X doesn't really matter, you often find that the ability to be able to block certain requests on the base of user agent or the request string itself or something else is useful. Okay, the other thing is rate limiting. I need to talk a little bit about rate limiting. Fact is, whether we like it or not, all applications, whether it's DHS2 or anything else, then they will have limits. How many parallel connections that are actually possible to be serviced and at what rate, right? How many of those connections per second? DHS2, this is gonna be primarily determined by how much RAM and how much CPU are available to process those connections. And so if you have a very, very busy server which is coming under substantial load, you can keep increasing your RAM, keep increasing your CPU, whichever one is the one that makes the difference until you reach a point where you can't keep increasing it anymore, right? And then you need to do something more drastic to stop becoming completely overwhelmed. I said this really doesn't matter, even if you're load balancing, the same principle applies, right? You can keep adding additional Tomcat workers, but there's a point of which it becomes uneconomical, right? Again, I think this is particularly the case with Android or mobile devices in general. One of the things that it's done is it's substantially increased the potential number of clients that you have out there, right? I mean, traditional DHS2 deployments, you had people entering aggregate data at the district level, right? Very, very small number of users, effectively.  Again, traffic not huge, but once you have tracker-based systems and you have them on mobile devices, then the potential rates of requests that you might be getting start to become huge. And sometimes this can be because of badly written apps, misbehaving, sending excessive requests that they don't need to. And sometimes it's simply because you've got a lot of devices out there. So regardless of what your server is running, how much RAM, CPU, how much load balancing you're doing, you're gonna reach a point eventually where you have to protect the server somehow from getting overwhelmed and going down for everybody. And yeah, there are some DHS2 API requests which historically have had particular performance overhead problems. There are some that still do. A lot of you on this session will be familiar with some of them, right? Some analytics query requests can be a little bit crazy in terms of their resource demands, posting events, tracked entity instances, things like that are sometimes very heavy. We saw yesterday and last week, how tools like Glowroot, in fact, are really, really useful for identifying some of these problematic requests that come up. But it's one thing to be able to identify them. There's another thing then to get developers to focus on fixing them as best they can. And then there's another thing again to actually try and do something about it.  So it can be important to identify some requests which are causing trouble. In the most extreme cases, you just block them, right?  In some cases, you can be a little bit more subtle than simply blocking them and just try to slow them down somehow. And that's really what rate limiting is about. Fortunately, I think the situation around performance in DHS-2 is starting to take some pretty drastic leaps forward. And the last few weeks have been particularly exciting to see. I think we can look forward to the release of 234 and 235. 235, maybe due out today, the XPatch release, actually have some quite substantial performance improvements. That is not quite significantly affected because people testing back in Oslo are actually using Glowroot to identify and measure some of the trouble spots and start to knock them on the head. So yeah, things are getting better regarding performance and I think we can expect over the coming weeks and months, things will continue to get even more better. But yeah, some of the optimizations that we see in 235 and 234 are probably the most optimized that we've seen in any recent DHS-2 releases. Good to look forward to that. But remember, no matter how good it gets, there's always gonna be outliers and there's always gonna be a problem that you potentially have got a million devices out there hammering on your server. So there's always finite limits. So how do we stop the server being brought down by friendly fire, if you like? I mean, this is not a distributed denial of service attack by some hostile actors. Is it just normal users, whether it's just too many of them and making too many requests which are heavy? Well, first we use the proxy. We can use the proxy to restrict the rate at which requests come by. We can use the proxy to restrict the degree of concurrency of those requests. We can restrict the bandwidth that particular API points have got, et cetera. And here's where we start to see differences.  It'll work to keep your server up if it just keeps dying but it's a bit of a blunt instrument. You can effectively rate limit your application but it's not very targeted at that. It's not a very tunable mechanism for rate limiting. It is possible to do more sophisticated rate limiting using Apache 2 together with modules like Mod Security but configuration overhead and doing that is a little bit unreasonable. EngineX does its better. It's got two very nice mechanisms. I'll say very nice, two mechanisms which are much better than the Apache ones anyway. For rate limiting and connection limiting the directives are called limits, underscore, rec, limit, underscore, on. You can read the official documentation on them there. I found them actually very useful. It was my main motivation in a couple of places to remove the Apache proxy and put in place an EngineX proxy simply because of the rate limiting. Everything else about Apache I preferred. But better than Apache 2 but it's still not ideal. One of the problems with EngineX rate limiting, it's kind of limitation I found is that you can limit on the basis of a URL. So if you've got a problem with API events, for example, you can make sure you've got no more than 10 parallel connections to API events and you can make sure you get no more than 20 requests per minute on those connections. But you can't discriminate between a get and a post. That's a really weird limit. I don't understand why it is like that but I Googled and Googled and asked questions on mailing lists and it seems that you just can't. You can put a limit on a URL but you can't discriminate between whether you're limiting a get or limiting a post. And that's quite an important limit because often the gets might be okay. The problem we have isn't with the gets but the problems are with the posts. And so unfortunately at the moment, EngineX also proves to be a little bit of a blunt instrument in this regard because we can only limit the URL. I've started looking around at other small projects which are particularly related to rate limiting. And interestingly enough, the GoLang community, Go programming language, there's quite a lot of libraries and little projects that very specifically aimed at rate limiting. And I think probably we're gonna find something there that's useful and one of the most useful looking libraries I came across this thing called toll booth. Any programmers out there who are familiar with Go want to learn Go, a great opportunity here. I think what we probably need to do is to make a little rate limiting program. We can stick that in between the proxy and the Tomcat, maybe installed it on the Tomcat server and make this very DHS2 specific, right? So that we can very finely tune exactly which requests and which request methods, maybe even which user agents and configure them with tunable limits. It's either that or it has to become part of the DHS2 core count that API points have gotten rate limiting functionality built into them. So that's where we are currently with rate limiting. At the moment, without further investigation, the best you can do if you need to rate limit is probably use an engine X server. I've got a little example, I think on the next slide. How are you set about doing that? Syntax, yeah, it's not too complex. Basically, all most of these rate limiting modules and algorithms make use of something called a leaky buckets, some variation even on the leaky bucket algorithm. People who are familiar with computer science would have come across the leaky bucket before, particularly if you've done computer networks. Principle of a leaky bucket is pretty much like what it sounds, imagine you've got a big bucket with a couple of holes in the bottom, and if you use the same principle when you're watering your plants with a watering can, right? You've got a bucket load of water that you want to put onto your peas, which are just sprouting now in the spring. You don't want to throw a big bucket of water straight onto your peas because the weight of the water is just gonna probably damage the plants. But if you can pour a whole bucket onto peas, if you run them through a watering can, so the water is restricted in its flow when it comes out in a little fine spray through the holes in the bottom. And no matter how much water you put into the bucket, the rate at which it comes out is determined by the little holes in the bottom. A leaky bucket algorithm works pretty much the same with networks as well. You have a bucket, all your network requests come in, come into the bucket and then they leak out at a controlled rate out the bottom. That way your server doesn't get overwhelmed, much like your pea plants. The trouble happens when your bucket gets full, right? Once your bucket is full of water, you can't accept any more bucket, any more water, any more water, any more network requests. And the size of your bucket is determined if I look here at engine X configuration here, this is the size, this is 10 megabytes, 10 megabytes, megabytes, these zones which are defined on the top of the config file, but they essentially determine the size of your buckets, right, and they have a name. And we've got three different types of zones defined here. First one is the connection zone. This is where we got to use this with limit con status with limit con down below to restrict the amount of concurrency, right? So we create a zone and we create the zone and this is for the whole server, right? So this means basically for the entire server we can control how much concurrency is on it. And on here, if you have my mouse is visible, on here we say limit connection using the zone called DVS con 75. What this lion is doing is saying at any moment in time we can allow 75 parallel connections to this server.  To get a 77th connection, it gets put into the bucket until my bucket overflows. When my bucket overflows, I'm gonna return an error code to the clients and the best error code return is this thing called 429. 429 is HTTP code, which basically says that your request has been rate limited. That's useful information to return maybe at some stage and if you're an Android app developer, for example, if you get this response, then you could display something to the user which is different to just saying there's an error on the server, right? If you're saying that then what you're saying instead to the user is everything is fine. Don't panic, go and have a cup of tea. It just happens to be that you're trying to access this server at the same time as 500,000 other people. And so you've got yourself rate limited, right? That's what a 429 means. So DVScon actually gives you a way of controlling the number of parallel connections. Limit request, limit request is used to limit the request rate, right? And you can do that. You can configure zones differently. There's two types of zones that you see here. One is determined by the IP, the binary remote address, right? That's the IP of the person connecting to you or the vice connecting to you. So what this zone is saying is that for any particular IP address, we're gonna allow them 20 requests per minute. Any more than 20 requests per minute that come in. So if I get a 21st request within the same minute it gets put into the buckets. You can also have a request rate for the entire server. Okay, these two are actually not very sensible values because you probably have a higher request rate for based on the whole server than you want to have for particular IP addresses. But then within a particular location block, like here we're saying, for example, if a value sets, we can apply those zones. And effectively control the concurrency. As you can see from the syntax here, the limitation is we can't say, can you apply those limits for only get data value sets or can you apply those limits only for post data value sets? As far as I can figure out, it simply cannot be done. And that's what's the various trawling of mailing lists and things seems to say. There's possibly some hacking you can do with maps that might make it work. But as I say, I think we actually need to make a little goal program to do this rate limiting more flexibly more DHIS to applicable. Either way, it's an important, potentially important functionality to have to protect your server against being overwhelmed. And as I say, currently the way NGINX does this is quite a bit more flexible than you can do out of the box with Apache. Okay.


So last night we saw the red mirage play out the scenario where Donald Trump would appear to be in the lead But then as mail-in ballots come in and they are counted that lead diminishes and a couple of weeks ago Bernie Sanders appeared on the tonight show with Jimmy Fallon and he predicted all this would happen with Pinpoint accuracy so I can't play the video clip for you because there's probably gonna be some copyright issues But here's the audio of Bernie Sanders basically saying Exactly what happened at the election is November 3rd, and it's been said that we won't know the results Until days later. When do you think we'll know the results? All right, Jimmy you raise an important point, and I hope the American people understand it because this is something I worry about My view is every vote must be counted For reasons which I don't have the time to get into tonight You're gonna have a situation. I suspect in states like Pennsylvania Michigan, Wisconsin other states Where they are going to be receiving huge amounts of mail-in ballots on unlike states like Florida or Vermont They're not being able For bad reasons to begin processing those ballots until I don't know election day Or maybe when the polls close that means you're gonna have states dealing with perhaps Millions of mail-in ballots here is my worry What polls show and what studies have shown is that for whatever reason? Democrats are more likely to use mail-in ballots Republicans are more likely to walk into polling booths on election day It is likely that the first votes that will be counted will be those people who came in on election day Which will be Republican and here is the fear and I hope everybody hears that it could well be And you know, I don't know what's gonna happen. Nobody does but it could well be that at 10 o'clock on election night Trump is winning in Michigan. He's winning in Pennsylvania He's winning in Wisconsin and it gets on the television. He says, thank you Americans for re-electing me. It's all over. Have a good day But then the next day and the day following all of those mail-in ballots are getting counted And it turns out that Biden has won those states at which point Trump says see I told you the whole thing was fraudulent I told you those mail-in ballots were crooked and I got you know, we're not gonna leave office So that is a worry that I and a lot of people have so the results, you know, we don't know what's happened Maybe the results will be when our election day So Bernie Sanders was accurate down to a T and I'm not showing you this video To toot Bernie Sanders horn and make it seem as if you know, he's some psychic because really the scenario that we all expected even though it was kind of a More worse case scenario it still played out exactly as we thought it would Trump appeared to be in the lead He declared victory and then as the mail-in ballots come in He's losing that lead because Democrats for whatever reason as Bernie Sanders pointed out are more likely to vote by mail Republicans not so much. That's more explainable Because Republicans have been told for months now by Donald Trump that vote by mail leads to fraud So of course, they're gonna show up to vote in person But Democrats not necessarily believing Donald Trump probably worrying about the pandemic not wanting to vote in person They voted by mail. So we're seeing everything play out Exactly as we expected, but I'm not showing you this to tell you how right Bernie Sanders was I'm playing this for you because it's really important that we all get this on the record We show Donald Trump supporters right now who probably won't believe us But maybe some will that this is not fraud. This is exactly what we expected This is not a nefarious ploy by Democrats to steal this election using mail-in ballots That just inexplicably appear out of nowhere if down if Joe Biden is down if that were the case Wouldn't it be happening in Florida? Wouldn't it be happen happening in states that he needed to win but didn't So we have to show them that this is something that we expected based on data not feelings based on what Voting habits are Democrats more likely to vote by mail Republicans not now this varies depending on the state depending on the demographic But by and large Democrats were more likely to vote by mail So Donald Trump was going to use this to his advantage to delegitimize the results of this election and knowing that Democrats more so would vote by mail. We saw what he did throughout the summer his postmaster general Louis DeJoy crippled the US Postal Service and When they were supposed to get at least 97% of ballots delivered on Tuesday They only hit 93% and now we're learning that 300,000 mail-in ballots are missing So if there's any attempt to steal this election going on, it's not Joe Biden and the Democrats It's Donald Trump who is using the institutional power that he has as an incumbent To do everything to not only delegitimize the process But use our government in his favor to win so it's important that we show people that this is predicted It's not some sudden plan to steal the selection away from Donald Trump if Donald Trump won legitimately I have no doubt that Democrats and Democratic Party voters would accept the results But because Donald Trump is a baby because there are reports that he literally threw a temper tantrum as he saw the results come in Well, he is going to bring the entire country down with him.  It's gonna be a bumpy ride folks


Consider this the beginning of the end like now with with reports like this that are coming out with the Ravens put in what they Put on their website a couple days ago about the whole Chuck Clark and be possibly being traded Consider this the beginning of the end and I don't say that to disrespect Chuck Clark at all But when stuff keeps coming out like this the player about 9.9 times out of 10 never stays with the team They don't it just doesn't happen like that not hey if Chuck Clark stays with the Ravens great But me personally my expectation is that literally any day now any day now Chuck Clark will be traded. Let's read this report though because this report is like First it starts off the Ravens and Eagles discuss the trade for safety Chuck Clark during the draft According to Adam Kaplan, okay, so, you know the Eagles they wanted Marcus Williams But the Ravens outbid them for Marcus Williams. Okay, look at Ravens outbidding somebody outbid him for somebody That's young and entering their prime check the Ravens out. But anyway So that part was cool that part. I was like, okay, so Eagles are trying to get Chuck Clark Okay, go to a new situation. All right, cool but the back part was Let's just read it Raven showed interest in wide receiver Jalen Rager, but the talks died Man, and I mean, I was just talking to somebody about this the other day because they hit me up on Instagram and I was just like man, um They were like, what do you think about Jalen Rager? I was like No, I just No, no, cuz they were like the Eagles just got AJ Brown. They already got Devontae Smith I mean, if you want to give us Devontae Smith ain't no problem. Hey, he's no problem, baby. I'm with that all day but And my guy was like, you like what? But what about a change of scenery a change of scenery could do wonders for him. Yeah, it could it could it really could But I don't know. Oh, just feel a little scared about that one. Just a little bit Um, so yeah, that that was that so hey Sooner or later, sooner or later Chuck Clark is gonna be traded Jalen Rager, I think what really like because it got to the point, you know, how um For some players, you know, man that player he dropped the ball. Oh, man. Oh, man Oh the hell but I remember watching I think it was last it wasn't the Eagles versus it was the Eagles versus Bucks game in the Player. Oh my gosh, man. I felt so bad for him for Jalen Ray I felt so bad because there were drops and it just it just looked like he was like Just out of it like his mind was like completely removed from the game And I just felt so bad for him as a player because and you just kept watching Over and over and over throughout that playoff game and it was like, oh, man This this is rough. It is rough and I felt like man I felt like he was gonna be forced to have a change of scenery after that game Because I thought at that point the Eagles might have been done with him Because it was it was so sad to see too man Because it was not and again it wasn't obviously I know it ain't coming from a place place from hate on my end gonna be hating on nobody but it just It's like you're watching and it's like wow He is just he's really going through it right now in a bad way So and not that that one playoff game defines him because I don't feel like one playoff game defines a player But just in that moment. It was it was just rough man. It was rough So hey, maybe a change of scenery would help him out just give him a fresh start, but I would probably Rather Ravens go to some other option. I mean, but I mean you never know man if he did go to the Ravens He would be the deep threat for the Ravens He would be there the speedy wide receiver and Man, I'm just in the space is talking about this yesterday Because I can understand both people but both sides argument When it comes to the Ravens and wide receivers, I know we've been talking about why receivers for so long but with I understand both sides and the reason I say that is because I understand one side because I was part of one side before To where it's like man. I don't want the Ravens to sign nobody We should just run with our guys all guys should show what we got our guys should be able to show themselves Because it's based off of all guys got potential. They should show their potential. We should show we've been drafting all these receivers They need to show what they got And it's like okay. Yeah, cuz I was I was like that too. I was like a Ravens Just they keep getting all these guys for a reason, right? Let them play let the young guys play That's why Ravens never develop wire receivers because they never play him But then there's another side Where it's like, hey, we should still go out and trade for a significant wire receiver somebody who's proven somebody Who's been that guy with a trade sign whichever we should go out and get more make Lamar's job easier And get and we should always be willing to upgrade the quality of the wire receivers Because that's usually something that the Ravens are kind of always like so-so about They'll upgrade the quality of the cornerback to upgrade the quality of line backs They upgrade the quality of defensive line upgrade quality offensive line upgrade quality of this dead in the third But with wire receivers, they usually like So my thing okay, hey and the timing the timing is what makes this thing so tricky is because Lamar is never ever ever going to be cheaper than he is right now And if you really are in win now mode, yes, like yeah, you do got some young guys a wide receiver And hey, they got some potential too, but it never hurts to upgrade the quality of your wire receivers Why not have more the more the Mary like a Indefensible offense But anyway with this deal not going through I'm not mad I'm not mad Another report says that the Ravens actually they approach the Eagles about so the Ravens were the ones that reached out I didn't realize that the Ravens were the ones that reached out to the Eagles about trading Chuck Clark So though that that means the Ravens are shopping Chuck Clark. Chuck Clark is not gonna be here He's not He's not and I know there's a lot of Ravens fans that are on offense about and I again I can understand both sides of that so people like a we keep Chuck Clark We should keep Chuck Clark. We got nice depth Chuck Clark. He knows the defense. He's been calling the defense Da-da-da smart player. Da-da-da Chuck Clark been around a team good teammate leader. Da-da-da. I get that and depth Oh, I get that But and then there's other side. It's like, okay. Bye Chuck. See you Chuck wish you well in future endeavors Chuck But all these reports are not coming out for no reason Chuck Clark is going to be traded in my opinion. I feel it's just it's a matter of time now, man It's a matter time. I feel as sure as We're probably gonna be live streaming later tonight for the schedule release I'm as sure and that as Chuck Clark is in being traded. I Don't think I said that right My confidence in doing a live stream tonight for the the schedule release and the official one because yeah I done seen every leak. I don't see all these different leaks, but I'm not getting excited over no schedule leaks Because it's like, uh, okay, because I remember last year a schedule leaked and It It was I think like two games were off though But it wasn't a big deal. But this year. I'm just I'm like, I I'm just waiting I keep seeing a oh, yeah, this kid came though. It came out Okay, cool. I'll wait till it's official then we'll talk about it and then we'll get into it later tonight, but I'm my confidence in us streaming doing a live stream tonight for the schedule release It's the same amount of confidence I have in Chuck Clark being traded from the Baltimore Ravens and Both with both is just a matter of time in my opinion Which and and either what you know when you think about it, too, like we talked about it Ravens signed Marcus Williams, then they drafted Kyle Hamilton So Chuck Clark knows man Ravens got their safeties of the next five years They got their safeties for the next five years. Well at least four years At least four years and they could do the fifth year option so Ravens they got stuff locked up there and Chuck Clark after he got two years left on his deal So he knows that he's not the future as safety for the Ravens So it's like all right if I know I'm not the future might as well get a head start now, right? And I'm sure Chuck wouldn't mind staying but at the same time Mmm, and I don't know how I don't know his personal thoughts obviously I don't know Chuck Clark, but he could be thinking a couple different things He could be thinking hey, I don't mind staying showing the showing the young guy the ropes and whatnot I'm sure he would if he stayed But at the same time it's like oh, but this this guy I'm training him to take my job He got drafted to replace me So it's like okay. That's uh That'll do it So he could also feel like you know what they drafted him replace me. You know what Ravens? Let's just let's just move forward now. Let's just get it over with now Let's just let's just start the process now. So I can be on my way He could do his thing for y'all. Hey, we had a wonderful time. We had a wonderful time I appreciate y'all for everything. Thank you all for everything so much appreciate the opportunity. Love y'all Ravens I'll love you all forever But it's time to go flap my wings somewhere else now. Will that be with another bird like the Eagles? It ain't looking like it now not that this came out But this report it lets it be known that Chuck Clark is on the move So like Chuck Clark is gonna be eventually again in my opinion, but I have a really strong opinion about this I'm very confident in this But like Chuck Clark is probably gonna be very soon when it comes to being with the Baltimore Ravens. I'm out