The sheer scale of the new space race is beginning to look like a space revolution, much like the process of industrial revolution. There is an overwhelming interest in everything, asteroids, comets, planets, the moon, the sun, space dust, and even old debris. It is as if they are searching for something, but what is it they seek? Mike Pence has just announced that the Space Force will be mobilized by 2020 with world governments supporting this move and even being involved in collaborations and missions planned. It just sets off the sense that there is something more going on with all this activity. The Pentagon is now backing President Trump's plan to create a new military branch dedicated to fighting in space. Vice President Mike Pence and Defense Secretary Jim Mattis laid out a blueprint yesterday for the United States Space Force. David Martin is at the Pentagon with the latest. David, good morning. Good morning. Only Congress can create a new military service, and as yet there is no estimate for how much the Space Force would cost. But according to Vice President Pence, it could happen by 2020. Today, other nations are seeking to disrupt our space-based systems and challenge American supremacy in space as never before. Like the Army, Navy, Air Force, Marines and Coast Guard, the new Space Force would prepare for war. A war in space against countries like Russia and China. The U.S. military depends on some 140 satellites, including this quarter-billion-dollar GPS satellite, for everything from communications to targeting the enemy. But Air Force Secretary Heather Wilson says many of those satellites were built at a time when there was no threat of attack. We built the glass houses before the invention of stones. The threat to U.S. satellites has been increasing since 2007 when China used a ground-based anti-satellite weapon to shoot down one of its old weather satellites 530 miles above the Earth. General Jay Raymond showed 60 minutes the results. This is about 3,000 pieces of debris just from that one event. That came just from that one collision? Just from that one collision. Defense Secretary James Mattis was originally opposed to the Space Force idea. Writing in a 2017 letter, it would add layers of bureaucracy. But his position seems to be changing. We are in complete alignment with the president's concern about protecting our assets in space. As the commander-in-chief continues to push the idea. Space Force! People love it! Currently about 18,000 men and women in uniform involved in space operations. Although President Trump says the Space Force would be separate and equal to the other branches, it would be by far the smallest of the military services. Glad? Alright David Martin, thank you very much. Anyway, welcome to Space News. This week, among other things, we are going to tell you about the stunning revelations that have been made in regards to an unknown intelligence onboard the Space Shuttle. Wait, can you hear this? It might sound like a plot to a Hollywood science fiction blockbuster, but an astronaut claims to have seen an organic alien-like creature on the Space Shuttle. NASA engineer Leland Melvin made the astonishing claims during a discussion on social media. The former NFL football player says he spotted the creature in the payload bay of the Space Shuttle Atlantis, which was an orbit above the Earth at the time. When he told NASA about the sighting, space agency experts told Melvin the most likely explanation was ice had broken off away from the shuttle and it was the movement of these floating shards that caught his eye. You would think, however, that a trained astronaut in space would be able to tell the difference between ice and organic life. Would you not? The revelation was made during an exchange on social network Twitter held between Mr. Melvin54, who worked as an engineer aboard the Shuttle, and Scott C. Warring, who runs the site UFO Sightings Daily. Mr. Warring asks, what's your outlook about the existence of intelligent alien life living in our solar system? Have you ever witnessed a UFO? Mr. Melvin responded he had seen something translucent, curved, organic looking. When he was working with colleague Randy Bresnik, he joked, I was about to say Houston, we have a problem, but now everyone spins up when those words are uttered from a space vehicle. Now isn't it stunning that he was aware that radio operations on Earth could have intercepted such a transmission, and therefore he refrained from doing so at the time, but years later he is openly telling people that he saw something on board that should not have been there. Are former astronauts playing their part in alien disclosure? Are they being collectively told that now they can speak of their experiences in an effort to prepare us for an announcement that almost seems like it has happened already? Anyway, with all these incidents being reported by astronauts and government workers. According to Sky News, NASA are reportedly funding private space agencies in an apparent attempt to develop 10 high technologies to further space exploration. The investments come as the space agency partner with six U.S. companies on projects including deep space rockets, cryogenics and a lunar lander. NASA considers a technology at the tipping point if an investment in a ground or flight demonstration would result in significantly improving its design and helping the company bring the product to market. While these key technologies will support NASA's science and human exploration missions in the future, these awards are yet another example of NASA's commitment to our nation's growing commercial space industry today. Also leading to speculation suggesting this is an effort to develop technology for a much bigger overall venture into space. One which although we do not know what is going on, there is no doubt that there is something major unfolding before our very eyes on this planet. Also, if you guys are interested, the per se meteors are this weekend between the 11th and 13th. Around 60 fireballs should be visible per hour at the peak, so get out and have a look if you have clear skies. And while most of us were sleeping last night, NASA launched for the sun, literally. But how the hell is this mission going to work? Launching from Cape Canaveral Air Force Station in Florida, Parker Solar Probe will make his journey all the way to the sun's atmosphere, or corona, closer to the sun than any spacecraft in history. Nestle, atop a United Launch Alliance Delta IV Heavy, one of the world's most powerful rockets, with a third stage added, Parker Solar Probe will blast off towards the sun with a whopping 55 times more energy than is required to reach Mars. About the size of a small car, it weighs a mere 1,400 pounds. That's a relatively light spacecraft, and it needs to be because it takes an immense amount of energy to get to its final orbit around the sun. Zooming through space in a highly elliptical orbit, Parker Solar Probe will reach speeds of up to 430,000 miles per hour. Fast enough to get from Philadelphia to Washington DC in one second, setting the record for the fastest spacecraft in history. During its normal mission lifetime of just under 7 years, Parker Solar Probe will complete 24 orbits of the sun reaching within 3.8 million miles of the sun's surface at closest approach. Earth being 93 million miles away, so pretty significant. It will be going where no spacecraft has dared go before within the corona of a star. With each orbit, we'll be seeing new regions of the sun's atmosphere and learning things about stellar mechanics that have never been explored before. But getting so close to the sun requires slowing down, for which Parker will use the gravity of our neighbor planet Venus. Parker Solar Probe uses Venus to adjust its course and slow down in orbit to put the spacecraft on the best trajectory. It will fly by Venus 7 times throughout the mission, each time it flies by it will get closer and closer to the sun. In an orbit this close to the sun, the real challenge is to keep the spacecraft from burning up. NASA was planning to send a mission to the solar corona for decades, however they did not have the technology that could protect a spacecraft and its instruments from the heat. Recent advances in materials science gave us the material to fashion a heat shield in front of the spacecraft, not only to withstand the extreme heat of the sun, but to remain cool on the backside. Parker Solar Probe is also the first NASA mission to be named after a living individual, Dr. Eugene Parker. Famed solar physicist who in 1958 first predicted the existence of the solar wind, the stream of charged particles and magnetic fields that flow continuously from the sun, bathing Earth. The spacecraft's path through the corona allows it to observe the acceleration of the very solar wind that Parker predicted, right as it makes a critical transition from slower than the speed of sound to faster than it. The corona is also where the solar material is heated to millions of degrees and where the most extreme events of the sun occur, such as solar flares and coronal mass ejections accelerating particles to a fraction of the speed of light. These explosions create space weather events that can pummel Earth with high energy particles, endangering astronauts interfering with GPS and communication satellites and, at their worst, disrupting our power grid. This will be the first time that solar scientists can see the objects of their study up close and personal. We can expect data in December this year as the first close approach is sure to cause a buzz back here on Earth. That's it for this week guys, just a bit of space news as requested. We will of course keep you up to date with strange and interesting events that are sure to take place in the coming weeks and months. Comments below guys and as always thank you for watching.


Hi, good afternoon, everybody. So yeah, as William said, we're going to be going over best practices for Kubernetes runners. This is lessons that a 3-person team learned while managing GitLab runners for around 1,800 users at F5 Networks. So we'll go over a bit of an introduction, go over a little bit about myself, F5 Networks, our install. Then we'll go over Mike and the DOS they performed. It could have happened to anybody, but Mike is our lucky volunteer to perform this DOS. We'll go over limits and why you need them and why setting limits is very difficult in your Kubernetes environment. Finally, we'll go over labeling workloads, which while it's not the most important thing you can do with your pipelines, it is a very useful tool when trying to debug or get more metrics and information out of them. So just to get started, a bit about me. I'm a software engineer at F5 Networks. I've been using GitLab since around 8.x. And even though I've done an rn-rf slash on our GitLab instance, I still have sudo access, so yay me. So yeah, like I said, we have about 1,800 users worldwide for our install. 7,500 projects roughly when I checked last month, that's excluding forks, so I'm not inflating those numbers by people forking it and then reintroducing merge requests. Somewhere around 350 to 400,000 CI jobs are kicked off every month, most of them going through the runners that my team manages. And for all of this, up until recently, we only had three engineers managing our whole CI pipeline, GitLab itself, as well as a few other tools on the side. So any requests that come in, any questions, issues, they all came to our three-person team. Recently, we've been expanding the team, so it's a lot easier. We've also introduced more people in India in order to give us a more 24 by 7 uptime and process for watching for tickets, but that's a newer introduction. So on the screen here, you can see we have a basic architecture diagram. In our case, we had to break up our GitLab instance. So originally, it was a single server, and it was actually taking a single VM that was taking up a whole hypervisor in our virtualization lab. Obviously, that was a little bit too big, so we broke it up into seven individual servers. We have two HA web servers, you could call them. And then on top of that, we have a Postgres server, Sidekick, Redis, and Gitaly all broken out into their own servers. Additionally, we have a Postgres replica. That is a read-only copy of Postgres. That allows us to do reporting, as well as monitoring, using that to look for anomalies. This is all fronted by an F5 big IP, which if you're familiar with F5, you probably have heard of. It doesn't really pertain to this talk, but we are using that for our load balancing. So GitLab runners. We manage two main types of GitLab runners. We have our Kubernetes runner, which has most of the jobs going through it. I'd say about 90% of CI jobs at F5 go through one of our Kubernetes runners. On top of that, we also have Docker machine that we're using, which is both on-prem as well as in the cloud. So being a networking company, sometimes you need to configure custom networks in your jobs for integration, as well as sometimes even unit testing. So in order to do that, Docker machine allows you to have your own virtual machine that is dedicated to your job and will allow you to set up your custom networking topology within Docker, thus allowing you to run your test and tear it down afterwards, not affecting other users. You can't really do that in Kubernetes, so we've chosen to give users a Docker machine type instance as well. So let's talk about Mike. So Mike, or at least that's what we're gonna call him, was given the task of setting up some automation to configure different settings on various projects. So he decided he would use a GitLab CI job to do this, which makes sense. You're going to be configuring GitLab. You want it in source control. That way you can iterate over it. Everybody can review it. It's an easy way to manage it. So use GitLab CI in order to do this, but unfortunately when he was setting this up, he didn't quite read the fine print in the library that he was using. So let's take a look at his source code. So this is Node.js. For those of you that are not familiar with it, don't worry, it's a pretty simple example. This is the actual code that he ran with the one exception of the project, not the project name, the group name. So all this does is it gets a project ID for your project. Assuming that it finds it, then it creates a scheduled pipeline on it to run once per hour at the 30 minute mark. Seems straightforward. The problem gets to be when you run this, when this code is run within the example and for a timer project. You might see where I'm going with this, but if you don't, you get something like this. You schedule a job, then the next hour you schedule another job, so now you've got two jobs scheduled, and then you get four jobs scheduled, and then eight. After 10 iterations, you get around 1,024 jobs scheduled, and after 15, 32,000 jobs. If this was allowed to run for 24 hours, you would end up with 16.7 million jobs being scheduled on that 24th hour, and with a CI pipeline capacity of 350 to 400,000 jobs per month, that will easily clog up your system, taking down your Kubernetes nodes, as well as your GitLab nodes, GitLab nodes, most likely. So it's a pretty easy mistake to make. His assumption was that this create function would not create a scheduled task if there already was one that matched. The way to fix this is to check all the scheduled jobs, see if the one you want exists, and then only create it if it doesn't. But something as simple as this and benign as this could easily take down your system. So what do you do when this happens? Well, first you have to find out who did it, or more specifically, what project did it. You don't necessarily need to know the user yet, but you need to know what to stop. Once you've determined who did it, or specifically what project has done this, you need to disable CI on that project, and the reason that you disable CI is so that they can't spawn off more jobs. So once the CI is disabled, kill all the pending jobs. So for that, we choose to use GitLab Rails, so we SSH directly into one of the nodes, and we use the GitLab Rails command, enter into an interactive shell, and run this script on the screen, which I have a link to a snippet for. This will cancel all pending jobs. You could also do this via the API, but because we have access to our machines, we chose to do it this way, this also gives the advantage that if the API is not responding, then we are able to connect to it directly and bypass that issue. You can also potentially kill the running jobs, and this one is more of a judgment call. So if the job is going to take 30 seconds, it might not be worth it to kill the ones that are currently running. If the job is going to take 30 minutes, you're probably going to want to kill those jobs and free up the resources for your users that are waiting. Oh, and in order to do that, you would just change the status, pending into status running. So what did we learn? Well, we need to monitor our queue size for GitLab. So we leveraged that read-only database that I mentioned earlier, and we set up a cron job that checks against it every two minutes and verifies that it's not exceeding some threshold, specifically how many jobs are in the pending state. So this is an actual notification that went out last month, I think it is, where we were backing up our job queue. And so you'll notice a few things in this. First off, we have a dashboard you can visit that I hid. You'll also notice that we include the playbook into this actual alert. And the reason we do this is if it's 3 a.m., you're not going to want to have to think. You want to have it right in front of you. You don't want to have to know what playbook to run. You don't want to have to know where to find that playbook. You just want to know exactly what you need to do. So we've included the playbook in our alerts. And so far, this has saved us from two additional DOS attacks, which happened over the summer from an overeager intern who put calls to triggers in a infinite loop. So all things considered, this has saved us two outages and makes it well worth it. At first, we did have a few false positives while we were still tuning it, but we've gotten it pretty well tuned, so now we don't really get those anymore. So let's talk about limits. Raise your hand if you have an unlimited cloud budget. That's what I thought. Just for people watching this later, nobody raised their hands. Nobody has an unlimited cloud budget. And if you're running on-prem, then it's even more constrained because it takes time to get new hardware in in order to give people access to it. You need to buy it, rack it, stack it, actually configure it, and install the software onto it. So to meet the demands of all your users, you need to put limits in place. Otherwise, one or two users can use up all of your resources. So while users may not like limits, I know I don't. It helps keep the environment healthy for all users. So setting limits while they are enforced, they are a conversation starter. Just because you set a limit does not mean it will not change. Projects grow, projects evolve, projects need more limits in the future. That's kind of the way things work. So this is a way to start a conversation with users when they come to you and say, hey, these limits don't work. So Kubernetes supports limits by default and the GitHub Runner for Kubernetes allows you to inject these limits, which is perfect. So when it comes to requests and limits, requests are used by the Kubernetes scheduler in order to determine which nodes to run a workload on. Limits are used to determine when to kill a job for exceeding that limit. You can have different requests and limits. If you don't specify the requests, though, and you do specify limits, that is what is used by the scheduler to determine the request value. So here is an example. This is what that link on the previous slide was linking to of the GitLab config Tommel that we have for our GitLab Runner, roughly what we have running in production. And this is our Kubernetes GitLab Runner. We have a concurrency of 200 jobs, so it'll let Macs spawn 200 jobs. And you'll see that we are limiting the CPUs on the build container to two and memory to six gigabytes and on the helper and service CPU and memory limits. We have one CPU and one gig of memory each. And so it gives you that flexibility to break it out because generally you don't necessarily need as much CPU or as much memory on a service that you're spinning up in your CI job. So for example, you spin up Postgres for some sort of integration test. You don't necessarily need as big of a container as you do for actually building your code. So this is powerful in the lighting you break that up and actually enforcing that. So adding limits after the fact. So adding limits after fact is in some ways easier than adding them before you've set up a Kubernetes Runner, although I would not necessarily recommend it. This is what we did. So we started off using this Kubernetes Runner when there were roughly 50 of us and say 100 projects in GitLab. It was very small and so we grew naturally and we didn't know better. We were also using it when the Kubernetes Runner was experimental and I don't know that necessarily supported limits at the time. But the way you do that is you enable the Kubernetes Metric server and then you start monitoring your resources. So you need to determine, well, what are users doing? So we have a few tools that we can use. This one I wouldn't recommend necessarily using for this purpose, but it is a useful tool. It's called K9s or K9s, depending on who you are. And it's essentially a top like interface for Kubernetes. So what it allows you to do is look at what is running at the time, see how many, for example, CPU cores it's using, how much memory. You can also see the percentage of CPU and percentage of memory based on the limit. So you'll notice it says zero for the percentage of CPU and memory. That's because I have not set any limits in this particular demo. So if you don't have any limits set, it'll always show zero for those particular columns. So they may not necessarily be useful columns to you, but it can be useful for indicating to your head, hey, this is at 0%, I don't think I have limits set. More likely you'll use something like Prometheus and Grafana. You probably are familiar with this. GitLab has a native integration with Prometheus and Grafana. In our case, in this case, we are getting it from the Kubernetes metrics server. So I have included on the right side of the legend, the actual values, just to make this easier for us. And also note that it's only about 40 minutes worth of data, so it's not enough to go make decisions based off of it. It's good enough for example purposes. So when it comes to Prometheus and Grafana, you will set up some sort of dashboard to look at, okay, how many CPU cores are my jobs using? And so in this case, we can see all the jobs are using around four CPU cores, and that's because of the way I specified them. So you can reasonably assume that four CPU cores will be enough for all of your CI jobs, and you can use that as a limit. You might wanna go a little higher to prevent resource constraints causing it to be killed, but we can argue that four CPUs is good enough for this. Then we come to memory, and similarly, I put the actual values, the maximum values in the legend, and we can see that we are somewhere between 500 megs to one gigs of memory being used by each job. Memory is generally cheaper than cores, so you can probably go with one and a half to two gigs of memory for each of these jobs, and you'll be safe. Arguably, you'll probably want more in your production environment, but this is just for demonstration purposes. So setting limits from the start. This is a very difficult discussion because part of DevOps is making data-driven decisions. Well, if you don't have data, how are you going to make decisions? So ideally, you are migrating from something, say, a Docker runner or a Shell runner or some other similar runner where you can have these metrics available to you and you can just use them to extrapolate what you would assign to your Kubernetes runner. If you're brand new to GitLab and brand new to GitLab's CI at all, then it's pretty much just a shot in the dark. How many builds do you... You can kind of make some decisions based on how many builds do you expect to be running at any given time. What are your resource constraints? So how much memory, how much CPU do you have? Is anything else running on your Kubernetes cluster if you have a Kubernetes cluster that is running some sort of hosted application and you're adding your GitLab CI jobs to it, you don't have access to that full Kubernetes cluster so don't make your number, don't decide on your limits based on the full Kubernetes cluster, base it on whatever is left over in the Kubernetes cluster after those applications are added. Ideally, for separation and reasons like Mike's DOS attack, you'll have a separate Kubernetes cluster just for your CI jobs, but that isn't always the case. So expect to be wrong when you're making decisions without data, which makes sense. You probably already knew that, but I do want to call it out. You're probably going to make mistakes when you try and decide on your limits without having any sort of data available to you. So what happens when a user hits their limits? Well, their job gets killed. That's not 100% of the time Kubernetes does some magic under the covers where if it hits a limit just for a moment and it goes back down, it won't necessarily kill it, but generally speaking, it gets killed. So this can go either one of two ways. The user goes, oh, wait, I did GCC-J16 instead of J8. Oh, I'll just go fix that and everything will work. That doesn't happen often. I think all the administrators and operations people in this room know that. What's more likely to happen is they ask for more resources, which I've definitely been on the receiving end many times. The first thing you need to do is determine what their actual need is. If they just say, I need more CPU or I need more RAM, you need them to come back to you with concrete numbers.  Yes, I did. You just needed more than what I guessed. So you need them to come back with concrete numbers. If you do not have the capacity to actually support them, don't. I know it's hard to say no, and many businesses don't like when you say no, but if you can't support them, then don't jeopardize your other users by trying to give them more resources than you physically have. Work with them and try and find a common ground. Maybe order more hardware and say, hey, we can give you this after this hardware comes in and it's installed. Try and find a way to work with them because, generally speaking, we're a reasonable human being, so we can probably work together. So labeling workloads. Labeling is not the most important thing. Labels, if you're familiar with Kubernetes, are key value pairs for identifying workloads. They can, in GitLab's case, in the Toml file, they can be expanded environment variables. So for example, like you can see on screen, you could set the job and the dollar sign CI job ID, and it'll expand the label so that job is whatever the GitLab CI job ID is. Likewise, you could do source or project. These are some examples of some useful labels that you could do. They can be used by admins when they're manually doing cubicle commands against Kubernetes. They could also be used in reporting tools like Prometheus and Grafana for doing limits on stuff. So you could say, I only want to look at things that are of source merge requests. They aren't the most useful, but if you run into an issue, it can be useful to have for debugging purposes. So there are some limitations with labels in Kubernetes. These aren't limitations of GitLab but Kubernetes itself. Not all characters are supported. So for example, if you put a question mark in there, it's going to be a problem. There's a maximum length. You're probably not going to hit this one, but you can only have labels of up to 63 characters long. And if you do put an unsupported value, whether it's via bad character or a long label, your job will not start. There won't be a very good indication to the user why the job does not start. There is a GitLab issue on it. There is an open merge request that I need to follow up on to actually get this fixed so that GitLab will clean up the values that you introduce and actually prevent it from having failed jobs that don't give users good input of why they failed. I learned about these limitations by doing some testing with this in our Dev environment, seeing it work perfectly, introducing it into our production environment, finding out I didn't do enough testing. Slash ended up in my thing because I didn't test against subgroups and I was just putting the whole path, or putting the group name of the job into the pod labels. I didn't think about subgroups and the fact that they have a slash in them and suddenly anybody with a job in a subgroup just didn't start. I've put a link there to the official documentation around the limitations for labels in Kubernetes at that bit.ly link if you would like to check those out. And I'm definitely running ahead of time. So I'll go over the summary and then if anybody has any questions, I can answer them. But basically mistakes happen and you need to be prepared for it. So basically Mike made a really simple mistake, a logic error that could have happened to anybody. You need to be prepared for it through active monitoring and alerting. Preferentially you'd have automated remediation but manual remediation is probably what you will end up with. Setting limits will annoy some users but it does help keep the ecosystem healthy. You are going to have jobs that will want to exceed those limits but if you can't support it, if you don't have the resources then you need to work with those users to either limit how much they're using or come up with a plan to get them to where they need to be. But limits are only starting point. You do need to revisit them from time to time. As projects grow, they need more resources to build. A project on day one is not the same as the project on day 100. That's obviously hyperbole in terms of complexity but you do get the idea. Hello World is not the same as GitLab. They take a little bit of a different amount of time to compile and test. When you are setting limits without any data it's okay to be wrong.   They need, say, one more core. Okay, I can afford that. I will give them one more core. Labeling workloads is powerful. It's not the most important thing in your environment but when you are trying to debug something it can be extremely useful for exposing data. So it's better to have it and not need it than need it and not have it in my opinion. It's something that I don't use every day and I'll be honest about that. But when I have needed it in terms of looking through logs it's been extremely useful for me at least. So thank you. If anybody has any questions I'm free to answer any questions. I have about five minutes left. You can go ahead and raise your hand if you have a question and I'll bring the mic to you. Do you have teams at F5 that manage their own runners? Yes, we do. So we have around 200 different individual team runners. For example, if you need, we don't have Mac hardware and we build stuff for iOS and Mac. Well, we don't have the physical hardware or the actual expertise in that. So some teams, for example, do run Mac minis to build their code and they import them. Ideally, they all run through ours but we do understand that sometimes it's just not possible. Here we have one out there. Sean, thanks for the talk. Welcome. I was curious if you had any thoughts on preventative measures. On what? Like preventative measures. Yeah. Like a reactive approach to like the... Yeah, so that's... Yeah, preventative measures would be great. We haven't come up with an idea of how to do that efficiently yet. So we're open to ideas on that. For example, the mic example, it's really hard to prevent that because you have to understand what the code is doing and the context around it and for us to prevent that ahead of time, ideally we'll have been prevented via merge request and his colleagues looking at and seeing, oh, hey, this is an obvious loop. In this case, it was a test project and so he was the only one working on it and so it didn't go through the normal merge request pipeline which would have most likely flagged that as, hey, we need to fix this. So ideally your users are going to prevent stuff like that from happening but if you're working in your own personal project, there's not necessarily going to be that merge request that happens. So ideally the users aren't doing stuff like this and don't cause these issues but I just, I don't honestly have any idea of how to automatically prevent that particular issue from happening. We do have some preventative measures in our GitLab runners for example and when you're building darker images in Kubernetes, generally you need one of two things, either a privileged runner which we are not allowing or you need some sort of VM that is exposing the Docker socket for you to actually use. So we are building them in that fashion. You could also use another tool but that's talk for another day and so we found that users occasionally would create networks on this because they would do a test and automatically the address of these VMs were getting injected and they were causing other jobs to fail. So what we did is we put a big IP in front of it in our case but any load balancer in theory could do this and we put special rules in place to say, for example, the slash networks endpoint of the Docker API is not allowed at all. So we leveraged tooling to prevent users from causing issues for others in that particular case but in the case like the mic example, I don't think there's an easy way for it to prevent that automatically other than a manual review process which their team would do. Welcome. And hey, thank you, Sean. Let's give Sean another round of applause. Yep.


Welcome back, everyone. This is Brian. And in this video, we're going to talk about a simple producer and consumer. This is going to demonstrate the queue a little bit better. I had some user feedback and they said, hey, I loved your video on locking threads. But you kind of said you don't like working with the queue. Can you go into it a little bit more? I don't really like working with the queue, but why not? Let's just do it. So that's what this video is really going to cover. We're going to make what's called a producer and consumer. And it is exactly what it sounds like if this is a new concept. A producer will produce some sort of work and the consumer will take that work and do something with it. But we're going to put the producer on one thread and the consumer on another thread. Let's take a look. To start off with, we need our imports. And I am going to save a whole lot of typing and we are just going to do copy and paste. I love copy and paste. Anyways, so we're going to use the random number generator threading and multi-threading because we're going to be threading these. And I want to show you which thread and which process they're in. We're going to use logging. Of course, we're configuring the logging before we do anything else from threading. We're going to import the thread class and from queue, we're going to import the queue class and we are going to stop time. We are not working with asynchronous. This is true threading. I need a basic function, something that's going to be usable by all the threads here. So I'm just going to say deaf display and we're going to is blatantly rip off the function we wrote in the last video. And if you skip that one, I'm going to type it out really quick here. We're going to get the thread name from the current thread. So I'm going to say threading dot and I want the current thread dot maybe name. There we go. Keyboard has betrayed me. And I want the process name from, you guessed it, the multi-threading current process name. And from here, it's very, very simple. I'm going to say logging and we've already configured logging by the time we hit this function. So we don't have to worry about any of that. But basically anything above debug, which means, well, everything, we want to capture that. So I'm going to say logging dot info, but format this. And I want to know the process name, we're running this all on one process. So it's kind of redundant. But I want to really know that thread name. And I want the actual message that we're printing out here. That way our logs look nice and beautiful. Now, because we're making a producer and consumer, the first thing we're going to do is make the producer. We need to actually produce the work. So I'm going to say def and we're going to create work. I want a queue. I want another queue and we're going to call it finished. And I want some sort of maximum number that we're going to get to here. Now we haven't actually created these yet. I didn't want to create global variables, but queue and finished are both going to be this queue class right here. Now queue is and somebody correct me if I'm wrong, as you often will, it is a thread safe way of getting and setting variables. So I'm going to say finished dot put. So we're going to put that out to the finished queue. I'm going to say we are not finished because this producer thread has just started. This function will be the entry point for the producer thread that we're going to be creating later. And then it's very simple. I'm going to say 4x in range, max. So whatever maximum we give it. Now I want some sort of value. This is where our buddy, the not the range, the random number generator is going to do a random int. And we want something between one and 100. It doesn't really matter what we do there. We just want some sort of value. And now we're going to take our other queue. I'm going to say queue dot put. I want to put that value out there. Let's go ahead and display something. So I'm going to set display. I want producing. We're going to say producing whatever iteration we're on and the value that we have here. Now once we're done with this, we want to shut the producer down. And we're going to do that by simply exiting out of this function. So we'll say display finished. Now we want to go back to our little buddy, the finished queue. This is a queue just like this one, but we're going to use it in a different fashion. So I'm going to say finished. And we're going to say through. So rather than storing a bull in a global variable and locking and unlocking and acquiring and releasing and all this other fun stuff, we're going to use the queue to just simply put that out there. Now that we have a producer who's going to take our queue and put some sort of value out there, we want to grab that value in a different thread. And this is going to be called the consumer. And this is going to actually perform the work. I want to say work. And we're going to use a another variable called finish, which is going to be that finished queue. I tried to keep the names very similar. Notice how I'm not putting a counter out in a global variable because then we got to use the global keyword. And then we've got to do all this locking and unlocking while true. If not work.empty. In case you're wondering what work is work is a queue. It's this little guy right here. I could have actually named it queue, but I wanted to name it work just to show you name and convention really does not matter. I'm going to say v equal work dot get. And under the hood, what happens here is our queue is going to say get the next item in the queue. Because you could store multiple items as we're going to see as we slow this thing way down here. All right, now that we got that, let's go ahead and say display f and let's say consuming whatever our counter is. And then whatever value we got out of the queue. Now I want to increment our counter because this is all in the same thread. I don't have to worry about locking and unlocking or any of that nonsense. Now, if the work is empty, I want to do something like this. I want to make sure we're not actually finished. Now remember in our producer, we have this finished queue. And when we're done, we're going to put out there a true. So I want to make sure we're actually done. I'm going to say q equal finished dot get. We're going to get the value of that finished queue. And I'm going to say if q equal true. And there's a billion different ways to do that, but I'm just going to just hard code it equals true or is true. Then we're going to go ahead and break out of here. And that'll break out of that loop. And we should let the user know we are of course finished. There's really only one thing left to do the main function. Let's stop playing around here and let's make this thing work. Now, before we really dive into threading, we're going to set some variables here. Some say max equals 50. This is going to be the maximum amount of work we want the producer to actually produce. Now we want a queue that's going to hold the work. So we call it work equal great instance of the queue class. And I want to be able to signal that other thread on a different queue when we're done. Some say finished equal queue. Let's go ahead and make a producer. This is going to be a thread and we've worked with threads before. So I'm not really going to take a whole lot of time to talk about this, but we're going to say the target is going to be the create work function. We want to give it some arguments here. Some say args equals. And we want our work queue, our finished queue. And the maximum amount of work to produce. And because we've learned a hard lesson in previous videos, we're going to set this as Damon equal true. So this thread will stop running when we stop. Ooh, all right, that is a lot of work here. So let's go ahead and let's grab this whole line. Say consumer. Now of course we need to change the target. So we're going to change this to perform work instead of create work. And we have a different set of parameters here. We have work and finish. So let's go ahead and get rid of that maximum there. That one's going to be a Damon as well. Now from here, it's very, very simple. We're just going to say producer dot start. We're going to start the producer thread. Now the producer thread has no idea the consumer even exists. So the producer is just going to immediately get up and start making stuff. Consumer. Same thing. It has no idea the producer even exists. These two threads are completely oblivious of each other. They work very independently. So we're going to start both of those. Now I want to wait on that producer. So I'm going to say join. Let's go ahead and display that the producer has finished. That way we know in our main thread something happened. Now we've got a subtle issue here. We need to now wait on the consumer. Now when we look at the consumer, if we didn't have this little bit here, this consumer will just run forever. So we want to actually have that finished queue there that we put that value in there.  So let's go ahead and say producer. Let's change that to consumer. Whoa, got a little copy and paste happy there. All right. So they can consumer join consumer finished and then last but not least, I want to know when the main thread is done. Last line of our application right there. All right. Moment of truth here. Assuming I didn't break anything. Let's run this. Whoa, that went fast. This is multi-threading at its finest. So let's deep dive and take a look. You can see immediately the producer starts producing and it's kicked out. A bunch of values into the queue before the consumers even started, but that's why I put that index there. So producing 029 consuming 029 and you can go through if you really want to, but it's first in first out. So it's going to be produce. Consume produce where is one consume one. There it is. So you can match those numbers up all day, but they're going to be identical. Now, because this is multi-threaded, you're going to see that this thing will start just plowing through this thing at blistering feet. And it's going to say producer has finished and the consumer still catching up. So now we're just consuming consuming consuming consuming. That is the producer consumer. Now, if you want to see that in action one more time, let's go ahead and clear this and let's explain this code in depth. So we've got all of our imports, which are well, pretty self-explanatory. We have a display function, which is going to show us the process name, the thread name and the message. Now, our producer, this produces the work. The first thing we're doing is saying, get a queue, we're calling it finished, and we're going to put a false in there saying we are not done. We're going to do some work. And then after we've done all that work, we're going to change that to a true. That way in the consumer, we can actually tell when we're done. While we're producing, we're going to go ahead and just update the work queue and put that value out there. And we're just going to notify the user we are producing something. Consumer does the polar opposite. We're just going to say loop forever. And then while we actually have work, go ahead and get the next item, display it, do something with it, whatever you want. And then if we're suddenly finished, then go ahead and break out of this. And last but not least, we're going to join those back into our main thread so we don't have to worry about leaking resources, crashing the application or any of that fun stuff. And the end result is well, this beautiful little dance of data. I hope you enjoyed this video. You can find the source code out on github.com. If you need additional help, myself and thousands of other developers are hanging out in the Voidrealms Facebook group. This is a large group with lots of developers and we talk about everything technology related, not just the technology that you just watched. And if you want official training, I do develop courses out on udemy.com. This is official classroom style training. If you go out there and the course you're looking for is just simply not there. Drop me a note. I'm either working on it or I will actually develop it. I will put a link down below for all three of those. And as always, help me help you smash that like and subscribe button. The more popular these videos become, the more I'll create and publish out on YouTube. Thank you for watching.


We're here today to talk about Indigenous affairs and some of your expertise, your long-standing commitment to Indigenous affairs and your work both on policy, research and education. And I guess the first question that comes to mind is really to talk to you about the Uluru Statement and that question of voice for Indigenous people. How do you view the Uluru Statement and where things have proceeded since then? Well the statement was significant on a whole range of fronts. The first being rarely have we seen within Indigenous Australia, which is very diverse, you're talking hundreds of nations. Rarely have we seen such a cogent collective articulation of our people's aspirations and they are a voice that's protected and enshrined in the Constitution, not at the whim of Parliament, not at the whim of the Executive Government, but enshrined in the Constitution. So that was significant. Now whilst there were some First Nations groups concerned by the statement, it enjoys an overwhelming support amongst First Nations people, as well as the voice to the Parliament, not within it and that's been a source of political contention. It's to provide First Nations people an opportunity at the highest level of policy making to influence policy in a way that benefits Aboriginal and Torres Strait Islander people as opposed to disadvantages them. Of course the voice is complemented by a process of mahrata and that's a process of healing of truth telling and in the longer term agreement making. Now whether that leads to a treaty or a series of treaties, that's important for Australia. Australia is kind of an outlier. Australia amongst colonised nations is kind of the poor cousin in so far as places such as New Zealand, Canada, the United States have all entered into treaties of some sort. So in Australia we have to catch up. In Australia we have to overcome long-standing legacies of prejudice and exclusion of First Nations people. I think the minute First Nations people are enshrined within the Constitution at the apex of Australian policy making, then Australia will be a better place for it. So last year we had the First Nations forum here and the messages that you're giving me now very much came through in that conversation with experts from overseas. So how do we move forward with the Uluru Statement? As you say it was incredibly important to get that consensus of a wide variety of groups across Australia. We haven't had since the release of the Statement, we haven't had bipartisan support in the country. The Conservative side of politics has been concerned that it creates a precedence where in their mind is discriminatory, that is that only First Nations people could be a part of that voice. But that fails to recognise Aboriginal sovereignty tens of thousands of years in the making and that in itself is deserved of a process of negotiation and goodwill relationship between First Nations and government. Now to come back to your question in terms of what we can learn from overseas and the fact that we haven't got the Uluru Statement landed where ideally it might be landed, I remain hopeful that we can achieve bipartisan support. If you have a look at for example one of the people who had a political allergic reaction to the Statement, Tony Abbott, it's interesting that according to Warren Mundean's biography Tony Abbott was comfortable with the idea of treaties as long as we didn't call them that. So I'm hoping our discourse and our dialogue can mature and that we can land in an area where all parties are supportive of it because the country needs it, the country is diminished, Australia is diminished until we get some sort of grand accord between First Nations and more recent arrivals. So at the moment a lot of the policy making at a federal level sits within Prime Minister and Cabinet. Do you think that that is the right place for it to be or do we need to think about sitting Indigenous policy separately again? I don't have a firm view either way. I think there are pluses and minuses either way. The plus in terms of having it in PM and C is that it's the head of government and PM and C fundamentally has a coordinating role and a strategic policy role. So to that that part of Indigenous affairs seems to me being a natural and important fit. However the Commonwealth also is in a unique situation in so far as it not only obviously creates policy but it is involved in certain service delivery. The oversight and coordination of that service delivery, I'm not sure where that might fit but certainly where it comes to strategic policy making, implementation oversight and monitoring including the important role of research it strikes me that Prime Minister's departments are natural home. So this is the year of Indigenous languages and I know here at the ANU we're offering several courses where people can learn Indigenous language which I think is incredibly important. We've heard a lot of dialogue at a federal level about Asian language instruction. How important is it that Indigenous language is taught and maintained within schools? I think when you in some ways they can't be compared other than that they are a language other than English and what do I mean by that? I mean I think Australia looks to students engaging in Asian languages because it's in our commercial and strategic interest. It's good for the economy that is with the world increasingly shrinking through information technology and of course international travel. It makes sense for Australia to be connected to its region and one of the benefits of that of course is economic growth. Where it comes to First Nations languages I think it's a different proposition. It's a proposition that ideally all Australian people will get insights in to a place, a human place that is very, very old, oldest continuing civilisation on earth. So as a teacher or a reform teacher I find great appeal in all students be they at primary, secondary or tertiary having some sort of exposure to languages because language is central to Aboriginal identity and it's an Aboriginal identity is central to place and so we all ideally want to see Australians get a stronger sense of place. One of the contentious issues that has been around for some time now is about the impact of mining particularly on places of environmental and artistic heritage for Aboriginal people. How do you see that playing out? Is there a need for a policy response to that? How do we protect those thousands of years of Indigenous culture? Mining is a contentious area and contentious within Aboriginal policy and political views. Some groups see mining as an important source of economy and income and revenue and jobs. So there are some groups that actively negotiate and seek agreement with mining companies. Then there are others with a view that it's almost the antithesis of looking after country and that mining hasn't had a good track record in some pockets of the country in terms of the restoration of lands and threats to the natural environment. When you look at proposed new big developments such as Adani in Queensland, many First Nations groups are concerned by the ecological impact not only at the site of the mine but the waters that are then connected in and around it and so we've got to be very careful including as that water makes way out to the Great Barrier Reef. Incidentally, First Nations groups up that way are older than the Great Barrier Reef but I'm absolutely confident that I want to see it in pristine condition. So there is a tension there in terms of our peoples need to find economic basis but equally and probably the voice that doesn't get heard as much as it should is nature and Indigenous rights need to be seen in conjunction in my mind and there's I think there's great policy scope here to bridge these two rights that is Indigenous rights and nature rights. So one of the areas I think most Australians are now pretty aware of is that Indigenous people have poorer health outcomes, poorer life expectancy, poorer education outcomes, higher rates of incarceration. How do we as policymakers and researchers look at those issues and make a difference? By we need to recalibrate. There's been quite a conversation over the last few years about closing the gap refresh. What I've seen is that refresh has resulted in just additional indicators being added on. It's like and apart I struggle with parts of that logic why so many of the initial goals weren't met. So let's add on some more. I would hope that policymakers are prepared to let go and by that I mean the strategic and cogent approach to empowering communities. Grass roots up always beats top-down. When communities are empowered and resourced to make change they will and there's international evidence of this throughout the world of communities that have both the will and the way to turn things around whether it's poverty, whether it's violence, whether it's you know truancy, all these social problems which are not unique to First Nations Australia. It's just that we're experiencing them the more because we've yet to overcome in this country the structural barriers that need to be overcome in order to for there to be empowerment of communities. So as we head to the next election and hopefully post-election whichever political party with whatever political stripes I think it's incumbent on them to think very hard about the strategic empowerment of communities and getting that grass roots action happening as opposed to top-down dreams.