and this time on a very much time on the off road. So those would be the kind of account that I want to get. It's kind of like a major distribution or distribution generator. So one level of interaction more. It's a, yeah, fairly flexible, very powerful system. Very simple. But it's very popular with, I haven't seen it doing a lot of projects. Like it has to be a machine performance, where you can buy this house at that time. It can be used to create extra device images, that go to the device, they can create packages, they can do all the updates, and what you need to do in case of a problem against your own customers. And I'm going to let you know that Qt is a kind of go-to toolkit for the task design. So, level 5 kind of naturally fits in there. Especially if you're alone, I have very much that you could use in a virtual environment. So if you're able to do this, let's make this easily usable, what you do in your code, by aligning the level 5 VCs in your code, so it's basically useful as easy as Qt. That's actually what the girls are talking about. A couple of years ago, a couple of years ago, we were taking startups in the first five days, and that's some sort of a Qt process, it's a lot of work, but it can be used to remind you. And so the other couple of months ago, we had a sort of basically a set of definitions for a design process, and a training process. And then there's a few things that we encountered that might be interesting for you to look at. Because essentially what you do in your code is authentication, because we already know that security is some sort of platform, and also something like that. And in order to support that, the other system, SCP has two sets of builds, one for your platform, where you go to the compiler, where you go to the scene, and one for time itself, that's the stuff that goes to your device, and you need to make the client actually execute all the materials. And that can be a problem as long as we look for the process, and we obviously try to minimize that as far as possible. In particular, it means not pulling in the external and the interactive stack, just because you're a cross-generator and it's all a Qt group. And that can be quite hard, but a few times it isn't very much in the case of many other protocols. Most of the time it is just adding a bit of some options so that you can pull the cross-generator without calling it a device.  And it's also useless, even if the entire thing doesn't have a cross-generator. So if you have a cross-generator as your framework, you might want to consider those points. Maybe not as far as possible. Don't do any of the cross-generators. There are a lot of time dependencies, but that is the cross-generator that is super annoying to support. And I think that should kind of be my favorite offender there. We still haven't figured that out in the end. And even after simple stuff to do, yes, tweaking languages like Python, that is a regular form of cross-generate. And a C-make, which you can also use to generate basic stuff. There's not a lot you can do about it being any special support for this kind of cross-generate. Another thing to note there is how we can, as with people, this was a framework that used to study the media. And the most important part there is don't have textures to be honest, absolutely. That might sound obvious, but if you look at the usual most of them have textures in some form to work on some cross-generations. And there's 5.8. So a lot of the girl, we finally got to the point where we had not seen no textures not seen any more. But since I've made an experimental series, there's a video from an instructor who was doing test build before I think working. But it's no longer recording up there in headshed. Okay, but how do we actually demonstrate that we have frameworks in the up to now, because essentially it's just a bunch of libraries. We don't actually see anything. We don't actually test it. It's working. So what we did as well is I can't show a lot of what's not in particular the last demo I've shown following the same ideas. So we have a nice showcase and hopefully we'll be up here. We've got a shell that's been starting here or I was trying to find it free. And I see the test case and the approval process. It's going to be sort of using as the end of this example so it takes a second or there it is. So it actually works. It has to be able to see people that like are working for more development for action plus more. Because I like it a lot in this world. So this is basically just the end of the show. But you know how all the the local tools available for example looking at anything that is being organized and stuff like that. And it's a nice starting point for custom and very nice work. And if you want to know more how this actually works and what you're going to do I'm going to ask for another talk on this later this morning. Okay, thank you.


Hello everyone, welcome to the next lecture in the course remote sensing principles and applications. Today, we are again going to continue with our topic of remote sensing image acquisitions and characteristics of a remote sensing system. In the last class, we discussed about how image is acquired from satellites that are in geostationary orbits like the scanner will have to move in both the north south as well as east-west direction in order to acquire a 2 dimensional image we saw. Then we saw some important concepts such as IFOV, GAFOV, FOV and GAFOV. So, IFOV is related to the angle subtended by single detector element at the point of like its objective space where the ray from the earth surface enters the lens system or the optical system, optical point ok. So, the angle subtended by single detector is called IFOV and if you take into effect of the orbital height into picture and combine with this IFOV, we will get the ground size subtended by each of the detector element. So, corresponding each detector element with the size will have a corresponding ground size that is determined by the IFOV angle and the orbital height we call that as IGIFOV, ground projected IFOV. So, whatever be the features on within that particular GAFOV size all those radiances will be averaged out and will be recorded as one single value in that particular detector. Then the next important concept we saw is FOV and GAFOV. So, that is the total scan angle subtended by or total scan angle that is the scanner made or if you take in the case of like a push broom sensor the total angle subtended by all the detectors in the cross track direction at the optical point where the light ray enters from the earth surface. So, this is FOV and correspondingly if you take with the height orbital height of the satellite we will get what is known as the swap width of the satellite. Today, we are going to see an important concept about spatial resolution. So, what exactly spatial resolution is? There is no clear cut definition for spatial resolution, but resolution the meaning will tell us that our ability to resolve something or distinguish between two features that is the concept of resolution our ability to resolve or distinguish between two features. So, spatial resolution means our ability to distinguish two features in the spatial domain that is if you take one particular the let us imagine like there are two different objects on the earth surface that are next to each other and you take a photograph from it. Are we able to distinguish these two objects separately or not? We will tell us information about the spatial resolution of the system. So, how close an object should be to be resolved or how far the object should be to get resolved all these depends on the spatial resolution. So, if you look at this meaning like our ability to resolve then our ability to resolve two features in the spatial domain will improve when the objects are far apart that is one case or the IFOV of the system is smaller, smaller in size that is say there are like two different objects here object A, object B. If the IFOV of the system covers the entire area surrounding these two objects together. So, this is JFOV, JFOV then the radiance coming out from these two objects will be collected together and it will be recorded in the sensor as one single value. So, it is highly possible that we may not be able to distinguish these two features they may appear as one single different signal in the image rather than appearing as two different features. On the other hand if these two features are there A, B and if the JFOV is something like this then the energy coming out of these two objects will be recorded as two separate values or will be reaching the sensor as two separate values using which we can determine or we will be able to determine these two objects separately. So, the spatial resolution is in turn determined or influenced to a large extent by the JFOV of the system our ability to resolve features on the image is highly dependent on the JFOV size. Course are the JFOV here course are I mean very large scale say in order of 500 meters, 1 kilometer, 5 kilometers and so on. Course are the JFOV the radiance from many different earth surface features will be averaged out and our ability to resolve them separately may decrease or if the JFOV is finer our ability to distinguish features on the earth surface may increase this is one thing. The next concept is pixel size in the image normally whenever we download a satellite image they will say each pixel is 30 meters 500 meters and so on. It depends on depending on the sensor we will have a characteristic size of a pixel that is each pixel in the image represents a certain area on the ground. Say if this is like one Landsat image I will take an example say image from Landsat 7 ETM sensor ETM plus sensor in band 1, 2, 5 if you take any bands each pixel sizes roughly 30 meters. So, that means each pixel corresponds to a ground area of 30 meter on x axis 30 meter on y axis it is like a square pixel. So, that is the meaning of pixel size. But what determines this pixel size? Does JFOV determine the pixel size means the answer is no JFOV does not determine the pixel size. The pixel size is determined by our scanning sampling time in case of line scanner or vis groom scanner or in case of a push groom scanner the pixel size is determined by the distance between two adjacent detector elements. I will make it clear with an explanation, detail explanation. Let us take an example there is like a vis groom scanner like this scanning the ground from this point to this point. So, this is like nadir when the vis groom sensor scans the ground surface what will do first it will have it will subtend like a small JFOV collect all the data from it then it will move little bit it will move continuously so it will move it will collect another JFOV signal. So, like this the movement of the scanner is continuous and it will be collecting energy continuously from the ground surface. So, that is the scanner moves like this or this while it is moving continuously it will be collecting whatever energy. So, but each time it will subtend a small JFOV on the ground whatever energy is coming within the JFOV will be collected but it will be a continuous process without any gaps. So, the signal coming out of a scanner will be like a continuous signal something like an example I will tell like the signal may be coming something like this actually I am drawing the signal from one of the examples we saw in earlier image acquisition process. So, let us say the signal coming from the ground for each one particular scan line on the earth surface something looks like this. So, here there is very high amount of incoming radiance here that it suddenly drops then it slowly increases drops again. So, it is a continuous stream of energy across one entire scan line. So, this is one scan line but digital systems cannot store this energy continuously we already saw and it will do what is known as sampling. So, at definite time intervals delta t the incoming energy will be sampled and that particular energy will be sent to the electronics part for amplification and quantization. This we already saw in the previous classes when we discussed about image acquisition and image formation process. So, this continuous stream of incoming energy will be sampled at certain time interval delta t and that value alone will be sent to the electronics for quantization and saving as image. So, within this time delta t the scanner would have moved certain distance on the ground that is for each delta t time period the scanner would have moved some distance here next delta t scanner would have moved here like this. So, the scanner will be continuously moving when for each enlarged delta t time period when the sampling occurs. So, corresponding to this particular delta t there will be a ground distance the sensor or the scanner that covers it is as a delta t increases the ground distance also like scanner will say the different points on the red surface let the scanner be here at point A. After delta t the scanner would have moved to point B after delta t time the scanner would have moved to point C etcetera. So, there is certain amount of distance the scanner moves for each delta t time period and the sampling that occurs after each delta t and the corresponding ground distance moved by the scanner will determine the pixel size that is say at delta t instant the scanner moves the distance of 30 meters on the ground. So, one sample is being collected for a distance of 30 meters. So, next delta t that is 2 delta t the scanner moves the scanner would have moved further 30 meters along in the across track direction. So, another sample is collected. So, every 30 meters on the ground distance we are collecting one-one sample. Hence, the pixel size will be 30 meters the pixel size is determined by our sampling interval or the corresponding ground distance covered by the scanner during that sampling interval. This is known as GSD ground sampling distance or GSI ground sampling interval. So, GSI refers to the time taken for sampling say in the order of like few microseconds in case of like with broom scanners ok. So, maybe in order of say 6 microseconds every 6 microseconds one sample is collected or every 10 milliseconds one sample is collected and so on that is GSI. But with that time interval for the time interval delta t what is the ground distance covered GSD ground sampling distance say within the time interval of say 6 microseconds the since the scanner has moved a distance of 30 meters in the across track direction. So, 1 delta t 30 meters let us say here 1 delta t 30 meters 2 delta t another 30 meters another 30 meters and so on. So, that GSD that distance travelled by the sensor will determine the pixel size of the whisk broom or line scanners. In case of push broom scanners where there will be no scanning involved there will be multiple detectors involved what will determine the pixel size. The pixel size will be determined by the distance between 2 adjacent detector points. Let us take an example a push broom sensor is looking like this this is the across track direction. Let us say we have some 4 detectors. So, each detector element from the top left point to the top left point of the next detector what is the distance that will determine the pixel size because whatever is so this is essentially the detector size. So, if the detectors are arranged just next to each other without any gap then essentially the distance between them is actually the size of the detector element. So, this will determine the pixel size in image essentially this will be the JFOV for that particular pixel and that will also determine the GSD or GSD. Let us say one hypothetical example where one detector element is here say width of W then there is a gap W without any detectors then there is another detector with W with size of W there is a gap then there is a third detector and so on. If a hypothetical sensor is designed like this like with gaps in between detectors the detectors are not placed continuously like here then the pixel size in the image will be determined by this distance to this distance that is W plus W 2W that is each pixel will have a size of 2W or its corresponding JFOV. If you talk in terms of JFOV it is 2 times the JFOV because there is one detector element there is a gap so whatever the data coming in will not be detected will not be detected. So, the same energy collected by this particular pixel has to be saved for these 2 detector elements or corresponding JFOV terms. If you talk about detector size or if you talk in terms of ground distance each detector element will have a JFOV size. So, this is the pixel size will be 2 times the GIFOV we call this as inter detector spacing what is the distance between 2 detector elements but most likely in push-proom sensors they will arrange everything together such that there will not be any gap in between the detector elements because we need to collect continuous stream of data without any gaps on the earth surface and hence the detector element size will be equal to the GSD or GSI but if there are in hypothetical case if there is a gap in between 2 detectors then the actual detector size plus that gap together will determine the pixel size in case of push-proom scanners. So, the pixel size is not actually determined by JFOV, JFOV is different that is the area covered by one single look of the detector on the ground whereas GSI is determined by the inter detector spacing in case of push-proom scanner or our sampling interval in case of line scanner or vis-broom scanner. So, a pixel and the JFOV need not be related the JFOV can be different a pixel size can be different are there any practical examples yes there are once a good example is the earlier Lancer satellite which was launched into space this is example of Lancer MSS multispectral scanner the earlier Lancer satellite that was launched into space or a sensor called ABHR. Let us take example of Lancer MSS Lancer multispectral scanner what it had is the JFOV of the Lancer MSS sensor was approximately 79 meters by 79 meters like JFOV squared I will say that is each size is 79 meters JFOV on the other hand the sampling times was set such that the GSD was equal to 60 meters that is each JFOV will cover area of 79 meters by 79 meters but the GSD the distance between 2 plus marks will be 60 meters so what essentially will happen let us take if one such scan line is here so it will collect the JFOV will look one large area a sample will be collected at 60 meters interval the JFOV will be like half like a overlap the second JFOV will look something like this like between this point and this point another sample might be collected here so what the sensor did is it did over sampling the JFOV the energy was averaged over a much larger area but the sampling occurred at a much shorter distance 60 meter by 60 meter so each pixel was 60 meter by 60 meter in size but JFOV was in order of like 79 meters so essentially the neighboring pixels say if you take an image and if you take 2 neighboring pixels pixel 1 and pixel 2 there will always be some overlapping region what I am highlighting here this region will be commonly overlapping between those 2 pixels because of the overlapping or larger size of JFOV so each pixel will have some common information with its neighboring pixel in both the direction that is a middle pixel here will have certain portion of overlapping with its pixel to the left similarly certain portion overlapping with its pixel to the right so only one particular portion of the image or one particular portion of the pixel will be unique the rest of these 2 sides will have a strong overlap with the adjacent pixels this is called over sampling why over sampling was done over sampling was done to increase the amount of signal that came in to the sensor that will improve the signal to noise station one sense and also the image quality will be better because of this larger signal that is coming in in order to improve the image quality they design the system like this but the one drawback is due to this common information between adjacent pixels each pixel is not independent of each other there is always some common data available between 2 adjacent pixels that means the pixels will be highly correlated in space that is whatever information contained in pixel A will be highly related with pixel information contained in B similarly the information contained between B and C will be highly correlated and so on so it leads to over sampling this is an example of over sampling and it is done in order to improve the signal quality incoming signal quality but it resulted in a lot of correlation between the pixels. A typical remote sensing system on the other hand a normal system nowadays what are launched will not have this kind of over sampling issue will have the GSD set equal to GIFOV the example is given here. So, whatever is the GIFOV of the sensor the GSI will be set exactly to sample that particular distance. So, there will not be any overlapping information nor there will not be any gaps. So, GIFOV and GSI will be made equal in a typical remote sensing system. So, this is an example answer is an example of GSI size smaller than GIFOV this is always preferred this was done just to improve the signal quality but adjacent pixels will be highly correlated. So, next we are going to see this in little bit more detail through an example. So, these pictures show image of same area processed using some digital image processing techniques where the GIFOV size was changed for each image here the GIFOV size was 10 meters here the GIFOV size was 30 meters here the GIFOV size was 80 meters but the pixel size in each image is just 10 meters. So, the pixel size is the same that is effectively the GSI is same but GIFOV kept on progressively becoming coarser and coarser 10 meter, 30 meter and 80 meter. From this image we can see that even though the pixel size is the same the amount of information we can get from these images actually decreases that is in this image we are able to resolve lot of features here there are like two structures we are able to resolve here there are like lot of small structures here that we are able to resolve one doll but here we are not able to resolve them they appear as single feature in this image nothing is clearly visible for us everything appears much because of the larger GIFOV and the averaging effect of this 80 meter size reservation size. Hence the GIFOV the size of GIFOV will influence our ability to resolve features but GSI will determine the pixel size they can be different they are not one and the same they need not be one and the same they can be different but in most remote sensing systems the GIFOV and GSI are set equal such that whatever the GIFOV sizes a sample will be made in the corresponding distance inside the sensor system but they can be different that is the aim of this particular exam. We have seen an example where GSI size was smaller than GIFOV but can the reverse may also happen yes it can happen that is the sampling the GSI can be occurring at a larger distance than GIFOV we will see an example for that. Let us take an example we have data arranged like this okay two scan lines okay so this is scan line one this is scan line two. Now I am going to have a system like let it be like a vis broom where the sampling interval in both across track and along track direction that is for vis broom the sampling has to occur in both the direction because the satellite in across track it is moving like this satellite also is moving like this in the along track direction so continuous stream of energy will always be coming in hence the sampling has to occur in two dimensions whereas in case of push broom sensors the sampling occurs only in the along track in the across track it is determined by the inter spacing the spacing between the detectors adjacent detectors but along in the along track direction a sampling will occur because satellite motion is continuous it will produce a continuous stream of energy as the satellite moves hence sampling has to be done. In case of vis broom or line scanners sampling has to be done in both the cases because here also the scanning happens continuously without any break system will sample certain points satellite does not is moving continuously without any break hence system has to do sampling in along track direction also hence sampling occurs in both the directions that is why we get like a GSI it is possible to produce a rectangular GSI also the sampling interval is different between across track and along track direction it is very highly possible for us to get a rectangular pixel most likely we will get a square pixel because the sampling interval is same in both along track and across track directions ok. In similarly in case of push broom the sampling interval in the along track direction will be made equal to the distance between the two adjacent detectors there also we can have rectangular pixels. So, here we let us assume we have a vis broom or some sort of like line scanner and where the GAFOV size let us say this distance each this is like ground area I am talking about ground area some features are there the features let us say for example I am taking 1 2 3 4 5 6 6 5 4 3 2 1 let us say each size is 30 meter the GAFOV of the system let us say is 30 meters which covers each area. So, this is 30 meters this is 30 meters and so on a hypothetical example. On the other hand the GSI is set at 60 meters in both X and Y direction. So, what will happen because the GSI is set is 60 meters in both X and Y direction the pixels rather than be this is 30 this is 30 30 30 the pixel rather than having 30 meter size will now have a 60 meter size in the both X and Y direction because of the GSI. So, this will be the size of pixel in both X and Y direction. So, let us take the example a sampling is starting from here. So, since the GAFOV is 30 meters in the both along track and across that direction all these numbers let us take this as 1 pixel this 4 pixels you consider 1 2 5 6 will be collector data will be collected by the scanner because of a GAFOV size. But since the GSI is set at 60 meters only one sample for this large area will be collected it will be collected as 1. Let us not take the center of pixels and we will not enter into that sort of argument we will make it simpler. So, let us say one sample is collected. So, this sample is collected here for the next pixel the sensor would have collected samples and the next GAFOV will be here it will collect 2 then it will move to 3. If you take these pixels then essentially the sample 3 will be recorded here. Similarly, here the sample 5 will be recorded because we are sampling only one our GSI is much larger twice in both X and Y direction but GAFOV is much smaller. So, the detector will effectively collect data from 1 2 3 4 everything separately, but we are not sampling it like that in this particular direction our sampling will take only 1 3 5 in this direction there will be no sampling because of the GSI set that is how it is actually happening that is scanner is continuously scanning. But the data is not being sampled properly data is being under sampled rather than collecting 4 samples we are now collecting like 1 1 sample like that. So, here we are doing what is known as an under sampling. So, if you look at like the pixel there is like a huge amount of loss of data instead of having like any one at least the information should be averaged out. Let us say there is like a numerical averaging say the information should be averaged out and we should have an averaged result here, but that is also not there we are having only certain data points we are not collecting the actual averaged value also because our GSI is much larger than GAFOV. So, we are not averaging over larger area we are averaging energy over smaller area, but we are missing in between area because of our poor sampling interval the data is now collected being 0.1 and it is sampled over 0.2 data is collected but not sampled over 0.3 data is collected and sampled and so on because GAFOV is 30 meters GSI is 60 meters. So, 1 1 ground point or GAFOV element is actually skipped by the sampling system. So, instead of having 1 2 3 4 5 6 we will have 1 3 5 and so on it is actually a loss of data for us. So, having GSI larger than GAFOV will lead to under sampling and it will also lead to a phenomenon known as spatial aliasing. So, spatial aliasing means there may be some unwanted artifacts or the image may look completely different from what it is there on the actual ground we may have like a completely different features on the image than what is actually present on the ground. So, here we are doing under sampling and we may encounter a phenomenon known as spatial aliasing. Example is given here in this slide. So, here we have taken an image GAFOV equal to 10 meters GSI equal to 10 meters. We are now making using some image processing techniques we are converting the image as such here GAFOV remain the same, but GSI is becoming coarser and coarser. Say 10 meters GSI size is 20 meters here of GAFOV 10 GSI is 40 meters you can see how the image looks here. So, here there is some loss of information from this image this image looks bit different this image looks totally different. Here, so this is like a 20 meter pixel image this is like a 40 meter pixel image. This is also example of 20 meter pixel image and 40 meter pixel image, but the GAFOV size is made equal to GSI. So, here in these 2 cases the GSI was much larger than GAFOV in these cases the GSI is equal to GAFOV. So, essentially let us label this images as A, B, C, D pixel size of A and C are equal, but if you look at the image C appears little bit better than A, A has lot of points of discontinuities. Similarly, if you look at image B and D both have same pixel resolution pixel size, but the image information content when you try to extract information out of image D will provide you better information than image B. Because in images A and B the GSI is larger than GAFOV. Hence, we are doing under sampling and we are not representing the ground truly, whereas in image C and D even though the pixel size is closer because of larger GAFOV we are actually collecting at least the average data. Let us say like the example where we took pixels 1 and 2 if the GAFOV is made equal to cover both 1 and 2 we will have like an average of the energy stored in the sensor. So, let us say if it is a numerical average we would have stored 1.5 instead of 1 because the sensor is doing like a proper averaging. So, by making equal GAFOV and GSI we are actually preserving the information even though we are making large scale averaging, but having a larger GSI and having a smaller GAFOV may cause data loss and may cause a phenomenon known as spatial aliasing. So, in this particular lecture we have covered the important concept of spatial resolution. What spatial resolution means and what a pixel size means and what determines this spatial resolution and what the pixel size. So, essentially our ability to resolve is determined by GAFOV but the pixel size is determined by GSI. So, there is no clear cut definition of the term spatial resolution because everything combined together the pixel size the GAFOV size everything combines together will finally determine our ability to identify objects in the image. When we finally get an image we would not have knowledge what happened inside the system. We have to just look at the image collect information out of it and all these things the GSI, GAFOV and all other sensor are scanning geometric properties will determine our ability to identify objects in the image space. So, there is no single point of definition for the term spatial resolution, but GAFOV determines our ability to distinguish objects and GSI determines the pixel size. The next class we will further go deeper into the topic and discuss about how our ability to identify objects changes with different properties contained within the image itself. With this we end this lecture. Thank you very much.


Hey everyone. How y'all doing? I know last couple sessions the kubecon winded down I'll give everyone like a couple seconds like some people are still flooding from the hallways So this talk is cluster in a box. That's the title on the schedule. So we'll be talking about Kind of be going over Different ways to utilize different types of containers to deploy different kinds of Kubernetes architectures so With that said, I guess we'll get started. I have to catch a flight very shortly after this as well. So We'll get going So my name is Ry Terrell. I'm a software engineer at canonical the company behind Ubuntu I work on the canonical distribution of Kubernetes Cool, and my name is Marco Chepi amongst other things. I do operations at the Silph Road How many people here I imagine most people here know what Ubuntu is they're part of Ubuntu I don't think we have to really give them much of an introduction Who here has ever heard of the Silph Road? Yeah, I just like two people. It's okay. I imagine most of you might actually know the the community that this this But this group panders to anyone here remember Pokemon Go? Remember that thing last summer is like this huge craze the media stopped reporting about it But there's still like a ton of active people. I promise they're out there the Silph Road's a site built to basically Support that community. So if you've ever seen it hordes of people wandering around odds are we're supplying them the data We're to wander towards And so we're the premier grassroots Pokemon Go community Which sounds spectacular for those of you who know What Pokemon Go is you're probably more familiar maybe with other titles from Niantic they produced ingress and they have a bunch of other Titles that are very interesting as well, but it's entirely volunteer run an operated community website Myself and the others that kind of lead the website all have full-time jobs outside of this We don't have like an income plan. We're not here to generate revenue We're just here to make sure the site stays afloat and we don't have to pay money for it We get a ton of hits to this thing You'd be surprised how popular and how continuous our traffic grows even after the initial dull and spike These numbers are a bit old now, but we're doing well over 30 plus million hits and tons of uniques every month And because we're volunteer running operated We have a very interesting kind of development cycle where we actually leverage people from the community at any one time It could be just me or someone else up to most recently kind of ramped up for this upcoming. There's an in-game release It's happening literally right now. I'm the operations guy giving a talk But somewhere behind the scene there's a bunch of code being deployed and I'm just gonna pretend like that's okay But we've scaled up to things like 55 even 60 developers and they're all volunteers from around the world We have a giant volunteer organization for leadership and community management There's a 1,000 plus research group that all gets together to collect and use scientific information To determine how mechanics work in game all these things are great, but it leads to a huge nightmare in that How do we make sure that a we keep running a really smooth operating website so that we can keep being that premier Grassroots Pokemon go community and how do we make it so that we can keep enticing volunteers to support and supply these things And so it was obvious containers would be a way for us to really expedite that development workflow and still keep sanity from us who are all volunteering our time So before I start talking about Linux containers machine containers doctors and everything I want to talk just really briefly about The kinds of containers that exist in the ecosystem today for a lot of you This may be a review for some of you This may be kind of new and we'll kind of walk through why these different types of containers are important and where We've found ways to best leverage those things so This is a this is a server. It's a machine could be your laptop could be a piece of bare metal It could be a cloud instance of VM running somewhere But most for the most part it is a machine in the most traditional sense you installed an operating system on it It's a boot to it's it's slack or it's Arch Linux or whatever it happens to be you've got a bunch of processes running You know things like a knit cron SSH Damon loggers things We've known to come in love as a core operating system processes It's got networking this kind of cute orange dot down here It's got things like discs where you can go and store and retrieve data from and from here You can kind of slice this up in a bunch of different ways the first that we're probably most familiar with his virtual machines VMS I mean they've been a staple for people from laptops Through to servers most famously all public clouds are basically just virtual machines running on big giant servers And the virtual machines are just that they're Virtualization of IO so it's emulation of hardware emulation of firmware emulation of all these bits and carving out physical resources from a machine In order to have them appear as a smaller chunk of isolated running operating system, and it's an operating system It's a knit process is its cron it's loggers. It's networking its discs thing. We've come to love Then we talk about things like Docker containers I imagine most people here are pretty familiar with things like Docker run C rocket the gambit of things that provide you with a Docker like experience a way to manage and run processes in confined fashions The most noticeable difference between things like process containers of virtual machines It's a quick review is that virtual machines run all these supporting processes and a knit process SSH loggers Cron's daemons, etc You're just running your process that you care about and then the dependencies required to bootstrap that process Instead of a Docker container So you're not running a kernel you're not running all of these additional ancillary support services just a process Inside of some form of disk I like to make it inverted because you can write to it, but you can't really persist it I mean you can you can commit snapshot, but they're really meant to be ephemeral And you can attach networking and most importantly is you could run these really dense the number of Docker containers You can run per host Trumps in the thousands compared to virtual machines You can spin up on that same host do that same workload because you're not physically carving out resources and virtualization hardware You're just simply confining and constraining a process The last container type is a machine container. This is the natural hybrid model of a virtual machine and a process container You've got a machine. It's running processes and knit, etc. It's got disk. It's got networking It's got your application. It's everything you've come to know and love from the last 20 years of Linux system administration But you're able to run it just as densely or if not as densely very close or more densely than virtual machines Because it's leveraging those same primitives that Docker containers are it's not doing virtualization of I.O It's simply using the same primitives Docker does to leverage things like isolation of processes and stuff But it gives you a machine context a disk you can write and persist to networking spaces and it processes You can SSH you can manage it just like you would a normal machine and that's kind of This kind of spread of what we see from containers today. So there's things like virtual machines There's things like machine containers. We're we talking about LXD in this case But you could count things like the clear containers project while not technically a container is a is a very similar idea to how to run Really lightweight isolation of machines and then you have things like Docker run C rocket OC ID The gambits of different kinds of container technologies and that's kind of how the space plays out So you wouldn't necessarily although you could run an init process inside of a process container like Docker But you could definitely and it's expected to and only runs inside of things like machine containers so That's Kind of where we started we said okay Docker is going to be a great thing for us But all of our developers all of our volunteers they all have different environments We've got people from literally around the world on max on arch on windows on free bsd On every platform you can imagine and to be able to present them with a Standard development environment saying here's what production looks like it looks like this operating system It's got these characteristics this software installed was really difficult So we turned to LXC as our first start and so LXC was the to us It was that og Linux container and I say Linux container because it's very similar to things like Solaris and jails. I'm sorry Solaris and zones and free bsd in jails Which all give you that same idea of an isolation of a machine? But this was for Linux and that's what we were using and we found out that things like Heroku the Paz I'm sure you may all be familiar with Heroku their entire infrastructure their entire Paz offering is built off of LXC So it gave us a nice comfort layer of knowing that this isn't going to be some spiraling security issue or some problem Where it's not going to be compatible Also found out Pibbles Cloud Foundry the engine behind that uses LXC is a way to confine and constrain for their delivery For their Paz as well So that's what we started. We started with LXC and I'd like right to kind of just walk through real quickly to show you all What LXC kind of looks like and how to interact with it so? Right what do we got here so if I run LXC list I Can get a list of my currently running containers. I have none right now. I can I can launch a container like this LXC you launch I'll launch an Ubuntu Xenio container, which is 1604 and in just a moment that'll come up So what's it doing right now? It's fetching I guess the Ubuntu image. That's right. Yeah, all right cool and so This this container this is effectively what we be talking about where it'll boot off and use the same host kernel It's basically kind of how you would normally do like a docker pull and a docker Run kind of a scenario set that's pulling an OS image down instead. Is that right? That's right. Yeah, cool. So Yeah Please please ask questions. Yeah Yeah, that's your Christian. What about the kernel? So This is an Ubuntu Xenio host machine. That's that's super awesome for us in this case. Yeah, exactly. That's really compatible But what if we were to launch say like I don't know sent us six machine or if a door machine I don't know does anyone know what the current sent us seven kernel is is it less than four four? Maybe three ten. There you go. Well to be honest We we tested this we literally spent all day testing these demos make sure that work and we'd already cached the Ubuntu images So I wasn't expecting it to download it again So we would have been well into your answer already because that was definitely part of the demo Because you're right. They're sharing the same host kernel So what happens with kernel mismatches? Turns out it's actually not that bad of a thing Most kernels going backwards are fine So as long as you have a new enough kernel running a user land that expects an older kernel tends to not break It's when you run a super old host kernel and expect newer user land features inside the containers to access that We like using Ubuntu. Riot may be I mean we may be a bit a bit bias But Ubuntu tends to ship very recent kernels even in Xenial we can install a 413 kernel if we really wanted to And I'm looking for the next long-term support release. So that's why we use it as a host OS But it's not limited to this LXC is a tool can be run on really any modern 4-0 plus kernel kind of where the sweet spot is for these things So 81% later Yeah, another question We're to download that image. That's a great question. So This is LXD installed on top of an Ubuntu machine So each distribution that installs LXC kind of also can install a preset of like in remote image repositories So there's an image repository called Ubuntu which points to a remote source that has a list of all the Ubuntu images If this wasn't downloading I would already answered your question as well How do you see all the list of images? Where are those things? How do you hook into them? That's another fantastic question. I'm really happy you guys are asking these questions It means I'm on the right track from a talk perspective. All right. Any other questions while we wait for the last 0.0 percent Yeah, hey, that's it so right, please show me what do we have now, okay? So if we take a look at LXC list again we'll see that we have a persistent machine with an IPv4 address Cool. Could you launch me another machine dare I ask sure? So you used X this time do you want to explain why you used X? Why didn't you type Xenial? Is it not is it that long? Yeah, sorry. It's it's too long. They have a shorthand in the Ubuntu namespace. I like to use okay cool So that launched a lot faster. We have the image cache. So it takes mere seconds at most And it looks like humane jawfish is now running as a container. It's got an IP address and everything Can you launch me another container sure? But instead of a boom to can you give me sentos? All right So hopefully this won't download the sentos image, but if it does I look at how fast that was man pre baking some of these things are wondrous So show me the containers we have them. We should have now an exciting piranha a very exciting piranha What um Can you prove that that sentos I guess is my next question just to make sure there's no doubt in the audience here Is that is that actually sentos machine proved to me that right sure? I can do that So I can exact into the machine context and we can take a look at the machine So let's see exact. I'm guessing this is kind of like how you can docker exec into yes as well very similar So I can do a yum update Cat it's the issue cat Etsy red hat release as well sure Right hat release That'll give us what we want And then finally to answer the last question. Can you run a you name? Yeah, absolutely So this is the entire sentos user lands everything you've known to come in love and enjoy from sentos But it's running on a fort a four dot four Linux kernel from the Ubuntu host Which is interesting actually, but we haven't had any problems at all running different types of images Right. Can you show us all the images that kind of come available by default? Yes, I mean we've launched sentos we launched in Ubuntu. What else can we launch out of the gate this more legible? Oh, yeah, that's super legible now So the for those of you in the back or those in the front squinting like me we've got let's see Alpine. What else is there? Arch Linux sentos six and seven all the debbians you can imagine Sid Jesse Weezy stretch fedora Open susa gentu oracle Linux List goes on down to Ubuntu beyond so let's see has like a default image repository as a part of the Linux containers project So they ship a bunch of these images that being said though It's you can take and create your own in images put them into your own image repository Just like you would maybe a Docker container and distribute kind of starting OS's or even golden images on top of that So that's super cool So this is how we started we use LXC and then ultimately we started using LXD So LXD is nothing different. It's just a hypervisor for LXC So before LXC was really burdensome you had to let go and run all these long commands LXD made it super hypervisor like creating containers starting and stopping them Migrating containers snapshotting them everything you want to do with a VM except at a machine container layer It's just a restful API with either a local client or over the network And we just integrate it into our pipeline So what we're able to do is on board a new developer create them a new Container that they can SSH into and then they have the tool chains that we use they had the environment They had everything that we basically have in production for them to play break repeat and rinse And that ultimately led to a better velocity for our developers So they have everything they expected everything they needed in order to start contributing and being effective as a volunteer Their time is precious to us. So the least amount of friction we can have the more we can get from contributions So on the cluster in a box so Switching tracks a little bit Ryan I had the pleasure of working with a really small team of people and the Linux Foundation to help design The certified Kubernetes administrator exam has anyone taken the CKA Kubernetes administrator exam. Yeah, wasn't it brutal? I'm sorry It's a really tough exam, but it's a great exam I recommend everyone here go out take the exam It's super tough But it kind of covers the gambit of things that you'd ever need to know in order to administer a Kubernetes cluster and that certification holds quite a lot of weight if you're looking for jobs or shopping around What CNCF yes in the Linux Foundation that produces and runs the exam So we had a bit of a problem when we were designing the exam in that this is kind of what you expect today When you run a cluster you've got a bunch of machines probably in a cloud somewhere Etsy D Masters workers, however you do your architecture whether you're self-hosting whether you're keeping components separate You've got VMs running So when we approached when we're working with the Linux Foundation team and the CNCF to kind of design this exam We said okay Well, we can't use just a single cluster because if the user messes up on a previous question We don't want that to impact the rest of the exam one wrong question shouldn't basically Discount you from being able to take the rest of the exam. So we said we're gonna need at least six clusters In fact, we ended up with something like eight or nine at the end of the day We designed all the questions to make sure that we had properly isolated all of these pieces and the Linux Foundation and the CNCF came Back and said that's great. However This exam is three hundred dollars So at that price point in order to run a five or six hour exam with the record was at setup times and burn and Review times that means that this infrastructure will cost well over the exam price in general so we went back to the drawing boards and I remember to talk from the last kube con this was given by Lin Sun who was formerly at IBM I think she's now at its Istio I remember this talk from Berlin where she talked about using LXC containers as a way to Do colocation of multiple clusters in a single machine? And so this was very fortuitous because we worked on this project over the summer So we started leveraging a lot of those primitives in order to drive that as let's deploy multiple clusters in a machine So the Linux Foundation CNCF said you have one VM. This is the side We're gonna run it in a cloud instance, and that's it make it work inside of there And so we did we set up an LXC cluster and we deployed all the components that you'd come to know and love They are all isolated from each other. They all have IP addresses. You can connect to them. They're all networked up It was effectively like we were doing in a previous diagram only it was on a single machine And then we were able to take that we were able to scale it up and we said, let's create just a bunch of clusters Let's destroy the IO on this machine and see how big we can stretch it at the end of day We were able to deploy quite a lot of consecutive clusters all isolated from each other all Independent of each other so that as exam takers move through the questions any wrong question against a bad cluster wouldn't affect the rest of the exam and As of right now if you go and take the exam you'll be taking it in an environment similar to this So I'm gonna have rye take us through what that kind of looks like in practice This isn't the actual exam environments It's similar in construction, but it would be disingenuous if we showed you what the exam environment looked like So this is just a using similar technologies and processes setting up something like that. So what do we have? Here right so I've SSH'd into an AWS box It's an m4 2x large It's running five Kubernetes clusters each cluster has about 10 machines For a total of approximately 50 machines So if we take a look at elixir list here, which might take a moment to come up because it's a long list what um And for for those m4 2x large what how many cores how many gigs of RAM is that it's eight cores and 32 gigs of RAM So what's the monthly if I'm just doing spot instances? What's the monthly cost on that like this most expensive? I'll see it that would be $300, okay, so this is a I forgot the cores and you said eight eight cores eight cores $300 a month a box. So that's five clusters about 50 LXC machine containers. That's right. Yep So this is a LXC list. So these are all the machine containers running You can kind of make out that some of these are the Kubernetes nodes because you can see the Docker and CNI and flannel bridges We could SSH into one of these the keys are already over there. So let's see and we can take a look at What's running under Docker we should see some kind of Kubernetes loads We can take a look at The pods the nodes cluster status So this is one of the clusters then it looks like there's three Kubernetes nodes in this cluster. It's got a three nodes of Xcd running as well and some workloads some pods I guess running Yeah, that's right cool What's the so What's the resource utilization like on this box? I guess is the next question I would have as a good question. Let's take a look So the load's a bit high right now. I usually see something around Five between five and seven, but we can take a look at each top as well So looks like the 15-minute load is around eight. So eight cores. That's pretty much about a hundred percent utilization. So that's five Kubernetes clusters across 50-ish machines running on a single instance at $300 a month. That's right cool I Imagine most of the bottleneck is not CPU, but actually disk IO or is it something else? Yeah, it is disk IO right now Cool Sweet, so that's that's an example of how we're able to do things like using LXC You can use whatever flavor of tool you like to deploy Kubernetes with Whether it be kube-adm Or the gambit kube-spray CDk etc to deploy a bunch of machines into a single cluster so from a silphrope perspective This was wildly successful for us We were able to set up a bunch of dev and test clusters on a single instance In Google's compute engine which gave us better economics And then we could basically test all of our pipelines with the same production model number of machines number of configurations Etc and then our actual production cluster was the actual where we actually spent the most of our money And this actually worked out pretty well for the Linux foundation as well cool so The last thing I want to talk about which is kind of the next Evolution for us at the silphrode is bare metal and large clusters As you can imagine we have quite a lot of code running for those of you views the site Which is none of you you should check it out if you don't that's fine, too But we have a lot of supporting services things like in a geolocation based game It's most important to help find people that are in similar areas to each other So we have lots of code that runs to help figure out who is close to you to help you achieve Certain tasks in game. We also have a giant world map Which we help display where points of interest are and other things of of importance within the game itself And so because of that we do a lot of traffic and if you've ever tried to embed Google maps or any other map Vendor onto your website. You might find it's pretty okay when you start pushing 30 million plus hits those things it actually becomes really expensive. It turns out maps and base maps are super costly And so we are into a problem where our map vendor was saying we can no longer support you at the free tier You're going to need to pay ten thousand dollars a month for utilization Considering we pull in less than two thousand a month in ad revenue That was not going to be sustainable for very long and I don't feel like paying out of pocket very much for these things So we figured okay, we'll run our own map servers. We use open street maps. There's open-source software do all of these things will take our Take our destiny destiny into our hands and we'll build and run these things ourselves Turns out it's super expensive to generate map tiles. No wonder. It's so frickin costly So we realized that running in the cloud is not very effective. We will invest in some bare metal servers We found a really decent Colo some good secondhand servers Two sockets about 400 gigs of RAM and tons of disks to store this data And then we hit our next roadblock, which is the 100 pod Limits does anyone know about this is anyone hit this yet in their clusters running clusters turns out as if we'd read the documentation that When it comes to building large clusters, there's some limits in the software itself The chief being that you can only run a hundred pods per cooblets now For those of you that running Decent capacity workloads most average web workloads don't really hit those kind of things especially in cloud instances We have like four to eight cores on average per machine or less and like less than 16 gigs of RAM a hundred pods in that space is actually quite dense But when you start looking at things like bare metal This was the average size of our server. It's a Two socket server kind of standard almost white label box We had 24 CPUs and 500 gigs of RAM and about 20 terabytes of space And if you start dividing that up and saying okay, I'm gonna run on this server is on average a hundred pods That's not gonna work very well It's awesome if you're doing machine learning or rendering farms or anything that requires tons of compute space But very little workloads, but if you're not it's kind of awful And so what we found is that well If this is the model today one piece of bare metal to one pod limit to 100 pods if we leverage that kind of same learning There's a trend here in this talk. I'm not sure if you guys are sensing it But we could leverage things like machine containers in Lexi to actually split up a piece of bare metal Because we're not incurring any virtualization overhead. I don't need to worry about installing anything to manage my VM layer I'm just using the traditional hypervisor API to say give me four machine containers There's no vert IO overhead So I'm not limiting the resources on my bare metal machine, but then I'm able to slice it up and say instead of one cooblet I can actually fit four cooblets at 400 pod limits. We went further and started playing with different architectures So we said alright Well, let's do it this way where instead of having this previous model where we JC said Each one of these was pinned to a CPU socket So to be at two machine containers per sockets and then half the RAM divided across them We went further. Let's do CPU sharing models Let's see how we can slice and dice and limit the quality assistance quality of service on these things We got up to a 900 pod limit. We never hit that. I don't we didn't have enough stuff to run 900 pods And so we actually ended up with was a mixture of architectures where we have one giant LXD Machine pinned to one socket and then the rest for more general purpose workloads that are running alongside of it And we were able to utilize about 600 pods per of one of our servers Which gave us a good sweet spot of density while still being able to leverage performance from the underlying hardware These are nodes so these are yeah, this is a cluster We have about three or four pieces of metal kind of split up in this architecture So we actually have a lot of capacity for pods, but we run on average around To serve all of our map tiles. We run probably 1200 pods Yes, it's the LXD itself. It doesn't see the bare metal. It just sees that you have in this case five Kubernetes nodes LXD containers It's less overhead. It's less overhead both in managing it and it's less overhead both in resources utilized the actual residual Computational processes used by LXD containers is managing the fractions of seconds of micro seconds of CPU time where VMs It's actually very heavy exactly Yes, and then dynamic resizing like we're able to reallocate the limits on these things like we'll say suddenly no longer Are you CPU pin just see if you share or you're now have a hard limit of memory all that stuff We can do dynamic without restarting anything. Yeah, exactly. Yes We haven't hit any We do this today in production. I know of a couple other customers that are doing something like this in production as well We just had to make sure we allowed and whitelist certain kernel modules into the machine container So things like Vlan IPv IPv6 and stuff but outside of those things we didn't have it had any problems and we view this as a safeguard for us We never touched the bare metal anymore. It's just very thin lightweight a boom to OS at this point And we're pretty confident that if we ever got an exploit because again, it's volunteers giving us code We do our best for security, but we're also not PCI HIPAA combined by any means But if a if a container did break out of confinement in Docker or whatever CRI we were using We feel okay knowing that it likely won't escape this isolation and affect the bare metal or eject bad firmware or something Yeah, we don't use live migration I've used it in the past, but we don't use it here. We just simply resize or just delete machines and create them. Oh I don't know. I've not tried Maybe worth a shot in a blog post though. Yeah, cool No, these are all connected to one cluster just spanned across multiple pieces of hardware So from our perspective, we've got it's actually just three pieces of metal in a colo It's not like a huge bare metal footprint, but we have a lot of capacity that we use So inside of Kubernetes, we actually see 15 Kubernetes nodes in our single cluster And they just happen to be spread across these it's almost as if we've gone to Amazon launched a bunch of Amazon instances We're just simply use utilizing machine containers to get the best resource utilization out of our bare metal We've got tooling that we do to automate like the creation of these things so we can repeat them So we just simply replay that automation there and then it stands up all of our little pieces and connects into the cluster with configuration stuff, yeah We use the combination of stuff and open source. It's not Super relevant. This will work anywhere. We've done it with kube-adm. We weren't quite ready for kube-adm There's some ha and upgrading problems. We wanted to make sure we always upgraded our kubernetes. So we used a combination of things like Canonicals distribution of kubernetes and some of our own scripts to manage how we do our bits So we just kind of borrowed from vendors said install stuff. Yeah Yeah, it's not at the end of my slide, but I'll add that I had a bunch of links in there I'll make sure that link is in there as well. Just some of the infrastructure code we have Yeah, LXC has a way to modify profiles So inside of LXC I can say here's a general profile that gets applied to all machines or this machine in particular has Vcpu limits these memory limits even down to network and disk IO Throttling as well. So all that's just built into the Linux kernel module for these namespaces basically Yeah Yeah It does it does a lot of this we had some pretty unique architecture stuff So we utilized a lot of the tools underneath conjure up to do that But we basically built and scripted our own way Conjure up works great. That's how we did a lot of our proof of concepts But when we went to production we we use the same tooling underneath conjure up to do that But we just kind of scripted our own way basically Yeah, so juju is one of those tools and then we bolted on some scripting and processes to kind of map how we Dynamically resize these lexine machines or says it handled in juju Yeah, so the end of the day sorry We did actually start with virtual machines initially and we found that the density was quite poor in fact, we ended up seeing about 10x more density with the same workload on these pieces of bare metal 10x probably more actually but as a result That's kind of how we ruled out virtual machines and still sit firmly in the land of containers. Just a varying range of them I think that's about it for our talk. Yeah, thank you. Thank you all for coming. I really appreciate it I don't know I Think we're out of time. Yeah, we're definitely out of time here Those links I mentioned before linux containers that org for everything LXC LXD Cncf.io certification experts all you should get certified. It's a great thing to have do study before you get certified It's like a four or six hour exam There's lots of questions about Kubernetes from both administration and usage and then that's that cluster large page I like to make sure I source it people know they always change those numbers and releases But sometimes Just to make sure you have the latest specs on how big clusters can get and I'll add their others links I mentioned to you as well. Thank you all enjoy the rest of kubecton


and close up the hand boogie and scroll down one more time and did i did i skip it here i'm too zoomed in i can't see right it's too zoomed in so now we'll look at the sales by customer detail now notes you also have this uh sales sales by a customer type and that would be applicable if you have you know customer types as as a category category section you can take a look at that but we're just going to look at this one sales by customer detail sales by customer detail so i'm sorry this one sales by customer type detail i'm going to go to this one sales by customer detail let's pick that one up and change the range again from 01 01 22 tab 1231 22 tab run it to refresh it so now you've got the customer information and the actual activity uh per customer so this is a nice report that gives you the detail but this is one of those reports that you might not use quite as often because you might look for this kind of information over here if you're if you're trying to communicate with a particular customer on this side where you would be going into the sales the sales side of things which would be kind of like the customer center i'm going to close up the hamburger and if you went to the sales transactions then here's where you can sort those transactions and possibly sort by the invoices for example or you can sort by the sales receipts so now we've got the sales receipts that we're sorting by and we can also go to the customer side of things and we could sort by the open invoices and so on so if i'm actually communicating with an invoice about the sales we made to an individual we probably find the individual in here and then look at that the detail that they have within it as opposed to say running a report like this and then looking for you know the detail within the report was my thought process on that but there is this