Hello everyone and thank you for tuning in. My name is Ritesh Naik and I'm here to talk about using Container Checkpoint Restore in Kubernetes to achieve fast port startup time. We at MathWorks have been using Container Checkpoint Restore at high scales since 2016. In this talk, I'll go over how we incorporated Checkpoint Restore in Kubernetes to achieve a scalable system. Hopefully, you can take something away from the session and apply to your own environment. On that note, before we get started, let me quickly introduce myself. My name is Ritesh Naik and I'm a senior software engineer at MathWorks. For those who are not familiar with MathWorks, MathWorks is a leading developer of mathematical computing software. Engineers, scientists, and researchers worldwide rely on its products to accelerate the piece of discovery, innovation, and development. I've been at MathWorks for over five years working on various areas of infrastructure and scalability. My team and I have been focusing on doing things in Kubernetes space since few years. If you'd like to reach out to me after this talk, feel free to ping me on Kubernetes Slack or via email. I'm happy to talk about all things containers, Kubernetes, Checkpoint Restore, Travel, or anything from that matter. So in this talk, I'll go with the motivation behind using Container Checkpoint Restore, quick introduction of Checkpoint Restore along with a demo. Our solution of using Container Checkpoint Restore in Kubernetes along with some of the lessons learned, best practices, and future enhancements. Finally, we'll wrap it up with Q&A at the end. As in any typical Kubernetes cluster, here we have a workload of containerized applications orchestrated by Kubernetes. This workload could include application serving concurrent requests like web service or each request could be handled by individual containerized application like in a function as a service or serverless setup. Applications could come up pretty fast or they might have initialization overhead. Now, keeping different workloads in mind, what are some of the things we need to consider to build a scalable system? First and foremost, system should be able to react quickly and scale fast to any unexpected traffic spikes. Secondly, there should not be any first use performance impact on the latency due to a cold start. This is basically the time spent on setting up the container environment along with the application initialization to come up ready. This could be avoided by keeping the containerized application warm and ready to go, and part of warming the containerized application involves initializing the application. We'll talk about this more in the upcoming slides. Third, to use the resources efficiently by maintaining low cost waste. So, how easy or difficult it is to achieve these goals? Let's start by addressing the elephant in the room, cold start. Yes, cold start is one of the main challenge that makes it difficult to achieve all three goals in the same time. This could also be seen in the function as a service or serverless use cases. Let me walk you through some of the options in the next couple of slides on why it becomes difficult to achieve all goals with container or application cold start-up time. Option one, scale out on demand with usage. Basically, wait for the usage demand and scale up the workload to meet the meet demand. Let's take this and stack it up against our goals. I'll give you a couple of seconds to guess. I'm sure most of you got it right. So, as you see here, this does well with the goal of using resources efficiently since new ports are spawned only when needed. However, this is prone to cold start pains and would have poor first use performance. Also, this wouldn't be able to scale quickly to meet the sudden burst in the workload. Can we do better? Let's see. How about we create a standby pool of pre-warmed pool? Basically, maintain a pool of pre-warmed containers and size of the pool could be calculated based on the historical traffic trend. Now, let's do the same exercise again and stack it against the goal. So, it does address the cold start pain. Setting up of a container environment and initializing the application is done ahead of time. But, it still suffers from handling the sudden burst in the workload. And since we are maintaining this pre-warmed ports, there would be a lot of waste in resources if these ports are sitting idle and traffic is slow. Now, can we achieve these seemingly contradictory goals? Can we get around the cold start pains to provide fast performance on first use with the ability to scale out fast to account for unexpected traffic spikes along with maintaining low utilization rate? It seems like a daunting task. But, yes, we can. And the answer is checkpoint restore. We used Linux utility called Cryo, which stands for checkpoint restore in user space to perform checkpoint restore of the containerized application. It's an open source project on GitHub with around 10,000 commits and hundreds of contributors. Now, idea here is to checkpoint the warmed container that is already initialized and restore as needed during scaling. With this approach, the startup time is reduced considerably. In some cases, we saw around 70 false improvement in the startup time by going from couple of minutes to couple of seconds. So, how checkpoint restore works? Checkpoint freezes a running application or any Linux processes for that matter and serializes the state to the disk as a collection of files. These files includes any files related in form, sockets, namespaces, memory, et cetera. Now, restore creates a process tree by reading these collection of files and restores the process state. Let's go over a quick demo on checkpoint restore to understand more about Cryo. In this demo, I have a containerized Go web application that I call it as GoPause. Basically, GoPause is simulating an app doing real startup and pre-warming that takes about 30 seconds before it starts listing over port 4000. And you can see how a checkpoint restore, this 30-second initialization has to happen just once during the checkpoint and every restore of the container would skip the initialization and be ready in couple of seconds. So, let's get started. So, we have a Go web application here that takes around 30 seconds to initialize. Waits for initialization to complete.  For this, we are using Run-C as container runtime for creating and managing container lifecycle and Cryo to enable checkpoint restore functionality. Run-C itself needs a config.decent file through which container options are passed and root file system to be used for containers. With that, let's take a look at the config file to see what is included in there. It includes details like OCI compliant version, command to run at the start of the container, part to the root file system, as you can see here, along with some of the mounts required on Linux. Basically, any container options that is required for container goes here. Let's start a container running the Go web application. As you see, it's waiting for the application to finish initialization and it will take around 30 seconds to initialize while we are waiting. Let's start pulling port 4000. This should get the response back once the Go app finishes initialization and starts listening on port 4000. So container is ready. We get this thing back as the response since the container is ready, let's checkpoint using Run-C checkpoint. This should stop the container as you see and writes the checkpoint files to the disk. Let's see what the status of, so yeah, container is stopped. Now let's restore the container from the checkpoint and you should see that it comes up ready in a second unlike running the container that would have spent 30 seconds to initialize and come up ready. So you should see the advantage of having restoring the container over just running the container if you have checkpoint available. So if you look at container lifecycle through the straight transition lens with checkpoint restored, we were able to skip the initialization and jump to ready. So to summarize, checkpoint restore is the answer to fast container startup type. Wonderful, isn't it? Let's leverage this in our world of Kubernetes. Unfortunately, as of today, there is no need for support of checkpoint restore in Kubernetes. Let's see what options we have here to bring in the checkpoint restore functionality to Kubernetes. One option would be to add a support of checkpoint restore in Kubernetes natively. This would involve creating Kubernetes enhancement proposal design MPI review. Second option is to take a non-native approach without touching the Kubernetes code base. And in this talk, I'll walk you through our solution to non-native approach. So checkpoint restore basically has different use cases. It could be used to address cold start that we talked about or migrate a container from one host to the other. Maybe during maintenance, among many other use cases. Given the different use cases and complexity of checkpoint restore in Kubernetes, it touches some of the core workflow and components. So adding checkpoint restore natively would involve making some code design decisions and might extend the timeline to make it to general availability. Time to GA was one of the factors that was considered to offer non-native approach over native approach. Having said that, there might be a case that could be made where non-native and native could coexist given the specific problems it is trying to solve. So our approach of introducing checkpoint restore capabilities in Kubernetes involved adding checkpoint and restore microservices. Checkpoints basic functionality is to checkpoint a running container. Restores basic functionality is to restore a container from the checkpoint. Checkpoint accepts container image and configurations required to run a container, waits for the container to start and completed its initialization and checkpoints the container. Restore uses this checkpoint image and restores the container to the state at which the container was checkpointed. Now, Kubernetes doesn't expose checkpoint restore functionality through its API. And in order to checkpoint or restore the containers in Kubernetes, we had to think little outside the box. We needed a way to introduce container runtime in Kubernetes that supports checkpoint restore functionality and not managed by Kubernetes. Hence, we went with the design of container runtime in container runtime. As the name suggests, it's running a container runtime inside the container of a pod spawned by container runtime that is managed by Kubernetes. Parent container that runs the checkpoint restore service runs the containerized application using the child container runtime. Parent container runtime could be any container runtime supported by Kubernetes and child container runtime needs to support checkpoint restore. For this doc, I would be using Docker as the parent container runtime and RunC as the child container runtime. Hence RunC in Docker. Let's take another look of the state from pre-checkpoint to post restore. In a pre-checkpoint state, pod running checkpoint service spawned that runs the inner container to be checkpointed. In a checkpoint state, checkpoint image is made ability. In a post restore state, there is a pod running restore service that has a restored container running inside the outer container using the checkpoint image. Now let's look from the sequence of events anchor. In checkpoint sequence, checkpoint service is deployed. Checkpoint service starts the application container running as a child container and waits for application to become ready. Once it is ready, checkpoint the container and save the checkpoint in the location that could be accessed by the restore service. Now moving on to the restore sequence. Restore service is deployed. Restore services looks for the checkpoint and waits for the checkpoint to become available if none exist. Once it is ready, restores the container from the checkpoint. Tying in all things together, in the first phase of an incremental approach, we made a design decision to just do checkpoint and restore per node. Now checkpoint service is scheduled as demon pods that would run on every node to run the containerized application and create a checkpoint. Meanwhile, restore service waits for the checkpoint to become available on the node and restores the container from the checkpoint. Henceforth, any new pods that are coming up on that particular node would start from the checkpoint that is available on the node. This should reduce the pod startup time from seconds to milliseconds and from minutes to couple of seconds. Let's see checkpoint and restore service in action. In this demo, we'll scale the application with and without checkpoint restore next to each other to see the difference checkpoint restore makes while scaling the application. We'll use the same Go app that we used in our previous demo and deploy them in the cluster as a vanilla container and as a container with checkpoint restore. We'll scale both the pods and you'll notice that restored pods should scale around 30 times faster compared to vanilla pod. For this demo, we are using the same Go application that we used before that takes around 30 seconds to initialize and listens on the port 4000. Now the sample Go application is containerized that just calls the binary executable when run and the same Go application is containerized and provided to the checkpoint restore service which uses RunC as the base image which has ways to run the container checkpoint and restore. Let's look into the deployment object. So this is a deployment object for deploying Go application as a vanilla container as you see it's using the vanilla Go app image and liveness and readiness probe is set to probe at port 4000. Let's look into checkpoint object. Now checkpoint here is of a type demon set that uses the image that has RunC helpers to run checkpoint and restore. The operation here is checkpoint and we are mounting some properties required to save the checkpoint artifact and get the checkpoint restore properties. Let's look into restore object. It's a deployment object exactly like Go application deployment object but using the image that has RunC helpers again. So the operation to be performed here is restore from the image and location for checkpoint artifacts and properties. This is the conflict map where properties are passed through it basically has a probe to check the readiness of the container running inside the parent container. Now let's deploy all the resources. On the right, you should see there are three ports. First one is checkpoint port that is starting an application basically running the container with the Go web app and waiting for it to initialize. Similarly, vanilla container or vanilla port is running with Go web container and restore container is waiting for checkpoint to become available. While it is coming up, let's scale the vanilla container to two. As you see on the right, both ports have come up ready and now let's scale Go web app restore port. And as you see on the right, the restore port came up ready in couple of seconds and the vanilla port is still waiting to come up because it is initializing whereas restored ports skipped that initialization part and came up ready in couple of seconds. As you see, all the ports are up for the restored port but the vanilla port is still coming up and it is initializing. So you can see how much of a difference this would need. Make if you are scaling 2000 ports or 10,000 ports and every port is trying to initialize whereas with restore you can skip that part and ports could come up ready in couple of seconds or in milliseconds in fact. Tying all things together on a high level, what does it look like to introduce checkpoint restore service for existing application? Please note that this is just one of the ways you could achieve this and there might be multiple ways but the overall concept remains same. Say you have an existing image that runs a Go app. You could create a respective checkpoint and restore image as a wrapper around the existing image. This is one way to hand over the existing app to checkpoint restore service. Now let's take the deployment object for that application. Configurable properties like a way to check in if the application is ready or a way to pass in the container option could be passed as config map along with that respective representation for checkpoint and restore service. If the checkpoint is per node the Kubernetes object for checkpoint would be a demon set here and restore would be a deployment. If not checkpoint could be a deployment or a cron job that runs once per deployment if checkpoint is saved in the external storage or performed as part of CI CD pipeline. This is a respective image used by checkpoint and restore. Checkpoint restore service does require privileges but the actual application container running in set doesn't. So application container shouldn't have any impact. The next one is the location information of the checkpoint artifacts. Here it is a local storage location but it could be an external storage or an OCI registry. Finally, some way to pass in the properties again here it is done locally. With this we wrap up the non-native solution for checkpoint restore in Kubernetes and this zero to checkpoint restore in Kubernetes should give a complete picture of what it looks like. For native support I would encourage you all to attend Adrian Reber's talk tomorrow to learn more on the vision of adding checkpoint restore support in Kubernetes natively. There is also an existing KEP created and owned by Adrian Reber on adding checkpoint restore support natively in Kubernetes. Adrian's talk and KEP focuses on one of the use cases of checkpoint restore called pod migration. It doesn't take on other use cases yet in its initial phase. Signored meetings are also one of the other good resource to involve in conversation about checkpoint restore. I've been involved in reviewing Adrian's proposal and design and have been attending signored meeting discussions around the native support. I highly recommend to check out these resources if this is something of interest to you and I welcome to review and contribute. Let's go over some of the lessons learned using checkpoint restore. One thing to note is that checkpoint restore behavior is sensitive and could have unexpected results if there are additional processes that's added or removed from the container or if there are new network connections at the time of checkpoint. It could especially become challenging if it is an application not owned by the team that consumes it and has to control over what goes into and out of the containerized application. Checkpoint restore has its capabilities and functionalities sprinkled across Cryo, the tool itself, container runtime and even in the kernel. So in the past, we had to look at different layers of the technological stacks to understand the root cause of some of the issues that we ran into. It has been a learning experience throughout this course. Designing it for specific container runtime would lock you down to the specific container runtime and specific implementation, and any breaking change in the basic functionality in some of the other components could break the workflow. For example, there was a change in the way checkpoint location could be specified during restore in Docker. And though there was no change in the lower container runtime like RunC, it broke the Docker workflow. Since we were so tied to using Docker runtime, it took us some time to make a transition to using different runtime or different version of Docker. Also trading off between portability and optimization is a difficult task. The more optimization in place makes portability more challenging that affects local workflows. This might create a gap and allow issues to slip through the cracks. And this could be part of late discovery. So write balance prevents any gaps that would create with developers, local workflow, and the cloud environment. Some of these practices listed down here should hold good for any cloud native application and holds good with checkpoint restore tool. Highly recommend enabling logs and metrics around checkpoint restore. These logs and metrics are really helpful. And at the same time, maintainers are well accessible to discuss if you run into any issues. I'll make this logs and metrics as easily as accessible as possible. Having that repeatability factor and including checkpoint restore in the CI CD pipeline would help for quicker detection of any failures or change in behavior. Since there are different components involved in bringing the checkpoint restore together, always make sure to update to the latest in order to get all the required security patches, updates, and bug fixes. There are some things to keep in mind about the limitations and boundaries of Cryo in order to design checkpoint restore around that. For example, one thing that comes to my mind is FS notify system. We ran into issues with checkpointing a process consisting of I notify, and it makes it difficult for Cryo to resolve some of these handles with the current implementation of FS notify. This is a known limitations and documented, but limitations like this should be good to be aware of. So what are some of the enhancements that we envision in future with a non-native approach? As I mentioned, with an initial approach, we decided to go with making checkpoint available on the node. This need not be the case, and it could be made available to the external storage and save on checkpointing for every node. This is something we plan to take on moving forward. Also, we can go one step further and make it part of CI CD pipeline and get it closer to the integration and avoid late discoveries. Current non-native targets the fast startup use case, but would eventually could be extended to port migration as well. Also the other way, current native support in Kubernetes targets port migration, but it could be extended to support fast or port startup use cases as well. This brings us to the end of this talk, and I'm happy to share that MathWorks is hiring. If you are interested to be part of MathWorks, feel free to use this QR code that should take you to the careers at MathWorks page that list down the job opportunities at MathWorks or get in touch with me. I'd like to thank you all for attending and hope that you got something to take away from this talk. We learned a lot during this journey, and I think this should be super helpful for building a scalable system. I'll be available for Q&A now. Thank you.


Okay, so here's your problem. I'll post the question on the website, and then what I'm going to do is break it into pieces so you can answer each piece. I'm going to solve the whole thing and give you places that you can pause. Remember, you should do it yourself first. If you get stuck, just look at one piece at a time. So that's what we want to do. Okay, so this is a pretty cool problem. Let me just go ahead and show it to you. Can you see that? I assume you can. This is a picture from Google Maps of the intersection between I-12 and 59, and it's got this really nice on-ramp right here. So you come down here, if you want to go from 12 east to 59 north, you've got to go around that loop like that. Cool. I picked it because I've been on that road a couple of times, and maybe you have that too. I don't know where you live, but you probably have something like that too. So what we want to do is calculate what should be a safe speed for cars to go around that curve. Okay, so we'll be like little city planners. So what's important here? Let's draw a picture. So I'll draw a small picture right here. There's the car, and it's going like that, and it's going around the curve. Now in this case, if it's going at a constant speed, it's still accelerating, okay, because it's changing directions. So we need to look at centripetal acceleration. And what keeps it, what pushes it towards the center to make it accelerate, friction. Friction does that. So in this case, I'm going to assume a flat road, and I want to calculate the speed that it can go without slipping. So if I look at this car head on, it would look like this. There's someone driving in the wheel, so the car is coming out this way. I drew it that way so that we could see all the forces acting on the car. Let me just do this in a blue. And so this is the center of the circle over here, okay. And let's call that R. Yeah, let's call it R. And I actually looked up on Google Maps, you can use a little measuring tool. For this case, I get R about 57 meters. It's not completely circular, it has different radiuses, other places too. And we also need to know the coefficient of friction between the car and the road. Now it depends on what kind of rubber you have, what kind of tires, and the road conditions. So let's do this for a wet road where it's the superior. So coefficient of static friction for a wet road is about .4, that's what we'll use. So now what forces are acting on this car at this instant as it's coming out, it's moving this way. So now I have the gravitational force, I have the normal force, let's call it N, and so that the net acceleration up and down, the net force is zero, so the acceleration is zero up and down. And then I have these tires have to be pushing it this way, so there's a frictional call force, I'll call it FF that way. Okay, so let's call this the X direction at this instant, and we'll call that the Y direction. Okay, so now I'm going to write down Newton's second law in both the X and the Y, oh wait, this would be a great place to pause, I made something too for you. So this would be a place to pause because to think about what to do next, I mean what did I do so far, all I did was write down what I had, I drew a picture and I drew the free body diagram, okay. So think about what goes next, and then pause, and work on it, and then come back, okay. And I assume that you've done that, otherwise you're really just, I mean you could be watching something else like, I don't know, TV, but I assume you came back, I assume you worked on it. Let me write down the F net in the X direction, and F net in the Y direction. So in the X direction I have just this frictional force, and I'll put that M, A, X, and in the Y direction I have N minus MG equals zero, because in the Y direction the horizontal is accelerating up or down, so the forces have to add up to zero, and that's not true in the X direction. And in fact I know the acceleration in the X direction, if it's moving, if it's acceleration due to something moving in a circle, this would be V squared over R, that's the acceleration of something moving in a circle, and the direction is towards the center of the circle, which is in the X direction, okay. Can I go ahead and solve this for V? No, I can't, because I don't know what that is, I know, and I don't know what the mass is, do I have to look that up? Okay, so here would be another great place to pause, okay, because what you want to do is think what do you do next, and once you set up these equations you've really made a lot of progress, but what you need to do next is somehow find this frictional force. Well we can use the magnitude of the frictional force is the coefficient times the normal force, and this is static friction because the tires and the road are not sliding next to each other, they're stationary. So if I use this up here then I just need to find N, so I can find N from this, so now I can take this and this and write that up here. Why don't you do that? I know I keep putting up the pause on, but okay you're back. Did you do it? Did you really do it? Okay, I'm just going to trust that you did. Okay, so now I'm going to put these things together, so I'm going to write this as muSN equals Mv squared over R, and then I'm going to use this and get muSMg equals Mv squared over R. Okay, I think you should be able to finish it from here, so and go do that. Okay, so you finished it and you want to check and make sure you did it right. What happened to the mass right here? Look at that. That's awesome because then we don't have to have the speed limit for light cars and speed limit for heavy cars. We have the speed limit, period. So now I can solve this for V squared equals muSgrv equals the square root of muSgr. Okay, so now I can put in my values. So I said 0.4 and then 9.8 Newtons per kilogram and then I said 57 meters. If you do this, you get 15 meters per second, which is equal to, this would be a great place for you to practice your unit conversions, 33 miles per hour, 34, okay, so what should I set this, I'm not going to set 34. I'm not going to set 35, so maybe 30. I would probably set the speed limit at 25 miles per hour because that way what if someone has, what if there's some oil on there or what if the race isn't completely constant or maybe if I put it at 25, people just go 30, okay, so there's your answer. Pretty cool, huh?


Hello, Didier Stevens here, Senior Handler at the Internet Storm Centre. So last weekend I wrote about some captured file that Brad Duncan posted about Cobblestrike traffic and I noticed that this traffic was unencrypted. And now this weekend I took a closer look, wrote a Python script to analyze it and that's what we are going to look at now. This is the diary I posted last week. So Brad Duncan, a fellow handler. He has a large collection of malware traffic and inside his collection I found traffic, so a malicious sample using Cobblestrike and traffic and in the pcap file. And that Cobblestrike traffic was not encrypted and that is because these malicious actors used a pirated trial version of Cobblestrike and then it will not encrypt its traffic. And since it's also HTTP, we can find some things and in this diary entry of last weekend I pointed out some things like for example here I see network data like IPv4 address, network mask, MTU and MAC address. So now in this video I'm going to show you the progress that I made. So you have the pcap file here and I also wrote a parser in Python for this traffic. So you can download this from Brad's site and I have it open here in Wireshark. Now the first thing that I'm going to do is to filter on that traffic on the IP address of the C2 like this and then also I'm going to look only at HTTP, so this filter. And here so this here is the download of the beacon itself, this too. And then you will see a lot of requests like this get visit.js. So that is the beacon contacting the C2 to see if there is anything to do. So I'm going also to filter that out all these requests. So no gets like this. And then you see a lot of answers. Now many of those answers there's actually nothing to do for the beacon. And then you can see that the content length is zero. So I'm going also to filter that one out. And then we end up with this. Here you see some posts. That's when the beacon is communicating its results back to the C2. And here you have some answers where the content is not zero. And that is where the beacon is sending some commands or sorry the C2 is sending some commands. So here this is a long length and you see that starts with mz. So that is probably the answer of the get to download the beacon itself. Here we have a small one. So this is the data that is sent by the C2 to get requests from the beacon. This data here. Now I found out that these four first bytes here are actually a Unix time epoch. And then with that information I found on GitHub repository of tools for parsing actually decrypting cobalt strike traffic when you have the keys to do the decryption. And there was also some parsing in this. And with that I could better understand what we have here. So this is like I said an epoch. Then you have a field with the length here. So 16 bytes. And this is the actual commands encoded. Then you have some padding here for the block cipher encryption with the AES. And then the last 16 bytes that is an HMAC authentication to the authenticator traffic. And here if we look at the post for example you don't see an epoch here. This is actually the size of what is transmitted again and terminated with an HMAC. And then you also have some different fields. And then here clear text like that IP address. So I wrote a parser to parse this. It's relying on Pyshark. So you need to install the Pyshark module, the Python Pyshark module. And then you give it the pcap file. And you just run it. And then you get output like this. So for each output you see the packet number. And if it was an HTTP response or an HTTP request. So you can see the first three are HTTP responses with a PE file. And then MD5 as you can see this is the same. So this is the actual download of the beacon. And this must be the initial download of the malware. And then next you can see an HTTP response 1369. So 1369 is this one here. And when this is decoded here you see the timestamp. So the epoch that is included. And you can see the date here corresponds with the date of the capture. This is in UTC here. Here this is also in UTC and there is a difference. So that must be the time zone. The packet is 16 bytes then. And this turns out to be a sleep command. Command number four is sleep. I found that in that PyBeacon source code. And the sleep command tells the beacon to sleep for 100 microseconds sorry milliseconds 100 milliseconds. And with jitter of 90 percent so variation of 90 percent. So this beacon here will be very responsive. Next there is again command coming from the C2 in packet 6658. You can find here 6658. And that is an unknown command. And it has arguments and this is the argument. I don't know exactly what this means. But in the following request so from the beacon to the C2 that is a post request so 6665 that's the packet number here you can see this is a post submit. So this is posting information back about the command that is executed here. In the PyBeacon code they call this a callback. So I call this also a callback. It's a callback 22 which is to do so the authors of PyBeacon don't know exactly what it is. But I observe that argument that was passed here 0 0 0 6 AD is also returned here. And then you have clear text. An IP address, a mask, an MTU and a MAC address. And then there is also some extra packet data that is not part of the communication. Next there is again a command coming from the C2 which is an unknown command. But there's a large piece of data that starts with MZ. So this might be again an executable 75K. And then another command also unknown but remark that there is a pipe here in the text in the data and that it ends with Portscanner. And next you will see that the post requests that follow are identified as callback 24 which is beacon output Portscanner. So we can see here the actual output of the Portscanner. So you see which machines are alive with our ARP. This one here. And then at the end you also see a machine which has a port opened 445 and this is the name of the machine. And then Scanner module is completed. This corresponds actually to what you can see in one of Cobalt-Strike's videos about using the Cobalt-Strike tool and performing a Portscan. So this is very similar. Next again we have another command that is unknown but again an executable or something like an executable is downloaded. This is the md5 hash and again a pipe and it ends here with bypass UAC. And that too is a command from Cobalt-Strike. You can find also a video on bypass UAC. And next the request, so the answer from the beacon to the C2 is again a callback 0 unknown but so callback 0 this must probably be clear text output because this looks exactly like what I saw in the videos, wrote hijackdll and then yeah okay this is another part of course than what I saw in the videos but this is a clear execution of UAC bypass. And then here at the end we have again a sleep command. Now you can also extract these things here, these payloads and you do this, let me take another command line. So parse option E for extract the pcap file, we get our output and here we have to wait a bit, it's how PyShark behaves. So here now you have the payload to disk. So let's go back here. So this must be the malware and this is the beacon that is downloaded twice, so 9E. So with my tool to analyze beacons 1768, so payload 9E here. You can see that this is indeed the configuration of that beacon and here you see the C2, the visit, the submit, the user agent string that is being used. And then let's look at the command here. So this is the port scan command. So F4, let's see if this is an executable, F4, yes indeed. So this is an unexecutable, let's get an overview, these are the sections. Let's find what's inside, let me see. So this is a port scan DLL, next, actually, let's look a bit more here. Let's look at the exports, I mean, okay, so here, yeah, port scan and here reflective loader. So this is a DLL that performs a port scan and it can be reflected to the world that is loaded reflectively. So that was the port scan and then we also have the UAC bypass, which is this one here, so 9C, so let's check if that too is a PE file, so 9C and that too is a PE file. Let's see what's inside. And yeah, we have a bypass 64-bit DLL, USC bypass, I mean, and also a temp.dll, so that's maybe the payload to execute. And let's look at the exports, so bypass UAC and also reflective loader. So this again is a reflective loader. So this is the progress that I was able to make to analyze this couple of strike traffic, there is still some parsing to do. For example, you have the unknowns here and also here you see bypass UAC, that is 9 characters and right before that, you have 0000 a backslash T, so a tap character and a tap is 9. So this is the length of the string here. So there is also some parsing to do here and this here wouldn't surprise me that that too is the length of the pipe string here. If we scroll back to port scanner, here you can see C, C that is 12 and port scanner that is indeed 12 letters.


Hello everyone, welcome to the NPTEL course on remote sensing and GIS for rural development. This is week 2 lecture 2. In this week, we are continuing our discussion with the challenges and issues of data for rural development. In the first week, we have already defined rural development and how data can aid in the different government schemes and more importantly provide a better outlook of the issues. We looked at lack of good quality data and quantity observation data which impacts the understanding of current and future scenarios which is called the baseline and the future scenarios which are built from the baseline and also we looked at understanding geophysical processes. There is also lack of cooperation between agencies, transboundary states, districts, countries and interstate. It is important to understand that everyone cannot collect the full scale of data. For example, the GANSes basin is there, I said poor countries are sharing the GANSes basin. India cannot be monitoring Nepal's GANSes basin part but the Nepal's GANSes basin part is also contributing to the GANSes flow in India. So, it is very important for us to understand how much water is incoming and outgoing which is leading for agricultural productivity and then rural development. So this cooperation has to exist if observation data is the key and lack of capacity is there as I mentioned for warning systems, models and building resilient prototypes. So moving on, in this week, we have been trying to introduce geospatial technologies and how it is important in rural development. The need comes in from the last slide. We understood that there is a need to augment observation data. For example, you have observation data, you need to add more data to make it full scale both in spatial or resolution. For example, if you have monthly data, can you downscale it to weekly by using satellite data or can you upscale it by to annual using remote sensing data or other data? We will get into how remote sensing can be that in this coming slide. Then there is a need to identify unbiased data. What do you mean by unbiased here? So there should be not any bias for a particular activity. For example, a country which is involved in tourism will always claim that it is a safe country to visit, it is a safe country to visit whatever the issues are or it will say it is a very clean ecosystem. The diseases are very less in this country, please visit. But when you go there and see it is different, so that is the bias nature I am saying. To understand rural development and the resources that contribute to rural development, there has to be an unbiased data and that comes from installation of instruments and the data procurement type, which has to be unbiased. We will come to examples of this. And for sure, from the last slide we say that there is less spatial and temporal data. So we need to identify better resolution data. Higher spatial resolution, so it is not only India scale but also it could go down to district and also states and sub districts, villages scale and higher temporal. So instead of annual and monthly, can you do daily, weekly, those kind of things. So there is a big need for higher spatial resolution data and temporal. I will also define officially what is spatial and temporal. We also need to identify less costly data, which means cost effective also and open source. Open source means everyone can use it. So it may not be only low cost data but also open source where everyone can use it. These are also part of the new data that we are going to use for this course. And we need to identify data with less lag. What do you mean by lag is there is a time lag. For example, if I go and collect the groundwater data in a village, I have to record it in an Excel sheet or a paper, bring it back to my office, work on it and then upload it to the farmers. So this is what a government agency normally does. And how long does it take for data collection to the report generation? It's normally one year or two years. However, within the one or two years, the impact of the groundwater is felt. So what is the use of such a delayed response is a question. So we need instantaneous reports. We need instantaneous data coming in. And that helps you to land for rural development. So there is a big need to identify data with less lag, captures phenomena quickly and converts it to info. So the data itself, if it is collecting time, takes time. You go to the field, you have to travel from say Mumbai to Pune, take the groundwater, come back from Pune to Mumbai and then work on the data and then publish it. Then it takes a lot of time. Rather than that, can you have data collection as a less time consuming activity and faster post processing? Not taking it in paper, not just typing it in a dashboard and then downloading it. Can you make it augmented very fast? And some of the times, you don't have data observation data. Like for example, for COVID, we didn't have specific data for rural population that is migrant. We didn't have density data, population density data. We didn't have the water need data. On those circumstances, you cannot go and put an observation system at once. You need to have proxy data, which is something that tells about it, but it not directly is involved. Normally proxy is the attendance you see in class, where I'm teaching as a faculty and I'm taking attendance. One person gives attendance instead of another person. That is proxy, duplicates. But here, the word proxy is kind of similar that it is not a data collected for a particular phenomena, but it can explain the phenomena. Let's look at some examples. So the first example is India at night. In this image, what you see is lights that is being lit up in the night. It could be your house light, street light, the highway light, anything. So this is kind of an illuminant picture. And the luminescence is taken from satellite as an image. And that is two different pictures, India at night 2012 and 2016, taken by the NASA US Agency. However, if we look at it for long and think, what can this be used for? That becomes a proxy. So for example, I could say, oh, between 2012 and 2016, the cluster size of the cities have become big. If you see Delhi, it has become bigger. You can see this dot. I'll put my pointer up. So you can see Delhi is becoming bigger. You could see all the major cities becoming bigger. One by is light is becoming bigger. And more sub-cities have been created. More importantly, the connectivity is more or less seen. You cannot see that here. There's one dot, dot, dot, dot, but the connectivity is less. But here you could see some connectivity tying up to the cities. So this could be a well-documented evidence that power supply from the Indian schemes, especially to rural India, has increased the availability of power. Only when there's power, you can light up your light house in terms of using electricity. So that could be a proxy data for the efficiency of a government scheme that focuses on bringing electricity to rural India. It can also say where the housings are there. Because there's a house, then you form a cluster. All these cannot be industries and then water coming and light coming from the industries. It could be a group of houses and those are community houses. So now we can also see that in regions where there is more light, maybe more inhabitants have come, more houses have come. And it also shows the economic viability because you have enough money to pay for the electricity. And we have good quality electricity that is coming throughout the night. The timing of this image is also important. We should note that. Is it all taken along the same night time? Let's look at another example. So this is a flood. So this is first, you can see the blue line is a river. And what you see is the Ganges River and another image, different days, between a week, okay? So you could see that the river is flowing peacefully. Small lines are being seen, which are the sub tributaries of the major river. So this is the major river and then you have tributaries coming on the other sides, here, here, et cetera. And the river also flows in multiple channels. However, during a flood, there's only one channel. You could see it's huge and the thickness is much bigger than the initial channel. So what does this tell about? This tells about the regions where the flooding could have happened because normally people would have had houses and agriculture field along the river banks. Now all the river bank is being taken up by the river's floods. Is this sustainable? It's not, right? So those kind of analysis you could do. You could do an analysis of the flood inundation areas, which areas along the river have been flooded. Let's look at another example of proxy data. This was taken by Grace Satellite from NASA and you could see different water storage levels in the ground. So normally the government records groundwater once in every four months or three months, depends on the agency. So let's say CGWP, they collected quarterly. So every three months they collect data and you plot it. So every monsoon, pre-monsoon, post-monsoon, et cetera, you have the data. However, monthly estimates are not available. Whereas in this picture, you could see that the Grace Satellite can take groundwater at monthly intervals and plot it. So this is very important to understand the water storage under the ground. So it becomes a proxy of observation data. The final image I would like to show is the Bangladesh coastal region. And you could see both the Ganges and the Brahmaputra coming into Bangladesh and existing out. While it's existing out, you could also see that a sea level rise is happening. So these are rural entities, these are agricultural lands. What is happening is water level is rising and gets into the land. So sea level, inclusion is there. Sea level rise has taken up a lot of land. And the inclusion not only happens on the surface but also on the ground. So this would not have been met earlier because they would not have data along the coastal regions. They may have for levels, but not on the ground to see if the land is getting inundated. Now you could see that the sea is actually swallowing part of the land because the sea is rising and those are becoming vulnerable for rural entities because those are mostly the agricultural land and the rural occupied land. So what has they been doing on a side note? They have been investing in hybrid rice crops that are flood resistance or they can stand in sea water. It may not be the same rise that we consume every day but what else can they do? The land is limited. The land is being taken away by the sea and so they have to eat a rice that is grown on stagnant water. So these are examples of proxy data or images that serve for a particular purpose. The images would have been taken for just as an imagery. Okay, you can take a picture and that can serve as a data. So for the picture itself but using it for another total angle is called proxy. So this has been used for rural water inundation, land inundation by sea level rise. The first one is rural electrification infrastructure and rural connectivity. Connectivity can be related to both road connectivity, electrical connectivity and also nowadays a communication connectivity like your cell networks, your mobile networks. And the central part is the ground water storage anomalies that can be taken by proxy data. So I want you to think more on how you could use such data for rural development where it may not be taken for a particular rural development initiative but under the proxy clause we can use it. All these are remote sensing images, satellite taken data. So all of them are remote sensing. Let us first define remote sensing because now we are getting into the introduction part. What does it mean? Okay, the maximal definition is given as remote sensing is the acquiring of a data about an object without touching it. So if, as I said, the temperature experience, if I'm the human and I'm having a temperature rise, I need to check the temperature. So there's two ways. The normal way you put a thermometer in the mouth or under the armpit, you take pressure that is touching the object. I'm the object, it is being touched. Whereas in, during the COVID, you would have noticed the gun type, those are remote sensing because you're not touching the body from a distance you're shooting and then taking the temperature. You're sensing the infrared actually, so heat. So this is remote sensing. So remote sensing again is collecting data from an object without touching it. Very, very simple definition. And a lot of examples exist. So your cameras are all taking pictures without touching the object, your cell phones because they are also used to take images, satellites because they take images from the atmosphere down to the earth or from the space to the earth. They're not touching. And now I gave the thermometer example where you have a temperature gun that can take temperature. There are multiple such instruments along the list and some are around the visible spectrum. So that is what imagery is. So if you take an image and the image is taken by remote sensing satellites and other products, there are wavelengths. I'll give you the link to the introductory part of remote sensing. You will learn more about what are the different wavelengths and how light gets reflected from a body and then captured by satellites. But here we're going to know that it's just a tool that collects data and then converts it into information. That can be used for rural development. So now we have defined it. What are the different platforms for remote sensing? So there is ground sensing. These are mostly for rural development initiatives. It could be used for other initiatives also. So there is ground sensing where a person is on the ground and takes data. For example, your cell phone, you can go in and map a crop type or take a crop image and then use it in your database. So that is ground sensing using your mobile phone or another camera that you take. You can also go on a Jeep, elevated Jeep, and take data from a convoy, from a pedestal. That is also ground sensing because you're on the ground. Now slowly you're going to take flight. You're going to detach from the ground. So the first thing that you see a lot nowadays is near sensing platforms. And the near sensing platforms include drones, unmanned air vehicles, UAVs, that are flying in a low altitude and taking data. Then you have some airplanes, some flights that take data and they also fly in a low elevation. Mostly the agricultural flights are smaller in size and they take data. Then you have bigger planes which collect data. For example, there are weather data that are collected by commercial planes. What I mean by commercial is what we carry the passengers from one place to another. So for example, if I'm going to Denmark, I would take a flight, commercial flight. So that flight can also collect data about rainfall, humidity, air temperature along the way. All these are near sensing. They're nearer to the ground compared to ground sensing but not there on on the ground. Then you have the remote remote sensing distance. And that is your satellites. So by definition, all are remote sensing but by definition of distance, we call satellites mostly remote sensing. The drones are called remote sensing under the near sensing category. And also there is ground sensing where remote sensing is possible. There are two types of remote sensing, major two types, okay? Which are called passive and active. We are going to talk about mostly the satellite base but it can be used for cameras and other data works. Why satellites is because that is the data that we'll be using throughout this course. We all know that the sun energy falls on the planet and then falls through a spectrum of wavelengths. There are some wavelengths we can see and those are called visible wavelengths. There are multiple others below and above the wavelength that we are not able to see or record. So however, the earth as a whole can observe most of it and then either observe it and get heated up or re-emit or reflect, okay? So observe, get heated up and then re-emit or just reflect bounce back, okay? So most of this energy comes in and then some interaction happens. So all the two remote sensing if there is a satellite, if you see on the top, the sun's energy comes, gets reflected or re-emitted to a satellite, then it is called passive sensor. The sensor on the satellite is called passive. The remote sensing tool is called passive remote sensing. Why? Because it is taking an energy from a different source which has been reflected, refracted, observed and re-emitted, whatever it is. What are the processes is fine. Your energy comes in, hits an object and then you take the data that is reflected, refracted, observed, what other projects you want to call. Bended, etc. Those are passive sensors because they are not supplying energy and taking the data. On the other hand, active sensors provide their own energy. For example, this satellite has an antenna and the antenna is both a transmitter and a receiver. In this example, the antenna was just a receiver. It was receiving the rays, not transmitting. Whereas here, it is both transmitting and receiving. So it transmits energy and it takes back the energy. So this is a very important satellite because many of you would have known that during night how we collect data when the sun is not there. So those are the times you use active sensors. It is very important for many, many aspects. However, it becomes more expensive because the satellite has to generate energy to send energy particles and then it gets bounced back and reads. Whereas in the previous passive sensor, it doesn't need that much energy because it is just collecting whatever is hitting on the surface. Active needs more energy, which means more cost because new sensors, new payloads, etc. The size is also big. Continuing on the remote sensing topic, there are two types of resolutions we need to talk about. As I mentioned, there is a spatial resolution and temporal resolution. Spatial defined as the size of the smallest feature that can be identified, detected by the sensor. What is the smallest we can see? That becomes your spatial resolution. What is the smallest for our eye? It's not constant because for my eye, maybe I could see a small ant. A very, very tiny ant. But others cannot see an ant. They have to put glasses. They can only see a coin. So our eyes are different. However, for satellites, it sees as a grid because all are digital. The sensors from a satellite sees an image and then converts to grids. The smallest grid size that you could see by the sensor of the satellite or the remote sensing object is called spatial resolution. It's mostly the pixel size. So everyone knows camera phones. So these camera phones are now bought as pixels. Or is it a 10 million pixel?  It's not the entire screen, but the entire screen is divided into grids. And the smallest grid, it's a uniform grid, but the smallest grid size that your cell phone can look at through the camera is called the spatial resolution. So as an example, I'll show you the Landsat data. You could see there is a ground data. And that ground, if you zoom in, it becomes gridded. So all these are pixels. It's pixelated. If you draw along the perimeter, the side, the side of the perimeter of the pixel, which is a square. So if it's a square, all sides are equal. So the side length is given as 30 meters. So people would say normally it's a 30 meter resolution for this example Landsat or 30 by 30 grid resolution. Because 30 by 30, you can say this is the size resolution. Temporal is how often images are called for the same area. So now I'll talk about a resolution of a particular image that I take using a camera. But if every week I come and take, that becomes my temporal resolution. So I'm monitoring a plant. Let's say I'm growing a plant in this bottle. I take an image today. I take an image next month. And then every month, the first I take, so then the temporal resolution is one month. Whereas if I take daily an image and analyze it, it becomes daily temporal reservation. So how often the image is recorded is called the temporal reservation. The satellite orbit repeats cycles. So normally your satellite comes to a particular location and then goes to the same location through the repeat cycle. And those are defines the temporal resolution. How many times data is taken for the same area? So as an example, you can see here, the satellite goes in this orbit, takes an image, goes to the next orbit, next orbit, and then comes back to the same orbit. So the time taken for the satellite to come back to the same orbit is the temporal resolution. Some examples of spatial resolution. You could see it depends on the sensor. It depends on the satellite payload. So you have 300 size, 130. So as the size goes smaller, the pixel size, you could see more detail can be seen. So 300 is too big. And within that just one color is there so you don't see much difference between houses and stuff. Then 100 meters per pixel. You can see some features coming up, but when you bring it down to 30 meters per pixel, you see a lot of more features. Temporal resolution, we already saw this example of a night image and a day image. I'm sorry, night image in two different years. So now 2012 and 2016, for this data set, the temporal resolution was four years. Maybe the day the satellite was taking every day. Then you say that for the satellite that it was used, it is daily or monthly depending on the satellite. For this image which I already explained in the proxy data, what is the temporal resolution? It should be monthly because every month I am taking a data. So remote sensing as a whole is a multidisciplinary. You could see remote sensing used in physical geography where we talk a lot of physics and geolocations, images, but it's also used in social geography and photography. You map people and the biogeography where you map the living things, humans, trees, plants, animals, et cetera. So it's a multidisciplinary by default. There are many resources to understand the basics of remote sensing and as I promised, there are some NPTEL courses. You feel that you need to update before you take my class. Please go through this notes and class lecture videos which are already available online by two eminent professors. There are two different courses, but similar I would say about remote sensing and GIS. This course cannot now you could understand that because rural development is the core, we need to spend more time on rural development. But since we're using remote sensing and GIS as a tool, you could still go ahead and look at this tool for updating or understanding the basics of remote sensing and GIS. With this, I would conclude today's lecture. Even though you don't know remote sensing and GIS, you can understand where and how you can use remote sensing and GIS in this course for rural development. And that understanding would lead you to these materials that I said that can let you know where and how you can learn remote sensing and GIS. Please don't keep that if you don't know remote sensing and GIS, you should not take this course, no, not at all. The basics that I'm going to teach should be enough also. With this, I will see you in the next class. Thank you.


Let's go to Victor in Paramus, New Jersey. Hey, Victor. What's going on? No, I'm March. Ah, FCX earnings come out tomorrow and then look at the symbol X why where do you think that's going? But FCX earnings come out tomorrow. It's taking a little beating. Yeah, so this is gonna be interesting The free three put Mac Moran has been on a tear I mean I see it on there Victor the Bloomberg is saying maybe Thursday pre-market for free tomorrow morning Yeah, this is almost Thursday, right? Yeah, yeah, so you have Let's see. So this is it's gonna be interesting. This is a complex ABC up it check out the B point Yeah, it's like a 15-something. Let's see you get 13 38 1070 so about three bucks which is 250. Yeah, so it's 15 bucks Yeah, so it's a complex ABC structure up, you know I mean I get hit out here today, but it looks like it's gonna reject lower price at this 1232 You know, it's been on a good tear man I mean I my take is that doesn't want to go up to this $15 And I'll see you know that that swing point just kind of hanging out there at both times that it went higher You know, you can see that expansion of volume. So let's still say it wants higher price You go over to the US stale. That's in trouble. That's always been in trouble Is that like gonna turn it back by him still and just go bankrupt and go bye-bye or what do you think? It seems that the you know, they've been losing money forever.  Yeah, and it's still lose the money So well, they're losing money that 2019 2020 2021, right? Yeah You know, but we need a steel company so they can't let them all go out of business That's the other side of us. That's you know, yeah, that's you know, true true So, you know, I wouldn't be buying it. That's for sure I mean this thing looks to me that yeah, I get six bucks on the agenda again. What is that right there eight? Well, 852 first But you can see that this is this has been in trouble for a long period of time And that's regardless of the tax that go put in You know, basically they were supposed to get some help Yeah, the administration basically did everything to keep the sale companies alive and it didn't work because it's Basically, it is what it is. You know what I mean? Which company no alcohol it came out with some bad alcohol. Yeah Yeah, that's been a dog forever. What happens with alcohol is that the Even with the energy cost coming down It's takes so much energy to make aluminum man that they just never had seemed to be able to really get to higher price You know, because they're in a business That's very expensive. Do you think IBM is going after the close IBM? Oh my god. You like all these old dogs That's a divot in there's supposed to be this new technology coming out bottom computing and Yeah, I know we've seen the and the the NFL. They were the main sponsor where they were weekend Yeah, I wouldn't be buying. I mean, this is already telling me is rejecting higher price out here this morning come into the gap Right couldn't hold price go to 126. There's a gap at 126 131 yeah, you get a 128th high ball and we put this on a weekly for a second weekly. They don't close the gap if the gap's down tomorrow And maybe it's the buy Yeah, well, you know, I can see that, you know, that's a big consolidation. I mean, you know IBM's not going away that's for sure, but I Don't think they've they did they certainly yeah one 27th game They had they haven't got the traction and we know that You know, what they should have done is this cloud computing had should have been growing like crazy and it's not And that's a big problem right the board red hat, too. That's yeah, you know, and that's you know They were in there early enough, but they just didn't perform like Google performed in like Microsoft and Amazon, you know, look at the revenue man They're stagnant from 2016 to 2021. Yeah, which is crazy in the business, isn't it? Yeah, and if you then if you take that and then this number keeps going down the number of employees And that's how they keep the at least the you know the profitability Yeah, yeah, somewhat. So yeah So thank God for thank God for share by that, right? That's right. Oh, that's how they've been doing it There's no doubt man cooking brother. Thanks Victor. Have a great one man. Have a safe one


Hello and welcome to the session. In this session we will discuss a question which says that a simulation of flipping a coin 5 times in a row is performed 2 random times. The results of this simulation are shown in the table below. Now in this table in the first column outcomes are given to us and in the second column the corresponding frequencies of various outcomes are given to us. Now in A part we have to make a background of the data. In the B part do you think that the simulation represents a coin that starts with the solution of the given question? Now we have given a simulation of tossing a coin 5 times in 200 trials if possible outcomes appear 1 time only 4 heads and 1 tail appear then 3 appear 54 times. Take a background for this data. Now here on horizontal axis we take outcomes and on vertical axis we take frequencies. We are representing frequencies. Here we write the frequency with scale factor of 10 so this is 0 then 10 20 30 upon the corresponding frequencies of these outcomes. Now let us draw the first one to represent the outcome is 1. Now this small bar is drawn the frequency of outcome 5 heads is 1 so we have drawn a bar of height 1. So here we have drawn a bar of height 5 then frequencies and 2 tails is 27. So here we have drawn a bar and then frequency of the highest bar appears more than head that the tail appears more frequency of number of tails is more than number of heads. The coin is not fair enough because much more than head that's all for the session. Hope you all have enjoyed the session.