Hi, folks. Thank you for coming into this session. And we know what the title is. I am Kaivan Balamoria, and I will be presenting the session on mitigating hackers with hardening. Of course, on our favorite OS Linux, we'll focus more on the buffer overflow because it's still one of the primary vulnerabilities that lead to exploits. So, guys, we'll get started. This is the blabber about who I am. Who am I? OK, so I'll leave it to you guys to read through. Some key things. This is my GitHub site and hang on a sec. This is the one. You'll find, hopefully, you'll find some useful repos over here. And there are plenty of them. It's all open source. Please have a look. Actually, this is something that I'm very happy about and a way of my giving back to the community, which has helped me so much in my career. I have authored these four books on Linux. And from 2018, I started with system programming, moved on to kernel programming, a little bit on drivers, but more on kernel internals, and last year on kernel debugging. So, guys, please have a look. Please buy it. And this is my Amazon author page, et cetera. So again, I leave it to you to check out these things. Several links regarding my stuff on the net and my work in general. But let's go to the agenda for this presentation. So folks, as you are well aware, we want to talk about the basics of security. This really is an introductory level talk, but it is technical. I divide it basically into three parts. Part one covers the very basics, terminology, the current state of affairs, stuff like that. So anyone interested will gain from this. Part two goes into the technical side of things. And we actually start understanding process image layout more importantly, most importantly, the layout of the stack, the stack frame. And then you will get to see why we do this. So that we can figure out how this pretty amazing buffer overflow vulnerability actually works. I mean, imagine having a bug in your code can enable a hacker to get a shell on your system, either locally or remotely. It leads to a privisk, a privilege escalation. That sounds pretty fanciful. So my job is to try and show you how and why this works. OK, now having seen that and convinced ourselves that, oh my goodness, it's scary. What do we do after that? We, of course, move into hardening. Hardening measures to prevent against such attacks. And the good news is in today's world, a lot of these are in place. But not everyone, and not everyone does it. And especially on newer stuff like IoT projects embedded, it's lacking in many ways. So I feel it's like my job to tell folks that please wake up to these realities and fix them. Time permitting, I'll do a quick proof of concept on a virtualized ARM machine to show you how we can change the flow of control via a BOF, a buffer overflow. Now, guys, I don't have that much time slated for this talk. In fact, I'm going to take advantage that this is a virtual session and not a physical one. I couldn't make it to Prague very unfortunately because of visa issues. But I'll take advantage of it being a virtual session and I'll spend a little more time. But we can't spend too much time, then you'll get bored. So I'll dramatically cut back on part one. And it's the easy stuff. So you all can quickly read through that. I will just skim through it. We'll really focus on part two and on part three. I think that's fair. So this is the layout of the GitHub site for this presentation. It's got the PDF doc of what you're seeing here. And it's got code and tools and articles. In fact, it's right here, guys. OK, this is the one. So please do a Git clone and you will have everything with you, including the presentation, which I just uploaded. Great. So this is part one. What is a vulnerability? Folks, it's kind of obvious. So and like I just said, we don't have too much time. Please allow me to skimp past. Please read this in detail at your convenience. But it's pretty straightforward. We often abbreviate it as the word VULN, one. One meaning vulnerability. OK, what is an exploit? An exploit is something that takes advantage of a vulnerability because no point in having a one that you can't exploit from the hacker's viewpoint, from the malicious attacker's viewpoint or the ethical hacker's viewpoint. OK, what is an exposure? So guys, these are kind of the technical terms used in the lingo. So I'm jotting them down. And please feel free to click on the links and read more about these things. It's amazing stuff. We have this thing called info leakage, leakage of information, which enables hackers to do some pretty amazing things. You can't believe it at first. There are physically, I mean, attacks that rely on the physics of a device. They call side channel attacks. In one such attack, which is very recent, this article is June this month. Hackers can steal cryptographic fees by video recording power leads. A very high quality video recording can give away what it's doing. Can you imagine? And such things. OK, being open source, people have written tools. One such tool is called the Linux exploit suggester. When you run this tool and I've included it in the GitHub repo for this presentation, when you run this tool, it's a Perl script. It will show you for your current kernel what exploits are possible. Now, here he searches 5.15 and he says no exploits. But don't believe this completely. When you look up the CVEs and NIST database, they probably have more to say about this. But this is a good way to start exploring these things. So I tried one thing out. I ran the Linux exploit suggester script. And I said, for the 4.1 kernel, please suggest some exploits to me. And lo and behold, I've got three exploits that I can take advantage of and attack such systems. You may have heard of Dirty Cow. It kind of became kind of famous. And here are details. And we'll see what the CVE number means. I mean, it's kind of obvious it describes the exposure. Check this out. Then we say, for the 4.1 kernel, download the proof of concept exploits. And it will ask you, shall I download all or one of them? And you choose. It's like that, guys. OK, let's move along. So you'll keep coming across the term CVE, common vulnerability and exposure. So essentially, it's called dictionary. It's kind of like a database analogous to that. It's a way for the InfoSec community to talk a common language and to say, hey, for the CVE, what do you suggest? So essentially, it's a number describing a vulnerability. OK, so guys, there's a related thing called a CVE, a common weakness enumeration. It's similar, but not exactly the same. And this tends to have more details on the whole deal. I'm in a bit of a hurry. So again, I'll leave it to you to read all of this. This is a famous one. This is an example of a CVE. So this is the year in which it was registered. And this is the number in that year. OK, this is a particular CVE called the Heartblade Vulnerability. Probably you've heard of it. And I'll show you one slide on that. By the way, today when building embedded Linux, we obviously have builder tools because we need some help here, right? We'd all agree. The industry's strength one today is probably Yachto. And Yachto is extremely powerful, but also has a steep learning curve. But do learn it. It's amazing. The other one is Buildroot. Now, both Buildroot and Yachto, they're fantastic products. They have a way to track vulnerabilities to tell you that the package version you're using has possibly got these vulnerabilities. So be careful. And it can even do more than that. So that's fantastic. It helps, because it's easy to lose track of these things. You know how it is, right? I want small and powerful web server on my IoT product. So Buildroot, Yachto, go and get it for me and install it. It's kind of straightforward to include that recipe and to get it built into your root of S. But is there a vulnerability? Do you know? Are you sure it's clean? These tools will help you with that. You don't have to do all the cross-checking yourself. And that's a big deal, guys. OK, let's take advantage of these things. Here's the Heartbleed one. It even has its own logo. Pretty cool. And essentially, it's a buffer overflow bug, the root cards. It's a buffer overread bug. You know, instead of trying to say more about this, I refer you to these links, especially the ComicCon XKCD, which beautifully explains the Heartbleed bug, how it works. Check it out, guys. OK. There's another website called CVE Details, which is obviously details on a CVE. So that's very precious here. I took a screenshot of the page on Heartbleed. And this is the link. And you will get all sorts of details. The other really interesting thing is they associate a score for it. I think it's between 1 to 10, if I'm not mistaken. And the higher it is, the more severe a vulnerability it is. The more easily it's exploited. They have a pretty sophisticated scoring system. So you could look by vendor, product, by store, by date, so many things. It's very helpful. OK. So fine. Now, the interesting thing for us programmers, us developers, and even DevOps people, everybody, really, is that why do we have these vulnerabilities? Well, they are usually vulnerable. They are usually root-caused as programming bugs. And the programming bug typically falls into one of these categories. See, guys, it's not some infinite number of things which cause these bugs, so we can't ever figure it out. It's not really like that. It's simpler than that in the sense that it's very likely to be due to one of these. But of course, these are very broad-based. No doubt you will spot that. But still, it gives us a starting point. OK. Folks, check it out. This book of mine does talk about memory issues and how we can catch them and therefore fix them. OK. From the nvd.nist website, we have a nice page which tells us, look, these are likely to be the root cause of your problems. These are assigned CWEs. And very often your problem is one of these. OK. So that's very interesting here. These things have been formalized over here. I'll show you more. And guess what? Even though it's perhaps one of the oldest bugs, buffer errors are a major percentage of modern vulnerabilities, even today. It's CWE 119. So guys, if you go to this link, right, you can get a view of software development heading and you can look at different views and you can look at different kinds of issues that crop up. While doing software development. So browse through these things. It will help you, for example, bad coding practices, authentication errors, and so on. OK. But let's look at this. Let's look at it in another manner. Let's ask the CVE details database. Show me the top 50 vendors vulnerability-wise. And guys, all the big names in our industry turn up, of course, everyone has bugs, including the venerable Linux OS. Of course it does. And this is the scoring here. So a score of, you know, 6 to 9 is dangerous. OK. So you can see for yourselves. This is some historic data. Fine. Overflow errors are a really significant percentage. OK. The CVE people keep track of the top 25 weaknesses every year. So guys, the previous year being 2022, I took a screenshot. This is the link which will get you there and you can read the details for yourself. OK, you can click on them and read what they are. OK. And look at it. The top one is out of bounds right, which is essentially an OOB overflow. Buffer overflow error. In this case, a right, which is out of bounds. OK. So guys, they're called CVEs, non-exploited vulnerabilities. OK. You can see the whole catalog of CVEs here. Each of these is documented in detail. For example, this one, here's the detail. So if I click on this, we get the complete details or you can see conceptual ops and so on. And these pages are very helpful guys because they will show you the hierarchy of bugs. They will show you the languages it affects, the scope. And I think best of all, examples in C, C++, Java, et cetera. So you read the examples and you start to really understand what is the bug, why is there a bug, and why is it so significant from a security viewpoint. It's explained here. OK. Let's move along. So folks, since we're talking about embedded and IoT, has this actually happened? Have buffer overflow bugs been exploited? Yeah, of course they have.  One of the earliest vulnerabilities is in 1988, can you imagine? The Morris worm, probably the first known worm. OK. Anyway, let's not spend time on all of this. These links are here. Have a look. This is a very recent one. OK, from the risks digest. In fact, it talks about our chat GPT thing. OK. So guys, we all know IoT devices like these lovely things over here. They're nothing but computers running software in disguise. To the layman, they look like a coffee machine or a music player. But to engineers like us, we know that they are computers and they are most likely running Linux or associated OSes. And they're running a lot of code, of which a fairly good percentage is hacky and buggy. And that's the problem, right? Especially because we're in a hurry. This is new stuff. And we want to be first to market. And our managers are saying, come on. We need to get there. Perhaps at the cost of bugs and leading to security failures, which is definitely not a good thing. OK, moving along. Guys, these are examples. I leave it to you to read through. OK, one of the big, it's better today than, say, 10 years back when IoT first came out. So when that happened, right? Firmware folks stuffed in a default username and password into these web-accessible devices. Various ports were open. And the hacker had a many time. Why? Because it either wasn't made clear to the consumer or the consumer had no clue how to change the default password and therefore ran with the default password almost forever. And guys, even today, how many of us take the trouble to change the password? Well, maybe I'm speaking to the wrong audience. I know you will. Do you? So check this out. This guy has GitHub repo with usernames and passwords commonly used as of 2019. Guys, look at this. These have been found in products. And you know what I'm going to say, right? With a distro like Kali Linux, you can run password crackers and tell it this is a database to use to try and crack the password. And who knows? One day, he'll find a match. It's scary. OK. OWASP, I don't know if I'm pronouncing it right, is well known for web application security. And they provide some really good guidelines. So guys, go through these links. Read these diagrams, 0.1, 0.2, 0.3. What is the problem? Why do we have insecure IoT and web applications? These are the problems. There's another article here which gives us this diagram, conceptually similar. But I think it's nicely graphically laid out. Read each of these points and think about them with regard to your product, to your application. Are you taking care of all of this? If not, please re-look. So let's take a practical case and again, a recent case. Last year, so I saw this article recently. And it's an amazing article, guys. Have a look. A real car, it's what's it called? The infotainment system. It was hacked. So this guy was able to hack it. He was able to get the Wi-Fi password because he found it in the logs. OK, now hang on. The car manufacturer didn't do a bad job. It was completely encrypted. But this hacker managed to break the encryption. Read the articles. OK, it's clever. I won't mention which car. You read the article. Now, why was he able to do this? Because 0.6 and 0.7 were violated. OK. Actually, it was encrypted, to be fair. But he was able to break it by getting the key. Fine. Now, why was he able to get the firmware encryption key? You know why, guys? Because the Yocto setup script was deployed as part of the firmware update package. I don't know why. It's a mistake. And we all do it, guys. Don't laugh too much. You may do it one day. We all make these mistakes. I remember many years back, the all-winner folks, they had a similar issue with an Android firmware. They left a back door open. So this is kind of like that and a much more recent occurrence. So what does it come down to? Code secrets were open. And you know what? I would use the word shocking. The shocking thing, the firmware encryption keys were in the setup script. The key values were from example code on the internet. It's crazy. I'm sure heads rolled. You know, the Linux Foundation, they kind of recently, they have a security platform. So check it out, guys. It will give us dashboards for security and help us keep track of these things with our open source projects. OK, so LF is going a lot in this regard. More examples of IoT devices being hacked or attempting to be hacked. So again, I'll leave it to you. The bottom line, we have to perform pen testing, penetration testing. It's not a trivial job. You need skilled people who have been trained in this or have learned it the hard way. You need professionals. If you don't have them outsourced the work, there are many, many companies which are happy to take this up. But you have to do something for your products. So here's another case study on baby monitors which had vulnerabilities that were hackable. So guys, these are examples from some of these PDFs, you know, some research on these products. And you know, many of them are from very reputed manufacturers. OK, the UK government released a Code of Practice guidelines for consumer IoT. And this is a few years back. But I think it's very good. And you know, look at these 13 points. They are an excellent summary of what do we do for our IoT products, security-wise? What must we take care of? OK, so again, with regard to the car hack, if point four was followed carefully, it wouldn't have happened. Guys, kind of similar things over here. So I'll just skip past it. So you know, sometimes people will ask me when I do a presentation like this. Being open source doesn't it make it easier? So the short answer is no. And you know, chat GPT tells us why. And the answer is pretty good here. But there are more points. For small projects, it can be problematic. For big projects like Linux, it's actually the best thing, being open source. So have a look at these points and you'll get the idea. See, guys, this is something the close source people should understand. You're not secure because you're close source, because with reverse engineering and decompiling, which there are tools from the NSA available at our fingertips today, we can reverse engineer your software. We don't need your source code to a good extent. Of course, it's not trivial stuff. But nevertheless, pros, they do this for a living. And you know, the good thing about open source. Right. OK, this is kind of more on the same. Developers have to take responsibility and stop root causing the whole issue of security. How? By eliminating bugs as far as is humanely possible. Notice I don't say eliminate because that's not realistically possible to eliminate every single bug. OK, let's move along.  You know what they say, right? Set a thief to catch a thief. We need to think like that, guys. So our traditional way of thinking is what does the software do? Is it doing something wrong? The hacker way of thinking is what can we make the software do? What can we force it to do? And that's how hackers think, and they can trick the code into doing things that the designer never thought of. And basically, that's hacking. OK.  It even means understanding things to a deep level. Anyway, guys, we'll talk about some of this stuff, OK? So now we come to the next main topic. Now we start getting into the technical side of things. Right, guys? You're still with me. Take a break if you'd like and come back. Let's go for the more technical part now. So what is a buffer overflow? So folks, to understand this, we need to understand the process image and particularly the stack. Here are a couple of points on, is it still relevant in today's world? And the short answer is yes, yes, yes. It is relevant. Why? See, folks, even if we are catching these problems, the fact that they still occur means that we have bugs. And Linus Torvalds has been shouting himself hoarse, as he has tended to do, from a very long time. And he's absolutely right. He says, make this your mantra, OK? It's true to a very large extent. So let's get to the techie part of it. You know, Linux is a Unix-like OS. One of the cornerstones of the Unix design philosophy, everything is a process. If it's not a process, it's a file. OK, a process has a virtual address space. Guys, this is not the time to go into what is virtual memory and how does it work. You can read my books for that. OK. The virtual address space is like a box in which everything to do with the process lives. So what lives in that box? The code, right? The code that we write, it's in that box. So we don't keep the stuff all over the place. The operating system organizes the memory depending on the type of memory. So for example, you write code in a high-level language, like C or C++ or whatever. Now, you compile it. The compiler pauses the language, invokes its code, generates assembly, and the assembler generates machine code. And machine code is all that the CPU understands. So the machine code is called the text. And it's kept in one portion of the virtual address space. The global data, the static data, the variables are kept in another portion. So we have text, we have data, we have other mappings. These are called mappings or segments. Mapping is a technically more correct word. Other mappings includes library text and data, shared memory mappings, and so on. And finally, we have the stack. If it's a multi-threaded process, we have multiple stacks. Because as you well know, each thread has its own personal stack. But we need to visualize this. So here's a visualization. And guys, to keep it simpler, I'm going to assume we're on a 32-bit Linux. So this is the low address. This is the high address. In any case, these are virtual addresses. Never confuse them with physical. The OS takes care of the mapping. And that's not our problem now. Now, within the virtual address space which every process alive has, it's divided into two parts. The user portion of the mapping and the kernel portion of the mapping. OK. The user portion of the mapping is unique to every process alive. The kernel portion of the mapping is the same for every process alive. In other words, every process maps into the same kernel. And that makes sense. OK. Within the user portion, what we call user mode or user space, we've got segments or mappings. Towards the low end, we have the text. This is the machine code. Immediately above it, we have three segments which together make up what we call data, which is, of course, read-write. Text is read-execute. There are three data segments, initialized data, uninitialized also called BSS, and finally, the heap. As you know, the heap is a dynamic segment. It can grow to higher addresses. It can shrink. In between, we have empty space. So for right now, imagine this is all empty space. At the top of the user space, we have another segment or mapping called the stack. It's also read-write memory, but it has a special property. It grows towards lower addresses. It's a dynamic segment, which grows down. This is a CPU property. It's true of all modern CPUs. XH6, ARM, MIPS, PAPI-C, et cetera. Between the stack and the heap, there exist other mappings. The most common ones are those of libraries, because as you know, even the famous KNRC Hello World depends on printf. And printf is found in the standard C library, which is mapped into the virtual address space here, which is why it works. See, guys, the basic thumb rule is this. If it's not in the box, if it's not in your virtual address space, you can't access it. You can't use it. So if printf isn't here, we can't call it. But it is here, because libc or whatever is memory mapped into virtual address space by the loader. So anyway, let's not go too deep into all of this. Just understanding the basics is fine. This is a nice, more colorful diagram of the same thing. So that's why I show it. OK, fine, guys. So for now, this is plenty. Guys, I did one what I feel is an interesting project. I wrote a utility called PROCMAP. With the PROCMAP process map, with the PROCMAP utility, it leverages the Linux's PROC file system. I need to show you guys this. Hey, by the way, this is the code repo on my machine. So guys, check this out. This is Bash. This is its PID. So as I'm sure you know, under PROC, we'll have information about it. And if you don't know about this, please, please learn about it. Look up section 5, PROC MAN page, and learn about it. It's very important. So one of these pseudo files is called maps. OK. And if you cat this file, guys, for every mapping in the virtual address space of Bash, it shows you one line. This line can be interpreted. Start address, end address. These are virtual addresses, not physical. I'm on a 64-bit Linux. That's why the numbers are big. This is the permissions. P for private mapping as for shared mapping. This is the offset from the beginning of the file. It's in hex, major number, minor number of device. I know number of the file, and this is the file. OK. So you know, guys, this is amazing. You can literally see the memory map, the virtual address space of every process. Now, this is brilliant, but it's not very visual. So I thought, let's make it more visual.  It's all open source. It's on my GitHub repo. This is the help screen. OK. I know there's quite a lot. But let me just show you an example. So for the same process, let's do this. PROC MAP minus minus PID equals, and we'll give it PID, and we'll run. And guys, it uses a kernel module to figure out the kernel memory map and the user mode virtual address space, and it shows you statistics. Let's check it out. Virtual address 0, virtual address 1000. This is the null trap page, the size, the permissions, the mapping type, the offset. This is empty space. It's called a sparse region, and we run 64-bit. It's enormous. Then we have a read-only section. Then we have the code of bash. We have another read-only section. We have data. We have another sparse region. We have the heap. It's currently just over 3MB, and it'll grow upwards. We have empty space to allow it to grow. It could even shrink. This address is called the program break, the top of the heap. These are the libraries. So guys, the PROC MAP utility allows you to see all of this in depth. I use it for debugging. I use it for learning, and I'm hoping others take advantage of it. It's all open source, guys. It's here. So please do a git clone. Give me feedback, and please contribute. I've currently got it running on X86 3264, ARM 3264. But of course, there'll be some bugs. Help me out by contributing. These are libraries. There's another big sparse region. Here's the stack. There are some other mappings, virtual dynamic shared object. This is an empty region, and that's the end of user mode virtual address space. On the 64-bit layout on X86 64, we have an enormous hole over here. It's called the non-canonical hole. Out of 16,384 petabytes of virtual address space, the vast majority of it is empty. That's the reality. And from here to here is the kernel virtual address space consisting of various regions. For example, all kernel modules are loaded here on the X86 64. This is the VMLock region. Kernel devs, driver devs, you'll be familiar. We have another hole. This is the low-mem region. The low-mem region is where platform RAM is mapped. On my system, it's 32 gig, but it shows up as 31. And this is actually where the kernel is mapped. I can even show you where. Because of course, I need root, and then I can get these things and show it to you. So I find it very helpful, guys. I hope you do. Okay, so please download and try out the proc map utility. It'll help you even debug and learn things. Let's come back to our topic. So guys, what has this to do with our talk?  Write a function foo. It accepts a username, email ID, employee number, and perhaps writes it into a flat file database. Nothing easier. So you are a good programmer. So you say, hey, no problem, boss. I'll do it, and you do it. So in this trivial example, and guys, I know this is very, you know, trivial. I've written some very simple code, and probably many of you will spot that it's not great. But let's go step by step. Okay, so we have a local buffer, 128 bytes. We ask for the name, and we accept the name into the local variable using a function called get string. It's a C library function. Okay, now, did you notice in the virtual address space there is a stack? In fact, here it is, right? Here's the stack. Okay, why is it there? Now, you've probably learned in school or college that a stack is nothing but memory with special semantics, push pop semantics. Last and first out, isn't it? Yes, but why is it there? We haven't answered the question. So here's the answer to that. We write code using a high-level language, right? But the processor under the hood does not understand when you say call function foo, pass this parameter, return this value. It has no clue what you're talking about. Okay, so the compiler comes to our rescue. The compiler, when seeing a function call invocation, it generates assembly code and it sets up what's called the stack frame or the call frame. It is essentially a way to set up metadata so that our purpose is fulfilled, so that we're able to think in this high-level conceptual way of calling a function, passing parameters to it, doing work inside the function and getting a return value. All of this is a concept that is implemented by the compiler as assembly, okay?  Okay, so guys, let's get into more details. Any variables you have within a function are local and they're allocated on the stack frame for that function. They're also called automatic variables because they come into being when the function is called and they disappear when the function returns. Cool. I talked about these things, okay? Let's move along. The stacks grow. They're a dynamic segment like the heap, unlike text. They grow towards lower addresses, not higher and this is important to keep in mind. Okay, this is a CPU attribute. Okay, this I talked about, why do we need a stack? Because the compiler will make use of it. Fine, how does it make use of it? So guys, it sets up a metadata system known as a stack frame or a call frame. It allows us to do things like this. Access parameters, use local variables for reading and writing, execute the code, return a value. What does the stack frame look like? So guys, I will show you a diagram which will make it easier to understand rather than reading all this text. Okay, but here's a key point. Then there is no one layout. The precise layout of the stack frame depends on the processor. It's a hardware feature. These features are described by the hardware OEM in a document called the ABI, the application binary interface. So people like compiler authors, they make use of the ABI. They have to study it, understand what the processor expects and accordingly write the arc specific portions of the compiler code. I'm not an expert on all this, but that's my basic understanding. Okay, guys, I have a tech blog and I've written a small article on my tech blog on what is the ABI? And I've kept it as simple as possible showing you what it is. Okay, for the Intel 32-bit, 64-bit and for the ARM 32-bit, 64-bit. So have a look when you can. Okay, please do subscribe. I write very rarely. Right. So folks, it's better to visualize it like this. Did you see that? Main calls foo and then the function foo runs, obviously. So what happens at runtime is the compiler and the system runtime, the C runtime, they set up what's called the stack or the call stack. When main is called, the stack frame for main gets pushed on the stack. Now, I hesitate to use the word pushed and popped because it doesn't technically work that way. But it's a conceptual way to think about it. Okay. And main calls the function foo. When it does, the frame foo becomes available on the stack. But did you notice something? It comes underneath. And this is the higher address. This is the lower address. See, the stack is growing downwards. This is the direction of increasing virtual addresses. The bottom, sorry, the lowest memory on the stack is non-intuitively called the top of the stack. And the computer needs to know where this is in order to do work, in order to understand what it's doing now because that's the function it's executing. So how does it know? It's in a register on the CPU, typically called the stack pointer or SP. The stack pointer register holds that value. So brilliant. This is how the computer knows what it's doing right now. Okay. Let's go further, guys. We mentioned the ABI. The ABI designates the precise stack frame layout. Guys, for the purpose of our presentation today, we're going to consider the simpler case. The ABI for 32-bit, x86 and ARM. And this is the layout of a stack frame on both those architectures. Okay. So at the lower address, you'll have the local variables, if any, followed by the saved frame pointer or the base pointer, followed by the return address, followed by the parameters, if any. This is the layout of a stack frame for a single function on the IA32 and the ARM32. Okay. So guys, now let's re-look. We had our code, when our code runs at runtime, the stack frames will get allocated on the stack. And in more detail, this is what it looks like. This is the frame for main, consisting of these metadata items. This is the stack frame for full, consisting of the same metadata items. But of course the size can vary because one may have parameters, one may not. Same is true of local variables, stuff like that. And remember, this is increasing virtual addresses. It grows downwards as the small animation here suggests. Hope it's clear, guys. Okay. Okay. Did you see that arrow appearing? I'll do it again. If you enable a compiler feature, then you can have the child frame pointing to its parent. This allows us to unwind the stack to accurately know who called whom. And that's very helpful for debugging. So this is called the saved frame pointer. Now, guys, this is good to have for the purpose of debug and for learning, but it's an overhead. And therefore in production software, it's usually disabled. And the compiler switch is called omit frame pointer. Let me show you. We all use, I mean, GCC is a really well-known compiler. We do also use Clang nowadays, but let's focus on GCC. Guys, GCC has a lot of options, option switches. If you look up the man page on GCC, it tells you the story. It's enormous. There are hundreds of options and they are described. Look at this one, minus F, no omit frame pointer. Check it out. Okay. So guys, if you use this option, it will not omit the frame pointer because the default is to omit it for performance. If you don't omit the frame pointer, you will get the frame pointer pointing to the previous frame. If you do omit it, in its place will be the base pointer value, which is used as an offset calculation base. The compiler makes use of this knowledge to figure out where are my local variables? Where's the return address? Where are the parameters by doing a plus or minus? It's as simple as that. Okay. See guys, I'll try and make this whole thing practical. Inside the GitHub repo for this presentation, you will find a folder called code. Inside there, I've kept a very simple C demo in a folder called stack C. Okay. I have a very simple C program called stack C. Here it is. So guys, look at this. Main calls testfunk. And testfunk has a couple of local variables. It does a men set and it prints out its return address by using a GCC feature or GCC macro. Okay. Now don't stress about all of that. Let's just look at the basics. Okay. We have a simple small C program here, a little better than hello world, of course. So look at what we'll do. This is the make file and it's very basic guys. The only thing of interest to point out, I am deliberately compiling it for 32 bit. And I will use this version with debug flags so that we can look it up in GDB. And look, I have said minus F, no omit frame pointer. Okay. So that the SFP value is present. Okay. So guys, look, let's do make. It builds. Let's ignore these warnings for now. In production, please don't. It's your job to fix all warnings. We've got the binary executables and it works. Yeah. Of course it works. I just want to reiterate the things we've seen. Within the tool chain, we've got a lot of utilities. It's not just the compiler. One of them is called OBJ dump. OBJ dump is fantastic. You can do a lot. Similarly with Redelf. Now I'll just focus on something simple. Disassemble with source. Okay. So guys, just see how amazing this stuff is. What's the name? Staxi debug 32. Okay. Now I got a lot of output and it looks scary. Don't worry. Let's put it into a file called temp. Now look, open temp and search for main, scroll down. And guys, things start to look familiar. Look, here's main, here's main. And you know what folks? The open brace resulted in this assembly language being generated. Okay. This is the assembly language for the open brace. It's called function prologue. And guess what? This is the corresponding machine code. The op codes and operands corresponding to this line of assembly, this is all obviously machine specific. But guys, this is, this is the bread and butter of the hacker. They don't depend on your source code. This is what they're looking at. So look, here's our calling the function testfunk. You remember? Recall testfunk. So here's our call to testfunk. And it became this assembly language. You can see the call instruction because we're X86. This is the machine code corresponding to each line. It's brilliant. This is exit. This is the machine and assembly of exit. Amazing, right? So guys, in the tech section, in the text section, so guys, in the tech section is all this machine code. Okay. And you know, here's the function testfunk. It's open brace and everything. You can literally see. Brilliant, right? So this is how we must start understanding things at a deeper level. Okay. That is something I encourage you to do. Try and go to the level of the machine to really understand. We'll see more. But folks, let's not lose track of what we're here for. What is a buffer overflow? It's an anomaly. It means something's wrong. When we are reading or writing data to a buffer, we have done the wrong thing. We have overrun the boundary or adjacent memory locations. Okay. Now, practically, in a C program, you do buffer equals ML log of 512 bytes. So we all know how this works. And we visualize a virtually contiguous buffer of 512 bytes. This is buffer 0. This is buffer 511. Now, if you do your job right, you will only access memory within these bounds, but it's possible to do your job wrong because with a language like C, it won't stop you. The compiler says, I don't care. Do what you like. So guys, we can go beyond the bounds of the buffer and we can access this memory for reading or writing. It's up to us. This is called a buffer overflow bug. It's called an OOB bug out of bounds because that's what it is, right? But you can even do it this way. You can do an underflow error. You can go to the negative side. Okay. Reading or writing. That too is an OOB bug. These are called out of bounds defects or bugs. So we can have read overflow, read underflow. We can have right overflow, right underflow. So guys, now look. Let's write a program getData.c as our manager asked us to do. Main calls foo. The function foo asks for your name and puts it in the local variable local. Now, guys, for the purpose of this demo, it looks okay, perhaps. Fine. I know the more experienced folk among you understand that it's not okay, but that's the whole point here. We'll see why it isn't okay. Fine. But on the face of it, it looks okay. So let's run it. Fine. We compile it and look how we run it. Instead of running it like this, I get the shell to give me 16 characters. Four A's, four B's, four C's, four D's. And I pipe it to get data process, which means this becomes the standard input, which means that getS will actually get this data. Now, you know what? We've got 128 bytes. We passed it 16 bytes. It's no problem. It'll work. Guys, I have a suggestion. Before we go into this, no, I will show you what the stack looks like by using this simple program and GDB. Then you will understand this a lot better. So guys, give me five minutes, but I think it'll be very useful. See folks, on the IA32 and the ARM32, this is the stack frame layout for every function. Okay. So watch. I'll show this to you with GDB. That's the cool thing. Okay. So let's not just look at it. Let's do something practical. Let's load it up. Let's look at the code. Now guys, look, this is testfunk. This is main. I'm going to set a breakpoint here. Break on line 32. Why? Because that's where it returns. Remember in GDB, when you set a breakpoint, it will stop before executing that line of code. Not after. Before. Okay. Let's run. We run. It runs. It starts my program. It runs. And it hits the breakpoint. Okay. Perfect. Has something not worked? We should have seen this printf, right? I don't know. It's a bit funny. Guys, for the time being, doesn't matter. No, no, I'm not happy with that. Let's do it again. So it does work. This is the return address. Just keep it in mind. Get into it. Set a breakpoint on line 32. Run it. This time it worked. I think we have to recompile. So guys, this is the return address of the function testfunk. It means this is the address in code text to which this function will return, which means here, guys, before exit, at exit. It's basically line 37 in C. Now look, back trace. You can see that main has called testfunk. Frame zero, frame one. Frame zero is the top of the stack. We've got two call frames like the presentation showed you. Okay. Amazing. But here's the interesting part. We can look up the registers. Remember we've compiled for 32 bit to keep the numbers smaller. This is the stack pointer, but much better. Let's look at 12 words on the stack using the examine command. So guys, now see, look at this code. We call testfunk passing one, two, three, four, four integers. It accepted the four integers. It has two local variables. Now look, this is the layout of the stack frame. At the low addresses, you'll find the local variables. Look, guys, this is the lower address. This is the higher address. Don't you agree with me? But this is called the top of the stack. It's non-intuitive. At the lower address, flag equals ff. Here it is. This is flag. You know what this 40X is? It's just padding. Memset a buffer of 10 bytes to AE. Look, guys, two bytes plus four, six plus four, 10. 10 AEs. These are the local variables.  These are the local variables. Next comes the base pointer or in the presence of no omit frame pointer, the frame pointer. So, guys, this is likely to be the frame pointer. Okay. Or it's this one because after that comes the return address. Now, folks, check this out. Isn't this the return address? 56556258. It's here. So, folks, it's not exactly as seen by GDB, but it's close. We can see that this is the return address, and we can see that it matches the code. So, here's the return address, guys. This is extra metadata, probably because I said no omit frame pointer. This is extra metadata on the stack. But this clearly is the return address. And after the return address come the parameters. Folks, it's clear. Passed in left to right order are the parameters 1, 2, 3, 4. Perfect. Do you see? So, guys, now we have a better understanding of how things actually work under the hood. Okay. Now, folks, it's not as easy as this in the real world because we compile with optimization and then it's a lot harder. Okay. But nevertheless, these are techniques used in debugging at the level of the machine when you're out of luck. Okay. Or you don't have symbolic information. Let's come back to the presentation. So, guys, now we can understand when we pass 12 bytes to get data, isn't this the layout of the stack frame on ARM32 or IA32? And we've got 128 bytes. So, look what happens. Cheesy animation. Now, look what happens. Hex 41 is A. Hex 42 is B. 43 is C. 44 is D. They fill up the low 16 bytes because remember, this is the low address. This is SP, top of the stack. Okay. We've got 128 bytes. We've got a lot of free space. And after that, it returns and then it exits. So, everything is fine. It's perfectly fine. So, guys, this is the good case. Hang on a sec, folks. Let's quit out of this. Let's go to the program, simple BoF because that is the code I've just been showing you. See? Call foo. foo has a local buffer of 128 bytes. It returns, it says okay, and it exits. And that's exactly what we did here. And it was perfectly fine. Correct? Okay. Now watch. We're going to overflow the stack. So, see, guys, instead of passing a safe number of bytes within 128 bytes, deliberately pass more. Let's pass 128 plus 4 plus 4 plus 4. In other words, 128 plus 12, 140 bytes. So, won't we overflow this? Of course we will. So, how do we do it? Any way you like. You can do it manually on the command line or you can make use of Perl or Python or anything you like. So, over here, we make use of Perl and we pass the letter A 140 times as standard input to get data. So, what will happen? Think about it. It will first populate the 128 bytes. Now, that's fine, but now look. Now it starts to overflow. It wipes out the SFP or the base pointer. It then wipes out the return address and it then wipes out the parameters and then it seg faults. You will get segmentation fault and if co-dumps are enabled in brackets co-dumped. Guys, wait. We need to do these things, not just talk about it. Guys, don't worry about all the different versions. Let's just take the basic version. If I don't overflow the buffer, it's perfectly fine. Do you agree with me? It's perfectly fine. Do overflow the buffer? Okay, hang on. What was that? Perl minus E print A 140 times. What am I doing wrong? Yeah. This way pipe to get data. Guys, now look. All right. This is modern Linux for you. Modern G-Lib C. Stack smashing detected. It detected a stack smashing attack. Guys, this is a hardening measure that we talk about later. Okay, I'll come to it. But it works. Now let me do this. This is the version of the same program compiled without such protections. So let's use that version called less product. And this time it seg faults as expected. But what is seg fault? Seg fault means the OS is saying, I can't do what you're asking me. I can't access memory where you asked me to. Guys, I'll show you why it's actually very clear. The processor tries to return to the designated return address. The designated return address is now a junk value, a bogus value. Hex 4, 1, 4, 1, 4, 1, 4, 1 in all likelihood. It's an empty or sparse region of the virtual address space. In fact, if you use my proc map utility, you will literally find that to be the case. It's an empty region. The... It's very interesting. When you look up a virtual address, the virtual address goes to the MMU. The MMU, the memory management unit, uses the paging tables of the process to perform runtime address translation. Translating the virtual address to physical, but it fails. Why? Because this address is not mapped. So it raises page fault. The MMU raises a page fault. It goes to the OS's exception handling code. And the OS's exception handling code running in process context says, you have made a bug. Kill the current process by sending it sig seg v. And we call it a seg fault. That is what has happened. Okay. So it looks like a bug. It is a bug, but there's more to it. And that is the really interesting thing that I am yet to come to. So hang in there, guys. Right. Now, look, I've shown you, right? It seg faults. Fine. Guys. No code dump. Let's set up the code dump as unlimited. Run it again. This time we get a code dump. Pretty cool. Why is it this big name? Because of this. Okay. It's all good stuff. The man page on core. And it explains all this. Okay. It's amazing. Now, why did it happen? What is wrong in our code here? I mean, we know we did something wrong. Let's, let's rewind. See, guys, we, we passed much more data than we should have. But shouldn't it have elegantly said, you have passed too much or something like that? It should have elegantly backtracked, but it did not. It crashed. And that is a problem. See what happened. This is what happened. It wasn't able to prevent this, which is really bad. We allowed the return address to get overwritten. The EBP, the parameters to get overwritten, it tried to return to this return address and it crashed. It seg faulted. Who's fault is it? Who's fault is it? This guy. I'm sure many of you know this. The get us is a very poorly written API. It's a very old API. Unfortunately, this wasn't forcing. And guys, that's the trouble with API is right. Once they're written, they're written in stone and they live forever because production code out there is using it even today. So it's a problem. We can't just remove it. David Wheeler is a very famous security consultant. In fact, he, as far as I know, he is with the Linux Foundation and he was even heading the OSSF at one point, you know, secure programming stuff. He wrote this free book, open source book. You can download it via this link. Okay. And look what he says. See users must avoid using dangerous functions that do not check bounds and get us is one of them. In fact, guys, this is not a surprise.  Where is it? Look here. Everyone knows about it. So what I'm doing is a very contrived example. No doubt, but it illustrates the point. Okay. So guys, it's not just the get us. There are many more APIs in C. And, you know, especially those of us that learn C 20 or 30 years ago. I'm sure there are many of you like me, old fellows. We get locked into those old APIs and we just keep using them. And that's a terrible mistake. So guys, don't you use string copy, string cat, Esperant F. I mean, Esperant F here who doesn't use it and all of this stuff. They're all wrong. They're all dangerous. They're all vulnerable. They all cause buffer overflow. What do we do? Don't use them. But don't use them means use what in their place. So guys, here's the basic funder. Don't use get us use F get us. Don't use string copy, use string and copy. Don't use string cat, use string and cat. Don't use Esperant F. Use SN print F because the N says there's one more parameter, the maximum number of bytes you can pass. So see guys, get us is a very bad idea. Don't use it. Okay. It's deprecated. Fine. Use what F get us. Why? Because look at the signature. Look at the prototype. The second parameter. It's a good API. Read the man page. It reads in at most one less than size characters from stream. So if we had used the F get us, we could have said sizes 127 bytes or 128. And it'll be fine. Okay. Similarly, don't use Esperant F. There are so many related routines. Don't use those that don't have the end. Use this one. Why? Because the second parameter is the maximum to allow. So guys, for God's sakes, if you don't take anything else away from here, take this point away. Okay. We have to reorient. We have to reteach ourselves. Basic C programming. This is not C programming. This is the library.  Security wise. Let's call a spade a spade. Don't use them. But I know you, you'll tell me, but hey, I'm on a legacy project and there is plenty of usage. What do we do? So there are hardening techniques. There are static analyzers which will catch these things. So let's use all of that. In fact, David Wheeler wrote a very simple static analyzer called floor finder. So it catches the bug straight away. Guys, now you understand. Just Google for CW e 120 and CW e 20 and see what the problem is. And you will learn. Okay. So these things are very critical here. And I think all of us as programmers must understand it. And you know what? It's not just Linux, right? It's, it's kind of over-sagnostic guys. Okay. Because hackers don't care. They'll catch you anywhere and everywhere. Fine. So it's a, it's an important lesson to learn. So the Linux foundation, no, I think just two or three years ago, they, they started the old SSF, the open secures was security foundation. It's to do exactly what we are talking about here to educate developers on secure coding techniques. Guys, go to open SSF.org.  They have working groups. You can contribute. They have town halls. And guess what? They've got a free training course. You can, um, you can take this course online course and you can get certified. Okay. Good stuff. This is from, uh, this is not mine. This is from David Wheeler's presentation. So guys, this is a physical buffer overflow. What do you say? Yeah. Now we, we do have a buffer overflow. It crashed. Yeah. We all got that. Um, let me once again reiterate. It crashed. Seg fault code. So what? It's not dangerous. It's just a bug. Fix the bug and you'll be fine. That's true guys. But it is dangerous. Why? Guys, why did it crash? It crashed because we gave a faulty return address. For one, for one, for one, for one hex a, a, a, a. And it was bogus and it caused the crash. Yeah. That's kind of obvious.  He or she crafts a return address pointing to code that they want executed. Guys, that is a crux of this presentation. That is the buffer overflow attack vector. So they won't pass some junk value like this. They will pass some meaningful value.  They land up with arbitrary code execution capabilities. So now watch instead of doing it like this, which causes a seg fault, maybe the hacker will study your program and do this. Guys, this is an address, but because of Indianness, you know, right?   So we might need to rearrange this, whatever it is. If this address is a crafted address, you know what that means? It's pointing to code that the hacker wants executed. So the hacker is very clever. The hacker is saying pass 132 is which will overwrite the stack and the base pointer and the next location because he 128 plus four. The next location is the red address. The hacker knows that. Overwrite this. Sorry, override this, override this and then override this with a special value. This is pointing to code that they want executed. Okay. That is the crux of it guys. This is known as a crafted buffer. It's also called an Easter egg. So guys, as an example, see, instead of just blindly passing hex 41, maybe we also pass this value. So see what happens. We overflow the SFP or the base pointer, but then we overflow the return address with a special value. It's pointing to code that we have put on the stack. I haven't yet shown you how we'll see. It leads to arbitrary code execution. So see guys, this stuff that we put onto the stack is known as the payload or the crafted buffer or the Easter egg. In other words, we are injecting machine code onto the stack, keeping the return address so that we come back to that machine code and execute it. That is the attack vector. Isn't it clever? Okay. Hang on a sec. What code are we going to execute? So see guys, what is the holy grail of the hacker root? We want a root shell. So to get a root shell, if we can successfully execute these two lines of C code, we'll get a root shell. The set UID system call sets the real and effective user ID of the process to zero, which means it's running as root. Now, of course, if you're not running as root already, this will fail. That's a security measure. But if you are, it will guarantee you're running as root. So if the hacker is able to attack a process that is running as root, which many set UID root processes do, then they can get a root shell because this is the next line of code. It spawns a shell. It execs a shell and the shell will be running as root. If this doesn't succeed, the shell is not running as root. It's running as the UID of the process, but it's still valuable. The hacker now has a shell with your account on your machine. Guys, this code put into assembly is called shell code because the objective is to get a shell. You know what? You don't have to do all the donkey work of getting this in assembly, although it's not hard to do. If you go to shellstorm.org, they've got it ready. Here's an example. See, guys, they are calling the function shell code via function pointer. C programmers understand all this. And this is the content of the array shell code. It's nothing but the machine language of set UID 0 exec bin S H. He's put the assembly language as comments so that you can understand what's going on, but it's only the machine language that actually goes on the stack. Isn't that amazing? This is an attack vector, guys. Okay. So that's how it's done. Now it's not just shellstorm. You can go to exploit DB, which is run by a very popular and, you know, good site offensive security. They have many training courses which are excellent on pen testing. Here's a link to shell code. Shall we try it out? See, guys, this is all shell code. Okay. Some of them work. In fact, all of these are unverified. So we are not in luck over here. We'll have to search by verified, but there are, there are lots of them and you can find it by, you know, taking a little trouble. There's probably a way to filter. Okay. I don't remember. Okay. These are not very good, but there's a way to find it. The Google hacking database has shell code. So guys, Google around and you'll find it. So, so here are some examples of verified ones. See, here's an unverified one, which adds, adds a root user with no password into the ETC password system database. And it's just a few bytes of shell code and it's being executed here. Amazing, right? Okay. So guys, that is one way to do it. It's known as injection of code, but there's another way to perform this buffer overflow hack. The other way is to run internal program functions like a secret function, which the user doesn't expect you will run. The programmer doesn't expect you'll run. We are forcibly changing the flow of control. So these are known as control flow attacks. Okay. CF. I will show you a demo of this on a virtualized arm machine. Cool, right? So let's get into that next guys. Okay. I will run a demo of how we will do a, what's the called internal program flow control flow attack using the buffer overflow vulnerability. In fact, why are the get us API? Okay. So folks, let's leave the presentation like this and let's move on to this demo. Now guys, I found something very interesting. Let me show you the demo in, in my code base. It's, it's in this folder. Buffer overflow proof of concept. So let's go to BOF POC and this is the content. This is the program. So let's look it up. Here we are. Guys, I've written detailed comments so that it's easy for you to read and remember. Okay. But just look at the code guys. It's extremely simple. In main, we call foo passing ugly of one. So that's actually this original idea I got from a YouTube video. It's not mine. Okay. So credit to that person.  So you can pass anything you like, including a buffer, which will overflow because all that the function foo does is call get as and getting local. But look, local is only 12. Okay. We are putting more into it. More than that, because, you know, it will read from standard input, right? So it will just take whatever you're passing it now completely away from all of this. There's a function called secret funk. So imagine this is some product and we have a so called secret function. This function does something over here. Of course, it's just a contrived thing. Okay. You know, the typical hacking catch the flag contest. Okay. Do you see my point? Nothing in this code called secret function, but we'll make it get called. Okay. That is the attack. We will call this function even though it's not being called. That is the attack vector by using the buffer overflow by exploiting that vulnerability. How the heck will we do that? Huh. So see, guys, you, I have shown you the code now. So now see my make file is fairly sophisticated. Oops. I made a mistake. Here's the make file. See the make file has many targets. Okay. Now look at this. This is called fortification. I'll talk about it later. This is stack protection. So if I use these, no, the attack will fail. If I use strong protection, the attack will fail. Guys, please use this for production. We want attacks to fail. These are hardening measures, compiler based hardening measures for God's sake, use them. But look at this. We will deliberately use less protection. If we do this, right? We are in effect saying don't protect it, even though modern Linux protects it by default. That's the good news. But we will purposely build a less protected version of the binary executable. Watch. Make bof one less protect. So guys, it runs it with these GCC flags, which ensures that it's not protected. Here's the binary executable. Now look, if I run it, you won't see anything unusual. Okay. It works. But what if we overflow it? So see what I did. I kept one simple ASCII text file called input one bof. Look at it, guys. One, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve days, four bees, four seas. How many bytes? 16. I'm sorry. 20 bytes, 20 bytes in total. Okay. Now the size of the buffer is how much? It's only 12 bytes. Okay. So if I send this now, it will overflow it. Okay. I'll show you something. If I make the regular version, if I compile the regular version and I run it, so this is the regular version that's going to overflowing its inputs. It does not get attacked. It detects the attack. Stack smashing attack detected and it deliberately terminates and co-dumps, allowing you to debug. See? But you know what, guys? With the less protected version, that won't happen. Instead, it says illegal instruction and it co-dumps. But that's not an attack. No, no, no. That is not an attack. So now I'll show you an attack. So see, folks, here is such a simple idea. This is our binary executable, correct? You agree with me? Now, in this binary executable, it's the compiled version of this code. So won't there be a function called secretfunk? Can we look for it? Of course we can. This is an easy way to do it. NM is main. So, guys, this is one way to do it, but I'll make it easier. Much easier. Awesome. This is the text address of the function secretfunk. And, guys, this is without debug info. So, you know, you can even figure this out in production. So this is the text address. Just note it. Let's do something. Let's note it. OK. We've got a note of it. Now, see what we do. I built one file called input2. And I'll show it to you in hexadecimal. Because if I show it in ASCII text, you can't read it. So I'll hex dump it. And, guys, oh, OK. It's ending in this. But I wanted to end in this. So I need to edit it. So I'll do hex. OK, good. Let's do all this stuff during the presentation. OK. Let's install hex edit. Now let's do this. Now, see, I've got 481620. And then I've got the virtual address of this function. But I have to update it for this value. So, see, guys, because it's a little Indian, I have to get this value. So, you know how I do it. 040111B6. So first put B6. Do you follow? Because it's a little Indian. So B6110400. This is what I want. So now I have updated the hex file. I must save it. Control, how do I save it here? Control X. Save changes. Yes, it saved it. Let's verify. Guys, isn't it now updated to what we wanted? 0401111B6. So it's in little Indian format. It's updated. Now watch. OK, guys, now look. Run the vulnerable program taking as input this buffer and look at the output. It failed. Excuse me. No, this is not how it should have worked. So guys, I think I've perhaps screwed something up in the hex dump. Let me do this manually here. It's really the whole point of the demo. Let me do it manually and I will show you. So let's do this with the debug version so that we can, sorry, let's go step by step. I build the debug version, then I load it. OK, now watch. Guys, I just realized something. I don't want you to think we need the debug version. We can even run without debug. It should work. I'm going to disassemble the function foo. So guys, this is the disassembly. OK, I realize I'm doing something wrong, not as per the demo I planned. The demo I planned was to run it on ARM, no? So let's switch to this window. Yeah. Guys, over here, I'm running on a Yocto built ARM machine. Shall we do one thing? Shall we start from scratch? I'll power it off. OK.  Ah, now I'm on my X86 64-bit Ubuntu. Now, guys, I wrote a shell script to use Quemu to emulate an ARM machine and the root FS and the kernel, all of that is from Yocto. And just, yeah, see, here's the kernel. OK. Here's the root FS. It's all over here. I'll show it to you. It's all over here. I did this a while back. OK. If I run the shell script, you will literally see an ARM machine booting up. It's a guest. It's running the 5.4 kernel. OK. Watch. See, these are the print case from the kernel. It's booting up. Over here, you can see it's this. You can see it's an ARM 32. OK. You can see that the device tree is present and it's loading up. So, guys, it's booted up. It's mounted the root FS and it's run the login program. And it's for development. So, there is no password. And now we are on a root shell. So, the first thing I do is get myself a bash shell, because I find it easier to work in. And I have a shell script which I source, which has aliases like L for LS minus L, which just make it easy for me. Yeah. And I already have all my stuff in the virtual machine. So, guys, now we run an ARM 32 bit. See. OK. We run an Cortex A15. For all practical purposes, we run an ARM machine. And now I have generated the program. I mean, we could do it again if you wish. It'll work. The compiler is installed on this particular Yocto build. OK. So, everything is there. Now let's run. In fact, now, guys. I think I set it up correctly over here. Give me a moment to test. Do you see the address? OK. Now see. Exdump input 2. Ah, guys. I have done it correctly over here. 010400104AC becomes this in Little Indian. I think I did it wrong there. That's why it didn't work. Now watch. BOF 1 less protect, less than input 2. Did you see that? This is the whole point, guys. We entered the secret function. Isn't that amazing? We got here even though it wasn't called. How did we do that? I'll show you. But, guys, I hope you got it. This is the demo. OK. Haven't we re-vectored control? Out of the normal program flow of control. It's an attack. It's an attack. Now watch. GDB minus Q. BOF 1 less protect. We don't need the debug version. Disassemble the function foo. Guys, this is ARM32 assembler. It's got a push and a pop. This is obviously the return where it's popping off the value into the program counter. So you know what I'm going to do? I'm going to set a breakpoint on an assembly instruction. We can do that. Now I run it with input 1, A, A, A, B, B, B, like that. I run it. It hits the breakpoint. It stops. I can see the back trace. I'm in function foo on frame zero. I can do info frame. And you will see details about the stack frame. This is all really good for debug purpose or for learning purpose or for hacking. Okay. Now. Check this out. P dollar SP. This is the stack. X slash ATX. Show me eight words of the stack. Guys. This is B. This is C. Because input 1 BOF has four Bs and four Cs at the end. But what about the legal part? Just do minus 12. Wait. Wait. Let's. Oh, no. This is perfect. Look, these are all the A's that we passed and this filled up the buffer because look guys, we've got only 12 bytes, but we have overflowed with four Bs and four Cs. Now see what have we overflowed? Do you remember? In fact, I must update this comment over here. I think it's a useful comment, right? This one. Let's get it updated even over here. This is the frame layout. The stack layout. So see guys. In ascending order from lower to higher. This is the layout. So we've got locals BP return address params. So we've got the locals. This is supposed to be the base pointer. Sorry, this one. And this is supposed to be the return address, but we've overwritten it. Do you see? So watch what will happen. This is the correct program counter. But see what we will do. We will step instruction, one instruction at a time. And look what has happened. The program counter has changed from the correct value to the wrong value because that's the value we gave it. This was on the return address. So it went here and now it will crash. Amazing, right? You can literally see what happened. But guys, a seg fault is not an attack. It's just a bug. So let's quit out and this time let's attack. So see what we'll do. Same thing. Set up a break point on the return. This is the equivalent of the close brace. Read in any data as long as it overflows. Now it's overflowed. Now we look it up. We look up 12 bytes before so that we can see everything. Now, if you think about it, no guys. This is the base pointer. This is the program counter return address. This is what's going to go popped. It'll get popped into the PC. This value. So guys, check this hack out. I'm going to set this memory address. But wait, this is C6. This is C6 plus four. Isn't that CA? I'm going to set it to null. It's become null. So when I step instruction, the program counter. Okay, it's slightly off. But it will still fail. It will still say for it. Okay. So guys, now let's do the full attack. See this time I rely on this. I wrote it down over here, right? So now watch. We set up a break point. We pass some data overflowing the stack. You know that it has been overflowed. This is the interesting one. So we set this value, this address, which is. Hey, I made a mistake, is it? See guys, I don't want to make assumptions. Assumptions are the killer. See, let's put it in hex. Let's do this plus four. Yeah, it is a BCA. Let's set this to guess what this value. So it doesn't like that because it's actually illegal memory. So guys, I think trying to show it to you manually over here. It did not work in this instance, but it's a bit funny. I have done this earlier. What if we change this to six? No, it didn't work. Forgive the demo guys. But you know what? I've put all this information into this document for you all. And I copy pasted it to show you. So allow me to just scroll down and show it to you. See, here's the secret function address and here's where I got it overwritten. Yeah, you know what? I have an idea. Probably it will work this way. Instead of using input one, we use input two. Input two has the address at the end. You remember guys, I did the hex dump. So now see. Now what are the values? Guys, now it's what we want. See, now it's the address or I hope it is of the secret function. Actually, it's not perfect. Yeah, something seems to have gone wrong slightly. But let's give it a shot. It worked when I did step instruction, we're in secret function. Amazing, right? Guys, it worked. So that is the demo. That is the proof that it actually works. Cool, guys. I hope you enjoyed this. Please try it out yourself. Okay. So folks, what I'm going to do is I'm going to give myself a short break. And come back and continue the session. Okay. So I'll see you soon. Thank you very much for listening and hope you'll look at the next part, which is the hardening measures. Okay. Interesting stuff. I'll see you there. Thank you. Hi guys. Welcome back. So we now move into the second part of this presentation on mitigating hackers with hardening on Linux. Focus on the buffer overflow. So in the previous part, we had looked at how to, I mean, we looked at the terminologies and the basics, CVE, CWEs and all of that. And more interestingly, we looked at process image layout using the proc map utility stack frame layout. I'm 32 X86 32 bit. And guys, a little Googling or looking at my tech blog will show you the layouts for the 64 bit as well. No worries there. And coming back to the point, because we understood these things, we saw the power of a buffer overflow. We essentially were able to overflow the return address with a crafted address leading to execution of a payload of code where we wanted control to go. And that essentially is the buffer overflow vector. Essentially how it works. So I showed you a demo on an, on a virtualized Yachto built arm 32 bit Cortex A15. And it actually worked. We found that we could re-vector control as we saw over here. It entered the secret function. So guys, let's move along. So these are just screenshots of those attacks. See, by the way, when you compile on Ubuntu Linux, I found that these are the compilers which is automatically passed by the compiler on exit 664. I wrote a utility to figure out these things. It's no big deal, but nevertheless, I won't talk about it now. It's probably going to be an upcoming talk. Okay. So I'll save it. This is one of the shell scripts. It's on the GitHub repo with which to try things. And you can see in the default case, Linux catches it, which is great. So what we do is we weaken the compile and then it works. Okay. There's this utility called Checksack, which I'll show you. It's able to show us these details. Okay. And this time it actually goes through. But in terms of demo showing that we can re-vector control to arbitrary code, this is the relevant screenshot running on the virtual machine. So you can see we got the function address. We incorporated the function address as the crafted return address in the buffer. And therefore, when we ran, control went to the secret function because we overwrote the return address with the value we wanted to. So amazing stuff. So guys, we've seen this. Now the second part is external library functions can be re-vector. Why not? See, if you can place any address, why not place the address of a library function? Now you might of course say, for what?  Think for a moment, guys. Haven't you come across this function? It's a library API system. The parameter is any Linux command. Anything. You put it as a string. PS minus A pipe to grep. But why not just put bin SH or bin dash? So guys, think about it. If we can get control to go here and we have control over the parameter string, we do. We can execute a shell or anything we like. We can set up a backdoor to enter that system on command. And that is the external library flow attack. This kind of attack is called the red to lip see hack because we are literally returning to lip see. These kinds of exploits are also called ROP exploits or hope return oriented programming. Okay. So folks moving along. I look up CV details.com historic data. Linux kernel as the product. Did you think there were no vulnerabilities in the kernel that there were only vulnerabilities in user space? It's not the case. There are plenty and a significant percentage of buffer overflow. I mean, besides buffer overflow, it's denial of service. Okay. DOS attacks. So it's a factor. Software is software. It has bugs. It has vulnerabilities. It gets attacked. Even if it's an operating system. Okay. Now, of course, the beauty being Linux and open sources. There's these things get fixed at a fast cadence. Anyway, just to give you a flavor of it. Again, you can look up several vulnerabilities in 2022. These are kernel vulnerabilities. Okay. And some of them have a fairly high score. In fact, they are privileged escalation attacks. But now guys, let's come to the, you know, another portion of key content. How can we counter measure? How can we harden the OS, the kernel? So and applications. So all this is to tell you that it's important to do because Linux is very important these days. And guys in a nutshell, these are the hardening measures. We look at each of them in some detail. Some of them we will focus more on as it's more relevant to developers. Okay. So guys, this is it. You, you do these six things and you're much. You dramatically improve the security posture of your Linux product or project. Okay. Now, of course we, we must do this, but check this out. Greg K H. Maintainer subsystem maintainer of a very long standing. Clearly tells us your main protection is to use the long-term kernel. In fact, to use the latest stable kernel. This is a code by Kess. Kess cook is essentially in charge of security at Google Android. If I'm not wrong. And he essentially says the same thing. So guys see, we all understand that especially with regard to enterprise and products, it's not always going to be even possible to use the very latest kernel. So we do the best we can. If you can manage to keep using the latest stable kernel. It's the best. But you know, especially for embedded products, IOT and enterprise products. That's very hard to do because we know, right? The cadence is extremely fast. Every two to three months, you'll have a new release, a stable release. So are you really going to do a firmware update for the whole OS that often it's unlikely. So the next best thing becomes the LTS kernel, which is of course the long-term stable kernel. The cadence is approximately one year. The last kernel at the end of the year stable kernel becomes the next LTS kernel. We are currently on 6.1 as the LTS ones. Okay. You must use them. You must at least use the LTS kernels. So, you know, embedded Linux builders like build root and yokto. There maintainers will tend to pick these kernels, which is great, which feeds into this whole ecosystem. Okay. However, there are companies and projects that are aiming to do even better. The LTS kernel will be kept stable for a minimum of, and I'm not kept stable. They will be updated, maintained for a minimum of two years. And as of now, the maximum is in the region of six years, which is, which is a lot. It's a lot of effort, right? So for example, 5.10 and 6.1 LTS, they are going to be maintained till December, 2026. If I'm not mistaken. That's wonderful. So let's quickly look up these things. In fact, I keep it bookmarked because I always want to look up these things. So see you guys. 6.1.x is the LTS kernel. December 2026. It will be maintained until then. That's it's a UL date. Okay. So wonderful. But guess what? There is a Linux foundation project adopted by a left called the CIP. And these folks, which includes companies like Toshiba Hitachi and more, they maintain SLTS kernels, super LTS. So the 4.4, the 5.10, for example, 5.10 SLTS, they want to maintain it until Jan, 2031. Excellent. So if you can use them, it might make a lot of sense because they are promising to put in the bug fix and security fixes right up to that date. So you keep feeding into that and then feeding it to your customers. Sure. I mean, without say, I mean, there's no need to say it. You need a really strong update, automated update path, right?  Fine. Moving along. So guys, now let's look at the hardening countermeasures. So the first one I mentioned, it's not too much in our control, but nevertheless, I shall mention it. If you can use a managed programming language, it'll save you a lot of headache security-wise. So we all are used to using C, C++, and it's great. There's nothing one can say that, you know, it's outdated, definitely not. However, it means we have to handle memory, which creates a lot of bugs there. Anyone programming for long enough on these languages knows this. Programmers manage memory. Programmers create bugs. It's almost like cause and effect. Okay. Unfortunately. But yeah, that is the downside. On the other hand, managed programming languages and Java is well known. The .NET framework, C sharp, they are well known, but nowadays even Rust has become kind of a big thing because from 6.1, we've got the bare minimum support in the Linux kernel. Linux isn't really running via the Rust programming language. Oops. It's C, by far the vast majority of code is C, but now there's module support for us. So slowly, it will creep in. And that's not a bad thing because it has this advantage of being security-wise superior. Okay. Now, they're in fact called memory safe languages. Okay. Now, guys, all this is great because a large amount of zero-day exploits, so-called zero-day exploits, are root cause to memory corruption bugs, which is exactly the problem with non-managed languages like C and C++. But the reality is we heavily depend on C, C++. All projects use a mix of languages. This includes Android. It uses Java, managed, but under the hood, C++ and C. The Linux kernel is mostly C and the middleware is mostly C++. Okay. Anyway, so guys, setting that aside, let's move on to the next one. This is a really important one and something we can do about, I mean, that can help us immediately. Compiler level protection. There's a lot of stuff we can do with the compiler. Our modern compilers, GCC and Clang, they've become extremely powerful. So folks, one of the first protections that have been available now for a long time is exactly the attack vector that we have been leveraging. The stack attacks. There is protection against this called the stack canary protection. It protects against stack smashing attacks. So these are older implementations. You can see they're really old. And from a long time, we have these flags that you can pass the compiler. So guys, again, do man GCC, look up minus F stack protector. In fact, bottom line, right? Please use it minus F stack protector strong. Use it in your make file. See, they'll always be ifs and buts. They'll always be corner cases where some project says no, we don't use it. But by and large, you should try and use it. Of course, there's a version dependency. GCC 4.9 onwards. Apparently you get the support not on older versions. So if you're using some really old tool chain, then you're out of luck. Hopefully, you know, 2023, we now are using modern tool chains, which certainly support this. Projects like Android have been using it from a long time. Okay. By the way, we saw how we call the secret function in effect diverting control flow. So we've got control flow integrity checking built into modern compilers. Use this option switch CF protection control flow. Ubuntu uses it by default. So guys, here's a snippet from the man page man GCC. You can see the stack protector flags. You can read the details. Okay. Now one thing that I am not very clear about if we use any kind of optimization. It seems to defeat this protection, but I'm not 100% sure on this. It's something I want to check if any of you can give me a heads up. Appreciate it. Okay. More on the same thing here. Stack protector. How is it implemented? Guys, it's straightforward. See, this is your basic stack layout, right? CT li is control information. Essentially, it's the stack frame pointer or the base pointer. Okay.  This random value is placed at runtime compiler generates code to do this at runtime. This is the function prologue at function call time. This is done when the function is returning. The canary is checked. If its value has changed, the runtime code determines that a stack smashing attack is in progress and it aborts the program. In fact, we saw this working, right? I will just show you again to remind you. So you remember, guys, if I use even the regular version and I try and divert control to the secret function, the attack is prevented. And why did it get prevented? Because of this feature. This is the feature in action. Okay. Because it was protected via compiler. So it's a hardening technique. Great. The overhead is minimal. You can try this as an exercise guys. Use it and don't use it and see the difference. By the way, the Linux kernel uses something called VMLock allocated stacks in recent versions. And that also helps as a stack guard. Okay. Even otherwise, it helps. So they are called V map stacks. It's another good feature to turn on. Okay. Next. It's crazy. Can you imagine the printf format string specifiers like %s, %d, %f? They can be used as an attack vector if you are not careful. Can you imagine? So I don't want you to just imagine. Let's try. So see guys, I don't have time to show you everything. Please have a look at the simple C program. In fact, there's nothing to it. Okay. Now this is the program. That's it. But watch this. I run a wrapper shell script which performs an attack.  It prints out 16 words of memory from the stack. In other words, no guys. We have done info leakage. You know what the shell script did? It did this format string issue %s, %s, %s 16 times. It gets these addresses. Okay. So guys, how come there is a problem here? We are allowing it to print anything it likes. Okay. So the fix for this is to use this GCC flag, minus W format security along with error format security. Treat this warning as an error. In fact, no guys, I'll show it to you in this next screenshot. When I compile with minus W format security, it flags this as an error. Why? Because this is the bug. Guys, did you notice we didn't give printf %s of one. We just did printf %s of one. So there's no string literal. There's no format arguments. The compiler catches the bug. Because this can be leveraged by a hacker. So now it treats it as an error and you are saved. But if you don't compile with this flag, it goes through and we can peek at memory. And being able to peek at memory in the stack is a major problem because we have information leakage. Okay. It is a security issue. Okay. Let's move along. Folks, you remember in the last video, when I explained how buffer overflow occurs, I said, but what API is at fault? And quickly we saw it's the getS, the getString. It's a poorly written API. We should avoid using it. In its place, we should use fgetS, getLine, stuff like that. In fact, folks, I didn't mention it. Even scanf is dangerous. If you do scanf %s, it can be overflowed. However, if you do scanf %20s, you are limiting it to 20 bytes. Okay. Keep that in mind. Coming back to this. Now, knowing that certain APIs, especially the older legacy ones in the C library, are vulnerable to these buffer overflows or underflows. There is one very interesting compiler measure called fortification. If you define this macro, fortifySource, and you give it an integer value, the compiler will generate code to do checking. It will use wrapper routines around these common APIs, allowing it to check in the C library, are they okay to use, or are they considered buggy? So guys, it's a protection mechanism.  The next slide shows us how to use it. So when you compile, you're supposed to compile like this. If you define fortifySource S2, you're supposed to give optimization level as minus of 2. Now, very recently, GCC12 has opted to fortifySource equals 3. You can look up this link, guys. In fact, there are a couple of articles that are very good. This is a good article. This is an older article, and I'll show you one more. So use these nowadays, yeah. Use this fortification. It's one more level of defense. Okay. Sorry. Yeah. No, no. I missed something, guys. Okay. Maybe this is the article I'm talking about. This is a recent article on fortification with GCC. Please have a look. He even gives an example. And he does mention that you can use the later version. Okay. Cool. Let's move along. So the next compiler protection, in fact, this is linker protection, but it's all part of tool chain. So it's called railroad, which is relocation read only. So folks, I'm sure you heard of this. You know, the ELF binary, the binary executable file, it has been attacked by hackers writing into sections which direct control to library functions. So if a hacker is able to change the address inside the binary executable, he can re-vector control to his own code. And obviously he can then take over. So to prevent that, right? We have a way to determine that after the process, C runtime loading has been done by the loader program, ldlinux.so or ld.so. We make the binary ELF sections read only the binary headers so that a hacker can't change it via the program. Okay. So guys, this is known as railroad. So using these linker switches, you put it in your make file. This is known as partial railroad. This one is used known as full railroad. Obviously this is the superior one to use. So I'd suggest you use it. Okay. Android uses it. And I'm just going to come to a utility called CheckSec. CheckSec is a bash script. It's a pretty powerful script. It allows us to check our binary executables and or our processes for hardening measures. So one of the things it checks for is railroad protection. So guys, it's from a developer point of view or from build team point of view. It's important to do this. But from a QA point of view or an audit point of view, how do I know is this binary or this process safe enough? Then you use utilities like CheckSec. So guys, let's move into a sidebar using CheckSec. It's a nice project.  You can clone it. Okay. It's fairly old, but it's still pretty powerful. So if you run CheckSec, this is the help screen. Why don't I quickly demo it? So guys, let's go into the repository tools security. I've put a lot of stuff here. I had mentioned this one in the last video. And there are many more. So for now I go to CheckSec.HS. I mean, SH and we're up to date and we can run it. So folks, it's a shell script and you can do stuff like this. So let's take an example. So we have the PS utility. So see what we can do. We can say CheckSec minus minus file equals and we can give the path to PS. So look at what CheckSec does. It shows us about PS. So guys, see what it's saying. It's full railroad, which is great. That means full protection on linker sections, ELF. It's got a stack canary, which means what you remember. This is the minus F stack protector. Probably the strong version. Okay. So stack smashing attacks will be detected. This I'm going to explain. This also I'm going to mention our path and run path are traditionally environment variables that are considered unsafe. So over here he's saying they won't be taken into consideration, which is good for security. No symbols means the binary executable has been stripped. Symbolic information is not available inside it, which is good for security. It has also been fortified. So guys, we just saw right minus defortify source equals two or three. So it has been fortified. Apparently four functions have been fortified and nine functions are fortifiable. Guys, this stuff, right? There is a bit of doubt. The CheckSec program seems to yield some false positives and there is an issue raised on their GitHub site against this. Even I find that it's not completely reliable. This information on fortification. So take it with a pinch of salt. Okay. It doesn't work fully. I'm hoping some fixes will come in. Okay. In fact, I've mentioned the issue in the subsequent slides. You can do a lot with CheckSec here. Here's an example, several examples of using CheckSec. See guys, interestingly, I ran it on a Windows program, any desk, which is running on Linux, probably with wine emulation. It didn't have a stack cannery. It doesn't support program independent executable, which means it can be relocated if PIE is on. For most of these, it's on, which is a good thing because it leads to randomization of addresses. But over here, it's not on. Over here, it's not fortified. As another example, the password program, it has all the protections, but it shows up in red color. This means that it runs as SecuID root. I'm sure you all understand. When you run this program, any user, it internally runs as root. It runs with an effective user ID of zero. This is dangerous because hackers are on the lookout for exactly this. Why? Because it's a nice juicy target to attack. Because if they can successfully attack it, they get a root shell and not just a regular shell. So guys, in today's world running SecuID root is considered legacy, bad practice, old stuff. I'll just say it without going into details. Please read up on using the POSIX capabilities model. It's the modern way in a nutshell. No, guys, read this man page, section seven info on capabilities. Read about this and start using it. It's a much superior thing. I won't say more now. Let's move along. Another way to use the CheckSec script is to specify a directory. It will pick up all the binary, I mean all the ELF objects and it will show you their protections. So these things can be useful for audit checking here. You can run these scripts and get a report. Fine. Even nicer in a way, you can even run CheckSec on processes currently alive, not necessarily program files by saying proc all. So if you run it like that, for processes currently alive, it shows you these things, which is interesting. So again, as an audit check, this could be something that you do. By the way, guys, we didn't come across this till now. Seccomp is a facility, a way to filter system calls, a way to say this process can only issue a certain subset of system calls, which is good for security. It's a hardening measure. Chrome is a very good example of an application using Seccomp. But you can see there are many others also. Hey, by the way, they mentioned that look at this documentation in the kernel source tree. But guys, this path name is the old one. This is the modern one. I've updated it here. So just use this to look this up. Let's move along. More on using CheckSec. So here I mentioned on fortification. You can check the fortification of a file in detail. However, this is not considered 100% reliable. False positives show up. There is an issue raised on the website. This is supposed to be the fix. But I found it doesn't work in full. So why MMV? CheckSec can even check the security posture of the Linux kernel to a certain extent.  So guys, we don't cover all of these things. A few of these I will be talking about, not all of them. And while we're on this topic, let me say there is an even more powerful script to check kernel level security. I'm going to come to it. But even CheckSec does a decent job. So guys, shall we try it out? See, CheckSec minus minus kernel. You can alternatively give the pointer to the path name of the kernel config file. If you just say kernel, it will check for the current machine. So guys, I am running this kernel because this is Ubuntu Linux on NX86 and it has found most of the things to be fine, but not all of them. So that's a good input. It's not completely hardened. Okay, cool. Because this is a desktop. Folks, I will skip this. But just to say, there is another script called Hardening Check. It's similar to CheckSec. It's just one more. Okay, it's a Perl script and it gives similar information to CheckSec. So it's kind of like an alternative to CheckSec. Check it out. On Ubuntu to install it, do sudo apt install devscripts and it gets installed. And then you just run Hardening Check as I've shown over here. Okay. So fine, guys. Running Hardening Check on a less protected executable. You remember we had done that and it finds problems. So, you know, it does work. It's able to detect that this is not very well protected. All right, great. Now I come to a... I would think a really more important thing. See, guys, very broadly speaking, no? Broadly speaking, there are two kinds of analysis that we perform on software. Static analysis and dynamic analysis. Static analysis is like the compiler and it gives you warnings and then you fix them and you are better. You've eliminated some potential bugs, some potential vulnerabilities. There are static analysis tools which do the same thing. They find potential bugs slash vulnerabilities. They warn you about them. The problem is they can be too verbose and give a lot of false positives, but they're worthwhile looking into. They are very much worthwhile. You must use some static analyzer. There are lots of them. Okay. You must use it. But what's dynamic analysis? It's runtime analysis. Now, there are many. You've probably heard of and used Valgrind. Valgrind is a very famous memory checker tool. It catches many memory bugs. But guys, in today's world, Valgrind is like the older way. In fact, Android has pretty much stopped using it. And in place of Valgrind, we use a contribution from who else, Google engineers to the compiler. And they are called sanitizers. So these have made their way into the modern compilers, GCC and Clang. These are the versions, guys. It's relatively recent. You need to be running recent compiler tool chains. And then you'll get this advantage and it's a fantastic advantage. Don't miss out on it. There are several sanitizers. This table tries to summarize them. Among all these sanitizers, in my opinion, perhaps the most powerful one is this one, address sanitizer, abbreviated as ASAN. It's a memory error detector. It catches memory bugs at runtime. So how do we turn it on? It's a compiler switch. When you compile your code, right, you do minus F, sanitize equals address. And then it's turned on. See, guys, man GCC, minus F, sanitize equals. And you will find them. It's not just sanitize equals address. There are many more. Okay. In fact, there are too many to see at a glance. You will find many of them. And they are very powerful stuff, guys. You should take advantage of them. There's a thread sanitizer to catch concurrency bugs in p-thread applications. It's a data race detector. There's a leak sanitizer to catch memory leakage bugs. There is an undefined behavior sanitizer. It's abbreviated as UB. UB SAN. It catches undefined behavior, which is essentially stuff like divide by zero, integer overflows, things like that. Okay. And related. So there's a lot of this stuff. Please take advantage of your compiler. There are several things that you can use. Data race detector, leak sanitizer, UB sanitizer. Now ASAN became so well regarded on Linux that Android made it the de facto way to compile and check for bugs as part of QA. And so do dozens and dozens of Linux projects. And guess what? Google engineers ported it to the Linux kernel. Marco Elver of Google, if I'm not mistaken, is maintaining it. In fact, I had some discussion with him where we found some issues with one of the runtime checking scripts. I forget now where exactly, but we had got something fixed there. So anyway, it's been ported and it's called KSAN, the kernel address sanitizer. It's supported from the 4.0 kernel for XH64 and 4.4 kernel for ARM64. It has caught a lot of bugs in the kernel. Guys, take advantage. Okay. Now it does mean you need to recompile your code and run all your test cases. But that's what we are here for, guys, to do work. So do your work. Okay. Do it. And catch bugs. And I told you, right? Many of these bugs are vulnerabilities that become the root cause of security issues. So it's a great idea to catch them. So in fact, my Linux kernel debugging book, this is a snippet of a page from there showing you that KSAN catches many types of memory bugs. See, guys, there aren't that many types of memory bugs. We have the UMR bug, which means what? You are using a variable uninitialized local variables. It's so common, but it's still deadly. So, you know, fix it. Make sure you use something to catch it. The compiler catches it. Static analysis catches it. Okay. OOB bugs. KSAN catches them. Slump debug catches them. See, this book is about kernel debugging. So I only talk about kernel side. UAF, use after free, use after scope. Very common, very dangerous. They are caught by these. But you have to turn them on in the kernel config and you have to run your test cases. There's no shortcut guides. UAR, use after return. Similar to UAS. Very important to catch them. Double free caught by KSAN. Leakage caught by Kmem leak, not by KSAN. Okay. So it's like that, folks. ASAN catches even memory leakage. So you have to do it. Okay. Righto. So this is how we do it. We use minus F, sanitize equals foo, where foo is one of these. You can't use all of them. They are mutually exclusive. You have to use one at a time. So it's more work. Okay. But it's very important. So guys, my system programming book has a series of test cases. All memory bugs are tested with and without ASAN. And you will find that ASAN catches most of them. So the code is all free here. You can look it up. David Wheeler writing an article about Heartbleed. He says ASAN is nothing short of amazing. It catches these bugs where pretty much everyone else didn't catch them at the time. Okay. Guys, this is more on stack. I'll leave it to you to have a look if you're interested. The third one in hardening techniques. Again, not too much in our control is what libraries do you use? So guys, usually this is decided beforehand. It's mostly G-Lib-C, good old G-Lib-C. But there are others. And the problem with G-Lib-C, of course, is that it's carrying a lot of legacy baggage, old APIs, which are really bad. So if you are using compiler protection, you will probably catch those bugs. If you are using hardening measures like fortification, you will probably catch it. That's good. But, you know, can you use a better library? If you can, then please do. So Android has been looking at this and they have made some enhancements. I don't know the details, guys. I just mentioned over here that you must make it a habit to use many tools. Static analysis, there are lots. Compiler protection, there is a lot. And use superior libraries. So you see, guys, I think you're starting to see. It's not just one thing or two things. You have to do many things. It's called depth and defense. That will put off the would-be hacker. Okay. One or two things they'll overcome. But if you have so many defenses, then there are probably easier targets out there to attack. And they'll say, let's go find them. This guy is too hard. Okay. That is our goal. Coming back to libraries, there are many libraries as a substitute for your traditional jealousy. In fact, muscle is a well-known one that was used many years back, a very small and compact. So, you know, we even reduce the attack surface by having a small and compact library. And it excludes the vulnerable APIs. So guys, you can read up more on these things. It's not fully in our control in most projects. Coming to the fourth point. We start moving into the domain of hardware protection. So think about it, guys. One of the most common attack vectors is to put shell code onto the stack or even on the heap. Those are known as heap spraying attacks. Wire and overflow are an underflow bug. Right? And we saw how those work. So the hacker can do privisk, gain privilege escalation, a shell, a backdoor, et cetera. It's a bad thing. Now, guys, think about it for a moment. The reason that the code on the shell or the heap runs is because it's able to run when the program counter points to it. But what if we can change the program table entries at the level of the OS paging tables, saying, don't run code on the stack or the heap or in the data segment. Refuse to run it because we won't mark it as executable. So guys, this is a hardware... I mean, this is a hardening countermeasure where we mark pages to be non-executable. It's abbreviated as NX, no execute. And this is supported on modern microprocessors. Intel, AMD, ARM, POPPC, they all support it. So guys, this is a huge thing because if we can guarantee this is enabled, then even if a hacker gets shell code onto the stack or the heap, no use, it won't work. So that's the brilliance of this thing, guys. Okay, so you know the key principle behind this? I mean, this is really the interesting thing. These are called WXRX pages. You understand. Executable pages are not writable. WXRX. This is implemented on modern Linux kernels. So guys, again, try and use the latest stable kernel you can. You will get all these protections. Okay, it's a big deal and it's taking advantage of hardware because it works at the level of hardware, at the PTE level. By the way, the Linux kernel has LSMs, security modules that load up early at boot. It's part of the kernel. It's not really a module in that sense. They can turn on these features, WXRX. Okay. More on this. These are processors that support them. So you can read about a few more details over here, guys. This is how you can check whether your machine has an X, a one-line check. By gripping PROC CPU info, or you can run this utility on Linux, check BIOS and X. Okay, cool. Let's move along. More on executable space protection. There's a lot of terminology, guys. I mean, this is where marketing comes in. Intel markets it as ExecuteDisable, AMD as EVP, Microsoft calls it DEP, ARM calls it ExecuteNever. They're quite smart. They have all different words for the same thing. Okay, but now to a more important thing. Recent Intel has come up with SMEP bit in the CR4 register, control register 4. ARM 2 has it, and they call it PXN or PAN, supervisor mode execution prevention. If this bit is turned on, if the SMEP bit is turned on, when code is running at ring zero privilege, which means at OS privilege, if the kernel code even tries to execute a user code, any page in user space, the MMU will raise a fault. In other words, we won't allow it. Why? Because guys, this is a favorite way to devise a kernel attack. The usual kernel exploiters map some shell code in user space, find some kernel vulnerability to exploit it, to be able to override kernel memory to point to that user mem location and run it. So kernel code will run something like exact bin SH and the hacker will get a shell. But if the SMEP bit is turned on, it won't work. So guys, again, at appropriate places in the kernel, modern kernels will leverage this functionality and it will make sure this bit is on turning off a whole class of attacks. See, guys, to deviate slightly, I subscribed to and I've done some very, very small contribution to the KSPP mailing list, the kernel self-protection project. The whole point of the KSPP project, which in fact is led by Cascook, is to try and defeat whole classes of attacks by setting up some protection measure in the kernel. So a lot of this kind of stuff has first been merged into the KSPP project and then merged upstream into Mainline Linux. That's the whole goal of the project. And therefore it also makes its way into sister projects like Android in terms of the kernel. Now, there are some close source projects which for some years were open source, GR security, PAX, which have done all this much earlier. The next thing to say now, guys, if you use one of these protections, the hacker, the experienced and motivated hacker can likely defeat it.  But if you combine SMEP with many other things, then it gets harder and harder for the attacker. Okay, moving along, Intel came up with an extension of SMEP called SMEP, Supervisor Mode Access Prevention. If you even try and access a user-made mode page from kernel mode, it will trigger a fault. This defeats a whole other class of attacks. It makes it even stronger. Now, of course, this is controllable. And I've mentioned the machine instructions that control it on the XZ6. Linux supports it from 3.8 onwards. John Corbett, as usual, has written a fantastic article on LWN. He's the editor. Linux Weekly News is a precious resource for all of us, guys. Please read the article for details. Right, folks. Give me a second. The fifth OS hardening countermeasure is called ASLAP. ASLR is the abbreviation, address space layout randomization. And you can see what they mean by that term. See, guys, if you use one protection, let's say NX. You remember? Non-executable pages. Wonderful. Arbitrary code injection will no longer work. However, think about it. What if I devise a Red 2 Lib C attack or a hope attack? Return-oriented programming. I'm trying to use the system function in G Lib C to execute something. It will work. Why? Because it's legal code. It's not going to be marked as non-executable. So it will work. So, you know, the hacker gets over this problem by attacking in this manner, by using the Red 2 Lib C vector or the hope vector. And guys, if you just do some Googling, people have written about these exploits. There are YouTube videos. There are plenty of tutorials and videos. You can check them out. So how do we protect against this guy? ASLR is, I won't say a protection. It's a kind of mitigation against these attacks. You know what ASLR does? Every time we run a process, it randomizes the memory regions inside the process by loading the text at a different offset, a random offset. Okay. That's all it is, guys. Okay. So it's not an optimum part at a different offset. Then everyone else also is offset that much. The data segments, the heap, the libraries, the stack. So what does it bias? It randomizes the layout of the VAS. So the attacker doesn't know in advance where the system function resides, which earlier he could figure out. But now he can't figure it out so easily. When you run a process, it changes. That is the protection. But guys, if you think about it for a moment, it's not protection. It's mitigation. It just makes it harder. But by a brute force approach, a hacker can figure out the random offset. Okay. It's not that hard, in fact. But moving along. They found that ASSLA is pretty good. So it's being used from 2005. The kernel config is called config randomize base. You should definitely have it on. However, from 2014 and the 3.14 kernel, we've now got CASSLA. Kernel address space layout randomization, which means what? The same thing for the kernel. So guys, even the kernel image is prone to these attacks because of information leakage. So by every time you boot the machine, if we can offset the start address essentially of the kernel text in memory by a random value, then the hacker won't know in advance the layout of the kernel. And therefore it becomes harder for him or her to attack. That's called CASSLA. So guys, you should enable it in kernels. Okay. I've already talked about these things. So I won't repeat them. By the way, just turning on ASSLA is not enough for a user mode process. Why? Because it needs to be built as a position independent executable minus FPI. If it's built like this, it won't work. Okay. So guys, a check set will show you. We had seen that earlier. Okay. Coming to the kernel side of it. No, sorry. This is to do with user space. This is the control knob. The control switch, which of course is under proxys. These are known as sys control features, S-Y-S-C-T-N. So under proxys kernel randomized VA space, let me show you on my machine. I get the value two. Now, what does two mean? So see guys, two means that ASSLA is on for pretty much everything. This is the OS default. You can turn it off by setting it to zero. So if you have root access, you can turn it off. Okay. So guys, you can use check set to check these things. And I wrote a small script called ASSLA check. You can also use it to try out these things. So I'll leave it to you to try out. Now, even though we have these mitigations, they are not foolproof by no stretch of the imagination. Because folks, the reality is for a clever and motivated hacker, even one kernel pointer becoming visible. If the hacker knows the virtual address of he has the variable name and the virtual address or the function name and its address. That is all he needs to be able to in effect reverse map the entire kernel layout. Guys, it's because like when an enterprise kernel or IoT embedded kernel ships, it's going to look the same. It's just offset by a different amount. So if I can fix on one thing, I can find all the rest. I can calculate the position of everything else. So, you know, these aren't great protections, but it's just more. It's just one more thorn for the potential hacker to overcome. See, it's like that. Now leakage is a big problem here. So let's take the example of kernel space. There's a lot of stuff that leaks addresses. Perhaps the best example is this. Have you heard of K all sims kernel all symbols. Now, guys, there is some protection. If you don't, will you run as root? All the addresses show up as zero. But if you do have root and, you know, a hacker is going to somehow get root, then you get the actual addresses here and you don't have protection any longer. You have access to all of this. So are there other ways? Yeah, there are other ways. The message, the kernel log might reveal stuff. So nowadays on Linux, you need to be root to run it. Otherwise, it won't run. Okay. It ran just now on my machine because I just ran super. Okay. Otherwise, it won't run. So guys, um, Boben Harding, um, added a Perl script called leaking addresses. I think we can find it. This is the kernel source tree on my machine. Here it is. Uh, he added this Perl script some years ago. And, uh, okay, that's good actually. So it's finding all sources of leaks. Okay. In fact, I helped out a little bit, uh, helping him add 32 bits support. Okay. Uh, for this script. Anyway, um, See guys, just as a real world example, very recently last month, I saw this article. It got registered as CW 532 insertion of sensitive information into a log file. So guys, this is essentially an info leak. Um, Samsung smartphone users were vulnerable to this vulnerability because hackers were beginning to exploit it. Okay. So an advisory was issued. And, you know, the nice thing, the day I was editing this slide talking about this, I got a security update on my Samsung phone. So that's, that's pretty cool guys. Okay. They, they addressed it quickly and they sent an update. Okay. Fine. So, uh, folks, this I've mentioned earlier, yeah, you know, that less protected version to run, um, a hack on modern Linux is not easy because we've got so many protections by default. So if you want to do it, you have to compile like this turning off protections, which is, which is actually nice. Right. Okay. Finally, the sixth point. Okay. We're getting close guys, almost there, but it's a really key point. Guys, we need to test, you know, testing is sometimes ignored, but it's very, very important. Okay. Let me not talk that way. We all know these things. Fine. So what I will point out is, um, many of us think in terms of running positive test cases, which is essentially saying has the desired desired outcome been yielded, but we don't run negative test cases. Negative test cases are things that the hacker will check. Okay. We need to run those kinds of tests and like really thoroughly. That'll help us. See guys, these are real photos of IOT products. So yeah, it's, it's funny, but it's also true. Um, we have to fix these bugs here. Right. So you've heard of integer overflow. Right. Guys, go into my GitHub repo and run the code here. I have a shell script to make it easy for you to run which demos and IOF bug. It's as simple as this using a signed integer when you should have used an unsigned integer. Okay. So guys, um, be careful. Yeah. These kinds of things can trip you up and hackers can take advantage of it. It might sound fanciful, but it's true folks. Uh, have a look at this. How is integer overflow exploitable? It is exploitable. Okay. Guys, we've all come across code like this, but have you thought about this? What if variable A and variable B get runtime values such that when they are multiplied and integer overflow occurs, isn't it possible? Are you checking for it? You must, you must write a test case for this, a negative test case. What if it happens? You know, in many products, right? You could ultimately land up with a much smaller buffer because this is an unsigned quantity guys. So it overflows to a smaller value and buffer overflow becomes a distinct possibility. Amazing. Right. Think about it. It becomes a good DOF attack candidate. So how do we catch these bugs? Guys, you have to catch them. Yeah. Do static analysis. It'll help you catch these use the sanitizer tools. They can also help you catch these, the UB sand, even the Colonel has UB sand by the way. And in a hardened Colonel, it should be on by default, not case and because case and is too expensive for a production Colonel, but there is something called, um, there is a lightweight one called K fence, which is used in production. Please use it. Uh, Colonel developers, driver developers, you would have come across using the atomic type, um, which allows you to do operations on integers without the need for locking. That's known as the atomic type. A little Googling will show you. But, uh, in recent kernels, we have a superior implementation called the ref count type guys. Ref count type protects against IOF integer overflow. So make it a habit to start using ref count type in place of atomic type. Okay. Uh, a little more on IOF. And finally, there is a software technique called fuzzing. This is what fuzzing means. See guys, essentially, you know, it's the monkey on the keyboard technique to put it simplistically. You, you throw huge amounts of random input at your process or your colonel or your driver and see if it fails. I mean fails badly, not gracefully. Okay. Why? Because hackers use this to figure out vulnerabilities, which they then try and exploit. And guys, there are numerous examples of exactly this kind of thing. In fact, I can't recollect just now, but I read about one recently where a hacker used Google syscaller to figure out a bug. I, sorry, I don't remember where now, but he figured out the bug and then attacked it. Fuzzing is necessary folks. So Google has a lot of stuff, including one called syscaller. It's, I think the powerful one now. In fact, see, let's look at this. This is the syscaller dashboard. You know, they are doing this continuous integration CICD as part of the DevOps functionality for the Linux kernel. So this is the, this is the syscaller dashboard showing you the results of recent fuzzing on the kernel. And, and see guys, this is two hours, five minutes ago. It, it found something two hours, 40 minutes ago. And, and here are possible bugs. So you can dive into these and see what is the report. For example, K-san has got something here. K-san has got something here. Ref count has got something here. Interesting. They may not be bugs. They may be false positives, but it's our job to check them out. In fact, this syscaller has caught a lot of stuff, which has subsequently been fixed. Okay. So guys, as a sidebar, Alexander Popov has written a script called K-config hardened check. It's really nice. It allows you to check the security posture of your Linux kernel.  It's a much easier way to install it. So see folks, I'll just do a quick demo since I've already used up a lot of time. We don't need, I've got it installed in my system. So I'll just run it guys. See, if I just run it like this, it shows me the help screen. So look at this. You can give it a kernel config file. You can give it a kernel command line in addition to the config file, or you can ask it to print security hardening recommendations or to generate a config fragment, security hardening options for a given type. Let's do minus C. It needs a config file. So see guys, on the X86, no, it's very easy here. This is the kernel I'm running. You will find it's config file under boot. See, here it is. So see what I'll do. I'll do K-config hardened check minus C and I'll give this. And there we go. It found 89 configs to be okay. And it found 98 of them to fail.  It tells you some information. These are all kernel configs. So you know how you read this? There's a config called config bug. Is it on? The desired value is yes. In our system, it is yes, which is why the result is okay. This is for self-protection. Now, on the other hand, it says we should have this one, but it failed. And it failed because we don't have the compiler feature, GCC plugins. That's why it's not there. Now, over here it failed because it's not set. So guys, look at this one. The treat warnings as errors. The desired value is yes, but we have it as no, which is why it failed. So like that you read and you figure out. Now you can do some cool things here. You can say this and you can say minus mode show fail. Now it will only show us failures. So now we can use this as an audit and we can take a printout and one by one, we can start fixing them. Now, of course, that's easier said than done in the real world, but that's the way you go about it. Then you can do other things, guys. You can say I am building a project for arm 64. So show me the recommendations, security recommendations. In fact, I will use minus G generate, and it will generate a config fragment. So guys, this is like for free. We get the kernel config. You must merge this with your existing kernel config for your IoT product or embedded or whatever. Interesting now. That's how this project helps us. So I've shown some screenshots, guys. Here's an example of how you can merge. Look up this link. I'll click on the link. And this is from his GitHub, Alexander Popovs. He's given an example of using the merge script to merge the fragment. The new fragment is what you use as your kernel config. Fine, guys. So concluding remarks. You need depth of defense. We have seen a lot of hardening techniques. Don't use one. Don't use to try and use as many as you can. Okay. Guys, I've highlighted this slide because it's one of the important slides. This is what you should aim for. In fact, this is a tweet from Cascook. Okay. So I'm adding on to that. Try and use ASLA, CASLA, use NX, use SMAP, SMAP. If you're on X86, otherwise use the equivalent on ARM64. Use all this in your make file as far as you can. All your make files. Okay. Turn on compiler protection. That's what this does. But of course, it does depend on kernel, I mean compiler version also. Guys, using Clang is a great idea. Clang sometimes does a superior job. Okay. Android uses Clang. Turn on full railroad, preferably. Turn on PIE, PIC. Try and use safer libraries. Use kconfig hardened check to figure out safer security posture for your kernel config. Use all of this and combine it with thorough testing. Run all your test cases against all of this stuff. And then you have confidence in the product. Okay. Guys, it's a lot of work. I'm not saying these things are easy. It's not easy. But you know, that's what we are paid to do. We must do it. Okay. So I think this is a key slide. Okay. This is more of the same guys. Guys, I've taken too much time already. Please read through all this. The principle of least privilege. Use the modern capabilities model. Try and read up about and use SecComp. You must have a secure update path. You must have encryption. You must have physical security. There's a lot more. I've given some links folks. These things are not simple. These are not trivial topics. It's not like we'll solve all our problems overnight. But it's a way to get started. I hope that you can make a powerful stuff with this. That is the hope. I've just given some extra notes over here, guys. You know, I was hoping to be there in person and attend Natali's talk because Natali is going to be talking at this conference and she is going to be saying things like this. Okay. Which is essentially saying that it's impossible to mitigate everything. So guys, that's what I've been telling you is the reality. So we need to do a lot of work. Okay. We need to harden our security posture. Then it becomes a lot harder. Thank you very much folks. Again, this is the Github repo. The latest version of this presentation also is here. More articles, code, everything's here. Guys, I do corporate training on Linux on a variety of topics. There's a link here. You can contact me here. These are my links. Thank you so much. I hope this has been useful. Thanks folks. Have a great time. Bye bye.