Welcome back everybody. We're here in Boston, Massachusetts at BMC Day. Hope we get some good B-roll here, Andy. Some good action out here in Boston Harbor. We're here with Brian Graceley and Stu Miniman. We got the part of the Silicon Angle Wikibon team out here. We've got our annual dinner tonight, legal seafood, looking forward to that. Check out the harbor side first floor. We'll be there if you're around. Dave, we're in the shadows of where the Cube first event was too. We're right back here in Boston. Chowda Fest, as we called it at BMC World back 2010. Yeah, so let's do a little wrap up on what's happened. You know, it's the time of season where we talk about what happened last year, or this year actually, and what's going to happen next year. So where do we start? I mean, what's the big news of 2015 from your takes too? Yeah, so I mean, Dave, when you brought me into the company almost six years ago, it was the discussion of convergence. You know, we talked everywhere about kind of, breaking away those silos. And the question we have is every new technology that comes out, is it just a tool? Does it just create another new silo? Convergence, hyperconvergence has grown quite a bit. We did a couple of years ago what we call server sand, and boy is that grown in leaps and bounds. One of the big news events of the year, of course, is Dell's acquisition of EMC. And I really see Dell's positioning to be ready for this next generation of how infrastructure is going to change, not only in data centers, but in the public clouds as getting ready for that. So what they get from EMC, how they're positioned from kind of the market power that they'll have and all of those pieces.  And you know, boy, my phone's running off the hook on all this stuff. So Brian, from your standpoint, first of all, you know, 2015, welcome to Wikibon. It's a pleasure to have you. You know, longtime industry watcher, participant, practitioner. What's your take on 2015? So for me, the big thing was, 2015 was the year that public cloud essentially reset the rules for IT. So, Amazon now announced their numbers, and while it's still, you know, eight billion dollars is a lot of money, it's not, you know, it's not everything, but it's forcing everybody else's hand. The on-demand nature of IT resources, the transparency of pricing, the speed at which new features are coming along. I mean, we're seeing an impact a lot of different things. We saw HP get out of the public cloud market, we saw the huge Dell and EMC Merge, you know, EMC Merge, well, acquisition, which we think, you know, cloud had an impact in that. So to me, the rules are kind of being rewritten and everybody is figuring out, how do I play in this new game, or how will these new rules affect me? Yeah, and I think I'd add to that. I mean, we're seeing, and we've seen this now for a decade, a slow-motion collapse in infrastructure hardware and software pricing, and it's really driven by two main factors, there may be others, there may be cloud and open-source software. And business models are changing, people are having to respond to that, not only from the vendor side, but also from the buyer side as well. And so to me, the Dell EMC acquisition was inevitable, the HP split was inevitable, companies like BMC going private, you know, have to happen in order for them to continue to be able to fund R&D. The big question is, okay, going forward, you know, what's the market look like? Stu, you're talking about server sand, you know, really completely changing that model, you know, the pendulum swinging back, we've talked about that a lot. The way in which companies leverage this digital fabric that we talk about all the time, we heard today all talk about digital transformation, really what that's about is taking advantage of all these technologies that are out there, it's not necessarily inventing them, it's applying them and creating new business models. Yeah, absolutely, Dave, you and I went to London with the MIT Sloan School and talked about the second machine age, so it's really about automation, it's about changing processes and you know, the discussion here at BMC Day has been that digital transformation, which is, you know, how much is the technology itself going to run itself, how much of it are we going to, you know, align our organization, our processes and our people to be able to take advantage of that and that, of course, is all going to drive the business. It's pulling together, I thought a great line today is it's no longer information technology, it's business technology, we have to talk about it, so the merging of those two, because as an infrastructure person, I always know the role of infrastructure was to deliver the apps and the role of apps is to help drive the business, so it's helped getting back to that kind of core of what we're working at, and it ties into kind of the discussion we've had of containers all year and all the platforms that are going on. Well, infrastructure as code is, you know, the hot buzz phrase, but people kind of kind of be careful what they wish for, right? I mean, it's great to talk about, but it's really certainly transformable, transforming from a developer standpoint, but it's really changing the economics of the IT business. Brian, I wonder if you could talk about that from both perspectives, from the IT organization that I should use, Evelyn's term, the business technology organization, what does it mean, and how do you see it affecting the industry? Well, I mean, we heard some numbers this morning, I'll sort of quote them, you know, this was talking about IT losing control, and we're at a BMC show, which is, you know, IT ops and trying to enable this, I mean, 40% of spend is going outside of IT, right? It's not that it's going away, it's going outside of IT. 80% of the people are trying to go around IT to make decisions, 60% are trying to bypass it, 30% of applications are moving to the cloud. IT needs to either kind of get on board or they're potentially in some big trouble, and the thing is, there's never been more technology, and in some cases, free open source technology to help them do that. They've got to get better at it because the baseline's being set by the guys in the public cloud, Amazon, Azure, Google, you know, how fast you can roll stuff out, and at the end of the day, if I run a business, I don't care how many ports you had to plug in or what you had to configure, I want the business to go faster, I want to grow it. There's a lot of change that's going to happen and people are going to have to figure out how to get better at these things. I think of that, you know, the old Microsoft commercials, what do you want to do today?  I mean, is that what the future holds for us? Well, yeah, absolutely, people want to consume it that way, and it's how fast we can respond to the business, and, you know, boy, you know, cycle times. Brian and I have been looking at this quite a bit. Think of the old, you know, typical software release. You know, it's 12 months. You look at what was going on in the open stack community. It's every six months, you look at what's going on in a lot of the open source, especially like, you know, Docker and that whole ecosystem, it's every two months. So, you know, we know that things are going to change constantly, just the speed of change and how fast we have to move at that. So, and absolutely, people want to consume it more in kind of that operational model. But it's, I think we don't agree maybe with the term bimodal, but I'm going to have, you know, some things that I purchase in a capital expense model and others that I manage in an OPEX model. So, we've been talking for years as well about the cartel. You know, the chess board is owned by a few, you know, select the handful, I should say, of large companies. And somebody makes a move and it ripples through the industry. I think we're seeing that with Dell EMC, but you can be basically IBM, you know, Oracle, HP splitting up, Dell EMC, Cisco. Let's talk about Cisco a little bit. You know, Brian's a former company. Just Chuck Robbins, you know, everybody in our little world is like, oh, they have to buy a storage company and maybe they do because if you don't have an integrated stack, maybe you're screwed. But if you're Chuck Robbins, I'm guessing your head is not in storage right now. He's thinking in and of everything and, you know, much bigger picture, but at the same time, there's a data center business there. So, with or Cisco? Well, so he's obviously brand new when he's been at Cisco for 20-some years, cleaned house a lot, you know, brought in a whole new management staff that, you know, whether he's doing it with John Chambers or not, we'll see.  I think we saw some data the other day.  So that phenomenon that has hit storage in terms of sort of commoditizing hasn't hit networking nearly as hard, but he's got to figure out how do I network those next billion, 10 billion devices?  Am I interacting with the analytics companies? And Cisco acquires well, but that's a big jump in the stack from an acquisition perspective. We'll see how quickly they give them the big checkbook to maybe go do something like that. Yeah, I mean, Brian mentioned earlier that Amazon's kind of rewritten the rules. In some spaces, we're seeing massive changes. Look at what's going on in the compute space. The ODMs made a big push. Dell moving with EMC, HP splitting, IBM selling off the X86 to Lenovo. The compute space is changing a lot. I mean, Dave, I worked at a big storage company for 10 years. If you saw more than a couple of 10% move in the market, everybody kind of freaked out. Today, in 2015, there were big moves. I mean, not just the acquisition moves. In 2016, you're going to see a couple of points move in a few directions, product lines are moving, going through massive transitions, but as Brian says, networking hasn't really seen that change. White box, not only are the white boxes in single digits, it's flat. The revenue has not been growing. I was talking to a Wall Street guy last week, and he kind of walked through through the numbers and he was like, you know, Stu, it's at 2% of the market, and it's been that way for the last two years. So while many of us thought that things like SDN and baking in networking functionality might really disrupt Cisco, most people at this point think that Cisco has weathered this last wave. Now, as IoT comes to the market and some other big waves come, everybody always wants to go after Cisco. And as public cloud, you look at what Oracle's doing, look at what Amazon's doing, look at what Microsoft's doing. There's edge cases where they can try to eat away at Cisco, but Cisco's got a pretty strong position. IVM's another interesting one. My tongue in cheek is Dell buys EMC, IVM buys the weather company, you know, what the heck's going on? A lot of IVM's competitors like to say, oh, IVM's getting out of hardware, they're giving up on hardware. You know, they're not, you know, it's still, you know, Z, which is the original converged infrastructure, high margin business and, you know, the power of the open power movement, kind of interesting, storage is sort of going toward software defined. But the real shift in IVM is around analytics, around Watson, IVM announced the big IoT initiative today. They hired Harriet Green from Thomas Cook as a new executive, Dion Newman's now the VPA marketing. Classic IVM, right, they opened up, I don't know, eight new facilities, IoT expertise centers, a thousand developers, a bunch of new APIs, it's like this engine. It's a Watson, it seems to be this really interesting beast that nobody in the traditional enterprise, you know, is able to replicate and IVM's pouring, you know, a lot of money, billions into Watson.  Interestingly, you saw, I think earlier this week, Facebook announced it's open sourcing, it's AI capability, and talk to the real alpha geeks and they say, look, Google and Facebook have the best AI, Watson's big, IBM-like, monolithic, you know, you've pointed at a problem like healthcare and it'll take some time and then solve that problem, but in terms of flexibility, agility, it's, we've seen this movie before, Brian. What's your take on all this? So with IBM, I think you're right. I think at some point they win. It's just a matter of how long is that journey, right? How long can Wall Street wait? How long can Ginny, you know, kind of have results that she has results?  That's essentially them saying, that is so important to my business. I need as many engineers around as I can get, I'm going to open source it. They don't have to sell it. They reap the benefits of it. It's a brilliant play by them. It's a brilliant play by the Googles of those guys. It's not so impactful to the IT industry, but it becomes, you know, kind of how do I feed this thing that drives what they do, so. The other big story, Stu, at 2015 that we haven't talked about much on theCUBE, every now and then, certainly Furrier talks about it, it's just the Microsoft, the relevance of Microsoft, the reemergence of Microsoft as a relevant player. I don't know if you saw the headlines where Bomber, you know, slapping down Satya, and who knows, it's media, but it's also Bomber, so it could be true. Was that sour grapes? I mean, don't you think Satya's doing a pretty good job? I do, and first of all, you know, talk about relevance in the marketplace. It was, you know, they were the punchline in the Silicon Valley jokes for many years, Dave, and Wikibon's research, you know, if you take SaaS infrastructure service and platform service, Microsoft's number one. I mean, you know, they might not be number one in SaaS standalone or infrastructure service standalone, but their number one overall, they have still, Dave, they own their applications, and you know, it's kind of say, okay, how many customers are using Office 365, all of them? Well, how many of them just got kind of moved into buying Office 365 and haven't moved to the cloud? Well, many of them are moving there, but boy, it's Microsoft saying it's okay to go to the cloud, we're going to be your good partner, you know, they've got so many different businesses. I mean, you know, Xbox is still one of the hottest gifts, you know, of the season here, so Microsoft is relevant, they're showing up a lot more places.  Let's go to the other end of the spectrum, Oracle. So, we're all at Oracle Open World. Donna Telly's there in force, bringing some of his old buddies back, you know, Barry Burke now, working at Oracle, obviously Chuck Hollis is there, you know, hiring like crazy, going all in on the red stack. It seems to me that, you know, Oracle's playbook is pretty clear, it's, you know, our way or the highway, and we're going to run better, Oracle better on Oracle, and we're going to vertically integrate, and that's what we're doing. And so, they make no apologies about it, it's a clear strategy, it's well articulated, and to me, the big thing about Oracle is, they've always spent money on R&D. Oracle's Achilles heel is that sort of cross-platform, you know, open horizontal play, but you know, Stu, you were there, what's your take on it? Yeah, so you know, we often, the term gets thrown around a lot, data gravity, but if you talk about stickiness of certain things in the environment, I've actually been surprised, you know, specific storage vendor. It's kind of easy to, you know, change what storage I have. Even virtualization, customers going, you know, from VMware over to say Microsoft, or you know, just buying some solution and KVMs baked into it is easier to move. The application is the toughest thing to change out there, and Oracle's got the biggest baddie of them all. So, you know, moving that application, Oracle's taking done a lot of work to help pull it into the cloud. You talked about Microsoft's application, IBM of course owns applications.   We were just, you know, okay, and then, you know, a year later, it's a new strategy, strategy du jour. So, Oracle's strategy, I think it's very sound. Now, let's face it, most or much of Oracle's, you know, cloud, certainly from the infrastructure side is hosting, much of it's SaaS apps are hosting. They're going to call that cloud. That's okay, that's the game everybody's playing. IBM does the same thing. And Oracle's got some work to do to get to sort of true cloud, and it's got, you know, bits and pieces, right? It's got the SaaS piece, it's got the past piece, it's got the infrastructure service, it's got the storage piece, the storage piece is in pieces. So, it's got to bring those together. As I say, when you have a leader who owns the budget, you know, there's opportunities exist there. I know you guys are tight on time. Okay, let's wrap, sort of looking ahead to 2016, what should we be looking for and, you know, any final thoughts? Well, the big thing for me for 2016 is looking at, you know, we're talking about, you know, sort of digitalization here. How fast do those new applications take off? I'm going to be looking at how big that market gets, who leads in that, is that, is that, you know, Docker, is that cloud foundry, is that those types of things? Because that has a direct impact, as we saw all this all day today, on changing the business, of impacting, you know, differentiating the business. That's a big area I want to be looking at, 2016. Yeah, for me, some of the architectural environments out there, we've been looking, David Floyd especially at, you know, the all flash arrays and some of the technologies underpinning it, some of those building block companies, if you look at, you know, Microns, inside a whole lot of devices, HPE just launched Synergy and, you know, there's Micron, all embedded in there. Western Digital and HGST just bought Sandis, so that's going to, you know, move them up from kind of the number three and four position up to, you know, almost tied with Intel for the lead in that market.  Well, flash has still got a lot of runway to take environments and those are impacting, you know, the storage architectures, the server sand architectures and the cloud environments that go on. So, you know, massive change. So for me, it's, you know, deep throughout fall of the money, my thing is for 2016 is follow the value. And I think it's, for organizations out there, it's not, you don't have to invent, it's all about really how you apply technologies within that digital fabric to create a business capability. That's where you're going to get differentiation. I think as industry watchers, we always look for who's the next Intel, the next Microsoft, the next Cisco, the next EMC. It's really going to come from how you apply technology. So that's what I'm going to look for in 2016. Who's really doing that? Well guys, thanks very much for holding down the fort. When I was on my call today, it was great to see you guys in Boston, looking forward to our dinner tonight and thanks everybody for watching. We're here at BMC Day, you know, covering all the angles. This is theCUBE. Check out all the action on Wikibon.com. That's where all the research lives. SiliconANGLE.tv is where all the CUBE videos are. And of course, SiliconANGLE.com for all the news, CrowdChat.net. This is theCUBE. Dave Vellante, Stu Miniman, Brian Graceley. We'll see you next time.


I think that Jack Ma offers an alternative look at globalization that creates a lot of jobs here. Now, you could argue that he's being a little naive that we can't really do those jobs. But I had Wix on last night. I have Adobe on tonight. We had Federal Express say things today. It is a global world if you're an entrepreneur. It's a global world if you do apps. It's a global world if you're able to create because you can do a fabulous website with Wix. You can be able to do e-commerce with Adobe with a creative cloud. There are so many different ways to make money, but you need an idea. I mean, a lot of people were watching Wix last night and said, I want to have a website. Well, the problem is you have to have a product. Product first, right? You have to have a product to sell.  Yeah, right, right, right. We should also point out that our own Brian Stasi is at the event covering Alibaba. Oh, fantastic.


There it goes. Hello everybody and welcome to our second talk in the 101 Essentials for Embedded Track. The next talk is going to be by Mike Anderson. He's a longtime speaker at the Embedded Linux conference and other Linux Foundation conferences. He often ends up giving far too many talks but we've managed to get him in to give this talk today on GDB on embedded devices and make sure that when you have many questions that you write them in the Q&A and he'll answer them at the end. Otherwise take it away, Mike. Thanks, Ben. Let's see here. Let's get going. Obviously, I hope that everybody's had an opportunity to at least download the charts. I think they're available online. If not, they will be available shortly and with any luck you will be able to get those and then have them for your very own copy. This is me, Mike Anderson. You can use my mail to there, mikeandersonptr at gmail.com. At the time I wrote these particular charts, I was unemployed, actually between jobs. I just started a new job last week so brand new on the job with the Aerospace Corporation. This is me. I've been in the industry for about 42 years, 20 or so of those with embedded Linux. I used to be chief scientist of the PTR group and many of you who have been in my presentations in the past probably know me from that particular role. I'm now an embedded systems architect at the Aerospace Corporation and I continue to be the mentor or a mentor for First Robotics Team 116, Epsilon Delta out of Herndon, Virginia. If any of you are interested in robotics at all, I highly recommend getting involved in First Robotics. It's a great way of being able to help the kids come up to speed and I look at it from my perspective. It's one of these days the software that those kids wrote may be running my ventilator and believe me, I want to make sure they did it right. So that's an important thing for me to make sure that they're all up to speed and embedded and are doing the right thing for us. LinkedIn, obviously if you want to get in touch with me on LinkedIn, you're more than welcome to do that as well. But enough about me, let's move on to talk about what we're here to all chat about, which is GCC and GDB and all the tools that go along with debugging and profiling inside of the kernel. Now, obviously part of the problem that we have with Linux is it's not that we don't have enough tools, it's that we have way too many tools, oftentimes with an overlap, a considerable overlap between some of the tools. So if I for whatever reason leave out your favorite tool, I appreciate that you can be upset about that. But unfortunately, we only have so much time to work with here. So we will press on and hope that we can cover most of the things that most people are interested in. We'll talk a little bit about the GNU project and GDB and where it came from and what we need to do in order to compile for debugging. We'll get into some special modes of GDB, in particular one that not very many people are aware of and that is the text user interface mode or the TUI mode. We'll also talk about some of the GDB front ends. We'll then dig into how we get help, some of the scripts and macro mechanisms, what we have to do to go about launching and kind of running applications within GDB, attaching to a running application, how we set breakpoints, watchpoints, trace points. We'll get into GDB server, which is a very useful tool for being able to debug code over on an embedded platform. The embedded platform that I have for use today is kind of a special one. It's the Udo Bolt, which happens to be based on the new APU from AMD. It is an X86 and it also has kind of an Arduino built into it as well. So it's kind of an interesting little beast, but that's what I'll be using for my embedded platform discussion when we get to GDB server. Then we'll kind of switch gears from the debug perspective over to the trace and profile perspective. We'll look at S-trace and L-trace, GPROF, GCOV, Valgrind, LTT, F-trace, and we'll talk a little bit about Perf, which is relatively new-ish. The issue that I have with Perf is that it only runs over on the Udo Bolt because I'm running a custom kernel on my platform. And as a consequence, it doesn't like running Perf out of the box, but then's the way it goes. So let's move on and start talking a little bit about debugging tools. Now, when we look at debugging tools, they basically fall into one of two different classes. Either they're a debugger like LLDB or GDB, live recorder, etc., or they're some sort of a checker. Now, in particular, checkers and profilers are focused on trying to find bad things, stack overflows, misuse, double freeing of memory, those kinds of things. How we're using the caches, address sanitizers, and even some of the static code analysis tools like Covarity kind of fall into that category, because they do have the ability to look at threading possibilities, the threading interactions and go, well, there could in fact be a problem here. And they'll try to bring it up and try to deal with it at something a little bit before you actually get into the debugging tools, before you actually get into the runtime stuff. So they tend to be a little bit on the static side, but that's okay, because we need to be able to do those too. Certainly, tools like Covarity have really helped tremendously with the Linux kernel, so that we could in fact get some, eliminate a lot of the potential race conditions and things of that sort that have existed in the kernel in the past. Now, with the GNU project, of course, GNU stands for GNU's Not Unix. It was originally developed in 1984. It was actually put out there in 1983, September 1983, it was proposed, and then the work didn't start until January of 1984. But one of the first processes, one of the first things they wanted to do was to build the tools that were required to build an operating system that would be a completely open source, free operating system that was very Unix-like, but was not Unix, and that's where the GNU's Not Unix came from. At first, there was just the GCC compiler, and then that was focused largely on just C. Then later, we got G++, which was C++, and now that's kind of finally manifest itself as the GCC with a capital GCC, and that's the GNU compiler collection. Now, the way the GNU compiler collection is built, it's got a front end that allows it to interpret the language and convert that into a back end, which will then actually do the compilation. We see front ends for C and C++, subjective C, Fortran, Ada, Go, etc., etc., and of course, libraries like libstandard C++. Those are all part of the GNU tools, and it's architected so that we do have that option of being able to work in terms of kind of a front end that's built to run on one machine and a back end that's built to run at a separate computer. For instance, we would have something that was x86 cross arm or Windows cross Linux. All of those kinds of things are different possibilities that exist with the GCC tool chains. Of course, as opposed to GCC, we've got LLVM, so there's lots of great compilers out there. I'm going to be focusing largely on GCC and the GNU compilers at this point, and GNU debugger rather at this point, because that's pretty much the topic of this particular talk. Perhaps I'll do an LLVM debugger at some point in the future, so that would all be kind of cool. Now, let me just go ahead and yes, the slides are available for download. They should be on the website. If I had failed to mention that earlier, if they don't show up on the website shortly, you can always email me, and I'll be able to get a copy as well. Now, the GNU debugger, GDB, it was built as a source debugger for the GNU compiler collection, which means it actually supports many, many different languages. This is one of the things that most people are not quite familiar with with GDB. GDB actually works with ADA and C and C++, obviously, but it also works in D, Fortran, Go, Objective C, Modula 2, Pascal, Rust, and several others. It is a very flexible debugger. It is not simply limited to C and C++. It actually works across the GCC, across the GNU compiler collection, into quite a nice debugger that works across multiple platforms. Now, when we get ready to compile something for GDB, we'll take a look at the command line debug options. When we get ready to build the code, we will pass a special compiler flag. This compiler flag is kind of ubiquitously known as the dash G option. However, there are more than one dash G option. We range anything from dash G0, which means don't do anything, which, I guess, makes more sense if we're going to make file and we're passing this as an environment variable. Because why would you go to the trouble of actually typing dash G0 if you didn't have anything out there to do if you weren't really trying to compile the code? Dash G1 includes minimal information, basically backtraces, but no information about local variables and line numbers and things of that sort. Dash G2 is the default debug level. If you just specify dash G on the command line, you will get G2. G2 produces symbols, line numbers, basically everything that's needed for symbolic and source debugging. And this is, like I say, the default if you use the dash G option in the compiler. But that is not all. It turns out there's a dash G3, which includes additional information such as all the macro definitions that may be present in the program. This is particularly important when we're looking at places where we're doing some large substitutions of macros. Like, for instance, in the Linux kernel, there are a number of commands that look like C commands, but they're actually inline assembly language commands. So all of those would certainly be something that could potentially be a macro expansion that we'd be interested in knowing. And we could see all that using the dash G3 option. But, you know, all of these options, these first three options, G1, G2, and G3, all produce output in what's referred to as dwarf format. And if you look at the output from the compiler, it's called ELF, embedded in linking format, or executable linking format. And then dwarf is debugging with attributed record formats, or so that's dwarf. So we have ELF and dwarf, very much a kind of Middle Earth sort of flavor there. But the mac daddy of all the options is dash GGDB3. This is like dash G3. However, it actually generates debugging information specifically for GDB, rather than the normal cough or X cough or dwarf to format that you come out with the dash G. So dash GGDB3 does in fact produce more information in the debug output, and it's specifically targeted at being used by GDB, rather than one of the other debuggers that are potentially out there. So here's an example of a compile for GDB. We have this ARM Linux GNU ABI GCC. We're using the dash GGDB3 option here, dash oh hello, and we're doing the compiler, the ubiquitous hello world. The example, if we were to take a look here in the ELF header, and we were using object dump as an example to show us all these things, we see all these things that begin with a dot debug. This is the code or the sections rather that get output when the dash G option is turned on to the compiler. Now, the nice thing about all these things is they are kept completely separate, and they're kept separate in the executable. Therefore, the code itself, the text segment in the executable does not get adulterated with debugging information. This is really handy because what this means is I can actually debug code on the target that has not been compiled for debugging as long as I have a version of the code that has been compiled for debugging online. So I will load the code compiled for debugging into the debugger, but the code that's running on the target does not have to be that code. It just has to be, well, obviously it doesn't have to be compiled for debugging, but it does have to be the same code base, obviously. Now, that's an important factor here because when we're dealing with extremely low memory footprints, and I have run Linux on a two megabyte machine before, it was a MIPS processor where they had already purchased all the RAM chips, and you couldn't get any more than the two megabytes that they had. When you're trying to run Linux in a very small memory footprint like that, you do not want to put anything into memory that has any of the debugging code in it because the debugging code adds about 30% to the size of the executable. So that's a very important thing to keep in mind, and we'll see an example of this when we get into a little bit of a session that I'm going to show you a little bit later here. So now we can run GDB from the command line interface. Now, obviously when GDB is built from the source code, we have to specify the host and target information, and the host and target information will specify whether it's Windows cross Linux or Linux cross Linux x86 to ARM, et cetera, et cetera. So here's the kind of output that we would expect to see from this kind of a compiler. We're going to output here, we're just going to run the GDB interface, and we'll see that this was particularly configured as x86, 64 Linux GNU. Now, if we had a cross compiler built on here, then we would see that it was configured as x86, 64 as the front end and potentially ARM as the back end. We did have a question come in, what's the advantage of G1, G2 if G3 does even more, especially if you can strip the executable from running on the target? The issue is that the more optimization, the more stuff you turn on here, especially if you turn on optimization, optimization rearranges the instruction stream. So if you turn on a very high level of debugging and a very high level of optimization at the same time, the debugger can handle it fine. Humans can't handle it too well when you see the instruction pointer jump backwards. So it's a little weird. It doesn't mean you can't debug it. It just simply means that you have to be mentally prepared to see things go backwards. Also, if we were to turn on all of the debugging options and we're running the debugger and we're actually interacting with the debugger, then, yes, having on higher levels of the debugging flag turned on will in fact slow the process down a little bit more because it's starting to keep track of more things. That does not necessarily mean that it's going to be like even three or four percent difference. It's going to be relatively small. So you're not going to see a whole lot of difference here because we really don't need to do anything to the target as far as being able to turn on the debug flags. We just need to make sure that the debug information is available to the debugger. So the text segment and the data segments and all the other important running segments of the application on the target don't change when we turn on the debugger. But it does change the size of the code. So when you start turning on debug level three, then the code is going to balloon up. So if you are just, if you're not really thinking in terms of being able to run the strip command and do all the things that you would normally do before you deployed the code, if you're just in a quick debug, compile debug test kind of model, then people don't like to have to go through the extra steps of doing the strip command and then doing the copy. It's just a little bit easier for them to just simply do everything inside of one make file. Now, personally, I would then put the strip command inside the make file, but we really do need to have the fully debug-enabled version of the code for the debugger. We don't need it to be the run on the target, but we do need it for the debugger. So it does get a little bit complicated and oftentimes it's just easier to not mess around with it. But fortunately, it really just changes the size of the executable as it exists on the disk, not so much the change in the runtime. All right, so if we wanted to run GDB in text UI mode, running GDB in text UI mode, the tui mode, uses curses. So obviously, if you don't have curses installed on your platform, it's not going to work all that well.  So for instance, let's go to the screen share here and let me see if I can get the right screen selected. So with any luck now, you're seeing a screen that shows my command line here. So let's go ahead and we'll run GDB. First of all, I had mentioned that we have the option for building GDB with different compilers. So if we took a look here at this output for ARM Linux, you'll see that the GDB was configured as the host is an x86 PC Linux GNU, so it's a x86 based host. But the target is an ARM processor running with hardware floating point turned on. So that's the kind of thing I was trying to show you a couple of pages ago, where we will configure a front end for GDB and a back end for GDB. Now, inside of GDB, I can automatically load commands. Let me just go ahead and I'll quit out and show you here. So we do GDB and I have a command. Let's just go ahead and compile it just so that we've got it all there. We'll do a GCC dash oh hello and dash G will turn on just level two. That's okay. We don't need to turn on any more than that for most cases. And we're going to compile hello.c and then we will run GDB and we'll run it in tui mode and we're going to look at hello. And now we see what it looks like when you're running in the text-based UI. So we have the upper window here, which is the source code and we can scroll around in the source code. We have the lower window, which is the command line. And inside of this, we actually have several different options. We can use the layout command. And if we just see layout next, it will then show me the next possible layout. In this particular case, I have the assembly language output. I can do layout next again. And now I see the source code on top and the assembly language. And then let's see here if I can get it to do this again. Registered values. I'm not running it right now, so there are no register values available. And then let's see here. I think it I want to see I think it's got a couple of oh then it goes back to the original one here. So if I wanted to just simply do a kind of a quick run of this, I can just tell it to run. And if I tell it to run, it will run to completion because I don't have any breakpoint set yet. But we see hello world and we see the value of i and now value i has been incremented. Notice one thing here. It says inferior one has exited normally. That is not a comment on the quality of your code. That is a comment on the way gdb sees your code. Whereas gdb is running it from outside of the code. It sees your code as an inferior process is something that is subject to the control of gdb. So in this particular case, we just ran it to completion and then it exited and that's the kind of message I would normally expect to see. There are a few other things that we can actually take a look here. We can do an info on the number of windows that are out here and this will show all the different windows and their respective heights. I can then specify the height. Let's say win height. Let's do source and we will subtract two from the source code. There it just changed the amount of window displayed by the source by two. And if I happen to then be looking at things like 2e, let's do register. And let's do register. Let's take a look at the floating point registers. And obviously it's not running. So let's do this. Let's do break at line 14. And then run. And now we see the floating point registers. So we can also see the general registers well. 2e, reg, gen, I think will allow us to our general, rather, general. And we can see the floating point register. I mean, we can see the general purpose registers here. So we see ax, bx, cx, etc. We also see the magic additional eight registers that were added to the x86 instruction set some time ago that you still don't see if you're in a 386 or 486 world. But it kind of gives you an idea of the relative flexibility associated with what you can see inside of just looking at it in the text UI mode. Now, as we move forward, we start looking at full up GUIs. Now, there are a lot of standalone IDE based front ends for GDB. The one that is the official supported front end is the data display debugger or DDD. You see a picture of that in the upper right hand corner there. It is very much like 2e mode except that it is an actual window that floats there and you can move things around inside of it and you can resize it. And it actually has quite a bit more flexibility for being able to display pieces of information like being able to see your watch variables and data structures and things of that sort.  Considering its age, it still holds up pretty well. Obviously, you can also use front ends like Eclipse. You can also use Microsoft Visual Studio Code. Microsoft Visual Studio Code is actually open source. It is not Visual Studio. It's the editor for Visual Studio, but it also has quite a few plugins that are available for it, including a GDB and an LLDB plugin. So we can actually do debugging using either the Clang compiler or using the GCC compiler. There are other ones, of course, KDevelop, SlickEdit, Code Warrior, many, many others are out there. So it really becomes a question of what your preference is in terms of which debugger interface you happen to like best. With the DDD front end, DDD supports JDB, Python, Perl, Tickle, PHP, and we can also specify a different back end. So for instance, if we are getting the dash debugger option, we can specify ARM-Linux, GNU-EBI-GDB and MyApp. And with that, we'll be able to run a different back end with the DDD front end that's running on the x86. The back end was running ARM and we'll then be able to reach out and contact, reach out and touch a different application to run here. And we'll show you how to do that when we're using things like a GDB server. And I'm getting some reports on the audio. Let me see what I can do here about trying to turn the audio down just a little bit. Maybe we can back it off a little bit. Hopefully that will help. Cool. All right. Excellent. Okay. So let's move on. Here's an example of the help output from GDB. So GDB has a very complete help inside of it. If there's any piece of information that you need to be able to get about how to run something within GDB, the help is actually quite complete. So we can ask for, for instance, help on break points and it'll then tell us everything we need to know about break points, trace points, watch points. It is very, very complete. It's like having the entire manual online and being able to pull it up at any point in time. So that's really quite handy at being able to see all of this. Now, as far as commands and definitions are concerned, one of the things that a lot of people are not quite familiar with in GDB is its ability to create your own commands and your own scripts. If you have a series of commands that you type frequently or if there is a long series of commands that you have to use in order to get started on a particular debugging session, you can actually define those as a command sequence within the debugger itself. And what we'll do with that is we will then define the name of the command that we want to define. And then once we hit return on that, it'll then prompt us for every single line that we want to type in. When we're finished, we'll type the single command end. And once we have done that, that will become an official command. So we can also then turn around and document those commands. And of course, all of us are low to do documentation. But as it turns out, we can in fact document our commands so that somebody else comes along and we can then check these commands into git. And then somebody else can check them out and have the same debugging tools and the same debugging scripts that you had used when you originally developed the code so that everybody kind of understands and they're all in the same, you know, basically all the same playing field. We can also go in and define c and c++ preprocessor macros. So these get into the macro define command. These are going to be visible to all of the inferior sources files. So if I'm trying to debug an application, and it has a pound of fine inside of it someplace, we can actually go in and manipulate that. So let me show you an example of how that's done.  And we have an application sitting here. We can define a new command. We can see let's define a command called Fred. And then it's going to ask me to type in a whole series of commands and then type in as the last one. So let's just do a print of I, that's our variable that we're messing around with in this particular program. And then we'll type in. And then from this point forward, all I need to do is just type command Fred, and it will then tell me the value of I. If I then specify that I want to step over, so let's do it next. And control L by the way automatically refreshes all of this. I will then do next again. There we go. So now if I type in the word Fred, I have the value one because I just incremented it up here online 16. So by being able to define your own commands by able to define the documentation for those commands, then if somebody said help Fred, they would then be able to see whatever documentation you had provided for that particular command. So it's kind of a clever sort of option here. Now one of the questions came in, can we set VIM to be the editor of choice in 2e mode? I've not seen anything that allows you to do that, but I wouldn't be surprised if there was some mode that you could put it in that would either be Emacs or VI. But I haven't been looking for that quite frankly, because I don't use 2e mode except when I'm communicating to a target. And the target doesn't have any GUI capabilities on it. So if I have just an arm target sitting out there that just uses a serial port, for instance, or just uses SSH over the ethernet, then 2e mode is really handy to be able to do debugging on that platform when I know I don't have the space, for instance, for having all the X-Windows libraries and things of that sort. So that is there a way to do it? There may be. I don't know what it is right off hand, however. All right, so let's go back to our code, or back to our charts here. We also have, in many cases, when we're trying to do a long debug session, we'll go home, we've had dinner, we come back, we then want to sit and we want to pick up where we left off, or we want to pick up and start everything over again, but we want to be able to key in a whole sequence of commands to get us in the right place, having loaded the code, having set the breakpoints, everything else that we need to have. So this is where the .gdbnit command comes into play. If you have a .gdbnit in the current directory, then basically what it will do is it will execute all the commands that it finds inside of there before it gives you the gdb prompt.  These are things that you will normally do inside of a gdb session. So rather than having to do this time after time after time, or just copy and paste between your environments, it's easy just to put this in a .gdbnit, and every time you start gdb, it will automatically load these commands, and you're off and running at that point. There is also a command, the dash x command line option that allows you to run scripts at gdb load time. So you can then specify, if you don't want to use .gdbnit, if you had your own script, for instance, then you could specify dash x, name of the script, and the application that you wanted to run, and it would automatically start gdb, load the application, and then execute the scripts so that everything was all set. You'd have your breakpoint set and everything else. Additionally, inside of gdb, if I need to be able to get out to the command line, I can use the shell command, and the shell command will then shell out to gdb out to the command line, and then execute whatever command you decided to do in the shell. So that's a really handy thing as well. Now, one of the things that is kind of a surprise to a lot of people is that gdb actually has an embedded, integrated Python engine inside of it. It doesn't actually just shell out what it does. There's actually a Python engine embedded inside of gdb. So we can execute Python expressions if we just simply prefaced them with the command Python, space, and then whatever Python expression we wanted to use, we can actually then execute Python commands. We can also write Python programs inside of gdb using the Python interpreter. Now, we see here a few examples, Python gdb.execute, that's going to execute some gdb commands. We can parse and evaluate things that's basically getting data from the inferior. We're basically going to reach out to the command line options. If we wanted to just get all of the possible things that Python can do, we can do Python help gdb, and that will then allow us to see all of the commands that are possible inside of the Python gdb package. But let me jump out here and I'll show you another example. And let's see if we can do this. So here at the gdb command prompt, what we can do is we can execute a command like Python print hello. There we go. So we just ran the Python, that Python command, nothing particularly magic about that command obviously. But we can then go into interactive mode. And whoops, let's not do that. Okay, so let's type Python. So now I'm in Python interactive mode. I can import class libraries. And then I can execute commands that require those class libraries. And type end. And there, it just ran it. And it said I am PID and it did the os.getpid command. So it's a way of being able to even get to the point of defining Python shortcuts that we can define as commands inside of the debugger. So just as we talked about the define command, we can also define commands that allow us to define Python commands and then execute extensive Python operations inside of the debugger. Very handy for us to be able to kind of mix and match. And many of us are very familiar with Python anyway. So it really helps to be able to pull that information and be able to get access to the Python interpreter without having to jump out of the debugger environment in order to get to the Python commands. So terribly handy. Then if we take a look at, of course, the bane of a lot of developers is the signal mechanism. And GDB is built on top of Ptrace, the process trace command. So when inferior gets a signal, the inferior is then suspended and the tracer gets notified. So what's going to happen, if we are interested in what signals we have, we can use the info signals command and that will print a table of what signals are out there and how GDB will handle each one of them. We can also then do an info handle and that will come back and tell us what those info signals, what that actually meant. So we can specify no stop, which means don't stop the program, but print that signal actually occurred. Useful for things like sig user one, sig user two. We can tell it to stop the program when the signal occurs. We can tell it to print a message when it occurs or no print a message. We can keep track of whether the program is going to pass it on. So this becomes important when we're dealing with threads and the ability to deliver a signal to the parent, which then gets replicated out to the individual threads from within the parent. So pass and no pass, that deals with being able to pass those signals on to threads that may be running in the same process address space. And we can then change a particular signal handler. We can do handle signal keywords. We can then turn around and deliver a signal on demand from GDB. So if we wanted to send the segmentation violation, basically the bus error signal, we could then signal sig seg v, and that would then send the signal to the main application. And then obviously if we had the pass turned on, then it would turn around and pass that signal on to the threads itself. So we can generate signals interactively from within GDB and test our applications that way if we happen to have signal handlers. Well, let's assume that for whatever reason we decided not to load the application when we ran GDB. There is a way of getting around that particular problem. It's called the file command. The file command, we specify the name of the file that we want to load, and it will then turn around and load that into GDB just as though you had invoked it from the command line. That is an important feature when we talk about debugging kernel code. If we're using KGDB, KGDB, when we run a KGDB of say the kernel, the kernel may not necessarily have had your dynamically loaded module in it when the kernel was compiled. Obviously it's not going to have dynamically loaded modules. So if we're debugging using KGDB, then what we'll do is we'll use the file command to load the driver into the debugger so that we can then start setting break points inside of the driver. We won't get into the details of doing debugging in the kernel in this particular session. We just don't have time to do that unfortunately. But understand that this file command also has some additional options in there, which tell me where to load the text segment, the data segment, read-only data, etc., etc., BSS. Those are all important things when we're trying to mess around with either code that has to be loaded in a specific location in memory, like for instance, if you're using U-boot, if you're trying to debug U-boot, U-boot has to have its text segments loaded at very specific locations in RAM for the board. So the file command here will then tell GDB where that code was actually loaded. And that allows GDB to reach out, generate the right addresses, and be able to debug that code as it exists on the target. So that's a very important command. Also, we've seen the run command. I did that a couple of times here. But we can set the arguments to the run command. So we can run the application, we can pass arguments that way, or we can use the set args command and set the arguments so that when we use the run command, it automatically passes the arguments we're interested in. If we did a show args command, that'll let you see the argument so that you can then go, oh, well, yeah, that's correct. Or no, that's not correct. Let me go ahead and change them and do another set command. So you can run it, and you can change the arguments, and you can have all of that set up inside of your .gdb init so that you don't have to type all that stuff all the time. Now, one of the real powers of GDB is the ability to reach out and attach to an already running program. So when we attach to an already running program, that program may in fact not have been compiled with debugging turned on. This is where you combine this gdb-p option and the ability to use the file command so that we can then attach to a running application. When we attach to it, it will stop the running application at its current execution point. So if it's say it's a server, it's running in background, it's a demon, we attach to it. When we attach to it, we will stop it. It will then have a breakpoint, basically, set at whatever place it was when we executed the gdb-p command. Then we can use the file command to load the executable with the debugging turned on. That is the executable that has the dash g option that was compiled with it. So again, this is usually targeted at applications where we have a very small memory footprint, either a small storage footprint or small RAM footprint on the target. And we still want to be able to do debugging on that target. Now, this is usually going to be focused on cross debugging and not necessarily native debugging. If we're in native debugging, we typically have enough memory that we can work with it without having to go through all this trouble. But the nice thing is that when we use the gdb-p option, we can then attach to any running application and it will stop it. And then we can start stepping through it. Now, if we don't have the source code for that application, then what we will see is everything in assembly language. So when you're looking, certainly you're doing debugging of U-boot or some of these low-level pieces of code, it helps to know the assembly language for the application that you're working with or for the instruction set architecture or the application processor that you're working on top of. So it's definitely handy to be able to have that kind of information and, you know, you don't necessarily have to speak assembly language, but at least be cognizant of it so that when you step into a piece of code that you don't have the source code for, you don't wig out when you see nothing but assembly language out there. It's okay. You can keep stepping and it'll eventually turn back into C or C++ or whatever language it was that you were actually using. Now, when we get ready to examine code, once the program gets loaded into the gdb, you can basically list any of the source files. So if the code is made up of multiple .os that are all linked together, that's fine. All you need to know is the name of the file and the line number or function name that you're interested in taking a look at. So when we use the list command, we can specify a line number, a file at a particular line number, a function name or a function inside of a file or a specific address. So all of those things work when I'm using the list command. Now, obviously, if I'm doing a list at an address, I'm typically going to be looking at assembly language code at that point, but that's okay. Again, that's not at all unusual when you're starting to do low-level debugging at this point. So you can specify the number of lines to list as a second parameter. So if I set the list size of, let's say, 15, then when I typed list and main, it would show me 15 lines instead of 10 lines. 10 lines is the default, by the way. You can also change things like the radix. Oftentimes, we'll be working in hexadecimal, but gdb defaults to decimal. So if I put in my .gdb init, if I put something like set output dash radix 16, then all the output will be in hex. Terribly handy to be able to translate all this stuff into something that makes more sense from a computer's perspective, maybe not necessarily from a human perspective, but most of us who've been programming for a while, we can equally think in hex as we do in decimal. It's like, how do you count in binary on your fingers? That's kind of interesting. You get to 31 on one hand. It's really handy to be able to think in those terms so that when you're looking at output code, you can do it in the radix that you need to have. Something still use octal, and of course, you can also switch that over to binary as well. If you want to know what options are available, then we can use the help set, and help set will then show you all of the potential options that are out there. It says the screen share is out of sync. I'm not sharing the screen at this point, so that would be why it's probably out of sync. Now I should be sharing the screen. Let's do something like this. Let's do a show. It shows me all the different options that are available to me here. Of course, I have to then do this layout. There's another thing that we can do. We can actually specify the focus, so we can specify focus and command. Now the focus is set to the command window, so now I can scroll around inside the command window. Those are the gdb command. Let's do a set output radix 16, and now it's set to hex. That would of course be something that we would typically want to be doing if we were trying to dump data structures and things of that sort that would all be done in hex. Let's go ahead and stop that screen share and go back to the actual charts. There's another thing you can do, which is kind of interesting, and that is once the code is loaded, you can actually call program functions from within gdb. The function will be called and the return will be printed and saved in the value history. It will use the same calling sequence that the language uses. Let's just go back out here to the screen share. I probably terminated that a little too soon. If we go here and let's just do a layout next, and there we go. All right, so now we're back to just the screen, the text itself, so let's do a list. You'll notice I have a function up here called sub, which is just a subtraction function. It's nothing particularly magic, but it's just a way of being able to show you how the command works. Hopefully you can see all that. Let's do this. Let's call sub, and we will pass it two parameters. Let's do, we take the first and we subtract the second. Let's pass it five comma three, and it returns the value two. Again, nothing particularly magic about that, but the nice thing is it's like, well, okay, just call this function. What is this function's inputs? What is this function's outputs? You can call it interactively from the command line. That's a really handy thing to be able to do, especially if the function that you're trying to call is in another file someplace that's been linked in. You can then list the function, so we can do list sub, and it would then show us the sub, the subtraction function up there towards the top. We can do that across any file that's been linked into this particular executable, so we can find that as well. All right, so let me stop screen share there, and that's the calling function. Setting variables. If we want to change the setting variables, any function, for instance, we can do set i equals five, or whatever language you happen to be using, and it has, let's see, I just got a message here. So it seems to be good for some people. Some of you are seeing the screen share fine, some of you are seeing it with a little bit of an issue. A lot of people are seeing it fine, so I suspect that may be a local problem. Refresh rates and things of that sort, depending on your individual data feeds and how much bandwidth you've got going out there. So I'll try to make sure that these things are as clean as possible. I'll just jump in and do a control L periodically to refresh the screen so that everybody has the latest and greatest, and hopefully that will help. But we can set any variable to any expression, and obviously we can use any syntax that's equivalent for whatever language you happen to be using. We can also, let's say that I have a function called call, a variable called call. Well, that overlaps with the call function inside of GDB. So how do I get to the call function, how do I get, how do I distinguish between the variable and the function? If I want the variable, then I'll use the set variable call equals expression. So the variable expression there, if I use the variable designator in front of it, then that says find the variable called call, not the GDB function called call. So if I happen to pick on one side, if I happen to pick a variable, and obviously, when we're writing our code, we don't necessarily know all the reserved words in GDB, we may actually pick a reserved word as a variable. It's not a reserved word in our language, but it just happens to be a reserved word in our debugger. If that's the case, then we just use this variable designator here, and it will find the variable in your code by that name rather than the GDB function by that name. Let's see here, printing expressions. Use the print command, obviously, that will print variable for me. I can then use, and this is a really interesting capability. Let's assume that I just updated the variable I, and what I want to do is I want to look at the previous value of I. I can then use dollar sign and the variable that I'm interested in, and it will jump back one version. I can also use multiple dollar signs, and I can continue to go backwards in all of the different values that the variable has. Obviously, there's a limit at some point, but it does give me the ability to take a look at the value of a variable before it changed, which is terribly handy. It's not quite the same thing as running backwards, which GDB does have the capability of doing in some cases, but in this particular case, we're just saying, give me the value that this variable had before I changed it. We can also use the at sign symbol. That's a binary operator for looking at consecutive data objects. Let's say I'm dumping a FIFO out of memory. I can use foo at number, some address, for instance, and then it will show me all the variables that are at that particular location, everything that's adjacent to foo. It's also very useful for being able to dump data structures, things of that sort. Definitely something of some use. If we have a known memory address, we can also use basic casting here. We have the X command, which allows me to show memory at a specific address. We're going to display it as a byte. It's a decimal byte, and we're looking at the address of test. That will then jump out, show me the address of test, and what it is as a byte. Definitely useful if I need to be able to cast it as a d-word, or a half-word, or a quad-word, something like that, then very useful to be able to specify that from the GDB command line. Breakpoints, of course, one of the big issues with GDB is our ability to set breakpoints. We have to understand that there are several different types of breakpoints inside a GDB. We have the standard software breakpoint, which is just the B command. That will go into the code and substitute the trap instruction for the breakpoint at the location that you just specified, and then it will store the original piece of code off in a buffer someplace, so that when I reach out with the B command, I'm going to actually reach into the executable, and I'm going to change the location at that variable, or at that line number, and make it the trap instruction that traps to the debugger. For instance, in the case of the X86, that's an int three, which is the trap to the debugger. This requires you to be able to actually modify the code. What happens here, if I'm running out of flash, then the normal software breakpoints don't work, because in order for me to set a breakpoint in flash, I would have to read the flash, change the variable, and then write the flash back out there. This is one of these cases where that becomes very problematic, because I'd be rewriting flash every time I wanted to step in instruction. That's not going to work very well. In order for me to be able to debug things in flash, I would then be using the hardware breakpoint. That's the H break option. Depending on your processor architecture, not all processor architectures have hardware breakpoints, and not all of them have a lot of hardware breakpoints. For instance, I think in the X86, I think there are three hardware breakpoints. In some cases, a PowerPC, there are five, ARM has three or four, I don't remember which ones, but in any case, the number of hardware breakpoints is dependent on your processor architecture. That's a useful thing in the sense that when I want to be able to set a breakpoint at a specific location in flash, I can do that, and what will happen is when the instruction pointer matches that hardware breakpoint mask, it will automatically stop the processor at that point and drop you back to the debugger. Extremely useful being able to debug things like Uboot, for instance, or an application. We'll show you here an example where you know that some piece of code is stepping on a variable, but you don't know which piece of code is doing it, so you can set a watch point up, and we'll talk about watch points in just a second, but you can set a watch point up that will then automatically stop if a piece of code writes to that location, and then you'll know exactly which piece of code is doing the writing, so if you've got some case where something is being stepped on, you're not sure what it is, then use a hardware breakpoint and you'll be able to find out exactly who's doing it, at what point in time, and where that code is located. Very useful to have. We also have temporary breakpoints. Temporary breakpoints, they'll stop once, and then they'll be removed, so if you just want to stop it as it's coming into a particular function call, but the function is a loop and you don't want to set there in loop forever, so we can absolutely then take into account, stop it once, check to make sure that everything's good going into the function, and then allow the function to continue to run and it won't hit that breakpoint again. There are also conditional breakpoints. Now, when we start dealing with conditional breakpoints, things can get a little wonky in the sense that you're basically after every instruction, you're looking to see whether that breakpoint has been triggered, so this can substantially slow down the execution of code. If you're trying to debug a race condition, this is not the way to do it, because by turning on this conditional breakpoint, you're going to slow the code down enough that the race condition may not happen, so use this sparingly and only if you need to have a particular, if it's a short duration run and it's not going to cause you any specific timing related issues, then obviously you can use conditional breakpoints. If it is a timing related problem, don't use conditional breakpoints, use hardware breakpoints instead. They're a much better way of being able to do some matching.  Yeah, that's a very handy thing to be able to do, just understand that that's going to change the execution speed of the code. The more of those things that you have in there and active at any one point in time, the slower the code will run, so use conditional breakpoints sparingly, obviously use them when you have to, but don't use them if you just because you want to use them only when you really need them. We can also issue a series of commands to be hit when a particular breakpoint is set, so if I do, for instance, here after the breakpoint in main, then execute the following commands, and then we'll type all the commands in and we'll end that last command with an end statement, and then every time it hits that breakpoint, it will automatically execute all of these commands. Absolutely handy for being able to hit a breakpoint and displaying a data structure, for instance, every time you hit this breakpoint, saves you a lot of typing. I mean, that's really the key thing here is to try and save people from having to do a lot of typing.  I see here the run command. I have the interrupt command, so if the application is running and I press the interrupt button, it will stop the application wherever it was when I hit the interrupt button. I have step next, so step will step one instruction. It will step one C source line, for instance, one source code line, but it steps into functions, so if I'm calling print and I issue the command step, it will then step into the print function itself. That, of course, you probably don't have the source code to, so you're looking at assembly language, and that's a fairly common mistake when we're just tinkering around with GDB and we're not necessarily used to all the functions. People will use the step command because it does step from one instruction to another from one source code line to the next, but it steps into functions, so if you're stepping into a function that you know is working correctly, then you'll want to use the finish command, so the finish command says run till you get to the return. Next, we'll call the next line, but then step over it, so if I was calling a function, my function sub, for instance, then if I used step, it would step into the sub function. If I used next, it would call the sub function, but not step into it, so we just simply say, yes, I know that works, go ahead and execute it and just continue after you've called that function. We have others, the continue command, if we've hit a breakpoint and we go, okay, I've seen everything I need to see here, press the continue button, it will continue running until it hits the next breakpoint.  I can also use these up and downs, these allow me to go up and down through stack frames, so let's say I have main calls sub and I'm in sub's stack frame, I want to look at the value of a variable up in main, so I would use the up button, the up command that would go up into main's stack frame and I would see main's variable and then I could use the down command and it would go back into sub's stack frame, so this allows me to move up and down through stack frames, especially if I happen to have a variable that has the same name, but in different stacks, not a good programming practice, I would remind you that, but sometimes it happens, somebody defines a variable called debug and another team is also working in another portion of the program and they define a variable called debug, well when I'm in a function and I do print debug, which am I looking at? Am I looking at the sub function's debug or the main function's debug? This up and down give me the ability to control which of those stack frames I'm looking at and if you're a real glutton for punishment, you can use the step I and next I, the steps instructions, so one assembly language instruction at a time, either step or next, so definitely if you're into assembly language instructions, this is a great way of being able to see those. One of the things that I talked about earlier is the hardware breakpoint and one of the basis that makes the hardware breakpoint what it does is this thing called a watch point. A watch point allows me to watch either an address or a symbol name and stop the application when one of the several things is happening. I can tell it to watch it, if I watch it, then if it gets written to, it will stop. If it's red, it doesn't stop. If I do an R watch address or symbol, then it will stop even if the variable is red and then I have some other variants here that says stop watch if thread three is modifying it. So if I have multiple threads inside the application, I can be very specific. I'm only interested in stopping if this variable is being modified by thread number three and then we can get into conditionals as well stops when the contents of a particular variable is greater than five. This would be another option to your conditional breakpoint, but in this case it uses hardware breakpoints and therefore doesn't have the same overhead that a conditional breakpoint would have if we were using standard software breakpoints. Here is an example of this. I have a breakpoint at main. I'm going to tell it to run and it hits the breakpoint and I then tell it to watch the variable x and continue. Well, when I modify x, it's going to immediately hit the breakpoint and show me the old value of x and the new value of x. So it shows me what it was before it got modified and what it was after it modified, but I'm still setting now at this location where it got modified. I can now do a little bit more debugging here in terms of figuring out, okay, what happened? Is everything okay? Why did it do this? Et cetera, et cetera. We can also debug threads. We can do an info threads. That will show me all the active thread IDs. I can then select a thread by ID and I can specify set a breakpoint in a particular thread. Other threads can reach out and touch that same breakpoint and it won't fire unless it is the thread that I specified the thread ID for. And then if I have a case where I've hit a thread breakpoint, I can then also tell it to show me all of the backtraces for the thread. So when I hit the breakpoint, I can have it set up the possibility of stopping the entire process and all of its sub threads. That's the default, by the way. Or I can specify using this set scheduler locking where I can then limit the execution to a particular thread. So I can either stop when I can stop the entire application when a thread hits the breakpoint. I can stop just the thread that hit the breakpoint. I can do either of those things by messing around with a scheduler locking mechanism. Now, also, we have the ability to generate core dumps. Now core dumps are kind of a lost art, if you will. The idea behind a core dump is it's going to take a memory snapshot of everything, the register sets, the instruction pointer, the stack pointer, everything that was executing at the time of the malfunction. By default, this is turned off inside of Linux. We can use the you limit command to turn them back on. And in you limit, we have to specify the maximum core file size in 512 byte blocks. Why 512 bytes? Well, because that used to be the sector size back in the old days. So this is normally turned off because if you have an application that terminates, it will leave a core dropping, which is a big chunk of memory that's basically been written off to disk. And people will forget about those things. And they have a tendency to collect. So the default case in Linux is to turn off core dumps. We can turn the core dump back on. And then when we encounter an application that terminates, it will then turn around dump the core. I will then be able to use GDB to load the application and the core file back into memory. And it will show me exactly where the application terminated. And I will see the reason that it terminated. And all of the memory and the register sets that were associated with the application at the time of termination will then be visible to me. Now, understand this is postmortem debug. This is not something that you're going to be able to then continue execution from that point. It's going to stop the application. The application is dead. But it does give me the ability to then turn around and look at what was happening inside the application at the time of termination. And this also works with most of the GDB front-ends. So whether it's DDD or Eclipse or Inside or any of those front-ends, it all works with all of them. You can also use it in 2D mode if you happen to be limited in terms of your graphics capability. So all of that just, it works the way it should work. Now, remote debugging with GDB and GDB server. This is another thing that happens. And this is a case where I don't necessarily want to be running the debugger on the target. The reason I don't want to run the debugger on the target is because the debugger is very intrusive to the target. What I want to be able to do is I want to be able to run a particular application under the auspices of the debugger, but not run the debugger itself on the target. That's where the GDB server comes in. Now, in the case of the GDB server, the GDB server is a very small executable. I think it's like four, six, actually maybe six k bytes in size. It's really, really tiny.  When I run that code under the auspices of the GDB server, I can then connect to it from the host via, say TCP IP or an RS232 link. It has several different mechanisms for being able to do this. And then I can set and debug the remote target. So as long as I have a network connection somewhere to the remote target, I can actually debug code on the remote target. And here's how something like that would work. I have GDB server. So this is on the target itself, GDB server, the IP address of the host, and a port number on the host. I then have the application that I'm going to be debugging. And in this case, I put it in background mode just simply so I get my console back. You don't really have to put it in background mode if you've got multiple SSH sessions going to the target for something along those lines. Then on the host, I can do target remote and the IP address of the target and the port number that it's sitting on. And these port numbers are just numbers that we made up. 1929, I have a good friend of mine from the old days who always used the date of the stock market crash here in the United States back in 1929 because nobody else uses that particular port number for any of their real applications. So he would always just pick that one. So it kind of hangs over from old times. But then at that point in time, as soon as I do the target remote, I'm connected to the target and I start seeing source code. I can also use the file command. And from the file command, I can then load the executable with all the debugging code in it into the debugger. And then once I've loaded it into the debugger, then I can start stepping, setting my break points and things of that sort. I can attach to a running program. So I'll specify GDB server, the host IP, whatever that is, and a port number, like 2345, just pick one, dash, dash, attach, and the process ID. That will then connect and wait for the target remote command to happen. And then you'll be connected in the debugger. If I wanted to do this from within DDD, then I would do DDD dash debugger, specify the debugger and the application. That then tells the DDD front end that I'm using the arm back end and the application that I want it to load. And then I would issue my target remote command, and I would be connected and debugging the application on the target from the host. So definitely a very handy thing to be able to do. Obviously, now all of that was related to debugging functions precisely. There are a ton of additional things that GDB can do. We just don't have enough time to go through all of them. So we're going to switch gears a little bit here, and we're going to take a look at some of the live tracing capabilities that are also useful for doing debugging. Now, functions like Strace and Ltrace, there are a lot of tools that are available to us, and Strace and Ltrace are ones that I use frequently because they give me the ability to take a look at all the system calls or all the library calls that are coming out of a particular application. There are no special compilation flags. You do not have to compile the code with the dash G option to use Strace and Ltrace. So what Strace does is it hooks into every system call. So let's assume that I'm running an application that's doing something, and when I click on this button, the application dies. But it doesn't give me any message, doesn't give me a core dump, doesn't give me any details, nothing that I can debug against. It was executing something when I pushed the button, but I don't know what. So what I can do is I can run Strace and attach to the application just before I push the button on the UI. And as soon as I push the button, I will then see all the interaction that application has with the system calls. Now, I can also put a timestamp on this if I want to have kind of more information about timing. And we see the example Strace output here. This is every function call that's being executed and its return status. So this, of course, very handy for being able to take a look at, I sent this function, what did I get back from it? These are all calling into the system call. Now, you can also use Strace to do things like, where am I spending all of my time? Here we see the dash C option, which counts the number of invocations. And we see that in this particular application, most of our time is being spent in the right call. So if my application is slow for whatever reason, then I know that this is probably the culprit that I need to look at. And I need to be able to figure out why it's so slow. We see the number of microseconds per call and the total number of calls. Now, obviously in this case, most of the time is consumed in the right call. But there's also a fair amount of time being, you know, we're seeing the select call being called frequently. And it's probably the next highest contributor to this overall execution time. Now, Ltrace does exactly what Strace does, except it does it for libraries. And in the case of Ltrace, we can Ltrace specific libraries. So let's assume that I have a third party library that was provided to me and I don't have the source code to it. But I want to look at the interaction between my application and that third party library. Well, I can hook an Ltrace up to it and watch every single function call going between my application and the library. The output looks just like Strace, except that now it's at the library level rather than the system level. I can limit the libraries that I'm interested in looking at. And I can also limit the functions out of a particular library. So maybe I'm only interested in the update me function in library XYZ. I can then specify an Ltrace only show me accesses to update me function in library XYZ. So I don't have to see everything. I just see a few things as I go along and I can limit what that output looks like. Now, PROC file system is a representation of the kernel's information. It's basically a dump of the system control block, the struct task struct that's associated with every application. And each process ID has its own unique directory information associated with it. So for instance, if I jump out here, go back to my command line and hopefully it will update for all of you. So let's do a clear here. sudo-i and then we'll cd into the PROC file system. Every one of those numbers that you see is corresponding to a PROC file system entry. So if I did a psax pipe to grep bash, and that would show me all the bash applications that are sitting out there. Let's take a look at PROC 22149. So 22149. This then tells me all of the information that the kernel is keeping about this particular process ID. So for instance, I can take a look at the memory map. This shows me the executable. That's the text segment. There's the read-only data segment. There's the read-write data segment. I have heap memory here, so I know exactly where that heap memory is being stored. I have locations associated with libraries. And again, we have the executable and read-only data, read-write data, etc. We also, this p here indicates that this is private versus an s, which would be shared. So if I have shared memory segments, then I would see them show up as an s. These are all private. I don't think I have any up. There's an s right there. So there's a shared one. And that's with a modules cache file of some sort. But we see the stack. We see the virtual system calls. So all that information, this is the virtual address that it shows up at. And this is the offset into this particular piece of code as to where this actually got mapped. So that's a terribly useful thing. I can also take a look at the status.  There's its task ID. Here are the number of file descriptors that it has, the owner who is running this particular application. This is the useful stuff down here. How much data is in the virtual environment, the virtual memory segment? How much is stack? How much is executable? How much are libraries? And how many page table entries? How much is being taken up in page table entries? So we can also see signals, things of that sort. Again, all very useful things when I'm trying to figure out exactly what's going on inside of an application. One other thing that I will show you is, if I go back out here to, whoops, I want to be in slash proc, I can look at interrupts. And this shows me all of the interrupts. Let me just do this all. Let me shrink it down just a little bit here. It shows me all the interrupts across all the processor cores. I have a hexachore on this platform. It shows me how they're routed. It shows me what kind of interrupt they are and what they're attached to. So this is very useful. I have fast EOIs. These are level triggered interrupts. These are edge triggered interrupts. These are message signaled interrupts. So these are PCI Express interrupts. So I can see exactly how my interrupts are happening and which processors are being used most often for servicing a particular interrupt. Let's see here. So let's go ahead and close that one and see what we can do about wrapping this particular discussion up. We have Valgrind, and it's Valgrind, not Valgrind, because it is the gate to Valhalla. There's Helgrind, Valgrind. There's a whole bunch of these little tools. We have Cashgrind, which basically looks at locality of reference inside of caches. I have heat profilers, and Helgrind deals with POSIX threads. So if there are race conditions inside of threads, Helgrind helps me find those race conditions. Valgrind is a complete of software. It is relatively complex, but it does work on ARM and it does work on x86, as well as PowerPC. So if you're trying to get into real fine grain debugging to find out exactly what's going on inside of the application processor and how it's interacting with your code, Valgrind is the key to being able to do some things like that. We have GPROF, which is a profiler. In the case of GPROF, we just simply have to specify the dash PG flag in order to turn on the profiler.  It will then produce an output that we can then take a look at in some detail. It'll tell us little bits and pieces. Let me show you here an example. So here we're looking at code coverage, which is also out of GCOV, the new coverage tool. So GCOV and GPROF, they're kind of old tools, but they're still very effective. Things like this, for instance, if I'm trying to get 100% test coverage, I now know that this particular line 13 is never being executed. It's never executed at all. So that says that my test case isn't exercising 100% of the code. And when you're doing safety critical types of applications, you want to be able to handle all 100% of the code. We can also take a look at branches, which branches have been taken, how often they've been taken, which ones were never executed, and how often those things were executed. So definitely a very useful tool in being able to figure out how all of that works. We have the Linux Trace Toolkit, Next Generation. This has been around for a while, but it's still a very useful tool because it gives us the ability to instrument the kernel and see the interaction between your host and the target and what's happening inside of the target is it executes a piece of code. So it's useful for debugging not only user space code, but also kernel space code. Now, LTT and G has been around, as I say, for a long time. It does have a user interface to it. Here's an example of that user interface running inside of an Eclipse plugin. So I can then see individual applications and see what their execution time was. I can then zoom in. Lots of very cool things for LTT and G. O Profile is kind of long in tooth these days. It is a statistical profiler. What it does is it allows me to sample the execution of the code and find out where it was running most of the time. So again, this is useful kind of like GPROF and the S Trace-C option, where I'm trying to figure out exactly how much time is being spent in what functions. Now, F Trace, this is Stephen Rosdett. He and a group of other folks were responsible for creating F Trace. It's a function tracer inside of the kernel. It was actually added way back in 2008. And it was one of the weird cases where the documentation was accepted into the kernel before the code was actually accepted into the kernel. But with F Trace, we have the ability to set up static trace points for event tracing. And we also have dynamic kernel function tracing. So we can look at the interaction between user space and kernel space or between two individual elements inside of kernel space. Very much like O Profile, it has to be enabled in the kernel. So you have to turn on F Trace in the kernel before you can actually use F Trace. F Trace has an output, a GUI front-end now called kernel shark, which looks very similar to what we were just looking at with LTTNG. It does give us the ability to kind of jump into a piece of code. And I can zoom in. So you see these little vertical bars here, these are kind of like a oscilloscope sampling areas. So I can then say, well, I want to look at just this piece of the code and it will zoom in on you. And then you can continue to zoom in and then you can zoom back out again to see exactly which pieces of code you're interested in. So you see the execution here across all the CPU cores.   It is available from Stephen's website. It's also available in most of the major distributions like Ubuntu, for instance, has this available. You can just download it and install it using your app yet. Now, the problem, as I mentioned here, isn't that Linux doesn't have enough tools. It's that Linux has too many tools. Debugging in user space at least is mostly done with GDB or it's equivalent LLDB if you're using the Clang compiler. Gprof and Gcov, Strace and Ltrace, they're really tools to provide you additional insight as to what's going on. Then tools like Ftrace and LTT and G give you extremely detailed interactions between the kernel space and your user applications or between kernel space entities for kind of a system-wide view. So there are a lot of execution information that's available to us. One of the questions that came up here, let's see here. Can we see the execution time using GDB? I don't remember anything in GDB that allows you to do that. I would then probably pop out to Strace. The reason I wouldn't do timing inside of GDB is because GDB, because it uses the Ptrace functions, it's going to affect the execution time of code. So this is not something that you can, if you're running something within GDB, you can guarantee that the execution time is going to be perturbed because GDB is sitting in there at all. So I don't think it would actually have a whole lot of meaning to you if you were to be able to see it inside of GDB. Let's see here. Oh, one other thing. I'm just about out of time here. So let me see if I can actually make this work. We've been talking a little bit about Ftrace functions. So let me cd here to, I always go a little bit long. Sorry about that. So let's cd to the tracing functions. So I've got Ftrace turned on. We see all the different options here inside of Ftrace. Let's take a look at the available tracers. So I have block IO tracers. I have wake up, real-time wake-ups. I have IRQ off time. I have function calls so I can get into detailed function trees. Let's just do something real quick. I'll turn on the IRQ off tracing. I will take a look here at some do a couple of things just to generate some interrupts. And then, okay, so let's take a look at the current tracer first. So it's IRQ is off is what's turned on. So let's go ahead and turn the tracing on. And then I will do a couple of ls's. And then I will turn the tracer off. And then let's take a look at the trace. And this shows me my interrupt off time and how long interrupts have been disabled. So there we go. That kind of gives you some idea. I wish I had a little more time. I'd show you Ftrace with kernel shark running. It's kind of cool. But unfortunately, I've run over my time again. So there we have it. We will open it up for Q&A. If there's any additional questions here, let me just make sure I've answered most of the questions. How does line editing work in TUI mode? Yes, so if you've selected the command line, then you can up arrow and it will allow you to scroll through the history. So you can then edit the histories there. You can just type over them again. Let me see here. Anything else? Okay, so we talked about execution time in GDB. Any other questions? I know that it was kind of fast here. Hopefully, everybody has been able to find the... I see I'm still doing screen sharing here. Everybody has been able to find the charts. I have here an application called Migrate. Let's go ahead and do a make on it. I'll just show you real quick what happens when we run kernel shark. This is going to run the application and it's going to keep it limited to just four different CPU cores. And then I'll run kernel shark. And it helps me if I can actually make toast plus and now go back and run kernel shark. And there is the application as it ran. This is actually one of Stephen's test cases where it actually deliberately kind of mashes the CPU to see how quickly things can migrate across. So it kind of gives you an idea of what it would look like if you were then trying to drill down into the actual execution of something using something like kernel shark. And you can get really, really detailed values here. I mean, you can tell exactly how much time, what time things took to execute, how often they did context switches, things of that sort. Very useful tool and comparable to what you see with LTT and G. But this one is a little bit easier to set up. LTT and G requires some additional drivers be loaded to the kernel. So it's fairly automatic these days. But F trace is a much faster and easier to work with. And that's my personal opinion. And it's probably worth what you paid for it. So all of 50 bucks, right? All right. So I think there's no more questions. And I don't want to run into anybody else's time. So if you have any questions, let me just go ahead and stop the screen share. And let's go back just to my information in case you want to try and get ahold of me. I'll get there. There we go. So there's my mail, mikeandersonptr at gmail.com. Or I'm on Twitter at at Hungar. Don't pay as much attention to Twitter as I should. But you can definitely reach me there. Or you can reach me over at LinkedIn. I'm on LinkedIn as well. If there are no more questions, then I will turn it back over to the powers that be. Thank you very much for your attention. I hope you had a good time here and got something out of it. See you later.