Hello everybody. Welcome to my talk. My name is Lev Estrovich, and I work at DHL Research. And today we're going to be talking about a disciplined approach to debugging. So first, a quick word about myself and my company. So I work at DHL Research. It's an independent research lab found in 2002 by David Shaw. He used to manage a hedge fund, also called DHL. But now he wants to advance science and specifically using molecular dynamics. So the high level goal is to advance, to make an impact in the fields of biology and chemistry using computer science and computer hardware. Specifically, we built a supercomputer called Amthon for molecular dynamics simulation. I'm not going to be talking about very much of that in this talk. But what I do is system software. So basically kind of operating system level software for this supercomputer that we built. And during the course of this, there's definitely a lot of things break. And you have to debug a lot of things. I've generally had about 20 years of experience in the field working at places like HP and Sienna Networking Company. So I've worked for everything from little embedded devices to obviously supercomputers. And hopefully I could provide some good background on debugging techniques and tools specifically under Linux and embedded Linux more particularly. So first of all, why do we want to talk about debugging and all? Well, about half, if you're a programmer, about half your time is spent doing administration tasks. So things like filing bug reports, talking to other people, HR, and all sorts of things that go along with working in an office or at home as the case may be now. And half of the time that you actually spend programming, half of that time is actually spent debugging. So these statistics you should probably take with a grain of salt. But there's about 1.5 million software developers in the US. And the total wages estimated by the US Department of Labor Statistics is about $150 billion. So about $37 billion at least is spent debugging in the US. And yet there's not that many resources really formally to teach debugging. If you look at the MIT course catalog, the word debugging only appears three times out of 10,000 words. I'm seeing some questions that people can't hear me. Is everybody can speak up or type in a question if anybody else can't hear me? OK. Sorry about that. Great. Thank you very much. I see some responses that we can hear you. So whoever can't, there must be something on your end. Sorry about that. OK. So let's continue. So before we even start talking about debugging, you really want to start first by testing. And even before you do any debugging, you should probably do testing. Unit tests, integration tests, basically the more expensive your software, the more testing you want to do, because debugging is always more expensive than testing in the beginning. And generally about writing software, I love this quote from Brian Kernigan, one of the authors of the original book on C. He says that everybody knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you are when you write it, as can be when you write it, how will you ever debug it? On the question of, will these slides be available? Yes, I will ask the Linux Foundation organizers, and we'll make them available after the talk on the skip page. So basically my approach to debugging is basically I wanted to give it a clever title. So to reproduce, observe, and bisect, which spells Rob, and this apparently is a character in the Nintendo video game called Rob. But basically the idea is we want to reproduce the problem that we're seeing. We want to observe and get as much visibility as we can into the problem and bisect is to find where the problem lies. So why do you want to reproduce instead of just starting to debug right away? Well, generally when the users report bugs or when you see a bug happen, you're generally are missing a lot of information. There's a very low signal to noise ratio. Users don't really know what you want to know about the bug or why it happens. They just say, oh, this broke. So to properly instrument and observe and figure out what happened, you really like the most important step is to reproduce the problem. In fact, even if you think that you can look at the code and find a fix and say, oh, I'm pretty sure this is what happened, you don't really know that you fixed it until you've reproduced it, then tested it after you fixed the bug and said, yeah, this really fixes the problem. A lot of times you can think that you found the problem, but you didn't actually. So until reproduced, you can't actually know for sure that you fixed it. So observe what am I talking about here. Basically, you want visibility. So the visibility is the first most important thing to produce a problem. And the second is to have visibility into the program into what you're trying to debug. So the picture over here is the very first bug found in a computer in the 1950s, I believe it was in the INEAC. And it was an actual bug that got fried in one of the relays and created a short. And the little bug was taped over there with a piece of tape to the paper. So to find these bugs, you need visibility into the program. You could use it in lots of different ways. The simplest one is just put a lot of print statements in your program. A better way is to use a logging framework. I have some links here to some very nice ones, like NCC, C++, Google's G-Log, and Python. There is a great logging module included. You can log to the system log from the shell using the logger command, or just redirect all your output to a file. You don't want things to disappear. So whenever you have something that could produce output, you want to drop it somewhere. And when there is logging aggregators, like Splunk or Elasticsearch, I really like Splunk. It's very nice. It's a way to aggregate logs from lots of machines. And then you can kind of see what's happening on many different machines at the same time if you're writing in a cluster and so forth. And the last point is Asserts. Asserts are your friends. If you're sure that something is true at a certain point of the program, put an Assert there, because you never know what can go wrong. And it'll provide a useful stopping point if things have gotten the same. So there's lots of great tools in Linux. I think in Linux, basically, you have tons and tons of tools to have visibility and the biggest problem is which tools they actually use to get visibility to your program. The first thing I do generally when I need to debug a Linux application is to use Strace, which traces the whole system calls that a program is making. GDB, of course. There is correct lots of great resources about GDB. GDB is also useful for debugging core files if a program crashes. Don't forget to compile with debug symbols. So there's the dash G option to GCC. You can do a man GCC and it'll describe all the various debugging options. But in general, debug symbols don't add. They add a little bit of size to your executable. They don't slow down your program at all. So unless you're in a very tight embedded system, you really do want to compile with symbols. Another trick that you can do is if your executable space is very tight on your platform, you can compile once with symbols and once without. So once you compile with minus G and once without, the two binaries will actually be interoperable, meaning if you use GDB server on the target and the GDB client on your developing platform, you can actually use the program file with symbols to debug the one without symbols. You can load up the symbol file from a different file. You can also use the file command in GDB. So if you started running your program with the one without symbols, you can then use the file command in GDB to load up the symbols from one. I have a question is how can we debug if from Pankaj, I believe? How can we debug if Linux is stuck in booting? For that, you need KGDB. There's good resources on that, but you'll need the second computer to attach to the first computer, or you can use the built-in KGDB debugger. So you should just Google KGDB and follow the instructions there. Always use WALL and WER, so that's to turn on all warnings and all errors in GCC. They're generally super useful. It doesn't hurt to turn them on. And the things it finds sometimes seem pedantic, but it's generally a good idea to fix all of them. A word on reverse debugging. It's pretty cool, but I practically haven't found it really super useful. Once you kind of know what you're looking for, you can restart your program and get to that place from the beginning instead of going back. Other people may have had different experiences. GDB does support some level of reverse debugging. There's also a tool called undoTP. We'll also talk about Wireshark and TCP Dump. And if you're debugging any hardware, you want to get visibility into that protocol. We'll skip that. So this is a Linux performance observability tools from Brendan Gregg. He's an engineer at Netflix, and he has some great resources on his web page about all the possible visibility tools in Linux. There's way too much to go into right now, but you can definitely go on his website. He gives great overviews of what to look at. And finally, bisect. So the idea is you want to do a binary search. Once you find that there is a problem, once you've reproduced it, and you can observe it, you now want to do a binary search to find out where in the code the problem actually is. So one way to do this is to kind of say, if you don't have an idea, make an educated guess. If you don't have an idea where it is, just guess. First half or second half of the program. And then you could just disable the other half, run it again, see if you hit the bug, and iterate. You could do other things to switch things up. You could switch compiler optimization levels. You could switch tool versions. Generally, whenever you do these things, do one at a time.  And you also generally want to, what I like to call the trust stack. You should trust your program the least. You should trust libraries that you're using or probably have been tested by more people than just you. So they're probably more trustworthy. The compiler, the OS, and the hardware in that order are less at fault. It's always kind of, you always want to blame the computer or the hardware or the OS being wrong, but most of the time that's not the case. Although sometimes it is. So I like to classify bugs into hard and soft errors. So the hard errors are actually easy to find. The hard errors is when you have a crash or unrecoverable error. So things just go off the rails and break. And it's generally pretty easy to produce and easy to figure out what happened. The soft error is kind of an intermittent error. And those are terrible because you don't really know that you fixed it because maybe it's just didn't get lucky and didn't happen again. So what you want to do is you want to try as hard as you can to reproduce the error and to turn that soft error into a hard error. So how do you do that? Well tracing and logging, you don't give you an idea. You could look at external events like logs, network, CPU load. You could try to replicate the error message that you see. You could stress test the application or of course you can use GDB or the bug error. So I'll give a few examples and I'm gonna answer, see there's a lot of questions I'm gonna try to answer most of them at the end. So this was an error that we saw that it's a weird error. It looks like an IO error in the program. So it's highlighted there in red, you could see the output. So we run an executable and it says boost file system reports input output error. So input output error is caused by an EIO. And generally this is very hard to reproduce because this generally only caused by bad hardware or bad network NFS shares. It's intermittent, meaning when it's bad hardware it won't always give you an EIO. So how do you reproduce something like this? Well, I actually tried to reproduce this first by just removing the file but it does not result in the same failure. The call works just fine and says the file is not there and doesn't cause an IO error or anything that causes the program to crash. Well, I looked at error.h that can be found in user include has some generic error.h and then you links distribution. Just look for an error code that's similar. There's one called elope, too many symbolic links. You could simulate that by making two links. You link file A to file B and there's some other thing from file B back to file A. And then when you cap A, you get this error too many levels of symbolic links. Well, I could try running a program against that. And then I wrote here a test program that does the call a fail boost file system exists on an non-existing file foo. And then red tried on this too many symbolic links file A. When we run the program, the first line outputs normally foo does not exist but on the second one, it throws an error and says terminate called. So that shows you that C++ through an exception. Now, why would checking for existence of a file through an exception? Well, if you look at the documentation for boost, it turns out that somebody decided that it's a really good idea to return one of something exists, zero of something doesn't exist. And in some cases as specified in a whole different long legalese page of call error recording, in some cases will throw errors. This is a really terrible idea, I think, but we found what the error was and there's a different version of the exists call this does not throw exceptions or just switched to using that. I don't know why they decided to do that. A different bug. We start a program and it fails to start and it gives us useful error message saying that Intel MKL, which is the Intel math library, could not load this library. Well, you know, that's straightforward enough but then you check, is that library actually there? Oh, yes it is. What? How, why is it not failing to load the library? This library is not on NFS. It's on a regular disk. So why would the program fail to start when that library is actually present? So second step would be to observe, to try to figure out what is going on here. So like I mentioned before, S-Trace is great for doing that. You can look, you can run your program under S-Trace if it's reproducible. You'll see what it's trying to do. So I took an S-Trace of a working run of this program and a failing one. And when they begin to diverge is the first red line when it does a get CWD. Get CWD is a system called it gets the current working directory. So when you type PWD in your shell, that's what it does. It says, where are we right now in the file system? And it's getting an email end. There's no such file or directory. So that's weird that it's checking its current directory and the current directory does not exist. So soon after you see the right call until MKA will fail error, which is the one that we're looking for. So why did that happen? It looks as though something in the MKA library really, really wants to know which directory you're in. And if it can't find, if it gets an email end on the current working directory, it fails with a really confusing error message. So let's try to find to check that this is really the problem. So when you base sect, we make like an educated guess. Let's make an educated guess and write a test program. So this little bash script makes a directory test. The parentheses here say in a subshell you wait for a one second and then you run the program. That was the failing program for us. Meanwhile, in the top shell script and that's the ampersand here says do this in the background. So it's going to execute another sub shell in that sub shell is going to wait a little bit and run the program in that directory. Meanwhile, in the main program, we're going to remove the directory that that sub shell is executing under and then we wait for everything to exit. And sure enough when we run this shell script we get the failing error message. So now we've reproduced it and found the actual error. So MKA really doesn't like when you remove that underlying directory. Homer Simpson is very upset. So GDB. So GDB is really great at debugging anything it can actually debug probably any language you can think of anything that's compiled by GCC which is not the GNU C compiler but now the GNU compiler collection which compiles basically any language and it can also debug things with LLVM if I'm not mistaken. So you can GDB a live process as well as you can start a new program for debugging or if you have a process that's running you can GDB the process with it. You do have to be careful because when you do that it stops the process. So if it's something that's important to keep running you probably don't want to run GDB on it directly. However, what you can do is you can use the G core tool that will generate a core dump. The core dump is basically an image of the register the memory of the running process. So G core will cause the program to dump core to generate that image without stopping the process. And then afterwards you can use GDB to debug it. In fact, GDB from the core file can even pick up which executable it was. So if you're not even sure, if you don't remember which version of the executable or which one you ran the path is actually will be in the core file. It has really good thread support. You can look at the man pages it just you could do commands that to look at all the threads or apply a command to all the threads. For example, the bottom one here is a thread apply all backtrace. So the backtrace command will then be done on all the threads in your program. It has very good support for debugging embedded for programs which is particularly useful for this conference for embedded systems. You wanna run something called GDB server which is a very small, several kilobyte stub that can just trace the program and then all the command line and all the other processing in GDB is done on a client. So the GDB connect command you could look at the manual pages for GDB server and explain exactly how to do this. A very useful feature if you have memory corruption that this happens a lot where you know that you've some memory has been trashed there is something called watch points. So this is kind of like break points except they break if something has written or read a certain value in memory. This is super useful for when you know that in your program something's gonna corrupt something's gonna write memory but you don't know what it is. You should keep in mind that what you really want to use are hardware watch points. The X86 architecture provides four hardware watch points other architectures provides different numbers some don't have any, you really don't want to if you use more than four or if your architecture does not provide any hardware watch points your program will get really slow because then GDB has to intercept basically every memory access and check if it's it for the address that you're watching for. But if you're using the hardware watch points and the processor at native speed we'll just check every memory access and break if you've tripped on a watch point. Conditional break points are also super useful you can break on a certain location right there's like a regular break point and you can give it a condition. So you say any condition that's evaluated by GDB so my variable equals five. So you say break on this location if and then you give the condition to give just keep in mind that GDB will actually take the break point and evaluate the condition every time. So if it's something it will slow down if you're going to be hitting that break point many, many times. You can observe, you can do commands at a break point. So if you want to print out a bunch of things every time you hit that break point you can use the command keyword and give it a bunch of lines to do after that break point. Finally, the useful features catch. This is useful for C++ specifically but could also be used to catch forks or signals. And C++ is very useful to catch exceptions. It will break point you as soon as anything throws any exception. So here's an example of debugging a hanging process with GDB. So we had a periodic hang where we run this big application and every like 10,000 jobs and this application runs for like 30 minutes. And periodically we see a hang where it just stops there and we have a timeout after two hours that kills the application. So the first thing we've tried to do is to reproduce but we couldn't figure out which inputs would actually cause it to hang. So we added code on the schedule level. So that's the process that starts our application to monitor the application. And if we see no output for about 10 minutes then we use G core to ask to get a core dump of the application and we can debug both line later. Now core files that are produced are really big because there's lots of memory that a replication uses. However, they zip very well. In fact, when your program runs, it'll use a lot of memory but a lot of that memory will be zeros or repeating bits of text and G zip will compress it really well. So we use GDB to analyze the core dump file of this hung process. And when we get into GDB, type info threads which will tell you where each thread application is. This one has lots of threads. All of them are in one thread. One is doing something interesting. The rest of them are in P thread can't wait. They're waiting on some P thread condition and one of them is stuck in read. Now read is a blocking system call meaning when you call read, you will not return until you get the results of the read. So that's interesting to follow up. That could be a reason why it's hanging. What is the read blocking on? So we'll switch to thread number three. The one that was blocked and see what's going on. Now you see this is in the loop P thread implementation that does not compile the debug symbol so it can't tell us which registers or sorry, what the arguments sorry, what the arguments for the read call currently are. However, we could Oh, I have a question. How do we know it's stuck in read? Well, the reason we don't really know it's stuck in read but we can look at the things that we think it could be stuck in. Blob 64 is the only other function here. That's not the thread can't wait. And Blob 64 is a very simple function. So that doesn't seem to be something that would be stuck in. Read is the only function here that should take any amount of time. So if you look at read, what you can do is you can type in for registers. When a core dump is generated it also dumps all the registers at the time. And if you look at the Wikipedia page for the x86 calling conventions it'll tell you which registers are used in the system call or actually in any function call. I've highlighted them here in the green, blue and purple and the read takes three arguments. And if we look at the registers of those arguments in that thread, the green one RDI is zero. The blue one second argument is some pointer and the purple one RDX is four. So that corresponds to file descriptor zero some pointer to a buffer and a count of four. And in fact, application does have a read of four bytes but it's supposed to be reading from a socket. How about file descriptor zero is standard in. And that's a good explanation of why it's waiting. It's waiting for input and standard in which is not actually connected to anything. This is a offline application. So looking at our server, it turns out what happened was in this application which is a server, there was a cleanup function that closed the socket and a different thread that was in a loop and waiting for the thread to exit. Every time and through the loop, it would do a read. If the read returns an error, it would exit. Otherwise it will go and process a data. And basically there was a race condition on exit. If you close the socket and then cleaned up the C++ object, the M socket variable could then be reused by something else. The thread that was looping will then use the M socket. It turns out that something set after the main thread exited, something you reuse the memory and set the M socket variable to zero which is just some garbage value. And the read from any other garbage value would have returned an error right away and the reader thread would have exited. However, a read from zero from standard in would get stuck waiting for somebody to take something in on the console. The fix here was easy. There was the two green lines that added an exit flag and a wait for the reader thread to exit. So I promised I'd talk a little bit about Wireshark. I'll try to get a little bit of that in. Wireshark is great at debugging any sort of network events when you have anything you have to dump with a network, you have to do with a network. You can use TCP dump which uses, which is a command line tool to gather network packets and then you can visualize them with Wireshark. Keep in mind that language to specify what you're capturing is slightly different for TCP dump and Wireshark. And in fact, Wireshark uses the same libpcap which is the underlying library that TCP dump uses. So the capture language on Wireshark is also gonna be different than the filtering language for viewing. So you can do a man seven on pcap filter that's for capturing and man four Wireshark filter that's for viewing. They're very similar but slightly different as you can see in the slides. But you can capture a specific port and a specific host and then analyze it later. You wanna capture as much as you can without overwhelming the network without overwhelming the kernel, sorry, when it's capturing all the packets so it doesn't drop packets. And then you can analyze later to see what you see. Conveniently Wireshark when you open it it's a nice GUI, it'll break down all the packets for you and show you in different colors, all of various things. It could follow TCP streams even or it could give you the whole conversation. So over here, I'm not sure if anybody can see it's pretty busy but the idea is you wanna look for the red and the black things are the errors. So over here there is a get of a HTTP get which Wireshark conveniently decodes and prints over here circled in blue. And then a few times later we see a TCP retransmission. So it's highlighted by Wireshark and I've put a little red highlight around it. Basically when you see a TCP retransmission that means the previous packet was lost. So we see there's packet loss on this link. Finally, the word about hardware debugging. Again, observability is what you really, really want if you're debugging iSquared C, PCIe, any sort of USB, any sort of hardware interface you want to get the tool that can give you visibility on that hardware interface. For iSquared C, I've listed several debuggers, Saley or Beagle analyzers are all good. USB, a little known fact is that Wireshark can actually display USB traffic and you can snoop traffic on the USB bus with the following commands. You do mod probe USB mon and then TCP dump and the interface is gonna be USB mon one, two, three or five depending on the number of your USB bus. So your computer can actually snoop traffic on the USB bus output it to a file like a regular TCP dump on the network it could just dump USB packets and Wireshark supports looking at those packets afterwards. If you're using PCIe or NVMe, SATA, any sort of high speed interface, you can use a large protocol analyzer. The CROI gets makes great protocol analyzers for PCIe, I've used them, they're super useful, also super expensive. So I'll give one example of a hardware debugger. So we had a Xilinx FPGA that was controlling an iSquared C bus. Once every several days in production, we would have devices that would just stop responding and we didn't know what was going on. So the one step was to reproduce and figure out what the heck is going on in this iSquared C bus. So we increased the polling frequency of our temperature sensors and so forth that were on iSquared C bus and to sense them every second every, sorry, 10th of a second instead of every 10 seconds so that increased the probability of hitting the error about a hundred times. Now we got hangs like every few minutes and we can actually produce this in the lab and attach a iSquared C analyzer or a logic analyzer to this. We couldn't get visibility into the Xilinx FPGA because the IP block wasn't encrypted. So we used an external logic analyzer and just took the probes onto the wires of an iSquared C bus and iSquared C is slow enough that you could do this on a regular analyzer. Now, this is not the capture from the actual analyzer that we took but this is a very similar problem I found online. So on the top in yellow, you could see the clock. The clock is one, the iSquared C has two lines, clock and data. Basically clock will always clock up, down, up, down, up, down and the data will be ones or zeros as you can see on the green at the bottom. So if you look at the zoom region that highlighted in red as the clock here is rising pretty slowly and as it rises past the threshold of where it switches from the clock switches from a zero to a one, there's a little glitch there. You could kind of see where it glitches past. Now, if your device and a lot of iSquared C devices are supposed to unglitch this, they're supposed to allow for slow rise times but not all of them really do. If you see glitches on your clock line like this, you could get an extra clock cycle detected by your hardware. So your hardware thinks that another clock has been issued on the bus and then everything is off by one because the thing that's sending the iSquared C signal thinks that it's only clocked once and thing receiving it thinks that things got clocked twice and everything breaks. Once we saw this, there was a very easy solution which is to put more pull-up resistors on the bus to increase the rise time. There's a good link here from debugging iSquared C from Texas Instruments that discusses similar problems to this. So again, the point of this is you want as much observability on the bus that you're developing as you can. We would have never caught this just in software without actually looking at the logic analyzer. So that's all I got in the time allotted and now let me try to go through some questions that we had. I think I answered some of them but we'll go through some of them. So, I'm sorry, advanced timer sprouts your name. So let's see here. We have some comments from, second. Okay, are the slides gonna be available? Yes, we'll post the slides. So there's a comment that's saying that from Steve that a cert is actually really easy to use and doesn't really provide a runtime cost because it compiles away in a release build. That's a very good point to make. If you have a release build where you really want the cert to keep your certain release build, you want to have your own version implementation of a cert that won't get compiled away in the release build. How does Leandro asks, how does LLVM Clang compare to GCC in terms of debugging? It GDB supports LLVM Clang. I think there's basically no difference in LLVM Clang but I could be wrong. I don't have that much experience with LLVM. Steve also points out that there's other useful warnings from GCC like W Shadow and W All that doesn't turn them all on. That's true. W All is basically a good starting point. It'll turn on most of the warnings that you care about. There are other warnings you might be more pedantic about and there's also other code coverage tools like Coverty and other tools that you could use to do static analysis on your code that could catch a bit more things. Larkin, I think, sorry, asked if I can elaborate on reverse debugging. So reverse debugging is basically it will allow you to step your program backwards. As your program executes, it records state of basically how everything was and then you can step backwards. So instead of doing next in GDB, the previous it'll step back in time. This does incur a cost because it has to record the state of your program at every point and it can't undo things like back as you sent in the network. It can't take those back. I'm not sure if it can then do file writes that also seems difficult in practice. I really haven't found it to be super useful. Brafadumbal, sorry, if I mispronounce your name, asks, well, will Gcore work even when the process is hung? Yes, Gcore will work even if your process is hung and in fact, that's very useful. If your process is hung, however, you can cause, you can just attach to it with GDB directly. You don't need to take a core dump, but if you do need to, you can use Gcore. Gavin asks, I experienced rare 3% memory corruption that is caused by a task race condition on the MIPS processor. Is there a good case for a GDB server and hardware watch points? Yes, if you have memory corruption, watch points are great. Make sure that your hardware again supports watch points because otherwise you're gonna impact the timing and your program run really slowly. You don't need GDB server if you can run GDB natively on the target.  F.Umson asks, what correlates the two log entries? I think you're referring to the TCP entries here, I think. What correlates them is you have to look at the addresses of the sender and the receiver. So look at the IP address of the source and the destination and the port. So basically your tuple in TCP is your source destination, source port and destination port and those are going to define a connection and Wireshark can also filter. You can just like right click on any one of these entries and say follow TCP stream and then it'll only give you packets that correspond to those particular posts in that particular TCP conversation. Joe asks, does any packets live for support IO on Windows? I don't know, I honestly don't know, I don't have that much Windows experience, I'm sorry. Shiva Marthi asks, am I covering F trace? I am not in this talk, but there is lots of great resources. You could look on Brendan Gregg's website, like I mentioned before, or you can just Google it, there was a good talk yesterday on the debugging GDP tutorial that covered F trace as well. If there's gonna be like an offline way to watch it, I would suggest watching that. Pankaj asks, why were the glitches in the SCL line occurring? So this is in an I2C. Basically the more devices, an I2C is a passive bus, the active device pulls down the line and then when it's not using the line, it lets go of the line and lets it flow back up to its full voltage. That's why the resistors on those lines will pull up resistors because they're pulling the voltage back up from zero to the line voltage. The more devices you put on there, the more capacitance there is in the line and the slower it will take to get back from voltage of zero to your desired voltage. So you need a lower resistance on the line, basically to pull it up quicker. This happens quite a lot in I2C buses. There is different solutions that are discussed in that TI document. He also says the drive strength of GPIO could have changed the rise time, that's true. However, again, the I2C is driven low, not driven high, so your drive strength would have caused the drop-off line to be slower. Here the drop-off line is pretty fast but the rise time is pretty slow. Rachel asks, do I use fault injection testing to support debugging? Yes. In fact, the TCP error here was done using fault injection testing. I find Docker containers are super useful for this. You can start things up in a Docker container and cause it to drop packets. You can cause it to power off the machine entirely or to hang it or to hold it or to cause all sorts of affairs. They are a bit messy and annoying to set up and they're not super useful on embedded systems because it's kind of hard to set up a Docker container for a different architecture on the host architecture. But if you're debugging in the same architecture, I think containers are super useful for a false injection testing. Jorge asks, how do you analyze a core dump? So basically what a core dump does is it takes a snapshot in time and then when you run GDB, you run GDB, your program and the core dump and the GDB will load up your program as though it was instantiated at that point in time when the core dump was taken and then you can look at the registers, you could look at memory. You can't advance time because it's just again a snapshot in time but you can use it to look at the current state of the entire program, examine all the variables and registers and threads and what everything was going on on that time. Backtraces will work. It'll tell you like for each thread, you could see what the backtrace is, who called, who and what your stack looks like at that point. Emil asks, recommendations for tools for debugging multi-threaded applications. Oh, GDB is really good at thread support. If you have race conditions and things like that, you can use Valgrind and Helgrind. Helgrind is a subset tool of Valgrind that tries to detect race conditions in your code and it's generally pretty good. It does slow down your code a lot by like a hundred times because it's effectively, it's almost like emulating the CPU. Jeremy asks, are there any tools for debugging PC transactions besides a PCIE analyzer? Are there any tools that can record data in a bus that could be seen by the root complex? I'm not aware of anything. You need really fast snoop sort of capability to log what's going on in the PCIE bus and most processors don't have that much, that visibility into the PCIE bus. So other than the PCIE analyzer, I'm not aware of anything that would capture all the transactions on a PCIE bus, unfortunately. We have a question of Thomas asks, GDB has a lot of hidden power, but there's very many accessible and clear resources isn't very many accessible and clear resources for discovering it. GDB has a super useful help interface. So just typing help in GDB and then perusing the different sections that gives you sections on break points and trace points and watch points are all really useful. The online manual is actually quite good. I would also just check out the GDB documentation on the GNU website. There's a good O'Reilly book on GDB as well. Andreas asks, is there any tool to dump and observe HTTPS traffic? Unfortunately, because of the nature of HTTPS, unless you capture it before it's encrypted, there's really no good way to observe HTTPS. It's supposed to be secure to prevent you from observing it. Orlin asks, are there any tools with a GUI on top of GDB? Yes, there's several GUIs on top of GDB. The most common one is DDD. The data displayed the bugger, I believe. It's pretty old and clunky, but it's workable. Eclipse supports GDB as a backend. So it can be a GDB GUI, Microsoft Visual Studio Code supports GDB and it can be the front, the GUI for GDB. There's so lots of tools support GDB on the backend to use your GDB text interface. I just like to use the text interface, but I know other people prefer the GUI. In fact, GDB has a kind of GUI text mode. I think it's minus Q. It shows you your program in like a curses window. So it's a combination of text mode and GUI. Andree asks, do you recommend using debugging flags and production baseline accommodations? You should always do minus G. That doesn't really add any time to your program. It does add symbols, so it adds a little bit in your image size. Compiling with less optimization, however, will make it easier to debug because things will be more sane as you step through them, but it will add a lot of, it'll slow down your programs. I don't recommend doing that in production. So Lechen asks, is there in general standpoint, any formal techniques to the bugger program? Like instead of test observable, okay, we can infer a bug for how a program was written formally. There are, there is some research about it. I don't know if anybody who actually does this much, there's basically static analysis tools that try to catch bugs. Basically, this is a way to write code that's less buggy, is what probably you're looking for. There is some research on that, but it's beyond the scope of the stock, as you've mentioned. Encore asks, are there any recommendations about debugging Linux kernel driver interrupt related issues? KGDB, which is built into the kernel, is useful. The kernel generally provides, oh, one interesting thing in debugging the kernel is there is a proc kcore file that will give you the current core. And you can use, if you have your Linux build, you can use, you can use GDB to actually debug the live kernel. You can't step through it, obviously, but you can look at the current, effectively core dump of the kernel and see what its state is. Or you can use the build-in KGDB tool. I'm gonna do one more question, and then I think we're out of time. So, CRA asks, in some cases, the process may get hung in the kernel that could prevent user space dump from either happening or completing a full kernel dump should be used in cases like that. Right, that is a good point. You can try to use the proc kcore, which gives you the kernel core file. Well, if you're in a state where your kernel is hung, it's pretty bad. Generally, those watch dogs effectively in the kernel that tasks that get scheduled periodically and will print out back traces in your VARLOG messages or VARLOG current or Dmessage. So all of those are kernel logging locations. And you can try to analyze what happened for bugs in drivers and so forth. So I think we're coming to the end a lot of time of the talk. If there is more questions, I see there's a few more that I wasn't able to get to. You can continue the conversation in Slack. There's a two dash track dash embedded Linux track Slack channel for this track where I'll monitor it for a while. And if anybody has specific questions, they can ask me there. Thank you very much for attending. I hope you learned something during this presentation. And I hope to see everybody in a year from now. And at an actual in-person conference, thank you very much.


The U.S. Agency for International Development utilizes U.S. Air Force airlift capabilities. To transport humanitarian aid throughout the Middle East, December 8, 2023. The U.S. government partners with the United Nations World Food Program. The United Nations Relief and Works Agency for Palestine Refugees. In the Near East, United Nations Relief and Works Agency. And the United Nations International Children's Emergency Fund, will distribute these supplies to meet the need in Gaza.


Let's start the day off today talking about shorthand notation of describing a chemical change or chemical process occurring. So in this case this is the shorthand notation that we as chemists use, we call it a chemical equation. In this case we're showing a reaction as opposed to a physical change, so a chemical change. How do I know that? It's because I'm having one substance, mercury-2 oxide, go to a different substance, make a different substance, that's mercury and oxygen, okay? So there's some essential features to this that we'll get into in a second, but I want to step back and recall that in the various, or in the past couple of weeks that you've been taking this class, you've seen chemical processes happen in the lab, you've talked about them, things like that, but we've never actually written them in shorthand how we do in chemistry class, okay? So this is the shorthand version of it, and we're going to have these essential features that you have to understand about this shorthand way of writing it. Well anyways, the first essential feature that you need to know is this thing in the middle is called the reaction arrow. The reaction arrow just tells us that some change has happened, okay? Chemical or physical change. In this case, it's a chemical change. To the left of the reaction arrow, you've got one set of molecules or compounds, atoms, compounds, particles, okay? To the left of it, we've got what we call the reactants. These are what was present in the in the mixture before anything happened, present in your reaction vessel before anything happened. On the right of the reaction arrow, we've got the product. So in this case, we've got process. So if I put a bunch of, if I put mercury to oxide under a bunch of heat, right, it's going to decompose into mercury and oxide. That's what this is telling me. Also some other essential features, particles in the reaction equation. At least the couple of them. There are these numbers, and in fact, there's nothing in front of there. Numbers are called co- if it's one, you don't put anything in there. Mercury to oxide decompose to form, and one molecule of oxygen. Crypto-oxygen just means that there's two oxygen atoms bonded together, and you guys know that. I haven't gone over. Are these things, okay? So after each particle, parentheses with a letter inside of it, okay? And those letters are either going to be S, Sands for solid. When it decomposes, it forms mercury metal, which is a liquid that isn't shown in this equation. It's called aqueous. This is when you form a solution where water is the solvent. We'll talk more about that later. Okay, so these are the essential features of a chemical equation.


10 tips for successfully working from home with your spouse. Now, none of this applies to me because my wife's in the next room. However, working with somebody you love can quickly turn into a nightmare if you don't approach it right. In this video, I'm gonna show you the best ways to work with your spouse or significant other and not have issues and be really protective. And we're gonna start that right now. Hey, I'm JR Fisher. Welcome to my channel. Take a look around. There's a lot of space here. Got a lot of videos, stuff for you to look at. Don't forget to subscribe. There's a big subscribe button down there. Don't forget, there's also a bell notification right next to it. Ring the bell. You gotta ring the bell because if you do and you turn on all notifications, I can let you know when I do a new video and you won't miss out on a single one. You won't have to go hunting for it. You'll be part of the VIP group, the Fisher family. So cool. So let's talk about working from home. Now, I don't normally work from home during the pandemic. I did, I had to. Something I did, I'm doing it right now. I do record all my videos pretty much from home, sometimes someplace else. But I normally go to a place like Starbucks and work. I also have an office down the street. I could go to that and work. But I don't really like working at home. But when you're forced to do so, there's certain things you should do in certain steps you should take to make it more enjoyable for both of you. Number one, separate business and pleasure. Now, this is kind of hard to do, but you've got to both be on the same wavelength there. Pleasure is when you're together as a couple and when you're working together. You know, one person may be a boss. The other one may be a boss. You may be equals, but try to keep it separate. One thing I do see when people have a business together and they work at home, they tend to use their personal accounts for business expenditures. Not a good idea. You want to keep that separate because you're going to end up arguing, well, who's going to buy the kids lunches this week or why don't we go on that trip and that vacation? And those are two separate things. So the first thing I'll tell you is you've got to separate business and pleasure. Those are two different things and you've got to separate the time that you spend on business and pleasure. And those are two different times too. Number two, practice smart role play. And what I mean by that is, especially if you're working on the same business together, if you're working on different businesses, it doesn't matter as much. But if you're working within the same business, you're going to find that one of you is better at one thing than the other. That's just the way it is. And you should lean on those talents and if you're the type of personnel that feels like you need to control everything, sometimes you need to step back a little bit. My wife is really, really good at doing accounting and watching money and knowing if the money's there and numbers and all that stuff. And she's really good at it. So I know that she's better at that than me. So that is her area. That's what she does. My thing is more marketing. It's more coming up with ideas and products and that type of thing. And she allows me to do that. Also understand that if one of you works for the other one, that does not extend to your personal life. Okay, so if you're the boss, if the wife's the boss and the husband has to work for the wife and she tells him what to do all day long, when they get off work, she should stop doing that because that's not the time to do it. Private life and business life are separate. And you both should really have this conversation about that and say, hey, this is a business thing. I don't want to offend you, but this is what we need to do and handle it in that manner. And then when you get off work, you're not in that position anymore, okay? You don't boss the other person. You don't tell the other person what to do. That could cause problems. Number three, get a room, okay? If you can get a room, that's great. If you can have a separate room to work, that's great.  This is my home office right here that you see right now. And when I'm in here, I'm certainly not watching TV shows. I don't come in here to watch TV.  I come in here to work. So I know when I'm in this room, there's nothing else to do but work. That's how it's set up. My wife also has an office in the house. She doesn't go in there to watch her TV shows. She goes to the living room to watch TV shows or in the bedroom, but she uses her office for an office and she knows when she's at that desk, she's there working. So make sure you set those boundaries up front, especially if you have to work together side by side, which can be even more difficult. I've done it, we've done it, but it is more difficult because you don't have your own space. So if you can get your own space, do it. I understand that all of you can do that, but if you can, hey, it's a real help. Number four, use safe words. Now we don't really have to do this, but I've seen couples do this. I know it's difficult. Maybe one person is, I don't know, whistling all day and the other person can't handle the person whistling all day. You need to have a safe word. It can be something along the lines of, okay, thank you. And that lets the other person know that they need to stop whatever they're doing, that they're irritating them. And set this up in advance, say it's not offensive, I don't wanna offend you, I don't wanna make you feel bad or anything, but some things do bother me. And if you have a safe word like that, the other person will know and then go, okay, I'll stop doing that, I get it, that bothers you. Number five, engage in conversation. Conversation is so very important that you communicate what each one of you is feeling. And you kinda need to take breaks away from work too, especially if you're working all day long together. I know during the pandemic when we were quarantined, and you know, I am right now actually, but you may be watching this at a later date, we always took time out for lunch, rarely would I eat at my desk. In fact, I never did, I think she did a few times, but we would sit down with a TV tray and turn on the TV and watch a show and relax and work was over with for that moment. Or we would go out to lunch when we could during the pandemic, couldn't do it, but we would go out to lunch, have a nice lunch somewhere and just, you know, get some away time from our desk and our work. It's good to take a break like that during the day. So have that conversation, engaging conversation and understand that different people work better in different environments. So try to pick that environment that you wanna work in and make sure you communicate that. Number six kinda flies in the face of the last thing I just said because number six is giving each other downtime. I know you wanna communicate and maybe you wanna talk to your spouse all day long, but maybe they don't want that, maybe that interrupts their work. So give each other some downtime where you leave the other person alone, let them get their stuff done if you can. If you're talking about work, that makes sense, but if you're interrupting constantly with videos or things you wanna show or Facebook or something like that, that can become annoying. So make sure you don't do that. I've been guilty of that a little bit as to where I wanna show or something or say, hey, look at this video or whatever. I try to do something like that now I wanna break instead of interrupting anything during the day. I'm probably more guilty of that than she is. She stays with her work a little bit more than I do. Number seven, stay on top of each other. Make sure you're actually working. It is very easy to be working at home and to not work at home. In other words, watch that show or start texting or surfing Facebook and you gotta keep each other in check. That's not the time for that.  So make sure that you stay on top of each other, you remind each other that this is work time and this is what we're gonna do now. Number eight, enjoy the closeness. You know, a lot of couples who don't work together may both be going off to jobs all day long and it may be stressful and hectic and all that and they don't even have the opportunity of spending the time with the person. They get very little time with them and they get home, maybe they have to take care of the kids at laundry and dinner, they don't even get a chance to speak to them. So sometimes it's nice just to sit back and realize that you have this opportunity to spend time with somebody that you care about so you should enjoy it and appreciate having that person around. Number nine, take a couple's retreat. What do I mean by that?  If you just work and then you go into the living room and sit down and watch TV all night, you're really not getting any enjoyment. So what you have to do is plan a trip somewhere, maybe a weekend getaway. Maybe like I said, it's a concert, maybe it's a play, but plan some time away where it's not just work and then the living room and then work and then the living room. Plan some time away. Number 10, give compliments. Compliments matter, okay? So don't just work at home all the time, especially if you're working in the same business and one of you never compliments the other person, that person's gonna feel like they're not needed, they're gonna feel inferior or something along those lines. I have a pretty good self-confidence so I don't need it as much, but some people really do need that. And if you give that person compliment when they've done something good, it's gonna make them feel good, it's gonna make them wanna do more for you, it's gonna make them wanna do more in the business if they see that you're appreciating what they're doing. So those 10 things should really help you out. If you're working at home with a significant other, it's really important to go through this list, make sure you're checking them off. They're all super, super important to being productive and to getting along at home. It really is. But if you're able to do this, you really need to stop and realize how lucky you are that you can do this because so many couples will spend 30, 40, 50 years working outside the home and then when they do spend time together, they don't know how to act together, they don't know what to do because they've been gone all this time and they don't even really know the other person. So if you have the ability to do this, it's really cool, just make sure you set it up right, compliment each other, have a good time, enjoy it and work hard. If you've enjoyed this video, give me a thumbs up, make sure you subscribe below, hit that subscribe button down there and the bell, ring the bell. Turn on all bell notifications and what that means is once I do one of these videos, I can actually notify you, you'll get a notification that I've done another one. You don't have to search for it and you don't miss out. You're in the front of the line, you're in the Fisher Family VIP Group that knows about this before anybody else. I also have a $97 course, it's an e-commerce course, you can get it absolutely free. All you have to do is click the link in the description there, click and learn, no credit card required, simple and easy. Thank you so much for watching this video. If you've got comments, put them below. If you've got any questions about this, put them below and I'll be happy to answer them for you and I'll see you in the next video. Hey, thanks for watching my video. Don't forget to subscribe to my channel and click that little bell right here so you can be notified every time I do a new video. Also, click on one of those videos there, keep watching on my channel.


Booker T was doing commentary and walked out as well. Later that week on SmackDown, Michael Cole called Booker out on it and Booker T's response was interesting. Booker, you said that this whole walkout was orchestrated by this small group of people.  Let me tell you, I had nothing to do with it. I'm totally neutral. I had to go to the bathroom. I thought the show was over with. I remember watching this, Michael Cole pressed him and his sticker Booker T said, I had nothing to do with that. I had to go to the restroom. I thought the show was. What's good, George? Bull Ross back again with another video. So we're going to check out 12 WWE Inside Jokes Explained. This should be a very interesting one by tap out corner. We're going to get right into this video. We're not going to waste any time. Definitely want to see what some of these inside jokes were. And if I even knew what they were in this particular video, appreciate all of the support. Let's get right into this one. WWE fans will understand all these jokes, especially the ones at the very end. But don't worry, I'll explain them. This first one, though, is hilarious if you understand it. Before the 2017 Battleground pay-per-view, the New Day were in the ring with their opponents, the Usos. Biggie wanted to let Jimmy and Jane know what they were coming for and said this. We want that gold, sucker. And that battleground, we're coming for you. Usos, Usos. What was Biggie really going to say? Oh, yeah, we are classic, classic, classic, classic promo from Booker T. Oh, we coming for you. Well, watch this clip. We take what we want. And after we take Lex Luga and the giant, we want the gold, sucker. Hulk Hogan, we coming for you. While recovering from an injury, CM Punk is having to take a commentary. If you knew anything about the best in the world, then you knew he was going to make some inside jokes while commenting. He was great on commentary. And this one. You know, I was the king of the ring, too, guys. I don't have you on my list of past winners. Yeah, yeah, I won the tournament we had in Rio de Janeiro. Oh, really? Yes. Of course, CM Punk never won King of the Ring. The joke Punk was making has to do with the origins of the Intercontinental Championship. The story that WWE told was that Pat Patterson became the first Intercontinental Champion in 1979 after we had a tournament in Rio de Janeiro. In reality, there was no tournament, and the whole thing was made up. Just like CM Punk's saying, he won the King of the Ring. Eight days before they fought at Crown Jewel, Ellie Knight and Roman Reigns met in the rain for a contract signing. During the signing, Ellie Knight said, Because while you failed over and over again, while you were busy doing suffering suckatash. Ellie Knight was referring to one of the cringiest things Roman Reigns ever had to do. Back in January 2015, Roman Reigns confronted Seth Rollins on SmackDown and dropped this infamous line.  When the Usos and the New Day faced off in a wrap- That was awful. That was one of the worst promos I have ever, unfortunately, had to witness. That was awful. That was literally one of the worst promos of all time. It was just bad. Whoever wrote that for Roman Reigns, you couldn't have had his best interest at heart. There's no way. But the fact that that was approved, ah. This is a young Roman, so he really had no other choice. He had to kind of play ball.  Bob, you knew things were going to get heated. However, I don't think anyone expected this. This was a classic segment. Big E, let's just keep it PG. You know what's good. Just don't get all rated R like your boy Xavier Woods. That was a good one. Not still one of the best segments they had going against each other, man. Here's where things get spicy. In January 2017, private photos of Page leaked online. Some of them showed her and Xavier Woods doing some, well, R rated stuff. On the topic of rap battles, listen to this shot John Cena took at the rock and see if you understand the reference. No, John Cena was not making that up. The rock did have a boob job, sort of. In the late 90s, the rock underwent gynecomastia surgery. It's a procedure for men that corrects overdeveloped or enlarged breasts. The People's Champ had an excess amount of tissue on his chest, so in 1908, he went under the knife. In fact, rumor has it that the reason the rock briefly wrestled with a shirt on was because his body was healing from the surgery. This next one has been a running joke for over 20 years. At the Beware of Dog pay-per-view in 1906, WWE was promoting their online chat by showing Shawn Michaels using the service. What made it hilarious was it looked like HPK was using a computer for the first time. This became a running joke. Anytime a computer was put in front of Shawn Michaels, you always had trouble or had no idea how it worked. Finally, until the 20th at NXT over in your house, it came full circle, with Michaels, Triple H, and Road Dog all having technical difficulties. Who says WWE doesn't do long-term storytelling? This next inside joke shows just how much of a team player Randy Orton is. In 2016, Edge and Christian got their own show on the WWE Network. It consisted of a bunch of skits and one episode featured Randy Orton. In the skit, Edge played a doctor helping Randy work through his issues. This was a spoof on Randy Orton taking anger management classes in real life back in 2006 due to him having trouble controlling his temper. However, see if you get this joke. I hate to tell you this, but Randy Orton actually did this in real life. Kind of. In 2003, a woman named Michelle Lohan joined WWE and worked as a valet. She wasn't familiar with wrestling and didn't know everyone's name, which led to a problem when Lohan bumped into Randy Orton backstage. Orton was upset that she didn't know who he was, so he stuck into the woman's locker room and put lotion and baby oil into one of Michelle Lohan's bags. Probably because it resembled it, the story became that Randy Orton crapped inside Lohan's bag. Now you see why Randy Orton needed to go to anger management class. So because she didn't know who Randy Orton was, this thing had decided to go into her personal belongings. Yeah, Randy was a menace on and off camera. I've never heard that story before. That's why I don't know who you are.  That's wild. Randy was wild. Take a listen to this next clip and see if you know who John Cena is referring to. Hey John. Hey listen up Braden Walker. Braden Maxx. Don't remember Braden Walker? I can't blame you. In 2008, a wrestler debuted on the ECW brand named Braden Walker. He had previously wrestled for WWE's rival, TNA, under the name Chris Harris. Walker debuted on July 8th, wrestled two matches, and was released on August 7th, making him one of the most obscure WWE wrestlers of all time. In 2011, as part of the storyline, Vince McMahon was relieved of his duties and Triple H took control of WWE. However, things turned to chaos under the game's leadership. This led to a vote of no confidence by the majority of wrestlers and staff and they all walked out. Booker T was doing commentary and walked out as well. Later that week on SmackDown, Michael Cole called Booker out on it and Booker T's response was interesting. Booker, you said that this whole walkout was orchestrated by this small group of people.  Let me tell you, I had nothing to do with that. I'm totally neutral. I had to go to the bathroom. I thought the show was over with. I remember watching this, Michael Cole pressed him and his sticker Booker T said, I had nothing to do with that. I had to go to the restroom. I thought the show was, I remember that because I'm like, wait a minute, why is Booker T here? I thought you walked out. No, I had to go to the restroom. I don't know what you're talking about, Michael.  I love Booker T. I had nothing to do with that. Okay, leave me out of it. Booker's response is based on something that happened in real life. The 2011 Illumination Chamber was the first time Booker T did commentary at a pay-per-view. The five-time WCW champion didn't prepare himself for the nearly three-hour event and had to get up in the middle of a match and go to the bathroom. At least Booker learned his lesson and waited until half the show the next time he had to go. John Cena is just full of obscure references. Towards the end of 2011, Kane returned to WWE and attacked Cena. The next week, John addressed it and said this. What I don't understand is the fact that there is a monster on the loose. A monster who wears a red mask, covers himself in fire and gets real and easy around the holidays or trips to the dentist. There are actually two jokes hidden in that line. The first about Kane getting uneasy during trips to the dentist is a reference to Kane's original WWE character. In 1995, Kendi and Hugh designs a gankum, DDS. The characters Jerry Lawler's private dentist and his appearance and entrance music were all based on that gimmick. Now, Cena's second line about Kane getting uneasy around the holidays is even more obscure. Very early in Kane's career, before he was signed by WWE, the bigger machine wrestled for a small company called USWA. While there, Kane briefly played a character called the Christmas creature. The persona was short-lived, making John Cena's reference even more obscure. On Raw in 2014, the creature was wrestling a match against Jack Swagger. During that, JBL made this comment about Captain Charisma. Okay, get ready to hear this joke explained. So, during a plane ride, a WWE writer was talking to Vince McMahon. McMahon shared with the writer they didn't let Christian's face and called it ratty. Vince then suggested they should do the Kennedy gimmick. No, not Mr. Kennedy. The Kennedy McMahon was referring to was William Kennedy Smith, who was charged in 1901 with raping a woman. During the trial, the woman who accused Kennedy had her face hidden by a blue dot on TV. When Vince McMahon said they should do the Kennedy gimmick, he meant playing a blue dot over Christian's face. Of course, this is a joke by Vince McMahon, but nevertheless, it carried on for years and years. Now, Christian and blue dots are inseparable. Speak! You know, Vince McMahon, I wonder if you're regretted making this joke. In the 80s, WWE introduced a wrestler named Virgil, who was the bodyguard slash manservant of the million-dollar man Ted DiBiase. Now, the name Virgil was very intentional. Dusty Rhodes was the booker for one of WWE's competitors, Jim Crockett Promotions, which eventually became WCW. Dusty Rhodes' real name was Virgil, so naming a wrestler after him was a joke. Ironically, Virgil the wrestler eventually started working for WCW, where the joke was kept alive. However, instead of being called Virgil, he was called Vincent, which was of course a jab at Vince McMahon. WCW eventually went even more direct with that and simply referred to Vincent as Vince. Then, McMahon's son got in on the joke too, when Vince was renamed to Shane. I guess the real Vince of Shane at the last laugh though, when they bought WCW. Funny enough, this sort of thing would happen again years later. In 2004, Simon Dean made his WWE debut. Well, his whole, we're not whole, but major part of his wrestling career, his character, his name was a rib at somebody else. Goes to another company, a rib of somebody else. Gets bought by that other company, a rib of somebody else. That's funny, bro. View. The name Simon Dean was a joke made at the expense of a former WWE wrestler. Dean Malenko was a retired wrestler and works as an agent at the time. Malenko's real name is Dean Simon. So, as a joke, his name was flipped around and given to Simon Dean. Now, to learn the secret of being a fan of fire, watch this video.  What is up with them giving each other wrestler's names or as jokes? That's wild, but it's the wrestling business. People make jokes out of certain things and you may not even realize it on screen, you know, unless someone tells you about it. But comment down below. Let me know some other WWE inside jokes that weren't listed on this video that you have fond memories of. That Booker T won. I didn't know that was an actual thing that happened to him, but it was still hilarious because he said, yo, I didn't know what was going on. I'm neutral. I thought the show was in there, so I had to go to the restroom. That was great. This is why Booker T is great, but I appreciate all the love and support you guys showing on channel Road to 150K, and I'm still here in speedy YouTube wrestling champion of the world. I appreciate y'all kicking the memes here next week, please.


Hey everyone, this is Dan with another episode of my crash protection videos. In the last few days, a lot of things happened related to Ukraine. The president of Russia, Vladimir Putin, announced on February 22 that Russia would recognize the two breakaway regions in Ukraine as independent states. And then the following day, he authorized military operations in eastern Ukraine. In response to that, the US, Germany and UK have imposed economic sanctions against Russia. More sanctions will likely be added. S&P already dropped 12% from its January peak. On top of all that, the US Federal Reserve Banks have been tapering QE. How much will the market drop in the next few weeks? And what should we do to protect our stock investments? I have lived through the 2000.com crash and every single market crash after that. And I posted a few YouTube videos in the last few months about crash protection. This is definitely the right time for me to talk about crash protection again, especially as related to the latest development in Ukraine and with regard to QE tapering. Let's get into the details. First, let's look at how the broad market has moved in the last couple of months. This is the chart showing SPY, which is the candlestick chart, and QQQ, which is the blue line. As you can see, SPY has dropped from its peak reach on January 4th. It has dropped almost 12%. Whereas QQQ has dropped more than 18%. And if you look at the latest trends of the two lines, they are definitely very bearish. The question is, how far will it drop before they recover? If you see the latest price as of February 23rd, the closing price for SPY is right at the previous low on January 24th. And actually, as of this morning, the market opened even lower. And therefore, QQQ has already dropped below the previous low that was reached around January 27th, 28th. Definitely a very bearish situation here. The situation in Ukraine started brewing in January with Russia deploying troops surrounding the eastern part of Ukraine. And then as of February 21st, Putin announced that Russia would recognize the two breakaway regions as independent states. As of yesterday, February 23rd, Putin announced special military operations in Ukraine. Definitely, the tension has been building and building. As of today, fighting already started. In response to all these, there have been sanctions imposed against Russia. Actually, even before the last few days, there had already been sanctions against Russia because of Russia's annexation of Crimea back in 2014. Although the sanctions up until a few days ago were limited to the Crimea region. As of a few days ago, the US issued sanctions on two major Russian banks and on the country's sovereign debt as well as several oligarchs close to Putin. Then Germany halted the Nord Stream 2 pipeline, which brings natural gas from Russia to Germany. In the meanwhile, the UK has announced the sanction of five Russian banks and three wealthy individuals linked to Moscow. I'm pretty sure there will be more sanctions announced in the next few days, which will definitely affect the stock market. That's why we need to look into protecting our stock investments. In the meanwhile, the Federal Reserve Banks, after their January 26 FOMC meeting, said that because of high inflation and low unemployment, now they want to reduce quantitative easing and bring the monthly increase of Fed assets eventually to zero in the next few months. Thus for February, they aim to increase the holdings of Treasury securities by $20 billion and the mortgage back securities by $10 billion a month. So the total is $30 billion of increase per month and that's still a big number, but compared to the number in less than six months ago, which was $120 billion a month, that is definitely a very significant reduction. And I mentioned more details in the video that I published on January 30. You can look it up in my YouTube channel. Eventually, the asset increase will be reduced to zero, which will be probably around April, May timeframe. And after that, the Fed might even start reducing the Fed assets. That means the Fed will be using liquidity in the marketplace, which will most likely bring down the stock market. In the market crash video I posted on January 30th, I did a deep dive analysis of what transpired between 1978 and 1984 when the Fed was aggressively trying to control inflation. Back then, the Fed allowed S&P to drop as much as 21% before they were able to bring the inflation rate from 5% to about 2.5%. In the meanwhile, the inflation rate did rise from 5% to 15% and the unemployment rate went from 5% up to 11%. Basically, there was a full-on recession. The Fed started to lower the interest rate after they achieved the inflation rate target and the economy covered. And the unemployment rate dropped down to about 7% as of 1984. The same thing could happen now, especially if the situation in Ukraine doesn't get any better. Then we have a combination of two major factors affecting the market. Number one, QE tapering and tightening of liquidity in the market. And number two, the buildup of tension in Ukraine and probably in the neighboring regions. We've got to get ready for that. I've been using Google Trends as a leading indicator to spot any imminent market corrections or market crashes. As of today around noon time, you can see the fear indicator, the red line, which is the Google search phrase stock market crash that really spiked up last night. And then it finally came down a little bit today. The blue line here is a bullish indicator, which is represented by the search phrase best stocks to buy. It's most likely that when additional sanctions are imposed against Russia, that the fear indicator, the red line will spike up again. I'll be monitoring Google Trends very diligently in the next few days. What are my strategies in light of what's been happening? First of all, I already saw some shares to lock in profits or to cut losses. I saw BioNTech shares, Moderna shares, ASML, AMD and Media, Royal Caribbean and TQQ. And I've been trading SQQQ to hedge against downturns. I will be monitoring SPY in terms of the daily 150 and 200 period exponential moving averages and the weekly 50 period exponential moving average and other key support resistance levels. I'll be doing the same as far as monitoring QQQ. I will definitely continue to monitor the situation in Ukraine. I will sell more long positions if critical support levels are breached. I will continue to monitor the Fed actions, Google Trends and the technical indicators, RSI, DMI and MACD. I will keep shares of stocks with good fundamentals such as Google, ASML, AMD and I've been trading TMV as well, which is the ETF index to the yield of the 20-year U.S. Treasury bill. And I've been buying and holding shares of energy stocks and recovery stocks such as ConocoPhillips, UCO, the oil ETF, UNG, the natural gas ETF, RCL, Royal Caribbean and LUV Southwest Airlines. I will continue to update my Twitter subscribers almost on a daily basis with regard to the latest news development as well as some of my trades. In the last few days, my portfolio looks like this. About 50 to 60 percent of my liquid assets, which include stocks, ETFs and cash, are actually in cash. I'm holding a lot of cash at this point. And up to 20 percent of my portfolio is in stocks with strong fundamentals. And up to 15 percent of my portfolio is in UNG, UCO, ConocoPhillips, XLE. These are energy-related stocks or ETFs. And also I hold GLD to go ETF and TMV, the interest rate-related ETF. Up to 15 percent of my portfolio will be reserved for swing trading with SQQ or TQQ because the market has been dropping the last few days of being trading mostly with SQQ. When would the market recover? First of all, if the Ukraine situation is resolved and when Russia withdraws its troops, maybe because Ukraine might recognize the independence of the two breakaway regions. Or if the tension in Ukraine continues to build up, but if the conflicts are just limited to a small region, then the market will generally recover within two to three months based on historical experiences. But this time around, the market slump might last longer because the sanctions imposed on Russia will also hurt the economy of the European Union, which will drag down the market in Europe. And it will have the ripple effect to the US stock market as well. Or when S&P has dropped close to 20 percent from its all-time high, then the Federal Reserve Banks might get worried about a potential major market crash and recession. Then the Federal Reserve Banks might inject liquidity into the market, create rebound artificially. How will we know that the market is starting to rebound? For that, I will be monitoring two technical indicators, DMI and MACD. So when the daily DMI and MACD indicators start turning bullish, it will probably then be a sign of a rebound. And of course, when the daily SPY and QQQ start making higher highs and higher lows, that will be also a very strong indication that the market is starting to recover. At this point, I'd like to suggest that you also subscribe to my Twitter account, which is DanMarketL. For example, on February 11th, I tweeted that I bought SQQQ again because I've been trading that. And then on February 14th, I said I sold my SQQQ at average 1.5 percent gain because at that point, the market was going up a little bit. And in meanwhile, I bought TMV shares. And then four days later, I bought more SQQQ because the market started to drop again. And then on February 22nd, I tweeted that I bought UNG, the natural gas ETF because of the tightening of tension in Ukraine. Again, I'd like to encourage you to click the like, subscribe and notification button if you like what you've seen so far. As usual, I very much welcome your comments, questions, and suggestions. I'd like to remind you that I'm not a financial advisor. I share my analyses and stock trading strategies for educational purposes only. If you want to buy or sell stocks, you should make your own decisions and you should definitely consult with your financial advisors before you do so. This wraps up my video for now. I will chat with you again in the next few days. In the meanwhile, I'd like to wish you the very best of luck with your financial investments.