Hi everybody, my name is Thierry Boutel, I'm a software architect working in a French company named IoT.BZDH. IoT stands for Internet of Things, obviously, and BZDH stands for Brittany, which is the place where we live. The subject of this presentation is cross debugging on Linux. In some past years, I was working for a wind river company, and we were developing development tools, and especially I was working on the Linux user mod debugger. We will see what is available today in terms of cross debugging and also what can be expected from modern tools today. The issues addressed in this talk are obviously the remote debugging, but also we will have a deeper look on the debugger entrances. For the ones who have ever wanted to know how a debugger works, it will be probably of interest. At the end of the talk, I also did a little bit about the ancient essential tools for the developers, like LTNG and Valgrind, and also we will see what and how ZEFIR OS works and what kind of debug it offers. IoT.BZDH is currently developing and recently released the official version of WetBask. WetBask is both a factory which helps developers and integrators to make products easier, and also WetBask OS, which is a new Linux distribution based on CentOS and that has long-term support. First, I would like to be a little more specific about what remote debugging means. Remote debugging means when you debug an application which is on a remote embedded target. Usually, this is a target which is of a different CPU architecture than your OS development machine, and as a rule, considering that the target architecture is different as your OS will prevent you from bad surprises. What it is not is just connecting SSH for instance on a target that has an opposite SSH server and to launch the debugger on it that doesn't suit with small embedded targets and that also won't work if you want to do system-mode debugging. In the past 15 years, I have often seen people that weren't using remote debugging. There were several reasons for that. Some people were just working on code that was not architecture-dependent or that was using board devices. In that case, they were able to reproduce their bug on their OS. Some others said that the debugger was badly integrated in popular IDs like this code and they were white. Also, GDB and GDB server have suffered from instabilities for years and were leading from crashes or bad performances. Some other people were very special ones because they weren't using the debugger at all simply because they didn't trust them or they could cop their bugs without using the debugger at all. There are two kinds of remote debugging, system-mode and user-mode. System-mode basically means kernel debugging when you are on the Linux. It can be performed either with hardware assistance, with a JTAG pub, through software, things like KGDB or software emulation for instance when you run KVM with a special debug port that emulates a big GDB server. One thing to keep in mind is when you perform a stop that suspends the whole system that means that you lose communication. In case of you are using software like KGDB, software debugger that means that you will have to keep your communication alive with AIRQ disabled. For doing that, the debugger implements a special communication channel based on pulling on the communication interface, like serial or Ethernet. It's interesting to notice that some high-level debugger may have what is called OS awareness. That means that they know that the underlying debug system is Linux and that they will be able to display some FWAD information which is the backtraces of FWAD local storage. User mode debugging means application debugging. Debugger relies on the P2S API that is built in the kernel. It's interesting to notice that since last year there have been several significant improvements in the P2S API. The signal handling is far better since Linux 3.4 and security has been improved because second comp is under Linux 4.8. For using P2S API with GDB, you have to get appropriate security credentials. If you have ever wanted what are the mechanics when you insert a breakpoint in your favorite debugger GUI the things behind are that the debugger inserts a special instruction into the process memory. That instruction is completely architecture-dependent and its role is to generate a software interrupt when the program counter comes on it. The debugger will catch that software interrupt. As it maintains a list of breakpoints and the original instructions that were at their places the debugger will be able to step over the breakpoint. Maybe the most important thing to remember here is that the process memory is modified and that can have significant side effects that I explained a little bit later. Hardware breakpoints use hardware support. The amount of available hardware breakpoints and how to use them depends on the CPU. There are two types of instruction breakpoints that just like software breakpoints are it when the program counter which is location and hardware breakpoints which are very useful to detect or to debug crashes due to altered memory. It is worth to notice that usually it's not a special address but rather a range of addresses. The main advantage of hardware breakpoints is that trace memory is not modified at all. Another important feature of a debugger backend is a single step. The single step consists in executing a single machine extension and stopping immediately after. That feature can be supported by the hardware or not. Usually the Linux kernel publishes that feature through the P2 single step call that can rely on hardware support or being emulated by the kernel itself. When there are no support at all it was the case on older kernel on the MIPS architecture. This is a trickiest issue because you have to figure out what will be the next program, location depending on the current instruction. You can imagine that jumps are the trickiest thing to handle. When you know the instruction where you will go after you have to put a temporary breakpoint on that location to resume your process to eat that location and to remove the breakpoint. In other words, your trace memory is modified twice and that can lead to annoying side effects. One of the most complicated problems that debugger has to deal with is multi-thread debugging, especially when there are a lot of shared code and when you want to use thread-specific breakpoints. In both cases, you are in the same address space. When you put a breakpoint, it will be eventually eat by a thread that was not aimed to. When this happens, the victim must be resumed. But when you want to do that, before modifying the process memory, you have to stop all the other ones. You have to suspend them, else you will eventually lose the breakpoint in the other threads. As you can guess, things can get even worse when you don't have hardware support for a single step, but that you emulate it with temporary breakpoints. The reason why I have been working on the Linux debugger is that when we were wanted a debugger with an ID, whose name was tornado, that has the same features as VXworks. So everything began in 2005. The first proof of concept of a unified software debugger for Linux that was dual, it could perform system and user mode. The first proof of concept was performed a couple of months after the beginning of 2005. It consisted of set of ugly patches on top of 2.6.x kernel. It was using VXworks WDB protocol and it was using serial and net polling routines because there was no netpoll API yet. And net everything think it was able to spawn processes from the kernel context, which is unusual. Today, nowadays, when the kernel wants for some reason to spawn a user process, it usually needs help from the user-land. After some months, the project was finally abandoned and a split was made between system mode and user mode. KGB was chosen for system mode. By the way, I would be interested to get some testimonials from the listeners if they know of themselves still use KGB today, because I don't know a lot of people that do so. For user mode, the P2A's API was chosen and we started to develop an application called the user-mode-agent that was leveraging, specifying and implementing the WDB protocol. It was using UDP and was supporting also the TPC protocol and also serial backend. All those architectures were supported, X86, 64, ARM32, PPC32, NM32 and MIX64. It was possible to get the Tracy IO-initiable console through the virtual console. There were support for hardware buy points, for ARM sub-mode and interworking one, and both flavors of threads that were available at that date, which were NPTEL and NNIC thread. And the key feature to my mind, it was the possibility to have five going debugging. I mean, stop, stepping, continuing on single thread without having to stop the whole process. At the end of the project, we have finally achieved the following result. Our debugger was able to debug up to 100 threads with lots of shared code, and each thread could be debugged individually or not. A set of more than 2,000 unit or threads could be one from the host debugger. Facing the complexity of the single step emulation problem, I decided to base my design on a finite state machine. That finite state machine was holding more than 80 states, and it was possible to design it with a GUI. The debugger was and has continued to be internally used by Win River developers to develop the JTAG Probs firmware of the company. What you can see here is the finite state machine displaying its GUI and its complexity. That tool is quite deprecated. It's no longer maintained and was originally written by a French engineering school. I recently published the code of that application on GitHub just for it's not being lost, but it's no longer maintained at all. The inconvenience of this design is that it doesn't suit for system mode debugging. Moreover, the problem of stepping over ghost boy points, ghost boy points are the ones that are not intended to be hit because specific to another thread, led to performances issues with SMP. So, the tweak used by the other project was to step over a break point of performance ghost type emulation by putting or executing the execution somewhere else in another advice space that was not shared with other threads. Let's go back to the present, if you don't mind. I will show some remote debug sessions example. In the first setup, the communication will be done by a GDB running in the WebPESC local builder. The local builder is a light container that has all the developing tools, including cross compiler when it's a GDB and so on, and what you need to package your application into the WebPESC factory. And it will connect to an instance of WebPESC OS running in a KVM. So what you can see in the running video is that the KVM instance is booted. Meanwhile, the developer compiles a very simple application. And as you can notice in the make file, there is a separation between the application program and the debug information. Once the target is booted, the GDB server instance is launched with the dash-dash-multi, which means that we don't want to close the connection when the GDB session exists, and we specify a special port. On the host we launch a GDB, we attach to the target with the special target extended remote and the given port, that has been mapped to my host because it is running on a KVM. And then we put a breakpoint on main and launch the program. We can see that the breakpoint on main is correctly hit. The following example matches the reality better. In this case, we still have GDB running on the host machine, and this time it will connect to an instance of GDB server running on an actual board. In this case, this is a Raspberry Pi 4. In this current video, I have booted the Raspberry Pi 4 and connected for SSH on it. First of all, we compile the simple application by having modified the... make file a little bit in order to invoke the cross-compiler. Thanks to the cross-compile suffix. We then upload the compile binary to the target without pushing the debug information. The debug information file will be kept on the host side. In the SSH session, we launch the GDB server just like we did in the previous example. We connect on it. Again, we will put a breakpoint on the main... on the three-point and execute the program. There is still the upload of the unknown target libraries from GDB server to GDB because it needs them to debug. We can see that the breakpoint is hit at the end, that we can continue hit to have the L1 output and that's it. As you can see, there is no significant loss of performances or efficiency despite of the fact that the target is less powerful than the host. In the next example, we will be debugging a much bigger process, still on the Raspberry Pi 4. This time, the process is the OpenCPN application. This is an application for displaying charts in navigation. It uses X11 and OpenGL and a lot of chart libraries. Up to 180 and it is written in C++. This time, since this is a graphical application, I will do the SSH port forwarding for the X11 port. What you can see also is that there is a completion for the C++ symbol. I am putting a breakpoint in chart canvas, column, column, chart canvas. This is the constructor of that class. Then, when the program is started, the thing which is happening here is that there is a synchronization with the missing library. It takes a while because there is a bunch of them, but this is an amount of time which is acceptable because it only happens once. After a while, the breakpoint we aim is finally hit. And we already see the first graphical windows that come. I am then stepping and exiting the current function and at the end we have the application which is launched displaying the map. By the way, this is the place where we live close to the Isle of Groix. I have wanted to make a kind of evaluation of alternatives to the GDB and GDB server couple. I have identified one interesting which is the LDDB from the Clang environment. LDDB can work with remote connection as well. It has the LDDB server. To my mind, the main advantage of this debugger is that it can perform an automatic upload of the debug binaries. The synchronization to the target is done when the binary is recompiled, when it changes on the host. However, I got a very bad experience from the Raspberry Pi. It was possible to debug my simple sample but it has crashed when I have wanted to debug a bigger application. When that happened, the loss connection was not detected. Also, the documentation went on how to list and delete pending or actual breakpoints didn't work. The documentation seems also outdated regarding the process launch command. As a conclusion, there is also a completion for the lookup of symbols including the C++ symbols but it is not user-friendly. Instead of suggesting the symbols on a single line, it presents them on a whole page in a pipe-more-way. So it is to my mind less practical that GDB does. Another alternative to GDB and GDB server that I had identified is TCF which means the target communication framework. The TCF project was initially launched at WinWeaver as a replacer of the user mode agent. That project was launched due to Felix Burton and is currently maintained by Eugene Tasav. This is a project which is hosted and integrated in Eclipse IDE. Unfortunately, I didn't have to perform a lot of tests with TCF as I won't have time to post that video but I just took some look at the code. It seems still maintained as I said. It supports a lot of backends including other backends than the ones for the Linux OS and also it has a very cool feature which is the auto-discovery via via UDP broadcast. TCF architecture is cool but it's still unclear to me why they removed all the multi-sweat logics that I had implemented in the user mode agent. I have finished to tell what I had to say about Linux debuggers. I would like to introduce something new for the ones that don't know which is the Zephyr OS and to make a small demo on how the debugging works on it. In short words, what is Zephyr OS? Zephyr OS is a real-time OS aimed for microcontrollers. It has a very nice documentation. Most API are very similar to the ones of Linux kernel. It also supports and has device trees but the device trees are statically compiled. Since we are on microcontrollers usually most of the time we don't have ethernet and there is no implemented software debug agent. That means that for debugging you need a GTAG. The setup of the following demo is as such. The target will be a WNSS H3ULCB board. That board has several CPUs on it. 4 Cortex-A 57 4 A53 and a Cortex-R7 on which we will run the Zephyr OS. We plug and only makes JTAG debug problem on the board and on the host we run OpenOCD that makes an abstraction of the JTAG communication and provides a GDB server on which we can plug we can connect a GDB debugger. OpenOCD offers one TCP port per debug call on the target. It also has one TCP port from internal monitoring and management and the demo in the coming demo we will see how Zephyr tools make it even easier with the WestDebug command. On the left terminal shows the project directory where the command is just WasteBuild to generate the firmware to inject on the target. On the right it's just a picocom running on the serial port. When you type WestDebug it performs several things. It uploads the firmware to the board and directly connects GDB to the OpenOCD where we put a waypoint on main, we perform a continue we have seen that the has started and after stepping over the point K you can see that the hello world is deployed so it's as simple as debugging Linux process. Before concluding my presentation about cause debuggers, I'd like to tell you a little bit about two tools that I find very cool and that have saved my life a number of times. Those tools are Valgrin and ATTNG. I think that most people that already have used Valgrin know it because of his memcheck tool which is a really useful tool when you want to track memory links for instance but it also has profiling tools that are maybe less popular but that can help you to track where bottlenecks in your code when you are facing performance issues. It is worth to notice that it works well but that kind of instrumentation has a big impact on performances. So don't expect your monitor code to be as fast as the production one. Using Valgrin is really simple. You just have to type the valgrin command the dash dash tool with the tool that you want to use. In this example it is callgrin you eventually had other options than your program name and your program options. When your program ends the callgrin.hout.pid file is generated that can be processed with the kcashgrin tool. Kcashgrin's graphical tool will show up the callgrin of your functions with the amount of time spent in each function. When should you use valgrin I would say all the time and that would must be in your CI. A special attention must be paid for the memcheck exclusion rules. For more information go to glue and see the valgrin documentation. Another tool which is really cool is LTTNG I really like their logo and find their mall very cute. The slogan of LTTNG can be instrument twice and investigate. It can perform either kernel or user space instrumentation. In my case it has saved my life in the past when I was chasing way nasty while scheduling issues. A typical example that LTTNG can solve is that when you are getting scheduling or priority inversion issues or when you want to chase higher queue storms. There are other known alternatives to LTTNG just like F2S, S2S and Perth but they are not as cool I mean they are not as easy to use and maybe the strength of LTTNG can combine both userland and kernel twice and it is especially useful when you have an application that relies on a dedicated kernel. LTTNG is composed of three things LTTNG tools, LTTNG modules which are kernel modules and LTTNG UST It was a real because it implements high performance buffers based on the RCU technology. Starting with LTTNG is quite easy. The documentation is very well written and you don't have to follow the mall. You need to add some specific kernel options, config modules of course but config IOS timers and config twice points of course. By the way, config twice points is not visible in kconfig but it is activated implicitly by config F2S. Config F2S just adds 5 by NOP's operation at each function tree. In fact, in another challenge on the other side, I show how with 4 commands you can get your first LTTNG twice. Got one file of twice by your cpu core. For using that twice you can either use bebel twice too or the best graphical application with this twice compass was formerly an eclipse plugin and now it is still at eclipse.org but it is a standalone application. As you can see the application shows which thread or which cpu is active at one time or when it is preempted by an IRCOO with some statistics. We have finally reached the end of the presentation. I hope it was interesting. I would like to thank everybody for their attention. We will now start the question and answer session. Feel free to ask anything you have got in mind and I will try to answer. If you come to France please visit us on the next picture to show again the place where we live. We will be happy to show you our beautiful country.


Hello everyone, I'm staunch doomsday prepper Dick Simmons. And I'm Dexter Griff. And the end is coming! We're all doomed. DOOOOOO! We have a few ways for you to prepare for several life-ending scenarios, besides our dead fast zombie plans of course. Sadly, not all problems can be solved with a dead Griff. To start, I've asked Lopez to join us as we discuss the impending AI takeover. Hola. As technology progresses and computers become more lifelike, it becomes more likely we'll see them gain sentience. And take over the world enslaving all of humanity! Dexter and me, least of the terriers. Before that happens, move to a small town with terrible internet access. The technology is bound to be decades behind, so use it as a weapon. Stand back! I've got dial-up internet, and I'm not afraid to use it. Are you sure you don't need that? I'll see how it goes. And if that doesn't work, I can always take my chances that the robots will accept my cyborg half and take me in as one of their own. No, es probable. For the lazier preppers, ride it out at the water park. Cool off with the pools and slides, while all the robots rust around you. Ha! Go ahead and try to break my chlorine shield. I dare ya. The way to stop robots from purging us is to purge them first. Ha! By launching imps high into the atmosphere, we can knock them all out at once. Yes, robots, come! Firing nuclear payload. Ha! Ha! See that, men? The nuclear option is always the best option. We'll be shocked back to the Stone Age, but it'll be well worth it. Foreign retaliatory strikes detected. Uh-oh! Unfortunately, this will probably bring radioactive armor getting upon the planet. Toxic wastelands, destroyed cities, searching for supplies will all become the norm. To bear these harsh conditions, take the leftover robot parts and radioactive materials and construct new robots for scavenging and protection. Ha! Ha! And there we go! Systemo de nuve in linea. Donde estoy? Donde esta el ejército mecanico superior? Glad to see you back in the land of the living, soldier. Now hop on! We've got fallout to avoid, terminators to salvage, and plenty of angry maxes out there that are bound to be pretty mad. Tu eras el demonio rojo y estas es mi inferno. That's the spirit! Once the bombs stop falling, I'm just going to take the lazy method again and do nothing. Yeah, saw that coming. Nothing but soak up as much radiation as possible. That way I could turn into the incredible griff, or super griff, or even... Here lies the incredibly super dead griff who died even more hideously mutated than he already was. We're not digging a hole on him. No, not a chance. Speaking of, though lethal to humans, a worldwide surplus of radiation will inevitably lead to the rise of gigantic mutant monsters. These towering beasts tend to terrorize the remains of society for reasons not fully understood. However, not all kaiju are bad-natured, some even having hearts of gold. The only way to know what you're dealing with is to directly communicate with them. Hey, up there! Do you ever wonder why you're here? I'm Simmons! Once diplomacy inevitably fails, here's the rest of your scavenged radioactive parts to construct giant telepathic-linked robots to fight against the beast's mano-a-mano. Let me present. Sergeant Locke! Oh, yeah! Back from the dead! Ah, sa-guarado, e-cargado. Oh, my God! I'm answered! Catastrophes tend to breed other world-ending catastrophes. Nerd monsters capable of wiping out cities! Congress! Doesn't really matter in the end. There's not much you can do to stop these doomsday scenarios once they're set into motion. Our advice for once the end does inevitably come? Just sit back, grab some popcorn, and watch it all unfold. That's right! And, if you do manage to make it through the end of the world, take comfort knowing that red versus blue will still be here right alongside you. Forever. For all of time. Wait, we're gonna be here forever? Oh, God, I- Thanks for checking out that episode of RVB. You can find the playlist to watch more below, or you can go to roostatif.com to check out the latest RT shows. Also, don't forget to like, subscribe, and hit that bell, or leave a comment.


But that said, John chapter 15. I'll begin reading at verse nine. I'll read to verse 11. And we'll get into our study. John chapter 15, beginning at verse nine. Jesus said, as a father loved me, I also have loved you, abide in my love. If you keep my commandments, you will abide in my love just as I have kept my father's commandments and abide in his love. These things I've spoken to you that my joy may remain in you and that your joy may be full. And so as we open up our study right now, remember that Jesus has been encouraging his disciples to remain faithful to him. He has just encouraged them to abide in him. And when he speaks concerning abiding, and that's what he had been doing in chapter 15, verses one through eight, when he was speaking concerning abiding, notice verse seven, how he said, if you abide in me, and my words abiding you, as he's been speaking of abiding, the word abide means to continue, to remain or to stay around. Obviously, Jesus is saying, continue with me, abide in me, remain in me. Abiding in Jesus' evidence that salvation has already taken place. When he speaks concerning abiding in him, there's an assumption there, an inference if you will, that salvation has taken place and therefore you abide in him, you remain in him, you continue in him. And this fruit of abiding in Christ, the fruit of being in Christ is gonna be demonstrated by continuing with him as well as an obedience to his word. You see, one of the ways we demonstrate that we're abiding is we have a heart of obedience. And that's why in chapter 14, verse 21, that's why he had said, he who has my commandments and keeps them, it is he who loves me, and he who loves me will be loved by my father and I will love him and manifest myself to him. In verse 23, he had said, if anyone loves me, he will keep my word and my father will love him and we will come to him and make our home with him. And so abiding in Christ is demonstrated by a love for him and his word. To abide in Christ is to live out the salvation that took place when you committed your heart to Christ and it's demonstrated again by continuing with him in obedience to him by his word. Remember in chapter eight, verse 31 and 32, how Jesus said to those Jews who believed him, if you abide in my word, you are my disciples indeed, you shall know the truth and the truth shall make you free. So he's speaking to those who claim to know him, he said, you need to continue with me, you need to do so because that demonstrates you're kind of abiding me and thus through your obedience, you're revealing that you have faith in me. And so when Jesus is speaking, concerning abiding in him and all of that and speaking of his words again in verse seven of chapter 15 where he says, my words abiding you, when he's speaking concerning that, he's speaking concerning the fact that we have heard him and we are obeying him. You see hearing his word and obeying his word is really one of the evidences that we're a believer. It's not just the hearing because people will come to Bible studies, perhaps you may be right now watching this Bible study because somebody's forcing you to or you're interested in what we have to say, preferably you're interested and you hear but just because a person hears doesn't mean that they adhere. You hear the word of God, but we're called to adhere to the word of God. We are called not to listen alone but to act on those things that we're hearing. That's the mark of a believer. There's a writer that has influenced me as a devotional writer, a man that many people are familiar with. His name is A.W. Tozer. And A.W. Tozer said this. He said, when the Holy Spirit is in full control of our lives, he will expect our obedience to the written word of God, but it is part of our human problem that we would like to be full of the spirit and yet go on and do as we please. And that's true.  A person who's really walking with the Lord is a person who hears what he has to say and obeys. Somebody who's abiding in Christ is somebody who's connected vitally to the true vine. And so we receive our nutrients, our life from the vine. And he is saying that if we actually are part of that vine, then one of the ways we demonstrate that we are in him is we obey his word, we live for him, we produce fruit. So obedience from a heart of faith is evidence that we are followers of Jesus and to obey his word is actually gonna reveal that we're saved when we obey from the heart in faith. Now to hear speaks of listening to something, to understand something, to give heed to something. So hearing God's word and obeying demonstrates a devotion, a devotion to his will. And that produces a fruitful prayer life because the Christian seeks to please God. We're not seeking to please ourselves. We desire to be in the center of his will as we pray. And that's why again in verse seven, he said, if you abide in me, my words abiding you, you will ask what you desire and it shall be done for you. And so to abiding his word, abiding in him, producing fruit in Christ is demonstrated in various ways, including the fact that our prayer life is fruitful because the Lord is listening to us as we pray according to his will. You see when we pray, we're not seeking to please ourselves. When we pray, we pray that we might be in the center of his will. In James in chapter four, verse three, in the New Testament, James writes, when you ask, you do not receive because you ask with wrong motives that you may spend what you get on your pleasure. And so our prayer life is fruitful when we're abiding in Christ and abiding in his word. And when we're abiding in his word, we pray according to his will. And as we pray according to his will, we receive that which we have asked of him because we're asking according to his will, receiving those things that he desires for us. And so we were looking at that together in the first eight verses of chapter 15. And now we move on into verse nine where Jesus continues and Jesus says this. He says, as a father loved me, I also have loved you. He goes on to say, abide, abide in my love. Now Jesus has already spoken of his father's love for him. In John 3.35, he had said, the father loves the son and has given all things into his hand. It is said there, the father loves the son and has given all things into his hand. In John 5.20, it reads, the father loves the son and shows him all things that he himself does. And so Jesus Christ speaks concerned in the fact that the father loves him. But notice in verse nine, he says, the father loved me, I also have loved you, abide in my love. So we're gonna be looking at a moment together. We're gonna be looking at how he gives his disciples insight about God's love. The first thing I wanna point out about the love of God is love originates with him. Love originates with God and love is intended to be shared with other people. The father loves him and he loves them. They are to abide in his love and they are to love one another. In 1 John 4.7 and 8, beloved, let us love one another for love is of God. And everyone who loves is born of God and knows God. And he who does not love does not know God. For God is love. If there's anything, guys, if there's anything that is contrary to true Christianity, anything that is contrary to true Christianity, it is when somebody claims to love God and doesn't love other people. It is so contradictory. How can I say I love God whom I have not seen and yet I hate my brother whom I do see? How can I claim to have the love of Christ in me when I don't love those whom he loves? And there are quite a number of people who have this attitude. They don't love other people. And it's sad, but it's true. And you see it all the time, I do. I've seen it all, I see it quite often. You see, love originates in God. God's love is demonstrated in Jesus Christ. Jesus loved his men, but it doesn't stop there. They're to love one another and they're to love other people. That's the mark of a true Christian. The fact that they have a discerning kind of love, a love that is a welcoming love. And if there's anything that the church needs to be reminded of, I would say it's the love one another. It's the love one another. I think there's an absence of love in the body of Christ today and it concerns me greatly to see that. You see, love for Jesus is gonna be demonstrated by keeping his commands. And obedience results in fellowship with God. And obedience results in resting in his love. In verse 10, he said, if you keep my commandments, you will abide in my love just as I have kept my father's commandments and abide in his love. So obedience, as mentioned a moment ago, obedience to him results in fellowship with God and abiding or resting, resting in his love. In verse 11, these things I've spoken to you that my joy may remain in you and that your joy may be full. These things I've spoken to you that your joy may be full. Remember in verse nine, as he said, the father loved me, I also have loved you, abiding my love, abiding in the love of God. When you're abiding in his love, you're abiding in various things guys.  It gives you a peace. It gives you a sense of God's presence in a way that is extremely powerful and that is the fount of joy. That's where you receive joy. It's knowing that you're in Christ and the love that God has for Jesus is the same kind of love that you can have with him. He loved Jesus, Jesus loves you and you love him in return and as a result, you're able to love other people. And when you have this settled kind of sense that God is for you and not against you, when you have this settled sense that God's presence is there and his power is there, then you have a peace that passes all understanding. If there's a time that we need to practice that again, it's right now, don't you think? Isn't it time for the church or isn't it past the time for the church to be able to rest in the joy of the Lord because the joy of the Lord is our strength and Jesus wants his joy to remain in us and he wants his joy in us to be full. And so as you abide in the love of God and as you're obeying him, well, your joy comes from God and you can rejoice for a variety of things, including the fact that Jesus is going to the Father and as he goes to the Father, even so one day, you will too. There's this thing that I think the church has forgotten and I'm speaking to myself as I speak to everybody else. This world is in my home, I'm just passing through. I'm a stranger and I'm a pilgrim, I'm a sojourner. Why am I looking to the world to give me peace? Why am I looking for somebody to pass some kind of law or eat it that's going to give me peace? Why am I doing that? Why do I think that peace is going to come from any other source other than God himself? Why do I think that? I want to have peace, how about you? Do you want joy? I do too. Where does peace come from? Where does joy come from? It doesn't come from our circumstances or surroundings, does it? It comes from a deeper source. It comes from your relationship with God through Jesus Christ. He gives you that peace that Paul would say passes all human understanding. It goes beyond any human comprehension and the ability on my part to really think through and to come to really understand. You know, there was a well-known atheist who was a father of existentialism who is reported, said, I don't want a peace that passes understanding. I want a peace that I can understand. Well, if you can understand the peace, maybe it's not real. The thing that passes understanding comes through Christ where you should be, according to the world standards, experiencing certain tensions and anxieties. And yet because you've learned in faith to cast your cares on the Lord, you've learned that he cares for you. You know, as I'm sharing, this isn't part of my notes and I'll say this briefly, but I have to tell you, you know, pastoring this church as long as I have, 38, 39 years, real soon, I can tell you that the Lord has shown me in so many different ways by his love and his provision that he raised up our fellowship with a reason. And that reason has yet to be fulfilled. In other words, continuing to work on the things that he intends for us to do. He wants us to work and continue to serve him so that the things he designed for this church may take place. And right now, you know, it's difficult for me. It's difficult, as I was mentioning a little bit earlier, it's difficult to walk into a sanctuary this large and to know that there was, we were seeing the church and it was healthy and the sanctuary had many people in it and there's so much going on that has been so good and fruitful and then suddenly to have an empty sanctuary and to speak to empty chairs more than usual, it's difficult, but you know what? I come in here like perhaps some of you do and I will come in, I'll stand behind this pulpit and none of you would know that more likely unless somehow you might have seen and I'll stand behind this pulpit and I'll look at an empty sanctuary and I pray over the seats and I say, God, in Jesus' name, I pray that you will fill those seats once again, that you'll bring people here that we can minister to and we can edify, we can equip and then they can go out and they can do the work of ministry and reach people for Jesus' sake, Lord. So please, Lord, in Jesus' name, may we once again be able to meet, may we once again be able to worship together and fellowship and hug each other and do the work of ministry and I've learned and I'm learning, have you? I'm learning to cast my cares on him. Why? Because he cares for me. Because he cares for us. You need to think of that and realize that God loves you and the joy that Jesus has, he said it's my joy and that joy originates in his relationship with his father. It's a joy that he has because he's united with his own father. It's a joy you can have. My joy can be your joy because you too are united with the father and God's plans for you are good. Even though we go through the valley, even though we go through difficulties, it all turns out for the good. If we understand that we can have peace as well as his joy. So he says, my word needs to abide in you. In other words, I need to by faith receive what he has to say and act on it to obey and joy will be the fruit that is produced. In Hebrews 12, verses one and two, the writer said, therefore, we also, since we are surrounded by so great a cloud of witnesses, let us lay aside every weight and the sin which so easily ensnares us and let us run with endurance the race that is set before us, looking unto Jesus, the author and finisher of our faith who for the joy that was set before him endured the cross despising the shame and has sat down at the right hand of the throne of God, the joy that was set before him and Jesus Christ is encouraging us with the same joy. In verse 12 and 13, this he said, it's my commandment that you'll love one another as I have loved you. Greater love has no one than this then to lay down one's life for his friends. Greater love has no man than this and to lay down one's life for his friends. The word friends there is a Greek word, it's philos, it speaks of those who are beloved, they are the loved ones, to lay down one's life for his loved ones, for his friends. You see, when we speak of the love of God and abiding in Christ in obedience, we need to remember that the essence of God's love is sacrificial. Laying down one's life for friends is the supreme test and expression of the love of God. It's the kind of love that Jesus revealed to them, it's the kind of love that he reveals to us. And it's the kind of love that believers exhibit to other people. In Galatians one verse four, Paul said, Jesus gave himself for our sins to rescue us from this present evil age. He gave himself. It's a sacrificial love. It's kind of love that we're to have for one another and Paul even refers to this kind of love when he's writing to husbands in Ephesians 525 when he says, husbands love your wives. Just as Christ loved the church and gave himself up for her. Love in its purest form, in its agape form is demonstrated by sacrifice. And so he's saying love one another, be sacrificial towards one another. And remember that greater love has no one than this than to lay down one's life for his friends. And this is what Jesus does. He's laying down his life for his friends. But he also speaks about what his friends do for him because in verse 14, he says, you are my friends if you do whatever I command you. So in verse 13, he spoke of those whom he loves and what he would do for them. But in verse 14, he speaks of how they respond to his love and they do that by obeying him. Their love, their friendship is revealed by obedience. In Luke 11, 27 and 28, it says, as Jesus was saying these things, a woman in the crowd called out, blessed is the mother who gave you birth and nursed you. And he replied, blessed rather are those who hear the word of God and obey it. In Matthew 12, 47 through 50, then one said to him, look, your mother and your brothers are standing outside seeking to speak with you. But he answered and said to the one who told him, who is my mother and who are my brothers?  Now notice for whoever does the will of my father in heaven is my brother and sister and mother. You are my friends if you do what I command you. I have laid down my life for you, demonstrating my love for you. You demonstrate your love for me by obeying, by doing what I have commanded you. You see, obedience is the evidence of friendship with Jesus Christ, because friendship is normally based on like-mindedness or common aims. The person that you call friends is, your friend is usually the person who agrees with you the most. It's a person that you get along with the best. And why do you get along with them the best? Because they usually agree with you and you usually agree with them. That's what friendship is. It's based on common likes, common philosophy, common direction. That's what friendship is. And so when we're a friend to Christ, that simply means we have things in common with him. The things he wants to do are the things we want to do. What he wants us to do is what we want to do. Why? Because we have common interests and common desires. We have a like-mindedness. We have common aims. It's like James 2.23, where it says the scripture was fulfilled, which says Abraham believed God. It was accounted to him for righteousness. He was called a friend of God. Abraham believed God, but God said to him, he accepted, desired, and acted upon. He had a common aim. He's the only person in the Old Testament that's referred to as a friend of God, Abraham. Because he did those things that God commanded him to do. James tells us that friendship with the world is enmity with God in James chapter four, verse four. So when I make friends with the world, I'm automatically in a hostile opposition to God. But when I want to do the things that are pleasing to God, I want to obey his commands. Then I'm demonstrating that he's my friend and I am his. Notice verse 15, where he says, no longer do I call you servants. For a servant doesn't know what his master's doing. But I've called your friends for all things that I've heard from my father I've made known to you. So Jesus is making clear that though they are servants, they are deeply loved by him. He's not saying that they don't serve him, but he is saying they're true brethren to him. It's interesting how he says a servant doesn't know what his master's doing. When he says a servant doesn't know what his master's doing, he doesn't know what he's doing or about to do. How do I illustrate that? Because he's saying that a servant is not informed of why something needs to be done. He just does what he's told. That's what servants do, right? What you're told. That's what the essence of a job is by the way, right? We're all, I'm seeking to employees, you know. What do you do on the job? What you want or what you're told? Well, you do what you're told, right? That's a job. And that's where I think sometimes confusion comes in because we start thinking that we can do what we want when in fact, no, we do what we're told. That's what a job is. And sometimes sometimes people haven't had real jobs. We call it real jobs. Meaning jobs outside of the church. And so what happens is we start thinking that we've got some kind of voting thing where, well, if I don't want to do that, or I think this is better, that's what we do. That's not how it works in the real world, right? I learned that very early and I think we all have to some degree, you know, in a church it can be a little bit awkward and sometimes even confusing because you're putting on, I put on the pastor's hat and then I have to put on a boss's hat. And it's very difficult to wear both. That's why it's, that's why I have administrators. That's why I have other people who help me with that to be honest with you. Because I don't want to be the boss. I want to be the pastor. I want to think of the spiritual things and lead in a spiritual way. But on the other hand, we all know this, we basically get hired to do whatever it is that we do. We get a wage for doing those things and that's what a job is, right? That's how it works. And so when Jesus is speaking about this he's saying that a servant in his day isn't necessarily told why things are being done. And again, I can apply that here to the church. I could say that if I walk in as the overseer of the boss or whatever you want to call me, you know, king, whatever makes you feel comfortable.  That's not a wise thing because I'll scratch my head and I'll wonder, did I ask you permission to make this decision? Did I put you on my board or are you? I'm open to hearing other sides because we ought to be, right? But do I tell you do this so that this can be done as a result? No, I shouldn't have to, right? When I was in the military I had to learn that the hard way. I learned that the hard way because when a sergeant told you to do something it wasn't a debate or a discussion. It was an order and you follow the order. If you don't follow the order you find yourself after five because we'd get off work at five doing some overtime, something you didn't want to do cleaning an office, picking up trash. And I've done that when I was in the military. They didn't like, I didn't like the order was given. I said, I'm not gonna do that. So I ended up doing something else. They're gonna trash because that's what they would do. You would work overtime. That's how it works. So I learned very early that it wasn't my pay grade to make the decisions that a sergeant was making or the captain was making or the general was making. It was my responsibility to simply follow an order. And that's basically what Jesus is referring to. Something we all understand. We all understand that a servant doesn't have to know the reasons why the boss said clean that room up. Servant doesn't need to know that. The servant doesn't need to be told because I'm gonna have some gas and the yes gonna need this and then they're gonna say this long and this is what we're gonna feed it. They don't need to know all of those things, right? But the servant needs to know is the room needs to be cleaned. And that's what Jesus is talking about here. Very basic thing, very true. He's not informed about everything. He's not informed about why something needs to be done. That's what Jesus says. He's simply saying do this. Look, if I hire an outside company to work on the campus, some outside company and I need something done, they don't come, I don't walk out there and say the reason I'm having you trim these trees or clean this is because they don't need to know that, right? They just get paid to do the job. And that's basically what Jesus is speaking about. They just pay, they get paid to do the job. They do the work. They're not told what we want to accomplish. They're not told why it's being done. And frankly, they don't really care. They just need to do it. In a servant-boss relationship, reasons Jesus is saying are not necessary to be revealed. But in a friendship relationship, reasons are important to reveal because those doing the work will actually take personal ownership of that work because they know that they're part of the overall plan and they will be blessed when the fruit is produced. And that's why with many of you, and many of you have been with me a long time, you'll know that sometimes I'll open my heart and tell you this is why we're doing this. I'll tell you that. The reason I want to do this is because of that. I will do that with my staff. And I will say, watch what happens. We're gonna see this. And that gets my staff excited to say, really? And then when it happens, then we all rejoice together. So if you look at it like that, if I bring an outsider in to do something, he doesn't care. So he's a servant. I don't owe them an explanation. But I treat people differently and I'll say, you know why we're doing that? And it may seem like it's not reasonable, but look at what's gonna happen. Then we all rejoice when the fruit is produced. And that's what Jesus is saying. It's not that they're not his servants. I mean, that's what Paul and the other apostles would refer to themselves as. The most many times when Paul was writing his letters, he'd say, Paul is servant of the Lord Jesus Christ. No, he knew what he was. He was a servant. But Jesus also made him a friend. And so the servant did what he was commanded, but he was also given insight into why this is taking place. Remember in Matthew, remember in the book of Matthew, how Jesus gave a series of parables. And as he was giving the series, as it's found in Matthew 13, when he was giving the series of parables, the people were there just listening to these mysterious teachings that they weren't understanding or grasping. The average people were not understanding. And in Matthew 13, 10 and 11, the disciples came to him and said, why do you speak to them in parables? And he answered and said to them, because it has been given to you to know the mysteries of the kingdom of heaven, but to them it has not been given. And so Jesus revealed to his men the things that he didn't reveal to others. This reminds me of something that the psalmist wrote in Psalm 103, verse seven, where it says he made known his ways to Moses, his acts to the children of Israel. That's one of my favorite Psalms, by the way, one of my favorite scriptures, because I understand it in a different way. The children of Israel knew what was being done by the Lord, and they could explain it. They could say this is what God has done, but Moses knew why God did that. He not only knew what was done, but he knew why it was done. His ways were made known to Moses. You reveal your ways, or you can reveal your acts. If you wanna know how I as a pastor feel, ask my kids.  My kids have a very clear view of why daddy does what he does. They know what I do, but I guarantee you, if you ask them why, 99.9% of the time, they can tell you why I do what I do. Many people in our church know that I do certain things, but my kids can tell you why. My ways have been made known to my children, but my acts have been made to the church, before the church. So Jesus is speaking to his men. He said, I'm not calling you servants, not that they're not, because they are, but I'm calling you friends, because a servant doesn't need an explanation to do the job, but I want you guys to know why I'm doing what I do. And I've taken this, by the way, and I'll make it a little personal for another moment here, and I will let you know that that's why I feel a need to always tell you guys what I do. I'll tell you why I'm doing what I do. Not everybody in this room is a staff member, and you may not know, but I will tell you in devotions. I'll say, this is why we're doing this, because I look at you as more than just servants in jobs, but I see you as brothers and sisters in a family, and that's how it works. And that's why Jesus said, I'm not gonna call you only a servant, I'm calling you my friend, because a servant doesn't need to know why the master does what he does, but a friend does. And that's what Jesus is disclosing to them. He goes on and he says, and notice at this time, they're really not able to understand exactly what he's saying. He says, in verse 15, all things that I heard from my father I have made known to you. Well, many of the things that he had made known to them, they were yet to truly understand, but later on it would be disclosed to them and they'd have a deeper understanding. They were not yet prepared to receive all that he had to give, but they were moving in that direction. And then he says in verse 16, you didn't choose me. I chose you and appointed you that you should go and bear fruit and that your fruit should remain that whatever you ask the father in my name, he may give you. And so Jesus reminds them that they were lost before he found them. They weren't seeking him, he actually came seeking them. They were the lost sheep. They were like that lost coin that Jesus speaks about, that lost son. And he's the one who sought them out. He chose them, he ordained them and he gave them a commission to go and notice and to bear fruit. They were to go and do the work of ministry but he would work with them. And Jeremiah, one of my favorite portions of the book of Jeremiah is found in chapter one verses five through seven, where God says, before I formed you in the womb I knew you, before you were born I sanctified you. I ordained you a prophet to the nations. And I said, oh Lord God, behold, I cannot speak for I'm a youth.  You didn't choose me, I chose you.  Your fruit will remain whatever you ask the father and my name he may give you. I'm the one who called you to do these things. You will go out and you will seek and you will find even as I sought you and found you and you will bring them to faith in Jesus Christ. You didn't choose me, I chose you. And I like that phrase where he says and I ordained you in the King James, I ordained you.  These things verse 17, I command you. Again, that you love one another. What is gonna make their ministry fruitful guys? What is gonna make the world see that God is truly amongst them? He says you're gonna love one another because love is what the world will see and love is what the world reacts to. When they see the church loving one another it demonstrates that there's a God who loves.  Part of the fruit will be antagonism towards you and your message. There's gonna be an angry rejection that leads to overt persecution fueled by hate. This world system constantly opposes God and the establishment of his kingdom on the earth. It's a system that is fueled by the rejection of God and all that he stands for. It especially provokes the enemy when God's children love one another. It's the love of God and one another that actually stimulates angry persecution.   And because this is true there's more reason to love one another and be united with one another in heart.  If there's anything different about us is that we love one another. God help us to do that. God help us to love each other because that's a mark of a Christian. That's what demonstrates that we really know the Lord.  He says in verse 19, if you were of the world the world would love its own because you are not of the world but I chose you out of the world therefore the world hates you. So John makes it clear that the world loves its own by repeating the word world five times. Very often that's true. The world loves its own.  You can have a public forum and a public forum will attract strangers who unite in opposition to the gospel. All you need to do is get a group of people and begin to discuss abortion or homosexuality. Talk about living together or pornography and see what happens. The world unites against the church. You see righteousness by contrast is always confrontational and those who follow Christ will live such lives that evil is exposed in contrast and the world's response is not always positive. In second Timothy 312 it simply says all who will live godly in Christ Jesus shall suffer persecution. And when you live for Christ those who don't will notice that you do and your life provides a contrast for their own lifestyles and often they respond with anger. The world doesn't have much of a problem with compromising Christians. As a matter of fact they like them because they can point them out as hypocrites but when you're sincere in your faith for Christ that's when you have some problems.  And so he reminds them of what he had been saying earlier he has been persecuted and they will have persecution also and the reason for rejection notice they do not know me. Persecution arises because they don't know God even if they're religious. Sometimes the strongest persecution has actually come from religious people. In verse 22 if I had not come and spoken to them they would have no sin but now they have no excuse for their sin. He's not saying they'd be sinless because all sin and fall short of the glory of God. What he is saying is they would not be guilty of rejecting Messiah. Knowledge of his message has given them accountability to God the more you know the more you oh you hear the message and Jesus is called and now you're responsible to respond to what you've heard perhaps there are some right now listening to this and you haven't committed your heart to Jesus Christ the more you know the more you give an account of and so ultimately you give an account of yourself to God for everything that he has deposited in you all that he has told you because knowledge of his message makes you accountable.  In other words you can't love the father if you hate his son. If I had not done among them the works which no one else did they would have no sin but now they have seen and also hated both me and my father but this happened that the word might be fulfilled which is written in their law. They hated me without a cause. They heard his message, they saw his works they still rejected him. In John 5 Jesus said at verse 36 he said the very works that I do bear witness of me that the father has sent me. In John 12 37 it reads although he had done so many signs before them they didn't believe in him. Remember always the greater the privilege the greater responsibility of response and this is something that is called willful rejection they're rejecting him even though he's done works and given them words that no one had ever done before. They are now held accountable for the things that they know even as those who are right now listening to this study even as you are also held accountable and Jesus Christ has called us to have relationship with him he laid his life down for us. He says if people follow me they're gonna have love for one another. The body of Christ has intended to love one another if you're having problems with a brother or sister in the Lord you need to deal with it you can't hide it you can't pretend it's not there you have to deal with it because if you're not loving your brother it's very difficult to say that you love an invisible God if you can't love the person who you can see how can you love the one you don't see? So God is speaking to our hearts and he's saying to us you gotta let these things go and you've gotta walk with me because there's enough hatred and antagonism in the world today the church shouldn't be a hotbed of that and believers need to learn to pray for one another and have compassion on one another care for one another, pray for one another and to forgive one another so that we can have unity in Christ that the world that is lost might see a living witness of those who've actually been bought by the blood and transformed by Jesus himself and in this we're demonstrating that we're his disciples greater love has no man than this than a man laid down his life for his friend and you're my friends Jesus said if you do what I commanded you. Father I ask that you would work in our hearts even now and I would lift up those who are having struggles in this particular area and I would ask Lord that we would obey you for your word teaches us to and has nothing to do with our feelings has everything to do with your word and obedience so I'm asking that you would work in all of us who are believers that we would learn to just not quench your spirit and allow the fruit of your spirit which is love to blossom in us. Lord again I ask that you would work within us and continue to move through us and as our eyes are closed and our heads about there may be some that need to get right with the Lord as you're viewing this right now the Holy Spirit speaking to you and you need to get right with God I wanna pray with you and if you need to get right with Jesus I wanna pray with you now if you open up your heart and say God be merciful to me I'm a sinner forgive me Lord because I am imperfect because I carry bitterness and anger because I've done other things because I'm tired of this and I need to get right with you and if you have a heart for that right this moment we can pray and you can say to the Lord forgive me a sinner God and if you desire to please I'm gonna share a simple prayer and I'm gonna ask you to pray along with me if you need to get right with God you can say God I'm a sinner Jesus Christ came to save sinners Jesus came to save me forgive me of my sins cleanse me give me a new life I will follow you every day from this day forward in Jesus name, amen. And if you prayed that simple prayer and you're sincerely in faith saying God be merciful to me come into my life all the Bible makes it very clear you became the temple of the spirit of God by faith you received Christ and if you just prayed that please let us know so we can contact you and minister to you further but if you got right with the Lord just now please walk with Christ but let us know and that way we can be of help to you and now we'll close the continuation and closing of this prayer in Jesus I pray that you would work in all of us to your glory and have your way in us and we would pray these things in your name, amen.


It's how we come in here. We're excited to play here. It's a fun place to play and obviously it's important that we play well this weekend and obviously tonight, you know, tonight didn't go our way but, you know, one loss is one loss and we've got more three more games against these guys the next three days so, you know, it's important to get some good rest tonight and come back tomorrow ready to play. You go about shaking this one off and getting back at it tomorrow.  It doesn't matter if it's five to four or two to one or twenty to three.  We don't have to wait a week to redeem ourselves. We get to come out here tomorrow afternoon and then get back after it. So we plan on doing that.  Both. Yesterday was a tough day as well. Did you sense any carryover from the loss of the Orioles tonight? Well, I mean, we just didn't play well. I mean, I don't know that it was any carryover or today had anything to do with yesterday but, you know, we just didn't play well. We came out and got an early lead, a big lead. Obviously couldn't hold on to it and from there they just kept down in the baseball and we kept trying to try to run it down and throw it back in.  They had a really good game offensively tonight and they ran away with them.