Okay. So, we see that towards the end of chapter 3, Paul is giving instructions to different sets of people. He's talking to wives, husband, then he's giving instructions to parents to and particularly the father to, you know, what not to do to their children so that they don't become discouraged and also bond servants and also talks about the employees, sorry. So, that is in chapter 4. So, chapter 4 starts like this. Let's say once. Okay. So, chapter 4 says, Masters, give your bond servants what is just and fair, knowing that you also have a master in heaven. Okay.  So, therefore, be like your master to others to whom you are masters. Right. And so he's saying, Masters, give your bond servants what is just and fair, knowing that you also have a master in heaven. Okay. Now, again, he's our example.  So, give your bond servants what is just and fair. Okay. Verse 2, continue earnestly in prayer, being vigilant in it with thanksgiving. Okay. And meanwhile, praying also for us, that God would open to us a door for the word to speak the mystery of Christ for which I am also in chains, that I may make it manifest as I ought to speak. Walk in wisdom towards those who are outside redeeming the time. Let your speech always be with grace, seasoned with salt, that you may know how you ought to answer each one. Okay. So, so here, all after giving the instruction to how earthly masters should treat their earthly servants or earthly employees, it's talking about prayer, and he's saying continue in prayer. Okay. Continue earnestly in prayer. And the word that he uses there is a Greek word, which means being diligent and continuing on, persevering on, to be steadfast, not to waver, but to be steadfast, and to persevere and not to stop, not to stop before the time. That is what it means. So continue earnestly in prayer. So you have that quality when it comes to praying, have that attribute when it comes to your prayers. Continue earnestly in prayer, being vigilant in it with thanksgiving. So being alert in it with thanksgiving, so alert for what? Alert for maybe the leading of the Lord, alert to do what he would want us to do. And he's saying being alert or vigilant in it with thanksgiving, along with thanksgiving. We read in Philippians also a similar instruction in prayer. A prayer which results, which directly changes our anxiety. Okay. A prayer which changes our anxiety and fear, maybe our worries, really. Philippians 4 says be anxious for nothing, but in Philippians 4 and verse 6, be anxious for nothing, but in everything by prayer and supplication with thanksgiving, let your request be made known to God. And he talks about how the peace of God will be the God or the security to our hearts and minds. It will guard your hearts and minds through Christ Jesus. So here similarly Paul is saying, praying earnestly, being vigilant in it, sorry, continue earnestly in prayer, being vigilant in it with thanksgiving. So along with prayer, even as you're praying, even as you're being alert and alert to the leading of the Holy Spirit about what you should pray for, what you should pray about, be alert in it with thanksgiving. So you're already thanking God in faith for those prayers that have been made in faith that are in line with his heart. And also thanking God for the fact that he is who he said he is, the prayer answering God, the one whose years are always open to our Christ. So continue in prayer, be diligent in it, be alert in it with thanksgiving. Meanwhile, praying also for us. So he's now giving his personal prayer request, saying praying also for us that God would open to us a door for the word to speak the mystery of God, for which I am also in chains that I may make it manifest as I ought to speak. So saying that God would give us an opportunity that he would give us a door. So a door actually opens out to another room. A door talks about accessibility, a closed door talks about a close, an opportunity that is not there. When an open door talks about an opportunity that is there, an opportunity for whatever. So here specifically saying an opportunity to minister, that God would open to us a door for the word to speak the mystery of Christ, the revelation that is hidden in Christ that is discovered by the Holy Spirit. I mean, that we discover by the power of the Holy Spirit, by the leading of the Spirit. So saying that God would open to us a door to speak the mystery of Christ, for which I am also in chains. So he has been speaking the mysteries of Christ and he's been sharing the gospel. And because of which he is in prison, his present condition is that he is in prison. And the reason is for the sake of the gospel. So for this reason, I am also in chains, that I may, sorry, that I may make it manifest that was for, that I may make it manifest as I ought to speak. So saying that this is where I may make it manifest that let it be displayed. The gospel, the mystery of Christ, let it be displayed, let it be put on display, let it be clear. It also means light. That's something that is exposed for that's not hidden. So he's saying, let it be made manifest that I may make it manifest as I ought to speak. So as I ought to communicate, may I do that so that this will be made manifest. Okay. So, yeah. Then he's saying, walk in wisdom towards those who are outside redeeming the time. So again, redeeming the time also is something that he has already addressed, he's already spoken about in an earlier episode that you walk wisely, walk circumspectly redeeming the time, reading from Ephesians 5 and verse 15 and 16. See then that you walk circumspectly, not as fools but as wise, redeeming the time because the days are evil and therefore do not be unwise but understand what the will of the Lord is. So same thing that he's also sharing here, walk in wisdom towards those who are outside. So towards those who are outside meaning towards those who are not walking according to the gospel, those who do not know Christ yet, to those who are unsaved, they are not saved yet. So saying, walk in wisdom towards those who are outside, meaning that when you, the way you live your life, the way you conduct your life, maybe in relation or in fellowship or not fellowship, when you're associating with, when you're meeting with, when you're associating with, when you're conducting your life along with them, walk in wisdom, be wise for various things, the values may not be the same, the common ground may not be the same, walk in wisdom and you do that, redeeming the time. Walk in wisdom, which means the same thing, saying walk circumspectly, redeeming the time, be mindful of the time, of the resource that you have in your hands. So redeeming the time. Verse six, let your speech always be with grace. Your speech, your conversations, your communications, let it be with grace. Okay. Let me just scroll up. Let it be with grace. Let it be seasoned with salt. Okay. So in the sense when you, when you speak gracious words and when we, when our speech or conversation is filled with edifying gracious words, it's, it's like, it's like salt, like, it's like seasoning with salt, so that there is no corruption of it. Right. There is, it is flavor some, you know, it is, so it doesn't, it's not corrupted. It is no decay. Right. That's, which is what salt does. Right. So let it be seasoned with salt. So the end result is this, that you may know how you ought to answer each one. Right. So when we speak with grace, there is this wisdom that we receive in order to answer each one. So you're saying that you may know how you ought to answer each one. And the thing is this, that when you walk in humility, you know, we receive the grace of God. When you walk in submission, we receive the grace of God. Right.  Right. Words and thoughts and our conversations when it's, when it's with the grace of God or seasoned with the, like how salt is seasoned or sprinkled. Right. Then we will also know how we ought to answer each person. Right. How it's necessary, how it fits in, that you will know how to answer each one. So, you know, of course, the obvious thing is that the answer is, is truth. Okay. It's not like, okay. Not being manipulative or not to give evasive answers. No, that you may know that there is a knowing how in truth that you may know how to answer each, each one. Okay. Okay. Then some final greetings that he shares.  So let's go through that. Verse seven, Taikika's beloved brother, faithful minister and fellow servant in the Lord will tell you all the news about me. So I'm sending him to you for this very purpose that he may know your circumstances and comfort your hearts with Onesimus, a faithful and beloved brother who is one of you. They will make known to you all things which are happening here. So he's sending Taikika's and he's sending Onesimus and Onesimus seems to be from Colosseum. So, so he's saying, you know, both these, both these brothers will come there and they are faithful and they will make known to me what is happening there.  He's coming to minister to comfort your hearts. And he's also saying, you know, he's that they will make known to you everything that is happening here. They'll just give a report, share what is happening, share our experiences with you. Okay. And Aristaakas is mentioned in Western Aristaakas. My fellow prisoner greets you with Mark, the cousin of Barnabas, about whom you received instructions. If he comes to you, welcome. So here we read about Mark and Aristaakas. Mark was also called John Mark. We read about in the Paul's first missionary journey and also just about just when they are just about to start the second missionary journey, you know, what happens there, right? So in the first missionary journey, John Mark actually goes halfway and then he stops. He goes back. He does not continue on. So because of which, when they're about to start the second missionary journey, Paul and Barnabas, they have a discussion.  The Bible records for us that they had differences because of John Mark, because John Mark was related to Barnabas. And then he wanted Barnabas wanted John Mark to come. But Paul said, no, he probably felt that he was not yet ready because he abandoned halfway. So, you know, that is the thing. So here Paul is writing about Mark. And then he's saying, you know, if he comes to Colosse, you know, you treat him well, welcome him, treat him well. So John Mark seems to have come around, you know, seems to have reconciled with Paul and also in another place also Paul mentions, you know, bring John Mark for he's useful to be in ministry. So seems to have, you know, reconciled and he seems to have come around, right? Okay, so yeah, so let's let's look at the other. He's also mentioning about someone called Justice. And his actually first name is Jesus.  So Jesus or Jesus was called Justice. And Paul mentions that they are his fellow workers who are, you know, only fellow workers of the kingdom of God, who are of the circumcision, like who are of Jewish origin, right? They have proved to be a comfort to me. Epiphras, I'm sorry, yeah, who are of the circumcision, you know, it could either be Jewish origin or, you know, they have come to, you know, know the Lord. So because, you know, aristocrats and, and especially Onesimus is, is from a Gentile church, right? So it doesn't mean that is from all of them are from a Jewish origin. So, yeah. So he's talking about Epiphras again. So Onesimus and Epiphras are from Colossae itself, right? Epiphras, who is one of you, verse 12, a bond servant of Christ greets you, always laboring fervently for you in prayers, that you may stand perfect and complete in all the will of God. So this was, this is mentioned about Epiphras, which is commendable. So saying Epiphras always, you know, he's, he's, first of all, he's referred to as a bond servant of Christ, which, you know, Paul calls himself in many places. So he's saying Epiphras is a bond servant of Christ. And he also says that Epiphras is someone who always labors fervently in prayer. So he's a man of prayer. He's a man of, a man who intercedes. And he's saying that this is what he prays, that you may stand perfect and complete and mature and stand perfect and complete in all the will of God. So whatever God's will is for you, that you may be perfect, that you may be complete. Right? So this is what you know, Paul's, sorry, Epiphras' prayer is for the church in Colossae. And he says in verse 13, for I bear him witness that he has a great zeal for you and those who are in Laodicea and those in Hierapolis. So he's saying, you know, I can testify, I can bear witness, I see that he has a great zeal for these churches, right? For the church in Colossae, Laodicea, and Hierapolis. So, you know, I bear him witness. And this is what he does. He prays, he intercedes, he labors. Labor is hard work, right? It's a lot of effort. So he's saying he labors fervently in prayer. So we learned something about praying in such, with such intensity, praying fervently and praying where it is, you know, it is hard labor. Okay. So, you know, when sometimes we, there are many kinds of prayers and sometimes, you know, people like look at one side of prayer, which is conversation with God, right?  Why should you, you know, why all this, I don't know, many years back, one person asking about another person, you know, why is this person getting all hot and bothered? And why is this person, you know, getting, why is there with so much intensity? Why? Why can't the person just relax and pray, right? So we see the different kinds of prayers. You see that prayer, you know, is when, especially the ministry of intercession and praying for others, you know, praying that Christ be formed in them, Paul writes it, right? He says, I labor in, I, you know, I labor in pain again, until Christ being formed in you.  So here also, Epaphras, who's laboring fervently that these believers would stand perfect and complete in all the will of God. So we see that it's a prayer with intensity, prayer with, you know, it's intercession, and that's fine. That's absolutely fine. It is warfare. It is laboring. It's with a lot of effort and being fervent, if that is fine. Okay. Then verse 14, Luke, the beloved physician and Demas, read to you, verse 15, greet the brethren who are in Laodicea and Nymphos and the church that is in his house. So this is the church in Laodicea, possibly verse 16 onwards. Now, when this episode is read among you, see that it is read also in the church of the Laodicians and that you likewise read the episode from Laodicea and say to our keepers, take heed to the ministry which you have received in the Lord, that you may fulfill it. This salutation by my own hand, Paul, remember my chains, grace be with you. Amen. So in those last three verses, so he's giving instruction, maybe read here and then the other place also, and he's asking them to also read a letter which he has written to the Laodicians. Now, we don't have a record of that. Just like how when we were studying Corinthians, we see that there was a letter that is already written to the Corinthians, which was actually three letters that he had written, but we have a record of these two which we call as first and second Corinthians. So similarly, here also he seems to have written an epistle to Laodicea and so he's giving them instruction that they read it in their assemblies, but we don't have that. And say to our keepers, he's talking to another person in Colosseum, a believer there, take heed to the ministry which you have received in the Lord. Take heed, be careful. Take heed of the ministry that this is verse 10. Sorry, not verse 10. Verse 17. Verse 17, take heed, be careful, be discerning. It means you look at something.  It could be you compare something, you look at it. So it means you perceive. See it. Be alert. Take heed to the ministry or the call that you have received in the Lord that you may fulfill it. So that's an instruction. So personally, we can apply it as well. Maybe there's a call, there's a ministry that we have received in the Lord and sometimes it's a timely word. Just comes, take heed, be careful. Discern what you're doing with your time, with your effort. Take heed to the ministry that you may fulfill it. If you want to fulfill it, if your objective is to fulfill it, then you need to take heed. You need to be careful. You need to be alert and discerning. And then the final salutation says, I write with my own hand, remember my chains, pray for the fact that I'm imprisoned and that his earlier request also at the beginning is that you pray, pray for that. So in the Philippian episode, we read that. So all requesting them to pray. And in this fourth chapter, we see that he's requesting them to pray for the door of opportunity to minister. He's in prison. He's still not free, but he is asking them to pray that there would be an opportunity. So such is the zeal and the expectation that he would continue to minister. So he's asking them to pray that there would be a door of opportunity so that I can make manifest the mystery of God and clearly communicate the mystery of God. And here also he's saying, pray, remember my chains and grace be with you. So this is how it ends here. Okay. So any questions before we move on to Thessalonians? Any questions that you might have about Colossians or anything that you noticed or anything that you highlighted to you that you learned that you want to talk about? You can do that. What is that one thing that you recall or one thing that was really highlighted to you as we studied Colossians? Okay. Anything? Feel free to share. Okay. So for me, chapter one really stands out. I mean, of course, there are the entire episode does, but particularly chapter one, because he's talking about the deity of Christ. So unmistakably, he's saying, you know, this is it. He is the image of the invisible God. He's talking about the deity of Christ and the first born over all creation, talking about the fact that he died and is buried and he came back to life, the first born over all creation and verse 16, by him all things were created that are in heaven and on earth, visible and invisible. And all things were created through him and for him. Very clearly establishing the fact that Christ is the creator, the living word, the word that John talks about in the beginning was the word. The word was with God and the word was God. And so it's talking about the living word, eternal word, Jesus being the eternal word and Jesus being the one who created everything. So it's talking about the deity that created everything and then says that for him and through him and in him, all things consist a very powerful revelation about Christ. So yeah, for me that stands out. Any thoughts, Dave, anything? Thayesh, Thomas, as we have studied, Kandan, Aaron, Kiran said any particular verse, any particular thing that or it could be an instruction that really struck you, nothing at all? Any question? Yeah, go ahead, Thomas. Yeah. In emotions, when he's talking about the Christ, he disarmed the powers, the rulers and the powers and he made public, the word is really touched me. There is no any power of the demon forces or Satan against us. We have the victory through the cross. Jesus made a victory through the cross. We are more than overcome by the demons. That word really touched me. Right. So yeah, chapter two is talking about the cross and that's a powerful, powerful revelation again. Who Christ is, of course, chapter one and what Christ did, chapter two and how it affects us as believers and especially in the realm of the Spirit, the fact that we have the authority and that's so powerful that we don't have to walk in fear and we have the authority because of what Christ did. So thank you for sharing that. That's so powerful. Yeah. Yeah. Okay. So 13 to 15, same thing, I think, Dave, what she was, yeah. And what we noticed there, yeah, yeah, go ahead. Yeah. Go ahead. Now, what do we see there in 13 to 15 is like, you know, he's talking about the past, right? The cross is referred to the believer as something in the past.   So that is also something that we notice here versus 13 to 15. Yeah. Yeah. Go ahead, Thomas. Trying to say the same words from the same chapter three, verse one, his second chapter about how we saved and not be dead. And we raised the Christ. So chapter three was one says, so you're raised from the dead and now if you want to seek anything, seek and look about like the worst, that's really nice touch to me. Yeah. Yeah. Like the focus now, what is our focus?  But the fact is that we are forever changed because of what happened, you know, on the cross, you know, we are changed. Saying that, you know, you are raised with Christ and seek those things which are above where Christ is sitting at the right hand of God. You know, Ephesians two also talks about the fact that we are raised with him and we are seated with him. Right. So that is our perspective. So as much as this physical life tries to crowd in, you know, with all kinds of maybe sometimes even worries and even sometimes some legitimate things, right?  Yeah. Yeah, Thaisha, you go ahead, please. Hi, Pastor. Thank you. I think when we meditate on these scriptures, we can understand the authority that we have Christ has given us as well. And it makes us trust more, makes us believe more bills or faith, because when though we have troubles, trials and all sorts of things in this world, the reassurance that we have Christ has given us, that's priceless. So we can tap into a supernatural power. And though things seem to go awire at times, we can speak the word and find comfort in the word because the collections, the book of collections are real. I never really looked at it. Sometimes to be honest, sometimes we, the Bible is there, but we sometimes breeze through and the Psalms get more noticed. But I never knew it spoke to me. It's when you sent an email about the new material. And I started looking through my email, but I never really, it jumped out on me. I never really meditated on the scripture like that before. And I said, oh my gosh, this is speaking to my very moment, that moment by moment life, what I'm going through, and the reassurance Christ has given me. I said, wow, it really jumped up off the pages. So thank you. Thank God for that. Thank you. Thanks for sharing that. Yeah, that's true. And also a couple of other things that are very practical, things that we see are practical instructions rather is a couple of them. One is about the allowing the peace of God to rule in our hearts, similar to Philippians four. One way to do it is to pray about everything, whatever might cause anxiety, pray about it, take it to the Lord with thanksgiving. And we will have the peace of God, rule and reign in our hearts, right? Yeah. Yeah, Kiran, you want to say that what Christ has done so far is teaching us, like, giving thanks to the Father, our Lord Jesus Christ for everything. He rescued us from the kingdom of darkness and Christ is supreme, supreme from invisible God. He did so many things and He rescues us. He's a sovereign God. Yes, sir. Right. Right. Yeah. Thomas, did you put your hands up? Yes, Pastor, in moreover Paul's instruction is really in chapter 3 verse 12, 13, 14 and 15 and 16. Especially in 13, he says that with one another and forgiving one another. If anyone has a complaint against us, he even tries to forgive you. So you also must do. But love all the one love. Put on love. This is on the bond of perfection. That's really a golden instruction for the believers. Yeah, absolutely. Yeah.  So it's not just a theoretical thing. And they actually lived it.    But when we actually allow, or when we allow the word of God to dwell in us, or rather, when we intentionally give space, make space in our minds and our hearts for the word of God, then the overflow of that, the outcome of that is, you know, what we see in verse 16. There's a teaching and in all wisdom teaching admonishing and singing and Psalms, hymn spiritual songs, just overflow, you know, of our hearts with grace in our hearts to the Lord. And verse 17, and also verse 23, you know, in whatever you do, in word or deed, you know, do all in the name of the Lord. Like, and that's a very, and very clear, simple, but it's a, it's a power packed instruction in the sense, you know, how can I, you know, if I want to do or indulge in sin, or even if I'm considering something, can I do it in the name of the Lord? Obviously, the answer is no, right? I can't. So saying, you know, giving thanks to the Father through Him, through Jesus, do it in the name of the Lord Jesus, right? So if we would reflect and say, okay, use that as a rule of thumb, you know, can I do this in the name of the Lord? Can I do this in the name of the Lord?  Of course, the context here is employer, employee, and, you know, for the employee, not to be a, you know, person pleaser, not to be a man pleaser, doing work only when things are, people are watching, observing, you know, that's the context. But if we would, you know, apply it broadly in our lives and see, okay, whatever I do, you know, can I do it wholeheartedly as unto the Lord?  Okay, I can't do it wholeheartedly, because I still have some doubts about this. And I can't do it as unto the Lord, because obviously, there are some, you know, there are some things here that, which are clearly, you know, not righteous, maybe, so, so that if that is our, you know, perspective, then even walking daily, in righteousness, that becomes very, very, it takes away all the other things, whatever doubts that I might have, or in what way I should walk.  In our spirit, there is so much freedom, you know, when we walk according to this, that, and that is what God wants for us as well, that will be sharp in our spirit, because sin clouds our thinking. Sin actually clouds our, you know, makes our spirits heavy, weary even, but when we walk in righteousness, when we walk in this manner, there is clarity in our thinking, there is clarity in our speech, there is a lightness and liberty in our spirit, and which is what the Lord wants for us, right, yeah, to walk in freedom. Okay, so, so we come to the end of, we've come to the end of Colossians, so we have about, yeah, yeah, thanks, Kiran, I see that, yeah. Okay, so we have about six more minutes. So, I guess we'll start off with the Thessalonians, and we look at the introduction of Thessalonians, and then we'll continue with it, you know, next class. I think we have about two more, yeah, two more sessions, which is basically four classes, and we should be able to complete Thessalonians by then, right. So, if you turn to Acts chapter 17, you know, Acts 17, that's where we read about Paul, and he goes to Thessalonica, and he's preaching over there, and it's not a very pleasant experience, right. So, so this is what we read, okay, so this is during Paul's second missionary journey, right. Acts chapter 15, we read about what happens just before they start, and John Mark, and all that. Acts chapter 16 is about Philippi, and then Acts chapter 17, we read about Thessalonica, right. So, let me just read a few verses here, 17 and verse one. Now, when they had passed through Amphipolis and Apollonia, they came to Thessalonica, where there was a synagogue of the Jews. Then Paul, as his custom was, went into them, and for three Sabbaths, reasoned with them from the scriptures, explaining and demonstrating that the Christ had to suffer and rise again from the dead, and saying, this Jesus whom I preach to you is the Christ, or is the Messiah. And some of them were persuaded, and a great multitude of the devout Jews, devout Greeks, and not a few of the leading women joined Paul and Silas. Okay, so this is what we see. So, they moved from Lydia's household, from Philippi. In Philippi, they were imprisoned, and the whole incident of singing and worshiping God in prison, and the earthquake, and the jailer saying, what must I do to be saved, and all that happened just prior to this. Then the morning, they are released from the prison, and they move from there. They go to the Lydia's house, the one who actually, a businesswoman who was selling fabric, says that she was a seller of purple. So, from there they come here, and they come to the synagogue, and he says three Sabbaths, so which is typically three weeks, roughly about three weeks, or maybe more. He is there, and is explaining, demonstrating. It's good to notice that he is explaining and demonstrating, which means that not only was he teaching from the scriptures, from the Old Testament text about the Messiah, probably all the prophecies and everything that was pointing to Christ, and talking about the earthly ministry of Christ, and how he fulfilled each one of them. So, he did that, and also demonstrated, which means that Christ is the healer, Christ is the deliverer, and Christ is the bondage breaker, and he's the saviour. So, as people put their faith or put their trust in him, they experienced. So, he took that, so demonstrated, meaning there was, it was made manifest, the power of God being made manifest. So, we know that in one Corinthians, from one Corinthians, we see that he taught about the gifts of the Spirit. So, we know that here demonstration about Christ would also mean that he moved in the gifts of the Spirit. So, word of knowledge, word of wisdom, prophecy, gift of faith, miracles, working of miracles, all that. So, there was a demonstration of that. So, he taught them, and he demonstrated to them the power of God. So, we see explaining and demonstrating that Christ had to suffer and rise again from the dead, and saying that, just this Jesus whom I preach to you is the Christ, or is the Messiah. Now, some of them were persuaded, and a great multitude of the devout Greeks, and some leading women, they obviously came to the Lord and they joined Paul and Silas, even as they were teaching. Okay, that was five. Now, the Jews who were not persuaded became envious, took some of the evil men from the marketplace, gathering a mob, set all the city in an uproar and attacked the house of Jason, and sought to bring them out to the people. But when they did not find them, they dragged Jason and some brethren to the rulers of the city, crying out, these who have turned the world upside down, have come here too. Jason has harbored them, and these are all acting contrary to the decrees of Caesar, saying there's another king, Jesus. And they troubled the crowd and the rulers of the city when they heard these things. So, when they had taken security from Jason and the rest, they let them go. So, we see a lot of confusion being created by the Jews who were not believing. Some of the Jews were persuaded, a lot of Greeks were persuaded. It says a multitude of Greeks, many people, but they created this confusion and they were brought before the, they brought Jason, who was properly extending hospitality and they stayed there with this house. So, they brought him, beat him and took, you know, extorted money from him and then they let them go. Okay. So, from there, they go to Berea. So, it's in a Macedonian region, there's Thessalonica, just a capital of that province and also, you know, Berea, which is just below Thessalonica. So, okay. So, we'll get into it next class and we'll look at first and second Thessalonians. Okay. Right. We'll stop here. God bless you. See you guys. Bye-bye. Thank you. See you. Bye-bye. Thank you. Bye-bye.


All right. Hi everyone. Welcome to literate the last session of the KubeCon. I hope you'll enjoy this KubeCon. Today's the six scheduling maintenance track. My name is Wei Huang. I'm the coach of six scheduling. I work for Apple as a software engineer in the AML department. My name is Ken. I'm from Dark Cloud. I also work on community staff and I'm also on the upstream. I work together with Wei. Yeah, that's all. Okay, let's dive into today's session. Today's session, we'll first of all go through what the schedule is and what the current framework looks like and then give you some recent update which is made sure it was completed in 129 and in the upcoming 130 releases and also mentioned some other notable updates and then can tell you I mentioned some sub-projects update which is sponsored by six scheduling. Okay, so first part is scheduling overview. You may have known that a scheduler since 119 firstly introduced the scheduling framework to orchestrate the whole scheduling workflow. So basically all the plug-ins right now shipped with the default Kubernetes offering are using this framework and a lot of auto tree plug-ins which is developer to satisfy in-house scheduling requirement are also using this framework. So basically I intentionally skipped the first part, the purple part I mentioned later in detail. And then you can see we start with a sorting interface which is internally we maintain a scheduling queue to accommodate incoming paths and the by default is sorting by the paths priority and then the schedulers the green part will choose the head of the queue and pick out for scheduling and to the scheduling cycle. So it will go through a series of six detection point. The first of two is called pre-filter and the filter which is majorly to decide whether the path can be scheduled according to predefined hard requirement or not. So by the end of this phase you will give you a result says, okay, how many nodes can fit this path? If yes, then continue to the happy path is to go continue to pre-score, score, etc, which is basically to prioritize this nodes candidate to give a final optimal node for the part to be scheduled on. And then once the last phase of the green box finish the scheduler will start from the head of scheduling queue again to pick up at the next part to schedule the next part. So why it separate the scheduling cycle and binding cycle is because binding cycle is sometimes time-consuming because it has to interact with the APS over to sending a binding request. So the yellow part which is called the binding cycle which does what it does is to do sending a binding request to the API server which is basically underneath to set the spec.nodeName for the path. So this is for the happy path. That means you have at least one node to accommodate the incoming path, but it can happens that there's no single node can host the path. So it goes to the upper layer if you look at the red sign here. So it calls post filter. That means no single node can accommodate the path. So it will trigger enter the post filter extension point. So right now a default implementation of post filter is called preemption. So basically it does is to evaluate whether there's any like less important low priority path can be preempted so that make room for the incoming path to be hosted on. So that is preemption. And because each Kubernetes path has default grace termination period which is defaulted to 30 seconds. So basically schedule cannot just sit there for 30 seconds for the path to be physically deleted and then place the path on. So basically if the path has to preempt at the path, it should be immediately returned at the end of post filter. So basically the output of post filter is to post temporary nominate a node name to the path and then return immediately. Wait for its next turn to be really scheduled. By that time it also will check whether the victim path has been really physically deleted. So this kind of feature to prefer, for example in the next round, they will prefer using the nominating node name set on the status field for the path. This feature is a little surprising. It's also used by some external component like Carpenter. They're using this to do the scheduling. You have a very fast path because they involve some node provisioning. So if they don't need to go through the regular scheduling flow to go through the previous, for example, 5,000 or 2,000 nodes because this path has been proved to not fit for the path. So that's basically optimization and how it's used by other components. So recent update section. We will mention a few caps. Cab, if you're not familiar, is short for Kubernetes enhancement proposal, which is a very formal process to submit the features and to track the importation progress. So the first one I want to introduce is called the path-scanning readiness. So as I mentioned earlier, for a pretty long time, this is the scheduling framework, which is just basically a path gets created. They will immediately go into scheduling queue and wait for it's turn to be pick up and scheduled. But it doesn't fit for some user scenario that is once the path gets created, it doesn't really mean the path is ready for scheduling. For example, the path maybe needs to go several external components to check its readiness, for example, whether it is quota, some in-house quota has been satisfied, et cetera, et cetera. So that is why we introduce an extension point called pre-in queue, which happens before the sorting and also also the whole scheduling life cycle. So once the path gets introduced, you can implement the pre-in queue extension point to add customized logic to pre-check the path. Once it's satisfied, it will push it to the scheduling queue. So this feature is alpha in 126 and beta in 27 and the GA in the upcoming 130 release. So this may be a little abstract to end user. So for the users that use this feature, we introduce the scheduling gates field in the path. So in the left rectangle box, you can see that it has two scheduling gates field. One is called full, one is called bar. These fields in the scheduling gates has to be specified at the beginning of your creative path. It can be your controller take the ownership of the path creation or it can be some mutation webhook. But once you set it and raise the path, create the path via API server, it's a one-way direction. So you can now say, okay, sometimes later, you add additional scheduling gates. That's not allowed. So if there's some scheduling gates out there in the bottom, you can see that the path will stay in scheduling gated phase. That means the path inside scheduler has not entered the regular scheduling cycle yet. You'll wait there for some external signal to remove the scheduling gates. For example, the full maybe has be checked, already satisfied, then it will be removed and then the bar gets removed. Then it will lift up the gate to push the part to the regular scheduling cycle. And then you will see your normally familiar with the part where it gets from pending to running. So basically, this feature introduce another scheduling phase, which is called the scheduling gated, for you to integrate with your components to do some very customized scheduling requirements. One typical thing is that some organization has some in-house quota requirement before the part can be scheduled. So this is used, I think in the Apache Unicorns pregimall is used a lot to increase their throughput. The next one, which is also a cap, is called the main domains in part topology spread. It's a sub feature of part topology spread. The motivation is that some users, I would say, a lot of users use cluster rows of scheduler in the cloud offerings. So as time goes, the project domains may shrink, for example, may shrink to one. And in that case, the part topology spread, the max excuse, is actually no OP because you only have one domain. So in that case, all your part will be stacked into the same domains. So that may be not the user one because they may also prioritize the application's resiliency, high availability over the others. So in this case, what they want maybe is that, okay, I don't want the part to be scheduled to the same domain, that doesn't make sense to me. So what they want is that, okay, just let the part be pending. I do want the part to literally to another domain. So what you need to do is that in the scheduling part topology spread constraints field, add the main domains equals two. To simplify, this in this case just mean domain equals two. So this kind of information and the pending part will be detected by your cluster of scheduler, either compender or vanilla cluster of scheduler. They will create an additional domain for you and then the part will be scheduled to the domain. That's what you want. So how to configure it is pretty simple. The main domain is optional field. So by default to set to one, you have to set a non-active number here. So another feature is called queuing HIN. It's very internal feature. It's not oriented for the end user, but I will mention a little bit. So in scheduler's history, before 122, how the part can be re-queued if the part is pending is pretty wild or aggressive. So basically for every system event, no matter it's a node and a part gets deleted, we'll trigger move all the parts back into the scheduling queue. So you can imagine it's super inefficient because you can put the high-level parts again and again, push back to the head of the queue, so that costs the low-priority part. It doesn't even have a chance to be scheduled. So this is classically called head-of-line blockings. And then in 122, we introduced some cost-grant approach by allowing the plugin developers to define which events are related, or we can say can help the failed part reschedulable. For example, in this case, you can define in your volume plugins that is only some PVC or PV event happens with the scheduler will move the part back otherwise just stay there. But this is sometimes not that fine-grained. So in the latest release, we introduced the queuing hint to not only allow the plugin developers to define related events instead of define related callback functions, which is we give enough context information to the callback function signature. So you can choose how to implement that callback functions. This is the original interface, which is very simple, is just return a series of the cluster events. A new interface looks like this, much complex, I would say, but it can achieve better throughput. It's also used in the DRA implementation. But right now, we found some bug, I would say. It's because of the internal implementation uses a double-link list, but sometimes the object cannot be garbage collected soon, so it causes excessive memory usage. So on one patch release of 128, we disable it by default. So we're still working on that to optimize the internal data structure to make the memory usage to the same level as before. So it's a queuing hint. The next feature is called match label keys in part affinity or part entire affinity. So if you're familiar with part body spread, there's also a field called match label keys. So basically part affinity here is adopting the same philosophy. So the requirement is that as time goes, upgrade the application is definitely a requirement. But Kubernetes, for example, the deployment's upgrade is that I will spin up a new one and then I delete a new replica and then set a new one and delete all the replica. This is by default the upgrade strategy. But think about a case that you are using part entire affinity as specified, two replicas on two hosts. And then when you want to upgrade the deployment, when the new replica comes, it actually cannot find a place because the old ones are still there and the new one match the older replicas. So in this case, your upgrade is stuck because the new one cannot be scheduled and the old ones are still staying there. So to resolve this, we introduce the match label keys. That is, you can specify in this case a part temporary hash as a key. And in runtime, schedule will inject the concrete value for you. You don't need to be aware of that. You just need to in your side specify which keys can be uniquely defined, the new replicas to differentiate it from the old replicas. So it can achieve what you want so that you are not stuck in the rolling upgrade. So the same case can happen to mismatch label keys. So it's in pair with the label keys. So basically in the runtime, the operator will be not in comparing to the previous match keys in. Okay, the last cab I want to mention is the 10 manager decoupled from node lifecycle controller. So basically this is a cross-seq feature which is mainly related with scheduling, mainly related to seek apps and signal, but historically we choose scheduling to host this. So basically the motivation is in the old implementation of node lifecycle controller, we don't differentiate, we just use one simple controller to do two things, to define how a part, sorry, how a node is tainted upon which conditions and when the, when the tints are applied, how the part should be evicted. So that is sort of makes the responsibilities in one single controller. So for some use cases, we want to separate them so that they can continue to use the upstream default implementation of define how the node should be tainted upon what kind of conditions and also use another component to replace the entry implementation to do what they want, like to see, okay, maybe this 10 should not trigger eviction or trigger eviction another way. So this feature is for that to basically it's more re-factor the code base so that give the users more freedom to do their customization. So under noted updates, one thing I want to mention is that the CUBE schedule configuration has been pretty stable along the years. So right now in the upcoming 130 release, we were just a simple third one version, which is v1. The old ones like v1 beta 3 are all removed in 129. Okay, I will hand over to continue to introduce the subprojects. Thanks, Wei. So next I'll go through the subprojects and see what progress we made in the past several months. So the first one is cork. So before we dig into the details, I would like to know how many people sitting here have ever used cork before. And if yes, please raise your hand. One, two, three, not much, but yeah, good. So cork is the two keys that have to build a cluster. So you can take it as a can, but the difference is can still consume resources. But cork, they are only API objects, so it will not use actual resources. Let's take a look at the picture. So actually, the controller plan will communicate to the cork directly, and there is no couplet. So you can fix the status of your API objects. And actually, that's why you can build up a cluster with thousands of nodes in a laptop, because there are only API objects, there are no real resources being used. And in the last list, we achieved several features. So the first one is about the simulation for the CPU and the memory usage. So actually, you can act as you are running the real pod in your cluster, in your cork cluster. And the second is about the state API. So state API is a CRD, so you can define the desired state you want. And you can actually define several states you want. They can transition from one state to another. And yeah, you can almost move the whole lap circle of your API object. And also, cork has two keys named cork control. So we add three new commands, the hack, the record, and the reply. So the hack command can modify the resources in ETCD directly by passing the API server. This does not set secure, because you are skipping the API validation, defaulting, so you should be careful when you want to use this command. And another one, you know, basically when you have a bug and it's hard to reproduce, so you can use the record command to write down all the events that happened in that moment, and replace them. And you can repeat this again and again until you find out the reason why, so the lose cost, yeah. Okay, the third one is about the integration with the metric server.  So all about cork is about simulation. So for the next step, we'd like to simulate like the GPU usage, because GPU is expensive, and also the volume provisional, and like the 4D port and the node scenarios. So this page is about how cork is widely used across the community. So we can see a lot of organizations, communities have used cork already, and yeah, cork has achieved, you know, got more than 2000 stars in today. It's quite popular, right? And the next about a queue, so still I'm also a contributor of a queue. So I would like to see, because in this coupon, I heard several topics mentioned about the queue. So I would like to know how many people here have ever used a queue before, if yes. One, two, oh yeah. Great to see that. So basically, queue is designed to, for the job management and job queueing. So it offers quite a lot of capacities, like the priority-based job ordering and job preemption. Yeah, and the queue is quite good at the resource quota management, like you can fair share your resources across different tenants and borrow resources from different tenants. This can somehow, you know, improve your resource utilization. And the next about the functionality is this can somehow, you know, for course saving, like you have a support instance, and if there is not enough, then you can fall to your own demand one. The next about the multi-class support, yes, queue right now is support the multi-class, and we'll talk a bit about that later. And in queue, we actually support a lot of jobs, like batch job, no doubt, and all kinds of coupon flow jobs, the rate job and the pause and the pause groups. Yeah. I think the most important thing about queue is that you can work queue with the community's native components smoothly. So there is no migration work there. Oh, I forgot that. So we got more than 1,000 stars a week ago. I think that's a big milestone for our project. So this is how queue works at the high level. So for short, the queue will have to manage how jobs are, when the job will run, when the job should be claimed, and then when the job is admitted, it will leave the scheduling to the default group schedule. And if there is not enough resource in the cluster, the cluster auto scalar will take place. Yeah. This is how queue works. And the weeks ago, we just released a new manual version, 0.6.0.  So the first one is about the multi-cluster support, aka multi-queue. So this is a master worker model. So you have to install the queue in each cluster, and the master queue will decide which cluster to dispatch the job to run. If you are interested with this feature, you can read the cap for more details. The next one is about visibility API. It's somehow for the insights of the pending jobs, pending workloads. So you can have an overview about how your cluster looks like, how many jobs are running, and how many jobs are pending right now. And next about the landing limit. So before this feature in queue, let's say we have two tenants, and one tenant can borrow the resource, can borrow all the resource from another tenant. So it's greedy. So with this landing limit feature, you can guarantee some resource for self-use CG, and it will never be borrowed by your neighbors. And now it's about all-or-nothing queueing. So I want to highlight that this is completely different from the coop scheduler pull group. It's only implicit in the queue queue. So the capacity is quite same. So you can annotate the pod. So a group of pods can be queued as a unit. Yes. And before we support the red job, but some of our customers say that I also have some red cluster in our queue. So I will also support a red cluster. So it comes. And we also support some enhancements for the pre-emission and some new policies. For the next step, we mostly focus on three things. The first one is about the more fairness resource sharing, like the dominated resource fairness. And right now, queue only supports two-level hierarchy. So the cohort, the class queue, but it's not enough in a big company. So we are now considered to support the hierarchical cohorts. It's already designed. And yeah. The last one is about the integration. Integrate with K-Serve so we can manage the online service, online influence service, and offline influence service together. Also the Kubeflow pipeline. And the next about what some extension, I guess a few people here has heard about this project. So today we have several ways to extend our schedule. The first one is the schedule framework. This is recommended. And the third one is about the multi-schedulers. And the third one is about the schedule extenders. So schedule framework is, you know, it's recommended, but it's not, whether you have to recompile your schedule once you have a new planning. And the schedule extender is quite flexible, but it's slow because it invokes further API requests.  So that's why we introduced the Watson extension. This is, you know, high-level picture about how Watson extension looks like. So we have Watson plugin in the schedule. We usually call the host plugin. And we have several guest plugins. So the guest plugin is actually where you're scheduling logics located. So we'll combine, let's say you have several guest plugins and you implement the functions and then we'll compile them to the Watson files. And the Watson plugin, the host plugin, will load the Watson files. So that's how it works generally. So with the Watson extension, so you don't need to recompile the schedule, you just need to replace the Watson files. Oh, yeah. And it's faster than the schedule extender. Also, because, you know, we have to implement the extension points in Watson plugin as well, but the interface is quite similar to the schedule framework. So you will have the same experience like developing the traditional schedule plugins. Also, it's a save send box because even if you panic in your guest plugin, the coops schedule will not be impacted. But yeah, it's still slower than the schedule framework right now. So if you want to use the Watson extension, how to use, how to run this. So first of all, you have to recompile the schedule with the host Watson plugin because it's still out of the tree. But one day when it became the material, we may make it an issue, I don't know. And the second, you should enable the guest plugin. It's quite similar to what we do today. So you just configure your coops schedule configuration. Here, we enable the two Watson plugins, the plugin one, plugin two, and we specify the guest path and then it works. We are very close to our first release. Yeah. And if you want to try in advance, so we have a sample here. So please. And the next about the schedule plugin. So we host some out of tree plugin in this repo because not all the plugins are fit for every user. So besides the stable ones like the coops scheduling, the capacity scheduling, we introduced two new plugins. So the first one is about the system core-based scheduling because some parts will share the, you know, the same host kernel. So the system calls may lead to some attack of a node. So the scheduling can make decisions based on the system core usage. So even if attack happens, there will be fewer, fewer victims in your cluster. Yeah. You can take a look if you want, if you have similar demands. The next is about the disk IO schedule. So it's quite straightforward. So you, it can make a decision, a schedule decision based on the real-time disk IO. So to achieve this, you have to install a disk driver in your each node. So it will report the real-time disk to the new CRD. And the schedule will read the information for while scheduling. Yeah. We also got more than 1,000 staff. So the next about the schedule. So, you know, a coops schedule make a schedule decisions at a certain time, a certain point. So, but you know, our cluster is truly so some nodes may come in, some parts may be deleted. So this will lead to unbalanced cluster. So this is what the schedule works here. It will evict the unbalanced parts and make them scheduling again by the default schedule. In the new release 0.29, we didn't introduce many big features, but we do have some reinforcements like respect to the node affinity policy, node tenant policy, and the match-level case in the port-to-project spread plugin. And also we respect the image pool back off in the port lifetime plugin. So yeah. We also have some significant fix like the user experience improvements, docks have installation logs, and yeah, several CVs. And each time I talk about the schedule, I will find that there are some new contributors in this schedule. And at this time as well, we have got five new contributors. So this is inspiring for the maintenance, right? Yeah. So that's all for today's session. And but if you have questions, if you have inspiring ideas, you can bring to our six slack. And you can also subscribe our mailing list and also join our meetings, the six scheduling, just schedule whatever. And thanks to all the contributors. Yeah. Thank you. So yeah, you can write and leave it for the back by scaling the QR code. And if you have any questions, yeah, please go ahead. Yeah. Thank you for the session. I have a question about scalability on the pre-enqueue plugin, the new plugin that you showed. Are there any tests done on how many pods it can hold? For pre-enqueue, actually, the pressure is not from schedule. Instead, this is the opposite way. Introduce pre-enqueue is to release the pressure on schedule. Because as I mentioned, pre-enqueue, the power of the life cycle, where it just goes through one direction cycle. So it will impose extra pressure to the API server and also no extra cycle to the rigorous scheduling cycle, because that's the pre-enqueue cycle. And then the pre-enqueue can happen very lightweight. So for the default schedule, the implementation, you can think of it as one line code. Whether check the lines of the scheduling gates equals zero or not. But when it comes to external plugins, it will quite depend on your implementation. For example, if you, in your plugin implementation, you're involved for API code to AWS or externally, never definitely as the latency to the schedule. But for scalability, it doesn't impose the pressure. It's just depends on your presentation, well, at the latency. Yeah. Yeah, you can hold. Yeah. Because I can, yeah, my share was this. The internal implementation of pre-enqueue is that it's only invoked by the power of gas-enqueued. So if the power, that means the power is only enqueued once if you want. So in the later cases, it's all based on the signals that it sends to the schedule via some other ways. For example, the scheduling case implementation is to remove the, remove the gaze. So if your implementation is pretty graceful, I don't think hosting a thousand pauses is the pain. You mentioned that there are some enhancements in the framework, right? So for example, the callback interface, if I recollect. So the existing plugins doesn't require to implement any enhancements to be able to cope with these extensions, enhancements in the actual framework. So schedule framework, the actual ones all design backwards compatible way. So that means if your existing plugin doesn't want to use the latest SDK interface, they can still work as is. That means the behavior is still as before. For example, for queuing him, you will still adapt to the older ones that are associated with some cross grain events, right? But you want to opt-in, you then adapt to the upgraded to the latest SDK and choose to opt-in. Yeah. This is always the design rationale. We don't break the order. Sure. So that also means that if... I think we are running out of time, right? Yeah. So I will be here to answer. Okay. Sure. Thank you. Thanks for everyone coming. I think we are the very last slot for the session. So enjoy your call. Thank you.


Hello, Dr. Hirsch and the Scholarship Selection Committee. My name is Joshua Sanchez and I just wanted to extend my sincerest gratitude for selecting me as one of the recipients for the San Jose State University iSchool Returning Students Scholarship. I'm deeply honored to be a recipient of this award and I'm very excited to start my second year at the iSchool. Looking back on my first year at San Jose, I have learned so much and there's been a lot that I've been able to apply to my work here at the Long Beach Public Library. Since submitting my application, my team and I have successfully promoted mental health programming for veterans as well as for our staff. I'm also looking forward at the end of the month to promoting my bibliotherapy and art therapy programs for people experiencing homelessness and LGBTQ youth. I'm also excited to continue my work with California Humanities to promote storytelling from the immigrant perspective. Thanks to the classes I've taken here at San Jose State, I have both the knowledge and confidence to take these projects on with a high level of execution. It's my goal to provide the highest quality programs and services to the citizens of Long Beach. I'm looking forward to continue serving this community as I enter my second year. As I begin planning, I've looked at classes that push the boundaries on what it means to be a public librarian in the 21st century. I hope to deepen my skills and knowledge of the profession and take courses that focus on the cutting edge of library services. I'm very excited to continue my education here at San Jose State and once again thank you very much to Dr. Sandra Hirsch and the Scholarship Selection Committee.