All right, now I begin by saying it's Mother's Day, and you know, I was telling First Service, I was saying, oh, what a, it's not a message I normally would want to give on Mother's Day, but I'm going to. So let's open up our Bibles to chapter 14, and let's look at it together here. Chapter 14, Matthew, we're going to be looking today at verses 1 through 13. I chose to entitle this, a tale of two men and two women, and you'll see why in just a moment. So beginning at verse 1, Matthew chapter 14, at that time, Herod the tetrarch heard the report about Jesus and said to his servants, this is John, the Baptist. He has risen from the dead, and therefore these powers are at work in him. For Herod had laid hold of John and bound him and put him in prison for the sake of Herodias, his brother Philip's wife. For John had said to him, it is not lawful for you to have her. And although he wanted to put him to death, he feared the multitude because they counted him as a prophet. But when Herod's birthday was celebrated, the daughter of Herodias danced before them and pleased Herod. Therefore he promised with an oath to give her whatever she might ask.  And the king was sorry. Nevertheless, because of the oaths and because of those who sat with him at the table, he commanded it to be given to her. So he sent and had John beheaded in prison. And his head was brought on a platter and given to the girl. She brought it to her mother. Then his disciples came and took away the body and buried it and went and told Jesus. When Jesus heard it, he departed from there by boat to a deserted place by himself. But when the multitudes heard it, they followed him on foot from the cities. At this point, Jesus' ministry has been widely spoken of and his ministry is now garnering great interest. His fame, his reputation has spread throughout the land of Israel. And there are many now who are hearing of Jesus Christ. All the way back in chapter four, we already have seen this, but I want to reiterate, when Jesus was baptized, the Bible tells us in chapter four verses 23 through 25 that he began going from place to place sharing his message. When you look at those verses, it says Jesus went about all Galilee teaching in their synagogues, preaching the gospel of the kingdom and healing all kinds of sickness and all kinds of disease among the people. Then his fame went throughout all Syria.  And great multitudes followed him from Galilee, from Decapolis, Jerusalem, Judea, and beyond the Jordan. And so Jesus' ministry is impacting and his fame is spreading throughout the land. This has been going on for some time. People are coming into contact with him. And after they come into contact with the Lord, they begin to do what people naturally do. When the Lord did something in their life that was impactful, they would tell other people, and that's what they've been doing. And so in Matthew 13, we saw at verse two, great multitudes are gathered together to him so that he got into a boat and sat, and the whole multitude stood on the shore. So there were great multitudes who were not only beginning to follow him, but continuing to follow him and increasing in numbers. And so as the word goes out that this man, Jesus Christ, is ministering throughout the land, Herod, the highest government official has heard of him. Now, Jesus' fame is going out not because he's self-broadcasting or self-promoting, but because people are being touched by him.  You see, Jesus was not into self-promotion. The Bible makes it very clear in Proverbs 27 verse two when it says, Let another man praise you, not your own mouth, a stranger, not your own lips. It wasn't that Jesus was out there promoting himself in some self-righteous, arrogant way, but when the Lord Jesus Christ would minister to somebody, that person who was ministered to could not help but go out and begin to speak concerning him. You see, you can't keep a good thing secret. And he had done much good and his reputation is growing because of that. And his name is well-known. It's so well-known that it has reached the ears of a man by the name of Herod Antipas. So what we have here are verses before us that give us insight into the lives, the lives of two men, Herod and John, as well as the lives of two women, Herodius and Salome, her daughter. We'll see that in just a moment. When you read your Bible, you begin in Matthew and you see that the name Herod is mentioned. And sometimes you may be thinking that the name Herod, when you see it in chapter two of Matthew, is the only Herod that is mentioned. But actually you see the name Herod used a variety of times and all. And this person here is a man by the name of Herod Antipas. Now Herod Antipas was the son of a man by the name of Herod the Great. So Herod was what is called a tetrarch. A tetrarch was a ruler over a fourth part. It speaks concerning the fact that Israel had been divided into sections. And so this speaks of a man who actually has rulership over a certain portion. Now Herod was not a Jewish man. He was half Idomian. Idomian would be a descendant of Esau, son of Isaac. And it was half Samaritan. He had a father by the name of Herod the Great. And Herod the Great was despised because Rome had placed him a non-Jew over Israel. When you read history, you see that Herod, Herod the Great, was a murderer. He's the one who ordered the slaughter of the babies in Bethlehem when Jesus was born. He also ordered the slaughter of the Sanhedrin, the Jewish religious council. In order to retain his power, he had one of his wives and two of his sons executed. So this was a very evil man. He was a father. So after the death of Herod the Great, Rome divided the kingdom amongst three of his sons, Antipas, Archelaus, and Philip. His half brother is also mentioned in this passage. Now you have Herod, but you also have someone by the name of John. Now Herod was a man who was a political official. John is a kingdom official. Herod was appointed by Rome. John represented God. And what we have here is a life, a life that is really to be held in contrast. You have the life of one who walks in the spirit and you have the other one who walks in the life of the flesh. And as you compare their lives, you have an opportunity of seeing the results of living the way that they did. Now, both of these men, as you look in Herod's life and you look in John's life, both of these men can be seen as examples. John is an example of godliness. But Herod is an example of the carnal flesh. Now we need in this world more people like John because we already have a nation of people like Herod. So at this time what we have is a man by the name of Herod and notice with me in verse one who hears a report. It says in verse one, Herod the tetrarch heard the report about Jesus and said to his servants, this is John the Baptist. He has risen from the dead and therefore these powers are at work in him. And so Herod hears these rumors and these rumors that were circulating have reached his ears. The report about Jesus has reached his ears and he immediately associates Jesus with John. Now Luke tells us that Jesus had sent his 12 disciples out to preach and to minister. And in chapter nine of Luke's gospel, at verse six it says they departed. They went through the towns preaching the gospel and healing everywhere. So as always happens reports of what was going on reached the ears of a political official. And so it says in Luke chapter nine verses seven through nine, Herod the tetrarch heard about all that was going on and he was perplexed because some were saying that John had been raised from the dead. Others that Elijah had appeared and still others that one of the prophets of long ago had come back to life. But Herod said, I beheaded John, who then is this I hear such things about? And he tried to see him. It's interesting how that this rumor that is given to us here in Luke chapter nine tells us that some are saying it's John, some are saying it's Elijah and some are saying it's one of the prophets. Because when we get to Matthew chapter 16 in about three years, when we get there, when we get to Matthew 16, we're going to be seeing that Jesus is in an area to the north of the Sea of Galilee. And he's in a place called Caesarea Philippi and he's seated there at the base of this particular hillside and he begins to question his disciples and asks them the question, who do men say that I the son of man am? And then at that point, they begin to respond and all they're doing is regurgitating what has been said about Christ that we see in Luke chapter nine verses seven through nine. Some say you're John the Baptist, some say you're Elijah, some say you're Jeremiah or one of the prophets. So Jesus was aware of the rumors that were circulating. Herod is also aware of those rumors that are circulating. And yet Jesus feels it right for him to correct those rumors when he spoke to his own men and when he said, who do men say that I the son of man am? And as I've mentioned to you a thousand times already, the term son of man, who do men say that I the son of man? The term son of man comes out of the Old Testament book of Daniel chapter seven verse 13 and is in reference to Messiah. So it was like an open book exam where Jesus says, who do men say that I the son of man am? He already supplied the answer to the question by referring to himself as son of man. Who do men say that I Messiah am? That's what he's actually saying. But they say John, they say Elijah, Jeremiah, one of the prophets and then he goes on and he says, but who do you say that I am? Which helps for us to understand that it's not enough for us to be able to repeat what others have said. We need to have a personal knowledge ourselves. And that's where John said, you are the Christ. The son of, I'm rather Peter said, you are the Christ, the son of the living God. And that's why the Lord Jesus Christ would question them. So this rumor that we have recorded in Luke chapter nine is something that Jesus later on will address in Matthew chapter 16. So the rumors, the rumors that are being addressed and the rumors that are going forth at this time is that John was raised from the dead. That's what Harry is saying. I beheaded John, who then is this I hear such things about? He's wondering who he is. Perhaps he thought that if John were raised from the dead he would have supernatural powers. Since the works were good, that would mean that the power would originate in heaven. So that's what's taking place here. That's our introduction in verses one and two of Matthew chapter 14. Again, at that time, Herod the tetrarch heard the report about Jesus, said to his servants, this is John the Baptist. He's risen from the dead. Therefore these powers are at work in him. So what you have between verse two and three is actually what we would call a flashback. It's providing for us events that happened previously. And Bible commentators would say that what's being spoken about here in verses three following, are events that took place over a year before. One commentator says at least a year another says up to two years. So what's being spoken of here in Matthew 14 verses three following, are events that took place over a year before up to two years before. And so this gives us a backdrop of what's happening here. Again, he says John the Baptist is risen from the dead. These powers are at work in him. And now Matthew supplies to us the context. He goes on verse three and says it, for Herod had laid hold of John and bound him, put him in prison for the sake of Herodias. Notice his brother, Philip's wife. John had said to him, it is not lawful for you to have her. And although he wanted to put him to death, he feared the multitude because they counted him as a prophet. But when Herod's birthday was celebrated, the daughter of Herodias danced before them and pleased Herod. Therefore he promised with an oath to give her whenever she might ask.  The king was sorry. Nevertheless, because of the oaths and because of those who sat with him at the table, he commanded it to be given to her. So he sent and had John beheaded in prison. And his head was brought on a platter and given to the girl. She brought it to her mother. John had been placed under arrest by Herod. He was in a prison to the Southeast, a prison that is called Macarus. The reason John had been imprisoned is because John had spoken to him and had called him to repentance. That was John's ministry. John's ministry was to go forth, preparing the way for Messiah and to call people to repentance. When you look into the gospel of Mark in chapter one and Mark is giving his introduction, he says in verses two through four as it is written in the prophets, behold, I send my messenger before your face who will prepare your way before you. The voice of one crying in the wilderness, prepare the way of the Lord. Make his path straight. John came baptizing in the wilderness and preaching a baptism of repentance for the remission of sins. And so when he said, prepare the way of the Lord, I was mentioning this on Wednesday night, how that when it speaks of preparing the way of the Lord, it is speaking about when a king would come to visit a city or province or whatever, they would send people before him. And as the royalty was to enter into the city, they would send people before him who they were supposed to clear out the debris. And so as they were coming, they were to fill in the potholes and they were to clear off all the rocks so that when the king made his entrance, the pathway was smoothed without stumbling. And so when John went forth preaching, what he was preaching was a baptism of repentance. What he was saying is prepare the way of the Lord. He was saying, remove the debris from your life. Fill in those potholes so that he has a straight entrance into you. That's what repentance is supposed to do. It's preparing a way for the Lord to have entrance into your life. That's how you get a relationship with God. It's not a natural thing. It's not something you were born with. It's something that you repent and you have because you repented and you deposited faith in God. So it's not an automatic thing.  That person gets a relationship with God when they personally bow their knee, their heart to God and say, God be merciful to be a sinner. And that comes through repentance. So we're called to clear the debris out of our life so that he may have an entrance in and that's what he did and that's what it says in the book of Isaiah chapter 40, verse three and Malachi makes mention of that in Malachi chapter three, verse one. That's what it's all about. It's having a relationship with the Lord and it's opening up so that he might have an entrance into us. And so the message of repentance is something that is not just for certain elements of society but for everybody. You see, even in our day to day we have people who will look and say, well, of course they should repent. Look at the kind of life they live. They're living, they're not good people. They're bad people. Look at these drug addicts. Look at these people who are living this way and that way. No, it's not for just the people that we have categorized and we have said, oh, those are the worst of the worst and therefore they really need to repent. It's all of us. The Bible speaks concerning all of us. All have sinned and fall short of the glory of God. There's not a single righteous one. No, not one. So we're all to repent and that's what John was calling. So it wasn't just of certain people. It was to everybody. We all need to get right with God. When you look in Luke chapter three, verses seven through 19 and you look at his ministry, John the Baptist goes forth and he's calling everybody to repentance. You'll see in incident, incidences where he speaks to the multitudes, the general population. But you also see him as he speaks to the tax collectors and then you see him as he speak into soldiers. So the message is to go to all people, including those in the position of power, including those like Herod. So Herod had arrested John. Herod imprisoned him and the reason he did so is because John had directly confronted him. He said, it is not a lawful for you to have your brother, Philip's wife. And I didn't notice that again. In verse three, Herod had laid hold of John, bound him, put him in prison for the sake of Herodius. Notice his brother, Philip's wife. It does not say Herodius Herod's wife. Notice that. It says his brother, Philip's wife because what he had done was wrong. You see over a year earlier, John had been arrested because he called Herod to repent from adultery. While on a visit to the city of Rome, Herod had stolen Herodius from his brother, Philip. So in order to marry her, Herod had to divorce his present wife. He divorced, it's said like this, out of convenience. He didn't want this one because he wanted that one. I had somebody approach me years ago and asked me a question. Does God forgive every sin? I was a young pastor at that time, young minister. And I wasn't quite aware of how devious we could be with our theological questions at that time. I thought he was asking an honest question. I thought he was one in a real biblical answer. He had called the office. This was 34 years ago, 33 years ago. He had called the office said, listen, it's my lunchtime. Can I see Pastor David for a minute? My secretary said he's preparing a Bible study. It'll only take a moment. It's an emergency. Can I see him? My secretary calls me and says to me, this man's got an emergency. He needs to see you. Can he come in and see you? I said, okay, of course, if it's an emergency, yes. And so he came in. I said, just let me know when he's there. So he came in. I still remember him coming into my office and sitting down with me. Pastor, yes? I would like to know if God forgives every sin. And I'm looking at him and I'm thinking he's sincere. So I said, well, of course, Jesus Christ died on a cross that set us free from sin. There's not a single sin that God doesn't forgive you of if you confess and repent and turn from it, turn to him. Thank you very much. That's all I needed to know. I thought, man, I'm the Bible answer man. That quick, huh? Well, little, I didn't know what he did. I found out later. He was having an affair. He wanted to get out of his present marriage. And so by asking, well, God forgive me of my sin and the answer is yes, he will. He took that as permission to get rid of his present wife so he could go off with this other woman and get married to her. And he wanted to say he did it right. So just because somebody has questions doesn't mean they're sincere. And just because men like Herod would sit down and listen, you'll see this in a moment to the things that John said and listen to him with great interest doesn't mean that his heart was really being moved in the direction that it needed to go. We'll look at that in just a moment, but we'll get back to it when we say that God commanded the nation of Israel not to commit adultery. That is what is called the seventh commandment. You see that in Exodus 20, verse 14 where it very simply says you shall not commit adultery. Now John had repeatedly brought this to his attention. He had said, it is not lawful for you to have your brother's wife. Now he wasn't saying that it's against civil law to divorce and remarry. He was saying it is against God's law. Now when you look in the Old Testament there's a book of Malachi. In Malachi, God gives us insight into God's declaration of marriage and its purpose. In Malachi, chapter two, verse 11, that verse speaks of marriage as notice the Lord's holy institution which he loves. We call it holy matrimony, but God's word says it's his holy institution which he loves. So it's not an arrangement invented by human beings. It's an institution that has been given to us by God. Now it's intended, marriage is intended to complete human beings because the two shall become one flesh. So you have the man, you have the woman, the two united in holy matrimony are now completing one another. The two become one flesh. But it's also intended for the continuation of the human race through procreation. And one of the purposes of reproduction is to produce a people who love the Lord. Still in Malachi in chapter two, verse 15, Malachi 2.15 tells us that God seeks godly offspring. So the process of producing godly offspring is undermined through adultery and easy divorce. If you want to have godly children, it's important to have a godly marriage. It's important for a man to know that this is a holy institution. For the woman to know it is a holy institution. And should God bless us with children, then it is part of the blessing of God that one, we had children, and two, that we were able to have them raised in a home that is godly. Not everybody has the chance to be raised in a godly home, of course. But those of us who are Christians and those of us who have had our children, we have the opportunity, the privilege and responsibility of raising them up in the knowledge and nurturing of the Lord. Because it's God's desire to have godly offspring. And it's his desire in this holy institution of marriage to produce children who love him. Now, John's a preacher of righteousness. He's a man who preached with conviction. He preached with courage. He preached without respect of persons. And it didn't matter to him that Herod was the ruler and that Herodias would be offended by the message. It didn't bother him at all. So that gives to us insight into why the spirit did not allow Matthew to refer to her as Herod's wife. They didn't have biblical reasons for divorce. They were guilty of adultery by divorcing out of convenience. Mark chapter 10, verses 11 and 12, he said to them, whoever divorces his wife and marries another commits adultery against her. If a woman divorces her husband, marries another, she commits adultery. This is speaking about non-biblical reasons for divorce. And if someone out of convenience just divorces, you're entering into an arrangement that isn't blessed of God. Now we know that John was a man sold out completely to the Lord. He was filled with the spirit of God from the womb. And we know that he was a faithful preacher of righteousness. He feared the Lord. He loved people. And because he feared the Lord and loved people, he shared with them God's message. That's what you're supposed to have whenever you share. It's a fear of the Lord and a love for people. That's why you take this message and you give it even though you know there are times when people will refuse to hear and sometimes even violently oppose. We see that in the life of John. But that's what we're called to do. If you're gonna be used by the Lord, you need to have a fear of the Lord and a love for people. That's what you need. There's a man by the name of Samuel Bregel. Samuel Bregel was an early leader of the Salvation Army. He wrote, the way to spiritual leadership. And this is what he said, the way to spiritual leadership. It is not won by promotion, but by many prayers and tears. It is attained by confession of sin and much heart searching and humbling before God. By self-surrender, a courageous sacrifice of every idol, a bold, uncompromising and uncomplaining embracing of the cross, and by an eternal, unfaltering, looking at Jesus crucified. It is not gained by seeking great things for ourselves, but rather like Paul, by counting those things that are gained to us as loss for Christ. That is a great price, but it must be unflinchingly paid by him who would be not merely a nominal, but a real spiritual leader of men, a leader whose power is recognized and felt in heaven, on earth, and in hell. And that was John. John was a man who was courageous, he was a man of conviction, he was a man who was faithful, a man filled with the spirit, and he spoke forth God's word. And he spoke and he said, it is not lawful for you to have your brother Philip's wife.  You're committing adultery. Of course, in our day adultery is so common, people don't even think it's wrong anymore, but they don't realize the pain it causes for all involved. They especially haven't taken into consideration the pain it causes the children. Now, John is speaking, and it says in verse four, John said to him, it's not lawful for you to have her. And although he wanted to put him to death, he feared the multitude because they counted him as a prophet. So sexual sin must be reproved, even when people think our concern is self-righteous judgmentalism. You know, as a Christian, you can be called many things, but one of the things that I have found today is common for Christians to be called when you say something about sin is they call us haters, we're haters. And there's nothing further from the truth than that. And we don't, I don't know about you. I don't hate people. I do love them. I hate sin because I see what sin does to people. I see the results of it. I see the pain that it causes. I have seen the pain of sexual immorality. I've seen as you have the pain of unfaithful husbands and unfaithful wives and the pain that it brings on the children and the families because when a man and woman get married, it's not just those two getting married, though they're the ones who are married, but it's impacting the entire family structure, the mothers and fathers of the children, the grandparents of the children born to that. And when you see the pain of divorce and the pain of unfaithfulness, well, it causes you within your heart to speak even more fervently and encourage people even more powerfully, remain faithful to God in your vows. And so this conviction that Herod had experienced when John was speaking to him, well, that conviction could have led him to repentance and he could have been saved, but instead, John's consistency produced the desire to have John put to death. In Proverbs 29, verse one, the Bible says, he who is often rebuked and hardens his neck will suddenly be destroyed in that without remedy. How many times does the Lord have to speak to our hearts before we listen? You see, not only was Herod angered by John's message, but Herodias was also. Mark tells us in chapter six, verses 19 and 20, Herodias nursed a grudge against John and wanted to kill him, but she was not able to because Herod feared John and protected him, knowing him to be a righteous and holy man. When Herod heard John, he was greatly puzzled yet he liked to listen to him. And so as that's going on, the Bible makes it very clear though that Herodias nursed a grudge. She held it against him. She wanted to kill him, but she could not. It would seem that Herod wanted to silence his conscience by listening, yet he was also aware of the sympathies of the people which caused him to be a little bit afraid because it says in verse five, he feared the multitude because they counted him as a prophet. And so what's going on? Well, verse six, when Herod's birthday was celebrated, the daughter of Herodias danced before them and pleased Herod. Now, when it speaks concerning that, pleased, it pleased him? That word pleased is a Greek word that means it stimulated him. It sexually excited him. She wasn't there doing the hokey pokey. She was dancing in a lascivious and lewd way and so not only do we see two men, John and Herod, but we see two women, Herodias and her daughter, Salome, and you see the influence of a mother on the life of a daughter. You see the influence that Herodias has on her daughter because it was at Herodias's prompting that Salome makes the request and is no doubt that this exhibition was prearranged by Herodias in order that she might be able to see that John is put to death. And so there is the influence that a mother can have on a daughter. That's a very, very powerful thing. Now it's Mother's Day.  God knows I love you and he does too. But isn't it something that we need to take to heart? The influence that parents have, the influences that mamas have. My wife has tremendous influence on her daughters. We have our two daughters. And our daughters in watching mom have learned certain things that I think their husbands are blessed by. Just the other day, I had our family over for a birthday celebration. And one of my granddaughters was standing next to the, in the kitchen next to the island and my daughter, Korean, spoke to her niece and said to her, because my niece was about to serve herself, I think. And my daughter said something like this, I wish I could give an exact quote, but I did hear a bit of it. She said, in the Rosales home, the ladies make sure to serve others before they themselves are served. And I looked at my daughter, Korean, and I said, when did you learn that? I've never seen that before. No. But it's true. But it's true. And it's not because mama, Marie, made the daughters watch her when she served me. It's simply that they saw her serve me. And I am spoiled, and I've said this before, but it's true. My wife, Marie, was raised in a situation where she cared for me because her mama cared for her dad. And the daughters watched the mother. And they learn how to communicate. And they learn how to love. And they learn how to respect. And they learn how to do all of that. No, I can hear you already. Yeah, but the man better be, okay. Bottom line is service is a choice that we make. We make a determination that I will reflect the spirit of the Lord in my service to God and to others. It just so happens it's a much more pleasant experience when the person being served is grateful. We all understand that. But my children are learning and they're adults. And now my grandchildren are learning how to be a Christian, how to be a servant, how to be married. They're watching as they watch their parents. And in the case of Herodias, she brought her daughter Salome into it when Salome did her dance. Now Herod had been drinking. He and his men with him. He's a bit high, no doubt. And he begins to make a drunken oath. Up to half of my kingdom I'll give you. Ask whatever it is you want. And that's just one of those boisterous, loud mouth drunk statements that people can make. But before she makes her request, she goes to her mom and says, what is it that I should ask for? He has made this huge promise to me. What is it that I should ask for?  Prompted by her mother. I want this. This is what we want. What a dangerous combination.  And an evil daughter. This is what I want. I don't even, this is so gruesome, I can't even imagine somebody actually asking for something like that.  What should I ask up to half the kingdom? We can have anything we want. Well, Herod is thinking I've already got everything. The only thing I don't have is his head. That's what I want. You tell him that we want the head of John the Baptist. We want it on a silver platter and we want it presented to us as a gift. You tell him that. She goes back and we're told that because of his drunken oath and because of those who sat at the table with them, he commanded it to be given. And that's what happens. He was caught in his own promise. It appears that he didn't want to do this, but his pride and the evil one out and he had John put to death. Now, here's something for us to practically apply. He was grieved, but he wasn't repentant. He was sorry for it, but he wasn't repentant. And so for the sake of his oath, he slaughtered John. Somebody said, for fear of a woman, for fear of his peers, for fear of his throne, and for a lack of fear of God, Herod damned his own soul forever. Herod damned his own soul. He had fear of everything except the one he should have feared, which is God himself. And so he said, let it be done. And they, according to verse 10, John was beheaded in prison. Verse 11, his head was brought on a platter given to the girl and she brought it to her mother. His disciples came and took away the body and buried it and went and told Jesus. When Jesus heard it, he departed from there by boat to a deserted place by himself. But when the multitudes heard it, they followed him on foot from the cities. He was grieved, but not repentant. For his oath's sake, he slaughtered John. And his beloved friends and disciples came. They took his body, buried it, and they went off and told the one that needed to hear. They went and told Jesus. So what do we have here? We have an illustration. The greatest illustration of a disciple of Jesus Christ. Remember in Luke chapter nine, how Jesus said in verses 23 through 25, if anyone desires to come after me, let him deny himself, take up his cross daily, follow me. For whoever desires to save his life will lose it. Whoever loses his life for my sake will save it. For what profit is it to a man if he gains the whole world and is himself destroyed or lost? What does it profit you if you have every single thing that you wanted? Herodias did you had everything that she wanted except for one thing she wanted, the head of John the Baptist. That's the one thing she wanted. That's how evil her heart was. What is it profit a man to gain everything in the world and lose himself in the process? To have it all and have nothing. To have it all and have nothing. Those choices are made every day, aren't they? And one form or another. No, we're not asking for the head of John the Baptist. We're simply saying those who are speaking to us and calling us to repent ought to be silenced. This whole nation does that. Those who are speaking to us and saying, get your heart right with God, they ought to be silenced. That's hate speech, isn't it? I mean, they're saying things that make me uncomfortable. Let's silence them all. There's John in his cell. And as John was in his cell earlier, his men had come to him and he said, I want you to go and speak to Jesus. And I want you to ask him, are you the coming one or should we be looking for another? Because John knew that his days were numbered. He knew that the time was coming and so he says, you go and you ask him. Are you the coming one? As I mentioned to you when we went through that passage, coming one, the coming one was another messy anecdote. Are you Messiah or should I be looking for somebody else? Yes, it's true. I was there baptizing and yes, it's true. I saw the spirit descend and remain on you. And yes, it's true that God told me the one you see the spirit descending and remaining on, this is he. This is the Lamb of God who takes away the sin of the world. Yeah, that's what the Word of God says and all and this is what God had spoken in and all and he says, I know that, but I want you to go because my time is short is what he's in essence saying. There's an inference there. My time is short. I don't want to lose my head for the wrong one. Go and ask him. Are you the coming one? Should we be looking for another?  You tell him about the healings that are doing, the miracles that I'm doing. And blessed is the one who's not offended because of me. John, you have an idea of what Messiah is supposed to be, but you don't have the whole picture.  You don't have the whole picture yet. Apparently those words when they returned and said, no, he's fulfilling Isaiah's prophecies and those things that speak concerning Messiah have brought comfort to the heart of John so that when he heard the sounds of those soldiers walking down that hallway and he heard the door as it was opened up and he saw the man with that sword and he knew his moment was over. That he'd close his eyes here but open him up looking in the face of God. Do you think he regretted that decision? Do you think he said, oh, I need another week in this prison? I don't think so. No, he said it's time. I fulfilled my purpose. It's over. In my race, the course is finished. I'll see him because I've been faithful. I wanna be someone who's able to say, I've been faithful. I've been faithful. I pray that's for all of us. I've been faithful. Is it easy? No. Is it easy standing up and saying, no, that's not right? No. Would I prefer being liked by everybody? Absolutely. Absolutely. But I still remember that when Jesus came in to Jerusalem on Palm Sunday, that there were people throwing their palm fronds and yelling out hosanna, blessed is he who comes in the name of the Lord. And I wouldn't be surprised if some of the voices that were crying out, blessed is he for the other ones later on, say in the next week, crucify him.  I used to wish that I could preach in easy believism. I used to wish that I could stand up and say, guess what, man, you get saved and you'll never have another bad day in your life. There'll be nothing but, well, I'll write a book. You know, this is your greatest day. I mean, everything's great. Nothing wrong. Your enemies will begin to love you. Money will drop into envelopes and show up at your house. You'll never have another headache.  Promoted and promoted and promoted because you're gonna be the head, not the tail. I wish I could say things like that, but the Bible doesn't seem to indicate that. Now does it?  And that's what faith is about. I mean with him and faithful. Well done, well done, my good and my faithful servant. Enter into the joy of your Lord. It's been prepared for you from the beginning of the world. That's what you want to hear when he says to you, well done, well done. And the tears and the sorrow, the disappointment and the pain, the various things that you've gone through, he wipes away every tear and there's nothing but joy at his right hand forevermore. All you Christians pie in the sky, buy and buy, no. The Lord is never gonna leave me nor will he forsake me and though I walk through the valley of the shadow of death, I will fear no evil for he is with me. His rod and his staff, they comfort me and he prepares the table in the presence of my enemies and God is a blessing God who will take us through every step of this journey and welcome us in and then we'll hear that well done and that's all that matters to me in the end. Well done, my good, my faithful servant. John the Baptist.


It is quite clear that the Caledonians pose fierce resistance to Roman rule. This is exemplified by the fact that the Romans built two walls, 100 miles apart, to separate themselves from the fierce Northern tribes of Caledonia. Firstly, in 122 AD, the Emperor Hadrian instructed his troops to construct a wall that would separate Caledonia from Roman Britannia. This wall is located in the very north of modern-day England, just south of the Scottish border. A few decades later, in 140 AD, the Romans built a second wall, further north, to once again protect against Caledonian attacks. Antonin's wall was built between the River Clyde and the River Forth and frequently came under attack. Even though Antonin's wall is further north, it still indicates that the Romans never had a major foothold in a large part of Caledonia, as the wall was relatively central in relation to the landmass of ancient Scotland. The likelihood is that the Romans never managed to have any significant presence that far north in Caledonia. Furthermore, the Romans had to retreat from Antonin's wall only a few years after it was constructed. Under pressure from the native forces, the Romans were forced to abandon Antonin's wall in around 165 AD. The Romans threw their weapons down wells and buried them in pits, and part of the wall was buried itself, perhaps to help stop these tools and barriers being used by the Caledonians. The Romans retreated to Hadrian's wall, around 100 miles south. Then, in the 3rd century AD, there was a notable Roman push north in subsequent genocide under Emperor Septimius Severus and his sons. In 207 AD, Severus received a letter from the Roman governor in Britannia, Sinesio, that the northern tribes of Caledonia were revolting, and the Romans at risk of losing its foothold entirely. Using guerrilla warfare tactics, the natives had proved quite effective in fighting the Romans. The following year, Severus arrived with 50,000 men and 7,000 sailors, the largest Roman force ever to touch down on the British Isles. In 209 and 210 AD, the Emperor led the first north of Hadrian's wall to fight two alliances that had formed amongst the native groups a few decades prior. The first alliance consisted of the Caledonians to the north, with the second known as the Matei, who were based around Fife and on either side of Antony's wall. With 50,000 land troops, however, Severus' massive Roman force ripped through the land, eventually forcing the natives to agree a peace treaty. The peace was short-lived, however. Shortly after the First Rebellion, the people of ancient Scotland revolted again. Severus, who was stationed at York, ordered his men to commit genocide in response. The Emperor sent his sons north, with troops under the order of killing everyone, even the unborn in the womb. The resulting slaughter is thought to have decimated the lowland population for a century. In 211 AD, Severus died, and his sons headed straight back to Rome. After the death of Severus, the Romans dropped back to Hadrian's wall once again. In the subsequent decades, the Romans launched numerous campaigns into modern-day Scotland but never managed to conquer and hold vast lands north of Hadrian's wall. It is also in the 3rd century AD that we see the first reference to the Picts in the historical record. In 297 AD, a Roman writer spoke of the Picts in the Irish, meaning Scots, attacking Hadrian's wall. It is thought that the Picts are the descendants of the Caledonians. However, as the Pict high, meaning painted people, was also a name that the Romans gave to the northern people of modern-day Scotland. The Caledonians and the Picts may have essentially been the same people. Unfortunately, we don't know what these people called themselves. Rome's failed attempt to conquer all of modern-day Scotland came to a conclusive end in 410 AD. The reality is that Rome never managed to conquer Caledonia due to a fusion of various factors. If the Caledonians weren't so fierce and proud of their way of life, the weather wasn't so bad, the geography wasn't so hilly, the potential riches from conquest not so little. And if Rome wasn't engaged in other conflicts and numerous other corners of the empire, then Rome potentially could have conquered Caledonia, or at least progressed further into Caledonian territory. Yet, as it stands, they didn't. Which leaves me to simply say that the Roman Empire, despite its power, wealth and fame, never conquered the people of ancient Scotland.


Good morning everybody. Thanks for showing up today. This is Computational Fluid Dynamics, Kubernetes, Kubeflow Open Phone. Before we get started, I just wanted to thank two folks. I don't think either of them are in the room today. Eduardo Orango and although Coquincador, they were super helpful when I was going through this sort of process of figuring all this stuff out. Eduardo's actually not at Red Hat anymore. He moved over to Nvidia. So either way, still helpful or thankful for his help. This is not a super serious presentation. It's Friday. We've been here all week. It's, you know, all tired. It's going to be fun. I promise. I hope if you want serious, there's a blog post that I wrote that details all the gory GitHub repos and all this other stuff. This presentation is on the SCED website. You don't have to worry. We can go back to this, whatever. So if you don't get a picture of this, it's fine. And like I said, this is going to be fun. Full of bad jokes, terrible memes, all that kind of stuff. So I'm getting at least one round of applause. So we're good. Okay. Here we go. This presentation is built like building blocks, right? We're just going to try to put things together, get deeper, deeper, deeper as we go to explain how I actually was able to achieve this because in reality it was mostly just kind of an experiment. Before we get started, just a couple of questions.  Everybody. Okay. That's not totally a surprise. Who's doing HPC not necessarily on Kubernetes, just anything that would qualify as HPC, only a few. Anybody doing MPI stuff? About the same number. And then lastly, open foam existing? One. Okay. Cool. How did we get here? So me, I've been at Red Hat for 14 and a half years. I'm not a software engineer. I'm not a Kubernetes Kubeflow open foam developer. I'm not an aerodynamicist, but I do have a race car and it does have a wing on it. So I ended up getting this email from an account team out of APAC and they were like, oh, this automaker is doing open foam in a container. They want to do it in VMs. And I was like, can we do this on Kubernetes? And I remember it. I have these friends who do aerodynamic stuff and they offered me some of their models to play with and I was like, cool. Okay. Let's do an experiment. Let me see if I can run their fluid dynamics wing job in a Kubernetes environment. So review open foam and MPI. What is MPI? It does parallel or distributed computing. So we have lots of things that we do then lots of little things in parallel. Open foam is for computational fluid dynamics. Air is a fluid. Water is a fluid. It can analyze how it flows around objects. So we have this big job. We want to break it into little jobs and that's where MPI comes. Well, but not really, right? MPI doesn't do the breaking up. It just does the processing. Open foam is a tool. Well, it's actually lots of tools to help do fluid dynamic stuff. Water is a fluid. So we take our big fluid dynamics thing and we break it up into little pieces with open foam and then we use MPI to do all the things. Those things are actually really like individual servers for most folks. They're used to HPC and clusters and etc. And it does this over SSH, right? So MPI finds all these systems with SSH, goes into them, does the things. We need some kind of shared storage because all the little pieces need to find the work that they have to do, so to speak. So we got to figure that out as well. Okay. It's not really that simple, but for the sake of simplicity, we're going to draw this picture. Open foam, MPI, lots of stuff, RWX. Cool. Clusters and Kubernetes. So how many of you have something that looks like this? Right? You got a rack of servers, those stuff. Okay, cool. So 2014, this cool Kubernetes thing happened and then people started trying to do this, right? We want to run all the workloads in Kubernetes. So this is a workload. Can we run it in Kubernetes? Let's see what we can do. Starting to draw parallels, right? So open foam, MPI, fluid dynamics job, we've got Kubernetes, Kubernetes runs pods. If you remember the previous picture, we actually have a bunch of little things that we're doing, fluid dynamics. That's a lot of pods. We have SSH. We have RWX storage. We have Kubernetes. We have RWX storage. We'll figure out the rest. I'm not really sure, but we'll get there, right? So let's go deeper. Cubeflow. How many of you know what Cubeflow is? Whoever heard of it? Oh, wow, lots of folks. Okay, cool. So Cubeflow project dedicated to making deployments of machine learning. Well, we're not doing machine learning. We're doing fluid dynamics, right? But it has this operator called the MPI operator. Now it says it's for all reduced distributed training. It's just MPI. Like we're doing an MPI thing. Let's do an MPI thing for one thing. So let's go deeper. Operators, how many of you are familiar with Kubernetes operators? Most everybody in the room. Okay. Operators are this, it's too complicated. So operators are just a pod, right? There is a controller pod that looks at instances of a custom resource, custom resource definition extends the Kubernetes API. And then it does stuff. In this case, we're talking about the MPI operator. The stuff that it does is MPI stuff. So we've got MPI operator. We have this MPI job. I'm running a wing CFD job. It's going to happen in a lot of pods. Okay. I don't know. So lots of pods, Kubernetes, RWX, all the things. I used OpenShift. That happens to be the Red Hat Kubernetes. We have a storage product. Well, it's moving to IBM, but we have a storage product open data foundation that gives me RWX. And then we had to do stuff in the middle with Cubeflow. So let's go deeper again. Wing CFD, MPI operator. When you use the MPI operator and you instantiate an MPI job object, it fires up a launcher pod. That launcher pod then runs MPI into all the job pods that do the work. They talk to storage if you have storage defined. And all this happens on top of your Kubernetes environment. Okay. It still does this over SSH though. That's gross. And I had to do some nasty tweaks that are in the blog post to make it work on OpenShift, but whatever. We've got it working. It's fine. Okay. Here we go. So what is the open foam process? So the first step is this meshing process. You take a 3D object, you break it up into like a meshy thing. Okay. Cool. Then you decompose this into lots of small jobs. That was the thing we talked about early on. Then you do the problem solving and figure out what the fluid job is. And then you put it all back together. Relatively trivial. So I'm really good at this. And fortunately, the open foam project provides a bunch of examples for how to do stuff. And I also have these guys to give me things to do. I actually helped Moreland build their open foam cluster like they're in Atlanta. I went over there and had lunch one day. It was fun. Okay. So Dan break is the example job that's provided by the open foam project. It's a very simple water flowing over a thing. Let's try to run that job and see if we can make it work. So what do we actually have to put in here? This is where the rubber finally meets the road. Trigger warning. There's actually code. It's barely code. So this is the Dan break example. So you do some stuff. You break it up. You run the parallel job. That's the meat. And then you put it back together. So how do I get that script into this launcher pod, right? Well, fortunately, I actually know a thing at least maybe about Kubernetes. We have this thing in Kubernetes called a config map. It makes it easy to inject files into the containers. Well, that's just a script. Why can't I just inject that as a file into the container and then make the MPI job run the whole script as the entry point to the work? So that's what I did. I put it in a config map that I defined against the Dan break job that went into the launcher pod that launched the wing CFD pods that talked to the storage on the container platform. Okay, cool. And then yes, it did work. This is what it looks like. It was doing things. The first one is the Dan break job. The second one is motorbike, which is another example within. This is the open shift user interface that just shows like, hey, they're using CPUs. There's pods stuff. That's the log output from open foam for the one person who's using open foam. They've seen that a lot. They spend a lot of time looking at that text. This is the more land job. This is not really a picture that I generated. That's when they sent me, but that's what I did. I ran this job and it worked. Well, sort of. Remember this picture? I don't have one of these. I got this. I wish that was me. That's not my laptop. It's not my background. It's a cool place to work every day, I guess, or just really humid and mosquitoes. Anyway, so I didn't have one of these. I did have one of these. Really, I had that. Unfortunately, I didn't spend a lot, but I did have this. Again, I just kind of was messing around. I used M5A 4x large, which are not compute optimized instances. I was like, all right, well, let me do two CPUs for pods. I'll just do a boatload of pods because more is always better, right? 512 CPUs, it took five hours, and more land was like, man, that's really slow. And so the reality is you have to know what you're doing, right? Tuning, tuning, tuning, tuning is important. I'm not really sure. I could have used the compute optimized instance type. I was probably, I probably should use bigger pods that are using the whole server instead of lots of tiny ones. Maybe I was over slicing the job. When you have the surface and you break it up into lots of little pieces, you end up with overlap, and then you're basically just reprocessing the same things over and over that get edited out at the end when you put it all back together. Or just know what you're doing, right? Actually, being an aerodynamicist know what any of these things mean. Where can we go from here, right? So PMIX is a new thing around open MPI that gets rid of SSH. There's an actual server daemon kind of deal. It can run on Kubernetes. So it would be great if we didn't have to legitimately run an SSH daemon in all the pods and then SSH via MPI from the launcher into, this is just gross. Of course, more tuning. NVIDIA is working on doing GPU enablement for this particular project for open foam so that you can use accelerated processing via GPUs. But ultimately, what was the point of this whole exercise? It really was an experiment. Can I even do this? So the reality is that I wasn't trying to test if AWS was a good place to do this because the reality is AWS really isn't a good place to do HPC. Your data is probably not there. So moving all your data into there, that's going to be ludicrously expensive. And it's just expensive to do. Your instance types might not be available if you need a boatload of them. There's all kinds of problems with this. Most of you, if you're doing HPC, you probably have clusters on site. That big picture of all this, like you can do this. It actually did work. We were able to run open foam and more generally, whatever MPI type job you have, we were able to do it.  It is a scheduler. I would like you to run my job. It will do it. It has no concept of priority, preemption, queuing, like that kind of stuff, like traditional slurmy, whatever your HPC tool of choice is. It's not doing that today. But there's a whole army of extensions and schedulers and other things that are coming about that are doing those things. Red Hat is investigating how we can do some of them. We've actually got the product manager for that in the room right now. But ultimately, the purpose of the job, the purpose of the experiment was like, can we do this? Yes, we could. Okay, great. That's it. That was really fast. Wow. I was like barely 12 minutes. So, yeah. Happy to answer questions. I realized, again, this was supposed to be just kind of a fun, happy one. But yeah. That's all I got for you. Questions? Okay, we got a few. Hold on. We got a microphone. Great talk. So, where are you going to go from here now? Like, is this project done? Was this just kind of like? Yeah. So, it's good. Thanks for asking that. I'm talking to the NVIDIA folks about it. They're interested in figuring out how we can do that. I've done NVIDIA GPU things on AWS before on OpenShift and Kubernetes. We know it works. So it's just a matter of how do we do that with OpenFoam. The other thing is, where's the script? So, like, these steps, I was thinking about it yesterday, actually. It doesn't really make sense to do all these steps in the launcher pod. Maybe it would make sense to, like, do the preprocessing and decomposition in a tecton pipeline and then take that and then attach that as a stored object into the launcher pod and then just run the MPI thing. And then when that job finishes, do another tecton pipeline to run reconstructs. Like, does that make sense? I don't know, maybe. It would probably work. But, like, how much effort would it be to actually automate it? Would I need to write a tecton task to launch the MPI job and then, like, another... So, I don't know. It's an experiment that I'd be interested in doing. But that's kind of the, like, where would it go from here? And then I'm trying to figure out some performance-y stuff. Yeah, here. So, re-jiggering some of this, make it look more like Morelin's environment and see if I can, you know, get close to the performance or whatever. We do have some clusters and labs at Red Hat that I could run some of these things in. And then just figure out, you know, like, is there more storage optimization that I need to do or those kinds of things? So, that's kind of where does it go. You know, again, it was really, it was a thaw-dish experiment. Like, can we run these types of jobs? Grav, like I mentioned, is the PM that's handling a lot of the HPC stuff. Just doing more HPC things like this, not necessarily more open-foam, but just other HPC things and see how do they work and just lots of blogs and other things. It's a long answer. I, sure. Go ahead. Yeah, I noticed under the MPI run command, you didn't have the allow run as root flag. Just curious, like, did you have any issues with setting it up to, like, run as a user instead of run as a user? Yeah, so where was it? Privilege. So many images. Lots of memes. Nope, that wasn't it. It was... I don't remember. Anyway, we'll, we'll, we'll say here. So the, the, most of the challenges I had were because of SSH and because of the open-foam container.  And so for the sake of just, like, get it done, I just kind of blew the permissions wide open and tried to do it however I could. Red Hat has done a lot of work around randomizing user IDs and other security enhancements in OpenShift to, to reduce the blast radius of, like, misbehaved workloads, so to speak, and be more secure. And so we have some weird tricks that we've done in our builder images to, like, do things to Etsy password as the container comes up so that things think they're running as the right user. Anyway, I just, I didn't want to spend the time to go through it. So TLDR and your question is, yeah, I just blew the permissions wide open to make it run, but you could probably spend a little bit more time to make it run not like so, so badly. You wanted to see one of the, this one? Thank you. You're welcome. No question about this slide, just wanted to see it. Yeah. Okay, cool. So are you or Red Hat doing more on Kubernetes and HPC or was this just kind of a one-off experiment? For me it was a one-off experiment because of that email that came in, like, you know, they wanted to do it in VMs and I said, well, you probably don't need to do it in VMs. You could probably just do it directly with Kubernetes. You know, Red Hat is not going to suddenly become an open-foam company, right? Like, that wasn't the purpose of this experiment at all. It was really mostly, like, what does it look like to use the MPI operator? Red Hat is invested in the Kubeflow project community for other reasons around AIML. So there happened to be people that I could talk to easily to fuss around with this thing. But, yeah, in terms of, like, any kind of strategic roadmap-y thing, like, it's not this specifically. It's more generally, like, we are exploring how to do HPC in a Kubernetes world and what we could maybe productize, you know, or just what we can support, who we can partner with, you know, and so on and so forth. Of course, we're doing a lot of great work with IBM in this arena, but there's a lot of other companies and projects that are out there that sort of play in this space as well. Okay, and I was going to do a follow-up. We got time. Okay. When you were running in AWS, were you using their MPI drivers? No. So, open-foam and open MPI and Kubeflow don't require any kind of special driver-y bits. It was really just, I did a vanilla open-gift installation on AWS. You probably could have also used EKS on AWS. I used the Open Data Foundation because I needed RWX. You probably could have used EFS. You know, you just need some central RWX storage that everybody can read and write to. And the only reason I needed RWX was because the launcher did the initial writing and decomposition of the job and then wrote that out to the shared storage, and then all the workers needed to access it and then write their stuff back. So like, if I had done it in a pipeline-y thing, I might have actually been able to get away with some other way to do it, but again, I just didn't explore that deeply. But yeah, there was nothing special about the way I was using AWS or what I was doing on AWS.  I just didn't feel like figuring out how to use it. This was easier. We have a system. I can order a cluster. It's like... Anyway, yes, sir. Question for context. So what is Morland's workflow right now then if they're not using something like this? What are they doing to get there? He has a rack of old Dell 1RUs. I think he's got like six or seven of them or whatever. That he was using again. And it was mostly like, I wasn't trying to get them to go Kubernetes or anything like that. It was more like, well, I know somebody that uses this tool, so I can at least poke them if I run into a jam. But yeah, their existing workload today is just 100% bare metal, no cube, no containers. It's just like a blunt do install directly on the servers and then open foam installed in the OS. Well, no, I mean, it's using MPI. So he's got a head server. I mean, it looks a lot like that in a way. Like there's a head server where he runs the preprocessing stuff. And then that's done on a NFS share. The head server is also the NFS server. And then they do MPI run, which then goes out to the other more powerful servers. And then they are pulling from the NFS share and doing the actual processing. And then the reconstruct happens at the end. And then it spits out the cool image. Hi, great talk. I was curious with regard to, and I'm not super familiar with open foam. Do you have to, how do you handle the scheduling of all the, yeah, so all the, all the parts on the, on the right hand side, do all of these need to be scheduled in at like one go? Essentially what I'm trying to say is, is this a gang scheduling problem that we are trying to solve here? Yeah. So I'm going to partially answer your question because I don't fully know. So, so take what I'm saying with a grain of salt. The MPI operator in the actual job definition, and I don't have the full job definition, you, you write a pod spec effectively for the launcher and a pod spec effectively for the workers. And you tell it how many workers you want when you run the job or sorry, when you instantiate the job instance of the CR, the operator creates the launcher pod and creates all of the worker pods. And it pretty much expects that they all can run at that time. And I think if it can't run all of the workers, it just sits there and waits. So it doesn't like run one of them ahead of time and wait for it to finish. And then, oh, now I can fit another one and run that on the back.  Like it wants it all to happen at once. I don't know if that's a limitation on the MPI operator. Like if, if there's a way to tell it, like it's okay to run something ahead of time. I just never looked into it any deeper. And when you ran this on a cloud provider AWS in this case, did you have to like pre scale the compute to be able to handle all those parts and then scale it down? So I, what I wanted to do was I actually wanted to do this as a cluster auto scaling job. And so the way open shift is built, and you can do this with, with most of the Kubernetes, um, there's a way to make open shift auto scale itself based on workload demand. And so what I was hoping to do was have like nowhere near enough actual nodes and then run this job. And then when the cluster auto scale are detected that there were pods waiting, it would start jacking up the number of nodes to actually run enough stuff, but I just, I didn't, I didn't get there. Um, so you wouldn't, you wouldn't necessarily have to have that ahead of time. The only caveat I guess is I don't know what the timeout is, if any, on the MPI job operator. And so if it took like a really long time to get enough nodes to then schedule the pods, I don't know if the job would have already failed because it was like, Oh, I didn't get to do it. Nevermind. I give up. Like I'm not sure what the timeout kind of deal is. Yeah, that makes sense. My next question is on, actually you mentioned that multiple pods were running on like one bare metal instance, one VM. Is that accurate? Did you see resource contention when multiple pods were running on the same VM? Yeah. Um, no, I didn't see resource contention because the pod, when you write the pod spec, you specify like a, uh, a request and a limit to do the best, sorry, guaranteed quality of service tier. So for those of you who've ever played with the quality of service tiers and Kubernetes, like you've got requests and you've got limits. If they're the same, it says like, make sure I absolutely have that much to run or don't run me. Um, and then underneath the covers, you know, the Linux kernel and C groups and the scheduler and all that other stuff, make sure that there's no contention. If you wrote the spec for these badly and didn't have a, um, a request and a limit to put you in the quality, uh, the guaranteed quality service tier and you ended up in either best effort or like, uh, whatever the other one is, like you could totally end up with resource contention where it just would throw two pods on the thing and they're both trying to use all the CPU and they both, one of them starves the other one out. So like you definitely want to put limits and requests to make sure you're in the guaranteed QoS tier. Um, otherwise you'll end up with contention. Yeah. But that's like normal. That's not specific to the HPC stuff. That's like generic Kubernetes stuff. Yeah. Just wanted to, uh, confirm. So when it came to the cube flow, it was, uh, providing your MPI operator. So then all the, uh, communication from the launcher pod to the compute pods, uh, was just being solely managed through whatever cube flow provided you. Correct. Uh, at that point it's all, uh, where's the SSH picture? So it's, it's all still SSH between the, um, the launcher and the workers. So the operator ensures that the launcher pod is created and the worker pods are created. Once all of the pods are up and running, the job or the script will run inside the launcher and that MPI run command MPI itself will then look at the host file to find, well, there, it'll look at the MPI host file to find all the targets, but those targets are all just pod names.  Yeah. So that MPI run MP four interphone. So everything after the four is what gets run, um, effectively on each of the workers. Um, so, so the, all the cube flow, all the MPI operator part of cube flow is doing is watching for those. Um, that's not bright enough. It just washes for these and then creates the pods and then executes the command. It's not managing any of the like communication stuff. That all happens effectively over the Kubernetes service layer. Well, over the Kubernetes software to find network, I don't think it's actually even using services because it's talking directly to each pod by host name. I think that's where I was curious where the obfuscation was happening or the, the, yeah, of the traffic. It sounds like, you know, cube, cube flow is providing you the MPI. So if you have questions regarding MPI, you got to look at cube flow source code and then regarding, uh, uh, like in the, the communication outside of that SSH or, or otherwise, um, you're looking more back to Kubernetes itself. Uh, yeah, sort of, I think, again, most of the communication is coming from what, what is happening in the MPI run commands from a like network or storage communication perspective, the, the SSH connections from the launcher to the workers is just over the local SDN, whatever network you've configured for your Kubernetes, it doesn't have to be a software to find network. Some people who do it on bare metal, they don't use an SDN. Um, and then of course you've got the storage communication, which in this case is, is network communication, um, between the workers and the, the storage that contains the little job bits and pieces. Um, from a tuning perspective, like open from is pretty good about the storagey bit because basically all the workers, like they, they load it up into memory once and then they do their thing and then they spit it out. They're not constantly talking to the storage, but if you have a different type of MPI workload that is constantly rewriting to storage, then of course that storage network is going to be important to ensure that it's got, you know, high performance, um, uh, characteristics or that the disks have high performance or both. And then I haven't had a chance to look at your blog posts, which I look forward to, but, um, when it came to open foam, I take it, you probably just pulled, uh, like a Docker file from somewhere or pulled the container. Yeah. I think I had to rebuild their image to make like a mine. Oh, I had to rebuild their image to change the SSH configuration to make some of that stuff work because the, the open, they provide a container for you to do stuff in the open foam project, but the container that they provide is like super hardlock to one specific UID and like behaves weirdly if you don't use that. And so I had to like mess with the bash RC to make it load the right path, these stuff to find all the tools and everything. It's a pretty minimal change, but basically, yes, I'm just using their, their container image and all anything I had to do from a container image perspective, it's all linked in the blog post and it's in a repo. I think there's a couple repos that have the different bits and pieces. Okay. Thank you. I have not, I know we have customers that do, do some of those more advanced network storage things. I don't even know that I have access to infinity. Somebody I read had somewhere in a lab has infinite band that I could try. But the reality is that like that's kind of invisible to the environment. It would be more of a like the beauty of Kubernetes into a certain extent storage and the kernel and all the other things is like that stuff shouldn't kind of matter. Like maybe I would have to define a PVC that exists as a, as a, as an end point on that infinite band storage, right? I don't know. Is infinite band like ice cozy-ish or is it more of a fiber channely or none of the above all of the above. Anyway, TLDR. Did you hear any of that? So you may have to attach an infinite band devices directly into the pods, which is, which is something that you can do. There's ways to attach devices into pods. It would probably make the, the wording of the, the job and like the pod definitions for the word launcher and the other things. It would make that kind of start to get hairy because you got to like be adding storage definitions to everything. I had to do that anyway to make these connect to the RWX. But if you also had to add devices to the pods, like it would, it would just get trickier. Yeah. And I don't know if open foams commit like tools speak infinite band. Yeah. All Kubernetes is doing is, is effectively, if you think about the, the path of how pods get launched, like Kubernetes tells a kubelet on a node to run a pod and then under the covers cryo and run C and all these other things are being instructed on how to bring up that container. And in that process that includes like, you need to attach this stuff from the host into this container at this point. So in the case of infinite band device, you know, it would, like if you think about a pod man or a Docker run command, it would be like, you know, attach slash dev slash whatever into the container at slash dev slash whatever. So there's no, there's no knowledge required of Kubernetes of infinite band, unless you're talking about like some CSI infinite band driver thing, but that would be baked into the CSI driver for infinite band. Anyway, so back to the original question. I did not try infinite band. I might be able to find some somewhere, but I don't know that, that open foam would know what to do with it. Even, even if I did, it does. Okay. Well then there you go. So you alluded to a couple of race conditions and the MPI bootstrapping process. Can you speak a little bit about those and whether you address those? No. So you said it was 512 ranks. Is that correct? What did I do? So I did 256 pods with two CPUs per pod. So that ended up being a total of 512 CPUs. And I think it ended up being, I don't know how many that ends up being per node. I'll make up a number. It was like, presumably it was one rank per CPU. One what? One rank per CPU. One rank? MPI rank.  I mean the MPI run command was pretty vanilla. And the host file that gets generated is also super vanilla. Yeah. So there's no, there's no fancy MPI stuff going on. The reason I ask is that we ran into bootstrapping problems with about 80 ranks or so. So we'd see race conditions where, whereby the workers would report ready. Would, sorry. The workers, the MPI launcher would start before all the workers were ready sometimes, which we caught a crash in the bootstrap. And then the other thing that would happen sometimes is the workers would start and would report ready before SSH was ready, which then would cause the MPI bootstrap to fail again. Okay. Yeah. Yeah. Yeah.  So MPI comes up and it's tries to do its thing. And it's like, I couldn't find half of these machines. I don't know what to do. And it crash loops and then kind of backs off. So I don't know that they could probably do things to make it smarter. One, number two, I'm not sure if you can define liveness and readiness probes in the template spec for the, for the job. I mean, it should support the full pod spec, but the challenge is like you'd have to write a really nasty liveness readiness probe for the worker to like SSH into itself. You can, you can do a post dark hook where you just like check to make sure the ports, port 22 is open. For example, yeah, I mean, you could do, you could do things. It's just, I never, I never got to that point. Fortunately, the, the pre-processing, and this is why doing it is a tecton pipeline could be beneficial because all the pre-processing steps would get done. And then the only thing that's failing is the MPI run. If you look at this, like blog mesh set fields decompose, they're happy to run multiple times while it fails because ultimately it does the first three things, then runs MPI run, and that's when it fails. So if you have other steps that come before the actual MPI that are not, I'll use the word idempotent, which is probably the wrong word to use here, but we'll, we'll go with it for now. Like if you have things that would blow up if they got run twice, this is going to be bad, bad news bears, right? Okay, we're getting a, we're getting a hook. Thanks very much. If you have other questions, I'm happy to answer them outside. Again, the presentation is uploaded to the portal. You can find me pretty much on lots of places on the internet. I appreciate it. Have a great trip home, travel home safe.


Senator Ted Cruz is back in Texas, admitting his whirlwind trip to Cancun was a mistake. When a nice storm comes rolling in, and folks might soon be shivering, you could be in the office governing, helping out your constituents. Or you could go down to Mexico, wait a minute, sip a drink on the sunny shore. You seem good at your job on Twitter, so you can go down to Mexico. We're gathered here for an impeachment, the president's lacking innocence. So we owe it to all the citizens to move forward with all the witnesses. Next week is recess. Or we could go down to Mexico. It's Valentine's weekend, don't you know? I look good at my job on Twitter, so perhaps I'll head down to Mexico. As a pandemic starts to rage, and we make a vaccine hurray. Here's a question for determining if the people you elect to lead are maybe not that great at governing. Do people prefer this to your currency? Increasingly. Then maybe go down to Mexico, the sunny shore. It's time to go to Mexico.


I'm Major Bashiru Bala, commanding officer of the Nigerian Army, 192 Battalion, in Wode Yewa Ogul State Express Concerns over the proliferation of arms and ammunition along the border with the Ben Air Republic. He spoke at a West African Social Activities event where a truck loaded with 263,300 rounds of Starr cartridges and 2,500 live cartridges were intercepted. Bala emphasised the need for collaboration with other security agencies and emphasised the importance of strengthening social relationships among officers and soldiers. May I assure you that the unit is ever ready and prepared to continue ensuring that maximum security is provided within its area of responsibility. Additionally, I want to use this opportunity to reaffirm our commitment to sustain the genocinatistic relationship with other collaborating security agencies, especially the lead agency for anti-smuggling operations, that is the Nigerian Custom Service, in checkmating transborder crimes and smuggling of contraband within our own area of responsibility. For the period under review, the unit recorded a lot of operational services, which include the interception of a truck loaded with 263,300 quantities of rounds and then 2,005 live cartridges. A guest of honour, Major-General, praise the President, Bala Tinibur and Amistad Chief for providing strategic leadership for the armed forces. The unit has evidence that from all the very entertaining displays we have witnessed today, this auspicious event has lead on to its desired expectations. At this juncture, I wish to express with utmost humility, my deepest gratitude to the President, Commander-in-Chief of the Armed Forces of the Federal Republic of Nigeria, His Excellency, President Bala Amitibur, Grand Commander of the Honor of the Federal Republic for the strategic guidance to the Armed Forces of Nigeria. Also to the Chief of Amistad, Lieutenant-General T.K. Labajan for his leadership, guidance and on whether support both in human capacity development and infrastructural development of the Nigerian Army, especially in 192 Battalion Rea, awarding as well as ensuring the preservation of the long-aid tradition of the Nigerian Army social activity. Hello, hope you enjoyed the news. Please do subscribe to our YouTube channel and don't forget to hit the notification button so you get notified about fresh news updates.